Manage pod-based indexes

This page shows you how to manage pod-based indexes.

For guidance on serverless indexes, see Manage serverless indexes.

​

Describe a pod-based index

Use the describe_index endpoint to get a complete description of a specific index:

from pinecone.grpc import PineconeGRPC as Pinecone

pc = Pinecone(api_key="YOUR_API_KEY")

pc.describe_index(name="example-index")

# Response:
# {'dimension': 1536,
#  'host': 'example-index-4mkljsz.svc.aped-4627-b74a.pinecone.io',
#  'metric': 'cosine',
#  'name': 'example-index',
#  'spec': {'pod': {'environment': 'us-east-1-aws',
#                   'pod_type': 's1.x1',
#                   'pods': 1,
#                   'replicas': 1,
#                   'shards': 1}},
#  'status': {'ready': True, 'state': 'Ready'}}

​

Delete a pod-based index

Use the delete_index operation to delete a pod-based index and all of its associated resources.

# pip install pinecone[grpc]
from pinecone.grpc import PineconeGRPC as Pinecone, PodSpec

pc = Pinecone(api_key="YOUR_API_KEY")

pc.delete_index(name="example-index")

If deletion protection is enabled on an index, requests to delete it will fail and return a 403 - FORBIDDEN status with the following error:

Deletion protection is enabled for this index. Disable deletion protection before retrying.

Before you can delete such an index, you must first disable deletion protection.

​

Selective metadata indexing

For pod-based indexes, Pinecone indexes all metadata fields by default. When metadata fields contains many unique values, pod-based indexes will consume significantly more memory, which can lead to performance issues, pod fullness, and a reduction in the number of possible vectors that fit per pod.

To avoid indexing high-cardinality metadata that is not needed for filtering your queries and keep memory utilization low, specify which metadata fields to index using the metadata_config parameter.

The value for the metadata_config parameter is a JSON object containing the names of the metadata fields to index.

{
    "indexed": [
        "metadata-field-1",
        "metadata-field-2",
        "metadata-field-n"
    ]
}

Example

The following example creates a pod-based index that only indexes the genre metadata field. Queries against this index that filter for the genre metadata field may return results; queries that filter for other metadata fields behave as though those fields do not exist.

from pinecone.grpc import PineconeGRPC as Pinecone, PodSpec

pc = Pinecone(api_key="YOUR_API_KEY")

pc.create_index(
  name="example-index",
  dimension=1536,
  metric="cosine",
  spec=PodSpec(
    environment="us-west1-gcp",
    pod_type="p1.x1",
    pods=1,
    metadata_config = {
      "indexed": ["genre"]
    }
  ),
  deletion_protection="disabled"

)

​

Prevent index deletion

You can prevent an index and its data from accidental deleting when creating a new index or when configuring an existing index. In both cases, you set the deletion_protection parameter to enabled.

To enable deletion protection when creating a new index:

# pip install pinecone[grpc]
from pinecone.grpc import PineconeGRPC as Pinecone, PodSpec

pc = Pinecone(api_key="YOUR_API_KEY")

pc.create_index(
  name="example-index",
  dimension=1536,
  metric="cosine",
  spec=PodSpec(
    environment="us-west1-gcp",
    pod_type="p1.x1",
    pods=1
  ),
    deletion_protection="enabled"
)

To enable deletion protection when configuring an existing index:

from pinecone.grpc import PineconeGRPC as Pinecone

pc = Pinecone(api_key="YOUR_API_KEY")

pc.configure_index(
   name="example-index", 
   deletion_protection="enabled"
)

When deletion protection is enabled on an index, requests to delete the index fail and return a 403 - FORBIDDEN status with the following error:

Deletion protection is enabled for this index. Disable deletion protection before retrying.

​

Disable deletion protection

Before you can delete an index with deletion protection enabled, you must first disable deletion protection as follows:

from pinecone.grpc import PineconeGRPC as Pinecone

pc = Pinecone(api_key="YOUR_API_KEY")

pc.configure_index(
   name="example-index", 
   deletion_protection="disabled"
)

​

Delete an entire namespace

In pod-based indexes, reads and writes share compute resources, so deleting an entire namespace with many records can increase the latency of read operations. In such cases, consider deleting records in batches.

​

Delete records in batches

In pod-based indexes, reads and writes share compute resources, so deleting an entire namespace or a large number of records can increase the latency of read operations. To avoid this, delete records in batches of up to 1000, with a brief sleep between requests. Consider using smaller batches if the index has active read traffic.

from pinecone import Pinecone
import numpy as np
import time

pc = Pinecone(api_key='API_KEY')

INDEX_NAME = 'INDEX_NAME'
NAMESPACE = 'NAMESPACE_NAME'
# Consider using smaller batches if you have a high RPS for read operations
BATCH = 1000

index = pc.Index(name=INDEX_NAME)
dimensions = index.describe_index_stats()['dimension']

# Create the query vector
query_vector = np.random.uniform(-1, 1, size=dimensions).tolist()
results = index.query(vector=query_vector, namespace=NAMESPACE, top_k=BATCH)

# Delete in batches until the query returns no results
while len(results['matches']) > 0:
    ids = [i['id'] for i in results['matches']]
    index.delete(ids=ids, namespace=NAMESPACE)
    time.sleep(0.01)
    results = index.query(vector=query_vector, namespace=NAMESPACE, top_k=BATCH)

​

Delete records by metadata

To delete records based on their metadata, pass a metadata filter expression to the delete operation. This deletes all vectors matching the metadata filter expression.

For example, to delete all vectors with genre “documentary” and year 2019 from an index, use the following code:

from pinecone.grpc import PineconeGRPC as Pinecone

pc = Pinecone(api_key="YOUR_API_KEY")

# To get the unique host for an index, 
# see https://docs.pinecone.io/guides/data/target-an-index
index = pc.Index(host="INDEX_HOST")

index.delete(
    filter={
        "genre": {"$eq": "documentary"}
    }
)

​

Tag an index

When configuring an index, you can tag the index to help with index organization and management. For more details, see Tag an index.

​

Troubleshoot index fullness errors

Serverless indexes automatically scale as needed.

However, pod-based indexes can run out of capacity. When that happens, upserting new records will fail with the following error:

Index is full, cannot accept data.

While a full pod-based index can still serve queries, you need to scale your index to accommodate more records.