Manage RAG documents

In Retrieval Augmented Generation (RAG) use cases, it is best practice to chunk large documents into smaller segments, embed each chunk separately, and then store each embedded chunk as a distinct record in Pinecone. This page shows you how to model, store, and manage such records in serverless indexes.

Use ID prefixes to reference parent documents

When you have multiple records representing chunks of a single document, use a common ID prefix to reference the document.

ℹ️

Note

You can use any prefix pattern you like, but make sure you use a consistent prefix pattern for all child records of a document. For example, doc1#chunk1, doc1_chunk1, doc1_chunk1, doc1___chunk1, doc1:chunk1, and doc1chunk1 are all valid prefixes for the first chunk of doc1. Prefixes can also be multi-level. For example, doc1#v1#chunk1 and doc1#v2#chunk1 could represent different versions of the first chunk of doc1.

from pinecone import Pinecone, ServerlessSpec

pc = Pinecone(api_key="YOUR_API_KEY")

pc.create_index(
  name="serverless-index",
  dimension=8,
  metric="cosine",
  spec=ServerlessSpec(
    cloud="aws",
    region="us-west-2"
  )
)

index = pc.Index("serverless-index")

index.upsert(
  vectors=[
    {"id": "doc1#chunk1", "values": [0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1]},
    {"id": "doc1#chunk2", "values": [0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2]},
    {"id": "doc1#chunk3", "values": [0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3]},
    {"id": "doc1#chunk4", "values": [0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4]}
  ],
  namespace="ns1"
)
import { Pinecone } from '@pinecone-database/pinecone'

const pc = new Pinecone({
  apiKey: 'YOUR_API_KEY'
});

await pc.createIndex({
  name: 'severless-index',
  dimension: 1536,
  metric: 'cosine',
  spec: {
    serverless: {
      cloud: 'aws',
      region: 'us-west-2'
    }
  }
});

const index = pc.index("serverless-index")

await index.namespace("ns1").upsert([
  {
    "id": "doc1#chunk1", 
    "values": [0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1]
  },
  {
    "id": "doc1#chunk2", 
    "values": [0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2]
  },
  {
    "id": "doc1#chunk3", 
    "values": [0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3]
  },
  {
    "id": "doc1#chunk4", 
    "values": [0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4]
  }
]);
PINECONE_API_KEY="YOUR_API_KEY"

curl -s -X POST "https://api.pinecone.io/indexes" \
  -H "Accept: application/json" \
  -H "Content-Type: application/json" \
  -H "Api-Key: $PINECONE_API_KEY" \
  -d '{
         "name": "serverless-index",
         "dimension": 1536,
         "metric": "cosine",
         "spec": {
            "serverless": {
               "cloud": "aws",
               "region": "us-west-2"
            }
         }
      }'

# The `GET` request below uses the unique endpoint for an index.
# See https://docs.pinecone.io/docs/get-index-endpoint for details.
INDEX_HOST="INDEX_HOST"

curl -X GET "https://$INDEX_HOST/vectors/upsert" \
  -H "Accept: application/json" \
  -H "Content-Type: application/json" \
  -H "Api-Key: $PINECONE_API_KEY" \
  -d '{
    "vectors": [
      {
        "id": "doc1#chunk1", 
        "values": [0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1]
      },
      {
        "id": "doc1#chunk2", 
        "values": [0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2]
      },
      {
        "id": "doc1#chunk3", 
        "values": [0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3]
      },
      {
        "id": "doc1#chunk4", 
        "values": [0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4]
      }
    ],
    "namespace": "ns1"
  }'

List all record IDs for a parent document

When all records related to a document use a common ID prefix, you can use the list operation, with the namespace and prefix parameters, to fetch the IDs of the records.

ℹ️

Note

The list operation is available only via the REST API at this time.

# The `GET` request below uses the unique endpoint for an index.
# See https://docs.pinecone.io/docs/get-index-endpoint for details.
PINECONE_API_KEY="YOUR_API_KEY"
INDEX_HOST="INDEX_HOST"

curl -X GET "https://$INDEX_HOST/vectors/list?namespace=ns1&prefix=doc1#" \
  -H "Api-Key: $PINECONE_API_KEY"

# Response:
# {
#   "vectors": [
#     { "id": "doc1#chunk1" },
#     { "id": "doc1#chunk2" },
#     { "id": "doc1#chunk3" },
#     { "id": "doc1#chunk4" },
    ...
#   ],
#   "pagination": {
#     "next": "c2Vjb25kY2FsbA=="
#   },
#   "namespace": "ns1",
#   "usage": {
#     "readUnits": 1
#   }
# }

When there are additional IDs to return, the response includes a pagination_token that you can use to get the next batch of IDs. For more details, see Paginate through list results

With the record IDs, you can then use the fetch operation to fetch the content of the records.

Delete all records for a parent document

To delete all records representing chunks of a single document, first list the record IDs based on their common ID prefix:

ℹ️

Note

The list operation is supported only for serverless indexes and only via the REST API.

# The `GET` request below uses the unique endpoint for an index.
# See https://docs.pinecone.io/docs/get-index-endpoint for details.
PINECONE_API_KEY="YOUR_API_KEY"
INDEX_HOST="INDEX_HOST"

curl -X GET "https://$INDEX_HOST/vectors/list?namespace=ns1&prefix=doc1#" \
  -H "Api-Key: $PINECONE_API_KEY"

# Response:
# {
#   "vectors": [
#     { "id": "doc1#chunk1" },
#     { "id": "doc1#chunk2" },
#     { "id": "doc1#chunk3" },
#     { "id": "doc1#chunk4" },
    ...
#   ],
#   "pagination": {
#     "next": "c2Vjb25kY2FsbA=="
#   },
#   "namespace": "ns1",
#   "usage": {
#     "readUnits": 1
#   }
# }

Then delete the records by ID:

curl -X POST "https://$INDEX_HOST/vectors/delete" \
  -H "Api-Key: $PINECONE_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "ids": [
      "doc1#chunk1", 
      "doc1#chunk2", 
      "doc1#chunk3", 
      "doc1#chunk4"
    ],
    "namespace": "ns1"
  }
'

Work with multi-level ID prefixes

The examples above are based on a simple ID prefix (doc1#), but it's also possible to work with more complex, multi-level prefixes.

For example, let's say you use the prefix pattern document#version#chunk to differentiate between different versions of a document. If you wanted to delete all records for one version of a document, you would first list the record IDs based on the relevant document#version# prefix:

# The `GET` request below uses the unique endpoint for an index.
# See https://docs.pinecone.io/docs/get-index-endpoint for details.
PINECONE_API_KEY="YOUR_API_KEY"
INDEX_HOST="INDEX_HOST"

curl -X GET "https://$INDEX_HOST/vectors/list?namespace=ns1&prefix=doc1%23v1%23" \
  -H "Api-Key: $PINECONE_API_KEY"

# Response:
# {
#   "vectors": [
#     { "id": "doc1#v1#chunk1" },
#     { "id": "doc1#v1#chunk2" },
#     { "id": "doc1#v1#chunk3" },
#     { "id": "doc1#v1#chunk4" },
    ...
#   ],
#   "pagination": {
#     "next": "c2Vjb25kY2FsbA=="
#   },
#   "namespace": "ns1",
#   "usage": {
#     "readUnits": 1
#   }
# }

You'd then delete the records by ID

However, if you wanted to delete all records across all versions of a document, you would list the record IDs based on the doc1# part of the prefix that is common to all versions:

# The `GET` request below uses the unique endpoint for an index.
# See https://docs.pinecone.io/docs/get-index-endpoint for details.
PINECONE_API_KEY="YOUR_API_KEY"
INDEX_HOST="INDEX_HOST"

curl -X GET "https://$INDEX_HOST/vectors/list?namespace=ns1&prefix=doc1%23" \
  -H "Api-Key: $PINECONE_API_KEY"

# Response:
# {
#   "vectors": [
#     { "id": "doc1#v1#chunk1" },
#     { "id": "doc1#v1#chunk2" },
#     { "id": "doc1#v1#chunk3" },
#     { "id": "doc1#v1#chunk4" },
#   ...
#     { "id": "doc1#v2#chunk1" },
#     { "id": "doc1#v2#chunk2" },
#     { "id": "doc1#v2#chunk3" },
#     { "id": "doc1#v2#chunk4" },
#   ...
#   ],
#   "pagination": {
#     "next": "c2Vjb25kY2FsbA=="
#   },
#   "namespace": "ns1",
#   "usage": {
#     "readUnits": 1
#   }
# }

You'd then delete the records by ID.

RAG using pod-based indexes

The list operation does not support pod-based indexes. Instead of using ID prefixes to reference parent documents, use a metadata key-value pair. If you later need to delete the records, you can pass a metadata filter expression to the delete operation.