Manage document chunks

When you split a long source document into chunks, each chunk is stored as its own document (in a document index) or record (in a vector index). To work with all of a source document’s chunks at once, give each chunk a structured ID prefixed with the source document’s ID, and a document_id metadata field. Each task shows a document index first, then a vector index. The document-index code assumes a chunk_text field declared with embed for integrated embedding, so Pinecone generates the vectors, and requires API version 2026-07 (Python SDK 10.1.0 or later). The vector-index code assumes you generate the vectors yourself.

Pinecone is eventually consistent, so a read right after a write (upsert, update, or delete) may not return the latest data. If you need to read immediately after writing, add a short delay or retry logic.

Upsert chunks

Upsert each chunk with its structured ID and document_id, plus any other metadata you want to filter on or display. On a document index with integrated embedding, each upsert is capped at 96 documents instead of 1,000:

Document index
Vector index

Python

from pinecone import Pinecone

pc = Pinecone(api_key="YOUR_API_KEY")

# To get the unique host for an index,
# see https://docs.pinecone.io/guides/manage-data/target-an-index
index = pc.Index(host="INDEX_HOST")

# Pinecone embeds `chunk_text` on write.
# The other fields are stored as metadata.
index.documents.upsert(
    namespace="example-namespace",
    documents=[
        {
            "_id": "document1#chunk1",
            "chunk_text": "First chunk of the document content...",
            "document_id": "document1",
            "document_title": "Introduction to Vector Databases",
            "chunk_number": 1,
            "document_url": "https://example.com/docs/document1",
            "created_at": "2024-01-15",
            "document_type": "tutorial"
        },
        {
            "_id": "document1#chunk2",
            "chunk_text": "Second chunk of the document content...",
            "document_id": "document1",
            "document_title": "Introduction to Vector Databases",
            "chunk_number": 2,
            "document_url": "https://example.com/docs/document1",
            "created_at": "2024-01-15",
            "document_type": "tutorial"
        },
        {
            "_id": "document1#chunk3",
            "chunk_text": "Third chunk of the document content...",
            "document_id": "document1",
            "document_title": "Introduction to Vector Databases",
            "chunk_number": 3,
            "document_url": "https://example.com/docs/document1",
            "created_at": "2024-01-15",
            "document_type": "tutorial"
        },
    ],
)

Python

from pinecone.grpc import PineconeGRPC as Pinecone

pc = Pinecone(api_key="YOUR_API_KEY")

# To get the unique host for an index,
# see https://docs.pinecone.io/guides/manage-data/target-an-index
index = pc.Index(host="INDEX_HOST")

index.upsert(
  namespace="example-namespace",
  vectors=[
    {
      "id": "document1#chunk1", 
      "values": [0.0236663818359375, -0.032989501953125, ..., -0.01041412353515625, 0.0086669921875], 
      "metadata": {
        "document_id": "document1",
        "document_title": "Introduction to Vector Databases",
        "chunk_number": 1,
        "chunk_text": "First chunk of the document content...",
        "document_url": "https://example.com/docs/document1",
        "created_at": "2024-01-15",
        "document_type": "tutorial"
      }
    },
    {
      "id": "document1#chunk2", 
      "values": [-0.0412445068359375, 0.028839111328125, ..., 0.01953125, -0.0174560546875],
      "metadata": {
        "document_id": "document1",
        "document_title": "Introduction to Vector Databases", 
        "chunk_number": 2,
        "chunk_text": "Second chunk of the document content...",
        "document_url": "https://example.com/docs/document1",
        "created_at": "2024-01-15",
        "document_type": "tutorial"
      }
    },
    {
      "id": "document1#chunk3", 
      "values": [0.0512237548828125, 0.041656494140625, ..., 0.02130126953125, -0.0394287109375],
      "metadata": {
        "document_id": "document1",
        "document_title": "Introduction to Vector Databases",
        "chunk_number": 3, 
        "chunk_text": "Third chunk of the document content...",
        "document_url": "https://example.com/docs/document1",
        "created_at": "2024-01-15",
        "document_type": "tutorial"
      }
    }
  ]
)

Search within a document

To search only one source document’s chunks, add a metadata filter on document_id. On a document index, you search with query text and Pinecone embeds it:

Document index
Vector index

Python

from pinecone import Pinecone

pc = Pinecone(api_key="YOUR_API_KEY")

# To get the unique host for an index,
# see https://docs.pinecone.io/guides/manage-data/target-an-index
index = pc.Index(host="INDEX_HOST")

filtered_results = index.documents.search(
    namespace="example-namespace",
    top_k=3,
    score_by=[
        {"type": "embed", "fields": ["chunk_text"], "query": "What is a vector database?"}
    ],
    filter={"document_id": {"$eq": "document1"}},
    include_fields=["chunk_text"],
)

for match in filtered_results.matches:
    print(match._id, match._score, match.chunk_text)

Python

from pinecone.grpc import PineconeGRPC as Pinecone

pc = Pinecone(api_key="YOUR_API_KEY")

# To get the unique host for an index,
# see https://docs.pinecone.io/guides/manage-data/target-an-index
index = pc.Index(host="INDEX_HOST")

filtered_results = index.query(
    namespace="example-namespace",
    vector=[0.0236663818359375,-0.032989501953125, ..., -0.01041412353515625,0.0086669921875], 
    top_k=3,
    filter={
        "document_id": {"$eq": "document1"}
    },
    include_metadata=True,
    include_values=False
)

print(filtered_results)

Fetch all chunks of a document

To retrieve all of a source document’s chunks, list their IDs by prefix, and then fetch them. On a document index, you can also fetch them by passing a filter such as {"document_id": {"$eq": "document1"}} to documents.fetch, which returns up to 100 matches per page by default (set limit for up to 10,000):

Document index
Vector index

Python

from pinecone import Pinecone

pc = Pinecone(api_key="YOUR_API_KEY")

# To get the unique host for an index,
# see https://docs.pinecone.io/guides/manage-data/target-an-index
index = pc.Index(host="INDEX_HOST")

# List all chunks for document1 using ID prefix
chunk_ids = []
for doc in index.documents.list(namespace="example-namespace", prefix="document1#"):
    chunk_ids.append(doc._id)

print(f"Found {len(chunk_ids)} chunks for document1")

# Fetch the complete documents by ID
if chunk_ids:
    response = index.documents.fetch(
        namespace="example-namespace",
        ids=chunk_ids,
        include_fields=["chunk_text", "chunk_number"],
    )

    for doc_id, doc in response.documents.items():
        print(f"Chunk ID: {doc_id}")
        print(f"Chunk text: {doc.chunk_text}")

Python

from pinecone.grpc import PineconeGRPC as Pinecone

pc = Pinecone(api_key="YOUR_API_KEY")

# To get the unique host for an index,
# see https://docs.pinecone.io/guides/manage-data/target-an-index
index = pc.Index(host="INDEX_HOST")

# List all chunks for document1 using ID prefix.
# Each iteration yields one page of IDs.
chunk_ids = []
for page in index.list(prefix='document1#', namespace='example-namespace'):
    chunk_ids.extend(item.id for item in page.vectors)

print(f"Found {len(chunk_ids)} chunks for document1")

# Fetch the complete records by ID
if chunk_ids:
    records = index.fetch(ids=chunk_ids, namespace='example-namespace')
    
    for record_id, record_data in records['vectors'].items():
        print(f"Chunk ID: {record_id}")
        print(f"Chunk text: {record_data['metadata']['chunk_text']}")
        # Process the vector values and metadata as needed

Delete chunks

To delete some or all of a source document’s chunks, filter on their metadata:

Document index
Vector index

Python

from pinecone import Pinecone

pc = Pinecone(api_key="YOUR_API_KEY")

# To get the unique host for an index,
# see https://docs.pinecone.io/guides/manage-data/target-an-index
index = pc.Index(host="INDEX_HOST")

# Delete chunks 1 and 3
index.documents.delete(
    namespace="example-namespace",
    filter={
        "document_id": {"$eq": "document1"},
        "chunk_number": {"$in": [1, 3]}
    }
)

# Delete all chunks for a document
index.documents.delete(
    namespace="example-namespace",
    filter={
        "document_id": {"$eq": "document1"}
    }
)

Python

from pinecone.grpc import PineconeGRPC as Pinecone

pc = Pinecone(api_key="YOUR_API_KEY")

# To get the unique host for an index,
# see https://docs.pinecone.io/guides/manage-data/target-an-index
index = pc.Index(host="INDEX_HOST")

# Delete chunks 1 and 3
index.delete(
    namespace="example-namespace",
    filter={
        "document_id": {"$eq": "document1"},
        "chunk_number": {"$in": [1, 3]}
    }
)

# Delete all chunks for a document
index.delete(
    namespace="example-namespace",
    filter={
        "document_id": {"$eq": "document1"}
    }
)

Replace a chunked document

When the number or order of a source document’s chunks changes, delete all of its chunks with a metadata filter, and then upsert the new ones. To change one chunk’s content without changing the chunk count, update it by ID instead.

To update most of the data in a large namespace, contact Support about a faster, cheaper approach based on an export.

Document index
Vector index

Python

from pinecone import Pinecone

pc = Pinecone(api_key="YOUR_API_KEY")

# To get the unique host for an index,
# see https://docs.pinecone.io/guides/manage-data/target-an-index
index = pc.Index(host="INDEX_HOST")

# Step 1: Delete all existing chunks for the document
index.documents.delete(
    namespace="example-namespace",
    filter={
        "document_id": {"$eq": "document1"}
    }
)

print("Deleted existing chunks for document1")

# Step 2: Upsert the updated document chunks
index.documents.upsert(
    namespace="example-namespace",
    documents=[
        {
            "_id": "document1#chunk1",
            "chunk_text": "Updated first chunk with new content...",
            "document_id": "document1",
            "document_title": "Introduction to Vector Databases - Updated Edition",
            "chunk_number": 1,
            "document_url": "https://example.com/docs/document1",
            "created_at": "2024-02-15",
            "document_type": "tutorial",
            "version": "2.0"
        },
        {
            "_id": "document1#chunk2",
            "chunk_text": "Updated second chunk with new content...",
            "document_id": "document1",
            "document_title": "Introduction to Vector Databases - Updated Edition",
            "chunk_number": 2,
            "document_url": "https://example.com/docs/document1",
            "created_at": "2024-02-15",
            "document_type": "tutorial",
            "version": "2.0"
        },
        # Add more chunks as needed for the updated document
    ],
)

print("Successfully updated document1 with new chunks")

Python

from pinecone.grpc import PineconeGRPC as Pinecone

pc = Pinecone(api_key="YOUR_API_KEY")

# To get the unique host for an index,
# see https://docs.pinecone.io/guides/manage-data/target-an-index
index = pc.Index(host="INDEX_HOST")

# Step 1: Delete all existing chunks for the document
index.delete(
    namespace="example-namespace",
    filter={
        "document_id": {"$eq": "document1"}
    }
)

print("Deleted existing chunks for document1")

# Step 2: Upsert the updated document chunks
chunk1_vector = ...  # from your embedding model
chunk2_vector = ...
index.upsert(
  namespace="example-namespace", 
  vectors=[
    {
      "id": "document1#chunk1",
      "values": chunk1_vector,
      "metadata": {
        "document_id": "document1",
        "document_title": "Introduction to Vector Databases - Updated Edition",
        "chunk_number": 1,
        "chunk_text": "Updated first chunk with new content...",
        "document_url": "https://example.com/docs/document1",
        "created_at": "2024-02-15",
        "document_type": "tutorial",
        "version": "2.0"
      }
    },
    {
      "id": "document1#chunk2",
      "values": chunk2_vector,
      "metadata": {
        "document_id": "document1",
        "document_title": "Introduction to Vector Databases - Updated Edition",
        "chunk_number": 2,
        "chunk_text": "Updated second chunk with new content...",
        "document_url": "https://example.com/docs/document1",
        "created_at": "2024-02-15",
        "document_type": "tutorial",
        "version": "2.0"
      }
    }
    # Add more chunks as needed for the updated document
  ]
)

print("Successfully updated document1 with new chunks")

Get started

Index data

Search

Optimize

Manage data

Manage cost

Move to production

Agents

Pod-based indexes (legacy)

Upsert chunks

Search within a document

Fetch all chunks of a document

Delete chunks

Replace a chunked document

​Upsert chunks

​Search within a document

​Fetch all chunks of a document

​Delete chunks

​Replace a chunked document

Upsert chunks

Search within a document

Fetch all chunks of a document

Delete chunks

Replace a chunked document