This page shows you how to upsert records into a namespace in an index. Namespaces let you partition records within an index and are essential for implementing multitenancy when you need to isolate the data of each customer/user.

If a record ID already exists, upserting overwrites the entire record. To change only part of a record, update the record.

To control costs when ingesting very large datasets (10,000,000+ records), import from data storage instead of upserting.

​
Upsert dense vectors

Upserting text is supported only for indexes with integrated embedding.

To upsert source text into a dense index, use the upsert_records operation. Pinecone converts the text to dense vectors automatically using the hosted dense embedding model associated with the index.

  • Specify the namespace to upsert into. If the namespace doesn’t exist, it is created. To use the default namespace, set the namespace to an empty string ("").
  • Format your input data as records, each with the following:
    • An _id field with a unique record identifier for the index namespace. id can be used as an alias for _id.
    • A field with the source text to convert to a vector. This field must match the field_map specified in the index.
    • Additional fields will be stored as record metadata and can be returned in search results or used to filter search results.

For example, the following code converts the sentences in the chunk_text fields to dense vectors and then upserts them into example-namespace in example-index. The additional category field is stored as metadata.

from pinecone import Pinecone

pc = Pinecone(api_key="YOUR_API_KEY")

# To get the unique host for an index, 
# see https://docs.pinecone.io/guides/data/target-an-index
index = pc.Index(host="INDEX_HOST")

# Upsert records into a namespace
# `chunk_text` fields are converted to dense vectors
# `category` fields are stored as metadata
index.upsert_records(
    "example-namespace",
    [
        {
            "_id": "rec1",
            "chunk_text": "Apples are a great source of dietary fiber, which supports digestion and helps maintain a healthy gut.",
            "category": "digestive system", 
        },
        {
            "_id": "rec2",
            "chunk_text": "Apples originated in Central Asia and have been cultivated for thousands of years, with over 7,500 varieties available today.",
            "category": "cultivation",
        },
        {
            "_id": "rec3",
            "chunk_text": "Rich in vitamin C and other antioxidants, apples contribute to immune health and may reduce the risk of chronic diseases.",
            "category": "immune system",
        },
        {
            "_id": "rec4",
            "chunk_text": "The high fiber content in apples can also help regulate blood sugar levels, making them a favorable snack for people with diabetes.",
            "category": "endocrine system",
        },
    ]
)

​
Upsert sparse vectors

This feature is in public preview.

It can take up to 35 seconds before upserted records are available to query.

Upserting text is supported only for indexes with integrated embedding.

To upsert source text into a sparse index, use the upsert_records operation. Pinecone converts the text to sparse vectors automatically using the hosted sparse embedding model associated with the index.

  • Specify the namespace to upsert into. If the namespace doesn’t exist, it is created. To use the default namespace, set the namespace to an empty string ("").
  • Format your input data as records, each with the following:
    • An _id field with a unique record identifier for the index namespace. id can be used as an alias for _id.
    • A field with the source text to convert to a vector. This field must match the field_map specified in the index.
    • Additional fields will be stored as record metadata and can be returned in search results or used to filter search results.

For example, the following code converts the sentences in the chunk_text fields to sparse vectors and then upserts them into example-namespace in example-index. The additional category and quarter fields are stored as metadata.

from pinecone import Pinecone

pc = Pinecone(api_key="YOUR_API_KEY")

# To get the unique host for an index, 
# see https://docs.pinecone.io/guides/data/target-an-index
index = pc.Index(host="INDEX_HOST")

# Upsert records into a namespace
# `chunk_text` fields are converted to sparse vectors
# `category` and `quarter` fields are stored as metadata
index.upsert_records(
    "example-namespace",
    [
        { 
            "_id": "vec1", 
            "chunk_text": "AAPL reported a year-over-year revenue increase, expecting stronger Q3 demand for its flagship phones.", 
            "category": "technology",
            "quarter": "Q3"
        },
        { 
            "_id": "vec2", 
            "chunk_text": "Analysts suggest that AAPL'\''s upcoming Q4 product launch event might solidify its position in the premium smartphone market.", 
            "category": "technology",
            "quarter": "Q4"
        },
        { 
            "_id": "vec3", 
            "chunk_text": "AAPL'\''s strategic Q3 partnerships with semiconductor suppliers could mitigate component risks and stabilize iPhone production.",
            "category": "technology",
            "quarter": "Q3"
        },
        { 
            "_id": "vec4", 
            "chunk_text": "AAPL may consider healthcare integrations in Q4 to compete with tech rivals entering the consumer wellness space.", 
            "category": "technology",
            "quarter": "Q4"
        }
    ]
)

time.sleep(10) # Wait for the upserted vectors to be indexed

​
Upsert in batches

When upserting larger amounts of data, it is recommended to upsert records in large batches. This should be as large as possible (up to 1000 records) without exceeding the maximum request size of 2MB. To understand the number of records you can fit into one batch, see the Upsert limits section.

To control costs when ingesting very large datasets (10,000,000+ records), use import instead of upsert.

import random
import itertools
from pinecone.grpc import PineconeGRPC as Pinecone

pc = Pinecone(api_key="YOUR_API_KEY")

# To get the unique host for an index, 
# see https://docs.pinecone.io/guides/data/target-an-index
index = pc.Index(host="INDEX_HOST")

def chunks(iterable, batch_size=200):
    """A helper function to break an iterable into chunks of size batch_size."""
    it = iter(iterable)
    chunk = tuple(itertools.islice(it, batch_size))
    while chunk:
        yield chunk
        chunk = tuple(itertools.islice(it, batch_size))

vector_dim = 128
vector_count = 10000

# Example generator that generates many (id, vector) pairs
example_data_generator = map(lambda i: (f'id-{i}', [random.random() for _ in range(vector_dim)]), range(vector_count))

# Upsert data with 200 vectors per upsert request
for ids_vectors_chunk in chunks(example_data_generator, batch_size=200):
    index.upsert(vectors=ids_vectors_chunk)

​
Upsert in parallel

Python SDK v6.0.0 and later provide async methods for use with asyncio. Asyncio support makes it possible to use Pinecone with modern async web frameworks such as FastAPI, Quart, and Sanic. For more details, see Asyncio support.

Send multiple upserts in parallel to help increase throughput. Vector operations block until the response has been received. However, they can be made asynchronously as follows:

# This example uses `async_req=True` and multiple threads.
# For a single-threaded approach compatible with modern async web frameworks, 
# see https://docs.pinecone.io/reference/python-sdk#asyncio-support
import random
import itertools
from pinecone import Pinecone

# Initialize the client with pool_threads=30. This limits simultaneous requests to 30.
pc = Pinecone(api_key="YOUR_API_KEY", pool_threads=30)

# To get the unique host for an index, 
# see https://docs.pinecone.io/guides/data/target-an-index
index = pc.Index(host="INDEX_HOST")

def chunks(iterable, batch_size=200):
    """A helper function to break an iterable into chunks of size batch_size."""
    it = iter(iterable)
    chunk = tuple(itertools.islice(it, batch_size))
    while chunk:
        yield chunk
        chunk = tuple(itertools.islice(it, batch_size))

vector_dim = 128
vector_count = 10000

example_data_generator = map(lambda i: (f'id-{i}', [random.random() for _ in range(vector_dim)]), range(vector_count))

# Upsert data with 200 vectors per upsert request asynchronously
# - Pass async_req=True to index.upsert()
with pc.Index(host="INDEX_HOST", pool_threads=30) as index:
    # Send requests in parallel
    async_results = [
        index.upsert(vectors=ids_vectors_chunk, async_req=True)
        for ids_vectors_chunk in chunks(example_data_generator, batch_size=200)
    ]
    # Wait for and retrieve responses (this raises in case of error)
    [async_result.get() for async_result in async_results]

​
Python SDK with gRPC

Using the Python SDK with gRPC extras can provide higher upsert speeds. Through multiplexing, gRPC is able to handle large amounts of requests in parallel without slowing down the rest of the system (HoL blocking), unlike REST. Moreover, you can pass various retry strategies to the gRPC SDK, including exponential backoffs.

To install the gRPC version of the SDK:

Shell
pip install "pinecone[grpc]"

To use the gRPC SDK, import the pinecone.grpc subpackage and target an index as usual:

Python
from pinecone.grpc import PineconeGRPC as Pinecone

# This is gRPC client aliased as "Pinecone"
pc = Pinecone(api_key='YOUR_API_KEY')  

# To get the unique host for an index, 
# see https://docs.pinecone.io/guides/data/target-an-index
index = pc.Index(host="INDEX_HOST")

To launch multiple read and write requests in parallel, pass async_req to the upsert operation:

Python
def chunker(seq, batch_size):
  return (seq[pos:pos + batch_size] for pos in range(0, len(seq), batch_size))

async_results = [
  index.upsert(vectors=chunk, async_req=True)
  for chunk in chunker(data, batch_size=200)
]

# Wait for and retrieve responses (in case of error)
[async_result.result() for async_result in async_results]

It is possible to get write-throttled faster when upserting using the gRPC SDK. If you see this often, we recommend you use a backoff algorithm(e.g., exponential backoffs)
while upserting.

The syntax for upsert, query, fetch, and delete with the gRPC SDK remain the same as the standard SDK.

​
Upsert a dataset as a dataframe

To quickly ingest data when using the Python SDK, use the upsert_from_dataframe method. The method includes retry logic andbatch_size, and is performant especially with Parquet file data sets.

The following example upserts the uora_all-MiniLM-L6-bm25 dataset as a dataframe.

Python
from pinecone import Pinecone, ServerlessSpec
from pinecone_datasets import list_datasets, load_dataset

pc = Pinecone(api_key="API_KEY")

dataset = load_dataset("quora_all-MiniLM-L6-bm25")

pc.create_index(
  name="example-index",
  dimension=384,
  metric="cosine",
  spec=ServerlessSpec(
    cloud="aws",
    region="us-east-1"
  )
)

# To get the unique host for an index, 
# see https://docs.pinecone.io/guides/data/target-an-index
index = pc.Index(host="INDEX_HOST")

index.upsert_from_dataframe(dataset.drop(columns=["blob"]))

​
Upsert limits

MetricLimit
Max upsert size2MB or 1000 records
Max metadata size per record40 KB
Max length for a record ID512 characters
Max dimensionality for dense vectors20,000
Max non-zero values for sparse vectors1000
Max dimensionality for sparse vectors4.2 billion

When upserting larger amounts of data, it is recommended to upsert records in large batches. A batch of upserts should be as large as possible (up to 1000 records) without exceeding the maximum request size of 2MB.

To understand the number of records you can fit into one batch based on the vector dimensions and metadata size, see the following table:

DimensionMetadata (bytes)Max batch size
38601000
768500559
15362000245

​
Data freshness

Pinecone is eventually consistent, so there can be a slight delay before new or changed records are visible to queries. You can view index stats to check data freshness.

Was this page helpful?

Query sparse-dense vectorsQuery data