Check data freshness

Pinecone is eventually consistent, so there can be a slight delay before new or changed records are visible to queries. This page describes two ways of checking the data freshness of a Pinecone index:

• To check if a serverless index queries reflect recent writes to the index, check the log sequence number.

• To check whether an index contains recently inserted or deleted vectors, verify the number of vectors in the index.

​

Check the log sequence number

​

Log sequence numbers

When you make a write request to a serverless index namespace, Pinecone assigns a monotonically increasing log sequence number (LSN) to the write operation. The LSN reflects upserts as well as updates and deletes to that namespace. Writes to one namespace do not increase the LSN for other namespaces.

You can use LSNs to verify that specific write operations are reflected in your query responses. If the LSN contained in the query response header is greater than or equal to the LSN of the relevant write operation, then that operation is reflected in the query response. If the LSN contained in the query response header is greater than the LSN of the relevant write operation, then subsequent operations are also reflected in the query response.

Follow the steps below to compare the LSNs for a write and a subsequent query.

​

1. Get the LSN for a write operation

Every time you modify records in your namespace, the HTTP response contains the LSN for the upsert. This is contained in a header called x-pinecone-request-lsn.

The following example demonstrates how to get the LSN for an upsert request using the curl option -i. This option tells curl to include headers in the displayed response. Use the same method to get the LSN for an update or delete request.

PINECONE_API_KEY="YOUR_API_KEY"
INDEX_HOST="INDEX_HOST"

curl -i "https://$INDEX_HOST/vectors/upsert" \
  -H "Api-Key: $PINECONE_API_KEY" \
  -H "content-type: application/json" \
  -H "X-Pinecone-API-Version: 2025-01" \
  -d '{
        "vectors": [
          {
            "id": "vec1",
            "values": [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8],
          }
        ],
        "namespace": "example-namespace"
      }'

The preceding request receives a response like the following example:

HTTP/2 200 
date: Wed, 21 Aug 2024 15:23:04 GMT
content-type: application/json
content-length: 66
x-pinecone-max-indexed-lsn: 4
x-pinecone-request-latency-ms: 1149
x-pinecone-request-id: 3687967458925971419
x-envoy-upstream-service-time: 1150
grpc-status: 0
server: envoy

{"upsertedCount":1}

In the preceding example response, the value of x-pinecone-max-indexed-lsn is 4. This means that the index has performed 4 write operations since its creation.

​

2. Get the LSN for a query

Every time you query your index, the HTTP response contains the LSN for the query. This is contained in a header called x-pinecone-max-indexed-lsn.

By checking the LSN in your query results, you can confirm that the LSN is greater than or equal to the LSN of the relevant write operation, indicating that the results of that operation are present in the query results.

The following example makes a query request to the index:

PINECONE_API_KEY="YOUR_API_KEY"
INDEX_HOST="INDEX_HOST"

curl -i "https://$INDEX_HOST/query" \
  -H "Api-Key: $PINECONE_API_KEY" \
  -H 'Content-Type: application/json' \
  -H "X-Pinecone-API-Version: 2025-01" \
  -d '{
    "vector": [0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3],
    "namespace": "example-namespace",
    "topK": 3,
    "includeValues": true
  }'

The preceding request receives a response like the following example:

HTTP/2 200 
date: Wed, 21 Aug 2024 15:33:36 GMT
content-type: application/json
content-length: 66
x-pinecone-max-indexed-lsn: 5
x-pinecone-request-latency-ms: 40
x-pinecone-request-id: 6683088825552978933
x-envoy-upstream-service-time: 41
grpc-status: 0
server: envoy

{
  "results":[],
  "matches":[
    {
      "id":"vec1",
      "score":0.891132772,
      "values":[0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8],
    }
  ],
  "namespace":"example-namespace",
  "usage":{"readUnits":6}
}

In the preceding example response, the value of x-pinecone-max-indexed-lsn is 5.

​

3. Compare LSNs for writes and queries

If the LSN of a query is greater than or equal to the LSN for a write operation, then the results of the query reflect the results of the write operation.

In step 1, the LSN contained in the response headers is 4.

In step 2, the LSN contained in the response headers is 5.

5 is greater than or equal to 4; therefore, the results of the query reflect the results of the upsert. However, this does not guarantee that the records upserted are still present or unmodified: the write operation with LSN of 5 may have updated or deleted these records, or upserted additional records.

​

Verify record counts

If you insert new records or delete records, the number of records in the index may change. This means that the record count for an index can indicate whether Pinecone has indexed your latest inserts and deletes: if the record count for the index matches the count you expect after inserting or deleting records, the index is probably up-to-date. However, this is not always true. For example, if you delete the same number of records that you insert, the expected record count may remain the same. Also, some write operations, such as updates to an index configuration or vector data values, do not change the number of records in the index.

To verify that your index contains the number of records you expect, view index stats:

# pip install "pinecone[grpc]"
from pinecone.grpc import PineconeGRPC as Pinecone

pc = Pinecone(api_key="YOUR_API_KEY")

# To get the unique host for an index, 
# see https://docs.pinecone.io/guides/data/target-an-index
index = pc.Index(host="INDEX_HOST")

index.describe_index_stats()

The response will look like this:

{'dimension': 1024,
 'index_fullness': 8e-05,
 'namespaces': {'example-namespace1': {'vector_count': 4}, 'example-namespace2': {'vector_count': 4}},
 'total_vector_count': 8}