Check data freshness

This topic describes two ways of checking the data freshness of your Pinecone index:

• To check if your serverless index queries reflect recent writes to your index, check the log sequence number.

• To check whether your index contains recently inserted or deleted vectors, verify the number of vectors in the index.

​

Check the log sequence number

You can use log sequence numbers (LSNs) to verify that specific write operations are reflected in your query responses.

Follow the steps below to compare the LSNs for a write and a subsequent query.

To learn more about LSNs, see Understanding data freshness.

​

1. Get the LSN for a write operation

The following example demonstrates how to get the LSN for an upsert request using the Pinecone API. Use the same method to get the LSN for an update or delete request.

The example shows an upsert request using the curl option -i. This option tells curl to include headers in the displayed response.

PINECONE_API_KEY="YOUR_API_KEY"
INDEX_HOST="INDEX_HOST"

curl -i "https://$INDEX_HOST/vectors/upsert" \
  -H "Api-Key: $PINECONE_API_KEY" \
  -H "content-type: application/json" \
  -H "X-Pinecone-API-Version: 2024-07" \
  -d '{
        "vectors": [
          {
            "id": "vec1",
            "values": [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8],
          }
        ],
        "namespace": "example-namespace"
      }'

The preceding request receives a response like the following example:

HTTP/2 200 
date: Wed, 21 Aug 2024 15:23:04 GMT
content-type: application/json
content-length: 66
x-pinecone-max-indexed-lsn: 4
x-pinecone-request-latency-ms: 1149
x-pinecone-request-id: 3687967458925971419
x-envoy-upstream-service-time: 1150
grpc-status: 0
server: envoy

{"upsertedCount":1}

In the preceding example response, the value of x-pinecone-max-indexed-lsn is 4. This means that the index has performed 4 write operations since its creation.

​

2. Get the LSN for a query

By checking the LSN in your query results, you can confirm that the LSN is greater than or equal to the LSN of the relevant write operation, indicating that the results of that operation are present in the query results.

The following example makes a query request to the index:

PINECONE_API_KEY="YOUR_API_KEY"
INDEX_HOST="INDEX_HOST"

curl -i "https://$INDEX_HOST/query" \
  -H "Api-Key: $PINECONE_API_KEY" \
  -H 'Content-Type: application/json' \
  -H "X-Pinecone-API-Version: 2024-07" \
  -d '{
    "vector": [0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3],
    "namespace": "example-namespace",
    "topK": 3,
    "includeValues": true
  }'

The preceding request receives a response like the following example:

HTTP/2 200 
date: Wed, 21 Aug 2024 15:33:36 GMT
content-type: application/json
content-length: 66
x-pinecone-max-indexed-lsn: 5
x-pinecone-request-latency-ms: 40
x-pinecone-request-id: 6683088825552978933
x-envoy-upstream-service-time: 41
grpc-status: 0
server: envoy

{
  "results":[],
  "matches":[
    {
      "id":"vec1",
      "score":0.891132772,
      "values":[0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8],
    }
  ],
  "namespace":"example-namespace",
  "usage":{"readUnits":6}
}

In the preceding example response, the value of x-pinecone-max-indexed-lsn is 5.

​

3. Compare LSNs for writes and queries

If the LSN of a query is greater than or equal to the LSN for a write operation, then the results of the query reflect the results of the write operation.

In step 1, the LSN contained in the response headers is 4.

In step 2, the LSN contained in the response headers is 5.

5 is greater than or equal to 4; therefore, the results of the query reflect the results of the upsert. However, this does not guarantee that the records upserted are still present or unmodified: the write operation with LSN of 5 may have updated or deleted these records, or upserted additional records.

​

Verify record counts

To verify that your index contains the number of records you expect, use the describe_index_stats operation to check if the current record count matches the number of records you expect.

Use describe_index_stats to retrieve the current total_vector_count for your index, as in the following example:

The response will look like this: