Understanding data freshness

Pinecone is eventually consistent, so there can be a slight delay before new or changed records are visible to queries. After adding, updating, or deleting records, you can check whether your index contains the required record by checking the log sequence number or verifying the number of records in the index.

​

Log sequence numbers

When you make a write request to a serverless index namespace, Pinecone assigns a monotonically increasing log sequence number (LSN) to the write operation. The LSN reflects upserts as well as updates and deletes to that namespace. Writes to one namespace do not increase the LSN for other namespaces.

​

LSN in response headers

Every time you modify records in your namespace, the HTTP response contains the LSN for the upsert. This is contained in a header called x-pinecone-request-lsn.

Every time you query your index, the HTTP response contains the LSN for the query. This is contained in a header called x-pinecone-max-indexed-lsn.

If the LSN contained in the query response header is greater than or equal to the LSN of the relevant write operation, then that operation is reflected in the query response. If the LSN contained in the query response header is greater than the LSN of the relevant write operation, then subsequent operations are also reflected in the query response.

To check the LSN in your HTTP responses using the Database API, see Check data freshness.

​

Record counts

If you insert new records or delete records, the number of record in the index may change. This means that the record count for an index can indicate whether Pinecone has indexed your latest inserts and deletes: if the record count for the index matches the count you expect after inserting or deleting records, the index is probably up-to-date.

However, this is not always true: for example, if you delete the same number of records that you insert, the expected record count may remain the same.

To verify that your index contains the number of record you expect, use the describe_index_stats operation to check if the current record count matches the number of records you expect.

Some write operations, such as updates to an index configuration or vector data values, do not change the number of records in the index. This means that you cannot use record counts to confirm that Pinecone has indexed these operations.

If you update records or insert and delete the same number of records, total record count does not change.

​

Replicas

Multiple replicas of pod-based indexes may not all become consistent at the same time. A describe_index_stats call to an index with multiple replicas immediately after performing an upsert can return different record counts depending on which replica issues the response. For this reason, you cannot rely on validating the record count from describe_index to determine the freshness of an index with multiple replicas.