Understanding indexes

An index is the highest-level organizational unit of vector data in Pinecone. It stores vectors, serves queries over the vectors it contains, and does other vector operations over its contents.

​

Index types

There are two types of serverless indexes, dense and sparse.

​

Dense indexes

Dense indexes store dense vectors, which are a series of numbers that represent the meaning and relationships of text, images, or other types of data. Each number in a dense vector corresponds to a point in a multidimensional space. Vectors that are closer together in that space are semantically similar.

When you query a dense index, Pinecone retrieves the dense vectors that are the most semantically similar to the query. This is often called semantic search, nearest neighbor search, similarity search, or just vector search.

Learn more:

• Create a dense index
• Upsert dense vectors
• Query a dense index

​

Sparse indexes

Sparse indexes store sparse vectors, which are a series of numbers that represent the words or phrases in a document. Sparse vectors have a very large number of dimensions, where only a small proportion of values are non-zero. The dimensions represent words from a dictionary, and the values represent the importance of these words in the document.

When you search a sparse index, Pinecone retrieves the sparse vectors that most exactly match the words or phrases in the query. Query terms are scored independently and then summed, with the most similar records scored highest. This is often called lexical search or keyword search.

Learn more:

• Create a sparse index
• Upsert sparse vectors
• Query a sparse index

​

Limitations

Sparse indexes have the following limitations:

• Max sparse records per namespace: 100,000,000

• Max non-zero values per sparse vector: 1000

• Max upserts per second per sparse index: 10

• Max queries per second per sparse index: 100

• Max top_k value per query: 1000

  You may get fewer than top_k results if top_k is larger than the number of sparse vectors in your index that match your query. That is, any vectors where the dotproduct score is 0 will be discarded.

• Max query results size: 4MB

• Limited performance with high cardinality metadata. Better metadata indexing is coming soon.

​

Namespaces

Within an index, records are partitioned into namespaces, and all upserts, queries, and other data operations always target one namespace. This has two main benefits:

• Multitenancy: When you need to isolate data between customers, you can use one namespace per customer and target each customer’s writes and queries to their dedicated namespace. See Implement multitenancy for end-to-end guidance.

• Faster queries: When you divide records into namespaces in a logical way, you speed up queries by ensuring only relevant records are scanned. The same applies to fetching records, listing record IDs, and other data operations.

Namespaces are created automatically during upsert. If a namespace doesn’t exist, it is created implicitly.

​

Vector embedding

You use an embedding model to convert data to vectors. You can either use Pinecone’s integrated embedding capabilities to convert your source data to vectors automatically, or you can use an external embedding model and bring your own vectors to Pinecone.

​

Use integrated embedding

1. Create an index that is integrated with one of Pinecone’s hosted embedding models.
2. Upsert your source text. Pinecone uses the integrated model to convert the text to vectors automatically.
3. Search with a query text. Again, Pinecone uses the integrated model to convert the text to a vector automatically.

​

Bring your own vectors

1. Use an external embedding model to convert your text to vectors.
2. Create an index that matches the characteristics of the model.
3. Upsert your vectors directly.
4. Use the same external embedding model to convert a query to a vector.
5. Search with your query vector directly.

​

Cloud regions

When creating an index, you must choose the cloud and region where you want the index to be hosted. The following table lists the available public clouds and regions and the plans that support them:

Cloud	Region	Supported plans	Availability phase
aws	us-east-1 (Virginia)	Starter, Standard, Enterprise	General availability
aws	us-west-2 (Oregon)	Standard, Enterprise	General availability
aws	eu-west-1 (Ireland)	Standard, Enterprise	General availability
gcp	us-central1 (Iowa)	Standard, Enterprise	General availability
gcp	europe-west4 (Netherlands)	Standard, Enterprise	General availability
azure	eastus2 (Virginia)	Standard, Enterprise	General availability

The cloud and region cannot be changed after a serverless index is created.

​

Similarity metrics

When creating a dense index, you can choose from the following similarity metrics. For the most accurate results, choose the similarity metric used to train the embedding model for your vectors. For more information, see Vector Similarity Explained.

​

euclidean

Querying indexes with this metric returns a similarity score equal to the squared Euclidean distance between the result and query vectors.

This metric calculates the square of the distance between two data points in a plane. It is one of the most commonly used distance metrics. For an example, see our IT threat detection example.

When you use metric='euclidean', the most similar results are those with the lowest similarity score.

​

cosine

This is often used to find similarities between different documents. The advantage is that the scores are normalized to [-1,1] range. For an example, see our generative question answering example.

​

dotproduct

This is used to multiply two vectors. You can use it to tell us how similar the two vectors are. The more positive the answer is, the closer the two vectors are in terms of their directions. For an example, see our semantic search example.