Understanding hybrid search
Pinecone supports vectors with sparse and dense values, which allows you to perform hybrid search on your Pinecone index. Hybrid search combines semantic and keyword search in one query for more relevant results. Semantic search results for outofdomain queries can be less relevant; combining these with keyword search results can improve relevance. This topic describes how hybrid search with sparsedense vectors works in Pinecone.
To see sparsedense embeddings in action, see the Ecommerce hybrid search example.
This feature is in public preview. Consider the current limitations and considerations for serverless indexes, and test thoroughly before using it in production.
Hybrid search in Pinecone
In Pinecone, you perform hybrid search with sparsedense vectors. Sparsedense vectors combine dense and sparse embeddings as a single vector. Sparse and dense vectors represent different types of information and enable distinct kinds of search.
Dense vectors
The basic vector type in Pinecone is a dense vector. Dense vectors enable semantic search. Semantic search returns the most similar results according to a specific distance metric even if no exact matches are present. This is possible because dense vectors generated by embedding models such as SBERT are numerical representations of semantic meaning.
Sparse vectors
Sparse vectors have very large number of dimensions, where only a small proportion of values are nonzero. When used for keywords search, each sparse vector represents a document; the dimensions represent words from a dictionary, and the values represent the importance of these words in the document. Keyword search algorithms like the BM25 algorithm compute the relevance of text documents based on the number of keyword matches, their frequency, and other factors.
Sparsedense workflow
Using sparsedense vectors involves the following general steps:
 Create dense vectors using an external embedding model.
 Create sparse vectors using an external model.
 Create an index with the
dotproduct
metric.  Upsert sparsedense vectors to your index.
 Search the index using sparsedense vectors.
 Pinecone returns sparsedense vectors.
Considerations for serverless indexes
The implementation of hybrid search is meaningfully different between podbased and serverless indexes. If you switch from one to the other, you may experience a regression in accuracy.
When you query a serverless index, the dense value of the query is used to retrieve the initial candidate records, and then the sparse value is considered when returning the final results. This process can affect accuracy in cases where records contain dense and sparse values that are from different data representations (e.g., image and text) or are otherwise not strongly correlated.
For more details about query execution, see Serverless architecture.
Limitations
Pinecone sparsedense vectors have the following limitations:

Records with sparse vector values must also contain dense vector values.

Sparse vector values can contain up to 1000 nonzero values and 4.2 billion dimensions.

Only indexes using the dotproduct distance metric support sparsedense vectors.
Upserting sparsedense vectors into indexes with a different distance metric will succeed, but querying will return an error.

Indexes created before February 22, 2023 do not support sparse vectors.
See also
Was this page helpful?