Understanding hybrid search

Overview

Pinecone supports vectors with sparse and dense values, which allows you to perform hybrid search on your Pinecone index. Hybrid search combines semantic and keyword search in one query for more relevant results. Semantic search results for out-of-domain queries can be less relevant; combining these with keyword search results can improve relevance. This topic describes how hybrid search with sparse-dense vectors works in Pinecone.

To see sparse-dense embeddings in action, see the Ecommerce hybrid search example.

Hybrid search in Pinecone

In Pinecone, you perform hybrid search with sparse-dense vectors. Sparse-dense vectors combine dense and sparse embeddings as a single vector. Sparse and dense vectors represent different types of information and enable distinct kinds of search.

Dense vectors

The basic vector type in Pinecone is a dense vector. Dense vectors enable semantic search. Semantic search returns the most similar results according to a specific distance metric even if no exact matches are present. This is possible because dense vectors generated by embedding models such as SBERT are numerical representations of semantic meaning.

Sparse vectors

Sparse vectors have very large number of dimensions, where only a small proportion of values are non-zero. When used for keywords search, each sparse vector represents a document; the dimensions represent words from a dictionary, and the values represent the importance of these words in the document. Keyword search algorithms like the BM25 algorithm compute the relevance of text documents based on the number of keyword matches, their frequency, and other factors.

Sparse-dense workflow

Using sparse-dense vectors involves the following general steps:

Create dense vectors using an external embedding model.
Create sparse vectors using an external model.
Create an index that supports sparse-dense vectors (s1 or p1 with the dotproduct metric).
Upsert dense and sparse vectors to your index.
Search the index using sparse-dense vectors.
Pinecone returns sparse-dense vectors.

Creating sparse vector embeddings

Keyword-aware semantic search requires vector representations of documents. Because Pinecone indexes accept sparse indexes rather than documents, you can control the generation of sparse vectors to represent documents.

Because Pinecone allows you to create your own sparse vectors, you can use sparse-dense queries to solve the Maximum Inner Product Search (MIPS) problem for sparse-dense vectors of any real values. This includes emerging use-cases such as retrieval over learnt sparse representations for text data using SPLADE.

For examples of sparse vector generation, see SPLADE for Sparse Vector Search Explained, our SPLADE generation notebook, and our BM25 generation notebook.

Creating sparse-dense vectors

In Pinecone, each vector consists of dense vector values and, optionally, sparse vector values as well. When you upsert records with sparse and dense vector values, Pinecone creates sparse-dense vectors from your sparse and dense embeddings. Pinecone does not support vectors with only sparse values. See Upserting sparse-dense records.

Querying sparse-dense vectors

To query your sparse-dense vectors, you provide a query vector containing both sparse and dense values. Pinecone ranks vectors in your index by considering the full dot product over the entire vector; the score of a vector is the sum of the dot product of its dense values with the dense part of the query, together with the dot product of its sparse values with the sparse part of the query. You may want to weight the sparse or dense vector component more heavily.

Limitations

Pinecone sparse-dense vectors have the following limitations:

Pinecone supports sparse vector values of sizes up to 1000 non-zero values.
Pinecone only supports upserting sparse-dense vectors to p1 and s1 indexes.
In order to query an index using sparse values, the index must use the dotproduct metric. Attempting to query any other index with sparse values returns an error.
Indexes created before February 22, 2023 do not support sparse values.

Next steps

Ecommerce hybrid search example

Upserting sparse-dense records