Key features
While Pinecone Database optimizes vector search for efficiency and accuracy, you can enhance it further with the features below.
Namespaces
Use namespaces to divide records within an index into separate groups. This has a few main benefits:
-
Multitenancy: When you need to isolate data between customers, use one namespace per customer and target each customer’s writes and queries to their dedicated namespace.
-
Faster queries: Every query targets one namespace in an index. When you divide records into namespaces in a logical way, you speed up queries by ensuring only relevant records are scanned. The same applies to fetching records, listing record IDs, and other data operations.
Learn more:
Metadata filtering
Every record in an index must contain an ID and a dense or sparse vector, depending on the type of index. In addition, you can include metadata key-value pairs to store related information or context. When you search the index, you can then include a metadata filter to limit the search to records matching a filter expression.
For example, if an index contains records about books, you could use a metadata field to associate each record with a genre, like "genre": "fiction"
or "genre": "poetry"
. When you query the index, you could then use a metadata filter to limit your search to records related to a specific genre.
Learn more:
Data ingestion
When loading large numbers of records into an index, consider the following methods:
-
Importing from object storage is the most efficient and cost-effective way to ingest large numbers of records. Store your data as Parquet files in object storage, integrate your object storage with Pinecone, and then start an asynchronous, long-running operation that imports and indexes your records.
-
Upserting in batches is another efficient way to ingest large numbers of records (up to 1000 per batch). Batch upserting is also a good option if you cannot work around bulk import’s current limitations.
Learn more:
Vector embedding
Dense vectors and sparse vectors are the basic units of data in Pinecone and what Pinecone was specially designed to store and work with. Dense vectors represents the semantics of data such as text, images, and audio recordings, while sparse vectors represent documents or queries in a way that captures keyword information.
To transform data into vector format, you use an embedding model. Pinecone hosts several embedding models so it’s easy to manage your vector storage and search process on a single platform, but there are many other embedding models available.
Learn more:
Reranking
Reranking is used as part of a two-stage vector retrieval process to improve the quality of results. You first query an index for a given number of relevant results, and then you send the query and results to a reranking model. The reranking model scores the results based on their semantic relevance to the query and returns a new, more accurate ranking. This approach is one the simplest methods for improving quality in retrieval augmented generation (RAG) pipelines.
Pinecone hosts several reranking models so it’s easy to manage two-stage vector retrieval on a single platform, but there are many other reranking models available.
Learn more:
Was this page helpful?