# Glossary Source: https://docs.pinecone.io/guides/get-started/glossary This page defines concepts in Pinecone and how they relate to each other. ## Organization A organization is a group of one or more [projects](#project) that use the same billing. Organizations allow one or more [users](#user) to control billing and permissions for all of the projects belonging to the organization. For more information, see [Understanding organizations](/guides/organizations/understanding-organizations). ## Project A project belongs to an [organization](#organization) and contains one or more [indexes](#index). Each project belongs to exactly one organization, but only [users](#user) who belong to the project can access the indexes in that project. [API keys](#api-key) and [Assistants](#assistant) are project-specific. For more information, see [Understanding projects](/guides/projects/understanding-projects). ## Index There are two types of [serverless indexes](/guides/index-data/indexing-overview), dense and sparse. For details on pod-based indexes, see [Using pods](/guides/indexes/pods/understanding-pod-based-indexes). ### Dense index A dense index stores [dense vectors](#dense-vector), which are a series of numbers that represent the meaning and relationships of text, images, or other types of data. Each number in a dense vector corresponds to a point in a multidimensional space. Vectors that are closer together in that space are semantically similar. When you query a dense index, Pinecone retrieves the dense vectors that are the most semantically similar to the query. This is often called **semantic search**, nearest neighbor search, similarity search, or just vector search. ### Sparse index A sparse index stores [sparse vectors](#sparse-vector), which are a series of numbers that represent the words or phrases in a document. Sparse vectors have a very large number of dimensions, where only a small proportion of values are non-zero. The dimensions represent words from a dictionary, and the values represent the importance of these words in the document. When you search a sparse index, Pinecone retrieves the sparse vectors that most exactly match the words or phrases in the query. Query terms are scored independently and then summed, with the most similar records scored highest. This is often called **lexical search** or **keyword search**. ## Namespace A namespace is a partition within a [dense](#dense-index) or [sparse index](#sparse-index). It divides [records](#record) in an index into separate groups. All [upserts](/guides/index-data/upsert-data), [queries](/guides/search/search-overview), and other [data operations](/reference/api/2024-10/data-plane) always target one namespace: For more information, see [Use namespaces](/guides/index-data/indexing-overview#namespaces). ## Record A record is a basic unit of data and consists of a [record ID](#record-id), a [dense vector](#dense-vector) or a [sparse vector](#sparse-vector) (depending on the type of index), and optional [metadata](#metadata). For more information, see [Upsert data](/guides/index-data/upsert-data). ### Record ID A record ID is a record's unique ID. [Use ID prefixes](/guides/manage-data/manage-document-chunks#use-id-prefixes) to segment your data beyond namespaces. ### Dense vector A dense vector, also referred to as a vector embedding or simply a vector, is a series of numbers that represent the meaning and relationships of data. Each number in a dense vector corresponds to a point in a multidimensional space. Vectors that are closer together in that space are semantically similar. Dense vectors are stored in [dense indexes](#dense-index). You use a dense embedding model to convert data to dense vectors. The embedding model can be external to Pinecone or [hosted on Pinecone infrastructure](/guides/index-data/create-an-index#embedding-models) and integrated with an index. For more information about dense vectors, see [What are vector embeddings?](https://www.pinecone.io/learn/vector-embeddings/). ### Sparse vector A sparse vector, also referred to as a sparse vector embedding, has a large number of dimensions, but only a small proportion of those values are non-zero. Sparse vectors are often used to represent documents or queries in a way that captures keyword information. Each dimension in a sparse vector typically represents a word from a dictionary, and the non-zero values represent the importance of these words in the document. Sparse vectors are store in [sparse indexes](#sparse-index). You use a sparse embedding model to convert data to sparse vectors. The embedding model can be external to Pinecone or [hosted on Pinecone infrastructure](/guides/index-data/create-an-index#embedding-models) and integrated with an index. For more information about sparse vectors, see [Sparse retrieval](https://www.pinecone.io/learn/sparse-retrieval/). ### Metadata Metadata is additional information that can be attached to vector embeddings to provide more context and enable additional [filtering capabilities](/guides/index-data/indexing-overview#metadata). For example, the original text of the embeddings can be stored in the metadata. ## Other concepts Although not represented in the diagram above, Pinecone also contains the following concepts: * [API key](#api-key) * [User](#user) * [Backup or collection](#backup-or-collection) * [Pinecone Inference](#pinecone-inference) ### API key An API key is a unique token that [authenticates](/reference/api/authentication) and authorizes access to the [Pinecone APIs](/reference/api/introduction). API keys are project-specific. ### User A user is a member of organizations and projects. Users are assigned specific roles at the organization and project levels that determine the user's permissions in the [Pinecone console](https://app.pinecone.io). For more information, see [Manage organization members](/guides/organizations/manage-organization-members) and [Manage project members](/guides/projects/manage-project-members). ### Backup or collection A backup is a static copy of a serverless index. Backups only consume storage. They are non-queryable representations of a set of records. You can create a backup from an index, and you can create a new index from that backup. The new index configuration can differ from the original source index: for example, it can have a different name. However, it must have the same number of dimensions and similarity metric as the source index. For more information, see [Understanding backups](/guides/manage-data/backups-overview). ### Pinecone Inference Pinecone Inference is an API service that provides access to [embedding models](/guides/index-data/create-an-index#embedding-models) and [reranking models](/guides/search/rerank-results#reranking-models) hosted on Pinecone's infrastructure. ## Learn more * [Vector database](https://www.pinecone.io/learn/vector-database/) * [Pinecone APIs](/reference/api/introduction) * [Approximate nearest neighbor (ANN) algorithms](https://www.pinecone.io/learn/a-developers-guide-to-ann-algorithms/) * [Retrieval augmented generation (RAG)](https://www.pinecone.io/learn/retrieval-augmented-generation/) * [Image search](https://www.pinecone.io/learn/series/image-search/) * [Tokenization](https://www.pinecone.io/learn/tokenization/) # Pinecone Database Source: https://docs.pinecone.io/guides/get-started/overview Pinecone is the leading vector database for building accurate and performant AI applications at scale in production. Set up a fully managed vector database for high-performance semantic search Create an AI assistant that answers complex questions about your proprietary data ## Inference Leading embedding and reranking models hosted by Pinecone. [Explore all models](/models). State of the art model 1B text embedding model State of the art reranking model for search Sparse vector model for keyword-style search ## Database workflows Use integrated embedding to upsert and search with text and have Pinecone generate vectors automatically. [Create an index](/guides/index-data/create-an-index) that is integrated with one of Pinecone's [hosted embedding models](/guides/index-data/create-an-index#embedding-models). Dense indexes and vectors enable semantic search, while sparse indexes and vectors enable lexical search. [Upsert](/guides/index-data/upsert-data) your source text and have Pinecone convert the text to vectors automatically. [Use namespaces to partition data](/guides/index-data/indexing-overview#namespaces) for faster queries and multitenant isolation between customers. [Search](/guides/search/search-overview) the index with a query text. Again, Pinecone uses the index's integrated model to convert the text to a vector automatically. [Filter by metadata](/guides/search/filter-by-metadata) to limit the scope of your search, [rerank results](/guides/search/rerank-results) to increase search accuracy, or add [lexical search](/guides/search/lexical-search) to capture both semantic understanding and precise keyword matches. If you use an external embedding model to generate vectors, you can upsert and search with vectors directly. Use an external embedding model to convert data into dense or sparse vectors. [Create an index](/guides/index-data/create-an-index) that matches the characteristics of your embedding model. Dense indexes and vectors enable semantic search, while sparse indexes and vectors enable lexical search. [Load your vectors](/guides/index-data/data-ingestion-overview) and metadata into your index using Pinecone's import or upsert feature. [Use namespaces to partition data](/guides/index-data/indexing-overview#namespaces) for faster queries and multitenant isolation between customers. Use an external embedding model to convert a query text to a vector and [search](/guides/search/search-overview) the index with the vector. [Filter by metadata](/guides/search/filter-by-metadata) to limit the scope of your search, [rerank results](/guides/search/rerank-results) to increase search accuracy, or add [lexical search](/guides/search/lexical-search) to capture both semantic understanding and precise keyword matches. ## Start building Comprehensive details about the Pinecone APIs, SDKs, utilities, and architecture. Simplify vector search with integrated embedding & reranking. Hands-on notebooks and sample apps with common AI patterns and tools. Pinecone's growing number of third-party integrations. Resolve common Pinecone issues with our troubleshooting guide. News about features and changes in Pinecone and related tools. # Pinecone Database quickstart Source: https://docs.pinecone.io/guides/get-started/quickstart This guide shows you how to set up and use Pinecone Database for high-performance semantic search. To get started in your browser, use the [Quickstart colab notebook](https://colab.research.google.com/github/pinecone-io/examples/blob/master/docs/pinecone-quickstart.ipynb). To try Pinecone Database locally before creating an account, use [Pinecone Local](/guides/operations/local-development). ## 1. Install an SDK [Pinecone SDKs](/reference/pinecone-sdks) provide convenient programmatic access to the [Pinecone APIs](/reference/api/introduction). Install the SDK for your preferred language: ```shell Python pip install pinecone ``` ```shell JavaScript npm install @pinecone-database/pinecone ``` ```shell Java # Maven io.pinecone pinecone-client 5.0.0 # Gradle implementation "io.pinecone:pinecone-client:5.0.0" ``` ```shell Go go get github.com/pinecone-io/go-pinecone/v3/pinecone ``` ```shell C# # .NET Core CLI dotnet add package Pinecone.Client # NuGet CLI nuget install Pinecone.Client ``` ## 2. Get an API key You need an API key to make calls to your Pinecone project. Create a new API key in the [Pinecone console](https://app.pinecone.io/organizations/-/keys), or use the widget below to generate a key. If you don't have a Pinecone account, the widget will sign you up for the free [Starter plan](https://www.pinecone.io/pricing/).
Your generated API key: ```shell "{{YOUR_API_KEY}}" ``` ## 3. Create an index In Pinecone, there are two types of indexes for storing vector data: [Dense indexes](/guides/index-data/indexing-overview#dense-indexes) store dense vectors for semantic search, and [sparse indexes](/guides/index-data/indexing-overview#sparse-indexes) store sparse vectors for lexical/keyword search. For this quickstart, [create a dense index](/guides/index-data/create-an-index#create-a-dense-index) that is integrated with an [embedding model hosted by Pinecone](/guides/index-data/create-an-index#embedding-models). With integrated models, you upsert and search with text and have Pinecone generate vectors automatically. If you prefer to use external embedding models, see [Bring your own vectors](/guides/index-data/indexing-overview#bring-your-own-vectors). ```python Python # Import the Pinecone library from pinecone import Pinecone # Initialize a Pinecone client with your API key pc = Pinecone(api_key="{{YOUR_API_KEY}}") # Create a dense index with integrated embedding index_name = "quickstart-py" if not pc.has_index(index_name): pc.create_index_for_model( name=index_name, cloud="aws", region="us-east-1", embed={ "model":"llama-text-embed-v2", "field_map":{"text": "chunk_text"} } ) ``` ```javascript JavaScript // Import the Pinecone library import { Pinecone } from '@pinecone-database/pinecone' // Initialize a Pinecone client with your API key const pc = new Pinecone({ apiKey: '{{YOUR_API_KEY}}' }); // Create a dense index with integrated embedding const indexName = 'quickstart-js'; await pc.createIndexForModel({ name: indexName, cloud: 'aws', region: 'us-east-1', embed: { model: 'llama-text-embed-v2', fieldMap: { text: 'chunk_text' }, }, waitUntilReady: true, }); ``` ```java Java import io.pinecone.clients.Index; import io.pinecone.clients.Pinecone; import org.openapitools.db_control.client.ApiException; import org.openapitools.db_control.client.model.CreateIndexForModelRequest; import org.openapitools.db_control.client.model.CreateIndexForModelRequestEmbed; import org.openapitools.db_control.client.model.DeletionProtection; import org.openapitools.db_control.client.model.IndexModel; import org.openapitools.db_data.client.model.SearchRecordsRequestQuery; import org.openapitools.db_data.client.model.SearchRecordsResponse; import io.pinecone.proto.DescribeIndexStatsResponse; import java.util.*; public class Quickstart { public static void main(String[] args) throws ApiException { Pinecone pc = new Pinecone.Builder("YOUR_API_KEY").build(); String indexName = "quickstart-java"; String region = "us-east-1"; HashMap fieldMap = new HashMap<>(); fieldMap.put("text", "chunk_text"); CreateIndexForModelRequestEmbed embed = new CreateIndexForModelRequestEmbed() .model("llama-text-embed-v2") .fieldMap(fieldMap); IndexModel index = pc.createIndexForModel( indexName, CreateIndexForModelRequest.CloudEnum.AWS, region, embed, DeletionProtection.DISABLED, null ); } } ``` ```go Go package main import ( "context" "encoding/json" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "{{YOUR_API_KEY}}", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } indexName := "quickstart-go" index, err := pc.CreateIndexForModel(ctx, &pinecone.CreateIndexForModelRequest{ Name: indexName, Cloud: pinecone.Aws, Region: "us-east-1", Embed: pinecone.CreateIndexForModelEmbed{ Model: "llama-text-embed-v2", FieldMap: map[string]interface{}{"text": "chunk_text"}, }, }) if err != nil { log.Fatalf("Failed to create serverless index: %v", idx.Name) } else { fmt.Printf("Successfully created serverless index: %v", idx.Name) } } // Function to prettify responses func prettifyStruct(obj interface{}) string { bytes, _ := json.MarshalIndent(obj, "", " ") return string(bytes) } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); var indexName = "quickstart-dotnet"; var createIndexRequest = await pinecone.CreateIndexForModelAsync( new CreateIndexForModelRequest { Name = indexName, Cloud = CreateIndexForModelRequestCloud.Aws, Region = "us-east-1", Embed = new CreateIndexForModelRequestEmbed { Model = "llama-text-embed-v2", FieldMap = new Dictionary() { { "text", "chunk_text" } } } ); ``` ## 4. Upsert text Prepare a sample dataset of factual statements from different domains like history, physics, technology, and music. Format the data as [records](/guides/get-started/glossary#record) with an ID, text, and category. ```python Python [expandable] records = [ { "_id": "rec1", "chunk_text": "The Eiffel Tower was completed in 1889 and stands in Paris, France.", "category": "history" }, { "_id": "rec2", "chunk_text": "Photosynthesis allows plants to convert sunlight into energy.", "category": "science" }, { "_id": "rec3", "chunk_text": "Albert Einstein developed the theory of relativity.", "category": "science" }, { "_id": "rec4", "chunk_text": "The mitochondrion is often called the powerhouse of the cell.", "category": "biology" }, { "_id": "rec5", "chunk_text": "Shakespeare wrote many famous plays, including Hamlet and Macbeth.", "category": "literature" }, { "_id": "rec6", "chunk_text": "Water boils at 100°C under standard atmospheric pressure.", "category": "physics" }, { "_id": "rec7", "chunk_text": "The Great Wall of China was built to protect against invasions.", "category": "history" }, { "_id": "rec8", "chunk_text": "Honey never spoils due to its low moisture content and acidity.", "category": "food science" }, { "_id": "rec9", "chunk_text": "The speed of light in a vacuum is approximately 299,792 km/s.", "category": "physics" }, { "_id": "rec10", "chunk_text": "Newton's laws describe the motion of objects.", "category": "physics" }, { "_id": "rec11", "chunk_text": "The human brain has approximately 86 billion neurons.", "category": "biology" }, { "_id": "rec12", "chunk_text": "The Amazon Rainforest is one of the most biodiverse places on Earth.", "category": "geography" }, { "_id": "rec13", "chunk_text": "Black holes have gravitational fields so strong that not even light can escape.", "category": "astronomy" }, { "_id": "rec14", "chunk_text": "The periodic table organizes elements based on their atomic number.", "category": "chemistry" }, { "_id": "rec15", "chunk_text": "Leonardo da Vinci painted the Mona Lisa.", "category": "art" }, { "_id": "rec16", "chunk_text": "The internet revolutionized communication and information sharing.", "category": "technology" }, { "_id": "rec17", "chunk_text": "The Pyramids of Giza are among the Seven Wonders of the Ancient World.", "category": "history" }, { "_id": "rec18", "chunk_text": "Dogs have an incredible sense of smell, much stronger than humans.", "category": "biology" }, { "_id": "rec19", "chunk_text": "The Pacific Ocean is the largest and deepest ocean on Earth.", "category": "geography" }, { "_id": "rec20", "chunk_text": "Chess is a strategic game that originated in India.", "category": "games" }, { "_id": "rec21", "chunk_text": "The Statue of Liberty was a gift from France to the United States.", "category": "history" }, { "_id": "rec22", "chunk_text": "Coffee contains caffeine, a natural stimulant.", "category": "food science" }, { "_id": "rec23", "chunk_text": "Thomas Edison invented the practical electric light bulb.", "category": "inventions" }, { "_id": "rec24", "chunk_text": "The moon influences ocean tides due to gravitational pull.", "category": "astronomy" }, { "_id": "rec25", "chunk_text": "DNA carries genetic information for all living organisms.", "category": "biology" }, { "_id": "rec26", "chunk_text": "Rome was once the center of a vast empire.", "category": "history" }, { "_id": "rec27", "chunk_text": "The Wright brothers pioneered human flight in 1903.", "category": "inventions" }, { "_id": "rec28", "chunk_text": "Bananas are a good source of potassium.", "category": "nutrition" }, { "_id": "rec29", "chunk_text": "The stock market fluctuates based on supply and demand.", "category": "economics" }, { "_id": "rec30", "chunk_text": "A compass needle points toward the magnetic north pole.", "category": "navigation" }, { "_id": "rec31", "chunk_text": "The universe is expanding, according to the Big Bang theory.", "category": "astronomy" }, { "_id": "rec32", "chunk_text": "Elephants have excellent memory and strong social bonds.", "category": "biology" }, { "_id": "rec33", "chunk_text": "The violin is a string instrument commonly used in orchestras.", "category": "music" }, { "_id": "rec34", "chunk_text": "The heart pumps blood throughout the human body.", "category": "biology" }, { "_id": "rec35", "chunk_text": "Ice cream melts when exposed to heat.", "category": "food science" }, { "_id": "rec36", "chunk_text": "Solar panels convert sunlight into electricity.", "category": "technology" }, { "_id": "rec37", "chunk_text": "The French Revolution began in 1789.", "category": "history" }, { "_id": "rec38", "chunk_text": "The Taj Mahal is a mausoleum built by Emperor Shah Jahan.", "category": "history" }, { "_id": "rec39", "chunk_text": "Rainbows are caused by light refracting through water droplets.", "category": "physics" }, { "_id": "rec40", "chunk_text": "Mount Everest is the tallest mountain in the world.", "category": "geography" }, { "_id": "rec41", "chunk_text": "Octopuses are highly intelligent marine creatures.", "category": "biology" }, { "_id": "rec42", "chunk_text": "The speed of sound is around 343 meters per second in air.", "category": "physics" }, { "_id": "rec43", "chunk_text": "Gravity keeps planets in orbit around the sun.", "category": "astronomy" }, { "_id": "rec44", "chunk_text": "The Mediterranean diet is considered one of the healthiest in the world.", "category": "nutrition" }, { "_id": "rec45", "chunk_text": "A haiku is a traditional Japanese poem with a 5-7-5 syllable structure.", "category": "literature" }, { "_id": "rec46", "chunk_text": "The human body is made up of about 60% water.", "category": "biology" }, { "_id": "rec47", "chunk_text": "The Industrial Revolution transformed manufacturing and transportation.", "category": "history" }, { "_id": "rec48", "chunk_text": "Vincent van Gogh painted Starry Night.", "category": "art" }, { "_id": "rec49", "chunk_text": "Airplanes fly due to the principles of lift and aerodynamics.", "category": "physics" }, { "_id": "rec50", "chunk_text": "Renewable energy sources include wind, solar, and hydroelectric power.", "category": "energy" } ] ``` ```javascript JavaScript [expandable] const records = [ { "_id": "rec1", "chunk_text": "The Eiffel Tower was completed in 1889 and stands in Paris, France.", "category": "history" }, { "_id": "rec2", "chunk_text": "Photosynthesis allows plants to convert sunlight into energy.", "category": "science" }, { "_id": "rec3", "chunk_text": "Albert Einstein developed the theory of relativity.", "category": "science" }, { "_id": "rec4", "chunk_text": "The mitochondrion is often called the powerhouse of the cell.", "category": "biology" }, { "_id": "rec5", "chunk_text": "Shakespeare wrote many famous plays, including Hamlet and Macbeth.", "category": "literature" }, { "_id": "rec6", "chunk_text": "Water boils at 100°C under standard atmospheric pressure.", "category": "physics" }, { "_id": "rec7", "chunk_text": "The Great Wall of China was built to protect against invasions.", "category": "history" }, { "_id": "rec8", "chunk_text": "Honey never spoils due to its low moisture content and acidity.", "category": "food science" }, { "_id": "rec9", "chunk_text": "The speed of light in a vacuum is approximately 299,792 km/s.", "category": "physics" }, { "_id": "rec10", "chunk_text": "Newton's laws describe the motion of objects.", "category": "physics" }, { "_id": "rec11", "chunk_text": "The human brain has approximately 86 billion neurons.", "category": "biology" }, { "_id": "rec12", "chunk_text": "The Amazon Rainforest is one of the most biodiverse places on Earth.", "category": "geography" }, { "_id": "rec13", "chunk_text": "Black holes have gravitational fields so strong that not even light can escape.", "category": "astronomy" }, { "_id": "rec14", "chunk_text": "The periodic table organizes elements based on their atomic number.", "category": "chemistry" }, { "_id": "rec15", "chunk_text": "Leonardo da Vinci painted the Mona Lisa.", "category": "art" }, { "_id": "rec16", "chunk_text": "The internet revolutionized communication and information sharing.", "category": "technology" }, { "_id": "rec17", "chunk_text": "The Pyramids of Giza are among the Seven Wonders of the Ancient World.", "category": "history" }, { "_id": "rec18", "chunk_text": "Dogs have an incredible sense of smell, much stronger than humans.", "category": "biology" }, { "_id": "rec19", "chunk_text": "The Pacific Ocean is the largest and deepest ocean on Earth.", "category": "geography" }, { "_id": "rec20", "chunk_text": "Chess is a strategic game that originated in India.", "category": "games" }, { "_id": "rec21", "chunk_text": "The Statue of Liberty was a gift from France to the United States.", "category": "history" }, { "_id": "rec22", "chunk_text": "Coffee contains caffeine, a natural stimulant.", "category": "food science" }, { "_id": "rec23", "chunk_text": "Thomas Edison invented the practical electric light bulb.", "category": "inventions" }, { "_id": "rec24", "chunk_text": "The moon influences ocean tides due to gravitational pull.", "category": "astronomy" }, { "_id": "rec25", "chunk_text": "DNA carries genetic information for all living organisms.", "category": "biology" }, { "_id": "rec26", "chunk_text": "Rome was once the center of a vast empire.", "category": "history" }, { "_id": "rec27", "chunk_text": "The Wright brothers pioneered human flight in 1903.", "category": "inventions" }, { "_id": "rec28", "chunk_text": "Bananas are a good source of potassium.", "category": "nutrition" }, { "_id": "rec29", "chunk_text": "The stock market fluctuates based on supply and demand.", "category": "economics" }, { "_id": "rec30", "chunk_text": "A compass needle points toward the magnetic north pole.", "category": "navigation" }, { "_id": "rec31", "chunk_text": "The universe is expanding, according to the Big Bang theory.", "category": "astronomy" }, { "_id": "rec32", "chunk_text": "Elephants have excellent memory and strong social bonds.", "category": "biology" }, { "_id": "rec33", "chunk_text": "The violin is a string instrument commonly used in orchestras.", "category": "music" }, { "_id": "rec34", "chunk_text": "The heart pumps blood throughout the human body.", "category": "biology" }, { "_id": "rec35", "chunk_text": "Ice cream melts when exposed to heat.", "category": "food science" }, { "_id": "rec36", "chunk_text": "Solar panels convert sunlight into electricity.", "category": "technology" }, { "_id": "rec37", "chunk_text": "The French Revolution began in 1789.", "category": "history" }, { "_id": "rec38", "chunk_text": "The Taj Mahal is a mausoleum built by Emperor Shah Jahan.", "category": "history" }, { "_id": "rec39", "chunk_text": "Rainbows are caused by light refracting through water droplets.", "category": "physics" }, { "_id": "rec40", "chunk_text": "Mount Everest is the tallest mountain in the world.", "category": "geography" }, { "_id": "rec41", "chunk_text": "Octopuses are highly intelligent marine creatures.", "category": "biology" }, { "_id": "rec42", "chunk_text": "The speed of sound is around 343 meters per second in air.", "category": "physics" }, { "_id": "rec43", "chunk_text": "Gravity keeps planets in orbit around the sun.", "category": "astronomy" }, { "_id": "rec44", "chunk_text": "The Mediterranean diet is considered one of the healthiest in the world.", "category": "nutrition" }, { "_id": "rec45", "chunk_text": "A haiku is a traditional Japanese poem with a 5-7-5 syllable structure.", "category": "literature" }, { "_id": "rec46", "chunk_text": "The human body is made up of about 60% water.", "category": "biology" }, { "_id": "rec47", "chunk_text": "The Industrial Revolution transformed manufacturing and transportation.", "category": "history" }, { "_id": "rec48", "chunk_text": "Vincent van Gogh painted Starry Night.", "category": "art" }, { "_id": "rec49", "chunk_text": "Airplanes fly due to the principles of lift and aerodynamics.", "category": "physics" }, { "_id": "rec50", "chunk_text": "Renewable energy sources include wind, solar, and hydroelectric power.", "category": "energy" } ]; ``` ```java Java [expandable] // Add to the Quickstart class: ArrayList> upsertRecords = new ArrayList<>(); HashMap record1 = new HashMap<>(); record1.put("_id", "rec1"); record1.put("chunk_text", "The Eiffel Tower was completed in 1889 and stands in Paris, France."); record1.put("category", "history"); HashMap record2 = new HashMap<>(); record2.put("_id", "rec2"); record2.put("chunk_text", "Photosynthesis allows plants to convert sunlight into energy."); record2.put("category", "science"); HashMap record3 = new HashMap<>(); record3.put("_id", "rec3"); record3.put("chunk_text", "Albert Einstein developed the theory of relativity."); record3.put("category", "science"); HashMap record4 = new HashMap<>(); record4.put("_id", "rec4"); record4.put("chunk_text", "The mitochondrion is often called the powerhouse of the cell."); record4.put("category", "biology"); HashMap record5 = new HashMap<>(); record5.put("_id", "rec5"); record5.put("chunk_text", "Shakespeare wrote many famous plays, including Hamlet and Macbeth."); record5.put("category", "literature"); HashMap record6 = new HashMap<>(); record6.put("_id", "rec6"); record6.put("chunk_text", "Water boils at 100°C under standard atmospheric pressure."); record6.put("category", "physics"); HashMap record7 = new HashMap<>(); record7.put("_id", "rec7"); record7.put("chunk_text", "The Great Wall of China was built to protect against invasions."); record7.put("category", "history"); HashMap record8 = new HashMap<>(); record8.put("_id", "rec8"); record8.put("chunk_text", "Honey never spoils due to its low moisture content and acidity."); record8.put("category", "food science"); HashMap record9 = new HashMap<>(); record9.put("_id", "rec9"); record9.put("chunk_text", "The speed of light in a vacuum is approximately 299,792 km/s."); record9.put("category", "physics"); HashMap record10 = new HashMap<>(); record10.put("_id", "rec10"); record10.put("chunk_text", "Newton's laws describe the motion of objects."); record10.put("category", "physics"); HashMap record11 = new HashMap<>(); record11.put("_id", "rec11"); record11.put("chunk_text", "The human brain has approximately 86 billion neurons."); record11.put("category", "biology"); HashMap record12 = new HashMap<>(); record12.put("_id", "rec12"); record12.put("chunk_text", "The Amazon Rainforest is one of the most biodiverse places on Earth."); record12.put("category", "geography"); HashMap record13 = new HashMap<>(); record13.put("_id", "rec13"); record13.put("chunk_text", "Black holes have gravitational fields so strong that not even light can escape."); record13.put("category", "astronomy"); HashMap record14 = new HashMap<>(); record14.put("_id", "rec14"); record14.put("chunk_text", "The periodic table organizes elements based on their atomic number."); record14.put("category", "chemistry"); HashMap record15 = new HashMap<>(); record15.put("_id", "rec15"); record15.put("chunk_text", "Leonardo da Vinci painted the Mona Lisa."); record15.put("category", "art"); HashMap record16 = new HashMap<>(); record16.put("_id", "rec16"); record16.put("chunk_text", "The internet revolutionized communication and information sharing."); record16.put("category", "technology"); HashMap record17 = new HashMap<>(); record17.put("_id", "rec17"); record17.put("chunk_text", "The Pyramids of Giza are among the Seven Wonders of the Ancient World."); record17.put("category", "history"); HashMap record18 = new HashMap<>(); record18.put("_id", "rec18"); record18.put("chunk_text", "Dogs have an incredible sense of smell, much stronger than humans."); record18.put("category", "biology"); HashMap record19 = new HashMap<>(); record19.put("_id", "rec19"); record19.put("chunk_text", "The Pacific Ocean is the largest and deepest ocean on Earth."); record19.put("category", "geography"); HashMap record20 = new HashMap<>(); record20.put("_id", "rec20"); record20.put("chunk_text", "Chess is a strategic game that originated in India."); record20.put("category", "games"); HashMap record21 = new HashMap<>(); record21.put("_id", "rec21"); record21.put("chunk_text", "The Statue of Liberty was a gift from France to the United States."); record21.put("category", "history"); HashMap record22 = new HashMap<>(); record22.put("_id", "rec22"); record22.put("chunk_text", "Coffee contains caffeine, a natural stimulant."); record22.put("category", "food science"); HashMap record23 = new HashMap<>(); record23.put("_id", "rec23"); record23.put("chunk_text", "Thomas Edison invented the practical electric light bulb."); record23.put("category", "inventions"); HashMap record24 = new HashMap<>(); record24.put("_id", "rec24"); record24.put("chunk_text", "The moon influences ocean tides due to gravitational pull."); record24.put("category", "astronomy"); HashMap record25 = new HashMap<>(); record25.put("_id", "rec25"); record25.put("chunk_text", "DNA carries genetic information for all living organisms."); record25.put("category", "biology"); HashMap record26 = new HashMap<>(); record26.put("_id", "rec26"); record26.put("chunk_text", "Rome was once the center of a vast empire."); record26.put("category", "history"); HashMap record27 = new HashMap<>(); record27.put("_id", "rec27"); record27.put("chunk_text", "The Wright brothers pioneered human flight in 1903."); record27.put("category", "inventions"); HashMap record28 = new HashMap<>(); record28.put("_id", "rec28"); record28.put("chunk_text", "Bananas are a good source of potassium."); record28.put("category", "nutrition"); HashMap record29 = new HashMap<>(); record29.put("_id", "rec29"); record29.put("chunk_text", "The stock market fluctuates based on supply and demand."); record29.put("category", "economics"); HashMap record30 = new HashMap<>(); record30.put("_id", "rec30"); record30.put("chunk_text", "A compass needle points toward the magnetic north pole."); record30.put("category", "navigation"); HashMap record31 = new HashMap<>(); record31.put("_id", "rec31"); record31.put("chunk_text", "The universe is expanding, according to the Big Bang theory."); record31.put("category", "astronomy"); HashMap record32 = new HashMap<>(); record32.put("_id", "rec32"); record32.put("chunk_text", "Elephants have excellent memory and strong social bonds."); record32.put("category", "biology"); HashMap record33 = new HashMap<>(); record33.put("_id", "rec33"); record33.put("chunk_text", "The violin is a string instrument commonly used in orchestras."); record33.put("category", "music"); HashMap record34 = new HashMap<>(); record34.put("_id", "rec34"); record34.put("chunk_text", "The heart pumps blood throughout the human body."); record34.put("category", "biology"); HashMap record35 = new HashMap<>(); record35.put("_id", "rec35"); record35.put("chunk_text", "Ice cream melts when exposed to heat."); record35.put("category", "food science"); HashMap record36 = new HashMap<>(); record36.put("_id", "rec36"); record36.put("chunk_text", "Solar panels convert sunlight into electricity."); record36.put("category", "technology"); HashMap record37 = new HashMap<>(); record37.put("_id", "rec37"); record37.put("chunk_text", "The French Revolution began in 1789."); record37.put("category", "history"); HashMap record38 = new HashMap<>(); record38.put("_id", "rec38"); record38.put("chunk_text", "The Taj Mahal is a mausoleum built by Emperor Shah Jahan."); record38.put("category", "history"); HashMap record39 = new HashMap<>(); record39.put("_id", "rec39"); record39.put("chunk_text", "Rainbows are caused by light refracting through water droplets."); record39.put("category", "physics"); HashMap record40 = new HashMap<>(); record40.put("_id", "rec40"); record40.put("chunk_text", "Mount Everest is the tallest mountain in the world."); record40.put("category", "geography"); HashMap record41 = new HashMap<>(); record41.put("_id", "rec41"); record41.put("chunk_text", "Octopuses are highly intelligent marine creatures."); record41.put("category", "biology"); HashMap record42 = new HashMap<>(); record42.put("_id", "rec42"); record42.put("chunk_text", "The speed of sound is around 343 meters per second in air."); record42.put("category", "physics"); HashMap record43 = new HashMap<>(); record43.put("_id", "rec43"); record43.put("chunk_text", "Gravity keeps planets in orbit around the sun."); record43.put("category", "astronomy"); HashMap record44 = new HashMap<>(); record44.put("_id", "rec44"); record44.put("chunk_text", "The Mediterranean diet is considered one of the healthiest in the world."); record44.put("category", "nutrition"); HashMap record45 = new HashMap<>(); record45.put("_id", "rec45"); record45.put("chunk_text", "A haiku is a traditional Japanese poem with a 5-7-5 syllable structure."); record45.put("category", "literature"); HashMap record46 = new HashMap<>(); record46.put("_id", "rec46"); record46.put("chunk_text", "The human body is made up of about 60% water."); record46.put("category", "biology"); HashMap record47 = new HashMap<>(); record47.put("_id", "rec47"); record47.put("chunk_text", "The Industrial Revolution transformed manufacturing and transportation."); record47.put("category", "history"); HashMap record48 = new HashMap<>(); record48.put("_id", "rec48"); record48.put("chunk_text", "Vincent van Gogh painted Starry Night."); record48.put("category", "art"); HashMap record49 = new HashMap<>(); record49.put("_id", "rec49"); record49.put("chunk_text", "Airplanes fly due to the principles of lift and aerodynamics."); record49.put("category", "physics"); HashMap record50 = new HashMap<>(); record50.put("_id", "rec50"); record50.put("chunk_text", "Renewable energy sources include wind, solar, and hydroelectric power."); record50.put("category", "energy"); upsertRecords.add(record1); upsertRecords.add(record2); upsertRecords.add(record3); upsertRecords.add(record4); upsertRecords.add(record5); upsertRecords.add(record6); upsertRecords.add(record7); upsertRecords.add(record8); upsertRecords.add(record9); upsertRecords.add(record10); upsertRecords.add(record11); upsertRecords.add(record12); upsertRecords.add(record13); upsertRecords.add(record14); upsertRecords.add(record15); upsertRecords.add(record16); upsertRecords.add(record17); upsertRecords.add(record18); upsertRecords.add(record19); upsertRecords.add(record20); upsertRecords.add(record21); upsertRecords.add(record22); upsertRecords.add(record23); upsertRecords.add(record24); upsertRecords.add(record25); upsertRecords.add(record26); upsertRecords.add(record27); upsertRecords.add(record28); upsertRecords.add(record29); upsertRecords.add(record30); upsertRecords.add(record31); upsertRecords.add(record32); upsertRecords.add(record33); upsertRecords.add(record34); upsertRecords.add(record35); upsertRecords.add(record36); upsertRecords.add(record37); upsertRecords.add(record38); upsertRecords.add(record39); upsertRecords.add(record40); upsertRecords.add(record41); upsertRecords.add(record42); upsertRecords.add(record43); upsertRecords.add(record44); upsertRecords.add(record45); upsertRecords.add(record46); upsertRecords.add(record47); upsertRecords.add(record48); upsertRecords.add(record49); upsertRecords.add(record50); ``` ```go Go [expandable] // Add to the main function: records := []*pinecone.IntegratedRecord{ { "_id": "rec1", "chunk_text": "The Eiffel Tower was completed in 1889 and stands in Paris, France.", "category": "history" }, { "_id": "rec2", "chunk_text": "Photosynthesis allows plants to convert sunlight into energy.", "category": "science" }, { "_id": "rec3", "chunk_text": "Albert Einstein developed the theory of relativity.", "category": "science" }, { "_id": "rec4", "chunk_text": "The mitochondrion is often called the powerhouse of the cell.", "category": "biology" }, { "_id": "rec5", "chunk_text": "Shakespeare wrote many famous plays, including Hamlet and Macbeth.", "category": "literature" }, { "_id": "rec6", "chunk_text": "Water boils at 100°C under standard atmospheric pressure.", "category": "physics" }, { "_id": "rec7", "chunk_text": "The Great Wall of China was built to protect against invasions.", "category": "history" }, { "_id": "rec8", "chunk_text": "Honey never spoils due to its low moisture content and acidity.", "category": "food science" }, { "_id": "rec9", "chunk_text": "The speed of light in a vacuum is approximately 299,792 km/s.", "category": "physics" }, { "_id": "rec10", "chunk_text": "Newton's laws describe the motion of objects.", "category": "physics" }, { "_id": "rec11", "chunk_text": "The human brain has approximately 86 billion neurons.", "category": "biology" }, { "_id": "rec12", "chunk_text": "The Amazon Rainforest is one of the most biodiverse places on Earth.", "category": "geography" }, { "_id": "rec13", "chunk_text": "Black holes have gravitational fields so strong that not even light can escape.", "category": "astronomy" }, { "_id": "rec14", "chunk_text": "The periodic table organizes elements based on their atomic number.", "category": "chemistry" }, { "_id": "rec15", "chunk_text": "Leonardo da Vinci painted the Mona Lisa.", "category": "art" }, { "_id": "rec16", "chunk_text": "The internet revolutionized communication and information sharing.", "category": "technology" }, { "_id": "rec17", "chunk_text": "The Pyramids of Giza are among the Seven Wonders of the Ancient World.", "category": "history" }, { "_id": "rec18", "chunk_text": "Dogs have an incredible sense of smell, much stronger than humans.", "category": "biology" }, { "_id": "rec19", "chunk_text": "The Pacific Ocean is the largest and deepest ocean on Earth.", "category": "geography" }, { "_id": "rec20", "chunk_text": "Chess is a strategic game that originated in India.", "category": "games" }, { "_id": "rec21", "chunk_text": "The Statue of Liberty was a gift from France to the United States.", "category": "history" }, { "_id": "rec22", "chunk_text": "Coffee contains caffeine, a natural stimulant.", "category": "food science" }, { "_id": "rec23", "chunk_text": "Thomas Edison invented the practical electric light bulb.", "category": "inventions" }, { "_id": "rec24", "chunk_text": "The moon influences ocean tides due to gravitational pull.", "category": "astronomy" }, { "_id": "rec25", "chunk_text": "DNA carries genetic information for all living organisms.", "category": "biology" }, { "_id": "rec26", "chunk_text": "Rome was once the center of a vast empire.", "category": "history" }, { "_id": "rec27", "chunk_text": "The Wright brothers pioneered human flight in 1903.", "category": "inventions" }, { "_id": "rec28", "chunk_text": "Bananas are a good source of potassium.", "category": "nutrition" }, { "_id": "rec29", "chunk_text": "The stock market fluctuates based on supply and demand.", "category": "economics" }, { "_id": "rec30", "chunk_text": "A compass needle points toward the magnetic north pole.", "category": "navigation" }, { "_id": "rec31", "chunk_text": "The universe is expanding, according to the Big Bang theory.", "category": "astronomy" }, { "_id": "rec32", "chunk_text": "Elephants have excellent memory and strong social bonds.", "category": "biology" }, { "_id": "rec33", "chunk_text": "The violin is a string instrument commonly used in orchestras.", "category": "music" }, { "_id": "rec34", "chunk_text": "The heart pumps blood throughout the human body.", "category": "biology" }, { "_id": "rec35", "chunk_text": "Ice cream melts when exposed to heat.", "category": "food science" }, { "_id": "rec36", "chunk_text": "Solar panels convert sunlight into electricity.", "category": "technology" }, { "_id": "rec37", "chunk_text": "The French Revolution began in 1789.", "category": "history" }, { "_id": "rec38", "chunk_text": "The Taj Mahal is a mausoleum built by Emperor Shah Jahan.", "category": "history" }, { "_id": "rec39", "chunk_text": "Rainbows are caused by light refracting through water droplets.", "category": "physics" }, { "_id": "rec40", "chunk_text": "Mount Everest is the tallest mountain in the world.", "category": "geography" }, { "_id": "rec41", "chunk_text": "Octopuses are highly intelligent marine creatures.", "category": "biology" }, { "_id": "rec42", "chunk_text": "The speed of sound is around 343 meters per second in air.", "category": "physics" }, { "_id": "rec43", "chunk_text": "Gravity keeps planets in orbit around the sun.", "category": "astronomy" }, { "_id": "rec44", "chunk_text": "The Mediterranean diet is considered one of the healthiest in the world.", "category": "nutrition" }, { "_id": "rec45", "chunk_text": "A haiku is a traditional Japanese poem with a 5-7-5 syllable structure.", "category": "literature" }, { "_id": "rec46", "chunk_text": "The human body is made up of about 60% water.", "category": "biology" }, { "_id": "rec47", "chunk_text": "The Industrial Revolution transformed manufacturing and transportation.", "category": "history" }, { "_id": "rec48", "chunk_text": "Vincent van Gogh painted Starry Night.", "category": "art" }, { "_id": "rec49", "chunk_text": "Airplanes fly due to the principles of lift and aerodynamics.", "category": "physics" }, { "_id": "rec50", "chunk_text": "Renewable energy sources include wind, solar, and hydroelectric power.", "category": "energy" }, } ``` ```csharp C# [expandable] var records = new List { new UpsertRecord { Id = "rec1", AdditionalProperties = { ["chunk_text"] = "The Eiffel Tower was completed in 1889 and stands in Paris, France.", ["category"] = "history", }, }, new UpsertRecord { Id = "rec2", AdditionalProperties = { ["chunk_text"] = "Photosynthesis allows plants to convert sunlight into energy.", ["category"] = "science", }, }, new UpsertRecord { Id = "rec3", AdditionalProperties = { ["chunk_text"] = "Albert Einstein developed the theory of relativity.", ["category"] = "science", }, }, new UpsertRecord { Id = "rec4", AdditionalProperties = { ["chunk_text"] = "The mitochondrion is often called the powerhouse of the cell.", ["category"] = "biology", }, }, new UpsertRecord { Id = "rec5", AdditionalProperties = { ["chunk_text"] = "Shakespeare wrote many famous plays, including Hamlet and Macbeth.", ["category"] = "literature", }, }, new UpsertRecord { Id = "rec6", AdditionalProperties = { ["chunk_text"] = "Water boils at 100°C under standard atmospheric pressure.", ["category"] = "physics", }, }, new UpsertRecord { Id = "rec7", AdditionalProperties = { ["chunk_text"] = "The Great Wall of China was built to protect against invasions.", ["category"] = "history", }, }, new UpsertRecord { Id = "rec8", AdditionalProperties = { ["chunk_text"] = "Honey never spoils due to its low moisture content and acidity.", ["category"] = "food science", }, }, new UpsertRecord { Id = "rec9", AdditionalProperties = { ["chunk_text"] = "The speed of light in a vacuum is approximately 299,792 km/s.", ["category"] = "physics", }, }, new UpsertRecord { Id = "rec10", AdditionalProperties = { ["chunk_text"] = "Newton's laws describe the motion of objects.", ["category"] = "physics", }, }, new UpsertRecord { Id = "rec11", AdditionalProperties = { ["chunk_text"] = "The human brain has approximately 86 billion neurons.", ["category"] = "biology", }, }, new UpsertRecord { Id = "rec12", AdditionalProperties = { ["chunk_text"] = "The Amazon Rainforest is one of the most biodiverse places on Earth.", ["category"] = "geography", }, }, new UpsertRecord { Id = "rec13", AdditionalProperties = { ["chunk_text"] = "Black holes have gravitational fields so strong that not even light can escape.", ["category"] = "astronomy", }, }, new UpsertRecord { Id = "rec14", AdditionalProperties = { ["chunk_text"] = "The periodic table organizes elements based on their atomic number.", ["category"] = "chemistry", }, }, new UpsertRecord { Id = "rec15", AdditionalProperties = { ["chunk_text"] = "Leonardo da Vinci painted the Mona Lisa.", ["category"] = "art", }, }, new UpsertRecord { Id = "rec16", AdditionalProperties = { ["chunk_text"] = "The internet revolutionized communication and information sharing.", ["category"] = "technology", }, }, new UpsertRecord { Id = "rec17", AdditionalProperties = { ["chunk_text"] = "The Pyramids of Giza are among the Seven Wonders of the Ancient World.", ["category"] = "history", }, }, new UpsertRecord { Id = "rec18", AdditionalProperties = { ["chunk_text"] = "Dogs have an incredible sense of smell, much stronger than humans.", ["category"] = "biology", }, }, new UpsertRecord { Id = "rec19", AdditionalProperties = { ["chunk_text"] = "The Pacific Ocean is the largest and deepest ocean on Earth.", ["category"] = "geography", }, }, new UpsertRecord { Id = "rec20", AdditionalProperties = { ["chunk_text"] = "Chess is a strategic game that originated in India.", ["category"] = "games", }, }, new UpsertRecord { Id = "rec21", AdditionalProperties = { ["chunk_text"] = "The Statue of Liberty was a gift from France to the United States.", ["category"] = "history", }, }, new UpsertRecord { Id = "rec22", AdditionalProperties = { ["chunk_text"] = "Coffee contains caffeine, a natural stimulant.", ["category"] = "food science", }, }, new UpsertRecord { Id = "rec23", AdditionalProperties = { ["chunk_text"] = "Thomas Edison invented the practical electric light bulb.", ["category"] = "inventions", }, }, new UpsertRecord { Id = "rec24", AdditionalProperties = { ["chunk_text"] = "The moon influences ocean tides due to gravitational pull.", ["category"] = "astronomy", }, }, new UpsertRecord { Id = "rec25", AdditionalProperties = { ["chunk_text"] = "DNA carries genetic information for all living organisms.", ["category"] = "biology", }, }, new UpsertRecord { Id = "rec26", AdditionalProperties = { ["chunk_text"] = "Rome was once the center of a vast empire.", ["category"] = "history", }, }, new UpsertRecord { Id = "rec27", AdditionalProperties = { ["chunk_text"] = "The Wright brothers pioneered human flight in 1903.", ["category"] = "inventions", }, }, new UpsertRecord { Id = "rec28", AdditionalProperties = { ["chunk_text"] = "Bananas are a good source of potassium.", ["category"] = "nutrition", }, }, new UpsertRecord { Id = "rec29", AdditionalProperties = { ["chunk_text"] = "The stock market fluctuates based on supply and demand.", ["category"] = "economics", }, }, new UpsertRecord { Id = "rec30", AdditionalProperties = { ["chunk_text"] = "A compass needle points toward the magnetic north pole.", ["category"] = "navigation", }, }, new UpsertRecord { Id = "rec31", AdditionalProperties = { ["chunk_text"] = "The universe is expanding, according to the Big Bang theory.", ["category"] = "astronomy", }, }, new UpsertRecord { Id = "rec32", AdditionalProperties = { ["chunk_text"] = "Elephants have excellent memory and strong social bonds.", ["category"] = "biology", }, }, new UpsertRecord { Id = "rec33", AdditionalProperties = { ["chunk_text"] = "The violin is a string instrument commonly used in orchestras.", ["category"] = "music", }, }, new UpsertRecord { Id = "rec34", AdditionalProperties = { ["chunk_text"] = "The heart pumps blood throughout the human body.", ["category"] = "biology", }, }, new UpsertRecord { Id = "rec35", AdditionalProperties = { ["chunk_text"] = "Ice cream melts when exposed to heat.", ["category"] = "food science", }, }, new UpsertRecord { Id = "rec36", AdditionalProperties = { ["chunk_text"] = "Solar panels convert sunlight into electricity.", ["category"] = "technology", }, }, new UpsertRecord { Id = "rec37", AdditionalProperties = { ["chunk_text"] = "The French Revolution began in 1789.", ["category"] = "history", }, }, new UpsertRecord { Id = "rec38", AdditionalProperties = { ["chunk_text"] = "The Taj Mahal is a mausoleum built by Emperor Shah Jahan.", ["category"] = "history", }, }, new UpsertRecord { Id = "rec39", AdditionalProperties = { ["chunk_text"] = "Rainbows are caused by light refracting through water droplets.", ["category"] = "physics", }, }, new UpsertRecord { Id = "rec40", AdditionalProperties = { ["chunk_text"] = "Mount Everest is the tallest mountain in the world.", ["category"] = "geography", }, }, new UpsertRecord { Id = "rec41", AdditionalProperties = { ["chunk_text"] = "Octopuses are highly intelligent marine creatures.", ["category"] = "biology", }, }, new UpsertRecord { Id = "rec42", AdditionalProperties = { ["chunk_text"] = "The speed of sound is around 343 meters per second in air.", ["category"] = "physics", }, }, new UpsertRecord { Id = "rec43", AdditionalProperties = { ["chunk_text"] = "Gravity keeps planets in orbit around the sun.", ["category"] = "astronomy", }, }, new UpsertRecord { Id = "rec44", AdditionalProperties = { ["chunk_text"] = "The Mediterranean diet is considered one of the healthiest in the world.", ["category"] = "nutrition", }, }, new UpsertRecord { Id = "rec45", AdditionalProperties = { ["chunk_text"] = "A haiku is a traditional Japanese poem with a 5-7-5 syllable structure.", ["category"] = "literature", }, }, new UpsertRecord { Id = "rec46", AdditionalProperties = { ["chunk_text"] = "The human body is made up of about 60% water.", ["category"] = "biology", }, }, new UpsertRecord { Id = "rec47", AdditionalProperties = { ["chunk_text"] = "The Industrial Revolution transformed manufacturing and transportation.", ["category"] = "history", }, }, new UpsertRecord { Id = "rec48", AdditionalProperties = { ["chunk_text"] = "Vincent van Gogh painted Starry Night.", ["category"] = "art", }, }, new UpsertRecord { Id = "rec49", AdditionalProperties = { ["chunk_text"] = "Airplanes fly due to the principles of lift and aerodynamics.", ["category"] = "physics", }, }, new UpsertRecord { Id = "rec50", AdditionalProperties = { ["chunk_text"] = "Renewable energy sources include wind, solar, and hydroelectric power.", ["category"] = "energy", }, }, }; ``` [Upsert](/guides/index-data/upsert-data) the sample dataset into a new [namespace](/guides/index-data/indexing-overview#namespaces) in your index. Because your index is integrated with an embedding model, you provide the textual statements and Pinecone converts them to dense vectors automatically. ```python Python # Target the index dense_index = pc.Index(index_name) # Upsert the records into a namespace dense_index.upsert_records("example-namespace", records) ``` ```javascript JavaScript // Target the index const index = pc.index(indexName).namespace("example-namespace"); // Upsert the records into a namespace await index.upsertRecords(records); ``` ```java Java // Add to the Quickstart class: // Target the index Index index = new Index(config, connection, "quickstart-java"); // Upsert the records into a namespace index.upsertRecords("example-namespace", upsertRecords); ``` ```go Go // Add to the main function: // Target the index idxModel, err := pc.DescribeIndex(ctx, indexName) if err != nil { log.Fatalf("Failed to describe index \"%v\": %v", indexName, err) } idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: idxModel.Host, Namespace: "example-namespace"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v: %v", idxModel.Host, err) } // Upsert the records into a namespace err = idxConnection.UpsertRecords(ctx, records) if err != nil { log.Fatalf("Failed to upsert vectors: %v", err) } ``` ```csharp C# // Upsert the records into a namespace await index.UpsertRecordsAsync( "example-namespace", records ); ``` To control costs when ingesting large datasets (10,000,000+ records), use [import](/guides/index-data/import-data) instead of upsert. Pinecone is eventually consistent, so there can be a slight delay before new or changed records are visible to queries. You can [view index stats](/guides/index-data/check-data-freshness) to check if the current vector count matches the number of vectors you upserted (50): ```python Python # Wait for the upserted vectors to be indexed import time time.sleep(10) # View stats for the index stats = dense_index.describe_index_stats() print(stats) ``` ```javascript JavaScript // Wait for the upserted vectors to be indexed await new Promise(resolve => setTimeout(resolve, 10000)); // View stats for the index const stats = await index.describeIndexStats(); console.log(stats); ``` ```java Java // Add to the Quickstart class: // Wait for upserted vectors to be indexed Thread.sleep(5000); // View stats for the index DescribeIndexStatsResponse indexStatsResponse = index.describeIndexStats(); System.out.println(indexStatsResponse); ``` ```go Go // Add to the main function: // View stats for the index stats, err := idxConnection.DescribeIndexStats(ctx) if err != nil { log.Fatalf("Failed to describe index \"%v\": %v", indexName, err) } else { fmt.Printf("%+v", prettifyStruct(*stats)) } ``` ```csharp C# var indexStatsResponse = await index.DescribeIndexStatsAsync(new DescribeIndexStatsRequest()); Console.WriteLine(indexStatsResponse); ``` The response looks like this: ```python Python {'dimension': 1024, 'index_fullness': 0.0, 'metric': 'cosine', 'namespaces': {'example-namespace': {'vector_count': 50}}, 'total_vector_count': 50, 'vector_type': 'dense'} ``` ```javascript JavaScript { namespaces: { 'example-namespace': { recordCount: 50 } }, dimension: 1024, indexFullness: 0, totalRecordCount: 50 } ``` ```java Java namespaces { key: "example-namespace" value { vector_count: 50 } } dimension: 1024 total_vector_count: 50 metric: "cosine" vector_type: "dense" ``` ```go Go { "dimension": 1024, "index_fullness": 0, "total_vector_count": 50, "namespaces": { "example-namespace": { "vector_count": 50 } } } ``` ```csharp C# { "namespaces": { "example-namespace": { "vectorCount": 50 } }, "dimension": 1024, "indexFullness": 0, "totalVectorCount": 50, "metric": "cosine", "vectorType": "dense" } ``` ## 5. Semantic search [Search the dense index](/guides/search/semantic-search) for ten records that are most semantically similar to the query, "Famous historical structures and monuments". Again, because your index is integrated with an embedding model, you provide the query as text and Pinecone converts the text to a dense vector automatically. ```python Python # Define the query query = "Famous historical structures and monuments" # Search the dense index results = dense_index.search( namespace="example-namespace", query={ "top_k": 10, "inputs": { 'text': query } } ) # Print the results for hit in results['result']['hits']: print(f"id: {hit['_id']:<5} | score: {round(hit['_score'], 2):<5} | category: {hit['fields']['category']:<10} | text: {hit['fields']['chunk_text']:<50}") ``` ```javascript JavaScript // Define the query const query = 'Famous historical structures and monuments'; // Search the dense index const results = await index.searchRecords({ query: { topK: 10, inputs: { text: query }, }, }); // Print the results results.result.hits.forEach(hit => { console.log(`id: ${hit.id}, score: ${hit.score.toFixed(2)}, category: ${hit.fields.category}, text: ${hit.fields.chunk_text}`); }); ``` ```java Java // Add to the Quickstart class: // Define the query String query = "Famous historical structures and monuments"; List fields = new ArrayList<>(); fields.add("category"); fields.add("chunk_text"); // Search the dense index SearchRecordsResponse recordsResponse = index.searchRecordsByText(query, "example-namespace", fields, 10, null, null); // Print the results System.out.println(recordsResponse); ``` ```go Go // Add to the main function: // Define the query query := "Famous historical structures and monuments" // Search the dense index res, err := idxConnection.SearchRecords(ctx, &pinecone.SearchRecordsRequest{ Query: pinecone.SearchRecordsQuery{ TopK: 10, Inputs: &map[string]interface{}{ "text": query, }, }, }) if err != nil { log.Fatalf("Failed to search records: %v", err) } fmt.Printf(prettifyStruct(res)) ``` ```csharp C# // Search the dense index var response = await index.SearchRecordsAsync( "example-namespace", new SearchRecordsRequest { Query = new SearchRecordsRequestQuery { TopK = 10, Inputs = new Dictionary { { "text", "Famous historical structures and monuments" } }, }, Fields = ["category", "chunk_text"], } ); Console.WriteLine(response); ``` Notice that most of the results are about historical structures and monuments. However, a few unrelated statements are included as well and are ranked high in the list, for example, a statement about Shakespeare. ```console Python id: rec17 | score: 0.24 | category: history | text: The Pyramids of Giza are among the Seven Wonders of the Ancient World. id: rec38 | score: 0.19 | category: history | text: The Taj Mahal is a mausoleum built by Emperor Shah Jahan. id: rec5 | score: 0.19 | category: literature | text: Shakespeare wrote many famous plays, including Hamlet and Macbeth. id: rec15 | score: 0.11 | category: art | text: Leonardo da Vinci painted the Mona Lisa. id: rec50 | score: 0.1 | category: energy | text: Renewable energy sources include wind, solar, and hydroelectric power. id: rec26 | score: 0.09 | category: history | text: Rome was once the center of a vast empire. id: rec47 | score: 0.08 | category: history | text: The Industrial Revolution transformed manufacturing and transportation. id: rec7 | score: 0.07 | category: history | text: The Great Wall of China was built to protect against invasions. id: rec1 | score: 0.07 | category: history | text: The Eiffel Tower was completed in 1889 and stands in Paris, France. id: rec3 | score: 0.07 | category: science | text: Albert Einstein developed the theory of relativity. ``` ```console JavaScript id: rec17, score: 0.24, text: The Pyramids of Giza are among the Seven Wonders of the Ancient World., category: history id: rec38, score: 0.19, text: The Taj Mahal is a mausoleum built by Emperor Shah Jahan., category: history id: rec5, score: 0.19, text: Shakespeare wrote many famous plays, including Hamlet and Macbeth., category: literature id: rec15, score: 0.11, text: Leonardo da Vinci painted the Mona Lisa., category: art id: rec50, score: 0.10, text: Renewable energy sources include wind, solar, and hydroelectric power., category: energy id: rec26, score: 0.09, text: Rome was once the center of a vast empire., category: history id: rec47, score: 0.08, text: The Industrial Revolution transformed manufacturing and transportation., category: history id: rec7, score: 0.07, text: The Great Wall of China was built to protect against invasions., category: history id: rec1, score: 0.07, text: The Eiffel Tower was completed in 1889 and stands in Paris, France., category: history id: rec3, score: 0.07, text: Albert Einstein developed the theory of relativity., category: science ``` ```java Java [expandable] class SearchRecordsResponse { result: class SearchRecordsResponseResult { hits: [class Hit { id: rec17 score: 0.77387625 fields: {category=history, chunk_text=The Pyramids of Giza are among the Seven Wonders of the Ancient World.} additionalProperties: null }, class Hit { id: rec1 score: 0.77372295 fields: {category=history, chunk_text=The Eiffel Tower was completed in 1889 and stands in Paris, France.} additionalProperties: null }, class Hit { id: rec38 score: 0.75988203 fields: {category=history, chunk_text=The Taj Mahal is a mausoleum built by Emperor Shah Jahan.} additionalProperties: null }, class Hit { id: rec5 score: 0.75516135 fields: {category=literature, chunk_text=Shakespeare wrote many famous plays, including Hamlet and Macbeth.} additionalProperties: null }, class Hit { id: rec26 score: 0.7550185 fields: {category=history, chunk_text=Rome was once the center of a vast empire.} additionalProperties: null }, class Hit { id: rec45 score: 0.73588645 fields: {category=literature, chunk_text=A haiku is a traditional Japanese poem with a 5-7-5 syllable structure.} additionalProperties: null }, class Hit { id: rec4 score: 0.730563 fields: {category=biology, chunk_text=The mitochondrion is often called the powerhouse of the cell.} additionalProperties: null }, class Hit { id: rec7 score: 0.73037535 fields: {category=history, chunk_text=The Great Wall of China was built to protect against invasions.} additionalProperties: null }, class Hit { id: rec32 score: 0.72860974 fields: {category=biology, chunk_text=Elephants have excellent memory and strong social bonds.} additionalProperties: null }, class Hit { id: rec47 score: 0.7285921 fields: {category=history, chunk_text=The Industrial Revolution transformed manufacturing and transportation.} additionalProperties: null }] additionalProperties: null } usage: class SearchUsage { readUnits: 6 embedTotalTokens: 13 rerankUnits: null additionalProperties: null } additionalProperties: null } ``` ```json Go [expandable] { "result": { "hits": [ { "_id": "rec17", "_score": 0.24442708, "fields": { "category": "history", "chunk_text": "The Pyramids of Giza are among the Seven Wonders of the Ancient World." } }, { "_id": "rec38", "_score": 0.1876694, "fields": { "category": "history", "chunk_text": "The Taj Mahal is a mausoleum built by Emperor Shah Jahan." } }, { "_id": "rec5", "_score": 0.18504046, "fields": { "category": "literature", "chunk_text": "Shakespeare wrote many famous plays, including Hamlet and Macbeth." } }, { "_id": "rec15", "_score": 0.109251045, "fields": { "category": "art", "chunk_text": "Leonardo da Vinci painted the Mona Lisa." } }, { "_id": "rec50", "_score": 0.098952696, "fields": { "category": "energy", "chunk_text": "Renewable energy sources include wind, solar, and hydroelectric power." } }, { "_id": "rec26", "_score": 0.085251465, "fields": { "category": "history", "chunk_text": "Rome was once the center of a vast empire." } }, { "_id": "rec47", "_score": 0.07533597, "fields": { "category": "history", "chunk_text": "The Industrial Revolution transformed manufacturing and transportation." } }, { "_id": "rec7", "_score": 0.06859385, "fields": { "category": "history", "chunk_text": "The Great Wall of China was built to protect against invasions." } }, { "_id": "rec1", "_score": 0.06831257, "fields": { "category": "history", "chunk_text": "The Eiffel Tower was completed in 1889 and stands in Paris, France." } }, { "_id": "rec3", "_score": 0.06689669, "fields": { "category": "science", "chunk_text": "Albert Einstein developed the theory of relativity." } } ] }, "usage": { "read_units": 6, "embed_total_tokens": 8 } } ``` ```csharp C# [expandable] { "result": { "hits": [ { "_id": "rec17", "_score": 0.27985704, "fields": { "category": "history", "chunk_text": "The Pyramids of Giza are among the Seven Wonders of the Ancient World." } }, { "_id": "rec38", "_score": 0.18836586, "fields": { "category": "history", "chunk_text": "The Taj Mahal is a mausoleum built by Emperor Shah Jahan." } }, { "_id": "rec5", "_score": 0.18140909, "fields": { "category": "literature", "chunk_text": "Shakespeare wrote many famous plays, including Hamlet and Macbeth." } }, { "_id": "rec15", "_score": 0.09603156, "fields": { "category": "art", "chunk_text": "Leonardo da Vinci painted the Mona Lisa." } }, { "_id": "rec50", "_score": 0.091406636, "fields": { "category": "energy", "chunk_text": "Renewable energy sources include wind, solar, and hydroelectric power." } }, { "_id": "rec1", "_score": 0.0828001, "fields": { "category": "history", "chunk_text": "The Eiffel Tower was completed in 1889 and stands in Paris, France." } }, { "_id": "rec26", "_score": 0.081794746, "fields": { "category": "history", "chunk_text": "Rome was once the center of a vast empire." } }, { "_id": "rec7", "_score": 0.078153394, "fields": { "category": "history", "chunk_text": "The Great Wall of China was built to protect against invasions." } }, { "_id": "rec47", "_score": 0.06604649, "fields": { "category": "history", "chunk_text": "The Industrial Revolution transformed manufacturing and transportation." } }, { "_id": "rec21", "_score": 0.056735568, "fields": { "category": "history", "chunk_text": "The Statue of Liberty was a gift from France to the United States." } } ] }, "usage": { "read_units": 6, "embed_total_tokens": 8 } } ``` ## 6. Rerank results To get a more accurate ranking, search again but this time [rerank the initial results](/guides/search/rerank-results) based on their relevance to the query. ```python Python {10-14} # Search the dense index and rerank results reranked_results = dense_index.search( namespace="example-namespace", query={ "top_k": 10, "inputs": { 'text': query } }, rerank={ "model": "bge-reranker-v2-m3", "top_n": 10, "rank_fields": ["chunk_text"] } ) # Print the reranked results for hit in reranked_results['result']['hits']: print(f"id: {hit['_id']}, score: {round(hit['_score'], 2)}, text: {hit['fields']['chunk_text']}, category: {hit['fields']['category']}") ``` ```javascript JavaScript {7-11} // Search the dense index and rerank results const rerankedResults = await index.searchRecords({ query: { topK: 10, inputs: { text: query }, }, rerank: { model: 'bge-reranker-v2-m3', topN: 10, rankFields: ['chunk_text'], }, }); // Print the reranked results rerankedResults.result.hits.forEach(hit => { console.log(`id: ${hit.id}, score: ${hit.score.toFixed(2)}, text: ${hit.fields.chunk_text}, category: ${hit.fields.category}`); }); ``` ```java Java {9} // Add to the Quickstart class: // Define the rerank parameters ListrankFields = new ArrayList<>(); rankFields.add("chunk_text"); SearchRecordsRequestRerank rerank = new SearchRecordsRequestRerank() .query(query) .model("bge-reranker-v2-m3") .topN(10) .rankFields(rankFields); // Search the dense index and rerank results SearchRecordsResponse recordsResponseReranked = index.searchRecordsByText(query, "example-namespace", fields, 10, null, rerank ); // Print the reranked results System.out.println(recordsResponseReranked); ``` ```go Go {11-15} // Add to the main function: // Search the dense index and rerank results topN := int32(10) resReranked, err := idxConnection.SearchRecords(ctx, &pinecone.SearchRecordsRequest{ Query: pinecone.SearchRecordsQuery{ TopK: 10, Inputs: &map[string]interface{}{ "text": query, }, }, Rerank: &pinecone.SearchRecordsRerank{ Model: "bge-reranker-v2-m3", TopN: &topN, RankFields: []string{"chunk_text"}, }, }) if err != nil { log.Fatalf("Failed to search records: %v", err) } fmt.Printf(prettifyStruct(resReranked)) ``` ```csharp C# {12-17} // Search the dense index and rerank results var responseReranked = await index.SearchRecordsAsync( "example-namespace", new SearchRecordsRequest { Query = new SearchRecordsRequestQuery { TopK = 10, Inputs = new Dictionary { { "text", "Famous historical structures and monuments" } }, }, Fields = ["category", "chunk_text"], Rerank = new SearchRecordsRequestRerank { Model = "bge-reranker-v2-m3", TopN = 10, RankFields = ["chunk_text"], } } ); Console.WriteLine(responseReranked); ``` Notice that all of the most relevant results about historical structures and monuments are now ranked highest. ```console Python id: rec1 | score: 0.11 | category: history | text: The Eiffel Tower was completed in 1889 and stands in Paris, France. id: rec38 | score: 0.06 | category: history | text: The Taj Mahal is a mausoleum built by Emperor Shah Jahan. id: rec7 | score: 0.06 | category: history | text: The Great Wall of China was built to protect against invasions. id: rec17 | score: 0.02 | category: history | text: The Pyramids of Giza are among the Seven Wonders of the Ancient World. id: rec26 | score: 0.01 | category: history | text: Rome was once the center of a vast empire. id: rec15 | score: 0.01 | category: art | text: Leonardo da Vinci painted the Mona Lisa. id: rec5 | score: 0.0 | category: literature | text: Shakespeare wrote many famous plays, including Hamlet and Macbeth. id: rec47 | score: 0.0 | category: history | text: The Industrial Revolution transformed manufacturing and transportation. id: rec50 | score: 0.0 | category: energy | text: Renewable energy sources include wind, solar, and hydroelectric power. id: rec3 | score: 0.0 | category: science | text: Albert Einstein developed the theory of relativity. ``` ```console JavaScript id: rec1, score: 0.11, text: The Eiffel Tower was completed in 1889 and stands in Paris, France., category: history id: rec38, score: 0.06, text: The Taj Mahal is a mausoleum built by Emperor Shah Jahan., category: history id: rec7, score: 0.06, text: The Great Wall of China was built to protect against invasions., category: history id: rec17, score: 0.02, text: The Pyramids of Giza are among the Seven Wonders of the Ancient World., category: history id: rec26, score: 0.01, text: Rome was once the center of a vast empire., category: history id: rec15, score: 0.01, text: Leonardo da Vinci painted the Mona Lisa., category: art id: rec5, score: 0.00, text: Shakespeare wrote many famous plays, including Hamlet and Macbeth., category: literature id: rec47, score: 0.00, text: The Industrial Revolution transformed manufacturing and transportation., category: history id: rec50, score: 0.00, text: Renewable energy sources include wind, solar, and hydroelectric power., category: energy id: rec3, score: 0.00, text: Albert Einstein developed the theory of relativity., category: science ``` ```java Java [expandable] class SearchRecordsResponse { result: class SearchRecordsResponseResult { hits: [class Hit { id: rec1 score: 0.10687689 fields: {category=history, chunk_text=The Eiffel Tower was completed in 1889 and stands in Paris, France.} additionalProperties: null }, class Hit { id: rec38 score: 0.06418265 fields: {category=history, chunk_text=The Taj Mahal is a mausoleum built by Emperor Shah Jahan.} additionalProperties: null }, class Hit { id: rec7 score: 0.062445287 fields: {category=history, chunk_text=The Great Wall of China was built to protect against invasions.} additionalProperties: null }, class Hit { id: rec17 score: 0.0153063545 fields: {category=history, chunk_text=The Pyramids of Giza are among the Seven Wonders of the Ancient World.} additionalProperties: null }, class Hit { id: rec26 score: 0.010652511 fields: {category=history, chunk_text=Rome was once the center of a vast empire.} additionalProperties: null }, class Hit { id: rec5 score: 3.194182E-5 fields: {category=literature, chunk_text=Shakespeare wrote many famous plays, including Hamlet and Macbeth.} additionalProperties: null }, class Hit { id: rec47 score: 1.7502925E-5 fields: {category=history, chunk_text=The Industrial Revolution transformed manufacturing and transportation.} additionalProperties: null }, class Hit { id: rec32 score: 1.631454E-5 fields: {category=biology, chunk_text=Elephants have excellent memory and strong social bonds.} additionalProperties: null }, class Hit { id: rec4 score: 1.6187581E-5 fields: {category=biology, chunk_text=The mitochondrion is often called the powerhouse of the cell.} additionalProperties: null }, class Hit { id: rec45 score: 1.6061611E-5 fields: {category=literature, chunk_text=A haiku is a traditional Japanese poem with a 5-7-5 syllable structure.} additionalProperties: null }] additionalProperties: null } usage: class SearchUsage { readUnits: 6 embedTotalTokens: 13 rerankUnits: 1 additionalProperties: null } additionalProperties: null } ``` ```json Go [expandable] { "result": { "hits": [ { "_id": "rec1", "_score": 0.10743748, "fields": { "category": "history", "chunk_text": "The Eiffel Tower was completed in 1889 and stands in Paris, France." } }, { "_id": "rec38", "_score": 0.064535476, "fields": { "category": "history", "chunk_text": "The Taj Mahal is a mausoleum built by Emperor Shah Jahan." } }, { "_id": "rec7", "_score": 0.062445287, "fields": { "category": "history", "chunk_text": "The Great Wall of China was built to protect against invasions." } }, { "_id": "rec17", "_score": 0.0153063545, "fields": { "category": "history", "chunk_text": "The Pyramids of Giza are among the Seven Wonders of the Ancient World." } }, { "_id": "rec26", "_score": 0.010652511, "fields": { "category": "history", "chunk_text": "Rome was once the center of a vast empire." } }, { "_id": "rec15", "_score": 0.007876706, "fields": { "category": "art", "chunk_text": "Leonardo da Vinci painted the Mona Lisa." } }, { "_id": "rec5", "_score": 0.00003194182, "fields": { "category": "literature", "chunk_text": "Shakespeare wrote many famous plays, including Hamlet and Macbeth." } }, { "_id": "rec47", "_score": 0.000017502925, "fields": { "category": "history", "chunk_text": "The Industrial Revolution transformed manufacturing and transportation." } }, { "_id": "rec50", "_score": 0.00001631454, "fields": { "category": "energy", "chunk_text": "Renewable energy sources include wind, solar, and hydroelectric power." } }, { "_id": "rec3", "_score": 0.000015936621, "fields": { "category": "science", "chunk_text": "Albert Einstein developed the theory of relativity." } } ] }, "usage": { "read_units": 6, "embed_total_tokens": 8, "rerank_units": 1 } } ``` ```csharp C# [expandable] { "result": { "hits": [ { "_id": "rec1", "_score": 0.10687689, "fields": { "category": "history", "chunk_text": "The Eiffel Tower was completed in 1889 and stands in Paris, France." } }, { "_id": "rec38", "_score": 0.064535476, "fields": { "category": "history", "chunk_text": "The Taj Mahal is a mausoleum built by Emperor Shah Jahan." } }, { "_id": "rec7", "_score": 0.062445287, "fields": { "category": "history", "chunk_text": "The Great Wall of China was built to protect against invasions." } }, { "_id": "rec21", "_score": 0.018511046, "fields": { "category": "history", "chunk_text": "The Statue of Liberty was a gift from France to the United States." } }, { "_id": "rec17", "_score": 0.0153063545, "fields": { "category": "history", "chunk_text": "The Pyramids of Giza are among the Seven Wonders of the Ancient World." } }, { "_id": "rec26", "_score": 0.010652511, "fields": { "category": "history", "chunk_text": "Rome was once the center of a vast empire." } }, { "_id": "rec15", "_score": 0.007876706, "fields": { "category": "art", "chunk_text": "Leonardo da Vinci painted the Mona Lisa." } }, { "_id": "rec5", "_score": 0.00003194182, "fields": { "category": "literature", "chunk_text": "Shakespeare wrote many famous plays, including Hamlet and Macbeth." } }, { "_id": "rec47", "_score": 0.000017502925, "fields": { "category": "history", "chunk_text": "The Industrial Revolution transformed manufacturing and transportation." } }, { "_id": "rec50", "_score": 0.00001631454, "fields": { "category": "energy", "chunk_text": "Renewable energy sources include wind, solar, and hydroelectric power." } } ] }, "usage": { "read_units": 6, "embed_total_tokens": 8, "rerank_units": 1 } } ``` ## 7. Improve results [Reranking results](/guides/search/rerank-results) is one of the most effective ways to improve search accuracy and relevance, but there are many other techniques to consider. For example: * [Filtering by metadata](/guides/search/filter-by-metadata): When records contain additional metadata, you can limit the search to records matching a [filter expression](/guides/index-data/indexing-overview#metadata-filter-expressions). * [Hybrid search](/guides/search/hybrid-search): You can add [lexical search](/guides/search/lexical-search) to capture precise keyword matches (e.g., product SKUs, email addresses, domain-specific terms) in addition to semantic matches. * [Chunking strategies](https://www.pinecone.io/learn/chunking-strategies/): You can chunk your content in different ways to get better results. Consider factors like the length of the content, the complexity of queries, and how results will be used in your application. ## 8. Clean up When you no longer need your example index, delete it as follows: ```python Python # Delete the index pc.delete_index(index_name) ``` ```javascript JavaScript // Delete the index await pc.deleteIndex(indexName); ``` ```java Java // Add to the Quickstart class: // Delete the index pc.deleteIndex(indexName); ``` ```go Go // Add to the main function: // Delete the index err = pc.DeleteIndex(ctx, indexName) if err != nil { log.Fatalf("Failed to delete index: %v", err) } else { fmt.Println("Index \"%v\" deleted successfully", indexName) } ``` ```csharp C# // Delete the index await pinecone.DeleteIndexAsync(indexName); ``` For production indexes, consider [enabling deletion protection](/guides/manage-data/manage-indexes#configure-deletion-protection). ## Next steps Learn more about storing data in Pinecone Explore different forms of vector search. Find out how to improve performance {/* Try example notebooks and sample apps Comprehensive details about the Pinecone APIs, SDKs, utilities, and architecture. Pinecone's growing number of third-party integrations. */} # Check data freshness Source: https://docs.pinecone.io/guides/index-data/check-data-freshness Pinecone is eventually consistent, so there can be a slight delay before new or changed records are visible to queries. This page describes two ways of checking the data freshness of a Pinecone index: * To check if a serverless index queries reflect recent writes to the index, [check the log sequence number](#check-the-log-sequence-number). * To check whether an index contains recently inserted or deleted vectors, [verify the number of vectors in the index](#verify-vector-counts). ## Check the log sequence number This method is only available for serverless indexes through the [Database API](https://docs.pinecone.io/reference/api/2024-10/data-plane/upsert). ### Log sequence numbers When you make a write request to a serverless index namespace, Pinecone assigns a monotonically increasing log sequence number (LSN) to the write operation. The LSN reflects upserts as well as updates and deletes to that namespace. Writes to one namespace do not increase the LSN for other namespaces. You can use LSNs to verify that specific write operations are reflected in your query responses. If the LSN contained in the query response header is greater than or equal to the LSN of the relevant write operation, then that operation is reflected in the query response. If the LSN contained in the query response header is *greater than* the LSN of the relevant write operation, then subsequent operations are also reflected in the query response. Follow the steps below to compare the LSNs for a write and a subsequent query. ### 1. Get the LSN for a write operation Every time you modify records in your namespace, the HTTP response contains the LSN for the upsert. This is contained in a header called `x-pinecone-request-lsn`. The following example demonstrates how to get the LSN for an `upsert` request using the `curl` option `-i`. This option tells curl to include headers in the displayed response. Use the same method to get the LSN for an `update` or `delete` request. ```shell curl PINECONE_API_KEY="YOUR_API_KEY" INDEX_HOST="INDEX_HOST" curl -i "https://$INDEX_HOST/vectors/upsert" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "content-type: application/json" \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "vectors": [ { "id": "vec1", "values": [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8], } ], "namespace": "example-namespace" }' ``` The preceding request receives a response like the following example: ```shell curl HTTP/2 200 date: Wed, 21 Aug 2024 15:23:04 GMT content-type: application/json content-length: 66 x-pinecone-max-indexed-lsn: 4 x-pinecone-request-latency-ms: 1149 x-pinecone-request-id: 3687967458925971419 x-envoy-upstream-service-time: 1150 grpc-status: 0 server: envoy {"upsertedCount":1} ``` In the preceding example response, the value of `x-pinecone-max-indexed-lsn` is 4. This means that the index has performed 4 write operations since its creation. ### 2. Get the LSN for a query Every time you query your index, the HTTP response contains the LSN for the query. This is contained in a header called `x-pinecone-max-indexed-lsn`. By checking the LSN in your query results, you can confirm that the LSN is greater than or equal to the LSN of the relevant write operation, indicating that the results of that operation are present in the query results. The following example makes a `query` request to the index: ```shell PINECONE_API_KEY="YOUR_API_KEY" INDEX_HOST="INDEX_HOST" curl -i "https://$INDEX_HOST/query" \ -H "Api-Key: $PINECONE_API_KEY" \ -H 'Content-Type: application/json' \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "vector": [0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3], "namespace": "example-namespace", "topK": 3, "includeValues": true }' ``` The preceding request receives a response like the following example: ```shell HTTP/2 200 date: Wed, 21 Aug 2024 15:33:36 GMT content-type: application/json content-length: 66 x-pinecone-max-indexed-lsn: 5 x-pinecone-request-latency-ms: 40 x-pinecone-request-id: 6683088825552978933 x-envoy-upstream-service-time: 41 grpc-status: 0 server: envoy { "results":[], "matches":[ { "id":"vec1", "score":0.891132772, "values":[0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8], } ], "namespace":"example-namespace", "usage":{"readUnits":6} } ``` In the preceding example response, the value of `x-pinecone-max-indexed-lsn` is 5. ### 3. Compare LSNs for writes and queries If the LSN of a query is greater than or equal to the LSN for a write operation, then the results of the query reflect the results of the write operation. In [step 1](#1-get-the-lsn-for-a-write-operation), the LSN contained in the response headers is 4. In [step 2](#2-get-the-lsn-for-a-query), the LSN contained in the response headers is 5. 5 is greater than or equal to 4; therefore, the results of the query reflect the results of the upsert. However, this does not guarantee that the records upserted are still present or unmodified: the write operation with LSN of 5 may have updated or deleted these records, or upserted additional records. ## Verify record counts If you insert new records or delete records, the number of records in the index may change. This means that the record count for an index can indicate whether Pinecone has indexed your latest inserts and deletes: if the record count for the index matches the count you expect after inserting or deleting records, the index is probably up-to-date. However, this is not always true. For example, if you delete the same number of records that you insert, the expected record count may remain the same. Also, some write operations, such as updates to an index configuration or vector data values, do not change the number of records in the index. To verify that your index contains the number of records you expect, [view index stats](/reference/api/2024-10/data-plane/describeindexstats): ```python Python # pip install "pinecone[grpc]" from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key="YOUR_API_KEY") # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index index = pc.Index(host="INDEX_HOST") index.describe_index_stats() ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }) // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index const index = pc.index("INDEX_NAME", "INDEX_HOST") const stats = await index.describeIndexStats(); ``` ```java Java import io.pinecone.clients.Index; import io.pinecone.configs.PineconeConfig; import io.pinecone.configs.PineconeConnection; import io.pinecone.proto.DescribeIndexStatsResponse; public class DescribeIndexStatsExample { public static void main(String[] args) { PineconeConfig config = new PineconeConfig("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index config.setHost("INDEX_HOST"); PineconeConnection connection = new PineconeConnection(config); Index index = new Index(connection, "INDEX_NAME"); DescribeIndexStatsResponse indexStatsResponse = index.describeIndexStats(); System.out.println(indexStatsResponse); } } ``` ```go Go package main import ( "context" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "INDEX_HOST"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v", err) } stats, err := idxConnection.DescribeIndexStats(ctx) if err != nil { log.Fatalf("Failed to describe index \"%v\": %v", idx.Name, err) } else { fmt.Printf("%+v", *stats) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index var index = pinecone.Index(host: "INDEX_HOST"); var indexStatsResponse = await index.DescribeIndexStatsAsync(new DescribeIndexStatsRequest()); Console.WriteLine(indexStatsResponse); ``` ```shell curl # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index PINECONE_API_KEY="YOUR_API_KEY" INDEX_HOST="INDEX_HOST" curl -X POST "https://$INDEX_HOST/describe_index_stats" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" ``` The response will look like this: ```Python Python {'dimension': 1024, 'index_fullness': 8e-05, 'namespaces': {'example-namespace1': {'vector_count': 4}, 'example-namespace2': {'vector_count': 4}}, 'total_vector_count': 8} ``` ```JavaScript JavaScript Returns: { namespaces: { example-namespace1: { recordCount: 4 }, example-namespace2: { recordCount: 4 } }, dimension: 1024, indexFullness: 0.00008, totalRecordCount: 8 } // Note: the value of totalRecordCount is the same as total_vector_count. ``` ```java Java namespaces { key: "example-namespace1" value { vector_count: 4 } } namespaces { key: "example-namespace2" value { vector_count: 4 } } dimension: 1024 total_vector_count: 8 ``` ```go Go { "dimension": 1024, "index_fullness": 0, "total_vector_count": 8, "namespaces": { "example-namespace1": { "vector_count": 4 }, "example-namespace2": { "vector_count": 4 } } } ``` ```csharp C# { "namespaces": { "example-namespace1": { "vectorCount": 4 }, "example-namespace2": { "vectorCount": 4 } }, "dimension": 1024, "indexFullness": 0, "totalVectorCount": 8 } ``` ```shell curl { "namespaces": { "example-namespace1": { "vectorCount": 4 }, "example-namespace2": { "vectorCount": 4 } }, "dimension": 1024, "indexFullness": 0.00008, "totalVectorCount": 8 } ``` # Create a serverless index Source: https://docs.pinecone.io/guides/index-data/create-an-index This page shows you how to create a dense or sparse serverless index. * **Dense indexes** store dense vectors, which are numerical representations of the meaning and relationships of text, images, or other types of data. You use dense indexes for [semantic search](/guides/search/semantic-search) or in combination with sparse indexes for [hybrid search](/guides/search/hybrid-search). * **Sparse indexes** store sparse vectors, which are numerical representations of the words or phrases in a document. You use sparse indexes for [lexical search](/guides/search/lexical-search), or in combination with dense indexes for [hybrid search](/guides/search/hybrid-search). You can create an index using the [Pinecone console](https://app.pinecone.io/organizations/-/projects/-/create-index/serverless). ## Create a dense index You can create a dense index with [integrated vector embedding](/guides/index-data/indexing-overview#integrated-embedding) or a dense index for storing vectors generated with an external embedding model. ### Integrated embedding Indexes with integrated embedding do not support [updating](/guides/manage-data/update-data) or [importing](/guides/index-data/import-data) with text. If you want to upsert and search with source text and have Pinecone convert it to dense vectors automatically, [create a dense index with integrated embedding](/reference/api/2025-01/control-plane/create_for_model) as follows: * Provide a `name` for the index. * Set `cloud` and `region` to the [cloud and region](/guides/index-data/create-an-index#cloud-regions) where the index should be deployed. * Set `embed.model` to one of [Pinecone's hosted embedding models](/guides/index-data/create-an-index#embedding-models). * Set `embed.field_map` to the name of the field in your source document that contains the data for embedding. Other parameters are optional. See the [API reference](/reference/api/2025-01/control-plane/create_for_model) for details. ```python Python from pinecone import Pinecone pc = Pinecone(api_key="YOUR_API_KEY") index_name = "integrated-dense-py" if not pc.has_index(index_name): pc.create_index_for_model( name=index_name, cloud="aws", region="us-east-1", embed={ "model":"llama-text-embed-v2", "field_map":{"text": "chunk_text"} } ) ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }); await pc.createIndexForModel({ name: 'integrated-dense-js', cloud: 'aws', region: 'us-east-1', embed: { model: 'llama-text-embed-v2', fieldMap: { text: 'chunk_text' }, }, waitUntilReady: true, }); ``` ```java Java import io.pinecone.clients.Pinecone; import org.openapitools.db_control.client.ApiException; import org.openapitools.db_control.client.model.CreateIndexForModelRequest; import org.openapitools.db_control.client.model.CreateIndexForModelRequestEmbed; import org.openapitools.db_control.client.model.DeletionProtection; import java.util.HashMap; import java.util.Map; public class CreateIntegratedIndex { public static void main(String[] args) throws ApiException { Pinecone pc = new Pinecone.Builder("YOUR_API_KEY").build(); String indexName = "integrated-dense-java"; String region = "us-east-1"; HashMap fieldMap = new HashMap<>(); fieldMap.put("text", "chunk_text"); CreateIndexForModelRequestEmbed embed = new CreateIndexForModelRequestEmbed() .model("llama-text-embed-v2") .fieldMap(fieldMap); Map tags = new HashMap<>(); tags.put("environment", "development"); pc.createIndexForModel( indexName, CreateIndexForModelRequest.CloudEnum.AWS, region, embed, DeletionProtection.DISABLED, tags ); } } ``` ```go Go package main import ( "context" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } indexName := "integrated-dense-go" DeletionProtection: &deletionProtection, index, err := pc.CreateIndexForModel(ctx, &pinecone.CreateIndexForModelRequest{ Name: indexName, Cloud: pinecone.Aws, Region: "us-east-1", Embed: pinecone.CreateIndexForModelEmbed{ Model: "llama-text-embed-v2", FieldMap: map[string]interface{}{"text": "chunk_text"}, }, DeletionProtection: &deletionProtection, Tags: &pinecone.IndexTags{ "environment": "development" }, }) if err != nil { log.Fatalf("Failed to create serverless integrated index: %v", idx.Name) } else { fmt.Printf("Successfully created serverless integrated index: %v", idx.Name) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); var createIndexRequest = await pinecone.CreateIndexForModelAsync( new CreateIndexForModelRequest { Name = "integrated-dense-dotnet", Cloud = CreateIndexForModelRequestCloud.Aws, Region = "us-east-1", Embed = new CreateIndexForModelRequestEmbed { Model = "llama-text-embed-v2", FieldMap = new Dictionary() { { "text", "chunk_text" } } }, DeletionProtection = DeletionProtection.Disabled, Tags = new Dictionary { { "environment", "development" } } } ); ``` ```json curl PINECONE_API_KEY="YOUR_API_KEY" curl "https://api.pinecone.io/indexes/create-for-model" \ -H "Content-Type: application/json" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "name": "integrated-dense-curl", "cloud": "aws", "region": "us-east-1", "embed": { "model": "llama-text-embed-v2", "field_map": { "text": "chunk_text" } } }' ``` ### Bring your own vectors If you use an external embedding model to convert your data to dense vectors, use the [create a dense index](/reference/api/2025-01/control-plane/create_index) as follows: * Provide a `name` for the index. * Set the `vector_type` to `dense`. * Specify the `dimension` and similarity `metric` of the vectors you'll store in the index. This should match the dimension and metric supported by your embedding model. * Set `spec.cloud` and `spec.region` to the [cloud and region](/guides/index-data/create-an-index#cloud-regions) where the index should be deployed. For Python, you also need to import the `ServerlessSpec` class. Other parameters are optional. See the [API reference](/reference/api/2025-01/control-plane/create_index) for details. ```python Python from pinecone.grpc import PineconeGRPC as Pinecone from pinecone import ServerlessSpec pc = Pinecone(api_key="YOUR_API_KEY") index_name = "standard-dense-py" if not pc.has_index(index_name): pc.create_index( name=index_name, vector_type="dense", dimension=1536, metric="cosine", spec=ServerlessSpec( cloud="aws", region="us-east-1" ), deletion_protection="disabled", tags={ "environment": "development" } ) ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }); await pc.createIndex({ name: 'standard-dense-js', vectorType: 'dense', dimension: 1536, metric: 'cosine', spec: { serverless: { cloud: 'aws', region: 'us-east-1' } }, deletionProtection: 'disabled', tags: { environment: 'development' }, }); ``` ```java Java import io.pinecone.clients.Pinecone; import org.openapitools.db_control.client.model.IndexModel; import org.openapitools.db_control.client.model.DeletionProtection; import java.util.HashMap; public class CreateServerlessIndexExample { public static void main(String[] args) { Pinecone pc = new Pinecone.Builder("YOUR_API_KEY").build(); String indexName = "standard-dense-java"; String cloud = "aws"; String region = "us-east-1"; String vectorType = "dense"; Map tags = new HashMap<>(); tags.put("environment", "development"); pc.createServerlessIndex( indexName, "cosine", 1536, cloud, region, DeletionProtection.DISABLED, tags, vectorType ); } } ``` ```go Go package main import ( "context" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // Serverless index indexName := "standard-dense-go" vectorType := "dense" dimension := int32(1536) metric := pinecone.Cosine deletionProtection := pinecone.DeletionProtectionDisabled idx, err := pc.CreateServerlessIndex(ctx, &pinecone.CreateServerlessIndexRequest{ Name: indexName, VectorType: &vectorType, Dimension: &dimension, Metric: &metric, Cloud: pinecone.Aws, Region: "us-east-1", DeletionProtection: &deletionProtection, Tags: &pinecone.IndexTags{ "environment": "development" }, }) if err != nil { log.Fatalf("Failed to create serverless index: %v", idx.Name) } else { fmt.Printf("Successfully created serverless index: %v", idx.Name) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); var createIndexRequest = await pinecone.CreateIndexAsync(new CreateIndexRequest { Name = "standard-dense-dotnet", VectorType = VectorType.Dense, Dimension = 1536, Metric = MetricType.Cosine, Spec = new ServerlessIndexSpec { Serverless = new ServerlessSpec { Cloud = ServerlessSpecCloud.Aws, Region = "us-east-1" } }, DeletionProtection = DeletionProtection.Disabled, Tags = new Dictionary { { "environment", "development" } } }); ``` ```shell curl PINECONE_API_KEY="YOUR_API_KEY" curl -s "https://api.pinecone.io/indexes" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "name": "standard-dense-curl", "vector_type": "dense", "dimension": 1536, "metric": "cosine", "spec": { "serverless": { "cloud": "aws", "region": "us-east-1" } }, "tags": { "environment": "development" }, "deletion_protection": "disabled" }' ``` ## Create a sparse index This feature is in [public preview](/release-notes/feature-availability). You can create a dense index with [integrated vector embedding](/guides/index-data/indexing-overview#integrated-embedding) or a dense index for storing vectors generated with an external embedding model. ### Integrated embedding If you want to upsert and search with source text and have Pinecone convert it to sparse vectors automatically, [create a sparse index with integrated embedding](/reference/api/2025-01/control-plane/create_for_model) as follows: * Provide a `name` for the index. * Set `cloud` and `region` to the [cloud and region](/guides/index-data/create-an-index#cloud-regions) where the index should be deployed. * Set `embed.model` to one of [Pinecone's hosted sparse embedding models](/guides/index-data/create-an-index#embedding-models). * Set `embed.field_map` to the name of the field in your source document that contains the text for embedding. Other parameters are optional. See the [API reference](/reference/api/2025-01/control-plane/create_for_model) for details. ```python Python from pinecone import Pinecone pc = Pinecone(api_key="YOUR_API_KEY") index_name = "integrated-sparse-py" if not pc.has_index(index_name): pc.create_index_for_model( name=index_name, cloud="aws", region="us-east-1", embed={ "model":"pinecone-sparse-english-v0", "field_map":{"text": "chunk_text"} } ) ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }); await pc.createIndexForModel({ name: 'integrated-sparse-js', cloud: 'aws', region: 'us-east-1', embed: { model: 'pinecone-sparse-english-v0', fieldMap: { text: 'chunk_text' }, }, waitUntilReady: true, }); ``` ```java Java import io.pinecone.clients.Pinecone; import org.openapitools.db_control.client.ApiException; import org.openapitools.db_control.client.model.CreateIndexForModelRequest; import org.openapitools.db_control.client.model.CreateIndexForModelRequestEmbed; import org.openapitools.db_control.client.model.DeletionProtection; import java.util.HashMap; import java.util.Map; public class CreateIntegratedIndex { public static void main(String[] args) throws ApiException { Pinecone pc = new Pinecone.Builder("YOUR_API_KEY").build(); String indexName = "integrated-sparse-java"; String region = "us-east-1"; HashMap fieldMap = new HashMap<>(); fieldMap.put("text", "chunk_text"); CreateIndexForModelRequestEmbed embed = new CreateIndexForModelRequestEmbed() .model("pinecone-sparse-english-v0") .fieldMap(fieldMap); Map tags = new HashMap<>(); tags.put("environment", "development"); pc.createIndexForModel( indexName, CreateIndexForModelRequest.CloudEnum.AWS, region, embed, DeletionProtection.DISABLED, tags ); } } ``` ```go Go package main import ( "context" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } indexName := "integrated-sparse-go" DeletionProtection: &deletionProtection, index, err := pc.CreateIndexForModel(ctx, &pinecone.CreateIndexForModelRequest{ Name: indexName, Cloud: pinecone.Aws, Region: "us-east-1", Embed: pinecone.CreateIndexForModelEmbed{ Model: "pinecone-sparse-english-v0", FieldMap: map[string]interface{}{"text": "chunk_text"}, }, DeletionProtection: &deletionProtection, Tags: &pinecone.IndexTags{ "environment": "development" }, }) if err != nil { log.Fatalf("Failed to create serverless integrated index: %v", idx.Name) } else { fmt.Printf("Successfully created serverless integrated index: %v", idx.Name) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); var createIndexRequest = await pinecone.CreateIndexForModelAsync( new CreateIndexForModelRequest { Name = "integrated-sparse-dotnet", Cloud = CreateIndexForModelRequestCloud.Aws, Region = "us-east-1", Embed = new CreateIndexForModelRequestEmbed { Model = "pinecone-sparse-english-v0", FieldMap = new Dictionary() { { "text", "chunk_text" } } }, DeletionProtection = DeletionProtection.Disabled, Tags = new Dictionary { { "environment", "development" } } } ); ``` ```shell curl PINECONE_API_KEY="YOUR_API_KEY" curl "https://api.pinecone.io/indexes/create-for-model" \ -H "Content-Type: application/json" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "name": "integrated-sparse-curl", "cloud": "aws", "region": "eu-west-1", "embed": { "model": "pinecone-sparse-english-v0", "field_map": { "text": "chunk_text" } } }' ``` ### Bring your own vectors If you use an external embedding model to convert your data to sparse vectors, [create a sparse index](/reference/api/2025-01/control-plane/create_index) as follows: * Provide a `name` for the index. * Set the `vector_type` to `sparse`. * Set the distance `metric` to `dotproduct`. Sparse indexes do not support other [distance metrics](/guides/index-data/indexing-overview#distance-metrics). * Set `spec.cloud` and `spec.region` to the cloud and region where the index should be deployed. Other parameters are optional. See the [API reference](/reference/api/2025-01/control-plane/create_index) for details. ```python Python from pinecone import Pinecone, ServerlessSpec pc = Pinecone(api_key="YOUR_API_KEY") index_name = "standard-sparse-py" if not pc.has_index(index_name): pc.create_index( name=index_name, vector_type="sparse", metric="dotproduct", spec=ServerlessSpec(cloud="aws", region="eu-west-1") ) ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }); await pc.createIndex({ name: 'standard-sparse-js', vectorType: 'sparse', metric: 'dotproduct', spec: { serverless: { cloud: 'aws', region: 'us-east-1' }, }, }); ``` ```java Java import io.pinecone.clients.Pinecone; import org.openapitools.db_control.client.model.DeletionProtection; import java.util.*; public class SparseIndex { public static void main(String[] args) throws InterruptedException { // Instantiate Pinecone class Pinecone pinecone = new Pinecone.Builder("YOUR_API_KEY").build(); // Create sparse Index String indexName = "standard-sparse-java"; String cloud = "aws"; String region = "us-east-1"; String vectorType = "sparse"; Map tags = new HashMap<>(); tags.put("env", "test"); pinecone.createSparseServelessIndex(indexName, cloud, region, DeletionProtection.DISABLED, tags, vectorType); } } ``` ```go Go package main import ( "context" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } indexName := "standard-sparse-go" vectorType := "sparse" metric := pinecone.Dotproduct deletionProtection := pinecone.DeletionProtectionDisabled idx, err := pc.CreateServerlessIndex(ctx, &pinecone.CreateServerlessIndexRequest{ Name: indexName, Metric: &metric, VectorType: &vectorType, Cloud: pinecone.Aws, Region: "us-east-1", DeletionProtection: &deletionProtection, }) if err != nil { log.Fatalf("Failed to create serverless index: %v", idx.Name) } else { fmt.Printf("Successfully created serverless index: %v", idx.Name) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); var createIndexRequest = await pinecone.CreateIndexAsync(new CreateIndexRequest { Name = "standard-sparse-dotnet", VectorType = VectorType.Sparse, Metric = MetricType.Dotproduct, Spec = new ServerlessIndexSpec { Serverless = new ServerlessSpec { Cloud = ServerlessSpecCloud.Aws, Region = "us-east-1" } } }); ``` ```shell curl PINECONE_API_KEY="YOUR_API_KEY" curl -s "https://api.pinecone.io/indexes" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "name": "standard-sparse-curl", "vector_type": "sparse", "metric": "dotproduct", "spec": { "serverless": { "cloud": "aws", "region": "eu-west-1" } } }' ``` ## Create an index from a backup You can create a dense or sparse index from a backup. For more details, see [Restore an index](/guides/manage-data/restore-an-index). ## Index options ### Cloud regions When creating an index, you must choose the cloud and region where you want the index to be hosted. The following table lists the available public clouds and regions and the plans that support them: | Cloud | Region | [Supported plans](https://www.pinecone.io/pricing/) | [Availability phase](/release-notes/feature-availability) | | ------- | ---------------------------- | --------------------------------------------------- | --------------------------------------------------------- | | `aws` | `us-east-1` (Virginia) | Starter, Standard, Enterprise | General availability | | `aws` | `us-west-2` (Oregon) | Standard, Enterprise | General availability | | `aws` | `eu-west-1` (Ireland) | Standard, Enterprise | General availability | | `gcp` | `us-central1` (Iowa) | Standard, Enterprise | General availability | | `gcp` | `europe-west4` (Netherlands) | Standard, Enterprise | General availability | | `azure` | `eastus2` (Virginia) | Standard, Enterprise | General availability | The cloud and region cannot be changed after a serverless index is created. On the free Starter plan, you can create serverless indexes in the `us-east-1` region of AWS only. To create indexes in other regions, [upgrade your plan](/guides/organizations/manage-billing/manage-your-billing-plan). ### Similarity metrics When creating a dense index, you can choose from the following similarity metrics. For the most accurate results, choose the similarity metric used to train the embedding model for your vectors. For more information, see [Vector Similarity Explained](https://www.pinecone.io/learn/vector-similarity/). [Sparse indexes](#sparse-indexes) must use the `dotproduct` metric. Querying indexes with this metric returns a similarity score equal to the squared Euclidean distance between the result and query vectors. This metric calculates the square of the distance between two data points in a plane. It is one of the most commonly used distance metrics. For an example, see our [IT threat detection example](https://colab.research.google.com/github/pinecone-io/examples/blob/master/docs/it-threat-detection.ipynb). When you use `metric='euclidean'`, the most similar results are those with the **lowest similarity score**. This is often used to find similarities between different documents. The advantage is that the scores are normalized to \[-1,1] range. For an example, see our [generative question answering example](https://colab.research.google.com/github/pinecone-io/examples/blob/master/docs/gen-qa-openai.ipynb). This is used to multiply two vectors. You can use it to tell us how similar the two vectors are. The more positive the answer is, the closer the two vectors are in terms of their directions. For an example, see our [semantic search example](https://colab.research.google.com/github/pinecone-io/examples/blob/master/docs/semantic-search.ipynb). ### Embedding models [Dense vectors](/guides/get-started/glossary#dense-vector) and [sparse vectors](/guides/get-started/glossary#sparse-vector) are the basic units of data in Pinecone and what Pinecone was specially designed to store and work with. Dense vectors represents the semantics of data such as text, images, and audio recordings, while sparse vectors represent documents or queries in a way that captures keyword information. To transform data into vector format, you use an embedding model. Pinecone hosts several embedding models so it's easy to manage your vector storage and search process on a single platform. You can use a hosted model to embed your data as an integrated part of upserting and querying, or you can use a hosted model to embed your data as a standalone operation. The following embedding models are hosted by Pinecone. To understand how cost is calculated for embedding, see [Understanding cost](/guides/manage-cost/understanding-cost#embed). To get model details via the API, see [List models](/reference/api/2025-04/inference/list_models) and [Describe a model](/reference/api/2025-04/inference/describe_model). [`multilingual-e5-large`](/models/multilingual-e5-large) is an efficient dense embedding model trained on a mixture of multilingual datasets. It works well on messy data and short queries expected to return medium-length passages of text (1-2 paragraphs). **Details** * Vector type: Dense * Modality: Text * Dimension: 1024 * Recommended similarity metric: Cosine * Max input tokens per sequence: 507 * Max sequences per batch: 96 **Parameters** The `multilingual-e5-large` model supports the following parameters: | Parameter | Type | Required/Optional | Description | Default | | :----------- | :----- | :---------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------ | | `input_type` | string | Required | The type of input data. Accepted values: `query` or `passage`. | | | `truncate` | string | Optional | How to handle inputs longer than those supported by the model. Accepted values: `END` or `NONE`.

`END` truncates the input sequence at the input token limit. `NONE` returns an error when the input exceeds the input token limit. | `END` | **Quotas** Quotas are defined at the project level and vary based on [pricing plan](https://www.pinecone.io/pricing/) and input type. | Input type | Starter plan | Paid plans | | :--------- | :--------------------- | :------------------------- | | `passage` | 250k tokens per minute | 1M tokens per minute | | `query` | 50k tokens per minute | 250k tokens per minute | | Combined | 5M tokens per month | Unlimited tokens per month | [`llama-text-embed-v2`](/models/llama-text-embed-v2) is a high-performance dense embedding model optimized for text retrieval and ranking tasks. It is trained on a diverse range of text corpora and provides strong performance on longer passages and structured documents. **Details** * Vector type: Dense * Modality: Text * Dimension: 1024 (default), 2048, 768, 512, 384 * Recommended similarity metric: Cosine * Max input tokens per sequence: 2048 * Max sequences per batch: 96 **Parameters** The `llama-text-embed-v2` model supports the following parameters: | Parameter | Type | Required/Optional | Description | Default | | :----------- | :------ | :---------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------ | | `input_type` | string | Required | The type of input data. Accepted values: `query` or `passage`. | | | `truncate` | string | Optional | How to handle inputs longer than those supported by the model. Accepted values: `END` or `NONE`.

`END` truncates the input sequence at the input token limit. `NONE` returns an error when the input exceeds the input token limit. | `END` | | `dimension` | integer | Optional | Dimension of the vector to return. | 1024 | **Quotas** Quotas are defined at the project level and vary based on [pricing plan](https://www.pinecone.io/pricing/) and input type. | Input type | Starter plan | Paid plans | | :--------- | :--------------------- | :------------------------- | | Combined | 250k tokens per minute | 1M tokens per minute | | Combined | 5M tokens per month | Unlimited tokens per month | [`pinecone-sparse-english-v0`](/models/pinecone-sparse-english-v0) is a sparse embedding model for converting text to [sparse vectors](/guides/get-started/glossary#sparse-vector) for keyword or hybrid semantic/keyword search. Built on the innovations of the [DeepImpact architecture](https://arxiv.org/pdf/2104.12016), the model directly estimates the lexical importance of tokens by leveraging their context, unlike traditional retrieval models like BM25, which rely solely on term frequency. **Details** * Vector type: Sparse * Modality: Text * Recommended similarity metric: Dotproduct * Max input tokens per sequence: 512 * Max sequences per batch: 96 **Parameters** The `pinecone-sparse-english-v0` model supports the following parameters: | Parameter | Type | Required/Optional | Description | Default | | :-------------- | :------ | :---------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------ | | `input_type` | string | Required | The type of input data. Accepted values: `query` or `passage`. | | | `truncate` | string | Optional | How to handle inputs longer than those supported by the model. Accepted values: `END` or `NONE`.

`END` truncates the input sequence at the input token limit. `NONE` returns an error when the input exceeds the input token limit. | `END` | | `return_tokens` | boolean | Optional | Whether to return the string tokens. | `False` | **Quotas** Quotas are defined at the project level and vary based on [pricing plan](https://www.pinecone.io/pricing/). | Limit type | Starter plan | Paid plans | | :---------------- | :----------- | :--------- | | Tokens per minute | 250K | 1M | | Tokens per month | 5M | Unlimited | # Data ingestion overview Source: https://docs.pinecone.io/guides/index-data/data-ingestion-overview To ingest data into an index, you can [import from object storage](#import-from-object-storage) or use the [upsert](#upsert) operation. To control costs when ingesting large datasets (10,000,000+ records), use [import](/guides/index-data/import-data) instead of upsert. ## Import from object storage [Importing from object storage](/guides/index-data/import-data) is most efficient and cost-effective method to load large numbers of records into an index. You store your data as Parquet files in object storage, integrate your object storage with Pinecone, and then start an asynchronous, long-running operation that imports and indexes your records. This feature is in [public preview](/release-notes/feature-availability) and available only on [Standard and Enterprise plans](https://www.pinecone.io/pricing/). ### Import limitations * Import does not support [integrated embedding](/guides/index-data/indexing-overview#vector-embedding). * Import only supports AWS S3 as a data source. * You cannot import data from S3 Express One Zone storage. * You cannot import data into existing namespaces. * Each import request can import up 1TB of data into a maximum of 100 namespaces. Note that you cannot import more than 10GB per file and no more than 100,000 files per import. * Each import will take at least 10 minutes to complete. ### Import cost * To understand how cost is calculated for imports, see [Understanding cost](/guides/manage-cost/understanding-cost#imports-and-storage). * For up-to-date pricing information, see [Pricing](https://www.pinecone.io/pricing/). ## Upsert For ongoing ingestion into an index, either one record at a time or in batches, use the [upsert](/guides/index-data/upsert-data) operation. [Batch uperting](/guides/index-data/upsert-data#upsert-in-batches) can improve throughput performance and is a good option for larger numbers of records (up to 1000 per batch) if you cannot work around import's current [limitations](#limitations). ### Upsert limits | Metric | Limit | | :------------------------------------- | :------------------ | | Max upsert size | 2MB or 1000 records | | Max metadata size per record | 40 KB | | Max length for a record ID | 512 characters | | Max dimensionality for dense vectors | 20,000 | | Max non-zero values for sparse vectors | 2048 | | Max dimensionality for sparse vectors | 4.2 billion | ### Upsert cost * To understand how cost is calculated for upserts, see [Understanding cost](/guides/manage-cost/understanding-cost#upsert). * For up-to-date pricing information, see [Pricing](https://www.pinecone.io/pricing/). ## Data freshness Pinecone is eventually consistent, so there can be a slight delay before new or changed records are visible to queries. You can view index stats to [check data freshness](/guides/index-data/check-data-freshness). # Implement multitenancy using namespaces Source: https://docs.pinecone.io/guides/index-data/implement-multitenancy [Multitenancy](https://en.wikipedia.org/wiki/Multitenancy) is a software architecture where a single instance of a system serves multiple customers, or tenants, while ensuring data isolation between them for privacy and security. This page shows you how to implement multitenancy in Pinecone using a **serverless index with one namespace per tenant**. ## How it works In Pinecone, an [index](/guides/index-data/indexing-overview) is the highest-level organizational unit of data, where you define the dimension of vectors to be stored in the index and the measure of similarity to be used when querying the index. Within an index, records are stored in [namespaces](/guides/index-data/indexing-overview#namespaces), and all [upserts](/guides/index-data/upsert-data), [queries](/guides/search/search-overview), and other [data plane operations](/reference/api/2024-10/data-plane) always target one namespace. This structure makes it easy to implement multitenancy. For example, for an AI-powered SaaS application where you need to isolate the data of each customer, you would assign each customer to a namespace and target their writes and queries to that namespace (diagram above). In cases where you have different workload patterns (e.g., RAG and semantic search), you would use a different index for each workload, with one namespace per customer in each index: * **Tenant isolation:** In the [serverless architecture](/reference/architecture/serverless-architecture), each namespace is stored separately, so using namespaces provides physical isolation of data between tenants/customers. * **No noisy neighbors:** Reads and writes always target a single namespace, so the behavior of one tenant/customer does not affect other tenants/customers. * **No maintenance effort:** Serverless indexes scale automatically based on usage; you don't configure or manage any compute or storage resources. * **Cost efficiency:** With serverless indexes, you pay only for the amount of data stored and operations performed. For queries in particular, the cost is partly based on the total number of records that must be scanned, so using namespaces can significantly reduce query costs. * **Simple tenant offboarding:** To offboard a tenant/customer, you just [delete the relevant namespace](/guides/manage-data/delete-data#delete-all-records-from-a-namespace). This is a lightweight and almost instant operation. ## 1. Create a serverless index Based on a [breakthrough architecture](/reference/architecture/serverless-architecture), serverless indexes scale automatically based on usage, and you pay only for the amount of data stored and operations performed. Combined with the isolation of tenant data using namespaces (next step), serverless indexes are ideal for multitenant use cases. To [create a serverless index](/guides/index-data/create-an-index#create-a-serverless-index), use the `spec` parameter to define the cloud and region where the index should be deployed. For Python, you also need to import the `ServerlessSpec` class. ```Python Python from pinecone.grpc import PineconeGRPC as Pinecone from pinecone import ServerlessSpec pc = Pinecone(api_key="YOUR_API_KEY") pc.create_index( name="multitenant-app", dimension=8, metric="cosine", spec=ServerlessSpec( cloud="aws", region="us-east-1" ) ) ``` ```JavaScript JavaScript import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }); await pc.createIndex({ name: 'multitenant-app', dimension: 8, metric: 'cosine', spec: { serverless: { cloud: 'aws', region: 'us-east-1' } } }); ``` ```java Java import io.pinecone.clients.Pinecone; public class CreateServerlessIndexExample { public static void main(String[] args) { Pinecone pc = new Pinecone.Builder("YOUR_API_KEY").build(); pc.createServerlessIndex("multitenant-app", "cosine", 8, "aws", "us-east-1"); } } ``` ```go Go package main import ( "context" "encoding/json" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func prettifyStruct(obj interface{}) string { bytes, _ := json.MarshalIndent(obj, "", " ") return string(bytes) } func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // Serverless index indexName := "multi-tenant-app" vectorType := "dense" dimension := int32(8) metric := pinecone.Cosine deletionProtection := pinecone.DeletionProtectionDisabled idx, err := pc.CreateServerlessIndex(ctx, &pinecone.CreateServerlessIndexRequest{ Name: indexName, VectorType: &vectorType, Dimension: &dimension, Metric: &metric, Cloud: pinecone.Aws, Region: "us-east-1", DeletionProtection: &deletionProtection, }) if err != nil { log.Fatalf("Failed to create serverless index: %v", idx.Name) } else { fmt.Printf("Successfully created serverless index: %v", idx.Name) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); var createIndexRequest = await pinecone.CreateIndexAsync(new CreateIndexRequest { Name = "multitenant-app", Dimension = 8, Metric = MetricType.Cosine, Spec = new ServerlessIndexSpec { Serverless = new ServerlessSpec { Cloud = ServerlessSpecCloud.Aws, Region = "us-east-1", } }, DeletionProtection = DeletionProtection.Disabled }); ``` ```bash curl PINECONE_API_KEY="YOUR_API_KEY" curl -s "https://api.pinecone.io/indexes" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "name": "multitenant-app", "dimension": 8, "metric": "cosine", "spec": { "serverless": { "cloud": "aws", "region": "us-east-1" } } }' ``` ## 2. Isolate tenant data In a multitenant solution, you need to isolate data between tenants. To achieve this in Pinecone, use one namespace per tenant. In the [serverless architecture](/reference/architecture/serverless-architecture), each namespace is stored separately, so this approach ensures physical isolation of each tenant's data. To [create a namespace for a tenant](/guides/index-data/indexing-overview#namespaces#creating-a-namespace), specify the `namespace` parameter when first [upserting](/guides/index-data/upsert-data) the tenant's records. For example, the following code upserts records for `tenant1` and `tenant2` into the `multitenant-app` index: ```Python Python from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key="YOUR_API_KEY") index = pc.Index("multitenant-app") index.upsert( vectors=[ {"id": "A", "values": [0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1]}, {"id": "B", "values": [0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2]}, {"id": "C", "values": [0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3]}, {"id": "D", "values": [0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4]} ], namespace="tenant1" ) index.upsert( vectors=[ {"id": "E", "values": [0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5]}, {"id": "F", "values": [0.6, 0.6, 0.6, 0.6, 0.6, 0.6, 0.6, 0.6]}, {"id": "G", "values": [0.7, 0.7, 0.7, 0.7, 0.7, 0.7, 0.7, 0.7]}, {"id": "H", "values": [0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8]} ], namespace="tenant2" ) ``` ```JavaScript JavaScript import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: "YOUR_API_KEY" }); const index = pc.index("multitenant-app"); await index.namespace("tenant1").upsert([ { "id": "A", "values": [0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1] }, { "id": "B", "values": [0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2] }, { "id": "C", "values": [0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3] }, { "id": "D", "values": [0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4] } ]); await index.namespace("tenant2").upsert([ { "id": "E", "values": [0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5] }, { "id": "F", "values": [0.6, 0.6, 0.6, 0.6, 0.6, 0.6, 0.6, 0.6] }, { "id": "G", "values": [0.7, 0.7, 0.7, 0.7, 0.7, 0.7, 0.7, 0.7] }, { "id": "H", "values": [0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8] } ]); ``` ```java Java import io.pinecone.clients.Index; import io.pinecone.clients.Pinecone; import java.util.Arrays; import java.util.List; public class UpsertExample { public static void main(String[] args) { Pinecone pc = new Pinecone.Builder("YOUR_API_KEY").build(); String indexName = "multitenant-app"; Index index = pc.getIndexConnection(indexName); List values1 = Arrays.asList(0.1f, 0.1f, 0.1f, 0.1f, 0.1f, 0.1f, 0.1f, 0.1f); List values2 = Arrays.asList(0.2f, 0.2f, 0.2f, 0.2f, 0.2f, 0.2f, 0.2f, 0.2f); List values3 = Arrays.asList(0.3f, 0.3f, 0.3f, 0.3f, 0.3f, 0.3f, 0.3f, 0.3f); List values4 = Arrays.asList(0.4f, 0.4f, 0.4f, 0.4f, 0.4f, 0.4f, 0.4f, 0.4f); List values5 = Arrays.asList(0.5f, 0.5f, 0.5f, 0.5f, 0.5f, 0.5f, 0.5f, 0.5f); List values6 = Arrays.asList(0.6f, 0.6f, 0.6f, 0.6f, 0.6f, 0.6f, 0.6f, 0.6f); List values7 = Arrays.asList(0.7f, 0.7f, 0.7f, 0.7f, 0.7f, 0.7f, 0.7f, 0.7f); List values8 = Arrays.asList(0.8f, 0.8f, 0.8f, 0.8f, 0.8f, 0.8f, 0.8f, 0.8f); index.upsert("A", values1, "tenant1"); index.upsert("B", values2, "tenant1"); index.upsert("C", values3, "tenant1"); index.upsert("D", values4, "tenant1"); index.upsert("E", values5, "tenant2"); index.upsert("F", values6, "tenant2"); index.upsert("G", values7, "tenant2"); index.upsert("H", values8, "tenant2"); } } ``` ```go Go // Add to the main function: idx, err := pc.DescribeIndex(ctx, indexName) if err != nil { log.Fatalf("Failed to describe index \"%v\": %v", idx.Name, err) } idxConnection1, err := pc.Index(pinecone.NewIndexConnParams{Host: idx.Host, Namespace: "tenant1"}) if err != nil { log.Fatalf("Failed to create IndexConnection1 for Host %v: %v", idx.Host, err) } idxConnection2, err := pc.Index(pinecone.NewIndexConnParams{Host: idx.Host, Namespace: "tenant2"}) if err != nil { log.Fatalf("Failed to create IndexConnection1 for Host %v: %v", idx.Host, err) } vectors1 := []*pinecone.Vector{ { Id: "A", Values: []float32{0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1}, }, { Id: "B", Values: []float32{0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2}, }, { Id: "C", Values: []float32{0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3}, }, { Id: "D", Values: []float32{0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4}, }, } vectors2 := []*pinecone.Vector{ { Id: "E", Values: []float32{0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5}, }, { Id: "F", Values: []float32{0.6, 0.6, 0.6, 0.6, 0.6, 0.6, 0.6, 0.6}, }, { Id: "G", Values: []float32{0.7, 0.7, 0.7, 0.7, 0.7, 0.7, 0.7, 0.7}, }, { Id: "H", Values: []float32{0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8}, }, } count1, err := idxConnection1.UpsertVectors(ctx, vectors1) if err != nil { log.Fatalf("Failed to upsert vectors: %v", err) } else { fmt.Printf("Successfully upserted %d vector(s)!\n", count1) } count2, err := idxConnection2.UpsertVectors(ctx, vectors2) if err != nil { log.Fatalf("Failed to upsert vectors: %v", err) } else { fmt.Printf("Successfully upserted %d vector(s)!\n", count2) } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); var index = pinecone.Index("multitenant-app"); var upsertResponse1 = await index.UpsertAsync(new UpsertRequest { Vectors = new[] { new Vector { Id = "A", Values = new[] { 0.1f, 0.1f, 0.1f, 0.1f, 0.1f, 0.1f, 0.1f, 0.1f }, }, new Vector { Id = "B", Values = new[] { 0.2f, 0.2f, 0.2f, 0.2f, 0.2f, 0.2f, 0.2f, 0.2f }, }, new Vector { Id = "C", Values = new[] { 0.3f, 0.3f, 0.3f, 0.3f, 0.3f, 0.3f, 0.3f, 0.3f }, }, new Vector { Id = "D", Values = new[] { 0.4f, 0.4f, 0.4f, 0.4f, 0.4f, 0.4f, 0.4f, 0.4f }, } }, Namespace = "tenant1", }); var upsertResponse2 = await index.UpsertAsync(new UpsertRequest { Vectors = new[] { new Vector { Id = "E", Values = new[] { 0.5f, 0.5f, 0.5f, 0.5f, 0.5f, 0.5f, 0.5f, 0.5f }, }, new Vector { Id = "F", Values = new[] { 0.6f, 0.6f, 0.6f, 0.6f, 0.6f, 0.6f, 0.6f, 0.6f }, }, new Vector { Id = "G", Values = new[] { 0.7f, 0.7f, 0.7f, 0.7f, 0.7f, 0.7f, 0.7f, 0.7f }, }, new Vector { Id = "H", Values = new[] { 0.8f, 0.8f, 0.8f, 0.8f, 0.8f, 0.8f, 0.8f, 0.8f }, } }, Namespace = "tenant2", }); ``` ```bash curl # The `POST` requests below uses the unique endpoint for an index. # See https://docs.pinecone.io/guides/manage-data/target-an-index for details. PINECONE_API_KEY="YOUR_API_KEY" INDEX_HOST="INDEX_HOST" curl "https://$INDEX_HOST/vectors/upsert" \ -H "Api-Key: $PINECONE_API_KEY" \ -H 'Content-Type: application/json' \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "vectors": [ { "id": "A", "values": [0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1] }, { "id": "B", "values": [0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2] }, { "id": "C", "values": [0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3] }, { "id": "D", "values": [0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4] } ], "namespace": "tenant1" }' curl "https://$INDEX_HOST/vectors/upsert" \ -H "Api-Key: $PINECONE_API_KEY" \ -H 'Content-Type: application/json' \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "vectors": [ { "id": "E", "values": [0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5] }, { "id": "F", "values": [0.6, 0.6, 0.6, 0.6, 0.6, 0.6, 0.6, 0.6] }, { "id": "G", "values": [0.7, 0.7, 0.7, 0.7, 0.7, 0.7, 0.7, 0.7] }, { "id": "H", "values": [0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8] } ], "namespace": "tenant2" }' ``` When upserting additional records for a tenant, or when [updating](/guides/manage-data/update-data) or [deleting](/guides/manage-data/delete-data) records for a tenant, specify the tenant's `namespace`. For example, the following code updates the dense vector value of record `A` in `tenant1`: ```Python Python from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key="YOUR_API_KEY") index = pc.Index("multitenant-app") index.update(id="A", values=[0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8], namespace="tenant1") ``` ```JavaScript JavaScript import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: "YOUR_API_KEY" }); const index = pc.index("multitenant-app"); await index.namespace('tenant1').update({ id: 'A', values: [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8] }); ``` ```java Java import io.pinecone.clients.Index; import io.pinecone.clients.Pinecone; import io.pinecone.proto.UpdateResponse; import java.util.Arrays; import java.util.List; public class UpdateExample { public static void main(String[] args) { Pinecone pc = new Pinecone.Builder("YOUR_API_KEY").build(); Index index = pc.getIndexConnection("multitenant-app"); List values = Arrays.asList(0.1f, 0.2f, 0.3f, 0.4f, 0.5f, 0.6f, 0.7f, 0.8f); UpdateResponse updateResponse = index.update("A", values, null, "tenant1", null, null); System.out.println(updateResponse); } } ``` ```go Go // Add to the main function: idxConn1.UpdateVector(ctx, &pinecone.UpdateVectorRequest{ Id: "A", Values: []float32{0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8}, }) if err != nil { log.Fatalf("Failed to update vector with ID %v: %v", id, err) } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); var index = pinecone.Index("multitenant-app"); var upsertResponse = await index.UpsertAsync(new UpsertRequest { Vectors = new[] { new Vector { Id = "A", Values = new[] { 0.1f, 0.2f, 0.3f, 0.4f, 0.5f, 0.6f, 0.7f, 0.8f }, } }, Namespace = "tenant1", }); ``` ```bash curl # The `POST` request below uses the unique endpoint for an index. # See https://docs.pinecone.io/guides/manage-data/target-an-index for details. PINECONE_API_KEY="YOUR_API_KEY" INDEX_HOST="INDEX_HOST" curl "https://$INDEX_HOST/vectors/update" \ -H "Api-Key: $PINECONE_API_KEY" \ -H 'Content-Type: application/json' \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "id": "A", "values": [01., 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8], "namespace": "tenant1" }' ``` ## 3. Query tenant data In a multitenant solution, you need to ensure that the queries of one tenant do not affect the experience of other tenants/customers. To achieve this in Pinecone, target each tenant's [queries](/guides/search/search-overview) at the namespace for the tenant. For example, the following code queries only `tenant2` for the 3 vectors that are most similar to an example query vector: ```python Python from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key="YOUR_API_KEY") index = pc.Index("multitenant-app") query_results = index.query( namespace="tenant2", vector=[0.7, 0.7, 0.7, 0.7, 0.7, 0.7, 0.7, 0.7], top_k=3, include_values=True ) print(query_results) # Returns: # {'matches': [{'id': 'F', # 'score': 1.00000012, # 'values': [0.6, 0.6, 0.6, 0.6, 0.6, 0.6, 0.6, 0.6]}, # {'id': 'G', # 'score': 1.0, # 'values': [0.7, 0.7, 0.7, 0.7, 0.7, 0.7, 0.7, 0.7]}, # {'id': 'E', # 'score': 1.0, # 'values': [0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5]}], # 'namespace': 'tenant2', # 'usage': {'read_units': 6}} ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: "YOUR_API_KEY" }); const index = pc.index("multitenant-app"); const queryResponse = await index.namespace("tenant2").query({ topK: 3, vector: [0.7, 0.7, 0.7, 0.7, 0.7, 0.7, 0.7, 0.7], includeValues: true }); console.log(queryResponse); // Returns: { "matches": [ { "id": "F", "score": 1.00000012, "values": [ 0.6, 0.6, 0.6, 0.6, 0.6, 0.6, 0.6, 0.6 ] }, { "id": "E", "score": 1, "values": [ 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5 ] }, { "id": "G", "score": 1, "values": [ 0.7, 0.7, 0.7, 0.7, 0.7, 0.7, 0.7, 0.7 ] } ], "namespace": "tenant2", "usage": { "readUnits": 6 } } ``` ```java Java import io.pinecone.clients.Index; import io.pinecone.clients.Pinecone; import io.pinecone.unsigned_indices_model.QueryResponseWithUnsignedIndices; import java.util.Arrays; import java.util.List; public class QueryExample { public static void main(String[] args) { Pinecone pc = new Pinecone.Builder("YOUR_API_KEY").build(); String indexName = "multitenant-app"; Index index = pc.getIndexConnection(indexName); List queryVector = Arrays.asList(0.7f, 0.7f, 0.7f, 0.7f, 0.7f, 0.7f, 0.7f, 0.7f); QueryResponseWithUnsignedIndices queryResponse = index.query(3, queryVector2, null, null, null, "tenant2", null, true, false); System.out.println(queryResponse); } } // Results: // class QueryResponseWithUnsignedIndices { // matches: [ScoredVectorWithUnsignedIndices { // score: 1.00000012 // id: F // values: [0.6, 0.6, 0.6, 0.6, 0.6, 0.6, 0.6, 0.6] // metadata: // sparseValuesWithUnsignedIndices: SparseValuesWithUnsignedIndices { // indicesWithUnsigned32Int: [] // values: [] // } // }, ScoredVectorWithUnsignedIndices { // score: 1 // id: E // values: [0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5] // metadata: // sparseValuesWithUnsignedIndices: SparseValuesWithUnsignedIndices { // indicesWithUnsigned32Int: [] // values: [] // } // }, ScoredVectorWithUnsignedIndices { // score: 0.07999992 // id: G // values: [0.7, 0.7, 0.7, 0.7, 0.7, 0.7, 0.7, 0.7] // metadata: // sparseValuesWithUnsignedIndices: SparseValuesWithUnsignedIndices { // indicesWithUnsigned32Int: [] // values: [] // } // }] // namespace: tenant2 // usage: read_units: 6 // } ``` ```go Go // Add to the main function: queryVector := []float32{0.7, 0.7, 0.7, 0.7, 0.7, 0.7, 0.7, 0.7} res, err := idxConnection2.QueryByVectorValues(ctx, &pinecone.QueryByVectorValuesRequest{ Vector: queryVector, TopK: 3, IncludeValues: true, }) if err != nil { log.Fatalf("Error encountered when querying by vector: %v", err) } else { fmt.Printf(prettifyStruct(res)) } // Returns: // { // "matches": [ // { // "vector": { // "id": "F", // "values": [ // 0.6, // 0.6, // 0.6, // 0.6, // 0.6, // 0.6, // 0.6, // 0.6 // ] // }, // "score": 1.0000001 // }, // { // "vector": { // "id": "G", // "values": [ // 0.7, // 0.7, // 0.7, // 0.7, // 0.7, // 0.7, // 0.7, // 0.7 // ] // }, // "score": 1 // }, // { // "vector": { // "id": "H", // "values": [ // 0.8, // 0.8, // 0.8, // 0.8, // 0.8, // 0.8, // 0.8, // 0.8 // ] // }, // "score": 1 // } // ], // "usage": { // "read_units": 6 // } // } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); var index = pinecone.Index("multitenant-app"); var queryResponse = await index.QueryAsync(new QueryRequest { Vector = new[] { 0.7f, 0.7f, 0.7f, 0.7f, 0.7f, 0.7f, 0.7f, 0.7f }, Namespace = "tenant2", TopK = 3, }); Console.WriteLine(queryRespnose); ``` ```shell curl # The `POST` requests below uses the unique endpoint for an index. # See https://docs.pinecone.io/guides/manage-data/target-an-index for details. PINECONE_API_KEY="YOUR_API_KEY" INDEX_HOST="INDEX_HOST" curl "https://$INDEX_HOST/query" \ -H "Api-Key: $PINECONE_API_KEY" \ -H 'Content-Type: application/json' \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "namespace": "tenant2", "vector": [0.7, 0.7, 0.7, 0.7, 0.7, 0.7, 0.7, 0.7], "topK": 3, "includeValues": true }' # # Output: # { # "matches": [ # { # "id": "F", # "score": 1.00000012, # "values": [0.6, 0.6, 0.6, 0.6, 0.6, 0.6, 0.6, 0.6] # }, # { # "id": "E", # "score": 1, # "values": [0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5] # }, # { # "id": "G", # "score": 1, # "values": [0.7, 0.7, 0.7, 0.7, 0.7, 0.7, 0.7, 0.7] # } # ], # "namespace": "tenant2", # "usage": {"read_units": 6} # } ``` ## 4. Offboard a tenant In a multitenant solution, you also need it to be quick and easy to offboard a tenant and delete all of its records. To achieve this in Pinecone, you just [delete the namespace](/guides/manage-data/delete-data#delete-an-entire-namespace) for the specific tenant. For example, the following code deletes the namespace and all records for `tenant1`: ```Python Python from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key="YOUR_API_KEY") index = pc.Index("multitenant-app") index.delete(delete_all=True, namespace='tenant1') ``` ```JavaScript JavaScript import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: "YOUR_API_KEY" }); const index = pc.index("multitenant-app"); await index.namespace('tenant1').deleteAll(); ``` ```java Java import io.pinecone.clients.Index; import io.pinecone.clients.Pinecone; public class DeleteVectorsExample { public static void main(String[] args) { Pinecone pc = new Pinecone.Builder("YOUR_API_KEY").build(); Index index = pc.getIndexConnection("multitenant-app"); index.deleteAll("tenant1"); } } ``` ```go Go // Add to the main function: idxConnection1.DeleteAllVectorsInNamespace(ctx) if err != nil { log.Fatalf("Failed to delete vectors in namespace \"%v\": %v", idxConnection2.Namespace, err) } ``` ```csharp C# var index = pinecone.Index("multitenant-app"); var deleteResponse = await index.DeleteAsync(new DeleteRequest { DeleteAll = true, Namespace = "tenant1", }); ``` ```bash curl # The `POST` request below uses the unique endpoint for an index. # See https://docs.pinecone.io/guides/manage-data/target-an-index for details. PINECONE_API_KEY="YOUR_API_KEY" INDEX_HOST="INDEX_HOST" curl "https://$INDEX_HOST/vectors/delete" \ -H "Api-Key: $PINECONE_API_KEY" \ -H 'Content-Type: application/json' \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "deleteAll": true, "namespace": "tenant1" } ' ``` ## Alternatives When tenant data isolation is not a requirement, or when you require the ability to query across tenants, you can store all records in a single index and use metadata fields to assign records to tenants/customers. At query time, you can them [filter by metadata](/guides/index-data/indexing-overview#metadata). For more guidance on this approach, see [Multitenancy in Vector Databases](https://www.pinecone.io/learn/series/vector-databases-in-production-for-busy-engineers/vector-database-multi-tenancy/). # Import records Source: https://docs.pinecone.io/guides/index-data/import-data This page shows you how to import records from object storage into an index and interact with the import. Importing from object storage is the most efficient and cost-effective way to load large numbers of records into an index. To run through this guide in your browser, see the [Bulk import colab notebook](https://colab.research.google.com/github/pinecone-io/examples/blob/master/docs/pinecone-import.ipynb). This feature is in [public preview](/release-notes/feature-availability) and available only on [Standard and Enterprise plans](https://www.pinecone.io/pricing/). ## Before you import Before you can import records, ensure you have a serverless index, a storage integration, and data formatted in a Parquet file and uploaded to the Amazon S3 bucket. ### Create an index [Create a serverless index](/guides/index-data/create-an-index) for your data. * Import does not support [integrated embedding](/guides/index-data/indexing-overview#vector-embedding), so make sure your index is not associated with an integrated embedding model. * Import only supports AWS S3 as a data source, so make sure your index is also on AWS. * You cannot import records into existing namespaces, so make sure your index does not have namespaces with the same name as the namespaces you want to import into. ### Add a storage integration To import records from a secure data source, you must create an integration to allow Pinecone access to data in your object storage. For information on how to add, edit, and delete a storage integration, see [Manage storage integrations](/guides/operations/integrations/manage-storage-integrations). To import records from a public data source, a storage integration is not required. ### Prepare your data For each namespace you want to import into, create a Parquet file and upload it to object storage. #### Dense index To import into a dense index, the Parquet file must contain the following columns: | Column name | Parquet type | Description | | ----------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------- | | `id` | `STRING` | Required. The unique [identifier for each record](/guides/get-started/glossary#record-id). | | `values` | `LIST` | Required. A list of floating-point values that make up the [dense vector embedding](/guides/get-started/glossary#dense-vector). | | `metadata` | `STRING` | Optional. Additional [metadata](/guides/get-started/glossary#metadata) for each record. To omit from specific rows, use `NULL`. | The Parquet file cannot contain additional columns. For example: ```parquet id | values | metadata -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1 | [ 3.82 2.48 -4.15 ... ] | {"year": 1984, "month": 6, "source": "source1", "title": "Example1", "text": "When ..."} 2 | [ 1.82 3.48 -2.15 ... ] | {"year": 1990, "month": 4, "source": "source2", "title": "Example2", "text": "Who ..."} ``` #### Sparse index To import into a sparse index, the Parquet file must contain the following columns: | Column name | Parquet type | Description | | --------------- | ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `id` | `STRING` | Required. The unique [identifier for each record](/guides/get-started/glossary#record-id). | | `sparse_values` | `LIST` and `LIST` | Required. A list of floating-point values (sparse values) and a list of integer values (sparse indices) that make up the [sparse vector embedding](/guides/get-started/glossary#sparse-vector). | | `metadata` | `STRING` | Optional. Additional [metadata](/guides/get-started/glossary#metadata) for each record. To omit from specific rows, use `NULL`. | The Parquet file cannot contain additional columns. For example: ```parquet id | sparse_values | metadata -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1 | {"indices": [ 822745112 1009084850 1221765879 ... ], "values": [1.7958984 0.41577148 2.828125 ...]} | {"year": 1984, "month": 6, "source": "source1", "title": "Example1", "text": "When ..."} 2 | {"indices": [ 504939989 1293001993 3201939490 ... ], "values": [1.4383747 0.72849722 1.384775 ...]} | {"year": 1990, "month": 4, "source": "source2", "title": "Example2", "text": "Who ..."} ``` In object storage, your directory structure determines which Pinecone namespaces your data is imported into. The files associated with each namespace must be in a separate prefix (or sub-directory). The namespace cannot be the same as an existing namespace in the index you are importing into. To import records, specify the URI of the prefix containing the namespace and Parquet files you want to import. For example: ``` s3://BUCKET_NAME/PATH/TO/NAMESPACES --/example_namespace/ ----0.parquet ----1.parquet ----2.parquet ----3.parquet ----.log --/example_namespace2/ ----4.parquet ----5.parquet ----6.parquet ----7.parquet ----.log ``` Pinecone then finds all `.parquet` files inside the namespace prefix and imports them into the namespace. All other file types are ignored. In the example above, the import is located at the top-level URL of `s3://BUCKET_NAME/PATH/TO/NAMESPACES/`. When scanning this directory, Pinecone finds the namespace `example_namespace`, which contains four `.parquet` files and one `.log` file. Pinecone ignores the `.log` file. Each import request can import up 1TB of data, or 100,000,000 records into a maximum of 100 namespaces, whichever limit is met first. ## Import records into an index Review current [limitations](#limitations) before starting an import. Use the [`start_import`](/reference/api/2025-01/data-plane/start_import) operation to start an asynchronous import of vectors from object storage into an index. To import from a private bucket, specify the Integration ID (`integration`) of the Amazon S3 integration you created. The ID is found on the [Storage integrations](https://app.pinecone.io/organizations/-/projects/-/storage) page of the Pinecone console. An ID is not needed to import from a public bucket. The operation returns an `id` that you can use to [check the status of the import](#list-imports). If you set the import to continue on error, the operation will skip records that fail to import and continue with the next record. The operation will complete, but there will not be any notification about which records, if any, failed to import. To see how many records were successfully imported, use the [`describe_import`](#describe-an-import) operation. ```python Python from pinecone import Pinecone, ImportErrorMode pc = Pinecone(api_key="YOUR_API_KEY") # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index index = pc.Index(host="INDEX_HOST") root = "s3://BUCKET_NAME/PATH/TO/DIR" index.start_import( uri=root, error_mode=ImportErrorMode.CONTINUE, # or ImportErrorMode.ABORT integration_id="a12b3d4c-47d2-492c-a97a-dd98c8dbefde" # Optional for public buckets ) ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone'; const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index const index = pc.index("INDEX_NAME", "INDEX_HOST") const storageURI = 's3://BUCKET_NAME/PATH/TO/DIR'; const errorMode = 'continue'; // or 'abort' const integrationID = 'a12b3d4c-47d2-492c-a97a-dd98c8dbefde'; // Optional for public buckets await index.startImport(storageURI, errorMode, integrationID); ``` ```java Java import io.pinecone.clients.Pinecone; import io.pinecone.clients.AsyncIndex; import org.openapitools.db_data.client.ApiException; import org.openapitools.db_data.client.model.ImportErrorMode; import org.openapitools.db_data.client.model.StartImportResponse; public class StartImport { public static void main(String[] args) throws ApiException { // Initialize a Pinecone client with your API key Pinecone pinecone = new Pinecone.Builder("YOUR_API_KEY").build(); // Get async imports connection object AsyncIndex asyncIndex = pinecone.getAsyncIndexConnection("docs-example"); // s3 uri String uri = "s3://BUCKET_NAME/PATH/TO/DIR"; // Integration ID (optional for public buckets) String integrationId = "a12b3d4c-47d2-492c-a97a-dd98c8dbefde"; // Start an import StartImportResponse response = asyncIndex.startImport(uri, integrationId, ImportErrorMode.OnErrorEnum.CONTINUE); } } ``` ```go Go package main import ( "context" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "INDEX_HOST"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v", err) } uri := "s3://BUCKET_NAME/PATH/TO/DIR" errorMode := "continue" // or "abort" importRes, err := idxConnection.StartImport(ctx, uri, nil, (*pinecone.ImportErrorMode)(&errorMode)) if err != nil { log.Fatalf("Failed to start import: %v", err) } fmt.Printf("Import started with ID: %s", importRes.Id) } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index var index = pinecone.Index(host: "INDEX_HOST"); var uri = "s3://BUCKET_NAME/PATH/TO/DIR"; var response = await index.StartBulkImportAsync(new StartImportRequest { Uri = uri, IntegrationId = "a12b3d4c-47d2-492c-a97a-dd98c8dbefde", ErrorMode = new ImportErrorMode { OnError = ImportErrorModeOnError.Continue } }); ``` ```bash curl # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index PINECONE_API_KEY="YOUR_API_KEY" INDEX_HOST="INDEX_HOST" curl "https://$INDEX_HOST/bulk/imports" \ -H 'Api-Key: $YOUR_API_KEY' \ -H 'Content-Type: application/json' \ -H 'X-Pinecone-API-Version: 2025-04' \ -d '{ "integrationId": "a12b3d4c-47d2-492c-a97a-dd98c8dbefde", "uri": "s3://BUCKET_NAME/PATH/TO/DIR", "errorMode": { "onError": "continue" } }' ``` ```json Response { "operation_id": "101" } ``` Once all the data is loaded, the [index builder](/reference/architecture/serverless-architecture#index-builder) will index the records, which usually takes at least 10 minutes. During this indexing process, the expected job status is `InProgress`, but `100.0` percent complete. Once all the imported records are indexed and fully available for querying, the import operation will be set to `Completed`. You can start a new import using the [Pinecone console](https://app.pinecone.io/organizations/-/projects/-/indexes). Find the index you want to import into, and click the **ellipsis (..) menu > Import data**. ## Manage imports ### List imports Use the [`list_imports`](/reference/api/2025-01/data-plane/list_imports) operation to list all of the recent and ongoing imports. By default, the operation returns up to 100 imports per page. If the `limit` parameter is passed, the operation returns up to that number of imports per page instead. For example, if `limit=3`, up to 3 imports are returned per page. Whenever there are additional imports to return, the response includes a `pagination_token` for fetching the next page of imports. When using the Python SDK, `list_import` paginates automatically. ```python Python from pinecone import Pinecone, ImportErrorMode pc = Pinecone(api_key="YOUR_API_KEY") # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index index = pc.Index(host="INDEX_HOST") # List using a generator that handles pagination for i in index.list_imports(): print(f"id: {i.id} status: {i.status}") # List using a generator that fetches all results at once operations = list(index.list_imports()) print(operations) ``` ```json Response { "data": [ { "id": "1", "uri": "s3://BUCKET_NAME/PATH/TO/DIR", "status": "Pending", "started_at": "2024-08-19T20:49:00.754Z", "finished_at": "2024-08-19T20:49:00.754Z", "percent_complete": 42.2, "records_imported": 1000000 } ], "pagination": { "next": "Tm90aGluZyB0byBzZWUgaGVyZQo=" } } ``` You can view the list of imports for an index in the [Pinecone console](https://app.pinecone.io/organizations/-/projects/-/indexes/). Select the index and navigate to the **Imports** tab. When using the Node.js SDK, Go SDK, .NET SDK, or REST API to list recent and ongoing imports, you must manually fetch each page of results. To view the next page of results, include the `paginationToken` provided in the response of the [`list_imports`](#list-imports) / `GET` request. ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone'; const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index const index = pc.index("INDEX_NAME", "INDEX_HOST") const results = await index.listImports({ limit: 10, paginationToken: 'Tm90aGluZyB0byBzZWUgaGVyZQo' }); console.log(results); ``` ```java Java import io.pinecone.clients.Pinecone; import io.pinecone.clients.AsyncIndex; import org.openapitools.db_data.client.ApiException; import org.openapitools.db_data.client.model.ListImportsResponse; public class ListImports { public static void main(String[] args) throws ApiException { // Initialize a Pinecone client with your API key Pinecone pinecone = new Pinecone.Builder("YOUR_API_KEY").build(); // Get async imports connection object AsyncIndex asyncIndex = pinecone.getAsyncIndexConnection("docs-example"); // List imports ListImportsResponse response = asyncIndex.listImports(10, "Tm90aGluZyB0byBzZWUgaGVyZQo"); System.out.println(response); } } ``` ```go Go package main import ( "context" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "INDEX_HOST"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v", err) } limit := int32(10) firstImportPage, err := idxConnection.ListImports(ctx, &limit, nil) if err != nil { log.Fatalf("Failed to list imports: %v", err) } fmt.Printf("First page of imports: %+v", firstImportPage.Imports) paginationToken := firstImportPage.NextPaginationToken nextImportPage, err := idxConnection.ListImports(ctx, &limit, paginationToken) if err != nil { log.Fatalf("Failed to list imports: %v", err) } fmt.Printf("Second page of imports: %+v", nextImportPage.Imports) } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index var index = pinecone.Index(host: "INDEX_HOST"); var imports = await index.ListBulkImportsAsync(new ListBulkImportsRequest { Limit = 10, PaginationToken = "Tm90aGluZyB0byBzZWUgaGVyZQo" }); ``` ```bash curl # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index PINECONE_API_KEY="YOUR_API_KEY" INDEX_HOST="INDEX_HOST" curl -X GET "https://$INDEX_HOST/bulk/imports?paginationToken==Tm90aGluZyB0byBzZWUgaGVyZQo" \ -H 'Api-Key: $YOUR_API_KEY' \ -H 'X-Pinecone-API-Version: 2025-04' ``` ### Describe an import Use the [`describe_import`](/reference/api/2025-01/data-plane/describe_import) operation to get details about a specific import. ```python Python from pinecone import Pinecone pc = Pinecone(api_key="YOUR_API_KEY") # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index index = pc.Index(host="INDEX_HOST") index.describe_import(id="101") ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone'; const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index const index = pc.index("INDEX_NAME", "INDEX_HOST") const results = await index.describeImport(id='101'); console.log(results); ``` ```java Java import io.pinecone.clients.Pinecone; import io.pinecone.clients.AsyncIndex; import org.openapitools.db_data.client.ApiException; import org.openapitools.db_data.client.model.ImportModel; public class DescribeImport { public static void main(String[] args) throws ApiException { // Initialize a Pinecone client with your API key Pinecone pinecone = new Pinecone.Builder("YOUR_API_KEY").build(); // Get async imports connection object AsyncIndex asyncIndex = pinecone.getAsyncIndexConnection("docs-example"); // Describe import ImportModel importDetails = asyncIndex.describeImport("101"); System.out.println(importDetails); } } ``` ```go Go package main import ( "context" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "INDEX_HOST"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v", err) } importID := "101" importDesc, err := idxConnection.DescribeImport(ctx, importID) if err != nil { log.Fatalf("Failed to describe import: %s - %v", importID, err) } fmt.Printf("Import ID: %s, Status: %s", importDesc.Id, importDesc.Status) } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index var index = pinecone.Index(host: "INDEX_HOST"); var importDetails = await index.DescribeBulkImportAsync("101"); ``` ```bash curl # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index PINECONE_API_KEY="YOUR_API_KEY" INDEX_HOST="INDEX_HOST" curl -X GET "https://{INDEX_HOST}/bulk/imports/101" \ -H 'Api-Key: $YOUR_API_KEY' \ -H 'X-Pinecone-API-Version: 2025-04' ``` ```json Response { "id": "101", "uri": "s3://BUCKET_NAME/PATH/TO/DIR", "status": "Pending", "created_at": "2024-08-19T20:49:00.754Z", "finished_at": "2024-08-19T20:49:00.754Z", "percent_complete": 42.2, "records_imported": 1000000 } ``` You can view the details of your import using the [Pinecone console](https://app.pinecone.io/organizations/-/projects/-/import). ### Cancel an import The [`cancel_import`](/reference/api/2025-01/data-plane/cancel_import) operation cancels an import if it is not yet finished. It has no effect if the import is already complete. ```python Python from pinecone import Pinecone pc = Pinecone(api_key="YOUR_API_KEY") # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index index = pc.Index(host="INDEX_HOST") index.cancel_import(id="101") ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone'; const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index const index = pc.index("INDEX_NAME", "INDEX_HOST") await index.cancelImport(id='101'); ``` ```java Java import io.pinecone.clients.Pinecone; import io.pinecone.clients.AsyncIndex; import org.openapitools.db_data.client.ApiException; public class CancelImport { public static void main(String[] args) throws ApiException { // Initialize a Pinecone client with your API key Pinecone pinecone = new Pinecone.Builder("YOUR_API_KEY").build(); // Get async imports connection object AsyncIndex asyncIndex = pinecone.getAsyncIndexConnection("docs-example"); // Cancel import asyncIndex.cancelImport("2"); } } ``` ```go Go package main import ( "context" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "INDEX_HOST"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v", err) } importID := "101" err = idxConnection.CancelImport(ctx, importID) if err != nil { log.Fatalf("Failed to cancel import: %s", importID) } importDesc, err := idxConnection.DescribeImport(ctx, importID) if err != nil { log.Fatalf("Failed to describe import: %s - %v", importID, err) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index var index = pinecone.Index(host: "INDEX_HOST"); var cancelResponse = await index.CancelBulkImportAsync("101"); ``` ```bash curl # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index PINECONE_API_KEY="YOUR_API_KEY" INDEX_HOST="INDEX_HOST" curl -X DELETE "https://{INDEX_HOST}/bulk/imports/101" \ -H 'Api-Key: $YOUR_API_KEY' \ -H "X-Pinecone-API-Version: 2025-04" ``` ```json Response {} ``` You can cancel your import using the [Pinecone console](https://app.pinecone.io/organizations/-/projects/-/import). To cancel an ongoing import, select the index you are importing into and navigate to the **Imports** tab. Then, click the **ellipsis (..) menu > Cancel**. # Indexing overview Source: https://docs.pinecone.io/guides/index-data/indexing-overview export const word_0 = "vectors" This page describes key concepts related to indexing data in Pinecone. ## Indexes In Pinecone, you store vector data in indexes. There are two types of indexes: dense and sparse. ### Dense indexes Dense indexes store dense vectors, which are a series of numbers that represent the meaning and relationships of text, images, or other types of data. Each number in a dense vector corresponds to a point in a multidimensional space. Vectors that are closer together in that space are semantically similar. When you query a dense index, Pinecone retrieves the dense vectors that are the most semantically similar to the query. This is often called **semantic search**, nearest neighbor search, similarity search, or just vector search. Learn more: * [Create a dense index](/guides/index-data/create-an-index#create-a-dense-index) * [Upsert dense vectors](/guides/index-data/upsert-data#upsert-dense-vectors) * [Query a dense index](/guides/search/semantic-search) ### Sparse indexes This feature is in [public preview](/release-notes/feature-availability). Sparse indexes store sparse vectors, which are a series of numbers that represent the words or phrases in a document. Sparse vectors have a very large number of dimensions, where only a small proportion of values are non-zero. The dimensions represent words from a dictionary, and the values represent the importance of these words in the document. When you search a sparse index, Pinecone retrieves the sparse vectors that most exactly match the words or phrases in the query. Query terms are scored independently and then summed, with the most similar records scored highest. This is often called **lexical search** or **keyword search**. Learn more: * [Create a sparse index](/guides/index-data/create-an-index#create-a-sparse-index) * [Upsert sparse vectors](/guides/index-data/upsert-data#upsert-sparse-vectors) * [Query a sparse index](/guides/search/lexical-search) #### Limitations These limitations are subject to change during the public preview period. Sparse indexes have the following limitations: * Max sparse records per namespace: 100,000,000 * Max non-zero values per sparse vector: 1000 * Max upserts per second per sparse index: 10 * Max queries per second per sparse index: 100 * Max `top_k` value per query: 1000 You may get fewer than `top_k` results if `top_k` is larger than the number of sparse vectors in your index that match your query. That is, any vectors where the dotproduct score is `0` will be discarded. * Max query results size: 4MB * Limited performance with high cardinality metadata. Better metadata indexing is coming soon. ## Namespaces Within an index, records are partitioned into namespaces, and all [upserts](/guides/index-data/upsert-data), [queries](/guides/search/search-overview), and other [data operations](/guides/index-data/upsert-data) always target one namespace. This has two main benefits: * **Multitenancy:** When you need to isolate data between customers, you can use one namespace per customer and target each customer's writes and queries to their dedicated namespace. See [Implement multitenancy](/guides/index-data/implement-multitenancy) for end-to-end guidance. * **Faster queries:** When you divide records into namespaces in a logical way, you speed up queries by ensuring only relevant records are scanned. The same applies to fetching records, listing record IDs, and other data operations. Namespaces are created automatically during [upsert](/guides/index-data/upsert-data). If a namespace doesn't exist, it is created implicitly. ## Vector embedding [Dense vectors](/guides/get-started/glossary#dense-vector) and [sparse vectors](/guides/get-started/glossary#sparse-vector) are the basic units of data in Pinecone and what Pinecone was specially designed to store and work with. Dense vectors represents the semantics of data such as text, images, and audio recordings, while sparse vectors represent documents or queries in a way that captures keyword information. To transform data into vector format, you use an embedding model. You can either use Pinecone's integrated embedding models to convert your source data to vectors automatically, or you can use an external embedding model and bring your own vectors to Pinecone. ### Integrated embedding 1. [Create an index](/guides/index-data/create-an-index) that is integrated with one of Pinecone's [hosted embedding models](/guides/index-data/create-an-index#embedding-models). 2. [Upsert](/guides/index-data/upsert-data) your source text. Pinecone uses the integrated model to convert the text to vectors automatically. 3. [Search](/guides/search/search-overview) with a query text. Again, Pinecone uses the integrated model to convert the text to a vector automatically. Indexes with integrated embedding do not support [updating](/guides/manage-data/update-data) or [importing](/guides/index-data/import-data) with text. ### Bring your own vectors 1. Use an embedding model to convert your text to vectors. The model can be [hosted by Pinecone](/reference/api/2025-04/inference/generate-embeddings) or an external provider. 2. [Create an index](/guides/index-data/create-an-index) that matches the characteristics of the model. 3. [Upsert](/guides/index-data/upsert-data) your vectors directly. 4. Use the same external embedding model to convert a query to a vector. 5. [Search](/guides/search/search-overview) with your query vector directly. ## Data ingestion To control costs when ingesting large datasets (10,000,000+ records), use [import](/guides/index-data/import-data) instead of upsert. There are two ways to ingest data into an index: * [Importing from object storage](/guides/index-data/import-data) is the most efficient and cost-effective way to load large numbers of records into an index. You store your data as Parquet files in object storage, integrate your object storage with Pinecone, and then start an asynchronous, long-running operation that imports and indexes your records. * [Upserting](/guides/index-data/upsert-data) is intended for ongoing writes to an index. [Batch upserting](/guides/index-data/upsert-data#upsert-in-batches) can improve throughput performance and is a good option for larger numbers of records (up to 1000 per batch) if you cannot work around import's current limitations. ## Metadata Every [record](/guides/get-started/glossary#record) in an index must contain an ID and a vector. In addition, you can include metadata key-value pairs to store additional information or context. When you query the index, you can then include a [metadata filter](/guides/search/filter-by-metadata) to limit the search to records matching a filter expression. Searches without metadata filters do not consider metadata and search the entire namespace. ### Metadata types Metadata payloads must be key-value pairs in a JSON object. Keys must be strings, and values can be one of the following data types: * String * Number (integer or floating point, gets converted to a 64 bit floating point) * Booleans (true, false) * List of strings Null metadata values are not supported. Instead of setting a key to hold a\ null value, we recommend you remove that key from the metadata payload. For example, the following would be valid metadata payloads: ```JSON JSON { "genre": "action", "year": 2020, "length_hrs": 1.5 } { "color": "blue", "fit": "straight", "price": 29.99, "is_jeans": true } ``` ### Metadata size Pinecone supports 40KB of metadata per vector. ### Metadata filter expressions Pinecone's filtering query language is based on [MongoDB's query and projection operators](https://docs.mongodb.com/manual/reference/operator/query/). Pinecone currently supports a subset of those selectors: | Filter | Description | Supported types | | --------- | ---------------------------------------------------------------------------------------------------------------------------------- | ----------------------- | | `$eq` | Matches {word_0} with metadata values that are equal to a specified value. Example: `{"genre": {"$eq": "documentary"}}` | Number, string, boolean | | `$ne` | Matches {word_0} with metadata values that are not equal to a specified value. Example: `{"genre": {"$ne": "drama"}}` | Number, string, boolean | | `$gt` | Matches {word_0} with metadata values that are greater than a specified value. Example: `{"year": {"$gt": 2019}}` | Number | | `$gte` | Matches {word_0} with metadata values that are greater than or equal to a specified value. Example:`{"year": {"$gte": 2020}}` | Number | | `$lt` | Matches {word_0} with metadata values that are less than a specified value. Example: `{"year": {"$lt": 2020}}` | Number | | `$lte` | Matches {word_0} with metadata values that are less than or equal to a specified value. Example: `{"year": {"$lte": 2020}}` | Number | | `$in` | Matches {word_0} with metadata values that are in a specified array. Example: `{"genre": {"$in": ["comedy", "documentary"]}}` | String, number | | `$nin` | Matches {word_0} with metadata values that are not in a specified array. Example: `{"genre": {"$nin": ["comedy", "documentary"]}}` | String, number | | `$exists` | Matches {word_0} with the specified metadata field. Example: `{"genre": {"$exists": true}}` | Number, string, boolean | | `$and` | Joins query clauses with a logical `AND`. Example: `{"$and": [{"genre": {"$eq": "drama"}}, {"year": {"$gte": 2020}}]}` | - | | `$or` | Joins query clauses with a logical `OR`. Example: `{"$or": [{"genre": {"$eq": "drama"}}, {"year": {"$gte": 2020}}]}` | - | For example, the following has a `"genre"` metadata field with a list of strings: ```JSON JSON { "genre": ["comedy", "documentary"] } ``` This means `"genre"` takes on both values, and requests with the following filters will match: ```JSON JSON {"genre":"comedy"} {"genre": {"$in":["documentary","action"]}} {"$and": [{"genre": "comedy"}, {"genre":"documentary"}]} ``` However, requests with the following filter will **not** match: ```JSON JSON { "$and": [{ "genre": "comedy" }, { "genre": "drama" }] } ``` Additionally, requests with the following filters will **not** match because they are invalid. They will result in a compilation error: ``` # INVALID QUERY: {"genre": ["comedy", "documentary"]} ``` ``` # INVALID QUERY: {"genre": {"$eq": ["comedy", "documentary"]}} ``` # Upsert records Source: https://docs.pinecone.io/guides/index-data/upsert-data This page shows you how to upsert records into a namespace in an index. [Namespaces](/guides/index-data/indexing-overview#namespaces) let you partition records within an index and are essential for [implementing multitenancy](/guides/index-data/implement-multitenancy) when you need to isolate the data of each customer/user. If a record ID already exists, upserting overwrites the entire record. To change only part of a record, [update ](/guides/manage-data/update-data) the record. To control costs when ingesting large datasets (10,000,000+ records), use [import](/guides/index-data/import-data) instead of upsert. ## Upsert dense vectors Upserting text is supported only for [indexes with integrated embedding](/guides/index-data/indexing-overview#integrated-embedding). To upsert source text into a [dense index with integrated embedding](/guides/index-data/create-an-index#create-a-dense-index), use the [`upsert_records`](/reference/api/2025-01/data-plane/upsert_records) operation. Pinecone converts the text to dense vectors automatically using the hosted dense embedding model associated with the index. * Specify the [`namespace`](/guides/index-data/indexing-overview#namespaces) to upsert into. If the namespace doesn't exist, it is created. To use the default namespace, set the namespace to `"__default__"`. * Format your input data as records, each with the following: * An `_id` field with a unique record identifier for the index namespace. `id` can be used as an alias for `_id`. * A field with the source text to convert to a vector. This field must match the `field_map` specified in the index. * Additional fields will be stored as record metadata and can be returned in search results or used to filter search results. For example, the following code converts the sentences in the `chunk_text` fields to dense vectors and then upserts them into `example-namespace` in an example index. The additional `category` field is stored as metadata. ```python Python from pinecone import Pinecone pc = Pinecone(api_key="YOUR_API_KEY") # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index index = pc.Index(host="INDEX_HOST") # Upsert records into a namespace # `chunk_text` fields are converted to dense vectors # `category` fields are stored as metadata index.upsert_records( "example-namespace", [ { "_id": "rec1", "chunk_text": "Apples are a great source of dietary fiber, which supports digestion and helps maintain a healthy gut.", "category": "digestive system", }, { "_id": "rec2", "chunk_text": "Apples originated in Central Asia and have been cultivated for thousands of years, with over 7,500 varieties available today.", "category": "cultivation", }, { "_id": "rec3", "chunk_text": "Rich in vitamin C and other antioxidants, apples contribute to immune health and may reduce the risk of chronic diseases.", "category": "immune system", }, { "_id": "rec4", "chunk_text": "The high fiber content in apples can also help regulate blood sugar levels, making them a favorable snack for people with diabetes.", "category": "endocrine system", }, ] ) ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: "YOUR_API_KEY" }) // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index const namespace = pc.index("INDEX_NAME", "INDEX_HOST").namespace("example-namespace"); // Upsert records into a namespace // `chunk_text` fields are converted to dense vectors // `category` ios stored as metadata await namespace.upsertRecords([ { "_id": "rec1", "chunk_text": "Apples are a great source of dietary fiber, which supports digestion and helps maintain a healthy gut.", "category": "digestive system", }, { "_id": "rec2", "chunk_text": "Apples originated in Central Asia and have been cultivated for thousands of years, with over 7,500 varieties available today.", "category": "cultivation", }, { "_id": "rec3", "chunk_text": "Rich in vitamin C and other antioxidants, apples contribute to immune health and may reduce the risk of chronic diseases.", "category": "immune system", }, { "_id": "rec4", "chunk_text": "The high fiber content in apples can also help regulate blood sugar levels, making them a favorable snack for people with diabetes.", "category": "endocrine system", } ]); ``` ```java Java import io.pinecone.clients.Index; import io.pinecone.configs.PineconeConfig; import io.pinecone.configs.PineconeConnection; import org.openapitools.db_data.client.ApiException; import java.util.*; public class UpsertText { public static void main(String[] args) throws ApiException { PineconeConfig config = new PineconeConfig("YOUR_API_KEY"); config.setHost("INDEX_HOST"); PineconeConnection connection = new PineconeConnection(config); Index index = new Index(config, connection, "integrated-dense-java"); ArrayList> upsertRecords = new ArrayList<>(); HashMap record1 = new HashMap<>(); record1.put("_id", "rec1"); record1.put("category", "digestive system"); record1.put("chunk_text", "Apples are a great source of dietary fiber, which supports digestion and helps maintain a healthy gut."); HashMap record2 = new HashMap<>(); record2.put("_id", "rec2"); record2.put("category", "cultivation"); record2.put("chunk_text", "Apples originated in Central Asia and have been cultivated for thousands of years, with over 7,500 varieties available today."); HashMap record3 = new HashMap<>(); record3.put("_id", "rec3"); record3.put("category", "immune system"); record3.put("chunk_text", "Rich in vitamin C and other antioxidants, apples contribute to immune health and may reduce the risk of chronic diseases."); HashMap record4 = new HashMap<>(); record4.put("_id", "rec4"); record4.put("category", "endocrine system"); record4.put("chunk_text", "The high fiber content in apples can also help regulate blood sugar levels, making them a favorable snack for people with diabetes."); upsertRecords.add(record1); upsertRecords.add(record2); upsertRecords.add(record3); upsertRecords.add(record4); index.upsertRecords("example-namespace", upsertRecords); } } ``` ```go Go package main import ( "context" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "INDEX_HOST", Namespace: "example-namespace"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v", err) } // Upsert records into a namespace // `chunk_text` fields are converted to dense vectors // `category` is stored as metadata records := []*pinecone.IntegratedRecord{ { "_id": "rec1", "chunk_text": "Apples are a great source of dietary fiber, which supports digestion and helps maintain a healthy gut.", "category": "digestive system", }, { "_id": "rec2", "chunk_text": "Apples originated in Central Asia and have been cultivated for thousands of years, with over 7,500 varieties available today.", "category": "cultivation", }, { "_id": "rec3", "chunk_text": "Rich in vitamin C and other antioxidants, apples contribute to immune health and may reduce the risk of chronic diseases.", "category": "immune system", }, { "_id": "rec4", "chunk_text": "The high fiber content in apples can also help regulate blood sugar levels, making them a favorable snack for people with diabetes.", "category": "endocrine system", }, } err = idxConnection.UpsertRecords(ctx, records) if err != nil { log.Fatalf("Failed to upsert vectors: %v", err) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); var index = pinecone.Index(host: "INDEX_HOST"); await index.UpsertRecordsAsync( "example-namespace", [ new UpsertRecord { Id = "rec1", AdditionalProperties = { ["chunk_text"] = "AAPL reported a year-over-year revenue increase, expecting stronger Q3 demand for its flagship phones.", ["category"] = "technology", ["quarter"] = "Q3", }, }, new UpsertRecord { Id = "rec2", AdditionalProperties = { ["chunk_text"] = "AAPL may consider healthcare integrations in Q4 to compete with tech rivals entering the consumer wellness space.", ["category"] = "technology", ["quarter"] = "Q4", }, }, new UpsertRecord { Id = "rec3", AdditionalProperties = { ["chunk_text"] = "AAPL may consider healthcare integrations in Q4 to compete with tech rivals entering the consumer wellness space.", ["category"] = "technology", ["quarter"] = "Q4", }, }, new UpsertRecord { Id = "rec4", AdditionalProperties = { ["chunk_text"] = "AAPL's strategic Q3 partnerships with semiconductor suppliers could mitigate component risks and stabilize iPhone production", ["category"] = "technology", ["quarter"] = "Q3", }, }, ] ); ``` ```shell curl # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index INDEX_HOST="INDEX_HOST" NAMESPACE="YOUR_NAMESPACE" PINECONE_API_KEY="YOUR_API_KEY" # Upsert records into a namespace # `chunk_text` fields are converted to dense vectors # `category` is stored as metadata curl "https://$INDEX_HOST/records/namespaces/$NAMESPACE/upsert" \ -H "Content-Type: application/x-ndjson" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{"_id": "rec1", "chunk_text": "Apples are a great source of dietary fiber, which supports digestion and helps maintain a healthy gut.", "category": "digestive system"} {"_id": "rec2", "chunk_text": "Apples originated in Central Asia and have been cultivated for thousands of years, with over 7,500 varieties available today.", "category": "cultivation"} {"_id": "rec3", "chunk_text": "Rich in vitamin C and other antioxidants, apples contribute to immune health and may reduce the risk of chronic diseases.", "category": "immune system"} {"_id": "rec4", "chunk_text": "The high fiber content in apples can also help regulate blood sugar levels, making them a favorable snack for people with diabetes.", "category": "endocrine system"}' ``` To upsert dense vectors into a [dense index](/guides/index-data/create-an-index#create-a-dense-index), use the [`upsert`](/reference/api/2025-01/data-plane/upsert) operation as follows: * Specify the [`namespace`](/guides/index-data/indexing-overview#namespaces) to upsert into. If the namespace doesn't exist, it is created. To use the default namespace, set the namespace to `"__default__"`. * Format your input data as records, each with the following: * An `id` field with a unique record identifier for the index namespace. * A `values` field with the dense vector values. * Optionally, a `metadata` field with key-value pairs to store additional information or context. When you query the index, you can then filter by metadata to ensure only relevant records are scanned. For more information, see [Metadata Filtering](/guides/index-data/indexing-overview#metadata). ```Python Python from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key="YOUR_API_KEY") # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index index = pc.Index(host="INDEX_HOST") index.upsert( vectors=[ { "id": "A", "values": [0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1], "metadata": {"genre": "comedy", "year": 2020} }, { "id": "B", "values": [0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2], "metadata": {"genre": "documentary", "year": 2019} }, { "id": "C", "values": [0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3], "metadata": {"genre": "comedy", "year": 2019} }, { "id": "D", "values": [0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4], "metadata": {"genre": "drama"} } ], namespace="example-namespace" ) ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: "YOUR_API_KEY" }) // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index const index = pc.index("INDEX_NAME", "INDEX_HOST") const records = [ { id: 'A', values: [0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1], metadata: { genre: "comedy", year: 2020 }, }, { id: 'B', values: [0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2], metadata: { genre: "documentary", year: 2019 }, }, { id: 'C', values: [0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3], metadata: { genre: "comedy", year: 2019 }, }, { id: 'D', values: [0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4], metadata: { genre: "drama" }, } ] await index.('example-namespace').upsert(records); ``` ```java Java import com.google.protobuf.Struct; import com.google.protobuf.Value; import io.pinecone.clients.Index; import io.pinecone.configs.PineconeConfig; import io.pinecone.configs.PineconeConnection; import java.util.Arrays; import java.util.List; public class UpsertExample { public static void main(String[] args) { PineconeConfig config = new PineconeConfig("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index config.setHost("INDEX_HOST"); PineconeConnection connection = new PineconeConnection(config); Index index = new Index(connection, "INDEX_NAME"); List values1 = Arrays.asList(0.1f, 0.1f, 0.1f, 0.1f, 0.1f, 0.1f, 0.1f, 0.1f); List values2 = Arrays.asList(0.2f, 0.2f, 0.2f, 0.2f, 0.2f, 0.2f, 0.2f, 0.2f); List values3 = Arrays.asList(0.3f, 0.3f, 0.3f, 0.3f, 0.3f, 0.3f, 0.3f, 0.3f); List values4 = Arrays.asList(0.4f, 0.4f, 0.4f, 0.4f, 0.4f, 0.4f, 0.4f, 0.4f); Struct metaData1 = Struct.newBuilder() .putFields("genre", Value.newBuilder().setStringValue("comedy").build()) .putFields("year", Value.newBuilder().setNumberValue(2020).build()) .build(); Struct metaData2 = Struct.newBuilder() .putFields("genre", Value.newBuilder().setStringValue("documentary").build()) .putFields("year", Value.newBuilder().setNumberValue(2019).build()) .build(); Struct metaData3 = Struct.newBuilder() .putFields("genre", Value.newBuilder().setStringValue("comedy").build()) .putFields("year", Value.newBuilder().setNumberValue(2019).build()) .build(); Struct metaData4 = Struct.newBuilder() .putFields("genre", Value.newBuilder().setStringValue("drama").build()) .build(); index.upsert("A", values1, null, null, metaData1, 'example-namespace'); index.upsert("B", values2, null, null, metaData2, 'example-namespace'); index.upsert("C", values3, null, null, metaData3, 'example-namespace'); index.upsert("D", values4, null, null, metaData4, 'example-namespace'); } } ``` ```go Go package main import ( "context" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" "google.golang.org/protobuf/types/known/structpb" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "INDEX_HOST", Namespace: "example-namespace"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v", err) } metadataMap1 := map[string]interface{}{ "genre": "comedy", "year": 2020, } metadata1, err := structpb.NewStruct(metadataMap1) if err != nil { log.Fatalf("Failed to create metadata map: %v", err) } metadataMap2 := map[string]interface{}{ "genre": "documentary", "year": 2019, } metadata2, err := structpb.NewStruct(metadataMap2) if err != nil { log.Fatalf("Failed to create metadata map: %v", err) } metadataMap3 := map[string]interface{}{ "genre": "comedy", "year": 2019, } metadata3, err := structpb.NewStruct(metadataMap3) if err != nil { log.Fatalf("Failed to create metadata map: %v", err) } metadataMap4 := map[string]interface{}{ "genre": "drama", } metadata4, err := structpb.NewStruct(metadataMap4) if err != nil { log.Fatalf("Failed to create metadata map: %v", err) } vectors := []*pinecone.Vector{ { Id: "A", Values: []float32{0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1}, Metadata: metadata1, }, { Id: "B", Values: []float32{0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2}, Metadata: metadata2, }, { Id: "C", Values: []float32{0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3}, Metadata: metadata3, }, { Id: "D", Values: []float32{0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4}, Metadata: metadata4, }, } count, err := idxConnection.UpsertVectors(ctx, vectors) if err != nil { log.Fatalf("Failed to upsert vectors: %v", err) } else { fmt.Printf("Successfully upserted %d vector(s)!\n", count) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index var index = pinecone.Index(host: "INDEX_HOST"); var upsertResponse = await index.UpsertAsync(new UpsertRequest { Vectors = new[] { new Vector { Id = "A", Values = new[] { 0.1f, 0.1f, 0.1f, 0.1f, 0.1f, 0.1f, 0.1f, 0.1f }, Metadata = new Metadata { ["genre"] = new("comedy"), ["year"] = new(2020), }, }, new Vector { Id = "B", Values = new[] { 0.2f, 0.2f, 0.2f, 0.2f, 0.2f, 0.2f, 0.2f, 0.2f }, Metadata = new Metadata { ["genre"] = new("documentary"), ["year"] = new(2019), }, }, new Vector { Id = "C", Values = new[] { 0.3f, 0.3f, 0.3f, 0.3f, 0.3f, 0.3f, 0.3f, 0.3f }, Metadata = new Metadata { ["genre"] = new("comedy"), ["year"] = new(2019), }, }, new Vector { Id = "D", Values = new[] { 0.4f, 0.4f, 0.4f, 0.4f, 0.4f, 0.4f, 0.4f, 0.4f }, Metadata = new Metadata { ["genre"] = new("drama"), }, } }, Namespace = "example-namespace", }); ``` ```bash curl # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index PINECONE_API_KEY="YOUR_API_KEY" INDEX_HOST="INDEX_HOST" curl "https://$INDEX_HOST/vectors/upsert" \ -H "Api-Key: $PINECONE_API_KEY" \ -H 'Content-Type: application/json' \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "vectors": [ { "id": "A", "values": [0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1], "metadata": {"genre": "comedy", "year": 2020} }, { "id": "B", "values": [0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2], "metadata": {"genre": "documentary", "year": 2019} }, { "id": "C", "values": [0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3], "metadata": {"genre": "comedy", "year": 2019} }, { "id": "D", "values": [0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4], "metadata": {"genre": "drama"} } ], "namespace": "example-namespace" }' ``` ## Upsert sparse vectors This feature is in [public preview](/release-notes/feature-availability). Upserting text is supported only for [indexes with integrated embedding](/guides/index-data/indexing-overview#integrated-embedding). To upsert source text into a [sparse index with integrated embedding](/guides/index-data/create-an-index#create-a-sparse-index), use the [`upsert_records`](/reference/api/2025-01/data-plane/upsert_records) operation. Pinecone converts the text to sparse vectors automatically using the hosted sparse embedding model associated with the index. * Specify the [`namespace`](/guides/index-data/indexing-overview#namespaces) to upsert into. If the namespace doesn't exist, it is created. To use the default namespace, set the namespace to `"__default__"`. * Format your input data as records, each with the following: * An `_id` field with a unique record identifier for the index namespace. `id` can be used as an alias for `_id`. * A field with the source text to convert to a vector. This field must match the `field_map` specified in the index. * Additional fields will be stored as record metadata and can be returned in search results or used to filter search results. For example, the following code converts the sentences in the `chunk_text` fields to sparse vectors and then upserts them into `example-namespace` in an example index. The additional `category` and `quarter` fields are stored as metadata. ```python Python from pinecone import Pinecone pc = Pinecone(api_key="YOUR_API_KEY") # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index index = pc.Index(host="INDEX_HOST") # Upsert records into a namespace # `chunk_text` fields are converted to sparse vectors # `category` and `quarter` fields are stored as metadata index.upsert_records( "example-namespace", [ { "_id": "vec1", "chunk_text": "AAPL reported a year-over-year revenue increase, expecting stronger Q3 demand for its flagship phones.", "category": "technology", "quarter": "Q3" }, { "_id": "vec2", "chunk_text": "Analysts suggest that AAPL'\''s upcoming Q4 product launch event might solidify its position in the premium smartphone market.", "category": "technology", "quarter": "Q4" }, { "_id": "vec3", "chunk_text": "AAPL'\''s strategic Q3 partnerships with semiconductor suppliers could mitigate component risks and stabilize iPhone production.", "category": "technology", "quarter": "Q3" }, { "_id": "vec4", "chunk_text": "AAPL may consider healthcare integrations in Q4 to compete with tech rivals entering the consumer wellness space.", "category": "technology", "quarter": "Q4" } ] ) time.sleep(10) # Wait for the upserted vectors to be indexed ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: "YOUR_API_KEY" }) // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index const namespace = pc.index("INDEX_NAME", "INDEX_HOST").namespace("example-namespace"); // Upsert records into a namespace // `chunk_text` fields are converted to sparse vectors // `category` and `quarter` fields are stored as metadata await namespace.upsertRecords([ { "_id": "vec1", "chunk_text": "AAPL reported a year-over-year revenue increase, expecting stronger Q3 demand for its flagship phones.", "category": "technology", "quarter": "Q3" }, { "_id": "vec2", "chunk_text": "Analysts suggest that AAPL'\''s upcoming Q4 product launch event might solidify its position in the premium smartphone market.", "category": "technology", "quarter": "Q4" }, { "_id": "vec3", "chunk_text": "AAPL'\''s strategic Q3 partnerships with semiconductor suppliers could mitigate component risks and stabilize iPhone production.", "category": "technology", "quarter": "Q3" }, { "_id": "vec4", "chunk_text": "AAPL may consider healthcare integrations in Q4 to compete with tech rivals entering the consumer wellness space.", "category": "technology", "quarter": "Q4" } ]); ``` ```java Java import io.pinecone.clients.Index; import io.pinecone.configs.PineconeConfig; import io.pinecone.configs.PineconeConnection; import org.openapitools.db_data.client.ApiException; import java.util.*; public class UpsertText { public static void main(String[] args) throws ApiException { PineconeConfig config = new PineconeConfig("YOUR_API_KEY"); config.setHost("INDEX_HOST"); PineconeConnection connection = new PineconeConnection(config); Index index = new Index(config, connection, "integrated-sparse-java"); ArrayList> upsertRecords = new ArrayList<>(); HashMap record1 = new HashMap<>(); record1.put("_id", "rec1"); record1.put("category", "digestive system"); record1.put("chunk_text", "Apples are a great source of dietary fiber, which supports digestion and helps maintain a healthy gut."); HashMap record2 = new HashMap<>(); record2.put("_id", "rec2"); record2.put("category", "cultivation"); record2.put("chunk_text", "Apples originated in Central Asia and have been cultivated for thousands of years, with over 7,500 varieties available today."); HashMap record3 = new HashMap<>(); record3.put("_id", "rec3"); record3.put("category", "immune system"); record3.put("chunk_text", "Rich in vitamin C and other antioxidants, apples contribute to immune health and may reduce the risk of chronic diseases."); HashMap record4 = new HashMap<>(); record4.put("_id", "rec4"); record4.put("category", "endocrine system"); record4.put("chunk_text", "The high fiber content in apples can also help regulate blood sugar levels, making them a favorable snack for people with diabetes."); upsertRecords.add(record1); upsertRecords.add(record2); upsertRecords.add(record3); upsertRecords.add(record4); index.upsertRecords("example-namespace", upsertRecords); } } ``` ```go Go package main import ( "context" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "INDEX_HOST", Namespace: "example-namespace"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v", err) } // Upsert records into a namespace // `chunk_text` fields are converted to sparse vectors // `category` and `quarter` fields are stored as metadata records := []*pinecone.IntegratedRecord{ { "_id": "vec1", "chunk_text": "AAPL reported a year-over-year revenue increase, expecting stronger Q3 demand for its flagship phones.", "category": "technology", "quarter": "Q3", }, { "_id": "vec2", "chunk_text": "Analysts suggest that AAPL's upcoming Q4 product launch event might solidify its position in the premium smartphone market.", "category": "technology", "quarter": "Q4", }, { "_id": "vec3", "chunk_text": "AAPL's strategic Q3 partnerships with semiconductor suppliers could mitigate component risks and stabilize iPhone production.", "category": "technology", "quarter": "Q3", }, { "_id": "vec4", "chunk_text": "AAPL may consider healthcare integrations in Q4 to compete with tech rivals entering the consumer wellness space.", "category": "technology", "quarter": "Q4", }, } err = idxConnection.UpsertRecords(ctx, records) if err != nil { log.Fatalf("Failed to upsert vectors: %v", err) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); var index = pinecone.Index(host: "INDEX_HOST"); await index.UpsertRecordsAsync( "example-namespace", [ new UpsertRecord { Id = "rec1", AdditionalProperties = { ["chunk_text"] = "AAPL reported a year-over-year revenue increase, expecting stronger Q3 demand for its flagship phones.", ["category"] = "technology", ["quarter"] = "Q3", }, }, new UpsertRecord { Id = "rec2", AdditionalProperties = { ["chunk_text"] = "AAPL may consider healthcare integrations in Q4 to compete with tech rivals entering the consumer wellness space.", ["category"] = "technology", ["quarter"] = "Q4", }, }, new UpsertRecord { Id = "rec3", AdditionalProperties = { ["chunk_text"] = "AAPL may consider healthcare integrations in Q4 to compete with tech rivals entering the consumer wellness space.", ["category"] = "technology", ["quarter"] = "Q4", }, }, new UpsertRecord { Id = "rec4", AdditionalProperties = { ["chunk_text"] = "AAPL's strategic Q3 partnerships with semiconductor suppliers could mitigate component risks and stabilize iPhone production", ["category"] = "technology", ["quarter"] = "Q3", }, }, ] ); ``` ```shell curl INDEX_HOST="INDEX_HOST" NAMESPACE="YOUR_NAMESPACE" PINECONE_API_KEY="YOUR_API_KEY" curl "https://$INDEX_HOST/records/namespaces/$NAMESPACE/upsert" \ -H "Content-Type: application/x-ndjson" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "_id": "vec1", "chunk_text": "AAPL reported a year-over-year revenue increase, expecting stronger Q3 demand for its flagship phones.", "category": "technology", "quarter": "Q3" } { "_id": "vec2", "chunk_text": "Analysts suggest that AAPL'\''s upcoming Q4 product launch event might solidify its position in the premium smartphone market.", "category": "technology", "quarter": "Q4" } { "_id": "vec3", "chunk_text": "AAPL'\''s strategic Q3 partnerships with semiconductor suppliers could mitigate component risks and stabilize iPhone production.", "category": "technology", "quarter": "Q3" } { "_id": "vec4", "chunk_text": "AAPL may consider healthcare integrations in Q4 to compete with tech rivals entering the consumer wellness space.", "category": "technology", "quarter": "Q4" }' ``` To upsert sparse vectors into a [sparse index](/guides/index-data/create-an-index#create-a-sparse-index), use the [`upsert`](/reference/api/2025-01/data-plane/upsert) operation as follows: * Specify the [`namespace`](/guides/index-data/indexing-overview#namespaces) to upsert into. If the namespace doesn't exist, it is created. To use the default namespace, set the namespace to `"__default__"`. * Format your input data as records, each with the following: * An `id` field with a unique record identifier for the index namespace. * A `sparse_values` field with the sparse vector values and indices. * Optionally, a `metadata` field with key-value pairs to store additional information or context. When you query the index, you can then filter by metadata to ensure only relevant records are scanned. For more information, see [Metadata Filtering](/guides/index-data/indexing-overview#metadata). For example, the following code upserts sparse vector representations of sentences related to the term "apple", with the source text and additional fields stored as metadata: ```python Python from pinecone import Pinecone, SparseValues, Vector pc = Pinecone(api_key="YOUR_API_KEY") # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index index = pc.Index(host="INDEX_HOST") index.upsert( namespace="example-namespace", vectors=[ { "id": "vec1", "sparse_values": { "values": [1.7958984, 0.41577148, 2.828125, 2.8027344, 2.8691406, 1.6533203, 5.3671875, 1.3046875, 0.49780273, 0.5722656, 2.71875, 3.0820312, 2.5019531, 4.4414062, 3.3554688], "indices": [822745112, 1009084850, 1221765879, 1408993854, 1504846510, 1596856843, 1640781426, 1656251611, 1807131503, 2543655733, 2902766088, 2909307736, 3246437992, 3517203014, 3590924191] }, "metadata": { "chunk_text": "AAPL reported a year-over-year revenue increase, expecting stronger Q3 demand for its flagship phones.", "category": "technology", "quarter": "Q3" } }, { "id": "vec2", "sparse_values": { "values": [0.4362793, 3.3457031, 2.7714844, 3.0273438, 3.3164062, 5.6015625, 2.4863281, 0.38134766, 1.25, 2.9609375, 0.34179688, 1.4306641, 0.34375, 3.3613281, 1.4404297, 2.2558594, 2.2597656, 4.8710938, 0.5605469], "indices": [131900689, 592326839, 710158994, 838729363, 1304885087, 1640781426, 1690623792, 1807131503, 2066971792, 2428553208, 2548600401, 2577534050, 3162218338, 3319279674, 3343062801, 3476647774, 3485013322, 3517203014, 4283091697] }, "metadata": { "chunk_text": "Analysts suggest that AAPL'\''s upcoming Q4 product launch event might solidify its position in the premium smartphone market.", "category": "technology", "quarter": "Q4" } }, { "id": "vec3", "sparse_values": { "values": [2.6875, 4.2929688, 3.609375, 3.0722656, 2.1152344, 5.78125, 3.7460938, 3.7363281, 1.2695312, 3.4824219, 0.7207031, 0.0826416, 4.671875, 3.7011719, 2.796875, 0.61621094], "indices": [8661920, 350356213, 391213188, 554637446, 1024951234, 1640781426, 1780689102, 1799010313, 2194093370, 2632344667, 2641553256, 2779594451, 3517203014, 3543799498, 3837503950, 4283091697] }, "metadata": { "chunk_text": "AAPL'\''s strategic Q3 partnerships with semiconductor suppliers could mitigate component risks and stabilize iPhone production", "category": "technology", "quarter": "Q3" } }, { "id": "vec4", "sparse_values": { "values": [0.73046875, 0.46972656, 2.84375, 5.2265625, 3.3242188, 1.9863281, 0.9511719, 0.5019531, 4.4257812, 3.4277344, 0.41308594, 4.3242188, 2.4179688, 3.1757812, 1.0224609, 2.0585938, 2.5859375], "indices": [131900689, 152217691, 441495248, 1640781426, 1851149807, 2263326288, 2502307765, 2641553256, 2684780967, 2966813704, 3162218338, 3283104238, 3488055477, 3530642888, 3888762515, 4152503047, 4177290673] }, "metadata": { "chunk_text": "AAPL may consider healthcare integrations in Q4 to compete with tech rivals entering the consumer wellness space.", "category": "technology", "quarter": "Q4" } } ] ) ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index const index = pc.index("INDEX_NAME", "INDEX_HOST") await index.namespace('example-namespace').upsert([ { id: 'vec1', sparseValues: { indices: [822745112, 1009084850, 1221765879, 1408993854, 1504846510, 1596856843, 1640781426, 1656251611, 1807131503, 2543655733, 2902766088, 2909307736, 3246437992, 3517203014, 3590924191], values: [1.7958984, 0.41577148, 2.828125, 2.8027344, 2.8691406, 1.6533203, 5.3671875, 1.3046875, 0.49780273, 0.5722656, 2.71875, 3.0820312, 2.5019531, 4.4414062, 3.3554688] }, metadata: { chunk_text: 'AAPL reported a year-over-year revenue increase, expecting stronger Q3 demand for its flagship phones.', category: 'technology', quarter: 'Q3' } }, { id: 'vec2', sparseValues: { indices: [131900689, 592326839, 710158994, 838729363, 1304885087, 1640781426, 1690623792, 1807131503, 2066971792, 2428553208, 2548600401, 2577534050, 3162218338, 3319279674, 3343062801, 3476647774, 3485013322, 3517203014, 4283091697], values: [0.4362793, 3.3457031, 2.7714844, 3.0273438, 3.3164062, 5.6015625, 2.4863281, 0.38134766, 1.25, 2.9609375, 0.34179688, 1.4306641, 0.34375, 3.3613281, 1.4404297, 2.2558594, 2.2597656, 4.8710938, 0.5605469] }, metadata: { chunk_text: "Analysts suggest that AAPL's upcoming Q4 product launch event might solidify its position in the premium smartphone market.", category: 'technology', quarter: 'Q4' } }, { id: 'vec3', sparseValues: { indices: [8661920, 350356213, 391213188, 554637446, 1024951234, 1640781426, 1780689102, 1799010313, 2194093370, 2632344667, 2641553256, 2779594451, 3517203014, 3543799498, 3837503950, 4283091697], values: [2.6875, 4.2929688, 3.609375, 3.0722656, 2.1152344, 5.78125, 3.7460938, 3.7363281, 1.2695312, 3.4824219, 0.7207031, 0.0826416, 4.671875, 3.7011719, 2.796875, 0.61621094] }, metadata: { chunk_text: "AAPL's strategic Q3 partnerships with semiconductor suppliers could mitigate component risks and stabilize iPhone production", category: 'technology', quarter: 'Q3' } }, { id: 'vec4', sparseValues: { indices: [131900689, 152217691, 441495248, 1640781426, 1851149807, 2263326288, 2502307765, 2641553256, 2684780967, 2966813704, 3162218338, 3283104238, 3488055477, 3530642888, 3888762515, 4152503047, 4177290673], values: [0.73046875, 0.46972656, 2.84375, 5.2265625, 3.3242188, 1.9863281, 0.9511719, 0.5019531, 4.4257812, 3.4277344, 0.41308594, 4.3242188, 2.4179688, 3.1757812, 1.0224609, 2.0585938, 2.5859375] }, metadata: { chunk_text: 'AAPL may consider healthcare integrations in Q4 to compete with tech rivals entering the consumer wellness space.', category: 'technology', quarter: 'Q4' } } ]); ``` ```java Java import io.pinecone.clients.Pinecone; import io.pinecone.clients.Index; import com.google.protobuf.Struct; import com.google.protobuf.Value; import java.util.*; public class UpsertSparseVectors { public static void main(String[] args) throws InterruptedException { // Instantiate Pinecone class Pinecone pinecone = new Pinecone.Builder("YOUR_API)KEY").build(); Index index = pinecone.getIndexConnection("docs-example"); // Record 1 ArrayList indices1 = new ArrayList<>(Arrays.asList( 822745112L, 1009084850L, 1221765879L, 1408993854L, 1504846510L, 1596856843L, 1640781426L, 1656251611L, 1807131503L, 2543655733L, 2902766088L, 2909307736L, 3246437992L, 3517203014L, 3590924191L )); ArrayList values1 = new ArrayList<>(Arrays.asList( 1.7958984f, 0.41577148f, 2.828125f, 2.8027344f, 2.8691406f, 1.6533203f, 5.3671875f, 1.3046875f, 0.49780273f, 0.5722656f, 2.71875f, 3.0820312f, 2.5019531f, 4.4414062f, 3.3554688f )); Struct metaData1 = Struct.newBuilder() .putFields("chunk_text", Value.newBuilder().setStringValue("AAPL reported a year-over-year revenue increase, expecting stronger Q3 demand for its flagship phones.").build()) .putFields("category", Value.newBuilder().setStringValue("technology").build()) .putFields("quarter", Value.newBuilder().setStringValue("Q3").build()) .build(); // Record 2 ArrayList indices2 = new ArrayList<>(Arrays.asList( 131900689L, 592326839L, 710158994L, 838729363L, 1304885087L, 1640781426L, 1690623792L, 1807131503L, 2066971792L, 2428553208L, 2548600401L, 2577534050L, 3162218338L, 3319279674L, 3343062801L, 3476647774L, 3485013322L, 3517203014L, 4283091697L )); ArrayList values2 = new ArrayList<>(Arrays.asList( 0.4362793f, 3.3457031f, 2.7714844f, 3.0273438f, 3.3164062f, 5.6015625f, 2.4863281f, 0.38134766f, 1.25f, 2.9609375f, 0.34179688f, 1.4306641f, 0.34375f, 3.3613281f, 1.4404297f, 2.2558594f, 2.2597656f, 4.8710938f, 0.5605469f )); Struct metaData2 = Struct.newBuilder() .putFields("chunk_text", Value.newBuilder().setStringValue("Analysts suggest that AAPL'\\''s upcoming Q4 product launch event might solidify its position in the premium smartphone market.").build()) .putFields("category", Value.newBuilder().setStringValue("technology").build()) .putFields("quarter", Value.newBuilder().setStringValue("Q4").build()) .build(); // Record 3 ArrayList indices3 = new ArrayList<>(Arrays.asList( 8661920L, 350356213L, 391213188L, 554637446L, 1024951234L, 1640781426L, 1780689102L, 1799010313L, 2194093370L, 2632344667L, 2641553256L, 2779594451L, 3517203014L, 3543799498L, 3837503950L, 4283091697L )); ArrayList values3 = new ArrayList<>(Arrays.asList( 2.6875f, 4.2929688f, 3.609375f, 3.0722656f, 2.1152344f, 5.78125f, 3.7460938f, 3.7363281f, 1.2695312f, 3.4824219f, 0.7207031f, 0.0826416f, 4.671875f, 3.7011719f, 2.796875f, 0.61621094f )); Struct metaData3 = Struct.newBuilder() .putFields("chunk_text", Value.newBuilder().setStringValue("AAPL'\\''s strategic Q3 partnerships with semiconductor suppliers could mitigate component risks and stabilize iPhone production").build()) .putFields("category", Value.newBuilder().setStringValue("technology").build()) .putFields("quarter", Value.newBuilder().setStringValue("Q3").build()) .build(); // Record 4 ArrayList indices4 = new ArrayList<>(Arrays.asList( 131900689L, 152217691L, 441495248L, 1640781426L, 1851149807L, 2263326288L, 2502307765L, 2641553256L, 2684780967L, 2966813704L, 3162218338L, 3283104238L, 3488055477L, 3530642888L, 3888762515L, 4152503047L, 4177290673L )); ArrayList values4 = new ArrayList<>(Arrays.asList( 0.73046875f, 0.46972656f, 2.84375f, 5.2265625f, 3.3242188f, 1.9863281f, 0.9511719f, 0.5019531f, 4.4257812f, 3.4277344f, 0.41308594f, 4.3242188f, 2.4179688f, 3.1757812f, 1.0224609f, 2.0585938f, 2.5859375f )); Struct metaData4 = Struct.newBuilder() .putFields("chunk_text", Value.newBuilder().setStringValue("AAPL may consider healthcare integrations in Q4 to compete with tech rivals entering the consumer wellness space").build()) .putFields("category", Value.newBuilder().setStringValue("technology").build()) .putFields("quarter", Value.newBuilder().setStringValue("Q4").build()) .build(); index.upsert("vec1", Collections.emptyList(), indices1, values1, metaData1, "example-namespace"); index.upsert("vec2", Collections.emptyList(), indices2, values2, metaData2, "example-namespace"); index.upsert("vec3", Collections.emptyList(), indices3, values3, metaData3, "example-namespace"); index.upsert("vec4", Collections.emptyList(), indices4, values4, metaData4, "example-namespace"); ``` ```go Go package main import ( "context" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" "google.golang.org/protobuf/types/known/structpb" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "INDEX_HOST", Namespace: "example-namespace"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v", err) } sparseValues1 := pinecone.SparseValues{ Indices: []uint32{822745112, 1009084850, 1221765879, 1408993854, 1504846510, 1596856843, 1640781426, 1656251611, 1807131503, 2543655733, 2902766088, 2909307736, 3246437992, 3517203014, 3590924191}, Values: []float32{1.7958984, 0.41577148, 2.828125, 2.8027344, 2.8691406, 1.6533203, 5.3671875, 1.3046875, 0.49780273, 0.5722656, 2.71875, 3.0820312, 2.5019531, 4.4414062, 3.3554688}, } metadataMap1 := map[string]interface{}{ "chunk_text": "AAPL reported a year-over-year revenue increase, expecting stronger Q3 demand for its flagship phones", "category": "technology", "quarter": "Q3", } metadata1, err := structpb.NewStruct(metadataMap1) if err != nil { log.Fatalf("Failed to create metadata map: %v", err) } sparseValues2 := pinecone.SparseValues{ Indices: []uint32{131900689, 592326839, 710158994, 838729363, 1304885087, 1640781426, 1690623792, 1807131503, 2066971792, 2428553208, 2548600401, 2577534050, 3162218338, 3319279674, 3343062801, 3476647774, 3485013322, 3517203014, 4283091697}, Values: []float32{0.4362793, 3.3457031, 2.7714844, 3.0273438, 3.3164062, 5.6015625, 2.4863281, 0.38134766, 1.25, 2.9609375, 0.34179688, 1.4306641, 0.34375, 3.3613281, 1.4404297, 2.2558594, 2.2597656, 4.8710938, 0.560546}, } metadataMap2 := map[string]interface{}{ "chunk_text": "Analysts suggest that AAPL's upcoming Q4 product launch event might solidify its position in the premium smartphone market.", "category": "technology", "quarter": "Q4", } metadata2, err := structpb.NewStruct(metadataMap2) if err != nil { log.Fatalf("Failed to create metadata map: %v", err) } sparseValues3 := pinecone.SparseValues{ Indices: []uint32{8661920, 350356213, 391213188, 554637446, 1024951234, 1640781426, 1780689102, 1799010313, 2194093370, 2632344667, 2641553256, 2779594451, 3517203014, 3543799498, 3837503950, 4283091697}, Values: []float32{2.6875, 4.2929688, 3.609375, 3.0722656, 2.1152344, 5.78125, 3.7460938, 3.7363281, 1.2695312, 3.4824219, 0.7207031, 0.0826416, 4.671875, 3.7011719, 2.796875, 0.61621094}, } metadataMap3 := map[string]interface{}{ "chunk_text": "AAPL's strategic Q3 partnerships with semiconductor suppliers could mitigate component risks and stabilize iPhone production", "category": "technology", "quarter": "Q3", } metadata3, err := structpb.NewStruct(metadataMap3) if err != nil { log.Fatalf("Failed to create metadata map: %v", err) } sparseValues4 := pinecone.SparseValues{ Indices: []uint32{131900689, 152217691, 441495248, 1640781426, 1851149807, 2263326288, 2502307765, 2641553256, 2684780967, 2966813704, 3162218338, 3283104238, 3488055477, 3530642888, 3888762515, 4152503047, 4177290673}, Values: []float32{0.73046875, 0.46972656, 2.84375, 5.2265625, 3.3242188, 1.9863281, 0.9511719, 0.5019531, 4.4257812, 3.4277344, 0.41308594, 4.3242188, 2.4179688, 3.1757812, 1.0224609, 2.0585938, 2.5859375}, } metadataMap4 := map[string]interface{}{ "chunk_text": "AAPL may consider healthcare integrations in Q4 to compete with tech rivals entering the consumer wellness space.", "category": "technology", "quarter": "Q4", } metadata4, err := structpb.NewStruct(metadataMap4) if err != nil { log.Fatalf("Failed to create metadata map: %v", err) } vectors := []*pinecone.Vector{ { Id: "vec1", SparseValues: &sparseValues1, Metadata: metadata1, }, { Id: "vec2", SparseValues: &sparseValues2, Metadata: metadata2, }, { Id: "vec3", SparseValues: &sparseValues3, Metadata: metadata3, }, { Id: "vec4", SparseValues: &sparseValues4, Metadata: metadata4, }, } count, err := idxConnection.UpsertVectors(ctx, vectors) if err != nil { log.Fatalf("Failed to upsert vectors: %v", err) } else { fmt.Printf("Successfully upserted %d vector(s)!\n", count) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); var index = pinecone.Index("docs-example"); var vector1 = new Vector { Id = "vec1", SparseValues = new SparseValues { Indices = new uint[] { 822745112, 1009084850, 1221765879, 1408993854, 1504846510, 1596856843, 1640781426, 1656251611, 1807131503, 2543655733, 2902766088, 2909307736, 3246437992, 3517203014, 3590924191 }, Values = new ReadOnlyMemory([1.7958984f, 0.41577148f, 2.828125f, 2.8027344f, 2.8691406f, 1.6533203f, 5.3671875f, 1.3046875f, 0.49780273f, 0.5722656f, 2.71875f, 3.0820312f, 2.5019531f, 4.4414062f, 3.3554688f]) }, Metadata = new Metadata { ["chunk_text"] = new("AAPL reported a year-over-year revenue increase, expecting stronger Q3 demand for its flagship phones."), ["category"] = new("technology"), ["quarter"] = new("Q3"), }, }; var vector2 = new Vector { Id = "vec2", SparseValues = new SparseValues { Indices = new uint[] { 131900689, 592326839, 710158994, 838729363, 1304885087, 1640781426, 1690623792, 1807131503, 2066971792, 2428553208, 2548600401, 2577534050, 3162218338, 3319279674, 3343062801, 3476647774, 3485013322, 3517203014, 4283091697 }, Values = new ReadOnlyMemory([0.4362793f, 3.3457031f, 2.7714844f, 3.0273438f, 3.3164062f, 5.6015625f, 2.4863281f, 0.38134766f, 1.25f, 2.9609375f, 0.34179688f, 1.4306641f, 0.34375f, 3.3613281f, 1.4404297f, 2.2558594f, 2.2597656f, 4.8710938f, 0.5605469f]) }, Metadata = new Metadata { ["chunk_text"] = new("Analysts suggest that AAPL'\''s upcoming Q4 product launch event might solidify its position in the premium smartphone market."), ["category"] = new("technology"), ["quarter"] = new("Q4"), }, }; var vector3 = new Vector { Id = "vec3", SparseValues = new SparseValues { Indices = new uint[] { 8661920, 350356213, 391213188, 554637446, 1024951234, 1640781426, 1780689102, 1799010313, 2194093370, 2632344667, 2641553256, 2779594451, 3517203014, 3543799498, 3837503950, 4283091697 }, Values = new ReadOnlyMemory([2.6875f, 4.2929688f, 3.609375f, 3.0722656f, 2.1152344f, 5.78125f, 3.7460938f, 3.7363281f, 1.2695312f, 3.4824219f, 0.7207031f, 0.0826416f, 4.671875f, 3.7011719f, 2.796875f, 0.61621094f]) }, Metadata = new Metadata { ["chunk_text"] = new("AAPL'\''s strategic Q3 partnerships with semiconductor suppliers could mitigate component risks and stabilize iPhone production"), ["category"] = new("technology"), ["quarter"] = new("Q3"), }, }; var vector4 = new Vector { Id = "vec4", SparseValues = new SparseValues { Indices = new uint[] { 131900689, 152217691, 441495248, 1640781426, 1851149807, 2263326288, 2502307765, 2641553256, 2684780967, 2966813704, 3162218338, 3283104238, 3488055477, 3530642888, 3888762515, 4152503047, 4177290673 }, Values = new ReadOnlyMemory([0.73046875f, 0.46972656f, 2.84375f, 5.2265625f, 3.3242188f, 1.9863281f, 0.9511719f, 0.5019531f, 4.4257812f, 3.4277344f, 0.41308594f, 4.3242188f, 2.4179688f, 3.1757812f, 1.0224609f, 2.0585938f, 2.5859375f]) }, Metadata = new Metadata { ["chunk_text"] = new("AAPL may consider healthcare integrations in Q4 to compete with tech rivals entering the consumer wellness space."), ["category"] = new("technology"), ["quarter"] = new("Q4"), }, }; // Upsert vector Console.WriteLine("Upserting vector..."); var upsertResponse = await index.UpsertAsync(new UpsertRequest { Vectors = new List { vector1, vector2, vector3, vector4 }, Namespace = "example-namespace" }); Console.WriteLine($"Upserted {upsertResponse.UpsertedCount} vector"); ``` ```shell curl INDEX_HOST="INDEX_HOST" PINECONE_API_KEY="YOUR_API_KEY" curl "http://$INDEX_HOST/vectors/upsert" \ -H "Content-Type: application/json" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "namespace": "example-namespace", "vectors": [ { "id": "vec1", "sparseValues": { "values": [1.7958984, 0.41577148, 2.828125, 2.8027344, 2.8691406, 1.6533203, 5.3671875, 1.3046875, 0.49780273, 0.5722656, 2.71875, 3.0820312, 2.5019531, 4.4414062, 3.3554688], "indices": [822745112, 1009084850, 1221765879, 1408993854, 1504846510, 1596856843, 1640781426, 1656251611, 1807131503, 2543655733, 2902766088, 2909307736, 3246437992, 3517203014, 3590924191] }, "metadata": { "chunk_text": "AAPL reported a year-over-year revenue increase, expecting stronger Q3 demand for its flagship phones.", "category": "technology", "quarter": "Q3" } }, { "id": "vec2", "sparseValues": { "values": [0.4362793, 3.3457031, 2.7714844, 3.0273438, 3.3164062, 5.6015625, 2.4863281, 0.38134766, 1.25, 2.9609375, 0.34179688, 1.4306641, 0.34375, 3.3613281, 1.4404297, 2.2558594, 2.2597656, 4.8710938, 0.5605469], "indices": [131900689, 592326839, 710158994, 838729363, 1304885087, 1640781426, 1690623792, 1807131503, 2066971792, 2428553208, 2548600401, 2577534050, 3162218338, 3319279674, 3343062801, 3476647774, 3485013322, 3517203014, 4283091697] }, "metadata": { "chunk_text": "Analysts suggest that AAPL'\''s upcoming Q4 product launch event might solidify its position in the premium smartphone market.", "category": "technology", "quarter": "Q4" } }, { "id": "vec3", "sparseValues": { "values": [2.6875, 4.2929688, 3.609375, 3.0722656, 2.1152344, 5.78125, 3.7460938, 3.7363281, 1.2695312, 3.4824219, 0.7207031, 0.0826416, 4.671875, 3.7011719, 2.796875, 0.61621094], "indices": [8661920, 350356213, 391213188, 554637446, 1024951234, 1640781426, 1780689102, 1799010313, 2194093370, 2632344667, 2641553256, 2779594451, 3517203014, 3543799498, 3837503950, 4283091697] }, "metadata": { "chunk_text": "AAPL'\''s strategic Q3 partnerships with semiconductor suppliers could mitigate component risks and stabilize iPhone production", "category": "technology", "quarter": "Q3" } }, { "id": "vec4", "sparseValues": { "values": [0.73046875, 0.46972656, 2.84375, 5.2265625, 3.3242188, 1.9863281, 0.9511719, 0.5019531, 4.4257812, 3.4277344, 0.41308594, 4.3242188, 2.4179688, 3.1757812, 1.0224609, 2.0585938, 2.5859375], "indices": [131900689, 152217691, 441495248, 1640781426, 1851149807, 2263326288, 2502307765, 2641553256, 2684780967, 2966813704, 3162218338, 3283104238, 3488055477, 3530642888, 3888762515, 4152503047, 4177290673] }, "metadata": { "chunk_text": "AAPL may consider healthcare integrations in Q4 to compete with tech rivals entering the consumer wellness space.", "category": "technology", "quarter": "Q4" } }, ] }' ``` ## Upsert in batches When upserting larger amounts of data, it is recommended to upsert records in large batches. A batch of upserts should be as large as possible (up to 1000 records) without exceeding the [maximum request size of 2MB](/reference/api/database-limits#upsert-limits). To understand the number of records you can fit into one batch based on the vector dimensions and metadata size, see the following table: | Dimension | Metadata (bytes) | Max batch size | | :-------- | :--------------- | :------------- | | 386 | 0 | 1000 | | 768 | 500 | 559 | | 1536 | 2000 | 245 | To control costs when ingesting very large datasets (10,000,000+ records), use [`import`](/guides/index-data/import-data) instead of `upsert`. ```Python Python import random import itertools from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key="YOUR_API_KEY") # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index index = pc.Index(host="INDEX_HOST") def chunks(iterable, batch_size=200): """A helper function to break an iterable into chunks of size batch_size.""" it = iter(iterable) chunk = tuple(itertools.islice(it, batch_size)) while chunk: yield chunk chunk = tuple(itertools.islice(it, batch_size)) vector_dim = 128 vector_count = 10000 # Example generator that generates many (id, vector) pairs example_data_generator = map(lambda i: (f'id-{i}', [random.random() for _ in range(vector_dim)]), range(vector_count)) # Upsert data with 200 vectors per upsert request for ids_vectors_chunk in chunks(example_data_generator, batch_size=200): index.upsert(vectors=ids_vectors_chunk) ``` ```JavaScript JavaScript import { Pinecone } from "@pinecone-database/pinecone"; const RECORD_COUNT = 10000; const RECORD_DIMENSION = 128; const client = new Pinecone({ apiKey: "YOUR_API_KEY" }); const index = client.index("docs-example"); // A helper function that breaks an array into chunks of size batchSize const chunks = (array, batchSize = 200) => { const chunks = []; for (let i = 0; i < array.length; i += batchSize) { chunks.push(array.slice(i, i + batchSize)); } return chunks; }; // Example data generation function, creates many (id, vector) pairs const generateExampleData = () => Array.from({ length: RECORD_COUNT }, (_, i) => { return { id: `id-${i}`, values: Array.from({ length: RECORD_DIMENSION }, (_, i) => Math.random()), }; }); const exampleRecordData = generateExampleData(); const recordChunks = chunks(exampleRecordData); // Upsert data with 200 records per upsert request for (const chunk of recordChunks) { await index.upsert(chunk) } ``` ```java Java import io.pinecone.clients.Index; import io.pinecone.configs.PineconeConfig; import io.pinecone.configs.PineconeConnection; import io.pinecone.unsigned_indices_model.VectorWithUnsignedIndices; import java.util.Arrays; import java.util.List; public class UpsertBatchExample { public static void main(String[] args) { PineconeConfig config = new PineconeConfig("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index config.setHost("INDEX_HOST"); PineconeConnection connection = new PineconeConnection(config); Index index = new Index(connection, "INDEX_NAME"); ArrayList vectors = generateVectors(); ArrayList> chunks = chunks(vectors, BATCH_SIZE); for (ArrayList chunk : chunks) { index.upsert(chunk, "example-namespace"); } } // A helper function that breaks an ArrayList into chunks of batchSize private static ArrayList> chunks(ArrayList vectors, int batchSize) { ArrayList> chunks = new ArrayList<>(); ArrayList chunk = new ArrayList<>(); for (int i = 0; i < vectors.size(); i++) { if (i % BATCH_SIZE == 0 && i != 0) { chunks.add(chunk); chunk = new ArrayList<>(); } chunk.add(vectors.get(i)); } return chunks; } // Example data generation function, creates many (id, vector) pairs private static ArrayList generateVectors() { Random random = new Random(); ArrayList vectors = new ArrayList<>(); for (int i = 0; i <= RECORD_COUNT; i++) { String id = "id-" + i; ArrayList values = new ArrayList<>(); for (int j = 0; j < RECORD_DIMENSION; j++) { values.add(random.nextFloat()); } VectorWithUnsignedIndices vector = new VectorWithUnsignedIndices(); vector.setId(id); vector.setValues(values); vectors.add(vector); } return vectors; } } ``` ```go Go package main import ( "context" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "INDEX_HOST"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v", err) } // Generate a large number of vectors to upsert vectorCount := 10000 vectorDim := idx.Dimension vectors := make([]*pinecone.Vector, vectorCount) for i := 0; i < int(vectorCount); i++ { randomFloats := make([]float32, vectorDim) for i := int32(0); i < vectorDim; i++ { randomFloats[i] = rand.Float32() } vectors[i] = &pinecone.Vector{ Id: fmt.Sprintf("doc1#-vector%d", i), Values: randomFloats, } } // Break the vectors into batches of 200 var batches [][]*pinecone.Vector batchSize := 200 for len(vectors) > 0 { batchEnd := batchSize if len(vectors) < batchSize { batchEnd = len(vectors) } batches = append(batches, vectors[:batchEnd]) vectors = vectors[batchEnd:] } // Upsert batches for i, batch := range batches { upsertResp, err := idxConn.UpsertVectors(context.Background(), batch) if err != nil { panic(err) } fmt.Printf("upserted %d vectors (%v of %v batches)\n", upsertResp, i+1, len(batches)) } } ``` ## Upsert in parallel Python SDK v6.0.0 and later provide `async` methods for use with [asyncio](https://docs.python.org/3/library/asyncio.html). Asyncio support makes it possible to use Pinecone with modern async web frameworks such as FastAPI, Quart, and Sanic. For more details, see [Asyncio support](/reference/python-sdk#asyncio-support). Send multiple upserts in parallel to help increase throughput. Vector operations block until the response has been received. However, they can be made asynchronously as follows: ```Python Python # This example uses `async_req=True` and multiple threads. # For a single-threaded approach compatible with modern async web frameworks, # see https://docs.pinecone.io/reference/python-sdk#asyncio-support import random import itertools from pinecone import Pinecone # Initialize the client with pool_threads=30. This limits simultaneous requests to 30. pc = Pinecone(api_key="YOUR_API_KEY", pool_threads=30) # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index index = pc.Index(host="INDEX_HOST") def chunks(iterable, batch_size=200): """A helper function to break an iterable into chunks of size batch_size.""" it = iter(iterable) chunk = tuple(itertools.islice(it, batch_size)) while chunk: yield chunk chunk = tuple(itertools.islice(it, batch_size)) vector_dim = 128 vector_count = 10000 example_data_generator = map(lambda i: (f'id-{i}', [random.random() for _ in range(vector_dim)]), range(vector_count)) # Upsert data with 200 vectors per upsert request asynchronously # - Pass async_req=True to index.upsert() with pc.Index(host="INDEX_HOST", pool_threads=30) as index: # Send requests in parallel async_results = [ index.upsert(vectors=ids_vectors_chunk, async_req=True) for ids_vectors_chunk in chunks(example_data_generator, batch_size=200) ] # Wait for and retrieve responses (this raises in case of error) [async_result.get() for async_result in async_results] ``` ```JavaScript JavaScript import { Pinecone } from "@pinecone-database/pinecone"; const RECORD_COUNT = 10000; const RECORD_DIMENSION = 128; const client = new Pinecone({ apiKey: "YOUR_API_KEY" }); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index const index = pc.index("INDEX_NAME", "INDEX_HOST") // A helper function that breaks an array into chunks of size batchSize const chunks = (array, batchSize = 200) => { const chunks = []; for (let i = 0; i < array.length; i += batchSize) { chunks.push(array.slice(i, i + batchSize)); } return chunks; }; // Example data generation function, creates many (id, vector) pairs const generateExampleData = () => Array.from({ length: RECORD_COUNT }, (_, i) => { return { id: `id-${i}`, values: Array.from({ length: RECORD_DIMENSION }, (_, i) => Math.random()), }; }); const exampleRecordData = generateExampleData(); const recordChunks = chunks(exampleRecordData); // Upsert data with 200 records per request asynchronously using Promise.all() await Promise.all(recordChunks.map((chunk) => index.upsert(chunk))); ``` ```java Java import com.google.protobuf.Struct; import com.google.protobuf.Value; import io.pinecone.clients.Index; import io.pinecone.configs.PineconeConfig; import io.pinecone.configs.PineconeConnection; import io.pinecone.proto.UpsertResponse; import io.pinecone.unsigned_indices_model.VectorWithUnsignedIndices; import java.util.ArrayList; import java.util.Arrays; import java.util.concurrent.ExecutorService; import java.util.concurrent.Executors; import java.util.List; public class UpsertExample { public static void main(String[] args) { PineconeConfig config = new PineconeConfig("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index config.setHost("INDEX_HOST"); PineconeConnection connection = new PineconeConnection(config); Index index = new Index(connection, "INDEX_NAME"); // Run 5 threads concurrently and upsert data into pinecone int numberOfThreads = 5; // Create a fixed thread pool ExecutorService executor = Executors.newFixedThreadPool(numberOfThreads); // Submit tasks to the executor for (int i = 0; i < numberOfThreads; i++) { // upsertData int batchNumber = i+1; executor.submit(() -> upsertData(index, batchNumber)); } // Shutdown the executor executor.shutdown(); } private static void upsertData(Index index, int batchNumber) { // Vector ids to be upserted String prefix = "v" + batchNumber; List upsertIds = Arrays.asList(prefix + "_1", prefix + "_2", prefix + "_3"); // List of values to be upserted List> values = new ArrayList<>(); values.add(Arrays.asList(1.0f, 2.0f, 3.0f)); values.add(Arrays.asList(4.0f, 5.0f, 6.0f)); values.add(Arrays.asList(7.0f, 8.0f, 9.0f)); // List of sparse indices to be upserted List> sparseIndices = new ArrayList<>(); sparseIndices.add(Arrays.asList(1L, 2L, 3L)); sparseIndices.add(Arrays.asList(4L, 5L, 6L)); sparseIndices.add(Arrays.asList(7L, 8L, 9L)); // List of sparse values to be upserted List> sparseValues = new ArrayList<>(); sparseValues.add(Arrays.asList(1000f, 2000f, 3000f)); sparseValues.add(Arrays.asList(4000f, 5000f, 6000f)); sparseValues.add(Arrays.asList(7000f, 8000f, 9000f)); List vectors = new ArrayList<>(3); // Metadata to be upserted Struct metadataStruct1 = Struct.newBuilder() .putFields("genre", Value.newBuilder().setStringValue("action").build()) .putFields("year", Value.newBuilder().setNumberValue(2019).build()) .build(); Struct metadataStruct2 = Struct.newBuilder() .putFields("genre", Value.newBuilder().setStringValue("thriller").build()) .putFields("year", Value.newBuilder().setNumberValue(2020).build()) .build(); Struct metadataStruct3 = Struct.newBuilder() .putFields("genre", Value.newBuilder().setStringValue("comedy").build()) .putFields("year", Value.newBuilder().setNumberValue(2021).build()) .build(); List metadataStructList = Arrays.asList(metadataStruct1, metadataStruct2, metadataStruct3); // Upsert data for (int i = 0; i < metadataStructList.size(); i++) { vectors.add(buildUpsertVectorWithUnsignedIndices(upsertIds.get(i), values.get(i), sparseIndices.get(i), sparseValues.get(i), metadataStructList.get(i))); } UpsertResponse upsertResponse = index.upsert(vectors, "example-namespace"); } } ``` ```go Go package main import ( "context" "fmt" "log" "math/rand" "sync" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index idxConn, err := pc.Index(pinecone.NewIndexConnParams{Host: "INDEX_HOST"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v", err) } // Generate a large number of vectors to upsert vectorCount := 10000 vectorDim := idx.Dimension vectors := make([]*pinecone.Vector, vectorCount) for i := 0; i < int(vectorCount); i++ { randomFloats := make([]float32, vectorDim) for i := int32(0); i < vectorDim; i++ { randomFloats[i] = rand.Float32() } vectors[i] = &pinecone.Vector{ Id: fmt.Sprintf("doc1#-vector%d", i), Values: randomFloats, } } // Break the vectors into batches of 200 var batches [][]*pinecone.Vector batchSize := 200 for len(vectors) > 0 { batchEnd := batchSize if len(vectors) < batchSize { batchEnd = len(vectors) } batches = append(batches, vectors[:batchEnd]) vectors = vectors[batchEnd:] } // Use channels to manage concurrency and possible errors maxConcurrency := 10 errChan := make(chan error, len(batches)) semaphore := make(chan struct{}, maxConcurrency) var wg sync.WaitGroup for i, batch := range batches { wg.Add(1) semaphore <- struct{}{} go func(batch []*pinecone.Vector, i int) { defer wg.Done() defer func() { <-semaphore }() upsertResp, err := idxConn.UpsertVectors(context.Background(), batch) if err != nil { errChan <- fmt.Errorf("batch %d failed: %v", i, err) return } fmt.Printf("upserted %d vectors (%v of %v batches)\n", upsertResp, i+1, len(batches)) }(batch, i) } wg.Wait() close(errChan) for err := range errChan { if err != nil { fmt.Printf("Error while upserting batch: %v\n", err) } } } ``` ### Python SDK with gRPC Using the Python SDK with gRPC extras can provide higher upsert speeds. Through multiplexing, gRPC is able to handle large amounts of requests in parallel without slowing down the rest of the system (HoL blocking), unlike REST. Moreover, you can pass various retry strategies to the gRPC SDK, including exponential backoffs. To install the gRPC version of the SDK: ```Shell Shell pip install "pinecone[grpc]" ``` To use the gRPC SDK, import the `pinecone.grpc` subpackage and target an index as usual: ```Python Python from pinecone.grpc import PineconeGRPC as Pinecone # This is gRPC client aliased as "Pinecone" pc = Pinecone(api_key='YOUR_API_KEY') # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index index = pc.Index(host="INDEX_HOST") ``` To launch multiple read and write requests in parallel, pass `async_req` to the `upsert` operation: ```Python Python def chunker(seq, batch_size): return (seq[pos:pos + batch_size] for pos in range(0, len(seq), batch_size)) async_results = [ index.upsert(vectors=chunk, async_req=True) for chunk in chunker(data, batch_size=200) ] # Wait for and retrieve responses (in case of error) [async_result.result() for async_result in async_results] ``` It is possible to get write-throttled faster when upserting using the gRPC SDK. If you see this often, we recommend you use a backoff algorithm(e.g., [exponential backoffs](https://www.pinecone.io/blog/working-at-scale/))\ while upserting. The syntax for upsert, query, fetch, and delete with the gRPC SDK remain the same as the standard SDK. # Back up a pod-based index Source: https://docs.pinecone.io/guides/indexes/pods/back-up-a-pod-based-index This page describes how to create a static copy of a pod-based index, also known as a [collection](/guides/indexes/pods/understanding-collections). ## Create a collection To create a backup of your pod-based index, use the [`create_collection`](/reference/api/2024-10/control-plane/create_collection) operation. The following example creates a [collection](/guides/indexes/pods/understanding-collections) named `example-collection` from an index named `docs-example`: ```python Python # pip install "pinecone[grpc]" from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key="API_KEY") pc.create_collection("example-collection", "docs-example") ``` ```javascript JavaScript // npm install @pinecone-database/pinecone import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }); await pc.createCollection({ name: "example-collection", source: "docs-example", }); ``` ```java Java import io.pinecone.clients.Pinecone; public class CreateCollectionExample { public static void main(String[] args) { Pinecone pc = new Pinecone.Builder("YOUR_API_KEY").build(); pc.createCollection("example-collection", "docs-example"); } } ``` ```go Go package main import ( "context" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } collection, err := pc.CreateCollection(ctx, &pinecone.CreateCollectionRequest{ Name: "example-collection", Source: "docs-example", }) if err != nil { log.Fatalf("Failed to create collection: %v", err) } else { fmt.Printf("Successfully created collection: %v", collection.Name) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); var collectionModel = await pinecone.CreateCollectionAsync(new CreateCollectionRequest { Name = "example-collection", Source = "docs-example", }); ``` ```shell curl PINECONE_API_KEY="YOUR_API_KEY" curl -s POST "https://api.pinecone.io/collections" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "name": "example-collection", "source": "docs-example" }' ``` You can create a collection using the [Pinecone console](https://app.pinecone.io/organizations/-/projects/-/backups). ## Check the status of a collection To retrieve the status of the process creating a collection and the size of the collection, use the [`describe_collection`](/reference/api/2024-10/control-plane/describe_collection) operation. Specify the name of the collection to check. You can only call `describe_collection` on a collection in the current project. The `describe_collection` operation returns an object containing key-value pairs representing the name of the collection, the size in bytes, and the creation status of the collection. The following example gets the creation status and size of a collection named `example-collection`. ```python Python # pip install "pinecone[grpc]" from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key='API_KEY') pc.describe_collection(name="example-collection") ``` ```javascript JavaScript // npm install @pinecone-database/pinecone import { Pinecone } from '@pinecone-database/pinecone'; const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }); await pc.describeCollection('example-collection'); ``` ```java Java import io.pinecone.clients.Pinecone; import org.openapitools.client.model.CollectionModel; public class DescribeCollectionExample { public static void main(String[] args) { Pinecone pc = new Pinecone.Builder("YOUR_API_KEY").build(); CollectionModel collectionModel = pc.describeCollection("example-collection"); System.out.println(collectionModel); } } ``` ```go Go package main import ( "context" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } collectionName := "example-collection" collection, err := pc.DescribeCollection(ctx, collectionName) if err != nil { log.Fatalf("Error describing collection %v: %v", collectionName, err) } else { fmt.Printf("Collection: %+v", collection) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); var collectionModel = await pinecone.DescribeCollectionAsync("example-collection"); Console.WriteLine(collectionModel); ``` ```shell curl PINECONE_API_KEY="YOUR_API_KEY" curl -i -X GET "https://api.pinecone.io/collections/example-collection" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" ``` You can check the status of a collection using the [Pinecone console](https://app.pinecone.io/organizations/-/projects/-/backups). ## List your collections To get a list of the collections in the current project, use the [`list_collections`](/reference/api/2024-10/control-plane/list_collections) operation. ```python Python from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key='API_KEY') pc.list_collections() ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }) await pc.listCollections(); ``` ```java Java import io.pinecone.clients.Pinecone; import org.openapitools.client.model.CollectionModel; public class ListCollectionsExample { public static void main(String[] args) { Pinecone pc = new Pinecone.Builder("YOUR_API_KEY").build(); List collectionList = pc.listCollections().getCollections(); } } ``` ```go Go package main import ( "context" "encoding/json" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func prettifyStruct(obj interface{}) string { bytes, _ := json.MarshalIndent(obj, "", " ") return string(bytes) } func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } collections, err := pc.ListCollections(ctx) if err != nil { log.Fatalf("Failed to list collections: %v", err) } else { if len(collections) == 0 { fmt.Printf("No collections found in project") } else { for _, collection := range collections { fmt.Printf("collection: %v\n", prettifyStruct(collection)) } } } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); var collectionList = await pinecone.ListCollectionsAsync(); Console.WriteLine(collectionList); ``` ```shell curl PINECONE_API_KEY="YOUR_API_KEY" curl -i -X GET "https://api.pinecone.io/collections" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" ``` You can view a list of your collections using the [Pinecone console](https://app.pinecone.io/organizations/-/projects/-/backups). You can view a list of your collections using the [Pinecone console](https://app.pinecone.io/organizations/-/projects/-/backups). ## Delete a collection To delete a collection, use the [`delete_collection`](/reference/api/2024-10/control-plane/delete_collection) operation. Specify the name of the collection to delete. Deleting the collection takes several minutes. During this time, the [`describe_collection`](#check-the-status-of-a-collection) operation returns the status "deleting". ```python Python # pip install "pinecone[grpc]" from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key='API_KEY') pc.delete_collection("example-collection") ``` ```javascript JavaScript // npm install @pinecone-database/pinecone import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }) await pc.deleteCollection("example-collection"); ``` ```java Java import io.pinecone.clients.Pinecone; public class DeleteCollectionExample { public static void main(String[] args) { Pinecone pc = new Pinecone.Builder("YOUR_API_KEY").build(); pc.deleteCollection("example-collection"); } } ``` ```go Go package main import ( "context" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } collectionName := "example-collection" err = pc.DeleteCollection(ctx, collectionName) if err != nil { log.Fatalf("Failed to delete collection: %v\n", err) } else { if len(collections) == 0 { fmt.Printf("No collections found in project") } else { fmt.Printf("Successfully deleted collection \"%v\"\n", collectionName) } } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); await pinecone.DeleteCollectionAsync("example-collection"); ``` ```shell curl PINECONE_API_KEY="YOUR_API_KEY" curl -i -X DELETE "https://api.pinecone.io/collections/example-collection" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" ``` You can delete a collection using the [Pinecone console](https://app.pinecone.io/organizations/-/projects/-/backups). # Choose a pod type and size Source: https://docs.pinecone.io/guides/indexes/pods/choose-a-pod-type-and-size This guidance applies to [pod-based indexes](/guides/index-data/indexing-overview#pod-based-indexes) only. With [serverless indexes](/guides/index-data/indexing-overview#serverless-indexes), you don't configure any compute or storage resources, and you don't manually manage those resources to meet demand, save on cost, or ensure high availability. Instead, serverless indexes scale automatically based on usage. When planning your Pinecone deployment, it is important to understand the approximate storage requirements of your vectors to choose the appropriate pod type and number. This page will give guidance on sizing to help you plan accordingly. As with all guidelines, these considerations are general and may not apply to your specific use case. We caution you to always test your deployment and ensure that the index configuration you are using is appropriate to your requirements. [Collections](/guides/indexes/pods/understanding-collections) allow you to create new versions of your index with different pod types and sizes. This also allows you to test different configurations. This guide is merely an overview of sizing considerations; test your index configuration before moving to production. Users on the Standard, Enterprise, and Enterprise Dedicated plans can [contact Support](https://app.pinecone.io/organizations/-/settings/support/ticket) for further help with sizing and testing. ## Overview There are five main considerations when deciding how to configure your Pinecone index: * Number of vectors * Dimensionality of your vectors * Size of metadata on each vector * Queries per second (QPS) throughput * Cardinality of indexed metadata Each of these considerations comes with requirements for index size, pod type, and replication strategy. ### Number of vectors The most important consideration in sizing is the [number of vectors](/guides/index-data/upsert-data) you plan on working with. As a rule of thumb, a single p1 pod can store approximately 1M vectors, while a s1 pod can store 5M vectors. However, this can be affected by other factors, such as dimensionality and metadata, which are explained below. ### Dimensionality of vectors The rules of thumb above for how many vectors can be stored in a given pod assumes a typical configuration of 768 [dimensions per vector](/guides/index-data/create-an-index). As your individual use case will dictate the dimensionality of your vectors, the amount of space required to store them may necessarily be larger or smaller. Each dimension on a single vector consumes 4 bytes of memory and storage per dimension, so if you expect to have 1M vectors with 768 dimensions each, that’s about 3GB of storage without factoring in metadata or other overhead. Using that reference, we can estimate the typical pod size and number needed for a given index. Table 1 below gives some examples of this. **Table 1: Estimated number of pods per 1M vectors by dimensionality** | Pod type | Dimensions | Estimated max vectors per pod | | -------- | ---------: | ----------------------------: | | **p1** | 512 | 1,250,000 | | | 768 | 1,000,000 | | | 1024 | 675,000 | | | 1536 | 500,000 | | **p2** | 512 | 1,250,000 | | | 768 | 1,100,000 | | | 1024 | 1,000,000 | | | 1536 | 550,000 | | **s1** | 512 | 8,000,000 | | | 768 | 5,000,000 | | | 1024 | 4,000,000 | | | 1536 | 2,500,000 | Pinecone does not support fractional pod deployments, so always round up to the next nearest whole number when choosing your pods. ## Queries per second (QPS) QPS speeds are governed by a combination of the [pod type](/guides/indexes/pods/understanding-pod-based-indexes#pod-types) of the index, the number of [replicas](/guides/indexes/pods/scale-pod-based-indexes#add-replicas), and the `top_k` value of queries. The pod type is the primary factor driving QPS, as the different pod types are optimized for different approaches. The [p1 pods](/guides/index-data/indexing-overview/#p1-pods) are performance-optimized pods which provide very low query latencies, but hold fewer vectors per pod than [s1 pods](/guides/index-data/indexing-overview/#s1-pods). They are ideal for applications with low latency requirements (\<100ms). The s1 pods are optimized for storage and provide large storage capacity and lower overall costs with slightly higher query latencies than p1 pods. They are ideal for very large indexes with moderate or relaxed latency requirements. The [p2 pod type](/guides/index-data/indexing-overview/#p2-pods) provides greater query throughput with lower latency. They support 200 QPS per replica and return queries in less than 10ms. This means that query throughput and latency are better than s1 and p1, especially for low dimension vectors (\<512D). As a rule, a single p1 pod with 1M vectors of 768 dimensions each and no replicas can handle about 20 QPS. It’s possible to get greater or lesser speeds, depending on the size of your metadata, number of vectors, the dimensionality of your vectors, and the `top_K` value for your search. See Table 2 below for more examples. **Table 2: QPS by pod type and `top_k` value**\* | Pod type | top\_k 10 | top\_k 250 | top\_k 1000 | | -------- | --------- | ---------- | ----------- | | p1 | 30 | 25 | 20 | | p2 | 150 | 50 | 20 | | s1 | 10 | 10 | 10 | \*The QPS values in Table 2 represent baseline QPS with 1M vectors and 768 dimensions. [Adding replicas](/guides/indexes/pods/scale-pod-based-indexes#add-replicas) is the simplest way to increase your QPS. Each replica increases the throughput potential by roughly the same QPS, so aiming for 150 QPS using p1 pods means using the primary pod and 5 replicas. Using threading or multiprocessing in your application is also important, as issuing single queries sequentially still subjects you to delays from any underlying latency. The [Pinecone gRPC SDK](/guides/index-data/upsert-data#grpc-python-sdk) can also be used to increase throughput of upserts. ### Metadata cardinality and size The last consideration when planning your indexes is the cardinality and size of your [metadata](/guides/index-data/upsert-data#inserting-vectors-with-metadata). While the increases are small when talking about a few million vectors, they can have a real impact as you grow to hundreds of millions or billions of vectors. Indexes with very high cardinality, like those storing a unique user ID on each vector, can have significant memory requirements, resulting in fewer vectors fitting per pod. Also, if the size of the metadata per vector is larger, the index requires more storage. Limiting which metadata fields are indexed using [selective metadata indexing](/guides/indexes/pods/manage-pod-based-indexes#selective-metadata-indexing) can help lower memory usage. ### Pod sizes You can also start with one of the larger [pod sizes](/guides/index-data/indexing-overview/#pod-size-and-performance), like p1.x2. Each step up in pod size doubles the space available for your vectors. We recommend starting with x1 pods and scaling as you grow. This way, you don’t start with too large a pod size and have nowhere else to go up, meaning you have to migrate to a new index before you’re ready. ### Example applications The following examples will showcase how to use the sizing guidelines above to choose the appropriate type, size, and number of pods for your index. #### Example 1: Semantic search of news articles In our first example, we’ll use the demo app for semantic search from our documentation. In this case, we’re only working with 204,135 vectors. The vectors use 300 dimensions each, well under the general measure of 768 dimensions. Using the rule of thumb above of up to 1M vectors per p1 pod, we can run this app comfortably with a single p1.x1 pod. #### Example 2: Facial recognition For this example, suppose you’re building an application to identify customers using facial recognition for a secure banking app. Facial recognition can work with as few as 128 dimensions, but in this case, because the app will be used for access to finances, we want to make sure we’re certain that the person using it is the right one. We plan for 100M customers and use 2048 dimensions per vector. We know from our rules of thumb above that 1M vectors with 768 dimensions fit nicely in a p1.x1 pod. We can just divide those numbers into the new targets to get the ratios we’ll need for our pod estimate: ``` 100M / 1M = 100 base p1 pods 2048 / 768 = 2.667 vector ratio 2.667 * 100 = 267 rounding up ``` So we need 267 p1.x1 pods. We can reduce that by switching to s1 pods instead, sacrificing latency by increasing storage availability. They hold five times the storage of p1.x1, so the math is simple: ``` 267 / 5 = 54 rounding up ``` So we estimate that we need 54 s1.x1 pods to store very high dimensional data for the face of each of the bank’s customers. # Create a pod-based index Source: https://docs.pinecone.io/guides/indexes/pods/create-a-pod-based-index This page shows you how to create a pod-based index. For guidance on serverless indexes, see [Create a serverless index](/guides/index-data/create-an-index). You can create an index using the [Pinecone console](https://app.pinecone.io/organizations/-/projects/-/create-index/serverless). ## Create a pod index To create a pod index, use the [`create_index`](/reference/api/2024-10/control-plane/create_index) operation as follows: * Provide a `name` for the index. * Specify the `dimension` and `metric` of the vectors you'll store in the index. This should match the dimension and metric supported by your embedding model. * Set `spec.environment` to the [environment](/guides/index-data/create-an-index#cloud-regions) where the index should be deployed. For Python, you also need to import the `ServerlessSpec` class. * Set `spec.pod_type` to the [pod type](/guides/indexes/pods/understanding-pod-based-indexes#pod-types) and [size](/guides/index-data/indexing-overview#pod-size-and-performance) that you want. Other parameters are optional. See the [API reference](/reference/api/2024-10/control-plane/create_index) for details. ```Python Python from pinecone.grpc import PineconeGRPC as Pinecone, PodSpec pc = Pinecone(api_key="YOUR_API_KEY") pc.create_index( name="docs-example", dimension=1536, metric="cosine", spec=PodSpec( environment="us-west1-gcp", pod_type="p1.x1", pods=1 ), deletion_protection="disabled" ) ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }); await pc.createIndex({ name: 'docs-example', dimension: 1536, metric: 'cosine', spec: { pod: { environment: 'us-west1-gcp', podType: 'p1.x1', pods: 1 } }, deletionProtection: 'disabled', }); ``` ```java Java import io.pinecone.clients.Pinecone; import org.openapitools.db_control.client.model.IndexModel; import org.openapitools.db_control.client.model.DeletionProtection; public class CreateIndexExample { public static void main(String[] args) { Pinecone pc = new Pinecone.Builder("YOUR_API_KEY").build(); pc.createPodsIndex("docs-example", 1536, "us-west1-gcp", "p1.x1", "cosine", DeletionProtection.DISABLED); } } ``` ```go Go package main import ( "context" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } indexName := "docs-example" metric := pinecone.Dotproduct deletionProtection := pinecone.DeletionProtectionDisabled idx, err := pc.CreatePodIndex(ctx, &pinecone.CreatePodIndexRequest{ Name: indexName, Metric: &metric, Dimension: 1536, Environment: "us-east1-gcp", PodType: "p1.x1", DeletionProtection: &deletionProtection, }) if err != nil { log.Fatalf("Failed to create pod-based index: %v", idx.Name) } else { fmt.Printf("Successfully created pod-based index: %v", idx.Name) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); var createIndexRequest = await pinecone.CreateIndexAsync(new CreateIndexRequest { Name = "docs-example", Dimension = 1536, Metric = MetricType.Cosine, Spec = new PodIndexSpec { Pod = new PodSpec { Environment = "us-east1-gcp", PodType = "p1.x1", Pods = 1, } }, DeletionProtection = DeletionProtection.Disabled }); ``` ```bash curl PINECONE_API_KEY="YOUR_API_KEY" curl -s "https://api.pinecone.io/indexes" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "name": "docs-example", "dimension": 1536, "metric": "cosine", "spec": { "pod": { "environment": "us-west1-gcp", "pod_type": "p1.x1", "pods": 1 } }, "deletion_protection": "disabled" }' ``` ## Create a pod index from a collection You can create a pod-based index from a collection. For more details, see [Restore an index](/guides/indexes/pods/restore-a-pod-based-index). # Manage pod-based indexes Source: https://docs.pinecone.io/guides/indexes/pods/manage-pod-based-indexes This page shows you how to manage pod-based indexes. For guidance on serverless indexes, see [Manage serverless indexes](/guides/manage-data/manage-indexes). ## Describe a pod-based index Use the [`describe_index`](/reference/api/2024-10/control-plane/describe_index/) endpoint to get a complete description of a specific index: ```Python Python from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key="YOUR_API_KEY") pc.describe_index(name="docs-example") # Response: # {'dimension': 1536, # 'host': 'docs-example-4mkljsz.svc.aped-4627-b74a.pinecone.io', # 'metric': 'cosine', # 'name': 'docs-example', # 'spec': {'pod': {'environment': 'us-east-1-aws', # 'pod_type': 's1.x1', # 'pods': 1, # 'replicas': 1, # 'shards': 1}}, # 'status': {'ready': True, 'state': 'Ready'}} ``` ```JavaScript JavaScript import { Pinecone } from '@pinecone-database/pinecone'; const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }); await pc.describeIndex('docs-example'); // Response: // { // "name": "docs-example", // "dimension": 1536, // "metric": "cosine", // "host": "docs-example-4mkljsz.svc.aped-4627-b74a.pinecone.io", // "deletionProtection": "disabled", // "spec": { // "serverless": { // "environment": "us-east-1-aws", // "pod_type": "s1.x1", // "pods": 1, // "replicas": 1, // "shards": 1 // } // }, // "status": { // "ready": true, // "state": "Ready" // } // } ``` ```java Java import io.pinecone.clients.Pinecone; import org.openapitools.db_control.client.model.*; public class DescribeIndexExample { public static void main(String[] args) { Pinecone pc = new Pinecone.Builder("YOURE_API_KEY").build(); IndexModel indexModel = pc.describeIndex("docs-example"); System.out.println(indexModel); } } // Response: // class IndexModel { // name: docs-example // dimension: 1536 // metric: cosine // host: docs-example-4mkljsz.svc.aped-4627-b74a.pinecone.io // deletionProtection: disabled // spec: class IndexModelSpec { // serverless: null // pod: class PodSpec { // cloud: aws // region: us-east-1 // environment: us-east-1-aws, // podType: s1.x1, // pods: 1, // replicas: 1, // shards: 1 // } // } // status: class IndexModelStatus { // ready: true // state: Ready // } // } ``` ```go Go package main import ( "context" "encoding/json" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func prettifyStruct(obj interface{}) string { bytes, _ := json.MarshalIndent(obj, "", " ") return string(bytes) } func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } idx, err := pc.DescribeIndex(ctx, "docs-example") if err != nil { log.Fatalf("Failed to describe index \"%v\": %v", idx.Name, err) } else { fmt.Printf("index: %v\n", prettifyStruct(idx)) } } // Response: // index: { // "name": "docs-example", // "dimension": 1536, // "host": "docs-example-4mkljsz.svc.aped-4627-b74a.pinecone.io", // "metric": "cosine", // "deletion_protection": "disabled", // "spec": { // "pod": { // "environment": "us-east-1-aws", // "pod_type": "s1.x1", // "pods": 1, // "replicas": 1, // "shards": 1 // } // }, // "status": { // "ready": true, // "state": "Ready" // } // } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); var indexModel = await pinecone.DescribeIndexAsync("docs-example"); Console.WriteLine(indexModel); // Response: // { // "name": "docs-example", // "dimension": 1536, // "metric": "cosine", // "host": "docs-example-4mkljsz.svc.aped-4627-b74a.pinecone.io", // "deletion_protection": "disabled", // "spec": { // "serverless": null, // "pod": { // "environment": "us-east-1-aws", // "pod_type": "s1.x1", // "pods": 1, // "replicas": 1, // "shards": 1 // } // }, // "status": { // "ready": true, // "state": "Ready" // } // } ``` ```bash curl PINECONE_API_KEY="YOUR_API_KEY" curl -i -X GET "https://api.pinecone.io/indexes/docs-example" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" # Response: # { # "name": "docs-example", # "metric": "cosine", # "dimension": 1536, # "status": { # "ready": true, # "state": "Ready" # }, # "host": "docs-example-4mkljsz.svc.aped-4627-b74a.pinecone.io", # "spec": { # "pod": { # "environment": "us-east-1-aws", # "pod_type": "s1.x1", # "pods": 1, # "replicas": 1, # "shards": 1 # } # } # } ``` ## Delete a pod-based index Use the [`delete_index`](reference/api/2024-10/control-plane/delete_index) operation to delete a pod-based index and all of its associated resources. You are billed for a pod-based index even when it is not in use. ```python Python # pip install "pinecone[grpc]" from pinecone.grpc import PineconeGRPC as Pinecone, PodSpec pc = Pinecone(api_key="YOUR_API_KEY") pc.delete_index(name="docs-example") ``` ```javascript JavaScript // npm install @pinecone-database/pinecone import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }); await pc.deleteIndex('docs-example'); ``` ```java Java import io.pinecone.clients.Pinecone; public class DeleteIndexExample { public static void main(String[] args) { Pinecone pc = new Pinecone.Builder("YOUR_API_KEY").build(); pc.deleteIndex("docs-example"); } } ``` ```go Go package main import ( "context" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } indexName := "docs-example" err = pc.DeleteIndex(ctx, indexName) if err != nil { log.Fatalf("Failed to delete index: %v", err) } else { fmt.Println("Index \"%v\" deleted successfully", indexName) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); await pinecone.DeleteIndexAsync("docs-example"); ``` ```shell curl PINECONE_API_KEY="YOUR_API_KEY" curl -i -X DELETE "https://api.pinecone.io/indexes/docs-example" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" ``` If deletion protection is enabled on an index, requests to delete it will fail and return a `403 - FORBIDDEN` status with the following error: ``` Deletion protection is enabled for this index. Disable deletion protection before retrying. ``` Before you can delete such an index, you must first [disable deletion protection](/guides/manage-data/manage-indexes#configure-deletion-protection). You can delete an index using the [Pinecone console](https://app.pinecone.io/organizations/-/projects/-/indexes). For the index you want to delete, click the three dots to the right of the index name, then click **Delete**. ## Selective metadata indexing For pod-based indexes, Pinecone indexes all metadata fields by default. When metadata fields contains many unique values, pod-based indexes will consume significantly more memory, which can lead to performance issues, pod fullness, and a reduction in the number of possible vectors that fit per pod. To avoid indexing high-cardinality metadata that is not needed for [filtering your queries](/guides/index-data/indexing-overview#metadata) and keep memory utilization low, specify which metadata fields to index using the `metadata_config` parameter. Since high-cardinality metadata does not cause high memory utilization in serverless indexes, selective metadata indexing is not supported. The value for the `metadata_config` parameter is a JSON object containing the names of the metadata fields to index. ```JSON JSON { "indexed": [ "metadata-field-1", "metadata-field-2", "metadata-field-n" ] } ``` **Example** The following example creates a pod-based index that only indexes the `genre` metadata field. Queries against this index that filter for the `genre` metadata field may return results; queries that filter for other metadata fields behave as though those fields do not exist. ```Python Python from pinecone.grpc import PineconeGRPC as Pinecone, PodSpec pc = Pinecone(api_key="YOUR_API_KEY") pc.create_index( name="docs-example", dimension=1536, metric="cosine", spec=PodSpec( environment="us-west1-gcp", pod_type="p1.x1", pods=1, metadata_config = { "indexed": ["genre"] } ), deletion_protection="disabled" ) ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }); await pc.createIndex({ name: 'docs-example', dimension: 1536, metric: 'cosine', spec: { pod: { environment: 'us-west1-gcp', podType: 'p1.x1', pods: 1, metadata_config: { indexed: ["genre"] } } }, deletionProtection: 'disabled', }); ``` ```java Java import io.pinecone.clients.Pinecone; import org.openapitools.db_control.client.model.IndexModel; import org.openapitools.db_control.client.model.DeletionProtection; public class CreateIndexExample { public static void main(String[] args) { Pinecone pc = new Pinecone.Builder("YOUR_API_KEY").build(); CreateIndexRequestSpecPodMetadataConfig podSpecMetadataConfig = new CreateIndexRequestSpecPodMetadataConfig(); List indexedItems = Arrays.asList("genre", "year"); podSpecMetadataConfig.setIndexed(indexedItems); pc.createPodsIndex("docs-example", 1536, "us-west1-gcp", "p1.x1", "cosine", podSpecMetadataConfig, DeletionProtection.DISABLED); } } ``` ```go Go package main import ( "context" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } podIndexMetadata := &pinecone.PodSpecMetadataConfig{ Indexed: &[]string{"genre"}, } indexName := "docs-example" metric := pinecone.Dotproduct deletionProtection := pinecone.DeletionProtectionDisabled idx, err := pc.CreatePodIndex(ctx, &pinecone.CreatePodIndexRequest{ Name: indexName, Metric: &metric, Dimension: 1536, Environment: "us-east1-gcp", PodType: "p1.x1", DeletionProtection: &deletionProtection, }) if err != nil { log.Fatalf("Failed to create pod-based index: %v", idx.Name) } else { fmt.Printf("Successfully created pod-based index: %v", idx.Name) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); var createIndexRequest = await pinecone.CreateIndexAsync(new CreateIndexRequest { Name = "docs-example", Dimension = 1536, Metric = MetricType.Cosine, Spec = new PodIndexSpec { Pod = new PodSpec { Environment = "us-east1-gcp", PodType = "p1.x1", Pods = 1, MetadataConfig = new PodSpecMetadataConfig { Indexed = new List { "genre" }, }, } }, DeletionProtection = DeletionProtection.Disabled }); ``` ```bash curl PINECONE_API_KEY="YOUR_API_KEY" curl -s https://api.pinecone.io/indexes \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "name": "docs-example", "dimension": 1536, "metric": "cosine", "spec": { "pod": { "environment": "us-west1-gcp", "pod_type": "p1.x1", "pods": 1, "metadata_config": { "indexed": ["genre"] } } }, "deletion_protection": "disabled" }' ``` ## Prevent index deletion This feature requires [Pinecone API version](/reference/api/versioning) `2024-07`, [Python SDK](/reference/python-sdk) v5.0.0, [Node.js SDK](/reference/node-sdk) v3.0.0, [Java SDK](/reference/java-sdk) v2.0.0, or [Go SDK](/reference/go-sdk) v1.0.0 or later. You can prevent an index and its data from accidental deleting when [creating a new index](/guides/index-data/create-an-index) or when [configuring an existing index](/guides/indexes/pods/manage-pod-based-indexes). In both cases, you set the `deletion_protection` parameter to `enabled`. To enable deletion protection when creating a new index: ```python Python # pip install "pinecone[grpc]" from pinecone.grpc import PineconeGRPC as Pinecone, PodSpec pc = Pinecone(api_key="YOUR_API_KEY") pc.create_index( name="docs-example", dimension=1536, metric="cosine", spec=PodSpec( environment="us-west1-gcp", pod_type="p1.x1", pods=1 ), deletion_protection="enabled" ) ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }); await pc.createIndex({ name: 'docs-example', dimension: 1536, metric: 'cosine', spec: { pod: { environment: 'us-west1-gcp', podType: 'p1.x1', pods: 1 } }, deletionProtection: 'enabled', }); ``` ```java Java import io.pinecone.clients.Pinecone; import org.openapitools.db_control.client.model.IndexModel; import org.openapitools.db_control.client.model.DeletionProtection; public class CreateIndexExample { public static void main(String[] args) { Pinecone pc = new Pinecone.Builder("YOUR_API_KEY").build(); pc.createPodsIndex("docs-example", 1536, "us-west1-gcp", "p1.x1", "cosine", DeletionProtection.ENABLED); } } ``` ```go Go package main import ( "context" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } indexName := "docs-example" metric := pinecone.Dotproduct deletionProtection := pinecone.DeletionProtectionDisabled idx, err := pc.CreatePodIndex(ctx, &pinecone.CreatePodIndexRequest{ Name: indexName, Metric: &metric, Dimension: 1536, Environment: "us-east1-gcp", PodType: "p1.x1", DeletionProtection: &deletionProtection, }) if err != nil { log.Fatalf("Failed to create pod-based index: %v", idx.Name) } else { fmt.Printf("Successfully created pod-based index: %v", idx.Name) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); var createIndexRequest = await pinecone.CreateIndexAsync(new CreateIndexRequest { Name = "docs-example", Dimension = 1536, Metric = MetricType.Cosine, Spec = new PodIndexSpec { Pod = new PodSpec { Environment = "us-east1-gcp", PodType = "p1.x1", Pods = 1, } }, DeletionProtection = DeletionProtection.Enabled }); ``` ```bash curl PINECONE_API_KEY="YOUR_API_KEY" curl -s "https://api.pinecone.io/indexes" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "name": "docs-example", "dimension": 1536, "metric": "cosine", "spec": { "pod": { "environment": "us-west1-gcp", "pod_type": "p1.x1", "pods": 1 } }, "deletion_protection": "enabled ``` To enable deletion protection when configuring an existing index: ```python Python from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key="YOUR_API_KEY") pc.configure_index( name="docs-example", deletion_protection="enabled" ) ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone'; const client = new Pinecone({ apiKey: 'YOUR_API_KEY' }); await client.configureIndex('docs-example', { deletionProtection: 'enabled' }); ``` ```java Java import io.pinecone.clients.Pinecone; import org.openapitools.db_control.client.model.*; public class ConfigureIndexExample { public static void main(String[] args) { Pinecone pc = new Pinecone.Builder("YOUR_API_KEY").build(); pc.configurePodsIndex("docs-example", DeletionProtection.ENABLED); } } ``` ```go Go package main import ( "context" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } idx, err := pc.ConfigureIndex(ctx, "docs-example", pinecone.ConfigureIndexParams{DeletionProtection: "enabled"}) if err != nil { log.Fatalf("Failed to configure index \"%v\": %v", idx.Name, err) } else { fmt.Printf("Successfully configured index \"%v\"", idx.Name) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); var indexMetadata = await pinecone.ConfigureIndexAsync("docs-example", new ConfigureIndexRequest { DeletionProtection = DeletionProtection.Enabled, }); ``` ```shell curl PINECONE_API_KEY="YOUR_API_KEY" curl -s -X PATCH "https://api.pinecone.io/indexes/docs-example" \ -H "Content-Type: application/json" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "deletion_protection": "enabled" }' ``` When deletion protection is enabled on an index, requests to delete the index fail and return a `403 - FORBIDDEN` status with the following error: ``` Deletion protection is enabled for this index. Disable deletion protection before retrying. ``` ## Disable deletion protection Before you can [delete an index](#delete-a-pod-based-index) with deletion protection enabled, you must first disable deletion protection as follows: ```python Python from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key="YOUR_API_KEY") pc.configure_index( name="docs-example", deletion_protection="disabled" ) ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone'; const client = new Pinecone({ apiKey: 'YOUR_API_KEY' }); await client.configureIndex('docs-example', { deletionProtection: 'disabled' }); ``` ```java Java import io.pinecone.clients.Pinecone; import org.openapitools.db_control.client.model.*; public class ConfigureIndexExample { public static void main(String[] args) { Pinecone pc = new Pinecone.Builder("YOUR_API_KEY").build(); pc.configurePodsIndex("docs-example", DeletionProtection.DISABLED); } } ``` ```go Go package main import ( "context" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } idx, err := pc.ConfigureIndex(ctx, "docs-example", pinecone.ConfigureIndexParams{DeletionProtection: "disabled"}) if err != nil { log.Fatalf("Failed to configure index \"%v\": %v", idx.Name, err) } else { fmt.Printf("Successfully configured index \"%v\"", idx.Name) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); var configureIndexRequest = await pinecone.ConfigureIndexAsync("docs-example", new ConfigureIndexRequest { DeletionProtection = DeletionProtection.Disabled, }); ``` ```shell curl PINECONE_API_KEY="YOUR_API_KEY" curl -s -X PATCH "https://api.pinecone.io/indexes/docs-example" \ -H "Content-Type: application/json" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "deletion_protection": "disabled" }' ``` ## Delete an entire namespace In pod-based indexes, reads and writes share compute resources, so deleting an entire namespace with many records can increase the latency of read operations. In such cases, consider [deleting records in batches](#delete-records-in-batches). ## Delete records in batches In pod-based indexes, reads and writes share compute resources, so deleting an entire namespace or a large number of records can increase the latency of read operations. To avoid this, delete records in batches of up to 1000, with a brief sleep between requests. Consider using smaller batches if the index has active read traffic. ```python Batch delete a namespace from pinecone import Pinecone import numpy as np import time pc = Pinecone(api_key='API_KEY') INDEX_NAME = 'INDEX_NAME' NAMESPACE = 'NAMESPACE_NAME' # Consider using smaller batches if you have a high RPS for read operations BATCH = 1000 index = pc.Index(name=INDEX_NAME) dimensions = index.describe_index_stats()['dimension'] # Create the query vector query_vector = np.random.uniform(-1, 1, size=dimensions).tolist() results = index.query(vector=query_vector, namespace=NAMESPACE, top_k=BATCH) # Delete in batches until the query returns no results while len(results['matches']) > 0: ids = [i['id'] for i in results['matches']] index.delete(ids=ids, namespace=NAMESPACE) time.sleep(0.01) results = index.query(vector=query_vector, namespace=NAMESPACE, top_k=BATCH) ``` ```python Batch delete by metadata from pinecone import Pinecone import numpy as np import time pc = Pinecone(api_key='API_KEY') INDEX_NAME = 'INDEX_NAME' NAMESPACE = 'NAMESPACE_NAME' # Consider using smaller batches if you have a high RPS for read operations BATCH = 1000 index = pc.Index(name=INDEX_NAME) dimensions = index.describe_index_stats()['dimension'] METADATA_FILTER = {} # Create the query vector with a filter query_vector = np.random.uniform(-1, 1, size=dimensions).tolist() results = index.query(vector=query_vector, namespace=NAMESPACE, filter=METADATA_FILTER, top_k=BATCH) # Delete in batches until the query returns no results while len(results['matches']) > 0: ids = [i['id'] for i in results['matches']] index.delete(ids=ids, namespace=NAMESPACE) time.sleep(0.01) results = index.query(vector=query_vector, namespace=NAMESPACE, filter=METADATA_FILTER, top_k=BATCH) ``` ## Delete records by metadata In pod-based indexes, if you are targeting a large number of records for deletion and the index has active read traffic, consider [deleting records in batches](#delete-records-in-batches). To delete records based on their metadata, pass a metadata filter expression to the `delete` operation. This deletes all vectors matching the metadata filter expression. For example, to delete all vectors with genre "documentary" and year 2019 from an index, use the following code: ```Python Python from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key="YOUR_API_KEY") # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index index = pc.Index(host="INDEX_HOST") index.delete( filter={ "genre": {"$eq": "documentary"} } ) ``` ```JavaScript JavaScript import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: "YOUR_API_KEY" }) // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index const index = pc.index("INDEX_NAME", "INDEX_HOST") await index.deleteMany({ genre: { $eq: "documentary" }, }); ``` ```java Java import com.google.protobuf.Struct; import com.google.protobuf.Value; import io.pinecone.clients.Index; import io.pinecone.configs.PineconeConfig; import io.pinecone.configs.PineconeConnection; import java.util.Arrays; import java.util.List; public class DeleteByMetadataExample { public static void main(String[] args) { PineconeConfig config = new PineconeConfig("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index config.setHost("INDEX_HOST"); PineconeConnection connection = new PineconeConnection(config); Index index = new Index(connection, "INDEX_NAME"); Struct filter = Struct.newBuilder() .putFields("genre", Value.newBuilder() .setStructValue(Struct.newBuilder() .putFields("$eq", Value.newBuilder() .setStringValue("documentary") .build())) .build()) .build(); index.deleteByFilter(filter, "example-namespace"); } } ``` ```go Go package main import ( "context" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" "google.golang.org/protobuf/types/known/structpb" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "INDEX_HOST", Namespace: "example-namespace"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v", err) } metadataFilter := map[string]interface{}{ "genre": "$eq": "documentary" } filter, err := structpb.NewStruct(metadataFilter) if err != nil { log.Fatalf("Failed to create metadata filter: %v", err) } err = idxConnection.DeleteVectorsByFilter(ctx, filter) if err != nil { log.Fatalf("Failed to delete vector(s) with filter %+v: %v", filter, err) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index var index = pinecone.Index(host: "INDEX_HOST"); var deleteResponse = await index.DeleteAsync(new DeleteRequest { Namespace = "example-namespace", Filter = new Metadata { ["genre"] = new Metadata { ["$eq"] = "documentary" } } }); ``` ```bash curl # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index PINECONE_API_KEY="YOUR_API_KEY" INDEX_HOST="INDEX_HOST" curl -i "https://$INDEX_HOST/vectors/delete" \ -H 'Api-Key: $PINECONE_API_KEY' \ -H 'Content-Type: application/json' \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "filter": {"genre": {"$eq": {"documentary"}} }' ``` ## Tag an index When configuring an index, you can tag the index to help with index organization and management. For more details, see [Tag an index](/guides/manage-data/manage-indexes#configure-index-tags). ## Troubleshoot index fullness errors Serverless indexes automatically scale as needed. However, pod-based indexes can run out of capacity. When that happens, upserting new records will fail with the following error: ```console console Index is full, cannot accept data. ``` While a full pod-based index can still serve queries, you need to [scale your index](/guides/indexes/pods/scale-pod-based-indexes) to accommodate more records. # Migrate a pod-based index to serverless Source: https://docs.pinecone.io/guides/indexes/pods/migrate-a-pod-based-index-to-serverless This page shows you how to use the Pinecone console to migrate a pod-based index to [serverless](/reference/architecture/serverless-architecture). The migration process is free; the standard costs of upserting records to a new serverless index are not applied. In most cases, migrating to serverless reduces costs significantly. However, costs can increase for read-heavy workloads with more than 1 query per second and for indexes with many records in a single namespace. Before migrating, consider [contacting Pinecone Support](/troubleshooting/contact-support) for help estimating and managing cost implications. ## Limitations Migration is supported for pod-based indexes on GCP and AWS with less than 25 million records and 20,000 namespaces. This is a limitation of the current migration tool, not Pinecone's [serverless architecture](/reference/architecture/serverless-architecture), which provides virtually limitless data scalability. Also, serverless indexes do not support the following features. If you were using these features for your pod-based index, you will need to adapt your code. If you are blocked by these limitations, [contact Pinecone Support](/troubleshooting/contact-support). * [Selective metadata indexing](/guides/indexes/pods/manage-pod-based-indexes#selective-metadata-indexing) * Because high-cardinality metadata in serverless indexes does not cause high memory utilization, this operation is not relevant. * [Filtering index statistics by metadata](/reference/api/2024-10/data-plane/describeindexstats) ## How it works To migrate a pod-based index to serverless, you just click a few buttons in the Pinecone console, but behind-the-scenes, it's a 2-step process: After migration, you will have both a new serverless index and the original pod-based index. Once you've switched your workload to the serverless index, you can delete the pod-based index to avoid paying for unused resources. ## 1. Understand cost implications In most cases, migrating to serverless reduces costs significantly. However, costs can increase for read-heavy workloads with more than 1 query per second and for indexes with many records in a single namespace. Before migrating, consider [contacting Pinecone Support](/troubleshooting/contact-support) for help estimating and managing cost implications. ## 2. Prepare for migration Migrating a pod-based index to serverless can take anywhere from a few minutes to several hours, depending on the size of the index. During that time, you can continue reading from the pod-based index. However, all [upserts](/guides/index-data/upsert-data), [updates](/guides/manage-data/update-data), and [deletes](/guides/manage-data/delete-data) to the pod-based index will not automatically be reflected in the new serverless index, so be sure to prepare in one of the following ways: * **Pause write traffic:** If downtime is acceptable, pause traffic to the pod-based index before starting migration. After migration, you will start sending traffic to the serverless index. * **Log your writes:** If you need to continue reading from the pod-based index during migration, send read traffic to the pod-based index, but log your writes to a temporary location outside of Pinecone (e.g., S3). After migration, you will replay the logged writes to the new serverless index and start sending all traffic to the serverless index. ## 3. Start migration 1. In the [Pinecone console](https://app.pinecone.io/), go to your pod-based index and click the **ellipsis (...) menu > Migrate to serverless**. The dropdown will not display **Migrate to serverless** if the index has any of the listed [limitations](#limitations). 2. To save the legacy index and create a new serverless index now, follow the prompts. Depending on the size of the index, migration can take anywhere from a few minutes to several hours. While migration is in progress, you'll see the yellow **Initializing** status: ![create index from collection - initializing status](https://mintlify.s3.us-west-1.amazonaws.com/pinecone/images/create-serverless-from-collection-initializing.png) When the new serverless index is ready, the status will change to green: ![create index from collection - ready status](https://mintlify.s3.us-west-1.amazonaws.com/pinecone/images/create-serverless-from-collection-ready.png) ## 4. Update SDKs If you are using an older version of the Python, Node.js, Java, or Go SDK, you must update the SDK to work with serverless indexes. 1. Check your SDK version: ```shell Python pip show pinecone ``` ```shell JavaScript npm list | grep @pinecone-database/pinecone ``` ```shell Java # Check your dependency file or classpath ``` ```shell Go go list -u -m all | grep go-pinecone ``` 2. If your SDK version is less than 3.0.0 for [Python](https://github.com/pinecone-io/pinecone-python-client/blob/main/README.md), 2.0.0 for [Node.js](https://sdk.pinecone.io/typescript/), 1.0.0 for [Java](https://github.com/pinecone-io/pinecone-java-client), or 1.0.0 for [Go](https://github.com/pinecone-io/go-pinecone), upgrade the SDK as follows: ```Python Python pip install "pinecone[grpc]" --upgrade ``` ```JavaScript JavaScript npm install @pinecone-database/pinecone@latest ``` ```shell Java # Maven io.pinecone pinecone-client 5.0.0 # Gradle implementation "io.pinecone:pinecone-client:5.0.0" ``` ```go Go go get -u github.com/pinecone-io/go-pinecone/v3/pinecone@latest ``` If you are using the [.NET SDK](/reference/dotnet-sdk), add a package reference to your project file: ```shell C# dotnet add package Pinecone.Client ``` ## 5. Adapt existing code You must make some minor code changes to work with serverless indexes. Serverless indexes do not support some features, as outlined in [Limitations](#limitations). If you were relying on these features for your pod-based index, you’ll need to adapt your code. 1. Change how you import the Pinecone library and authenticate and initialize the client: ```Python Python from pinecone.grpc import PineconeGRPC as Pinecone from pinecone import ServerlessSpec, PodSpec # ServerlessSpec and PodSpec are required only when # creating serverless and pod-based indexes. pc = Pinecone(api_key="YOUR_API_KEY") ``` ```JavaScript JavaScript import { Pinecone } from '@pinecone-database/pinecone'; const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }); ``` ```java Java import io.pinecone.clients.Pinecone; import org.openapitools.db_control.client.model.*; public class InitializeClientExample { public static void main(String[] args) { Pinecone pc = new Pinecone.Builder("YOUR_API_KEY").build(); } } ``` ```go Go package main import ( "context" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); ``` 2. [Listing indexes](/guides/manage-data/manage-indexes) now fetches a complete description of each index. If you were relying on the output of this operation, you'll need to adapt your code. ```Python Python from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key="YOUR_API_KEY") index_list = pc.list_indexes() print(index_list) ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }) const indexList = await pc.listIndexes(); console.log(indexList); ``` ```java Java import io.pinecone.clients.Pinecone; import org.openapitools.db_control.client.model.*; public class ListIndexesExample { public static void main(String[] args) { Pinecone pc = new Pinecone.Builder("YOUR_API_KEY").build(); IndexList indexList = pc.listIndexes(); System.out.println(indexList); } } ``` ```go Go package main import ( "context" "encoding/json" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func prettifyStruct(obj interface{}) string { bytes, _ := json.MarshalIndent(obj, "", " ") return string(bytes) } func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } idxs, err := pc.ListIndexes(ctx) if err != nil { log.Fatalf("Failed to list indexes: %v", err) } else { for _, index := range idxs { fmt.Printf("index: %v\n", prettifyStruct(index)) } } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); var indexList = await pinecone.ListIndexesAsync(); Console.WriteLine(indexList); ``` ```shell curl PINECONE_API_KEY="YOUR_API_KEY" curl -i -X GET "https://api.pinecone.io/indexes" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" ``` The `list_indexes` operation now returns a response like the following: ```python Python [{ "name": "docs-example-sparse", "metric": "dotproduct", "host": "docs-example-sparse-govk0nt.svc.aped-4627-b74a.pinecone.io", "spec": { "serverless": { "cloud": "aws", "region": "us-east-1" } }, "status": { "ready": true, "state": "Ready" }, "vector_type": "sparse", "dimension": null, "deletion_protection": "disabled", "tags": { "environment": "development" } }, { "name": "docs-example-dense", "metric": "cosine", "host": "docs-example-dense-govk0nt.svc.aped-4627-b74a.pinecone.io", "spec": { "serverless": { "cloud": "aws", "region": "us-east-1" } }, "status": { "ready": true, "state": "Ready" }, "vector_type": "dense", "dimension": 1536, "deletion_protection": "disabled", "tags": { "environment": "development" } }] ``` ```javascript JavaScript { indexes: [ { name: 'docs-example-sparse', dimension: undefined, metric: 'dotproduct', host: 'docs-example-sparse-govk0nt.svc.aped-4627-b74a.pinecone.io', deletionProtection: 'disabled', tags: { environment: 'development', example: 'tag' }, embed: undefined, spec: { pod: undefined, serverless: { cloud: 'aws', region: 'us-east-1' } }, status: { ready: true, state: 'Ready' }, vectorType: 'sparse' }, { name: 'docs-example-dense', dimension: 1536, metric: 'cosine', host: 'docs-example-dense-govk0nt.svc.aped-4627-b74a.pinecone.io', deletionProtection: 'disabled', tags: { environment: 'development', example: 'tag' }, embed: undefined, spec: { pod: undefined, serverless: { cloud: 'aws', region: 'us-east-1' } }, status: { ready: true, state: 'Ready' }, vectorType: 'dense' } ] } ``` ```java Java class IndexList { indexes: [class IndexModel { name: docs-example-sparse dimension: null metric: dotproduct host: docs-example-sparse-govk0nt.svc.aped-4627-b74a.pinecone.io deletionProtection: disabled tags: {environment=development} embed: null spec: class IndexModelSpec { pod: null serverless: class ServerlessSpec { cloud: aws region: us-east-1 additionalProperties: null } additionalProperties: null } status: class IndexModelStatus { ready: true state: Ready additionalProperties: null } vectorType: sparse additionalProperties: null }, class IndexModel { name: docs-example-dense dimension: 1536 metric: cosine host: docs-example-dense-govk0nt.svc.aped-4627-b74a.pinecone.io deletionProtection: disabled tags: {environment=development} embed: null spec: class IndexModelSpec { pod: null serverless: class ServerlessSpec { cloud: aws region: us-east-1 additionalProperties: null } additionalProperties: null } status: class IndexModelStatus { ready: true state: Ready additionalProperties: null } vectorType: dense additionalProperties: null }] additionalProperties: null } ``` ```go Go index: { "name": "docs-example-sparse", "host": "docs-example-sparse-govk0nt.svc.aped-4627-b74a.pinecone.io", "metric": "dotproduct", "vector_type": "sparse", "deletion_protection": "disabled", "dimension": null, "spec": { "serverless": { "cloud": "aws", "region": "us-east-1" } }, "status": { "ready": true, "state": "Ready" }, "tags": { "environment": "development" } } index: { "name": "docs-example-dense", "host": "docs-example-dense-govk0nt.svc.aped-4627-b74a.pinecone.io", "metric": "cosine", "vector_type": "dense", "deletion_protection": "disabled", "dimension": 1536, "spec": { "serverless": { "cloud": "aws", "region": "us-east-1" } }, "status": { "ready": true, "state": "Ready" }, "tags": { "environment": "development" } } ``` ```csharp C# { "indexes": [ { "name": "docs-example-sparse", "metric": "dotproduct", "host": "docs-example-sparse-govk0nt.svc.aped-4627-b74a.pinecone.io", "deletion_protection": "disabled", "tags": { "environment": "development" }, "spec": { "serverless": { "cloud": "aws", "region": "us-east-1" } }, "status": { "ready": true, "state": "Ready" }, "vector_type": "sparse" }, { "name": "docs-example-dense", "dimension": 1536, "metric": "cosine", "host": "docs-example-dense-govk0nt.svc.aped-4627-b74a.pinecone.io", "deletion_protection": "disabled", "tags": { "environment": "development" }, "spec": { "serverless": { "cloud": "aws", "region": "us-east-1" } }, "status": { "ready": true, "state": "Ready" }, "vector_type": "dense" } ] } ``` ```json curl { "indexes": [ { "name": "docs-example-sparse", "vector_type": "sparse", "metric": "dotproduct", "dimension": null, "status": { "ready": true, "state": "Ready" }, "host": "docs-example-sparse-govk0nt.svc.aped-4627-b74a.pinecone.io", "spec": { "serverless": { "region": "us-east-1", "cloud": "aws" } }, "deletion_protection": "disabled", "tags": { "environment": "development" } }, { "name": "docs-example-dense", "vector_type": "dense", "metric": "cosine", "dimension": 1536, "status": { "ready": true, "state": "Ready" }, "host": "docs-example-dense-govk0nt.svc.aped-4627-b74a.pinecone.io", "spec": { "serverless": { "region": "us-east-1", "cloud": "aws" } }, "deletion_protection": "disabled", "tags": { "environment": "development" } } ] } ``` 3. [Describing an index](/guides/manage-data/manage-indexes) now returns a description of an index in a different format. It also returns the index host needed to run data plane operations against the index. If you were relying on the output of this operation, you'll need to adapt your code. ```Python Python from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key="YOUR_API_KEY") pc.describe_index(name="docs-example") ``` ```JavaScript JavaScript import { Pinecone } from '@pinecone-database/pinecone'; const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }); await pc.describeIndex('docs-example'); ``` ```java Java import io.pinecone.clients.Pinecone; import org.openapitools.db_control.client.model.*; public class DescribeIndexExample { public static void main(String[] args) { Pinecone pc = new Pinecone.Builder("YOURE_API_KEY").build(); IndexModel indexModel = pc.describeIndex("docs-example"); System.out.println(indexModel); } } ``` ```go Go package main import ( "context" "encoding/json" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func prettifyStruct(obj interface{}) string { bytes, _ := json.MarshalIndent(obj, "", " ") return string(bytes) } func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } idx, err := pc.DescribeIndex(ctx, "docs-example") if err != nil { log.Fatalf("Failed to describe index \"%v\": %v", idx.Name, err) } else { fmt.Printf("index: %v\n", prettifyStruct(idx)) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); var indexModel = await pinecone.DescribeIndexAsync("docs-example"); Console.WriteLine(indexModel); ``` ```bash curl PINECONE_API_KEY="YOUR_API_KEY" curl -i -X GET "https://api.pinecone.io/indexes/docs-example" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" ``` ## 6. Use your new index When you're ready to cutover to your new serverless index: 1. Your new serverless index has a different name and unique endpoint than your pod-based index. Update your code to target the new serverless index: ```Python Python index = pc.Index("YOUR_SERVERLESS_INDEX_NAME") ``` ```JavaScript JavaScript const index = pc.index("YOUR_SERVERLESS_INDEX_NAME"); ``` ```java Java import io.pinecone.clients.Index; import io.pinecone.clients.Pinecone; public class TargetIndexExample { public static void main(String[] args) { Pinecone pc = new Pinecone.Builder("YOUR_API_KEY").build(); Index index = pc.getIndexConnection("YOUR_SERVERLESS_INDEX_NAME"); ``` ```go Go package main import ( "context" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } idx, err := pc.DescribeIndex(ctx, "YOUR_SERVERLESS_INDEX_NAME") if err != nil { log.Fatalf("Failed to describe index \"%v\": %v", idx.Name, err) } idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: idx.Host, Namespace: "example-namespace"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host %v: %v", idx.Host, err) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); var index = pinecone.Index("YOUR_SERVERLESS_INDEX_NAME"); ``` ```bash curl # When using the API directly, you need the unique endpoint for your new serverless index. # See https://docs.pinecone.io/guides/manage-data/target-an-index for details. PINECONE_API_KEY="YOUR_API_KEY" INDEX_HOST="INDEX_HOST" curl -X POST "https://$INDEX_HOST/describe_index_stats" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" ``` 2. Reinitialize your clients. 3. If you logged writes to the pod-based index during migration, replay the logged writes to your serverless index. 4. [Delete the pod-based index](/guides/manage-data/manage-indexes#delete-an-index) to avoid paying for unused resources. It is not possible to save a serverless index as a collection, so if you want to retain the option to recreate your pod-based index, be sure to keep the collection you created earlier. ## See also * [Limits](/reference/api/database-limits) * [Serverless architecture](/reference/architecture/serverless-architecture) * [Understanding serverless cost](/guides/manage-cost/understanding-cost) # Restore a pod-based index Source: https://docs.pinecone.io/guides/indexes/pods/restore-a-pod-based-index You can restore a pod-based index by creating a new index from a [collection](/guides/indexes/pods/understanding-collections). ## Create a pod-based index from a collection To create a pod-based index from a [collection](/guides/manage-data/back-up-an-index#pod-based-index-backups-using-collections), use the [`create_index`](/reference/api/2024-10/control-plane/create_index) endpoint and provide a [`source_collection`](/reference/api/2024-10/control-plane/create_index#!path=source%5Fcollection\&t=request) parameter containing the name of the collection from which you wish to create an index. The new index can differ from the original source index: the new index can have a different name, number of pods, or pod type. The new index is queryable and writable. ```Python Python from pinecone.grpc import PineconeGRPC as Pinecone, PodSpec pc = Pinecone(api_key="YOUR_API_KEY") pc.create_index( name="docs-example", dimension=128, metric="cosine", spec=PodSpec( environment="us-west-1-gcp", pod_type="p1.x1", pods=1, source_collection="example-collection" ) ) ``` ```JavaScript JavaScript import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }); await pc.createIndex({ name: 'docs-example', dimension: 128, metric: 'cosine', spec: { pod: { environment: 'us-west-1-gcp', podType: 'p1.x1', pods: 1, sourceCollection: 'example-collection' } } }); ``` ```java Java import io.pinecone.clients.Pinecone; import org.openapitools.db_control.client.model.IndexModel; import org.openapitools.db_control.client.model.DeletionProtection; public class CreateIndexFromCollectionExample { public static void main(String[] args) { Pinecone pc = new Pinecone.Builder("YOUR_API_KEY").build(); pc.createPodsIndex("docs-example", 1536, "us-west1-gcp", "p1.x1", "cosine", "example-collection", DeletionProtection.DISABLED); } } ``` ```go Go package main import ( "context" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } indexName := "docs-example" metric := pinecone.Dotproduct deletionProtection := pinecone.DeletionProtectionDisabled idx, err := pc.CreatePodIndex(ctx, &pinecone.CreatePodIndexRequest{ Name: indexName, Metric: &metric, Dimension: 1536, Environment: "us-east1-gcp", PodType: "p1.x1", SourceCollection: "example-collection", DeletionProtection: &deletionProtection, }) if err != nil { log.Fatalf("Failed to create pod-based index: %v", idx.Name) } else { fmt.Printf("Successfully created pod-based index: %v", idx.Name) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); var createIndexRequest = await pinecone.CreateIndexAsync(new CreateIndexRequest { Name = "docs-example", Dimension = 1538, Metric = MetricType.Cosine, Spec = new PodIndexSpec { Pod = new PodSpec { Environment = "us-east1-gcp", PodType = "p1.x1", Pods = 1, Replicas = 1, Shards = 1, SourceCollection = "example-collection", } }, DeletionProtection = DeletionProtection.Enabled, }); ``` ```bash curl PINECONE_API_KEY="YOUR_API_KEY" curl -s "https://api.pinecone.io/indexes" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "name": "docs-example", "dimension": 128, "metric": "cosine", "spec": { "pod": { "environment": "us-west-1-gcp", "pod_type": "p1.x1", "pods": 1, "source_collection": "example-collection" } } }' ``` # Scale pod-based indexes Source: https://docs.pinecone.io/guides/indexes/pods/scale-pod-based-indexes This guidance applies to [pod-based indexes](/guides/index-data/indexing-overview#pod-based-indexes) only. With [serverless indexes](/guides/index-data/indexing-overview#serverless-indexes), you don't configure any compute or storage resources, and you don't manually manage those resources to meet demand, save on cost, or ensure high availability. Instead, serverless indexes scale automatically based on usage. While your index can still serve queries, new upserts may fail as the capacity becomes exhausted. If you need to scale your environment to accommodate more vectors, you can modify your existing index and scale it vertically or create a new index and scale horizontally. This page explains how you can scale your [pod-based indexes](/guides/index-data/indexing-overview#pod-based-indexes) horizontally and vertically. ## Vertical vs. horizontal scaling If you need to scale your environment to accommodate more vectors, you can modify your existing index to scale it vertically or create a new index and scale horizontally. This article will describe both methods and how to scale your index effectively. ## Vertical scaling [Vertical scaling](https://www.pinecone.io/learn/testing-p2-collections-scaling/#vertical-scaling-on-p1-and-s1) is fast and involves no downtime. This is a good choice when you can't pause upserts and must continue serving traffic. It also allows you to double your capacity instantly. However, there are some factors to consider. ### Increase pod size The default [pod size](/guides/index-data/indexing-overview#pod-size-and-performance) is `x1`. You can increase the size to `x2, `x4, or `x8`. Moving up to the next size effectively doubles the capacity of the index. If you need to scale by smaller increments, then consider horizontal scaling. Increasing the pod size of your index does not result in downtime. Reads and writes continue uninterrupted during the scaling process, which completes in about 10 minutes. Currently, you cannot reduce the pod size of your indexes. The number of base pods you specify when you initially create the index is static and cannot be changed. For example, if you start with 10 pods of `p1.x1` and vertically scale to `p1.x2`, this equates to 20 pods worth of usage. Pod types (performance versus storage pods) also cannot be changed with vertical scaling. If you want to change your pod type while scaling, then horizontal scaling is the better option. You can only scale index sizes up and cannot scale them back down. To increase the pod size of an existing index, use the [`configure_index`](/reference/api/2024-10/control-plane/configure_index) operation and append the new size to the `pod_type` parameter, separated by a period (.). **Example** The following example assumes that `docs-example` has size `x1` and increases the size to `x2`. ```Python Python from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key="YOUR_API_KEY") pc.configure_index("docs-example", pod_type="s1.x2") ``` ```JavaScript JavaScript import { Pinecone } from '@pinecone-database/pinecone' const pinecone = new Pinecone({ apiKey: 'YOUR_API_KEY' }); await pc.configureIndex('docs-example', { spec: { pod: { podType: 's1.x2', }, }, }); ``` ```java Java import io.pinecone.clients.Pinecone; public class ConfigureIndexExample { public static void main(String[] args) { Pinecone pc = new Pinecone.Builder("PINECONE_API_KEY").build(); pc.configurePodsIndex("docs-example", "s1.x2"); } } ``` ```go Go package main import ( "context" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } idx, err := pc.ConfigureIndex(ctx, "docs-example", pinecone.ConfigureIndexParams{PodType: "s1.x2"}) if err != nil { log.Fatalf("Failed to configure index \"%v\": %v", idx.Name, err) } else { fmt.Printf("Successfully configured index \"%v\"", idx.Name) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); var indexMetadata = await pinecone.ConfigureIndexAsync("docs-example", new ConfigureIndexRequest { Spec = new ConfigureIndexRequestSpec { Pod = new ConfigureIndexRequestSpecPod { PodType = "s1.x2", } } }); ``` ```bash curl PINECONE_API_KEY="YOUR_API_KEY" curl -s -X PATCH "https://api.pinecone.io/indexes/docs-example-curl" \ -H "Content-Type: application/json" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "pod_type": "s1.x2" }' ``` ### Check the status of a pod size change To check the status of a pod size change, use the [`describe_index`](/reference/api/2024-10/control-plane/describe_index/) endpoint. The `status` field in the results contains the key-value pair `"state":"ScalingUp"` or `"state":"ScalingDown"` during the resizing process and the key-value pair `"state":"Ready"` after the process is complete. The index fullness metric provided by [`describe_index_stats`](/reference/api/2024-10/control-plane/describe_index) may be inaccurate until the resizing process is complete. **Example** The following example uses `describe_index` to get the index status of the index `docs-example`. The `status` field contains the key-value pair `"state":"ScalingUp"`, indicating that the resizing process is still ongoing. ```Python Python from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key="YOUR_API_KEY") pc.describe_index(name="docs-example") ``` ```JavaScript JavaScript import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }); await pc.describeIndex({ name: "docs-example", }); ``` ```java Java import io.pinecone.clients.Pinecone; import org.openapitools.db_control.client.model.*; public class DescribeIndexExample { public static void main(String[] args) { Pinecone pc = new Pinecone.Builder("YOUR_API_KEY").build(); IndexModel indexModel = pc.describeIndex("docs-example"); } } ``` ```go Go package main import ( "context" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } idx, err := pc.DescribeIndex(ctx, "docs-example") if err != nil { log.Fatalf("Failed to describe index %v: %v", idx.Name, err) } else { fmt.Printf("Successfully found index: %v", idx.Name) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); var indexModel = await pinecone.DescribeIndexAsync("docs-example"); Console.WriteLine(indexModel); ``` ```bash curl PINECONE_API_KEY="YOUR_API_KEY" curl -s -X GET "https://api.pinecone.io/indexes/docs-example-curl" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" ``` ## Horizontal scaling There are two approaches to horizontal scaling in Pinecone: adding pods and adding replicas. Adding pods increases all resources but requires a pause in upserts; adding replicas only increases throughput and requires no pause in upserts. ### Add pods Adding pods to an index increases all resources, including available capacity. Adding pods to an existing index is possible using our [collections](/guides/indexes/pods/understanding-collections) feature. A collection is an immutable snapshot of your index in time: a collection stores the data but not the original index configuration. When you [create an index from a collection](/guides/indexes/pods/create-a-pod-based-index#create-a-pod-index-from-a-collection), you define the new index configuration. This allows you to scale the base pod count horizontally without scaling vertically. The main advantage of this approach is that you can scale incrementally instead of doubling capacity as with vertical scaling. Also, you can redefine pod types if you are experimenting or if you need to use a different pod type, such as performance-optimized pods or storage-optimized pods. Another advantage of this method is that you can change your [metadata configuration](/guides/indexes/pods/manage-pod-based-indexes#selective-metadata-indexing) to redefine metadata fields as indexed or stored-only. This is important when tuning your index for the best throughput. Here are the general steps to make a copy of your index and create a new index while changing the pod type, pod count, metadata configuration, replicas, and all typical parameters when creating a new collection: 1. Pause upserts. 2. Create a collection from the current index. 3. Create an index from the collection with new parameters. 4. Continue upserts to the newly created index. Note: the URL has likely changed. 5. Delete the old index if desired. ### Add replicas Each replica duplicates the resources and data in an index. This means that adding additional replicas increases the throughput of the index but not its capacity. However, adding replicas does not require downtime. Throughput in terms of queries per second (QPS) scales linearly with the number of replicas per index. To add replicas, use the `configure_index` endpoint to [increase the number of replicas for your index](/guides/indexes/pods/scale-pod-based-indexes#add-replicas). ```Python Python from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key="YOUR_API_KEY") pc.configure_index("docs-example", replicas=4) ``` ```JavaScript JavaScript import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }); await pc.configureIndex('docs-example', { spec: { pod: { replicas: 4, }, }, }); ``` ```java Java import io.pinecone.clients.Pinecone; public class ConfigureIndexExample { public static void main(String[] args) { Pinecone pc = new Pinecone.Builder("PINECONE_API_KEY").build(); pc.configurePodsIndex("docs-example", 4, DeletionProtection.DISABLED); } } ``` ```go Go package main import ( "context" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } idx, err := pc.ConfigureIndex(ctx, "docs-example", pinecone.ConfigureIndexParams{Replicas: 4}) if err != nil { log.Fatalf("Failed to configure index \"%v\": %v", idx.Name, err) } else { fmt.Printf("Successfully configured index \"%v\"", idx.Name) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); var configureIndexRequest = await pinecone.ConfigureIndexAsync("docs-example", new ConfigureIndexRequest { Spec = new ConfigureIndexRequestSpec { Pod = new ConfigureIndexRequestSpecPod { Replicas = 4, } } }); ``` ```bash curl PINECONE_API_KEY="YOUR_API_KEY" curl -s -X PATCH "https://api.pinecone.io/indexes/docs-example-curl" \ -H "Content-Type: application/json" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "replicas": 4 }' ``` ## Next steps * See our learning center for more information on [vertical scaling](https://www.pinecone.io/learn/testing-p2-collections-scaling/#vertical-scaling-on-p1-and-s1). * Learn more about [collections](/guides/indexes/pods/understanding-collections). # Understanding collections Source: https://docs.pinecone.io/guides/indexes/pods/understanding-collections A collection is a static copy of a pod-based index that only consumes storage. It is a non-queryable representation of a set of records. You can [create a collection](/guides/indexes/pods/back-up-a-pod-based-index) of a pod-based index, and you can [create a new pod-based index from a collection](/guides/manage-data/restore-an-index). This allows you to restore the index with the same or different configurations. Once a collection is created, it cannot be moved to a different project. ## Use cases Creating a collection is useful when performing tasks like the following: * Protecting an index from manual or system failures. * Temporarily shutting down an index. * Copying the data from one index into a different index. * Making a backup of your index. * Experimenting with different index configurations. ## Performance Collections operations perform differently, depending on the pod type of the index: * Creating a `p1` or `s1` index from a collection takes approximately 10 minutes. * Creating a `p2` index from a collection can take several hours when the number of vectors is on the order of 1,000,000. ## Limitations Collection limitations are as follows: * You can only perform operations on collections in the current Pinecone project. ## Pricing See [Pricing](https://www.pinecone.io/pricing/) for up-to-date pricing information. # Understanding pod-based indexes Source: https://docs.pinecone.io/guides/indexes/pods/understanding-pod-based-indexes With pod-based indexes, you choose one or more pre-configured units of hardware (pods). Depending on the pod type, pod size, and number of pods used, you get different amounts of storage and higher or lower latency and throughput. Be sure to [choose an appropriate pod type and size](/guides/indexes/pods/choose-a-pod-type-and-size) for your dataset and workload. ## Pod types Different pod types are priced differently. See [Understanding cost](/guides/manage-cost/understanding-cost) for more details. Once a pod-based index is created, you cannot change its pod type. However, you can create a collection from an index and then [create a new index with a different pod type](/guides/indexes/pods/create-a-pod-based-index#create-a-pod-index-from-a-collection) from the collection. ### s1 pods These storage-optimized pods provide large storage capacity and lower overall costs with slightly higher query latencies than p1 pods. They are ideal for very large indexes with moderate or relaxed latency requirements. Each s1 pod has enough capacity for around 5M vectors of 768 dimensions. ### p1 pods These performance-optimized pods provide very low query latencies, but hold fewer vectors per pod than s1 pods. They are ideal for applications with low latency requirements (\<100ms). Each p1 pod has enough capacity for around 1M vectors of 768 dimensions. ### p2 pods The p2 pod type provides greater query throughput with lower latency. For vectors with fewer than 128 dimension and queries where `topK` is less than 50, p2 pods support up to 200 QPS per replica and return queries in less than 10ms. This means that query throughput and latency are better than s1 and p1. Each p2 pod has enough capacity for around 1M vectors of 768 dimensions. However, capacity may vary with dimensionality. The data ingestion rate for p2 pods is significantly slower than for p1 pods; this rate decreases as the number of dimensions increases. For example, a p2 pod containing vectors with 128 dimensions can upsert up to 300 updates per second; a p2 pod containing vectors with 768 dimensions or more supports upsert of 50 updates per second. Because query latency and throughput for p2 pods vary from p1 pods, test p2 pod performance with your dataset. The p2 pod type does not support sparse vector values. ## Pod size and performance Each pod type supports four pod sizes: `x1`, `x2`, `x4`, and `x8`. Your index storage and compute capacity doubles for each size step. The default pod size is `x1`. You can increase the size of a pod after index creation. To learn about changing the pod size of an index, see [Configure an index](/guides/indexes/pods/scale-pod-based-indexes#increase-pod-size). ## Pod environments When creating a pod-based index, you must choose the cloud environment where you want the index to be hosted. The project environment can affect your [pricing](https://pinecone.io/pricing). The following table lists the available cloud regions and the corresponding values of the `environment` parameter for the [`create_index`](/guides/index-data/create-an-index#create-a-pod-based-index) endpoint: | Cloud | Region | Environment | | ----- | ---------------------------- | ----------------------------- | | GCP | us-west-1 (N. California) | `us-west1-gcp` | | GCP | us-central-1 (Iowa) | `us-central1-gcp` | | GCP | us-west-4 (Las Vegas) | `us-west4-gcp` | | GCP | us-east-4 (Virginia) | `us-east4-gcp` | | GCP | northamerica-northeast-1 | `northamerica-northeast1-gcp` | | GCP | asia-northeast-1 (Japan) | `asia-northeast1-gcp` | | GCP | asia-southeast-1 (Singapore) | `asia-southeast1-gcp` | | GCP | us-east-1 (South Carolina) | `us-east1-gcp` | | GCP | eu-west-1 (Belgium) | `eu-west1-gcp` | | GCP | eu-west-4 (Netherlands) | `eu-west4-gcp` | | AWS | us-east-1 (Virginia) | `us-east-1-aws` | | Azure | eastus (Virginia) | `eastus-azure` | [Contact us](http://www.pinecone.io/contact/) if you need a dedicated deployment in other regions. The environment cannot be changed after the index is created. # Manage cost Source: https://docs.pinecone.io/guides/manage-cost/manage-cost This page provides guidance on managing the cost of Pinecone. For the latest pricing details, see our [pricing page](https://www.pinecone.io/pricing/). For help estimating total cost, see [Understanding total cost](/guides/manage-cost/understanding-cost). To see a calculation of your current usage and costs, see the [usage dashboard](/guides/manage-cost/monitor-your-usage) in the Pinecone console. ## Set a monthly spend alert To receive an email notification when your organization's spending reaches a specified limit, set a monthly spend alert: 1. Go to [**Settings > Usage**](https://app.pinecone.io/organizations/-/settings/usage) in the Pinecone console. 2. In the **Monthly spend alert** section, click **Create spend alert**. 3. Enter the dollar amount at which you want to receive an alert. 4. Click **Update alert**. ## Choose the right index * **Serverless:** With serverless indexes, you don't configure or manage any compute or storage resources. Instead, based on a [breakthrough architecture](/reference/architecture/serverless-architecture), serverless indexes scale automatically based on usage, and you pay only for the amount of data stored and operations performed, with no minimums. This means that there's no extra cost for having additional indexes. * **Pod-based:** Pod sizes are designed for different applications, and some are more expensive than others. [Choose the appropriate pod type and size](/guides/indexes/pods/choose-a-pod-type-and-size), so you pay for the resources you need. For example, the `s1` pod type provides large storage capacity and lower overall costs with slightly higher query latencies than `p1` pods. By switching to a different pod type, you may be able to reduce costs while still getting the performance your application needs. ## List by ID prefix `list` is supported only for serverless indexes. By using a hierarchical ID schema, you can retrieve records without performing a query. To do so, you can use [`list`](/reference/api/2024-10/data-plane/list) to retrieve records by ID prefix, then use `fetch` to retrieve the records you need. This can reduce costs, because [`query` consumes more RUs when scanning a larger namespace](/guides/manage-cost/understanding-cost#query), while [`fetch` consumes a fixed ratio of RUs to records retrieved](/guides/manage-cost/understanding-cost#fetch). Similarly, you can use `list` and then [delete by ID prefix](/guides/manage-data/manage-document-chunks#delete-all-records-for-a-parent-document). To learn more, see [Manage document chunks](/guides/manage-data/manage-document-chunks). ## Back up inactive indexes Serverless indexes do not support collections at this time. When a specific index is not in use, [back it up using collections](/guides/manage-data/back-up-an-index) and delete the inactive index. When you're ready to use these vectors again, you can [create a new index from the collection](/guides/indexes/pods/create-a-pod-based-index#create-a-pod-index-from-a-collection). This new index can also use a different index type or size. Because it's relatively cheap to store collections, you can reduce costs by only running an index when it's in use. ## Use namespaces for multitenancy If your application requires you to isolate the data of each customer/user, consider [implementing multitenancy with serverless indexes and namespaces](/guides/index-data/implement-multitenancy). With serverless indexes, you pay only for the amount of data stored and operations performed. For queries in particular, the cost is partly based on the total number of records that must be scanned, so using namespaces can significantly reduce query costs. ## Commit to annual spend Users who commit to an annual contract may qualify for discounted rates. To learn more, [contact Pinecone sales](https://www.pinecone.io/contact/). ## Talk to support Users on the Standard and Enterprise plans can [contact Support](https://app.pinecone.io/organizations/-/settings/support/ticket) for help in optimizing costs. ## See also * [Understanding cost](/guides/manage-cost/understanding-cost) * [Monitoring usage](/guides/manage-cost/monitor-your-usage) # Monitor your usage Source: https://docs.pinecone.io/guides/manage-cost/monitor-your-usage This page shows you how to monitor the overall usage and costs for your Pinecone organization as well as usage and performance metrics for individual indexes. ## Monitor organization-level usage You must be the [organization owner](/guides/organizations/understanding-organizations#organization-owners) to view usage across your Pinecone organization. Also, this feature is available only to organizations on the Standard or Enterprise plans. To view and download a report of your usage and costs for your Pinecone organization, go to [**Settings > Usage**](https://app.pinecone.io/organizations/-/settings/usage) in the Pinecone console. All dates are given in UTC to match billing invoices. ## Monitor index-level usage You can monitor index-level usage directly in the Pinecone console, or you can pull them into [Prometheus](https://prometheus.io/). For more details, see [Monitoring](/guides/production/monitoring). ## Monitor operation-level usage ### Read units Read operations like `query` and `fetch` return a `usage` parameter with the [read unit](/guides/manage-cost/understanding-cost#read-units) consumption of each request that is made. For example, a query to an example index might return this result and summary of read unit usage: ```Python Python from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key="YOUR_API_KEY") index = pc.Index("pinecone-index") index.query( vector=[0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3], top_k=3, include_values=True ) # Returns: # { # "matches": [ # { # "id": "C", # "score": -1.76717265e-07, # "values": [0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3], # }, # { # "id": "B", # "score": 0.080000028, # "values": [0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2], # }, # { # "id": "D", # "score": 0.0800001323, # "values": [0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4], # }, # ], # "namespace": "", # "usage": {"read_units": 5} # } ``` ```JavaScript JavaScript import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: "YOUR_API_KEY" }) const index = pc.index("pinecone-index") const queryResponse = await index.query({ vector: [0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3], topK: 3, includeValues: true, }); // Returns: // { // matches: [ // { // id: 'C', // score: 0.000072891, // values: [0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3] // }, // { // id: 'B', // score: 0.080000028, // values: [0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2] // }, // { // id: 'D', // score: 0.0800001323, // values: [0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4] // } // ], // namespace: '', // usage: {readUnits: 5} // } ``` For a more in-depth demonstration of how to use read units to inspect read costs, see [this notebook](https://github.com/pinecone-io/examples/blob/master/docs/read-units-demonstrated.ipynb). ### Embedding tokens Requests to one of [Pinecone's hosted embedding models](/guides/index-data/create-an-index#embedding-models), either directly via the [`embed` operation](/reference/api/2025-01/inference/generate-embeddings) or automatically when upserting or querying an [index with integrated embedding](/guides/index-data/indexing-overview#integrated-embedding), return a `usage` parameter with the total tokens generated. For example, the following request to use the `multilingual-e5-large` model to generate embeddings for sentences related to the word “apple” might return this request and summary of embedding tokens generated: ```python Python # Import the Pinecone library from pinecone.grpc import PineconeGRPC as Pinecone from pinecone import ServerlessSpec import time # Initialize a Pinecone client with your API key pc = Pinecone(api_key="YOUR_API_KEY") # Define a sample dataset where each item has a unique ID and piece of text data = [ {"id": "vec1", "text": "Apple is a popular fruit known for its sweetness and crisp texture."}, {"id": "vec2", "text": "The tech company Apple is known for its innovative products like the iPhone."}, {"id": "vec3", "text": "Many people enjoy eating apples as a healthy snack."}, {"id": "vec4", "text": "Apple Inc. has revolutionized the tech industry with its sleek designs and user-friendly interfaces."}, {"id": "vec5", "text": "An apple a day keeps the doctor away, as the saying goes."}, {"id": "vec6", "text": "Apple Computer Company was founded on April 1, 1976, by Steve Jobs, Steve Wozniak, and Ronald Wayne as a partnership."} ] # Convert the text into numerical vectors that Pinecone can index embeddings = pc.inference.embed( model="llama-text-embed-v2", inputs=[d['text'] for d in data], parameters={"input_type": "passage", "truncate": "END"} ) print(embeddings) ``` ```javascript JavaScript // Import the Pinecone library import { Pinecone } from '@pinecone-database/pinecone'; // Initialize a Pinecone client with your API key const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }); // Define a sample dataset where each item has a unique ID and piece of text const data = [ { id: 'vec1', text: 'Apple is a popular fruit known for its sweetness and crisp texture.' }, { id: 'vec2', text: 'The tech company Apple is known for its innovative products like the iPhone.' }, { id: 'vec3', text: 'Many people enjoy eating apples as a healthy snack.' }, { id: 'vec4', text: 'Apple Inc. has revolutionized the tech industry with its sleek designs and user-friendly interfaces.' }, { id: 'vec5', text: 'An apple a day keeps the doctor away, as the saying goes.' }, { id: 'vec6', text: 'Apple Computer Company was founded on April 1, 1976, by Steve Jobs, Steve Wozniak, and Ronald Wayne as a partnership.' } ]; // Convert the text into numerical vectors that Pinecone can index const model = 'llama-text-embed-v2'; const embeddings = await pc.inference.embed( model, data.map(d => d.text), { inputType: 'passage', truncate: 'END' } ); console.log(embeddings); ``` ```java Java // Import the required classes import io.pinecone.clients.Index; import io.pinecone.clients.Inference; import io.pinecone.clients.Pinecone; import org.openapitools.inference.client.ApiException; import org.openapitools.inference.client.model.Embedding; import org.openapitools.inference.client.model.EmbeddingsList; import java.math.BigDecimal; import java.util.*; import java.util.stream.Collectors; public class GenerateEmbeddings { public static void main(String[] args) throws ApiException { // Initialize a Pinecone client with your API key Pinecone pc = new Pinecone.Builder("YOUR_API_KEY").build(); Inference inference = pc.getInferenceClient(); // Prepare input sentences to be embedded List data = Arrays.asList( new DataObject("vec1", "Apple is a popular fruit known for its sweetness and crisp texture."), new DataObject("vec2", "The tech company Apple is known for its innovative products like the iPhone."), new DataObject("vec3", "Many people enjoy eating apples as a healthy snack."), new DataObject("vec4", "Apple Inc. has revolutionized the tech industry with its sleek designs and user-friendly interfaces."), new DataObject("vec5", "An apple a day keeps the doctor away, as the saying goes."), new DataObject("vec6", "Apple Computer Company was founded on April 1, 1976, by Steve Jobs, Steve Wozniak, and Ronald Wayne as a partnership.") ); List inputs = data.stream() .map(DataObject::getText) .collect(Collectors.toList()); // Specify the embedding model and parameters String embeddingModel = "llama-text-embed-v2"; Map parameters = new HashMap<>(); parameters.put("input_type", "passage"); parameters.put("truncate", "END"); // Generate embeddings for the input data EmbeddingsList embeddings = inference.embed(embeddingModel, parameters, inputs); // Get embedded data List embeddedData = embeddings.getData(); } private static List convertBigDecimalToFloat(List bigDecimalValues) { return bigDecimalValues.stream() .map(BigDecimal::floatValue) .collect(Collectors.toList()); } } class DataObject { private String id; private String text; public DataObject(String id, String text) { this.id = id; this.text = text; } public String getId() { return id; } public String getText() { return text; } } ``` ```go Go package main // Import the required packages import ( "context" "encoding/json" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) type Data struct { ID string Text string } type Query struct { Text string } func prettifyStruct(obj interface{}) string { bytes, _ := json.MarshalIndent(obj, "", " ") return string(bytes) } func main() { ctx := context.Background() // Initialize a Pinecone client with your API key pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // Define a sample dataset where each item has a unique ID and piece of text data := []Data{ {ID: "vec1", Text: "Apple is a popular fruit known for its sweetness and crisp texture."}, {ID: "vec2", Text: "The tech company Apple is known for its innovative products like the iPhone."}, {ID: "vec3", Text: "Many people enjoy eating apples as a healthy snack."}, {ID: "vec4", Text: "Apple Inc. has revolutionized the tech industry with its sleek designs and user-friendly interfaces."}, {ID: "vec5", Text: "An apple a day keeps the doctor away, as the saying goes."}, {ID: "vec6", Text: "Apple Computer Company was founded on April 1, 1976, by Steve Jobs, Steve Wozniak, and Ronald Wayne as a partnership."}, } // Specify the embedding model and parameters embeddingModel := "llama-text-embed-v2" docParameters := pinecone.EmbedParameters{ InputType: "passage", Truncate: "END", } // Convert the text into numerical vectors that Pinecone can index var documents []string for _, d := range data { documents = append(documents, d.Text) } docEmbeddingsResponse, err := pc.Inference.Embed(ctx, &pinecone.EmbedRequest{ Model: embeddingModel, TextInputs: documents, Parameters: docParameters, }) if err != nil { log.Fatalf("Failed to embed documents: %v", err) } else { fmt.Printf(prettifyStruct(docEmbeddingsResponse)) } } ``` ```csharp C# using Pinecone; using System; using System.Collections.Generic; // Initialize a Pinecone client with your API key var pinecone = new PineconeClient("YOUR_API_KEY"); // Prepare input sentences to be embedded var data = new[] { new { Id = "vec1", Text = "Apple is a popular fruit known for its sweetness and crisp texture." }, new { Id = "vec2", Text = "The tech company Apple is known for its innovative products like the iPhone." }, new { Id = "vec3", Text = "Many people enjoy eating apples as a healthy snack." }, new { Id = "vec4", Text = "Apple Inc. has revolutionized the tech industry with its sleek designs and user-friendly interfaces." }, new { Id = "vec5", Text = "An apple a day keeps the doctor away, as the saying goes." }, new { Id = "vec6", Text = "Apple Computer Company was founded on April 1, 1976, by Steve Jobs, Steve Wozniak, and Ronald Wayne as a partnership." } }; // Specify the embedding model and parameters var embeddingModel = "llama-text-embed-v2"; // Generate embeddings for the input data var embeddings = await pinecone.Inference.EmbedAsync(new EmbedRequest { Model = embeddingModel, Inputs = data.Select(item => new EmbedRequestInputsItem { Text = item.Text }), Parameters = new Dictionary { ["input_type"] = "passage", ["truncate"] = "END" } }); Console.WriteLine(embeddings); ``` ```shell curl PINECONE_API_KEY="YOUR_API_KEY" curl https://api.pinecone.io/embed \ -H "Api-Key: $PINECONE_API_KEY" \ -H "Content-Type: application/json" \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "model": "llama-text-embed-v2", "parameters": { "input_type": "passage", "truncate": "END" }, "inputs": [ {"text": "Apple is a popular fruit known for its sweetness and crisp texture."}, {"text": "The tech company Apple is known for its innovative products like the iPhone."}, {"text": "Many people enjoy eating apples as a healthy snack."}, {"text": "Apple Inc. has revolutionized the tech industry with its sleek designs and user-friendly interfaces."}, {"text": "An apple a day keeps the doctor away, as the saying goes."}, {"text": "Apple Computer Company was founded on April 1, 1976, by Steve Jobs, Steve Wozniak, and Ronald Wayne as a partnership."} ] }' ``` The returned object looks like this: ```python Python EmbeddingsList( model='llama-text-embed-v2', data=[ {'values': [0.04925537109375, -0.01313018798828125, -0.0112762451171875, ...]}, ... ], usage={'total_tokens': 130} ) ``` ```javascript JavaScript EmbeddingsList(1) [ { values: [ 0.04925537109375, -0.01313018798828125, -0.0112762451171875, ... ] }, ... model: 'llama-text-embed-v2', data: [ { values: [Array] } ], usage: { totalTokens: 130 } ] ``` ```java Java class EmbeddingsList { model: llama-text-embed-v2 data: [class Embedding { values: [0.04925537109375, -0.01313018798828125, -0.0112762451171875, ...] additionalProperties: null }, ...] usage: class EmbeddingsListUsage { totalTokens: 130 additionalProperties: null } additionalProperties: null } ``` ```go Go { "data": [ { "values": [ 0.03942871, -0.010177612, -0.046051025, ... ] }, ... ], "model": "llama-text-embed-v2", "usage": { "total_tokens": 130 } } ``` ```csharp C# { "model": "llama-text-embed-v2", "data": [ { "values": [ 0.04913330078125, -0.01306915283203125, -0.01116180419921875, ... ] }, ... ], "usage": { "total_tokens": 130 } } ``` ```json curl { "data": [ { "values": [ 0.04925537109375, -0.01313018798828125, -0.0112762451171875, ... ] }, ... ], "model": "llama-text-embed-v2", "usage": { "total_tokens": 130 } } ``` ## See also * [Understanding cost](/guides/manage-cost/understanding-cost) * [Managing cost](/guides/manage-cost/manage-cost) # Understanding cost Source: https://docs.pinecone.io/guides/manage-cost/understanding-cost This page describes how costs are incurred for Pinecone Database, Pinecone Inference, and Pinecone Assistant. Pinecone Database pricing is calculated on a per-index basis, not per project or organization; if you run more than one index in a project, you'll be charged for each index based on its size or usage. Likewise, Pinecone Assistant pricing is calculated on a per-assistant basis. Pinecone Inference pricing is based on usage. For the latest pricing details, please see [our pricing page](https://www.pinecone.io/pricing/). ## Platform fee Your plan determines the features, quotas, support, and usage credits you have access to. Aside from your platform subscription fee, all Pinecone usage is pay-as-you-go for only what you use. | Plan | Platform fee | Included usage credits | | ---------- | ------------ | ---------------------- | | Starter | N/A | N/A | | Standard | \$25/month | \$15/month | | Enterprise | \$500/month | \$150/month | Please note that usage credits do not roll over from month to month. Platform fees do not apply to organizations with annual commitments. ## Serverless indexes With [serverless indexes](/guides/index-data/indexing-overview#serverless-indexes), you don't configure or manage any compute or storage resources. Instead, based on a [breakthrough architecture](/reference/architecture/serverless-architecture), serverless indexes scale automatically based on usage, and you pay only for the amount of data stored and operations performed, based on three usage metrics: * **Read units (RUs):** Read units measure the resources consumed by read operations such as query, fetch, and list. * **Write units (WUs):** Write units measure the resources used by write operations such as upsert, update, and delete. * **Storage:** The size of an index on a per-Gigabyte (GB) hourly rate. For each usage metric, pricing varies by cloud provider, region, and plan (see our [pricing page](https://www.pinecone.io/pricing/)). ### Read units Read operations consume read units (RUs). Read units measure the compute, I/O, and network resources used during the read process. The following operations consume RUs: * [Fetch](/reference/api/2024-10/data-plane/fetch) * [Query](/reference/api/2024-10/data-plane/query) * [List](/reference/api/2024-10/data-plane/list) The number of RUs used by a specific request is always included in its response. For a demonstration of how to use read units to inspect read costs, see [this notebook](https://github.com/pinecone-io/examples/blob/master/docs/read-units-demonstrated.ipynb). #### Fetch A fetch request uses 1 RU for every 10 fetched records. | # of fetched records | RU usage | | -------------------- | -------- | | 10 | 1 | | 50 | 5 | | 107 | 11 | Specifying a non-existent ID or adding the same ID more than once does not increase the number of RUs used. However, a fetch request will always use at least 1 RU. #### Query The number of RUs used by a query is proportional to the following factors: * Record count: The number of vectors contained in the target index. Only vectors stored in the relevant namespace are used. * Record size: Higher dimensionality or larger metadata increases the size of each scanned vector. Because serverless indexes organize vectors in similarity-based clusters, only a fraction of each index will be read for each query. The number of RUs a query uses therefore increases much more slowly than the index size. The following table contains the RU cost of a query at different namespace sizes and record dimensionality, assuming an average metadata size around 500 bytes: | Records per namespace | Dimension=384 | Dimension=768 | Dimension=1536 | | --------------------- | ------------- | ------------- | -------------- | | 100,000 | 5 RUs | 5 RUs | 6 RUs | | 1,000,000 | 6 RUs | 10 RUs | 18 RUs | | 10,000,000 | 18 RUs | 32 RUs | 59 RUs | Scanning a namespace has a minimal cost of 5 RUs. When either `include_metadata` or `include_values` are specified, an internal fetch call retrieves the full record values for the IDs returned in the initial scan. This stage consumes RUs equal to a matching fetch call - 1 RU per 10 records in the result set. | TopK value | Additional RUs used | | ---------- | ------------------- | | TopK=5 | 1 | | TopK=10 | 1 | | TopK=50 | 5 | Hybrid retrieval searches over more data and incurs additional RUs. This additional cost is a factor of the size of the namespace you search and the number of non-zero sparse dimensions. Similar costs are incurred for sparse-only searches. For example, for sparse vectors with 100 non-zero sparse dimensions, approximate RUs consumed for sparse vectors are as follows: | Records per namespace | RUs for sparse vector retrieval | | --------------------- | ------------------------------- | | 10,000,000 | 17 RUs | | 100,000,000 | 54 RUs | #### List List has a fixed cost of 1 RU per call, with an additional 1 RU per paginated call. ### Write units Write operations consume write units (WUs). Write units measure the storage and compute resources used to persist a record, make it available for querying, and update the index to reflect its addition. The following operations consume WUs: * [Upsert](/reference/api/2024-10/data-plane/upsert) * [Update](/reference/api/2024-10/data-plane/update) * [Delete](/reference/api/2024-10/data-plane/delete) #### Upsert When upserting a new record, the number of WUs used is proportional to the total size of the record, with a minimum of 1 WU. When an upsert modifies an existing record, the number of WUs used is proportional to the total size of both the new and previous versions of the record. The following table contains the WU cost of an upsert request at different batch sizes and record dimensionality, assuming all records are new and have an average metadata size around 500 bytes: | Records per batch | Dimension=384 | Dimension=768 | Dimension=1536 | | ----------------- | ------------- | ------------- | -------------- | | 1 | 3 WUs | 4 WUs | 7 WUs | | 10 | 30 WUs | 40 WUs | 70 WUs | | 100 | 300 WUs | 400 WUs | 700 WUs | #### Update The number of WUs used when updating a record is proportional to the total size of both the new and previous versions of the record, with a minimum of 1 WU. The following table contains the WU cost of an update request at different dimensionalities and metadata sizes, with WUs based on the new and previous metadata size: | Dimension | Previous metada size | New metadata size | WUs | | --------- | -------------------- | ----------------- | ------ | | 768 | 400 bytes | 500 bytes | 8 WUs | | 1536 | 400 bytes | 500 bytes | 14 WUs | | 1536 | 4000 bytes | 2000 bytes | 19 WUs | #### Delete The number of WUs used by a delete request is proportional to the total size of records it deletes, with a minimum of 1 WU. The following table contains the WU cost of a delete request at different batch sizes and record dimensionality, assuming an average metadata size around 500 bytes: | Records per batch | Dimension=384 | Dimension=768 | Dimension=1536 | | ----------------- | ------------- | ------------- | -------------- | | 1 | 3 WUs | 4 WUs | 7 WUs | | 10 | 30 WUs | 40 WUs | 70 WUs | | 100 | 300 WUs | 400 WUs | 700 WUs | Specifying a non-existent ID or adding the same ID more than once does not increase WU use. Deleting all records in a namespace using the `deleteAll` flag always consumes 1 WU. ### Imports and storage The size of an index is defined as the total size of its vectors across all namespaces. The size of a single vector is defined as the sum of three components: * ID size * Embedding size (equal to 4 times the vector's dimensions) * Total metadata size (equal to the total size of all metadata fields) The following table demonstrates a typical index size at different vector counts and dimensionality: | Records per namespace | Dimension=384 | Dimension=768 | Dimension=1536 | | --------------------- | ------------- | ------------- | -------------- | | 100,000 | 0.20 GB | 0.35 GB | 0.66 GB | | 1,000,000 | 2.00 GB | 3.50 GB | 6.60 GB | | 10,000,000 | 20.00 GB | 35.00 GB | 66.00 GB | The cost of an import is based on the size of the records read, whether the records were imported successfully or not. If the import operation fails (e.g., after encountering a vector of the wrong dimension in an import with `on_error="abort"`), you will still be charged for the records read. However, if the import fails because of an internal system error, you will not incur charges. In this case, the import will return the error message `"We were unable to process your request. If the problem persists, please contact us at https://support.pinecone.io"`. ### Monitoring usage * [Index-level monitoring](/guides/manage-cost/monitor-your-usage#monitor-index-level-usage): In the Pinecone console, you can track usage and performance metrics for each index. * [Operation-level monitoring](/guides/manage-cost/monitor-your-usage#monitor-operation-level-usage): The response to read operations like `query`, `fetch`, and `list` include the number of read units consumed. ## Pod-based indexes ### Cost calculation For each [pod-based index](/guides/index-data/indexing-overview#pod-based-indexes), billing is determined by the per-minute price per pod and the number of pods the index uses, regardless of index activity. The per-minute price varies by pod type, pod size, account plan, and cloud region. Total cost depends on a combination of factors: * **Pod type.** Each [pod type](/guides/indexes/pods/understanding-pod-based-indexes#pod-types) has different per-minute pricing. * **Number of pods.** This includes replicas, which duplicate pods. * **Pod size.** Larger pod sizes have proportionally higher costs per minute. * **Total pod-minutes.** This includes the total time each pod is running, starting at pod creation and rounded up to 15-minute increments. * **Cloud provider.** The cost per pod-type and pod-minute varies depending on the cloud provider you choose for your project. * **Collection storage.** Collections incur costs per GB of data per minute in storage, rounded up to 15-minute increments. * **Plan.** The free plan incurs no costs; the Standard or Enterprise plans incur different costs per pod-type, pod-minute, cloud provider, and collection storage. The following equation calculates the total costs accrued over time: ``` (Number of pods) * (pod size) * (number of replicas) * (minutes pod exists) * (pod price per minute) + (collection storage in GB) * (collection storage time in minutes) * (collection storage price per GB per minute) ``` To see a calculation of your current usage and costs, go to [**Settings > Usage**](https://app.pinecone.io/organizations/-/settings/usage) in the Pinecone console. ### Pricing For pod-based index pricing rates, see our [pricing page](https://www.pinecone.io/pricing/). ### Example While our pricing page lists rates on an hourly basis for ease of comparison, this example lists prices per minute, as this is how Pinecone calculates billing. An example application has the following requirements: * 1,000,000 vectors with 1536 dimensions * 150 queries per second with `top_k` = 10 * Deployment in an EU region * Ability to store 1GB of inactive vectors [Based on these requirements](/guides/indexes/pods/choose-a-pod-type-and-size), the organization chooses to configure the project to use the Standard billing plan to host one `p1.x2` pod with three replicas and a collection containing 1 GB of data. This project runs continuously for the month of January on the Standard plan. The components of the total cost for this example are given in Table 1 below: **Table 1: Example billing components** | Billing component | Value | | ----------------------------- | ------------ | | Number of pods | 1 | | Number of replicas | 3 | | Pod size | x2 | | Total pod count | 6 | | Minutes in January | 44,640 | | Pod-minutes (pods \* minutes) | 267,840 | | Pod price per minute | \$0.0012 | | Collection storage | 1 GB | | Collection storage minutes | 44,640 | | Price per storage minute | \$0.00000056 | The invoice for this example is given in Table 2 below: **Table 2: Example invoice** | Product | Quantity | Price per unit | Charge | | ------------- | -------- | -------------- | -------- | | Collections | 44,640 | \$0.00000056 | \$0.025 | | P2 Pods (AWS) | 0 | | \$0.00 | | P2 Pods (GCP) | 0 | | \$0.00 | | S1 Pods | 0 | | \$0.00 | | P1 Pods | 267,840 | \$0.0012 | \$514.29 | Amount due \$514.54 ## Pinecone Inference Pinecone Inference is a service that gives you access to [embedding models](/guides/index-data/create-an-index#embedding-models) and [reranking models](/guides/search/rerank-results#reranking-models) hosted on Pinecone's infrastructure. ### Embed Embedding costs are determined by how many [tokens](https://www.pinecone.io/learn/tokenization/) are in a request. In general, the more words contained in your passage or query, the more tokens you generate. For example, if you generate embeddings for the query, "What is the maximum diameter of a red pine?", Pinecone Inference generates 10 tokens, then converts them into an embedding. If the price per token for your billing plan is \$.08 per million tokens, then this API call costs \$.00001. To learn more about tokenization, see [Choosing an embedding model](https://www.pinecone.io/learn/series/rag/embedding-models-rundown/). For up-to-date embed pricing, see [Pricing](https://www.pinecone.io/pricing/). Embedding requests returns the total tokens generated. You can use this information to [monitor and manage embedding costs](/guides/manage-cost/monitor-your-usage#embedding-tokens). ### Rerank Reranking costs are determined by the number of requests to the [reranking model](/guides/search/rerank-results#reranking-models). For up-to-date rerank pricing, see [Pricing](https://www.pinecone.io/pricing/). ## Pinecone Assistant The cost of using Pinecone Assistant is determined by the following factors: | Invoice line item | Description | | -------------------------------------- | ------------------------------------------------------------------------------------------------ | | Assistants Context Tokens Processed | Number of tokens processed for [context retrieval](/guides/assistant/context-snippets-overview). | | Assistants Evaluation Tokens Out | Number of tokens used to calculate [evaluation metrics](/guides/assistant/evaluation-overview). | | Assistants Evaluation Tokens Processed | Number of tokens used to prompt [evaluation metrics](/guides/assistant/evaluation-overview). | | Assistants Hourly Count | Number of hours the assistant is available. | | Assistants Input Tokens | Number of [tokens processed](/guides/assistant/pricing-and-limits#token-usage) by the assistant. | | Assistants Output Tokens | Number of [tokens output](/guides/assistant/pricing-and-limits#token-usage) by the assistant. | | Assistants Total Storage GB/Hours | Total size of [files stored in the assistant](/guides/assistant/files-overview) per month. | See [Pricing](https://www.pinecone.io/pricing/) for up-to-date pricing information. ## Cost controls Pinecone offers tools to help you understand and control your costs. * **Monitoring usage.** You can use the [usage dashboard](/guides/manage-cost/monitor-your-usage) in the Pinecone console to monitor your Pinecone usage and costs as these accrue. * **Pod limits.** For pod-based indexes, project owners can [set limits for the total number of pods](/reference/api/database-limits#pods-per-project) across all indexes in the project. The default pod limit is 5. ## See also * [Monitoring usage](/guides/manage-cost/monitor-your-usage) * [Managing cost](/guides/manage-cost/manage-cost) * [Pricing](https://www.pinecone.io/pricing/) # Back up an index Source: https://docs.pinecone.io/guides/manage-data/back-up-an-index This page describes how to create a static copy of a serverless index, also known as a [backup](/guides/manage-data/backups-overview). Serverless index [backups](/guides/manage-data/backups-overview) are in [public preview](/release-notes/feature-availability) and available only on [Standard and Enterprise plans](https://www.pinecone.io/pricing/). ## Create a backup You can [create a backup from a serverless index](/reference/api/2025-04/control-plane/create_backup), as in the following example: ```python Python from pinecone import Pinecone pc = Pinecone(api_key="YOUR_API_KEY") backup = pc.create_backup( index_name="docs-example", backup_name="example-backup", description="Monthly backup of production index" ) print(backup) ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone'; const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }) const backup = await pc.createBackup({ indexName: 'docs-example', name: 'example-backup', description: 'Monthly backup of production index', }); console.log(backup); ``` ```java Java import io.pinecone.clients.Pinecone; import org.openapitools.db_control.client.ApiException; import org.openapitools.db_control.client.model.*; public class CreateBackup { public static void main(String[] args) throws ApiException { Pinecone pc = new Pinecone.Builder("YOUR_API_KEY").build(); String indexName = "docs-example"; String backupName = "example-backup"; String backupDescription = "Monthly backup of production index"; BackupModel backup = pc.createBackup(indexName,backupName, backupDescription); System.out.println(backup); } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("PINECONE_API_KEY"); var backup = await pinecone.Backups.BackupIndexAsync( "docs-example", new BackupIndexRequest { Name = "example-backup", Description = "Monthly backup of production index" } ); Console.WriteLine(backup); ``` ```bash curl PINECONE_API_KEY="YOUR_API_KEY" INDEX_NAME="docs-example" curl "https://api.pinecone.io/indexes/$INDEX_NAME/backups" \ -H "Api-Key: $PINECONE_API_KEY" \ -H 'accept: application/json' \ -H 'Content-Type: application/json' \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "name": "example-backup", "description": "Monthly backup of production index" }' ``` The example returns a response like the following: ```python Python {'backup_id': '8c85e612-ed1c-4f97-9f8c-8194e07bcf71', 'cloud': 'aws', 'created_at': '2025-05-15T00:52:10.809305882Z', 'description': 'Monthly backup of production index', 'dimension': 1024, 'name': 'example-backup', 'namespace_count': 3, 'record_count': 98, 'region': 'us-east-1', 'size_bytes': 1069169, 'source_index_id': 'f73b36c9-faf5-4a2c-b1d6-4013d8b1cc74', 'source_index_name': 'docs-example', 'status': 'Ready', 'tags': {}} ``` ```javascript JavaScript { backupId: '8c85e612-ed1c-4f97-9f8c-8194e07bcf71', sourceIndexName: 'docs-example', sourceIndexId: 'f73b36c9-faf5-4a2c-b1d6-4013d8b1cc74', name: 'example-backup', description: 'Monthly backup of production index', status: 'Ready', cloud: 'aws', region: 'us-east-1', dimension: 1024, metric: undefined, recordCount: 98, namespaceCount: 3, sizeBytes: 1069169, tags: {}, createdAt: '2025-05-14T16:37:25.625540Z' } ``` ```java Java class BackupModel { backupId: 0d75b99f-be61-4a93-905e-77201286c02e sourceIndexName: docs-example sourceIndexId: f73b36c9-faf5-4a2c-b1d6-4013d8b1cc74 name: example-backup description: Monthly backup of production index status: Initializing cloud: aws region: us-east-1 dimension: null metric: null recordCount: null namespaceCount: null sizeBytes: null tags: {} createdAt: 2025-05-16T19:42:23.804787550Z additionalProperties: null } ``` ```csharp C# { "backup_id": "8c85e612-ed1c-4f97-9f8c-8194e07bcf71", "source_index_name": "docs-example", "source_index_id": "f73b36c9-faf5-4a2c-b1d6-4013d8b1cc74", "name": "example-backup", "description": "Monthly backup of production index", "status": "Ready", "cloud": "aws", "region": "us-east-1", "tags": {}, "created_at": "2025-05-15T00:52:10.809305882Z" } ``` ```json curl { "backup_id":"8c85e612-ed1c-4f97-9f8c-8194e07bcf71", "source_index_id":"f73b36c9-faf5-4a2c-b1d6-4013d8b1cc74", "source_index_name":"docs-example", "tags":{}, "name":"example-backup", "description":"Monthly backup of production index", "status":"Ready", "cloud":"aws", "region":"us-east-1", "dimension":1024, "record_count":96, "namespace_count":3, "size_bytes":1069169, "created_at":"2025-05-14T16:37:25.625540Z" } ``` You can create a backup using the [Pinecone console](https://app.pinecone.io/organizations/-/projects/-/backups). ## View all backups for an index You can [view all backups for an index](/reference/api/2025-04/control-plane/list_index_backups) as follows. Up to 100 backups are returned at a time by default, in sorted order (bitwise “C” collation). If the `limit` parameter is set, up to that number of backups are returned instead. Whenever there are additional backups to return, the response also includes a `pagination_token` that you can use to get the next batch of backups. When the response does not include a `pagination_token`, there are no more backups to return. ```python Python from pinecone import Pinecone pc = Pinecone(api_key="YOUR_API_KEY") index_backups = pc.list_backups(index_name="docs-example") print(index_backups) ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone'; const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }) const indexBackups = await pc.listBackups({ indexName: 'docs-example' }); console.log(indexBackups); ``` ```java Java import io.pinecone.clients.Pinecone; import org.openapitools.db_control.client.ApiException; import org.openapitools.db_control.client.model.*; public class CreateBackup { public static void main(String[] args) throws ApiException { Pinecone pc = new Pinecone.Builder("YOUR_API_KEY").build(); String indexName = "docs-example"; BackupList indexBackupList = pc.listIndexBackups(indexName); System.out.println(indexBackupList); } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); var indexBackups = await pinecone.Backups.ListByIndexAsync( "docs-example", new ListBackupsByIndexRequest()); Console.WriteLine(indexBackups); ``` ```bash curl PINECONE_API_KEY="YOUR_API_KEY" INDEX_NAME="docs-example" curl -X GET "https://api.pinecone.io/indexes/$INDEX_NAME/backups" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" \ -H "accept: application/json" ``` The example returns a response like the following: ```python Python [{ "backup_id": "8c85e612-ed1c-4f97-9f8c-8194e07bcf71", "source_index_name": "docs-example", "source_index_id": "f73b36c9-faf5-4a2c-b1d6-4013d8b1cc74", "status": "Ready", "cloud": "aws", "region": "us-east-1", "tags": {}, "name": "example-backup", "description": "Monthly backup of production index", "dimension": 1024, "record_count": 98, "namespace_count": 3, "size_bytes": 1069169, "created_at": "2025-05-15T00:52:10.809305882Z" }] ``` ```javascript JavaScript { data: [ { backupId: '8c85e612-ed1c-4f97-9f8c-8194e07bcf71', sourceIndexName: 'docs-example', sourceIndexId: 'f73b36c9-faf5-4a2c-b1d6-4013d8b1cc74', name: 'example-backup', description: 'Monthly backup of production index', status: 'Ready', cloud: 'aws', region: 'us-east-1', dimension: 1024, metric: undefined, recordCount: 98, namespaceCount: 3, sizeBytes: 1069169, tags: {}, createdAt: '2025-05-14T16:37:25.625540Z' } ], pagination: undefined } ``` ```java Java class BackupList { data: [class BackupModel { backupId: 8c85e612-ed1c-4f97-9f8c-8194e07bcf71 sourceIndexName: docs-example sourceIndexId: f73b36c9-faf5-4a2c-b1d6-4013d8b1cc74 name: example-backup description: Monthly backup of production index status: Initializing cloud: aws region: us-east-1 dimension: null metric: null recordCount: null namespaceCount: null sizeBytes: null tags: {} createdAt: 2025-05-16T19:46:26.248428Z additionalProperties: null }] pagination: null additionalProperties: null } ``` ```csharp C# { "data": [ { "backup_id":"9947520e-d5a1-4418-a78d-9f464c9969da", "source_index_id":"8433941a-dae7-43b5-ac2c-d3dab4a56b2b", "source_index_name":"docs-example", "tags":{}, "name":"example-backup", "description":"Monthly backup of production index", "status":"Pending", "cloud":"aws", "region":"us-east-1", "dimension":1024, "record_count":98, "namespace_count":3, "size_bytes":1069169, "created_at":"2025-03-11T18:29:50.549505Z" } ] } ``` ```json curl { "data": [ { "backup_id":"9947520e-d5a1-4418-a78d-9f464c9969da", "source_index_id":"8433941a-dae7-43b5-ac2c-d3dab4a56b2b", "source_index_name":"docs-example", "tags":{}, "name":"example-backup", "description":"Monthly backup of production index", "status":"Pending", "cloud":"aws", "region":"us-east-1", "dimension":1024, "record_count":98, "namespace_count":3, "size_bytes":1069169, "created_at":"2025-03-11T18:29:50.549505Z" } ], "pagination":null } ``` You can view the backups for a specific index from either the [Backups](https://app.pinecone.io/organizations/-/projects/-/backups) tab or the [Indexes](https://app.pinecone.io/organizations/-/projects/-/indexes) tab in the Pinecone console. ## View all backups in a project You can [view backups for all indexes in a project](/reference/api/2025-04/control-plane/list_project_backups) as follows. Up to 100 backups are returned at a time by default, in sorted order (bitwise “C” collation). If the `limit` parameter is set, up to that number of backups are returned instead. Whenever there are additional backups to return, the response also includes a `pagination_token` that you can use to get the next batch of backups. When the response does not include a `pagination_token`, there are no more backups to return. ```python Python from pinecone import Pinecone pc = Pinecone(api_key="YOUR_API_KEY") project_backups = pc.list_backups() print(project_backups) ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone'; const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }) const projectBackups = await pc.listBackups(); console.log(projectBackups); ``` ```java Java import io.pinecone.clients.Pinecone; import org.openapitools.db_control.client.ApiException; import org.openapitools.db_control.client.model.*; public class CreateBackup { public static void main(String[] args) throws ApiException { Pinecone pc = new Pinecone.Builder("YOUR_API_KEY").build(); String indexName = "docs-example"; BackupList projectBackupList = pc.listProjectBackups(); System.out.println(projectBackupList); } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("PINECONE_API_KEY"); var backups = await pinecone.Backups.ListAsync(); Console.WriteLine(backups); ``` ```bash curl PINECONE_API_KEY="YOUR_API_KEY" curl -X GET "https://api.pinecone.io/backups" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" \ -H "accept: application/json" ``` The example returns a response like the following: ```python Python [{ "backup_id": "8c85e612-ed1c-4f97-9f8c-8194e07bcf71", "source_index_name": "docs-example", "source_index_id": "f73b36c9-faf5-4a2c-b1d6-4013d8b1cc74", "status": "Ready", "cloud": "aws", "region": "us-east-1", "tags": {}, "name": "example-backup", "description": "Monthly backup of production index", "dimension": 1024, "record_count": 98, "namespace_count": 3, "size_bytes": 1069169, "created_at": "2025-05-15T20:26:21.248515Z" }, { "backup_id": "95707edb-e482-49cf-b5a5-312219a51a97", "source_index_name": "docs-example2", "source_index_id": "b49f27d1-1bf3-49c6-82b5-4ae46f00f0e6", "status": "Ready", "cloud": "aws", "region": "us-east-1", "tags": {}, "name": "example-backup2", "description": "Monthly backup of production index", "dimension": 1024, "record_count": 97, "namespace_count": 2, "size_bytes": 1069169, "created_at": "2025-05-15T00:52:10.809354Z" }, { "backup_id": "8c85e612-ed1c-4f97-9f8c-8194e07bcf71", "source_index_name": "docs-example3", "source_index_id": "f73b36c9-faf5-4a2c-b1d6-4013d8b1cc74", "status": "Ready", "cloud": "aws", "region": "us-east-1", "tags": {}, "name": "example-backup3", "description": "Monthly backup of production index", "dimension": 1024, "record_count": 98, "namespace_count": 3, "size_bytes": 1069169, "created_at": "2025-05-14T16:37:25.625540Z" }] ``` ```javascript JavaScript { data: [ { backupId: 'e12269b0-a29b-4af0-9729-c7771dec03e3', sourceIndexName: 'docs-example', sourceIndexId: 'bcb5b3c9-903e-4cb6-8b37-a6072aeb874f', name: 'example-backup', description: undefined, status: 'Ready', cloud: 'aws', region: 'us-east-1', dimension: 0, metric: undefined, recordCount: 96, namespaceCount: 1, sizeBytes: 86393, tags: undefined, createdAt: '2025-05-14T17:00:45.803146Z' }, { backupId: 'd686451d-1ede-4004-9f72-7d22cc799b6e', sourceIndexName: 'docs-example2', sourceIndexId: 'b49f27d1-1bf3-49c6-82b5-4ae46f00f0e6', name: 'example-backup2', description: undefined, status: 'Ready', cloud: 'aws', region: 'us-east-1', dimension: 1024, metric: undefined, recordCount: 50, namespaceCount: 1, sizeBytes: 545171, tags: undefined, createdAt: '2025-05-14T17:00:34.814371Z' }, { backupId: '8c85e612-ed1c-4f97-9f8c-8194e07bcf71', sourceIndexName: 'docs-example3', sourceIndexId: 'f73b36c9-faf5-4a2c-b1d6-4013d8b1cc74', name: 'example-backup3', description: 'Monthly backup of production index', status: 'Ready', cloud: 'aws', region: 'us-east-1', dimension: 1024, metric: undefined, recordCount: 98, namespaceCount: 3, sizeBytes: 1069169, tags: {}, createdAt: '2025-05-14T16:37:25.625540Z' } ], pagination: undefined } ``` ```java Java class BackupList { data: [class BackupModel { backupId: 13761d20-7a0b-4778-ac27-36dd91c4be43 sourceIndexName: example-dense-index sourceIndexId: f73b36c9-faf5-4a2c-b1d6-4013d8b1cc74 name: example-backup-java2 description: Monthly backup of production index status: Initializing cloud: aws region: us-east-1 dimension: null metric: null recordCount: null namespaceCount: null sizeBytes: null tags: {} createdAt: 2025-05-16T19:46:26.248428Z additionalProperties: null }, class BackupModel { backupId: 0d75b99f-be61-4a93-905e-77201286c02e sourceIndexName: example-dense-index sourceIndexId: f73b36c9-faf5-4a2c-b1d6-4013d8b1cc74 name: example-backup-java description: Monthly backup of production index status: Initializing cloud: aws region: us-east-1 dimension: null metric: null recordCount: null namespaceCount: null sizeBytes: null tags: {} createdAt: 2025-05-16T19:42:23.804820Z additionalProperties: null }, class BackupModel { backupId: bf2cda5d-b233-4a0a-aae9-b592780ad3ff sourceIndexName: example-sparse-index sourceIndexId: bcb5b3c9-903e-4cb6-8b37-a6072aeb874f name: example-sparse-python description: Monthly backup of production index status: Ready cloud: aws region: us-east-1 dimension: 0 metric: null recordCount: 96 namespaceCount: 1 sizeBytes: 86393 tags: {} createdAt: 2025-05-16T18:01:51.531129Z additionalProperties: null }] pagination: null additionalProperties: null } ``` ```csharp C# { "data": [ { "backup_id": "95707edb-e482-49cf-b5a5-312219a51a97", "source_index_name": "docs-example", "source_index_id": "f73b36c9-faf5-4a2c-b1d6-4013d8b1cc74", "name": "example-backup", "description": "Monthly backup of production index", "status": "Ready", "cloud": "aws", "region": "us-east-1", "dimension": 1024, "record_count": 97, "namespace_count": 2, "size_bytes": 1069169, "tags": {}, "created_at": "2025-05-15T00:52:10.809354Z" }, { "backup_id": "e12269b0-a29b-4af0-9729-c7771dec03e3", "source_index_name": "docs-example2", "source_index_id": "bcb5b3c9-903e-4cb6-8b37-a6072aeb874f", "name": "example-backup2", "status": "Ready", "cloud": "aws", "region": "us-east-1", "dimension": 0, "record_count": 96, "namespace_count": 1, "size_bytes": 86393, "created_at": "2025-05-14T17:00:45.803146Z" }, { "backup_id": "d686451d-1ede-4004-9f72-7d22cc799b6e", "source_index_name": "docs-example3", "source_index_id": "b49f27d1-1bf3-49c6-82b5-4ae46f00f0e6", "name": "example-backup3", "status": "Ready", "cloud": "aws", "region": "us-east-1", "dimension": 1024, "record_count": 50, "namespace_count": 1, "size_bytes": 545171, "created_at": "2025-05-14T17:00:34.814371Z" } ] } ``` ```json curl { "data": [ { "backup_id": "e12269b0-a29b-4af0-9729-c7771dec03e3", "source_index_id": "bcb5b3c9-903e-4cb6-8b37-a6072aeb874f", "source_index_name": "docs-example", "tags": null, "name": "example-backup", "description": null, "status": "Ready", "cloud": "aws", "region": "us-east-1", "dimension": 0, "record_count": 96, "namespace_count": 1, "size_bytes": 86393, "created_at": "2025-05-14T17:00:45.803146Z" }, { "backup_id": "d686451d-1ede-4004-9f72-7d22cc799b6e", "source_index_id": "b49f27d1-1bf3-49c6-82b5-4ae46f00f0e6", "source_index_name": "docs-example2", "tags": null, "name": "example-backup2", "description": null, "status": "Ready", "cloud": "aws", "region": "us-east-1", "dimension": 1024, "record_count": 50, "namespace_count": 1, "size_bytes": 545171, "created_at": "2025-05-14T17:00:34.814371Z" }, { "backup_id": "8c85e612-ed1c-4f97-9f8c-8194e07bcf71", "source_index_id": "f73b36c9-faf5-4a2c-b1d6-4013d8b1cc74", "source_index_name": "docs-example3", "tags": {}, "name": "example-backup3", "description": "Monthly backup of production index", "status": "Ready", "cloud": "aws", "region": "us-east-1", "dimension": 1024, "record_count": 98, "namespace_count": 3, "size_bytes": 1069169, "created_at": "2025-05-14T16:37:25.625540Z" } ], "pagination": null } ``` You can view all backups in a project using the [Pinecone console](https://app.pinecone.io/organizations/-/projects-/backups). ## View backup details You can [view the details of a backup](/reference/api/2025-04/control-plane/describe_backup), as in the following example: ```python Python from pinecone import Pinecone pc = Pinecone(api_key="YOUR_API_KEY") backup = pc.describe_backup(backup_id="8c85e612-ed1c-4f97-9f8c-8194e07bcf71") print(backup) ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone'; const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }) const backupDesc = await pc.describeBackup('8c85e612-ed1c-4f97-9f8c-8194e07bcf71'); console.log(backupDesc); ``` ```java Java import io.pinecone.clients.Pinecone; import org.openapitools.db_control.client.ApiException; import org.openapitools.db_control.client.model.*; public class CreateBackup { public static void main(String[] args) throws ApiException { Pinecone pc = new Pinecone.Builder("YOUR_API_KEY").build(); BackupModel backupModel = pc.describeBackup("8c85e612-ed1c-4f97-9f8c-8194e07bcf71"); System.out.println(backupModel); } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("PINECONE_API_KEY"); var backup = await pinecone.Backups.GetAsync("8c85e612-ed1c-4f97-9f8c-8194e07bcf71"); Console.WriteLine(backup); ``` ```bash curl PINECONE_API_KEY="YOUR_API_KEY" BACKUP_ID="8c85e612-ed1c-4f97-9f8c-8194e07bcf71" curl -X GET "https://api.pinecone.io/backups/$BACKUP_ID" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" \ -H "accept: application/json" ``` The example returns a response like the following: ```python Python {'backup_id': '8c85e612-ed1c-4f97-9f8c-8194e07bcf71', 'cloud': 'aws', 'created_at': '2025-05-15T00:52:10.809354Z', 'description': 'Monthly backup of production index', 'dimension': 1024, 'name': 'example-backup', 'namespace_count': 3, 'record_count': 98, 'region': 'us-east-1', 'size_bytes': 1069169, 'source_index_id': 'f73b36c9-faf5-4a2c-b1d6-4013d8b1cc74', 'source_index_name': 'docs-example', 'status': 'Ready', 'tags': {}} ``` ```javascript JavaScript { backupId: '8c85e612-ed1c-4f97-9f8c-8194e07bcf71', sourceIndexName: 'docs-example', sourceIndexId: 'f73b36c9-faf5-4a2c-b1d6-4013d8b1cc74', name: 'example-backup', description: 'Monthly backup of production index', status: 'Ready', cloud: 'aws', region: 'us-east-1', dimension: 1024, metric: undefined, recordCount: 98, namespaceCount: 3, sizeBytes: 1069169, tags: {}, createdAt: '2025-05-14T16:37:25.625540Z' } ``` ```java Java class BackupList { data: [class BackupModel { backupId: 95707edb-e482-49cf-b5a5-312219a51a97 sourceIndexName: docs-example sourceIndexId: f73b36c9-faf5-4a2c-b1d6-4013d8b1cc74 name: example-backup description: Monthly backup of production index status: Initializing cloud: aws region: us-east-1 dimension: null metric: null recordCount: null namespaceCount: null sizeBytes: null tags: {} createdAt: 2025-05-16T19:46:26.248428Z additionalProperties: null }] pagination: null additionalProperties: null } ``` ```csharp C# { "backup_id": "95707edb-e482-49cf-b5a5-312219a51a97", "source_index_name": "docs-example", "source_index_id": "f73b36c9-faf5-4a2c-b1d6-4013d8b1cc74", "name": "example-backup", "description": "Monthly backup of production index", "status": "Ready", "cloud": "aws", "region": "us-east-1", "dimension": 1024, "record_count": 97, "namespace_count": 2, "size_bytes": 1069169, "tags": {}, "created_at": "2025-05-15T00:52:10.809354Z" } ``` ```json curl { "backup_id":"8c85e612-ed1c-4f97-9f8c-8194e07bcf71", "source_index_id":"f73b36c9-faf5-4a2c-b1d6-4013d8b1cc74", "source_index_name":"docs-example", "tags":{}, "name":"example-backup", "description":"Monthly backup of production index", "status":"Ready", "cloud":"aws", "region":"us-east-1", "dimension":1024, "record_count":98, "namespace_count":3, "size_bytes":1069169, "created_at":"2025-03-11T18:29:50.549505Z" } ``` You can view backup details using the [Pinecone console](https://app.pinecone.io/organizations/-/projects-/backups). ## Delete a backup You can [delete a backup](/reference/api/2025-04/control-plane/delete_backup), as in the following example: ```python Python from pinecone import Pinecone pc = Pinecone(api_key="YOUR_API_KEY") pc.delete_backup(backup_id="9947520e-d5a1-4418-a78d-9f464c9969da") ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone'; const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }) await pc.deleteBackup('9947520e-d5a1-4418-a78d-9f464c9969da'); ``` ```java Java import io.pinecone.clients.Pinecone; import org.openapitools.db_control.client.ApiException; import org.openapitools.db_control.client.model.*; public class CreateBackup { public static void main(String[] args) throws ApiException { Pinecone pc = new Pinecone.Builder("YOUR_API_KEY").build(); pc.deleteBackup("9947520e-d5a1-4418-a78d-9f464c9969da"); } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("PINECONE_API_KEY"); await pinecone.Backups.DeleteAsync("9947520e-d5a1-4418-a78d-9f464c9969da"); ``` ```bash curl PINECONE_API_KEY="YOUR_API_KEY" BACKUP_ID="9947520e-d5a1-4418-a78d-9f464c9969da" curl -X DELETE "https://api.pinecone.io/backups/$BACKUP_ID" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" ``` You can delete a backup using the [Pinecone console](https://app.pinecone.io/organizations/-/projects-/backups). # Backups overview Source: https://docs.pinecone.io/guides/manage-data/backups-overview A backup is a static copy of a serverless [index](/guides/index-data/indexing-overview) that only consumes storage. It is a non-queryable representation of a set of records. You can [create a backup](/guides/manage-data/back-up-an-index) of a serverless index, and you can [create a new serverless index from a backup](/guides/manage-data/restore-an-index). This allows you to restore the index with the same or different configurations. Once a backup is created, it cannot be moved to a different project. Serverless index [backups](/guides/manage-data/backups-overview) are in [public preview](/release-notes/feature-availability) and available only on [Standard and Enterprise plans](https://www.pinecone.io/pricing/). ## Use cases Creating a backup is useful when performing tasks like the following: * Protecting an index from manual or system failures. * Temporarily shutting down an index. * Copying the data from one index into a different index. * Making a backup of your index. * Experimenting with different index configurations. ## Performance Backup and restore times depend upon the size of the index and number of namespaces: * For less than 1M vectors in a namespace, backups and restores take approximately 10 minutes. * For 100,000,000 vectors, backups and restores can take up to 5 hours. ## Quotas | Metric | Starter plan | Standard plan | Enterprise plan | | :-------------------- | :----------- | :------------ | :-------------- | | Backups per project | N/A | 500 | 1000 | | Namespaces per backup | N/A | 2000 | 2000 | ## Limitations Backup limitations are as follows: * Backups are stored in the same region as the source index. * You can only restore an index to the same region as the source index. * Backups only include vectors that were in the index at least 15 minutes prior to the backup time. This means that if a vector was inserted into an index and a backup was immediately taken after, the recently inserted vector may not be backed up. More specifically, if a backup is created only a few minutes after the source index was created, the backup may have 0 vectors. * You can only perform operations on backups in the current Pinecone project. ## Pricing See [Pricing](https://www.pinecone.io/pricing/) for up-to-date pricing information. # Delete records Source: https://docs.pinecone.io/guides/manage-data/delete-data This page shows you how to use the [`delete`](/reference/api/2024-10/data-plane/delete) endpoint to remove records from an index [namespace](/guides/index-data/indexing-overview#namespaces). ## Delete records by ID Since Pinecone records can always be efficiently accessed using their ID, deleting by ID is the most efficient way to remove specific records. To remove records from the default namespace, specify `"__default__"` as the namespace in your request. ```Python Python # pip install "pinecone[grpc]" from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key="YOUR_API_KEY") # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index index = pc.Index(host="INDEX_HOST") index.delete(ids=["id-1", "id-2"], namespace='example-namespace') ``` ```JavaScript JavaScript import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: "YOUR_API_KEY" }) // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index const index = pc.index("INDEX_NAME", "INDEX_HOST") const ns = index.namespace('example-namespace') // Delete one record by ID. await ns.deleteOne('id-1'); // Delete more than one record by ID. await ns.deleteMany(['id-2', 'id-3']); ``` ```java Java import io.pinecone.clients.Index; import io.pinecone.configs.PineconeConfig; import io.pinecone.configs.PineconeConnection; import java.util.Arrays; import java.util.List; public class DeleteExample { public static void main(String[] args) { PineconeConfig config = new PineconeConfig("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index config.setHost("INDEX_HOST"); PineconeConnection connection = new PineconeConnection(config); Index index = new Index(connection, "INDEX_NAME"); List ids = Arrays.asList("id-1", "id-2"); index.deleteByIds(ids, "example-namespace"); } } ``` ```go Go package main import ( "context" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "INDEX_HOST", Namespace: "example-namespace"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v", err) } id1 := "id-1" id2 := "id-2" err = idxConnection.DeleteVectorsById(ctx, []string{id1, id2}) if err != nil { log.Fatalf("Failed to delete vector with ID %v: %v", id, err) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index var index = pinecone.Index(host: "INDEX_HOST"); var deleteResponse = await index.DeleteAsync(new DeleteRequest { Ids = new List { "id-1", "id-2" }, Namespace = "example-namespace", }); ``` ```bash curl # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index PINECONE_API_KEY="YOUR_API_KEY" INDEX_HOST="INDEX_HOST" curl "https://$INDEX_HOST/vectors/delete" \ -H "Api-Key: $PINECONE_API_KEY" \ -H 'Content-Type: application/json' \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "ids": [ "id-1", "id-2" ], "namespace": "example-namespace" } ' ``` ## Delete records by metadata Serverless indexes do not support deleting by metadata. You can [delete records by ID prefix](/guides/manage-data/manage-document-chunks#delete-all-records-for-a-parent-document) instead. ## Delete records in a namespace In serverless indexes, it's safe to delete an entire namespace or a large number of records at once because reads and writes do not share compute resources. There are two ways to delete records from a namespace: * [Delete all records in a namespace](#delete-all-records-in-a-namespace) * [Delete an entire namespace](#delete-an-entire-namespace) ### Delete all records in a namespace To delete all of the records in a namespace but not the namespace itself, provide a `namespace` parameter and specify the appropriate `deleteAll` parameter for your SDK. To target the default namespace, set `namespace` to `"__default__"`. ```Python Python # pip install "pinecone[grpc]" from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key="YOUR_API_KEY") # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index index = pc.Index(host="INDEX_HOST") index.delete(delete_all=True, namespace='example-namespace') ``` ```JavaScript JavaScript import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: "YOUR_API_KEY" }) // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index const index = pc.index("INDEX_NAME", "INDEX_HOST") await index.namespace('example-namespace').deleteAll(); ``` ```java Java import io.pinecone.clients.Index; import io.pinecone.configs.PineconeConfig; import io.pinecone.configs.PineconeConnection; import java.util.Arrays; import java.util.List; public class DeleteExample { public static void main(String[] args) { PineconeConfig config = new PineconeConfig("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index config.setHost("INDEX_HOST"); PineconeConnection connection = new PineconeConnection(config); Index index = new Index(connection, "INDEX_NAME"); index.deleteAll("example-namespace"); } } ``` ```go Go package main import ( "context" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "INDEX_HOST", Namespace: "example-namespace"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v", err) } err = idxConnection.DeleteAllVectorsInNamespace(ctx) if err != nil { log.Fatalf("Failed to delete all vectors in namespace %v: %v", namespace, err) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index var index = pinecone.Index(host: "INDEX_HOST"); var deleteResponse = await index.DeleteAsync(new DeleteRequest { DeleteAll = true, Namespace = "example-namespace", }); ``` ```bash curl # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index PINECONE_API_KEY="YOUR_API_KEY" INDEX_HOST="INDEX_HOST" curl "https://$INDEX_HOST/vectors/delete" \ -H "Api-Key: $PINECONE_API_KEY" \ -H 'Content-Type: application/json' \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "deleteAll": true, "namespace": "example-namespace" } ' ``` ### Delete an entire namespace To delete an entire namespace and all of its records, see [Delete a namespace](/guides/manage-data/manage-namespaces#delete-a-namespace). ## Delete an entire index To remove all records from an index, [delete the index](/guides/manage-data/manage-indexes#delete-an-index) and [recreate it](/guides/index-data/create-an-index). ## Data freshness Pinecone is eventually consistent, so there can be a slight delay before new or changed records are visible to queries. You can view index stats to [check data freshness](/guides/index-data/check-data-freshness). # Fetch records Source: https://docs.pinecone.io/guides/manage-data/fetch-data This page shows you how to [fetch records](/reference/api/2024-10/data-plane/fetch) by ID from a dense or sparse index [namespace](/guides/index-data/indexing-overview#namespaces). The returned records are complete, including all relevant vector values and metadata. You can fetch data using the [Pinecone console](https://app.pinecone.io/organizations/-/projects/-/indexes/-/browser). ## Fetch records To fetch records, specify the record IDs and the namespace. To use the default namespace, specify the record IDs and `"__default__"` as the namespace. ```Python Python from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key="YOUR_API_KEY") # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index index = pc.Index(host="INDEX_HOST") index.fetch(ids=["id-1", "id-2"], namespace="example-namespace") ``` ```JavaScript JavaScript import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: "YOUR_API_KEY" }) // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index const index = pc.index("INDEX_NAME", "INDEX_HOST") const fetchResult = await index.namespace('example-namespace').fetch(['id-1', 'id-2']); ``` ```java Java import io.pinecone.clients.Index; import io.pinecone.configs.PineconeConfig; import io.pinecone.configs.PineconeConnection; import io.pinecone.proto.FetchResponse; import java.util.Arrays; import java.util.List; public class FetchExample { public static void main(String[] args) { PineconeConfig config = new PineconeConfig("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index config.setHost("INDEX_HOST"); PineconeConnection connection = new PineconeConnection(config); Index index = new Index(connection, "INDEX_NAME"); List ids = Arrays.asList("id-1", "id-2"); FetchResponse fetchResponse = index.fetch(ids, "example-namespace"); System.out.println(fetchResponse); } } ``` ```go Go package main import ( "context" "encoding/json" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func prettifyStruct(obj interface{}) string { bytes, _ := json.MarshalIndent(obj, "", " ") return string(bytes) } func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "INDEX_HOST", Namespace: "example-namespace"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v", err) } res, err := idxConnection.FetchVectors(ctx, []string{"id-1", "id-2"}) if err != nil { log.Fatalf("Failed to fetch vectors: %v", err) } else { fmt.Printf(prettifyStruct(res)) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index var index = pinecone.Index(host: "INDEX_HOST"); var fetchResponse = await index.FetchAsync(new FetchRequest { Ids = new List { "id-1", "id-2" }, Namespace = "example-namespace" }); ``` ```bash curl # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index PINECONE_API_KEY="YOUR_API_KEY" INDEX_HOST="INDEX_HOST" curl -X GET "https://$INDEX_HOST/vectors/fetch?ids=id-1&ids=id-2&namespace=example-namespace" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" ``` The response looks like this: ```Python Python {'namespace': 'example-namespace', 'usage': {'readUnits': 1}, 'vectors': {'id-1': {'id': 'id-1', 'values': [0.568879, 0.632687092, 0.856837332, ...]}, 'id-2': {'id': 'id-2', 'values': [0.00891787093, 0.581895, 0.315718859, ...]}}} ``` ```JavaScript JavaScript {'namespace': 'example-namespace', 'usage': {'readUnits': 1}, 'records': {'id-1': {'id': 'id-1', 'values': [0.568879, 0.632687092, 0.856837332, ...]}, 'id-2': {'id': 'id-2', 'values': [0.00891787093, 0.581895, 0.315718859, ...]}}} ``` ```java Java namespace: "example-namespace" vectors { key: "id-1" value { id: "id-1" values: 0.568879 values: 0.632687092 values: 0.856837332 ... } } vectors { key: "id-2" value { id: "id-2" values: 0.00891787093 values: 0.581895 values: 0.315718859 ... } } usage { read_units: 1 } ``` ```go Go { "vectors": { "id-1": { "id": "id-1", "values": [ -0.0089730695, -0.020010853, -0.0042787646, ... ] }, "id-2": { "id": "id-2", "values": [ -0.005380766, 0.00215196, -0.014833462, ... ] } }, "usage": { "read_units": 1 } } ``` ```csharp C# { "vectors": { "id-1": { "id": "id-1", "values": [ -0.0089730695, -0.020010853, -0.0042787646, ... ], "sparseValues": null, "metadata": null }, "vec1": { "id": "id-2", "values": [ -0.005380766, 0.00215196, -0.014833462, ... ], "sparseValues": null, "metadata": null } }, "namespace": "example-namespace", "usage": { "readUnits": 1 } ``` ```json curl { "vectors": { "id-1": { "id": "id-1", "values": [0.568879, 0.632687092, 0.856837332, ...] }, "id-2": { "id": "id-2", "values": [0.00891787093, 0.581895, 0.315718859, ...] } }, "namespace": "example-namespace", "usage": {"readUnits": 1}, } ``` ## Data freshness Pinecone is eventually consistent, so there can be a slight delay before new or changed records are visible to queries. You can view index stats to [check data freshness](/guides/index-data/check-data-freshness). # List record IDs Source: https://docs.pinecone.io/guides/manage-data/list-record-ids This page shows you how to use the [`list`](/reference/api/2024-10/data-plane/list) endpoint to list the IDs of records in an index [namespace](/guides/index-data/indexing-overview#namespaces). You can list the IDs of all records in a namespace or just the records with a common ID prefix. Using `list` to get record IDs and not the associated data is a cheap and fast way to check [upserts](/guides/index-data/upsert-data). The `list` endpoint is supported only for [serverless indexes](/guides/index-data/indexing-overview#serverless-indexes). ## List the IDs of all records in a namespace To list the IDs of all records in the namespace of a serverless index, pass only the `namespace` parameter: ```python Python from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key='YOUR_API_KEY') # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index index = pc.Index(host="INDEX_HOST") for ids in index.list(namespace='example-namespace'): print(ids) # Response: # ['doc1#chunk1', 'doc1#chunk2', 'doc1#chunk3'] ``` ```js JavaScript import { Pinecone } from '@pinecone-database/pinecone'; const pc = new Pinecone(); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index const index = pc.index("INDEX_NAME", "INDEX_HOST").namespace("example-namespace"); const results = await index.listPaginated(); console.log(results); // { // vectors: [ // { id: 'doc1#01' }, { id: 'doc1#02' }, { id: 'doc1#03' }, // { id: 'doc1#04' }, { id: 'doc1#05' }, { id: 'doc1#06' }, // { id: 'doc1#07' }, { id: 'doc1#08' }, { id: 'doc1#09' }, // ... // ], // pagination: { // next: 'eyJza2lwX3Bhc3QiOiJwcmVUZXN0LS04MCIsInByZWZpeCI6InByZVRlc3QifQ==' // }, // namespace: 'example-namespace', // usage: { readUnits: 1 } // } // Fetch the next page of results await index.listPaginated({ prefix: 'doc1#', paginationToken: results.pagination.next}); ``` ```java Java import io.pinecone.clients.Index; import io.pinecone.configs.PineconeConfig; import io.pinecone.configs.PineconeConnection; import io.pinecone.proto.ListResponse; public class ListExample { public static void main(String[] args) { PineconeConfig config = new PineconeConfig("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index config.setHost("INDEX_HOST"); PineconeConnection connection = new PineconeConnection(config); Index index = new Index(connection, "INDEX_NAME"); // get the pagination token String paginationToken = index.list("example-namespace", 3).getPagination().getNext(); // get vectors with limit 3 with the paginationToken obtained from the previous step ListResponse listResponse = index.list("example-namespace", 3, paginationToken); } } // Response: // vectors { // id: "doc1#chunk1" // } // vectors { // id: "doc1#chunk2" // } // vectors { // id: "doc2#chunk1" // } // vectors { // id: "doc3#chunk1" // } // pagination { // next: "eyJza2lwX3Bhc3QiOiJhbHN0cm9lbWVyaWEtcGVydXZpYW4iLCJwcmVmaXgiOm51bGx9" // } // namespace: "example-namespace" // usage { // read_units: 1 // } ``` ```go Go package main import ( "context" "encoding/json" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func prettifyStruct(obj interface{}) string { bytes, _ := json.MarshalIndent(obj, "", " ") return string(bytes) } func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "INDEX_HOST", Namespace: "example-namespace"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v", err) } limit := uint32(3) res, err := idxConnection.ListVectors(ctx, &pinecone.ListVectorsRequest{ Limit: &limit, }) if len(res.VectorIds) == 0 { fmt.Println("No vectors found") } else { fmt.Printf(prettifyStruct(res)) } } // Response: // { // "vector_ids": [ // "doc1#chunk1", // "doc1#chunk2", // "doc1#chunk3" // ], // "usage": { // "read_units": 1 // }, // "next_pagination_token": "eyJza2lwX3Bhc3QiOiIwMDBkMTc4OC0zMDAxLTQwZmMtYjZjNC0wOWI2N2I5N2JjNDUiLCJwcmVmaXgiOm51bGx9" // } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index var index = pinecone.Index(host: "INDEX_HOST"); var listResponse = await index.ListAsync(new ListRequest { Namespace = "example-namespace", }); Console.WriteLine(listResponse); // Response: // { // "vectors": [ // { // "id": "doc1#chunk1" // }, // { // "id": "doc1#chunk2" // }, // { // "id": "doc1#chunk3" // } // ], // "pagination": "eyJza2lwX3Bhc3QiOiIwMDBkMTc4OC0zMDAxLTQwZmMtYjZjNC0wOWI2N2I5N2JjNDUiLCJwcmVmaXgiOm51bGx9", // "namespace": "example-namespace", // "usage": { // "readUnits": 1 // } // } ``` ```shell curl # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index PINECONE_API_KEY="YOUR_API_KEY" INDEX_HOST="INDEX_HOST" curl -X GET "https://$INDEX_HOST/vectors/list?namespace=example-namespace" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" # Response: # { # "vectors": [ # { "id": "doc1#chunk1" }, # { "id": "doc1#chunk2" }, # { "id": "doc1#chunk3" }, # { "id": "doc1#chunk4" }, # ... # ], # "pagination": { # "next": "c2Vjb25kY2FsbA==" # }, # "namespace": "example-namespace", # "usage": { # "readUnits": 1 # } # } ``` ## List the IDs of records with a common prefix ID prefixes enable you to query segments of content. Use the `list` endpoint to list all of the records with the common prefix. For more details, see [Use ID prefixes](/guides/manage-data/manage-document-chunks#use-id-prefixes). ## Paginate through results The `list` endpoint returns up to 100 IDs per page at a time by default. If the `limit` parameter is passed, `list` returns up to that number of IDs per page instead. For example, if `limit=3`, up to 3 IDs be returned per page. Whenever there are additional IDs to return, the response also includes a `pagination_token` for fetching the next page of IDs. ### Implicit pagination When using the Python SDK, `list` paginates automatically. ```python Python from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key='YOUR_API_KEY') # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index index = pc.Index(host="INDEX_HOST") for ids in index.list(namespace='example-namespace'): print(ids) # Response: # ['doc1#chunk1', 'doc1#chunk2', 'doc1#chunk3'] # ['doc1#chunk4', 'doc1#chunk5', 'doc1#chunk6'] # ... ``` ### Manual pagination When using the Node.js SDK, Java SDK, Go SDK, .NET SDK, or REST API, you must manually fetch each page of results. You can also manually paginate with the Python SDK using `list_paginated()`. ```python Python from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key='YOUR_API_KEY') # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index index = pc.Index(host="INDEX_HOST") namespace = 'example-namespace' # For manual control over pagination results = index.list_paginated( prefix='pref', limit=3, namespace='example-namespace' ) print(results.namespace) print([v.id for v in results.vectors]) print(results.pagination.next) print(results.usage) # Results: # ['10103-0', '10103-1', '10103-10'] # eyJza2lwX3Bhc3QiOiIxMDEwMy0= # {'read_units': 1} ``` ```js JavaScript import { Pinecone } from '@pinecone-database/pinecone'; const pc = new Pinecone(); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index const index = pc.index("INDEX_NAME", "INDEX_HOST").namespace("example-namespace"); const results = await index.listPaginated({ prefix: 'doc1#', limit: 3 }); console.log(results); // Response: // { // vectors: [ // { id: 'doc1#01' }, { id: 'doc1#02' }, { id: 'doc1#03' } // ], // pagination: { // next: 'eyJza2lwX3Bhc3QiOiJwcmVUZXN0LSCIsInByZWZpeCI6InByZVRlc3QifQ==' // }, // namespace: 'example-namespace', // usage: { readUnits: 1 } // } ``` ```java Java import io.pinecone.clients.Index; import io.pinecone.configs.PineconeConfig; import io.pinecone.configs.PineconeConnection; import io.pinecone.proto.ListResponse; public class ListExample { public static void main(String[] args) { PineconeConfig config = new PineconeConfig("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index config.setHost("INDEX_HOST"); PineconeConnection connection = new PineconeConnection(config); Index index = new Index(connection, "INDEX_NAME"); ListResponse listResponse = index.list("example-namespace", "doc1#" 2); /* Note: Currently, you must include an ID prefix to list vector IDs. */ System.out.println(listResponse.getVectorsList()); System.out.println(listResponse.getPagination()); } } // Response: // vectors { // id: "doc1#chunk1" // } // vectors { // id: "doc1#chunk2" // } // pagination { // next: "eyJza2lwX3Bhc3QiOiJhbHN0cm9lbWVyaWEtcGVydXZpYW4iLCJwcmVmaXgiOm51bGx9" // } // namespace: "example-namespace" // usage { // read_units: 1 // } ``` ```go Go package main import ( "context" "encoding/json" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func prettifyStruct(obj interface{}) string { bytes, _ := json.MarshalIndent(obj, "", " ") return string(bytes) } func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "INDEX_HOST", Namespace: "example-namespace"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v", err) } limit := uint32(3) res, err := idxConnection.ListVectors(ctx, &pinecone.ListVectorsRequest{ Limit: &limit, }) if len(res.VectorIds) == 0 { fmt.Println("No vectors found") } else { fmt.Printf(prettifyStruct(res)) } } // Response: // { // "vector_ids": [ // "doc1#chunk1", // "doc1#chunk2", // "doc1#chunk3" // ], // "usage": { // "read_units": 1 // }, // "next_pagination_token": "eyJza2lwX3Bhc3QiOiIwMDBkMTc4OC0zMDAxLTQwZmMtYjZjNC0wOWI2N2I5N2JjNDUiLCJwcmVmaXgiOm51bGx9" // } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index var index = pinecone.Index(host: "INDEX_HOST"); var listResponse = await index.ListAsync(new ListRequest { Namespace = "example-namespace", Prefix = "document1#", }); Console.WriteLine(listResponse); // Response: // { // "vectors": [ // { // "id": "doc1#chunk1" // }, // { // "id": "doc1#chunk2" // }, // { // "id": "doc1#chunk3" // } // ], // "pagination": "eyJza2lwX3Bhc3QiOiIwMDBkMTc4OC0zMDAxLTQwZmMtYjZjNC0wOWI2N2I5N2JjNDUiLCJwcmVmaXgiOm51bGx9", // "namespace": "example-namespace", // "usage": { // "readUnits": 1 // } // } ``` ```shell curl # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index PINECONE_API_KEY="YOUR_API_KEY" INDEX_HOST="INDEX_HOST" curl -X GET "https://$INDEX_HOST/vectors/list?namespace=example-namespace" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" # Response: # { # "vectors": [ # { "id": "doc1#chunk1" }, # { "id": "doc1#chunk2" }, # { "id": "doc1#chunk3" }, # { "id": "doc1#chunk4" }, # ... # ], # "pagination": { # "next": "c2Vjb25kY2FsbA==" # }, # "namespace": "example-namespace", # "usage": { # "readUnits": 1 # } # } ``` Then, to get the next batch of IDs, use the returned `pagination_token`: ```python Python from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key='YOUR_API_KEY') # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index index = pc.Index(host="INDEX_HOST") namespace = 'example-namespace' results = index.list_paginated( prefix='pref', limit=3, namespace='example-namespace', pagination_token='eyJza2lwX3Bhc3QiOiIxMDEwMy0=' ) print(results.namespace) print([v.id for v in results.vectors]) print(results.pagination.next) print(results.usage) # Response: # ['10103-0', '10103-1', '10103-10'] # xndlsInByZWZpeCI6IjEwMTAzIn0== # {'read_units': 1} ``` ```js JavaScript await index.listPaginated({ prefix: 'doc1#', limit: 3, paginationToken: results.pagination.next}); // Response: // { // vectors: [ // { id: 'doc1#10' }, { id: 'doc1#11' }, { id: 'doc1#12' } // ], // pagination: { // next: 'dfajlkjfdsoijeowjoDJFKLJldLIFf34KFNLDSndaklqoLQJORN45afdlkJ==' // }, // namespace: 'example-namespace', // usage: { readUnits: 1 } // } ``` ```java Java listResponse = index.list("example-namespace", "doc1#", "eyJza2lwX3Bhc3QiOiJ2MTg4IiwicHJlZml4IjpudWxsfQ=="); System.out.println(listResponse.getVectorsList()); // Response: // vectors { // id: "doc1#chunk3" // } // vectors { // id: "doc1#chunk4" // } // vectors { // id: "doc1#chunk5" // } // vectors { // id: "doc1#chunk6" // } // vectors { // id: "doc1#chunk7" // } // vectors { // id: "doc1#chunk8" // } // pagination { // next: "eyJza2lwX3Bhc3QiOiJhbHN0cm9lbWVyaWEtcGVydXZpYW4iLCJwcmVmaXgiOm51bGx9" // } // namespace: "example-namespace" // usage { // read_units: 1 // } ``` ```go Go package main import ( "context" "encoding/json" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func prettifyStruct(obj interface{}) string { bytes, _ := json.MarshalIndent(obj, "", " ") return string(bytes) } func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "INDEX_HOST", Namespace: "example-namespace"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v", err) } limit := uint32(3) paginationToken := "dfajlkjfdsoijeowjoDJFKLJldLIFf34KFNLDSndaklqoLQJORN45afdlkJ==" res, err := idxConnection.ListVectors(ctx, &pinecone.ListVectorsRequest{ Limit: &limit, PaginationToken: &paginationToken, }) if len(res.VectorIds) == 0 { fmt.Println("No vectors found") } else { fmt.Printf(prettifyStruct(res)) } } // Response: // { // "vector_ids": [ // "doc1#chunk4", // "doc1#chunk5", // "doc1#chunk6" // ], // "usage": { // "read_units": 1 // }, // "next_pagination_token": "eyJza2lwX3Bhc3QiOiIwMDBkMTc4OC0zMDAxLTQwZmMtYjZjNC0wOWI2N2I5N2JjNDUiLCJwcmVmaXgiOm51bGx9" // } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index var index = pinecone.Index(host: "INDEX_HOST"); var listResponse = await index.ListAsync(new ListRequest { Namespace = "example-namespace", Prefix = "document1#", PaginationToken= "eyJza2lwX3Bhc3QiOiIwMDBkMTc4OC0zMDAxLTQwZmMtYjZjNC0wOWI2N2I5N2JjNDUiLCJwcmVmaXgiOm51bGx9", }); Console.WriteLine(listResponse); // Response: // { // "vectors": [ // { // "id": "doc1#chunk4" // }, // { // "id": "doc1#chunk5" // }, // { // "id": "doc1#chunk6" // } // ], // "pagination": "dfajlkjfdsoijeowjoDJFKLJldLIFf34KFNLDSndaklqoLQJORN45afdlkJ==", // "namespace": "example-namespace", // "usage": { // "readUnits": 1 // } // } ``` ```shell curl # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index PINECONE_API_KEY="YOUR_API_KEY" INDEX_HOST="INDEX_HOST" curl -X GET "https://$INDEX_HOST/vectors/list?namespace=example-namespace&paginationToken=c2Vjb25kY2FsbA%3D%3D" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" # Response: # { # "vectors": [ # { "id": "doc2#chunk1" }, # { "id": "doc2#chunk1" }, # { "id": "doc2#chunk1" }, # { "id": "doc2#chunk1" }, # ... # ], # "pagination": { # "next": "mn23b4jB3Y9jpsS1" # }, # "namespace": "example-namespace", # "usage": { # "readUnits": 1 # } # } ``` When there are no more IDs to return, the response does not includes a `pagination_token`: ```python Python from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key='YOUR_API_KEY') index = pc.Index(host="INDEX_HOST") namespace = 'example-namespace' results = index.list_paginated( prefix='10103', limit=3, pagination_token='xndlsInByZWZpeCI6IjEwMTAzIn0==' ) print(results.namespace) print([v.id for v in results.vectors]) print(results.pagination.next) print(results.usage) # Response: # ['10103-4', '10103-5', '10103-6'] # {'read_units': 1} ``` ```js JavaScript import { Pinecone } from '@pinecone-database/pinecone'; const pc = new Pinecone(); const index = pc.index("INDEX_NAME", "INDEX_HOST").namespace("example-namespace"); const results = await index.listPaginated({ prefix: 'doc1#' }); console.log(results); // Response: // { // vectors: [ // { id: 'doc1#19' }, { id: 'doc1#20' }, { id: 'doc1#21' } // ], // namespace: 'example-namespace', // usage: { readUnits: 1 } // } ``` ```go Go package main import ( "context" "encoding/json" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func prettifyStruct(obj interface{}) string { bytes, _ := json.MarshalIndent(obj, "", " ") return string(bytes) } func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "INDEX_HOST", Namespace: "example-namespace"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v", err) } limit := uint32(3) paginationToken := "eyJza2lwX3Bhc3QiOiIwMDBkMTc4OC0zMDAxLTQwZmMtYjZjNC0wOWI2N2I5N2JjNDUiLCJwcmVmaXgiOm51bGx9" res, err := idxConnection.ListVectors(ctx, &pinecone.ListVectorsRequest{ Limit: &limit, paginationToken: &paginationToken, }) if len(res.VectorIds) == 0 { fmt.Println("No vectors found") } else { fmt.Printf(prettifyStruct(res)) } } // Response: // { // "vector_ids": [ // "doc1#chunk7", // "doc1#chunk8", // "doc1#chunk9" // ], // "usage": { // "read_units": 1 // } // } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); var index = pinecone.Index(host: "INDEX_HOST"); var listResponse = await index.ListAsync(new ListRequest { Namespace = "example-namespace", Prefix = "document1#", PaginationToken= "dfajlkjfdsoijeowjoDJFKLJldLIFf34KFNLDSndaklqoLQJORN45afdlkJ==", }); Console.WriteLine(listResponse); // Response: // { // "vectors": [ // { // "id": "doc1#chunk7" // }, // { // "id": "doc1#chunk8" // }, // { // "id": "doc1#chunk9" // } // ], // "pagination": null, // "namespace": "example-namespace", // "usage": { // "readUnits": 1 // } // } ``` ```shell curl # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index PINECONE_API_KEY="YOUR_API_KEY" INDEX_HOST="INDEX_HOST" curl -X GET "https://$INDEX_HOST/vectors/list?namespace=example-namespace&paginationToken=mn23b4jB3Y9jpsS1" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" # Response: # { # "vectors": [ # { "id": "doc3#chunk1" }, # { "id": "doc5#chunk2" }, # { "id": "doc5#chunk3" }, # { "id": "doc5#chunk4" }, # ... # ], # "namespace": "example-namespace", # "usage": { # "readUnits": 1 # } # } ``` # Manage document chunks Source: https://docs.pinecone.io/guides/manage-data/manage-document-chunks In [Retrieval Augmented Generation (RAG)](https://www.pinecone.io/learn/retrieval-augmented-generation/) use cases, it is best practice to [chunk large documents](https://www.pinecone.io/learn/chunking-strategies/) into smaller segments, embed each chunk separately, and then store each embedded chunk as a distinct record in Pinecone. This page shows you how to model, store, and manage such records in serverless indexes. ## Use ID prefixes ID prefixes enable you to query segments of content, which is especially useful for lists and [mass deletion](/guides/manage-data/delete-data). Prefixes are commonly used to represent the following: * **Hierarchical relationships**: When you have multiple records representing chunks of a single document, use a [common ID prefix](#use-id-prefixes-to-reference-parent-documents) to reference the document. This is the main use of ID prefixes for RAG. * **Versioning**: Assign a [multi-level ID prefix](#work-with-multi-level-id-prefixes) to denote the version of the content. * **Content typing**: For [multi-modal search](https://www.pinecone.io/solutions/multi-modal/), assign an ID prefix to identify different kind of objects (e.g., text, images, videos) in the database. * **Source identification**: Assign an ID prefix to denote the source of the content. For example, if you want to disconnect a given user's account that was a data source, you can easily find and delete all of the records associated with the user. ### Use ID prefixes to reference parent documents When you have multiple records representing chunks of a single document, use a common ID prefix to reference the document. You can use any prefix pattern you like, but make sure you use a consistent prefix pattern for all child records of a document. For example, the following are all valid prefixes for the first chunk of `doc1`: * `doc1#chunk1` * `doc1_chunk1` * `doc1___chunk1` * `doc1:chunk1` * `doc1chunk1` Prefixes can also be [multi-level](#work-with-multi-level-id-prefixes). For example, `doc1#v1#chunk1` and `doc1#v2#chunk1` can represent different versions of the first chunk of `doc1`. ID prefixes are not validated on upsert or update. It is useful to pick a unique and consistent delimiter that will not be used in the ID elsewhere. ```python Python from pinecone.grpc import PineconeGRPC as Pinecone from pinecone import ServerlessSpec pc = Pinecone(api_key="YOUR_API_KEY") pc.create_index( name="docs-example", dimension=8, metric="cosine", spec=ServerlessSpec( cloud="aws", region="us-east-1" ) ) # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index index = pc.Index(host="INDEX_HOST") index.upsert( vectors=[ {"id": "doc1#chunk1", "values": [0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1]}, {"id": "doc1#chunk2", "values": [0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2]}, {"id": "doc1#chunk3", "values": [0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3]}, {"id": "doc1#chunk4", "values": [0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4]} ], namespace="example-namespace" ) ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }); await pc.createIndex({ name: 'docs-example', dimension: 1536, metric: 'cosine', spec: { serverless: { cloud: 'aws', region: 'us-east-1' } } }); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index const index = pc.index("INDEX_NAME", "INDEX_HOST") await index.namespace("example-namespace").upsert([ { "id": "doc1#chunk1", "values": [0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1] }, { "id": "doc1#chunk2", "values": [0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2] }, { "id": "doc1#chunk3", "values": [0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3] }, { "id": "doc1#chunk4", "values": [0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4] } ]); ``` ```java Java // Upsert vectors import com.google.protobuf.Struct; import io.pinecone.clients.Index; import io.pinecone.configs.PineconeConfig; import io.pinecone.configs.PineconeConnection; import io.pinecone.unsigned_indices_model.VectorWithUnsignedIndices; import java.util.ArrayList; import java.util.Arrays; import java.util.List; import static io.pinecone.commons.IndexInterface.buildUpsertVectorWithUnsignedIndices; import java.util.Arrays; import java.util.List; public class UpsertExample { public static void main(String[] args) { PineconeConfig config = new PineconeConfig("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index config.setHost("INDEX_HOST"); PineconeConnection connection = new PineconeConnection(config); Index index = new Index(connection, "INDEX_NAME"); // Vector ids to be upserted List upsertIds = Arrays.asList("doc1#chunk1", "doc1#chunk2", "doc1#chunk3"); // List of values to be upserted List> values = new ArrayList<>(); values.add(Arrays.asList(1.0f, 2.0f, 3.0f)); values.add(Arrays.asList(4.0f, 5.0f, 6.0f)); values.add(Arrays.asList(7.0f, 8.0f, 9.0f)); ... } } ``` ```go Go package main import ( "context" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // Serverless index indexName := "docs-example" vectorType := "dense" dimension := int32(1536) metric := pinecone.Cosine deletionProtection := pinecone.DeletionProtectionDisabled idx, err := pc.CreateServerlessIndex(ctx, &pinecone.CreateServerlessIndexRequest{ Name: indexName, VectorType: &vectorType, Dimension: &dimension, Metric: &metric, Cloud: pinecone.Aws, Region: "us-east-1", DeletionProtection: &deletionProtection, }) if err != nil { log.Fatalf("Failed to create serverless index: %v", idx.Name) } else { fmt.Printf("Successfully created serverless index: %v", idx.Name) } // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "INDEX_HOST"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v", err) } vectors := []*pinecone.Vector{ { Id: "doc1#chunk1", Values: []float32{0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1}, }, { Id: "doc1#chunk2", Values: []float32{0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2}, }, { Id: "doc1#chunk3", Values: []float32{0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3}, }, { Id: "doc1#chunk4", Values: []float32{0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4}, }, } count, err := idxConnection.UpsertVectors(ctx, vectors) if err != nil { log.Fatalf("Failed to upsert vectors: %v", err) } else { fmt.Printf("Successfully upserted %d vector(s)", count) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); var createIndexRequest = await pinecone.CreateIndexAsync(new CreateIndexRequest { Name = "docs-example", Dimension = 1536, Metric = MetricType.Cosine, Spec = new ServerlessIndexSpec { Serverless = new ServerlessSpec { Cloud = ServerlessSpecCloud.Aws, Region = "us-east-1", } }, DeletionProtection = DeletionProtection.Disabled }); var upsertResponse = await index.UpsertAsync(new UpsertRequest { Vectors = new[] { new Vector { Id = "doc1#chunk1", Values = new[] { 0.1f, 0.1f, 0.1f, 0.1f, 0.1f, 0.1f, 0.1f, 0.1f }, }, new Vector { Id = "doc1#chunk2", Values = new[] { 0.2f, 0.2f, 0.2f, 0.2f, 0.2f, 0.2f, 0.2f, 0.2f }, }, new Vector { Id = "doc1#chunk3", Values = new[] { 0.3f, 0.3f, 0.3f, 0.3f, 0.3f, 0.3f, 0.3f, 0.3f }, }, new Vector { Id = "doc1#chunk4", Values = new[] { 0.4f, 0.4f, 0.4f, 0.4f, 0.4f, 0.4f, 0.4f, 0.4f }, }, }, Namespace = "example-namespace", }); ``` ```shell curl PINECONE_API_KEY="YOUR_API_KEY" curl -s "https://api.pinecone.io/indexes" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "name": "docs-example", "dimension": 1536, "metric": "cosine", "spec": { "serverless": { "cloud": "aws", "region": "us-east-1" } } }' # The `GET` request below uses the unique endpoint for an index. # See https://docs.pinecone.io/guides/manage-data/target-an-index for details. INDEX_HOST="INDEX_HOST" curl -X GET "https://$INDEX_HOST/vectors/upsert" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "vectors": [ { "id": "doc1#chunk1", "values": [0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1] }, { "id": "doc1#chunk2", "values": [0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2] }, { "id": "doc1#chunk3", "values": [0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3] }, { "id": "doc1#chunk4", "values": [0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4] } ], "namespace": "example-namespace" }' ``` ## List all record IDs for a parent document When all records related to a document use a common ID prefix, you can use the `list` operation with the `namespace` and `prefix` parameters to fetch the IDs of the records. The `list` operation is supported only for [serverless indexes](/guides/index-data/indexing-overview#serverless-indexes). ```python Python from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key='YOUR_API_KEY') # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index index = pc.Index(host="INDEX_HOST") # To iterate over all result pages using a generator function for ids in index.list(prefix='doc1#', namespace='example-namespace'): print(ids) ``` ```js JavaScript import { Pinecone } from '@pinecone-database/pinecone'; const pc = new Pinecone(); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index const index = pc.index("INDEX_NAME", "INDEX_HOST").namespace("example-namespace"); const results = await index.listPaginated({ prefix: 'doc1#' }); console.log(results); ``` ```java Java import io.pinecone.clients.Index; import io.pinecone.configs.PineconeConfig; import io.pinecone.configs.PineconeConnection; import io.pinecone.proto.ListResponse; PineconeConfig config = new PineconeConfig("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index config.setHost("INDEX_HOST"); PineconeConnection connection = new PineconeConnection(config); Index index = new Index(connection, "INDEX_NAME"); ListResponse listResponse = index.list("example-namespace", "doc1#"); /* Note: Currently, you must include an ID prefix to list vector IDs. */ System.out.println("Response: " + listResponse.getVectorsList()); ``` ```go Go package main import ( "context" "encoding/json" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func prettifyStruct(obj interface{}) string { bytes, _ := json.MarshalIndent(obj, "", " ") return string(bytes) } func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "INDEX_HOST", Namespace: "example-namespace"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v", err) } limit := uint32(3) prefix := "doc1#" res, err := idxConnection.ListVectors(ctx, &pinecone.ListVectorsRequest{ Limit: &limit, Prefix: &prefix, }) if len(res.VectorIds) == 0 { fmt.Println("No vectors found") } else { fmt.Printf(prettifyStruct(res)) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index var index = pinecone.Index(host: "INDEX_HOST"); var listResponse = await index.ListAsync(new ListRequest { Namespace = "example-namespace", Prefix = "doc1#", }); Console.WriteLine(listResponse); ``` ```shell curl # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index PINECONE_API_KEY="YOUR_API_KEY" INDEX_HOST="INDEX_HOST" curl -X GET "https://$INDEX_HOST/vectors/list?namespace=example-namespace&prefix=doc1#" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" ``` The response looks like this: ```python Python ['doc1#chunk1', 'doc1#chunk2', 'doc1#chunk3'] ``` ```js JavaScript { vectors: [ { id: 'doc1#01' }, { id: 'doc1#02' }, { id: 'doc1#03' }, { id: 'doc1#04' }, { id: 'doc1#05' }, { id: 'doc1#06' }, { id: 'doc1#07' }, { id: 'doc1#08' }, { id: 'doc1#09' }, ... ], pagination: { next: 'eyJza2lwX3Bhc3QiOiJwcmVUZXN0LS04MCIsInByZWZpeCI6InByZVRlc3QifQ==' }, namespace: 'example-namespace', usage: { readUnits: 1 } } ``` ```java Java vectors { id: "doc1#chunk1" } vectors { id: "doc1#chunk2" } vectors { id: "doc1#chunk3" } ... pagination { next: "eyJza2lwX3Bhc3QiOiJhbHN0cm9lbWVyaWEtcGVydXZpYW4iLCJwcmVmaXgiOm51bGx9" } namespace: "example-namespace" usage { read_units: 1 } ``` ```go Go { "vector_ids": [ "doc1#chunk1", "doc1#chunk2", "doc1#chunk3" ], "usage": { "read_units": 1 } } ``` ```csharp C# { "vectors": [ { "id": "doc1#chunk1" }, { "id": "doc1#chunk2" }, { "id": "doc1#chunk3" } ], "pagination": "eyJza2lwX3Bhc3QiOiIwMDBkMTc4OC0zMDAxLTQwZmMtYjZjNC0wOWI2N2I5N2JjNDUiLCJwcmVmaXgiOm51bGx9", "namespace": "example-namespace", "usage": { "readUnits": 1 } } ``` ```json curl { "vectors": [ { "id": "doc1#chunk1" }, { "id": "doc1#chunk2" }, { "id": "doc1#chunk3" }, { "id": "doc1#chunk4" }, ... ], "pagination": { "next": "c2Vjb25kY2FsbA==" }, "namespace": "example-namespace", "usage": { "readUnits": 1 } } ``` When there are additional IDs to return, the response includes a `pagination_token` that you can use to get the next batch of IDs. For more details, see [Paginate through list results](/guides/manage-data/list-record-ids#paginate-through-results) With the record IDs, you can then use the `fetch` endpoint to [fetch the content of the records](/guides/manage-data/fetch-data). ## Delete all records for a parent document To delete all records representing chunks of a single document, first list the record IDs based on their common ID prefix, and then [delete the records by ID](/guides/manage-data/delete-data#delete-specific-records-by-id): ```python Python from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key='YOUR_API_KEY') # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index index = pc.Index(host="INDEX_HOST") for ids in index.list(prefix='doc1#', namespace='example-namespace'): print(ids) # ['doc1#chunk1', 'doc1#chunk2', 'doc1#chunk3'] index.delete(ids=ids, namespace=namespace) ``` ```js JavaScript import { Pinecone } from '@pinecone-database/pinecone'; const pc = new Pinecone(); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index const index = pc.index("INDEX_NAME", "INDEX_HOST").namespace("example-namespace"); const pageOneList = await index.listPaginated({ prefix: 'doc1#' }); const pageOneVectorIds = pageOneList.vectors.map((vector) => vector.id); // Then, delete the first page of records by ID: await index.deleteMany(pageOneVectorIds); // For a second page of returned records: const pageTwoList = await index.listPaginated({ prefix: 'doc1#', paginationToken: pageOneList.pagination.next }); const pageTwoVectorIds = pageTwoList.vectors.map((vector) => vector.id); await index.deleteMany(pageTwoVectorIds); ``` ```java Java import io.pinecone.clients.Index; import io.pinecone.configs.PineconeConfig; import io.pinecone.configs.PineconeConnection; import java.util.Arrays; import java.util.List; ... PineconeConfig config = new PineconeConfig("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index config.setHost("INDEX_HOST"); PineconeConnection connection = new PineconeConnection(config); Index index = new Index(connection, "INDEX_NAME"); List ids = Arrays.asList("doc1#"); DeleteResponse deleteResponse = index.deleteByIds(ids, "example-namespace"); ``` ```go Go package main import ( "context" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func main() { ctx := context.Background() client, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "INDEX_HOST", Namespace: "example-namespace"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v", err) } recordPrefix := "doc1#" var limit uint32 = 100 listResp, err := indexConn.ListVectors(context.Background(), &pinecone.ListVectorsRequest{ Prefix: &recordPrefix, Limit: &limit, }) if err != nil { panic(err) } fmt.Printf("List response: %+v\n", listResp) for listResp.NextPaginationToken != nil { vectorIdsToDelete := make([]string, len(listResp.VectorIds)) for i, vectorId := range listResp.VectorIds { if vectorId != nil { vectorIdsToDelete[i] = *vectorId } } err = indexConn.DeleteVectorsById(context.Background(), vectorIdsToDelete) if err != nil { panic(err) } fmt.Printf("Deleted %d vectors\n", len(vectorIdsToDelete)) listResp, err = indexConn.ListVectors(context.Background(), &pinecone.ListVectorsRequest{ Prefix: &recordPrefix, Limit: &limit, PaginationToken: listResp.NextPaginationToken, }) if err != nil { panic(err) } } } ``` ```shell curl # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index PINECONE_API_KEY="YOUR_API_KEY" INDEX_HOST="INDEX_HOST" curl -X GET "https://$INDEX_HOST/vectors/list?namespace=example-namespace&prefix=doc1#" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" # Response: # { # "vectors": [ # { "id": "doc1#chunk1" }, # { "id": "doc1#chunk2" }, # { "id": "doc1#chunk3" }, # { "id": "doc1#chunk4" }, # ... # ], # "pagination": { # "next": "c2Vjb25kY2FsbA==" # }, # "namespace": "example-namespace", # "usage": { # "readUnits": 1 # } # } # Then, delete the records by ID: curl "https://$INDEX_HOST/vectors/delete" \ -H "Api-Key: $PINECONE_API_KEY" \ -H 'Content-Type: application/json' \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "ids": [ "doc1#chunk1", "doc1#chunk2", "doc1#chunk3", "doc1#chunk4" ], "namespace": "example-namespace" } ' ``` ## Work with multi-level ID prefixes The examples above are based on a simple ID prefix (`doc1#`), but it's also possible to work with more complex, multi-level prefixes. For example, let's say you use the prefix pattern `doc#v#chunk` to differentiate between different versions of a document. If you wanted to delete all records for one version of a document, first list the record IDs based on the relevant `doc#v#` prefix and then [delete the records by ID](/guides/manage-data/delete-data#delete-specific-records-by-id): ```python Python from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key='YOUR_API_KEY') # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index index = pc.Index(host="INDEX_HOST") for ids in index.list(prefix='doc1#v1', namespace='example-namespace'): print(ids) # ['doc1#v1#chunk1', 'doc1#v1#chunk2', 'doc1#v1#chunk3'] index.delete(ids=ids, namespace=namespace) ``` ```js JavaScript import { Pinecone } from '@pinecone-database/pinecone'; const pc = new Pinecone(); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index const index = pc.index("INDEX_NAME", "INDEX_HOST").namespace("example-namespace"); const results = await index.listPaginated({ prefix: 'doc1#v1' }); console.log(results); // { // vectors: [ // { id: 'doc1#v1#01' }, { id: 'doc1#v1#02' }, { id: 'doc1#v1#03' } // ], // pagination: { // next: 'eyJza2lwX3Bhc3QiOiJwcmVUZXN0LS04MCIsInByZWZpeCI6InByZVRlc3QifQ==' // }, // namespace: 'example-namespace', // usage: { readUnits: 1 } // } // Then, delete the records by ID: const vectorIds = results.vectors.map((vector) => vector.id); await index.deleteMany(vectorIds); ``` ```java Java import io.pinecone.clients.Index; import io.pinecone.configs.PineconeConfig; import io.pinecone.configs.PineconeConnection; import java.util.Arrays; import java.util.List; ... PineconeConfig config = new PineconeConfig("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index config.setHost("INDEX_HOST"); PineconeConnection connection = new PineconeConnection(config); Index index = new Index(connection, "INDEX_NAME"); List ids = Arrays.asList("doc1#v1"); /* [id: "doc1#v1#chunk1", id: "doc1#v1#chunk2", id: "doc1#v1#chunk3" ... */ /* Then, delete the records by ID: */ DeleteResponse deleteResponse = index.deleteByIds(ids, "example-namespace"); ``` ```go Go package main import ( "context" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func main() { ctx := context.Background() client, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "INDEX_HOST", Namespace: "example-namespace"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v", err) } recordPrefix := "doc1#v1" var limit uint32 = 100 listResp, err := indexConn.ListVectors(context.Background(), &pinecone.ListVectorsRequest{ Prefix: &recordPrefix, Limit: &limit, }) if err != nil { panic(err) } fmt.Printf("List response: %+v\n", listResp) for listResp.NextPaginationToken != nil { vectorIdsToDelete := make([]string, len(listResp.VectorIds)) for i, vectorId := range listResp.VectorIds { if vectorId != nil { vectorIdsToDelete[i] = *vectorId } } err = indexConn.DeleteVectorsById(context.Background(), vectorIdsToDelete) if err != nil { panic(err) } fmt.Printf("Deleted %d vectors\n", len(vectorIdsToDelete)) listResp, err = indexConn.ListVectors(context.Background(), &pinecone.ListVectorsRequest{ Prefix: &recordPrefix, Limit: &limit, PaginationToken: listResp.NextPaginationToken, }) if err != nil { panic(err) } } } ``` ```shell curl # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index PINECONE_API_KEY="YOUR_API_KEY" INDEX_HOST="INDEX_HOST" curl -X GET "https://$INDEX_HOST/vectors/list?namespace=example-namespace&prefix=doc1%23v1%23" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" # Response: # { # "vectors": [ # { "id": "doc1#v1#chunk1" }, # { "id": "doc1#v1#chunk2" }, # { "id": "doc1#v1#chunk3" }, # { "id": "doc1#v1#chunk4" }, # ... # ], # "pagination": { # "next": "c2Vjb25kY2FsbA==" # }, # "namespace": "example-namespace", # "usage": { # "readUnits": 1 # } # } # Then, delete the records by ID: curl "https://$INDEX_HOST/vectors/delete" \ -H "Api-Key: $PINECONE_API_KEY" \ -H 'Content-Type: application/json' \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "ids": [ "doc1#v1#chunk1", "doc1#v1#chunk2", "doc1#v1#chunk3", "doc1#v1#chunk4" ], "namespace": "example-namespace" } ' ``` However, if you wanted to delete all records across all versions of a document, you would list the record IDs based on the `doc1#` part of the prefix that is common to all versions and then [delete the records by ID](/guides/manage-data/delete-data#delete-specific-records-by-id): ```python Python from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key='YOUR_API_KEY') # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index index = pc.Index(host="INDEX_HOST") for ids in index.list(prefix='doc1#', namespace='example-namespace'): print(ids) # ['doc1#v1#chunk1', 'doc1#v1#chunk2', 'doc1#v1#chunk3', 'doc1#v2#chunk1', 'doc1#v2#chunk2', 'doc1#v2#chunk3'] index.delete(ids=ids, namespace=namespace) ``` ```js JavaScript import { Pinecone } from '@pinecone-database/pinecone'; const pc = new Pinecone(); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index const index = pc.index("INDEX_NAME", "INDEX_HOST").namespace("example-namespace"); const results = await index.listPaginated({ prefix: 'doc1#' }); console.log(results); // { // vectors: [ // { id: 'doc1#v1#01' }, { id: 'doc1#v1#02' }, { id: 'doc1#v1#03' }, // { id: 'doc1#v2#01' }, { id: 'doc1#v2#02' }, { id: 'doc1#v2#03' } // ], // pagination: { // next: 'eyJza2lwX3Bhc3QiOiJwcmVUZXN0LS04MCIsInByZWZpeCI6InByZVRlc3QifQ==' // }, // namespace: 'example-namespace', // usage: { readUnits: 1 } // } // Then, delete the records by ID: const vectorIds = results.vectors.map((vector) => vector.id); await index.deleteMany(vectorIds); ``` ```java Java import io.pinecone.clients.Index; import io.pinecone.configs.PineconeConfig; import io.pinecone.configs.PineconeConnection; import java.util.Arrays; import java.util.List; ... PineconeConfig config = new PineconeConfig("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index config.setHost("INDEX_HOST"); PineconeConnection connection = new PineconeConnection(config); Index index = new Index(connection, "INDEX_NAME"); List ids = Arrays.asList("doc1"); /* [id: "doc1#v1#chunk1", id: "doc1#v1#chunk2", id: "doc1#v2#chunk1" ... */ /* Then, delete the records by ID:*/ DeleteResponse deleteResponse = index.deleteByIds(ids, "example-namespace"); ``` ```go Go package main import ( "context" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func main() { ctx := context.Background() client, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "INDEX_HOST", Namespace: "example-namespace"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v", err) } recordPrefix := "doc1#" var limit uint32 = 100 listResp, err := indexConn.ListVectors(context.Background(), &pinecone.ListVectorsRequest{ Prefix: &recordPrefix, Limit: &limit, }) if err != nil { panic(err) } fmt.Printf("List response: %+v\n", listResp) for listResp.NextPaginationToken != nil { vectorIdsToDelete := make([]string, len(listResp.VectorIds)) for i, vectorId := range listResp.VectorIds { if vectorId != nil { vectorIdsToDelete[i] = *vectorId } } err = indexConn.DeleteVectorsById(context.Background(), vectorIdsToDelete) if err != nil { panic(err) } fmt.Printf("Deleted %d vectors\n", len(vectorIdsToDelete)) listResp, err = indexConn.ListVectors(context.Background(), &pinecone.ListVectorsRequest{ Prefix: &recordPrefix, Limit: &limit, PaginationToken: listResp.NextPaginationToken, }) if err != nil { panic(err) } } } ``` ```shell curl # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index PINECONE_API_KEY="YOUR_API_KEY" INDEX_HOST="INDEX_HOST" curl -X GET "https://$INDEX_HOST/vectors/list?namespace=example-namespace&prefix=doc1%23" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" # Response: # { # "vectors": [ # { "id": "doc1#v1#chunk1" }, # { "id": "doc1#v1#chunk2" }, # { "id": "doc1#v1#chunk3" }, # { "id": "doc1#v1#chunk4" }, # ... # { "id": "doc1#v2#chunk1" }, # { "id": "doc1#v2#chunk2" }, # { "id": "doc1#v2#chunk3" }, # { "id": "doc1#v2#chunk4" }, # ... # ], # "pagination": { # "next": "c2Vjb25kY2FsbA==" # }, # "namespace": "example-namespace", # "usage": { # "readUnits": 1 # } # } # Then, delete the records by ID: curl "https://$INDEX_HOST/vectors/delete" \ -H "Api-Key: $PINECONE_API_KEY" \ -H 'Content-Type: application/json' \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "ids": [ "doc1#v1#chunk1", "doc1#v1#chunk2", "doc1#v1#chunk3", "doc1#v1#chunk4", ... "doc1#v2#chunk1", "doc1#v2#chunk2", "doc1#v2#chunk3", "doc1#v2#chunk4", ], "namespace": "example-namespace" } ' ``` {/* ## RAG using pod-based indexes The `list` endpoint does not support pod-based indexes. Instead of using ID prefixes to reference parent documents, [use a metadata key-value pair](/guides/index-data/upsert-data#upsert-with-metadata-filters). If you later need to delete the records, you can [pass a metadata filter expression to the `delete` endpoint](/guides/manage-data/delete-data#delete-specific-records-by-metadata). */} # Manage serverless indexes Source: https://docs.pinecone.io/guides/manage-data/manage-indexes This page shows you how to manage your existing serverless indexes. For guidance on pod-based indexes, see [Manage pod-based indexes](/guides/indexes/pods/manage-pod-based-indexes). ## List indexes Use the [`list_indexes`](/reference/api/2024-10/control-plane/list_indexes) operation to get a complete description of all indexes in a project: ```Python Python from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key="YOUR_API_KEY") index_list = pc.list_indexes() print(index_list) ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }) const indexList = await pc.listIndexes(); console.log(indexList); ``` ```java Java import io.pinecone.clients.Pinecone; import org.openapitools.db_control.client.model.*; public class ListIndexesExample { public static void main(String[] args) { Pinecone pc = new Pinecone.Builder("YOUR_API_KEY").build(); IndexList indexList = pc.listIndexes(); System.out.println(indexList); } } ``` ```go Go package main import ( "context" "encoding/json" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func prettifyStruct(obj interface{}) string { bytes, _ := json.MarshalIndent(obj, "", " ") return string(bytes) } func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } idxs, err := pc.ListIndexes(ctx) if err != nil { log.Fatalf("Failed to list indexes: %v", err) } else { for _, index := range idxs { fmt.Printf("index: %v\n", prettifyStruct(index)) } } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); var indexList = await pinecone.ListIndexesAsync(); Console.WriteLine(indexList); ``` ```shell curl PINECONE_API_KEY="YOUR_API_KEY" curl -i -X GET "https://api.pinecone.io/indexes" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" ``` The response will look like this: ```python Python [{ "name": "docs-example-sparse", "metric": "dotproduct", "host": "docs-example-sparse-govk0nt.svc.aped-4627-b74a.pinecone.io", "spec": { "serverless": { "cloud": "aws", "region": "us-east-1" } }, "status": { "ready": true, "state": "Ready" }, "vector_type": "sparse", "dimension": null, "deletion_protection": "disabled", "tags": { "environment": "development" } }, { "name": "docs-example-dense", "metric": "cosine", "host": "docs-example-dense-govk0nt.svc.aped-4627-b74a.pinecone.io", "spec": { "serverless": { "cloud": "aws", "region": "us-east-1" } }, "status": { "ready": true, "state": "Ready" }, "vector_type": "dense", "dimension": 1536, "deletion_protection": "disabled", "tags": { "environment": "development" } }] ``` ```javascript JavaScript { indexes: [ { name: 'docs-example-sparse', dimension: undefined, metric: 'dotproduct', host: 'docs-example-sparse-govk0nt.svc.aped-4627-b74a.pinecone.io', deletionProtection: 'disabled', tags: { environment: 'development', example: 'tag' }, embed: undefined, spec: { pod: undefined, serverless: { cloud: 'aws', region: 'us-east-1' } }, status: { ready: true, state: 'Ready' }, vectorType: 'sparse' }, { name: 'docs-example-dense', dimension: 1536, metric: 'cosine', host: 'docs-example-dense-govk0nt.svc.aped-4627-b74a.pinecone.io', deletionProtection: 'disabled', tags: { environment: 'development', example: 'tag' }, embed: undefined, spec: { pod: undefined, serverless: { cloud: 'aws', region: 'us-east-1' } }, status: { ready: true, state: 'Ready' }, vectorType: 'dense' } ] } ``` ```java Java class IndexList { indexes: [class IndexModel { name: docs-example-sparse dimension: null metric: dotproduct host: docs-example-sparse-govk0nt.svc.aped-4627-b74a.pinecone.io deletionProtection: disabled tags: {environment=development} embed: null spec: class IndexModelSpec { pod: null serverless: class ServerlessSpec { cloud: aws region: us-east-1 additionalProperties: null } additionalProperties: null } status: class IndexModelStatus { ready: true state: Ready additionalProperties: null } vectorType: sparse additionalProperties: null }, class IndexModel { name: docs-example-dense dimension: 1536 metric: cosine host: docs-example-dense-govk0nt.svc.aped-4627-b74a.pinecone.io deletionProtection: disabled tags: {environment=development} embed: null spec: class IndexModelSpec { pod: null serverless: class ServerlessSpec { cloud: aws region: us-east-1 additionalProperties: null } additionalProperties: null } status: class IndexModelStatus { ready: true state: Ready additionalProperties: null } vectorType: dense additionalProperties: null }] additionalProperties: null } ``` ```go Go index: { "name": "docs-example-sparse", "host": "docs-example-sparse-govk0nt.svc.aped-4627-b74a.pinecone.io", "metric": "dotproduct", "vector_type": "sparse", "deletion_protection": "disabled", "dimension": null, "spec": { "serverless": { "cloud": "aws", "region": "us-east-1" } }, "status": { "ready": true, "state": "Ready" }, "tags": { "environment": "development" } } index: { "name": "docs-example-dense", "host": "docs-example-dense-govk0nt.svc.aped-4627-b74a.pinecone.io", "metric": "cosine", "vector_type": "dense", "deletion_protection": "disabled", "dimension": 1536, "spec": { "serverless": { "cloud": "aws", "region": "us-east-1" } }, "status": { "ready": true, "state": "Ready" }, "tags": { "environment": "development" } } ``` ```csharp C# { "indexes": [ { "name": "docs-example-sparse", "metric": "dotproduct", "host": "docs-example-sparse-govk0nt.svc.aped-4627-b74a.pinecone.io", "deletion_protection": "disabled", "tags": { "environment": "development" }, "spec": { "serverless": { "cloud": "aws", "region": "us-east-1" } }, "status": { "ready": true, "state": "Ready" }, "vector_type": "sparse" }, { "name": "docs-example-dense", "dimension": 1536, "metric": "cosine", "host": "docs-example-dense-govk0nt.svc.aped-4627-b74a.pinecone.io", "deletion_protection": "disabled", "tags": { "environment": "development" }, "spec": { "serverless": { "cloud": "aws", "region": "us-east-1" } }, "status": { "ready": true, "state": "Ready" }, "vector_type": "dense" } ] } ``` ```json curl { "indexes": [ { "name": "docs-example-sparse", "vector_type": "sparse", "metric": "dotproduct", "dimension": null, "status": { "ready": true, "state": "Ready" }, "host": "docs-example-sparse-govk0nt.svc.aped-4627-b74a.pinecone.io", "spec": { "serverless": { "region": "us-east-1", "cloud": "aws" } }, "deletion_protection": "disabled", "tags": { "environment": "development" } }, { "name": "docs-example-dense", "vector_type": "dense", "metric": "cosine", "dimension": 1536, "status": { "ready": true, "state": "Ready" }, "host": "docs-example-dense-govk0nt.svc.aped-4627-b74a.pinecone.io", "spec": { "serverless": { "region": "us-east-1", "cloud": "aws" } }, "deletion_protection": "disabled", "tags": { "environment": "development" } } ] } ``` With the Python SDK, you can use the `.names()` helper function to iterate over the index names in the `list_indexes()` response, for example: ```Python Python from pinecone.grpc import PineconeGRPC as Pinecone from pinecone import ServerlessSpec for index_name in pc.list_indexes().names: print(index_name) ``` ## Describe an index Use the [`describe_index`](/reference/api/2024-10/control-plane/describe_index/) endpoint to get a complete description of a specific index: ```Python Python from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key="YOUR_API_KEY") pc.describe_index(name="docs-example") ``` ```JavaScript JavaScript import { Pinecone } from '@pinecone-database/pinecone'; const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }); await pc.describeIndex('docs-example'); ``` ```java Java import io.pinecone.clients.Pinecone; import org.openapitools.db_control.client.model.*; public class DescribeIndexExample { public static void main(String[] args) { Pinecone pc = new Pinecone.Builder("YOURE_API_KEY").build(); IndexModel indexModel = pc.describeIndex("docs-example"); System.out.println(indexModel); } } ``` ```go Go package main import ( "context" "encoding/json" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func prettifyStruct(obj interface{}) string { bytes, _ := json.MarshalIndent(obj, "", " ") return string(bytes) } func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } idx, err := pc.DescribeIndex(ctx, "docs-example") if err != nil { log.Fatalf("Failed to describe index \"%v\": %v", idx.Name, err) } else { fmt.Printf("index: %v\n", prettifyStruct(idx)) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); var indexModel = await pinecone.DescribeIndexAsync("docs-example"); Console.WriteLine(indexModel); ``` ```bash curl PINECONE_API_KEY="YOUR_API_KEY" curl -i -X GET "https://api.pinecone.io/indexes/docs-example" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" ``` The response will look like this: ```Python Python {'deletion_protection': 'disabled', 'dimension': 1536, 'host': 'docs-example-dense-govk0nt.svc.aped-4627-b74a.pinecone.io', 'metric': 'cosine', 'name': 'docs-example-dense', 'spec': {'serverless': {'cloud': 'aws', 'region': 'us-east-1'}}, 'status': {'ready': True, 'state': 'Ready'}, 'tags': {'environment': 'development'}, 'vector_type': 'dense'} ``` ```javaScript JavaScript { name: 'docs-example-dense', dimension: 1536, metric: 'cosine', host: 'docs-example-dense-govk0nt.svc.aped-4627-b74a.pinecone.io', deletionProtection: 'disabled', tags: { environment: 'development', example: 'tag' }, embed: undefined, spec: { pod: undefined, serverless: { cloud: 'aws', region: 'us-east-1' } }, status: { ready: true, state: 'Ready' }, vectorType: 'dense' } ``` ```java Java class IndexModel { name: docs-example-dense dimension: 1536 metric: cosine host: docs-example-dense-govk0nt.svc.aped-4627-b74a.pinecone.io deletionProtection: disabled tags: {environment=development} embed: null spec: class IndexModelSpec { pod: null serverless: class ServerlessSpec { cloud: aws region: us-east-1 additionalProperties: null } additionalProperties: null } status: class IndexModelStatus { ready: true state: Ready additionalProperties: null } vectorType: dense additionalProperties: null } ``` ```go Go index: { "name": "docs-example-dense", "host": "docs-example-dense-govk0nt.svc.aped-4627-b74a.pinecone.io", "metric": "cosine", "vector_type": "dense", "deletion_protection": "disabled", "dimension": 1536, "spec": { "serverless": { "cloud": "aws", "region": "us-east-1" } }, "status": { "ready": true, "state": "Ready" }, "tags": { "environment": "development" } } ``` ```csharp C# { "name": "docs-example-dense", "dimension": 1536, "metric": "cosine", "host": "docs-example-dense-govk0nt.svc.aped-4627-b74a.pinecone.io", "deletion_protection": "disabled", "tags": { "environment": "development" }, "spec": { "serverless": { "cloud": "aws", "region": "us-east-1" } }, "status": { "ready": true, "state": "Ready" }, "vector_type": "dense" } ``` ```json curl { "name": "docs-example-dense", "vector_type": "dense", "metric": "cosine", "dimension": 1536, "status": { "ready": true, "state": "Ready" }, "host": "docs-example-dense-govk0nt.svc.aped-4627-b74a.pinecone.io", "spec": { "serverless": { "region": "us-east-1", "cloud": "aws" } }, "deletion_protection": "disabled", "tags": { "environment": "development" } } ``` **Do not target an index by name in production.** When you target an index by name for data operations such as `upsert` and `query`, the SDK gets the unique DNS host for the index using the `describe_index` operation. This is convenient for testing but should be avoided in production because `describe_index` uses a different API than data operations and therefore adds an additional network call and point of failure. Instead, you should get an index host once and cache it for reuse or specify the host directly. ## Delete an index Use the [`delete_index`](reference/api/2024-10/control-plane/delete_index) operation to delete an index and all of its associated resources. ```python Python # pip install "pinecone[grpc]" from pinecone.grpc import PineconeGRPC as Pinecone, PodSpec pc = Pinecone(api_key="YOUR_API_KEY") pc.delete_index(name="docs-example") ``` ```javascript JavaScript // npm install @pinecone-database/pinecone import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }); await pc.deleteIndex('docs-example'); ``` ```java Java import io.pinecone.clients.Pinecone; public class DeleteIndexExample { public static void main(String[] args) { Pinecone pc = new Pinecone.Builder("YOUR_API_KEY").build(); pc.deleteIndex("docs-example"); } } ``` ```go Go package main import ( "context" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } indexName := "docs-example" err = pc.DeleteIndex(ctx, indexName) if err != nil { log.Fatalf("Failed to delete index: %v", err) } else { fmt.Println("Index \"%v\" deleted successfully", indexName) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); await pinecone.DeleteIndexAsync("docs-example"); ``` ```shell curl PINECONE_API_KEY="YOUR_API_KEY" curl -i -X DELETE "https://api.pinecone.io/indexes/docs-example" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" ``` If deletion protection is enabled on an index, requests to delete it will fail and return a `403 - FORBIDDEN` status with the following error: ``` Deletion protection is enabled for this index. Disable deletion protection before retrying. ``` Before you can delete such an index, you must first [disable deletion protection](/guides/manage-data/manage-indexes#configure-deletion-protection). You can delete an index using the [Pinecone console](https://app.pinecone.io/organizations/-/projects/-/indexes). For the index you want to delete, click the three dots to the right of the index name, then click **Delete**. ## Associate an embedding model [Integrated inference](/guides/index-data/indexing-overview#integrated-embedding) lets you upsert and search without extra steps for embedding data and reranking results. To configure an existing serverless index for an embedding model, use the [`configure_index`](/reference/api/2025-01/control-plane/configure_index) operation as follows: * Set `embed.model` to one of [Pinecone's hosted embedding models](/guides/index-data/create-an-index#embedding-models). * Set `embed.field_map` to the name of the field in your source document that contains the data for embedding. The `vector_type`, `metric`, and `dimension` of the index must be supported by the specified embedding model. ```python Python # pip install --upgrade pinecone from pinecone import Pinecone pc = Pinecone(api_key="YOUR_API_KEY") pc.configure_index( name="docs-example", embed={ "model":"llama-text-embed-v2", "field_map":{"text": "chunk_text"} } ) ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone'; const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }); await pc.configureIndex('docs-example', { embed: { model: 'llama-text-embed-v2', fieldMap: { text: 'chunk_text' }, }, }); ``` ```json curl PINECONE_API_KEY="YOUR_API_KEY" curl -s -X PATCH "https://api.pinecone.io/indexes/docs-example" \ -H "Content-Type: application/json" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "embed": { "model": "llama-text-embed-v2", "field_map": { "text": "chunk_text" } } }' ``` ## Configure deletion protection This feature requires [Pinecone API version](/reference/api/versioning) `2024-07`, [Python SDK](/reference/python-sdk) v5.0.0, [Node.js SDK](/reference/node-sdk) v3.0.0, [Java SDK](/reference/java-sdk) v2.0.0, or [Go SDK](/reference/go-sdk) v1.0.0 or later. ### Enable deletion protection You can prevent an index and its data from accidental deleting when [creating a new index](/guides/index-data/create-an-index) or after its been created. In both cases, you set the `deletion_protection` parameter to `enabled`. Enabling deletion protection does *not* prevent [namespace deletions](/guides/manage-data/manage-namespaces#delete-a-namespace). To enable deletion protection when creating a new index: ```python Python # pip install "pinecone[grpc]" # Serverless index from pinecone.grpc import PineconeGRPC as Pinecone from pinecone import ServerlessSpec pc = Pinecone(api_key="YOUR_API_KEY") pc.create_index( name="docs-example", dimension=1536, metric="cosine", spec=ServerlessSpec( cloud="aws", region="us-east-1" ), deletion_protection="enabled" ) ``` ```javascript JavaScript // npm install @pinecone-database/pinecone // Serverles index import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }); await pc.createIndex({ name: 'docs-example', dimension: 1536, metric: 'cosine', spec: { serverless: { cloud: 'aws', region: 'us-east-1' } }, deletionProtection: 'enabled', }); ``` ```java Java import io.pinecone.clients.Pinecone; import org.openapitools.db_control.client.model.IndexModel; import org.openapitools.db_control.client.model.DeletionProtection; // Serverless index public class CreateServerlessIndexExample { public static void main(String[] args) { Pinecone pc = new Pinecone.Builder("YOUR_API_KEY").build(); pc.createServerlessIndex("docs-example", "cosine", 1536, "aws", "us-east-1", DeletionProtection.enabled); } } ``` ```go Go package main import ( "context" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // Serverless index indexName := "docs-example" vectorType := "dense" dimension := int32(1536) metric := pinecone.Cosine deletionProtection := pinecone.DeletionProtectionDisabled idx, err := pc.CreateServerlessIndex(ctx, &pinecone.CreateServerlessIndexRequest{ Name: indexName, VectorType: &vectorType, Dimension: &dimension, Metric: &metric, Cloud: pinecone.Aws, Region: "us-east-1", DeletionProtection: &deletionProtection, Tags: &pinecone.IndexTags{ "environment": "development" }, }) if err != nil { log.Fatalf("Failed to create serverless index: %v", idx.Name) } else { fmt.Printf("Successfully created serverless index: %v", idx.Name) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); // Serverless index var createIndexRequest = await pinecone.CreateIndexAsync(new CreateIndexRequest { Name = "docs-example", Dimension = 1536, Metric = MetricType.Cosine, Spec = new ServerlessIndexSpec { Serverless = new ServerlessSpec { Cloud = ServerlessSpecCloud.Aws, Region = "us-east-1", } }, DeletionProtection = DeletionProtection.Enabled }); ``` ```shell curl PINECONE_API_KEY="YOUR_API_KEY" # Serverless index curl -s "https://api.pinecone.io/indexes" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "name": "docs-example", "dimension": 1536, "metric": "cosine", "spec": { "serverless": { "cloud": "aws", "region": "us-east-1" } }, "deletion_protection": "enabled" }' ``` To enable deletion protection when configuring an existing index: ```python Python from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key="YOUR_API_KEY") pc.configure_index( name="docs-example", deletion_protection="enabled" ) ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone'; const client = new Pinecone({ apiKey: 'YOUR_API_KEY' }); await client.configureIndex('docs-example', { deletionProtection: 'enabled' }); ``` ```java Java import io.pinecone.clients.Pinecone; import org.openapitools.db_control.client.model.*; public class ConfigureIndexExample { public static void main(String[] args) { Pinecone pc = new Pinecone.Builder("YOUR_API_KEY").build(); pc.configureServerlessIndex("docs-example", DeletionProtection.ENABLED); } } ``` ```go Go package main import ( "context" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } idx, err := pc.ConfigureIndex(ctx, "docs-example", pinecone.ConfigureIndexParams{DeletionProtection: "enabled"}) if err != nil { log.Fatalf("Failed to configure index \"%v\": %v", idx.Name, err) } else { fmt.Printf("Successfully configured index \"%v\"", idx.Name) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); var indexMetadata = await pinecone.ConfigureIndexAsync("docs-example", new ConfigureIndexRequest { DeletionProtection = DeletionProtection.Enabled, }); ``` ```shell curl PINECONE_API_KEY="YOUR_API_KEY" curl -s -X PATCH "https://api.pinecone.io/indexes/docs-example-curl" \ -H "Content-Type: application/json" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "deletion_protection": "enabled" }' ``` When deletion protection is enabled on an index, requests to delete the index fail and return a `403 - FORBIDDEN` status with the following error: ``` Deletion protection is enabled for this index. Disable deletion protection before retrying. ``` ### Disable deletion protection Before you can [delete an index](#delete-an-index) with deletion protection enabled, you must first disable deletion protection as follows: ```python Python from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key="YOUR_API_KEY") pc.configure_index( name="docs-example", deletion_protection="disabled" ) ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone'; const client = new Pinecone({ apiKey: 'YOUR_API_KEY' }); await client.configureIndex('docs-example', { deletionProtection: 'disabled' }); ``` ```java Java import io.pinecone.clients.Pinecone; import org.openapitools.db_control.client.model.*; public class ConfigureIndexExample { public static void main(String[] args) { Pinecone pc = new Pinecone.Builder("YOUR_API_KEY").build(); pc.configureServerlessIndex("docs-example", DeletionProtection.DISABLED); } } ``` ```go Go package main import ( "context" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } idx, err := pc.ConfigureIndex(ctx, "docs-example", pinecone.ConfigureIndexParams{DeletionProtection: "disabled"}) if err != nil { log.Fatalf("Failed to configure index \"%v\": %v", idx.Name, err) } else { fmt.Printf("Successfully configured index \"%v\"", idx.Name) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); var configureIndexRequest = await pinecone.ConfigureIndexAsync("docs-example", new ConfigureIndexRequest { DeletionProtection = DeletionProtection.Disabled, }); ``` ```shell curl PINECONE_API_KEY="YOUR_API_KEY" curl -s -X PATCH "https://api.pinecone.io/indexes/docs-example-curl" \ -H "Content-Type: application/json" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "deletion_protection": "disabled" }' ``` ## Configure index tags Tags are key-value pairs that you can use to categorize and identify the index. ### Add tags To add tags to an index, use the `tags` parameter when [creating a new index](/guides/index-data/create-an-index) or when [configuring an existing index](/guides/indexes/pods/manage-pod-based-indexes). To add tags when creating a new index: ```python Python from pinecone.grpc import PineconeGRPC as Pinecone from pinecone import ServerlessSpec pc = Pinecone(api_key="YOUR_API_KEY") pc.create_index( name="docs-example", dimension=1536, metric="cosine", spec=ServerlessSpec( cloud="aws", region="us-east-1" ), deletion_protection="disabled", tags={ "example": "tag", "environment": "development" } ) ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone' const client = new Pinecone({ apiKey: 'YOUR_API_KEY' }); await pc.createIndex({ name: 'docs-example', dimension: 1536, metric: 'cosine', spec: { serverless: { cloud: 'aws', region: 'us-east-1' } }, deletionProtection: 'disabled', tags: { example: 'tag', environment: 'development' }, }); ``` ```java Java import io.pinecone.clients.Pinecone; import org.openapitools.db_control.client.model.IndexModel; import org.openapitools.db_control.client.model.DeletionProtection; import java.util.HashMap; // Serverless index public class CreateServerlessIndexExample { public static void main(String[] args) { Pinecone pc = new Pinecone.Builder("YOUR_API_KEY").build(); HashMap tags = new HashMap<>(); tags.put("tag", "development"); pc.createServerlessIndex("docs-example", "cosine", 1536, "aws", "us-east-1", DeletionProtection.DISABLED, tags); } } ``` ```go Go package main import ( "context" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // Serverless index idx, err := pc.CreateServerlessIndex(ctx, &pinecone.CreateServerlessIndexRequest{ Name: "docs-example", Dimension: 1536, Metric: pinecone.Cosine, Cloud: pinecone.Aws, Region: "us-east-1", DeletionProtection: "disabled", Tags: &pinecone.IndexTags{ "example": "tag", "environment": "development" }, }) if err != nil { log.Fatalf("Failed to create serverless index: %v", idx.Name) } else { fmt.Printf("Successfully created serverless index: %v", idx.Name) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); var createIndexRequest = await pinecone.CreateIndexAsync(new CreateIndexRequest { Name = "docs-example", Dimension = 1536, Metric = MetricType.Cosine, Spec = new ServerlessIndexSpec { Serverless = new ServerlessSpec { Cloud = ServerlessSpecCloud.Aws, Region = "us-east-1" } }, DeletionProtection = DeletionProtection.Disabled, Tags = new Dictionary { { "example", "tag" }, { "environment", "development" } } }); ``` ```shell curl PINECONE_API_KEY="YOUR_API_KEY" # Serverless index curl -s "https://api.pinecone.io/indexes" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "name": "docs-example", "dimension": 1536, "metric": "cosine", "spec": { "serverless": { "cloud": "aws", "region": "us-east-1" } }, "tags": { "example": "tag", "environment": "development" }, "deletion_protection": "disabled" }' ``` You can add tags during index creation using the [Pinecone console](https://app.pinecone.io/organizations/-/projects/-/create-index/). To add or update tags when configuring an existing index: ```python Python from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key="YOUR_API_KEY") pc.configure_index( name="docs-example", tags={ example: "tag", environment: "development" } ) ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone'; const client = new Pinecone({ apiKey: 'YOUR_API_KEY' }); await client.configureIndex('docs-example', { tags: { example: 'tag', environment: 'development' }}); ``` ```java Java import io.pinecone.clients.Pinecone; import org.openapitools.db_control.client.model.*; import java.util.HashMap; public class ConfigureIndexExample { public static void main(String[] args) { Pinecone pc = new Pinecone.Builder("YOUR_API_KEY").build(); HashMap tags = new HashMap<>(); tags.put("tag", "development"); pc.configureServerlessIndex("docs-example", DeletionProtection.ENABLED, tags); } } ``` ```go Go package main import ( "context" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } idx, err := pc.ConfigureIndex(ctx, "docs-example", pinecone.ConfigureIndexParams{ Tags: pinecone.IndexTags{ "example": "tag", "environment": "development", }, }, ) if err != nil { log.Fatalf("Failed to configure index \"%v\": %v", idx.Name, err) } else { fmt.Printf("Successfully configured index \"%v\"", idx.Name) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); var configureIndexRequest = await pinecone.ConfigureIndexAsync("docs-example", new ConfigureIndexRequest { Tags = new Dictionary { { "example", "tag" }, { "environment", "development" } } }); ``` ```shell curl PINECONE_API_KEY="YOUR_API_KEY" curl -s -X PATCH "https://api.pinecone.io/indexes/docs-example-curl" \ -H "Content-Type: application/json" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "tags": { "example": "tag", "environment": "development" } }' ``` You can add or update tags when configuring an existing index using the [Pinecone console](https://app.pinecone.io/organizations/-/projects/-/indexes). Find the index to edit and click the **ellipsis (..) menu > Add tags**. ### View tags To view the tags of an index, [list all indexes](/guides/manage-data/manage-indexes) in a project or [get information about a specific index](/guides/manage-data/manage-indexes). ### Remove tags To remove a tag from an index, [configure the index](/reference/api/2024-10/control-plane/configure_index) and use the `tags` parameter to send the tag key with an empty value (`""`). The following example removes the `example: tag` tag from `docs-example`: ```python Python from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key="YOUR_API_KEY") pc.configure_index( name="docs-example", tags={"example": ""} ) ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone'; const client = new Pinecone({ apiKey: 'YOUR_API_KEY' }); await client.configureIndex('docs-example', { tags: { example: '' }}); ``` ```java Java import io.pinecone.clients.Pinecone; import org.openapitools.db_control.client.model.*; import java.util.HashMap; public class ConfigureIndexExample { public static void main(String[] args) { Pinecone pc = new Pinecone.Builder("YOUR_API_KEY").build(); HashMap tags = new HashMap<>(); tags.put("example", ""); pc.configureServerlessIndex("docs-example", DeletionProtection.ENABLED, tags); } } ``` ```go Go package main import ( "context" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } idx, err := pc.ConfigureIndex(ctx, "docs-example", pinecone.ConfigureIndexParams{ Tags: pinecone.IndexTags{ "example": "", }, }, ) if err != nil { log.Fatalf("Failed to configure index \"%v\": %v", idx.Name, err) } else { fmt.Printf("Successfully configured index \"%v\"", idx.Name) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); var configureIndexRequest = await pinecone.ConfigureIndexAsync("docs-example", new ConfigureIndexRequest { Tags = new Dictionary { { "example", "" } } }); ``` ```shell curl PINECONE_API_KEY="YOUR_API_KEY" curl -s -X PATCH "https://api.pinecone.io/indexes/docs-example-curl" \ -H "Content-Type: application/json" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "tags": { "example": "" } }' ``` You can remove tags from an index using the [Pinecone console](https://app.pinecone.io/organizations/-/projects/-/indexes). Find the index to edit and click the **ellipsis (..) menu > \_\_ tags**. ## View backups for an index Serverless indexes can be [backed up](/guides/manage-data/back-up-an-index). You can [view all backups for a specific index](/reference/api/2025-04/control-plane/list_index_backups), as in the following example: ```python Python from pinecone import Pinecone pc = Pinecone(api_key="YOUR_API_KEY") index_backups = pc.list_backups(index_name="docs-example") print(index_backups) ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone'; const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }) const indexBackups = await pc.listBackups({ indexName: 'docs-example' }); console.log(indexBackups); ``` ```java Java import io.pinecone.clients.Pinecone; import org.openapitools.db_control.client.ApiException; import org.openapitools.db_control.client.model.*; public class CreateBackup { public static void main(String[] args) throws ApiException { Pinecone pc = new Pinecone.Builder("YOUR_API_KEY").build(); String indexName = "docs-example"; BackupList indexBackupList = pc.listIndexBackups(indexName); System.out.println(indexBackupList); } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); var indexBackups = await pinecone.Backups.ListByIndexAsync( "docs-example", new ListBackupsByIndexRequest()); Console.WriteLine(indexBackups); ``` ```bash curl PINECONE_API_KEY="YOUR_API_KEY" INDEX_NAME="docs-example" curl -X GET "https://api.pinecone.io/indexes/$INDEX_NAME/backups" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" \ -H "accept: application/json" ``` The example returns a response like the following: ```python Python [{ "backup_id": "8c85e612-ed1c-4f97-9f8c-8194e07bcf71", "source_index_name": "docs-example", "source_index_id": "f73b36c9-faf5-4a2c-b1d6-4013d8b1cc74", "status": "Ready", "cloud": "aws", "region": "us-east-1", "tags": {}, "name": "example-backup", "description": "Monthly backup of production index", "dimension": 1024, "record_count": 98, "namespace_count": 3, "size_bytes": 1069169, "created_at": "2025-05-15T00:52:10.809305882Z" }] ``` ```javascript JavaScript { data: [ { backupId: '8c85e612-ed1c-4f97-9f8c-8194e07bcf71', sourceIndexName: 'docs-example', sourceIndexId: 'f73b36c9-faf5-4a2c-b1d6-4013d8b1cc74', name: 'example-backup', description: 'Monthly backup of production index', status: 'Ready', cloud: 'aws', region: 'us-east-1', dimension: 1024, metric: undefined, recordCount: 98, namespaceCount: 3, sizeBytes: 1069169, tags: {}, createdAt: '2025-05-14T16:37:25.625540Z' } ], pagination: undefined } ``` ```java Java class BackupList { data: [class BackupModel { backupId: 8c85e612-ed1c-4f97-9f8c-8194e07bcf71 sourceIndexName: docs-example sourceIndexId: f73b36c9-faf5-4a2c-b1d6-4013d8b1cc74 name: example-backup description: Monthly backup of production index status: Initializing cloud: aws region: us-east-1 dimension: null metric: null recordCount: null namespaceCount: null sizeBytes: null tags: {} createdAt: 2025-05-16T19:46:26.248428Z additionalProperties: null }] pagination: null additionalProperties: null } ``` ```csharp C# { "data": [ { "backup_id":"9947520e-d5a1-4418-a78d-9f464c9969da", "source_index_id":"8433941a-dae7-43b5-ac2c-d3dab4a56b2b", "source_index_name":"docs-example", "tags":{}, "name":"example-backup", "description":"Monthly backup of production index", "status":"Pending", "cloud":"aws", "region":"us-east-1", "dimension":1024, "record_count":98, "namespace_count":3, "size_bytes":1069169, "created_at":"2025-03-11T18:29:50.549505Z" } ] } ``` ```json curl { "data": [ { "backup_id":"9947520e-d5a1-4418-a78d-9f464c9969da", "source_index_id":"8433941a-dae7-43b5-ac2c-d3dab4a56b2b", "source_index_name":"docs-example", "tags":{}, "name":"example-backup", "description":"Monthly backup of production index", "status":"Pending", "cloud":"aws", "region":"us-east-1", "dimension":1024, "record_count":98, "namespace_count":3, "size_bytes":1069169, "created_at":"2025-03-11T18:29:50.549505Z" } ], "pagination":null } ``` You can view the backups for a specific index from either the [Backups](https://app.pinecone.io/organizations/-/projects/-/backups) tab or the [Indexes](https://app.pinecone.io/organizations/-/projects/-/indexes) tab in the Pinecone console. # Manage namespaces Source: https://docs.pinecone.io/guides/manage-data/manage-namespaces This page describes how to list, describe, and delete a [namespace](/guides/index-data/indexing-overview#namespaces) in Pinecone. These operations are available in API versions `2025-04` and later. ## List all namespaces in an index You can [list all namespaces](/reference/api/2025-04/data-plane/listnamespaces) in a serverless index as follows. Up to 100 namespaces are returned at a time by default, in sorted order (bitwise “C” collation). If the `limit` parameter is set, up to that number of namespaces are returned instead. Whenever there are additional namespaces to return, the response also includes a `pagination_token` that you can use to get the next batch of namespaces. When the response does not include a `pagination_token`, there are no more namespaces to return. ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone'; const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }) // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index var index = pinecone.Index(host: "INDEX_HOST"); const namespaceList = await index.listNamespaces(); console.log(namespaceList); ``` ```java Java import io.pinecone.clients.AsyncIndex; import io.pinecone.clients.Index; import io.pinecone.configs.PineconeConfig; import io.pinecone.configs.PineconeConnection; import io.pinecone.proto.ListNamespacesResponse; import org.openapitools.db_data.client.ApiException; public class Namespaces { public static void main(String[] args) throws ApiException { PineconeConfig config = new PineconeConfig("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index config.setHost("INDEX_HOST"); PineconeConnection connection = new PineconeConnection(config); Index index = new Index(config, connection, "docs-example"); ListNamespacesResponse listNamespacesResponse = index.listNamespaces(); // List all restore jobs with default pagination limit (100) ListNamespacesResponse listNamespacesResponse = index.listNamespaces(null, null); // List all restore jobs with pagination limit of 2 ListNamespacesResponse listNamespacesResponseWithLimit = index.listNamespaces(2); // List all restore jobs with pagination limit and token ListNamespacesResponse listNamespacesResponsePaginated = index.listNamespaces(5, "eyJza2lwX3Bhc3QiOiIxMDEwMy0="); System.out.println(restoreJobListWithLimit); } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("PINECONE_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index var index = pinecone.Index(host: "INDEX_HOST"); var namespaces = await index.ListNamespacesAsync(new ListNamespacesRequest()); Console.WriteLine(namespaces); ``` ```bash curl PINECONE_API_KEY="YOUR_API_KEY" INDEX_HOST="INDEX_HOST" curl -X GET "https://$INDEX_HOST/namespaces" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" ``` The response will look like the following: ```javascript JavaScript { namespaces: [ { name: 'example-namespace', recordCount: '20000' }, { name: 'example-namespace2', recordCount: '10500' }, ... ], pagination: "Tm90aGluZyB0byBzZWUgaGVyZQo=" } ``` ```java Java namespaces { name: "example-namespace" record_count: 20000 } namespaces { name: "example-namespace2" record_count: 10500 } pagination { next: "eyJza2lwX3Bhc3QiOiJlZDVhYzFiNi1kMDFiLTQ2NTgtYWVhZS1hYjJkMGI2YzBiZjQiLCJwcmVmaXgiOm51bGx9" } ``` ```csharp C# { "namespaces":[ {"name":"example-namespace","recordCount":20000}, {"name":"example-namespace2","recordCount":10500}, ... ], "pagination":"Tm90aGluZyB0byBzZWUgaGVyZQo=" } ``` ```json curl { "namespaces": [ { "name": "example-namespace", "record_count": 20000 }, { "name": "example-namespace-2", "record_count": 10500 }, ... ], "pagination": { "next": "Tm90aGluZyB0byBzZWUgaGVyZQo=" } } ``` ## Describe a namespace To [get details about a namespace](/reference/api/2025-04/data-plane/describenamespace) in a serverless index, including the total number of vectors in the namespace: ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone'; const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }) const index = pc.index('docs-example'); const namespace = await index.describeNamespace('example-namespace'); console.log(namespace); ``` ```java Java import io.pinecone.clients.Index; import io.pinecone.configs.PineconeConfig; import io.pinecone.configs.PineconeConnection; import io.pinecone.proto.NamespaceDescription; import org.openapitools.db_data.client.ApiException; public class Namespaces { public static void main(String[] args) throws ApiException { PineconeConfig config = new PineconeConfig("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index config.setHost("INDEX_HOST"); PineconeConnection connection = new PineconeConnection(config); Index index = new Index(config, connection, "docs-example"); NamespaceDescription namespaceDescription = index.describeNamespace("example-namespace"); System.out.println(namespaceDescription); } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("PINECONE_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index var index = pinecone.Index(host: "INDEX_HOST"); var @namespace = await index.DescribeNamespaceAsync("example-namespace"); Console.WriteLine(@namespace); ``` ```bash curl PINECONE_API_KEY="YOUR_API_KEY" INDEX_HOST="YOUR_INDEX_HOST" NAMESPACE="YOUR_NAMESPACE" # To target the default namespace, use "__default__". curl -X GET "https://$INDEX_HOST/namespaces/$NAMESPACE" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" ``` The response will look like the following: ```javascript JavaScript { name: 'example-namespace', recordCount: '20000' } ``` ```java Java name: "example-namespace" record_count: 20000 ``` ```csharp C# {"name":"example-namespace","recordCount":20000} ``` ```json curl { "name": "example-namespace", "record_count": 20000 } ``` ## Delete a namespace To [delete a namespace](/reference/api/2025-04/data-plane/deletenamespace) in a serverless index: Deleting a namespace is irreversible. All data in the namespace will be permanently deleted. ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone'; const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }) const index = pc.index('docs-example'); const namespace = await index.deleteNamespace('example-namespace'); console.log(namespace); ``` ```java Java import io.pinecone.clients.Index; import io.pinecone.configs.PineconeConfig; import io.pinecone.configs.PineconeConnection; import java.util.concurrent.ExecutionException; public class DeleteNamespace { public static void main(String[] args) throws ExecutionException, InterruptedException { PineconeConfig config = new PineconeConfig("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index config.setHost("INDEX_HOST"); PineconeConnection connection = new PineconeConnection(config); Index index = new Index(config, connection, "docs-example"); index.deleteNamespace("example-namespace"); } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("PINECONE_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index var index = pinecone.Index(host: "INDEX_HOST"); await index.DeleteNamespaceAsync("example-namespace"); ``` ```bash curl PINECONE_API_KEY="YOUR_API_KEY" INDEX_HOST="YOUR_INDEX_HOST" NAMESPACE="YOUR_NAMESPACE" # To target the default namespace, use "__default__". curl -X DELETE "https://$INDEX_HOST/namespaces/$NAMESPACE" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" ``` ## Rename a namespace Pinecone does not support renaming namespaces directly. Instead, you must [delete the records](/guides/manage-data/delete-data) in the namespace and [upsert the records](/guides/index-data/upsert-data) to a new namespace. ## Move records to a new namespace Pinecone does not support moving records between namespaces directly. Instead, you must [delete the records](/guides/manage-data/delete-data) in the old namespace and [upsert the records](/guides/index-data/upsert-data) to the new namespace. # Restore an index Source: https://docs.pinecone.io/guides/manage-data/restore-an-index You can restore a serverless index by creating a new index from a [backup](/guides/manage-data/backups-overview). ## Create a serverless index from a backup Serverless index [backups](/guides/manage-data/backups-overview) are in [public preview](/release-notes/feature-availability) and available only on [Standard and Enterprise plans](https://www.pinecone.io/pricing/). You can create a new serverless index from a backup. The new index can differ from the source index in name, but it must be created in the same cloud provider and region, and have the same dimensions and similarity metric as the source index. The new index is queryable and writable. You cannot create a pod-based index from a backup of a serverless index. To [create a serverless index from a backup](/reference/api/2025-04/control-plane/create_index_from_backup), provide the `backup_id` parameter containing the ID of the backup from which you wish to create an index: ```python Python from pinecone import Pinecone pc = Pinecone(api_key="YOUR_API_KEY") pc.create_index_from_backup( backup_id="a65ff585-d987-4da5-a622-72e19a6ed5f4", name="restored-index", tags={ "tag0": "val0", "tag1": "val1" }, deletion_protection="enabled" ) ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone'; const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }) const response = await pc.createIndexFromBackup({ backupId: 'a65ff585-d987-4da5-a622-72e19a6ed5f4', name: 'restored-index', tags: { tag0: 'val0', tag1: 'val1' }, deletionProtection: 'enabled' }); console.log(response); ``` ```java Java import io.pinecone.clients.Pinecone; import org.openapitools.db_control.client.ApiException; import org.openapitools.db_control.client.model.*; public class CreateIndexFromBackup { public static void main(String[] args) throws ApiException { Pinecone pc = new Pinecone.Builder("YOUR_API_KEY").build(); String backupID = "a65ff585-d987-4da5-a622-72e19a6ed5f4"; String indexName = "restored-index"; CreateIndexFromBackupResponse backupResponse = pc.createIndexFromBackup(backupID, indexName); System.out.println(backupResponse); } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); var response = await pinecone.Backups.CreateIndexFromBackupAsync( "a65ff585-d987-4da5-a622-72e19a6ed5f4", new CreateIndexFromBackupRequest { Name = "restored-index", Tags = new Dictionary { { "tag0", "val0" }, { "tag1", "val1" } }, DeletionProtection = DeletionProtection.Enabled } ); Console.WriteLine(response); ``` ```bash curl PINECONE_API_KEY="YOUR_API_KEY" BACKUP_ID="a65ff585-d987-4da5-a622-72e19a6ed5f4" curl "https://api.pinecone.io/backups/$BACKUP_ID/create-index" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" \ -H 'Content-Type: application/json' \ -d '{ "name": "restored-index", "tags": { "tag0": "val0", "tag1": "val1" }, "deletion_protection": "enabled" }' ``` The example returns a response like the following: ```python Python {'deletion_protection': 'enabled', 'dimension': 1024, 'embed': {'dimension': 1024, 'field_map': {'text': 'chunk_text'}, 'metric': 'cosine', 'model': 'multilingual-e5-large', 'read_parameters': {'input_type': 'query', 'truncate': 'END'}, 'vector_type': 'dense', 'write_parameters': {'input_type': 'passage', 'truncate': 'END'}}, 'host': 'example-dense-index-python3-govk0nt.svc.aped-4627-b74a.pinecone.io', 'metric': 'cosine', 'name': 'example-dense-index-python3', 'spec': {'serverless': {'cloud': 'aws', 'region': 'us-east-1'}}, 'status': {'ready': True, 'state': 'Ready'}, 'tags': {'tag0': 'val0', 'tag1': 'val1'}, 'vector_type': 'dense'} ``` ```javascript JavaScript { restoreJobId: 'e9ba8ff8-7948-4cfa-ba43-34227f6d30d4', indexId: '025117b3-e683-423c-b2d1-6d30fbe5027f' } ``` ```java Java class CreateIndexFromBackupResponse { restoreJobId: e9ba8ff8-7948-4cfa-ba43-34227f6d30d4 indexId: 025117b3-e683-423c-b2d1-6d30fbe5027f additionalProperties: null } ``` ```csharp C# { "restore_job_id":"e9ba8ff8-7948-4cfa-ba43-34227f6d30d4", "index_id":"025117b3-e683-423c-b2d1-6d30fbe5027f" } ``` ```json curl { "restore_job_id":"e9ba8ff8-7948-4cfa-ba43-34227f6d30d4", "index_id":"025117b3-e683-423c-b2d1-6d30fbe5027f" } ``` You can create a serverless index from a backup using the [Pinecone console](https://app.pinecone.io/organizations/-/projects). ## Create a serverless index from a collection You can migrate a pod-based index to serverless by creating a new serverless index from a collection. For more information, see [Migrate a pod-based index to serverless](/guides/indexes/pods/migrate-a-pod-based-index-to-serverless). ## List restore jobs You can [list all restore jobs](/reference/api/2025-04/control-plane/list_restore_jobs) as follows. Up to 100 restore jobs are returned at a time by default, in sorted order (bitwise “C” collation). If the `limit` parameter is set, up to that number of restore jobs are returned instead. Whenever there are additional restore jobs to return, the response also includes a `pagination_token` that you can use to get the next batch of jobs. When the response does not include a `pagination_token`, there are no more restore jobs to return. ```python Python from pinecone import Pinecone pc = Pinecone(api_key="YOUR_API_KEY") restore_jobs = pc.list_restore_jobs() print(restore_jobs) ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone'; const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }) const restoreJobs = await pc.listRestoreJobs(); console.log(restoreJobs); ``` ```java Java import io.pinecone.clients.Pinecone; import org.openapitools.db_control.client.ApiException; import org.openapitools.db_control.client.model.*; public class CreateIndexFromBackup { public static void main(String[] args) throws ApiException { Pinecone pc = new Pinecone.Builder("YOUR_API-KEY").build(); // List all restore jobs with default pagination limit RestoreJobList restoreJobList = pc.listRestoreJobs(null, null); // List all restore jobs with pagination limit of 5 RestoreJobList restoreJobListWithLimit = pc.listRestoreJobs(5); // List all restore jobs with pagination limit and token RestoreJobList restoreJobListPaginated = pc.listRestoreJobs(5, "eyJza2lwX3Bhc3QiOiIxMDEwMy0="); System.out.println(restoreJobList); System.out.println(restoreJobListWithLimit); System.out.println(restoreJobListPaginated); } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); var jobs = await pinecone.RestoreJobs.ListAsync(new ListRestoreJobsRequest()); Console.WriteLine(jobs); ``` ```bash curl PINECONE_API_KEY="YOUR_API_KEY" curl "https://api.pinecone.io/restore-jobs" \ -H "X-Pinecone-Api-Version: 2025-04" \ -H "Api-Key: $PINECONE_API_KEY" ``` The example returns a response like the following: ```python Python [{ "restore_job_id": "06b08366-a0a9-404d-96c2-e791c71743e5", "backup_id": "95707edb-e482-49cf-b5a5-312219a51a97", "target_index_name": "restored-index", "target_index_id": "027aff93-de40-4f48-a573-6dbcd654f961", "status": "Completed", "created_at": "2025-05-15T13:59:51.439479+00:00", "completed_at": "2025-05-15T14:00:09.222998+00:00", "percent_complete": 100.0 }, { "restore_job_id": "4902f735-b876-4e53-a05c-bc01d99251cb", "backup_id": "8c85e612-ed1c-4f97-9f8c-8194e07bcf71", "target_index_name": "restored-index2", "target_index_id": "027aff93-de40-4f48-a573-6dbcd654f961", "status": "Completed", "created_at": "2025-05-15T21:06:19.906074+00:00", "completed_at": "2025-05-15T21:06:39.360509+00:00", "percent_complete": 100.0 }] ``` ```javascript JavaScript { data: [ { restoreJobId: '69acc1d0-9105-4fcb-b1db-ebf97b285c5e', backupId: '8c85e612-ed1c-4f97-9f8c-8194e07bcf71', targetIndexName: 'restored-index2', targetIndexId: 'e6c0387f-33db-4227-9e91-32181106e56b', status: 'Completed', createdAt: 2025-05-14T17:25:59.378Z, completedAt: 2025-05-14T17:26:23.997Z, percentComplete: 100 }, { restoreJobId: '9857add2-99d4-4399-870e-aa7f15d8d326', backupId: '94a63aeb-efae-4f7a-b059-75d32c27ca57', targetIndexName: 'restored-index', targetIndexId: '0d8aed24-adf8-4b77-8e10-fd674309dc85', status: 'Completed', createdAt: 2025-04-25T18:14:05.227Z, completedAt: 2025-04-25T18:14:11.074Z, percentComplete: 100 } ], pagination: undefined } ``` ```java Java class RestoreJobList { data: [class RestoreJobModel { restoreJobId: cf597d76-4484-4b6c-b07c-2bfcac3388aa backupId: 0d75b99f-be61-4a93-905e-77201286c02e targetIndexName: restored-index targetIndexId: 8a810881-1505-46c0-b906-947c048b15f5 status: Completed createdAt: 2025-05-16T20:09:18.700631Z completedAt: 2025-05-16T20:11:30.673296Z percentComplete: 100.0 additionalProperties: null }, class RestoreJobModel { restoreJobId: 4902f735-b876-4e53-a05c-bc01d99251cb backupId: 8c85e612-ed1c-4f97-9f8c-8194e07bcf71 targetIndexName: restored-index2 targetIndexId: 710cb6e6-bfb4-4bf5-a425-9754e5bbc832 status: Completed createdAt: 2025-05-15T21:06:19.906074Z completedAt: 2025-05-15T21:06:39.360509Z percentComplete: 100.0 additionalProperties: null }] pagination: class PaginationResponse { next: eyJsaW1pdCI6Miwib2Zmc2V0IjoyfQ== additionalProperties: null } additionalProperties: null } ``` ```csharp C# { "data": [ { "restore_job_id": "9857add2-99d4-4399-870e-aa7f15d8d326", "backup_id": "94a63aeb-efae-4f7a-b059-75d32c27ca57", "target_index_name": "restored-index", "target_index_id": "0d8aed24-adf8-4b77-8e10-fd674309dc85", "status": "Completed", "created_at": "2025-04-25T18:14:05.227526Z", "completed_at": "2025-04-25T18:14:11.074618Z", "percent_complete": 100 }, { "restore_job_id": "69acc1d0-9105-4fcb-b1db-ebf97b285c5e", "backup_id": "8c85e612-ed1c-4f97-9f8c-8194e07bcf71", "target_index_name": "restored-index2", "target_index_id": "e6c0387f-33db-4227-9e91-32181106e56b", "status": "Completed", "created_at": "2025-05-14T17:25:59.378989Z", "completed_at": "2025-05-14T17:26:23.997284Z", "percent_complete": 100 } ] } ``` ```json curl { "data": [ { "restore_job_id": "9857add2-99d4-4399-870e-aa7f15d8d326", "backup_id": "94a63aeb-efae-4f7a-b059-75d32c27ca57", "target_index_name": "restored-index", "target_index_id": "0d8aed24-adf8-4b77-8e10-fd674309dc85", "status": "Completed", "created_at": "2025-04-25T18:14:05.227526Z", "completed_at": "2025-04-25T18:14:11.074618Z", "percent_complete": 100 }, { "restore_job_id": "69acc1d0-9105-4fcb-b1db-ebf97b285c5e", "backup_id": "8c85e612-ed1c-4f97-9f8c-8194e07bcf71", "target_index_name": "restored-index2", "target_index_id": "e6c0387f-33db-4227-9e91-32181106e56b", "status": "Completed", "created_at": "2025-05-14T17:25:59.378989Z", "completed_at": "2025-05-14T17:26:23.997284Z", "percent_complete": 100 } ], "pagination": null } ``` ## View restore job details You can [view the details of a specific restore job](/reference/api/2025-04/control-plane/describe_restore_job), as in the following example: ```python Python from pinecone import Pinecone pc = Pinecone(api_key="YOUR_API_KEY") restore_job = pc.describe_restore_job(job_id="9857add2-99d4-4399-870e-aa7f15d8d326") print(restore_job) ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone'; const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }) const restoreJob = await pc.describeRestoreJob('9857add2-99d4-4399-870e-aa7f15d8d326'); console.log(restoreJob); ``` ```java Java import io.pinecone.clients.Pinecone; import org.openapitools.db_control.client.ApiException; import org.openapitools.db_control.client.model.*; public class CreateIndexFromBackup { public static void main(String[] args) throws ApiException { Pinecone pc = new Pinecone.Builder("YOUR_API-KEY").build(); RestoreJobModel restoreJob = pc.describeRestoreJob("9857add2-99d4-4399-870e-aa7f15d8d326"); System.out.println(restoreJob); } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); var job = await pinecone.RestoreJobs.GetAsync("9857add2-99d4-4399-870e-aa7f15d8d326"); Console.WriteLine(job); ``` ```bash curl PINECONE_API_KEY="YOUR_API_KEY" JOB_ID="9857add2-99d4-4399-870e-aa7f15d8d326" curl "https://api.pinecone.io/restore-jobs/$JOB_ID" \ -H "X-Pinecone-Api-Version: 2025-04" \ -H "Api-Key: $PINECONE_API_KEY" \ -H 'accept: application/json' ``` The example returns a response like the following: ```python Python {'backup_id': '94a63aeb-efae-4f7a-b059-75d32c27ca57', 'completed_at': datetime.datetime(2025, 4, 25, 18, 14, 11, 74618, tzinfo=tzutc()), 'created_at': datetime.datetime(2025, 4, 25, 18, 14, 5, 227526, tzinfo=tzutc()), 'percent_complete': 100.0, 'restore_job_id': '9857add2-99d4-4399-870e-aa7f15d8d326', 'status': 'Completed', 'target_index_id': '0d8aed24-adf8-4b77-8e10-fd674309dc85', 'target_index_name': 'restored-index'} ``` ```javascript JavaScript { restoreJobId: '9857add2-99d4-4399-870e-aa7f15d8d326', backupId: '94a63aeb-efae-4f7a-b059-75d32c27ca57', targetIndexName: 'restored-index', targetIndexId: '0d8aed24-adf8-4b77-8e10-fd674309dc85', status: 'Completed', createdAt: 2025-04-25T18:14:05.227Z, completedAt: 2025-04-25T18:14:11.074Z, percentComplete: 100 } ``` ```java Java class RestoreJobModel { restoreJobId: cf597d76-4484-4b6c-b07c-2bfcac3388aa backupId: 0d75b99f-be61-4a93-905e-77201286c02e targetIndexName: restored-index targetIndexId: 0d8aed24-adf8-4b77-8e10-fd674309dc85 status: Completed createdAt: 2025-05-16T20:09:18.700631Z completedAt: 2025-05-16T20:11:30.673296Z percentComplete: 100.0 additionalProperties: null } ``` ```csharp C# { "restore_job_id": "9857add2-99d4-4399-870e-aa7f15d8d326", "backup_id": "94a63aeb-efae-4f7a-b059-75d32c27ca57", "target_index_name": "restored-index", "target_index_id": "0d8aed24-adf8-4b77-8e10-fd674309dc85", "status": "Completed", "created_at": "2025-04-25T18:14:05.227526Z", "completed_at": "2025-04-25T18:14:11.074618Z", "percent_complete": 100 } ``` ```json curl { "restore_job_id": "9857add2-99d4-4399-870e-aa7f15d8d326", "backup_id": "94a63aeb-efae-4f7a-b059-75d32c27ca57", "target_index_name": "restored-index", "target_index_id": "0d8aed24-adf8-4b77-8e10-fd674309dc85", "status": "Completed", "created_at": "2025-04-25T18:14:05.227526Z", "completed_at": "2025-04-25T18:14:11.074618Z", "percent_complete": 100 } ``` # Target an index Source: https://docs.pinecone.io/guides/manage-data/target-an-index This page shows you how to target an index for data operations such as `upsert` and `query`. **Do not target an index by name in production.** When you target an index by name for data operations such as `upsert` and `query`, the SDK gets the unique DNS host for the index using the `describe_index` operation. This is convenient for testing but should be avoided in production because `describe_index` uses a different API than data operations and therefore adds an additional network call and point of failure. Instead, you should get an index host once and cache it for reuse or specify the host directly. ## Target by index host (recommended) This method is recommended for production: When using Private Endpoints for private connectivity between your application and Pinecone, you must target the index using the [Private Endpoint URL](/guides/production/connect-to-aws-privatelink#run-data-plane-commands) for the host. ```Python Python {5} from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key="YOUR_API_KEY") index = pc.Index(host="INDEX_HOST") ``` ```javascript JavaScript {6} import { Pinecone } from '@pinecone-database/pinecone'; const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }); // For the Node.js SDK, you must specify both the index host and name. const index = pc.index("INDEX_NAME", "INDEX_HOST"); ``` ```java Java {11} import io.pinecone.clients.Index; import io.pinecone.configs.PineconeConfig; import io.pinecone.configs.PineconeConnection; public class TargetIndexByHostExample { public static void main(String[] args) { PineconeConfig config = new PineconeConfig("YOUR_API_KEY"); config.setHost("INDEX_HOST"); PineconeConnection connection = new PineconeConnection(config); // For the Java SDK, you must specify both the index host and name. Index index = new Index(connection, "INDEX_NAME"); } } ``` ```go Go {21} package main import ( "context" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "INDEX_HOST", Namespace: "example-namespace"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host %v: %v", idx.Host, err) } } ``` ```csharp C# {5} using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); var index = pinecone.Index(host: "INDEX_HOST"); ``` ### Get an index host You can get the unique DNS host for an index from the Pinecone console or the Pinecone API. To get an index host from the Pinecone console: 1. Open the [Pinecone console](https://app.pinecone.io/organizations/-/projects). 2. Select the project containing the index. 3. Select the index. 4. Copy the URL under **HOST**. To get an index host from the Pinecone API, use the [`describe_index`](/reference/api/2024-10/control-plane/describe_index) operation, which returns the index host as the `host` value: ```Python Python from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key="YOUR_API_KEY") pc.describe_index(name="docs-example") # Response: # {'deletion_protection': 'disabled', # 'dimension': 1536, # 'host': 'docs-example-4zo0ijk.svc.us-east1-aws.pinecone.io', # 'metric': 'cosine', # 'name': 'docs-example', # 'spec': {'serverless': {'cloud': 'aws', 'region': 'us-east-1'}}, # 'status': {'ready': True, 'state': 'Ready'}} ``` ```JavaScript JavaScript import { Pinecone } from '@pinecone-database/pinecone'; const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }); await pc.describeIndex('docs-example'); // Response: // { // "name": "docs-example", // "dimension": 1536, // "metric": "cosine", // "host": "docs-example-4zo0ijk.svc.us-east1-aws.pinecone.io", // "deletionProtection": "disabled", // "spec": { // "serverless": { // "cloud": "aws", // "region": "us-east-1" // } // }, // "status": { // "ready": true, // "state": "Ready" // } // } ``` ```java Java import io.pinecone.clients.Pinecone; import org.openapitools.db_control.client.model.*; public class DescribeIndexExample { public static void main(String[] args) { Pinecone pc = new Pinecone.Builder("YOUR_API_KEY").build(); IndexModel indexModel = pc.describeIndex("docs-example"); System.out.println(indexModel); } } // Response: // class IndexModel { // name: docs-example-java // dimension: 1536 // metric: cosine // host: docs-example-4zo0ijk.svc.us-west2-aws.pinecone.io // deletionProtection: enabled // spec: class IndexModelSpec { // pod: null // serverless: class ServerlessSpec { // cloud: aws // region: us-east-1 // } // } // status: class IndexModelStatus { // ready: true // state: Ready // } // } ``` ```go Go package main import ( "context" "encoding/json" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func prettifyStruct(obj interface{}) string { bytes, _ := json.MarshalIndent(obj, "", " ") return string(bytes) } func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } idx, err := pc.DescribeIndex(ctx, "docs-example") if err != nil { log.Fatalf("Failed to describe index \"%v\": %v", idx.Name, err) } else { fmt.Printf("index: %v\n", prettifyStruct(idx)) } } // Response: // index: { // "name": "docs-example", // "dimension": 1536, // "host": "docs-example-govk0nt.svc.apw5-4e34-81fa.pinecone.io", // "metric": "cosine", // "deletion_protection": "disabled", // "spec": { // "serverless": { // "cloud": "aws", // "region": "us-east-1" // } // }, // "status": { // "ready": true, // "state": "Ready" // } // } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); var indexModel = await pinecone.DescribeIndexAsync("docs-example"); Console.WriteLine(indexModel); // Response: // { // "name": "docs-example", // "dimension": 1536, // "metric": "cosine", // "host": "docs-example-govk0nt.svc.aped-4627-b74a.pinecone.io", // "deletion_protection": "disabled", // "spec": { // "pod": null, // "serverless": { // "cloud": "aws", // "region": "us-east-1" // } // }, // "status": { // "ready": true, // "state": "Ready" // } // } ``` ```bash curl PINECONE_API_KEY="YOUR_API_KEY" curl -i -X GET "https://api.pinecone.io/indexes/docs-example-curl" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" # Response: # { # "name": "docs-example", # "metric": "cosine", # "dimension": 1536, # "status": { # "ready": true, # "state": "Ready" # }, # "host": "docs-example-4zo0ijk.svc.us-east1-aws.pinecone.io", # "spec": { # "serverless": { # "region": "us-east-1", # "cloud": "aws" # } # } # } ``` ## Target by index name This method is convenient for testing but is not recommended for production: ```Python Python from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key="YOUR_API_KEY") index = pc.Index("docs-example") ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone'; const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }); // For the Node.js SDK, you must specify both the index host and name. const index = pc.index('docs-example'); ``` ```java Java import io.pinecone.clients.Index; import io.pinecone.clients.Pinecone; public class GenerateEmbeddings { public static void main(String[] args) throws ApiException { Pinecone pc = new Pinecone.Builder("YOUR_API_KEY").build(); Index index = pc.getIndexConnection("docs-example"); } } ``` ```go Go // It is not possible to target an index by name in the Go SDK. // You must target an index by host. ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); var index = pinecone.Index("docs-example"); ``` # Update records Source: https://docs.pinecone.io/guides/manage-data/update-data This page shows you how to use the [`update`](/reference/api/2024-10/data-plane/update) operation to update parts of existing records in dense or sparse indexes. To update entire records, use the [`upsert`](/guides/index-data/upsert-data) operation instead. The `update` operation does not validate the existence of IDs within an index. If a non-existent ID is specified, no records are affected and a `200 OK` status is returned. ## Update dense vector values To update the dense vector value of a record in a [dense index](/guides/index-data/indexing-overview#dense-indexes), specify the `namespace`, record `id`, and the new dense vector `values`. The new dense vector values must have the same length as the existing dense vector values. In this example, assume you want to update the dense vector values of the following record in the `example-namespace` namespace: ``` ( id="id-3", values=[1.0, 2.0], metadata={"type": "doc", "genre": "drama"} ) ``` ```Python Python from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key="YOUR_API_KEY") # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index index = pc.Index(host="INDEX_HOST") index.update(id="id-3", values=[4.0, 2.0], namespace="example-namespace") ``` ```JavaScript JavaScript import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: "YOUR_API_KEY" }) // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index const index = pc.index("INDEX_NAME", "INDEX_HOST") await index.namespace('example-namespace').update({ id: 'id-3', values: [4.0, 2.0] }); ``` ```java Java import io.pinecone.clients.Index; import io.pinecone.configs.PineconeConfig; import io.pinecone.configs.PineconeConnection; import io.pinecone.proto.UpdateResponse; import java.util.Arrays; import java.util.List; public class UpdateExample { public static void main(String[] args) { PineconeConfig config = new PineconeConfig("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index config.setHost("INDEX_HOST"); PineconeConnection connection = new PineconeConnection(config); Index index = new Index(connection, "INDEX_NAME"); List values = Arrays.asList(4.0f, 2.0f); UpdateResponse updateResponse = index.update("id-3", values, null, "example-namespace", null, null); System.out.println(updateResponse); } } ``` ```go Go package main import ( "context" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "INDEX_HOST", Namespace: "example-namespace"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v", err) } id := "id-3" err = idxConnection.UpdateVector(ctx, &pinecone.UpdateVectorRequest{ Id: id, Values: []float32{4.0, 2.0}, }) if err != nil { log.Fatalf("Failed to update vector with ID %v: %v", id, err) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index var index = pinecone.Index(host: "INDEX_HOST"); var updateResponse = await index.UpdateAsync(new UpdateRequest { Id = "id-3", Namespace = "example-namespace", Values = new[] { 4.0f, 2.0f } }); ``` ```bash curl # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index PINECONE_API_KEY="YOUR_API_KEY" INDEX_HOST="INDEX_HOST" curl "https://$INDEX_HOST/vectors/update" \ -H "Api-Key: $PINECONE_API_KEY" \ -H 'Content-Type: application/json' \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "id": "id-3", "values": [ 4.0, 2.0 ], "namespace": "example-namespace" }' ``` After the update, the dense vector values are changed, but the metadata is unchanged: ``` ( id="id-3", values=[4.0, 2.0], metadata={"type": "doc", "genre": "drama"} ) ``` ## Update sparse vector values To update the sparse vector value of a record in a [sparse index](/guides/index-data/indexing-overview#sparse-indexes), specify the `namespace`, record `id`, and the new `sparse_values`. In this example, assume you are updating the sparse vector values of the following record in the `example-namespace` namespace: ``` ( id="id-3", sparse_values={"indices": [1, 5], "values": [0.5, 0.5]} ) ``` ```Python Python from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key="YOUR_API_KEY") # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index index = pc.Index(host="INDEX_HOST") index.update( id="id-3", sparse_values={"indices": [2, 6], "values": [0.5, 0.5]}, namespace="example-namespace" ) ``` ```JavaScript JavaScript import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: "YOUR_API_KEY" }) // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index const index = pc.index("INDEX_NAME", "INDEX_HOST") await index.namespace('example-namespace').update({ id: 'id-3', sparseValues: {'indices': [2, 6], 'values': [0.5, 0.5]}, }); ``` ```java Java import com.google.protobuf.Struct; import com.google.protobuf.Value; import io.pinecone.clients.Index; import io.pinecone.configs.PineconeConfig; import io.pinecone.configs.PineconeConnection; import io.pinecone.proto.UpdateResponse; import java.util.Arrays; import java.util.List; public class UpdateExample { public static void main(String[] args) { PineconeConfig config = new PineconeConfig("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index config.setHost("INDEX_HOST"); PineconeConnection connection = new PineconeConnection(config); Index index = new Index(connection, "INDEX_NAME"); List sparseIndices = Arrays.asList(2L, 6L); List sparseValues = Arrays.asList(0.5f, 0.5f); UpdateResponse updateResponse = index.update("id-3", null, null, "example-namespace", sparseIndices, sparseValues); System.out.println(updateResponse); } } ``` ```go Go package main import ( "context" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "INDEX_HOST", Namespace: "example-namespace"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v", err) } id := "id-3" sparseValues := pinecone.SparseValues{ Indices: []uint32{2, 6}, Values: []float32{0.5, 0.5}, } err = idxConnection.UpdateVector(ctx, &pinecone.UpdateVectorRequest{ Id: id, SparseValues: &sparseValues, }) if err != nil { log.Fatalf("Failed to update vector with ID %v: %v", id, err) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index var index = pinecone.Index(host: "INDEX_HOST"); var updateResponse = await index.UpdateAsync(new UpdateRequest { Id = "id-3", Namespace = "example-namespace", SparseValues = new SparseValues { Indices = [2, 6], Values = new[] { 0.5f, 0.5f } } }); ``` ```bash curl # The `POST` request below uses the unique endpoint for an index. # See https://docs.pinecone.io/guides/manage-data/target-an-index for details. PINECONE_API_KEY="YOUR_API_KEY" INDEX_HOST="INDEX_HOST" curl "https://$INDEX_HOST/vectors/update" \ -H "Api-Key: $PINECONE_API_KEY" \ -H 'Content-Type: application/json' \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "id": "id-3", "sparse_values": {"indices": [2, 6], "values": [0.5, 0.5]}, "namespace": "example-namespace" }' ``` After the update, the sparse value `indices` array is changed, but the rest of the record is unchanged: ``` ( id="id-3", sparse_values={"indices": [2, 6], "values": [0.5, 0.5]} ) ``` ## Update metadata values When updating metadata, only the specified metadata fields are modified, and if a specified metadata file does not exist, it is added. In this example, assume you are updating the metadata values of following record in the `example-namespace` namespace: ``` ( id="id-3", values=[4.0, 2.0], metadata={"type": "doc", "genre": "drama"} ) ``` ```Python Python from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key="YOUR_API_KEY") # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index index = pc.Index(host="INDEX_HOST") index.update( id="id-3", set_metadata={"type": "web", "new": True}, namespace="example-namespace" ) ``` ```JavaScript JavaScript import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: "YOUR_API_KEY" }) // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index const index = pc.index("INDEX_NAME", "INDEX_HOST") await index.namespace('example-namespace').update({ id: 'id-3', metadata: { type: "web", new: true, }, }); ``` ```java Java import com.google.protobuf.Struct; import com.google.protobuf.Value; import io.pinecone.clients.Index; import io.pinecone.configs.PineconeConfig; import io.pinecone.configs.PineconeConnection; import io.pinecone.proto.UpdateResponse; import java.util.Arrays; import java.util.List; public class UpdateExample { public static void main(String[] args) { PineconeConfig config = new PineconeConfig("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index config.setHost("INDEX_HOST"); PineconeConnection connection = new PineconeConnection(config); Index index = new Index(connection, "INDEX_NAME"); Struct metaData = Struct.newBuilder() .putFields("web", Value.newBuilder().setBoolValue(true).build()) .build(); UpdateResponse updateResponse = index.update("id-3", null, metaData, "example-namespace", null, null); System.out.println(updateResponse); } } ``` ```go Go package main import ( "context" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" "google.golang.org/protobuf/types/known/structpb" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "INDEX_HOST", Namespace: "example-namespace"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v", err) } id := "id-3" metadataMap := map[string]interface{}{ "type": "web", "new": True, } metadataFilter, err := structpb.NewStruct(metadataMap) if err != nil { log.Fatalf("Failed to create metadata map: %v", err) } err = idxConnection.UpdateVector(ctx, &pinecone.UpdateVectorRequest{ Id: id, Metadata: metadataFilter, }) if err != nil { log.Fatalf("Failed to update vector with ID %v: %v", id, err) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index var index = pinecone.Index(host: "INDEX_HOST"); var updateResponse = await index.UpdateAsync(new UpdateRequest { Id = "id-3", Namespace = "example-namespace", Values = new[] { 4.0f, 2.0f }, SetMetadata = new Metadata { ["type"] = new("web"), ["new"] = new(true) } }); ``` ```bash curl # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index PINECONE_API_KEY="YOUR_API_KEY" INDEX_HOST="INDEX_HOST" curl "https://$INDEX_HOST/vectors/update" \ -H "Api-Key: $PINECONE_API_KEY" \ -H 'Content-Type: application/json' \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "id": "id-3", "setMetadata": { "type": "web", "new": true }, "namespace": "example-namespace" }' ``` After the update, the `type` metadata field is `web`, the `new` property is added with the value `true`, and the `genre` property is unchanged: ``` ( id="id-3", values=[4.0, 2.0], metadata={"type": "web", "new": true, "genre": "drama"} ) ``` ## Update a combination of values To update an entire record, use the [`upsert`](/guides/index-data/upsert-data) operation instead. In this example, assume you are updating the dense vector values and one metadata value of the following record in the `example-namespace` namespace: ``` ( id="id-3", values=[4.0, 2.0], metadata={"type": "doc", "genre": "drama"} ) ``` ```Python Python from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key="YOUR_API_KEY") # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index index = pc.Index(host="INDEX_HOST") index.update( id="id-3", values=[5.0, 3.0], set_metadata={"genre": "comedy"}, namespace="example-namespace" ) ``` ```JavaScript JavaScript import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: "YOUR_API_KEY" }) // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index const index = pc.index("INDEX_NAME", "INDEX_HOST") await index.namespace('example-namespace').update({ id: 'id-3', values: [5.0, 3.0], metadata: { type: "comedy", }, }); ``` ```java Java import com.google.protobuf.Struct; import com.google.protobuf.Value; import io.pinecone.clients.Index; import io.pinecone.configs.PineconeConfig; import io.pinecone.configs.PineconeConnection; import io.pinecone.proto.UpdateResponse; import java.util.Arrays; import java.util.List; public class UpdateExample { public static void main(String[] args) { PineconeConfig config = new PineconeConfig("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index config.setHost("INDEX_HOST"); PineconeConnection connection = new PineconeConnection(config); Index index = new Index(connection, "INDEX_NAME"); List values = Arrays.asList(5.0f, 3.0f); Struct metaData = Struct.newBuilder() .putFields("type", Value.newBuilder().setStringValue("comedy").build()) .build(); UpdateResponse updateResponse = index.update("id-3", values, metaData, "example-namespace", null, null); System.out.println(updateResponse); } } ``` ```go Go package main import ( "context" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" "google.golang.org/protobuf/types/known/structpb" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "INDEX_HOST", Namespace: "example-namespace"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v", err) } id := "id-3" metadataMap := map[string]interface{}{ "genre": "comedy", } metadataFilter, err := structpb.NewStruct(metadataMap) if err != nil { log.Fatalf("Failed to create metadata map: %v", err) } err = idxConnection.UpdateVector(ctx, &pinecone.UpdateVectorRequest{ Id: id, Values: []float32{5.0, 3.0}, Metadata: metadataFilter, }) if err != nil { log.Fatalf("Failed to update vector with ID %v: %v", id, err) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index var index = pinecone.Index(host: "INDEX_HOST"); var updateResponse = await index.UpdateAsync(new UpdateRequest { Id = "id-3", Namespace = "example-namespace", Values = new[] { 5.0f, 3.0f }, SetMetadata = new Metadata { ["genre"] = new("comedy") } }); ``` ```bash curl # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index PINECONE_API_KEY="YOUR_API_KEY" INDEX_HOST="INDEX_HOST" # Update both values and metadata curl "https://$INDEX_HOST/vectors/update" \ -H "Api-Key: $PINECONE_API_KEY" \ -H 'Content-Type: application/json' \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "id": "id-3", "values": [5.0, 3.0], "setMetadata": {"type": "comedy"}, "namespace": "example-namespace" }' ``` After the update, the dense vector values and the `genre` metadata value are changed, but the `type` metadata value is unchanged: ``` ( id="id-3", values=[5.0, 2.0], metadata={"type": "doc", "genre": "comedy"} ) ``` ## Data freshness Pinecone is eventually consistent, so there can be a slight delay before new or changed records are visible to queries. You can view index stats to [check data freshness](/guides/index-data/check-data-freshness). # Integrate with Amazon S3 Source: https://docs.pinecone.io/guides/operations/integrations/integrate-with-amazon-s3 This feature is in [public preview](/release-notes/feature-availability) and available only on [Standard and Enterprise plans](https://www.pinecone.io/pricing/). To connect Pinecone to an Amazon S3 bucket, you need to create an IAM policy and role. Then, you can [configure audit logs](/guides/production/configure-audit-logs) or [add a storage integration](/guides/operations/integrations/manage-storage-integrations) to be used to [import data](/guides/index-data/import-data). ## Before you begin Ensure you have the following: * A [Pinecone account](https://app.pinecone.io/). * An [Amazon S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-buckets.html). ## 1. Create an IAM policy In the [AWS IAM console](https://console.aws.amazon.com/iam/home): 1. In the navigation pane, click **Policies**. 2. Click **Create policy**. 3. In **Select a service** section, select **S3**. 4. Select the following actions to allow: * `ListBucket`: Permission to list some or all of the objects in an S3 bucket. Required for [importing data](/guides/index-data/import-data) and [exporting audit logs](/guides/production/configure-audit-logs). * `GetObject`: Permission to retrieve objects from an S3 bucket. Required for [importing data](/guides/index-data/import-data). * `PutObject`: Permission to add an object to an S3 bucket. Required for [exporting audit logs](/guides/production/configure-audit-logs). 5. In the **Resources** section, select **Specific**. 6. For the **bucket**, specify the ARN of the bucket you created. For example: `arn:aws:s3:::example-bucket-name` 7. For the **object**, specify an object ARN as the target resource. For example: `arn:aws:s3:::example-bucket-name/*` 8. Click **Next**. 9. Specify the name of your policy. For example: "Pinecone-S3-Access". 10. Click **Create policy**. ## 2. Set up access using an IAM role In the [AWS IAM console](https://console.aws.amazon.com/iam/home): 1. In the navigation pane, click **Roles**. 2. Click **Create role**. 3. In the **Trusted entity type** section, select **AWS account**. 4. Select **Another AWS account**. 5. Enter the Pinecone AWS VPC account ID: `713131977538` 6. Click **Next**. 7. Select the [policy you created](#1-create-an-iam-policy). 8. Click **Next**. 9. Specify the role name. For example: "Pinecone". 10. Click **Create role**. 11. Click the role you created. 12. On the **Summary** page for the role, find the **ARN**. For example: `arn:aws:iam::123456789012:role/PineconeAccess` 13. Copy the **ARN**. You will need to enter the ARN into Pinecone later. ## Next steps After you have configured your IAM policy and role, you can use them to do the following: * [Add a storage integration](/guides/operations/integrations/manage-storage-integrations) and [import data](/guides/index-data/import-data) from your Amazon S3 bucket. * [Configure audit logs](/guides/production/configure-audit-logs) to export logs to your Amazon S3 bucket. # Manage storage integrations Source: https://docs.pinecone.io/guides/operations/integrations/manage-storage-integrations This feature is in [public preview](/release-notes/feature-availability) and available only on [Standard and Enterprise plans](https://www.pinecone.io/pricing/). You can connect your Pinecone project to cloud storage by creating a storage integration. This integration allows Pinecone to access the data in your cloud storage. Once your integration is set up, you can use it to import data from your cloud storage into a Pinecone index. For information on how to set up cloud storage for integration with Pinecone, see the following guides: * [Amazon S3](/guides/operations/integrations/integrate-with-amazon-s3) ## Add a storage integration After you have set up cloud storage for integration with Pinecone, you can add the storage integration through the [Pinecone console](https://app.pinecone.io/organizations/-/projects): 1. Select your project. 2. Go to [**Storage integrations**](https://app.pinecone.io/organizations/-/projects/-/storage). 3. Click **Add integration**. 4. Enter a unique integration name. 5. Select **Amazon S3**. 6. Enter the **ARN** of the [IAM role you created](/guides/operations/integrations/integrate-with-amazon-s3#2-set-up-access-using-an-iam-role). 7. Click **Add integration**. ## Update an integration To update information for a storage integration through the [Pinecone console](https://app.pinecone.io/organizations/-/projects), take the following steps: 1. Select your project. 2. Go to [**Manage > Storage integrations**](https://app.pinecone.io/organizations/-/projects/-/storage). 3. For the integration you want to update, click the *...* (Actions) icon. 4. Click **Manage**. 5. Update the integration details as needed. 6. Click **Add integration**. ## Delete an integration To delete a storage integration through the [Pinecone console](https://app.pinecone.io/organizations/-/projects), take the following steps: 1. Select your project. 2. Go to [**Manage > Storage integrations**](https://app.pinecone.io/organizations/-/projects/-/storage). 3. For the integration you want to update, click the *...* (Actions) icon. 4. Click **Delete**. 5. Enter the integration name. 6. Click **Confirm deletion**. # Local development with Pinecone Local Source: https://docs.pinecone.io/guides/operations/local-development Pinecone Local is an in-memory Pinecone emulator available as a Docker image. This page shows you how to use Pinecone Local to develop your applications locally without connecting to your Pinecone account or incurring usage or storage fees. Pinecone Local is not suitable for production. See [Limitations](#limitations) for details. This feature is in [public preview](/release-notes/feature-availability). ## Limitations Pinecone Local has the following limitations: * Pinecone Local is available in Docker only. * Pinecone Local is an in-memory emulator and is not suitable for production. Records loaded into Pinecone Local do not persist after it is stopped. * Pinecone Local does not authenticate client requests. API keys are ignored. * Max number of records per index: 100,000. Pinecone Local does not currently support the following features: * [Bulk import](/guides/index-data/import-data) * [Backups](/guides/manage-data/backups-overview) * [Collections](/guides/indexes/pods/understanding-collections) * [Pinecone Inference](/reference/api/introduction#inference) * [Pinecone Assistant](/guides/assistant/overview) ## 1. Start Pinecone Local You can configure Pinecone Local as an index emulator or database emulator: * **Index emulator** - This approach uses the `pinecone-index` Docker image to create and configure indexes on startup. This is recommended when you want to quickly experiment with reading and writing data without needing to manage the index lifecycle. With index emulation, you can only read and write data to the indexes created at startup. You cannot create new indexes, list indexes, or run other operations that do not involve reading and writing data. * **Database emulator** - This approach uses the `pinecone-local` Docker image to emulate Pinecone Database more broadly. This is recommended when you want to test your production app or manually create and manage indexes. ### Index emulator Make sure [Docker](https://docs.docker.com/get-docker/) is installed and running on your local machine. Create a `docker-compose.yaml` file that defines a service for each index that you want Pinecone Local to create on startup, including the `pinecone-index` Docker image, the host and port that the index with run on, and other index details: ```yaml services: dense-index: image: ghcr.io/pinecone-io/pinecone-index:latest container_name: dense-index environment: PORT: 5081 INDEX_TYPE: serverless VECTOR_TYPE: dense DIMENSION: 2 METRIC: cosine ports: - "5081:5081" platform: linux/amd64 sparse-index: image: ghcr.io/pinecone-io/pinecone-index:latest container_name: sparse-index environment: PORT: 5082 INDEX_TYPE: serverless VECTOR_TYPE: sparse DIMENSION: 0 METRIC: dotproduct ports: - "5082:5082" platform: linux/amd64 ``` For each index, update the environment variables as needed: * `PORT`: Specify the port number for the index to listen on. * `INDEX_TYPE`: Specify the type of Pinecone index to create. Accepted values: `serverless` or `pod`. * `VECTOR_TYPE`: Specify the [type of vectors](/guides/index-data/indexing-overview#indexes) you will store in the index. Accepted values: `dense` or `sparse`. Sparse is supported only with serverless indexes. * `DIMENSION`: Specify the dimension of vectors you will store in the index. For sparse indexes, this must be set to `0`. * `METRIC`: Specify the [distance metric](/guides/index-data/indexing-overview#distance-metrics) for calculating the similarity between vectors in the index. Accepted values for dense indexes: `cosine`, `euclidean`, or `dotproduct`. Accepted value for sparse indexes: `dotproduct`. To start Pinecone Local, run the following command: ```shell docker compose up -d ``` You'll see a message with details about each index. Make sure [Docker](https://docs.docker.com/get-docker/) is installed and running on your local machine. Download the latest `pinecone-index` Docker image: ```shell docker pull ghcr.io/pinecone-io/pinecone-index:latest ``` Start Pinecone Local with one or more indexes: ```shell docker run -d \ --name dense-index \ -e PORT=5081 \ -e INDEX_TYPE=serverless \ -e VECTOR_TYPE=dense \ -e DIMENSION=2 \ -e METRIC=cosine \ -p 5081:5081 \ --platform linux/amd64 \ ghcr.io/pinecone-io/pinecone-index:latest ``` ```shell docker run -d \ --name sparse-index \ -e PORT=5082 \ -e INDEX_TYPE=serverless \ -e VECTOR_TYPE=sparse \ -e DIMENSION=0 \ -e METRIC=dotproduct \ -p 5082:5082 \ --platform linux/amd64 \ ghcr.io/pinecone-io/pinecone-index:latest ``` For each index, update the environment variables as needed: * `PORT`: Specify the port number for the index to listen on. * `INDEX_TYPE`: Specify the type of Pinecone index to create. Accepted values: `serverless` or `pod`. * `VECTOR_TYPE`: Specify the [type of vectors](/guides/index-data/indexing-overview#indexes) you will store in the index. Accepted values: `dense` or `sparse`. Sparse is supported only with serverless indexes. * `DIMENSION`: Specify the dimension of vectors you will store in the index. For sparse indexes, this must be set to `0`. * `METRIC`: Specify the [distance metric](/guides/index-data/indexing-overview#distance-metrics) for calculating the similarity between vectors in the index. Accepted values for dense indexes: `cosine`, `euclidean`, or `dotproduct`. Accepted value for sparse indexes: `dotproduct`. ### Database emulator Make sure [Docker](https://docs.docker.com/get-docker/) is installed and running on your local machine. Create a `docker-compose.yaml` file that defines a service for Pinecone local, including the `pinecone-local` Docker image, the host and port that Pinecone Local will run on and the range of ports that will be available for indexes: ```yaml services: pinecone: image: ghcr.io/pinecone-io/pinecone-local:latest environment: PORT: 5080 PINECONE_HOST: localhost ports: - "5080-5090:5080-5090" platform: linux/amd64 ``` To start Pinecone Local, run the following command: ```shell docker compose up -d ``` You'll see a message with details about the Pinecone Local instance. Make sure [Docker](https://docs.docker.com/get-docker/) is installed and running on your local machine. Download the latest `pinecone-local` Docker image: ```shell docker pull ghcr.io/pinecone-io/pinecone-local:latest ``` Start Pinecone Local: ```shell docker run -d \ --name pinecone-local \ -e PORT=5080 \ -e PINECONE_HOST=localhost \ -p 5080-5090:5080-5090 \ --platform linux/amd64 \ ghcr.io/pinecone-io/pinecone-local:latest ``` This command defines the host and port that Pinecone Local will run on, as well as the range of ports that will be available for indexes. ## 2. Develop your app Running code against Pinecone Local is just like running code against your Pinecone account, with the following differences: * Pinecone Local does not authenticate client requests. API keys are ignored. * The latest version of Pinecone Local uses [Pinecone API version](/reference/api/versioning) `2025-01` and requires [Python SDK](/reference/python-sdk) `v6.x` or later, [Node.js SDK](/reference/node-sdk) `v5.x` or later, [Java SDK](/reference/java-sdk) `v4.x` or later, [Go SDK](/reference/go-sdk) `v3.x` or later, and [.NET SDK](/reference/dotnet-sdk) `v3.x` or later. Be sure to review the [limitations](#limitations) of Pinecone Local before using it for development or testing. **Example** The following example assumes that you have [started Pinecone Local without indexes](/guides/operations/local-development#database-emulator). It initializes a client, creates a [dense index](/guides/index-data/indexing-overview#dense-indexes) and a [sparse index](/guides/index-data/indexing-overview#sparse-indexes), upserts records into the indexes, checks their record counts, and queries the indexes. ```python Python from pinecone.grpc import PineconeGRPC, GRPCClientConfig from pinecone import ServerlessSpec # Initialize a client. # API key is required, but the value does not matter. # Host and port of the Pinecone Local instance # is required when starting without indexes. pc = PineconeGRPC( api_key="pclocal", host="http://localhost:5080" ) # Create two indexes, one dense and one sparse dense_index_name = "dense-index" sparse_index_name = "sparse-index" if not pc.has_index(dense_index_name): dense_index_model = pc.create_index( name=dense_index_name, vector_type="dense", dimension=2, metric="cosine", spec=ServerlessSpec(cloud="aws", region="us-east-1"), deletion_protection="disabled", tags={"environment": "development"} ) print("Dense index model:\n", dense_index_model) if not pc.has_index(sparse_index_name): sparse_index_model = pc.create_index( name=sparse_index_name, vector_type="sparse", metric="dotproduct", spec=ServerlessSpec(cloud="aws", region="us-east-1"), deletion_protection="disabled", tags={"environment": "development"} ) print("\nSparse index model:\n", sparse_index_model) # Target each index, disabling tls dense_index_host = pc.describe_index(name=dense_index_name).host dense_index = pc.Index(host=dense_index_host, grpc_config=GRPCClientConfig(secure=False)) sparse_index_host = pc.describe_index(name=sparse_index_name).host sparse_index = pc.Index(host=sparse_index_host, grpc_config=GRPCClientConfig(secure=False)) # Upsert records into the dense index dense_index.upsert( vectors=[ { "id": "vec1", "values": [1.0, -2.5], "metadata": {"genre": "drama"} }, { "id": "vec2", "values": [3.0, -2.0], "metadata": {"genre": "documentary"} }, { "id": "vec3", "values": [0.5, -1.5], "metadata": {"genre": "documentary"} } ], namespace="example-namespace" ) # Upsert records into the sparse index sparse_index.upsert( namespace="example-namespace", vectors=[ { "id": "vec1", "sparse_values": { "values": [1.7958984, 0.41577148, 2.828125, 2.8027344, 2.8691406, 1.6533203, 5.3671875, 1.3046875, 0.49780273, 0.5722656, 2.71875, 3.0820312, 2.5019531, 4.4414062, 3.3554688], "indices": [822745112, 1009084850, 1221765879, 1408993854, 1504846510, 1596856843, 1640781426, 1656251611, 1807131503, 2543655733, 2902766088, 2909307736, 3246437992, 3517203014, 3590924191] }, "metadata": { "chunk_text": "AAPL reported a year-over-year revenue increase, expecting stronger Q3 demand for its flagship phones.", "category": "technology", "quarter": "Q3" } }, { "id": "vec2", "sparse_values": { "values": [0.4362793, 3.3457031, 2.7714844, 3.0273438, 3.3164062, 5.6015625, 2.4863281, 0.38134766, 1.25, 2.9609375, 0.34179688, 1.4306641, 0.34375, 3.3613281, 1.4404297, 2.2558594, 2.2597656, 4.8710938, 0.5605469], "indices": [131900689, 592326839, 710158994, 838729363, 1304885087, 1640781426, 1690623792, 1807131503, 2066971792, 2428553208, 2548600401, 2577534050, 3162218338, 3319279674, 3343062801, 3476647774, 3485013322, 3517203014, 4283091697] }, "metadata": { "chunk_text": "Analysts suggest that AAPL'\''s upcoming Q4 product launch event might solidify its position in the premium smartphone market.", "category": "technology", "quarter": "Q4" } }, { "id": "vec3", "sparse_values": { "values": [2.6875, 4.2929688, 3.609375, 3.0722656, 2.1152344, 5.78125, 3.7460938, 3.7363281, 1.2695312, 3.4824219, 0.7207031, 0.0826416, 4.671875, 3.7011719, 2.796875, 0.61621094], "indices": [8661920, 350356213, 391213188, 554637446, 1024951234, 1640781426, 1780689102, 1799010313, 2194093370, 2632344667, 2641553256, 2779594451, 3517203014, 3543799498, 3837503950, 4283091697] }, "metadata": { "chunk_text": "AAPL'\''s strategic Q3 partnerships with semiconductor suppliers could mitigate component risks and stabilize iPhone production", "category": "technology", "quarter": "Q3" } } ] ) # Check the number of records in each index print("\nDense index stats:\n", dense_index.describe_index_stats()) print("\nSparse index stats:\n", sparse_index.describe_index_stats()) # Query the dense index with a metadata filter dense_response = dense_index.query( namespace="example-namespace", vector=[3.0, -2.0], filter={"genre": {"$eq": "documentary"}}, top_k=1, include_values=False, include_metadata=True ) print("\nDense query response:\n", dense_response) # Query the sparse index with a metadata filter sparse_response = sparse_index.query( namespace="example-namespace", sparse_vector={ "values": [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0], "indices": [767227209, 1640781426, 1690623792, 2021799277, 2152645940, 2295025838, 2443437770, 2779594451, 2956155693, 3476647774, 3818127854, 4283091697] }, filter={ "quarter": {"$eq": "Q4"} }, top_k=1, include_values=False, include_metadata=True ) print("/nSparse query response:\n", sparse_response) # Delete the indexes pc.delete_index(name=dense_index_name) pc.delete_index(name=sparse_index_name) ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone'; // Initialize a client. // API key is required, but the value does not matter. // Host and port of the Pinecone Local instance // is required when starting without indexes. const pc = new Pinecone({ apiKey: 'pclocal', controllerHostUrl: 'http://localhost:5080' }); // Create two indexes, one dense and one sparse const denseIndexName = 'dense-index'; const sparseIndexName = 'sparse-index'; const denseIndexModel = await pc.createIndex({ name: denseIndexName, vectorType: 'dense', dimension: 2, metric: 'cosine', spec: { serverless: { cloud: 'aws', region: 'us-east-1' } }, deletionProtection: 'disabled', tags: { environment: 'development' }, }); console.log('Dense index model:', denseIndexModel); const sparseIndexModel = await pc.createIndex({ name: sparseIndexName, vectorType: 'sparse', metric: 'dotproduct', spec: { serverless: { cloud: 'aws', region: 'us-east-1' } }, deletionProtection: 'disabled', tags: { environment: 'development' }, }); console.log('\nSparse index model:', sparseIndexModel); // Target each index const denseIndexHost = (await pc.describeIndex(denseIndexName)).host; const denseIndex = await pc.index(denseIndexName, 'http://' + denseIndexHost); const sparseIndexHost = (await pc.describeIndex(sparseIndexName)).host; const sparseIndex = await pc.index(sparseIndexName, 'http://' + sparseIndexHost); // Upsert records into the dense index await denseIndex.namespace('example-namespace').upsert([ { id: 'vec1', values: [1.0, -2.5], metadata: { genre: 'drama' }, }, { id: 'vec2', values: [3.0, -2.0], metadata: { genre: 'documentary' }, }, { id: 'vec3', values: [0.5, -1.5], metadata: { genre: 'documentary' }, } ]); // Upsert records into the sparse index await sparseIndex.namespace('example-namespace').upsert([ { id: 'vec1', sparseValues: { indices: [822745112, 1009084850, 1221765879, 1408993854, 1504846510, 1596856843, 1640781426, 1656251611, 1807131503, 2543655733, 2902766088, 2909307736, 3246437992, 3517203014, 3590924191], values: [1.7958984, 0.41577148, 2.828125, 2.8027344, 2.8691406, 1.6533203, 5.3671875, 1.3046875, 0.49780273, 0.5722656, 2.71875, 3.0820312, 2.5019531, 4.4414062, 3.3554688] }, metadata: { chunk_text: 'AAPL reported a year-over-year revenue increase, expecting stronger Q3 demand for its flagship phones.', category: 'technology', quarter: 'Q3' } }, { id: 'vec2', sparseValues: { indices: [131900689, 592326839, 710158994, 838729363, 1304885087, 1640781426, 1690623792, 1807131503, 2066971792, 2428553208, 2548600401, 2577534050, 3162218338, 3319279674, 3343062801, 3476647774, 3485013322, 3517203014, 4283091697], values: [0.4362793, 3.3457031, 2.7714844, 3.0273438, 3.3164062, 5.6015625, 2.4863281, 0.38134766, 1.25, 2.9609375, 0.34179688, 1.4306641, 0.34375, 3.3613281, 1.4404297, 2.2558594, 2.2597656, 4.8710938, 0.5605469] }, metadata: { chunk_text: "Analysts suggest that AAPL's upcoming Q4 product launch event might solidify its position in the premium smartphone market.", category: 'technology', quarter: 'Q4' } }, { id: 'vec3', sparseValues: { indices: [8661920, 350356213, 391213188, 554637446, 1024951234, 1640781426, 1780689102, 1799010313, 2194093370, 2632344667, 2641553256, 2779594451, 3517203014, 3543799498, 3837503950, 4283091697], values: [2.6875, 4.2929688, 3.609375, 3.0722656, 2.1152344, 5.78125, 3.7460938, 3.7363281, 1.2695312, 3.4824219, 0.7207031, 0.0826416, 4.671875, 3.7011719, 2.796875, 0.61621094] }, metadata: { chunk_text: "AAPL's strategic Q3 partnerships with semiconductor suppliers could mitigate component risks and stabilize iPhone production", category: 'technology', quarter: 'Q3' } } ]); // Check the number of records in each index console.log('\nDense index stats:', await denseIndex.describeIndexStats()); console.log('\nSparse index stats:', await sparseIndex.describeIndexStats()); // Query the dense index with a metadata filter const denseQueryResponse = await denseIndex.namespace('example-namespace').query({ vector: [3.0, -2.0], filter: { 'genre': {'$eq': 'documentary'} }, topK: 1, includeValues: false, includeMetadata: true, }); console.log('\nDense query response:', denseQueryResponse); const sparseQueryResponse = await sparseIndex.namespace('example-namespace').query({ sparseVector: { indices: [767227209, 1640781426, 1690623792, 2021799277, 2152645940, 2295025838, 2443437770, 2779594451, 2956155693, 3476647774, 3818127854, 4283091697], values: [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] }, topK: 1, includeValues: false, includeMetadata: true }); console.log('\nSparse query response:', sparseQueryResponse); // Delete the index await pc.deleteIndex(denseIndexName); await pc.deleteIndex(sparseIndexName); ``` ```java Java import com.google.protobuf.Struct; import com.google.protobuf.Value; import io.pinecone.clients.Index; import io.pinecone.clients.Pinecone; import io.pinecone.proto.DescribeIndexStatsResponse; import org.openapitools.db_control.client.model.DeletionProtection; import io.pinecone.unsigned_indices_model.QueryResponseWithUnsignedIndices; import java.util.*; public class PineconeLocalExample { public static void main(String[] args) { // Initialize a client. // API key is required, but the value does not matter. // When starting without indexes, disable TLS and // provide the host and port of the Pinecone Local instance. String host = "http://localhost:5080"; Pinecone pc = new Pinecone.Builder("pclocal") .withHost(host) .withTlsEnabled(false) .build(); // Create two indexes, one dense and one sparse String denseIndexName = "dense-index"; String sparseIndexName = "sparse-index"; HashMap tags = new HashMap<>(); tags.put("environment", "development"); pc.createServerlessIndex( denseIndexName, "cosine", 2, "aws", "us-east-1", DeletionProtection.DISABLED, tags ); pc.createSparseServelessIndex( sparseIndexName, "aws", "us-east-1", DeletionProtection.DISABLED, tags, "sparse" ); // Get index connection objects Index denseIndexConnection = pc.getIndexConnection(denseIndexName); Index sparseIndexConnection = pc.getIndexConnection(sparseIndexName); // Upsert records into the dense index Struct metaData1 = Struct.newBuilder() .putFields("genre", Value.newBuilder().setStringValue("drama").build()) .build(); Struct metaData2 = Struct.newBuilder() .putFields("genre", Value.newBuilder().setStringValue("documentary").build()) .build(); Struct metaData3 = Struct.newBuilder() .putFields("genre", Value.newBuilder().setStringValue("documentary").build()) .build(); denseIndexConnection.upsert("vec1", Arrays.asList(1.0f, -2.5f), null, null, metaData1, "example-namespace"); denseIndexConnection.upsert("vec2", Arrays.asList(3.0f, -2.0f), null, null, metaData2, "example-namespace"); denseIndexConnection.upsert("vec3", Arrays.asList(0.5f, -1.5f), null, null, metaData3, "example-namespace"); // Upsert records into the sparse index ArrayList indices1 = new ArrayList<>(Arrays.asList( 822745112L, 1009084850L, 1221765879L, 1408993854L, 1504846510L, 1596856843L, 1640781426L, 1656251611L, 1807131503L, 2543655733L, 2902766088L, 2909307736L, 3246437992L, 3517203014L, 3590924191L )); ArrayList values1 = new ArrayList<>(Arrays.asList( 1.7958984f, 0.41577148f, 2.828125f, 2.8027344f, 2.8691406f, 1.6533203f, 5.3671875f, 1.3046875f, 0.49780273f, 0.5722656f, 2.71875f, 3.0820312f, 2.5019531f, 4.4414062f, 3.3554688f )); Struct sparseMetaData1 = Struct.newBuilder() .putFields("chunk_text", Value.newBuilder().setStringValue("AAPL reported a year-over-year revenue increase, expecting stronger Q3 demand for its flagship phones.").build()) .putFields("category", Value.newBuilder().setStringValue("technology").build()) .putFields("quarter", Value.newBuilder().setStringValue("Q3").build()) .build(); ArrayList indices2 = new ArrayList<>(Arrays.asList( 131900689L, 592326839L, 710158994L, 838729363L, 1304885087L, 1640781426L, 1690623792L, 1807131503L, 2066971792L, 2428553208L, 2548600401L, 2577534050L, 3162218338L, 3319279674L, 3343062801L, 3476647774L, 3485013322L, 3517203014L, 4283091697L )); ArrayList values2 = new ArrayList<>(Arrays.asList( 0.4362793f, 3.3457031f, 2.7714844f, 3.0273438f, 3.3164062f, 5.6015625f, 2.4863281f, 0.38134766f, 1.25f, 2.9609375f, 0.34179688f, 1.4306641f, 0.34375f, 3.3613281f, 1.4404297f, 2.2558594f, 2.2597656f, 4.8710938f, 0.5605469f )); Struct sparseMetaData2 = Struct.newBuilder() .putFields("chunk_text", Value.newBuilder().setStringValue("Analysts suggest that AAPL'\\''s upcoming Q4 product launch event might solidify its position in the premium smartphone market.").build()) .putFields("category", Value.newBuilder().setStringValue("technology").build()) .putFields("quarter", Value.newBuilder().setStringValue("Q4").build()) .build(); ArrayList indices3 = new ArrayList<>(Arrays.asList( 8661920L, 350356213L, 391213188L, 554637446L, 1024951234L, 1640781426L, 1780689102L, 1799010313L, 2194093370L, 2632344667L, 2641553256L, 2779594451L, 3517203014L, 3543799498L, 3837503950L, 4283091697L )); ArrayList values3 = new ArrayList<>(Arrays.asList( 2.6875f, 4.2929688f, 3.609375f, 3.0722656f, 2.1152344f, 5.78125f, 3.7460938f, 3.7363281f, 1.2695312f, 3.4824219f, 0.7207031f, 0.0826416f, 4.671875f, 3.7011719f, 2.796875f, 0.61621094f )); Struct sparseMetaData3 = Struct.newBuilder() .putFields("chunk_text", Value.newBuilder().setStringValue("AAPL'\\''s strategic Q3 partnerships with semiconductor suppliers could mitigate component risks and stabilize iPhone production").build()) .putFields("category", Value.newBuilder().setStringValue("technology").build()) .putFields("quarter", Value.newBuilder().setStringValue("Q3").build()) .build(); sparseIndexConnection.upsert("vec1", Collections.emptyList(), indices1, values1, sparseMetaData1, "example-namespace"); sparseIndexConnection.upsert("vec2", Collections.emptyList(), indices2, values2, sparseMetaData2, "example-namespace"); sparseIndexConnection.upsert("vec3", Collections.emptyList(), indices3, values3, sparseMetaData3, "example-namespace"); // Check the number of records each the index DescribeIndexStatsResponse denseIndexStatsResponse = denseIndexConnection.describeIndexStats(null); System.out.println("Dense index stats:"); System.out.println(denseIndexStatsResponse); DescribeIndexStatsResponse sparseIndexStatsResponse = sparseIndexConnection.describeIndexStats(null); System.out.println("Sparse index stats:"); System.out.println(sparseIndexStatsResponse); // Query the dense index with a metadata filter List queryVector = Arrays.asList(1.0f, 1.5f); QueryResponseWithUnsignedIndices denseQueryResponse = denseIndexConnection.query(1, queryVector, null, null, null, "example-namespace", null, false, true); System.out.println("Dense query response:"); System.out.println(denseQueryResponse); // Query the sparse index with a metadata filter List sparseIndices = Arrays.asList( 767227209L, 1640781426L, 1690623792L, 2021799277L, 2152645940L, 2295025838L, 2443437770L, 2779594451L, 2956155693L, 3476647774L, 3818127854L, 428309169L); List sparseValues = Arrays.asList( 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f); QueryResponseWithUnsignedIndices sparseQueryResponse = sparseIndexConnection.query(1, null, sparseIndices, sparseValues, null, "example-namespace", null, false, true); System.out.println("Sparse query response:"); System.out.println(sparseQueryResponse); // Delete the indexes pc.deleteIndex(denseIndexName); pc.deleteIndex(sparseIndexName); } } ``` ```go Go package main import ( "context" "encoding/json" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" "google.golang.org/protobuf/types/known/structpb" ) func prettifyStruct(obj interface{}) string { bytes, _ := json.MarshalIndent(obj, "", " ") return string(bytes) } func main() { ctx := context.Background() // Initialize a client. // No API key is required. // Host and port of the Pinecone Local instance // is required when starting without indexes. pc, err := pinecone.NewClientBase(pinecone.NewClientBaseParams{ Host: "http://localhost:5080", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // Create two indexes, one dense and one sparse denseIndexName := "dense-index" denseVectorType := "dense" dimension := int32(2) denseMetric := pinecone.Cosine deletionProtection := pinecone.DeletionProtectionDisabled denseIdx, err := pc.CreateServerlessIndex(ctx, &pinecone.CreateServerlessIndexRequest{ Name: denseIndexName, VectorType: &denseVectorType, Dimension: &dimension, Metric: &denseMetric, Cloud: pinecone.Aws, Region: "us-east-1", DeletionProtection: &deletionProtection, Tags: &pinecone.IndexTags{"environment": "development"}, }) if err != nil { log.Fatalf("Failed to create serverless index: %v", denseIdx.Name) } else { fmt.Printf("Successfully created serverless index: %v\n", denseIdx.Name) } sparseIndexName := "sparse-index" sparseVectorType := "sparse" sparseMetric := pinecone.Dotproduct sparseIdx, err := pc.CreateServerlessIndex(ctx, &pinecone.CreateServerlessIndexRequest{ Name: sparseIndexName, VectorType: &sparseVectorType, Metric: &sparseMetric, Cloud: pinecone.Aws, Region: "us-east-1", DeletionProtection: &deletionProtection, Tags: &pinecone.IndexTags{"environment": "development"}, }) if err != nil { log.Fatalf("Failed to create serverless index: %v", sparseIdx.Name) } else { fmt.Printf("\nSuccessfully created serverless index: %v\n", sparseIdx.Name) } // Get the index hosts denseIdxModel, err := pc.DescribeIndex(ctx, denseIndexName) if err != nil { log.Fatalf("Failed to describe index \"%v\": %v", denseIndexName, err) } sparseIdxModel, err := pc.DescribeIndex(ctx, sparseIndexName) if err != nil { log.Fatalf("Failed to describe index \"%v\": %v", sparseIndexName, err) } // Target the indexes. // Make sure to prefix the hosts with http:// to let the SDK know to disable tls. denseIdxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "http://" + denseIdxModel.Host, Namespace: "example-namespace"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v", err) } sparseIdxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "http://" + sparseIdxModel.Host, Namespace: "example-namespace"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v", err) } // Upsert records into the dense index denseMetadataMap1 := map[string]interface{}{ "genre": "drama", } denseMetadata1, err := structpb.NewStruct(denseMetadataMap1) if err != nil { log.Fatalf("Failed to create metadata map: %v", err) } denseMetadataMap2 := map[string]interface{}{ "genre": "documentary", } denseMetadata2, err := structpb.NewStruct(denseMetadataMap2) if err != nil { log.Fatalf("Failed to create metadata map: %v", err) } denseMetadataMap3 := map[string]interface{}{ "genre": "documentary", } denseMetadata3, err := structpb.NewStruct(denseMetadataMap3) if err != nil { log.Fatalf("Failed to create metadata map: %v", err) } denseVectors := []*pinecone.Vector{ { Id: "vec1", Values: &[]float32{1.0, -2.5}, Metadata: denseMetadata1, }, { Id: "vec2", Values: &[]float32{3.0, -2.0}, Metadata: denseMetadata2, }, { Id: "vec3", Values: &[]float32{0.5, -1.5}, Metadata: denseMetadata3, }, } denseCount, err := denseIdxConnection.UpsertVectors(ctx, denseVectors) if err != nil { log.Fatalf("Failed to upsert vectors: %v", err) } else { fmt.Printf("\nSuccessfully upserted %d vector(s)!\n", denseCount) } // Upsert records into the sparse index sparseValues1 := pinecone.SparseValues{ Indices: []uint32{822745112, 1009084850, 1221765879, 1408993854, 1504846510, 1596856843, 1640781426, 1656251611, 1807131503, 2543655733, 2902766088, 2909307736, 3246437992, 3517203014, 3590924191}, Values: []float32{1.7958984, 0.41577148, 2.828125, 2.8027344, 2.8691406, 1.6533203, 5.3671875, 1.3046875, 0.49780273, 0.5722656, 2.71875, 3.0820312, 2.5019531, 4.4414062, 3.3554688}, } sparseMetadataMap1 := map[string]interface{}{ "chunk_text": "AAPL reported a year-over-year revenue increase, expecting stronger Q3 demand for its flagship phones", "category": "technology", "quarter": "Q3", } sparseMetadata1, err := structpb.NewStruct(sparseMetadataMap1) if err != nil { log.Fatalf("Failed to create metadata map: %v", err) } sparseValues2 := pinecone.SparseValues{ Indices: []uint32{131900689, 592326839, 710158994, 838729363, 1304885087, 1640781426, 1690623792, 1807131503, 2066971792, 2428553208, 2548600401, 2577534050, 3162218338, 3319279674, 3343062801, 3476647774, 3485013322, 3517203014, 4283091697}, Values: []float32{0.4362793, 3.3457031, 2.7714844, 3.0273438, 3.3164062, 5.6015625, 2.4863281, 0.38134766, 1.25, 2.9609375, 0.34179688, 1.4306641, 0.34375, 3.3613281, 1.4404297, 2.2558594, 2.2597656, 4.8710938, 0.560546}, } sparseMetadataMap2 := map[string]interface{}{ "chunk_text": "Analysts suggest that AAPL's upcoming Q4 product launch event might solidify its position in the premium smartphone market.", "category": "technology", "quarter": "Q4", } sparseMetadata2, err := structpb.NewStruct(sparseMetadataMap2) if err != nil { log.Fatalf("Failed to create metadata map: %v", err) } sparseValues3 := pinecone.SparseValues{ Indices: []uint32{8661920, 350356213, 391213188, 554637446, 1024951234, 1640781426, 1780689102, 1799010313, 2194093370, 2632344667, 2641553256, 2779594451, 3517203014, 3543799498, 3837503950, 4283091697}, Values: []float32{2.6875, 4.2929688, 3.609375, 3.0722656, 2.1152344, 5.78125, 3.7460938, 3.7363281, 1.2695312, 3.4824219, 0.7207031, 0.0826416, 4.671875, 3.7011719, 2.796875, 0.61621094}, } sparseMetadataMap3 := map[string]interface{}{ "chunk_text": "AAPL's strategic Q3 partnerships with semiconductor suppliers could mitigate component risks and stabilize iPhone production", "category": "technology", "quarter": "Q3", } sparseMetadata3, err := structpb.NewStruct(sparseMetadataMap3) if err != nil { log.Fatalf("Failed to create metadata map: %v", err) } sparseVectors := []*pinecone.Vector{ { Id: "vec1", SparseValues: &sparseValues1, Metadata: sparseMetadata1, }, { Id: "vec2", SparseValues: &sparseValues2, Metadata: sparseMetadata2, }, { Id: "vec3", SparseValues: &sparseValues3, Metadata: sparseMetadata3, }, } sparseCount, err := sparseIdxConnection.UpsertVectors(ctx, sparseVectors) if err != nil { log.Fatalf("Failed to upsert vectors: %v", err) } else { fmt.Printf("\nSuccessfully upserted %d vector(s)!\n", sparseCount) } // Check the number of records in each index denseStats, err := denseIdxConnection.DescribeIndexStats(ctx) if err != nil { log.Fatalf("Failed to describe index: %v", err) } else { fmt.Printf("\nDense index stats: %+v\n", prettifyStruct(*denseStats)) } sparseStats, err := sparseIdxConnection.DescribeIndexStats(ctx) if err != nil { log.Fatalf("Failed to describe index: %v", err) } else { fmt.Printf("\nSparse index stats: %+v\n", prettifyStruct(*sparseStats)) } // Query the dense index with a metadata filter queryVector := []float32{3.0, -2.0} queryMetadataMap := map[string]interface{}{ "genre": map[string]interface{}{ "$eq": "documentary", }, } metadataFilter, err := structpb.NewStruct(queryMetadataMap) if err != nil { log.Fatalf("Failed to create metadata map: %v", err) } denseRes, err := denseIdxConnection.QueryByVectorValues(ctx, &pinecone.QueryByVectorValuesRequest{ Vector: queryVector, TopK: 1, MetadataFilter: metadataFilter, IncludeValues: false, IncludeMetadata: true, }) if err != nil { log.Fatalf("Error encountered when querying by vector: %v", err) } else { fmt.Printf("\nDense query response: %v\n", prettifyStruct(denseRes)) } // Query the sparse index with a metadata filter sparseValues := pinecone.SparseValues{ Indices: []uint32{767227209, 1640781426, 1690623792, 2021799277, 2152645940, 2295025838, 2443437770, 2779594451, 2956155693, 3476647774, 3818127854, 4283091697}, Values: []float32{1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0}, } sparseRes, err := sparseIdxConnection.QueryByVectorValues(ctx, &pinecone.QueryByVectorValuesRequest{ SparseValues: &sparseValues, TopK: 1, IncludeValues: false, IncludeMetadata: true, }) if err != nil { log.Fatalf("Error encountered when querying by vector: %v", err) } else { fmt.Printf("\nSparse query response: %v\n", prettifyStruct(sparseRes)) } // Delete the indexes err = pc.DeleteIndex(ctx, denseIndexName) if err != nil { log.Fatalf("Failed to delete index: %v", err) } else { fmt.Printf("\nIndex \"%v\" deleted successfully\n", denseIndexName) } err = pc.DeleteIndex(ctx, sparseIndexName) if err != nil { log.Fatalf("Failed to delete index: %v", err) } else { fmt.Printf("\nIndex \"%v\" deleted successfully\n", sparseIndexName) } } ``` ```csharp C# using Pinecone; // Initialize a client. // API key is required, but the value does not matter. // When starting without indexes, disable TLS and // provide the host and port of the Pinecone Local instance. var pc = new PineconeClient("pclocal", new ClientOptions { BaseUrl = "http://localhost:5080", IsTlsEnabled = false } ); // Create two indexes, one dense and one sparse var denseIndexName = "dense-index"; var sparseIndexName = "sparse-index"; var createDenseIndexRequest = await pc.CreateIndexAsync(new CreateIndexRequest { Name = denseIndexName, VectorType = VectorType.Dense, Dimension = 2, Metric = MetricType.Cosine, Spec = new ServerlessIndexSpec { Serverless = new ServerlessSpec { Cloud = ServerlessSpecCloud.Aws, Region = "us-east-1" } }, DeletionProtection = DeletionProtection.Disabled, Tags = new Dictionary { { "environment", "development" } } }); Console.WriteLine("Dense index model:" + createDenseIndexRequest); var createSparseIndexRequest = await pc.CreateIndexAsync(new CreateIndexRequest { Name = sparseIndexName, VectorType = VectorType.Sparse, Metric = MetricType.Dotproduct, Spec = new ServerlessIndexSpec { Serverless = new ServerlessSpec { Cloud = ServerlessSpecCloud.Aws, Region = "us-east-1" } }, DeletionProtection = DeletionProtection.Disabled, Tags = new Dictionary { { "environment", "development" } } }); Console.WriteLine("\nSparse index model:" + createSparseIndexRequest); // Target the indexes var denseIndex = pc.Index(denseIndexName); var sparseIndex = pc.Index(sparseIndexName); // Upsert records into the dense index var denseUpsertResponse = await denseIndex.UpsertAsync(new UpsertRequest() { Namespace = "example-namespace", Vectors = new List { new Vector { Id = "vec1", Values = new ReadOnlyMemory([1.0f, -2.5f]), Metadata = new Metadata { ["genre"] = new("drama"), }, }, new Vector { Id = "vec2", Values = new ReadOnlyMemory([3.0f, -2.0f]), Metadata = new Metadata { ["genre"] = new("documentary"), }, }, new Vector { Id = "vec3", Values = new ReadOnlyMemory([0.5f, -1.5f]), Metadata = new Metadata { ["genre"] = new("documentary"), } } } }); Console.WriteLine($"\nUpserted {denseUpsertResponse.UpsertedCount} dense vectors"); // Upsert records into the sparse index var sparseVector1 = new Vector { Id = "vec1", SparseValues = new SparseValues { Indices = new uint[] { 822745112, 1009084850, 1221765879, 1408993854, 1504846510, 1596856843, 1640781426, 1656251611, 1807131503, 2543655733, 2902766088, 2909307736, 3246437992, 3517203014, 3590924191 }, Values = new ReadOnlyMemory([1.7958984f, 0.41577148f, 2.828125f, 2.8027344f, 2.8691406f, 1.6533203f, 5.3671875f, 1.3046875f, 0.49780273f, 0.5722656f, 2.71875f, 3.0820312f, 2.5019531f, 4.4414062f, 3.3554688f]) }, Metadata = new Metadata { ["chunk_text"] = new("AAPL reported a year-over-year revenue increase, expecting stronger Q3 demand for its flagship phones."), ["category"] = new("technology"), ["quarter"] = new("Q3"), }, }; var sparseVector2 = new Vector { Id = "vec2", SparseValues = new SparseValues { Indices = new uint[] { 131900689, 592326839, 710158994, 838729363, 1304885087, 1640781426, 1690623792, 1807131503, 2066971792, 2428553208, 2548600401, 2577534050, 3162218338, 3319279674, 3343062801, 3476647774, 3485013322, 3517203014, 4283091697 }, Values = new ReadOnlyMemory([0.4362793f, 3.3457031f, 2.7714844f, 3.0273438f, 3.3164062f, 5.6015625f, 2.4863281f, 0.38134766f, 1.25f, 2.9609375f, 0.34179688f, 1.4306641f, 0.34375f, 3.3613281f, 1.4404297f, 2.2558594f, 2.2597656f, 4.8710938f, 0.5605469f]) }, Metadata = new Metadata { ["chunk_text"] = new("Analysts suggest that AAPL'\''s upcoming Q4 product launch event might solidify its position in the premium smartphone market."), ["category"] = new("technology"), ["quarter"] = new("Q4"), }, }; var sparseVector3 = new Vector { Id = "vec3", SparseValues = new SparseValues { Indices = new uint[] { 8661920, 350356213, 391213188, 554637446, 1024951234, 1640781426, 1780689102, 1799010313, 2194093370, 2632344667, 2641553256, 2779594451, 3517203014, 3543799498, 3837503950, 4283091697 }, Values = new ReadOnlyMemory([2.6875f, 4.2929688f, 3.609375f, 3.0722656f, 2.1152344f, 5.78125f, 3.7460938f, 3.7363281f, 1.2695312f, 3.4824219f, 0.7207031f, 0.0826416f, 4.671875f, 3.7011719f, 2.796875f, 0.61621094f]) }, Metadata = new Metadata { ["chunk_text"] = new("AAPL'\''s strategic Q3 partnerships with semiconductor suppliers could mitigate component risks and stabilize iPhone production"), ["category"] = new("technology"), ["quarter"] = new("Q3"), }, }; var sparseUpsertResponse = await sparseIndex.UpsertAsync(new UpsertRequest { Vectors = new List { sparseVector1, sparseVector2, sparseVector3 }, Namespace = "example-namespace" }); Console.WriteLine($"\nUpserted {sparseUpsertResponse.UpsertedCount} sparse vectors"); // Check the number of records in each index var denseIndexStatsResponse = await denseIndex.DescribeIndexStatsAsync(new DescribeIndexStatsRequest()); Console.WriteLine("\nDense index stats:" + denseIndexStatsResponse); var sparseIndexStatsResponse = await sparseIndex.DescribeIndexStatsAsync(new DescribeIndexStatsRequest()); Console.WriteLine("\nSparse index stats:" + sparseIndexStatsResponse); // Query the dense index with a metadata filter var denseQueryResponse = await denseIndex.QueryAsync(new QueryRequest { Vector = new ReadOnlyMemory([3.0f, -2.0f]), TopK = 1, Namespace = "example-namespace", Filter = new Metadata { ["genre"] = new Metadata { ["$eq"] = "documentary", } }, IncludeValues = false, IncludeMetadata = true }); Console.WriteLine("\nDense query response:" + denseQueryResponse); // Query the sparse index with a metadata filter var sparseQueryResponse = await sparseIndex.QueryAsync(new QueryRequest { Namespace = "example-namespace", TopK = 1, SparseVector = new SparseValues { Indices = [767227209, 1640781426, 1690623792, 2021799277, 2152645940, 2295025838, 2443437770, 2779594451, 2956155693, 3476647774, 3818127854, 4283091697], Values = new[] { 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f }, }, Filter = new Metadata { ["quarter"] = new Metadata { ["$eq"] = "Q4", } }, IncludeValues = false, IncludeMetadata = true }); Console.WriteLine("\nSparse query response:" + sparseQueryResponse); // Delete the indexes await pc.DeleteIndexAsync(denseIndexName); await pc.DeleteIndexAsync(sparseIndexName); ``` {/* ```rust Rust use pinecone_sdk::models::{Cloud, DeletionProtection, Kind, Metadata, Metric, Value, Vector, WaitPolicy}; use pinecone_sdk::pinecone::PineconeClientConfig; use pinecone_sdk::utils::errors::PineconeError; use std::collections::BTreeMap; #[tokio::main] async fn main() -> Result<(), PineconeError> { // Initialize a client. // No API key is required. // Host and port of the Pinecone Local instance // is required when starting without indexes. let config = PineconeClientConfig { api_key: Some(String::default()), control_plane_host: Some("http://localhost:5080".to_string()), ..Default::default() }; let pinecone = config.client()?; // Create an index let index_model = pinecone.create_serverless_index( "docs-example", // Name of the index 2, // Dimension of the vectors Metric::Cosine, // Distance metric Cloud::Aws, // Cloud provider "us-east-1", // Region DeletionProtection::Disabled, // Deletion protection WaitPolicy::NoWait, // Timeout ).await?; let index_host = format!("http://{}", index_model.host); // Target the index let mut index = pinecone.index(&index_host).await?; // Upsert records into the index let vectors = [Vector { id: "vec1".to_string(), values: vec![1.0, -2.5], sparse_values: None, metadata: Some(Metadata { fields: vec![( "genre".to_string(), Value { kind: Some(Kind::StringValue("drama".to_string())), }, )] .into_iter() .collect(), }), }, Vector { id: "vec2".to_string(), values: vec![3.0, -2.0], sparse_values: None, metadata: Some(Metadata { fields: vec![( "genre".to_string(), Value { kind: Some(Kind::StringValue("documentary".to_string())), }, )] .into_iter() .collect(), }), }, Vector { id: "vec3".to_string(), values: vec![0.5, -1.5], sparse_values: None, metadata: Some(Metadata { fields: vec![( "genre".to_string(), Value { kind: Some(Kind::StringValue("documentary".to_string())), }, )] .into_iter() .collect(), }), }]; index.upsert(&vectors, &"example-namespace".into()).await?; // Check the number of records in the index let index_stats = index.describe_index_stats(None).await?; println!("Index stats:\n{:?}", index_stats); // Query the index with a metadata filter let vector = vec![3.0, -2.0]; let mut fields = BTreeMap::new(); let kind = Some(Kind::StringValue("documentary".to_string())); fields.insert("genre".to_string(), Value { kind }); let query_response = index.query_by_value( vector, None, 3, &"example-namespace".into(), Some(Metadata { fields }), None, Some(true), ).await?; println!("Query response:\n{:?}", query_response); // Delete the index pinecone.delete_index(&index_model.name).await?; Ok(()) } ``` */} ```shell curl PINECONE_LOCAL_HOST="localhost:5080" DENSE_INDEX_HOST="localhost:5081" SPARSE_INDEX_HOST="localhost:5082" # Create two indexes, one dense and one sparse curl -X POST "http://$PINECONE_LOCAL_HOST/indexes" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "X-Pinecone-API-Version: 2025-01" \ -d '{ "name": "dense-index", "vector_type": "dense", "dimension": 2, "metric": "cosine", "spec": { "serverless": { "cloud": "aws", "region": "us-east-1" } }, "tags": { "environment": "development" }, "deletion_protection": "disabled" }' curl -X POST "http://$PINECONE_LOCAL_HOST/indexes" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "X-Pinecone-API-Version: 2025-01" \ -d '{ "name": "sparse-index", "vector_type": "sparse", "metric": "dotproduct", "spec": { "serverless": { "cloud": "aws", "region": "us-east-1" } }, "tags": { "environment": "development" }, "deletion_protection": "disabled" }' # Upsert records into the dense index curl -X POST "http://$DENSE_INDEX_HOST/vectors/upsert" \ -H "Content-Type: application/json" \ -H "X-Pinecone-API-Version: 2025-01" \ -d '{ "namespace": "example-namespace", "vectors": [ { "id": "vec1", "values": [1.0, -2.5], "metadata": {"genre": "drama"} }, { "id": "vec2", "values": [3.0, -2.0], "metadata": {"genre": "documentary"} }, { "id": "vec3", "values": [0.5, -1.5], "metadata": {"genre": "documentary"} } ] }' # Upsert records into the sparse index curl -X POST "http://$SPARSE_INDEX_HOST/vectors/upsert" \ -H "Content-Type: application/json" \ -H "X-Pinecone-API-Version: 2025-01" \ -d '{ "namespace": "example-namespace", "vectors": [ { "id": "vec1", "sparseValues": { "values": [1.7958984, 0.41577148, 2.828125, 2.8027344, 2.8691406, 1.6533203, 5.3671875, 1.3046875, 0.49780273, 0.5722656, 2.71875, 3.0820312, 2.5019531, 4.4414062, 3.3554688], "indices": [822745112, 1009084850, 1221765879, 1408993854, 1504846510, 1596856843, 1640781426, 1656251611, 1807131503, 2543655733, 2902766088, 2909307736, 3246437992, 3517203014, 3590924191] }, "metadata": { "chunk_text": "AAPL reported a year-over-year revenue increase, expecting stronger Q3 demand for its flagship phones.", "category": "technology", "quarter": "Q3" } }, { "id": "vec2", "sparseValues": { "values": [0.4362793, 3.3457031, 2.7714844, 3.0273438, 3.3164062, 5.6015625, 2.4863281, 0.38134766, 1.25, 2.9609375, 0.34179688, 1.4306641, 0.34375, 3.3613281, 1.4404297, 2.2558594, 2.2597656, 4.8710938, 0.5605469], "indices": [131900689, 592326839, 710158994, 838729363, 1304885087, 1640781426, 1690623792, 1807131503, 2066971792, 2428553208, 2548600401, 2577534050, 3162218338, 3319279674, 3343062801, 3476647774, 3485013322, 3517203014, 4283091697] }, "metadata": { "chunk_text": "Analysts suggest that AAPL'\''s upcoming Q4 product launch event might solidify its position in the premium smartphone market.", "category": "technology", "quarter": "Q4" } }, { "id": "vec3", "sparseValues": { "values": [2.6875, 4.2929688, 3.609375, 3.0722656, 2.1152344, 5.78125, 3.7460938, 3.7363281, 1.2695312, 3.4824219, 0.7207031, 0.0826416, 4.671875, 3.7011719, 2.796875, 0.61621094], "indices": [8661920, 350356213, 391213188, 554637446, 1024951234, 1640781426, 1780689102, 1799010313, 2194093370, 2632344667, 2641553256, 2779594451, 3517203014, 3543799498, 3837503950, 4283091697] }, "metadata": { "chunk_text": "AAPL'\''s strategic Q3 partnerships with semiconductor suppliers could mitigate component risks and stabilize iPhone production", "category": "technology", "quarter": "Q3" } } ] }' # Check the number of records in each index curl -X POST "http://$DENSE_INDEX_HOST/describe_index_stats" \ -H "Content-Type: application/json" \ -H "X-Pinecone-API-Version: 2025-01" \ -d '{}' curl -X POST "http://$SPARSE_INDEX_HOST/describe_index_stats" \ -H "Content-Type: application/json" \ -H "X-Pinecone-API-Version: 2025-01" \ -d '{}' # Query the dense index with a metadata filter curl "http://$DENSE_INDEX_HOST/query" \ -H "Content-Type: application/json" \ -H "X-Pinecone-API-Version: 2025-01" \ -d '{ "vector": [3.0, -2.0], "filter": {"genre": {"$eq": "documentary"}}, "topK": 1, "includeMetadata": true, "includeValues": false, "namespace": "example-namespace" }' # Query the sparse index with a metadata filter curl "http://$SPARSE_INDEX_HOST/query" \ -H "Content-Type: application/json" \ -H "X-Pinecone-API-Version: 2025-01" \ -d '{ "sparseVector": { "values": [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0], "indices": [767227209, 1640781426, 1690623792, 2021799277, 2152645940, 2295025838, 2443437770, 2779594451, 2956155693, 3476647774, 3818127854, 4283091697] }, "filter": {"quarter": {"$eq": "Q4"}}, "namespace": "example-namespace", "topK": 1, "includeMetadata": true, "includeValues": false }' # Delete the index curl -X DELETE "http://$PINECONE_LOCAL_HOST/indexes/dense-index" \ -H "X-Pinecone-API-Version: 2025-01" curl -X DELETE "http://$PINECONE_LOCAL_HOST/indexes/sparse-index" \ -H "X-Pinecone-API-Version: 2025-01" ``` ## 3. Stop Pinecone Local Pinecone Local is an in-memory emulator. Records loaded into Pinecone Local do not persist after Pinecone Local is stopped. To stop and remove the resources for Pinecone Local, run the following command: ```shell Docker Compose docker compose down ``` ```shell Docker CLI # If you started Pinecone Local with indexes: docker stop dense-index sparse-index docker rm dense-index sparse-index # If you started Pinecone Local without indexes: docker stop pinecone-local docker rm pinecone-local ``` ## Moving from Pinecone Local to your Pinecone account When you're ready to run your application against your Pinecone account, be sure to do the following: * Update your application to [use your Pinecone API key](/reference/api/authentication). * Update your application to [target your Pinecone indexes](/guides/manage-data/target-an-index). * [Use Pinecone's import feature](/guides/index-data/import-data) to efficiently load large amounts of data into your indexes and then [use batch upserts](/guides/index-data/upsert-data#upsert-in-batches) for ongoing writes. * Follow Pinecone's [production best practices](/guides/production/production-checklist). # Use the Pinecone MCP server Source: https://docs.pinecone.io/guides/operations/mcp-server This feature is in [early access](/release-notes/feature-availability) and is not intended for production usage. The Pinecone MCP server enables AI agents to interact directly with Pinecone's functionality and documentation via the standardized [Model Context Protocol (MCP)](https://modelcontextprotocol.io/). Using the MCP server, agents can search Pinecone documentation, manage indexes, upsert data, and query indexes for relevant information. This page shows you how to configure [Cursor](https://www.cursor.com/) and [Claude Desktop](https://claude.ai/download) to connect with the Pinecone MCP server. ## Tools The Pinecone MCP server provides the following tools: * `search-docs`: Search the official Pinecone documentation. * `list-indexes`: Lists all Pinecone indexes. * `describe-index`: Describes the configuration of an index. * `describe-index-stats`: Provides statistics about the data in the index, including the number of records and available namespaces. * `create-index-for-model`: Creates a new index that uses an integrated inference model to embed text as vectors. * `upsert-records`: Inserts or updates records in an index with integrated inference. * `search-records`: Searches for records in an index based on a text query, using integrated inference for embedding. Has options for metadata filtering and reranking. * `cascading-search`: Searches for records across multiple indexes, deduplicating and reranking the results. * `rerank-documents`: Reranks a collection of records or text documents using a specialized reranking model. The Pinecone MCP supports only [indexes with integrated embedding](/guides/index-data/indexing-overview#vector-embedding). Indexes for vectors you create with external embedding models are not supported. ## Before you begin Ensure you have the following: * A [Pinecone API key](https://app.pinecone.io/organizations/-/keys) * [Node.js](https://nodejs.org/en) installed, with `node` and `npx` available on your `PATH` ## Configure Cursor In your project root, create a `.cursor/mcp.json` file, if it doesn't exist, and add the following configuration: ```json { "mcpServers": { "pinecone": { "command": "npx", "args": [ "-y", "@pinecone-database/mcp" ], "env": { "PINECONE_API_KEY": "YOUR_API_KEY" } } } } ``` Replace `YOUR_API_KEY` with your Pinecone API key. Go to **Cursor Settings > MCP**. You should see the server and its list of tools. The Pinecone MCP server works well out-of-the-box. However, you can add explicit rules to ensure the server behaves as expected. In your project root, create a `.cursor/rules/pinecone.mdc` file and add the following: ```mdx [expandable] ### Tool Usage for Code Generation - When generating code related to Pinecone, always use the `pinecone` MCP and the `search_docs` tool. - Perform at least two distinct searches per request using different, relevant questions to ensure comprehensive context is gathered before writing code. ### Error Handling - If an error occurs while executing Pinecone-related code, immediately invoke the `pinecone` MCP and the `search_docs` tool. - Search for guidance on the specific error encountered and incorporate any relevant findings into your resolution strategy. ### Syntax and Version Accuracy - Before writing any code, verify and use the correct syntax for the latest stable version of the Pinecone SDK. - Prefer official code snippets and examples from documentation over generated or assumed field values. - Do not fabricate field names, parameter values, or request formats. ### SDK Installation Best Practices - When providing installation instructions, always reference the current official package name. - For Pinecone, use `pip install pinecone` not deprecated packages like `pinecone-client`. ``` Press `Command + i` to open the Agent chat. Test the Pinecone MCP server with prompts that required the server to generate Pinceone-compatible code and perform tasks in your Pinecone account. Generate code: > Write a Python script that creates a dense index with integrated embedding, upserts 20 sentences about dogs, waits 10 seconds, searches the index, and reranks the results. Perform tasks: > Create a dense index with integrated embedding, upsert 20 sentences about dogs, waits 10 seconds, search the index, and reranks the results. ## Configure Claude Desktop Go to **Settings > Developer > Edit Config** and add the following configuration: ```json { "mcpServers": { "pinecone": { "command": "npx", "args": [ "-y", "@pinecone-database/mcp" ], "env": { "PINECONE_API_KEY": "YOUR_API_KEY" } } } } ``` Replace `YOUR_API_KEY` with your Pinecone API key. Restart Claude Desktop. On the new chat screen, you should see a hammer (MCP) icon appear with the new MCP tools available. Test the Pinecone MCP server with prompts that required the server to generate Pinceone-compatible code and perform tasks in your Pinecone account. Generate code: > Write a Python script that creates a dense index with integrated embedding, upserts 20 sentences about dogs, waits 10 seconds, searches the index, and reranks the results. Perform tasks: > Create a dense index with integrated embedding, upsert 20 sentences about dogs, waits 10 seconds, search the index, and reranks the results. # Decrease latency Source: https://docs.pinecone.io/guides/optimize/decrease-latency This page describes helpful techniques for decreasing latency for upserts, searches, and other data operations. ## Use namespaces When you divide records into [namespaces](/guides/index-data/indexing-overview#namespaces) in a logical way, you speed up queries by ensuring only relevant records are scanned. The same applies to [fetching records](/guides/manage-data/fetch-data), [listing record IDs](/guides/manage-data/list-record-ids), and other data operations. ## Filter by metadata In addition to increasing search accuracy and relevance, [searching with metadata filters](/guides/search/filter-by-metadata) can also help decrease latency by retrieving only records that match the filter. ## Target indexes by host When you target an index by name for data operations such as `upsert` and `query`, the SDK gets the unique DNS host for the index using the `describe_index` operation. This is convenient for testing but should be avoided in production because `describe_index` uses a different API than data operations and therefore adds an additional network call and point of failure. Instead, you should get an index host once and cache it for reuse or specify the host directly. You can get index hosts in the [Pinecone console](https://app.pinecone.io/organizations/-/projects/-/indexes) or using the [`describe_index`](/guides/manage-data/manage-indexes#describe-an-index) operation. The following example shows how to target an index by host directly: When using Private Endpoints for private connectivity between your application and Pinecone, you must target the index using the [Private Endpoint URL](/guides/production/connect-to-aws-privatelink#run-data-plane-commands) for the host. ```Python Python {5} from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key="YOUR_API_KEY") index = pc.Index(host="INDEX_HOST") ``` ```javascript JavaScript {6} import { Pinecone } from '@pinecone-database/pinecone'; const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }); // For the Node.js SDK, you must specify both the index host and name. const index = pc.index("INDEX_NAME", "INDEX_HOST"); ``` ```java Java {11} import io.pinecone.clients.Index; import io.pinecone.configs.PineconeConfig; import io.pinecone.configs.PineconeConnection; public class TargetIndexByHostExample { public static void main(String[] args) { PineconeConfig config = new PineconeConfig("YOUR_API_KEY"); config.setHost("INDEX_HOST"); PineconeConnection connection = new PineconeConnection(config); // For the Java SDK, you must specify both the index host and name. Index index = new Index(connection, "INDEX_NAME"); } } ``` ```go Go {21} package main import ( "context" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "INDEX_HOST", Namespace: "example-namespace"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host %v: %v", idx.Host, err) } } ``` ```csharp C# {5} using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); var index = pinecone.Index(host: "INDEX_HOST"); ``` ## Reuse connections When you target an index for upserting or querying, the client establishes a TCP connection, which is a three-step process. To avoid going through this process on every request, and reduce average request latency, [cache and reuse the index connection object](/reference/api/authentication#initialize-a-client) whenever possible. ## Use a cloud environment If you experience slow uploads or high query latencies, it might be because you are accessing Pinecone from your home network. To decrease latency, access Pinecone/deploy your application from a cloud environment instead, ideally from the same [cloud and region](/guides/index-data/create-an-index#cloud-regions) as your index. ## Avoid database limits Pinecone has [rate limits](/reference/api/database-limits#rate-limits) that restrict the frequency of requests within a specified period of time. Rate limits vary based on pricing plan and apply to serverless indexes only. # Increase search relevance Source: https://docs.pinecone.io/guides/optimize/increase-relevance This page describes helpful techniques for improving search accuracy and relevance. ## Rerank results [Reranking](/guides/search/rerank-results) is used as part of a two-stage vector retrieval process to improve the quality of results. You first query an index for a given number of relevant results, and then you send the query and results to a reranking model. The reranking model scores the results based on their semantic relevance to the query and returns a new, more accurate ranking. This approach is one of the simplest methods for improving quality in retrieval augmented generation (RAG) pipelines. Pinecone provides [hosted reranking models](/guides/search/rerank-results#reranking-models) so it's easy to manage two-stage vector retrieval on a single platform. You can use a hosted model to rerank results as an integrated part of a query, or you can use a hosted model to rerank results as a standalone operation. ## Filter by metadata Every [record](/guides/get-started/glossary#record) in an index must contain an ID and a dense or sparse vector, depending on the [type of index](/guides/index-data/indexing-overview#indexes). In addition, you can include metadata key-value pairs to store related information or context. When you search the index, you can then include a metadata filter to limit the search to records matching a filter expression. For example, if an index contains records about books, you could use a metadata field to associate each record with a genre, like `"genre": "fiction"` or `"genre": "poetry"`. When you query the index, you could then use a metadata filter to limit your search to records related to a specific genre. For more details, see [Filter by metadata](/guides/search/filter-by-metadata). ## Use hybrid search [Semantic search](/guides/search/semantic-search) and [lexical search](/guides/search/lexical-search) are powerful information retrieval techniques, but each has notable limitations. For example, semantic search can miss results based on exact keyword matches, especially in scenarios involving domain-specific terminology, while lexical search can miss results based on relationships, such as synonyms and paraphrases. To lift these limitations, you can search both dense and sparse indexes, combine the results from both, and use one of Pinecone’s hosted reranking models to assign a unified relevance score, reorder the result accordingly, and return the most relevant matches. This is often called hybrid search or cascading retrieval. For more details, see [Hybrid search](/guides/search/hybrid-search). ## Explore chunking strategies You can chunk your content in different ways to get better results. Consider factors like the length of the content, the complexity of queries, and how results will be used in your application. For more details, see [Chunking strategies](https://www.pinecone.io/learn/chunking-strategies/). # Increase throughput Source: https://docs.pinecone.io/guides/optimize/increase-throughput This page describes helpful techniques for increasing throughput for upserts, searches, and other data operations. ## Import from object storage [Importing from object storage](/guides/index-data/import-data) is the most efficient and cost-effective method to load large numbers of records into an index. You store your data as Parquet files in object storage, integrate your object storage with Pinecone, and then start an asynchronous, long-running operation that imports and indexes your records. ## Upsert in batches [Upserting in batches](/guides/index-data/upsert-data#upsert-in-batches) is another efficient way to ingest large numbers of records (up to 1000 per batch). Batch upserting is also a good option if you cannot work around bulk import's current [limitations](/guides/index-data/data-ingestion-overview#import-limitations). ## Upsert/search in parallel Pinecone is thread-safe, so you can send multiple [upsert](/guides/index-data/upsert-data#upsert-in-parallel) requests and multiple [query](/guides/search/semantic-search#parallel-queries) requests in parallel to help increase throughput. ## Python SDK options ### Use gRPC Use the [Python SDK with gRPC extras](/reference/python-sdk) to run data operations such as upserts and queries over [gRPC](https://grpc.io/) rather than HTTP for a modest performance improvement. ### Upsert from a dataframe To quickly ingest data when using the Python SDK, use the [`upsert_from_dataframe` method](/reference/python-sdk#upsert-from-a-dataframe). The method includes retry logic and `batch_size`, and is performant especially with Parquet file data sets. ## See also Read more about [high-throughput optimizations](https://www.pinecone.io/blog/working-at-scale/) on our blog. # Manage your billing plan Source: https://docs.pinecone.io/guides/organizations/manage-billing/manage-your-billing-plan This page describes how to access and change billing plan information for your Pinecone organization through the Pinecone console. To change your billing plan, you must be the [organization owner or billing admin](/guides/organizations/understanding-organizations#organization-owners) for your organization. For information on how to set up billing through a cloud marketplace, see [AWS Marketplace](/guides/organizations/manage-billing/setting-up-billing-through-aws-marketplace), [Azure Marketplace](/guides/organizations/manage-billing/setting-up-billing-through-azure-marketplace), or [GCP Marketplace](/guides/organizations/manage-billing/setting-up-billing-through-gcp-marketplace). Billing data and management is only available through the Pinecone console. ## Upgrade to pay-as-you-go billing You can upgrade from the free [Starter plan](https://www.pinecone.io/pricing/) to pay-as-you-go billing directly in the Pinecone console: 1. In the Pinecone console, go to [**Settings > Billing > Plans**](https://app.pinecone.io/organizations/-/settings/billing/plans). 2. Click **Upgrade** in the **Standard** or **Enterprise** plan section. After upgrading, you will immediately start paying for usage of your Pinecone indexes, including the serverless indexes that were free on the Starter plan. For more details about how costs are calculated, see [Understanding cost](/guides/manage-cost/understanding-cost). If your organization requires custom contracts, annual commitments, or tailored features, [contact Pinecone](https://www.pinecone.io/contact). ## Downgrade to the Starter plan To stop recurring charges, you can downgrade to the free Starter plan. Before you can downgrade, your organization must be under the [Starter plan quotas](/reference/api/database-limits): * No more than 5 indexes, all serverless and in the `us-east-1` region of AWS. * If you have pod-based indexes, [save them as collections](/guides/manage-data/back-up-an-index#create-a-backup-using-a-collection) and then [delete them](/guides/manage-data/manage-indexes#delete-an-index). * If you have serverless indexes in a region other than `us-east-1`, [create a new serverless index](/guides/index-data/create-an-index#create-a-serverless-index) in `us-east-1`, [re-upsert your data](/guides/index-data/upsert-data) into the new index, and [delete the old index](/guides/manage-data/manage-indexes#delete-an-index). * If you have more than 5 serverless indexes, [delete enough indexes](/guides/manage-data/manage-indexes#delete-an-index) to bring you within the limit. * No more than 1 project. * If you have more than 1 project, delete enough projects to bring you within the limit. * Before you can delete a project, you must [delete all indexes](/guides/manage-data/manage-indexes#delete-an-index) and [delete all collections](/guides/manage-data/back-up-an-index#delete-a-collection) in the project. * No more than 2 GB of data across all of your serverless indexes. * If you are storing more than 2 GB of data, [delete enough records](/guides/manage-data/delete-data) to bring you within the limit. * No more than 100 namespaces per serverless index. * If any serverless index has more than 100 namespaces, [delete enough namespaces](/guides/manage-data/delete-data#delete-all-records-from-a-namespace) to bring you within the limit. * No more than 3 [Assistants](/guides/assistant/overview). * If you have more than 3 assistants, [delete assistants](/guides/assistant/manage-assistants#delete-an-assistant) until you are below the limit. * No more than 10 files per assistant. * If you have more than 10 files uploaded to an assistant, [delete uploaded files](/guides/assistant/manage-files#delete-a-file) until you are below the limit. * No more than 1GB of Assistant storage. * If you have more than 1 GB of Assistant storage, [delete uploaded files](https://docs.pinecone.io/guides/assistant/manage-files#delete-a-file) until you are below the limit. Once you meet the requirements above, downgrade to the Starter plan as follows: 1. In the Pinecone console, got to [**Settings > Billing > Plans**](https://app.pinecone.io/organizations/-/settings/billing/plans). 2. Click **Downgrade** in the **Starter** plan section. Your billing will end immediately; however, you will receive a final invoice for any charges accrued in the current month. ## Access your billing history and invoices You can access your billing history and invoices in the Pinecone console: 1. Go to [**Settings > Billing > Overview**](https://app.pinecone.io/organizations/-/settings/billing). 2. Scroll down to the **Payment history and invoices** section. 3. For each billing period, you can download the invoice by clicking the **Download** button. ## Update your credit card information You can update your credit card information in the Pinecone console: 1. Go to [**Settings > Billing > Overview**](https://app.pinecone.io/organizations/-/settings/billing). 2. In the **Billing Contact** section, click **Edit**. 3. Enter your new credit card information. 4. Click **Update**. # Set up billing through AWS Marketplace Source: https://docs.pinecone.io/guides/organizations/manage-billing/setting-up-billing-through-aws-marketplace This page shows you how to set up pay-as-you-go billing for your Pinecone organization through the Amazon Web Services (AWS) Marketplace. To commit to annual spending, [contact Pinecone](https://www.pinecone.io/contact). ## Upgrade to a paid plan and connect to AWS You must be an [organization owner](/guides/organizations/understanding-organizations#organization-roles) to connect your organization to a cloud marketplace. If you are upgrading from the [Starter plan](https://www.pinecone.io/pricing/), you can connect your organization to AWS: 1. In the Pinecone console, go to [**Settings > Billing**](https://app.pinecone.io/organizations/-/settings/billing). 2. In the **Billing Contact** section, click **Connect to cloud marketplace**. 3. Select **Amazon Web Services**. 4. Click **Continue to marketplace**. 5. Click the **Set up your account** button in the top right. This takes you to an AWS-specific Pinecone sign-up page. If the [Pinecone subscription page](https://aws.amazon.com/marketplace/saas/ordering?productId=738798c3-eeca-494a-a2a9-161bee9450b2) shows a message stating, “You are currently subscribed to this offer,” contact your team members to request an invitation to the existing AWS-linked organization. The **Set up your account** button is clickable, but Pinecone does not create a new AWS-linked organization. 6. Choose an authentication method. Use the same authentication method as your existing Pinecone organization. 7. Select an organization from the list. You can only connect to organizations that are on the [Starter plan](https://www.pinecone.io/pricing/). Alternatively, you can opt to create a new organization. 8. Click **Connect to Pinecone**. Follow the prompts. Once your organization is connected, you will receive a confirmation message. ## Create a new organization connected to AWS You can create a new Pinecone organization linked to AWS Marketplace billing: 1. Go to [the Pinecone listing](https://aws.amazon.com/marketplace/pp/prodview-xhgyscinlz4jk) on the AWS Marketplace. 2. Click **View purchase options** in the top right. 3. Click **Subscribe**. 4. Click the **Set up your account** button in the top right. This takes you to an AWS-specific Pinecone sign up page. 5. Choose an authentication method. 6. Create a new organization. 7. Click **Connect to Pinecone**. Follow the prompts. Once your organization is connected, you will receive a confirmation message. ## Next steps * [Create an index](/guides/index-data/create-an-index) * [Upsert data](/guides/index-data/upsert-data) * [Query data](/guides/search/search-overview) # Set up billing through Azure Marketplace Source: https://docs.pinecone.io/guides/organizations/manage-billing/setting-up-billing-through-azure-marketplace This page shows you how to configure pay-as-you-go billing for your Pinecone organization through the Azure Marketplace. To commit to annual spending, [contact Pinecone](https://www.pinecone.io/contact). ## Upgrade to a paid plan and connect to Azure You must be an [organization owner](/guides/organizations/understanding-organizations#organization-roles) to connect your organization to a cloud marketplace. If you are upgrading from the [Starter plan](https://www.pinecone.io/pricing/), you can connect your organization to Azure: 1. In the Pinecone console, go to [**your profile > Billing and settings > Billing**](https://app.pinecone.io/organizations/-/settings/billing). 2. In the **Billing Contact** section, click **Connect to cloud marketplace**. 3. Select **Azure Cloud Platform**. 4. Click **Continue to marketplace**. 5. Click **Subscribe**. 6. Select the Subscription and Resource group you would like your Pinecone Azure Marketplace Subscription to use. 7. Confirm that recurring billing is enabled. 8. Continue to the **Review and Subscribe** section to review your information. 9. Click **Subscribe**. This takes you to an Azure-specific Pinecone sign-up page. 10. Choose an authentication method. Use the same authentication method as your existing Pinecone organization. 11. Select an organization from the list. You can only connect to organizations that are on the [Starter plan](https://www.pinecone.io/pricing/). Alternatively, you can opt to create a new organization. 12. Click **Connect to Pinecone**. Follow the prompts. Once your organization is connected, you will receive a confirmation message. ## Create a new organization connected to Azure You can create a new Pinecone organization linked to Azure Marketplace billing: 1. Go to [the Pinecone listing](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/pineconesystemsinc1688761585469.pineconesaas?tab=Overview) on the Azure Marketplace. 2. Click **Get it now**. 3. Select the **Pinecone - Pay As You Go** plan. 4. Click **Subscribe**. 5. After the subscription is approved, click **Configure Account**. This redirects you to an Azure-specific Pinecone login page. 6. Choose an authentication method. 7. Create a new organization. 8. Click **Connect to Pinecone**. Follow the prompts. Once your organization is connected, you will receive a confirmation message. ## Next steps * [Create an index](/guides/index-data/create-an-index) * [Upsert data](/guides/index-data/upsert-data) * [Query data](/guides/search/search-overview) # Set up billing through GCP Marketplace Source: https://docs.pinecone.io/guides/organizations/manage-billing/setting-up-billing-through-gcp-marketplace This page shows you how to set up pay-as-you-go billing for your Pinecone organization through the Google Cloud Platform (GCP) Marketplace. To commit to annual spending, [contact Pinecone](https://www.pinecone.io/contact). ## Upgrade to a paid plan and connect to GCP You must be an [organization owner](/guides/organizations/understanding-organizations#organization-roles) to connect your organization to a cloud marketplace. If you are upgrading from the [Starter plan](https://www.pinecone.io/pricing/), you can connect your organization to GCP from the Pinecone console: 1. In the Pinecone console, go to [**Settings > Billing**](https://app.pinecone.io/organizations/-/settings/billing). 2. In the **Billing Contact** section, click **Connect to cloud marketplace**. 3. Select **Google Cloud Platform**. 4. Click **Continue to marketplace**. 5. Click **Subscribe**. 6. Select a billing account. 7. Read and agree to the terms and conditions. 8. Click **Subscribe**. 9. In the **Your order request has been sent to Pinecone** modal, click **Sign up with Pinecone**. This takes you to a GCP-specific Pinecone sign-up page. 10. Choose an authentication method. Use the same authentication method as your existing Pinecone organization. 11. Select an organization from the list. You can only connect to organizations that are on the [Starter plan](https://www.pinecone.io/pricing/). Alternatively, you can opt to create a new organization. 12. Click **Connect to Pinecone**. Follow the prompts. Once your organization is connected, you will receive a confirmation message. ## Create a new organization connected to GCP You can create a new Pinecone organization linked to GCP Marketplace billing: 1. Go to [the Pinecone listing](https://console.cloud.google.com/marketplace/product/pinecone-public/pinecone) on the GCP Marketplace. 2. Click **Subscribe**. 3. Select a billing account. 4. Read and agree to the terms and conditions. 5. Click **Subscribe**. Follow the prompts. Once your organization is connected, you will receive a confirmation message. 6. In the **Your order request has been sent to Pinecone** modal, click **Sign up with Pinecone**. This takes you to a GCP-specific Pinecone sign-up page. 7. Choose an authentication method. 8. Create a new organization. 9. Click **Connect to Pinecone**. Follow the prompts. Once your organization is connected, you will receive a confirmation message. ## Next steps * [Create an index](/guides/index-data/create-an-index) * [Upsert data](/guides/index-data/upsert-data) * [Query data](/guides/search/search-overview) # Manage organization members Source: https://docs.pinecone.io/guides/organizations/manage-organization-members This page shows how [organization owners](/guides/organizations/understanding-organizations#organization-roles) can add and manage organization members. For information about managing members at the **project-level**, see [Manage project members](/guides/projects/manage-project-members). ## Add a member to an organization You can add members to your organization in the [Pinecone console](https://app.pinecone.io): 1. Go to [**Settings > Access > Members**](https://app.pinecone.io/organizations/-/settings/access/members). 2. In the **Invite by email** field, enter the member's email address. 3. Choose an [**Organization role**](/guides/organizations/understanding-organizations#organization-roles) for the member. The role determines the member's permissions within Pinecone. 4. Click **Invite**. When you invite a member to join your organization, Pinecone sends them an email containing a link that enables them to gain access to the organization or project. If they already have a Pinecone account, they still receive an email, but they can also immediately view the project. ## Change a member's role You can change a member's role in the [Pinecone console](https://app.pinecone.io): 1. Go to [**Settings > Access > Members**](https://app.pinecone.io/organizations/-/settings/access/members). 2. In the row of the member whose role you want to change, click **ellipsis (..) menu > Edit role**. 3. Select a [**Project role**](/guides/projects/understanding-projects#project-roles) for the member. 4. Click **Edit role**. ## Remove a member You can remove a member from your organization in the [Pinecone console](https://app.pinecone.io): 1. Go to [**Settings > Access > Members**](https://app.pinecone.io/organizations/-/settings/access/members). 2. In the row of the member you want to remove, click **ellipsis (..) menu > Remove member**. 3. Click **Remove Member**. To remove yourself from an organization, click the **Leave organization** button in your user's row and confirm. # Manage service accounts at the organization-level Source: https://docs.pinecone.io/guides/organizations/manage-service-accounts This feature is in [public preview](/release-notes/feature-availability) and available only on [Enterprise plans](https://www.pinecone.io/pricing/). This page shows how [organization owners](/guides/organizations/understanding-organizations#organization-roles) can add and manage service accounts at the organization-level. Service accounts enable programmatic access to Pinecone's Admin API, which can be used to create and manage projects and API keys. Once a service account is added at the organization-level, it can be added to a project. For more information, see [Manage service accounts at the project-level](/guides/projects/manage-service-accounts). ## Create a service account You can create a service account in the [Pinecone console](https://app.pinecone.io): 1. Go to [**Settings > Access > Service accounts**](https://app.pinecone.io/organizations/-/settings/access/service-accounts). 2. Enter a **Name** for the service account. 3. Choose an [**Organization Role**](/guides/organizations/understanding-organizations#organization-roles) for the service account. The role determines the service account's permissions within Pinecone. 4. Click **Create**. 5. Copy and save the **Client secret** in a secure place for future use. You will need the client secret to retrieve an access token. You will not be able to see the client secret again after you close the dialog. 6. Click **Close**. Once you have created a service account, [add it to a project](/guides/projects/manage-service-accounts#add-a-service-account-to-a-project) to allow it access to the project's resources. ## Retrieve an access token To access the Admin API, you must provide an access token to authenticate. Retrieve the access token using the client secret of a service account, which was [provided at time of creation](#create-a-service-account). You can retrieve an access token for a service account from the `https://login.pinecone.io/oauth/token` endpoint, as shown in the following example: ```bash curl curl "https://login.pinecone.io/oauth/token" \ # Note: Base URL is login.pinecone.io -H "X-Pinecone-Api-Version: 2025-04" \ -H "Content-Type: application/json" \ -d '{ "grant_type": "client_credentials", "client_id": "YOUR_CLIENT_ID", "client_secret": "YOUR_CLIENT_SECRET", "audience": "https://api.pinecone.io/" }' ``` The response will include an `access_token` field, which you can use to authenticate with the Admin API. ``` { "access_token":"YOUR_ACCESS_TOKEN", "expires_in":86400, "token_type":"Bearer" } ``` ## Change a service account's role You can change a service account's role in the [Pinecone console](https://app.pinecone.io): 1. Go to [**Settings > Access > Service accounts**](https://app.pinecone.io/organizations/-/settings/service-accounts). 2. In the row of the service account you want to update, click **ellipsis (..) menu > Manage**. 3. Select an [**Organization role**](/guides/organizations/understanding-organizations#organization-roles) for the service account. 4. Click **Update**. ## Update service account name You can change a service account's name in the [Pinecone console](https://app.pinecone.io): 1. Go to [**Settings > Access > Service accounts**](https://app.pinecone.io/organizations/-/settings/service-accounts). 2. In the row of the service account you want to update, click **ellipsis (..) menu > Manage**. 3. Enter a new **Service account name**. 4. Click **Update**. ## Rotate a service account's secret You can rotate a service account's client secret in the [Pinecone console](https://app.pinecone.io): 1. Go to [**Settings > Access > Service accounts**](https://app.pinecone.io/organizations/-/settings/service-accounts). 2. In the row of the service account you want to update, click **ellipsis (..) menu > Rotate secret**. 3. **Enter the service account name** to confirm. 4. Click **Rotate client secret**. 5. Copy and save the **Client secret** in a secure place for future use. You will not be able to see the client secret again after you close the dialog. 6. Click **Close**. ## Delete a service account Deleting a service account will remove it from all projects and will disrupt any applications using it to access Pinecone. You delete a service account in the [Pinecone console](https://app.pinecone.io): 1. Go to [**Settings > Access > Service accounts**](https://app.pinecone.io/organizations/-/settings/service-accounts). 2. In the row of the service account you want to update, click **ellipsis (..) menu > Delete**. 3. **Enter the service account name** to confirm. 4. Click **Delete service account**. # Understanding organizations Source: https://docs.pinecone.io/guides/organizations/understanding-organizations A Pinecone organization is a set of [projects](/guides/projects/understanding-projects) that use the same billing. Organizations allow one or more users to control billing and project permissions for all of the projects belonging to the organization. Each project belongs to an organization. While an email address can be associated with multiple organizations, it cannot be used to create more than one organization. For information about managing organization members, see [Manage organization members](/guides/organizations/manage-organization-members). ## Projects in an organization Each organization contains one or more projects that share the same organization owners and billing settings. Each project belongs to exactly one organization. If you need to move a project from one organization to another, [contact Support](https://app.pinecone.io/organizations/-/settings/support/ticket). ## Billing settings All of the projects in an organization share the same billing method and settings. The billing settings for the organization are controlled by the organization owners. Organization owners can update the billing contact information, update the payment method, and view and download invoices using the [Pinecone console](https://app.pinecone.io/organizations/-/settings/billing). ## Organization roles Organization owners can manage access to their organizations and projects by assigning roles to organization members and service accounts. The role determines the entity's permissions within Pinecone. The organization roles are as follows: * **Organization owner**: Organization owners have global permissions across the organization. This includes managing billing details, organization members, and all projects. Organization owners are automatically [project owners](/guides/projects/understanding-projects#project-roles) and, therefore, have all project owner permissions as well. * **Organization user**: Organization users have restricted organization-level permissions. When inviting organization users, you also choose the projects they belong to and the project role they should have. Organization users are automatically [project owners](/guides/projects/understanding-projects#project-roles) and, therefore, have all project owner permissions as well. * **Billing admin**: Billing admins have permissions to view and update billing details, but they cannot manage organization members. Billing admins cannot manage projects unless they are also [project owners](/guides/projects/understanding-projects#project-roles). The following table summarizes the permissions for each organization role: | Permission | Org Owner | Org User | Billing Admin | | ------------------------------------ | --------- | -------- | ------------- | | View account details | ✓ | ✓ | ✓ | | Update organization name | ✓ | | | | Delete the organization | ✓ | | | | View billing details | ✓ | | ✓ | | Update billing details | ✓ | | ✓ | | View usage details | ✓ | | ✓ | | View support plans | ✓ | | ✓ | | Invite members to the organization | ✓ | | | | Delete pending member invites | ✓ | | | | Remove members from the organization | ✓ | | | | Update organization member roles | ✓ | | | | Create projects | ✓ | ✓ | | ## Organization single sign-on (SSO) SSO allows organizations to manage their teams' access to Pinecone through their identity management solution. Once your integration is configured, you can specify a default role for teammates when they sign up. For more information, see [Configure single sign-on](/guides/production/configure-single-sign-on/okta). SSO is available only on Enterprise plans. ## Service accounts This feature is in [public preview](/release-notes/feature-availability) and available only on [Enterprise plans](https://www.pinecone.io/pricing/). [Service accounts](/guides/organizations/manage-service-accounts) enable programmatic access to Pinecone's Admin API, which can be used to create and manage projects and API keys. Use service accounts to automate infrastructure management and integrate Pinecone into your deployment workflows, rather than through manual actions in the Pinecone console. Service accounts use the [organization roles](/guides/organizations/understanding-organizations#organization-roles) and [project role](/guides/projects/understanding-projects#project-roles) for permissioning, and provide a secure and auditable way to handle programmatic access. ## See also * [Manage organization members](/guides/organizations/manage-organization-members) * [Manage project members](/guides/projects/manage-project-members) # CI/CD with Pinecone Local and GitHub Actions Source: https://docs.pinecone.io/guides/production/automated-testing Pinecone Local is an in-memory Pinecone Database emulator available as a Docker image. This page shows you how to build a CI/CD workflow with Pinecone Local and [GitHub Actions](https://docs.github.com/en/actions) to test your integration without connecting to your Pinecone account, affecting production data, or incurring any usage or storage fees. Pinecone Local is not suitable for production. See [Limitations](#limitations) for details. This feature is in [public preview](/release-notes/feature-availability). ## Limitations Pinecone Local has the following limitations: * Pinecone Local is available in Docker only. * Pinecone Local is an in-memory emulator and is not suitable for production. Records loaded into Pinecone Local do not persist after it is stopped. * Pinecone Local does not authenticate client requests. API keys are ignored. * Max number of records per index: 100,000. Pinecone Local does not currently support the following features: * [Bulk import](/guides/index-data/import-data) * [Backups](/guides/manage-data/backups-overview) * [Collections](/guides/indexes/pods/understanding-collections) * [Pinecone Inference](/reference/api/introduction#inference) * [Pinecone Assistant](/guides/assistant/overview) ## 1. Write your tests Running code against Pinecone Local is just like running code against your Pinecone account, with the following differences: * Pinecone Local does not authenticate client requests. API keys are ignored. * The latest version of Pinecone Local uses [Pinecone API version](/reference/api/versioning) `2025-01` and requires [Python SDK](/reference/python-sdk) `v6.x` or later, [Node.js SDK](/reference/node-sdk) `v5.x` or later, [Java SDK](/reference/java-sdk) `v4.x` or later, [Go SDK](/reference/go-sdk) `v3.x` or later, and [.NET SDK](/reference/dotnet-sdk) `v3.x` or later. Be sure to review the [limitations](#limitations) of Pinecone Local before using it for development or testing. **Example** The following example assumes that you have [started Pinecone Local without indexes](/guides/operations/local-development#database-emulator). It initializes a client, creates a [dense index](/guides/index-data/indexing-overview#dense-indexes) and a [sparse index](/guides/index-data/indexing-overview#sparse-indexes), upserts records into the indexes, checks their record counts, and queries the indexes. ```python Python from pinecone.grpc import PineconeGRPC, GRPCClientConfig from pinecone import ServerlessSpec # Initialize a client. # API key is required, but the value does not matter. # Host and port of the Pinecone Local instance # is required when starting without indexes. pc = PineconeGRPC( api_key="pclocal", host="http://localhost:5080" ) # Create two indexes, one dense and one sparse dense_index_name = "dense-index" sparse_index_name = "sparse-index" if not pc.has_index(dense_index_name): dense_index_model = pc.create_index( name=dense_index_name, vector_type="dense", dimension=2, metric="cosine", spec=ServerlessSpec(cloud="aws", region="us-east-1"), deletion_protection="disabled", tags={"environment": "development"} ) print("Dense index model:\n", dense_index_model) if not pc.has_index(sparse_index_name): sparse_index_model = pc.create_index( name=sparse_index_name, vector_type="sparse", metric="dotproduct", spec=ServerlessSpec(cloud="aws", region="us-east-1"), deletion_protection="disabled", tags={"environment": "development"} ) print("\nSparse index model:\n", sparse_index_model) # Target each index, disabling tls dense_index_host = pc.describe_index(name=dense_index_name).host dense_index = pc.Index(host=dense_index_host, grpc_config=GRPCClientConfig(secure=False)) sparse_index_host = pc.describe_index(name=sparse_index_name).host sparse_index = pc.Index(host=sparse_index_host, grpc_config=GRPCClientConfig(secure=False)) # Upsert records into the dense index dense_index.upsert( vectors=[ { "id": "vec1", "values": [1.0, -2.5], "metadata": {"genre": "drama"} }, { "id": "vec2", "values": [3.0, -2.0], "metadata": {"genre": "documentary"} }, { "id": "vec3", "values": [0.5, -1.5], "metadata": {"genre": "documentary"} } ], namespace="example-namespace" ) # Upsert records into the sparse index sparse_index.upsert( namespace="example-namespace", vectors=[ { "id": "vec1", "sparse_values": { "values": [1.7958984, 0.41577148, 2.828125, 2.8027344, 2.8691406, 1.6533203, 5.3671875, 1.3046875, 0.49780273, 0.5722656, 2.71875, 3.0820312, 2.5019531, 4.4414062, 3.3554688], "indices": [822745112, 1009084850, 1221765879, 1408993854, 1504846510, 1596856843, 1640781426, 1656251611, 1807131503, 2543655733, 2902766088, 2909307736, 3246437992, 3517203014, 3590924191] }, "metadata": { "chunk_text": "AAPL reported a year-over-year revenue increase, expecting stronger Q3 demand for its flagship phones.", "category": "technology", "quarter": "Q3" } }, { "id": "vec2", "sparse_values": { "values": [0.4362793, 3.3457031, 2.7714844, 3.0273438, 3.3164062, 5.6015625, 2.4863281, 0.38134766, 1.25, 2.9609375, 0.34179688, 1.4306641, 0.34375, 3.3613281, 1.4404297, 2.2558594, 2.2597656, 4.8710938, 0.5605469], "indices": [131900689, 592326839, 710158994, 838729363, 1304885087, 1640781426, 1690623792, 1807131503, 2066971792, 2428553208, 2548600401, 2577534050, 3162218338, 3319279674, 3343062801, 3476647774, 3485013322, 3517203014, 4283091697] }, "metadata": { "chunk_text": "Analysts suggest that AAPL'\''s upcoming Q4 product launch event might solidify its position in the premium smartphone market.", "category": "technology", "quarter": "Q4" } }, { "id": "vec3", "sparse_values": { "values": [2.6875, 4.2929688, 3.609375, 3.0722656, 2.1152344, 5.78125, 3.7460938, 3.7363281, 1.2695312, 3.4824219, 0.7207031, 0.0826416, 4.671875, 3.7011719, 2.796875, 0.61621094], "indices": [8661920, 350356213, 391213188, 554637446, 1024951234, 1640781426, 1780689102, 1799010313, 2194093370, 2632344667, 2641553256, 2779594451, 3517203014, 3543799498, 3837503950, 4283091697] }, "metadata": { "chunk_text": "AAPL'\''s strategic Q3 partnerships with semiconductor suppliers could mitigate component risks and stabilize iPhone production", "category": "technology", "quarter": "Q3" } } ] ) # Check the number of records in each index print("\nDense index stats:\n", dense_index.describe_index_stats()) print("\nSparse index stats:\n", sparse_index.describe_index_stats()) # Query the dense index with a metadata filter dense_response = dense_index.query( namespace="example-namespace", vector=[3.0, -2.0], filter={"genre": {"$eq": "documentary"}}, top_k=1, include_values=False, include_metadata=True ) print("\nDense query response:\n", dense_response) # Query the sparse index with a metadata filter sparse_response = sparse_index.query( namespace="example-namespace", sparse_vector={ "values": [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0], "indices": [767227209, 1640781426, 1690623792, 2021799277, 2152645940, 2295025838, 2443437770, 2779594451, 2956155693, 3476647774, 3818127854, 4283091697] }, filter={ "quarter": {"$eq": "Q4"} }, top_k=1, include_values=False, include_metadata=True ) print("/nSparse query response:\n", sparse_response) # Delete the indexes pc.delete_index(name=dense_index_name) pc.delete_index(name=sparse_index_name) ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone'; // Initialize a client. // API key is required, but the value does not matter. // Host and port of the Pinecone Local instance // is required when starting without indexes. const pc = new Pinecone({ apiKey: 'pclocal', controllerHostUrl: 'http://localhost:5080' }); // Create two indexes, one dense and one sparse const denseIndexName = 'dense-index'; const sparseIndexName = 'sparse-index'; const denseIndexModel = await pc.createIndex({ name: denseIndexName, vectorType: 'dense', dimension: 2, metric: 'cosine', spec: { serverless: { cloud: 'aws', region: 'us-east-1' } }, deletionProtection: 'disabled', tags: { environment: 'development' }, }); console.log('Dense index model:', denseIndexModel); const sparseIndexModel = await pc.createIndex({ name: sparseIndexName, vectorType: 'sparse', metric: 'dotproduct', spec: { serverless: { cloud: 'aws', region: 'us-east-1' } }, deletionProtection: 'disabled', tags: { environment: 'development' }, }); console.log('\nSparse index model:', sparseIndexModel); // Target each index const denseIndexHost = (await pc.describeIndex(denseIndexName)).host; const denseIndex = await pc.index(denseIndexName, 'http://' + denseIndexHost); const sparseIndexHost = (await pc.describeIndex(sparseIndexName)).host; const sparseIndex = await pc.index(sparseIndexName, 'http://' + sparseIndexHost); // Upsert records into the dense index await denseIndex.namespace('example-namespace').upsert([ { id: 'vec1', values: [1.0, -2.5], metadata: { genre: 'drama' }, }, { id: 'vec2', values: [3.0, -2.0], metadata: { genre: 'documentary' }, }, { id: 'vec3', values: [0.5, -1.5], metadata: { genre: 'documentary' }, } ]); // Upsert records into the sparse index await sparseIndex.namespace('example-namespace').upsert([ { id: 'vec1', sparseValues: { indices: [822745112, 1009084850, 1221765879, 1408993854, 1504846510, 1596856843, 1640781426, 1656251611, 1807131503, 2543655733, 2902766088, 2909307736, 3246437992, 3517203014, 3590924191], values: [1.7958984, 0.41577148, 2.828125, 2.8027344, 2.8691406, 1.6533203, 5.3671875, 1.3046875, 0.49780273, 0.5722656, 2.71875, 3.0820312, 2.5019531, 4.4414062, 3.3554688] }, metadata: { chunk_text: 'AAPL reported a year-over-year revenue increase, expecting stronger Q3 demand for its flagship phones.', category: 'technology', quarter: 'Q3' } }, { id: 'vec2', sparseValues: { indices: [131900689, 592326839, 710158994, 838729363, 1304885087, 1640781426, 1690623792, 1807131503, 2066971792, 2428553208, 2548600401, 2577534050, 3162218338, 3319279674, 3343062801, 3476647774, 3485013322, 3517203014, 4283091697], values: [0.4362793, 3.3457031, 2.7714844, 3.0273438, 3.3164062, 5.6015625, 2.4863281, 0.38134766, 1.25, 2.9609375, 0.34179688, 1.4306641, 0.34375, 3.3613281, 1.4404297, 2.2558594, 2.2597656, 4.8710938, 0.5605469] }, metadata: { chunk_text: "Analysts suggest that AAPL's upcoming Q4 product launch event might solidify its position in the premium smartphone market.", category: 'technology', quarter: 'Q4' } }, { id: 'vec3', sparseValues: { indices: [8661920, 350356213, 391213188, 554637446, 1024951234, 1640781426, 1780689102, 1799010313, 2194093370, 2632344667, 2641553256, 2779594451, 3517203014, 3543799498, 3837503950, 4283091697], values: [2.6875, 4.2929688, 3.609375, 3.0722656, 2.1152344, 5.78125, 3.7460938, 3.7363281, 1.2695312, 3.4824219, 0.7207031, 0.0826416, 4.671875, 3.7011719, 2.796875, 0.61621094] }, metadata: { chunk_text: "AAPL's strategic Q3 partnerships with semiconductor suppliers could mitigate component risks and stabilize iPhone production", category: 'technology', quarter: 'Q3' } } ]); // Check the number of records in each index console.log('\nDense index stats:', await denseIndex.describeIndexStats()); console.log('\nSparse index stats:', await sparseIndex.describeIndexStats()); // Query the dense index with a metadata filter const denseQueryResponse = await denseIndex.namespace('example-namespace').query({ vector: [3.0, -2.0], filter: { 'genre': {'$eq': 'documentary'} }, topK: 1, includeValues: false, includeMetadata: true, }); console.log('\nDense query response:', denseQueryResponse); const sparseQueryResponse = await sparseIndex.namespace('example-namespace').query({ sparseVector: { indices: [767227209, 1640781426, 1690623792, 2021799277, 2152645940, 2295025838, 2443437770, 2779594451, 2956155693, 3476647774, 3818127854, 4283091697], values: [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] }, topK: 1, includeValues: false, includeMetadata: true }); console.log('\nSparse query response:', sparseQueryResponse); // Delete the index await pc.deleteIndex(denseIndexName); await pc.deleteIndex(sparseIndexName); ``` ```java Java import com.google.protobuf.Struct; import com.google.protobuf.Value; import io.pinecone.clients.Index; import io.pinecone.clients.Pinecone; import io.pinecone.proto.DescribeIndexStatsResponse; import org.openapitools.db_control.client.model.DeletionProtection; import io.pinecone.unsigned_indices_model.QueryResponseWithUnsignedIndices; import java.util.*; public class PineconeLocalExample { public static void main(String[] args) { // Initialize a client. // API key is required, but the value does not matter. // When starting without indexes, disable TLS and // provide the host and port of the Pinecone Local instance. String host = "http://localhost:5080"; Pinecone pc = new Pinecone.Builder("pclocal") .withHost(host) .withTlsEnabled(false) .build(); // Create two indexes, one dense and one sparse String denseIndexName = "dense-index"; String sparseIndexName = "sparse-index"; HashMap tags = new HashMap<>(); tags.put("environment", "development"); pc.createServerlessIndex( denseIndexName, "cosine", 2, "aws", "us-east-1", DeletionProtection.DISABLED, tags ); pc.createSparseServelessIndex( sparseIndexName, "aws", "us-east-1", DeletionProtection.DISABLED, tags, "sparse" ); // Get index connection objects Index denseIndexConnection = pc.getIndexConnection(denseIndexName); Index sparseIndexConnection = pc.getIndexConnection(sparseIndexName); // Upsert records into the dense index Struct metaData1 = Struct.newBuilder() .putFields("genre", Value.newBuilder().setStringValue("drama").build()) .build(); Struct metaData2 = Struct.newBuilder() .putFields("genre", Value.newBuilder().setStringValue("documentary").build()) .build(); Struct metaData3 = Struct.newBuilder() .putFields("genre", Value.newBuilder().setStringValue("documentary").build()) .build(); denseIndexConnection.upsert("vec1", Arrays.asList(1.0f, -2.5f), null, null, metaData1, "example-namespace"); denseIndexConnection.upsert("vec2", Arrays.asList(3.0f, -2.0f), null, null, metaData2, "example-namespace"); denseIndexConnection.upsert("vec3", Arrays.asList(0.5f, -1.5f), null, null, metaData3, "example-namespace"); // Upsert records into the sparse index ArrayList indices1 = new ArrayList<>(Arrays.asList( 822745112L, 1009084850L, 1221765879L, 1408993854L, 1504846510L, 1596856843L, 1640781426L, 1656251611L, 1807131503L, 2543655733L, 2902766088L, 2909307736L, 3246437992L, 3517203014L, 3590924191L )); ArrayList values1 = new ArrayList<>(Arrays.asList( 1.7958984f, 0.41577148f, 2.828125f, 2.8027344f, 2.8691406f, 1.6533203f, 5.3671875f, 1.3046875f, 0.49780273f, 0.5722656f, 2.71875f, 3.0820312f, 2.5019531f, 4.4414062f, 3.3554688f )); Struct sparseMetaData1 = Struct.newBuilder() .putFields("chunk_text", Value.newBuilder().setStringValue("AAPL reported a year-over-year revenue increase, expecting stronger Q3 demand for its flagship phones.").build()) .putFields("category", Value.newBuilder().setStringValue("technology").build()) .putFields("quarter", Value.newBuilder().setStringValue("Q3").build()) .build(); ArrayList indices2 = new ArrayList<>(Arrays.asList( 131900689L, 592326839L, 710158994L, 838729363L, 1304885087L, 1640781426L, 1690623792L, 1807131503L, 2066971792L, 2428553208L, 2548600401L, 2577534050L, 3162218338L, 3319279674L, 3343062801L, 3476647774L, 3485013322L, 3517203014L, 4283091697L )); ArrayList values2 = new ArrayList<>(Arrays.asList( 0.4362793f, 3.3457031f, 2.7714844f, 3.0273438f, 3.3164062f, 5.6015625f, 2.4863281f, 0.38134766f, 1.25f, 2.9609375f, 0.34179688f, 1.4306641f, 0.34375f, 3.3613281f, 1.4404297f, 2.2558594f, 2.2597656f, 4.8710938f, 0.5605469f )); Struct sparseMetaData2 = Struct.newBuilder() .putFields("chunk_text", Value.newBuilder().setStringValue("Analysts suggest that AAPL'\\''s upcoming Q4 product launch event might solidify its position in the premium smartphone market.").build()) .putFields("category", Value.newBuilder().setStringValue("technology").build()) .putFields("quarter", Value.newBuilder().setStringValue("Q4").build()) .build(); ArrayList indices3 = new ArrayList<>(Arrays.asList( 8661920L, 350356213L, 391213188L, 554637446L, 1024951234L, 1640781426L, 1780689102L, 1799010313L, 2194093370L, 2632344667L, 2641553256L, 2779594451L, 3517203014L, 3543799498L, 3837503950L, 4283091697L )); ArrayList values3 = new ArrayList<>(Arrays.asList( 2.6875f, 4.2929688f, 3.609375f, 3.0722656f, 2.1152344f, 5.78125f, 3.7460938f, 3.7363281f, 1.2695312f, 3.4824219f, 0.7207031f, 0.0826416f, 4.671875f, 3.7011719f, 2.796875f, 0.61621094f )); Struct sparseMetaData3 = Struct.newBuilder() .putFields("chunk_text", Value.newBuilder().setStringValue("AAPL'\\''s strategic Q3 partnerships with semiconductor suppliers could mitigate component risks and stabilize iPhone production").build()) .putFields("category", Value.newBuilder().setStringValue("technology").build()) .putFields("quarter", Value.newBuilder().setStringValue("Q3").build()) .build(); sparseIndexConnection.upsert("vec1", Collections.emptyList(), indices1, values1, sparseMetaData1, "example-namespace"); sparseIndexConnection.upsert("vec2", Collections.emptyList(), indices2, values2, sparseMetaData2, "example-namespace"); sparseIndexConnection.upsert("vec3", Collections.emptyList(), indices3, values3, sparseMetaData3, "example-namespace"); // Check the number of records each the index DescribeIndexStatsResponse denseIndexStatsResponse = denseIndexConnection.describeIndexStats(null); System.out.println("Dense index stats:"); System.out.println(denseIndexStatsResponse); DescribeIndexStatsResponse sparseIndexStatsResponse = sparseIndexConnection.describeIndexStats(null); System.out.println("Sparse index stats:"); System.out.println(sparseIndexStatsResponse); // Query the dense index with a metadata filter List queryVector = Arrays.asList(1.0f, 1.5f); QueryResponseWithUnsignedIndices denseQueryResponse = denseIndexConnection.query(1, queryVector, null, null, null, "example-namespace", null, false, true); System.out.println("Dense query response:"); System.out.println(denseQueryResponse); // Query the sparse index with a metadata filter List sparseIndices = Arrays.asList( 767227209L, 1640781426L, 1690623792L, 2021799277L, 2152645940L, 2295025838L, 2443437770L, 2779594451L, 2956155693L, 3476647774L, 3818127854L, 428309169L); List sparseValues = Arrays.asList( 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f); QueryResponseWithUnsignedIndices sparseQueryResponse = sparseIndexConnection.query(1, null, sparseIndices, sparseValues, null, "example-namespace", null, false, true); System.out.println("Sparse query response:"); System.out.println(sparseQueryResponse); // Delete the indexes pc.deleteIndex(denseIndexName); pc.deleteIndex(sparseIndexName); } } ``` ```go Go package main import ( "context" "encoding/json" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" "google.golang.org/protobuf/types/known/structpb" ) func prettifyStruct(obj interface{}) string { bytes, _ := json.MarshalIndent(obj, "", " ") return string(bytes) } func main() { ctx := context.Background() // Initialize a client. // No API key is required. // Host and port of the Pinecone Local instance // is required when starting without indexes. pc, err := pinecone.NewClientBase(pinecone.NewClientBaseParams{ Host: "http://localhost:5080", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // Create two indexes, one dense and one sparse denseIndexName := "dense-index" denseVectorType := "dense" dimension := int32(2) denseMetric := pinecone.Cosine deletionProtection := pinecone.DeletionProtectionDisabled denseIdx, err := pc.CreateServerlessIndex(ctx, &pinecone.CreateServerlessIndexRequest{ Name: denseIndexName, VectorType: &denseVectorType, Dimension: &dimension, Metric: &denseMetric, Cloud: pinecone.Aws, Region: "us-east-1", DeletionProtection: &deletionProtection, Tags: &pinecone.IndexTags{"environment": "development"}, }) if err != nil { log.Fatalf("Failed to create serverless index: %v", denseIdx.Name) } else { fmt.Printf("Successfully created serverless index: %v\n", denseIdx.Name) } sparseIndexName := "sparse-index" sparseVectorType := "sparse" sparseMetric := pinecone.Dotproduct sparseIdx, err := pc.CreateServerlessIndex(ctx, &pinecone.CreateServerlessIndexRequest{ Name: sparseIndexName, VectorType: &sparseVectorType, Metric: &sparseMetric, Cloud: pinecone.Aws, Region: "us-east-1", DeletionProtection: &deletionProtection, Tags: &pinecone.IndexTags{"environment": "development"}, }) if err != nil { log.Fatalf("Failed to create serverless index: %v", sparseIdx.Name) } else { fmt.Printf("\nSuccessfully created serverless index: %v\n", sparseIdx.Name) } // Get the index hosts denseIdxModel, err := pc.DescribeIndex(ctx, denseIndexName) if err != nil { log.Fatalf("Failed to describe index \"%v\": %v", denseIndexName, err) } sparseIdxModel, err := pc.DescribeIndex(ctx, sparseIndexName) if err != nil { log.Fatalf("Failed to describe index \"%v\": %v", sparseIndexName, err) } // Target the indexes. // Make sure to prefix the hosts with http:// to let the SDK know to disable tls. denseIdxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "http://" + denseIdxModel.Host, Namespace: "example-namespace"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v", err) } sparseIdxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "http://" + sparseIdxModel.Host, Namespace: "example-namespace"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v", err) } // Upsert records into the dense index denseMetadataMap1 := map[string]interface{}{ "genre": "drama", } denseMetadata1, err := structpb.NewStruct(denseMetadataMap1) if err != nil { log.Fatalf("Failed to create metadata map: %v", err) } denseMetadataMap2 := map[string]interface{}{ "genre": "documentary", } denseMetadata2, err := structpb.NewStruct(denseMetadataMap2) if err != nil { log.Fatalf("Failed to create metadata map: %v", err) } denseMetadataMap3 := map[string]interface{}{ "genre": "documentary", } denseMetadata3, err := structpb.NewStruct(denseMetadataMap3) if err != nil { log.Fatalf("Failed to create metadata map: %v", err) } denseVectors := []*pinecone.Vector{ { Id: "vec1", Values: &[]float32{1.0, -2.5}, Metadata: denseMetadata1, }, { Id: "vec2", Values: &[]float32{3.0, -2.0}, Metadata: denseMetadata2, }, { Id: "vec3", Values: &[]float32{0.5, -1.5}, Metadata: denseMetadata3, }, } denseCount, err := denseIdxConnection.UpsertVectors(ctx, denseVectors) if err != nil { log.Fatalf("Failed to upsert vectors: %v", err) } else { fmt.Printf("\nSuccessfully upserted %d vector(s)!\n", denseCount) } // Upsert records into the sparse index sparseValues1 := pinecone.SparseValues{ Indices: []uint32{822745112, 1009084850, 1221765879, 1408993854, 1504846510, 1596856843, 1640781426, 1656251611, 1807131503, 2543655733, 2902766088, 2909307736, 3246437992, 3517203014, 3590924191}, Values: []float32{1.7958984, 0.41577148, 2.828125, 2.8027344, 2.8691406, 1.6533203, 5.3671875, 1.3046875, 0.49780273, 0.5722656, 2.71875, 3.0820312, 2.5019531, 4.4414062, 3.3554688}, } sparseMetadataMap1 := map[string]interface{}{ "chunk_text": "AAPL reported a year-over-year revenue increase, expecting stronger Q3 demand for its flagship phones", "category": "technology", "quarter": "Q3", } sparseMetadata1, err := structpb.NewStruct(sparseMetadataMap1) if err != nil { log.Fatalf("Failed to create metadata map: %v", err) } sparseValues2 := pinecone.SparseValues{ Indices: []uint32{131900689, 592326839, 710158994, 838729363, 1304885087, 1640781426, 1690623792, 1807131503, 2066971792, 2428553208, 2548600401, 2577534050, 3162218338, 3319279674, 3343062801, 3476647774, 3485013322, 3517203014, 4283091697}, Values: []float32{0.4362793, 3.3457031, 2.7714844, 3.0273438, 3.3164062, 5.6015625, 2.4863281, 0.38134766, 1.25, 2.9609375, 0.34179688, 1.4306641, 0.34375, 3.3613281, 1.4404297, 2.2558594, 2.2597656, 4.8710938, 0.560546}, } sparseMetadataMap2 := map[string]interface{}{ "chunk_text": "Analysts suggest that AAPL's upcoming Q4 product launch event might solidify its position in the premium smartphone market.", "category": "technology", "quarter": "Q4", } sparseMetadata2, err := structpb.NewStruct(sparseMetadataMap2) if err != nil { log.Fatalf("Failed to create metadata map: %v", err) } sparseValues3 := pinecone.SparseValues{ Indices: []uint32{8661920, 350356213, 391213188, 554637446, 1024951234, 1640781426, 1780689102, 1799010313, 2194093370, 2632344667, 2641553256, 2779594451, 3517203014, 3543799498, 3837503950, 4283091697}, Values: []float32{2.6875, 4.2929688, 3.609375, 3.0722656, 2.1152344, 5.78125, 3.7460938, 3.7363281, 1.2695312, 3.4824219, 0.7207031, 0.0826416, 4.671875, 3.7011719, 2.796875, 0.61621094}, } sparseMetadataMap3 := map[string]interface{}{ "chunk_text": "AAPL's strategic Q3 partnerships with semiconductor suppliers could mitigate component risks and stabilize iPhone production", "category": "technology", "quarter": "Q3", } sparseMetadata3, err := structpb.NewStruct(sparseMetadataMap3) if err != nil { log.Fatalf("Failed to create metadata map: %v", err) } sparseVectors := []*pinecone.Vector{ { Id: "vec1", SparseValues: &sparseValues1, Metadata: sparseMetadata1, }, { Id: "vec2", SparseValues: &sparseValues2, Metadata: sparseMetadata2, }, { Id: "vec3", SparseValues: &sparseValues3, Metadata: sparseMetadata3, }, } sparseCount, err := sparseIdxConnection.UpsertVectors(ctx, sparseVectors) if err != nil { log.Fatalf("Failed to upsert vectors: %v", err) } else { fmt.Printf("\nSuccessfully upserted %d vector(s)!\n", sparseCount) } // Check the number of records in each index denseStats, err := denseIdxConnection.DescribeIndexStats(ctx) if err != nil { log.Fatalf("Failed to describe index: %v", err) } else { fmt.Printf("\nDense index stats: %+v\n", prettifyStruct(*denseStats)) } sparseStats, err := sparseIdxConnection.DescribeIndexStats(ctx) if err != nil { log.Fatalf("Failed to describe index: %v", err) } else { fmt.Printf("\nSparse index stats: %+v\n", prettifyStruct(*sparseStats)) } // Query the dense index with a metadata filter queryVector := []float32{3.0, -2.0} queryMetadataMap := map[string]interface{}{ "genre": map[string]interface{}{ "$eq": "documentary", }, } metadataFilter, err := structpb.NewStruct(queryMetadataMap) if err != nil { log.Fatalf("Failed to create metadata map: %v", err) } denseRes, err := denseIdxConnection.QueryByVectorValues(ctx, &pinecone.QueryByVectorValuesRequest{ Vector: queryVector, TopK: 1, MetadataFilter: metadataFilter, IncludeValues: false, IncludeMetadata: true, }) if err != nil { log.Fatalf("Error encountered when querying by vector: %v", err) } else { fmt.Printf("\nDense query response: %v\n", prettifyStruct(denseRes)) } // Query the sparse index with a metadata filter sparseValues := pinecone.SparseValues{ Indices: []uint32{767227209, 1640781426, 1690623792, 2021799277, 2152645940, 2295025838, 2443437770, 2779594451, 2956155693, 3476647774, 3818127854, 4283091697}, Values: []float32{1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0}, } sparseRes, err := sparseIdxConnection.QueryByVectorValues(ctx, &pinecone.QueryByVectorValuesRequest{ SparseValues: &sparseValues, TopK: 1, IncludeValues: false, IncludeMetadata: true, }) if err != nil { log.Fatalf("Error encountered when querying by vector: %v", err) } else { fmt.Printf("\nSparse query response: %v\n", prettifyStruct(sparseRes)) } // Delete the indexes err = pc.DeleteIndex(ctx, denseIndexName) if err != nil { log.Fatalf("Failed to delete index: %v", err) } else { fmt.Printf("\nIndex \"%v\" deleted successfully\n", denseIndexName) } err = pc.DeleteIndex(ctx, sparseIndexName) if err != nil { log.Fatalf("Failed to delete index: %v", err) } else { fmt.Printf("\nIndex \"%v\" deleted successfully\n", sparseIndexName) } } ``` ```csharp C# using Pinecone; // Initialize a client. // API key is required, but the value does not matter. // When starting without indexes, disable TLS and // provide the host and port of the Pinecone Local instance. var pc = new PineconeClient("pclocal", new ClientOptions { BaseUrl = "http://localhost:5080", IsTlsEnabled = false } ); // Create two indexes, one dense and one sparse var denseIndexName = "dense-index"; var sparseIndexName = "sparse-index"; var createDenseIndexRequest = await pc.CreateIndexAsync(new CreateIndexRequest { Name = denseIndexName, VectorType = VectorType.Dense, Dimension = 2, Metric = MetricType.Cosine, Spec = new ServerlessIndexSpec { Serverless = new ServerlessSpec { Cloud = ServerlessSpecCloud.Aws, Region = "us-east-1" } }, DeletionProtection = DeletionProtection.Disabled, Tags = new Dictionary { { "environment", "development" } } }); Console.WriteLine("Dense index model:" + createDenseIndexRequest); var createSparseIndexRequest = await pc.CreateIndexAsync(new CreateIndexRequest { Name = sparseIndexName, VectorType = VectorType.Sparse, Metric = MetricType.Dotproduct, Spec = new ServerlessIndexSpec { Serverless = new ServerlessSpec { Cloud = ServerlessSpecCloud.Aws, Region = "us-east-1" } }, DeletionProtection = DeletionProtection.Disabled, Tags = new Dictionary { { "environment", "development" } } }); Console.WriteLine("\nSparse index model:" + createSparseIndexRequest); // Target the indexes var denseIndex = pc.Index(denseIndexName); var sparseIndex = pc.Index(sparseIndexName); // Upsert records into the dense index var denseUpsertResponse = await denseIndex.UpsertAsync(new UpsertRequest() { Namespace = "example-namespace", Vectors = new List { new Vector { Id = "vec1", Values = new ReadOnlyMemory([1.0f, -2.5f]), Metadata = new Metadata { ["genre"] = new("drama"), }, }, new Vector { Id = "vec2", Values = new ReadOnlyMemory([3.0f, -2.0f]), Metadata = new Metadata { ["genre"] = new("documentary"), }, }, new Vector { Id = "vec3", Values = new ReadOnlyMemory([0.5f, -1.5f]), Metadata = new Metadata { ["genre"] = new("documentary"), } } } }); Console.WriteLine($"\nUpserted {denseUpsertResponse.UpsertedCount} dense vectors"); // Upsert records into the sparse index var sparseVector1 = new Vector { Id = "vec1", SparseValues = new SparseValues { Indices = new uint[] { 822745112, 1009084850, 1221765879, 1408993854, 1504846510, 1596856843, 1640781426, 1656251611, 1807131503, 2543655733, 2902766088, 2909307736, 3246437992, 3517203014, 3590924191 }, Values = new ReadOnlyMemory([1.7958984f, 0.41577148f, 2.828125f, 2.8027344f, 2.8691406f, 1.6533203f, 5.3671875f, 1.3046875f, 0.49780273f, 0.5722656f, 2.71875f, 3.0820312f, 2.5019531f, 4.4414062f, 3.3554688f]) }, Metadata = new Metadata { ["chunk_text"] = new("AAPL reported a year-over-year revenue increase, expecting stronger Q3 demand for its flagship phones."), ["category"] = new("technology"), ["quarter"] = new("Q3"), }, }; var sparseVector2 = new Vector { Id = "vec2", SparseValues = new SparseValues { Indices = new uint[] { 131900689, 592326839, 710158994, 838729363, 1304885087, 1640781426, 1690623792, 1807131503, 2066971792, 2428553208, 2548600401, 2577534050, 3162218338, 3319279674, 3343062801, 3476647774, 3485013322, 3517203014, 4283091697 }, Values = new ReadOnlyMemory([0.4362793f, 3.3457031f, 2.7714844f, 3.0273438f, 3.3164062f, 5.6015625f, 2.4863281f, 0.38134766f, 1.25f, 2.9609375f, 0.34179688f, 1.4306641f, 0.34375f, 3.3613281f, 1.4404297f, 2.2558594f, 2.2597656f, 4.8710938f, 0.5605469f]) }, Metadata = new Metadata { ["chunk_text"] = new("Analysts suggest that AAPL'\''s upcoming Q4 product launch event might solidify its position in the premium smartphone market."), ["category"] = new("technology"), ["quarter"] = new("Q4"), }, }; var sparseVector3 = new Vector { Id = "vec3", SparseValues = new SparseValues { Indices = new uint[] { 8661920, 350356213, 391213188, 554637446, 1024951234, 1640781426, 1780689102, 1799010313, 2194093370, 2632344667, 2641553256, 2779594451, 3517203014, 3543799498, 3837503950, 4283091697 }, Values = new ReadOnlyMemory([2.6875f, 4.2929688f, 3.609375f, 3.0722656f, 2.1152344f, 5.78125f, 3.7460938f, 3.7363281f, 1.2695312f, 3.4824219f, 0.7207031f, 0.0826416f, 4.671875f, 3.7011719f, 2.796875f, 0.61621094f]) }, Metadata = new Metadata { ["chunk_text"] = new("AAPL'\''s strategic Q3 partnerships with semiconductor suppliers could mitigate component risks and stabilize iPhone production"), ["category"] = new("technology"), ["quarter"] = new("Q3"), }, }; var sparseUpsertResponse = await sparseIndex.UpsertAsync(new UpsertRequest { Vectors = new List { sparseVector1, sparseVector2, sparseVector3 }, Namespace = "example-namespace" }); Console.WriteLine($"\nUpserted {sparseUpsertResponse.UpsertedCount} sparse vectors"); // Check the number of records in each index var denseIndexStatsResponse = await denseIndex.DescribeIndexStatsAsync(new DescribeIndexStatsRequest()); Console.WriteLine("\nDense index stats:" + denseIndexStatsResponse); var sparseIndexStatsResponse = await sparseIndex.DescribeIndexStatsAsync(new DescribeIndexStatsRequest()); Console.WriteLine("\nSparse index stats:" + sparseIndexStatsResponse); // Query the dense index with a metadata filter var denseQueryResponse = await denseIndex.QueryAsync(new QueryRequest { Vector = new ReadOnlyMemory([3.0f, -2.0f]), TopK = 1, Namespace = "example-namespace", Filter = new Metadata { ["genre"] = new Metadata { ["$eq"] = "documentary", } }, IncludeValues = false, IncludeMetadata = true }); Console.WriteLine("\nDense query response:" + denseQueryResponse); // Query the sparse index with a metadata filter var sparseQueryResponse = await sparseIndex.QueryAsync(new QueryRequest { Namespace = "example-namespace", TopK = 1, SparseVector = new SparseValues { Indices = [767227209, 1640781426, 1690623792, 2021799277, 2152645940, 2295025838, 2443437770, 2779594451, 2956155693, 3476647774, 3818127854, 4283091697], Values = new[] { 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f }, }, Filter = new Metadata { ["quarter"] = new Metadata { ["$eq"] = "Q4", } }, IncludeValues = false, IncludeMetadata = true }); Console.WriteLine("\nSparse query response:" + sparseQueryResponse); // Delete the indexes await pc.DeleteIndexAsync(denseIndexName); await pc.DeleteIndexAsync(sparseIndexName); ``` {/* ```rust Rust use pinecone_sdk::models::{Cloud, DeletionProtection, Kind, Metadata, Metric, Value, Vector, WaitPolicy}; use pinecone_sdk::pinecone::PineconeClientConfig; use pinecone_sdk::utils::errors::PineconeError; use std::collections::BTreeMap; #[tokio::main] async fn main() -> Result<(), PineconeError> { // Initialize a client. // No API key is required. // Host and port of the Pinecone Local instance // is required when starting without indexes. let config = PineconeClientConfig { api_key: Some(String::default()), control_plane_host: Some("http://localhost:5080".to_string()), ..Default::default() }; let pinecone = config.client()?; // Create an index let index_model = pinecone.create_serverless_index( "docs-example", // Name of the index 2, // Dimension of the vectors Metric::Cosine, // Distance metric Cloud::Aws, // Cloud provider "us-east-1", // Region DeletionProtection::Disabled, // Deletion protection WaitPolicy::NoWait, // Timeout ).await?; let index_host = format!("http://{}", index_model.host); // Target the index let mut index = pinecone.index(&index_host).await?; // Upsert records into the index let vectors = [Vector { id: "vec1".to_string(), values: vec![1.0, -2.5], sparse_values: None, metadata: Some(Metadata { fields: vec![( "genre".to_string(), Value { kind: Some(Kind::StringValue("drama".to_string())), }, )] .into_iter() .collect(), }), }, Vector { id: "vec2".to_string(), values: vec![3.0, -2.0], sparse_values: None, metadata: Some(Metadata { fields: vec![( "genre".to_string(), Value { kind: Some(Kind::StringValue("documentary".to_string())), }, )] .into_iter() .collect(), }), }, Vector { id: "vec3".to_string(), values: vec![0.5, -1.5], sparse_values: None, metadata: Some(Metadata { fields: vec![( "genre".to_string(), Value { kind: Some(Kind::StringValue("documentary".to_string())), }, )] .into_iter() .collect(), }), }]; index.upsert(&vectors, &"example-namespace".into()).await?; // Check the number of records in the index let index_stats = index.describe_index_stats(None).await?; println!("Index stats:\n{:?}", index_stats); // Query the index with a metadata filter let vector = vec![3.0, -2.0]; let mut fields = BTreeMap::new(); let kind = Some(Kind::StringValue("documentary".to_string())); fields.insert("genre".to_string(), Value { kind }); let query_response = index.query_by_value( vector, None, 3, &"example-namespace".into(), Some(Metadata { fields }), None, Some(true), ).await?; println!("Query response:\n{:?}", query_response); // Delete the index pinecone.delete_index(&index_model.name).await?; Ok(()) } ``` */} ```shell curl PINECONE_LOCAL_HOST="localhost:5080" DENSE_INDEX_HOST="localhost:5081" SPARSE_INDEX_HOST="localhost:5082" # Create two indexes, one dense and one sparse curl -X POST "http://$PINECONE_LOCAL_HOST/indexes" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "X-Pinecone-API-Version: 2025-01" \ -d '{ "name": "dense-index", "vector_type": "dense", "dimension": 2, "metric": "cosine", "spec": { "serverless": { "cloud": "aws", "region": "us-east-1" } }, "tags": { "environment": "development" }, "deletion_protection": "disabled" }' curl -X POST "http://$PINECONE_LOCAL_HOST/indexes" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "X-Pinecone-API-Version: 2025-01" \ -d '{ "name": "sparse-index", "vector_type": "sparse", "metric": "dotproduct", "spec": { "serverless": { "cloud": "aws", "region": "us-east-1" } }, "tags": { "environment": "development" }, "deletion_protection": "disabled" }' # Upsert records into the dense index curl -X POST "http://$DENSE_INDEX_HOST/vectors/upsert" \ -H "Content-Type: application/json" \ -H "X-Pinecone-API-Version: 2025-01" \ -d '{ "namespace": "example-namespace", "vectors": [ { "id": "vec1", "values": [1.0, -2.5], "metadata": {"genre": "drama"} }, { "id": "vec2", "values": [3.0, -2.0], "metadata": {"genre": "documentary"} }, { "id": "vec3", "values": [0.5, -1.5], "metadata": {"genre": "documentary"} } ] }' # Upsert records into the sparse index curl -X POST "http://$SPARSE_INDEX_HOST/vectors/upsert" \ -H "Content-Type: application/json" \ -H "X-Pinecone-API-Version: 2025-01" \ -d '{ "namespace": "example-namespace", "vectors": [ { "id": "vec1", "sparseValues": { "values": [1.7958984, 0.41577148, 2.828125, 2.8027344, 2.8691406, 1.6533203, 5.3671875, 1.3046875, 0.49780273, 0.5722656, 2.71875, 3.0820312, 2.5019531, 4.4414062, 3.3554688], "indices": [822745112, 1009084850, 1221765879, 1408993854, 1504846510, 1596856843, 1640781426, 1656251611, 1807131503, 2543655733, 2902766088, 2909307736, 3246437992, 3517203014, 3590924191] }, "metadata": { "chunk_text": "AAPL reported a year-over-year revenue increase, expecting stronger Q3 demand for its flagship phones.", "category": "technology", "quarter": "Q3" } }, { "id": "vec2", "sparseValues": { "values": [0.4362793, 3.3457031, 2.7714844, 3.0273438, 3.3164062, 5.6015625, 2.4863281, 0.38134766, 1.25, 2.9609375, 0.34179688, 1.4306641, 0.34375, 3.3613281, 1.4404297, 2.2558594, 2.2597656, 4.8710938, 0.5605469], "indices": [131900689, 592326839, 710158994, 838729363, 1304885087, 1640781426, 1690623792, 1807131503, 2066971792, 2428553208, 2548600401, 2577534050, 3162218338, 3319279674, 3343062801, 3476647774, 3485013322, 3517203014, 4283091697] }, "metadata": { "chunk_text": "Analysts suggest that AAPL'\''s upcoming Q4 product launch event might solidify its position in the premium smartphone market.", "category": "technology", "quarter": "Q4" } }, { "id": "vec3", "sparseValues": { "values": [2.6875, 4.2929688, 3.609375, 3.0722656, 2.1152344, 5.78125, 3.7460938, 3.7363281, 1.2695312, 3.4824219, 0.7207031, 0.0826416, 4.671875, 3.7011719, 2.796875, 0.61621094], "indices": [8661920, 350356213, 391213188, 554637446, 1024951234, 1640781426, 1780689102, 1799010313, 2194093370, 2632344667, 2641553256, 2779594451, 3517203014, 3543799498, 3837503950, 4283091697] }, "metadata": { "chunk_text": "AAPL'\''s strategic Q3 partnerships with semiconductor suppliers could mitigate component risks and stabilize iPhone production", "category": "technology", "quarter": "Q3" } } ] }' # Check the number of records in each index curl -X POST "http://$DENSE_INDEX_HOST/describe_index_stats" \ -H "Content-Type: application/json" \ -H "X-Pinecone-API-Version: 2025-01" \ -d '{}' curl -X POST "http://$SPARSE_INDEX_HOST/describe_index_stats" \ -H "Content-Type: application/json" \ -H "X-Pinecone-API-Version: 2025-01" \ -d '{}' # Query the dense index with a metadata filter curl "http://$DENSE_INDEX_HOST/query" \ -H "Content-Type: application/json" \ -H "X-Pinecone-API-Version: 2025-01" \ -d '{ "vector": [3.0, -2.0], "filter": {"genre": {"$eq": "documentary"}}, "topK": 1, "includeMetadata": true, "includeValues": false, "namespace": "example-namespace" }' # Query the sparse index with a metadata filter curl "http://$SPARSE_INDEX_HOST/query" \ -H "Content-Type: application/json" \ -H "X-Pinecone-API-Version: 2025-01" \ -d '{ "sparseVector": { "values": [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0], "indices": [767227209, 1640781426, 1690623792, 2021799277, 2152645940, 2295025838, 2443437770, 2779594451, 2956155693, 3476647774, 3818127854, 4283091697] }, "filter": {"quarter": {"$eq": "Q4"}}, "namespace": "example-namespace", "topK": 1, "includeMetadata": true, "includeValues": false }' # Delete the index curl -X DELETE "http://$PINECONE_LOCAL_HOST/indexes/dense-index" \ -H "X-Pinecone-API-Version: 2025-01" curl -X DELETE "http://$PINECONE_LOCAL_HOST/indexes/sparse-index" \ -H "X-Pinecone-API-Version: 2025-01" ``` ## 2. Set up GitHub Actions [Set up a GitHub Actions workflow](https://docs.github.com/en/actions/writing-workflows/quickstart) to do the following: 1. Pull the Pinecone Local Docker image. 2. Start a Pinecone Local instance for each test run. 3. Execute tests against the local instance. 4. Tear down the instance after tests complete. Here's a sample GitHub Actions workflow that you can extend for your own needs: ```yaml name: CI/CD with Pinecone Local on: pull_request: branches: - main push: branches: - main jobs: pc-local-tests: name: Pinecone Local tests runs-on: ubuntu-latest services: pc-local: image: ghcr.io/pinecone-io/pinecone-local:latest env: PORT: 5080 ports: - "5080-6000:5080-6000" steps: - name: Check out repository code uses: actions/checkout@v4 - name: Install dependencies run: | python -m pip install --upgrade pip pip install "pinecone[grpc]" - name: Run tests run: | pytest test/ ``` ## 3. Run your tests GitHub Actions will automaticaly run your tests against Pinecone Local when the events you specified in your workflow occur. For a list of the events that can trigger a workflow and more details about using GitHub Actions for CI/CD, see the [GitHub Actions documentation](https://docs.github.com/en/actions). # Bring your own cloud Source: https://docs.pinecone.io/guides/production/bring-your-own-cloud Bring your own cloud (BYOC) lets you deploy Pinecone Database in your own AWS or GCP account to ensure data sovereignty and compliance, with Pinecone handling provisioning, operations, and maintenance. BYOC is in [public preview](/release-notes/feature-availability) on AWS and GCP. To learn more about the offering, [contact Pinecone](https://www.pinecone.io/contact/?contact_form_inquiry_type=Product+Information). ## Use cases Pinecone BYOC is designed for organizations with high security and compliance requirements, for example: * **Data sovereignty**: If your organization has strict data governance policies, Pinecone BYOC can help ensure that all data is stored and processed locally and does not leave your security perimeter. * **Data residency**: The standard Pinecone managed service can be deployed in several [AWS or GCP cloud regions](/guides/index-data/create-an-index#cloud-regions). If your organization has specific data residency or latency constraints that require you to deploy in regions that Pinecone does not yet support, Pinecone BYOC gives you that flexibility. ## Architecture The BYOC architecture employs a split model: * **Data plane**: The data plane is responsible for storing and processing your records, executing queries, and interacting with object storage for index data. In a BYOC deployment, the data plane is hosted in your own AWS or GCP account within a dedicated VPC, ensuring that all data is stored and processed locally and does not leave your organizational boundaries. You use a [private endpoint](#configure-a-private-endpoint) (AWS PrivateLink or GCP Private Service Connect) as an additional measure to secure requests to your indexes. * **Control plane**: The control plane is responsible for managing the index lifecycle as well as region-agnostic services such as user management, authentication, and billing. The control plane does not hold or process any records. In a BYOC deployment, the control plane is managed by Pinecone and hosted globally. Communication between the data plane and control plane is encrypted using TLS and employs role-based access control (RBAC) with minimal IAM permissions. ## Onboarding The onboarding process for BYOC in AWS or GCP involves the following general stages: If you don't already have an AWS or GCP account where you want to deploy Pinecone, you create one for this purpose. You download and run a Terraform template provided by Pinecone. This template creates essential resources, including an IAM role with scoped-down permissions and a trust relationship with Pinecone's AWS or GCP account. Pinecone deploys a data plane cluster within a dedicated VPC in your AWS or GCP account, and you [configure a private endpoint](#configure-a-private-endpoint) for securely connecting to your indexes via AWS PrivateLink or GCP Private Service Connect. Once the environment is operational, Pinecone performs validation tests to ensure proper functionality. ## Configure a private endpoint You use a private endpoint to securely connect to your BYOC indexes. On AWS, you use the [AWS PrivateLink](https://docs.aws.amazon.com/vpc/latest/privatelink/what-is-privatelink.html) service; on GCP, you use the [GCP Private Service Connect](https://cloud.google.com/vpc/docs/private-service-connect) service. Follow the instructions in the AWS documentation to [create a VPC endpoint](https://docs.aws.amazon.com/vpc/latest/privatelink/create-interface-endpoint.html#create-interface-endpoint-aws) for connecting to your indexes via AWS PrivateLink. * For **Resource configurations**, select the relevant resource for your Pinecone BYOC deployment. * For **Network settings**, select the VPC for your BYOC deployment. * In **Additional settings**, select **Enable DNS name** to allow you to access your indexes using a DNS name. Follow the instructions in the GCP documentation to [create a private endpoint](https://cloud.google.com/vpc/docs/configure-private-service-connect-services#create-endpoint) for connecting to your indexes via GCP Private Service Connect. * Set the **Target service** to the following: ``` projects//regions//serviceAttachments/pinecone-psc ``` * Copy the IP address of the private endpoint. You'll need it later. Follow the instructions in the GCP documentation to [create a private DNS zone](https://cloud.google.com/dns/docs/zones#create-private-zone). * Set the **DNS name** to the following: ``` private..pinecone.io ``` * Select the same VPC network as the private endpoint. Follow the instructions in the GCP documentation to [add a resource record set](https://cloud.google.com/dns/docs/records#add-rrset). * Set the **DNS name** to **\***. * Set the **Resource record type** to **A**. * Set the **Ipv4 Address** to the IP address of the private endpoint. ## Create an index Once your BYOC environment is ready, you can create a BYOC index in the [Pinecone console](https://app.pinecone.io/organizations/-/projects/-/indexes) or via Pinecone API or [Python SDK](/reference/python-sdk). To create a BYOC index, set the `spec` parameter to the environment name provided to you during onboarding, for example: ```python Python {9-11} from pinecone import Pinecone, ByocSpec pc = Pinecone(api_key="YOUR_API_KEY") pc.create_index( name="example-byoc-index", dimension=1536, metric="cosine", spec=ByocSpec( environment="aws-us-east-1-b921" ), deletion_protection="disabled", tags={ "example": "tag" } ) ``` ```shell curl {11-15} curl -s "https://api.pinecone.io/indexes" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "name": "example-byoc-index", "vector_type": "dense", "dimension": 1536, "metric": "cosine", "spec": { "byoc": { "environment": "aws-us-east-1-b921" } }, "tags"={ "example": "tag" }, "deletion_protection": "disabled" }' ``` ## Read and write data Once your [private endpoint](#configure-a-private-endpoint) is configured, you can run data operations against a BYOC index as usual, but you must target the index using its private endpoint URL. The only difference in the URL is that `.svc.` is changed to `svc.private.` as shown in the example below. BYOC does not support reading and writing data from the index browser in the [Pinecone console](https://app.pinecone.io/organizations/-/projects/-/indexes/-/browser). ```python Python {8} # pip install "pinecone[grpc]" from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key="YOUR_API_KEY") # Use the Private Endpoint URL for host, which can be found in the # Pinecone console after you select the index to view more details. index = pc.Index(host="https://docs-example-a1b234c.svc.private.aped-4627-b74a.pinecone.io") upsert_response = index.upsert( vectors=[ { "id": "I", "values": [0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1] }, { "id": "J", "values": [0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2] }, { "id": "K", "values": [0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3] }, { "id": "L", "values": [0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4] } ] ) ``` ```bash curl {3} # Use the Private Endpoint URL for host, which can be found in the # Pinecone console after you select the index to view more details. INDEX_PRIVATE_HOST="https://docs-example-a1b234c.svc.private.aped-4627-b74a.pinecone.io" curl "https://$INDEX_PRIVATE_HOST/vectors/upsert" \ -H "Api-Key: $PINECONE_API_KEY" \ -H 'Content-Type: application/json' \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "vectors": [ { "id": "I", "values": [0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1] }, { "id": "J", "values": [0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2] }, { "id": "K", "values": [0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3] }, { "id": "L", "values": [0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4] } ] }' ``` ## Monitoring Pinecone engineers monitor the state of your BYOC deployment and manage incidents if they arise. In addition, you can [monitor performance metrics](/guides/production/monitoring) for your BYOC indexes in the Pinecone Console or with Prometheus or Datadog. To use Prometheus, your monitoring tool must have access to your VPC. ## FAQs In the standard service, Pinecone manages all cloud resources and includes their cost in the service fee. In BYOC, customers provision and pay for cloud resources directly through their AWS or GCP account, providing greater control and data sovereignty as well as access to available AWS or GCP credits or discounts. Also, BYOC does not support integrated inference, which relies on Pinecone's infrastructure for model hosting. Data is stored and processed exclusively within the customer's AWS or GCP account, with encryption applied at rest and in transit. Communication between the data plane and control plane is encrypted using TLS, and access is controlled via RBAC and scoped IAM permissions. AWS PrivateLink or GCP Private Service Connect is used for secure data plane API calls. Currently, BYOC is available in AWS and GCP. Support for Azure is planned for future releases. * BYOC does not support using [integrated embedding](/guides/index-data/indexing-overview#integrated-embedding) to upsert and search with text and have Pinecone generate vectors automatically. Integrated embedding relies on models hosted by Pinecone that are outsite of your AWS or GCP account. * BYOC does not support reading and writing data from the index browser in the [Pinecone console](https://app.pinecone.io/organizations/-/projects/-/indexes/-/browser). You must use the Pinecone API or SDKs instead. # Configure audit logs Source: https://docs.pinecone.io/guides/production/configure-audit-logs This page describes how to configure audit logs in Pinecone. Audit logs provide a detailed record of user, service account, and API actions that occur within Pinecone. Pinecone supports Amazon S3 as a destination for audit logs. To enable and manage audit logs, you must be an [organization owner](/guides/organizations/understanding-organizations#organization-roles). This feature is in [public preview](/release-notes/feature-availability) and available only on [Enterprise plans](https://www.pinecone.io/pricing/). ## Enable audit logs 1. Set up a [IAM policy and role in Amazon S3](/guides/operations/integrations/integrate-with-amazon-s3). 2. Go to [**Settings > Audit logs**](https://app.pinecone.io/organizations/-/settings/logging) in the Pinecone console. 3. Enter the **Role ARN** of the IAM role you created. 4. Enter the name of the Amazon S3 bucket you created. 5. Click **Enable audit logging**. Once you enable audit logs, Pinecone will start writing logs to the S3 bucket. In your bucket, you will also see a file named `audit-log-access-test`, which is a test file that Pinecone writes to verify that it has the necessary permissions to write logs to the bucket. ## View audit logs Logs are written to the S3 bucket approximately every 30 minutes. Each log batch will be saved into its own file as a JSON blob, keyed by the time of the log to be written. Only logs since the integration was created and enabled will be saved. For more information about the log schema and captured events, see [Understanding security - Audit logs](/guides/production/security-overview#audit-logs). ## Edit audit log integration details You can edit the details of the audit log integration in the Pinecone console: 1. Go to [**Settings > Audit logs**](https://app.pinecone.io/organizations/-/settings/logging). 2. Enter the new **Role ARN** or **AWS Bucket**. 3. Click **Update settings**. ## Disable audit logs If you disable audit logs, logs not yet saved will be lost. You can disable audit logs in the Pinecone console: 1. Go to [**Settings > Audit logs**](https://app.pinecone.io/organizations/-/settings/logging). 2. Click the toggle next to **Audit logs are active**. 3. Click **Confirm**. ## Remove audit log integration If you remove the audit log integration, logs not yet saved will be lost. You can remove the audit log integration in the Pinecone console: 1. Go to [**Settings > Audit logs**](https://app.pinecone.io/organizations/-/settings/logging). 2. At the top of the page, click the **ellipsis (...) menu > Remove integration**. 3. Click **Remove integration**. # Configure customer-managed encryption keys Source: https://docs.pinecone.io/guides/production/configure-cmek This page describes how to set up and use customer-managed encryption keys (CMEK) to secure data within a Pinecone project. CMEK allows you to encrypt your data using keys that you manage in your cloud provider's key management system (KMS). Pinecone supports CMEK using Amazon Web Services (AWS) KMS. This feature is in [public preview](/release-notes/feature-availability). ## Set up CMEK using AWS KMS ### Before you begin The following steps assume you have: * Access to the [AWS console](https://console.aws.amazon.com/console/home). * A [Pinecone Enterprise plan](https://www.pinecone.io/pricing/). ### 1. Create a role In the [AWS console](https://console.aws.amazon.com/console/home), create a role that Pinecone can use to access the AWS Key Management System (KMS) key. You can either grant Pinecone access to a key in your account, or if your customers provide their own keys, you can grant access to keys that are outside of your account. 1. Open the [Amazon Identity and Access Management (IAM) console](https://console.aws.amazon.com/iam/). 2. In the navigation pane, click **Roles**. 3. Click **Create role**. 4. In the **Trusted entity type** section, select **Custom trust policy**. 5. In the **Custom trust policy** section, enter one of the following JSON snippets. Pick a snippet based on whether you want to allow Pinecone to assume a role from all regions or from explicit regions. Add an optional external ID for additional security. If you use an external ID, you must provide it to Pinecone when [adding a CMEK key](#add-a-key). ```json JSON { "Version": "2012-10-17", "Statement": [ { "Sid": "AllowPineconeToAssumeIntoRoleFromExplicitRegionswithID", "Effect": "Allow", "Principal": { "AWS": [ // Explicit role per Pinecone region. Replace XXXXXXXXXXXX with Pinecone's AWS account number. "arn:aws:iam::XXXXXXXXXXXX:role/pinecone_cmek_access_us-east-1", "arn:aws:iam::XXXXXXXXXXXX:role/pinecone_cmek_access_us-west-2", "arn:aws:iam::XXXXXXXXXXXX:role/pinecone_cmek_access_eu-west-1" ] }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { // Optional. Replace with a UUID v4 for additional security. If you use an external ID, you must provide it to Pinecone when adding an API key. "sts:ExternalId": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX" } } } ] } ``` ```json JSON { "Version": "2012-10-17", "Statement": [ { "Sid": "AllowPineconeToAssumeIntoRoleFromExplicitRegions", "Effect": "Allow", "Principal": { "AWS": [ // Explicit role per Pinecone region. Replace XXXXXXXXXXXX with Pinecone's AWS account number. "arn:aws:iam::XXXXXXXXXXXX:role/pinecone_cmek_access_us-east-1", "arn:aws:iam::XXXXXXXXXXXX:role/pinecone_cmek_access_us-west-2", "arn:aws:iam::XXXXXXXXXXXX:role/pinecone_cmek_access_eu-west-1" ] }, "Action": "sts:AssumeRole" } ] } ``` ```json JSON { "Version": "2012-10-17", "Statement": [ { "Sid": "AllowPineconeToAssumeIntoRoleFromAllRegions", "Effect": "Allow", "Principal": { "AWS": "*" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { // Optional. Replace with a UUID v4 for additional security. If you use an external ID, you must provide it to Pinecone when adding an API key. "sts:ExternalId": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX" }, "StringLike": { // Replace XXXXXXXXXXXX with Pinecone's AWS account number. "aws:PrincipalArn": "arn:aws:iam::XXXXXXXXXXXX:role/pinecone_cmek_access_*" } } } ] } ``` Replace `XXXXXXXXXXXX` with Pinecone's AWS account number, which can be found by going to [**Manage > CMEK**](https://app.pinecone.io/organizations/-/projects/-/cmek-encryption) in the Pinecone console and clicking **Add CMEK**. 6. Click **Next**. 7. Keep the default permissions as is and click **Next**. 8. Enter a **Role name** and click **Create role**. 9. Copy the **Role ARN** (e.g., `arn:aws:iam::XXXXXX:role/YYYYYY`). This will be used to [create a CMEK-enabled project](#3-create-a-cmek-enabled-project). 1. Open the [Amazon Identity and Access Management (IAM) console](https://console.aws.amazon.com/iam/). 2. In the navigation pane, click **Roles**. 3. Click **Create role**. 4. In the **Trusted entity type** section, select **Custom trust policy**. 5. In the **Custom trust policy** section, enter the following JSON: ```json JSON { "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": [ "kms:Decrypt", "kms:Encrypt" ], "Resource": "arn:aws:kms:*:XXXXXX:key/*" } ] } ``` * Replace `XXXXXX` with the account ID of the customer who owns the key. * Add a `Statement` array for each customer account ID. 6. Click **Next**. 7. Keep the default permissions as is and click **Next**. 8. Enter a **Role name** and click **Create role**. 9. Copy the **Role ARN** (e.g., `arn:aws:iam::XXXXXX:role/YYYYYY`). This will be used to [create a CMEK-enabled project](#3-create-a-cmek-enabled-project). ### 2. Create an AWS KMS key In the [AWS console](https://console.aws.amazon.com/console/home), create the KMS key that Pinecone will use to encrypt your data: 1. Open the [Amazon Key Management Service (KMS) console](https://console.aws.amazon.com/kms/home). 2. In the navigation pane, click **Customer managed keys**. 3. Click **Create key**. 4. In the **Key type** section, select **Symmetric**. 5. In the **Key usage** section, select **Encrypt and decrypt**. 6. Under **Advanced options > Key material origin**, select **KMS**. 7. In the **Regionality** section, select **Single-Region key**. You can create a multi-regional key to safeguard against data loss in case of regional failure. However, Pinecone only accepts one Key ARN per project. If you set a multi-regional key and need to change the Key ARN to switch region, please [contact Support](https://app.pinecone.io/organizations/-/settings/support/ticket) for help. 8. Click **Next**. 9. Enter an **Alias** and click **Next**. 10. Keep the default administrators as is and click **Next**. 11. Select the [role you created](#1-create-a-role) from the **Key users** list and click **Next**. 12. Click **Finish**. 13. Copy the **Key ARN** (e.g., `arn:aws:kms:us-east-1:XXXXXXX:key/YYYYYYY`). This will be used to [create a CMEK-enabled project](#create-a-cmek-enabled-project). ### 3. Create a CMEK-enabled project Once your [role and key is configured](#set-up-cmek-using-aws-kms), you can create a CMEK-enabled project using the Pinecone console: 1. Go to [**Settings > Organization settings > Projects**](https://app.pinecone.io/organizations/-/settings/projects). 2. Click **+Create project**. 3. Enter a **Name**. 4. Select **Encrypt with Customer Managed Encryption Key**. 5. Click **Create project**. 6. Copy and save the generated API key in a secure place for future use. You will not be able to see the API key again after you close the dialog. 7. Click **Close**. ## Add a key To start encrypting your data with a customer-managed key, you need to add the key to the [CMEK-enabled project](#3-create-a-cmek-enabled-project) using the Pinecone console: 1. Go to [**Manage > CMEK**](https://app.pinecone.io/organizations/-/projects/-/cmek-encryption) for the CMEK-enabled project. 2. Click **Add CMEK**. You can only add one key per project, and you cannot change the key in Pinecone once it is set. 3. Enter a **Key name**. 4. Enter the **Role ARN** for the [role you created](#1-create-a-role). 5. Enter a **Key ARN** for the [key you created](#2-create-a-aws-kms-key). 6. If you [created a role](#1-create-a-role) with an external ID, enter the **External ID**. If not, leave this field blank. 7. Click **Create key**. ## Delete a key Before a key can be deleted from a project, all indexes in the project must be deleted. Then, you can delete the key using the Pinecone console: 1. Go to the [Manage > CMEK tab](https://app.pinecone.io/organizations/-/projects/-/cmek-encryption) for the project in which the key was created. 2. For the key you want to delete, click the **ellipsis (..) menu > Delete**. 3. Enter the key name to confirm deletion. 4. Click **Delete key**. ## Limitations * CMEK can be enabled for serverless indexes in AWS regions only. * [Backups](/guides/manage-data/back-up-an-index) are unavailable for indexes created in a CMEK-enabled project. * You cannot change a key once it is set. * You can add only one key per project. # Configure SSO with Okta Source: https://docs.pinecone.io/guides/production/configure-single-sign-on/okta This page shows you how to set up Pinecone with Okta as the single sign-on (SSO) provider. These instructions can be adapted for any provider with SAML 2.0 support. SSO is available only on Enterprise plans. ## Before you begin This page assumes you have the following: * Access to your organization's [Pinecone console](https://login.pinecone.io) as an [organization owner](/guides/organizations/understanding-organizations#organization-owners). * Access to your organization's [Okta Admin console](https://login.okta.com/). ## 1. Create an app integration in Okta 1. In the [Okta Admin console](https://login.okta.com/), navigate to **Applications > Applications**. 2. Click **Create App Integrations**. 3. Select **SAML 2.0**. 4. Click **Next**. 5. Enter the **General Settings**: * **App name**: `Pinecone` * **App logo** (optional) 6. Click **Next**. 7. Enter the **SAML Settings**. For now, use placeholder values: * **Single sign-on URL**: `https:changeme.com` * **Audience URI (SP Entity ID)**: `urn:auth0:production-v2-pinecone-io:changeme` * **Name ID format**: `Unspecified` * **Application username**: `Okta username` * **Update application username on**: `Create and update` The placeholder values will be updated once the SSO keys are created. 8. Click **Finish**. ## 2. Get the application sign on URL 1. In a separate window, navigate to **Applications > Pinecone > Sign On**. 2. Click **More details**. 3. Copy the **Sign on URL**. You will enter this URL in [Step 4](#4-enable-sso-in-pinecone). ## 3. Generate the SAML signing certificate 1. In **Applications > Pinecone > Sign On**, click **Generate new certificate**. 2. For the new certification, click the **ellipsis (...) menu > Download certificate**. You will need to enter this certificate value in [Step 4](#4-enable-sso-in-pinecone). ## 4. Enable SSO in Pinecone 1. In the Pinecone console, go to [**Settings > Manage**](https://app.pinecone.io/organizations/-/settings/manage). 2. In the **Single Sign-On** section, click **Enable SSO**. 3. In the **Login URL** field, enter the URL copied in [Step 2](#2-get-the-application-sign-on-url). 4. In the **Email domain** field, enter your company's email domain. To target multiple domains, enter each domain separated by a comma. 5. In the **Certificate** field, enter the certificate value downloaded in [Step 3](#3-generate-the-saml-signing-certificate). Be sure to enter all of the certificate value, including the `--BEGIN--` and `--END--` tags. 6. Click **Enable SSO**. **SSO Keys** displays. You will need to enter these values in [Step 5](#5-update-the-saml-settings-in-okta). ## 5. Update the SAML settings in Okta 1. Back in the [Okta Admin console](https://login.okta.com/), navigate to **Application Settings > Applications > General**. 2. In the **SAML Settings** section, click **Edit**. 3. Replace the placeholder values with the information from the **SSO Keys** section in [Step 4](#4-enable-sso-in-pinecone): * **Single sign-on URL**: Enter the **AssertionConsumerServiceLocation** value. * **Audience URI (SP Entity ID)**: Enter the **entityId** value. * **Name ID format**: `EmailAddress` 4. In the **Attribute Statements** section, enter the following: * **Name**: `email` * **Value**: `user.email` 5. Click **Next**. 6. Click **Finish**. Okta is now ready to be used for single sign-on. Follow the Okta docs to learn how to [add users and groups](https://help.okta.com/en-us/content/topics/users-groups-profiles/usgp-main.htm). # Configure Private Endpoints for AWS PrivateLink Source: https://docs.pinecone.io/guides/production/connect-to-aws-privatelink This page describes how to create and use [Private Endpoints](/guides/production/security-overview#private-endpoints-for-aws-privatelink) to connect AWS PrivateLink to Pinecone while keeping your VPC private from the public internet. ## Use Private Endpoints to connect to PrivateLink ### Before you begin The following steps assume you have: * Access to the [AWS console](https://console.aws.amazon.com/console/home). * [Created an Amazon VPC](https://docs.aws.amazon.com/vpc/latest/userguide/create-vpc.html#create-vpc-and-other-resources) in the same AWS [region](/guides/index-data/create-an-index#cloud-regions) as the index you want to connect to. You can optionally enable DNS hostnames and resolution, if you want your VPC to automatically discover the DNS CNAME for your PrivateLink and do not want configure a CNAME. * To [configure the routing](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/routing-to-vpc-interface-endpoint.html) yourself, use one of Pinecone's DNS entry for the corresponding region: | Index region | Pinecone DNS entry | | ------------------------- | -------------------------------------- | | `us-east-1` (N. Virginia) | `*.private.aped-4627-b74a.pinecone.io` | | `us-west-2` (Oregon) | `*.private.apw5-4e34-81fa.pinecone.io` | | `eu-west-1` (Ireland) | `*.private.apu-57e2-42f6.pinecone.io` | * A [Pinecone Enterprise plan](https://www.pinecone.io/pricing/). * [Created a serverless index](/guides/index-data/create-an-index#create-a-serverless-index) in the same AWS [region](/guides/index-data/create-an-index#cloud-regions) as your Amazon VPC. Private Endpoints are configured at the project-level and you can add up to 10 endpoints per project. If you have multiple projects in your organization, Private Endpoints need to be set up separately for each. ### 1. Create an Amazon VPC endpoint In the [AWS console](https://console.aws.amazon.com/console/home): 1. Open the [Amazon VPC console](https://console.aws.amazon.com/vpc/). 2. In the navigation pane, click **Endpoint**. 3. Click **Create endpoint**. 4. For **Service category**, select **Other endpoint services**. 5. In **Service settings**, enter the **Service name**, based on the region your Pinecone index is in: | Index region | Service name | | ------------------------- | --------------------------------------------------------- | | `us-east-1` (N. Virginia) | `com.amazonaws.vpce.us-east-1.vpce-svc-05ef6f1f0b9130b54` | | `us-west-2` (Oregon) | `com.amazonaws.vpce.us-west-2.vpce-svc-04ecb9a0e0d5aab01` | | `eu-west-1` (Ireland) | `com.amazonaws.vpce.eu-west-1.vpce-svc-03c6b7e17ff02a70f` | 6. Click **Verify service**. 7. Select the **VPC** to host the endpoint. 8. (Optional) In **Additional settings**, **Enable DNS name**. The enables you to access our service with the DNS name we configure. An additional CNAME record is needed if you disable this option. 9. Select the **Subnets** and **Subnet ID** for the endpoint. 10. Select the **Security groups** to apply to the endpoint. 11. Click **Create endpoint**. 12. Copy the **VPC endpoint ID** (e.g., `vpce-XXXXXXX`). This will be used to [add a Private Endpoint in Pinecone](#2-add-a-private-endpoint-in-pinecone). ### 2. Add a Private Endpoint in Pinecone To add a Private Endpoint using the [Pinecone console](https://app.pinecone.io/organizations/-/projects): 1. Select your project. 2. Go to **Manage > Network**. 3. Click **Add a connection**. 4. Select your VPC region. Only indexes in the selected region in this project will be affected. 5. Click **Next**. 6. Enter the AWS VPC endpoint ID you copied in the [section above](#create-an-amazon-vpc-endpoint). 7. Click **Next**. 8. (optional) To **enable VPC endpoint access only**, turn the toggle on. This can also be enabled later. For more information, see [Manage internet access to your project](#optional-manage-internet-access-to-your-project). 9. Click **Finish setup**. Private Endpoints only affect [data plane](/reference/api/2024-10/data-plane) access. [Control plane](/reference/api/2024-10/control-plane) access will continue over the public internet. ## Run data plane commands Once your Private Endpoint is configured, to run data operations against an index (e.g., [`upsert`](/guides/index-data/upsert-data), [`query`](/guides/search/search-overview), [`update`](/guides/manage-data/update-data), [`delete`](/guides/manage-data/delete-data), etc.), you must target the index using the Private Endpoint URL for the index. The only difference in the URL is that `.svc.` is changed to `svc.private.` as shown in the example below. ```python Python {8} # pip install "pinecone[grpc]" from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key="YOUR_API_KEY") # Use the Private Endpoint URL for host, which can be found in the # Pinecone console after you select the index to view more details. index = pc.Index(host="https://docs-example-a1b234c.svc.private.aped-4627-b74a.pinecone.io") upsert_response = index.upsert( vectors=[ { "id": "I", "values": [0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1] }, { "id": "J", "values": [0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2] }, { "id": "K", "values": [0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3] }, { "id": "L", "values": [0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4] } ] ) ``` ```bash curl {3} # Use the Private Endpoint URL for host, which can be found in the # Pinecone console after you select the index to view more details. INDEX_PRIVATE_HOST="https://docs-example-a1b234c.svc.private.aped-4627-b74a.pinecone.io" curl "https://$INDEX_PRIVATE_HOST/vectors/upsert" \ -H "Api-Key: $PINECONE_API_KEY" \ -H 'Content-Type: application/json' \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "vectors": [ { "id": "I", "values": [0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1] }, { "id": "J", "values": [0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2] }, { "id": "K", "values": [0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3] }, { "id": "L", "values": [0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4] } ] }' ``` If you run the same command from outside of the Private Endpoint, you will get an `Unauthorized` response. ## Manage internet access to your project Once your Private Endpoint is configured, you can turn off internet access to your project. To enable VPC endpoint access only: 1. Open the [Pinecone console](https://app.pinecone.io/organizations/-/projects). 2. Select your project. 3. Go to **Network > Access**. 4. Turn the **VPC endpoint access only** toggle on. This will turn off internet access to the project. This can be turned off at any point. This access control is set at the *project-level* and can unintentionally affect Pinecone indexes that communicate via the internet in the same project. Only indexes communicating through Private Endpoints will continue to work. ## Manage Private Endpoints In addition to [creating Private Endpoints](#2-add-a-private-endpoint-in-pinecone), you can also: * [View Private Endpoints](#view-private-endpoints) * [Delete a Private Endpoint](#delete-a-private-endpoint) ### View Private Endpoints To view Private Endpoints using the [Pinecone console](https://app.pinecone.io/organizations/-/projects): 1. Select your project. 2. Go to **Manage > Network**. A list of Private Endpoints displays with the associated **VPC ID** and **Cloud** provider. ### Delete a Private Endpoint To delete a Private Endpoint using the [Pinecone console](https://app.pinecone.io/organizations/-/projects): 1. Select your project. 2. Go to **Manage > Network**. 3. For the Private Endpoint you want to delete, click the *...* (Actions) icon. 4. Click **Delete**. 5. Enter the endpoint name. 6. Click **Delete Endpoint**. # Data deletion on Pinecone Source: https://docs.pinecone.io/guides/production/data-deletion Pinecone follows a secure process to ensure that customer data is permanently deleted from our system. This page gives an overview of the process. As defined in the [Master Subscription Agreement](https://www.pinecone.io/legal/master-subscription-agreement/), customer data is data that you provide to Pinecone through the services of the Pinecone system, or such data provided on your behalf by connected systems. This includes objects such as [records](/guides/get-started/glossary#record), [indexes](/guides/get-started/glossary#index), [backups](/guides/get-started/glossary#backup-or-collection), [projects](/guides/get-started/glossary#project), [API keys](/guides/get-started/glossary#api-key), [users](/guides/get-started/glossary#user), [assistants](/guides/get-started/glossary#pinecone-assistant), and [organizations](/guides/get-started/glossary#organization). ## Deletion request The deletion of customer data begins when you initiate a deletion request through the Pinecone API, console, or a connected service. A deletion request can delete a single resource, such as a record, or can delete a resource and all its dependent resources, such as an index and all its records. Deletion of your customer data also occurs automatically when you end your relationship with Pinecone. ## Soft deletion After you initiate a deletion request, Pinecone marks the data for deletion. The data is not immediately removed from the system. Instead, Pinecone retains the data for a maximum of 90 days. During this period, the data is not accessible to you or any other user. ## Permanent deletion Before the end of the 90-day retention window, Pinecone permanently deletes the data from its system. Once the data is permanently deleted, it is no longer recoverable. Pinecone creates an [audit log](/guides/production/security-overview#audit-logs) of user, service account, and API events. Events are captured within two hours of occurrence and are retained for 90 days, after which they are permanently deleted. ## See also * [Delete records](/guides/manage-data/delete-data) * [Delete an index](/guides/manage-data/manage-indexes#delete-an-index) * [Delete a project](/guides/projects/manage-projects#delete-a-project) * [Delete an API key](/guides/projects/manage-api-keys#delete-an-api-key) * [Delete a user](/guides/projects/manage-project-members#remove-members) * [Delete an organization](/troubleshooting/delete-your-organization) * [Master Subscription Agreement](https://www.pinecone.io/legal/master-subscription-agreement/) # Monitor performance Source: https://docs.pinecone.io/guides/production/monitoring Pinecone generates time-series performance metrics for each Pinecone index. You can monitor these metrics directly in the Pinecone console or with tools like Prometheus or Datadog. ## Monitor in the Pinecone Console To view performance metrics in the Pinecone console: 1. Open the [Pinecone console](https://app.pinecone.io/organizations/-/projects). 2. Select the project containing the index you want to monitor. 3. Go to **Database > Indexes**. 4. Select the index. 5. Go to the **Metrics** tab. ## Monitor with Datadog To monitor Pinecone with Datadog, use Datadog's [Pinecone integration](/integrations/datadog). This feature is available on the [Standard, Enterprise, and Dedicated plans](https://www.pinecone.io/pricing/). ## Monitor with Prometheus This feature is available on the [Standard, Enterprise, and Dedicated plans](https://www.pinecone.io/pricing/). On the Dedicated plan (i.e., [Bring Your Own Cloud](/guides/production/bring-your-own-cloud)), you must configure Prometheus monitoring within your VPC. ### Configuration To monitor all serverless indexes in a project, insert the following snippet into the [`scrape_configs`](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config) section of your `prometheus.yml` file and update it with values for your Prometheus integration: This method uses [HTTP service discovery](https://prometheus.io/docs/prometheus/latest/http_sd/) to automatically discover and target all serverless indexes across all regions in a project. ```YAML global: scrape_interval: 15s scrape_configs: - job_name: 'pinecone-serverless-metrics' http_sd_configs: - url: https://api.pinecone.io/prometheus/projects/PROJECT_ID/metrics/discovery refresh_interval: 1m authorization: type: Bearer credentials: API_KEY authorization: type: Bearer credentials: API_KEY ``` * Replace `PROJECT_ID` with the unique ID of the project you want to monitor. You can [find the project ID](/guides/projects/understanding-projects#project-ids) in the Pinecone console. * Replace both instances of `API_KEY` with an API key for the project you want to monitor. The first instance is for service discovery, and the second instance is for the discovered targets. If necessary, you can [create an new API key](/guides/projects/manage-api-keys) in the Pinecone console. To monitor all pod-based indexes in a specific region of a project, insert the following snippet into the [`scrape_configs`](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config) section of your `prometheus.yml` file and update it with values for your Prometheus integration: ```YAML scrape_configs: - job_name: "pinecone-pod-metrics" scheme: https metrics_path: '/metrics' authorization: credentials: API_KEY static_configs: - targets: ["metrics.ENVIRONMENT.pinecone.io" ] ``` * Replace `API_KEY` with an API key for the project you want to monitor. If necessary, you can [create an new API key](/reference/api/authentication) in the Pinecone console. * Replace `ENVIRONMENT` with the [environment](/guides/indexes/pods/understanding-pod-based-indexes#pod-environments) of the pod-based indexes you want to monitor. For more configuration details, see the [Prometheus docs](https://prometheus.io/docs/prometheus/latest/configuration/configuration/). ### Available metrics The following metrics are available when you integrate Pinecone with Prometheus: | Name | Type | Description | | :------------------------------------- | :------ | :------------------------------------------------------------------------------------------------------------- | | `pinecone_db_record_total` | gauge | The total number of records in the index. | | `pinecone_db_op_upsert_total` | counter | The number of [upsert](/guides/index-data/upsert-data) requests made to an index. | | `pinecone_db_op_upsert_duration_total` | counter | The total time taken processing upsert requests for an index in milliseconds. | | `pinecone_db_op_query_total` | counter | The number of [query](/guides/search/search-overview) requests made to an index. | | `pinecone_db_op_query_duration_total` | counter | The total time taken processing [query](/guides/search/search-overview) requests for an index in milliseconds. | | `pinecone_db_op_fetch_total` | counter | The number of [fetch](/guides/manage-data/fetch-data) requests made to an index. | | `pinecone_db_op_fetch_duration_total` | counter | The total time taken processing fetch requests for an index in milliseconds. | | `pinecone_db_op_update_total` | counter | The number of [update](/guides/manage-data/update-data) requests made to an index. | | `pinecone_db_op_update_duration_total` | counter | The total time taken processing update requests for an index in milliseconds. | | `pinecone_db_op_delete_total` | counter | The number of [delete](/guides/manage-data/delete-data) requests made to an index. | | `pinecone_db_op_delete_duration_total` | counter | The total time taken processing delete requests for an index in milliseconds. | | `pinecone_db_write_unit_total` | counter | The total number of [write units](/guides/manage-cost/understanding-cost#write-units) consumed by an index. | | `pinecone_db_read_unit_total` | counter | The total number of [read units](/guides/manage-cost/understanding-cost#read-units) consumed by an index. | | `pinecone_db_storage_size_bytes` | gauge | The total size of the index in bytes. | | Name | Type | Description | | :----------------------------------- | :-------- | :-------------------------------------------------------------------------------- | | `pinecone_vector_count` | gauge | The number of records per pod in the index. | | `pinecone_request_count_total` | counter | The number of data plane calls made by clients. | | `pinecone_request_error_count_total` | counter | The number of data plane calls made by clients that resulted in errors. | | `pinecone_request_latency_seconds` | histogram | The distribution of server-side processing latency for pinecone data plane calls. | | `pinecone_index_fullness` | gauge | The fullness of the index on a scale of 0 to 1. | ### Metric labels Each metric contains the following labels: | Label | Description | | :-------------- | :----------------------------------------------------------- | | `index_name` | Name of the index to which the metric applies. | | `cloud` | Cloud where the index is deployed: `aws`, `gcp`, or `azure`. | | `region` | Region where the index is deployed. | | `capacity_mode` | Type of index: `serverless` or `byoc`. | | Label | Description | | :------------- | :--------------------------------------------------------------------------------------------------------------------------------------------- | | `pid` | Process identifier. | | `index_name` | Name of the index to which the metric applies. | | `project_name` | Name of the project containing the index. | | `request_type` | Type of request: `upsert`, `delete`, `fetch`, `query`, or `describe_index_stats`. This label is included only in `pinecone_request_*` metrics. | ### Example queries Return the total number of records per index: ```shell sum by (index_name) (pinecone_db_record_total) ``` Return the total number of records in Pinecone index `docs-example`: ```shell pinecone_db_record_total{index_name="docs-example"} ``` Return the total number of upsert requests per index: ```shell sum by (index_name) (pinecone_db_op_upsert_total) ``` Return the average processing time in millisconds for upsert requests per index: ```shell sum by (index_name) (pinecone_db_op_upsert_duration_total/pinecone_db_op_upsert_total) ``` Return the total read units consumed per index: ```shell sum by (index_name) (pinecone_db_read_unit_total) ``` Return the total write units consumed for the Pinecone index `docs-example`: ```shell pinecone_db_write_unit_total{index_name="docs-example"} ``` Return the average latency in seconds for all requests against the Pinecone index `docs-example`: ```shell avg by (request_type) (pinecone_request_latency_seconds{index_name="docs-example"}) ``` Return the vector count for the Pinecone index `docs-example`: ```shell sum ((avg by (app) (pinecone_vector_count{index_name="docs-example"}))) ``` Return the total number of requests against the Pinecone index `docs-example` over one minute: ```shell sum by (request_type)(increase(pinecone_request_count_total{index_name="docs-example"}[60s])) ``` Return the total number of upsert requests against the Pinecone index `docs-example` over one minute: ```shell sum by (request_type)(increase(pinecone_request_count_total{index_name="docs-example", request_type="upsert"}[60s])) ``` Return the total errors returned by the Pinecone index `docs-example` over one minute: ```shell sum by (request_type) (increase(pinecone_request_error_count{ index_name="docs-example"}[60s])) ``` Return the index fullness metric for the Pinecone index `docs-example`: ``` round(max (pinecone_index_fullness{index_name="docs-example"} * 100)) ``` # Production checklist Source: https://docs.pinecone.io/guides/production/production-checklist This page provides recommendations and best practices for preparing your Pinecone indexes for production, anticipating production issues, and enabling reliability and growth. For high-scale use cases, consider using the [Pinecone AWS Reference Architecture](https://github.com/pinecone-io/aws-reference-architecture-pulumi) as a starting point, and read up on [code best practices](https://www.pinecone.io/blog/working-at-scale). ## Prepare your project structure One of the first steps towards building a production-ready Pinecone index is configuring your project correctly. * Consider [creating a separate project](/guides/projects/create-a-project) for your development and production indexes, to allow for testing changes to your index before deploying them to production. * Ensure that you have properly [configured user access](/guides/projects/understanding-projects#project-roles) to the Pinecone console, so that only those users who need to access the production index can do so. * Ensure that you have properly configured access through the API by [managing API keys](/guides/projects/manage-api-keys) and using API key permissions. Consider how best to [manage the API keys](/guides/projects/manage-api-keys) associated with your production project. In order to [make calls to the Pinecone API](/guides/get-started/quickstart), you must provide a valid API key for the relevant Pinecone project. ## Test your query results Before you move your index to production, make sure that your index is returning accurate results in the context of your application by [identifying the appropriate metrics](https://www.pinecone.io/learn/offline-evaluation/) for evaluating your results. ## Target indexes by host Before moving your project to production, make sure that you are [targeting indexes by host](/guides/manage-data/target-an-index) rather than by name. ## Backup up your indexes In order to enable long-term retention, compliance archiving, and deployment of new indexes, consider backing up your production indexes by [creating a backup or collection](/guides/manage-data/back-up-an-index). ## Enforce security Use Pinecone's [security features](/guides/production/security-overview) to protect your production data: * Data security * Private endpoints * Customer-managed encryption keys (CMEK) * Authorization * API keys * Role-based access control (RBAC) * Organization single sign-on (SSO) * Audit logs * Bring your own cloud ## Tune for performance Before serving production workloads, identify ways to [increase search relevance](/guides/optimize/increase-relevance), [increase throughput](/guides/optimize/increase-throughput), and [decrease latency](/guides/optimize/decrease-latency). ## Configure monitoring Prepare to [monitor the production performance and availability of your indexes](/guides/production/monitoring). ## Configure CI/CD Use [Pinecone in CI/CD](/guides/production/automated-testing) to safely test changes before deploying them to production. ## Know how to get support If you need help, [contact Support](https://app.pinecone.io/organizations/-/settings/support/ticket), or talk to the [Pinecone community](https://www.pinecone.io/community/). Ensure that your [plan tier](https://www.pinecone.io/pricing/) matches the support and availability SLAs you need. This may require you to upgrade to Enterprise. # Security overview Source: https://docs.pinecone.io/guides/production/security-overview This page describes Pinecone's security protocols, practices, and features. ## Access management ### API keys Each Pinecone [project](/guides/projects/understanding-projects) has one or more [API keys](/guides/projects/manage-api-keys). In order to make calls to the Pinecone API, a user must provide a valid API key for the relevant Pinecone project. You can [manage API key permissions](/guides/projects/manage-api-keys) in the [Pinecone console](https://app.pinecone.io/organizations/-/projects/-/keys). The available permission roles are as follows: #### General permissions | Role | Permissions | | :--- | :---------------------------------------------- | | All | Permissions to read and write all project data. | #### Control plane permissions | Role | Permissions | | :-------- | :---------------------------------------------------------------------------------------------------------- | | ReadWrite | Permissions to list, describe, create, delete, and configure indexes, backups, collections, and assistants. | | ReadOnly | Permissions to list and describe indexes, backups, collections, and assistants. | | None | No control plane permissions. | #### Data plane permissions For pod-based indexes, the data plane is limited to ReadWrite. | Role | Permissions | | :-------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ReadWrite |
  • Indexes: Permissions to query, import, fetch, add, update, and delete index data.
  • Pinecone Assistant: Permissions to add, list, view, and delete files; chat with an assistant, and evaluate responses.
  • Pinecone Inference: Permissions to generate embeddings and rerank documents.
| | ReadOnly |
  • Indexes: Permissions to query, fetch, list ID, and view stats.
  • Pinecone Assistant: Permissions to list and view files, chat with an assistant, and evaluate responses.
| | None | No data plane permissions. | ### Organization single sign-on (SSO) SSO allows organizations to manage their teams' access to Pinecone through their identity management solution. Once your integration is configured, you can require that users from your domain sign in through SSO, and you can specify a default role for teammates when they sign up. Only organizations in the Enterprise dedicated tier can set up SSO. For more information, see [configure single sign on](/guides/production/configure-single-sign-on/okta). ### Role-based access controls (RBAC) Pinecone uses role-based access controls (RBAC) to manage access to resources. Service accounts, API keys, and users are all *principals*. A principal's access is determined by the *roles* assigned to it. Roles are assigned to a principal for a *resource*, either a project or an organization. The roles available to be assigned depend on the type of principal and resource. #### Service account roles A service account can be assigned roles for the organization it belongs to, and any projects within that organization. For more information, see [Organization roles](/guides/organizations/understanding-organizations#organization-roles) and [Project roles](/guides/projects/understanding-projects#project-roles). #### API key roles An API key can only be assigned permissions for the projects it belongs to. For more information, see [API keys](#api-keys). #### User roles A user can be assigned roles for each organization they belong to, and any projects within that organization. For more information, see [Organization roles](/guides/organizations/understanding-organizations#organization-roles) and [Project roles](/guides/projects/understanding-projects#project-roles). ## Compliance To learn more about data privacy and compliance at Pinecone, visit the [Pinecone Trust and Security Center](https://security.pinecone.io/). ### Audit logs This feature is in [public preview](/release-notes/feature-availability) and available only on [Enterprise plans](https://www.pinecone.io/pricing/). [Audit logs](/guides/production/configure-audit-logs) provide a detailed record of user and API actions that occur within Pinecone. Events are captured every 30 minutes and each log batch will be saved into its own file as a JSON blob, keyed by the time of the log to be written. Only logs since the integration was created and enabled will be saved. Audit log events adhere to a standard JSON schema and include the following fields: ```json JSON { "id": "00000000-0000-0000-0000-000000000000", "organization_id": "AA1bbbbCCdd2EEEe3FF", "organization_name": "example-org", "client": { "userAgent": "rawUserAgent" }, "actor": { "principal_id": "00000000-0000-0000-0000-000000000000", "principal_name": "example@pinecone.io", "principal_type": "user", // user, api_key, service_account "display_name": "Example Person" // Only in case of user }, "event": { "time": "2024-10-21T20:51:53.697Z", "action": "create", "resource_type": "index", "resource_id": "uuid", "resource_name": "docs-example", "outcome": { "result": "success", "reason": "", // Only displays for "result": "failure" "error_code": "", // Only displays for "result": "failure" }, "parameters": { // Varies based on event } } } ``` The following events are captured in the audit logs: * [Organization events](#organization-events) * [Project events](#project-events) * [Index events](#index-events) * [User and API key events](#user-and-api-key-events) * [Security and governance events](#security-and-governance-events) #### Organization events | Action | Query parameters | | ----------------- | -------------------------------------------------------------------------------------------------------------- | | Rename org | `event.action: update`, `event.resource_type: organization`, `event.resource_id: NEW_ORG_NAME` | | Delete org | `event.action: delete`, `event.resource_type: organization`, `event.resource_id: DELETED_ORG_NAME` | | Create org member | `event.action: create`, `event.resource_type: user`, `event.resource_id: [ARRAY_OF_USER_EMAILS]` | | Update org member | `event.action: update`, `event.resource_type: user`, `event.resource_id: { user: USER_EMAIL, role: NEW_ROLE }` | | Delete org member | `event.action: delete`, `event.resource_type: user`, `event.resource_id: USER_EMAIL` | #### Project events | Action | Query parameters | | -------------------------- | ------------------------------------------------------------------------------------------------------------------ | | Create project | `event.action: create`, `event.resource_type: project`, `event.resouce_id: PROJ_NAME` | | Update project | `event.action: update`, `event.resource_type: project`, `event.resource_id: PROJECT_NAME` | | Delete project | `event.action: delete`, `event.resource_type: project`, `event.resource_id: PROJECT_NAME` | | Invite project member | `event.action: create`, `event.resource_type: user`, `event.resource_id: [ARRAY_OF_USER_EMAILS]` | | Update project member role | `event.action: update`, `event.resource_type: user`, `event.resource_id: { user: USER_EMAIL, role: NEW_ROLE }` | | Delete project member | `event.action: delete`, `event.resource_type: user`, `event.resource_id: { user: USER_EMAIL, project: PROJ_NAME }` | #### Index events | Action | Query parameters | | ------------- | --------------------------------------------------------------------------------------- | | Create index | `event.action: create`, `event.resource_type: index`, `event.resouce_id: INDEX_NAME` | | Update index | `event.action: update`, `event.resource_type: index`, `event.resource_id: INDEX_NAME` | | Delete index | `event.action: delete`, `event.resource_type: index`, `event.resource_id: INDEX_NAME` | | Create backup | `event.action: create`, `event.resource_type: backup`, `event.resource_id: BACKUP_NAME` | | Delete backup | `event.action: delete`, `event.resource_type: backup`, `event.resource_id: BACKUP_NAME` | #### User and API key events | Action | Query parameters | | -------------- | --------------------------------------------------------------------------------------- | | User login | `event.action: login`, `event.resource_type: user`, `event.resouce_id: USERNAME` | | Create API key | `event.action: create`, `event.resource_type: api-key`, `event.resource_id: API_KEY_ID` | | Delete API key | `event.action: delete`, `event.resource_type: api-key`, `event.resource_id: API_KEY_ID` | #### Security and governance events | Action | Query parameters | | ----------------------- | ---------------------------------------------------------------------------------------------------------- | | Create Private Endpoint | `event.action: create`, `event.resource_type: private-endpoints`, `event.resource_id: PRIVATE_ENDPOINT_ID` | | Delete Private Endpoint | `event.action: delete`, `event.resource_type: private-endpoints`, `event.resource_id: PRIVATE_ENDPOINT_ID` | ## Data protection ### Customer-managed encryption keys (CMEK) This feature is in [public preview](/release-notes/feature-availability). Data within a Pinecone project can be encrypted using [customer-managed encryption keys (CMEK)](/guides/production/configure-cmek). This allows you to encrypt your data using keys that you manage in your cloud provider's key management system (KMS). Pinecone supports CMEK using Amazon Web Services (AWS) KMS. ### Backup and recovery Serverless index [backups](/guides/manage-data/backups-overview) are in [public preview](/release-notes/feature-availability) and available only on [Standard and Enterprise plans](https://www.pinecone.io/pricing/). A backup or collection is a static copy of your index that only consumes storage. It is a non-queryable representation of a set of records. You can [create a backup or collection](/guides/manage-data/back-up-an-index) of an index, and you can [create a new index from a backup or collection](/guides/manage-data/restore-an-index). This allows you to restore the index with the same or different configurations. For more information, see [Understanding backups](/guides/manage-data/backups-overview) and [Understanding collections](/guides/indexes/pods/understanding-collections). ### Encryption at rest Pinecone encrypts stored data using the 256-bit Advanced Encryption Standard (AES-256) encryption algorithm. ### Encryption in transit Pinecone uses standard protocols to encrypt user data in transit. Clients open HTTPS or gRPC connections to the Pinecone API; the Pinecone API gateway uses gRPC connections to user deployments in the cloud. These HTTPS and gRPC connections use the TLS 1.2 protocol with 256-bit Advanced Encryption Standard (AES-256) encryption. ![Diagram showing encryption protocols for user data in transit](https://mintlify.s3.us-west-1.amazonaws.com/pinecone/images/encryption-in-transit-2.png) Traffic is also encrypted in transit between the Pinecone backend and cloud infrastructure services, such as S3 and GCS. For more information, see [Google Cloud Platform](https://cloud.google.com/docs/security/encryption-in-transit) and [AWS security documentation](https://docs.aws.amazon.com/AmazonS3/userguide/UsingEncryption.html). ## Network security ### Private Endpoints for AWS PrivateLink Use [Private Endpoints to connect to Amazon Web Services (AWS) PrivateLink](/guides/production/connect-to-aws-privatelink). This establishes private connectivity between your Pinecone [serverless indexes](/guides/index-data/indexing-overview#serverless-indexes) and supported AWS services while keeping your VPC private from the public internet. ![PrivateLink diagram](https://mintlify.s3.us-west-1.amazonaws.com/pinecone/images/privatelink.png) Private Endpoints are additive to other Pinecone security features: data is also [encrypted in transit](#encryption-in-transit), [encrypted at rest](#encryption-at-rest), and an [API key](#api-keys) is required to authenticate. ### Proxies The following Pinecone SDKs support the use of proxies: * [Python SDK](/reference/python-sdk#proxy-configuration) * [Node.js SDK](/reference/node-sdk#proxy-configuration) # Create a project Source: https://docs.pinecone.io/guides/projects/create-a-project This page shows you how to create a project. If you are an [organization owner or user](/guides/organizations/understanding-organizations#organization-roles), you can create a project in your organization: 1. In the Pinecone console, got to [**your profile > Organization settings > Projects**](https://app.pinecone.io/organizations/-/settings/projects). 2. Click **+ Create Project**. 3. Enter a **Name**. A project name can contain up to 512 characters. For more information, see [Object identifiers](/reference/api/database-limits#identifier-limits). 4. (Optional) Select **Encrypt with Customer Managed Encryption Key**. For more information, see [Configure CMEK](/guides/production/configure-cmek). 5. Click **Create project**. To load an index with a [sample dataset](/guides/data/use-sample-datasets), click **Load sample data** and follow the prompts. Organizations on the Starter plan are limited to 1 project. To create additional projects, [upgrade to the Standard or Enterprise plan](/guides/organizations/manage-billing/manage-your-billing-plan). An [access token](/guides/organizations/manage-service-accounts#retrieve-an-access-token) must be provided to complete this action through the Admin API. The Admin API is in [public preview](/release-notes/feature-availability). ```bash curl PINECONE_ACCESS_TOKEN="YOUR_ACCESS_TOKEN" curl "https://api.pinecone.io/admin/projects" \ -H "X-Pinecone-Api-Version: 2025-04" \ -H "Authorization: Bearer $PINECONE_ACCESS_TOKEN" \ -d '{ "name":"example-project" }' ``` The example returns a response like the following: ```json { "id": "3fa85f64-5717-4562-b3fc-2c963f66afa6", "name": "example-project", "max_pods": 0, "force_encryption_with_cmek": false, "organization_id": "string", "created_at": "2025-03-16T22:46:45.030Z" } ``` ## Next steps * [Add users to your project](/guides/projects/manage-project-members#add-members-to-a-project) * [Set a project-wide pod limit](/reference/api/database-limits#pods-per-project) for pod-based indexes * [Create an index](/guides/index-data/create-an-index) # Manage API keys Source: https://docs.pinecone.io/guides/projects/manage-api-keys Each Pinecone [project](/guides/projects/understanding-projects) has one or more API keys. In order to [make calls to the Pinecone API](/guides/get-started/quickstart), you must provide a valid API key for the relevant Pinecone project. This page shows you how to [create](#create-an-api-key), [view](#view-api-keys), [change permissions for](#change-api-key-permissions), and [delete](#delete-an-api-key) API keys. If you use custom API key permissions, ensure that you [target your index by host](/guides/manage-data/target-an-index#target-by-index-host-recommended) when performing data operations such as `upsert` and `query`. ## Create an API key You can create a new API key for your project, as follows: 1. Open the [Pinecone console](https://app.pinecone.io/organizations/-/projects). 2. Select your project. 3. Go to **API keys**. 4. Click **Create API key**. 5. Enter an **API key name**. 6. Select the **Permissions** to grant to the API key. For a description of the permission roles, see [API key permissions](/guides/production/security-overview#api-keys). Users on the Starter plan can set the permissions to **All** only. To customize the permissions further, [upgrade to the Standard or Enterprise plan](/guides/organizations/manage-billing/manage-your-billing-plan). 7. Click **Create key**. 8. Copy and save the generated API key in a secure place for future use. You will not be able to see the API key again after you close the dialog. 9. Click **Close**. An [access token](/guides/organizations/manage-service-accounts#retrieve-an-access-token) must be provided to complete this action through the Admin API. The Admin API is in [public preview](/release-notes/feature-availability). ```bash curl PINECONE_ACCESS_TOKEN="YOUR_ACCESS_TOKEN" PINECONE_PROJECT_ID="YOUR_PROJECT_ID" curl "https://api.pinecone.io/admin/projects/$PINECONE_PROJECT_ID/api_keys" \ -H "X-Pinecone-Api-Version: 2025-04" \ -H "Authorization: Bearer $PINECONE_ACCESS_TOKEN" \ -d '{ "name": "Example API Key", "roles": ["ProjectEditor"] }' ``` The example returns a response like the following: ```json { "key": { "id": "3fa85f64-5717-4562-b3fc-2c963f66afa6", "name": "Example API key", "project_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6", "roles": [ "ProjectEditor" ] }, "value": "string" } ``` ## View project API keys You can [view the API keys](/reference/api/2025-04/admin/list_api_keys) for your project as in the following example: An [access token](/guides/organizations/manage-service-accounts#retrieve-an-access-token) must be provided to complete this action through the Admin API. The Admin API is in [public preview](/release-notes/feature-availability). ```bash curl PINECONE_ACCESS_TOKEN="YOUR_ACCESS_TOKEN" PROJECT_ID="3fa85f64-5717-4562-b3fc-2c963f66afa6" curl -X GET "https://api.pinecone.io/admin/projects/$PROJECT_ID/api-keys" \ -H "Authorization: Bearer $PINECONE_ACCESS_TOKEN" \ -H "accept: application/json" \ -H "X-Pinecone-Api-Version: 2025-04" ``` The example returns a response like the following: ```json { "data": [ { "id": "3fa85f64-5717-4562-b3fc-2c963f66afa6", "name": "string", "project_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6", "roles": [ "ProjectEditor" ] } ] } ``` You can view the API keys for your project in the Pinecone console, on the [**API keys** tab](https://app.pinecone.io/organizations/-/projects/-/keys). ## View API key details You can [view the details of an API key](/reference/api/2025-04/admin/fetch_api_key), as shown in the following example: An [access token](/guides/organizations/manage-service-accounts#retrieve-an-access-token) must be provided to complete this action through the Admin API. The Admin API is in [public preview](/release-notes/feature-availability). ```bash curl PINECONE_ACCESS_TOKEN="YOUR_ACCESS_TOKEN" PINECONE_API_KEY_ID="3fa85f64-5717-4562-b3fc-2c963f66afa6" curl -X GET "https://api.pinecone.io/admin/api-keys/$PINECONE_API_KEY_ID" \ -H "Authorization: Bearer $PINECONE_ACCESS_TOKEN" \ -H "accept: application/json" \ -H "X-Pinecone-Api-Version: 2025-04" ``` The example returns a response like the following: ```json { "id": "3fa85f64-5717-4562-b3fc-2c963f66afa6", "name": "string", "project_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6", "roles": [ "ProjectEditor" ] } ``` ## Change API key permissions Users on the Starter plan cannot change API key permissions once they are set. Instead, [create a new API key](#create-an-api-key) or [upgrade to the Standard or Enterprise plan](/guides/organizations/manage-billing/manage-your-billing-plan). If you are a [project owner](/guides/projects/understanding-projects#project-roles), you can change API key permissions: 1. Open the [Pinecone console](https://app.pinecone.io/organizations/-/projects). 2. Select your project. 3. Go to the **API keys** tab. 4. In the row of the API key you want to change, click the **ellipsis (...) menu > Manage**. 5. Change the permissions for the API key as needed. For information about the different API key permissions, refer to [Understanding security - API keys](/guides/production/security-overview#api-keys). 6. Click **Update**. ## Delete an API key If you are a [project owner](/guides/projects/understanding-projects#project-roles), you can delete your API key: 1. Open the [Pinecone console](https://app.pinecone.io/organizations/-/projects). 2. Select your project. 3. Go to the **API keys** tab. 4. In the row of the API key you want to change, click the **ellipsis (...) menu > Delete**. 5. Enter the **API key name**. 6. Click **Confirm deletion**. Deleting an API key is irreversible and will immediately disable any applications using the API key. An [access token](/guides/organizations/manage-service-accounts#retrieve-an-access-token) must be provided to complete this action through the Admin API. The Admin API is in [public preview](/release-notes/feature-availability). ```bash curl PINECONE_ACCESS_TOKEN="YOUR_ACCESS_TOKEN" PINECONE_API_KEY_ID="YOUR_KEY_ID" curl -X DELETE "https://api.pinecone.io/admin/api-keys/$PINECONE_API_KEY_ID" \ -H "X-Pinecone-Api-Version: 2025-04" \ -H "Authorization: Bearer $PINECONE_ACCESS_TOKEN" ``` # Manage project members Source: https://docs.pinecone.io/guides/projects/manage-project-members This page shows how to add and manage project members. [Organization owners](/guides/organizations/understanding-organizations#organization-roles) or [project owners](#project-roles) can manage members in a project. Members can be added to a project with different [roles](/guides/projects/understanding-projects#project-roles), which determine their permissions within the project. For information about managing members at the **organization-level**, see [Manage organization members](/guides/organizations/manage-organization-members). ## Add members to a project You can add members to a project in the [Pinecone console](https://app.pinecone.io/organizations/-/projects): 1. Select your project. 2. Go to the [**Manage > Access > Members** tab](https://app.pinecone.io/organizations/-/projects/-/access/members). 3. Enter the member's email address or name. 4. Select a [**Project role**](/guides/projects/understanding-projects#project-roles) for the member. The role determines the member's permissions within Pinecone. 5. Click **Invite**. When you invite a member to join your project, Pinecone sends them an email containing a link that enables them to gain access to the project. If they already have a Pinecone account, they still receive an email, but they can also immediately view the project. ## Change a member's role You can change a member's role in the [Pinecone console](https://app.pinecone.io/organizations/-/projects): 1. Select your project. 2. Go to the [**Manage > Access > Members** tab](https://app.pinecone.io/organizations/-/projects/-/access/members). 3. In the row of the member you want to edit, click **ellipsis (..) menu > Edit role**. 4. Select a [**Project role**](/guides/projects/understanding-projects#project-roles) for the member. 5. Click **Edit role**. ## Remove a member You can remove a member from a project in the [Pinecone console](https://app.pinecone.io/organizations/-/projects): 1. Select your project. 2. Go to the [**Manage > Access > Members** tab](https://app.pinecone.io/organizations/-/projects/-/access/members). 3. In the row of the member you want to remove, click **ellipsis (..) menu > Remove member**. 4. Click **Remove member**. To remove yourself from a project, click the **Leave project** button in your user's row and confirm. # Manage projects Source: https://docs.pinecone.io/guides/projects/manage-projects This page shows you how to rename a project, set a project pod limit, and delete a project. You must be an [organization owner](/guides/organizations/understanding-organizations#organization-roles) or [project owner](/guides/projects/understanding-projects#project-roles) to edit project details or delete a project. ## View project details You can view the details of a project, as in the following example: An [access token](/guides/organizations/manage-service-accounts#retrieve-an-access-token) must be provided to complete this action through the Admin API. The Admin API is in [public preview](/release-notes/feature-availability). ```bash curl PINECONE_ACCESS_TOKEN="YOUR_ACCESS_TOKEN" PROJECT_ID="3fa85f64-5717-4562-b3fc-2c963f66afa6" curl -X GET "https://api.pinecone.io/admin/projects/$PROJECT_ID" \ -H "Authorization: Bearer $PINECONE_ACCESS_TOKEN" \ -H "X-Pinecone-Api-Version: 2025-04" \ -H "accept: application/json" ``` The example returns a response like the following: ```json { "id": "3fa85f64-5717-4562-b3fc-2c963f66afa6", "name": "example-project", "max_pods": 0, "force_encryption_with_cmek": false, "organization_id": "string", "created_at": "2025-03-17T00:30:23.262Z" } ``` You can view project details using the [Pinecone console](https://app.pinecone.io/organizations/-/settings/projects/-/indexes). ## Rename a project You can change the name of your project: 1. In the Pinecone console, got to [**Settings > Projects**](https://app.pinecone.io/organizations/-/settings/projects). 2. Click the **ellipsis (...) menu > Configure** icon next to the project you want to update. 3. Enter a new **Project Name**. A project name can contain up to 512 characters. For more information, see [Identifier limits](/reference/api/database-limits#identifier-limits). 4. Click **Save Changes**. An [access token](/guides/organizations/manage-service-accounts#retrieve-an-access-token) must be provided to complete this action through the Admin API. The Admin API is in [public preview](/release-notes/feature-availability). ```bash curl PINECONE_ACCESS_TOKEN="YOUR_ACCESS_TOKEN" PROJECT_ID="YOUR_PROJECT_ID" curl -X PATCH "https://api.pinecone.io/admin/projects/$PROJECT_ID" \ -H "accept: application/json" \ -H "Content-Type: application/json" \ -H "X-Pinecone-Api-Version: 2025-04" \ -d '{ "name": "updated-example-project" }' ``` The example returns a response like the following: ```json { "id": "3fa85f64-5717-4562-b3fc-2c963f66afa6", "name": "updated-example-project", "max_pods": 0, "force_encryption_with_cmek": false, "organization_id": "string", "created_at": "2025-03-17T00:42:31.912Z" } ``` ## Set a project pod limit Pod limits do not apply to [serverless indexes](/guides/index-data/indexing-overview#serverless-indexes). Pinecone auto-scales serverless indexes based on usage, and you pay only for what you use. You can set or change the default limit on the number of [pods](/guides/indexes/pods/understanding-pod-based-indexes) per project for organizations on Standard and Enterprise plans: 1. Go to [Settings > Projects](https://app.pinecone.io/organizations/-/settings/projects). 2. For the project you want to update, click the **ellipsis (...) menu > Configure**. 3. In the **Pod Limit** section, update the number of pods. 4. Click **Save Changes**. An [access token](/guides/organizations/manage-service-accounts#retrieve-an-access-token) must be provided to complete this action through the Admin API. The Admin API is in [public preview](/release-notes/feature-availability). ```bash curl PINECONE_ACCESS_TOKEN="YOUR_ACCESS_TOKEN" PROJECT_ID="YOUR_PROJECT_ID" curl -X PATCH "https://api.pinecone.io/admin/projects/$PROJECT_ID" \ -H "accept: application/json" \ -H "Content-Type: application/json" \ -H "X-Pinecone-Api-Version: 2025-04" \ -d '{ "max_pods": 5 }' ``` The example returns a response like the following: ```json { "id": "3fa85f64-5717-4562-b3fc-2c963f66afa6", "name": "example-project", "max_pods": 5, "force_encryption_with_cmek": false, "organization_id": "string", "created_at": "2025-03-17T00:42:31.912Z" } ``` ## Delete a project To delete a project, you must first [delete all data](/guides/manage-data/delete-data), [indexes](/guides/manage-data/manage-indexes#delete-an-index), [collections](/guides/indexes/pods/back-up-a-pod-based-index#delete-a-collection), [backups](/guides/manage-data/back-up-an-index#delete-a-backup) and [assistants](/guides/assistant/manage-assistants#delete-an-assistant) associated with the project. Then, you can delete the project itself: 1. In the Pinecone console, got to [**Settings > Projects**](https://app.pinecone.io/organizations/-/settings/projects). 2. For the project you want to delete, click the **ellipsis (...) menu > Delete**. 3. Enter the project name to confirm the deletion. 4. Click **Delete Project**. An [access token](/guides/organizations/manage-service-accounts#retrieve-an-access-token) must be provided to complete this action through the Admin API. The Admin API is in [public preview](/release-notes/feature-availability). ```bash curl PINECONE_ACCESS_TOKEN="YOUR_ACCESS_TOKEN" PROJECT_ID="YOUR_KEY_ID" curl -X DELETE "https://api.pinecone.io/admin/projects/$PROJECT_ID" \ -H "X-Pinecone-Api-Version: 2025-04" \ -H "Authorization: Bearer $PINECONE_ACCESS_TOKEN" ``` # Manage service accounts at the project-level Source: https://docs.pinecone.io/guides/projects/manage-service-accounts This feature is in [public preview](/release-notes/feature-availability) and available only on [Enterprise plans](https://www.pinecone.io/pricing/). This page shows how [organization owners](/guides/organizations/understanding-organizations#organization-roles) and [project owners](/guides/projects/understanding-projects#project-roles) can add and manage service accounts at the project-level. Service accounts enable programmatic access to Pinecone's Admin API, which can be used to create and manage projects and API keys. ## Add a service account to a project After a service account has been [added to an organization](/guides/organizations/manage-service-accounts#create-a-service-account), it can be added to a project in the [Pinecone console](https://app.pinecone.io/organizations/-/projects): 1. Select your project. 2. Go to the [**Manage > Access > Service accounts** tab](https://app.pinecone.io/organizations/-/projects/-/access/service-accounts). 3. Select the service account to add. 4. Select a [**Project role**](/guides/projects/understanding-projects#project-roles) for the service account. The role determines its permissions within Pinecone. 5. Click **Connect**. ## Change project role To change a service account's role in the [Pinecone console](https://app.pinecone.io/organizations/-/projects): 1. Select your project. 2. Go to the [**Manage > Access > Service accounts** tab](https://app.pinecone.io/organizations/-/projects/-/access/service-accounts). 3. In the row of the service account you want to edit, click **ellipsis (..) menu > Edit role**. 4. Select a [**Project role**](/guides/projects/understanding-projects#project-roles) for the service account. 5. Click **Edit role**. ## Remove a service account from a project To remove a service account from a project in the [Pinecone console](https://app.pinecone.io/organizations/-/projects): 1. Select your project. 2. Go to the [**Manage > Access > Service accounts** tab](https://app.pinecone.io/organizations/-/projects/-/access/service-accounts). 3. In the row of the service account you want to remove, click **ellipsis (..) menu > Disconnect**. 4. Enter the service account name to confirm. 5. Click **Disconnect**. # Understanding projects Source: https://docs.pinecone.io/guides/projects/understanding-projects A Pinecone project belongs to an [organization](/guides/organizations/understanding-organizations) and contains a number of [indexes](/guides/index-data/indexing-overview) and users. Only a user who belongs to the project can access the indexes in that project. Each project also has at least one project owner. ## Project environments You choose a cloud environment for each index in a project. This makes it easy to manage related resources across environments and use the same API key to access them. ## Project roles If you are an [organization owner](/guides/organizations/understanding-organizations#organization-roles) or project owner, you can manage members in your project. Project members are assigned a role, which determines their permissions within the project. The project roles are as follows: * **Project owner**: Project owners have global permissions across projects they own. * **Project user**: Project users have restricted permissions for the specific projects they are invited to. The following table summarizes the permissions for each project role: | Permission | Owner | User | | :-------------------------- | ----- | ---- | | Update project names | ✓ | | | Delete projects | ✓ | | | View project members | ✓ | ✓ | | Update project member roles | ✓ | | | Delete project members | ✓ | | | View API keys | ✓ | ✓ | | Create API keys | ✓ | | | Delete API keys | ✓ | | | View indexes | ✓ | ✓ | | Create indexes | ✓ | ✓ | | Delete indexes | ✓ | ✓ | | Upsert vectors | ✓ | ✓ | | Query vectors | ✓ | ✓ | | Fetch vectors | ✓ | ✓ | | Update a vector | ✓ | ✓ | | Delete a vector | ✓ | ✓ | | List vector IDs | ✓ | ✓ | | Get index stats | ✓ | ✓ | Specific to pod-based indexes: | Permission | Owner | User | | :------------------------ | ----- | ---- | | Update project pod limits | ✓ | | | View project pod limits | ✓ | ✓ | | Update index size | ✓ | ✓ | ## API keys Each Pinecone [project](/guides/projects/understanding-projects) has one or more API keys. In order to [make calls to the Pinecone API](/guides/get-started/quickstart), you must provide a valid API key for the relevant Pinecone project. For more information, see [Manage API keys](/guides/projects/manage-api-keys). ## Project IDs Each Pinecone project has a unique product ID. To find the ID of a project, go to the project list in the [Pinecone console](https://app.pinecone.io/organizations/-/projects). ## Project pod limit To control costs, [project owners](/guides/projects/understanding-projects#project-roles) can [set the maximum total number of pods](/reference/api/database-limits#pods-per-project) allowed across all pod-based indexes in a project. Pod limits do not apply to [serverless indexes](/guides/index-data/indexing-overview#serverless-indexes). Serverless indexes auto-scale based on usage. ## See also * [Understanding organizations](guides/organizations/understanding-organizations) * [Manage organization members](guides/organizations/manage-organization-members) # Filter by metadata Source: https://docs.pinecone.io/guides/search/filter-by-metadata export const word_0 = "vectors" Every [record](/guides/get-started/glossary#record) in an index must contain an ID and a dense or sparse vector. In addition, you can include [metadata key-value pairs](/guides/index-data/indexing-overview#metadata) to store related information or context. When you search the index, you can then include a metadata filter to limit the search to records matching the filter expression. ## Search with a metadata filter The following code searches for the 3 records that are most semantically similar to a query and that have a `category` metadata field with the value `digestive system`. Searching with text is supported only for [indexes with integrated embedding](/guides/index-data/indexing-overview#integrated-embedding). ```python Python from pinecone import Pinecone pc = Pinecone(api_key="YOUR_API_KEY") # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index index = pc.Index(host="INDEX_HOST") filtered_results = index.search( namespace="example-namespace", query={ "inputs": {"text": "Disease prevention"}, "top_k": 3, "filter": {"category": "digestive system"}, }, fields=["category", "chunk_text"] ) print(filtered_results) ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: "YOUR_API_KEY" }) // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index const namespace = pc.index("INDEX_NAME", "INDEX_HOST").namespace("example-namespace"); const response = await namespace.searchRecords({ query: { topK: 3, inputs: { text: "Disease prevention" }, filter: { category: "digestive system" } }, fields: ['chunk_text', 'category'] }); console.log(response); ``` ```java Java import io.pinecone.clients.Index; import io.pinecone.configs.PineconeConfig; import io.pinecone.configs.PineconeConnection; import org.openapitools.db_data.client.ApiException; import org.openapitools.db_data.client.model.SearchRecordsResponse; import java.util.*; public class SearchText { public static void main(String[] args) throws ApiException { PineconeConfig config = new PineconeConfig("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index config.setHost("INDEX_HOST"); PineconeConnection connection = new PineconeConnection(config); Index index = new Index(config, connection, "integrated-dense-java"); String query = "Disease prevention"; List fields = new ArrayList<>(); fields.add("category"); fields.add("chunk_text"); Map filter = new HashMap<>(); filter.put("category", "digestive system"); // Search the dense index SearchRecordsResponse recordsResponse = index.searchRecordsByText(query, "example-namespace", fields, 3, filter, null); // Print the results System.out.println(recordsResponse); } } ``` ```go Go package main import ( "context" "encoding/json" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func prettifyStruct(obj interface{}) string { bytes, _ := json.MarshalIndent(obj, "", " ") return string(bytes) } func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "INDEX_HOST", Namespace: "example-namespace"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v", err) } metadataMap := map[string]interface{}{ "category": map[string]interface{}{ "$eq": "digestive system", }, } res, err := idxConnection.SearchRecords(ctx, &pinecone.SearchRecordsRequest{ Query: pinecone.SearchRecordsQuery{ TopK: 3, Inputs: &map[string]interface{}{ "text": "Disease prevention", }, Filter: &metadataMap, }, Fields: &[]string{"chunk_text", "category"}, }) if err != nil { log.Fatalf("Failed to search records: %v", err) } fmt.Printf(prettifyStruct(res)) } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index var index = pinecone.Index(host: "INDEX_HOST"); var response = await index.SearchRecordsAsync( "example-namespace", new SearchRecordsRequest { Query = new SearchRecordsRequestQuery { TopK = 4, Inputs = new Dictionary { { "text", "Disease prevention" } }, Filter = new Dictionary { ["category"] = new Dictionary { ["$eq"] = "digestive system" } } }, Fields = ["category", "chunk_text"], } ); Console.WriteLine(response); ``` ```shell curl INDEX_HOST="INDEX_HOST" NAMESPACE="YOUR_NAMESPACE" PINECONE_API_KEY="YOUR_API_KEY" curl "https://$INDEX_HOST/records/namespaces/$NAMESPACE/search" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: unstable" \ -d '{ "query": { "inputs": {"text": "Disease prevention"}, "top_k": 3, "filter": {"category": "digestive system"} }, "fields": ["category", "chunk_text"] }' ``` ```Python Python from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key="YOUR_API_KEY") # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index index = pc.Index(host="INDEX_HOST") index.query( namespace="example-namespace", vector=[0.0236663818359375,-0.032989501953125, ..., -0.01041412353515625,0.0086669921875], top_k=3, filter={ "category": {"$eq": "digestive system"} }, include_metadata=True, include_values=False ) ``` ```JavaScript JavaScript import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: "YOUR_API_KEY" }) // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index const index = pc.index("INDEX_NAME", "INDEX_HOST") const queryResponse = await index.namespace('example-namespace').query({ vector: [0.0236663818359375,-0.032989501953125,...,-0.01041412353515625,0.0086669921875], topK: 3, filter: { "category": { "$eq": "digestive system" } } includeValues: false, includeMetadata: true, }); ``` ```java Java import com.google.protobuf.Struct; import com.google.protobuf.Value; import io.pinecone.clients.Index; import io.pinecone.configs.PineconeConfig; import io.pinecone.configs.PineconeConnection; import io.pinecone.unsigned_indices_model.QueryResponseWithUnsignedIndices; import java.util.Arrays; import java.util.List; public class QueryExample { public static void main(String[] args) { PineconeConfig config = new PineconeConfig("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index config.setHost("INDEX_HOST"); PineconeConnection connection = new PineconeConnection(config); Index index = new Index(connection, "INDEX_NAME"); List query = Arrays.asList(0.0236663818359375f, -0.032989501953125f, ..., -0.01041412353515625f, 0.0086669921875f); Struct filter = Struct.newBuilder() .putFields("category", Value.newBuilder() .setStructValue(Struct.newBuilder() .putFields("$eq", Value.newBuilder() .setStringValue("digestive system") .build())) .build()) .build(); QueryResponseWithUnsignedIndices queryResponse = index.query(1, query, null, null, null, "example-namespace", filter, false, true); System.out.println(queryResponse); } } ``` ```go Go package main import ( "context" "encoding/json" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func prettifyStruct(obj interface{}) string { bytes, _ := json.MarshalIndent(obj, "", " ") return string(bytes) } func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "INDEX_HOST", Namespace: "example-namespace"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v", err) } queryVector := []float32{0.0236663818359375,-0.032989501953125,...,-0.01041412353515625,0.0086669921875} metadataMap := map[string]interface{}{ "category": map[string]interface{}{ "$eq": "digestive system", } } metadataFilter, err := structpb.NewStruct(metadataMap) if err != nil { log.Fatalf("Failed to create metadata map: %v", err) } res, err := idxConnection.QueryByVectorValues(ctx, &pinecone.QueryByVectorValuesRequest{ Vector: queryVector, TopK: 3, MetadataFilter: metadataFilter, IncludeValues: false, includeMetadata: true, }) if err != nil { log.Fatalf("Error encountered when querying by vector: %v", err) } else { fmt.Printf(prettifyStruct(res)) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index var index = pinecone.Index(host: "INDEX_HOST"); var queryResponse = await index.QueryAsync(new QueryRequest { Vector = new[] { 0.0236663818359375f ,-0.032989501953125f, ..., -0.01041412353515625f, 0.0086669921875f }, Namespace = "example-namespace", TopK = 3, Filter = new Metadata { ["category"] = new Metadata { ["$eq"] = "digestive system", } }, IncludeMetadata = true, }); Console.WriteLine(queryResponse); ``` ```bash curl # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index PINECONE_API_KEY="YOUR_API_KEY" INDEX_HOST="INDEX_HOST" curl "https://$INDEX_HOST/query" \ -H "Api-Key: $PINECONE_API_KEY" \ -H 'Content-Type: application/json' \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "vector": [0.0236663818359375,-0.032989501953125,...,-0.01041412353515625,0.0086669921875], "namespace": "example-namespace", "topK": 3, "filter": {"category": {"$eq": "digestive system"}}, "includeMetadata": true, "includeValues": false }' ``` ## Metadata filter expressions Pinecone's filtering query language is based on [MongoDB's query and projection operators](https://docs.mongodb.com/manual/reference/operator/query/). Pinecone currently supports a subset of those selectors: | Filter | Description | Supported types | | --------- | ---------------------------------------------------------------------------------------------------------------------------------- | ----------------------- | | `$eq` | Matches {word_0} with metadata values that are equal to a specified value. Example: `{"genre": {"$eq": "documentary"}}` | Number, string, boolean | | `$ne` | Matches {word_0} with metadata values that are not equal to a specified value. Example: `{"genre": {"$ne": "drama"}}` | Number, string, boolean | | `$gt` | Matches {word_0} with metadata values that are greater than a specified value. Example: `{"year": {"$gt": 2019}}` | Number | | `$gte` | Matches {word_0} with metadata values that are greater than or equal to a specified value. Example:`{"year": {"$gte": 2020}}` | Number | | `$lt` | Matches {word_0} with metadata values that are less than a specified value. Example: `{"year": {"$lt": 2020}}` | Number | | `$lte` | Matches {word_0} with metadata values that are less than or equal to a specified value. Example: `{"year": {"$lte": 2020}}` | Number | | `$in` | Matches {word_0} with metadata values that are in a specified array. Example: `{"genre": {"$in": ["comedy", "documentary"]}}` | String, number | | `$nin` | Matches {word_0} with metadata values that are not in a specified array. Example: `{"genre": {"$nin": ["comedy", "documentary"]}}` | String, number | | `$exists` | Matches {word_0} with the specified metadata field. Example: `{"genre": {"$exists": true}}` | Number, string, boolean | | `$and` | Joins query clauses with a logical `AND`. Example: `{"$and": [{"genre": {"$eq": "drama"}}, {"year": {"$gte": 2020}}]}` | - | | `$or` | Joins query clauses with a logical `OR`. Example: `{"$or": [{"genre": {"$eq": "drama"}}, {"year": {"$gte": 2020}}]}` | - | For example, the following has a `"genre"` metadata field with a list of strings: ```JSON JSON { "genre": ["comedy", "documentary"] } ``` This means `"genre"` takes on both values, and requests with the following filters will match: ```JSON JSON {"genre":"comedy"} {"genre": {"$in":["documentary","action"]}} {"$and": [{"genre": "comedy"}, {"genre":"documentary"}]} ``` However, requests with the following filter will **not** match: ```JSON JSON { "$and": [{ "genre": "comedy" }, { "genre": "drama" }] } ``` Additionally, requests with the following filters will **not** match because they are invalid. They will result in a compilation error: ``` # INVALID QUERY: {"genre": ["comedy", "documentary"]} ``` ``` # INVALID QUERY: {"genre": {"$eq": ["comedy", "documentary"]}} ``` # Hybrid search Source: https://docs.pinecone.io/guides/search/hybrid-search [Semantic search](/guides/search/semantic-search) and [lexical search](/guides/search/lexical-search) are powerful information retrieval techniques, but each has notable limitations. For example, semantic search can miss results based on exact keyword matches, especially in scenarios involving domain-specific terminology, while lexical search can miss results based on relationships, such as synonyms and paraphrases. This page shows you how to lift these limitations by combining semantic and lexical search. This is often called hybrid search. ## Hybrid search approaches There are two ways to perform hybrid search in Pinecone: * [Use separate dense and sparse indexes](#use-separate-dense-and-sparse-indexes). This is the **recommended** approach because it provides the most flexibility. * [Use a single hybrid index](#use-a-single-hybrid-index). This approach is simpler to implement but doesn't support a few useful features. The following table summarizes the pros and cons between the two approaches: | Approach | Pros | Cons | | :-------------------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Separate dense and sparse indexes |
  • You can start with dense for semantic search and add sparse for lexical search later.
  • You can do sparse-only queries.
  • You can rerank at multiple levels (for each index and for merged results).
  • You can use integrated embedding and reranking.
|
  • You need to manage and make requests to two separate indexes.
  • You need to maintain the linkage between sparse and dense vectors in different indexes.
| | Single hybrid index |
  • You make requests to only a single index.
  • The linkage between dense and sparse vectors is implicit.
|
  • You can't do sparse-only queries.
  • You can't use integrated embedding and reranking.
| ## Use separate dense and sparse indexes This is the recommended way to perform hybrid search in Pinecone. You create separate dense and sparse indexes, upsert dense vectors into the dense index and sparse vectors into the sparse index, and search each index separately. Then you combine and deduplicate the results, use one of Pinecone's [hosted reranking models](/guides/search/rerank-results#reranking-models) to rerank them based on a unified relevance score, and return the most relevant matches. [Create a dense index](/guides/index-data/create-an-index#create-a-dense-index) and [create a sparse index](/guides/index-data/create-an-index#create-a-sparse-index), either with integrated embedding or for vectors created with external models. For example, the following code creates indexes with integrated embedding models. ```python Python from pinecone import Pinecone pc = Pinecone(api_key="YOUR_API_KEY") dense_index_name = "dense-for-hybrid-py" sparse_index_name = "sparse-for-hybrid-py" if not pc.has_index(dense_index_name): pc.create_index_for_model( name=dense_index_name, cloud="aws", region="us-east-1", embed={ "model":"llama-text-embed-v2", "field_map":{"text": "chunk_text"} } ) if not pc.has_index(sparse_index_name): pc.create_index_for_model( name=sparse_index_name, cloud="aws", region="us-east-1", embed={ "model":"pinecone-sparse-english-v0", "field_map":{"text": "chunk_text"} } ) ``` [Upsert dense vectors](/guides/index-data/upsert-data#upsert-dense-vectors) into the dense index and [upsert sparse vectors](/guides/index-data/upsert-data#upsert-sparse-vectors) into the sparse index. Make sure to establish a linkage between the dense and sparse vectors so you can merge and deduplicate search results later. For example, the following uses `_id` as the linkage, but you can use any other custom field as well. Because the indexes are integrated with embedding models, you provide the source texts and Pinecone converts them to vectors automatically. ```python Python [expandable] # Define the records records = [ { "_id": "vec1", "chunk_text": "Apple Inc. issued a $10 billion corporate bond in 2023." }, { "_id": "vec2", "chunk_text": "ETFs tracking the S&P 500 outperformed active funds last year." }, { "_id": "vec3", "chunk_text": "Tesla's options volume surged after the latest earnings report." }, { "_id": "vec4", "chunk_text": "Dividend aristocrats are known for consistently raising payouts." }, { "_id": "vec5", "chunk_text": "The Federal Reserve raised interest rates by 0.25% to curb inflation." }, { "_id": "vec6", "chunk_text": "Unemployment hit a record low of 3.7% in Q4 of 2024." }, { "_id": "vec7", "chunk_text": "The CPI index rose by 6% in July 2024, raising concerns about purchasing power." }, { "_id": "vec8", "chunk_text": "GDP growth in emerging markets outpaced developed economies." }, { "_id": "vec9", "chunk_text": "Amazon's acquisition of MGM Studios was valued at $8.45 billion." }, { "_id": "vec10", "chunk_text": "Alphabet reported a 20% increase in advertising revenue." }, { "_id": "vec11", "chunk_text": "ExxonMobil announced a special dividend after record profits." }, { "_id": "vec12", "chunk_text": "Tesla plans a 3-for-1 stock split to attract retail investors." }, { "_id": "vec13", "chunk_text": "Credit card APRs reached an all-time high of 22.8% in 2024." }, { "_id": "vec14", "chunk_text": "A 529 college savings plan offers tax advantages for education." }, { "_id": "vec15", "chunk_text": "Emergency savings should ideally cover 6 months of expenses." }, { "_id": "vec16", "chunk_text": "The average mortgage rate rose to 7.1% in December." }, { "_id": "vec17", "chunk_text": "The SEC fined a hedge fund $50 million for insider trading." }, { "_id": "vec18", "chunk_text": "New ESG regulations require companies to disclose climate risks." }, { "_id": "vec19", "chunk_text": "The IRS introduced a new tax bracket for high earners." }, { "_id": "vec20", "chunk_text": "Compliance with GDPR is mandatory for companies operating in Europe." }, { "_id": "vec21", "chunk_text": "What are the best-performing green bonds in a rising rate environment?" }, { "_id": "vec22", "chunk_text": "How does inflation impact the real yield of Treasury bonds?" }, { "_id": "vec23", "chunk_text": "Top SPAC mergers in the technology sector for 2024." }, { "_id": "vec24", "chunk_text": "Are stablecoins a viable hedge against currency devaluation?" }, { "_id": "vec25", "chunk_text": "Comparison of Roth IRA vs 401(k) for high-income earners." }, { "_id": "vec26", "chunk_text": "Stock splits and their effect on investor sentiment." }, { "_id": "vec27", "chunk_text": "Tech IPOs that disappointed in their first year." }, { "_id": "vec28", "chunk_text": "Impact of interest rate hikes on bank stocks." }, { "_id": "vec29", "chunk_text": "Growth vs. value investing strategies in 2024." }, { "_id": "vec30", "chunk_text": "The role of artificial intelligence in quantitative trading." }, { "_id": "vec31", "chunk_text": "What are the implications of quantitative tightening on equities?" }, { "_id": "vec32", "chunk_text": "How does compounding interest affect long-term investments?" }, { "_id": "vec33", "chunk_text": "What are the best assets to hedge against inflation?" }, { "_id": "vec34", "chunk_text": "Can ETFs provide better diversification than mutual funds?" }, { "_id": "vec35", "chunk_text": "Unemployment hit at 2.4% in Q3 of 2024." }, { "_id": "vec36", "chunk_text": "Unemployment is expected to hit 2.5% in Q3 of 2024." }, { "_id": "vec37", "chunk_text": "In Q3 2025 unemployment for the prior year was revised to 2.2%"}, { "_id": "vec38", "chunk_text": "Emerging markets witnessed increased foreign direct investment as global interest rates stabilized." }, { "_id": "vec39", "chunk_text": "The rise in energy prices significantly impacted inflation trends during the first half of 2024." }, { "_id": "vec40", "chunk_text": "Labor market trends show a declining participation rate despite record low unemployment in 2024." }, { "_id": "vec41", "chunk_text": "Forecasts of global supply chain disruptions eased in late 2024, but consumer prices remained elevated due to persistent demand." }, { "_id": "vec42", "chunk_text": "Tech sector layoffs in Q3 2024 have reshaped hiring trends across high-growth industries." }, { "_id": "vec43", "chunk_text": "The U.S. dollar weakened against a basket of currencies as the global economy adjusted to shifting trade balances." }, { "_id": "vec44", "chunk_text": "Central banks worldwide increased gold reserves to hedge against geopolitical and economic instability." }, { "_id": "vec45", "chunk_text": "Corporate earnings in Q4 2024 were largely impacted by rising raw material costs and currency fluctuations." }, { "_id": "vec46", "chunk_text": "Economic recovery in Q2 2024 relied heavily on government spending in infrastructure and green energy projects." }, { "_id": "vec47", "chunk_text": "The housing market saw a rebound in late 2024, driven by falling mortgage rates and pent-up demand." }, { "_id": "vec48", "chunk_text": "Wage growth outpaced inflation for the first time in years, signaling improved purchasing power in 2024." }, { "_id": "vec49", "chunk_text": "China's economic growth in 2024 slowed to its lowest level in decades due to structural reforms and weak exports." }, { "_id": "vec50", "chunk_text": "AI-driven automation in the manufacturing sector boosted productivity but raised concerns about job displacement." }, { "_id": "vec51", "chunk_text": "The European Union introduced new fiscal policies in 2024 aimed at reducing public debt without stifling growth." }, { "_id": "vec52", "chunk_text": "Record-breaking weather events in early 2024 have highlighted the growing economic impact of climate change." }, { "_id": "vec53", "chunk_text": "Cryptocurrencies faced regulatory scrutiny in 2024, leading to volatility and reduced market capitalization." }, { "_id": "vec54", "chunk_text": "The global tourism sector showed signs of recovery in late 2024 after years of pandemic-related setbacks." }, { "_id": "vec55", "chunk_text": "Trade tensions between the U.S. and China escalated in 2024, impacting global supply chains and investment flows." }, { "_id": "vec56", "chunk_text": "Consumer confidence indices remained resilient in Q2 2024 despite fears of an impending recession." }, { "_id": "vec57", "chunk_text": "Startups in 2024 faced tighter funding conditions as venture capitalists focused on profitability over growth." }, { "_id": "vec58", "chunk_text": "Oil production cuts in Q1 2024 by OPEC nations drove prices higher, influencing global energy policies." }, { "_id": "vec59", "chunk_text": "The adoption of digital currencies by central banks increased in 2024, reshaping monetary policy frameworks." }, { "_id": "vec60", "chunk_text": "Healthcare spending in 2024 surged as governments expanded access to preventive care and pandemic preparedness." }, { "_id": "vec61", "chunk_text": "The World Bank reported declining poverty rates globally, but regional disparities persisted." }, { "_id": "vec62", "chunk_text": "Private equity activity in 2024 focused on renewable energy and technology sectors amid shifting investor priorities." }, { "_id": "vec63", "chunk_text": "Population aging emerged as a critical economic issue in 2024, especially in advanced economies." }, { "_id": "vec64", "chunk_text": "Rising commodity prices in 2024 strained emerging markets dependent on imports of raw materials." }, { "_id": "vec65", "chunk_text": "The global shipping industry experienced declining freight rates in 2024 due to overcapacity and reduced demand." }, { "_id": "vec66", "chunk_text": "Bank lending to small and medium-sized enterprises surged in 2024 as governments incentivized entrepreneurship." }, { "_id": "vec67", "chunk_text": "Renewable energy projects accounted for a record share of global infrastructure investment in 2024." }, { "_id": "vec68", "chunk_text": "Cybersecurity spending reached new highs in 2024, reflecting the growing threat of digital attacks on infrastructure." }, { "_id": "vec69", "chunk_text": "The agricultural sector faced challenges in 2024 due to extreme weather and rising input costs." }, { "_id": "vec70", "chunk_text": "Consumer spending patterns shifted in 2024, with a greater focus on experiences over goods." }, { "_id": "vec71", "chunk_text": "The economic impact of the 2008 financial crisis was mitigated by quantitative easing policies." }, { "_id": "vec72", "chunk_text": "In early 2024, global GDP growth slowed, driven by weaker exports in Asia and Europe." }, { "_id": "vec73", "chunk_text": "The historical relationship between inflation and unemployment is explained by the Phillips Curve." }, { "_id": "vec74", "chunk_text": "The World Trade Organization's role in resolving disputes was tested in 2024." }, { "_id": "vec75", "chunk_text": "The collapse of Silicon Valley Bank raised questions about regulatory oversight in 2024." }, { "_id": "vec76", "chunk_text": "The cost of living crisis has been exacerbated by stagnant wage growth and rising inflation." }, { "_id": "vec77", "chunk_text": "Supply chain resilience became a top priority for multinational corporations in 2024." }, { "_id": "vec78", "chunk_text": "Consumer sentiment surveys in 2024 reflected optimism despite high interest rates." }, { "_id": "vec79", "chunk_text": "The resurgence of industrial policy in Q1 2024 focused on decoupling critical supply chains." }, { "_id": "vec80", "chunk_text": "Technological innovation in the fintech sector disrupted traditional banking in 2024." }, { "_id": "vec81", "chunk_text": "The link between climate change and migration patterns is increasingly recognized." }, { "_id": "vec82", "chunk_text": "Renewable energy subsidies in 2024 reduced the global reliance on fossil fuels." }, { "_id": "vec83", "chunk_text": "The economic fallout of geopolitical tensions was evident in rising defense budgets worldwide." }, { "_id": "vec84", "chunk_text": "The IMF's 2024 global outlook highlighted risks of stagflation in emerging markets." }, { "_id": "vec85", "chunk_text": "Declining birth rates in advanced economies pose long-term challenges for labor markets." }, { "_id": "vec86", "chunk_text": "Digital transformation initiatives in 2024 drove productivity gains in the services sector." }, { "_id": "vec87", "chunk_text": "The U.S. labor market's resilience in 2024 defied predictions of a severe recession." }, { "_id": "vec88", "chunk_text": "New fiscal measures in the European Union aimed to stabilize debt levels post-pandemic." }, { "_id": "vec89", "chunk_text": "Venture capital investments in 2024 leaned heavily toward AI and automation startups." }, { "_id": "vec90", "chunk_text": "The surge in e-commerce in 2024 was facilitated by advancements in logistics technology." }, { "_id": "vec91", "chunk_text": "The impact of ESG investing on corporate strategies has been a major focus in 2024." }, { "_id": "vec92", "chunk_text": "Income inequality widened in 2024 despite strong economic growth in developed nations." }, { "_id": "vec93", "chunk_text": "The collapse of FTX highlighted the volatility and risks associated with cryptocurrencies." }, { "_id": "vec94", "chunk_text": "Cyberattacks targeting financial institutions in 2024 led to record cybersecurity spending." }, { "_id": "vec95", "chunk_text": "Automation in agriculture in 2024 increased yields but displaced rural workers." }, { "_id": "vec96", "chunk_text": "New trade agreements signed 2022 will make an impact in 2024"}, ] ``` ```python Python # Target the dense and sparse indexes # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index dense_index = pc.Index(host="INDEX_HOST") sparse_index = pc.Index(host="INDEX_HOST") # Upsert the records # The `chunk_text` fields are converted to dense and sparse vectors dense_index.upsert_records("example-namespace", records) sparse_index.upsert_records("example-namespace", records) ``` Perform a [semantic search](/guides/search/semantic-search) on the dense index. For example, the following code searches the dense index for 40 records most semantically related to the query "Q3 2024 us economic data". Because the index is integrated with an embedding model, you provide the query as text and Pinecone converts the text to a dense vector automatically. ```python Python query = "Q3 2024 us economic data" dense_results = dense_index.search( namespace="example-namespace", query={ "top_k": 40, "inputs": { "text": query } } ) print(dense_results) ``` ```python Response [expandable] {'result': {'hits': [{'_id': 'vec35', '_score': 0.8629686832427979, 'fields': {'chunk_text': 'Unemployment hit at 2.4% in Q3 ' 'of 2024.'}}, {'_id': 'vec36', '_score': 0.8573639988899231, 'fields': {'chunk_text': 'Unemployment is expected to ' 'hit 2.5% in Q3 of 2024.'}}, {'_id': 'vec6', '_score': 0.8535352945327759, 'fields': {'chunk_text': 'Unemployment hit a record low ' 'of 3.7% in Q4 of 2024.'}}, {'_id': 'vec42', '_score': 0.8336166739463806, 'fields': {'chunk_text': 'Tech sector layoffs in Q3 2024 ' 'have reshaped hiring trends ' 'across high-growth ' 'industries.'}}, {'_id': 'vec48', '_score': 0.8328524827957153, 'fields': {'chunk_text': 'Wage growth outpaced inflation ' 'for the first time in years, ' 'signaling improved purchasing ' 'power in 2024.'}}, {'_id': 'vec55', '_score': 0.8322604298591614, 'fields': {'chunk_text': 'Trade tensions between the ' 'U.S. and China escalated in ' '2024, impacting global supply ' 'chains and investment flows.'}}, {'_id': 'vec45', '_score': 0.8309446573257446, 'fields': {'chunk_text': 'Corporate earnings in Q4 2024 ' 'were largely impacted by ' 'rising raw material costs and ' 'currency fluctuations.'}}, {'_id': 'vec72', '_score': 0.8275909423828125, 'fields': {'chunk_text': 'In early 2024, global GDP ' 'growth slowed, driven by ' 'weaker exports in Asia and ' 'Europe.'}}, {'_id': 'vec29', '_score': 0.8270887136459351, 'fields': {'chunk_text': 'Growth vs. value investing ' 'strategies in 2024.'}}, {'_id': 'vec46', '_score': 0.8263787627220154, 'fields': {'chunk_text': 'Economic recovery in Q2 2024 ' 'relied heavily on government ' 'spending in infrastructure and ' 'green energy projects.'}}, {'_id': 'vec79', '_score': 0.8258304595947266, 'fields': {'chunk_text': 'The resurgence of industrial ' 'policy in Q1 2024 focused on ' 'decoupling critical supply ' 'chains.'}}, {'_id': 'vec87', '_score': 0.8257324695587158, 'fields': {'chunk_text': "The U.S. labor market's " 'resilience in 2024 defied ' 'predictions of a severe ' 'recession.'}}, {'_id': 'vec40', '_score': 0.8253997564315796, 'fields': {'chunk_text': 'Labor market trends show a ' 'declining participation rate ' 'despite record low ' 'unemployment in 2024.'}}, {'_id': 'vec37', '_score': 0.8235862255096436, 'fields': {'chunk_text': 'In Q3 2025 unemployment for ' 'the prior year was revised to ' '2.2%'}}, {'_id': 'vec58', '_score': 0.8233317136764526, 'fields': {'chunk_text': 'Oil production cuts in Q1 2024 ' 'by OPEC nations drove prices ' 'higher, influencing global ' 'energy policies.'}}, {'_id': 'vec47', '_score': 0.8231339454650879, 'fields': {'chunk_text': 'The housing market saw a ' 'rebound in late 2024, driven ' 'by falling mortgage rates and ' 'pent-up demand.'}}, {'_id': 'vec41', '_score': 0.8187897801399231, 'fields': {'chunk_text': 'Forecasts of global supply ' 'chain disruptions eased in ' 'late 2024, but consumer prices ' 'remained elevated due to ' 'persistent demand.'}}, {'_id': 'vec56', '_score': 0.8155254125595093, 'fields': {'chunk_text': 'Consumer confidence indices ' 'remained resilient in Q2 2024 ' 'despite fears of an impending ' 'recession.'}}, {'_id': 'vec63', '_score': 0.8136948347091675, 'fields': {'chunk_text': 'Population aging emerged as a ' 'critical economic issue in ' '2024, especially in advanced ' 'economies.'}}, {'_id': 'vec52', '_score': 0.8129132390022278, 'fields': {'chunk_text': 'Record-breaking weather events ' 'in early 2024 have highlighted ' 'the growing economic impact of ' 'climate change.'}}, {'_id': 'vec23', '_score': 0.8126378655433655, 'fields': {'chunk_text': 'Top SPAC mergers in the ' 'technology sector for 2024.'}}, {'_id': 'vec62', '_score': 0.8116977214813232, 'fields': {'chunk_text': 'Private equity activity in ' '2024 focused on renewable ' 'energy and technology sectors ' 'amid shifting investor ' 'priorities.'}}, {'_id': 'vec64', '_score': 0.8109902739524841, 'fields': {'chunk_text': 'Rising commodity prices in ' '2024 strained emerging markets ' 'dependent on imports of raw ' 'materials.'}}, {'_id': 'vec54', '_score': 0.8092231154441833, 'fields': {'chunk_text': 'The global tourism sector ' 'showed signs of recovery in ' 'late 2024 after years of ' 'pandemic-related setbacks.'}}, {'_id': 'vec96', '_score': 0.8075559735298157, 'fields': {'chunk_text': 'New trade agreements signed ' '2022 will make an impact in ' '2024'}}, {'_id': 'vec49', '_score': 0.8062589764595032, 'fields': {'chunk_text': "China's economic growth in " '2024 slowed to its lowest ' 'level in decades due to ' 'structural reforms and weak ' 'exports.'}}, {'_id': 'vec7', '_score': 0.8034461140632629, 'fields': {'chunk_text': 'The CPI index rose by 6% in ' 'July 2024, raising concerns ' 'about purchasing power.'}}, {'_id': 'vec84', '_score': 0.8027160167694092, 'fields': {'chunk_text': "The IMF's 2024 global outlook " 'highlighted risks of ' 'stagflation in emerging ' 'markets.'}}, {'_id': 'vec13', '_score': 0.8010239601135254, 'fields': {'chunk_text': 'Credit card APRs reached an ' 'all-time high of 22.8% in ' '2024.'}}, {'_id': 'vec53', '_score': 0.8007135391235352, 'fields': {'chunk_text': 'Cryptocurrencies faced ' 'regulatory scrutiny in 2024, ' 'leading to volatility and ' 'reduced market ' 'capitalization.'}}, {'_id': 'vec60', '_score': 0.7980866432189941, 'fields': {'chunk_text': 'Healthcare spending in 2024 ' 'surged as governments expanded ' 'access to preventive care and ' 'pandemic preparedness.'}}, {'_id': 'vec91', '_score': 0.7980680465698242, 'fields': {'chunk_text': 'The impact of ESG investing on ' 'corporate strategies has been ' 'a major focus in 2024.'}}, {'_id': 'vec68', '_score': 0.797269880771637, 'fields': {'chunk_text': 'Cybersecurity spending reached ' 'new highs in 2024, reflecting ' 'the growing threat of digital ' 'attacks on infrastructure.'}}, {'_id': 'vec59', '_score': 0.795337438583374, 'fields': {'chunk_text': 'The adoption of digital ' 'currencies by central banks ' 'increased in 2024, reshaping ' 'monetary policy frameworks.'}}, {'_id': 'vec39', '_score': 0.793889045715332, 'fields': {'chunk_text': 'The rise in energy prices ' 'significantly impacted ' 'inflation trends during the ' 'first half of 2024.'}}, {'_id': 'vec66', '_score': 0.7919396162033081, 'fields': {'chunk_text': 'Bank lending to small and ' 'medium-sized enterprises ' 'surged in 2024 as governments ' 'incentivized ' 'entrepreneurship.'}}, {'_id': 'vec57', '_score': 0.7917722463607788, 'fields': {'chunk_text': 'Startups in 2024 faced tighter ' 'funding conditions as venture ' 'capitalists focused on ' 'profitability over growth.'}}, {'_id': 'vec75', '_score': 0.7907494306564331, 'fields': {'chunk_text': 'The collapse of Silicon Valley ' 'Bank raised questions about ' 'regulatory oversight in ' '2024.'}}, {'_id': 'vec51', '_score': 0.790622889995575, 'fields': {'chunk_text': 'The European Union introduced ' 'new fiscal policies in 2024 ' 'aimed at reducing public debt ' 'without stifling growth.'}}, {'_id': 'vec89', '_score': 0.7899052500724792, 'fields': {'chunk_text': 'Venture capital investments in ' '2024 leaned heavily toward AI ' 'and automation startups.'}}]}, 'usage': {'embed_total_tokens': 12, 'read_units': 1}} ``` Perform a [lexical search](/guides/search/lexical-search). For example, the following code searches the sparse index for 40 records that most exactly match the words in the query. Again, because the index is integrated with an embedding model, you provide the query as text and Pinecone converts the text to a sparse vector automatically. ```python Python sparse_results = sparse_index.search( namespace="example-namespace", query={ "top_k": 40, "inputs": { "text": query } } ) print(sparse_results) ``` ```python Response [expandable] {'result': {'hits': [{'_id': 'vec35', '_score': 7.0625, 'fields': {'chunk_text': 'Unemployment hit at 2.4% in Q3 ' 'of 2024.'}}, {'_id': 'vec46', '_score': 7.041015625, 'fields': {'chunk_text': 'Economic recovery in Q2 2024 ' 'relied heavily on government ' 'spending in infrastructure and ' 'green energy projects.'}}, {'_id': 'vec36', '_score': 6.96875, 'fields': {'chunk_text': 'Unemployment is expected to ' 'hit 2.5% in Q3 of 2024.'}}, {'_id': 'vec42', '_score': 6.9609375, 'fields': {'chunk_text': 'Tech sector layoffs in Q3 2024 ' 'have reshaped hiring trends ' 'across high-growth ' 'industries.'}}, {'_id': 'vec49', '_score': 6.65625, 'fields': {'chunk_text': "China's economic growth in " '2024 slowed to its lowest ' 'level in decades due to ' 'structural reforms and weak ' 'exports.'}}, {'_id': 'vec63', '_score': 6.4765625, 'fields': {'chunk_text': 'Population aging emerged as a ' 'critical economic issue in ' '2024, especially in advanced ' 'economies.'}}, {'_id': 'vec92', '_score': 5.72265625, 'fields': {'chunk_text': 'Income inequality widened in ' '2024 despite strong economic ' 'growth in developed nations.'}}, {'_id': 'vec52', '_score': 5.599609375, 'fields': {'chunk_text': 'Record-breaking weather events ' 'in early 2024 have highlighted ' 'the growing economic impact of ' 'climate change.'}}, {'_id': 'vec89', '_score': 4.0078125, 'fields': {'chunk_text': 'Venture capital investments in ' '2024 leaned heavily toward AI ' 'and automation startups.'}}, {'_id': 'vec62', '_score': 3.99609375, 'fields': {'chunk_text': 'Private equity activity in ' '2024 focused on renewable ' 'energy and technology sectors ' 'amid shifting investor ' 'priorities.'}}, {'_id': 'vec57', '_score': 3.93359375, 'fields': {'chunk_text': 'Startups in 2024 faced tighter ' 'funding conditions as venture ' 'capitalists focused on ' 'profitability over growth.'}}, {'_id': 'vec69', '_score': 3.8984375, 'fields': {'chunk_text': 'The agricultural sector faced ' 'challenges in 2024 due to ' 'extreme weather and rising ' 'input costs.'}}, {'_id': 'vec37', '_score': 3.89453125, 'fields': {'chunk_text': 'In Q3 2025 unemployment for ' 'the prior year was revised to ' '2.2%'}}, {'_id': 'vec60', '_score': 3.822265625, 'fields': {'chunk_text': 'Healthcare spending in 2024 ' 'surged as governments expanded ' 'access to preventive care and ' 'pandemic preparedness.'}}, {'_id': 'vec51', '_score': 3.783203125, 'fields': {'chunk_text': 'The European Union introduced ' 'new fiscal policies in 2024 ' 'aimed at reducing public debt ' 'without stifling growth.'}}, {'_id': 'vec55', '_score': 3.765625, 'fields': {'chunk_text': 'Trade tensions between the ' 'U.S. and China escalated in ' '2024, impacting global supply ' 'chains and investment flows.'}}, {'_id': 'vec70', '_score': 3.76171875, 'fields': {'chunk_text': 'Consumer spending patterns ' 'shifted in 2024, with a ' 'greater focus on experiences ' 'over goods.'}}, {'_id': 'vec90', '_score': 3.70703125, 'fields': {'chunk_text': 'The surge in e-commerce in ' '2024 was facilitated by ' 'advancements in logistics ' 'technology.'}}, {'_id': 'vec87', '_score': 3.69140625, 'fields': {'chunk_text': "The U.S. labor market's " 'resilience in 2024 defied ' 'predictions of a severe ' 'recession.'}}, {'_id': 'vec78', '_score': 3.673828125, 'fields': {'chunk_text': 'Consumer sentiment surveys in ' '2024 reflected optimism ' 'despite high interest rates.'}}, {'_id': 'vec82', '_score': 3.66015625, 'fields': {'chunk_text': 'Renewable energy subsidies in ' '2024 reduced the global ' 'reliance on fossil fuels.'}}, {'_id': 'vec53', '_score': 3.642578125, 'fields': {'chunk_text': 'Cryptocurrencies faced ' 'regulatory scrutiny in 2024, ' 'leading to volatility and ' 'reduced market ' 'capitalization.'}}, {'_id': 'vec94', '_score': 3.625, 'fields': {'chunk_text': 'Cyberattacks targeting ' 'financial institutions in 2024 ' 'led to record cybersecurity ' 'spending.'}}, {'_id': 'vec45', '_score': 3.607421875, 'fields': {'chunk_text': 'Corporate earnings in Q4 2024 ' 'were largely impacted by ' 'rising raw material costs and ' 'currency fluctuations.'}}, {'_id': 'vec47', '_score': 3.576171875, 'fields': {'chunk_text': 'The housing market saw a ' 'rebound in late 2024, driven ' 'by falling mortgage rates and ' 'pent-up demand.'}}, {'_id': 'vec84', '_score': 3.5703125, 'fields': {'chunk_text': "The IMF's 2024 global outlook " 'highlighted risks of ' 'stagflation in emerging ' 'markets.'}}, {'_id': 'vec41', '_score': 3.5546875, 'fields': {'chunk_text': 'Forecasts of global supply ' 'chain disruptions eased in ' 'late 2024, but consumer prices ' 'remained elevated due to ' 'persistent demand.'}}, {'_id': 'vec65', '_score': 3.537109375, 'fields': {'chunk_text': 'The global shipping industry ' 'experienced declining freight ' 'rates in 2024 due to ' 'overcapacity and reduced ' 'demand.'}}, {'_id': 'vec96', '_score': 3.53125, 'fields': {'chunk_text': 'New trade agreements signed ' '2022 will make an impact in ' '2024'}}, {'_id': 'vec86', '_score': 3.52734375, 'fields': {'chunk_text': 'Digital transformation ' 'initiatives in 2024 drove ' 'productivity gains in the ' 'services sector.'}}, {'_id': 'vec95', '_score': 3.5234375, 'fields': {'chunk_text': 'Automation in agriculture in ' '2024 increased yields but ' 'displaced rural workers.'}}, {'_id': 'vec64', '_score': 3.51171875, 'fields': {'chunk_text': 'Rising commodity prices in ' '2024 strained emerging markets ' 'dependent on imports of raw ' 'materials.'}}, {'_id': 'vec79', '_score': 3.51171875, 'fields': {'chunk_text': 'The resurgence of industrial ' 'policy in Q1 2024 focused on ' 'decoupling critical supply ' 'chains.'}}, {'_id': 'vec66', '_score': 3.48046875, 'fields': {'chunk_text': 'Bank lending to small and ' 'medium-sized enterprises ' 'surged in 2024 as governments ' 'incentivized ' 'entrepreneurship.'}}, {'_id': 'vec6', '_score': 3.4765625, 'fields': {'chunk_text': 'Unemployment hit a record low ' 'of 3.7% in Q4 of 2024.'}}, {'_id': 'vec58', '_score': 3.39453125, 'fields': {'chunk_text': 'Oil production cuts in Q1 2024 ' 'by OPEC nations drove prices ' 'higher, influencing global ' 'energy policies.'}}, {'_id': 'vec80', '_score': 3.390625, 'fields': {'chunk_text': 'Technological innovation in ' 'the fintech sector disrupted ' 'traditional banking in 2024.'}}, {'_id': 'vec75', '_score': 3.37109375, 'fields': {'chunk_text': 'The collapse of Silicon Valley ' 'Bank raised questions about ' 'regulatory oversight in ' '2024.'}}, {'_id': 'vec67', '_score': 3.357421875, 'fields': {'chunk_text': 'Renewable energy projects ' 'accounted for a record share ' 'of global infrastructure ' 'investment in 2024.'}}, {'_id': 'vec56', '_score': 3.341796875, 'fields': {'chunk_text': 'Consumer confidence indices ' 'remained resilient in Q2 2024 ' 'despite fears of an impending ' 'recession.'}}]}, 'usage': {'embed_total_tokens': 9, 'read_units': 1}} ``` Merge the dense and sparse results and deduplicated them based on the field you used to link sparse and dense vectors. For example, the following code merges the results and deduplicates them based on the `_id` field. ```python Python def merge_chunks(h1, h2): """Get the unique hits from two search results and return them as single array of {'_id', 'chunk_text'} dicts, printing each dict on a new line.""" # Deduplicate by _id deduped_hits = {hit['_id']: hit for hit in h1['result']['hits'] + h2['result']['hits']}.values() # Sort by _score descending sorted_hits = sorted(deduped_hits, key=lambda x: x['_score'], reverse=True) # Transform to format for reranking result = [{'_id': hit['_id'], 'chunk_text': hit['fields']['chunk_text']} for hit in sorted_hits] return result merged_results = merge_chunks(sparse_results, dense_results) print('[\n ' + ',\n '.join(str(obj) for obj in merged_results) + '\n]') ``` ```console Response [expandable] [ {'_id': 'vec92', 'chunk_text': 'Income inequality widened in 2024 despite strong economic growth in developed nations.'}, {'_id': 'vec69', 'chunk_text': 'The agricultural sector faced challenges in 2024 due to extreme weather and rising input costs.'}, {'_id': 'vec70', 'chunk_text': 'Consumer spending patterns shifted in 2024, with a greater focus on experiences over goods.'}, {'_id': 'vec90', 'chunk_text': 'The surge in e-commerce in 2024 was facilitated by advancements in logistics technology.'}, {'_id': 'vec78', 'chunk_text': 'Consumer sentiment surveys in 2024 reflected optimism despite high interest rates.'}, {'_id': 'vec82', 'chunk_text': 'Renewable energy subsidies in 2024 reduced the global reliance on fossil fuels.'}, {'_id': 'vec94', 'chunk_text': 'Cyberattacks targeting financial institutions in 2024 led to record cybersecurity spending.'}, {'_id': 'vec65', 'chunk_text': 'The global shipping industry experienced declining freight rates in 2024 due to overcapacity and reduced demand.'}, {'_id': 'vec86', 'chunk_text': 'Digital transformation initiatives in 2024 drove productivity gains in the services sector.'}, {'_id': 'vec95', 'chunk_text': 'Automation in agriculture in 2024 increased yields but displaced rural workers.'}, {'_id': 'vec80', 'chunk_text': 'Technological innovation in the fintech sector disrupted traditional banking in 2024.'}, {'_id': 'vec67', 'chunk_text': 'Renewable energy projects accounted for a record share of global infrastructure investment in 2024.'}, {'_id': 'vec35', 'chunk_text': 'Unemployment hit at 2.4% in Q3 of 2024.'}, {'_id': 'vec36', 'chunk_text': 'Unemployment is expected to hit 2.5% in Q3 of 2024.'}, {'_id': 'vec6', 'chunk_text': 'Unemployment hit a record low of 3.7% in Q4 of 2024.'}, {'_id': 'vec42', 'chunk_text': 'Tech sector layoffs in Q3 2024 have reshaped hiring trends across high-growth industries.'}, {'_id': 'vec48', 'chunk_text': 'Wage growth outpaced inflation for the first time in years, signaling improved purchasing power in 2024.'}, {'_id': 'vec55', 'chunk_text': 'Trade tensions between the U.S. and China escalated in 2024, impacting global supply chains and investment flows.'}, {'_id': 'vec45', 'chunk_text': 'Corporate earnings in Q4 2024 were largely impacted by rising raw material costs and currency fluctuations.'}, {'_id': 'vec72', 'chunk_text': 'In early 2024, global GDP growth slowed, driven by weaker exports in Asia and Europe.'}, {'_id': 'vec29', 'chunk_text': 'Growth vs. value investing strategies in 2024.'}, {'_id': 'vec46', 'chunk_text': 'Economic recovery in Q2 2024 relied heavily on government spending in infrastructure and green energy projects.'}, {'_id': 'vec79', 'chunk_text': 'The resurgence of industrial policy in Q1 2024 focused on decoupling critical supply chains.'}, {'_id': 'vec87', 'chunk_text': "The U.S. labor market's resilience in 2024 defied predictions of a severe recession."}, {'_id': 'vec40', 'chunk_text': 'Labor market trends show a declining participation rate despite record low unemployment in 2024.'}, {'_id': 'vec37', 'chunk_text': 'In Q3 2025 unemployment for the prior year was revised to 2.2%'}, {'_id': 'vec58', 'chunk_text': 'Oil production cuts in Q1 2024 by OPEC nations drove prices higher, influencing global energy policies.'}, {'_id': 'vec47', 'chunk_text': 'The housing market saw a rebound in late 2024, driven by falling mortgage rates and pent-up demand.'}, {'_id': 'vec41', 'chunk_text': 'Forecasts of global supply chain disruptions eased in late 2024, but consumer prices remained elevated due to persistent demand.'}, {'_id': 'vec56', 'chunk_text': 'Consumer confidence indices remained resilient in Q2 2024 despite fears of an impending recession.'}, {'_id': 'vec63', 'chunk_text': 'Population aging emerged as a critical economic issue in 2024, especially in advanced economies.'}, {'_id': 'vec52', 'chunk_text': 'Record-breaking weather events in early 2024 have highlighted the growing economic impact of climate change.'}, {'_id': 'vec23', 'chunk_text': 'Top SPAC mergers in the technology sector for 2024.'}, {'_id': 'vec62', 'chunk_text': 'Private equity activity in 2024 focused on renewable energy and technology sectors amid shifting investor priorities.'}, {'_id': 'vec64', 'chunk_text': 'Rising commodity prices in 2024 strained emerging markets dependent on imports of raw materials.'}, {'_id': 'vec54', 'chunk_text': 'The global tourism sector showed signs of recovery in late 2024 after years of pandemic-related setbacks.'}, {'_id': 'vec96', 'chunk_text': 'New trade agreements signed 2022 will make an impact in 2024'}, {'_id': 'vec49', 'chunk_text': "China's economic growth in 2024 slowed to its lowest level in decades due to structural reforms and weak exports."}, {'_id': 'vec7', 'chunk_text': 'The CPI index rose by 6% in July 2024, raising concerns about purchasing power.'}, {'_id': 'vec84', 'chunk_text': "The IMF's 2024 global outlook highlighted risks of stagflation in emerging markets."}, {'_id': 'vec13', 'chunk_text': 'Credit card APRs reached an all-time high of 22.8% in 2024.'}, {'_id': 'vec53', 'chunk_text': 'Cryptocurrencies faced regulatory scrutiny in 2024, leading to volatility and reduced market capitalization.'}, {'_id': 'vec60', 'chunk_text': 'Healthcare spending in 2024 surged as governments expanded access to preventive care and pandemic preparedness.'}, {'_id': 'vec91', 'chunk_text': 'The impact of ESG investing on corporate strategies has been a major focus in 2024.'}, {'_id': 'vec68', 'chunk_text': 'Cybersecurity spending reached new highs in 2024, reflecting the growing threat of digital attacks on infrastructure.'}, {'_id': 'vec59', 'chunk_text': 'The adoption of digital currencies by central banks increased in 2024, reshaping monetary policy frameworks.'}, {'_id': 'vec39', 'chunk_text': 'The rise in energy prices significantly impacted inflation trends during the first half of 2024.'}, {'_id': 'vec66', 'chunk_text': 'Bank lending to small and medium-sized enterprises surged in 2024 as governments incentivized entrepreneurship.'}, {'_id': 'vec57', 'chunk_text': 'Startups in 2024 faced tighter funding conditions as venture capitalists focused on profitability over growth.'}, {'_id': 'vec75', 'chunk_text': 'The collapse of Silicon Valley Bank raised questions about regulatory oversight in 2024.'}, {'_id': 'vec51', 'chunk_text': 'The European Union introduced new fiscal policies in 2024 aimed at reducing public debt without stifling growth.'}, {'_id': 'vec89', 'chunk_text': 'Venture capital investments in 2024 leaned heavily toward AI and automation startups.'} ] ``` Use one of Pinecone's [hosted reranking models](/guides/search/rerank-results#reranking-models) to rerank the merged and deduplicated results based on a unified relevance score. For example, the following code sends the merged and deduplicated results to the `bge-reranker-v2-m3` reranking model and returns the top 40 results. ```python Python result = pc.inference.rerank( model="bge-reranker-v2-m3", query=query, documents=merged_results, rank_fields=["chunk_text"], top_n=40, return_documents=True, parameters={ "truncate": "END" } ) print("Query", query) print('-----') for row in result.data: print(f"{row['document']['_id']} - {round(row['score'], 2)} - {row['document']['chunk_text']}") ``` ```console Response [expandable] Query Q3 2024 us economic data ----- vec36 - 0.84 - Unemployment is expected to hit 2.5% in Q3 of 2024. vec35 - 0.76 - Unemployment hit at 2.4% in Q3 of 2024. vec48 - 0.33 - Wage growth outpaced inflation for the first time in years, signaling improved purchasing power in 2024. vec37 - 0.25 - In Q3 2025 unemployment for the prior year was revised to 2.2% vec42 - 0.21 - Tech sector layoffs in Q3 2024 have reshaped hiring trends across high-growth industries. vec87 - 0.2 - The U.S. labor market's resilience in 2024 defied predictions of a severe recession. vec63 - 0.08 - Population aging emerged as a critical economic issue in 2024, especially in advanced economies. vec92 - 0.08 - Income inequality widened in 2024 despite strong economic growth in developed nations. vec72 - 0.07 - In early 2024, global GDP growth slowed, driven by weaker exports in Asia and Europe. vec46 - 0.06 - Economic recovery in Q2 2024 relied heavily on government spending in infrastructure and green energy projects. vec6 - 0.06 - Unemployment hit a record low of 3.7% in Q4 of 2024. vec7 - 0.05 - The CPI index rose by 6% in July 2024, raising concerns about purchasing power. vec40 - 0.05 - Labor market trends show a declining participation rate despite record low unemployment in 2024. vec55 - 0.05 - Trade tensions between the U.S. and China escalated in 2024, impacting global supply chains and investment flows. vec70 - 0.04 - Consumer spending patterns shifted in 2024, with a greater focus on experiences over goods. vec41 - 0.04 - Forecasts of global supply chain disruptions eased in late 2024, but consumer prices remained elevated due to persistent demand. vec47 - 0.03 - The housing market saw a rebound in late 2024, driven by falling mortgage rates and pent-up demand. vec84 - 0.02 - The IMF's 2024 global outlook highlighted risks of stagflation in emerging markets. vec69 - 0.02 - The agricultural sector faced challenges in 2024 due to extreme weather and rising input costs. vec39 - 0.02 - The rise in energy prices significantly impacted inflation trends during the first half of 2024. vec78 - 0.01 - Consumer sentiment surveys in 2024 reflected optimism despite high interest rates. vec59 - 0.01 - The adoption of digital currencies by central banks increased in 2024, reshaping monetary policy frameworks. vec95 - 0.01 - Automation in agriculture in 2024 increased yields but displaced rural workers. vec52 - 0.01 - Record-breaking weather events in early 2024 have highlighted the growing economic impact of climate change. vec56 - 0.01 - Consumer confidence indices remained resilient in Q2 2024 despite fears of an impending recession. vec54 - 0.01 - The global tourism sector showed signs of recovery in late 2024 after years of pandemic-related setbacks. vec45 - 0.01 - Corporate earnings in Q4 2024 were largely impacted by rising raw material costs and currency fluctuations. vec86 - 0.01 - Digital transformation initiatives in 2024 drove productivity gains in the services sector. vec66 - 0.01 - Bank lending to small and medium-sized enterprises surged in 2024 as governments incentivized entrepreneurship. vec29 - 0.01 - Growth vs. value investing strategies in 2024. vec49 - 0.01 - China's economic growth in 2024 slowed to its lowest level in decades due to structural reforms and weak exports. vec60 - 0.01 - Healthcare spending in 2024 surged as governments expanded access to preventive care and pandemic preparedness. vec90 - 0.0 - The surge in e-commerce in 2024 was facilitated by advancements in logistics technology. vec64 - 0.0 - Rising commodity prices in 2024 strained emerging markets dependent on imports of raw materials. vec96 - 0.0 - New trade agreements signed 2022 will make an impact in 2024 vec57 - 0.0 - Startups in 2024 faced tighter funding conditions as venture capitalists focused on profitability over growth. vec91 - 0.0 - The impact of ESG investing on corporate strategies has been a major focus in 2024. vec62 - 0.0 - Private equity activity in 2024 focused on renewable energy and technology sectors amid shifting investor priorities. vec79 - 0.0 - The resurgence of industrial policy in Q1 2024 focused on decoupling critical supply chains. vec65 - 0.0 - The global shipping industry experienced declining freight rates in 2024 due to overcapacity and reduced demand. ``` ## Use a single hybrid index You can perform hybrid search with a single hybrid index as follows: To store both dense and sparse vectors in a single index, use the [`create_index`](/reference/api/2025-01/control-plane/create_index) operation, setting the `vector_type` to `dense` and the `metric` to `dotproduct`. This is the only combination that supports dense/sparse search on a single index. ```python Python from pinecone.grpc import PineconeGRPC as Pinecone from pinecone import ServerlessSpec pc = Pinecone(api_key="YOUR_API_KEY") index_name = "hybrid-index" if not pc.has_index(index_name): pc.create_index( name=index_name, vector_type="dense", dimension=1024, metric="dotproduct", spec=ServerlessSpec( cloud="aws", region="us-east-1" ) ) ``` Use Pinecone's [hosted embedding models](/guides/index-data/create-an-index#embedding-models) to [convert data into dense and sparse vectors](/reference/api/2025-01/inference/generate-embeddings). ```python Python [expandable] # Define the records data = [ { "_id": "vec1", "chunk_text": "Apple Inc. issued a $10 billion corporate bond in 2023." }, { "_id": "vec2", "chunk_text": "ETFs tracking the S&P 500 outperformed active funds last year." }, { "_id": "vec3", "chunk_text": "Tesla's options volume surged after the latest earnings report." }, { "_id": "vec4", "chunk_text": "Dividend aristocrats are known for consistently raising payouts." }, { "_id": "vec5", "chunk_text": "The Federal Reserve raised interest rates by 0.25% to curb inflation." }, { "_id": "vec6", "chunk_text": "Unemployment hit a record low of 3.7% in Q4 of 2024." }, { "_id": "vec7", "chunk_text": "The CPI index rose by 6% in July 2024, raising concerns about purchasing power." }, { "_id": "vec8", "chunk_text": "GDP growth in emerging markets outpaced developed economies." }, { "_id": "vec9", "chunk_text": "Amazon's acquisition of MGM Studios was valued at $8.45 billion." }, { "_id": "vec10", "chunk_text": "Alphabet reported a 20% increase in advertising revenue." }, { "_id": "vec11", "chunk_text": "ExxonMobil announced a special dividend after record profits." }, { "_id": "vec12", "chunk_text": "Tesla plans a 3-for-1 stock split to attract retail investors." }, { "_id": "vec13", "chunk_text": "Credit card APRs reached an all-time high of 22.8% in 2024." }, { "_id": "vec14", "chunk_text": "A 529 college savings plan offers tax advantages for education." }, { "_id": "vec15", "chunk_text": "Emergency savings should ideally cover 6 months of expenses." }, { "_id": "vec16", "chunk_text": "The average mortgage rate rose to 7.1% in December." }, { "_id": "vec17", "chunk_text": "The SEC fined a hedge fund $50 million for insider trading." }, { "_id": "vec18", "chunk_text": "New ESG regulations require companies to disclose climate risks." }, { "_id": "vec19", "chunk_text": "The IRS introduced a new tax bracket for high earners." }, { "_id": "vec20", "chunk_text": "Compliance with GDPR is mandatory for companies operating in Europe." }, { "_id": "vec21", "chunk_text": "What are the best-performing green bonds in a rising rate environment?" }, { "_id": "vec22", "chunk_text": "How does inflation impact the real yield of Treasury bonds?" }, { "_id": "vec23", "chunk_text": "Top SPAC mergers in the technology sector for 2024." }, { "_id": "vec24", "chunk_text": "Are stablecoins a viable hedge against currency devaluation?" }, { "_id": "vec25", "chunk_text": "Comparison of Roth IRA vs 401(k) for high-income earners." }, { "_id": "vec26", "chunk_text": "Stock splits and their effect on investor sentiment." }, { "_id": "vec27", "chunk_text": "Tech IPOs that disappointed in their first year." }, { "_id": "vec28", "chunk_text": "Impact of interest rate hikes on bank stocks." }, { "_id": "vec29", "chunk_text": "Growth vs. value investing strategies in 2024." }, { "_id": "vec30", "chunk_text": "The role of artificial intelligence in quantitative trading." }, { "_id": "vec31", "chunk_text": "What are the implications of quantitative tightening on equities?" }, { "_id": "vec32", "chunk_text": "How does compounding interest affect long-term investments?" }, { "_id": "vec33", "chunk_text": "What are the best assets to hedge against inflation?" }, { "_id": "vec34", "chunk_text": "Can ETFs provide better diversification than mutual funds?" }, { "_id": "vec35", "chunk_text": "Unemployment hit at 2.4% in Q3 of 2024." }, { "_id": "vec36", "chunk_text": "Unemployment is expected to hit 2.5% in Q3 of 2024." }, { "_id": "vec37", "chunk_text": "In Q3 2025 unemployment for the prior year was revised to 2.2%"}, { "_id": "vec38", "chunk_text": "Emerging markets witnessed increased foreign direct investment as global interest rates stabilized." }, { "_id": "vec39", "chunk_text": "The rise in energy prices significantly impacted inflation trends during the first half of 2024." }, { "_id": "vec40", "chunk_text": "Labor market trends show a declining participation rate despite record low unemployment in 2024." }, { "_id": "vec41", "chunk_text": "Forecasts of global supply chain disruptions eased in late 2024, but consumer prices remained elevated due to persistent demand." }, { "_id": "vec42", "chunk_text": "Tech sector layoffs in Q3 2024 have reshaped hiring trends across high-growth industries." }, { "_id": "vec43", "chunk_text": "The U.S. dollar weakened against a basket of currencies as the global economy adjusted to shifting trade balances." }, { "_id": "vec44", "chunk_text": "Central banks worldwide increased gold reserves to hedge against geopolitical and economic instability." }, { "_id": "vec45", "chunk_text": "Corporate earnings in Q4 2024 were largely impacted by rising raw material costs and currency fluctuations." }, { "_id": "vec46", "chunk_text": "Economic recovery in Q2 2024 relied heavily on government spending in infrastructure and green energy projects." }, { "_id": "vec47", "chunk_text": "The housing market saw a rebound in late 2024, driven by falling mortgage rates and pent-up demand." }, { "_id": "vec48", "chunk_text": "Wage growth outpaced inflation for the first time in years, signaling improved purchasing power in 2024." }, { "_id": "vec49", "chunk_text": "China's economic growth in 2024 slowed to its lowest level in decades due to structural reforms and weak exports." }, { "_id": "vec50", "chunk_text": "AI-driven automation in the manufacturing sector boosted productivity but raised concerns about job displacement." }, { "_id": "vec51", "chunk_text": "The European Union introduced new fiscal policies in 2024 aimed at reducing public debt without stifling growth." }, { "_id": "vec52", "chunk_text": "Record-breaking weather events in early 2024 have highlighted the growing economic impact of climate change." }, { "_id": "vec53", "chunk_text": "Cryptocurrencies faced regulatory scrutiny in 2024, leading to volatility and reduced market capitalization." }, { "_id": "vec54", "chunk_text": "The global tourism sector showed signs of recovery in late 2024 after years of pandemic-related setbacks." }, { "_id": "vec55", "chunk_text": "Trade tensions between the U.S. and China escalated in 2024, impacting global supply chains and investment flows." }, { "_id": "vec56", "chunk_text": "Consumer confidence indices remained resilient in Q2 2024 despite fears of an impending recession." }, { "_id": "vec57", "chunk_text": "Startups in 2024 faced tighter funding conditions as venture capitalists focused on profitability over growth." }, { "_id": "vec58", "chunk_text": "Oil production cuts in Q1 2024 by OPEC nations drove prices higher, influencing global energy policies." }, { "_id": "vec59", "chunk_text": "The adoption of digital currencies by central banks increased in 2024, reshaping monetary policy frameworks." }, { "_id": "vec60", "chunk_text": "Healthcare spending in 2024 surged as governments expanded access to preventive care and pandemic preparedness." }, { "_id": "vec61", "chunk_text": "The World Bank reported declining poverty rates globally, but regional disparities persisted." }, { "_id": "vec62", "chunk_text": "Private equity activity in 2024 focused on renewable energy and technology sectors amid shifting investor priorities." }, { "_id": "vec63", "chunk_text": "Population aging emerged as a critical economic issue in 2024, especially in advanced economies." }, { "_id": "vec64", "chunk_text": "Rising commodity prices in 2024 strained emerging markets dependent on imports of raw materials." }, { "_id": "vec65", "chunk_text": "The global shipping industry experienced declining freight rates in 2024 due to overcapacity and reduced demand." }, { "_id": "vec66", "chunk_text": "Bank lending to small and medium-sized enterprises surged in 2024 as governments incentivized entrepreneurship." }, { "_id": "vec67", "chunk_text": "Renewable energy projects accounted for a record share of global infrastructure investment in 2024." }, { "_id": "vec68", "chunk_text": "Cybersecurity spending reached new highs in 2024, reflecting the growing threat of digital attacks on infrastructure." }, { "_id": "vec69", "chunk_text": "The agricultural sector faced challenges in 2024 due to extreme weather and rising input costs." }, { "_id": "vec70", "chunk_text": "Consumer spending patterns shifted in 2024, with a greater focus on experiences over goods." }, { "_id": "vec71", "chunk_text": "The economic impact of the 2008 financial crisis was mitigated by quantitative easing policies." }, { "_id": "vec72", "chunk_text": "In early 2024, global GDP growth slowed, driven by weaker exports in Asia and Europe." }, { "_id": "vec73", "chunk_text": "The historical relationship between inflation and unemployment is explained by the Phillips Curve." }, { "_id": "vec74", "chunk_text": "The World Trade Organization's role in resolving disputes was tested in 2024." }, { "_id": "vec75", "chunk_text": "The collapse of Silicon Valley Bank raised questions about regulatory oversight in 2024." }, { "_id": "vec76", "chunk_text": "The cost of living crisis has been exacerbated by stagnant wage growth and rising inflation." }, { "_id": "vec77", "chunk_text": "Supply chain resilience became a top priority for multinational corporations in 2024." }, { "_id": "vec78", "chunk_text": "Consumer sentiment surveys in 2024 reflected optimism despite high interest rates." }, { "_id": "vec79", "chunk_text": "The resurgence of industrial policy in Q1 2024 focused on decoupling critical supply chains." }, { "_id": "vec80", "chunk_text": "Technological innovation in the fintech sector disrupted traditional banking in 2024." }, { "_id": "vec81", "chunk_text": "The link between climate change and migration patterns is increasingly recognized." }, { "_id": "vec82", "chunk_text": "Renewable energy subsidies in 2024 reduced the global reliance on fossil fuels." }, { "_id": "vec83", "chunk_text": "The economic fallout of geopolitical tensions was evident in rising defense budgets worldwide." }, { "_id": "vec84", "chunk_text": "The IMF's 2024 global outlook highlighted risks of stagflation in emerging markets." }, { "_id": "vec85", "chunk_text": "Declining birth rates in advanced economies pose long-term challenges for labor markets." }, { "_id": "vec86", "chunk_text": "Digital transformation initiatives in 2024 drove productivity gains in the services sector." }, { "_id": "vec87", "chunk_text": "The U.S. labor market's resilience in 2024 defied predictions of a severe recession." }, { "_id": "vec88", "chunk_text": "New fiscal measures in the European Union aimed to stabilize debt levels post-pandemic." }, { "_id": "vec89", "chunk_text": "Venture capital investments in 2024 leaned heavily toward AI and automation startups." }, { "_id": "vec90", "chunk_text": "The surge in e-commerce in 2024 was facilitated by advancements in logistics technology." }, { "_id": "vec91", "chunk_text": "The impact of ESG investing on corporate strategies has been a major focus in 2024." }, { "_id": "vec92", "chunk_text": "Income inequality widened in 2024 despite strong economic growth in developed nations." }, { "_id": "vec93", "chunk_text": "The collapse of FTX highlighted the volatility and risks associated with cryptocurrencies." }, { "_id": "vec94", "chunk_text": "Cyberattacks targeting financial institutions in 2024 led to record cybersecurity spending." }, { "_id": "vec95", "chunk_text": "Automation in agriculture in 2024 increased yields but displaced rural workers." }, { "_id": "vec96", "chunk_text": "New trade agreements signed 2022 will make an impact in 2024"}, ] ``` ```python Python # Convert the chunk_text into dense vectors dense_embeddings = pc.inference.embed( model="llama-text-embed-v2", inputs=[d['chunk_text'] for d in data], parameters={"input_type": "passage", "truncate": "END"} ) # Convert the chunk_text into sparse vectors sparse_embeddings = pc.inference.embed( model="pinecone-sparse-english-v0", inputs=[d['chunk_text'] for d in data], parameters={"input_type": "passage", "truncate": "END"} ) ``` Use the [`upsert`](/reference/api/2025-01/data-plane/upsert) operation, specifying dense values in the `value` parameter and sparse values in the `sparse_values` parameter. Only dense indexes using the [dotproduct distance metric](/guides/index-data/indexing-overview#dotproduct) support dense and sparse vectors. Upserting records with dense and sparse vectors into dense indexes with a different distance metric will succeed, but querying will return an error. ```Python Python # Target the hybrid index # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index index = pc.Index(host="INDEX_HOST") # Each record contains an ID, a dense vector, a sparse vector, and the original text as metadata records = [] for d, de, se in zip(data, dense_embeddings, sparse_embeddings): records.append({ "id": d['_id'], "values": de['values'], "sparse_values": {'indices': se['sparse_indices'], 'values': se['sparse_values']}, "metadata": {'text': d['chunk_text']} }) # Upsert the records into the hybrid index index.upsert( vectors=records, namespace="example-namespace" ) ``` Use the [`embed`](/reference/api/2025-01/inference/generate-embeddings) operation to convert your query into a dense vector and a sparse vector, and then use the [`query`](/reference/api/2025-01/data-plane/query) operation to search the hybrid index for the 40 most relevant records. ```Python Python query = "Q3 2024 us economic data" # Convert the query into a dense vector dense_query_embedding = pc.inference.embed( model="llama-text-embed-v2", inputs=query, parameters={"input_type": "query", "truncate": "END"} ) # Convert the query into a sparse vector sparse_query_embedding = pc.inference.embed( model="pinecone-sparse-english-v0", inputs=query, parameters={"input_type": "query", "truncate": "END"} ) for d, s in zip(dense_query_embedding, sparse_query_embedding): query_response = index.query( namespace="example-namespace", top_k=40, vector=d['values'], sparse_vector={'indices': s['sparse_indices'], 'values': s['sparse_values']}, include_values=False, include_metadata=True ) print(query_response) ``` ```python Response [expandable] {'matches': [{'id': 'vec35', 'metadata': {'text': 'Unemployment hit at 2.4% in Q3 of 2024.'}, 'score': 7.92519569, 'values': []}, {'id': 'vec46', 'metadata': {'text': 'Economic recovery in Q2 2024 relied ' 'heavily on government spending in ' 'infrastructure and green energy projects.'}, 'score': 7.86733627, 'values': []}, {'id': 'vec36', 'metadata': {'text': 'Unemployment is expected to hit 2.5% in Q3 ' 'of 2024.'}, 'score': 7.82636, 'values': []}, {'id': 'vec42', 'metadata': {'text': 'Tech sector layoffs in Q3 2024 have ' 'reshaped hiring trends across high-growth ' 'industries.'}, 'score': 7.79465914, 'values': []}, {'id': 'vec49', 'metadata': {'text': "China's economic growth in 2024 slowed to " 'its lowest level in decades due to ' 'structural reforms and weak exports.'}, 'score': 7.46323156, 'values': []}, {'id': 'vec63', 'metadata': {'text': 'Population aging emerged as a critical ' 'economic issue in 2024, especially in ' 'advanced economies.'}, 'score': 7.29055929, 'values': []}, {'id': 'vec92', 'metadata': {'text': 'Income inequality widened in 2024 despite ' 'strong economic growth in developed ' 'nations.'}, 'score': 6.51210213, 'values': []}, {'id': 'vec52', 'metadata': {'text': 'Record-breaking weather events in early ' '2024 have highlighted the growing economic ' 'impact of climate change.'}, 'score': 6.4125514, 'values': []}, {'id': 'vec62', 'metadata': {'text': 'Private equity activity in 2024 focused on ' 'renewable energy and technology sectors ' 'amid shifting investor priorities.'}, 'score': 4.8084693, 'values': []}, {'id': 'vec89', 'metadata': {'text': 'Venture capital investments in 2024 leaned ' 'heavily toward AI and automation ' 'startups.'}, 'score': 4.7974205, 'values': []}, {'id': 'vec57', 'metadata': {'text': 'Startups in 2024 faced tighter funding ' 'conditions as venture capitalists focused ' 'on profitability over growth.'}, 'score': 4.72518444, 'values': []}, {'id': 'vec37', 'metadata': {'text': 'In Q3 2025 unemployment for the prior year ' 'was revised to 2.2%'}, 'score': 4.71824408, 'values': []}, {'id': 'vec69', 'metadata': {'text': 'The agricultural sector faced challenges ' 'in 2024 due to extreme weather and rising ' 'input costs.'}, 'score': 4.66726208, 'values': []}, {'id': 'vec60', 'metadata': {'text': 'Healthcare spending in 2024 surged as ' 'governments expanded access to preventive ' 'care and pandemic preparedness.'}, 'score': 4.62045908, 'values': []}, {'id': 'vec55', 'metadata': {'text': 'Trade tensions between the U.S. and China ' 'escalated in 2024, impacting global supply ' 'chains and investment flows.'}, 'score': 4.59764862, 'values': []}, {'id': 'vec51', 'metadata': {'text': 'The European Union introduced new fiscal ' 'policies in 2024 aimed at reducing public ' 'debt without stifling growth.'}, 'score': 4.57397079, 'values': []}, {'id': 'vec70', 'metadata': {'text': 'Consumer spending patterns shifted in ' '2024, with a greater focus on experiences ' 'over goods.'}, 'score': 4.55043507, 'values': []}, {'id': 'vec87', 'metadata': {'text': "The U.S. labor market's resilience in 2024 " 'defied predictions of a severe recession.'}, 'score': 4.51785707, 'values': []}, {'id': 'vec90', 'metadata': {'text': 'The surge in e-commerce in 2024 was ' 'facilitated by advancements in logistics ' 'technology.'}, 'score': 4.47754288, 'values': []}, {'id': 'vec78', 'metadata': {'text': 'Consumer sentiment surveys in 2024 ' 'reflected optimism despite high interest ' 'rates.'}, 'score': 4.46246624, 'values': []}, {'id': 'vec53', 'metadata': {'text': 'Cryptocurrencies faced regulatory scrutiny ' 'in 2024, leading to volatility and reduced ' 'market capitalization.'}, 'score': 4.4435873, 'values': []}, {'id': 'vec45', 'metadata': {'text': 'Corporate earnings in Q4 2024 were largely ' 'impacted by rising raw material costs and ' 'currency fluctuations.'}, 'score': 4.43836403, 'values': []}, {'id': 'vec82', 'metadata': {'text': 'Renewable energy subsidies in 2024 reduced ' 'the global reliance on fossil fuels.'}, 'score': 4.43601322, 'values': []}, {'id': 'vec94', 'metadata': {'text': 'Cyberattacks targeting financial ' 'institutions in 2024 led to record ' 'cybersecurity spending.'}, 'score': 4.41334057, 'values': []}, {'id': 'vec47', 'metadata': {'text': 'The housing market saw a rebound in late ' '2024, driven by falling mortgage rates and ' 'pent-up demand.'}, 'score': 4.39900732, 'values': []}, {'id': 'vec41', 'metadata': {'text': 'Forecasts of global supply chain ' 'disruptions eased in late 2024, but ' 'consumer prices remained elevated due to ' 'persistent demand.'}, 'score': 4.37389421, 'values': []}, {'id': 'vec84', 'metadata': {'text': "The IMF's 2024 global outlook highlighted " 'risks of stagflation in emerging markets.'}, 'score': 4.37335157, 'values': []}, {'id': 'vec96', 'metadata': {'text': 'New trade agreements signed 2022 will make ' 'an impact in 2024'}, 'score': 4.33860636, 'values': []}, {'id': 'vec79', 'metadata': {'text': 'The resurgence of industrial policy in Q1 ' '2024 focused on decoupling critical supply ' 'chains.'}, 'score': 4.33784199, 'values': []}, {'id': 'vec6', 'metadata': {'text': 'Unemployment hit a record low of 3.7% in ' 'Q4 of 2024.'}, 'score': 4.33008051, 'values': []}, {'id': 'vec65', 'metadata': {'text': 'The global shipping industry experienced ' 'declining freight rates in 2024 due to ' 'overcapacity and reduced demand.'}, 'score': 4.3228569, 'values': []}, {'id': 'vec64', 'metadata': {'text': 'Rising commodity prices in 2024 strained ' 'emerging markets dependent on imports of ' 'raw materials.'}, 'score': 4.32269621, 'values': []}, {'id': 'vec95', 'metadata': {'text': 'Automation in agriculture in 2024 ' 'increased yields but displaced rural ' 'workers.'}, 'score': 4.31127262, 'values': []}, {'id': 'vec86', 'metadata': {'text': 'Digital transformation initiatives in 2024 ' 'drove productivity gains in the services ' 'sector.'}, 'score': 4.30181122, 'values': []}, {'id': 'vec66', 'metadata': {'text': 'Bank lending to small and medium-sized ' 'enterprises surged in 2024 as governments ' 'incentivized entrepreneurship.'}, 'score': 4.27241945, 'values': []}, {'id': 'vec58', 'metadata': {'text': 'Oil production cuts in Q1 2024 by OPEC ' 'nations drove prices higher, influencing ' 'global energy policies.'}, 'score': 4.21715498, 'values': []}, {'id': 'vec80', 'metadata': {'text': 'Technological innovation in the fintech ' 'sector disrupted traditional banking in ' '2024.'}, 'score': 4.17712116, 'values': []}, {'id': 'vec75', 'metadata': {'text': 'The collapse of Silicon Valley Bank raised ' 'questions about regulatory oversight in ' '2024.'}, 'score': 4.16192341, 'values': []}, {'id': 'vec56', 'metadata': {'text': 'Consumer confidence indices remained ' 'resilient in Q2 2024 despite fears of an ' 'impending recession.'}, 'score': 4.15782213, 'values': []}, {'id': 'vec67', 'metadata': {'text': 'Renewable energy projects accounted for a ' 'record share of global infrastructure ' 'investment in 2024.'}, 'score': 4.14623, 'values': []}], 'namespace': 'example-namespace', 'usage': {'read_units': 9}} ``` Because Pinecone views your sparse-dense vector as a single vector, it does not offer a built-in parameter to adjust the weight of a query's dense part against its sparse part; the index is agnostic to density or sparsity of coordinates in your vectors. You may, however, incorporate a linear weighting scheme by customizing your query vector, as we demonstrate in the function below. The following example transforms vector values using an alpha parameter. ```Python Python def hybrid_score_norm(dense, sparse, alpha: float): """Hybrid score using a convex combination alpha * dense + (1 - alpha) * sparse Args: dense: Array of floats representing sparse: a dict of `indices` and `values` alpha: scale between 0 and 1 """ if alpha < 0 or alpha > 1: raise ValueError("Alpha must be between 0 and 1") hs = { 'indices': sparse['indices'], 'values': [v * (1 - alpha) for v in sparse['values']] } return [v * alpha for v in dense], hs ``` The following example transforms a vector using the above function, then queries a Pinecone index. ```Python Python sparse_vector = { 'indices': [10, 45, 16], 'values': [0.5, 0.5, 0.2] } dense_vector = [0.1, 0.2, 0.3] hdense, hsparse = hybrid_score_norm(dense_vector, sparse_vector, alpha=0.75) query_response = index.query( namespace="example-namespace", top_k=10, vector=hdense, sparse_vector=hsparse ) ``` # Lexical search Source: https://docs.pinecone.io/guides/search/lexical-search This page shows you how to search a [sparse index](/guides/index-data/indexing-overview#sparse-indexes) for records that most exactly match the words or phrases in a query. This is often called lexical search or keyword search. Lexical search uses [sparse vectors](https://www.pinecone.io/learn/sparse-retrieval/), which have a very large number of dimensions, where only a small proportion of values are non-zero. The dimensions represent words from a dictionary, and the values represent the importance of these words in the document. Words are scored independently and then summed, with the most similar records scored highest. This feature is in [public preview](/release-notes/feature-availability). ## Search with text Searching with text is supported only for [indexes with integrated embedding](/guides/index-data/indexing-overview#integrated-embedding). To search a sparse index with a query text, use the [`search_records`](/reference/api/2025-01/data-plane/search_records) operation with the following parameters: * The `namespace` to query. To use the default namespace, set the namespace to `"__default__"`. * The `query.inputs.text` parameter with the query text. Pinecone uses the embedding model integrated with the index to convert the text to a sparse vector automatically. * The `query.top_k` parameter with the number of similar records to return. * Optionally, you can specify the `fields` to return in the response. If not specified, the response will include all fields. For example, the following code converts the query “What is AAPL's outlook, considering both product launches and market conditions?” to a sparse vector and then searches for the 3 most similar vectors in the `example-namespaces` namespace: ```python Python from pinecone import Pinecone pc = Pinecone(api_key="YOUR_API_KEY") # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index index = pc.Index(host="INDEX_HOST") results = index.search( namespace="example-namespace", query={ "inputs": {"text": "What is AAPL's outlook, considering both product launches and market conditions?"}, "top_k": 3 }, fields=["chunk_text", "quarter"] ) print(results) ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: "YOUR_API_KEY" }) // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index const namespace = pc.index("INDEX_NAME", "INDEX_HOST").namespace("example-namespace"); const response = await namespace.searchRecords({ query: { topK: 3, inputs: { text: "What is AAPL's outlook, considering both product launches and market conditions?" }, }, fields: ['chunk_text', 'quarter'] }); console.log(response); ``` ```java Java import io.pinecone.clients.Index; import io.pinecone.configs.PineconeConfig; import io.pinecone.configs.PineconeConnection; import org.openapitools.db_data.client.ApiException; import org.openapitools.db_data.client.model.SearchRecordsResponse; import java.util.*; public class SearchText { public static void main(String[] args) throws ApiException { PineconeConfig config = new PineconeConfig("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index config.setHost("INDEX_HOST"); PineconeConnection connection = new PineconeConnection(config); Index index = new Index(config, connection, "integrated-sparse-java"); String query = "What is AAPL's outlook, considering both product launches and market conditions?"; List fields = new ArrayList<>(); fields.add("category"); fields.add("chunk_text"); // Search the sparse index SearchRecordsResponse recordsResponse = index.searchRecordsByText(query, "example-namespace", fields, 3, null, null); // Print the results System.out.println(recordsResponse); } } ``` ```go Go package main import ( "context" "encoding/json" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func prettifyStruct(obj interface{}) string { bytes, _ := json.MarshalIndent(obj, "", " ") return string(bytes) } func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "INDEX_HOST", Namespace: "example-namespace"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v", err) } res, err := idxConnection.SearchRecords(ctx, &pinecone.SearchRecordsRequest{ Query: pinecone.SearchRecordsQuery{ TopK: 3, Inputs: &map[string]interface{}{ "text": "What is AAPL's outlook, considering both product launches and market conditions?", }, }, Fields: &[]string{"chunk_text", "category"}, }) if err != nil { log.Fatalf("Failed to search records: %v", err) } fmt.Printf(prettifyStruct(res)) } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index var index = pinecone.Index(host: "INDEX_HOST"); var response = await index.SearchRecordsAsync( "example-namespace", new SearchRecordsRequest { Query = new SearchRecordsRequestQuery { TopK = 3, Inputs = new Dictionary { { "text", "What is AAPL's outlook, considering both product launches and market conditions?" } }, }, Fields = ["category", "chunk_text"], } ); Console.WriteLine(response); ``` ```shell curl PINECONE_API_KEY="YOUR_API_KEY" INDEX_HOST="INDEX_HOST" curl "https://$INDEX_HOST/records/namespaces/example-namespace/search" \ -H "Content-Type: application/json" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "query": { "inputs": { "text": "What is AAPL'\''s outlook, considering both product launches and market conditions?" }, "top_k": 3 }, "fields": ["chunk_text", "quarter"] }' ``` The results will look as follows. The most similar records are scored highest. ```python Python {'result': {'hits': [{'_id': 'vec2', '_score': 10.77734375, 'fields': {'chunk_text': "Analysts suggest that AAPL'''s " 'upcoming Q4 product launch ' 'event might solidify its ' 'position in the premium ' 'smartphone market.', 'quarter': 'Q4'}}, {'_id': 'vec3', '_score': 6.49066162109375, 'fields': {'chunk_text': "AAPL'''s strategic Q3 " 'partnerships with ' 'semiconductor suppliers could ' 'mitigate component risks and ' 'stabilize iPhone production.', 'quarter': 'Q3'}}, {'_id': 'vec1', '_score': 5.3671875, 'fields': {'chunk_text': 'AAPL reported a year-over-year ' 'revenue increase, expecting ' 'stronger Q3 demand for its ' 'flagship phones.', 'quarter': 'Q3'}}]}, 'usage': {'embed_total_tokens': 18, 'read_units': 1}} ``` ```javascript JavaScript { result: { hits: [ { _id: "vec2", _score: 10.82421875, fields: { chunk_text: "Analysts suggest that AAPL'''s upcoming Q4 product launch event might solidify its position in the premium smartphone market.", quarter: "Q4" } }, { _id: "vec3", _score: 6.49066162109375, fields: { chunk_text: "AAPL'''s strategic Q3 partnerships with semiconductor suppliers could mitigate component risks and stabilize iPhone production.", quarter: "Q3" } }, { _id: "vec1", _score: 5.3671875, fields: { chunk_text: "AAPL reported a year-over-year revenue increase, expecting stronger Q3 demand for its flagship phones.", quarter: "Q3" } } ] }, usage: { readUnits: 1, embedTotalTokens: 18 } } ``` ```java Java class SearchRecordsResponse { result: class SearchRecordsResponseResult { hits: [class Hit { id: vec2 score: 10.82421875 fields: {chunk_text=Analysts suggest that AAPL's upcoming Q4 product launch event might solidify its position in the premium smartphone market., quarter=Q4} additionalProperties: null }, class Hit { id: vec3 score: 6.49066162109375 fields: {chunk_text=AAAPL'''s strategic Q3 partnerships with semiconductor suppliers could mitigate component risks and stabilize iPhone production., quarter=Q3} additionalProperties: null }, class Hit { id: vec1 score: 5.3671875 fields: {chunk_text=AAPL reported a year-over-year revenue increase, expecting stronger Q3 demand for its flagship phones., quarter=Q3} additionalProperties: null }] additionalProperties: null } usage: class SearchUsage { readUnits: 1 embedTotalTokens: 18 } additionalProperties: null } ``` ```go Go { "result": { "hits": [ { "_id": "vec2", "_score": 10.833984, "fields": { "chunk_text": "Analysts suggest that AAPL's upcoming Q4 product launch event might solidify its position in the premium smartphone market.", "quarter": "Q4" } }, { "_id": "vec3", "_score": 6.473572, "fields": { "chunk_text": "AAPL's strategic Q3 partnerships with semiconductor suppliers could mitigate component risks and stabilize iPhone production.", "quarter": "Q3" } }, { "_id": "vec1", "_score": 5.3710938, "fields": { "chunk_text": "AAPL reported a year-over-year revenue increase, expecting stronger Q3 demand for its flagship phones.", "quarter": "Q3" } } ] }, "usage": { "read_units": 6, "embed_total_tokens": 18 } } ``` ```csharp C# { "result": { "hits": [ { "_id": "vec2", "_score": 10.833984, "fields": { "chunk_text": "Analysts suggest that AAPL's upcoming Q4 product launch event might solidify its position in the premium smartphone market.", "quarter": "Q4" } }, { "_id": "vec3", "_score": 6.473572, "fields": { "chunk_text": "AAPL's strategic Q3 partnerships with semiconductor suppliers could mitigate component risks and stabilize iPhone production.", "quarter": "Q3" } }, { "_id": "vec1", "_score": 5.3710938, "fields": { "chunk_text": "AAPL reported a year-over-year revenue increase, expecting stronger Q3 demand for its flagship phones.", "quarter": "Q3" } } ] }, "usage": { "read_units": 6, "embed_total_tokens": 18 } } ``` ```json curl { "result": { "hits": [ { "_id": "vec2", "_score": 10.82421875, "fields": { "chunk_text": "Analysts suggest that AAPL'''s upcoming Q4 product launch event might solidify its position in the premium smartphone market.", "quarter": "Q4" } }, { "_id": "vec3", "_score": 6.49066162109375, "fields": { "chunk_text": "AAPL'''s strategic Q3 partnerships with semiconductor suppliers could mitigate component risks and stabilize iPhone production.", "quarter": "Q3" } }, { "_id": "vec1", "_score": 5.3671875, "fields": { "chunk_text": "AAPL reported a year-over-year revenue increase, expecting stronger Q3 demand for its flagship phones.", "quarter": "Q3" } } ] }, "usage": { "embed_total_tokens": 18, "read_units": 1 } } ``` ## Search with a sparse vector To search a sparse index with a sparse vector representation of a query, use the [`query`](/reference/api/2025-01/data-plane/query) operation with the following parameters: * The `namespace` to query. To use the default namespace, set the namespace to `"__default__"`. * The `sparse_vector` parameter with the sparse vector values and indices. * The `top_k` parameter with the number of results to return. * Optionally, you can set `include_values` and/or `include_metadata` to `true` to include the vector values and/or metadata of the matching records in the response. However, when querying with `top_k` over 1000, avoid returning vector data or metadata for optimal performance. For example, the following code uses a sparse vector representation of the query "What is AAPL's outlook, considering both product launches and market conditions?" to search for the 3 most similar vectors in the `example-namespaces` namespace: ```python Python from pinecone import Pinecone pc = Pinecone(api_key="YOUR_API_KEY") # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index index = pc.Index(host="INDEX_HOST") results = index.query( namespace="example-namespace", sparse_vector={ "values": [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0], "indices": [767227209, 1640781426, 1690623792, 2021799277, 2152645940, 2295025838, 2443437770, 2779594451, 2956155693, 3476647774, 3818127854, 4283091697] }, top_k=3, include_metadata=True, include_values=False ) print(results) ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: "YOUR_API_KEY" }) // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index const index = pc.index("INDEX_NAME", "INDEX_HOST") const queryResponse = await index.namespace('example-namespace').query({ sparseVector: { indices: [767227209, 1640781426, 1690623792, 2021799277, 2152645940, 2295025838, 2443437770, 2779594451, 2956155693, 3476647774, 3818127854, 4283091697], values: [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] }, topK: 3, includeValues: false, includeMetadata: true }); console.log(queryResponse); ``` ```java Java import io.pinecone.clients.Pinecone; import io.pinecone.unsigned_indices_model.QueryResponseWithUnsignedIndices; import io.pinecone.clients.Index; import java.util.*; public class SearchSparseIndex { public static void main(String[] args) throws InterruptedException { // Instantiate Pinecone class Pinecone pinecone = new Pinecone.Builder("YOUR_API_KEY").build(); String indexName = "docs-example"; Index index = pinecone.getIndexConnection(indexName); List sparseIndices = Arrays.asList( 767227209L, 1640781426L, 1690623792L, 2021799277L, 2152645940L, 2295025838L, 2443437770L, 2779594451L, 2956155693L, 3476647774L, 3818127854L, 428309169L); List sparseValues = Arrays.asList( 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f); QueryResponseWithUnsignedIndices queryResponse = index.query(3, null, sparseIndices, sparseValues, null, "example-namespace", null, false, true); System.out.println(queryResponse); } } ``` ```go Go package main import ( "context" "encoding/json" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func prettifyStruct(obj interface{}) string { bytes, _ := json.MarshalIndent(obj, "", " ") return string(bytes) } func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "INDEX_HOST", Namespace: "example-namespace"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v", err) } sparseValues := pinecone.SparseValues{ Indices: []uint32{767227209, 1640781426, 1690623792, 2021799277, 2152645940, 2295025838, 2443437770, 2779594451, 2956155693, 3476647774, 3818127854, 4283091697}, Values: []float32{1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0}, } res, err := idxConnection.QueryByVectorValues(ctx, &pinecone.QueryByVectorValuesRequest{ SparseValues: &sparseValues, TopK: 3, IncludeValues: false, IncludeMetadata: true, }) if err != nil { log.Fatalf("Error encountered when querying by vector: %v", err) } else { fmt.Printf(prettifyStruct(res)) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); var index = pinecone.Index("docs-example"); var queryResponse = await index.QueryAsync(new QueryRequest { Namespace = "example-namespace", TopK = 4, SparseVector = new SparseValues { Indices = [767227209, 1640781426, 1690623792, 2021799277, 2152645940, 2295025838, 2443437770, 2779594451, 2956155693, 3476647774, 3818127854, 4283091697], Values = new[] { 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f }, }, IncludeValues = false, IncludeMetadata = true }); Console.WriteLine(queryResponse); ``` ```shell curl PINECONE_API_KEY="YOUR_API_KEY" INDEX_HOST="INDEX_HOST" curl "https://$INDEX_HOST/query" \ -H "Content-Type: application/json" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "sparseVector": { "values": [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0], "indices": [767227209, 1640781426, 1690623792, 2021799277, 2152645940, 2295025838, 2443437770, 2779594451, 2956155693, 3476647774, 3818127854, 4283091697] }, "namespace": "example-namespace", "topK": 4, "includeMetadata": true, "includeValues": false }' ``` The results will look as follows. The most similar records are scored highest. ```python Python {'matches': [{'id': 'vec2', 'metadata': {'category': 'technology', 'quarter': 'Q4', 'chunk_text': "Analysts suggest that AAPL'''s " 'upcoming Q4 product launch event ' 'might solidify its position in the ' 'premium smartphone market.'}, 'score': 10.9042969, 'values': []}, {'id': 'vec3', 'metadata': {'category': 'technology', 'quarter': 'Q3', 'chunk_text': "AAPL'''s strategic Q3 partnerships " 'with semiconductor suppliers could ' 'mitigate component risks and ' 'stabilize iPhone production'}, 'score': 6.48010254, 'values': []}, {'id': 'vec1', 'metadata': {'category': 'technology', 'quarter': 'Q3', 'chunk_text': 'AAPL reported a year-over-year ' 'revenue increase, expecting ' 'stronger Q3 demand for its flagship ' 'phones.'}, 'score': 5.3671875, 'values': []}], 'namespace': 'example-namespace', 'usage': {'read_units': 1}} ``` ```javascript JavaScript { matches: [ { id: 'vec2', score: 10.9042969, values: [], metadata: { chunk_text: "Analysts suggest that AAPL'''s upcoming Q4 product launch event might solidify its position in the premium smartphone market.", category: 'technology', quarter: 'Q4' } }, { id: 'vec3', score: 6.48010254, values: [], metadata: { chunk_text: "AAPL'''s strategic Q3 partnerships with semiconductor suppliers could mitigate component risks and stabilize iPhone production.", category: 'technology', quarter: 'Q3' } }, { id: 'vec1', score: 5.3671875, values: [], metadata: { chunk_text: 'AAPL reported a year-over-year revenue increase, expecting stronger Q3 demand for its flagship phones.', category: 'technology', quarter: 'Q3' } } ], namespace: 'example-namespace', usage: {readUnits: 1} } ``` ```java Java class QueryResponseWithUnsignedIndices { matches: [ScoredVectorWithUnsignedIndices { score: 10.34375 id: vec2 values: [] metadata: fields { key: "category" value { string_value: "technology" } } fields { key: "chunk_text" value { string_value: "Analysts suggest that AAPL\'\\\'\'s upcoming Q4 product launch event might solidify its position in the premium smartphone market." } } fields { key: "quarter" value { string_value: "Q4" } } sparseValuesWithUnsignedIndices: SparseValuesWithUnsignedIndices { indicesWithUnsigned32Int: [] values: [] } }, ScoredVectorWithUnsignedIndices { score: 5.8638916 id: vec3 values: [] metadata: fields { key: "category" value { string_value: "technology" } } fields { key: "chunk_text" value { string_value: "AAPL\'\\\'\'s strategic Q3 partnerships with semiconductor suppliers could mitigate component risks and stabilize iPhone production" } } fields { key: "quarter" value { string_value: "Q3" } } sparseValuesWithUnsignedIndices: SparseValuesWithUnsignedIndices { indicesWithUnsigned32Int: [] values: [] } }, ScoredVectorWithUnsignedIndices { score: 5.3671875 id: vec1 values: [] metadata: fields { key: "category" value { string_value: "technology" } } fields { key: "chunk_text" value { string_value: "AAPL reported a year-over-year revenue increase, expecting stronger Q3 demand for its flagship phones." } } fields { key: "quarter" value { string_value: "Q3" } } sparseValuesWithUnsignedIndices: SparseValuesWithUnsignedIndices { indicesWithUnsigned32Int: [] values: [] } }] namespace: example-namespace usage: read_units: 1 } ``` ```go Go { "matches": [ { "vector": { "id": "vec2", "metadata": { "category": "technology", "quarter": "Q4", "chunk_text": "Analysts suggest that AAPL's upcoming Q4 product launch event might solidify its position in the premium smartphone market." } }, "score": 10.904296 }, { "vector": { "id": "vec3", "metadata": { "category": "technology", "quarter": "Q3", "chunk_text": "AAPL's strategic Q3 partnerships with semiconductor suppliers could mitigate component risks and stabilize iPhone production" } }, "score": 6.4801025 }, { "vector": { "id": "vec1", "metadata": { "category": "technology", "quarter": "Q3", "chunk_text": "AAPL reported a year-over-year revenue increase, expecting stronger Q3 demand for its flagship phones" } }, "score": 5.3671875 } ], "usage": { "read_units": 1 }, "namespace": "example-namespace" } ``` ```csharp C# { "results": [], "matches": [ { "id": "vec2", "score": 10.904297, "values": [], "metadata": { "category": "technology", "chunk_text": "Analysts suggest that AAPL\u0027\u0027\u0027s upcoming Q4 product launch event might solidify its position in the premium smartphone market.", "quarter": "Q4" } }, { "id": "vec3", "score": 6.4801025, "values": [], "metadata": { "category": "technology", "chunk_text": "AAPL\u0027\u0027\u0027s strategic Q3 partnerships with semiconductor suppliers could mitigate component risks and stabilize iPhone production", "quarter": "Q3" } }, { "id": "vec1", "score": 5.3671875, "values": [], "metadata": { "category": "technology", "chunk_text": "AAPL reported a year-over-year revenue increase, expecting stronger Q3 demand for its flagship phones.", "quarter": "Q3" } } ], "namespace": "example-namespace", "usage": { "readUnits": 1 } } ``` ```json curl { "results": [], "matches": [ { "id": "vec2", "score": 10.9042969, "values": [], "metadata": { "chunk_text": "Analysts suggest that AAPL'''s upcoming Q4 product launch event might solidify its position in the premium smartphone market.", "category": "technology", "quarter": "Q4" } }, { "id": "vec3", "score": 6.48010254, "values": [], "metadata": { "chunk_text": "AAPL'''s strategic Q3 partnerships with semiconductor suppliers could mitigate component risks and stabilize iPhone production.", "category": "technology", "quarter": "Q3" } }, { "id": "vec1", "score": 5.3671875, "values": [], "metadata": { "chunk_text": "AAPL reported a year-over-year revenue increase, expecting stronger Q3 demand for its flagship phones.", "category": "technology", "quarter": "Q3" } } ], "namespace": "example-namespace", "usage": { "readUnits": 1 } } ``` ## Search with a record ID When you search with a record ID, Pinecone uses the sparse vector associated with the record as the query. To search a sparse index with a record ID, use the [`query`](/reference/api/2025-01/data-plane/query) operation with the following parameters: * The `namespace` to query. To use the default namespace, set the namespace to `"__default__"`. * The `id` parameter with the unique record ID containing the sparse vector to use as the query. * The `top_k` parameter with the number of results to return. * Optionally, you can set `include_values` and/or `include_metadata` to `true` to include the vector values and/or metadata of the matching records in the response. However, when querying with `top_k` over 1000, avoid returning vector data or metadata for optimal performance. For example, the following code uses an ID to search for the 3 records in the `example-namespace` namespace that best match the sparse vector in the record: ```Python Python from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key="YOUR_API_KEY") # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index index = pc.Index(host="INDEX_HOST") index.query( namespace="example-namespace", id="rec2", top_k=3, include_metadata=True, include_values=False ) ``` ```JavaScript JavaScript import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: "YOUR_API_KEY" }) // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index const index = pc.index("INDEX_NAME", "INDEX_HOST") const queryResponse = await index.namespace('example-namespace').query({ id: 'rec2', topK: 3, includeValues: false, includeMetadata: true, }); ``` ```java Java import io.pinecone.clients.Index; import io.pinecone.configs.PineconeConfig; import io.pinecone.configs.PineconeConnection; import io.pinecone.unsigned_indices_model.QueryResponseWithUnsignedIndices; public class QueryExample { public static void main(String[] args) { PineconeConfig config = new PineconeConfig("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index config.setHost("INDEX_HOST"); PineconeConnection connection = new PineconeConnection(config); Index index = new Index(connection, "INDEX_NAME"); QueryResponseWithUnsignedIndices queryRespone = index.queryByVectorId(3, "rec2", "example-namespace", null, false, true); System.out.println(queryResponse); } } ``` ```go Go package main import ( "context" "encoding/json" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func prettifyStruct(obj interface{}) string { bytes, _ := json.MarshalIndent(obj, "", " ") return string(bytes) } func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "INDEX_HOST", Namespace: "example-namespace"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v", err) } vectorId := "rec2" res, err := idxConnection.QueryByVectorId(ctx, &pinecone.QueryByVectorIdRequest{ VectorId: vectorId, TopK: 3, IncludeValues: false, IncludeMetadata: true, }) if err != nil { log.Fatalf("Error encountered when querying by vector ID `%v`: %v", vectorId, err) } else { fmt.Printf(prettifyStruct(res.Matches)) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index var index = pinecone.Index(host: "INDEX_HOST"); var queryResponse = await index.QueryAsync(new QueryRequest { Id = "rec2", Namespace = "example-namespace", TopK = 3, IncludeValues = false, IncludeMetadata = true }); Console.WriteLine(queryResponse); ``` ```bash curl # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index PINECONE_API_KEY="YOUR_API_KEY" INDEX_HOST="INDEX_HOST" curl "https://$INDEX_HOST/query" \ -H "Api-Key: $PINECONE_API_KEY" \ -H 'Content-Type: application/json' \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "id": "rec2", "namespace": "example-namespace", "topK": 3, "includeMetadata": true, "includeValues": false }' ``` # Rerank results Source: https://docs.pinecone.io/guides/search/rerank-results Reranking is used as part of a two-stage vector retrieval process to improve the quality of results. You first query an index for a given number of relevant results, and then you send the query and results to a reranking model. The reranking model scores the results based on their semantic relevance to the query and returns a new, more accurate ranking. This approach is one of the simplest methods for improving quality in retrieval augmented generation (RAG) pipelines. Pinecone provides [hosted reranking models](#reranking-models) so it's easy to manage two-stage vector retrieval on a single platform. You can use a hosted model to rerank results as an integrated part of a query, or you can use a hosted model or external model to rerank results as a standalone operation. {/* To run through this guide in your browser, see the [Rerank example notebook](https://colab.research.google.com/github/pinecone-io/examples/blob/master/docs/pinecone-reranker.ipynb). */} ## Integrated reranking To rerank initial results as an integrated part of a query, without any extra steps, use the [`search`](/reference/api/2025-01/data-plane/search_records) operation with the `rerank` parameter, including the [hosted reranking model](#reranking-models) you want to use, the number of reranked results to return, and the fields to use for reranking, if different than the main query. For example, the following code searches for the 3 records most semantically related to a query text and uses the `hosted bge-reranker-v2-m3` model to rerank the results and return only the 2 most relevant documents: ```python Python from pinecone import Pinecone pc = Pinecone(api_key="YOUR_API_KEY") # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index index = pc.Index(host="INDEX_HOST") ranked_results = index.search( namespace="example-namespace", query={ "inputs": {"text": "Disease prevention"}, "top_k": 4 }, rerank={ "model": "bge-reranker-v2-m3", "top_n": 2, "rank_fields": ["chunk_text"] }, fields=["category", "chunk_text"] ) print(ranked_results) ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: "YOUR_API_KEY" }) // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index const namespace = pc.index("INDEX_NAME", "INDEX_HOST").namespace("example-namespace"); const response = await namespace.searchRecords({ query: { topK: 2, inputs: { text: 'Disease prevention' }, }, fields: ['chunk_text', 'category'], rerank: { model: 'bge-reranker-v2-m3', rankFields: ['chunk_text'], topN: 2, }, }); console.log(response); ``` ```java Java import io.pinecone.clients.Index; import io.pinecone.configs.PineconeConfig; import io.pinecone.configs.PineconeConnection; import org.openapitools.db_data.client.ApiException; import org.openapitools.db_data.client.model.SearchRecordsRequestRerank; import org.openapitools.db_data.client.model.SearchRecordsResponse; import java.util.*; public class SearchText { public static void main(String[] args) throws ApiException { PineconeConfig config = new PineconeConfig("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index config.setHost("INDEX_HOST"); PineconeConnection connection = new PineconeConnection(config); Index index = new Index(config, connection, "integrated-dense-java"); String query = "Disease prevention"; List fields = new ArrayList<>(); fields.add("category"); fields.add("chunk_text"); ListrankFields = new ArrayList<>(); rankFields.add("chunk_text"); SearchRecordsRequestRerank rerank = new SearchRecordsRequestRerank() .query(query) .model("bge-reranker-v2-m3") .topN(2) .rankFields(rankFields); SearchRecordsResponse recordsResponseReranked = index.searchRecordsByText(query, "example-namespace", fields,4, null, rerank); System.out.println(recordsResponseReranked); } } ``` ```go Go package main import ( "context" "encoding/json" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func prettifyStruct(obj interface{}) string { bytes, _ := json.MarshalIndent(obj, "", " ") return string(bytes) } func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "INDEX_HOST", Namespace: "example-namespace"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v", err) } topN := int32(2) res, err := idxConnection.SearchRecords(ctx, &pinecone.SearchRecordsRequest{ Query: pinecone.SearchRecordsQuery{ TopK: 3, Inputs: &map[string]interface{}{ "text": "Disease prevention", }, }, Rerank: &pinecone.SearchRecordsRerank{ Model: "bge-reranker-v2-m3", TopN: &topN, RankFields: []string{"chunk_text"}, }, Fields: &[]string{"chunk_text", "category"}, }) if err != nil { log.Fatalf("Failed to search records: %v", err) } fmt.Printf(prettifyStruct(res)) } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index var index = pinecone.Index(host: "INDEX_HOST"); var response = await index.SearchRecordsAsync( "example-namespace", new SearchRecordsRequest { Query = new SearchRecordsRequestQuery { TopK = 4, Inputs = new Dictionary { { "text", "Disease prevention" } }, }, Fields = ["category", "chunk_text"], Rerank = new SearchRecordsRequestRerank { Model = "bge-reranker-v2-m3", TopN = 2, RankFields = ["chunk_text"], } } ); Console.WriteLine(response); ``` ```shell curl INDEX_HOST="INDEX_HOST" NAMESPACE="YOUR_NAMESPACE" PINECONE_API_KEY="YOUR_API_KEY" curl "https://$INDEX_HOST/records/namespaces/$NAMESPACE/search" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: unstable" \ -d '{ "query": { "inputs": {"text": "Disease prevention"}, "top_k": 4 }, "rerank": { "model": "bge-reranker-v2-m3", "top_n": 2, "rank_fields": ["chunk_text"] }, "fields": ["category", "chunk_text"] }' ``` The response looks as follows. For each hit, the `_score` represents the relevance of a document to the query, normalized between 0 and 1, with scores closer to 1 indicating higher relevance. ```python Python {'result': {'hits': [{'_id': 'rec3', '_score': 0.004399413242936134, 'fields': {'category': 'immune system', 'chunk_text': 'Rich in vitamin C and other ' 'antioxidants, apples ' 'contribute to immune health ' 'and may reduce the risk of ' 'chronic diseases.'}}, {'_id': 'rec4', '_score': 0.0029235430993139744, 'fields': {'category': 'endocrine system', 'chunk_text': 'The high fiber content in ' 'apples can also help regulate ' 'blood sugar levels, making ' 'them a favorable snack for ' 'people with diabetes.'}}]}, 'usage': {'embed_total_tokens': 8, 'read_units': 6, 'rerank_units': 1}} ``` ```javascript JavaScript { result: { hits: [ { _id: 'rec3', _score: 0.004399413242936134, fields: { category: 'immune system', chunk_text: 'Rich in vitamin C and other antioxidants, apples contribute to immune health and may reduce the risk of chronic diseases.' } }, { _id: 'rec4', _score: 0.0029235430993139744, fields: { category: 'endocrine system', chunk_text: 'The high fiber content in apples can also help regulate blood sugar levels, making them a favorable snack for people with diabetes.' } } ] }, usage: { readUnits: 6, embedTotalTokens: 8, rerankUnits: 1 } } ``` ```java Java class SearchRecordsResponse { result: class SearchRecordsResponseResult { hits: [class Hit { id: rec3 score: 0.004399413242936134 fields: {category=immune system, chunk_text=Rich in vitamin C and other antioxidants, apples contribute to immune health and may reduce the risk of chronic diseases.} additionalProperties: null }, class Hit { id: rec4 score: 0.0029235430993139744 fields: {category=endocrine system, chunk_text=The high fiber content in apples can also help regulate blood sugar levels, making them a favorable snack for people with diabetes.} additionalProperties: null }] additionalProperties: null } usage: class SearchUsage { readUnits: 6 embedTotalTokens: 13 rerankUnits: 1 additionalProperties: null } additionalProperties: null } ``` ```go Go { "result": { "hits": [ { "_id": "rec3", "_score": 0.13683891, "fields": { "category": "immune system", "chunk_text": "Rich in vitamin C and other antioxidants, apples contribute to immune health and may reduce the risk of chronic diseases." } }, { "_id": "rec4", "_score": 0.0029235430993139744, "fields": { "category": "endocrine system", "chunk_text": "The high fiber content in apples can also help regulate blood sugar levels, making them a favorable snack for people with diabetes." } } ] }, "usage": { "read_units": 6, "embed_total_tokens": 8, "rerank_units": 1 } } ``` ```csharp C# { "result": { "hits": [ { "_id": "rec3", "_score": 0.004399413242936134, "fields": { "category": "immune system", "chunk_text": "Rich in vitamin C and other antioxidants, apples contribute to immune health and may reduce the risk of chronic diseases." } }, { "_id": "rec4", "_score": 0.0029121784027665854, "fields": { "category": "endocrine system", "chunk_text": "The high fiber content in apples can also help regulate blood sugar levels, making them a favorable snack for people with diabetes." } } ] }, "usage": { "read_units": 6, "embed_total_tokens": 8, "rerank_units": 1 } } ``` ```json curl { "result": { "hits": [ { "_id": "rec3", "_score": 0.004433765076100826, "fields": { "category": "immune system", "chunk_text": "Rich in vitamin C and other antioxidants, apples contribute to immune health and may reduce the risk of chronic diseases." } }, { "_id": "rec4", "_score": 0.0029121784027665854, "fields": { "category": "endocrine system", "chunk_text": "The high fiber content in apples can also help regulate blood sugar levels, making them a favorable snack for people with diabetes." } } ] }, "usage": { "embed_total_tokens": 8, "read_units": 6, "rerank_units": 1 } } ``` ## Standalone reranking To rerank initial results as a standalone operation, use the [`rerank`](/reference/api/2025-01/inference/rerank) operation with the [hosted reranking model](#reranking-models) you want to use, the query results and the query, the number of ranked results to return, the field to use for reranking, and any other model-specific parameters. For example, the following code uses the hosted `bge-reranker-v2-m3` model to rerank the values of the `documents.chunk_text` fields based on their relevance to the query and return only the 2 most relevant documents, along with their score: ```python Python from pinecone import Pinecone pc = Pinecone(api_key="YOUR_API_KEY") ranked_results = pc.inference.rerank( model="bge-reranker-v2-m3", query="What is AAPL's outlook, considering both product launches and market conditions?", documents=[ {"id": "vec2", "chunk_text": "Analysts suggest that AAPL'\''s upcoming Q4 product launch event might solidify its position in the premium smartphone market."}, {"id": "vec3", "chunk_text": "AAPL'\''s strategic Q3 partnerships with semiconductor suppliers could mitigate component risks and stabilize iPhone production."}, {"id": "vec1", "chunk_text": "AAPL reported a year-over-year revenue increase, expecting stronger Q3 demand for its flagship phones."}, ], top_n=2, rank_fields=["chunk_text"], return_documents=True, parameters={ "truncate": "END" } ) print(ranked_results) ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone'; const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }); const rerankingModel = 'bge-reranker-v2-m3'; const query = "What is AAPL's outlook, considering both product launches and market conditions?"; const documents = [ { id: 'vec2', chunk_text: "Analysts suggest that AAPL's upcoming Q4 product launch event might solidify its position in the premium smartphone market." }, { id: 'vec3', chunk_text: "AAPL's strategic Q3 partnerships with semiconductor suppliers could mitigate component risks and stabilize iPhone production." }, { id: 'vec1', chunk_text: "AAPL reported a year-over-year revenue increase, expecting stronger Q3 demand for its flagship phones." }, ]; const rerankOptions = { topN: 2, rankFields: ['chunk_text'], returnDocuments: true, parameters: { truncate: 'END' }, }; const rankedResults = await pc.inference.rerank( rerankingModel, query, documents, rerankOptions ); console.log(rankedResults); ``` ```java Java import io.pinecone.clients.Inference; import io.pinecone.clients.Pinecone; import org.openapitools.inference.client.model.RerankResult; import org.openapitools.inference.client.ApiException; import java.util.*; public class RerankExample { public static void main(String[] args) throws ApiException { Pinecone pc = new Pinecone.Builder("YOUR_API_KEY").build(); Inference inference = pc.getInferenceClient(); // The model to use for reranking String model = "bge-reranker-v2-m3"; // The query to rerank documents against String query = "What is AAPL's outlook, considering both product launches and market conditions?"; // Add the documents to rerank List> documents = new ArrayList<>(); Map doc1 = new HashMap<>(); doc1.put("id", "vec2"); doc1.put("chunk_text", "Analysts suggest that AAPL's upcoming Q4 product launch event might solidify its position in the premium smartphone market."); documents.add(doc1); Map doc2 = new HashMap<>(); doc2.put("id", "vec3"); doc2.put("chunk_text", "AAPL's strategic Q3 partnerships with semiconductor suppliers could mitigate component risks and stabilize iPhone production"); documents.add(doc2); Map doc3 = new HashMap<>(); doc3.put("id", "vec1"); doc3.put("chunk_text", "AAPL reported a year-over-year revenue increase, expecting stronger Q3 demand for its flagship phones."); documents.add(doc3); // The fields to rank the documents by. If not provided, the default is "text" List rankFields = Arrays.asList("chunk_text"); // The number of results to return sorted by relevance. Defaults to the number of inputs int topN = 2; // Whether to return the documents in the response boolean returnDocuments = true; // Additional model-specific parameters for the reranker Map parameters = new HashMap<>(); parameters.put("truncate", "END"); // Send ranking request RerankResult result = inference.rerank(model, query, documents, rankFields, topN, returnDocuments, parameters); // Get ranked data System.out.println(result.getData()); } } ``` ```go Go package main import ( "context" "encoding/json" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func prettifyStruct(obj interface{}) string { bytes, _ := json.MarshalIndent(obj, "", " ") return string(bytes) } func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } rerankModel := "bge-reranker-v2-m3" topN := 2 returnDocuments := true documents := []pinecone.Document{ {"id": "vec2", "chunk_text": "Analysts suggest that AAPL's upcoming Q4 product launch event might solidify its position in the premium smartphone market."}, {"id": "vec3", "chunk_text": "AAPL's strategic Q3 partnerships with semiconductor suppliers could mitigate component risks and stabilize iPhone production."}, {"id": "vec1", "chunk_text": "AAPL reported a year-over-year revenue increase, expecting stronger Q3 demand for its flagship phones."}, } ranking, err := pc.Inference.Rerank(ctx, &pinecone.RerankRequest{ Model: rerankModel, Query: "What is AAPL's outlook, considering both product launches and market conditions?", ReturnDocuments: &returnDocuments, TopN: &topN, RankFields: &[]string{"chunk_text"}, Documents: documents, }) if err != nil { log.Fatalf("Failed to rerank: %v", err) } fmt.Printf(prettifyStruct(ranking)) } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); // Add the documents to rerank var documents = new List> { new() { ["id"] = "vec2", ["chunk_text"] = "Analysts suggest that AAPL's upcoming Q4 product launch event might solidify its position in the premium smartphone market." }, new() { ["id"] = "vec3", ["chunk_text"] = "AAPL's strategic Q3 partnerships with semiconductor suppliers could mitigate component risks and stabilize iPhone production." }, new() { ["id"] = "vec1", ["chunk_text"] = "AAPL reported a year-over-year revenue increase, expecting stronger Q3 demand for its flagship phones." } }; // The fields to rank the documents by. If not provided, the default is "text" var rankFields = new List { "chunk_text" }; // Additional model-specific parameters for the reranker var parameters = new Dictionary { ["truncate"] = "END" }; // Send ranking request var result = await pinecone.Inference.RerankAsync( new RerankRequest { Model = "bge-reranker-v2-m3", Query = "What is AAPL's outlook, considering both product launches and market conditions?", Documents = documents, RankFields = rankFields, TopN = 2, ReturnDocuments = true, Parameters = parameters }); Console.WriteLine(result); ``` ```shell curl PINECONE_API_KEY="YOUR_API_KEY" curl https://api.pinecone.io/rerank \ -H "Content-Type: application/json" \ -H "Accept: application/json" \ -H "X-Pinecone-API-Version: 2025-04" \ -H "Api-Key: $PINECONE_API_KEY" \ -d '{ "model": "bge-reranker-v2-m3", "query": "What is AAPL'\''s outlook, considering both product launches and market conditions?", "documents": [ {"id": "vec2", "chunk_text": "Analysts suggest that AAPL'\''s upcoming Q4 product launch event might solidify its position in the premium smartphone market."}, {"id": "vec3", "chunk_text": "AAPL'\''s strategic Q3 partnerships with semiconductor suppliers could mitigate component risks and stabilize iPhone production."}, {"id": "vec1", "chunk_text": "AAPL reported a year-over-year revenue increase, expecting stronger Q3 demand for its flagship phones."} ], "top_n": 2, "rank_fields": ["chunk_text"], "return_documents": true, "parameters": { "truncate": "END" } }' ``` The response looks as follows. For each hit, the \_score represents the relevance of a document to the query, normalized between 0 and 1, with scores closer to 1 indicating higher relevance. ```python Python RerankResult( model='bge-reranker-v2-m3', data=[{ index=0, score=0.004166256, document={ id='vec2', chunk_text="Analysts suggest that AAPL'''s upcoming Q4 product launch event might solidify its position in the premium smartphone market." } },{ index=2, score=0.0011513996, document={ id='vec1', chunk_text='AAPL reported a year-over-year revenue increase, expecting stronger Q3 demand for its flagship phones.' } }], usage={'rerank_units': 1} ) ``` ```javascript JavaScript { model: 'bge-reranker-v2-m3', data: [ { index: 0, score: 0.004166256, document: [id: 'vec2', chunk_text: "Analysts suggest that AAPL'''s upcoming Q4 product launch event might solidify its position in the premium smartphone market."] }, { index: 2, score: 0.0011513996, document: [id: 'vec1', chunk_text: 'AAPL reported a year-over-year revenue increase, expecting stronger Q3 demand for its flagship phones.'] } ], usage: { rerankUnits: 1 } } ``` ```java Java [class RankedDocument { index: 0 score: 0.0063143647 document: {id=vec2, chunk_text=Analysts suggest that AAPL's upcoming Q4 product launch event might solidify its position in the premium smartphone market.} additionalProperties: null }, class RankedDocument { index: 2 score: 0.0011513996 document: {id=vec1, chunk_text=AAPL reported a year-over-year revenue increase, expecting stronger Q3 demand for its flagship phones.} additionalProperties: null }] ``` ```go Go { "data": [ { "document": { "id": "vec2", "chunk_text": "Analysts suggest that AAPL's upcoming Q4 product launch event might solidify its position in the premium smartphone market." }, "index": 0, "score": 0.0063143647 }, { "document": { "id": "vec1", "chunk_text": "AAPL reported a year-over-year revenue increase, expecting stronger Q3 demand for its flagship phones." }, "index": 2, "score": 0.0011513996 } ], "model": "bge-reranker-v2-m3", "usage": { "rerank_units": 1 } } ``` ```csharp C# { "model": "bge-reranker-v2-m3", "data": [ { "index": 0, "score": 0.006289902, "document": { "chunk_text": "Analysts suggest that AAPL\u0027s upcoming Q4 product launch event might solidify its position in the premium smartphone market.", "id": "vec2" } }, { "index": 3, "score": 0.0011513996, "document": { "chunk_text": "AAPL reported a year-over-year revenue increase, expecting stronger Q3 demand for its flagship phones.", "id": "vec1" } } ], "usage": { "rerank_units": 1 } } ``` ```json curl { "model": "bge-reranker-v2-m3", "data": [ { "index": 0, "document": { "chunk_text": "Analysts suggest that AAPL's upcoming Q4 product launch event might solidify its position in the premium smartphone market.", "id": "vec2" }, "score": 0.007606672 }, { "index": 3, "document": { "chunk_text": "AAPL reported a year-over-year revenue increase, expecting stronger Q3 demand for its flagship phones.", "id": "vec1" }, "score": 0.0013406205 } ], "usage": { "rerank_units": 1 } } ``` {/* ## Rerank results on the default field To [rerank search results](/reference/api/2024-10/inference/rerank), specify a [supported reranking model](/guides/search/rerank-results#reranking-models), and provide documents and a query as well as other model-specific parameters. By default, Pinecone expects the documents to be in the `documents.text` field. For example, the following request uses the `bge-reranker-v2-m3` reranking model to rerank the values of the `documents.text` field based on their relevance to the query, `"The tech company Apple is known for its innovative products like the iPhone."`. With `truncate` set to `"END"`, the input sequence (`query` + `document`) is truncated at the token limit (`1024`); to return an error instead, you'd set `truncate` to `"NONE"` or leave the parameter out. ```python Python from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key="YOUR_API_KEY") result = pc.inference.rerank( model="bge-reranker-v2-m3", query="The tech company Apple is known for its innovative products like the iPhone.", documents=[ {"id": "vec1", "text": "Apple is a popular fruit known for its sweetness and crisp texture."}, {"id": "vec2", "text": "Many people enjoy eating apples as a healthy snack."}, {"id": "vec3", "text": "Apple Inc. has revolutionized the tech industry with its sleek designs and user-friendly interfaces."}, {"id": "vec4", "text": "An apple a day keeps the doctor away, as the saying goes."}, ], top_n=4, return_documents=True, parameters={ "truncate": "END" } ) print(result) ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone'; const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }); const rerankingModel = 'bge-reranker-v2-m3'; const query = 'The tech company Apple is known for its innovative products like the iPhone.'; const documents = [ { id: 'vec1', text: 'Apple is a popular fruit known for its sweetness and crisp texture.' }, { id: 'vec2', text: 'Many people enjoy eating apples as a healthy snack.' }, { id: 'vec3', text: 'Apple Inc. has revolutionized the tech industry with its sleek designs and user-friendly interfaces.' }, { id: 'vec4', text: 'An apple a day keeps the doctor away, as the saying goes.' }, ]; const rerankOptions = { topN: 4, returnDocuments: true, parameters: { truncate: 'END' }, }; const response = await pc.inference.rerank( rerankingModel, query, documents, rerankOptions ); console.log(response); ``` ```java Java import io.pinecone.clients.Inference; import io.pinecone.clients.Pinecone; import org.openapitools.inference.client.model.RerankResult; import org.openapitools.inference.client.ApiException; import java.util.*; public class RerankExample { public static void main(String[] args) throws ApiException { Pinecone pc = new Pinecone.Builder("YOUR_API_KEY").build(); Inference inference = pc.getInferenceClient(); // The model to use for reranking String model = "bge-reranker-v2-m3"; // The query to rerank documents against String query = "The tech company Apple is known for its innovative products like the iPhone."; // Add the documents to rerank List> documents = new ArrayList<>(); Map doc1 = new HashMap<>(); doc1.put("id", "vec1"); doc1.put("text", "Apple is a popular fruit known for its sweetness and crisp texture."); documents.add(doc1); Map doc2 = new HashMap<>(); doc2.put("id", "vec2"); doc2.put("text", "Many people enjoy eating apples as a healthy snack."); documents.add(doc2); Map doc3 = new HashMap<>(); doc3.put("id", "vec3"); doc3.put("text", "Apple Inc. has revolutionized the tech industry with its sleek designs and user-friendly interfaces."); documents.add(doc3); Map doc4 = new HashMap<>(); doc4.put("id", "vec4"); doc4.put("text", "An apple a day keeps the doctor away, as the saying goes."); documents.add(doc4); // The fields to rank the documents by. If not provided, the default is "text" List rankFields = Arrays.asList("text"); // The number of results to return sorted by relevance. Defaults to the number of inputs int topN = 4; // Whether to return the documents in the response boolean returnDocuments = true; // Additional model-specific parameters for the reranker Map parameters = new HashMap<>(); parameters.put("truncate", "END"); // Send ranking request RerankResult result = inference.rerank(model, query, documents, rankFields, topN, returnDocuments, parameters); // Get ranked data System.out.println(result.getData()); } } ``` ```go Go package main import ( "context" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } rerankModel := "bge-reranker-v2-m3" topN := 4 returnDocuments := true documents := []pinecone.Document{ {"id": "vec1", "text": "Apple is a popular fruit known for its sweetness and crisp texture."}, {"id": "vec2", "text": "Many people enjoy eating apples as a healthy snack."}, {"id": "vec3", "text": "Apple Inc. has revolutionized the tech industry with its sleek designs and user-friendly interfaces."}, {"id": "vec4", "text": "An apple a day keeps the doctor away, as the saying goes."}, } ranking, err := pc.Inference.Rerank(ctx, &pinecone.RerankRequest{ Model: rerankModel, Query: "The tech company Apple is known for its innovative products like the iPhone.", ReturnDocuments: &returnDocuments, TopN: &topN, RankFields: &[]string{"text"}, Documents: documents, }) if err != nil { log.Fatalf("Failed to rerank: %v", err) } fmt.Printf("Rerank result: %+v\n", ranking) } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); // The model to use for reranking var model = "bge-reranker-v2-m3"; // The query to rerank documents against var query = "The tech company Apple is known for its innovative products like the iPhone."; // Add the documents to rerank var documents = new List> { new() { ["id"] = "vec1", ["my_field"] = "Apple is a popular fruit known for its sweetness and crisp texture." }, new() { ["id"] = "vec2", ["my_field"] = "Many people enjoy eating apples as a healthy snack." }, new() { ["id"] = "vec3", ["my_field"] = "Apple Inc. has revolutionized the tech industry with its sleek designs and user-friendly interfaces." }, new() { ["id"] = "vec4", ["my_field"] = "An apple a day keeps the doctor away, as the saying goes." } }; // The fields to rank the documents by. If not provided, the default is "text" var rankFields = new List { "my_field" }; // The number of results to return sorted by relevance. Defaults to the number of inputs int topN = 4; // Whether to return the documents in the response bool returnDocuments = true; // Additional model-specific parameters for the reranker var parameters = new Dictionary { ["truncate"] = "END" }; // Send ranking request var result = await pinecone.Inference.RerankAsync( new RerankRequest { Model = model, Query = query, Documents = documents, RankFields = rankFields, TopN = topN, ReturnDocuments = true, Parameters = parameters }); Console.WriteLine(result); ``` ```shell curl PINECONE_API_KEY="YOUR_API_KEY" curl https://api.pinecone.io/rerank \ -H "Content-Type: application/json" \ -H "Accept: application/json" \ -H "X-Pinecone-API-Version: 2025-04" \ -H "Api-Key: $PINECONE_API_KEY" \ -d '{ "model": "bge-reranker-v2-m3", "query": "The tech company Apple is known for its innovative products like the iPhone.", "return_documents": true, "top_n": 4, "documents": [ {"id": "vec1", "text": "Apple is a popular fruit known for its sweetness and crisp texture."}, {"id": "vec2", "text": "Many people enjoy eating apples as a healthy snack."}, {"id": "vec3", "text": "Apple Inc. has revolutionized the tech industry with its sleek designs and user-friendly interfaces."}, {"id": "vec4", "text": "An apple a day keeps the doctor away, as the saying goes."} ], "parameters": { "truncate": "END" } }' ``` The returned object contains documents with relevance scores: Normalized between 0 and 1, the `score` represents the relevance of a passage to the query, with scores closer to 1 indicating higher relevance. ```python Python RerankResult( model='bge-reranker-v2-m3', data=[ { index=2, score=0.48357219, document={id="vec3", text="Apple Inc. has re..."} }, { index=0, score=0.048405956, document={id="vec1", text="Apple is a popula..."} }, { index=3, score=0.007846239, document={id="vec4", text="An apple a day ke..."} }, { index=1, score=0.0006563728, document={id="vec2", text="Many people enjoy..."} } ], usage={'rerank_units': 1} ) ``` ```javascript JavaScript { model: 'bge-reranker-v2-m3', data: [ { index: 2, score: 0.48357219, document: [Object] }, { index: 0, score: 0.048405956, document: [Object] }, { index: 3, score: 0.007846239, document: [Object] }, { index: 1, score: 0.0006563728, document: [Object] } ], usage: { rerankUnits: 1 } } ``` ```java Java [class RankedDocument { index: 2 score: 0.48357219 document: {id=vec3, text=Apple Inc. has revolutionized the tech industry with its sleek designs and user-friendly interfaces.} additionalProperties: null }, class RankedDocument { index: 0 score: 0.048405956 document: {id=vec1, text=Apple is a popular fruit known for its sweetness and crisp texture.} additionalProperties: null }, class RankedDocument { index: 3 score: 0.007846239 document: {id=vec4, text=An apple a day keeps the doctor away, as the saying goes.} additionalProperties: null }, class RankedDocument { index: 1 score: 0.0006563728 document: {id=vec2, text=Many people enjoy eating apples as a healthy snack.} additionalProperties: null }] ``` ```go Go Rerank result: { "data": [ { "document": { "id": "vec3", "text": "Apple Inc. has revolutionized the tech industry with its sleek designs and user-friendly interfaces." }, "index": 2, "score": 0.48357219 }, { "document": { "id": "vec1", "text": "Apple is a popular fruit known for its sweetness and crisp texture." }, "index": 0, "score": 0.048405956 }, { "document": { "id": "vec4", "text": "An apple a day keeps the doctor away, as the saying goes." }, "index": 3, "score": 0.007846239 }, { "document": { "id": "vec2", "text": "Many people enjoy eating apples as a healthy snack." }, "index": 1, "score": 0.0006563728 } ], "model": "bge-reranker-v2-m3", "usage": { "rerank_units": 1 } } ``` ```csharp C# { "model": "bge-reranker-v2-m3", "data": [ { "index": 2, "score": 0.48357219, "document": { "id": "vec3", "my_field": "Apple Inc. has revolutionized the tech industry with its sleek designs and user-friendly interfaces." } }, { "index": 0, "score": 0.048405956, "document": { "id": "vec1", "my_field": "Apple is a popular fruit known for its sweetness and crisp texture." } }, { "index": 3, "score": 0.007846239, "document": { "id": "vec4", "my_field": "An apple a day keeps the doctor away, as the saying goes." } }, { "index": 1, "score": 0.0006563728, "document": { "id": "vec2", "my_field": "Many people enjoy eating apples as a healthy snack." } } ], "usage": { "rerank_units": 1 } } ``` ```JSON curl { "data":[ { "index":2, "document":{ "id":"vec3", "text":"Apple Inc. has revolutionized the tech industry with its sleek designs and user-friendly interfaces." }, "score":0.47654688 }, { "index":0, "document":{ "id":"vec1", "text":"Apple is a popular fruit known for its sweetness and crisp texture." }, "score":0.047963805 }, { "index":3, "document":{ "id":"vec4", "text":"An apple a day keeps the doctor away, as the saying goes." }, "score":0.007587992 }, { "index":1, "document":{ "id":"vec2", "text":"Many people enjoy eating apples as a healthy snack." }, "score":0.0006491712 } ], "usage":{ "rerank_units":1 } } ``` ## Rerank results on a custom field To [rerank results](/reference/api/2024-10/inference/rerank) on a field other than `documents.text`, provide the `rank_fields` parameter to specify the fields on which to rerank. The [`bge-reranker-v2-m3`](#bge-reranker-v2-m3) and [`pinecone-rerank-v0`](#pinecone-rerank-v0) models support only a single rerank field. [`cohere-rerank-3.5`](#cohere-rerank-3-5) supports multiple rerank fields, ranked based on the order of the fields specified. For example, the following request reranks documents based on the values of the `documents.my_field` field: ```python Python from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key="YOUR_API_KEY") result = pc.inference.rerank( model="bge-reranker-v2-m3", query="The tech company Apple is known for its innovative products like the iPhone.", documents=[ {"id": "vec1", "my_field": "Apple is a popular fruit known for its sweetness and crisp texture."}, {"id": "vec2", "my_field": "Many people enjoy eating apples as a healthy snack."}, {"id": "vec3", "my_field": "Apple Inc. has revolutionized the tech industry with its sleek designs and user-friendly interfaces."}, {"id": "vec4", "my_field": "An apple a day keeps the doctor away, as the saying goes."}, ], rank_fields=["my_field"], top_n=4, return_documents=True, parameters={ "truncate": "END" } ) ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone'; const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }); const rerankingModel = 'bge-reranker-v2-m3'; const query = 'The tech company Apple is known for its innovative products like the iPhone.'; const documents = [ { id: 'vec1', my_field: 'Apple is a popular fruit known for its sweetness and crisp texture.' }, { id: 'vec2', my_field: 'Many people enjoy eating apples as a healthy snack.' }, { id: 'vec3', my_field: 'Apple Inc. has revolutionized the tech industry with its sleek designs and user-friendly interfaces.' }, { id: 'vec4', my_field: 'An apple a day keeps the doctor away, as the saying goes.' }, ]; const rerankOptions = { rankFields: ['my_field'], topN: 4, returnDocuments: true, parameters: { truncate: "END" }, }; const response = await pc.inference.rerank( rerankingModel, query, documents, rerankOptions ); console.log(response); ``` ```java Java import io.pinecone.clients.Inference; import io.pinecone.clients.Pinecone; import org.openapitools.inference.client.model.RerankResult; import org.openapitools.inference.client.ApiException; import java.util.*; public class RerankExample { public static void main(String[] args) throws ApiException { Pinecone pc = new Pinecone.Builder("YOUR_API_KEY").build(); Inference inference = pc.getInferenceClient(); // The model to use for reranking String model = "bge-reranker-v2-m3"; // The query to rerank documents against String query = "The tech company Apple is known for its innovative products like the iPhone."; // Add the documents to rerank List> documents = new ArrayList<>(); Map doc1 = new HashMap<>(); doc1.put("id", "vec1"); doc1.put("my_field", "Apple is a popular fruit known for its sweetness and crisp texture."); documents.add(doc1); Map doc2 = new HashMap<>(); doc2.put("id", "vec2"); doc2.put("my_field", "Many people enjoy eating apples as a healthy snack."); documents.add(doc2); Map doc3 = new HashMap<>(); doc3.put("id", "vec3"); doc3.put("my_field", "Apple Inc. has revolutionized the tech industry with its sleek designs and user-friendly interfaces."); documents.add(doc3); Map doc4 = new HashMap<>(); doc4.put("id", "vec4"); doc4.put("my_field", "An apple a day keeps the doctor away, as the saying goes."); documents.add(doc4); // The fields to rank the documents by. If not provided, the default is "text" List rankFields = Arrays.asList("my_field"); // The number of results to return sorted by relevance. Defaults to the number of inputs int topN = 2; // Whether to return the documents in the response boolean returnDocuments = true; // Additional model-specific parameters for the reranker Map parameters = new HashMap<>(); parameters.put("truncate", "END"); // Send ranking request RerankResult result = inference.rerank(model, query, documents, rankFields, topN, returnDocuments, parameters); // Get ranked data System.out.println(result.getData()); } } ``` ```go Go package main import ( "context" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } rerankModel := "bge-reranker-v2-m3" topN := 4 returnDocuments := true documents := []pinecone.Document{ {"id": "vec1", "my_field": "Apple is a popular fruit known for its sweetness and crisp texture."}, {"id": "vec2", "my_field": "Many people enjoy eating apples as a healthy snack."}, {"id": "vec3", "my_field": "Apple Inc. has revolutionized the tech industry with its sleek designs and user-friendly interfaces."}, {"id": "vec4", "my_field": "An apple a day keeps the doctor away, as the saying goes."}, } ranking, err := pc.Inference.Rerank(ctx, &pinecone.RerankRequest{ Model: rerankModel, Query: "The tech company Apple is known for its innovative products like the iPhone.", ReturnDocuments: &returnDocuments, TopN: &topN, RankFields: &[]string{"my_field"}, Documents: documents, }) if err != nil { log.Fatalf("Failed to rerank: %v", err) } fmt.Printf("Rerank result: %+v\n", ranking) } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); // The model to use for reranking var model = "bge-reranker-v2-m3"; // The query to rerank documents against var query = "The tech company Apple is known for its innovative products like the iPhone."; // Add the documents to rerank var documents = new List> { new() { ["id"] = "vec1", ["my_field"] = "Apple is a popular fruit known for its sweetness and crisp texture." }, new() { ["id"] = "vec2", ["my_field"] = "Many people enjoy eating apples as a healthy snack." }, new() { ["id"] = "vec3", ["my_field"] = "Apple Inc. has revolutionized the tech industry with its sleek designs and user-friendly interfaces." }, new() { ["id"] = "vec4", ["my_field"] = "An apple a day keeps the doctor away, as the saying goes." } }; // The fields to rank the documents by. If not provided, the default is "text" var rankFields = new List { "my_field" }; // The number of results to return sorted by relevance. Defaults to the number of inputs int topN = 2; // Whether to return the documents in the response bool returnDocuments = true; // Additional model-specific parameters for the reranker var parameters = new Dictionary { ["truncate"] = "END" }; // Send ranking request var result = await pinecone.Inference.RerankAsync( new RerankRequest { Model = model, Query = query, Documents = documents, RankFields = rankFields, TopN = topN, ReturnDocuments = true, Parameters = parameters }); // Get ranked data var data = result.Data; Console.WriteLine(data); ``` ```shell curl PINECONE_API_KEY="YOUR_API_KEY" curl "https://api.pinecone.io/rerank" \ -H "Content-Type: application/json" \ -H "Accept: application/json" \ -H "X-Pinecone-API-Version: 2025-04" \ -H "Api-Key: $PINECONE_API_KEY" \ -d '{ "model": "bge-reranker-v2-m3", "query": "The tech company Apple is known for its innovative products like the iPhone.", "return_documents": true, "top_n": 4, "rank_fields": ["my_field"], "documents": [ {"id": "vec1", "my_field": "Apple is a popular fruit known for its sweetness and crisp texture."}, {"id": "vec2", "my_field": "Many people enjoy eating apples as a healthy snack."}, {"id": "vec3", "my_field": "Apple Inc. has revolutionized the tech industry with its sleek designs and user-friendly interfaces."}, {"id": "vec4", "my_field": "An apple a day keeps the doctor away, as the saying goes."} ], "parameters": { "truncate": "END" } }' ``` */} ## Reranking models Pinecone hosts several reranking models so it's easy to manage two-stage vector retrieval on a single platform. You can use a hosted model to rerank results as an integrated part of a query, or you can use a hosted model to rerank results as a standalone operation. The following reranking models are hosted by Pinecone. To understand how cost is calculated for reranking, see [Understanding cost](/guides/manage-cost/understanding-cost#rerank). To get model details via the API, see [List models](/reference/api/2025-04/inference/list_models) and [Describe a model](/reference/api/2025-04/inference/describe_model). [`cohere-rerank-3.5`](/models/cohere-rerank-3.5) is Cohere's leading reranking model, balancing performance and latency for a wide range of enterprise search applications. **Details** * Modality: Text * Max tokens per query and document pair: 40,000 * Max documents: 200 **Parameters** The `cohere-rerank-3.5` model supports the following parameters: | Parameter | Type | Required/Optional | Description | | | :------------------- | :--------------- | :---------------- | :-------------------------------------------------------------------------------------------------------------------------------------- | ---------- | | `max_chunks_per_doc` | integer | Optional | Long documents will be automatically truncated to the specified number of chunks. Accepted range: `1 - 3072`. | | | `rank_fields` | array of strings | Optional | The fields to use for reranking. The model reranks based on the order of the fields specified (e.g., `["field1", "field2", "field3"]`). | `["text"]` | **Quotas** Quotas are defined at the project level and vary based on [pricing plan](https://www.pinecone.io/pricing/). To request a rate increase, [contact Support](https://app.pinecone.io/organizations/-/settings/support/ticket). | Limit type | Starter plan | Paid plans | | :------------------ | :----------- | :--------- | | Requests per minute | N/A | 300 | | Requests per month | N/A | Unlimited | [`bge-reranker-v2-m3`](/models/bge-reranker-v2-m3) is a high-performance, multilingual reranking model that works well on messy data and short queries expected to return medium-length passages of text (1-2 paragraphs). **Details** * Modality: Text * Max tokens per query and document pair: 1024 * Max documents: 100 **Parameters** The `bge-reranker-v2-m3` model supports the following parameters: | Parameter | Type | Required/Optional | Description | Default | | :------------ | :--------------- | :---------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :--------- | | `truncate` | string | Optional | How to handle inputs longer than those supported by the model. Accepted values: `END` or `NONE`.

`END` truncates the input sequence at the input token limit. `NONE` returns an error when the input exceeds the input token limit. | `NONE` | | `rank_fields` | array of strings | Optional | The field to use for reranking. The model supports only a single rerank field. | `["text"]` | **Quotas** Quotas are defined at the project level and vary based on [pricing plan](https://www.pinecone.io/pricing/). To request a rate increase, [contact Support](https://app.pinecone.io/organizations/-/settings/support/ticket). | Limit type | Starter plan | Paid plans | | :------------------ | :----------- | :--------- | | Requests per minute | 60 | 60 | | Requests per month | 500 | Unlimited | [`pinecone-rerank-v0`](/models/pinecone-rerank-v0) is a state of the art reranking model that out-performs competitors on widely accepted benchmarks. It can handle chunks up to 512 tokens (1-2 paragraphs). **Details** * Modality: Text * Max tokens per query and document pair: 512 * Max documents: 100 **Parameters** The `pinecone-rerank-v0` model supports the following parameters: | Parameter | Type | Required/Optional | Description | Default | | :------------ | :--------------- | :---------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :--------- | | `truncate` | string | Optional | How to handle inputs longer than those supported by the model. Accepted values: `END` or `NONE`.

`END` truncates the input sequence at the input token limit. `NONE` returns an error when the input exceeds the input token limit. | `END` | | `rank_fields` | array of strings | Optional | The field to use for reranking. The model supports only a single rerank field. | `["text"]` | **Quotas** Quotas are defined at the project level and vary based on [pricing plan](https://www.pinecone.io/pricing/). To request a rate increase, [contact Support](https://app.pinecone.io/organizations/-/settings/support/ticket). | Limit type | Starter plan | Paid plans | | :------------------ | :----------- | :--------- | | Requests per minute | 60 | 60 | | Requests per month | 500 | Unlimited | # Search overview Source: https://docs.pinecone.io/guides/search/search-overview This section of the documentation shows you the different ways to search your data in Pinecone. ## Search types * [Semantic search](/guides/search/semantic-search) * [Lexical search](/guides/search/lexical-search) * [Hybrid search](/guides/search/hybrid-search) ## Optimization * [Filter by metadata](/guides/search/filter-by-metadata) * [Rerank results](/guides/search/rerank-results) * [Parallel queries](/guides/search/semantic-search#parallel-queries) ## Limits | Metric | Limit | | :---------------- | :----- | | Max `top_k` value | 10,000 | | Max result size | 4MB | The query result size is affected by the dimension of the dense vectors and whether or not dense vector values and metadata are included in the result. If a query fails due to exceeding the 4MB result size limit, choose a lower `top_k` value, or use `include_metadata=False` or `include_values=False` to exclude metadata or values from the result. ## Cost * To understand how cost is calculated for queries, see [Understanding cost](/guides/manage-cost/understanding-cost#query). * For up-to-date pricing information, see [Pricing](https://www.pinecone.io/pricing/). ## Data freshness Pinecone is eventually consistent, so there can be a slight delay before new or changed records are visible to queries. You can view index stats to [check data freshness](/guides/index-data/check-data-freshness). # Semantic search Source: https://docs.pinecone.io/guides/search/semantic-search This page shows you how to search a [dense index](/guides/index-data/indexing-overview#dense-indexes) for records that are most similar in meaning and context to a query. This is often called semantic search, nearest neighbor search, similarity search, or just vector search. Semantic search uses [dense vectors](https://www.pinecone.io/learn/vector-embeddings/). Each number in a dense vector corresponds to a point in a multidimensional space. Vectors that are closer together in that space are semantically similar. ## Search with text Searching with text is supported only for [indexes with integrated embedding](/guides/index-data/indexing-overview#integrated-embedding). To search a dense index with a query text, use the [`search_records`](/reference/api/2025-01/data-plane/search_records) operation with the following parameters: * The `namespace` to query. To use the default namespace, set the namespace to `"__default__"`. * The `query.inputs.text` parameter with the query text. Pinecone uses the embedding model integrated with the index to convert the text to a dense vector automatically. * The `query.top_k` parameter with the number of similar records to return. * Optionally, you can specify the `fields` to return in the response. If not specified, the response will include all fields. For example, the following code searches for the 2 records most semantically related to a query text: ```python Python from pinecone import Pinecone pc = Pinecone(api_key="YOUR_API_KEY") # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index index = pc.Index(host="INDEX_HOST") results = index.search( namespace="example-namespace", query={ "inputs": {"text": "Disease prevention"}, "top_k": 2 }, fields=["category", "chunk_text"] ) print(results) ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: "YOUR_API_KEY" }) // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index const namespace = pc.index("INDEX_NAME", "INDEX_HOST").namespace("example-namespace"); const response = await namespace.searchRecords({ query: { topK: 2, inputs: { text: 'Disease prevention' }, }, fields: ['chunk_text', 'category'], }); console.log(response); ``` ```java Java import io.pinecone.clients.Index; import io.pinecone.configs.PineconeConfig; import io.pinecone.configs.PineconeConnection; import org.openapitools.db_data.client.ApiException; import org.openapitools.db_data.client.model.SearchRecordsResponse; import java.util.*; public class SearchText { public static void main(String[] args) throws ApiException { PineconeConfig config = new PineconeConfig("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index config.setHost("INDEX_HOST"); PineconeConnection connection = new PineconeConnection(config); Index index = new Index(config, connection, "integrated-dense-java"); String query = "Disease prevention"; List fields = new ArrayList<>(); fields.add("category"); fields.add("chunk_text"); // Search the dense index SearchRecordsResponse recordsResponse = index.searchRecordsByText(query, "example-namespace", fields, 2, null, null); // Print the results System.out.println(recordsResponse); } } ``` ```go Go package main import ( "context" "encoding/json" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func prettifyStruct(obj interface{}) string { bytes, _ := json.MarshalIndent(obj, "", " ") return string(bytes) } func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "INDEX_HOST", Namespace: "example-namespace"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v", err) } res, err := idxConnection.SearchRecords(ctx, &pinecone.SearchRecordsRequest{ Query: pinecone.SearchRecordsQuery{ TopK: 2, Inputs: &map[string]interface{}{ "text": "Disease prevention", }, }, Fields: &[]string{"chunk_text", "category"}, }) if err != nil { log.Fatalf("Failed to search records: %v", err) } fmt.Printf(prettifyStruct(res)) } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index var index = pinecone.Index(host: "INDEX_HOST"); var response = await index.SearchRecordsAsync( "example-namespace", new SearchRecordsRequest { Query = new SearchRecordsRequestQuery { TopK = 4, Inputs = new Dictionary { { "text", "Disease prevention" } }, }, Fields = ["category", "chunk_text"], } ); Console.WriteLine(response); ``` ```shell curl INDEX_HOST="INDEX_HOST" NAMESPACE="YOUR_NAMESPACE" PINECONE_API_KEY="YOUR_API_KEY" curl "https://$INDEX_HOST/records/namespaces/$NAMESPACE/search" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: unstable" \ -d '{ "query": { "inputs": {"text": "Disease prevention"}, "top_k": 2 }, "fields": ["category", "chunk_text"] }' ``` The response will look as follows. Each record is returned with a similarity score that represents its distance to the query vector, calculated according to the [similarity metric](/guides/index-data/create-an-index#similarity-metrics) for the index. ```python Python {'result': {'hits': [{'_id': 'rec3', '_score': 0.8204272389411926, 'fields': {'category': 'immune system', 'chunk_text': 'Rich in vitamin C and other ' 'antioxidants, apples ' 'contribute to immune health ' 'and may reduce the risk of ' 'chronic diseases.'}}, {'_id': 'rec1', '_score': 0.7931625843048096, 'fields': {'category': 'digestive system', 'chunk_text': 'Apples are a great source of ' 'dietary fiber, which supports ' 'digestion and helps maintain a ' 'healthy gut.'}}]}, 'usage': {'embed_total_tokens': 8, 'read_units': 6}} ``` ```javascript JavaScript { result: { hits: [ { _id: 'rec3', _score: 0.82042724, fields: { category: 'immune system', chunk_text: 'Rich in vitamin C and other antioxidants, apples contribute to immune health and may reduce the risk of chronic diseases.' } }, { _id: 'rec1', _score: 0.7931626, fields: { category: 'digestive system', chunk_text: 'Apples are a great source of dietary fiber, which supports digestion and helps maintain a healthy gut.' } } ] }, usage: { readUnits: 6, embedTotalTokens: 8 } } ``` ```java Java class SearchRecordsResponse { result: class SearchRecordsResponseResult { hits: [class Hit { id: rec3 score: 0.8204272389411926 fields: {category=immune system, chunk_text=Rich in vitamin C and other antioxidants, apples contribute to immune health and may reduce the risk of chronic diseases.} additionalProperties: null }, class Hit { id: rec1 score: 0.7931625843048096 fields: {category=endocrine system, chunk_text=Apples are a great source of dietary fiber, which supports digestion and helps maintain a healthy gut.} additionalProperties: null }] additionalProperties: null } usage: class SearchUsage { readUnits: 6 embedTotalTokens: 13 } additionalProperties: null } ``` ```go Go { "result": { "hits": [ { "_id": "rec3", "_score": 0.82042724, "fields": { "category": "immune system", "chunk_text": "Rich in vitamin C and other antioxidants, apples contribute to immune health and may reduce the risk of chronic diseases." } }, { "_id": "rec1", "_score": 0.7931626, "fields": { "category": "digestive system", "chunk_text": "Apples are a great source of dietary fiber, which supports digestion and helps maintain a healthy gut." } } ] }, "usage": { "read_units": 6, "embed_total_tokens": 8 } } ``` ```csharp C# { "result": { "hits": [ { "_id": "rec3", "_score": 0.13741668, "fields": { "category": "immune system", "chunk_text": "Rich in vitamin C and other antioxidants, apples contribute to immune health and may reduce the risk of chronic diseases." } }, { "_id": "rec1", "_score": 0.0023413408, "fields": { "category": "digestive system", "chunk_text": "Apples are a great source of dietary fiber, which supports digestion and helps maintain a healthy gut." } } ] }, "usage": { "read_units": 6, "embed_total_tokens": 5, "rerank_units": 1 } } ``` ```json curl { "result": { "hits": [ { "_id": "rec3", "_score": 0.82042724, "fields": { "category": "immune system", "chunk_text": "Rich in vitamin C and other antioxidants, apples contribute to immune health and may reduce the risk of chronic diseases." } }, { "_id": "rec1", "_score": 0.7931626, "fields": { "category": "digestive system", "chunk_text": "Apples are a great source of dietary fiber, which supports digestion and helps maintain a healthy gut." } } ] }, "usage": { "embed_total_tokens": 8, "read_units": 6 } } ``` ## Search with a dense vector To search a dense index with a dense vector representation of a query, use the [`query`](/reference/api/2025-01/data-plane/query) operation with the following parameters: * The `namespace` to query. To use the default namespace, set the namespace to `"__default__"`. * The `vector` parameter with the dense vector values representing your query. * The `top_k` parameter with the number of results to return. * Optionally, you can set `include_values` and/or `include_metadata` to `true` to include the vector values and/or metadata of the matching records in the response. However, when querying with `top_k` over 1000, avoid returning vector data or metadata for optimal performance. For example, the following code uses a dense vector representation of the query “Disease prevention” to search for the 3 most semantically similar records in the `example-namespaces` namespace: ```Python Python from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key="YOUR_API_KEY") # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index index = pc.Index(host="INDEX_HOST") index.query( namespace="example-namespace", vector=[0.0236663818359375,-0.032989501953125, ..., -0.01041412353515625,0.0086669921875], top_k=3, include_metadata=True, include_values=False ) ``` ```JavaScript JavaScript import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: "YOUR_API_KEY" }) // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index const index = pc.index("INDEX_NAME", "INDEX_HOST") const queryResponse = await index.namespace('example-namespace').query({ vector: [0.0236663818359375,-0.032989501953125,...,-0.01041412353515625,0.0086669921875], topK: 3, includeValues: false, includeMetadata: true, }); ``` ```java Java import io.pinecone.clients.Index; import io.pinecone.configs.PineconeConfig; import io.pinecone.configs.PineconeConnection; import io.pinecone.unsigned_indices_model.QueryResponseWithUnsignedIndices; import java.util.Arrays; import java.util.List; public class QueryExample { public static void main(String[] args) { PineconeConfig config = new PineconeConfig("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index config.setHost("INDEX_HOST"); PineconeConnection connection = new PineconeConnection(config); Index index = new Index(connection, "INDEX_NAME"); List query = Arrays.asList(0.0236663818359375f, -0.032989501953125f, ..., -0.01041412353515625f, 0.0086669921875f); QueryResponseWithUnsignedIndices queryResponse = index.query(3, query, null, null, null, "example-namespace", null, false, true); System.out.println(queryResponse); } } ``` ```go Go package main import ( "context" "encoding/json" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func prettifyStruct(obj interface{}) string { bytes, _ := json.MarshalIndent(obj, "", " ") return string(bytes) } func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "INDEX_HOST", Namespace: "example-namespace"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v", err) } queryVector := []float32{0.0236663818359375,-0.032989501953125,...,-0.01041412353515625,0.0086669921875} res, err := idxConnection.QueryByVectorValues(ctx, &pinecone.QueryByVectorValuesRequest{ Vector: queryVector, TopK: 3, IncludeValues: false, includeMetadata: true, }) if err != nil { log.Fatalf("Error encountered when querying by vector: %v", err) } else { fmt.Printf(prettifyStruct(res)) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index var index = pinecone.Index(host: "INDEX_HOST"); var queryResponse = await index.QueryAsync(new QueryRequest { Vector = new[] { 0.0236663818359375f ,-0.032989501953125f, ..., -0.01041412353515625f, 0.0086669921875f }, Namespace = "example-namespace", TopK = 3, IncludeMetadata = true, }); Console.WriteLine(queryResponse); ``` ```bash curl # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index PINECONE_API_KEY="YOUR_API_KEY" INDEX_HOST="INDEX_HOST" curl "https://$INDEX_HOST/query" \ -H "Api-Key: $PINECONE_API_KEY" \ -H 'Content-Type: application/json' \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "vector": [0.0236663818359375,-0.032989501953125,...,-0.01041412353515625,0.0086669921875], "namespace": "example-namespace", "topK": 3, "includeMetadata": true, "includeValues": false }' ``` The response will look as follows. Each record is returned with a similarity score that represents its distance to the query vector, calculated according to the [similarity metric](/guides/index-data/create-an-index#similarity-metrics) for the index. ```python Python {'matches': [{'id': 'rec3', 'metadata': {'category': 'immune system', 'chunk_text': 'Rich in vitamin C and other ' 'antioxidants, apples contribute to ' 'immune health and may reduce the ' 'risk of chronic diseases.'}, 'score': 0.82026422, 'values': []}, {'id': 'rec1', 'metadata': {'category': 'digestive system', 'chunk_text': 'Apples are a great source of ' 'dietary fiber, which supports ' 'digestion and helps maintain a ' 'healthy gut.'}, 'score': 0.793068111, 'values': []}, {'id': 'rec4', 'metadata': {'category': 'endocrine system', 'chunk_text': 'The high fiber content in apples ' 'can also help regulate blood sugar ' 'levels, making them a favorable ' 'snack for people with diabetes.'}, 'score': 0.780169606, 'values': []}], 'namespace': 'example-namespace', 'usage': {'read_units': 6}} ``` ```JavaScript JavaScript { matches: [ { id: 'rec3', score: 0.819709897, values: [], sparseValues: undefined, metadata: [Object] }, { id: 'rec1', score: 0.792900264, values: [], sparseValues: undefined, metadata: [Object] }, { id: 'rec4', score: 0.780068815, values: [], sparseValues: undefined, metadata: [Object] } ], namespace: 'example-namespace', usage: { readUnits: 6 } } ``` ```java Java class QueryResponseWithUnsignedIndices { matches: [ScoredVectorWithUnsignedIndices { score: 0.8197099 id: rec3 values: [] metadata: fields { key: "category" value { string_value: "immune system" } } fields { key: "chunk_text" value { string_value: "Rich in vitamin C and other antioxidants, apples contribute to immune health and may reduce the risk of chronic diseases." } } sparseValuesWithUnsignedIndices: SparseValuesWithUnsignedIndices { indicesWithUnsigned32Int: [] values: [] } }, ScoredVectorWithUnsignedIndices { score: 0.79290026 id: rec1 values: [] metadata: fields { key: "category" value { string_value: "digestive system" } } fields { key: "chunk_text" value { string_value: "Apples are a great source of dietary fiber, which supports digestion and helps maintain a healthy gut." } } sparseValuesWithUnsignedIndices: SparseValuesWithUnsignedIndices { indicesWithUnsigned32Int: [] values: [] } }, ScoredVectorWithUnsignedIndices { score: 0.7800688 id: rec4 values: [] metadata: fields { key: "category" value { string_value: "endocrine system" } } fields { key: "chunk_text" value { string_value: "The high fiber content in apples can also help regulate blood sugar levels, making them a favorable snack for people with diabetes." } } sparseValuesWithUnsignedIndices: SparseValuesWithUnsignedIndices { indicesWithUnsigned32Int: [] values: [] } }] namespace: example-namespace usage: read_units: 6 } ``` ```go Go { "matches": [ { "vector": { "id": "rec3", "metadata": { "category": "immune system", "chunk_text": "Rich in vitamin C and other antioxidants, apples contribute to immune health and may reduce the risk of chronic diseases." } }, "score": 0.8197099 }, { "vector": { "id": "rec1", "metadata": { "category": "digestive system", "chunk_text": "Apples are a great source of dietary fiber, which supports digestion and helps maintain a healthy gut." } }, "score": 0.79290026 }, { "vector": { "id": "rec4", "metadata": { "category": "endocrine system", "chunk_text": "The high fiber content in apples can also help regulate blood sugar levels, making them a favorable snack for people with diabetes." } }, "score": 0.7800688 } ], "usage": { "read_units": 6 }, "namespace": "example-namespace" } ``` ```csharp C# { "results": [], "matches": [ { "id": "rec3", "score": 0.8197099, "values": [], "metadata": { "category": "immune system", "chunk_text": "Rich in vitamin C and other antioxidants, apples contribute to immune health and may reduce the risk of chronic diseases." } }, { "id": "rec1", "score": 0.79290026, "values": [], "metadata": { "category": "digestive system", "chunk_text": "Apples are a great source of dietary fiber, which supports digestion and helps maintain a healthy gut." } }, { "id": "rec4", "score": 0.7800688, "values": [], "metadata": { "category": "endocrine system", "chunk_text": "The high fiber content in apples can also help regulate blood sugar levels, making them a favorable snack for people with diabetes." } } ], "namespace": "example-namespace", "usage": { "readUnits": 6 } } ``` ```json curl { "results": [], "matches": [ { "id": "rec3", "score": 0.820593238, "values": [], "metadata": { "category": "immune system", "chunk_text": "Rich in vitamin C and other antioxidants, apples contribute to immune health and may reduce the risk of chronic diseases." } }, { "id": "rec1", "score": 0.792266726, "values": [], "metadata": { "category": "digestive system", "chunk_text": "Apples are a great source of dietary fiber, which supports digestion and helps maintain a healthy gut." } }, { "id": "rec4", "score": 0.780045748, "values": [], "metadata": { "category": "endocrine system", "chunk_text": "The high fiber content in apples can also help regulate blood sugar levels, making them a favorable snack for people with diabetes." } } ], "namespace": "example-namespace", "usage": { "readUnits": 6 } } ``` ## Search with a record ID When you search with a record ID, Pinecone uses the dense vector associated with the record as the query. To search a dense index with a record ID, use the [`query`](/reference/api/2025-01/data-plane/query) operation with the following parameters: * The `namespace` to query. To use the default namespace, set the namespace to `"__default__"`. * The `id` parameter with the unique record ID containing the vector to use as the query. * The `top_k` parameter with the number of results to return. * Optionally, you can set `include_values` and/or `include_metadata` to `true` to include the vector values and/or metadata of the matching records in the response. However, when querying with `top_k` over 1000, avoid returning vector data or metadata for optimal performance. For example, the following code uses an ID to search for the 3 records in the `example-namespace` namespace that are most semantically similar to the dense vector in the record: ```Python Python from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key="YOUR_API_KEY") # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index index = pc.Index(host="INDEX_HOST") index.query( namespace="example-namespace", id="rec2", top_k=3, include_metadata=True, include_values=False ) ``` ```JavaScript JavaScript import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: "YOUR_API_KEY" }) // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index const index = pc.index("INDEX_NAME", "INDEX_HOST") const queryResponse = await index.namespace('example-namespace').query({ id: 'rec2', topK: 3, includeValues: false, includeMetadata: true, }); ``` ```java Java import io.pinecone.clients.Index; import io.pinecone.configs.PineconeConfig; import io.pinecone.configs.PineconeConnection; import io.pinecone.unsigned_indices_model.QueryResponseWithUnsignedIndices; public class QueryExample { public static void main(String[] args) { PineconeConfig config = new PineconeConfig("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index config.setHost("INDEX_HOST"); PineconeConnection connection = new PineconeConnection(config); Index index = new Index(connection, "INDEX_NAME"); QueryResponseWithUnsignedIndices queryRespone = index.queryByVectorId(3, "rec2", "example-namespace", null, false, true); System.out.println(queryResponse); } } ``` ```go Go package main import ( "context" "encoding/json" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func prettifyStruct(obj interface{}) string { bytes, _ := json.MarshalIndent(obj, "", " ") return string(bytes) } func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "INDEX_HOST", Namespace: "example-namespace"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v", err) } vectorId := "rec2" res, err := idxConnection.QueryByVectorId(ctx, &pinecone.QueryByVectorIdRequest{ VectorId: vectorId, TopK: 3, IncludeValues: false, IncludeMetadata: true, }) if err != nil { log.Fatalf("Error encountered when querying by vector ID `%v`: %v", vectorId, err) } else { fmt.Printf(prettifyStruct(res.Matches)) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index var index = pinecone.Index(host: "INDEX_HOST"); var queryResponse = await index.QueryAsync(new QueryRequest { Id = "rec2", Namespace = "example-namespace", TopK = 3, IncludeValues = false, IncludeMetadata = true }); Console.WriteLine(queryResponse); ``` ```bash curl # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index PINECONE_API_KEY="YOUR_API_KEY" INDEX_HOST="INDEX_HOST" curl "https://$INDEX_HOST/query" \ -H "Api-Key: $PINECONE_API_KEY" \ -H 'Content-Type: application/json' \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "id": "rec2", "namespace": "example-namespace", "topK": 3, "includeMetadata": true, "includeValues": false }' ``` ## Parallel queries Python SDK v6.0.0 and later provide `async` methods for use with [asyncio](https://docs.python.org/3/library/asyncio.html). Async support makes it possible to use Pinecone with modern async web frameworks such as FastAPI, Quart, and Sanic, and can significantly increase the efficiency of running queries in parallel. For more details, see the [Async requests](/reference/python-sdk#async-requests). # Configure an index Source: https://docs.pinecone.io/reference/api/2025-04/control-plane/configure_index db_control_2025-04.oas patch /indexes/{index_name} Configure an existing index. For serverless indexes, you can configure index deletion protection, tags, and integrated inference embedding settings for the index. For pod-based indexes, you can configure the pod size, number of replicas, tags, and index deletion protection. It is not possible to change the pod type of a pod-based index. However, you can create a collection from a pod-based index and then [create a new pod-based index with a different pod type](http://docs.pinecone.io/guides/indexes/pods/create-a-pod-based-index#create-a-pod-index-from-a-collection) from the collection. For guidance and examples, see [Configure an index](http://docs.pinecone.io/guides/indexes/pods/manage-pod-based-indexes). ```python Python from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key="YOUR_API_KEY") pc.configure_index( name="docs-example", pod_type="p1.x2", replicas=4, deletion_protection="enabled" ) ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone' const pinecone = new Pinecone({ apiKey: 'YOUR_API_KEY' }); await pinecone.configureIndex('docs-example', { podType: 'p1.x2', replicas: 4, deletionProtection: 'enabled' }); ``` ```java Java import io.pinecone.clients.Pinecone; public class ConfigureIndexExample { public static void main(String[] args) { Pinecone pc = new Pinecone.Builder("YOUR_API_KEY").build(); pc.configureIndex("docs-example", "p1.x2", 4, DeletionProtection.ENABLED); } } ``` ```go Go package main import ( "context" "fmt" "log" "github.com/pinecone-io/go-pinecone/v2/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } idx, err := pc.ConfigureIndex(ctx, "docs-example", pinecone.ConfigureIndexParams{PodType: "p1.x2", Replicas: 4, DeletionProtection: "enabled"}) if err != nil { log.Fatalf("Failed to configure index \"%v\": %v", idx.Name, err) } else { fmt.Printf("Successfully configured index \"%v\"", idx.Name) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); var indexMetadata = await pinecone.ConfigureIndexAsync("docs-example", new ConfigureIndexRequest { Spec = new ConfigureIndexRequestSpec { Pod = new ConfigureIndexRequestSpecPod { Replicas = 4, PodType = "p1.x2", } }, DeletionProtection = DeletionProtection.Disabled, }); ``` ```shell curl PINECONE_API_KEY="YOUR_API_KEY" curl -s -X PATCH "https://api.pinecone.io/indexes/docs-example" \ -H "Content-Type: application/json" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "spec": { "pod": { "pod_type": "p1.x2", "replicas": 4 } }, deletion_protection": "enabled" }' ``` ```json { "name": "docs-example", "metric": "cosine", "dimension": 1536, "status": { "ready": true, "state": "Ready" }, "host": "docs-example1-4zo0ijk.svc.dev-us-west2-aws.pinecone.io", "spec": { "serverless": { "region": "us-east-1", "cloud": "aws" }, "tags": { "example": "tag", "environment": "production" } } } ``` # Create a backup of an index Source: https://docs.pinecone.io/reference/api/2025-04/control-plane/create_backup db_control_2025-04.oas post /indexes/{index_name}/backups Create a backup of an index. ```python Python from pinecone import Pinecone pc = Pinecone(api_key="YOUR_API_KEY") backup = pc.create_backup( index_name="docs-example", backup_name="example-backup", description="Monthly backup of production index" ) print(backup) ``` ```javascript JavaScript const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }) const backup = await pc.createBackup({ indexName: 'docs-example', name: 'example-backup', description: 'Monthly backup of production index', }); console.log(backup); ``` ```java Java import io.pinecone.clients.Pinecone; import org.openapitools.db_control.client.ApiException; import org.openapitools.db_control.client.model.*; public class CreateBackup { public static void main(String[] args) throws ApiException { Pinecone pc = new Pinecone.Builder("YOUR_API_KEY").build(); String indexName = "docs-example"; String backupName = "example-backup"; String backupDescription = "Monthly backup of production index"; BackupModel backup = pc.createBackup(indexName,backupName, backupDescription); System.out.println(backup); } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("PINECONE_API_KEY"); var backup = await pinecone.Backups.BackupIndexAsync( "docs-example", new BackupIndexRequest { Name = "example-backup", Description = "Monthly backup of production index" } ); Console.WriteLine(backup); ``` ```bash curl PINECONE_API_KEY="YOUR_API_KEY" INDEX_NAME="docs-example" curl "https://api.pinecone.io/indexes/$INDEX_NAME/backups" \ -H "Api-Key: $PINECONE_API_KEY" \ -H 'accept: application/json' \ -H 'Content-Type: application/json' \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "name": "example-backup", "description": "Monthly backup of production index" }' ``` ```python Python {'backup_id': '8c85e612-ed1c-4f97-9f8c-8194e07bcf71', 'cloud': 'aws', 'created_at': '2025-05-15T00:52:10.809305882Z', 'description': 'Monthly backup of production index', 'dimension': 1024, 'name': 'example-backup', 'namespace_count': 3, 'record_count': 98, 'region': 'us-east-1', 'size_bytes': 1069169, 'source_index_id': 'f73b36c9-faf5-4a2c-b1d6-4013d8b1cc74', 'source_index_name': 'docs-example', 'status': 'Ready', 'tags': {}} ``` ```javascript JavaScript { backupId: '8c85e612-ed1c-4f97-9f8c-8194e07bcf71', sourceIndexName: 'docs-example', sourceIndexId: 'f73b36c9-faf5-4a2c-b1d6-4013d8b1cc74', name: 'example-backup', description: 'Monthly backup of production index', status: 'Ready', cloud: 'aws', region: 'us-east-1', dimension: 1024, metric: undefined, recordCount: 98, namespaceCount: 3, sizeBytes: 1069169, tags: {}, createdAt: '2025-05-14T16:37:25.625540Z' } ``` ```java Java class BackupModel { backupId: 0d75b99f-be61-4a93-905e-77201286c02e sourceIndexName: docs-example sourceIndexId: f73b36c9-faf5-4a2c-b1d6-4013d8b1cc74 name: example-backup description: Monthly backup of production index status: Initializing cloud: aws region: us-east-1 dimension: null metric: null recordCount: null namespaceCount: null sizeBytes: null tags: {} createdAt: 2025-05-16T19:42:23.804787550Z additionalProperties: null } ``` ```csharp C# { "backup_id": "8c85e612-ed1c-4f97-9f8c-8194e07bcf71", "source_index_name": "docs-example", "source_index_id": "f73b36c9-faf5-4a2c-b1d6-4013d8b1cc74", "name": "example-backup", "description": "Monthly backup of production index", "status": "Ready", "cloud": "aws", "region": "us-east-1", "tags": {}, "created_at": "2025-05-15T00:52:10.809305882Z" } ``` ```json curl { "backup_id":"8c85e612-ed1c-4f97-9f8c-8194e07bcf71", "source_index_id":"f73b36c9-faf5-4a2c-b1d6-4013d8b1cc74", "source_index_name":"docs-example", "tags":{}, "name":"example-backup", "description":"Monthly backup of production index", "status":"Ready", "cloud":"aws", "region":"us-east-1", "dimension":1024, "record_count":96, "namespace_count":3, "size_bytes":1069169, "created_at":"2025-05-14T16:37:25.625540Z" } ``` # Create an index with integrated embedding Source: https://docs.pinecone.io/reference/api/2025-04/control-plane/create_for_model db_control_2025-04.oas post /indexes/create-for-model Create an index with integrated embedding. With this type of index, you provide source text, and Pinecone uses a [hosted embedding model](https://docs.pinecone.io/guides/index-data/create-an-index#embedding-models) to convert the text automatically during [upsert](https://docs.pinecone.io/reference/api/2025-01/data-plane/upsert_records) and [search](https://docs.pinecone.io/reference/api/2025-01/data-plane/search_records). For guidance and examples, see [Create an index](https://docs.pinecone.io/guides/index-data/create-an-index#integrated-embedding). ```python Python # pip install --upgrade pinecone from pinecone import Pinecone pc = Pinecone(api_key="YOUR_API_KEY") index_name = "integrated-dense-py" index_model = pc.create_index_for_model( name=index_name, cloud="aws", region="us-east-1", embed={ "model":"llama-text-embed-v2", "field_map":{"text": "chunk_text"} } ) # Import specific classes to get type hints and autocompletions from pinecone import CloudProvider, AwsRegion, IndexEmbed, EmbedModel index_model = pc.create_index_for_model( name=index_name, cloud=CloudProvider.AWS, region=AwsRegion.US_EAST_1, embed=IndexEmbed( model=EmbedModel.Multilingual_E5_Large, field_map={"text": "chunk_text"}, metric='cosine' ) ) ``` ```javascript JavaScript // npm install @pinecone-database/pinecone import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }); await pc.createIndexForModel({ name: 'integrated-dense-js', cloud: 'aws', region: 'us-east-1', embed: { model: 'llama-text-embed-v2', fieldMap: { text: 'chunk_text' }, }, waitUntilReady: true, }); ``` ```java Java import io.pinecone.clients.Pinecone; import org.openapitools.db_control.client.ApiException; import org.openapitools.db_control.client.model.CreateIndexForModelRequest; import org.openapitools.db_control.client.model.CreateIndexForModelRequestEmbed; import org.openapitools.db_control.client.model.DeletionProtection; import org.openapitools.db_control.client.model.IndexModel; import java.util.HashMap; import java.util.Map; public class CreateIntegratedIndex { public static void main(String[] args) throws ApiException { Pinecone pc = new Pinecone.Builder("pcsk_oGz4L_NkfDbacGU9f6BWG6ioVm1RZyPJELzY53nLEeqUfhMkn6kptgEHSFhLfDB8xzhFb").build(); String indexName = "integrated-dense-java"; String region = "us-east-1"; HashMap fieldMap = new HashMap<>(); fieldMap.put("text", "chunk_text"); CreateIndexForModelRequestEmbed embed = new CreateIndexForModelRequestEmbed() .model("llama-text-embed-v2") .fieldMap(fieldMap); Map tags = new HashMap<>(); tags.put("environment", "development"); IndexModel index = pc.createIndexForModel( indexName, CreateIndexForModelRequest.CloudEnum.AWS, region, embed, DeletionProtection.DISABLED, tags ); System.out.println(index); } } ``` ```go Go package main import ( "context" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } indexName := "integrated-dense-go" DeletionProtection: &deletionProtection, index, err := pc.CreateIndexForModel(ctx, &pinecone.CreateIndexForModelRequest{ Name: indexName, Cloud: pinecone.Aws, Region: "us-east-1", Embed: pinecone.CreateIndexForModelEmbed{ Model: "llama-text-embed-v2", FieldMap: map[string]interface{}{"text": "chunk_text"}, }, DeletionProtection: &deletionProtection, Tags: &pinecone.IndexTags{ "environment": "development" }, }) if err != nil { log.Fatalf("Failed to create serverless integrated index: %v", idx.Name) } else { fmt.Printf("Successfully created serverless integrated index: %v", idx.Name) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); var createIndexRequest = await pinecone.CreateIndexForModelAsync( new CreateIndexForModelRequest { Name = "integrated-dense-dotnet", Cloud = CreateIndexForModelRequestCloud.Aws, Region = "us-east-1", Embed = new CreateIndexForModelRequestEmbed { Model = "llama-text-embed-v2", FieldMap = new Dictionary() { { "text", "chunk_text" } }, }, DeletionProtection = DeletionProtection.Disabled, Tags = new Dictionary { { "environment", "development" } } } ); ``` ```shell curl PINECONE_API_KEY="YOUR_API_KEY" curl https://api.pinecone.io/indexes/create-for-model \ -H "Content-Type: application/json" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "name": "integrated-dense-curl", "cloud": "aws", "region": "us-east-1", "embed": { "model": "llama-text-embed-v2", "metric": "cosine", "field_map": { "text": "chunk_text" }, "write_parameters": { "input_type": "passage", "truncate": "END" }, "read_parameters": { "input_type": "query", "truncate": "END" } } }' ``` ```python Python {'deletion_protection': 'disabled', 'dimension': 1024, 'embed': {'dimension': 1024, 'field_map': {'text': 'chunk_text'}, 'metric': 'cosine', 'model': 'llama-text-embed-v2', 'read_parameters': {'input_type': 'query', 'truncate': 'END'}, 'write_parameters': {'input_type': 'passage', 'truncate': 'END'}}, 'host': 'integrated-dense-py-govk0nt.svc.aped-4627-b74a.pinecone.io', 'id': '9dabb7cb-ec0a-4e2e-b79e-c7c997e592ce', 'metric': 'cosine', 'name': 'integrated-dense-py', 'spec': {'serverless': {'cloud': 'aws', 'region': 'us-east-1'}}, 'status': {'ready': True, 'state': 'Ready'}, 'tags': None} ``` ```javascript JavaScript { name: 'integrated-dense-js', dimension: 1024, metric: 'cosine', host: 'integrated-dense-js-govk0nt.svc.aped-4627-b74a.pinecone.io', deletionProtection: 'disabled', tags: undefined, embed: { model: 'llama-text-embed-v2', metric: 'cosine', dimension: 1024, vectorType: 'dense', fieldMap: { text: 'chunk_text' }, readParameters: { input_type: 'query', truncate: 'END' }, writeParameters: { input_type: 'passage', truncate: 'END' } }, spec: { pod: undefined, serverless: { cloud: 'aws', region: 'us-east-1' } }, status: { ready: true, state: 'Ready' }, vectorType: 'dense' } ``` ```java Java class IndexModel { name: integrated-dense-java dimension: 1024 metric: cosine host: integrated-dense-java-govk0nt.svc.aped-4627-b74a.pinecone.io deletionProtection: disabled tags: {environment=development} embed: class ModelIndexEmbed { model: llama-text-embed-v2 metric: cosine dimension: 1024 vectorType: dense fieldMap: {text=chunk_text} readParameters: {dimension=1024.0, input_type=query, truncate=END} writeParameters: {dimension=1024.0, input_type=passage, truncate=END} additionalProperties: null } spec: class IndexModelSpec { byoc: null pod: null serverless: class ServerlessSpec { cloud: aws region: us-east-1 additionalProperties: null } additionalProperties: null } status: class IndexModelStatus { ready: false state: Initializing additionalProperties: null } vectorType: dense additionalProperties: null } ``` ```go Go { "name": "integrated-dense-go", "host": "integrated-dense-go-govk0nt.svc.aped-4627-b74a.pinecone.io", "metric": "cosine", "vector_type": "dense", "deletion_protection": "disabled", "dimension": 1024, "spec": { "serverless": { "cloud": "aws", "region": "us-east-1" } }, "status": { "ready": true, "state": "Ready" }, "embed": { "model": "llama-text-embed-v2", "dimension": 1024, "metric": "cosine", "vector_type": "dense", "field_map": { "text": "chunk_text" }, "read_parameters": { "input_type": "query", "truncate": "END" }, "write_parameters": { "input_type": "passage", "truncate": "END" } } } ``` ```csharp C# { "name": "integrated-dense-dotnet", "dimension": 1024, "metric": "cosine", "host": "integrated-dense-dotnet-govk0nt.svc.aped-4627-b74a.pinecone.io", "deletion_protection": "disabled", "tags": { "environment": "development" }, "embed": { "model": "llama-text-embed-v2", "metric": "cosine", "dimension": 1024, "vector_type": "dense", "field_map": { "text": "chunk_text" }, "read_parameters": { "dimension": 1024, "input_type": "query", "truncate": "END" }, "write_parameters": { "dimension": 1024, "input_type": "passage", "truncate": "END" } }, "spec": { "serverless": { "cloud": "aws", "region": "us-east-1" } }, "status": { "ready": true, "state": "Ready" }, "vector_type": "dense" } ``` ```json curl { "id": "9dabb7cb-ec0a-4e2e-b79e-c7c997e592ce", "name": "integrated-dense-curl", "metric": "cosine", "dimension": 1024, "status": { "ready": false, "state": "Initializing" }, "host": "integrated-dense-curl-govk0nt.svc.aped-4627-b74a.pinecone.io", "spec": { "serverless": { "region": "us-east-1", "cloud": "aws" } }, "deletion_protection": "disabled", "tags": null, "embed": { "model": "llama-text-embed-v2", "field_map": { "text": "chunk_text" }, "dimension": 1024, "metric": "cosine", "write_parameters": { "input_type": "passage", "truncate": "END" }, "read_parameters": { "input_type": "query", "truncate": "END" } } } ``` # Create an index Source: https://docs.pinecone.io/reference/api/2025-04/control-plane/create_index db_control_2025-04.oas post /indexes Create a Pinecone index. This is where you specify the measure of similarity, the dimension of vectors to be stored in the index, which cloud provider you would like to deploy with, and more. For guidance and examples, see [Create an index](https://docs.pinecone.io/guides/index-data/create-an-index). ```python Python # pip install "pinecone[grpc]" # Serverless index from pinecone.grpc import PineconeGRPC as Pinecone from pinecone import ServerlessSpec pc = Pinecone(api_key="YOUR_API_KEY") pc.create_index( name="docs-example1", dimension=1536, metric="cosine", spec=ServerlessSpec( cloud="aws", region="us-east-1", ), deletion_protection="disabled" ) # Pod-based index from pinecone.grpc import PineconeGRPC as Pinecone, PodSpec pc = Pinecone(api_key="YOUR_API_KEY") pc.create_index( name="docs-example2", dimension=1536, metric="cosine", spec=PodSpec( environment="us-west1-gcp", pod_type="p1.x1", pods=1, ), deletion_protection="disabled" ) ``` ```javascript JavaScript // npm install @pinecone-database/pinecone // Serverles index import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }); await pc.createIndex({ name: 'serverless-index', dimension: 1536, metric: 'cosine', spec: { serverless: { cloud: 'aws', region: 'us-east-1' } }, deletionProtection: 'disabled', }); // Pod-based index await pc.createIndex({ name: 'docs-example2', dimension: 1536, metric: 'cosine', spec: { pod: { environment: 'us-west1-gcp', podType: 'p1.x1', pods: 1 } }, deletionProtection: 'disabled', }); ``` ```java Java import io.pinecone.clients.Pinecone; // Serverless index public class CreateServerlessIndexExample { public static void main(String[] args) { Pinecone pc = new Pinecone.Builder("YOUR_API_KEY").build(); pc.createServerlessIndex("docs-example1", "cosine", 1536, "aws", "us-east-1", DeletionProtection.disabled); } } // Pod-based index public class CreatePodIndexExample { public static void main(String[] args) { Pinecone pc = new Pinecone.Builder("YOUR_API_KEY").build(); pc.createPodsIndex("docs-example2", 1536, "us-west1-gcp", "p1.x1", "cosine", DeletionProtection.disabled); } } ``` ```go Go package main import ( "context" "fmt" "log" "github.com/pinecone-io/go-pinecone/v2/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // Serverless index idx, err := pc.CreateServerlessIndex(ctx, &pinecone.CreateServerlessIndexRequest{ Name: "docs-example1", Dimension: 1536, Metric: pinecone.Cosine, Cloud: pinecone.Aws, Region: "us-east-1", DeletionProtection: "disabled", }) if err != nil { log.Fatalf("Failed to create serverless index: %v", idx.Name) } else { fmt.Printf("Successfully created serverless index: %v", idx.Name) } // Pod-based index idx, err := pc.CreatePodIndex(ctx, &pinecone.CreatePodIndexRequest{ Name: "docs-example2", Dimension: 1536, Metric: pinecone.Cosine, Environment: "us-east1-gcp", PodType: "p1.x1", DeletionProtection: "disabled", }) if err != nil { log.Fatalf("Failed to create pod-based index: %v", idx.Name) } else { fmt.Printf("Successfully created pod-based index: %v", idx.Name) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); // Serverless index var createIndexRequest = await pinecone.CreateIndexAsync(new CreateIndexRequest { Name = "docs-example1", Dimension = 1536, Metric = MetricType.Cosine, Spec = new ServerlessIndexSpec { Serverless = new ServerlessSpec { Cloud = ServerlessSpecCloud.Aws, Region = "us-east-1", } }, DeletionProtection = DeletionProtection.Disabled }); // Pod-based index var createIndexRequest = await pinecone.CreateIndexAsync(new CreateIndexRequest { Name = "pod index", Dimension = 1536, Metric = MetricType.Cosine, Spec = new PodIndexSpec { Pod = new PodSpec { Environment = "us-east1-gcp", PodType = "p1.x1", Pods = 1, } }, DeletionProtection = DeletionProtection.Disabled }); ``` ```shell curl PINECONE_API_KEY="YOUR_API_KEY" # Serverless index curl -s "https://api.pinecone.io/indexes" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "name": "example-serverless-index", "vector_type": "dense", "dimension": 1536, "metric": "cosine", "spec": { "serverless": { "cloud": "aws", "region": "us-east-1" } }, "tags"={ "example": "tag" }, "deletion_protection": "disabled" }' # BYOC index curl -s "https://api.pinecone.io/indexes" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "name": "example-byoc-index", "vector_type": "dense", "dimension": 1536, "metric": "cosine", "spec": { "byoc": { "environment": "aws-us-east-1-b921" } }, "tags"={ "example": "tag" }, "deletion_protection": "disabled" }' # Pod-based index curl -s "https://api.pinecone.io/indexes" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "name": "example-pod-index", "dimension": 1536, "metric": "cosine", "spec": { "pod": { "environment": "us-west1-gcp", "pod_type": "p1.x1", "pods": 1 } }, "tags": { "example": "tag" }, "deletion_protection": "disabled" }' ``` ```shell # Serverless index { "name": "example-serverless-index", "vector_type": "dense", "metric": "cosine", "dimension": 1536, "status": { "ready": true, "state": "Ready" }, "host": "example-serverless-index-govk0nt.svc.aped-4627-b74a.pinecone.io", "spec": { "serverless": { "region": "us-east-1", "cloud": "aws" } }, "deletion_protection": "disabled", "tags": { "example": "tag" } } # BYOC index { "name": "example-byoc-index", "vector_type": "dense", "metric": "cosine", "dimension": 1536, "status": { "ready": true, "state": "Ready" }, "host": "example-byoc-index-govk0nt.svc.private.aped-4627-b74a.pinecone.io", "spec": { "byoc": { "environment": "aws-us-east-1-b921" } }, "deletion_protection": "disabled", "tags": { "example": "tag" } } # Pod-based index { "name": "example-pod-index", "metric": "cosine", "dimension": 1536, "status": { "ready": true, "state": "Ready" }, "host": "example-pod-index-4zo0ijk.svc.us-west1-gcp.pinecone.io", "spec": { "pod": { "replicas": 1, "shards": 1, "pods": 1, "pod_type": "p1.x1", "environment": "us-west1-gcp" } }, "deletion_protection": "disabled", "tags": { "example": "tag" } } ``` # Delete an index Source: https://docs.pinecone.io/reference/api/2025-04/control-plane/delete_index db_control_2025-04.oas delete /indexes/{index_name} Delete an existing index. ```python Python # pip install "pinecone[grpc]" from pinecone.grpc import PineconeGRPC as Pinecone, PodSpec pc = Pinecone(api_key="YOUR_API_KEY") pc.delete_index(name="docs-example") ``` ```javascript JavaScript // npm install @pinecone-database/pinecone import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }); await pc.deleteIndex('docs-example'); ``` ```java Java import io.pinecone.clients.Pinecone; public class DeleteIndexExample { public static void main(String[] args) { Pinecone pc = new Pinecone.Builder("YOUR_API_KEY").build(); pc.deleteIndex("docs-example"); } } ``` ```go Go package main import ( "context" "fmt" "log" "github.com/pinecone-io/go-pinecone/v2/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } indexName := "docs-example" err = pc.DeleteIndex(ctx, indexName) if err != nil { log.Fatalf("Failed to delete index: %v", err) } else { fmt.Println("Index \"%v\" deleted successfully", indexName) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); await pinecone.DeleteIndexAsync("docs-example"); ``` ```shell curl PINECONE_API_KEY="YOUR_API_KEY" curl -i -X DELETE "https://api.pinecone.io/indexes/docs-example" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" ``` # Describe an index Source: https://docs.pinecone.io/reference/api/2025-04/control-plane/describe_index db_control_2025-04.oas get /indexes/{index_name} Get a description of an index. ```python Python # pip install "pinecone[grpc]" from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key="YOUR_API_KEY") pc.describe_index(name="movie-recommendations") ``` ```javascript JavaScript // npm install @pinecone-database/pinecone import { Pinecone } from '@pinecone-database/pinecone'; const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }); await pc.describeIndex('movie-recommendations'); ``` ```java Java import io.pinecone.clients.Pinecone; import org.openapitools.db_control.client.model.*; public class DescribeIndexExample { public static void main(String[] args) { Pinecone pc = new Pinecone.Builder("YOURE_API_KEY").build(); IndexModel indexModel = pc.describeIndex("movie-recommendations"); System.out.println(indexModel); } } ``` ```go Go package main import ( "context" "encoding/json" "fmt" "log" "github.com/pinecone-io/go-pinecone/v2/pinecone" ) func prettifyStruct(obj interface{}) string { bytes, _ := json.MarshalIndent(obj, "", " ") return string(bytes) } func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } idx, err := pc.DescribeIndex(ctx, "movie-recommendations") if err != nil { log.Fatalf("Failed to describe index \"%v\": %v", idx.Name, err) } else { fmt.Printf("index: %v\n", prettifyStruct(idx)) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); var indexModel = await pinecone.DescribeIndexAsync("docs-example"); Console.WriteLine(indexModel); ``` ```shell curl PINECONE_API_KEY="YOUR_API_KEY" curl -i -X GET "https://api.pinecone.io/indexes/movie-recommendations" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" ``` ```Python Python {'deletion_protection': 'disabled', 'dimension': 1536, 'host': 'docs-example-dense-govk0nt.svc.aped-4627-b74a.pinecone.io', 'metric': 'cosine', 'name': 'docs-example-dense', 'spec': {'serverless': {'cloud': 'aws', 'region': 'us-east-1'}}, 'status': {'ready': True, 'state': 'Ready'}, 'tags': {'environment': 'development'}, 'vector_type': 'dense'} ``` ```javaScript JavaScript { name: 'docs-example-dense', dimension: 1536, metric: 'cosine', host: 'docs-example-dense-govk0nt.svc.aped-4627-b74a.pinecone.io', deletionProtection: 'disabled', tags: { environment: 'development', example: 'tag' }, embed: undefined, spec: { pod: undefined, serverless: { cloud: 'aws', region: 'us-east-1' } }, status: { ready: true, state: 'Ready' }, vectorType: 'dense' } ``` ```java Java class IndexModel { name: docs-example-dense dimension: 1536 metric: cosine host: docs-example-dense-govk0nt.svc.aped-4627-b74a.pinecone.io deletionProtection: disabled tags: {environment=development} embed: null spec: class IndexModelSpec { pod: null serverless: class ServerlessSpec { cloud: aws region: us-east-1 additionalProperties: null } additionalProperties: null } status: class IndexModelStatus { ready: true state: Ready additionalProperties: null } vectorType: dense additionalProperties: null } ``` ```go Go index: { "name": "docs-example-dense", "host": "docs-example-dense-govk0nt.svc.aped-4627-b74a.pinecone.io", "metric": "cosine", "vector_type": "dense", "deletion_protection": "disabled", "dimension": 1536, "spec": { "serverless": { "cloud": "aws", "region": "us-east-1" } }, "status": { "ready": true, "state": "Ready" }, "tags": { "environment": "development" } } ``` ```csharp C# { "name": "docs-example-dense", "dimension": 1536, "metric": "cosine", "host": "docs-example-dense-govk0nt.svc.aped-4627-b74a.pinecone.io", "deletion_protection": "disabled", "tags": { "environment": "development" }, "spec": { "serverless": { "cloud": "aws", "region": "us-east-1" } }, "status": { "ready": true, "state": "Ready" }, "vector_type": "dense" } ``` ```json curl { "name": "docs-example-dense", "vector_type": "dense", "metric": "cosine", "dimension": 1536, "status": { "ready": true, "state": "Ready" }, "host": "docs-example-dense-govk0nt.svc.aped-4627-b74a.pinecone.io", "spec": { "serverless": { "region": "us-east-1", "cloud": "aws" } }, "deletion_protection": "disabled", "tags": { "environment": "development" } } ``` # List indexes Source: https://docs.pinecone.io/reference/api/2025-04/control-plane/list_indexes db_control_2025-04.oas get /indexes List all indexes in a project. ```Python Python from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key="YOUR_API_KEY") index_list = pc.list_indexes() print(index_list) ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }) const indexList = await pc.listIndexes(); console.log(indexList); ``` ```java Java import io.pinecone.clients.Pinecone; import org.openapitools.db_control.client.model.*; public class ListIndexesExample { public static void main(String[] args) { Pinecone pc = new Pinecone.Builder("YOUR_API_KEY").build(); IndexList indexList = pc.listIndexes(); System.out.println(indexList); } } ``` ```go Go package main import ( "context" "encoding/json" "fmt" "log" "github.com/pinecone-io/go-pinecone/v2/pinecone" ) func prettifyStruct(obj interface{}) string { bytes, _ := json.MarshalIndent(obj, "", " ") return string(bytes) } func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } idxs, err := pc.ListIndexes(ctx) if err != nil { log.Fatalf("Failed to list indexes: %v", err) } else { for _, index := range idxs { fmt.Printf("index: %v\n", prettifyStruct(index)) } } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); var indexList = await pinecone.ListIndexesAsync(); Console.WriteLine(indexList); ``` ```shell curl PINECONE_API_KEY="YOUR_API_KEY" curl -i -X GET "https://api.pinecone.io/indexes" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" ``` ```json { "indexes": [ { "name": "docs-example2", "metric": "cosine", "dimension": 1536, "status": { "ready": true, "state": "Ready" }, "host": "docs-example2-govk0nt.svc.aped-4627-b74a.pinecone.io", "spec": { "serverless": { "region": "us-east-1", "cloud": "aws" } }, "deletion_protection": "disabled", "tags": { "environment": "production", "example": "tag2" }, "vector_type": "dense" }, { "name": "docs-example1", "metric": "cosine", "dimension": 1536, "status": { "ready": true, "state": "Ready" }, "host": "docs-example1-govk0nt.svc.aped-4627-b74a.pinecone.io", "spec": { "serverless": { "region": "us-east-1", "cloud": "aws" } }, "deletion_protection": "disabled", "tags": { "environment": "development", "example": "tag" }, "vector_type": "dense" } ] } ``` # List backups for all indexes in a project Source: https://docs.pinecone.io/reference/api/2025-04/control-plane/list_project_backups db_control_2025-04.oas get /backups List all backups for a project. ```python Python from pinecone import Pinecone pc = Pinecone(api_key="YOUR_API_KEY") project_backups = pc.list_backups() print(project_backups) ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone'; const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }) const projectBackups = await pc.listBackups(); console.log(projectBackups); ``` ```java Java import io.pinecone.clients.Pinecone; import org.openapitools.db_control.client.ApiException; import org.openapitools.db_control.client.model.*; public class CreateBackup { public static void main(String[] args) throws ApiException { Pinecone pc = new Pinecone.Builder("YOUR_API_KEY").build(); String indexName = "docs-example"; BackupList projectBackupList = pc.listProjectBackups(); System.out.println(projectBackupList); } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("PINECONE_API_KEY"); var backups = await pinecone.Backups.ListAsync(); Console.WriteLine(backups); ``` ```bash curl PINECONE_API_KEY="YOUR_API_KEY" curl -X GET "https://api.pinecone.io/backups" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" \ -H "accept: application/json" ``` ```python Python [{ "backup_id": "8c85e612-ed1c-4f97-9f8c-8194e07bcf71", "source_index_name": "docs-example", "source_index_id": "f73b36c9-faf5-4a2c-b1d6-4013d8b1cc74", "status": "Ready", "cloud": "aws", "region": "us-east-1", "tags": {}, "name": "example-backup", "description": "Monthly backup of production index", "dimension": 1024, "record_count": 98, "namespace_count": 3, "size_bytes": 1069169, "created_at": "2025-05-15T20:26:21.248515Z" }, { "backup_id": "95707edb-e482-49cf-b5a5-312219a51a97", "source_index_name": "docs-example2", "source_index_id": "b49f27d1-1bf3-49c6-82b5-4ae46f00f0e6", "status": "Ready", "cloud": "aws", "region": "us-east-1", "tags": {}, "name": "example-backup2", "description": "Monthly backup of production index", "dimension": 1024, "record_count": 97, "namespace_count": 2, "size_bytes": 1069169, "created_at": "2025-05-15T00:52:10.809354Z" }, { "backup_id": "8c85e612-ed1c-4f97-9f8c-8194e07bcf71", "source_index_name": "docs-example3", "source_index_id": "f73b36c9-faf5-4a2c-b1d6-4013d8b1cc74", "status": "Ready", "cloud": "aws", "region": "us-east-1", "tags": {}, "name": "example-backup3", "description": "Monthly backup of production index", "dimension": 1024, "record_count": 98, "namespace_count": 3, "size_bytes": 1069169, "created_at": "2025-05-14T16:37:25.625540Z" }] ``` ```javascript JavaScript { data: [ { backupId: 'e12269b0-a29b-4af0-9729-c7771dec03e3', sourceIndexName: 'docs-example', sourceIndexId: 'bcb5b3c9-903e-4cb6-8b37-a6072aeb874f', name: 'example-backup', description: undefined, status: 'Ready', cloud: 'aws', region: 'us-east-1', dimension: 0, metric: undefined, recordCount: 96, namespaceCount: 1, sizeBytes: 86393, tags: undefined, createdAt: '2025-05-14T17:00:45.803146Z' }, { backupId: 'd686451d-1ede-4004-9f72-7d22cc799b6e', sourceIndexName: 'docs-example2', sourceIndexId: 'b49f27d1-1bf3-49c6-82b5-4ae46f00f0e6', name: 'example-backup2', description: undefined, status: 'Ready', cloud: 'aws', region: 'us-east-1', dimension: 1024, metric: undefined, recordCount: 50, namespaceCount: 1, sizeBytes: 545171, tags: undefined, createdAt: '2025-05-14T17:00:34.814371Z' }, { backupId: '8c85e612-ed1c-4f97-9f8c-8194e07bcf71', sourceIndexName: 'docs-example3', sourceIndexId: 'f73b36c9-faf5-4a2c-b1d6-4013d8b1cc74', name: 'example-backup3', description: 'Monthly backup of production index', status: 'Ready', cloud: 'aws', region: 'us-east-1', dimension: 1024, metric: undefined, recordCount: 98, namespaceCount: 3, sizeBytes: 1069169, tags: {}, createdAt: '2025-05-14T16:37:25.625540Z' } ], pagination: undefined } ``` ```java Java class BackupList { data: [class BackupModel { backupId: 13761d20-7a0b-4778-ac27-36dd91c4be43 sourceIndexName: example-dense-index sourceIndexId: f73b36c9-faf5-4a2c-b1d6-4013d8b1cc74 name: example-backup-java2 description: Monthly backup of production index status: Initializing cloud: aws region: us-east-1 dimension: null metric: null recordCount: null namespaceCount: null sizeBytes: null tags: {} createdAt: 2025-05-16T19:46:26.248428Z additionalProperties: null }, class BackupModel { backupId: 0d75b99f-be61-4a93-905e-77201286c02e sourceIndexName: example-dense-index sourceIndexId: f73b36c9-faf5-4a2c-b1d6-4013d8b1cc74 name: example-backup-java description: Monthly backup of production index status: Initializing cloud: aws region: us-east-1 dimension: null metric: null recordCount: null namespaceCount: null sizeBytes: null tags: {} createdAt: 2025-05-16T19:42:23.804820Z additionalProperties: null }, class BackupModel { backupId: bf2cda5d-b233-4a0a-aae9-b592780ad3ff sourceIndexName: example-sparse-index sourceIndexId: bcb5b3c9-903e-4cb6-8b37-a6072aeb874f name: example-sparse-python description: Monthly backup of production index status: Ready cloud: aws region: us-east-1 dimension: 0 metric: null recordCount: 96 namespaceCount: 1 sizeBytes: 86393 tags: {} createdAt: 2025-05-16T18:01:51.531129Z additionalProperties: null }] pagination: null additionalProperties: null } ``` ```csharp C# { "data": [ { "backup_id": "95707edb-e482-49cf-b5a5-312219a51a97", "source_index_name": "docs-example", "source_index_id": "f73b36c9-faf5-4a2c-b1d6-4013d8b1cc74", "name": "example-backup", "description": "Monthly backup of production index", "status": "Ready", "cloud": "aws", "region": "us-east-1", "dimension": 1024, "record_count": 97, "namespace_count": 2, "size_bytes": 1069169, "tags": {}, "created_at": "2025-05-15T00:52:10.809354Z" }, { "backup_id": "e12269b0-a29b-4af0-9729-c7771dec03e3", "source_index_name": "docs-example2", "source_index_id": "bcb5b3c9-903e-4cb6-8b37-a6072aeb874f", "name": "example-backup2", "status": "Ready", "cloud": "aws", "region": "us-east-1", "dimension": 0, "record_count": 96, "namespace_count": 1, "size_bytes": 86393, "created_at": "2025-05-14T17:00:45.803146Z" }, { "backup_id": "d686451d-1ede-4004-9f72-7d22cc799b6e", "source_index_name": "docs-example3", "source_index_id": "b49f27d1-1bf3-49c6-82b5-4ae46f00f0e6", "name": "example-backup3", "status": "Ready", "cloud": "aws", "region": "us-east-1", "dimension": 1024, "record_count": 50, "namespace_count": 1, "size_bytes": 545171, "created_at": "2025-05-14T17:00:34.814371Z" } ] } ``` ```json curl { "data": [ { "backup_id": "e12269b0-a29b-4af0-9729-c7771dec03e3", "source_index_id": "bcb5b3c9-903e-4cb6-8b37-a6072aeb874f", "source_index_name": "docs-example", "tags": null, "name": "example-backup", "description": null, "status": "Ready", "cloud": "aws", "region": "us-east-1", "dimension": 0, "record_count": 96, "namespace_count": 1, "size_bytes": 86393, "created_at": "2025-05-14T17:00:45.803146Z" }, { "backup_id": "d686451d-1ede-4004-9f72-7d22cc799b6e", "source_index_id": "b49f27d1-1bf3-49c6-82b5-4ae46f00f0e6", "source_index_name": "docs-example2", "tags": null, "name": "example-backup2", "description": null, "status": "Ready", "cloud": "aws", "region": "us-east-1", "dimension": 1024, "record_count": 50, "namespace_count": 1, "size_bytes": 545171, "created_at": "2025-05-14T17:00:34.814371Z" }, { "backup_id": "8c85e612-ed1c-4f97-9f8c-8194e07bcf71", "source_index_id": "f73b36c9-faf5-4a2c-b1d6-4013d8b1cc74", "source_index_name": "docs-example3", "tags": {}, "name": "example-backup3", "description": "Monthly backup of production index", "status": "Ready", "cloud": "aws", "region": "us-east-1", "dimension": 1024, "record_count": 98, "namespace_count": 3, "size_bytes": 1069169, "created_at": "2025-05-14T16:37:25.625540Z" } ], "pagination": null } ``` # Cancel an import Source: https://docs.pinecone.io/reference/api/2025-04/data-plane/cancel_import db_data_2025-04.oas delete /bulk/imports/{id} Cancel an import operation if it is not yet finished. It has no effect if the operation is already finished. For guidance and examples, see [Import data](https://docs.pinecone.io/guides/index-data/import-data). This feature is in [public preview](/release-notes/feature-availability) and available only on [Standard and Enterprise plans](https://www.pinecone.io/pricing/). ```python Python from pinecone import Pinecone pc = Pinecone(api_key="YOUR_API_KEY") # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index index = pc.Index(host="INDEX_HOST") index.cancel_import(id="101") ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone'; const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index const index = pc.index("INDEX_NAME", "INDEX_HOST") await index.cancelImport(id='101'); ``` ```java Java import io.pinecone.clients.Pinecone; import io.pinecone.clients.AsyncIndex; import org.openapitools.db_data.client.ApiException; public class CancelImport { public static void main(String[] args) throws ApiException { // Initialize a Pinecone client with your API key Pinecone pinecone = new Pinecone.Builder("YOUR_API_KEY").build(); // Get async imports connection object AsyncIndex asyncIndex = pinecone.getAsyncIndexConnection("docs-example"); // Cancel import asyncIndex.cancelImport("2"); } } ``` ```go Go package main import ( "context" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "INDEX_HOST"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v", err) } importID := "101" err = idxConnection.CancelImport(ctx, importID) if err != nil { log.Fatalf("Failed to cancel import: %s", importID) } importDesc, err := idxConnection.DescribeImport(ctx, importID) if err != nil { log.Fatalf("Failed to describe import: %s - %v", importID, err) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index var index = pinecone.Index(host: "INDEX_HOST"); var cancelResponse = await index.CancelBulkImportAsync("101"); ``` ```bash curl # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index PINECONE_API_KEY="YOUR_API_KEY" INDEX_HOST="INDEX_HOST" curl -X DELETE "https://{INDEX_HOST}/bulk/imports/101" \ -H 'Api-Key: $YOUR_API_KEY' \ -H "X-Pinecone-API-Version: 2025-04" ``` ```json {} ``` # Delete vectors Source: https://docs.pinecone.io/reference/api/2025-04/data-plane/delete db_data_2025-04.oas post /vectors/delete Delete vectors by id from a single namespace. For guidance and examples, see [Delete data](https://docs.pinecone.io/guides/manage-data/delete-data). ```python Python # pip install "pinecone[grpc]" from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key="YOUR_API_KEY") # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index index = pc.Index(host="INDEX_HOST") index.delete(ids=["id-1", "id-2"], namespace="example-namespace") ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }) // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index const index = pc.index("INDEX_NAME", "INDEX_HOST") const ns = index.namespace('example-namespace') // Delete one record by ID. await ns.deleteOne('id-1'); // Delete more than one record by ID. await ns.deleteMany(['id-2', 'id-3']); ``` ```java Java import io.pinecone.clients.Index; import io.pinecone.configs.PineconeConfig; import io.pinecone.configs.PineconeConnection; import java.util.Arrays; import java.util.List; public class DeleteVectorsExample { public static void main(String[] args) { PineconeConfig config = new PineconeConfig("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index config.setHost("INDEX_HOST"); PineconeConnection connection = new PineconeConnection(config); Index index = new Index(connection, "INDEX_NAME"); List ids = Arrays.asList("id-1 ", "id-2"); index.deleteByIds(ids, "example-namespace"); } } ``` ```go Go package main import ( "context" "log" "github.com/pinecone-io/go-pinecone/v2/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "INDEX_HOST", Namespace: "example-namespace"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v", err) } id1 := "id-1" id2 := "id-2" err = idxConnection.DeleteVectorsById(ctx, []string{id1, id2}) if err != nil { log.Fatalf("Failed to delete vector with ID %v: %v", id, err) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index var index = pinecone.Index(host: "INDEX_HOST"); var deleteResponse = await index.DeleteAsync(new DeleteRequest { Ids = new List { "id-1", "id-2" }, Namespace = "example-namespace", }); ``` ```shell curl # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index PINECONE_API_KEY="YOUR_API_KEY" INDEX_HOST="INDEX_HOST" curl "https://$INDEX_HOST/vectors/delete" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "Content-Type: application/json" \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "ids": [ "id-1", "id-2" ], "namespace": "example-namespace" } ' ``` ```json {} ``` # Delete a namespace Source: https://docs.pinecone.io/reference/api/2025-04/data-plane/deletenamespace db_data_2025-04.oas delete /namespaces/{namespace} Delete a namespace from an index. ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone'; const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }) const index = pc.index('docs-example'); const namespace = await index.deleteNamespace('example-namespace'); console.log(namespace); ``` ```java Java import io.pinecone.clients.Index; import io.pinecone.configs.PineconeConfig; import io.pinecone.configs.PineconeConnection; import java.util.concurrent.ExecutionException; public class DeleteNamespace { public static void main(String[] args) throws ExecutionException, InterruptedException { PineconeConfig config = new PineconeConfig("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index config.setHost("INDEX_HOST"); PineconeConnection connection = new PineconeConnection(config); Index index = new Index(config, connection, "docs-example"); index.deleteNamespace("example-namespace"); } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("PINECONE_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index var index = pinecone.Index(host: "INDEX_HOST"); await index.DeleteNamespaceAsync("example-namespace"); ``` ```bash curl PINECONE_API_KEY="YOUR_API_KEY" INDEX_HOST="YOUR_INDEX_HOST" NAMESPACE="YOUR_NAMESPACE" # To target the default namespace, use "__default__". curl -X DELETE "https://$INDEX_HOST/namespaces/$NAMESPACE" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" ``` # Describe an import Source: https://docs.pinecone.io/reference/api/2025-04/data-plane/describe_import db_data_2025-04.oas get /bulk/imports/{id} Return details of a specific import operation. For guidance and examples, see [Import data](https://docs.pinecone.io/guides/index-data/import-data). This feature is in [public preview](/release-notes/feature-availability) and available only on [Standard and Enterprise plans](https://www.pinecone.io/pricing/). ```python Python from pinecone import Pinecone pc = Pinecone(api_key="YOUR_API_KEY") # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index index = pc.Index(host="INDEX_HOST") index.describe_import(id="101") ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone'; const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index const index = pc.index("INDEX_NAME", "INDEX_HOST") const results = await index.describeImport(id='101'); console.log(results); ``` ```java Java import io.pinecone.clients.Pinecone; import io.pinecone.clients.AsyncIndex; import org.openapitools.db_data.client.ApiException; import org.openapitools.db_data.client.model.ImportModel; public class DescribeImport { public static void main(String[] args) throws ApiException { // Initialize a Pinecone client with your API key Pinecone pinecone = new Pinecone.Builder("YOUR_API_KEY").build(); // Get async imports connection object AsyncIndex asyncIndex = pinecone.getAsyncIndexConnection("docs-example"); // Describe import ImportModel importDetails = asyncIndex.describeImport("101"); System.out.println(importDetails); } } ``` ```go Go package main import ( "context" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "INDEX_HOST"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v", err) } importID := "101" importDesc, err := idxConnection.DescribeImport(ctx, importID) if err != nil { log.Fatalf("Failed to describe import: %s - %v", importID, err) } fmt.Printf("Import ID: %s, Status: %s", importDesc.Id, importDesc.Status) } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index var index = pinecone.Index(host: "INDEX_HOST"); var importDetails = await index.DescribeBulkImportAsync("101"); ``` ```bash curl # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index PINECONE_API_KEY="YOUR_API_KEY" INDEX_HOST="INDEX_HOST" curl -X GET "https://{INDEX_HOST}/bulk/imports/101" \ -H 'Api-Key: $YOUR_API_KEY' \ -H 'X-Pinecone-API-Version: 2025-04' ``` ```json { "id": "101", "uri": "s3://BUCKET_NAME/PATH/TO/DIR", "status": "Pending", "created_at": "2024-08-19T20:49:00.754Z", "finished_at": "2024-08-19T20:49:00.754Z", "percent_complete": 42.2, "records_imported": 1000000 } ``` # Get index stats Source: https://docs.pinecone.io/reference/api/2025-04/data-plane/describeindexstats db_data_2025-04.oas post /describe_index_stats Return statistics about the contents of an index, including the vector count per namespace, the number of dimensions, and the index fullness. Serverless indexes scale automatically as needed, so index fullness is relevant only for pod-based indexes. ```python Python # pip install "pinecone[grpc]" from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key="YOUR_API_KEY") # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index index = pc.Index(host="INDEX_HOST") index.describe_index_stats() ``` ```javascript JavaScript // npm install @pinecone-database/pinecone import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }) // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index const index = pc.index("INDEX_NAME", "INDEX_HOST") const stats = await index.describeIndexStats(); ``` ```java Java import io.pinecone.clients.Index; import io.pinecone.configs.PineconeConfig; import io.pinecone.configs.PineconeConnection; import io.pinecone.proto.DescribeIndexStatsResponse; public class DescribeIndexStatsExample { public static void main(String[] args) { PineconeConfig config = new PineconeConfig("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index config.setHost("INDEX_HOST"); PineconeConnection connection = new PineconeConnection(config); Index index = new Index(connection, "INDEX_NAME"); DescribeIndexStatsResponse indexStatsResponse = index.describeIndexStats(); System.out.println(indexStatsResponse); } } ``` ```go Go package main import ( "context" "log" "github.com/pinecone-io/go-pinecone/v2/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "INDEX_HOST"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v", err) } stats, err := idxConnection.DescribeIndexStats(ctx) if err != nil { log.Fatalf("Failed to describe index \"%v\": %v", idx.Name, err) } else { fmt.Printf("%+v", *stats) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index var index = pinecone.Index(host: "INDEX_HOST"); var indexStatsResponse = await index.DescribeIndexStatsAsync(new DescribeIndexStatsRequest()); Console.WriteLine(indexStatsResponse); ``` ```shell curl # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index PINECONE_API_KEY="YOUR_API_KEY" INDEX_HOST="INDEX_HOST" curl -X POST "https://$INDEX_HOST/describe_index_stats" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" ``` ```Python Python {'dimension': 1024, 'index_fullness': 8e-05, 'namespaces': {'example-namespace1': {'vector_count': 4}, 'example-namespace2': {'vector_count': 4}}, 'total_vector_count': 8} ``` ```JavaScript JavaScript Returns: { namespaces: { example-namespace1: { recordCount: 4 }, example-namespace2: { recordCount: 4 } }, dimension: 1024, indexFullness: 0.00008, totalRecordCount: 8 } // Note: the value of totalRecordCount is the same as total_vector_count. ``` ```java Java namespaces { key: "example-namespace1" value { vector_count: 4 } } namespaces { key: "example-namespace2" value { vector_count: 4 } } dimension: 1024 total_vector_count: 8 ``` ```go Go { "dimension": 1024, "index_fullness": 0, "total_vector_count": 8, "namespaces": { "example-namespace1": { "vector_count": 4 }, "example-namespace2": { "vector_count": 4 } } } ``` ```csharp C# { "namespaces": { "example-namespace1": { "vectorCount": 4 }, "example-namespace2": { "vectorCount": 4 } }, "dimension": 1024, "indexFullness": 0, "totalVectorCount": 8 } ``` ```shell curl { "namespaces": { "example-namespace1": { "vectorCount": 4 }, "example-namespace2": { "vectorCount": 4 } }, "dimension": 1024, "indexFullness": 0.00008, "totalVectorCount": 8 } ``` # Describe a namespace Source: https://docs.pinecone.io/reference/api/2025-04/data-plane/describenamespace db_data_2025-04.oas get /namespaces/{namespace} Describe a [namespace](https://docs.pinecone.io/guides/index-data/indexing-overview#namespaces) in a serverless index, including the total number of vectors in the namespace. ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone'; const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }) const index = pc.index('docs-example'); const namespace = await index.describeNamespace('example-namespace'); console.log(namespace); ``` ```java Java import io.pinecone.clients.Index; import io.pinecone.configs.PineconeConfig; import io.pinecone.configs.PineconeConnection; import io.pinecone.proto.NamespaceDescription; import org.openapitools.db_data.client.ApiException; public class Namespaces { public static void main(String[] args) throws ApiException { PineconeConfig config = new PineconeConfig("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index config.setHost("INDEX_HOST"); PineconeConnection connection = new PineconeConnection(config); Index index = new Index(config, connection, "docs-example"); NamespaceDescription namespaceDescription = index.describeNamespace("example-namespace"); System.out.println(namespaceDescription); } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("PINECONE_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index var index = pinecone.Index(host: "INDEX_HOST"); var @namespace = await index.DescribeNamespaceAsync("example-namespace"); Console.WriteLine(@namespace); ``` ```bash curl PINECONE_API_KEY="YOUR_API_KEY" INDEX_HOST="YOUR_INDEX_HOST" NAMESPACE="YOUR_NAMESPACE" # To target the default namespace, use "__default__". curl "https://$INDEX_HOST/namespaces/$NAMESPACE" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" ``` ```javascript JavaScript { name: 'example-namespace', recordCount: '20000' } ``` ```java Java name: "example-namespace" record_count: 20000 ``` ```csharp C# {"name":"example-namespace","recordCount":20000} ``` ```json curl { "name": "example-namespace", "record_count": 20000 } ``` # Fetch vectors Source: https://docs.pinecone.io/reference/api/2025-04/data-plane/fetch db_data_2025-04.oas get /vectors/fetch Look up and return vectors by ID from a single namespace. The returned vectors include the vector data and/or metadata. For guidance and examples, see [Fetch data](https://docs.pinecone.io/guides/manage-data/fetch-data). ```python Python # pip install "pinecone[grpc]" from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key="YOUR_API_KEY") # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index index = pc.Index(host="INDEX_HOST") index.fetch(ids=["id-1", "id-2"], namespace="example-namespace") ``` ```javascript JavaScript // npm install @pinecone-database/pinecone import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }) // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index const index = pc.index("INDEX_NAME", "INDEX_HOST") const fetchResult = await index.namespace('example-namespace').fetch(['id-1', 'id-2']); ``` ```java Java import io.pinecone.clients.Index; import io.pinecone.configs.PineconeConfig; import io.pinecone.configs.PineconeConnection; import io.pinecone.proto.FetchResponse; import java.util.Arrays; import java.util.List; public class FetchExample { public static void main(String[] args) { PineconeConfig config = new PineconeConfig("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index config.setHost("INDEX_HOST"); PineconeConnection connection = new PineconeConnection(config); Index index = new Index(connection, "INDEX_NAME"); List ids = Arrays.asList("id-1", "id-2"); FetchResponse fetchResponse = index.fetch(ids, "example-namespace"); System.out.println(fetchResponse); } } ``` ```go Go package main import ( "context" "encoding/json" "fmt" "log" "github.com/pinecone-io/go-pinecone/v2/pinecone" ) func prettifyStruct(obj interface{}) string { bytes, _ := json.MarshalIndent(obj, "", " ") return string(bytes) } func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "INDEX_HOST", Namespace: "example-namespace"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v", err) } res, err := idxConnection.FetchVectors(ctx, []string{"id-1", "id-2"}) if err != nil { log.Fatalf("Failed to fetch vectors: %+v", err) } else { fmt.Printf(prettifyStruct(res)) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index var index = pinecone.Index(host: "INDEX_HOST"); var fetchResponse = await index.FetchAsync(new FetchRequest { Ids = new List { "id-1", "id-2" }, Namespace = "example-namespace", }); Console.WriteLine(fetchResponse); ``` ```shell curl # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index PINECONE_API_KEY="YOUR_API_KEY" INDEX_HOST="INDEX_HOST" curl -X GET "https://$INDEX_HOST/vectors/fetch?ids=id-1&ids=id-2&namespace=example-namespace" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" ``` ```Python Python {'namespace': 'example-namespace', 'usage': {'readUnits': 1}, 'vectors': {'id-1': {'id': 'id-1', 'values': [0.568879, 0.632687092, 0.856837332, ...]}, 'id-2': {'id': 'id-2', 'values': [0.00891787093, 0.581895, 0.315718859, ...]}}} ``` ```JavaScript JavaScript {'namespace': 'example-namespace', 'usage': {'readUnits': 1}, 'records': {'id-1': {'id': 'id-1', 'values': [0.568879, 0.632687092, 0.856837332, ...]}, 'id-2': {'id': 'id-2', 'values': [0.00891787093, 0.581895, 0.315718859, ...]}}} ``` ```java Java namespace: "example-namespace" vectors { key: "id-1" value { id: "id-1" values: 0.568879 values: 0.632687092 values: 0.856837332 ... } } vectors { key: "id-2" value { id: "id-2" values: 0.00891787093 values: 0.581895 values: 0.315718859 ... } } usage { read_units: 1 } ``` ```go Go { "vectors": { "id-1": { "id": "id-1", "values": [ -0.0089730695, -0.020010853, -0.0042787646, ... ] }, "id-2": { "id": "id-2", "values": [ -0.005380766, 0.00215196, -0.014833462, ... ] } }, "usage": { "read_units": 1 } } ``` ```csharp C# { "vectors": { "id-1": { "id": "id-1", "values": [ -0.0089730695, -0.020010853, -0.0042787646, ... ], "sparseValues": null, "metadata": null }, "vec1": { "id": "id-2", "values": [ -0.005380766, 0.00215196, -0.014833462, ... ], "sparseValues": null, "metadata": null } }, "namespace": "example-namespace", "usage": { "readUnits": 1 } ``` ```json curl { "vectors": { "id-1": { "id": "id-1", "values": [0.568879, 0.632687092, 0.856837332, ...] }, "id-2": { "id": "id-2", "values": [0.00891787093, 0.581895, 0.315718859, ...] } }, "namespace": "example-namespace", "usage": {"readUnits": 1}, } ``` # List vector IDs Source: https://docs.pinecone.io/reference/api/2025-04/data-plane/list db_data_2025-04.oas get /vectors/list List the IDs of vectors in a single namespace of a serverless index. An optional prefix can be passed to limit the results to IDs with a common prefix. Returns up to 100 IDs at a time by default in sorted order (bitwise "C" collation). If the `limit` parameter is set, `list` returns up to that number of IDs instead. Whenever there are additional IDs to return, the response also includes a `pagination_token` that you can use to get the next batch of IDs. When the response does not include a `pagination_token`, there are no more IDs to return. For guidance and examples, see [List record IDs](https://docs.pinecone.io/guides/manage-data/list-record-ids). **Note:** `list` is supported only for serverless indexes. ```python Python # pip install "pinecone[grpc]" from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key='YOUR_API_KEY') # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index index = pc.Index(host="INDEX_HOST") # Implicit pagination using a generator function for ids in index.list(prefix="doc1#", namespace="example-namespace"): print(ids) # Manual pagination results = index.list_paginated( prefix="doc1#", limit=3, namespace="example-namespace", pagination_token="eyJza2lwX3Bhc3QiOiIxMDEwMy0=" ) print(results) ``` ```javascript JavaScript // npm install @pinecone-database/pinecone import { Pinecone } from '@pinecone-database/pinecone'; const pc = new Pinecone(); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index const index = pc.index("INDEX_NAME", "INDEX_HOST").namespace("example-namespace"); const results = await index.listPaginated({ prefix: 'doc1#', limit: 3 }); console.log(results); // Fetch the next page of results await index.listPaginated({ prefix: 'doc1#', paginationToken: results.pagination.next}); ``` ```java Java import io.pinecone.clients.Index; import io.pinecone.configs.PineconeConfig; import io.pinecone.configs.PineconeConnection; import io.pinecone.proto.ListResponse; public class ListExample { public static void main(String[] args) { PineconeConfig config = new PineconeConfig("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index config.setHost("INDEX_HOST"); PineconeConnection connection = new PineconeConnection(config); Index index = new Index(connection, "INDEX_NAME"); ListResponse listResponse = index.list("example-namespace", "doc1#", 3); System.out.println(listResponse); } } ``` ```go Go package main import ( "context" "encoding/json" "fmt" "log" "github.com/pinecone-io/go-pinecone/v2/pinecone" ) func prettifyStruct(obj interface{}) string { bytes, _ := json.MarshalIndent(obj, "", " ") return string(bytes) } func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "INDEX_HOST", Namespace: "example-namespace"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v", err) } limit := uint32(3) prefix := "doc1#" res, err := idxConnection.ListVectors(ctx, &pinecone.ListVectorsRequest{ Limit: &limit, Prefix: &prefix, }) if len(res.VectorIds) == 0 { fmt.Println("No vectors found") } else { fmt.Printf(prettifyStruct(res)) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index var index = pinecone.Index(host: "INDEX_HOST"); var listResponse = await index.ListAsync(new ListRequest { Namespace = "example-namespace", Prefix = "doc1#", Limit = 3, }); Console.WriteLine(listResponse); ``` ```shell curl # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index PINECONE_API_KEY="YOUR_API_KEY" INDEX_HOST="INDEX_HOST" curl -X GET "https://$INDEX_HOST/vectors/list?namespace=example-namespace&prefix=doc1#&limit=3" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" ``` ```python Python # Implicit pagination ['doc1#chunk1', 'doc1#chunk2', 'doc1#chunk3', 'doc1chunk4', ...] # Manual pagination {'namespace': 'example-namespace', 'pagination': {'next': 'eyJza2lwX3Bhc3QiOiJkb2MxI2NodW5rMiIsInByZWZpeCI6ImRvYzEjIn0='}, 'usage': {'read_units': 1}, 'vectors': [{'id': 'doc1#chunk1'}, {'id': 'doc1#chunk2'}, {'id': 'doc1#chunk3'}]} ``` ```js JavaScript { vectors: [ { id: 'doc1#chunk1' }, { id: 'doc1#chunk2' }, { id: 'doc1#chunk3' } ], pagination: { next: 'eyJza2lwX3Bhc3QiOiJwcmVUZXN0LS04MCIsInByZWZpeCI6InByZVRlc3QifQ==' }, namespace: 'example-namespace', usage: { readUnits: 1 } } ``` ```java Java vectors { id: "doc1#chunk1" } vectors { id: "doc1#chunk2" } vectors { id: "doc1#chunk3" } pagination { next: "eyJza2lwX3Bhc3QiOiJhbHN0cm9lbWVyaWEtcGVydXZpYW4iLCJwcmVmaXgiOm51bGx9" } namespace: "example-namespace" usage { read_units: 1 } ``` ```go Go { "vector_ids": [ "doc1#chunk1", "doc1#chunk2", "doc1#chunk3" ], "usage": { "read_units": 1 }, "next_pagination_token": "eyJza2lwX3Bhc3QiOiIwMDBkMTc4OC0zMDAxLTQwZmMtYjZjNC0wOWI2N2I5N2JjNDUiLCJwcmVmaXgiOm51bGx9" } ``` ```csharp C# { "vectors": [ { "id": "doc1#chunk1" }, { "id": "doc1#chunk2" }, { "id": "doc1#chunk3" } ], "pagination": "eyJza2lwX3Bhc3QiOiIwMDBkMTc4OC0zMDAxLTQwZmMtYjZjNC0wOWI2N2I5N2JjNDUiLCJwcmVmaXgiOm51bGx9", "namespace": "example-namespace", "usage": { "readUnits": 1 } } ``` ```json curl { "vectors": [ { "id": "doc1#chunk1" }, { "id": "doc1#chunk2" }, { "id": "doc1#chunk3" } ], "pagination": { "next": "c2Vjb25kY2FsbA==" }, "namespace": "example-namespace", "usage": { "readUnits": 1 } } ``` # List imports Source: https://docs.pinecone.io/reference/api/2025-04/data-plane/list_imports db_data_2025-04.oas get /bulk/imports List all recent and ongoing import operations. By default, `list_imports` returns up to 100 imports per page. If the `limit` parameter is set, `list` returns up to that number of imports instead. Whenever there are additional IDs to return, the response also includes a `pagination_token` that you can use to get the next batch of imports. When the response does not include a `pagination_token`, there are no more imports to return. For guidance and examples, see [Import data](https://docs.pinecone.io/guides/index-data/import-data). This feature is in [public preview](/release-notes/feature-availability) and available only on [Standard and Enterprise plans](https://www.pinecone.io/pricing/). ```python Python from pinecone import Pinecone, ImportErrorMode pc = Pinecone(api_key="YOUR_API_KEY") # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index index = pc.Index(host="INDEX_HOST") # List using a generator that handles pagination for i in index.list_imports(): print(f"id: {i.id} status: {i.status}") # List using a generator that fetches all results at once operations = list(index.list_imports()) print(operations) ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone'; const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index const index = pc.index("INDEX_NAME", "INDEX_HOST") const results = await index.listImports({ limit: 10, paginationToken: 'Tm90aGluZyB0byBzZWUgaGVyZQo' }); console.log(results); ``` ```java Java import io.pinecone.clients.Pinecone; import io.pinecone.clients.AsyncIndex; import org.openapitools.db_data.client.ApiException; import org.openapitools.db_data.client.model.ListImportsResponse; public class ListImports { public static void main(String[] args) throws ApiException { // Initialize a Pinecone client with your API key Pinecone pinecone = new Pinecone.Builder("YOUR_API_KEY").build(); // Get async imports connection object AsyncIndex asyncIndex = pinecone.getAsyncIndexConnection("docs-example"); // List imports ListImportsResponse response = asyncIndex.listImports(10, "Tm90aGluZyB0byBzZWUgaGVyZQo"); System.out.println(response); } } ``` ```go Go package main import ( "context" "fmt" "log" "github.com/pinecone-io/go-pinecone/v2/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "INDEX_HOST"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v", err) } limit := int32(10) firstImportPage, err := idxConnection.ListImports(ctx, &limit, nil) if err != nil { log.Fatalf("Failed to list imports: %v", err) } fmt.Printf("First page of imports: %+v", firstImportPage.Imports) paginationToken := firstImportPage.NextPaginationToken nextImportPage, err := idxConnection.ListImports(ctx, &limit, paginationToken) if err != nil { log.Fatalf("Failed to list imports: %v", err) } fmt.Printf("Second page of imports: %+v", nextImportPage.Imports) } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index var index = pinecone.Index(host: "INDEX_HOST"); var imports = await index.ListBulkImportsAsync(new ListBulkImportsRequest { Limit = 10, PaginationToken = "Tm90aGluZyB0byBzZWUgaGVyZQo" }); ``` ```bash curl # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index PINECONE_API_KEY="YOUR_API_KEY" INDEX_HOST="INDEX_HOST" curl -X GET "https://$INDEX_HOST/bulk/imports?paginationToken==Tm90aGluZyB0byBzZWUgaGVyZQo" \ -H 'Api-Key: $YOUR_API_KEY' \ -H 'X-Pinecone-API-Version: 2025-04' ``` ```json { "data": [ { "id": "1", "uri": "s3://BUCKET_NAME/PATH/TO/DIR", "status": "Pending", "started_at": "2024-08-19T20:49:00.754Z", "finished_at": "2024-08-19T20:49:00.754Z", "percent_complete": 42.2, "records_imported": 1000000 } ], "pagination": { "next": "Tm90aGluZyB0byBzZWUgaGVyZQo=" } } ``` # List namespaces Source: https://docs.pinecone.io/reference/api/2025-04/data-plane/listnamespaces db_data_2025-04.oas get /namespaces Get a list of all [namespaces](https://docs.pinecone.io/guides/index-data/indexing-overview#namespaces) in a serverless index. Up to 100 namespaces are returned at a time by default, in sorted order (bitwise “C” collation). If the `limit` parameter is set, up to that number of namespaces are returned instead. Whenever there are additional namespaces to return, the response also includes a `pagination_token` that you can use to get the next batch of namespaces. When the response does not include a `pagination_token`, there are no more namespaces to return. ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone'; const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }) // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index const index = pc.index("INDEX_NAME", "INDEX_HOST") const namespaceList = await index.listNamespaces(); console.log(namespaceList); ``` ```java Java import io.pinecone.clients.AsyncIndex; import io.pinecone.clients.Index; import io.pinecone.configs.PineconeConfig; import io.pinecone.configs.PineconeConnection; import io.pinecone.proto.ListNamespacesResponse; import org.openapitools.db_data.client.ApiException; public class Namespaces { public static void main(String[] args) throws ApiException { PineconeConfig config = new PineconeConfig("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index config.setHost("INDEX_HOST"); PineconeConnection connection = new PineconeConnection(config); Index index = new Index(config, connection, "docs-example"); // List all namespaces with default pagination limit (100) ListNamespacesResponse listNamespacesResponse = index.listNamespaces(null, null); // List all namespaces with pagination limit of 2 ListNamespacesResponse listNamespacesResponseWithLimit = index.listNamespaces(null,2); // List all namespaces with pagination limit and token ListNamespacesResponse listNamespacesResponsePaginated = index.listNamespaces("eyJza2lwX3Bhc3QiOiIxMDEwMy0=", 5); System.out.println(restoreJobListWithLimit); } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("PINECONE_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index var index = pinecone.Index(host: "INDEX_HOST"); var namespaces = await index.ListNamespacesAsync(new ListNamespacesRequest()); Console.WriteLine(namespaces); ``` ```bash curl # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index PINECONE_API_KEY="YOUR_API_KEY" INDEX_HOST="INDEX_HOST" curl "https://$INDEX_HOST/namespaces" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" ``` ```javascript JavaScript { namespaces: [ { name: 'example-namespace', recordCount: '20000' }, { name: 'example-namespace2', recordCount: '10500' } ], pagination: "Tm90aGluZyB0byBzZWUgaGVyZQo=" } ``` ```java Java namespaces { name: "example-namespace" record_count: 20000 } namespaces { name: "example-namespace2" record_count: 10500 } ``` ```csharp C# { "namespaces":[ {"name":"example-namespace","recordCount":20000}, {"name":"example-namespace2","recordCount":10500}, ... ], "pagination":"Tm90aGluZyB0byBzZWUgaGVyZQo=" } ``` ```json curl { "namespaces": [ { "name": "example-namespace", "record_count": 20000 }, { "name": "example-namespace-2", "record_count": 10500 } ], "pagination": { "next": "Tm90aGluZyB0byBzZWUgaGVyZQo=" } } ``` # Search with a vector Source: https://docs.pinecone.io/reference/api/2025-04/data-plane/query db_data_2025-04.oas post /query Search a namespace using a query vector. It retrieves the ids of the most similar items in a namespace, along with their similarity scores. For guidance and examples, see [Search](https://docs.pinecone.io/guides/search/semantic-search). ```python Python # pip install "pinecone[grpc]" from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key="YOUR_API_KEY") # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index index = pc.Index(host="INDEX_HOST") index.query( namespace="example-namespace", vector=[0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3], filter={ "genre": {"$eq": "documentary"} }, top_k=3, include_values=True ) ``` ```javascript JavaScript // npm install @pinecone-database/pinecone import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }) // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index const index = pc.index("INDEX_NAME", "INDEX_HOST") const queryResponse = await index.namespace('example-namespace').query({ vector: [0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3], filter: { 'genre': {'$eq': 'documentary'} }, topK: 3, includeValues: true }); ``` ```java Java import com.google.protobuf.Struct; import com.google.protobuf.Value; import io.pinecone.clients.Index; import io.pinecone.configs.PineconeConfig; import io.pinecone.configs.PineconeConnection; import io.pinecone.unsigned_indices_model.QueryResponseWithUnsignedIndices; import java.util.Arrays; import java.util.List; public class QueryExample { public static void main(String[] args) { PineconeConfig config = new PineconeConfig("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index config.setHost("INDEX_HOST"); PineconeConnection connection = new PineconeConnection(config); Index index = new Index(connection, "INDEX_NAME"); List query = Arrays.asList(0.3f, 0.3f, 0.3f, 0.3f, 0.3f, 0.3f, 0.3f, 0.3f); Struct filter = Struct.newBuilder() .putFields("genre", Value.newBuilder() .setStructValue(Struct.newBuilder() .putFields("$eq", Value.newBuilder() .setStringValue("documentary") .build())) .build()) .build(); QueryResponseWithUnsignedIndices queryResponse = index.query(3, query, null, null, null, "example-namespace", filter, false, true); System.out.println(queryResponse); } } ``` ```go Go package main import ( "context" "encoding/json" "fmt" "log" "github.com/pinecone-io/go-pinecone/v2/pinecone" "google.golang.org/protobuf/types/known/structpb" ) func prettifyStruct(obj interface{}) string { bytes, _ := json.MarshalIndent(obj, "", " ") return string(bytes) } func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "INDEX_HOST", Namespace: "example-namespace"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v", err) } queryVector := []float32{0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3} metadataMap := map[string]interface{}{ "genre": map[string]interface{}{ "$eq": "documentary", }, } metadataFilter, err := structpb.NewStruct(metadataMap) if err != nil { log.Fatalf("Failed to create metadata map: %v", err) } res, err := idxConnection.QueryByVectorValues(ctx, &pinecone.QueryByVectorValuesRequest{ Vector: queryVector, TopK: 3, MetadataFilter: metadataFilter, IncludeValues: true, }) if err != nil { log.Fatalf("Error encountered when querying by vector: %v", err) } else { fmt.Printf(prettifyStruct(res)) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index var index = pinecone.Index(host: "INDEX_HOST"); var queryResponse = await index.QueryAsync(new QueryRequest { Vector = new[] { 0.3f, 0.3f, 0.3f, 0.3f, 0.3f, 0.3f, 0.3f, 0.3f }, Namespace = "example-namespace", TopK = 3, Filter = new Metadata { ["genre"] = new Metadata { ["$eq"] = "documentary", } } }); Console.WriteLine(queryResponse); ``` ```shell curl PINECONE_API_KEY="YOUR_API_KEY" INDEX_HOST="INDEX_HOST" curl "https://$INDEX_HOST/query" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "Content-Type: application/json" \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "namespace": "example-namespace", "vector": [0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3], "filter": {"genre": {"$eq": "documentary"}}, "topK": 3, "includeValues": true }' ``` ```shell { "matches":[ { "id": "vec3", "score": 0, "values": [0.3,0.3,0.3,0.3,0.3,0.3,0.3,0.3] }, { "id": "vec2", "score": 0.0800000429, "values": [0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2] }, { "id": "vec4", "score": 0.0799999237, "values": [0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4] } ], "namespace": "example-namespace", "usage": {"read_units": 6} } ``` # Search with text Source: https://docs.pinecone.io/reference/api/2025-04/data-plane/search_records db_data_2025-04.oas post /records/namespaces/{namespace}/search Search a namespace with a query text, query vector, or record ID and return the most similar records, along with their similarity scores. Optionally, rerank the initial results based on their relevance to the query. Searching with text is supported only for [indexes with integrated embedding](https://docs.pinecone.io/guides/indexes/create-an-index#integrated-embedding). Searching with a query vector or record ID is supported for all indexes. For guidance and examples, see [Search](https://docs.pinecone.io/guides/search/semantic-search). ```python Python from pinecone import Pinecone pc = Pinecone(api_key="YOUR_API_KEY") # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index index = pc.Index("docs-example") # Search with a query text and rerank the results # Supported only for indexes with integrated embedding search_with_text = index.search( namespace="example-namespace", query={ "inputs": {"text": "Disease prevention"}, "top_k": 4 }, fields=["category", "chunk_text"], rerank={ "model": "bge-reranker-v2-m3", "top_n": 2, "rank_fields": ["chunk_text"] # Specified field must also be included in 'fields' } ) print(search_with_text) # Search with a query vector and rerank the results search_with_vector = index.search( namespace="example-namespace", query={ "vector": { "values": [0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3] }, "top_k": 4 }, fields=["category", "chunk_text"], rerank={ "query": "Disease prevention", "model": "bge-reranker-v2-m3", "top_n": 2, "rank_fields": ["chunk_text"] # Specified field must also be included in 'fields' } ) print(search_with_vector) # Search with a record ID and rerank the results search_with_id = index.search( namespace="example-namespace", query={ "id": "rec1", "top_k": 4 }, fields=["category", "chunk_text"], rerank={ "query": "Disease prevention", "model": "bge-reranker-v2-m3", "top_n": 2, "rank_fields": ["chunk_text"] # Specified field must also be included in 'fields' } ) print(search_with_id) ``` ```javascript JavaScript // npm install @pinecone-database/pinecone import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: "YOUR_API_KEY" }) // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index const namespace = pc.index("INDEX_NAME", "INDEX_HOST").namespace("example-namespace"); // Search with a query text and rerank the results // Supported only for indexes with integrated embedding const searchWithText = await namespace.searchRecords({ query: { topK: 4, inputs: { text: 'Disease prevention' }, }, fields: ['chunk_text', 'category'], rerank: { model: 'bge-reranker-v2-m3', rankFields: ['chunk_text'], topN: 2, }, }); console.log(searchWithText); // Search with a query vector and rerank the results const searchWithVector = await namespace.searchRecords({ query: { topK: 4, vector: { values: [0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3] }, inputs: { text: 'Disease prevention' }, }, fields: ['chunk_text', 'category'], rerank: { query: "Disease prevention", model: 'bge-reranker-v2-m3', rankFields: ['chunk_text'], topN: 2, }, }); console.log(searchWithVector); // Search with a record ID and rerank the results const searchWithId = await namespace.searchRecords({ query: { topK: 4, id: 'rec1', }, fields: ['chunk_text', 'category'], rerank: { query: "Disease prevention", model: 'bge-reranker-v2-m3', rankFields: ['chunk_text'], topN: 2, }, }); console.log(searchWithId); ``` ```java Java import io.pinecone.clients.Index; import io.pinecone.configs.PineconeConfig; import io.pinecone.configs.PineconeConnection; import org.openapitools.db_data.client.ApiException; import org.openapitools.db_data.client.model.SearchRecordsRequestRerank; import org.openapitools.db_data.client.model.SearchRecordsResponse; import org.openapitools.db_data.client.model.SearchRecordsVector; import java.util.*; public class SearchText { public static void main(String[] args) throws ApiException { PineconeConfig config = new PineconeConfig("YOUR_API_KEY"); config.setHost("INDEX_HOST"); PineconeConnection connection = new PineconeConnection(config); Index index = new Index(config, connection, "integrated-dense-java"); String query = "Famous historical structures and monuments"; List fields = new ArrayList<>(); fields.add("category"); fields.add("chunk_text"); ListrankFields = new ArrayList<>(); rankFields.add("chunk_text"); SearchRecordsRequestRerank rerank = new SearchRecordsRequestRerank() .query(query) .model("bge-reranker-v2-m3") .topN(2) .rankFields(rankFields); // Search with a query text and rerank the results // Supported only for indexes with integrated embedding SearchRecordsResponse searchWithText = index.searchRecordsByText(query, "example-namespace", fields, 10, null, rerank); System.out.println(searchWithText); // Search with a query vector and rerank the results SearchRecordsVector queryVector = new SearchRecordsVector(); queryVector.setValues(Arrays.asList(0.3f, 0.3f, 0.3f, 0.3f, 0.3f, 0.3f, 0.3f, 0.3f)); SearchRecordsResponse searchWithVector = index.searchRecordsByVector(queryVector, "example-namespace", fields, 4, null, rerank); System.out.println(searchWithVector); // Search with a record ID and rerank the results SearchRecordsResponse searchWithID = index.searchRecordsById("rec1", "example-namespace", fields, 4, null, rerank); System.out.println(searchWithID); } } ``` ```go Go package main import ( "context" "encoding/json" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func prettifyStruct(obj interface{}) string { bytes, _ := json.MarshalIndent(obj, "", " ") return string(bytes) } func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "INDEX_HOST", Namespace: "example-namespace"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v", err) } // Search with a query text and rerank the results // Supported only for indexes with integrated embedding topN := int32(2) searchWithText, err := idxConnection.SearchRecords(ctx, &pinecone.SearchRecordsRequest{ Query: pinecone.SearchRecordsQuery{ TopK: 4, Inputs: &map[string]interface{}{ "text": "Disease prevention", }, }, Rerank: &pinecone.SearchRecordsRerank{ Model: "bge-reranker-v2-m3", TopN: &topN, RankFields: []string{"chunk_text"}, }, Fields: &[]string{"chunk_text", "category"}, }) if err != nil { log.Fatalf("Failed to search records: %v", err) } fmt.Printf(prettifyStruct(searchWithText)) // Search with a query vector and rerank the results topN := int32(2) searchWithVector, err := idxConnection.SearchRecords(ctx, &pinecone.SearchRecordsRequest{ Query: pinecone.SearchRecordsQuery{ TopK: 4, Vector: pinecone.SearchRecordsVector{ Values: []float32{0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3}, }, }, Rerank: &pinecone.SearchRecordsRerank{ Model: "bge-reranker-v2-m3", TopN: &topN, RankFields: []string{"chunk_text"}, }, Fields: &[]string{"chunk_text", "category"}, }) if err != nil { log.Fatalf("Failed to search records: %v", err) } fmt.Printf(prettifyStruct(resSearchWithVector)) // Search with a query ID and rerank the results topN := int32(2) searchWithId, err := idxConnection.SearchRecords(ctx, &pinecone.SearchRecordsRequest{ Query: pinecone.SearchRecordsQuery{ TopK: 4, Id: "rec1", }, Rerank: &pinecone.SearchRecordsRerank{ Model: "bge-reranker-v2-m3", TopN: &topN, RankFields: []string{"chunk_text"}, }, Fields: &[]string{"chunk_text", "category"}, }) if err != nil { log.Fatalf("Failed to search records: %v", err) } fmt.Printf(prettifyStruct(searchWithId)) } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); var index = pinecone.Index(host: "INDEX_HOST"); // Search with a query text and rerank the results var searchWithText = await index.SearchRecordsAsync( "example-namespace", new SearchRecordsRequest { Query = new SearchRecordsRequestQuery { TopK = 4, Inputs = new Dictionary { { "text", "Disease prevention" } }, }, Fields = ["category", "chunk_text"], Rerank = new SearchRecordsRequestRerank { Model = "bge-reranker-v2-m3", TopN = 2, RankFields = ["chunk_text"], }, } ); Console.WriteLine(searchWithText); // Search with a query vector and rerank the results var searchWithVector = await index.SearchRecordsAsync( "example-namespace", new SearchRecordsRequest { Query = new SearchRecordsRequestQuery { TopK = 4, Vector = new SearchRecordsVector { Values = new float[] { 0.3f, 0.3f, 0.3f, 0.3f, 0.3f, 0.3f, 0.3f, 0.3f }, }, }, Fields = ["category", "chunk_text"], Rerank = new SearchRecordsRequestRerank { Model = "bge-reranker-v2-m3", TopN = 2, RankFields = ["chunk_text"], }, } ); Console.WriteLine(searchWithVector); // Search with a query ID and rerank the results var searchWithId = await index.SearchRecordsAsync( "example-namespace", new SearchRecordsRequest { Query = new SearchRecordsRequestQuery { TopK = 4, Id = "rec1", }, Fields = ["category", "chunk_text"], Rerank = new SearchRecordsRequestRerank { Model = "bge-reranker-v2-m3", TopN = 2, RankFields = ["chunk_text"], }, } ); Console.WriteLine(searchWithId); ``` ```shell curl INDEX_HOST="INDEX_HOST" NAMESPACE="YOUR_NAMESPACE" PINECONE_API_KEY="YOUR_API_KEY" # Search with a query text and rerank the results # Supported only for indexes with integrated embedding curl "https://$INDEX_HOST/records/namespaces/$NAMESPACE/search" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-01" \ -d '{ "query": { "inputs": {"text": "Disease prevention"}, "top_k": 4, }, "fields": ["category", "chunk_text"] "rerank": { "model": "bge-reranker-v2-m3", "top_n": 2, "rank_fields": ["chunk_text"] # Specified field must also be included in 'fields' } }' # Search with a query vector and rerank the results curl "https://$INDEX_HOST/records/namespaces/$NAMESPACE/search" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-01" \ -d '{ "query": { "vector": { "values": [0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3] }, "top_k": 4, }, "fields": ["category", "chunk_text"] "rerank": { "query": "Disease prevention", "model": "bge-reranker-v2-m3", "top_n": 2, "rank_fields": ["chunk_text"] # Specified field must also be included in 'fields' } }' # Search with a query text and rerank the results # Supported only for indexes with integrated embedding curl "https://$INDEX_HOST/records/namespaces/$NAMESPACE/search" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-01" \ -d '{ "query": { "id": "rec1", "top_k": 4, }, "fields": ["category", "chunk_text"] "rerank": { "query": "Disease prevention", "model": "bge-reranker-v2-m3", "top_n": 2, "rank_fields": ["chunk_text"] } }' ``` ```python Python {'result': {'hits': [{'_id': 'rec3', '_score': 0.004399413242936134, 'fields': {'category': 'immune system', 'chunk_text': 'Rich in vitamin C and other ' 'antioxidants, apples ' 'contribute to immune health ' 'and may reduce the risk of ' 'chronic diseases.'}}, {'_id': 'rec4', '_score': 0.0029235430993139744, 'fields': {'category': 'endocrine system', 'chunk_text': 'The high fiber content in ' 'apples can also help regulate ' 'blood sugar levels, making ' 'them a favorable snack for ' 'people with diabetes.'}}]}, 'usage': {'embed_total_tokens': 8, 'read_units': 6, 'rerank_units': 1}} ``` ```javascript JavaScript { result: { hits: [ { _id: 'rec3', _score: 0.004399413242936134, fields: { category: 'immune system', chunk_text: 'Rich in vitamin C and other antioxidants, apples contribute to immune health and may reduce the risk of chronic diseases.' } }, { _id: 'rec4', _score: 0.0029235430993139744, fields: { category: 'endocrine system', chunk_text: 'The high fiber content in apples can also help regulate blood sugar levels, making them a favorable snack for people with diabetes.' } } ] }, usage: { readUnits: 6, embedTotalTokens: 8, rerankUnits: 1 } } ``` ```java Java class SearchRecordsResponse { result: class SearchRecordsResponseResult { hits: [class Hit { id: rec3 score: 0.004399413242936134 fields: {category=immune system, chunk_text=Rich in vitamin C and other antioxidants, apples contribute to immune health and may reduce the risk of chronic diseases.} additionalProperties: null }, class Hit { id: rec4 score: 0.0029235430993139744 fields: {category=endocrine system, chunk_text=The high fiber content in apples can also help regulate blood sugar levels, making them a favorable snack for people with diabetes.} additionalProperties: null }] additionalProperties: null } usage: class SearchUsage { readUnits: 6 embedTotalTokens: 13 rerankUnits: 1 additionalProperties: null } additionalProperties: null } ``` ```go Go { "result": { "hits": [ { "_id": "rec3", "_score": 0.004399413242936134, "fields": { "category": "immune system", "chunk_text": "Rich in vitamin C and other antioxidants, apples contribute to immune health and may reduce the risk of chronic diseases." } }, { "_id": "rec4", "_score": 0.0029235430993139744, "fields": { "category": "endocrine system", "chunk_text": "The high fiber content in apples can also help regulate blood sugar levels, making them a favorable snack for people with diabetes." } } ] }, "usage": { "read_units": 6, "embed_total_tokens": 8, "rerank_units": 1 } } ``` ```csharp C# { "result": { "hits": [ { "_id": "rec3", "_score": 0.13741668, "fields": { "category": "immune system", "chunk_text": "Rich in vitamin C and other antioxidants, apples contribute to immune health and may reduce the risk of chronic diseases." } }, { "_id": "rec1", "_score": 0.0023413408, "fields": { "category": "digestive system", "chunk_text": "Apples are a great source of dietary fiber, which supports digestion and helps maintain a healthy gut." } } ] }, "usage": { "read_units": 6, "embed_total_tokens": 5, "rerank_units": 1 } } ``` ```json curl { "result": { "hits": [ { "_id": "rec3", "_score": 0.004433765076100826, "fields": { "category": "immune system", "chunk_text": "Rich in vitamin C and other antioxidants, apples contribute to immune health and may reduce the risk of chronic diseases." } }, { "_id": "rec4", "_score": 0.0029121784027665854, "fields": { "category": "endocrine system", "chunk_text": "The high fiber content in apples can also help regulate blood sugar levels, making them a favorable snack for people with diabetes." } } ] }, "usage": { "embed_total_tokens": 8, "read_units": 6, "rerank_units": 1 } } ``` # Start import Source: https://docs.pinecone.io/reference/api/2025-04/data-plane/start_import db_data_2025-04.oas post /bulk/imports Start an asynchronous import of vectors from object storage into an index. For guidance and examples, see [Import data](https://docs.pinecone.io/guides/index-data/import-data). This feature is in [public preview](/release-notes/feature-availability) and available only on [Standard and Enterprise plans](https://www.pinecone.io/pricing/). ```python Python from pinecone import Pinecone, ImportErrorMode pc = Pinecone(api_key="YOUR_API_KEY") # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index index = pc.Index(host="INDEX_HOST") root = "s3://BUCKET_NAME/PATH/TO/DIR" index.start_import( uri=root, error_mode=ImportErrorMode.CONTINUE, # or ImportErrorMode.ABORT integration_id="a12b3d4c-47d2-492c-a97a-dd98c8dbefde" # Optional for public buckets ) ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone'; const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index const index = pc.index("INDEX_NAME", "INDEX_HOST") const storageURI = 's3://BUCKET_NAME/PATH/TO/DIR'; const errorMode = 'continue'; // or 'abort' const integrationID = 'a12b3d4c-47d2-492c-a97a-dd98c8dbefde'; // Optional for public buckets await index.startImport(storageURI, errorMode, integrationID); ``` ```java Java import io.pinecone.clients.Pinecone; import io.pinecone.clients.AsyncIndex; import org.openapitools.db_data.client.ApiException; import org.openapitools.db_data.client.model.ImportErrorMode; import org.openapitools.db_data.client.model.StartImportResponse; public class StartImport { public static void main(String[] args) throws ApiException { // Initialize a Pinecone client with your API key Pinecone pinecone = new Pinecone.Builder("YOUR_API_KEY").build(); // Get async imports connection object AsyncIndex asyncIndex = pinecone.getAsyncIndexConnection("docs-example"); // s3 uri String uri = "s3://BUCKET_NAME/PATH/TO/DIR"; // Integration ID (optional for public buckets) String integrationId = "a12b3d4c-47d2-492c-a97a-dd98c8dbefde"; // Start an import StartImportResponse response = asyncIndex.startImport(uri, integrationId, ImportErrorMode.OnErrorEnum.CONTINUE); } } ``` ```go Go package main import ( "context" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "INDEX_HOST"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v", err) } uri := "s3://BUCKET_NAME/PATH/TO/DIR" errorMode := "continue" // or "abort" importRes, err := idxConnection.StartImport(ctx, uri, nil, (*pinecone.ImportErrorMode)(&errorMode)) if err != nil { log.Fatalf("Failed to start import: %v", err) } fmt.Printf("Import started with ID: %s", importRes.Id) } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index var index = pinecone.Index(host: "INDEX_HOST"); var uri = "s3://BUCKET_NAME/PATH/TO/DIR"; var response = await index.StartBulkImportAsync(new StartImportRequest { Uri = uri, IntegrationId = "a12b3d4c-47d2-492c-a97a-dd98c8dbefde", ErrorMode = new ImportErrorMode { OnError = ImportErrorModeOnError.Continue } }); ``` ```bash curl # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index PINECONE_API_KEY="YOUR_API_KEY" INDEX_HOST="INDEX_HOST" curl "https://$INDEX_HOST/bulk/imports" \ -H 'Api-Key: $YOUR_API_KEY' \ -H 'Content-Type: application/json' \ -H 'X-Pinecone-API-Version: 2025-04' \ -d '{ "integrationId": "a12b3d4c-47d2-492c-a97a-dd98c8dbefde", "uri": "s3://BUCKET_NAME/PATH/TO/DIR", "errorMode": { "onError": "continue" } }' ``` ```json { "operation_id": "101" } ``` # Update a vector Source: https://docs.pinecone.io/reference/api/2025-04/data-plane/update db_data_2025-04.oas post /vectors/update Update a vector in a namespace. If a value is included, it will overwrite the previous value. If a `set_metadata` is included, the values of the fields specified in it will be added or overwrite the previous value. For guidance and examples, see [Update data](https://docs.pinecone.io/guides/manage-data/update-data). ```python Python # pip install "pinecone[grpc]" from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key="YOUR_API_KEY") # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index index = pc.Index(host="INDEX_HOST") index.update( id="id-3", values=[4.0, 2.0], set_metadata={"genre": "comedy"}, namespace="example-namespace" ) ``` ```javascript JavaScript // npm install @pinecone-database/pinecone import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: "YOUR_API_KEY" }) // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index const index = pc.index("INDEX_NAME", "INDEX_HOST") await index.namespace('example-namespace').update({ id: 'id-3', values: [4.0, 2.0], metadata: { genre: "comedy", }, }); ``` ```java Java import com.google.protobuf.Struct; import com.google.protobuf.Value; import io.pinecone.clients.Index; import io.pinecone.configs.PineconeConfig; import io.pinecone.configs.PineconeConnection; import io.pinecone.proto.UpdateResponse; import java.util.Arrays; import java.util.List; public class UpdateExample { public static void main(String[] args) { PineconeConfig config = new PineconeConfig("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index config.setHost("INDEX_HOST"); PineconeConnection connection = new PineconeConnection(config); Index index = new Index(connection, "INDEX_NAME"); List values = Arrays.asList(4.0f, 2.0f); Struct metaData = Struct.newBuilder() .putFields("genre", Value.newBuilder().setStringValue("comedy").build()) .build(); UpdateResponse updateResponse = index.update("id-3", values, metaData, "example-namespace", null, null); System.out.println(updateResponse); } } ``` ```go Go package main import ( "context" "log" "github.com/pinecone-io/go-pinecone/v2/pinecone" "google.golang.org/protobuf/types/known/structpb" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "INDEX_HOST", Namespace: "example-namespace"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v", err) } id := "id-3" metadataMap := map[string]interface{}{ "genre": "comedy", } metadataFilter, err := structpb.NewStruct(metadataMap) if err != nil { log.Fatalf("Failed to create metadata map: %v", err) } err = idxConnection.UpdateVector(ctx, &pinecone.UpdateVectorRequest{ Id: id, Metadata: metadataFilter, }) if err != nil { log.Fatalf("Failed to update vector with ID %v: %v", id, err) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index var index = pinecone.Index(host: "INDEX_HOST"); var updateResponse = await index.UpdateAsync(new UpdateRequest { Id = "id-3", Values = new[] { 4.0f, 2.0f }, SetMetadata = new Metadata { ["genre"] = "comedy" }, Namespace = "example-namespace", }); ``` ```shell curl # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index PINECONE_API_KEY="YOUR_API_KEY" INDEX_HOST="INDEX_HOST" curl "https://$INDEX_HOST/vectors/update" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "Content-Type: application/json" \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "id": "id-3", "values": [4.0, 2.0], "setMetadata": {"type": "comedy"}, "namespace": "example-namespace" }' ``` ```json {} ``` # Upsert vectors Source: https://docs.pinecone.io/reference/api/2025-04/data-plane/upsert db_data_2025-04.oas post /vectors/upsert Upsert vectors into a namespace. If a new value is upserted for an existing vector ID, it will overwrite the previous value. For guidance and examples, see [Upsert data](https://docs.pinecone.io/guides/index-data/upsert-data#upsert-vectors). To control costs when ingesting large datasets (10,000,000+ records), use [import](/guides/index-data/import-data) instead of upsert. ```python Python # pip install "pinecone[grpc]" from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key="YOUR_API_KEY") # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index index = pc.Index(host="INDEX_HOST") index.upsert( vectors=[ { "id": "vec1", "values": [0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1], "metadata": {"genre": "comedy", "year": 2020} }, { "id": "vec2", "values": [0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2], "metadata": {"genre": "documentary", "year": 2019} } ], namespace="example-namespace" ) ``` ```javascript JavaScript // npm install @pinecone-database/pinecone import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: "YOUR_API_KEY" }) // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index const index = pc.index("INDEX_NAME", "INDEX_HOST") const records = [ { id: 'vec1', values: [0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1], metadata: { genre: "comedy", year: 2020 }, }, { id: 'vec2', values: [0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2], metadata: { genre: "documentary", year: 2019 }, } ] await index.('example-namespace').upsert(records); ``` ```go Go package main import ( "context" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "INDEX_HOST", Namespace: "example-namespace"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v", err) } // Upsert records into a namespace // `chunk_text` fields are converted to sparse vectors // `category` and `quarter` fields are stored as metadata records := []*pinecone.IntegratedRecord{ { "_id": "vec1", "chunk_text": "AAPL reported a year-over-year revenue increase, expecting stronger Q3 demand for its flagship phones.", "category": "technology", "quarter": "Q3", }, { "_id": "vec2", "chunk_text": "Analysts suggest that AAPL's upcoming Q4 product launch event might solidify its position in the premium smartphone market.", "category": "technology", "quarter": "Q4", }, { "_id": "vec3", "chunk_text": "AAPL's strategic Q3 partnerships with semiconductor suppliers could mitigate component risks and stabilize iPhone production.", "category": "technology", "quarter": "Q3", }, { "_id": "vec4", "chunk_text": "AAPL may consider healthcare integrations in Q4 to compete with tech rivals entering the consumer wellness space.", "category": "technology", "quarter": "Q4", }, } err = idxConnection.UpsertRecords(ctx, records) if err != nil { log.Fatalf("Failed to upsert vectors: %v", err) } } ``` ```java Java import com.google.protobuf.Struct; import com.google.protobuf.Value; import io.pinecone.clients.Index; import io.pinecone.configs.PineconeConfig; import io.pinecone.configs.PineconeConnection; import java.util.Arrays; import java.util.List; public class UpsertExample { public static void main(String[] args) { PineconeConfig config = new PineconeConfig("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index config.setHost("INDEX_HOST"); PineconeConnection connection = new PineconeConnection(config); Index index = new Index(connection, "INDEX_NAME"); List values1 = Arrays.asList(0.1f, 0.1f, 0.1f, 0.1f, 0.1f, 0.1f, 0.1f, 0.1f); List values2 = Arrays.asList(0.2f, 0.2f, 0.2f, 0.2f, 0.2f, 0.2f, 0.2f, 0.2f); Struct metaData1 = Struct.newBuilder() .putFields("genre", Value.newBuilder().setStringValue("comedy").build()) .putFields("year", Value.newBuilder().setNumberValue(2020).build()) .build(); Struct metaData2 = Struct.newBuilder() .putFields("genre", Value.newBuilder().setStringValue("documentary").build()) .putFields("year", Value.newBuilder().setNumberValue(2019).build()) .build(); index.upsert("vec1", values1, null, null, metaData1, 'example-namespace'); index.upsert("vec2", values2, null, null, metaData2, 'example-namespace'); } } ``` ```go Go package main import ( "context" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" "google.golang.org/protobuf/types/known/structpb" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "INDEX_HOST", Namespace: "example-namespace"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v", err) } metadataMap1 := map[string]interface{}{ "genre": "comedy", "year": 2020, } metadata1, err := structpb.NewStruct(metadataMap1) if err != nil { log.Fatalf("Failed to create metadata map: %v", err) } metadataMap2 := map[string]interface{}{ "genre": "documentary", "year": 2019, } metadata2, err := structpb.NewStruct(metadataMap2) if err != nil { log.Fatalf("Failed to create metadata map: %v", err) } vectors := []*pinecone.Vector{ { Id: "vec1", Values: []float32{0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1}, Metadata: metadata1, }, { Id: "vec2", Values: []float32{0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2}, Metadata: metadata2, }, } count, err := idxConnection.UpsertVectors(ctx, vectors) if err != nil { log.Fatalf("Failed to upsert vectors: %v", err) } else { fmt.Printf("Successfully upserted %d vector(s)!\n", count) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index var index = pinecone.Index(host: "INDEX_HOST"); var upsertResponse = await index.UpsertAsync(new UpsertRequest { Vectors = new[] { new Vector { Id = "vec1", Values = new[] { 0.1f, 0.1f, 0.1f, 0.1f, 0.1f, 0.1f, 0.1f, 0.1f }, Metadata = new Metadata { ["genre"] = new("comedy"), ["year"] = new(2020), }, }, new Vector { Id = "vec2", Values = new[] { 0.2f, 0.2f, 0.2f, 0.2f, 0.2f, 0.2f, 0.2f, 0.2f }, Metadata = new Metadata { ["genre"] = new("documentary"), ["year"] = new(2019), }, } }, Namespace = "example-namespace", }); ``` ```shell curl # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index PINECONE_API_KEY="YOUR_API_KEY" INDEX_HOST="INDEX_HOST" curl "https://$INDEX_HOST/vectors/upsert" \ -H "Api-Key: $PINECONE_API_KEY" \ -H 'Content-Type: application/json' \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "vectors": [ { "id": "vec1", "values": [0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1], "metadata": {"genre": "comedy", "year": 2020} }, { "id": "vec2", "values": [0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2], "metadata": {"genre": "documentary", "year": 2019} } ], "namespace": "example-namespace" }' ``` ```json {"upsertedCount":2} ``` # Upsert text Source: https://docs.pinecone.io/reference/api/2025-04/data-plane/upsert_records db_data_2025-04.oas post /records/namespaces/{namespace}/upsert Upsert text into a namespace. Pinecone converts the text to vectors automatically using the hosted embedding model associated with the index. Upserting text is supported only for [indexes with integrated embedding](https://docs.pinecone.io/reference/api/2025-01/control-plane/create_for_model). For guidance and examples, see [Upsert data](https://docs.pinecone.io/guides/index-data/upsert-data#upsert-text). ```python Python # pip install --upgrade pinecone from pinecone import Pinecone pc = Pinecone(api_key="YOUR_API_KEY") # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index index = pc.Index(host="INDEX_HOST") # Upsert records into a namespace # `chunk_text` fields are converted to dense vectors # `category` fields are stored as metadata index.upsert_records( "example-namespace", [ { "_id": "rec1", "chunk_text": "Apples are a great source of dietary fiber, which supports digestion and helps maintain a healthy gut.", "category": "digestive system", }, { "_id": "rec2", "chunk_text": "Apples originated in Central Asia and have been cultivated for thousands of years, with over 7,500 varieties available today.", "category": "cultivation", }, { "_id": "rec3", "chunk_text": "Rich in vitamin C and other antioxidants, apples contribute to immune health and may reduce the risk of chronic diseases.", "category": "immune system", }, { "_id": "rec4", "chunk_text": "The high fiber content in apples can also help regulate blood sugar levels, making them a favorable snack for people with diabetes.", "category": "endocrine system", }, ] ) ``` ```javascript JavaScript // npm install @pinecone-database/pinecone import { Pinecone } from '@pinecone-database/pinecone' const pc = new Pinecone({ apiKey: "YOUR_API_KEY" }) // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index const namespace = pc.index("INDEX_NAME", "INDEX_HOST").namespace("example-namespace"); // Upsert records into a namespace // `chunk_text` fields are converted to dense vectors // `category` ios stored as metadata await namespace.upsertRecords([ { "_id": "rec1", "chunk_text": "Apples are a great source of dietary fiber, which supports digestion and helps maintain a healthy gut.", "category": "digestive system", }, { "_id": "rec2", "chunk_text": "Apples originated in Central Asia and have been cultivated for thousands of years, with over 7,500 varieties available today.", "category": "cultivation", }, { "_id": "rec3", "chunk_text": "Rich in vitamin C and other antioxidants, apples contribute to immune health and may reduce the risk of chronic diseases.", "category": "immune system", }, { "_id": "rec4", "chunk_text": "The high fiber content in apples can also help regulate blood sugar levels, making them a favorable snack for people with diabetes.", "category": "endocrine system", } ]); ``` ```java Java import io.pinecone.clients.Index; import io.pinecone.configs.PineconeConfig; import io.pinecone.configs.PineconeConnection; import org.openapitools.db_data.client.ApiException; import java.util.*; public class UpsertText { public static void main(String[] args) throws ApiException { PineconeConfig config = new PineconeConfig("YOUR_API_KEY"); config.setHost("INDEX_HOST"); PineconeConnection connection = new PineconeConnection(config); Index index = new Index(config, connection, "integrated-dense-java"); ArrayList> upsertRecords = new ArrayList<>(); HashMap record1 = new HashMap<>(); record1.put("_id", "rec1"); record1.put("category", "digestive system"); record1.put("chunk_text", "Apples are a great source of dietary fiber, which supports digestion and helps maintain a healthy gut."); HashMap record2 = new HashMap<>(); record2.put("_id", "rec2"); record2.put("category", "cultivation"); record2.put("chunk_text", "Apples originated in Central Asia and have been cultivated for thousands of years, with over 7,500 varieties available today."); HashMap record3 = new HashMap<>(); record3.put("_id", "rec3"); record3.put("category", "immune system"); record3.put("chunk_text", "Rich in vitamin C and other antioxidants, apples contribute to immune health and may reduce the risk of chronic diseases."); HashMap record4 = new HashMap<>(); record4.put("_id", "rec4"); record4.put("category", "endocrine system"); record4.put("chunk_text", "The high fiber content in apples can also help regulate blood sugar levels, making them a favorable snack for people with diabetes."); upsertRecords.add(record1); upsertRecords.add(record2); upsertRecords.add(record3); upsertRecords.add(record4); index.upsertRecords("example-namespace", upsertRecords); } } ``` ```go Go package main import ( "context" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } // To get the unique host for an index, // see https://docs.pinecone.io/guides/manage-data/target-an-index idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: "INDEX_HOST", Namespace: "example-namespace"}) if err != nil { log.Fatalf("Failed to create IndexConnection for Host: %v", err) } // Upsert records into a namespace // `chunk_text` fields are converted to dense vectors // `category` is stored as metadata records := []*pinecone.IntegratedRecord{ { "_id": "rec1", "chunk_text": "Apples are a great source of dietary fiber, which supports digestion and helps maintain a healthy gut.", "category": "digestive system", }, { "_id": "rec2", "chunk_text": "Apples originated in Central Asia and have been cultivated for thousands of years, with over 7,500 varieties available today.", "category": "cultivation", }, { "_id": "rec3", "chunk_text": "Rich in vitamin C and other antioxidants, apples contribute to immune health and may reduce the risk of chronic diseases.", "category": "immune system", }, { "_id": "rec4", "chunk_text": "The high fiber content in apples can also help regulate blood sugar levels, making them a favorable snack for people with diabetes.", "category": "endocrine system", }, } err = idxConnection.UpsertRecords(ctx, records) if err != nil { log.Fatalf("Failed to upsert vectors: %v", err) } } ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY"); var index = pinecone.Index(host: "INDEX_HOST"); await index.UpsertRecordsAsync( "example-namespace", [ new UpsertRecord { Id = "rec1", AdditionalProperties = { ["chunk_text"] = "Apples are a great source of dietary fiber, which supports digestion and helps maintain a healthy gut.", ["category"] = "digestive system", }, }, new UpsertRecord { Id = "rec2", AdditionalProperties = { ["chunk_text"] = "Apples originated in Central Asia and have been cultivated for thousands of years, with over 7,500 varieties available today.", ["category"] = "cultivation", }, }, new UpsertRecord { Id = "rec3", AdditionalProperties = { ["chunk_text"] = "Rich in vitamin C and other antioxidants, apples contribute to immune health and may reduce the risk of chronic diseases.", ["category"] = "immune system", }, }, new UpsertRecord { Id = "rec4", AdditionalProperties = { ["chunk_text"] = "The high fiber content in apples can also help regulate blood sugar levels, making them a favorable snack for people with diabetes.", ["category"] = "endocrine system", }, }, ] ); ``` ```shell curl # To get the unique host for an index, # see https://docs.pinecone.io/guides/manage-data/target-an-index INDEX_HOST="INDEX_HOST" NAMESPACE="YOUR_NAMESPACE" PINECONE_API_KEY="YOUR_API_KEY" # Upsert records into a namespace # `chunk_text` fields are converted to dense vectors # `category` fields are stored as metadata curl "https://$INDEX_HOST/records/namespaces/$NAMESPACE/upsert" \ -H "Content-Type: application/x-ndjson" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{"_id": "rec1", "chunk_text": "Apples are a great source of dietary fiber, which supports digestion and helps maintain a healthy gut.", "category": "digestive system"} {"_id": "rec2", "chunk_text": "Apples originated in Central Asia and have been cultivated for thousands of years, with over 7,500 varieties available today.", "category": "cultivation"} {"_id": "rec3", "chunk_text": "Rich in vitamin C and other antioxidants, apples contribute to immune health and may reduce the risk of chronic diseases.", "category": "immune system"} {"_id": "rec4", "chunk_text": "The high fiber content in apples can also help regulate blood sugar levels, making them a favorable snack for people with diabetes.", "category": "endocrine system"}' ``` # Authentication Source: https://docs.pinecone.io/reference/api/authentication All requests to [Pinecone APIs](/reference/api/introduction) must contain a valid [API key](/guides/production/security-overview#api-keys) for the target project. ## Get an API key [Create a new API key](https://app.pinecone.io/organizations/-/projects/-/keys) in the Pinecone console, or use the connect widget below to generate a key.
Copy your generated key: ``` PINECONE_API_KEY="{{YOUR_API_KEY}}" # This API key has ReadWrite access to all indexes in your project. ``` ## Initialize a client When using a [Pinecone SDK](/reference/pinecone-sdks), initialize a client object with your API key and then reuse the authenicated client in subsquent function calls. For example: ```Python Python from pinecone.grpc import PineconeGRPC as Pinecone from pinecone import ServerlessSpec pc = Pinecone(api_key='YOUR_API_KEY') # Creates an index using the API key stored in the client 'pc'. pc.create_index( name="docs-example", dimension=1536, metric="cosine", spec=ServerlessSpec( cloud='aws', region='us-east-1' ) ) ``` ```JavaScript JavaScript import { Pinecone } from '@pinecone-database/pinecone'; const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }); // Creates an index using the API key stored in the client 'pc'. await pc.createIndex({ name: 'docs-example', dimension: 1536, metric: 'cosine', spec: { serverless: { cloud: 'aws', region: 'us-east-1' } } }) ``` ```java Java import io.pinecone.clients.Pinecone; import org.openapitools.db_control.client.model.IndexModel; import org.openapitools.db_control.client.model.DeletionProtection; public class CreateServerlessIndexExample { public static void main(String[] args) { Pinecone pc = new Pinecone.Builder("YOUR_API_KEY").build(); // Creates an index using the API key stored in the client 'pc'. pc.createServerlessIndex("docs-example", "cosine", 1536, "aws", "us-east-1"); } } ``` ```go Go package main import ( "context" "fmt" "log" "github.com/pinecone-io/go-pinecone/v3/pinecone" ) func main() { ctx := context.Background() pc, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", }) if err != nil { log.Fatalf("Failed to create Client: %v", err) } indexName := "docs-example" vectorType := "dense" dimension := int32(1536) metric := pinecone.Cosine deletionProtection := pinecone.DeletionProtectionDisabled idx, err := pc.CreateServerlessIndex(ctx, &pinecone.CreateServerlessIndexRequest{ Name: indexName, VectorType: &vectorType, Dimension: &dimension, Metric: &metric, Cloud: pinecone.Aws, Region: "us-east-1", DeletionProtection: &deletionProtection, }) if err != nil { log.Fatalf("Failed to create serverless index: %v", idx.Name) } else { fmt.Printf("Successfully created serverless index: %v", idx.Name) } } ``` ```shell curl curl -s "https://api.pinecone.io/indexes" \ -H "Api-Key: YOUR_API_KEY" \ -H "Content-Type: application/json" \ -H "X-Pinecone-API-Version: 2025-04" \ -d '{ "name": "docs-example", "dimension": 1536, "metric": "cosine", "spec": { "serverless": { "cloud":"aws", "region": "us-east-1" } } }' ``` ## Add headers to an HTTP request All HTTP requests to Pinecone APIs must contain an `Api-Key` header that specifies a valid [API key](/guides/production/security-overview#api-keys) and must be encoded as JSON with the `Content-Type: application/json` header. For example: ```shell curl curl https://api.pinecone.io/indexes \ -H "Content-Type: application/json" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2025-01" \ -d '{ "name": "docs-example", "dimension": 1536, "metric": "cosine", "spec": { "serverless": { "cloud":"aws", "region": "us-east-1" } } }' ``` ## Troubleshooting Older versions of Pinecone required you to initialize a client with an `init` method that takes both `api_key` and `environment` parameters, for example: ```python Python # Legacy initialization import pinecone pc = pinecone.init( api_key="PINECONE_API_KEY", environment="PINECONE_ENVIRONMENT" ) ``` ```javascript JavaScript // Legacy initialization import { Pinecone } from '@pinecone-database/pinecone'; const pineconeClient = new PineconeClient(); await pineconeClient.init({ apiKey: 'PINECONE_API_KEY', environment: 'PINECONE_ENVIRONMENT', }); ``` In more recent versions of Pinecone, this has changed. Initialization no longer requires an `init` step, and cloud environment is defined for each index rather than an entire project. Client initialization now only requires an `api_key` parameter, for example: ```python Python # New initialization from pinecone import Pinecone pc = Pinecone(api_key="YOUR_API_KEY") ``` ```javascript JavaScript // New initialization import { Pinecone } from '@pinecone-database/pinecone'; const pc = new Pinecone({ apiKey: 'YOUR_API_KEY' }); ``` If you are receiving errors about initialization, upgrade your [Pinecone SDK](/reference/pinecone-sdks) to the latest version, for example: ```shell Python # Upgrade Pinecone SDK pip install pinecone --upgrade ``` ```shell JavaScript # Upgrade Pinecone SDK npm install @pinecone-database/pinecone@latest ``` Also, note that some third-party tutorials and examples still reference the older initialization method. In such cases, follow the example above and the examples throughout the Pinecone documentation instead. # Pinecone Database limits Source: https://docs.pinecone.io/reference/api/database-limits This page describes different types of limits for Pinecone Database. ## Rate limits Rate limits are restrictions on the frequency of requests within a specified period of time. Rate limits vary based on [pricing plan](https://www.pinecone.io/pricing/) and apply to [serverless indexes](/guides/index-data/indexing-overview#serverless-indexes) only. | Metric | Starter plan | Standard plan | Enterprise plan | | :-------------------------------------------------------------------------------------------------------- | :----------- | :------------ | :-------------- | | [Read units per month per project](#read-units-per-month-per-project) | 1,000,000 | N/A | N/A | | [Write units per month per project](#write-units-per-month-per-project) | 2,000,000 | N/A | N/A | | [Upsert size per second per namespace](#upsert-size-per-second-per-namespace) | 50 MB | 50 MB | 50 MB | | [Query read units per second per index](#query-read-units-per-second-per-index) | 2,000 | 2,000 | 2,000 | | [Update records per second per namespace](#update-records-per-second-per-namespace) | 100 | 100 | 100 | | [Fetch requests per second per index](#fetch-requests-per-second-per-index) | 100 | 100 | 100 | | [List requests per second per index](#list-requests-per-second-per-index) | 200 | 200 | 200 | | [Describe index stats requests per second per index](#describe-index-stats-requests-per-second-per-index) | 100 | 100 | 100 | | Delete records per second per namespace | 5,000 | 5,000 | 5,000 | | Delete records per second per index | 5,000 | 5,000 | 5,000 | ### Read units per month per project This limit applies to organizations on the Starter plan only. | Starter plan | Standard plan | Enterprise plan | | ------------ | ------------- | --------------- | | 1,000,000 | N/A | N/A | [Read units](/guides/manage-cost/understanding-cost#read-units) measure the compute, I/O, and network resources used by [fetch](/guides/manage-data/fetch-data), [query](/guides/search/search-overview), and [list](/guides/manage-data/list-record-ids) requests to serverless indexes. When you reach the monthly read unit limit for a project, fetch, query, and list requests to serverless indexes in the project will fail and return a `429 - TOO_MANY_REQUESTS` status with the following error: ``` Request failed. You've reached your read unit limit for the current month limit. To continue reading data, upgrade your plan. ``` To continue reading from serverless indexes in the project, [upgrade your plan](/guides/organizations/manage-billing/manage-your-billing-plan). To check how close you are to the monthly read unit limit for a project, do the following: 1. Open the [Pinecone console](https://app.pinecone.io/organizations/-/projects). 2. Select the project. 3. Select any index in the project. 4. Look under **Starter Usage**. ### Write units per month per project This limit applies to organizations on the Starter plan only. | Starter plan | Standard plan | Enterprise plan | | ------------ | ------------- | --------------- | | 2,000,000 | N/A | N/A | [Write units](/guides/manage-cost/understanding-cost#write-units) measure the storage and compute resources used by [upsert](/guides/index-data/upsert-data), [update](/guides/manage-data/update-data), and [delete](/guides/manage-data/delete-data) requests to serverless indexes. When you reach the monthly write unit limit for a project, upsert, update, and delete requests to serverless indexes in the project will fail and return a `429 - TOO_MANY_REQUESTS` status with the following error: ``` Request failed. You've reached your write unit limit for the current month. To continue writing data, upgrade your plan. ``` To continue writing data to serverless indexes in the project, [upgrade your plan](/guides/organizations/manage-billing/manage-your-billing-plan). To check how close you are to the monthly read unit limit for a project, do the following: 1. Open the [Pinecone console](https://app.pinecone.io/organizations/-/projects). 2. Select the project. 3. Select any index in the project. 4. Look under **Starter Usage**. ### Upsert size per second per namespace | Starter plan | Standard plan | Enterprise plan | | ------------ | ------------- | --------------- | | 50 MB | 50 MB | 50 MB | When you reach the per second [upsert](/guides/index-data/upsert-data) size for a namespace in an index, additional upserts will fail and return a `429 - TOO_MANY_REQUESTS` status with the following error: ``` Request failed. You've reached the max upsert size limit per second for index . Pace your upserts or upgrade your plan. ``` To increase this limit, [upgrade your plan](/guides/organizations/manage-billing/manage-your-billing-plan). Otherwise, you can handle this limit by [automatically retrying requests with an exponential backoff](https://www.pinecone.io/blog/working-at-scale/). ### Query read units per second per index | Starter plan | Standard plan | Enterprise plan | | ------------ | ------------- | --------------- | | 2,000 | 2,000 | 2,000 | Pinecone measures [query](/guides/search/search-overview) usage in [read units](/guides/manage-cost/understanding-cost#read-units). When you reach the per second limit for queries across all namespaces in an index, additional queries will fail and return a `429 - TOO_MANY_REQUESTS` status with the following error: ``` Request failed. You've reached the max query read units per second for index . Pace your queries. ``` To handle this limit, [automatically retry requests with an exponential backoff](https://www.pinecone.io/blog/working-at-scale/). To check how many read units a query consumes, [check the query response](/guides/manage-cost/monitor-your-usage). ### Update records per second per namespace | Starter plan | Standard plan | Enterprise plan | | ------------ | ------------- | --------------- | | 100 | 100 | 100 | When you reach the per second [update](/guides/manage-data/update-data) limit for a namespace in an index, additional updates will fail and return a `429 - TOO_MANY_REQUESTS` status with the following error: ``` Request failed. You've reached the max update records per second for namespace . Pace your update requests or upgrade your plan. ``` To increase this limit, [upgrade your plan](/guides/organizations/manage-billing/manage-your-billing-plan). Otherwise, you can handle this limit by [automatically retrying requests with an exponential backoff](https://www.pinecone.io/blog/working-at-scale/). ### Fetch requests per second per index | Starter plan | Standard plan | Enterprise plan | | ------------ | ------------- | --------------- | | 100 | 100 | 100 | When you reach the per second [fetch](/guides/manage-data/fetch-data) limit across all namespaces in an index, additional fetch requests will fail and return a `429 - TOO_MANY_REQUESTS` status with the following error: ``` Request failed. You've reached the max fetch requests per second for index . Pace your fetch requests. ``` To handle this limit, [automatically retry requests with an exponential backoff](https://www.pinecone.io/blog/working-at-scale/). ### List requests per second per index | Starter plan | Standard plan | Enterprise plan | | ------------ | ------------- | --------------- | | 200 | 200 | 200 | When you reach the per second [list](/guides/manage-data/list-record-ids) limit across all namespaces in an index, additional list requests will fail and return a `429 - TOO_MANY_REQUESTS` status with the following error: ``` Request failed. You've reached the max list requests per second for index . Pace your list requests. ``` To handle this limit, [automatically retry requests with an exponential backoff](https://www.pinecone.io/blog/working-at-scale/). ### Describe index stats requests per second per index | Starter plan | Standard plan | Enterprise plan | | ------------ | ------------- | --------------- | | 100 | 100 | 100 | When you reach the per second [describe index stats](/reference/api/2024-10/data-plane/describeindexstats) limit across all namespaces in an index, additional list requests will fail and return a `429 - TOO_MANY_REQUESTS` status with the following error: ``` Request failed. You've reached the max describe_index_stats requests per second for index . Pace your describe_index_stats requests. ``` To handle this limit, [automatically retry requests with an exponential backoff](https://www.pinecone.io/blog/working-at-scale/). ## Object limits Object limits are restrictions on the number or size of objects in Pinecone. Object limits vary based on [pricing plan](https://www.pinecone.io/pricing/). | Metric | Starter plan | Standard plan | Enterprise plan | | :----------------------------------------------------------------------------- | :----------- | :------------ | :-------------- | | [Projects per organization](#projects-per-organization) | 1 | 20 | 100 | | [Pods per organization](#pods-per-organization) | 0 | 100 | 100 | | [Serverless indexes per project](#serverless-indexes-per-project) 1 | 5 | 20 | 200 | | [Serverless index storage per project](#serverless-index-storage-per-project) | 2 GB | N/A | N/A | | [Namespaces per serverless index](#namespaces-per-serverless-index) | 100 | 25,000 | 100,000 | | [Serverless backups per project](#serveless-backups-per-project) | N/A | 500 | 1000 | | [Namespaces per serverless backup](#namespaces-per-serverless-backup) | N/A | 2000 | 2000 | | [Pod-based indexes per project](#pod-based-indexes-per-project) | 0 | N/A | N/A | | [Pods per project](#pods-per-project) 2 | 0 | 2 | 2 | | [Collections per project](#collections-per-project) | 100 | N/A | N/A | 1 On the Starter plan, all serverless must be in the `us-east-1` region of AWS.
2 The limit on the number of [pods per project](#pods-per-project) can be customized for organizations on Standard and Enterprise plans after creating a project. ### Projects per organization | Starter plan | Standard plan | Enterprise plan | | ------------ | ------------- | --------------- | | 1 | 20 | 100 | When you reach this quota for an organization, trying to [create projects](/guides/projects/create-a-project) will fail and return a `403 - QUOTA_EXCEEDED` status with the following error: ``` Request failed. You've reached the max projects allowed in organization . To add more projects, upgrade your plan. ``` To increase this quota, [upgrade your plan](/guides/organizations/manage-billing/manage-your-billing-plan) or [contact Support](https://app.pinecone.io/organizations/-/settings/support/ticket). ### Pods per organization | Starter plan | Standard plan | Enterprise plan | | ------------ | ------------- | --------------- | | 0 | 100 | 100 | When you reach this quota for an organization, trying to [create pod-based indexes](/guides/index-data/create-an-index#create-a-pod-based-index) in any project in the organization will fail and return a `403 - QUOTA_EXCEEDED` status with the following error: ``` Request failed. You've reached the max pods allowed in organization ORGANIZATION_NAME (LIMIT). To increase this limit, contact support@pinecone.io. ``` To increase this quota, [contact Support](https://app.pinecone.io/organizations/-/settings/support/ticket). ### Serverless indexes per project | Starter plan | Standard plan | Enterprise plan | | ------------ | ------------- | --------------- | | 5 | 20 | 200 | When you reach this quota for a project, trying to [create serverless indexes](/guides/index-data/create-an-index#create-a-serverless-index) in the project will fail and return a `403 - QUOTA_EXCEEDED` status with the following error: ``` Request failed. You've reached the max serverless indexes allowed in project . To add more serverless indexes, upgrade your plan. ``` On the Starter plan, all serverless must be in the `us-east-1` region of AWS. To create indexes in different regions or to increase this quota, [upgrade your plan](/guides/organizations/manage-billing/manage-your-billing-plan) or [contact Support](https://app.pinecone.io/organizations/-/settings/support/ticket). ### Serverless index storage per project This limit applies to organizations on the Starter plan only. | Starter plan | Standard plan | Enterprise plan | | ------------ | ------------- | --------------- | | 2 GB | N/A | N/A | When you've reached this quota for a project, updates and upserts into serverless indexes will fail and return a `403 - QUOTA_EXCEEDED` status with the following error: ``` Request failed. You've reached the max storage allowed for project . To update or upsert new data, delete records or upgrade your plan. ``` To continue writing data into your serverless indexes, [delete records](/guides/manage-data/delete-data) to bring you under the limit or [upgrade your plan](/guides/organizations/manage-billing/manage-your-billing-plan). ### Namespaces per serverless index | Starter plan | Standard plan | Enterprise plan | | ------------ | ------------- | --------------- | | 100 | 25,000 | 100,000 | When you reach this quota for a serverless index, trying to [upsert records into a new namespace](/guides/index-data/upsert-data) in the index will fail and return a `403 - QUOTA_EXCEEDED` status with the following error: ``` Request failed. You've reached the max namespaces allowed in serverless index . To add more namespaces, upgrade your plan. ``` To increase this quota, [upgrade your plan](/guides/organizations/manage-billing/manage-your-billing-plan). These quotas are intended to provide reasonable boundaries and prevent unexpected or unintentional misuse. To increase your quota beyond the standard allotment, [contact Support](https://app.pinecone.io/organizations/-/settings/support/ticket). ### Serverless backups per project | Starter plan | Standard plan | Enterprise plan | | ------------ | ------------- | --------------- | | N/A | 500 | 1000 | When you reach this quota for a project, trying to [create serverless backups](/guides/manage-data/back-up-an-index) in the project will fail and return a `403 - QUOTA_EXCEEDED` status with the following error: ``` Backup failed to create. Quota for number of backups per index exceeded. ``` ### Namespaces per serverless backup | Starter plan | Standard plan | Enterprise plan | | ------------ | ------------- | --------------- | | N/A | 2000 | 2000 | When you reach this quota for a backup, trying to [create serverless backups](/guides/manage-data/back-up-an-index) will fail and return a `403 - QUOTA_EXCEEDED` status. ### Pod-based indexes per project This limit applies to organizations on the Starter plan only. | Starter plan | Standard plan | Enterprise plan | | ------------ | ------------- | --------------- | | 0 | N/A | N/A | When you try to create a pod-based index on the Starter plan, the request will fail and return a `403 - QUOTA_EXCEEDED` status with the following error: ``` Request failed. You've reach the max pod-based indexes allowed in project . To add more pod-based indexes, upgrade your plan. ``` To create pod-based indexes, [upgrade your plan](/guides/organizations/manage-billing/manage-your-billing-plan). ### Pods per project | Starter plan | Standard plan | Enterprise plan | | ------------ | ------------- | --------------- | | 0 | 2 | 2 | When you reach this quota for a project, trying to [create pod-based indexes](/guides/index-data/create-an-index#create-a-pod-based-index) in the project will fail and return a `403 - QUOTA_EXCEEDED` status with the following error: ``` Request failed. You've reached the max pods allowed in project PROJECT_NAME. To increase this limit, adjust your project settings in the console. Contact a project owner if you don't have permission. ``` To set or change the default limit, [set a project pod limit](/guides/projects/manage-projects#set-a-project-pod-limit). ### Collections per project | Starter plan | Standard plan | Enterprise plan | | ------------ | ------------- | --------------- | | 100 | N/A | N/A | When you reach this quota for a project, trying to [create collections](/guides/manage-data/back-up-an-index) in the project will fail and return a `403 - QUOTA_EXCEEDED` status with the following error: ``` Request failed. You've reached the max collections allowed in project . To add more collections, upgrade your plan. ``` To increase this quota, [upgrade your plan](/guides/organizations/manage-billing/manage-your-billing-plan). ## Operation limits Operation limits are restrictions on the size, number, or other characteristics of operations in Pinecone. Operation limits are fixed and do not vary based on pricing plan. ### Upsert limits | Metric | Limit | | :------------------------------------- | :------------------ | | Max upsert size | 2MB or 1000 records | | Max metadata size per record | 40 KB | | Max length for a record ID | 512 characters | | Max dimensionality for dense vectors | 20,000 | | Max non-zero values for sparse vectors | 2048 | | Max dimensionality for sparse vectors | 4.2 billion | ### Query limits | Metric | Limit | | :---------------- | :----- | | Max `top_k` value | 10,000 | | Max result size | 4MB | The query result size is affected by the dimension of the dense vectors and whether or not dense vector values and metadata are included in the result. If a query fails due to exceeding the 4MB result size limit, choose a lower `top_k` value, or use `include_metadata=False` or `include_values=False` to exclude metadata or values from the result. ### Fetch limits | | Limit | | :---------------------------- | :---- | | Max records per fetch request | 1,000 | ### Delete limits | Delete | Limit | | :----------------------------- | :---- | | Max records per delete request | 1,000 | ## Identifier limits An identifier is a string of characters (up to 255 characters in length) used to identify "named" [objects in Pinecone](/guides/get-started/glossary). The following Pinecone objects use strings as identifiers: | Object | Field | Max # characters | Allowed characters | | --------------------------------------------------------- | ----------- | ---------------- | ---------------------------- | | [Organization](/guides/get-started/glossary#organization) | `name` | 512 | UTF-8 except `\0` | | [Project](/guides/get-started/glossary#project) | `name` | 512 | UTF-8 except `\0` | | [Index](/guides/get-started/glossary#index) | `name` | 45 | `A-Z`, `a-z`, `0-9`, and `-` | | [Namespace](/guides/get-started/glossary#namespace) | `namespace` | 512 | ASCII except `\0` | | [Record](/guides/get-started/glossary#record) | `id` | 512 | ASCII except `\0` | # Errors Source: https://docs.pinecone.io/reference/api/errors Pinecone uses conventional HTTP response codes to indicate the success or failure of an API request. In general, codes in the `2xx` range indicate success, codes in the `4xx` range indicate an error that failed given the information provided, and codes in the `5xx` range indicate an error with Pinecone's servers. ## 200 - OK The request succeeded. ## 201 - CREATED The request succeeded and a new resource was created. ## 202 - NO CONTENT The request succeeded, but there is no content to return. ## 400 - INVALID ARGUMENT The request failed due to an invalid argument. ## 401 - UNAUTHENTICATED The request failed due to a missing or invalid [API key](/guides/projects/understanding-projects#api-keys). ## 402 - PAYMENT REQUIRED The request failed due to delinquent payment. ## 403 - FORBIDDEN The request failed due to an exceeded [quota](/reference/api/database-limits#object-limits) or [index deletion protection](/guides/manage-data/manage-indexes#configure-deletion-protection). ## 404 - NOT FOUND The request failed because the resource was not found. ## 409 - ALREADY EXISTS The request failed because the resource already exists. ## 412 - FAILED PRECONDITIONS The request failed due to preconditions not being met. | ## 422 - UNPROCESSABLE ENTITY The request failed because the server was unable to process the contained instructions. ## 429 - TOO MANY REQUESTS The request was [rate-limited](/reference/api/database-limits#rate-limits). ## 500 - UNKNOWN An internal server error occurred. ## 503 - UNAVAILABLE The server is currently unavailable. # Pinecone Inference limits Source: https://docs.pinecone.io/reference/api/inference-limits Limits for Pinecone Inference are model-specific. For model limits, see [Embedding models](/guides/index-data/create-an-index#embedding-models) and [Reranking models](/guides/search/rerank-results#reranking-models). # API reference Source: https://docs.pinecone.io/reference/api/introduction Pinecone's APIs let you interact programmatically with your Pinecone account. [SDK versions](/reference/pinecone-sdks#sdk-versions) are pinned to specific API versions. ## Database Use the Database API to store and query records in [Pinecone Database](/guides/get-started/quickstart). The following Pinecone SDKs support the Database API: ## Inference Use the Inference API to generate vector embeddings and rerank results using [embedding models](/guides/index-data/create-an-index#embedding-models) and [reranking models](/guides/search/rerank-results#reranking-models) hosted on Pinecone's infrastructure. You can use the Inference API as a standalone service, via the [`embed`](/reference/api/2025-01/inference/generate-embeddings) and [`rerank`](/reference/api/2025-01/inference/rerank.mdx) endpoints, or integrated with Pinecone's database operations, via the [`create_for_model`](/reference/api/2025-01/control-plane/create_for_model.mdx), [`upsert_records`](/reference/api/2025-01/data-plane/upsert_records), and [`search_records`](/reference/api/2025-01/data-plane/search_records) endpoints. The following Pinecone SDKs support using the Inference API: # Known limitations Source: https://docs.pinecone.io/reference/api/known-limitations This page describes known limitations and feature restrictions in Pinecone. ## General * [Upserts](/guides/index-data/upsert-data) * Pinecone is eventually consistent, so there can be a slight delay before upserted records are available to query. After upserting records, use the [`describe_index_stats`](/reference/api/2024-10/data-plane/describeindexstats) operation to check if the current vector count matches the number of records you expect, although this method may not work for pod-based indexes with multiple replicas. * Only indexes using the [dotproduct distance metric](/guides/index-data/indexing-overview#dotproduct) support querying sparse-dense vectors. Upserting, updating, and fetching sparse-dense vectors in indexes with a different distance metric will succeed, but querying will return an error. * Indexes created before February 22, 2023 do not support sparse vectors. * [Metadata](/guides/index-data/upsert-data#upsert-with-metadata-filters) * Null metadata values are not supported. Instead of setting a key to hold a null value, remove the key from the metadata payload. ## Serverless indexes Serverless indexes do not support the following features: * [Deleting records by metadata](/guides/manage-data/delete-data#delete-specific-records-by-metadata) * Instead, you can [delete records by ID prefix](/guides/manage-data/manage-document-chunks#delete-all-records-for-a-parent-document). * [Selective metadata indexing](/guides/indexes/pods/manage-pod-based-indexes#selective-metadata-indexing) * Because high-cardinality metadata in serverless indexes does not cause high memory utilization, this operation is not relevant. * [Filtering index statistics by metadata](/reference/api/2024-10/data-plane/describeindexstats) * [Private endpoints](/guides/production/connect-to-aws-privatelink) * This feature is available on AWS only. ## Pod-based indexes * [Pod storage capacity](/guides/indexes/pods/understanding-pod-based-indexes#pod-types) * Each **p1** pod has enough capacity for 1M vectors with 768 dimensions. * Each **s1** pod has enough capacity for 5M vectors with 768 dimensions. * [Metadata](/guides/index-data/upsert-data#upsert-with-metadata-filters) * Metadata with high cardinality, such as a unique value for every vector in a large index, uses more memory than expected and can cause the pods to become full. * [Collections](/guides/manage-data/back-up-an-index#pod-based-index-backups-using-collections) * You cannot query or write to a collection after its creation. For this reason, a collection only incurs storage costs. * You can only perform operations on collections in the current Pinecone project. * [Sparse-dense vectors](/guides/search/hybrid-search#use-a-single-hybrid-index) * Only `s1` and `p1` [pod-based indexes](/guides/indexes/pods/understanding-pod-based-indexes#pod-types) using the dotproduct distance metric support sparse-dense vectors. # API versioning Source: https://docs.pinecone.io/reference/api/versioning Pinecone's APIs are versioned to ensure that your applications continue to work as expected as the platform evolves. Versions are named by release date in the format `YYYY-MM`, for example, `2025-04`. ## Release schedule On a quarterly basis, Pinecone releases a new **stable** API version as well as a **release candidate** of the next stable version. * **Stable:** Each stable version remains unchanged and supported for a minimum of 12 months. Since stable versions are released every 3 months, this means you have at least 9 months to test and migrate your app to the newest stable version before support for the previous version is removed. * **Release candidate:** The release candidate gives you insight into the upcoming changes in the next stable version. It is available for approximately 3 months before the release of the stable version and can include new features, improvements, and [breaking changes](#breaking-changes). Below is an example of Pinecone's release schedule: ## Specify an API version When using the API directly, it is important to specify an API version in your requests. If you don't, requests default to the oldest supported stable version. Once support for that version ends, your requests will default to the next oldest stable version, which could include breaking changes that require you to update your integration. To specify an API version, set the `X-Pinecone-API-Version` header to the version name. For example, based on the version support diagram above, if it is currently July 2024 and you want to use the latest stable version to describe an index, you would set `"X-Pinecone-API-Version: 2024-07"`: ```shell curl PINECONE_API_KEY="YOUR_API_KEY" curl -i -X GET "https://api.pinecone.io/indexes/movie-recommendations" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2024-07" ``` If you want to use the release candidate of the next stable version instead, you would set `"X-Pinecone-API-Version: 2024-10"`: ```shell curl PINECONE_API_KEY="YOUR_API_KEY" curl -i -X GET "https://api.pinecone.io/indexes/movie-recommendations" \ -H "Api-Key: $PINECONE_API_KEY" \ -H "X-Pinecone-API-Version: 2024-10" ``` ## SDK versions Official [Pinecone SDKs](/reference/pinecone-sdks) provide convenient access to Pinecone APIs. SDK versions are pinned to specific API versions. When a new API version is released, a new version of the SDK is also released. For the mapping between SDK and API versions, see [SDK versions](/reference/pinecone-sdks#sdk-versions). ## Breaking changes Breaking changes are changes that can potentially break your integration with a Pinecone API. Breaking changes include: * Removing an entire operation * Removing or renaming a parameter * Removing or renaming a response field * Adding a new required parameter * Making a previously optional parameter required * Changing the type of a parameter or response field * Removing enum values * Adding a new validation rule to an existing parameter * Changing authentication or authorization requirements ## Non-breaking changes Non-breaking changes are additive and should not break your integration. Additive changes include: * Adding an operation * Adding an optional parameter * Adding an optional request header * Adding a response field * Adding a response header * Adding enum values ## Get updates To ensure you always know about upcoming API changes, follow the [Release notes](/release-notes/). # Notebooks Source: https://docs.pinecone.io/examples/notebooks export const UtilityExampleCard = ({title, text, link}) => { return

{title}

{text}

; }; export const ExampleCard = ({title, text, link, children, arrow, vectors, namespaces}) => { return

{title}

{text}

{children &&
{children}
} {arrow && } {(vectors || namespaces) &&
{vectors && {vectors} vectors} {namespaces && {namespaces} namespaces}
} ; }; export const Tag = ({text, icon}) => { return {icon && } {text} ; };
{/* ## Utility Notebooks
*/} # Reference architectures Source: https://docs.pinecone.io/examples/reference-architectures
The official AWS Reference Architecture for high-scale systems using Pinecone.
[Documentation](https://github.com/pinecone-io/aws-reference-architecture-pulumi/blob/main/README.md) [Video tutorial](https://youtu.be/ySznARngHts) [Source code](https://github.com/pinecone-io/aws-reference-architecture-pulumi) # Sample apps Source: https://docs.pinecone.io/examples/sample-apps export const Tag = ({text, icon}) => { return {icon && } {text} ; }; export const UtilityExampleCard = ({title, text, link}) => { return

{title}

{text}

; }; export const ExampleCard = ({title, text, link, children, arrow, vectors, namespaces}) => { return

{title}

{text}

{children &&
{children}
} {arrow && } {(vectors || namespaces) &&
{vectors && {vectors} vectors} {namespaces && {namespaces} namespaces}
} ; };

More code examples

# Airbyte Source: https://docs.pinecone.io/integrations/airbyte export const PrimarySecondaryCTA = ({primaryLabel, primaryHref, primaryTarget, secondaryLabel, secondaryHref, secondaryTarget}) =>
{primaryLabel && primaryHref && } {secondaryLabel && secondaryHref && }
; Airbyte offers a platform for creating ETL pipelines without writing integration code. It supports integrations with hundreds of systems, including databases, data warehouses, and SaaS products. The Pinecone connector for Airbyte allows users to connect these systems directly into Pinecone. The connector fetches data from the connected source, embeds the data using an large language model (LLM), and then upserts it into Pinecone. From enhancing semantic search capabilities to building intelligent recommendation systems, the Pinecone Airbyte connector offers a versatile solution. By tapping into Airbyte's extensive array of source connectors, you can explore new ways to enrich your data-driven projects and achieve your specific goals. ## Related articles * [Data Sync and Search: Pinecone and Airbyte](https://www.pinecone.io/learn/series/airbyte/) * [Introduction to Airbyte and the Pinecone connector](https://www.pinecone.io/learn/series/airbyte/airbyte-and-pinecone-intro/) * [Postgres to Pinecone Syncing](https://www.pinecone.io/learn/series/airbyte/airbyte-postgres-to-pinecone/) # Amazon Bedrock Source: https://docs.pinecone.io/integrations/amazon-bedrock Pinecone as a Knowledge Base for Amazon Bedrock export const PrimarySecondaryCTA = ({primaryLabel, primaryHref, primaryTarget, secondaryLabel, secondaryHref, secondaryTarget}) =>
{primaryLabel && primaryHref && } {secondaryLabel && secondaryHref && }
; Users can now select Pinecone as a Knowledge Base for [Amazon Bedrock](https://aws.amazon.com/bedrock/), a fully managed service from Amazon Web Services (AWS) for building GenAI applications. The Pinecone vector database is a key component of the AI tech stack, helping companies solve one of the biggest challenges in deploying GenAI solutions — hallucinations — by allowing them to store, search, and find the most relevant and up-to-date information from company data and send that context to Large Language Models (LLMs) with every query. This workflow is called Retrieval Augmented Generation (RAG), and with Pinecone, it aids in providing relevant, accurate, and fast responses from search or GenAI applications to end users. With the release of Knowledge Bases for Amazon Bedrock, developers can integrate their enterprise data into Amazon Bedrock using Pinecone as the fully-managed vector database to build GenAI applications that are: * **Highly performant:** Speed through data in milliseconds. Leverage metadata filters and sparse-dense index support for top-notch relevance, ensuring quick, accurate, and grounded results across diverse search tasks. * **Cost effective at scale:** Start for free on the starter plan and seamlessly scale usage with transparent usage-based pricing. Add or remove resources to meet your desired capacity and performance, upwards of billions of embeddings. * **Enterprise ready:** Launch, use, and scale your AI solution without needing to maintain infrastructure, monitor services, or troubleshoot algorithms. Pinecone meets the security and operational requirements of enterprises. ## What are Agents for Amazon Bedrock? In Bedrock, users interact with **Agents** that are capable of combining the natural language interface of the supported LLMs with those of a **Knowledge Base.** Bedrock's Knowledge Base feature uses the supported LLMs to generate **embeddings** from the original data source. These embeddings are stored in Pinecone, and the Pinecone index is used to retrieve semantically relevant content upon the user's query to the agent. **Note:** the LLM used for embeddings may be different than the one used for the natural language generation. For example, you may choose to use Amazon Titan to generate embeddings and use Anthropic's Claude to generate natural language responses. Additionally, Agents for Amazon Bedrock may be configured to execute various actions in the context of responding to a user's query - but we won't get into this functionality in this post. ## What is a Knowledge Base for Amazon Bedrock? A Bedrock Knowledge base ingests raw text data or documents found in Amazon S3, embeds the content and upserts the embeddings into Pinecone. Then, a Bedrock agent can interact with the knowledge base to retrieve the most semantically relevant content given a user's query. Overall, the Knowledge Base feature is a valuable tool for users who want to improve their AI models' performance. With Bedrock's LLMs and Pinecone, users can easily integrate their data from AWS storage solutions and enhance the accuracy and relevance of their AI models. In this post, we'll go through the steps required for creating a Knowledge Base for Amazon Bedrock as well as an agent that will retrieve information from the knowledge base. ![](https://mintlify.s3.us-west-1.amazonaws.com/pinecone/images/AWS_Bedrock_Specific.png) ## Setup guide The process of using a Bedrock knowledge base with Pinecone works as follows: Create a Pinecone index with an embedding model in mind. Upload sample data to Amazon S3. Sync data with Bedrock to create embeddings saved in Pinecone. Use the knowledge base to reference the data saved in Pinecone. Agents can interact directly with the Bedrock knowledge base, which will retrieve the semantically relevant content. ### 1. Create a Pinecone index The knowledge base stores data in a Pinecone index. Decide which [supported embedding model](https://docs.aws.amazon.com/bedrock/latest/userguide/models-supported.html) to use with Bedrock before you create the index, as your index's dimensions will need to match the model's. For example, the AWS Titan Text Embeddings V2 model can use dimension sizes 1024, 384, and 256. After signing up to Pinecone, follow the [quickstart guide](/guides/get-started/quickstart) to create your Pinecone index and retrieve your `apiKey` and index host from the [Pinecone console](https://app.pinecone.io). Ensure that you create your index with the same dimensions as the model you will later select for creating your embeddings. ### 2. Set up your data source #### Set up secrets After setting up your Pinecone index, you'll have to create a secret in [AWS Secrets Manager](https://console.aws.amazon.com/secretsmanager/newsecret): 1. In the **Secret type** section, select **Other type of secret**. 2. In the **Key/value pairs** section, enter a key value pair for the Pinecone API key name and its respective value. For example, use `apiKey` and the API key value. ![](https://mintlify.s3.us-west-1.amazonaws.com/pinecone/images/bedrock/kv.png) 3. Click **Next**. 4. Enter a **Secret name** and **Description**. 5. Click **Next** to save your key. 6. On the **Configure rotation** page, select all the default options in the next screen, and click **Next**. 7. Click **Store**. 8. Click on the new secret you created and save the secret ARN for a later step. #### Set up S3 The knowledge base is going to draw on data saved in S3. For this example, we use a [sample of research papers](https://huggingface.co/datasets/jamescalam/ai-arxiv2-semantic-chunks) obtained from a dataset. This data will be embedded and then saved in Pinecone. 1. Create a new general purpose bucket in [Amazon S3](https://console.aws.amazon.com/s3/home). 2. Once the bucket is created, upload a CSV file. The CSV file must have a field for text that will be embedded, and a field for metadata to upload with each embedded text. 3. Save your bucket's address (`s3://…`) for the following configuration steps. ### 3. Create a Bedrock knowledge base To [create a Bedrock knowledge base](https://console.aws.amazon.com/bedrock/home?#/knowledge-bases/create-knowledge-base), use the following steps: 1. Enter a **Knowledge Base name**. 2. In the **Choose data source** section, select **Amazon S3**. 3. Click **Next**. 4. On the **Configure data source** page, enter the **S3 URI** for the bucket you created. 5. If you do not want to use the default chunking strategy, select a chunking strategy. 6. Click **Next**. ### 4. Connect Pinecone to the knowledge base Now you will need to select an embedding model to configure with Bedrock and configure the data sources. 1. Select the embedding model you decided on earlier. 2. For the **Vector database**, select **Choose a vector store you have created** and select **Pinecone**. 3. Mark the check box for authorizing AWS to access your Pinecone index. 4. For the **Endpoint URL**, enter the the Pinecone index host retrieved from the Pinecone console. 5. For the **Credentials secret ARN**, enter the secret ARN you created earlier. 6. In the **Metadata field mapping** section, enter the **Text field name** you want to embed and the **Bedrock-managed metadata field name** that will be used for metadata managed by Bedrock (e.g., `metadata`). 7. Click **Next**. 8. Review your selections and complete the creation of the knowledge base. 9. On the [Knowledge Bases](https://console.aws.amazon.com/bedrock/home?#/knowledge-bases) page select the knowledge base you just created to view its details. 10. Click **Sync** for the newly created data source. Sync the data source whenever you add new data to the data source to start the ingestion workflow of converting your Amazon S3 data into vector embeddings and upserting the embeddings into the vector database. Depending on the amount of data, this whole workflow can take some time. ### 5. Create and link an agent to Bedrock Lastly, [create an agent](https://console.aws.amazon.com/bedrock/home?#/agents) that will use the knowledge base for retrieval: 1. Click **Create Agent**. 2. Enter an **Name** and **Description**. 3. Click **Create**. 4. Select the LLM provider and model you'd like to use. 5. Provide instructions for the agent. These will define what the agent is trying to accomplish. 6. In the **Knowledge Bases** section, select the knowledge base you created. 7. Prepare the agent by clicking **Prepare** near the top of the builder page. 8. Test the agent after preparing it to verify it is using the knowledge base. 9. Click **Save and exit**. Your agent is now set up and ready to go! In the next section, we'll show how to interact with the newly created agent. #### Create an alias for your agent In order to deploy the agent, create an alias for it that points to a specific version of the agent. Once the alias is created, it will display in the agent view. 1. On the [Agents](https://console.aws.amazon.com/bedrock/home?#/agents) page, select the agent you created. 2. Click **Create Alias**. 3. Enter an **Alias name** and **Description**. 4. Click **Create alias**. #### Test the Bedrock agent To test the newly created agent, use the playground on the right of the screen when we open the agent. In this example, we used a dataset of research papers for our source data. We can ask a question about those papers and retrieve a detailed response, this time with the deployed version. ![](https://mintlify.s3.us-west-1.amazonaws.com/pinecone/images/bedrock/alias-testing-query.png) By inspecting the trace, we can see what chunks were used by the Agent and diagnose issues with responses. ![](https://mintlify.s3.us-west-1.amazonaws.com/pinecone/images/bedrock/alias-testing-trace.png) ## Related articles * [Pinecone as a Knowledge Base for Amazon Bedrock](https://www.pinecone.io/blog/amazon-bedrock-integration/) # Amazon SageMaker Source: https://docs.pinecone.io/integrations/amazon-sagemaker export const PrimarySecondaryCTA = ({primaryLabel, primaryHref, primaryTarget, secondaryLabel, secondaryHref, secondaryTarget}) =>
{primaryLabel && primaryHref && } {secondaryLabel && secondaryHref && }
; Amazon SageMaker is a fully managed service that brings together a broad set of tools to enable high-performance, low-cost machine learning (ML) for any use case. With SageMaker, you can build, train and deploy ML models at scale-- all in one integrated development environment (IDE). SageMaker supports governance requirements with simplified access control and transparency over your ML projects. Amazon SageMaker offers access to hundreds of pretrained models, including publicly available foundational models (FMs), and you can build your own FMs with purpose-built tools to fine-tune, experiment, retrain, and deploy FMs. Amazon SageMaker and Pinecone can be used together for high-performance, scalable, and reliable retrieval augmented generation (RAG) use cases. The integration uses Amazon SageMaker to compute and host models for large language Models (LLMs), and uses Pinecone as the knowledge base that keeps the LLMs up-to-date with the latest information, reducing the likelihood of hallucinations. ## Related articles * [Mitigate hallucinations through Retrieval Augmented Generation using Pinecone vector database & Llama-2 from Amazon SageMaker JumpStart](https://aws.amazon.com/blogs/machine-learning/mitigate-hallucinations-through-retrieval-augmented-generation-using-pinecone-vector-database-llama-2-from-amazon-sagemaker-jumpstart/) # Anyscale Source: https://docs.pinecone.io/integrations/anyscale export const PrimarySecondaryCTA = ({primaryLabel, primaryHref, primaryTarget, secondaryLabel, secondaryHref, secondaryTarget}) =>
{primaryLabel && primaryHref && } {secondaryLabel && secondaryHref && }
; Anyscale Endpoints offers open-source large language models (LLMs) as fully managed API endpoints. This allows you to focus on building applications powered by LLMs without the need to worry about the underlying infrastructure. Use Anyscale Endpoints with Canopy, an open source SDK that allows you to test, build, and package retrieval augmented generation (RAG) applications with Pinecone vector database. # Apify Source: https://docs.pinecone.io/integrations/apify export const PrimarySecondaryCTA = ({primaryLabel, primaryHref, primaryTarget, secondaryLabel, secondaryHref, secondaryTarget}) =>
{primaryLabel && primaryHref && } {secondaryLabel && secondaryHref && }
; [Apify](https://apify.com) is a web scraping and data extraction platform. It provides an app store with more than a thousand ready-made cloud tools called Actors. These tools are suitable for use cases including extracting structured data from e-commerce sites, social media, search engines, online maps, or any other website. For example, the [Website Content Crawler](https://apify.com/apify/website-content-crawler) Actor can deeply crawl websites, clean their HTML by removing a cookies modal, footer, or navigation, and then transform the HTML into Markdown. This Markdown can then be used as training data for AI models or to feed LLM and generative AI applications with web content. The Apify integration for Pinecone makes it easy to transfer results from Actors to the Pinecone vector database, enabling Retrieval-Augmented Generation (RAG) or semantic search over data extracted from the web. # Aryn Source: https://docs.pinecone.io/integrations/aryn export const PrimarySecondaryCTA = ({primaryLabel, primaryHref, primaryTarget, secondaryLabel, secondaryHref, secondaryTarget}) =>
{primaryLabel && primaryHref && } {secondaryLabel && secondaryHref && }
; Aryn is an AI-powered ETL system for complex, unstructured documents like PDFs, HTML, presentations, and more. It's purpose-built for building RAG and GenAI applications, providing up to 6x better accuracy in chunking and extracting information from documents. This can lead to 30% better recall and 2x improvement in answer accuracy for real-world use cases. Aryn's ETL system has two components: Sycamore and the Aryn Partitioning Service. Sycamore is Aryn's open source document processing engine, available as a Python library. It contains a set of transforms for information extraction, LLM-powered enrichment, data cleaning, creating vector embeddings, and loading Pinecone indexes. The Aryn Partitioning Service is used as a first step in a Sycamore data processing pipeline, and it identifies and extracts parts of documents, like text, tables, images, and more. It uses a state-of-the-art vision segmentation AI model, trained on hundreds of thousands of human-annotated documents. The Pinecone integration with Aryn enables developers to easily chunk documents, create vector embeddings, and load Pinecone with high-quality data. # Amazon Web Services (AWS) Source: https://docs.pinecone.io/integrations/aws export const PrimarySecondaryCTA = ({primaryLabel, primaryHref, primaryTarget, secondaryLabel, secondaryHref, secondaryTarget}) =>
{primaryLabel && primaryHref && } {secondaryLabel && secondaryHref && }
; Access Pinecone through our Amazon Web Services (AWS) Marketplace listing. AWS Marketplace allows you to manage Pinecone and other third-party software from a centralized location, and simplifies software licensing and procurement with flexible pricing options and multiple deployment methods. You can set up pay-as-you-go billing for a Pinecone organization through the AWS marketplace. # Azure Source: https://docs.pinecone.io/integrations/azure export const PrimarySecondaryCTA = ({primaryLabel, primaryHref, primaryTarget, secondaryLabel, secondaryHref, secondaryTarget}) =>
{primaryLabel && primaryHref && } {secondaryLabel && secondaryHref && }
; Access Pinecone through our Microsoft Azure marketplace listing. Azure Marketplace allows you to manage Pinecone and other third-party software from a centralized location, and simplifies software licensing and procurement with flexible pricing options and multiple deployment methods. You can set up pay-as-you-go billing for a Pinecone organization through the Azure marketplace. # Box Source: https://docs.pinecone.io/integrations/box export const PrimarySecondaryCTA = ({primaryLabel, primaryHref, primaryTarget, secondaryLabel, secondaryHref, secondaryTarget}) =>
{primaryLabel && primaryHref && } {secondaryLabel && secondaryHref && }
; Connect a [Box](https://www.box.com/) account to a Pinecone vector database. This integration allows embeddings generation based on content stored in a particular account and folder in Box. By default, the Pinecone Inference API is used, as well as OpenAI for the LLM. The integration can be used within a larger AI agent or workflow. # Attribute usage to your integration Source: https://docs.pinecone.io/integrations/build-integration/attribute-usage-to-your-integration Once you have created your integration with Pinecone, specify a **source tag** when instantiating clients with Pinecone SDKs, or pass a source tag as part of the `User-Agent` header when using the API directly. Anyone can create an integration, but [becoming an official Pinecone partner](/integrations/build-integration/become-a-partner) can help accelerate your go-to-market and add value to your customers. ### Source tag naming conventions Your source tag must follow these conventions: * Clearly identify your integration. * Use only lowercase letters, numbers, underscores, and colons. For example, for an integration called "New Framework", `"new_framework"` is valid, but `"new framework"` and `"New_framework"` are not valid. ### Specify a source tag | Pinecone SDK | Required version | | ------------------------------- | ---------------- | | [Python](/reference/python-sdk) | v3.2.1+ | | [Node.js](/reference/node-sdk) | v2.2.0+ | | [Java](/reference/java-sdk) | v1.0.0+ | | [Go](/reference/go-sdk) | v0.4.1+ | | [.NET](/reference/dotnet-sdk) | v1.0.0+ | ```python Python # REST client from pinecone import Pinecone pc = Pinecone( api_key="YOUR_API_KEY", source_tag="YOUR_SOURCE_TAG" ) # gRPC client from pinecone.grpc import PineconeGRPC pc = PineconeGRPC( api_key="YOUR_API_KEY", source_tag="YOUR_SOURCE_TAG" ) ``` ```javascript JavaScript import { Pinecone } from '@pinecone-database/pinecone'; const pc = new Pinecone({ apiKey: 'YOUR_API_KEY', sourceTag: 'YOUR_SOURCE_TAG' }); ``` ```java Java import io.pinecone.clients.Pinecone; public class IntegrationExample { public static void main(String[] args) { Pinecone pc = new Pinecone.Builder("YOUR_API_KEY") .withSourceTag("YOUR_SOURCE_TAG") .build(); } } ``` ```go Go import "github.com/pinecone-io/go-pinecone/v2/pinecone" client, err := pinecone.NewClient(pinecone.NewClientParams{ ApiKey: "YOUR_API_KEY", SourceTag: "YOUR_SOURCE_TAG", }) ``` ```csharp C# using Pinecone; var pinecone = new PineconeClient("YOUR_API_KEY", new ClientOptions { SourceTag = "YOUR_SOURCE_TAG", }); ``` ```shell curl curl -i -X GET "https://api.pinecone.io/indexes" \ -H "Accept: application/json" \ -H "Api-Key: YOUR_API_KEY" \ -H "User-Agent: source_tag=YOUR_SOURCE_TAG" \ -H "X-Pinecone-API-Version: 2025-04" ``` # Become a Pinecone partner Source: https://docs.pinecone.io/integrations/build-integration/become-a-partner Anyone can use the [Pinecone SDKs](/reference/pinecone-sdks) or the [Pinecone API](/reference/api/introduction) to build a native Pinecone integration into a third-party service or tool. However, becoming an official Pinecone partner can help accelerate your go-to-market and add value to your customers. To start the process, fill out our [application form](https://www.pinecone.io/partners/#sales-contact-form-submissions). ## Additional information * [Attribute usage to your integration](/integrations/build-integration/attribute-usage-to-your-integration) * [Connect your users to Pinecone](/integrations/build-integration/connect-your-users-to-pinecone) # Connect your users to Pinecone Source: https://docs.pinecone.io/integrations/build-integration/connect-your-users-to-pinecone To reduce friction for users using your integration, you can create a [custom object](#custom-object), like a button or link, to trigger a **Connect to Pinecone** popup from your app, website, or [Colab](https://colab.google/) notebook. Within this popup, your users can sign up for or log in to Pinecone, select or create an organization and project to connect to, and generate an API key. The API key is then communicated back to the user to copy or directly sent to the hosting page, app, or notebook. Alternatively, you can embed our [pre-built widget](#pre-built-widget), which provides the same functionality, but with the ease of a drop-in component. To start, [create an integration ID](#create-an-integration-id) for your app. Only [organization owners](/guides/organizations/manage-organization-members) can add or manage integrations. ## Create an integration ID Create a unique `integrationId` to enable usage of the **Connect to Pinecone** [popup](#custom-object) and [widget](#pre-built-widget): 1. On the the [**Integrations**](https://app.pinecone.io/organizations/-/settings/integrations) tab in the Pinecone console, click the **Create Integration** button. The **Integrations** tab does not display unless your organization already has integrations. [Follow this link to create your first integration](https://app.pinecone.io/organizations/-/settings/integrations?create=true). 2. Fill out the **Create integration** form: * **Integration name**: Give your integration a name. * **URL Slug**: This is your `integrationID`. Enter a human-readable string that uniquely identifies your integration and that may appear in URLs. Your integration URL slug is public and cannot be changed. * **Logo**: Upload a logo for your integration. * **Return mechanism**: Select one of the following return methods for the generated API key: * **Web Message**: Your application will receive the Pinecone API key via a web message. Select this option if you are using the [@pinecone-database/connect library](/integrations/build-integration/connect-your-users-to-pinecone#javascript). The API key will only be provided to the allowed origin(s) specified below. * **Copy/Paste**: The API key will display in the success message, and users will need to copy and paste their Pinecone API keys into your application. * **Allowed origin**: If you selected **Web Message** as your **Return mechanism**, list the URL origin(s) where your integration is hosted. The [origin](https://developer.mozilla.org/en-US/docs/Glossary/Origin) is the part of the URL that specifies the protocol, hostname, and port. 3. Click **Create**. Anyone can create an integration, but [becoming an official Pinecone partner](/integrations/build-integration/become-a-partner) can help accelerate your go-to-market and add value to your customers. ## Custom object [Once you have created your `integrationId`](#create-an-integration-id), you can create a custom object, like a button or link, that loads a **Connect to Pinecone** popup that displays as follows: ![Connect popup](https://mintlify.s3.us-west-1.amazonaws.com/pinecone/images/connect-popup.png) The `ConnectPopup` function can be called with either the JavaScript library or script. The JavaScript library is the most commonly used method, but the script can be used in instances where you cannot build and use a custom library, like within the constraints of a content management system (CMS). The function includes the following **required** configuration option: * `integrationId`: The slug assigned to the integration. If `integrationId` is not passed, the widget will not render. To create a unique `integrationId`, fill out the [Create Integration form](#create-an-integration-id). The function returns an object containing the following: * `open`: A function that opens the popup. Suitable for use as an on-click handler. Example usage of the library and script: ```javascript JavaScriptlibrary import { ConnectPopup } from '@pinecone-database/connect' /* Define a function called connectWithAPIKey */ const connectWithAPIKey = () => { return new Promise((resolve, reject) => { /* Call ConnectPopup function with an object containing options */ const popup = ConnectPopup({ onConnect: (key) => { resolve(key); }, integrationId: 'myApp' }).open(); }); }; /* Handle button click event */ document.getElementById('connectButton').addEventListener('click', () => { connectWithAPIKey() .then(apiKey => { console.log("API Key:", apiKey); }) .catch(error => { console.error("Error:", error); }); }); ``` ```html JavaScript script ... ... ... ... ``` Once you have created your integration, be sure to [attribute usage to your integration](/integrations/build-integration/attribute-usage-to-your-integration). ## Pre-built widget The pre-built **Connect** widget displays as follows: ![Connect widget](https://mintlify.s3.us-west-1.amazonaws.com/pinecone/images/connect-widget.png) [Once you have created your `integrationId`](#create-an-integration-id), you can embed the **Connect** widget multiple ways: * [JavaScript](#javascript) library (`@pinecone-database/connect`) or script: Renders the widget in apps and websites. * [Colab](#colab) (`pinecone-notebooks`): Renders the widget in Colab notebooks using Python. Once you have created your integration, be sure to [attribute usage to your integration](/integrations/build-integration/attribute-usage-to-your-integration). ### JavaScript To embed the **Connect to Pinecone** widget in your app or website using the [`@pinecone-database/connect` library](https://www.npmjs.com/package/@pinecone-database/connect), install the necessary dependencies: ```shell Shell # Install dependencies npm i -S @pinecone-database/connect ``` You can use the JavaScript library to render the **Connect to Pinecone** widget and obtain the API key with the [`connectToPinecone` function](#connecttopinecone-function). It displays the widget and calls the provided callback function with the Pinecone API key, once the user completes the flow. The function includes the following **required** configuration options: * `integrationId`: The slug assigned to the integration. If `integrationId` is not passed, the widget will not render. To create a unique `integrationId`, [fill out the Create Integration form](#create-an-integration-id) with Pinecone. * `container`: The HTML element where the **Connect** widget will render. Example usage: ```JavaScript JavaScript import {connectToPinecone} from '@pinecone-database/connect' const setupPinecone = (apiKey) => { /* Set up a Pinecone client using the API key */ } connectToPinecone( setupPinecone, { integrationId: 'myApp', container: document.getElementById('connect-widget') } ) ``` If you cannot use the JavaScript library, you can directly call the script. For example: ```html HTML ... ... ``` ### Colab To embed the **Connect** widget in your Colab notebook, use the [`pinecone-notebooks` Python library](https://pypi.org/project/pinecone-notebooks/#description): ```shell # Install dependencies using Colab syntax pip install -qU pinecone-notebooks pinecone[grpc] ``` ```python # Render the Connect widget for the user to authenticate and generate an API key from pinecone_notebooks.colab import Authenticate Authenticate() # The generated API key is available in the PINECONE_API_KEY environment variable from pinecone.grpc import PineconeGRPC as Pinecone from pinecone import ServerlessSpec import os api_key = os.environ.get('PINECONE_API_KEY') # Use the API key to initialize the Pinecone client pc = Pinecone(api_key=api_key) ``` To see this flow in practice, see our [example notebook](https://colab.research.google.com/drive/1VZ-REFRbleJG4tfJ3waFIrSveqrYQnNx?usp=sharing). ## Manage generated API keys Your users can [manage the API keys](/guides/projects/manage-api-keys) generated by your integration in the Pinecone console. # Cloudera AI Source: https://docs.pinecone.io/integrations/cloudera Vector embedding, RAG, and semantic search at scale export const PrimarySecondaryCTA = ({primaryLabel, primaryHref, primaryTarget, secondaryLabel, secondaryHref, secondaryTarget}) =>
{primaryLabel && primaryHref && } {secondaryLabel && secondaryHref && }
; [Cloudera AI](https://www.cloudera.com/) is an enterprise data cloud experience that provides scalable, secure, and agile machine learning and AI workflows. It leverages the power of Python, Apache Spark, R, and a host of other runtimes for distributed data processing, enabling the efficient creation, ingestion, and updating of vector embeddings at scale. The primary advantage of Cloudera AI lies in its integration with the Cloudera ecosystem, which facilitates seamless data flow and processing across various stages of machine learning and AI pipelines. Cloudera AI offers interactive sessions, collaborative projects, model hosting capabilities, and application hosting features, all within a Python-centric development environment. This multifaceted approach enables users to efficiently develop, train, and deploy machine learning and AI models at scale. Integrating Pinecone with Cloudera AI elevates the potential of Retrieval-Augmented Generation (RAG) models by providing a robust, scalable vector search platform. Pinecone's strength in handling vector embeddings—characterized by its ultra-low query latency, dynamic index updates, and scalability to billions of vector embeddings—make it the perfect match for the nuanced needs of RAG applications built on Cloudera AI. Within the Cloudera AI ecosystem, Pinecone acts as a first-class citizen for RAG by efficiently retrieving relevant context from massive datasets, enhancing the generation capabilities of models with relevant, real-time data. This integration enables the development of sophisticated machine learning and AI applications that combine the predictive power of Cloudera AI's hosted models with the dynamic retrieval capabilities of Pinecone, offering unparalleled accuracy and relevance for generated outputs. By leveraging Cloudera AI's project and session management features, developers can prototype, develop, and deploy these complex systems more effectively, making advanced machine learning and AI applications more accessible and practical for enterprise use. Cloudera's Accelerators for Machine Learning Projects (AMPs) drive efficient deployment of RAG architectures by doing the development work for you. This AMP serves as a prototype for fully integrating Pinecone into a RAG use case and illustrates semantic search with RAG at scale. ## Additional resources * [Python script](https://github.com/cloudera/CML_llm-hol/blob/main/2_populate_vector_db/pinecone_vectordb_insert.py) - Example of creating vectors in Pinecone * [Jupyter notebook](https://github.com/cloudera/CML_llm-hol/blob/main/3_query_vector_db/pinecone_vectordb_query.ipynb) - Example of querying Pinecone collections # Cohere Source: https://docs.pinecone.io/integrations/cohere Using Cohere and Pinecone to generate and index high-quality vector embeddings export const PrimarySecondaryCTA = ({primaryLabel, primaryHref, primaryTarget, secondaryLabel, secondaryHref, secondaryTarget}) =>
{primaryLabel && primaryHref && } {secondaryLabel && secondaryHref && }
; The Cohere platform builds natural language processing and generation into your product with a few lines of code. Cohere's large language models (LLMs) can solve a broad spectrum of natural language use cases, including classification, semantic search, paraphrasing, summarization, and content generation. Use the Cohere Embed API endpoint to generate language embeddings, and then index those embeddings in the Pinecone vector database for fast and scalable vector search. ## Setup guide [View source](https://github.com/pinecone-io/examples/blob/master/integrations/cohere/) [Open in Colab](https://colab.research.google.com/github/pinecone-io/examples/blob/master/integrations/cohere/semantic%5Fsearch%5Ftrec.ipynb) In this guide, you will learn how to use the [Cohere Embed API endpoint](https://docs.cohere.ai/reference/embed) to generate language embeddings, and then index those embeddings in the [Pinecone vector database](https://www.pinecone.io) for fast and scalable vector search. This is a powerful and common combination for building semantic search, question-answering, threat-detection, and other applications that rely on NLP and search over a large corpus of text data. The basic workflow looks like this: * Embed and index * Use the Cohere Embed API endpoint to generate vector embeddings of your documents (or any text data). * Upload those vector embeddings into Pinecone, which can store and index millions/billions of these vector embeddings, and search through them at ultra-low latencies. * Search * Pass your query text or document through the Cohere Embed API endpoint again. * Take the resulting vector embedding and send it as a [query](/guides/search/search-overview) to Pinecone. * Get back semantically similar documents, even if they don't share any keywords with the query. ![Basic workflow of Cohere with Pinecone](https://mintlify.s3.us-west-1.amazonaws.com/pinecone/images/fd0ba7b-pinecone-cohere-overview.png) ### Set up the environment Start by installing the Cohere and Pinecone clients and HuggingFace *Datasets* for downloading the TREC dataset used in this guide: ```shell Shell pip install -U cohere pinecone datasets ``` ### Create embeddings Sign up for an API key at [Cohere](https://dashboard.cohere.com/api-keys) and then use it to initialize your connection. ```Python Python import cohere co = cohere.Client("") ``` Load the **T**ext **RE**trieval **C**onference (TREC) question classification dataset, which contains 5.5K labeled questions. You will take only the first 1K samples for this walkthrough, but this can be scaled to millions or even billions of samples. ```Python Python from datasets import load_dataset # load the first 1K rows of the TREC dataset trec = load_dataset('trec', split='train[:1000]') ``` Each sample in `trec` contains two label features and the *text* feature. Pass the questions from the *text* feature to Cohere to create embeddings. ```Python Python embeds = co.embed( texts=trec['text'], model='embed-english-v3.0', input_type='search_document', truncate='END' ).embeddings ``` Check the dimensionality of the returned vectors. You will need to save the embedding dimensionality from this to be used when initializing your Pinecone index later ```Python Python import numpy as np shape = np.array(embeds).shape print(shape) # [Out]: # (1000, 1024) ``` You can see the `1024` embedding dimensionality produced by Cohere's `embed-english-v3.0` model, and the `1000` samples you built embeddings for. ### Store the Embeddings Now that you have your embeddings, you can move on to indexing them in the Pinecone vector database. For this, you need a [Pinecone API key](/guides/projects/manage-api-keys). You first initialize our connection to Pinecone and then create a new index called `cohere-pinecone-trec` for storing the embeddings. When creating the index, you specify that you would like to use the cosine similarity metric to align with Cohere's embeddings, and also pass the embedding dimensionality of `1024`. ```Python Python from pinecone import Pinecone # initialize connection to pinecone (get API key at app.pinecone.io) pc = Pinecone(api_key='YOUR_API_KEY') index_name = 'cohere-pinecone-trec' # if the index does not exist, we create it if not pc.has_index(index_name): pc.create_index( name=index_name, dimension=shape[1], metric="cosine", spec=ServerlessSpec( cloud='aws', region='us-east-1' ) ) # connect to index index = pc.Index(index_name) ``` Now you can begin populating the index with your embeddings. Pinecone expects you to provide a list of tuples in the format *(id, vector, metadata)*, where the *metadata* field is an optional extra field where you can store anything you want in a dictionary format. For this example, you will store the original text of the embeddings. While uploading your data, you will batch everything to avoid pushing too much data in one go. ```Python Python batch_size = 128 ids = [str(i) for i in range(shape[0])] # create list of metadata dictionaries meta = [{'text': text} for text in trec['text']] # create list of (id, vector, metadata) tuples to be upserted to_upsert = list(zip(ids, embeds, meta)) for i in range(0, shape[0], batch_size): i_end = min(i+batch_size, shape[0]) index.upsert(vectors=to_upsert[i:i_end]) # let's view the index statistics print(index.describe_index_stats()) # [Out]: # {'dimension': 1024, # 'index_fullness': 0.0, # 'namespaces': {'': {'vector_count': 1000}}, # 'total_vector_count': 1000} ``` You can see from `index.describe_index_stats` that you have a *1024-dimensionality* index populated with *1000* embeddings. Note that serverless indexes scale automatically as needed, so the `index_fullness` metric is relevant only for pod-based indexes. ### Semantic search Now that you have your indexed vectors, you can perform a few search queries. When searching, you will first embed your query using Cohere, and then search using the returned vector in Pinecone. ```Python Python query = "What caused the 1929 Great Depression?" # create the query embedding xq = co.embed( texts=[query], model='embed-english-v3.0', input_type='search_query', truncate='END' ).embeddings print(np.array(xq).shape) # query, returning the top 5 most similar results res = index.query(vector=xq, top_k=5, include_metadata=True) ``` The response from Pinecone includes your original text in the `metadata` field. Let's print out the `top_k` most similar questions and their respective similarity scores. ```Python Python for match in res['matches']: print(f"{match['score']:.2f}: {match['metadata']['text']}") # [Out]: # 0.62: Why did the world enter a global depression in 1929 ? # 0.49: When was `` the Great Depression '' ? # 0.38: What crop failure caused the Irish Famine ? # 0.32: What caused Harry Houdini 's death ? # 0.31: What causes pneumonia ? ``` Looks good! Let's make it harder and replace *"depression"* with the incorrect term *"recession"*. ```Python Python query = "What was the cause of the major recession in the early 20th century?" # create the query embedding xq = co.embed( texts=[query], model='embed-english-v3.0', input_type='search_query', truncate='END' ).embeddings # query, returning the top 5 most similar results res = index.query(vector=xq, top_k=5, include_metadata=True) for match in res['matches']: print(f"{match['score']:.2f}: {match['metadata']['text']}") # [Out]: # 0.43: When was `` the Great Depression '' ? # 0.40: Why did the world enter a global depression in 1929 ? # 0.39: When did World War I start ? # 0.35: What are some of the significant historical events of the 1990s ? # 0.32: What crop failure caused the Irish Famine ? ``` Let's perform one final search using the definition of depression rather than the word or related words. ```Python Python query = "Why was there a long-term economic downturn in the early 20th century?" # create the query embedding xq = co.embed( texts=[query], model='embed-english-v3.0', input_type='search_query', truncate='END' ).embeddings # query, returning the top 10 most similar results res = index.query(vector=xq, top_k=10, include_metadata=True) for match in res['matches']: print(f"{match['score']:.2f}: {match['metadata']['text']}") # [Out]: # 0.40: When was `` the Great Depression '' ? # 0.39: Why did the world enter a global depression in 1929 ? # 0.35: When did World War I start ? # 0.32: What are some of the significant historical events of the 1990s ? # 0.31: What war did the Wanna-Go-Home Riots occur after ? # 0.31: What do economists do ? # 0.29: What historical event happened in Dogtown in 1899 ? # 0.28: When did the Dow first reach ? # 0.28: Who earns their money the hard way ? # 0.28: What were popular songs and types of songs in the 1920s ? ``` It's clear from this example that the semantic search pipeline is clearly able to identify the meaning between each of your queries. Using these embeddings with Pinecone allows you to return the most semantically similar questions from the already indexed TREC dataset. # Confluent Source: https://docs.pinecone.io/integrations/confluent export const PrimarySecondaryCTA = ({primaryLabel, primaryHref, primaryTarget, secondaryLabel, secondaryHref, secondaryTarget}) =>
{primaryLabel && primaryHref && } {secondaryLabel && secondaryHref && }
; Confluent allows you to connect and process all of your data in real time with a cloud-native and complete data streaming platform available everywhere you need it. Confluent's Kafka Connect is a tool for scalably and reliably streaming data between Apache Kafka and other data systems. It makes it simple to quickly define connectors that move large data sets in and out of Kafka. Use the Pinecone Sink Connector, a Kafka Connect plugin for Pinecone, to take content from Confluent Cloud, convert it into vector embeddings using large language models (LLMs), and then store these embeddings in a Pinecone vector database. # Context Data Source: https://docs.pinecone.io/integrations/context-data export const PrimarySecondaryCTA = ({primaryLabel, primaryHref, primaryTarget, secondaryLabel, secondaryHref, secondaryTarget}) =>
{primaryLabel && primaryHref && } {secondaryLabel && secondaryHref && }
; Using [Context Data](https://contextdata.ai/), you can easily create end-to-end data flows by connecting to a myriad of data sources (PostgreSQL, MySQL, Amazon S3, Salesforce, etc.), seamlessly embedding and writing the results to Pinecone using Context Data's super simple no-code web interface. These flows can also be configured to be triggered and run on a user-defined schedule. Additionally, Context Data provides the ability to create transformations like aggregations, joins, and feature engineering using SQL common table expressions before writing to Pinecone. You also have the ability to "chat" with your Pinecone indexes directly from Context Data's privacy-focused Query Studio. # Databricks Source: https://docs.pinecone.io/integrations/databricks Using Databricks and Pinecone to create and index vector embeddings at scale export const PrimarySecondaryCTA = ({primaryLabel, primaryHref, primaryTarget, secondaryLabel, secondaryHref, secondaryTarget}) =>
{primaryLabel && primaryHref && } {secondaryLabel && secondaryHref && }
; Databricks is a Unified Analytics Platform on top of Apache Spark. The primary advantage of using Spark is its ability to distribute workloads across a cluster of machines. By adding more machines or increasing the number of cores on each machine, it is easy to horizontally scale a cluster to handle computationally intensive tasks like vector embedding, where parallelization can save many hours of precious computation time and resources. Leveraging GPUs with Spark can produce even better results — enjoying the benefits of the fast computation of a GPU combined with parallelization will ensure optimal performance. Efficiently create, ingest, and update vector embeddings at scale with Databricks and Pinecone. ## Setup guide In this guide, you'll create embeddings based on the [sentence-transformers/all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) model from [Hugging Face](https://huggingface.co/), but the approach demonstrated here should work with any other model and dataset. ### Before you begin Ensure you have the following: * A [Databricks cluster](https://docs.databricks.com/en/compute/configure.html) * A [Pinecone account](https://app.pinecone.io/) * A [Pinecone API key](/guides/projects/understanding-projects#api-keys) ### 1. Install the Spark-Pinecone connector 1. [Install the Spark-Pinecone connector as a library](https://docs.databricks.com/en/libraries/cluster-libraries.html#install-a-library-on-a-cluster). 2. Configure the library as follows: 1. Select **File path/S3** as the **Library Source**. 2. Enter the S3 URI for the Pinecone assembly JAR file: ``` s3://pinecone-jars/1.1.0/spark-pinecone-uberjar.jar ``` Databricks platform users must use the Pinecone assembly jar listed above to ensure that the proper dependecies are installed. 3. Click **Install**. 1. [Install the Spark-Pinecone connector as a library](https://docs.databricks.com/en/libraries/cluster-libraries.html#install-a-library-on-a-cluster). 2. Configure the library as follows: 1. Select **File path/S3** as the **Library Source**. 2. Enter the S3 URI for the Pinecone assembly JAR file: ``` s3://pinecone-jars/1.1.0/spark-pinecone-uberjar.jar ``` 3. Click **Install**. 1. [Install the Spark-Pinecone connector as a library](https://docs.databricks.com/en/libraries/cluster-libraries.html#install-a-library-on-a-cluster). 2. Configure the library as follows: 1. [Download the Pinecone assembly JAR file](https://repo1.maven.org/maven2/io/pinecone/spark-pinecone_2.12/1.1.0/). 2. Select **Workspace** as the **Library Source**. 3. Upload the JAR file. 4. Click **Install**. ### 2. Load the dataset into partitions As your example dataset, use a collection of news articles from Hugging Face's datasets library: 1. [Create a new notebook](https://docs.databricks.com/en/notebooks/notebooks-manage.html#create-a-notebook) attached to your cluster. 2. Install dependencies: ``` pip install datasets transformers pinecone torch ``` 3. Load the dataset: ```Python Python from datasets import list_datasets, load_dataset dataset_name = "allenai/multinews_sparse_max" dataset = load_dataset(dataset_name, split="train") ``` 4. Convert the dataset from the Hugging Face format and repartition it: ```Python Python dataset.to_parquet("/dbfs/tmp/dataset_parquet.pq") num_workers = 10 dataset_df = spark.read.parquet("/tmp/dataset_parquet.pq").repartition(num_workers) ``` Once the repartition is complete, you get back a DataFrame, which is a distributed collection of the data organized into named columns. It is conceptually equivalent to a table in a relational database or a dataframe in R/Python, but with richer optimizations under the hood. As mentioned above, each partition in the dataframe has an equal amount of the original data. 5. The dataset doesn't have identifiers associated with each document, so add them: ```Python Python from pyspark.sql.types import StringType from pyspark.sql.functions import monotonically_increasing_id dataset_df = dataset_df.withColumn("id", monotonically_increasing_id().cast(StringType())) ``` As its name suggests, `withColumn` adds a column to the dataframe, containing a simple increasing identifier that you cast to a string. ### 3. Create the vector embeddings 1. Create a UDF (User-Defined Function) to create the embeddings, using the AutoTokenizer and AutoModel classes from the Hugging Face transformers library: ```Python Python from transformers import AutoTokenizer, AutoModel def create_embeddings(partitionData): tokenizer = AutoTokenizer.from_pretrained("sentence-transformers/all-MiniLM-L6-v2") model = AutoModel.from_pretrained("sentence-transformers/all-MiniLM-L6-v2") for row in partitionData: document = str(row.document) inputs = tokenizer(document, padding=True, truncation=True, return_tensors="pt", max_length=512) result = model(**inputs) embeddings = result.last_hidden_state[:, 0, :].cpu().detach().numpy() lst = embeddings.flatten().tolist() yield [row.id, lst, "", "{}", None] ``` 2. Apply the UDF to the data: ```Python Python embeddings = dataset_df.rdd.mapPartitions(create_embeddings) ``` A dataframe in Spark is a higher-level abstraction built on top of a more fundamental building block called a resilient distributed dataset (RDD). Here, you use the `mapPartitions` function, which provides finer control over the execution of the UDF by explicitly applying it to each partition of the RDD. 3. Convert the resulting RDD back into a dataframe with the schema required by Pinecone: ```Python Python from pyspark.sql.types import StructType, StructField, StringType, ArrayType, FloatType, IntegerType schema = StructType([ StructField("id",StringType(),True), StructField("values",ArrayType(FloatType()),True), StructField("namespace",StringType(),True), StructField("metadata", StringType(), True), StructField("sparse_values", StructType([ StructField("indices", ArrayType(LongType(), False), False), StructField("values", ArrayType(FloatType(), False), False) ]), True) ]) embeddings_df = spark.createDataFrame(data=embeddings,schema=schema) ``` ### 4. Save the embeddings in Pinecone 1. Initialize the connection to Pinecone: ```Python Python from pinecone.grpc import PineconeGRPC as Pinecone from pinecone import ServerlessSpec pc = Pinecone(api_key="YOUR_API_KEY") ``` 2. Create an index for your embeddings: ```Python Python pc.create_index( name="news", dimension=1536, metric="cosine", spec=ServerlessSpec( cloud="aws", region="us-east-1" ) ) ``` 3. Use the Spark-Pinecone connector to save the embeddings to your index: ```Python Python ( embeddings_df.write .option("pinecone.apiKey", api_key) .option("pinecone.indexName", index_name) .format("io.pinecone.spark.pinecone.Pinecone") .mode("append") .save() ) ``` The process of writing the embeddings to Pinecone should take approximately 15 seconds. When it completes, you'll see the following: ``` spark: org.apache.spark.sql.SparkSession = org.apache.spark.sql.SparkSession@41638051 pineconeOptions: scala.collection.immutable.Map[String,String] = Map(pinecone.apiKey ->, pinecone.indexName -> "news") ``` This means the process was completed successfully and the embeddings have been stored in Pinecone. 4. Perform a similarity search using the embeddings you loaded into Pinecone by providing a set of vector values or a vector ID. The [query endpoint](https://docs.pinecone.io/reference/api/2024-10/data-plane/query) will return the IDs of the most similar records in the index, along with their similarity scores: ```Python Python index.query( namespace="example-namespace", vector=[0.3, 0.3, 0.3, 0.3, ... 0.3], top_k=3, include_values=True ) ``` If you want to make a query with a text string (e.g., `"Summarize this article"`), use the [`search` endpoint via integrated inference](/reference/api/2025-01/data-plane/search_records). # Datadog Source: https://docs.pinecone.io/integrations/datadog Monitoring Pinecone with Datadog export const PrimarySecondaryCTA = ({primaryLabel, primaryHref, primaryTarget, secondaryLabel, secondaryHref, secondaryTarget}) =>
{primaryLabel && primaryHref && } {secondaryLabel && secondaryHref && }
; This feature is available on the [Standard, Enterprise, and Dedicated plans](https://www.pinecone.io/pricing/). Datadog is a monitoring and analytics tool that can be used to determine performance metrics as well as event monitoring for infrastructure and cloud services. Use Datadog to: * Optimize performance and control usage: Observe and track specific actions (e.g., request count) within Pinecone to identify application requests with high latency or usage. Monitor trends and gain actionable insights to improve resource utilization and reduce spend. * Automatically alert on metrics: Get alerted when index fullness reaches a certain threshold. You can also create your own customized monitors to alert on specific metrics and thresholds. * Locate and triage unexpected spikes in usage or latency: Quickly visualize anomalies in usage or latency in Pinecone's Datadog dashboard. View metrics over time to better understand trends and determine the severity of a spike. ## Setup guide Follow these steps to monitor a Pinecone project with Datadog: 1. Go to the [Pinecone integration](https://app.datadoghq.com/integrations/pinecone) tile in Datadog. 2. Go to the **Configure** tab. 3. Click **+ Add New**. 4. Enter a project name to identify your project in Datadog. 5. Do not select an environment. This is a legacy setting. 6. Enter an [API key](/guides/projects/understanding-projects#api-keys) for the Pinecone project you want to monitor. 7. Enter the [project ID](/guides/projects/understanding-projects#project-ids) of the Pinecone project you want to monitor. 8. Save the configuration. On the **Monitoring Resources** tab, you'll find dashboards for the pod-based and serverless indexes in your project and recommendations for [configuring monitors](https://docs.datadoghq.com/monitors/configuration/?tab=thresholdalert) using [Pinecone's metrics](/guides/production/monitoring#available-metrics). # Datavolo Source: https://docs.pinecone.io/integrations/datavolo export const PrimarySecondaryCTA = ({primaryLabel, primaryHref, primaryTarget, secondaryLabel, secondaryHref, secondaryTarget}) =>
{primaryLabel && primaryHref && } {secondaryLabel && secondaryHref && }
; [Datavolo](https://datavolo.io/) helps data teams build multimodal data pipelines to support their AI initiatives. Every organization has their own private data that they need to incorporate into their AI apps, and a predominant pattern to do so has emerged: retrieval augmented generation (RAG). Datavolo sources, transforms, and enriches data in a continuous, composable and customizable manner, landing the data in Pinecone for retrieval. This ensures organizations can securely access their unstructured data. # Estuary Source: https://docs.pinecone.io/integrations/estuary export const PrimarySecondaryCTA = ({primaryLabel, primaryHref, primaryTarget, secondaryLabel, secondaryHref, secondaryTarget}) =>
{primaryLabel && primaryHref && } {secondaryLabel && secondaryHref && }
; [Estuary](https://estuary.dev/) builds real-time data pipelines that focus on moving data from sources to destinations with millisecond latency. It supports integrations with hundreds of systems including databases, warehouses, SaaS products and streaming solutions. The Pinecone connector for Estuary enables users to source from these systems and push data to Pinecone, for an always up-to-date view. It incrementally updates source data to ensure that minimal credits are used when reaching out to get embeddings from providers prior to pushing them to Pinecone. Estuary's Pinecone connector enables a variety of use cases like enabling LLM-based search across your organizations data and building intelligent recommendation systems. It can be set up in 5 minutes, without any engineering effort to maximize efficiency. # Fleak Source: https://docs.pinecone.io/integrations/fleak export const PrimarySecondaryCTA = ({primaryLabel, primaryHref, primaryTarget, secondaryLabel, secondaryHref, secondaryTarget}) =>
{primaryLabel && primaryHref && } {secondaryLabel && secondaryHref && }
; Fleak simplifies the process of building, deploying, and managing data workflows. As a low-code platform, Fleak lets users create and deploy complex workflows using SQL and pre-configured processing nodes. The platform facilitates seamless data processing, microservice interactions, and the inference of large language models (LLMs) within a single, intuitive environment. Fleak makes advanced technology accessible and manageable for Pinecone users without requiring extensive coding hours or infrastructure knowledge. The platform provides serverless, autoscaling HTTP API endpoints, ensuring that workflows are robust, reliable, and scalable to meet the complex data needs of enterprises. This setup allows businesses to automate and enhance their operations, driving productivity and innovation through powerful, user-friendly tools. By integrating Pinecone into Fleak's platform, users can access and leverage their vector data to enrich their workflows without additional engineering overhead, enabling seamless data-driven decision-making, advanced analytics, and the integration of AI-driven insights. # FlowiseAI Source: https://docs.pinecone.io/integrations/flowise export const PrimarySecondaryCTA = ({primaryLabel, primaryHref, primaryTarget, secondaryLabel, secondaryHref, secondaryTarget}) =>
{primaryLabel && primaryHref && } {secondaryLabel && secondaryHref && }
; Flowise is a low-code LLM apps development platform. It supports integrations with dozens of systems, including databases and chat models. The Pinecone integration with Flowise allows users to build RAG apps, including upserting and querying documents. # Gathr Source: https://docs.pinecone.io/integrations/gathr export const PrimarySecondaryCTA = ({primaryLabel, primaryHref, primaryTarget, secondaryLabel, secondaryHref, secondaryTarget}) =>
{primaryLabel && primaryHref && } {secondaryLabel && secondaryHref && }
; [Gathr](https://www.gathr.one/) is the world's first and only "data to outcome" platform. Leading enterprises use Gathr to build and operationalize data and AI-driven solutions at scale. Gathr unifies data engineering, machine learning, generative AI, actionable analytics, and process automation on a single platform. With Gen AI capabilities and no-code rapid application development, Gathr significantly boosts productivity for all. The unified experience fosters seamless handoff and collaboration between teams, accelerating the journey from prototype to production. Users have achieved success with Gathr, from ingesting petabyte-scale data in real time to orchestrating thousands of complex data processing pipelines in months and delivering actionable insights and xOps solutions to multiply business impact. Additionally, Gathr helps enterprises architect Gen AI solutions for use cases like document summarization, sentiment analysis, next best action, insider threat detection, predictive maintenance, custom chatbots, and more. Gathr Gen AI Fabric is designed to build enterprise-grade Gen AI solutions end-to-end on a unified platform. It offers production-ready building blocks for creating Gen AI solutions, out-of-the-box Gen AI solution templates, and GathrIQ, a data-to-outcome copilot. One of the building blocks is integration with Vector DB and Knowledge Graphs. Gathr supports reading and writing from Pinecone using a built-in, ready-to-use connector to support use cases requiring knowledge graphs. # Google Cloud Platform (GCP) Source: https://docs.pinecone.io/integrations/gcp export const PrimarySecondaryCTA = ({primaryLabel, primaryHref, primaryTarget, secondaryLabel, secondaryHref, secondaryTarget}) =>
{primaryLabel && primaryHref && } {secondaryLabel && secondaryHref && }
; Access Pinecone through our Google Cloud Marketplace listing. Google Cloud Marketplace allows you to manage Pinecone and other third-party software from a centralized location, and simplifies software licensing and procurement with flexible pricing options and multiple deployment methods. You can set up pay-as-you-go billing for a Pinecone organization through the Google Cloud Marketplace. # Genkit Source: https://docs.pinecone.io/integrations/genkit export const PrimarySecondaryCTA = ({primaryLabel, primaryHref, primaryTarget, secondaryLabel, secondaryHref, secondaryTarget}) =>
{primaryLabel && primaryHref && } {secondaryLabel && secondaryHref && }
; The [Genkit](https://firebase.google.com/docs/genkit) Pinecone plugin empowers developers to reduce the complexity of integrating AI components through simple indexers, embedders and retrievers abstractions. Through the Genkit Pinecone plugin, developers can integrate AI models with their own custom logic and data to build AI features optimized for their businesses. Additionally, developers can analyze unstructured text, generate creative content, select tasks, and send results back to their app as structured type-safe objects. The plugin provides a common format for content that supports combinations of text, data, and other media. Developers can use Genkit for models that perform any generative task (such as image generation), not just LLMs. # GitHub Copilot Source: https://docs.pinecone.io/integrations/github-copilot export const PrimarySecondaryCTA = ({primaryLabel, primaryHref, primaryTarget, secondaryLabel, secondaryHref, secondaryTarget}) =>
{primaryLabel && primaryHref && } {secondaryLabel && secondaryHref && }
; Access the Pinecone Copilot Extension through our GitHub Marketplace listing. The Pinecone Copilot Extension serves as a seamless bridge between you and your Pinecone data-- providing product information, coding assistance, troubleshooting capabilities and streamlining the debugging process. This extension offers personalized recommendations right to your fingertips, enabling you to swiftly retrieve relevant data and collaborate effectively with Copilot. # Haystack Source: https://docs.pinecone.io/integrations/haystack Using Haystack and Pinecone to keep your NLP-driven apps up-to-date export const PrimarySecondaryCTA = ({primaryLabel, primaryHref, primaryTarget, secondaryLabel, secondaryHref, secondaryTarget}) =>
{primaryLabel && primaryHref && } {secondaryLabel && secondaryHref && }
; Haystack is the open source Python framework by Deepset for building custom apps with large language models (LLMs). It lets you quickly try out the latest models in natural language processing (NLP) while being flexible and easy to use. Their community of users and builders has helped shape Haystack into what it is today: a complete framework for building production-ready NLP apps. Haystack and Pinecone integration can be used to keep your NLP-driven apps up-to-date with Haystack's indexing pipelines that help you prepare and maintain your data. ## Setup guide In this guide we will see how to integrate Pinecone and the popular [Haystack library](https://github.com/deepset-ai/haystack) for *Question-Answering*. ### Install Haystack We start by installing the latest version of Haystack with all dependencies required for the `PineconeDocumentStore`. ```Python Python pip install -U farm-haystack>=1.3.0 pinecone[grpc] datasets ``` ### Initialize the PineconeDocumentStore We initialize a `PineconeDocumentStore` by providing an API key and environment name. [Create an account](https://app.pinecone.io) to get your free API key. ```Python Python from haystack.document_stores import PineconeDocumentStore document_store = PineconeDocumentStore( api_key='', index='haystack-extractive-qa', similarity="cosine", embedding_dim=384 ) ``` ``` INFO - haystack.document_stores.pinecone - Index statistics: name: haystack-extractive-qa, embedding dimensions: 384, record count: 0 ``` ### Prepare data Before adding data to the document store, we must download and convert data into the Document format that Haystack uses. We will use the SQuAD dataset available from Hugging Face Datasets. ```Python Python from datasets import load_dataset # load the squad dataset data = load_dataset("squad", split="train") ``` Next, we remove duplicates and unecessary columns. ```Python Python # convert to a pandas dataframe df = data.to_pandas() # select only title and context column df = df[["title", "context"]] # drop rows containing duplicate context passages df = df.drop_duplicates(subset="context") df.head() ``` | title | context | | | ----- | --------------------------- | ------------------------------------------------- | | 0 | University\_of\_Notre\_Dame | Architecturally, the school has a Catholic cha... | | 5 | University\_of\_Notre\_Dame | As at most other universities, Notre Dame's st... | | 10 | University\_of\_Notre\_Dame | The university is the major seat of the Congre... | | 15 | University\_of\_Notre\_Dame | The College of Engineering was established in ... | | 20 | University\_of\_Notre\_Dame | All of Notre Dame's undergraduate students are... | Then convert these records into the Document format. ```Python Python from haystack import Document docs = [] for d in df.iterrows(): d = d[1] # create haystack document object with text content and doc metadata doc = Document( content=d["context"], meta={ "title": d["title"], 'context': d['context'] } ) docs.append(doc) ``` This `Document` format contains two fields; *'content'* for the text content or paragraphs, and *'meta'* where we can place any additional information that can later be used to apply metadata filtering in our search. Now we upsert the documents to Pinecone. ```Python Python # upsert the data document to pinecone index document_store.write_documents(docs) ``` ### Initialize retriever The next step is to create embeddings from these documents. We will use Haystacks `EmbeddingRetriever` with a SentenceTransformer model (`multi-qa-MiniLM-L6-cos-v1`) which has been designed for question-answering. ```Python Python from haystack.retriever.dense import EmbeddingRetriever retriever = EmbeddingRetriever( document_store=document_store, embedding_model="multi-qa-MiniLM-L6-cos-v1", model_format="sentence_transformers" ) ``` Then we run the `PineconeDocumentStore.update_embeddings` method with the `retriever` provided as an argument. GPU acceleration can greatly reduce the time required for this step. ```Python Python document_store.update_embeddings( retriever, batch_size=16 ) ``` ### Inspect documents and embeddings We can get documents by their ID with the `PineconeDocumentStore.get_documents_by_id` method. ```Python Python d = document_store.get_documents_by_id(ids=['49091c797d2236e73fab510b1e9c7f6b'], return_embedding=True)[0] ``` From here we return can view document content with `d.content` and the document embedding with `d.embedding`. ### Initialize an extractive QA pipeline An `ExtractiveQAPipeline` contains three key components by default: * a document store (`PineconeDocumentStore`) * a retriever model * a reader model We use the `deepset/electra-base-squad2` model from the HuggingFace model hub as our reader model. ```Python Python from haystack.nodes import FARMReader reader = FARMReader( model_name_or_path='deepset/electra-base-squad2', use_gpu=True ) ``` We are now ready to initialize the `ExtractiveQAPipeline`. ```Python Python from haystack.pipelines import ExtractiveQAPipeline pipe = ExtractiveQAPipeline(reader, retriever) ``` ### Ask Questions Using our QA pipeline we can begin querying with `pipe.run`. ```Python Python from haystack.utils import print_answers query = "What was Albert Einstein famous for?" # get the answer answer = pipe.run( query=query, params={ "Retriever": {"top_k": 1}, } ) # print the answer(s) print_answers(answer) ``` ``` Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 3.53 Batches/s] Query: What was Albert Einstein famous for? Answers: [ ] ``` ```Python Python query = "How much oil is Egypt producing in a day?" # get the answer answer = pipe.run( query=query, params={ "Retriever": {"top_k": 1}, } ) # print the answer(s) print_answers(answer) ``` ``` Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 3.81 Batches/s] Query: How much oil is Egypt producing in a day? Answers: [ ] ``` ```Python Python query = "What are the first names of the youtube founders?" # get the answer answer = pipe.run( query=query, params={ "Retriever": {"top_k": 1}, } ) # print the answer(s) print_answers(answer) ``` ``` Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 3.83 Batches/s] Query: What are the first names of the youtube founders? Answers: [ ] ``` We can return multiple answers by setting the `top_k` parameter. ```Python Python query = "Who was the first person to step foot on the moon?" # get the answer answer = pipe.run( query=query, params={ "Retriever": {"top_k": 3}, } ) # print the answer(s) print_answers(answer) ``` ``` Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 3.71 Batches/s] Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 3.78 Batches/s] Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 3.88 Batches/s] Query: Who was the first person to step foot on the moon? Answers: [ , , ] ``` # Hugging Face Inference Endpoints Source: https://docs.pinecone.io/integrations/hugging-face-inference-endpoints Using Hugging Face Inference Endpoints and Pinecone to generate and index high-quality vector embeddings export const PrimarySecondaryCTA = ({primaryLabel, primaryHref, primaryTarget, secondaryLabel, secondaryHref, secondaryTarget}) =>
{primaryLabel && primaryHref && } {secondaryLabel && secondaryHref && }
; Hugging Face Inference Endpoints offers a secure production solution to easily deploy any Hugging Face Transformers, Sentence-Transformers and Diffusion models from the Hub on dedicated and autoscaling infrastructure managed by Hugging Face. Coupled with Pinecone, you can use Hugging Face to generate and index high-quality vector embeddings with ease. ## Setup guide Hugging Face Inference Endpoints allows access to straightforward model inference. Coupled with Pinecone we can generate and index high-quality vector embeddings with ease. Let's get started by initializing an Inference Endpoint for generating vector embeddings. ### Create an endpoint We start by heading over to the [Hugging Face Inference Endpoints homepage](https://ui.endpoints.huggingface.co/endpoints) and signing up for an account if needed. After, we should find ourselves on this page: ![endpoints 0](https://mintlify.s3.us-west-1.amazonaws.com/pinecone/images/hf-endpoints-0.png) We click on **Create new endpoint**, choose a model repository (eg name of the model), endpoint name (this can be anything), and select a cloud environment. Before moving on it is *very important* that we set the **Task** to **Sentence Embeddings** (found within the *Advanced configuration* settings). ![endpoints 1](https://mintlify.s3.us-west-1.amazonaws.com/pinecone/images/hf-endpoints-1.png) ![endpoints 2](https://mintlify.s3.us-west-1.amazonaws.com/pinecone/images/hf-endpoints-2.png) Other important options include the *Instance Type*, by default this uses CPU which is cheaper but also slower. For faster processing we need a GPU instance. And finally, we set our privacy setting near the end of the page. After setting our options we can click **Create Endpoint** at the bottom of the page. This action should take use to the next page where we will see the current status of our endpoint. ![endpoints 3](https://mintlify.s3.us-west-1.amazonaws.com/pinecone/images/hf-endpoints-3.png) Once the status has moved from **Building** to **Running** (this can take some time), we're ready to begin creating embeddings with it. ## Create embeddings Each endpoint is given an **Endpoint URL**, it can be found on the endpoint **Overview** page. We need to assign this endpoint URL to the `endpoint_url` variable. ![endpoints 4](https://mintlify.s3.us-west-1.amazonaws.com/pinecone/images/hf-endpoints-4.png) ```Python Python endpoint = "" ``` We will also need the organization API token, we find this via the organization settings on Hugging Face (`https://huggingface.co/organizations//settings/profile`). This is assigned to the `api_org` variable. ![endpoints 5](https://mintlify.s3.us-west-1.amazonaws.com/pinecone/images/hf-endpoints-5.png) ```Python Python api_org = "" ``` Now we're ready to create embeddings via Inference Endpoints. Let's start with a toy example. ```Python Python import requests # add the api org token to the headers headers = { 'Authorization': f'Bearer {api_org}' } # we add sentences to embed like so json_data = {"inputs": ["a happy dog", "a sad dog"]} # make the request res = requests.post( endpoint, headers=headers, json=json_data ) ``` We should see a `200` response. ```Python Python res ``` ``` ``` Inside the response we should find two embeddings... ```Python Python len(res.json()['embeddings']) ``` ``` 2 ``` We can also see the dimensionality of our embeddings like so: ```Python Python dim = len(res.json()['embeddings'][0]) dim ``` ``` 768 ``` We will need more than two items to search through, so let's download a larger dataset. For this we will use Hugging Face datasets. ```Python Python from datasets import load_dataset snli = load_dataset("snli", split='train') snli ``` ``` Downloading: 100%|██████████| 1.93k/1.93k [00:00<00:00, 992kB/s] Downloading: 100%|██████████| 1.26M/1.26M [00:00<00:00, 31.2MB/s] Downloading: 100%|██████████| 65.9M/65.9M [00:01<00:00, 57.9MB/s] Downloading: 100%|██████████| 1.26M/1.26M [00:00<00:00, 43.6MB/s] Dataset({ features: ['premise', 'hypothesis', 'label'], num_rows: 550152 }) ``` SNLI contains 550K sentence pairs, many of these include duplicate items so we will take just one set of these (the *hypothesis*) and deduplicate them. ```Python passages = list(set(snli['hypothesis'])) len(passages) ``` ``` 480042 ``` We will drop to 50K sentences so that the example is quick to run, if you have time, feel free to keep the full 480K. ```Python Python passages = passages[:50_000] ``` ## Create a Pinecone index With our endpoint and dataset ready, all that we're missing is a vector database. For this, we need to initialize our connection to Pinecone, this requires a [free API key](https://app.pinecone.io/). ```Python Python import pinecone # initialize connection to pinecone (get API key at app.pinecone.io) pinecone.init(api_key="YOUR_API_KEY", environment="YOUR_ENVIRONMENT") ``` Now we create a new index called `'hf-endpoints'`, the name isn't important *but* the `dimension` must align to our endpoint model output dimensionality (we found this in `dim` above) and the model metric (typically `cosine` is okay, but not for all models). ```Python Python index_name = 'hf-endpoints' # check if the hf-endpoints index exists if index_name not in pinecone.list_indexes(): # create the index if it does not exist pinecone.create_index( index_name, dimension=dim, metric="cosine" ) # connect to hf-endpoints index we created index = pinecone.Index(index_name) ``` ## Create and index embeddings Now we have all of our components ready; endpoints, dataset, and Pinecone. Let's go ahead and create our dataset embeddings and index them within Pinecone. ```Python Python from tqdm.auto import tqdm # we will use batches of 64 batch_size = 64 for i in tqdm(range(0, len(passages), batch_size)): # find end of batch i_end = min(i+batch_size, len(passages)) # extract batch batch = passages[i:i_end] # generate embeddings for batch via endpoints res = requests.post( endpoint, headers=headers, json={"inputs": batch} ) emb = res.json()['embeddings'] # get metadata (just the original text) meta = [{'text': text} for text in batch] # create IDs ids = [str(x) for x in range(i, i_end)] # add all to upsert list to_upsert = list(zip(ids, emb, meta)) # upsert/insert these records to pinecone _ = index.upsert(vectors=to_upsert) # check that we have all vectors in index index.describe_index_stats() ``` ``` 100%|██████████| 782/782 [11:02<00:00, 1.18it/s] {'dimension': 768, 'index_fullness': 0.1, 'namespaces': {'': {'vector_count': 50000}}, 'total_vector_count': 50000} ``` With everything indexed we can begin querying. We will take a few examples from the *premise* column of the dataset. ```Python Python query = snli['premise'][0] print(f"Query: {query}") # encode with HF endpoints res = requests.post(endpoint, headers=headers, json={"inputs": query}) xq = res.json()['embeddings'] # query and return top 5 xc = index.query(xq, top_k=5, include_metadata=True) # iterate through results and print text print("Answers:") for match in xc['matches']: print(match['metadata']['text']) ``` ``` Query: A person on a horse jumps over a broken down airplane. Answers: The horse jumps over a toy airplane. a lady rides a horse over a plane shaped obstacle A person getting onto a horse. person rides horse A woman riding a horse jumps over a bar. ``` These look good, let's try a couple more examples. ```Python Python query = snli['premise'][100] print(f"Query: {query}") # encode with HF endpoints res = requests.post(endpoint, headers=headers, json={"inputs": query}) xq = res.json()['embeddings'] # query and return top 5 xc = index.query(xq, top_k=5, include_metadata=True) # iterate through results and print text print("Answers:") for match in xc['matches']: print(match['metadata']['text']) ``` ``` Query: A woman is walking across the street eating a banana, while a man is following with his briefcase. Answers: A woman eats a banana and walks across a street, and there is a man trailing behind her. A woman eats a banana split. A woman is carrying two small watermelons and a purse while walking down the street. The woman walked across the street. A woman walking on the street with a monkey on her back. ``` And one more... ```Python Python query = snli['premise'][200] print(f"Query: {query}") # encode with HF endpoints res = requests.post(endpoint, headers=headers, json={"inputs": query}) xq = res.json()['embeddings'] # query and return top 5 xc = index.query(xq, top_k=5, include_metadata=True) # iterate through results and print text print("Answers:") for match in xc['matches']: print(match['metadata']['text']) ``` ``` Query: People on bicycles waiting at an intersection. Answers: A pair of people on bikes are waiting at a stoplight. Bike riders wait to cross the street. people on bicycles Group of bike riders stopped in the street. There are bicycles outside. ``` All of these results look excellent. If you are not planning on running your endpoint and vector DB beyond this tutorial, you can shut down both. ## Clean up Shut down the endpoint by navigating to the Inference Endpoints **Overview** page and selecting **Delete endpoint**. Delete the Pinecone index with: ```Python Python pinecone.delete_index(index_name) ``` Once the index is deleted, you cannot use it again. # Instill AI Source: https://docs.pinecone.io/integrations/instill export const PrimarySecondaryCTA = ({primaryLabel, primaryHref, primaryTarget, secondaryLabel, secondaryHref, secondaryTarget}) =>
{primaryLabel && primaryHref && } {secondaryLabel && secondaryHref && }
; Instill AI specializes in developing cutting-edge solutions for data, models, and pipeline orchestration. Their flagship source-available product, Instill Core, is a no-code/low-code platform designed to facilitate the development, deployment, and management of AI workflows. By simplifying the integration of various AI models and data sources, they enable businesses to harness the power of AI without requiring extensive technical expertise. Their solutions cater to a wide range of applications, from predictive analytics and autonomous AI agents to enterprise private knowledge bases, AI assistants, and beyond. The Pinecone integration with Instill allows developers to incorporate its API for vector upsert and query tasks into AI pipelines. Developers configure their Pinecone component within Instill Core by providing the necessary API key and base URL. They can then perform tasks such as querying vector similarities to retrieve the most relevant results, complete with metadata and similarity scores, or upserting new vector data to keep their datasets up-to-date. This integration enables the addition of knowledge to LLMs via Retrieval Augmented Generation (RAG), significantly enhancing the capabilities of autonomous agents, chatbots, question-answering systems, and multi-agent systems. # Jina AI Source: https://docs.pinecone.io/integrations/jina export const PrimarySecondaryCTA = ({primaryLabel, primaryHref, primaryTarget, secondaryLabel, secondaryHref, secondaryTarget}) =>
{primaryLabel && primaryHref && } {secondaryLabel && secondaryHref && }
; Jina Embeddings leverage powerful models to generate high-quality text embeddings that can process inputs up to 8,000 tokens. Jina Embeddings are designed to be highly versatile, catering to both domain-specific use cases, such as e-commerce, and language-specific needs, including Chinese and German. By providing robust models and the expertise to fine-tune them for specific requirements, Jina AI empowers developers to enhance their search functionalities, improve natural language understanding, and drive more insightful data analysis. By integrating Pinecone with Jina, you can add knowledge to LLMs via retrieval augmented generation (RAG), greatly enhancing LLM ability for autonomous agents, chatbots, question-answering, and multi-agent systems. # LangChain Source: https://docs.pinecone.io/integrations/langchain Using LangChain and Pinecone to add knowledge to LLMs export const PrimarySecondaryCTA = ({primaryLabel, primaryHref, primaryTarget, secondaryLabel, secondaryHref, secondaryTarget}) =>
{primaryLabel && primaryHref && } {secondaryLabel && secondaryHref && }
; LangChain provides modules for managing and optimizing the use of large language models (LLMs) in applications. Its core philosophy is to facilitate data-aware applications where the language model interacts with other data sources and its environment. This framework consists of several parts that simplify the entire application lifecycle: * Write your applications in LangChain/LangChain.js. Get started quickly by using Templates for reference. * Use LangSmith to inspect, test, and monitor your chains to constantly improve and deploy with confidence. * Turn any chain into an API with LangServe. By integrating Pinecone with LangChain, you can add knowledge to LLMs via retrieval augmented generation (RAG), greatly enhancing LLM ability for autonomous agents, chatbots, question-answering, and multi-agent systems. ## Setup guide This guide shows you how to integrate Pinecone, a high-performance vector database, with [LangChain](https://www.langchain.com/), a framework for building applications powered by large language models (LLMs). Pinecone enables developers to build scalable, real-time recommendation and search systems based on vector similarity search. LangChain, on the other hand, provides modules for managing and optimizing the use of language models in applications. Its core philosophy is to facilitate data-aware applications where the language model interacts with other data sources and its environment. By integrating Pinecone with LangChain, you can add knowledge to LLMs via [Retrieval Augmented Generation (RAG)](https://www.pinecone.io/learn/series/rag/), greatly enhancing LLM ability for autonomous agents, chatbots, question-answering, and multi-agent systems. This guide demonstrates only one way out of many that you can use LangChain and Pinecone together. For additional examples, see: * [LangChain AI Handbook](https://www.pinecone.io/learn/series/langchain/) * [Retrieval Augmentation for LLMs](https://github.com/pinecone-io/examples/blob/master/learn/generation/langchain/handbook/05-langchain-retrieval-augmentation.ipynb) * [Retrieval Augmented Conversational Agent](https://github.com/pinecone-io/examples/blob/master/learn/generation/langchain/handbook/08-langchain-retrieval-agent.ipynb) ## Key concepts The `PineconeVectorStore` class provided by LangChain can be used to interact with Pinecone indexes. It's important to remember that you must have an existing Pinecone index before you can create a `PineconeVectorStore` object. ### Initializing a vector store To initialize a `PineconeVectorStore` object, you must provide the name of the Pinecone index and an `Embeddings` object initialized through LangChain. There are two general approaches to initializing a `PineconeVectorStore` object: 1. Initialize without adding records: ```Python Python import os from langchain_pinecone import PineconeVectorStore from langchain_openai import OpenAIEmbeddings os.environ['OPENAI_API_KEY'] = '' os.environ['PINECONE_API_KEY'] = '' index_name = "" embeddings = OpenAIEmbeddings() vectorstore = PineconeVectorStore(index_name=index_name, embedding=embeddings) ``` You can also use the `from_existing_index` method of LangChain's `PineconeVectorStore` class to initialize a vector store. 2. Initialize while adding records: The `from_documents` and `from_texts` methods of LangChain's `PineconeVectorStore` class add records to a Pinecone index and return a `PineconeVectorStore` object. The `from_documents` method accepts a list of LangChain's `Document` class objects, which can be created using LangChain's `CharacterTextSplitter` class. The `from_texts` method accepts a list of strings. Similarly to above, you must provide the name of an existing Pinecone index and an `Embeddings` object. Both of these methods handle the embedding of the provided text data and the creation of records in your Pinecone index. ```Python Python import os from langchain_pinecone import PineconeVectorStore from langchain_openai import OpenAIEmbeddings from langchain_community.document_loaders import TextLoader from langchain_text_splitters import CharacterTextSplitter os.environ['OPENAI_API_KEY'] = '' os.environ['PINECONE_API_KEY'] = '' index_name = "" embeddings = OpenAIEmbeddings() # path to an example text file loader = TextLoader("../../modules/state_of_the_union.txt") documents = loader.load() text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0) docs = text_splitter.split_documents(documents) vectorstore_from_docs = PineconeVectorStore.from_documents( docs, index_name=index_name, embedding=embeddings ) texts = ["Tonight, I call on the Senate to: Pass the Freedom to Vote Act.", "ne of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court.", "One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence."] vectorstore_from_texts = PineconeVectorStore.from_texts( texts, index_name=index_name, embedding=embeddings ) ``` ### Add more records Once you have initialized a `PineconeVectorStore` object, you can add more records to the underlying Pinecone index (and thus also the linked LangChain object) using either the `add_documents` or `add_texts` methods. Like their counterparts that also initialize a `PineconeVectorStore` object, both of these methods also handle the embedding of the provided text data and the creation of records in your Pinecone index. ```Python Python # path to an example text file loader = TextLoader("../../modules/inaugural_address.txt") documents = loader.load() text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0) docs = text_splitter.split_documents(documents) vectorstore = PineconeVectorStore(index_name=index_name, embedding=embeddings) vectorstore.add_documents(docs) ``` ```Python Python vectorstore = PineconeVectorStore(index_name=index_name, embedding=embeddings) vectorstore.add_texts(["More text to embed and add to the index!"]) ``` ### Perform a similarity search A `similarity_search` on a `PineconeVectorStore` object returns a list of LangChain `Document` objects most similar to the query provided. While the `similarity_search` uses a Pinecone query to find the most similar results, this method includes additional steps and returns results of a different type. The `similarity_search` method accepts raw text and automatically embeds it using the `Embedding` object provided when you initialized the `PineconeVectorStore`. You can also provide a `k` value to determine the number of LangChain `Document` objects to return. The default value is `k=4`. ```Python Python query = "Who is Ketanji Brown Jackson?" vectorstore.similarity_search(query) # Response: # [ # Document(page_content='Ketanji Onyika Brown Jackson is an American lawyer and jurist who is an associate justice of the Supreme Court of the United...', metadata={'chunk': 0.0, 'source': 'https://en.wikipedia.org/wiki/Ketanji_Brown_Jackson', 'title': 'Ketanji Brown Jackson', 'wiki-id': '6573'}), # Document(page_content='Jackson was nominated to the Supreme Court by President Joe Biden on February 25, 2022, and confirmed by the U.S. Senate...', metadata={'chunk': 1.0, 'source': 'https://en.wikipedia.org/wiki/Ketanji_Brown_Jackson', 'title': 'Ketanji Brown Jackson', 'wiki-id': '6573'}), # Document(page_content='Jackson grew up in Miami and attended Miami Palmetto Senior High School. She distinguished herself as a champion debater...', metadata={'chunk': 3.0, 'source': 'https://en.wikipedia.org/wiki/Ketanji_Brown_Jackson', 'title': 'Ketanji Brown Jackson', 'wiki-id': '6573'}), # Document(page_content='After high school, Jackson matriculated at Harvard University to study government, having applied despite her guidance...', metadata={'chunk': 5.0, 'source': 'https://en.wikipedia.org/wiki/Ketanji_Brown_Jackson', 'title': 'Ketanji Brown Jackson', 'wiki-id': '6573'}) # ] ``` You can also optionally apply a metadata filter to your similarity search. The filtering query language is the same as for Pinecone queries, as detailed in [Filtering with metadata](https://docs.pinecone.io/guides/index-data/indexing-overview#metadata). ```Python Python query = "Tell me more about Ketanji Brown Jackson." vectorstore.similarity_search(query, filter={'source': 'https://en.wikipedia.org/wiki/Ketanji_Brown_Jackson'}) ``` ### Namespaces Several methods of the `PineconeVectorStore` class support using [namespaces](https://docs.pinecone.io/guides/indexes/use-namespaces). You can also initialize your `PineconeVectorStore` object with a namespace to restrict all further operations to that space. ```Python Python index_name = "" embeddings = OpenAIEmbeddings() vectorstore = PineconeVectorStore(index_name=index_name, embedding=embeddings, namespace="example-namespace") ``` If you initialize your `PineconeVectorStore` object without a namespace, you can specify the target namespace within the operation. ```Python Python # path to an example text file loader = TextLoader("../../modules/congressional_address.txt") documents = loader.load() text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0) docs = text_splitter.split_documents(documents) vectorstore_from_docs = PineconeVectorStore.from_documents( docs, index_name=index_name, embedding=embeddings, namespace="example-namespace" ) vectorstore_from_texts = PineconeVectorStore.from_texts( texts, index_name=index_name, embedding=embeddings, namespace="example-namespace" ) vectorstore_from_docs.add_documents(docs, namespace="example-namespace") vectorstore_from_texts.add_texts(["More text!"], namespace="example-namespace") ``` ```Python Python query = "Who is Ketanji Brown Jackson?" vectorstore.similarity_search(query, namespace="example-namespace") ``` ## Tutorial ### 1. Set up your environment Before you begin, install some necessary libraries and set environment variables for your Pinecone and OpenAI API keys: ```Shell pip install -qU \ "pinecone[grpc]"==5.1.0 \ pinecone-datasets==0.7.0 \ langchain-pinecone==0.1.2 \ langchain-openai==0.1.23 \ langchain==0.2.15 ``` ```Shell # Set environment variables for API keys export PINECONE_API_KEY= export OPENAI_API_KEY= ``` ```Python Python import os pinecone_api_key = os.environ.get('PINECONE_API_KEY') openai_api_key = os.environ.get('OPENAI_API_KEY') ``` ### 2. Build the knowledge base 1. Load a [sample Pinecone dataset](/guides/data/use-public-pinecone-datasets) into memory: ```Python Python import pinecone_datasets dataset = pinecone_datasets.load_dataset('wikipedia-simple-text-embedding-ada-002-100K') len(dataset) # Response: # 100000 ``` 2. Reduce the dataset and format it for upserting into Pinecone: ```Python Python # we will use rows of the dataset up to index 30_000 dataset.documents.drop(dataset.documents.index[30_000:], inplace=True) # we drop sparse_values as they are not needed for this example dataset.documents.drop(['metadata'], axis=1, inplace=True) dataset.documents.rename(columns={'blob': 'metadata'}, inplace=True) ``` ### 3. Index the data in Pinecone 1. Decide whether to use a [serverless](/guides/index-data/indexing-overview#serverless-indexes) or [pod-based](/guides/index-data/indexing-overview#pod-based-indexes) index. ```python Python use_serverless = True ``` 2. Initialize your client connection to Pinecone and create an index. This step uses the Pinecone API key you set as an environment variable [earlier](#1-set-up-your-environment). ```Python Python from pinecone.grpc import PineconeGRPC as Pinecone from pinecone import ServerlessSpec, PodSpec import time # configure client pc = Pinecone(api_key=pinecone_api_key) if use_serverless: spec = ServerlessSpec(cloud='aws', region='us-east-1') else: # if not using a starter index, you should specify a pod_type too spec = PodSpec() # check for and delete index if already exists index_name = 'langchain-retrieval-augmentation-fast' if pc.has_index(index_name): pc.delete_index(name=index_name) # create a new index pc.create_index( index_name, dimension=1536, # dimensionality of text-embedding-ada-002 metric='dotproduct', spec=spec ) ``` 3. Target the index and check its current stats: ```Python Python index = pc.Index(index_name) index.describe_index_stats() # Response: # {'dimension': 1536, # 'index_fullness': 0.0, # 'namespaces': {}, # 'total_vector_count': 0} ``` You'll see that the index has a `total_vector_count` of `0`, as you haven't added any vectors yet. 4. Now upsert the data to Pinecone: ```Python Python for batch in dataset.iter_documents(batch_size=100): index.upsert(batch) ``` 5. Once the data is indexed, check the index stats once again: ```Python Python index.describe_index_stats() # Response: # {'dimension': 1536, # 'index_fullness': 0.0, # 'namespaces': {}, # 'total_vector_count': 70000} ``` ### 4. Initialize a LangChain vector store Now that you've built your Pinecone index, you need to initialize a LangChain vector store using the index. This step uses the OpenAI API key you set as an environment variable [earlier](#1-set-up-your-environment). Note that OpenAI is a paid service and so running the remainder of this tutorial may incur some small cost. 1. Initialize a LangChain embedding object: ```Python Python from langchain_openai import OpenAIEmbeddings # get openai api key from platform.openai.com model_name = 'text-embedding-ada-002' embeddings = OpenAIEmbeddings( model=model_name, openai_api_key=openai_api_key ) ``` 2. Initialize the LangChain vector store: The `text_field` parameter sets the name of the metadata field that stores the raw text when you upsert records using a LangChain operation such as `vectorstore.from_documents` or `vectorstore.add_texts`. This metadata field is used as the `page_content` in the `Document` objects retrieved from query-like LangChain operations such as `vectorstore.similarity_search`. If you do not specify a value for `text_field`, it will default to `"text"`. ```Python Python from langchain_pinecone import PineconeVectorStore text_field = "text" vectorstore = PineconeVectorStore( index, embeddings, text_field ) ``` 3. Now you can query the vector store directly using `vectorstore.similarity_search`: ```Python Python query = "who was Benito Mussolini?" vectorstore.similarity_search( query, # our search query k=3 # return 3 most relevant docs ) # Response: # [Document(page_content='Benito Amilcare Andrea Mussolini KSMOM GCTE (29 July 1883 – 28 April 1945) was an Italian politician and journalist...', metadata={'chunk': 0.0, 'source': 'https://simple.wikipedia.org/wiki/Benito%20Mussolini', 'title': 'Benito Mussolini', 'wiki-id': '6754'}), # Document(page_content='Fascism as practiced by Mussolini\nMussolini\'s form of Fascism, "Italian Fascism"- unlike Nazism, the racist ideology...', metadata={'chunk': 1.0, 'source': 'https://simple.wikipedia.org/wiki/Benito%20Mussolini', 'title': 'Benito Mussolini', 'wiki-id': '6754'}), # Document(page_content='Veneto was made part of Italy in 1866 after a war with Austria. Italian soldiers won Latium in 1870. That was when...', metadata={'chunk': 5.0, 'source': 'https://simple.wikipedia.org/wiki/Italy', 'title': 'Italy', 'wiki-id': '363'})] ``` All of these sample results are good and relevant. But what else can you do with this? There are many tasks, one of the most interesting (and well supported by LangChain) is called "Generative Question-Answering" or GQA. ### 5. Use Pinecone and LangChain for RAG In RAG, you take the query as a question that is to be answered by a LLM, but the LLM must answer the question based on the information it is seeing from the vectorstore. 1. To do this, initialize a `RetrievalQA` object like so: ```Python Python from langchain_openai import ChatOpenAI from langchain.chains import RetrievalQA # completion llm llm = ChatOpenAI( openai_api_key=OPENAI_API_KEY, model_name='gpt-3.5-turbo', temperature=0.0 ) qa = RetrievalQA.from_chain_type( llm=llm, chain_type="stuff", retriever=vectorstore.as_retriever() ) qa.invoke(query) # Response: # Benito Mussolini was an Italian politician and journalist who served as the Prime Minister of Italy from 1922 until 1943. He was the leader of the National Fascist Party and played a significant role in the rise of fascism in Italy... ``` 2. You can also include the sources of information that the LLM is using to answer your question using a slightly different version of `RetrievalQA` called `RetrievalQAWithSourcesChain`: ```Python Python from langchain.chains import RetrievalQAWithSourcesChain qa_with_sources = RetrievalQAWithSourcesChain.from_chain_type( llm=llm, chain_type="stuff", retriever=vectorstore.as_retriever() ) qa_with_sources.invoke(query) # Response: # {'question': 'who was Benito Mussolini?', # 'answer': "Benito Mussolini was an Italian politician and journalist who served as the Prime Minister of Italy from 1922 until 1943. He was the leader of the National Fascist Party and played a significant role in the rise of fascism in Italy...", # 'sources': 'https://simple.wikipedia.org/wiki/Benito%20Mussolini'} ``` ### 6. Clean up When you no longer need the index, use the `delete_index` operation to delete it: ```Python Python pc.delete_index(name=index_name) ``` ## Related articles * [LangChain AI Handbook](https://www.pinecone.io/learn/series/langchain/) # Langtrace Source: https://docs.pinecone.io/integrations/langtrace export const PrimarySecondaryCTA = ({primaryLabel, primaryHref, primaryTarget, secondaryLabel, secondaryHref, secondaryTarget}) =>
{primaryLabel && primaryHref && } {secondaryLabel && secondaryHref && }
; Scale3 Labs recently launched Langtrace AI, an open-source monitoring and evaluation platform for LLM-powered applications. Langtrace is built based on Open Telemetry(OTEL) standards and supports native tracing for the most popular LLM vendors, VectorDBs, and frameworks(like Langchain and LlamaIndex). Langtrace AI supports tracing Pinecone natively, which means the Langtrace SDK can generate OTEL standard traces with automatic instrumentation in just 2 lines of code. These traces can be ingested by an observability tool that supports OTEL, such as Datadog, Grafana/Prometheus, SigNoz, Sentry, etc. Langtrace also has a visualization client that is optimized for visualizing the traces generated in an LLM stack. By having a Pinecone integration, Pinecone users can get access to rich and high cardinal tracing for the Pinecone API calls using Langtrace, which they can ingest into their observability tool of choice. This helps customers gain insights into the DB calls and help with debugging and troubleshooting applications in case of incidents. # LlamaIndex Source: https://docs.pinecone.io/integrations/llamaindex Using LlamaIndex and Pinecone to build semantic search and RAG applications export const PrimarySecondaryCTA = ({primaryLabel, primaryHref, primaryTarget, secondaryLabel, secondaryHref, secondaryTarget}) =>
{primaryLabel && primaryHref && } {secondaryLabel && secondaryHref && }
; LlamaIndex is a framework for connecting data sources to LLMs, with its chief use case being the end-to-end development of retrieval augmented generation (RAG) applications. LlamaIndex provides the essential abstractions to more easily ingest, structure, and access private or domain-specific data in order to inject these safely and reliably into LLMs for more accurate text generation. It’s available in Python and Typescript. Seamlessly integrate Pinecone vector database with LlamaIndex to build semantic search and RAG applications. ## Setup guide [View source](https://github.com/pinecone-io/examples/blob/master/learn/generation/llama-index/using-llamaindex-with-pinecone.ipynb) [Open in Colab](https://colab.research.google.com/github/pinecone-io/examples/blob/master/learn/generation/llama-index/using-llamaindex-with-pinecone.ipynb) [LlamaIndex](https://www.llamaindex.ai/) is a framework for connecting data sources to LLMs, with its chief use case being the end-to-end development of [RAG applications](https://www.pinecone.io/learn/retrieval-augmented-generation/). Compared to other similar frameworks, LlamaIndex offers a wide variety of tools for pre- and post-processing your data. This guide shows you how to use LlamaIndex and Pinecone to both perform traditional semantic search and build a RAG pipeline. Specifically, you will: * Load, transform, and vectorize sample data with LlamaIndex * Index and store the vectorized data in Pinecone * Search the data in Pinecone and use the results to augment an LLM call * Evaluate the answer you get back from the LLM This guide demonstrates only one way out of many that you can use LlamaIndex as part of a RAG pipeline. See LlamaIndex's section on [Advanced RAG](https://docs.llamaindex.ai/en/stable/optimizing/advanced%5Fretrieval/advanced%5Fretrieval.html) to learn more about what's possible. ### Set up your environment Before you begin, install some necessary libraries and set environment variables for your Pinecone and OpenAI API keys: ```Shell Shell # Install libraries pip install -qU \ "pinecone[grpc]"==5.1.0 \ llama-index==0.11.4 \ llama-index-vector-stores-pinecone==0.2.1 \ llama-index-readers-file==0.2.0 \ arxiv==2.1.3 \ setuptools # (Optional) ``` ```Shell Shell # Set environment variables for API keys export PINECONE_API_KEY= export OPENAI_API_KEY= pinecone_api_key = os.environ.get('PINECONE_API_KEY') openai_api_key = os.environ.get('OPENAI_API_KEY') ``` Also note that all code on this page is run on Python 3.11. ### Load the data In this guide, you will use the [canonical HNSW paper](https://arxiv.org/pdf/1603.09320.pdf) by Yuri Malkov (PDF) as your sample dataset. Your first step is to download the PDF from arXiv.org and load it into a LlamaIndex loader called [PDF Loader](https://llamahub.ai/l/file-pdf?from=all). This Loader is available (along with many more) on the [LlamaHub](https://llamahub.ai/), which is a directory of data loaders. ```Python Python import arxiv from pathlib import Path from llama_index.readers.file import PDFReader # Download paper to local file system (LFS) # `id_list` contains 1 item that matches our PDF's arXiv ID paper = next(arxiv.Client().results(arxiv.Search(id_list=["1603.09320"]))) paper.download_pdf(filename="hnsw.pdf") # Instantiate `PDFReader` from LlamaHub loader = PDFReader() # Load HNSW PDF from LFS documents = loader.load_data(file=Path('./hnsw.pdf')) # Preview one of our documents documents[0] # Response: # Document(id_='e25106d2-bde5-41f0-83fa-5cbfa8234bef', embedding=None, metadata={'page_label': '1', 'file_name': 'hnsw.pdf'}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={}, text="IEEE TRANSACTIONS ON JOURNAL NAME, MANUS CRIPT ID 1 \n Efficient and robust approximate nearest \nneighbor search using Hierarchical Navigable \nSmall World graphs \nYu. A. Malkov, D. A. Yashunin \nAbstract — We present a new approach for the approximate K -nearest neighbor search based on navigable small world \ngraphs with controllable hierarchy (Hierarchical NSW , HNSW ) and tree alg o-\nrithms", start_char_idx=None, end_char_idx=None, text_template='{metadata_str}\n\n{content}', metadata_template='{key}: {value}', metadata_seperator='\n') ``` You can see above that each `Document` has a ton of useful information, but depending on which Loader you choose, you may have to clean your data. In this case, you need to remove things like remaining `\n` characters and broken, hyphenated words (e.g., `alg o-\nrithms` → `algorithms`). ```Python Python # Clean up our Documents' content import re def clean_up_text(content: str) -> str: """ Remove unwanted characters and patterns in text input. :param content: Text input. :return: Cleaned version of original text input. """ # Fix hyphenated words broken by newline content = re.sub(r'(\w+)-\n(\w+)', r'\1\2', content) # Remove specific unwanted patterns and characters unwanted_patterns = [ "\\n", " —", "——————————", "—————————", "—————", r'\\u[\dA-Fa-f]{4}', r'\uf075', r'\uf0b7' ] for pattern in unwanted_patterns: content = re.sub(pattern, "", content) # Fix improperly spaced hyphenated words and normalize whitespace content = re.sub(r'(\w)\s*-\s*(\w)', r'\1-\2', content) content = re.sub(r'\s+', ' ', content) return content # Call function cleaned_docs = [] for d in documents: cleaned_text = clean_up_text(d.text) d.text = cleaned_text cleaned_docs.append(d) # Inspect output cleaned_docs[0].get_content() # Response: # "IEEE TRANSACTIONS ON JOURNAL NAME, MANUS CRIPT ID 1 Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs Yu. A. Malkov, D. A. Yashunin Abstract We present a new approach for the approximate K-nearest neighbor search based on navigable small world graphs with controllable hierarchy (Hierarchical NSW , HNSW ) and tree algorithms." # Great! ``` The value-add of using a file loader from LlamaHub is that your PDF is already broken down into LlamaIndex [Documents](https://docs.llamaindex.ai/en/stable/module%5Fguides/loading/documents%5Fand%5Fnodes/root.html#documents-nodes). Along with each Document object comes a [customizable](https://docs.llamaindex.ai/en/stable/module%5Fguides/loading/documents%5Fand%5Fnodes/usage%5Fdocuments.html#metadata) metadata dictionary and a hash ID, among other useful artifacts. ### Transform the data #### Metadata Now, if you look at one of your cleaned Document objects, you'll see that the default values in your metadata dictionary are not particularly useful. ```Python Python cleaned_docs[0].metadata # Response: # {'page_label': '1', 'file_name': 'hnsw.pdf'} ``` To add some metadata that would be more helpful, let's add author name and the paper's title. Note that whatever metadata you add to the metadata dictionary will apply to all [Nodes](https://docs.llamaindex.ai/en/stable/module%5Fguides/loading/documents%5Fand%5Fnodes/root.html#nodes), so you want to keep your additions high-level. LlamaIndex also provides [advanced customizations](https://docs.llamaindex.ai/en/stable/module%5Fguides/loading/documents%5Fand%5Fnodes/usage%5Fdocuments.html#advanced-metadata-customization) for what metadata the LLM can see vs the embedding, etc. ```Python Python # Iterate through `documents` and add our new key:value pairs metadata_additions = {"authors": ["Yu. A. Malkov", "D. A. Yashunin"], "title": "Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs"} # Update dict in place [cd.metadata.update(metadata_additions) for cd in cleaned_docs] # Let\'s confirm everything worked: cleaned_docs[0].metadata # Response: # {'page_label': '1', # 'file_name': 'hnsw.pdf', # 'authors': ['Yu. A. Malkov', 'D. A. Yashunin'], # 'title': 'Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs'} # Great! ``` #### Ingestion pipeline The easiest way to turn your data into indexable vectors and put those into Pinecone is to make what's called an [Ingestion Pipeline](https://docs.llamaindex.ai/en/stable/module%5Fguides/loading/ingestion%5Fpipeline/root.html). Ingestion Pipelines are how you will build a pipeline that will take your list of Documents, parse them into Nodes (or “[chunks](https://www.pinecone.io/learn/chunking-strategies/)” in non-LlamaIndex contexts), vectorize each Node's content, and upsert them into Pinecone. In the following pipeline, you'll use one of LlamaIndex's newer parsers: the [SemanticSplitterNodeParser](https://docs.llamaindex.ai/en/stable/module%5Fguides/loading/node%5Fparsers/modules.html#semanticsplitternodeparser), which uses OpenAI's [ada-002 embedding model](https://github.com/run-llama/llama%5Findex/blob/47b34d1fdfde2ded134a373b620c3e7a694e8380/llama%5Findex/embeddings/openai.py#L216) to split Documents into semantically coherent Nodes. This step uses the OpenAI API key you set as an environment variable [earlier](#set-up-your-environment). ```Python Python import os from llama_index.node_parser import SemanticSplitterNodeParser from llama_index.embeddings import OpenAIEmbedding from llama_index.ingestion import IngestionPipeline # This will be the model we use both for Node parsing and for vectorization embed_model = OpenAIEmbedding(api_key=openai_api_key) # Define the initial pipeline pipeline = IngestionPipeline( transformations=[ SemanticSplitterNodeParser( buffer_size=1, breakpoint_percentile_threshold=95, embed_model=embed_model, ), embed_model, ], ) ``` Hold off on running this pipeline; you will modify it below. ### Upsert the data Above, you defined an Ingestion Pipeline. There's one thing missing, though: a vector database into which you can upsert your transformed data. LlamaIndex lets you declare a [VectorStore](https://docs.llamaindex.ai/en/stable/examples/vector%5Fstores/pinecone%5Fmetadata%5Ffilter.html) and add that right into the pipeline for super easy ingestion. Let's do that with Pinecone below. This step uses the Pinecone API key you set as an environment variable [earlier](#set-up-your-environment). ```Python Python from pinecone.grpc import PineconeGRPC from pinecone import ServerlessSpec from llama_index.vector_stores.pinecone import PineconeVectorStore # Initialize connection to Pinecone pc = PineconeGRPC(api_key=pinecone_api_key) index_name = "llama-integration-example" # Create your index (can skip this step if your index already exists) pc.create_index( index_name, dimension=1536, spec=ServerlessSpec(cloud="aws", region="us-east-1"), ) # Initialize your index pinecone_index = pc.Index(index_name) # Initialize VectorStore vector_store = PineconeVectorStore(pinecone_index=pinecone_index) ``` With your PineconeVectorStore now initialized, you can pop that into your `pipeline` and run it. ```Python Python # Our pipeline with the addition of our PineconeVectorStore pipeline = IngestionPipeline( transformations=[ SemanticSplitterNodeParser( buffer_size=1, breakpoint_percentile_threshold=95, embed_model=embed_model, ), embed_model, ], vector_store=vector_store # Our new addition ) # Now we run our pipeline! pipeline.run(documents=cleaned_docs) ``` Now ensure your index is up and running with some Pinecone-native methods like `.describe_index_stats()`: ```Python Python pinecone_index.describe_index_stats() # Response: # {'dimension': 1536, # 'index_fullness': 0.0, # 'namespaces': {'': {'vector_count': 46}}, # 'total_vector_count': 46} ``` Awesome, your index now has vectors in it. Since you have 46 vectors, you can infer that your `SemanticSplitterNodeParser` split your list of Documents into 46 Nodes. #### Query the data To fetch search results from Pinecone itself, you need to make a [VectorStoreIndex](https://docs.llamaindex.ai/en/stable/module%5Fguides/indexing/vector%5Fstore%5Findex.html) object and a [VectorIndexRetriever](https://github.com/run-llama/llama%5Findex/blob/main/llama%5Findex/indices/vector%5Fstore/retrievers/retriever.py#L21) object. You can then pass natural language queries to your Pinecone index and receive results. ```Python Python from llama_index.core import VectorStoreIndex from llama_index.core.retrievers import VectorIndexRetriever # Instantiate VectorStoreIndex object from your vector_store object vector_index = VectorStoreIndex.from_vector_store(vector_store=vector_store) # Grab 5 search results retriever = VectorIndexRetriever(index=vector_index, similarity_top_k=5) # Query vector DB answer = retriever.retrieve('How does logarithmic complexity affect graph construction?') # Inspect results print([i.get_content() for i in answer]) # Response: # ['some relevant search result 1', 'some relevant search result 1'...] ``` These search results can now be plugged into any downstream task you want. One of the most common ways to use vector database search results is as additional context to augment a query sent to an LLM. This workflow is what's commonly referred to as a [RAG application](https://www.pinecone.io/learn/retrieval-augmented-generation/). ### Build a RAG app with the data Building a RAG app with LlamaIndex is very simple. In theory, you could create a simple [Query Engine](https://docs.llamaindex.ai/en/stable/module%5Fguides/deploying/query%5Fengine/usage%5Fpattern.html#usage-pattern) out of your `vector_index` object by calling `vector_index.as_query_engine().query(‘some query')`, but then you wouldn't be able to specify the number of Pinecone search results you'd like to use as context. To control how many search results your RAG app uses from your Pinecone index, you will instead create your Query Engine using the [RetrieverQueryEngine](https://github.com/run-llama/llama%5Findex/blob/main/llama%5Findex/query%5Fengine/retriever%5Fquery%5Fengine.py#L21) class. This class allows you to pass in the `retriever` created above, which you configured to retrieve the top 5 search results. ```Python Python from llama_index.core.query_engine import RetrieverQueryEngine # Pass in your retriever from above, which is configured to return the top 5 results query_engine = RetrieverQueryEngine(retriever=retriever) # Now you query: llm_query = query_engine.query('How does logarithmic complexity affect graph construction?') llm_query.response # Response: # 'Logarithmic complexity in graph construction affects the construction process by organizing the graph into different layers based on their length scale. This separation of links into layers allows for efficient and scalable routing in the graph. The construction algorithm starts from the top layer, which contains the longest links, and greedily traverses through the elements until a local minimum is reached. Then, the search switches to the lower layer with shorter links, and the process repeats. By keeping the maximum number of connections per element constant in all layers, the routing complexity in the graph scales logarithmically. This logarithmic complexity is achieved by assigning an integer level to each element, determining the maximum layer it belongs to. The construction algorithm incrementally builds a proximity graph for each layer, consisting of "short" links that approximate the Delaunay graph. Overall, logarithmic complexity in graph construction enables efficient and robust approximate nearest neighbor search.' ``` You can even inspect the context (Nodes) that informed your LLM's answer using the `.source_nodes` attribute. Let's inspect the first Node: ```Python Python llm_response_source_nodes = [i.get_content() for i in llm_query.source_nodes] llm_response_source_nodes # Response: # ["AUTHOR ET AL.: TITL E 7 be auto-configured by using sample data. The construction process can be easily and efficiently parallelized with only few synchronization points (as demonstrated in Fig. 9) and no measurable effect on index quality. Construction speed/index q uality tradeoff is co ntrolled via the efConstruction parameter. The tradeoff between the search time and the index construction time is presented in Fig. 10 for a 10M SIFT dataset and shows that a reasonable quality index can be constructed for efConstruct ion=100 on a 4X 2.4 GHz 10-core X..."] ``` ### Evaluate the data Now that you've made a RAG app and queried your LLM, you need to evaluate its response. With LlamaIndex, there are [many ways](https://docs.llamaindex.ai/en/module%5Fguides/evaluating/usage%5Fpattern.html#) to evaluate the results your RAG app generates. A great way to get started with evaluation is to confirm (or deny) that your LLM's responses are relevant, given the context retrieved from your vector database. To do this, you can use LlamaIndex's [RelevancyEvaluator](https://docs.llamaindex.ai/en/stable/examples/evaluation/relevancy%5Feval.html#relevancy-evaluator) class. The great thing about this type of evaluation is that there is no need for [ground truth data](https://dtunkelang.medium.com/evaluating-search-using-human-judgement-fbb2eeba37d9) (i.e., labeled datasets to compare answers with). ```Python Python from llama_index.core.evaluation import RelevancyEvaluator # (Need to avoid peripheral asyncio issues) import nest_asyncio nest_asyncio.apply() # Define evaluator evaluator = RelevancyEvaluator() # Issue query llm_response = query_engine.query( "How does logarithmic complexity affect graph construction?" ) # Grab context used in answer query & make it pretty llm_response_source_nodes = [i.get_content() for i in llm_response.source_nodes] # # Take your previous question and pass in the response youwe got above eval_result = evaluator.evaluate_response(query="How does logarithmic complexity affect graph construction?", response=llm_response) # Print response print(f'\nGiven the {len(llm_response_source_nodes)} chunks of content (below), is your \ LLM\'s response relevant? {eval_result.passing}\n \ \n ----Contexts----- \n \ \n{llm_response_source_nodes}') # Response: # "Given the 5 chunks of content (below), is your LLM's response relevant? True # ----Contexts----- # ['AUTHOR ET AL.: TITL E 7 be auto-configured by using sample data. The construction process can be easily and efficiently parallelized with only few synchronization points (as demonstrated in Fig...']" ``` You can see that there are various attributes you can inspect on your evaluator's result in order to ascertain what's going on behind the scenes. To get a quick binary True/False signal as to whether your LLM is producing relevant results given your context, inspect the `.passing` attribute. Let's see what happens when we send a totally out of scope query through your RAG app. Issue a random query you know your RAG app won't be able to answer, given what's in your index: ```Python Python query = "Why did the chicken cross the road?" response = query_engine.query(query) print(response.response) # Response: # "I'm sorry, but I cannot answer that question based on the given context information." # Evaluate eval_result = evaluator.evaluate_response(query=query, response=response) print(str(eval_result.passing)) # Response: # False # Our LLM is not taking our context into account, as expected :) ``` As expected, when you send an out-of-scope question through your RAG pipeline, your evaluator says the LLM's answer is not relevant to the retrieved context. ### Summary As you have seen, LlamaIndex is a powerful framework to use when building semantic search and RAG applications – and we have only gotten to the tip of the iceberg! [Explore more](https://docs.llamaindex.ai/en/index.html) on your own and [let us know how it goes](https://community.pinecone.io/). # Matillion Source: https://docs.pinecone.io/integrations/matillion export const PrimarySecondaryCTA = ({primaryLabel, primaryHref, primaryTarget, secondaryLabel, secondaryHref, secondaryTarget}) =>
{primaryLabel && primaryHref && } {secondaryLabel && secondaryHref && }
; [Matillion Data Productivity Cloud](https://www.matillion.com/) is a unified platform that helps your team move faster with one central place to build and manage graphical, low-code data pipelines. It allows data teams to use structured, semi-structured, and unstructured data in analytics; build AI pipelines for new use cases; and be more productive. Matillion Data Productivity Cloud and Pinecone can be used together for retrieval augmented generation (RAG) use cases, helping to contextualize business insights without code. Matillion supports 150+ pre-built data source connectors, as well as the ability to build custom connectors to any REST API source system, making it easy to chunk unstructured datasets, create embeddings, and upsert to Pinecone. Matillion's graphical AI Prompt Components integrate with large language models (LLM) running in OpenAI, Amazon Bedrock, Azure OpenAI, and Snowpark Container Services. They enable no-code lookup of external knowledge stored in Pinecone, enabling data engineers to enrich GenAI answers with contextualized and proprietary data. ## Additional resources * Video: [Use RAG with a Pinecone Vector database on the Data Productivity Cloud](https://www.youtube.com/watch?v=BsH7WlJdoFs) * Video: [How to upsert to your Pinecone Vector database](https://www.youtube.com/watch?v=l9qt-EzLkgY) * [Unlock the power of AI in Data Engineering](https://www.matillion.com/blog/matillion-new-ai-capabilities-for-data-engineering) # New Relic Source: https://docs.pinecone.io/integrations/new-relic export const PrimarySecondaryCTA = ({primaryLabel, primaryHref, primaryTarget, secondaryLabel, secondaryHref, secondaryTarget}) =>
{primaryLabel && primaryHref && } {secondaryLabel && secondaryHref && }
; New Relic is an all-in-one observability platform and provides the industry’s first APM solution for AI-powered applications. New Relic is pioneering AI observability with AIM to provide engineers unprecedented visibility and insights across the AI application stack, making it easier to troubleshoot and optimize their AI applications for performance, quality, cost, and responsible use of AI. Implement monitoring and integrate your Pinecone application with New Relic for performance analysis and insights. The New Relic for Pinecone (Prometheus) quickstart contains one dashboard. These interactive visualizations let you easily explore your data, understand context, and resolve problems faster. It also includes three alerts to detect changes in key performance metrics. Integrate these alerts with your favorite tools (like Slack, PagerDuty, etc.) and New Relic will let you know when something needs your attention. # Nexla Source: https://docs.pinecone.io/integrations/nexla export const PrimarySecondaryCTA = ({primaryLabel, primaryHref, primaryTarget, secondaryLabel, secondaryHref, secondaryTarget}) =>
{primaryLabel && primaryHref && } {secondaryLabel && secondaryHref && }
; Nexla is a Data + AI Integration Platform that makes it easy for users to build data pipelines in a no-code/low-code manner. The Pinecone integration with Nexla makes it easy for enterprise users to ingest data from systems like Sharepoint, OneDrive, Cloud Storage, Data Warehouses, and 500+ other connectors that Nexla supports natively. # Nuclia Source: https://docs.pinecone.io/integrations/nuclia export const PrimarySecondaryCTA = ({primaryLabel, primaryHref, primaryTarget, secondaryLabel, secondaryHref, secondaryTarget}) =>
{primaryLabel && primaryHref && } {secondaryLabel && secondaryHref && }
; [Nuclia](https://nuclia.com/) RAG-as-a-Service automatically indexes files and documents from both internal and external sources, powering diverse company use cases with large language models (LLMs). This comprehensive indexing capability ensures that organizations can leverage unstructured data effectively, transforming it into actionable insights. With Nuclia's modular Retrieval-Augmented Generation (RAG) system, you can deploy solutions tailored to various operational needs across different deployment options, enhancing flexibility and efficiency. The modular RAG system from Nuclia is designed to fit specific use cases, allowing you to customize your RAG pipeline to meet your unique requirements. Whether it's defining your own retrieval and chunking strategies or choosing from various embedding models, Nuclia's RAG-as-a-Service makes it easy to bring your tailored solutions into production. This customization not only improves the value of your products but also helps you stay competitive by automating tasks and making your data smarter with LLMs, saving hundreds of hours in the process. When you create a knowledge box at Nuclia, choose to store the index in Pinecone. This is especially useful for large datasets where full text search is not key on the retrieval phase. # OctoAI Source: https://docs.pinecone.io/integrations/octoai export const PrimarySecondaryCTA = ({primaryLabel, primaryHref, primaryTarget, secondaryLabel, secondaryHref, secondaryTarget}) =>
{primaryLabel && primaryHref && } {secondaryLabel && secondaryHref && }
; Harness value from the latest AI innovations by delievering efficient, reliable, and customizable AI systems for your apps. Run your models or checkpoints on OctoAI's cost-effective API endpoints, or run OctoAI's optimized GenAI stack in your environment. Choose from the best models that OctoAI has to offer, including GTE Large embedding model, the best foundational open source LLMs such as Mixtral-8x7B from Mistral AI, Llama2 from Meta, and highly capable model fine tunes like Nous Hermes 2 Pro Mistral from Nous Research. As a fully open source solution, Pinecone Canopy and OctoAI is one of the fastest ways and more affordable ways to get started on your RAG journey. Canopy uses Pinecone vector database for storage and retrieval, which is free to use for up to 100k vectors (that's about 30k pages of text). # OpenAI Source: https://docs.pinecone.io/integrations/openai Using OpenAI and Pinecone to combine deep learning capabilities for embedding generation with efficient vector storage and retrieval export const PrimarySecondaryCTA = ({primaryLabel, primaryHref, primaryTarget, secondaryLabel, secondaryHref, secondaryTarget}) =>
{primaryLabel && primaryHref && } {secondaryLabel && secondaryHref && }
; OpenAI's large language models (LLMs) enhance semantic search or “long-term memory” for LLMs. This combo utilizes LLMs' embedding and completion (or generation) endpoints alongside Pinecone's vector search capabilities for nuanced information retrieval. By integrating OpenAI's LLMs with Pinecone, you can combine deep learning capabilities for embedding generation with efficient vector storage and retrieval. This approach surpasses traditional keyword-based search, offering contextually-aware, precise results. ## Setup guide [View source](https://github.com/pinecone-io/examples/blob/master/integrations/openai/) [Open in Colab](https://colab.research.google.com/github/pinecone-io/examples/blob/master/integrations/openai/semantic_search_openai.ipynb) This guide covers the integration of OpenAI's Large Language Models (LLMs) with Pinecone (referred to as the **OP stack**), enhancing semantic search or 'long-term memory' for LLMs. This combo utilizes LLMs' embedding and completion (or generation) endpoints alongside Pinecone's vector search capabilities for nuanced information retrieval. LLMs like OpenAI's `text-embedding-ada-002` generate vector embeddings, i.e., numerical representations of text semantics. These embeddings facilitate semantic-based rather than literal textual matches. Additionally, LLMs like `gpt-4` or `gpt-3.5-turbo` can predict text completions based on information provided from these contexts. Pinecone is a vector database designed for storing and querying high-dimensional vectors. It provides fast, efficient semantic search over these vector embeddings. By integrating OpenAI's LLMs with Pinecone, we combine deep learning capabilities for embedding generation with efficient vector storage and retrieval. This approach surpasses traditional keyword-based search, offering contextually-aware, precise results. There are many ways of integrating these two tools and we have several guides focusing on specific use-cases. If you already know what you'd like to do you can jump to these specific materials: * [ChatGPT Plugins Walkthrough](https://youtu.be/hpePPqKxNq8) * [Ask Lex ChatGPT Plugin](https://github.com/pinecone-io/examples/tree/master/learn/generation/openai/chatgpt/plugins/ask-lex) * [Generative Question-Answering](https://github.com/pinecone-io/examples/blob/master/docs/gen-qa-openai.ipynb) * [Retrieval Augmentation using LangChain](https://github.com/pinecone-io/examples/blob/master/learn/generation/langchain/handbook/05-langchain-retrieval-augmentation.ipynb) ### Introduction to Embeddings At the core of the OP stack we have embeddings which are supported via the [OpenAI Embedding API](https://beta.openai.com/docs/guides/embeddings). We index those embeddings in the [Pinecone vector database](https://www.pinecone.io) for fast and scalable retrieval augmentation of our LLMs or other information retrieval use-cases. *This example demonstrates the core OP stack. It is the simplest workflow and is present in each of the other workflows, but is not the only way to use the stack. Please see the links above for more advanced usage.* The OP stack is built for semantic search, question-answering, threat-detection, and other applications that rely on language models and a large corpus of text data. The basic workflow looks like this: * Embed and index * Use the OpenAI Embedding API to generate vector embeddings of your documents (or any text data). * Upload those vector embeddings into Pinecone, which can store and index millions/billions of these vector embeddings, and search through them at ultra-low latencies. * Search * Pass your query text or document through the OpenAI Embedding API again. * Take the resulting vector embedding and send it as a [query](/guides/search/search-overview) to Pinecone. * Get back semantically similar documents, even if they don't share any keywords with the query. ![Basic workflow of OpenAI and Pinecone](https://mintlify.s3.us-west-1.amazonaws.com/pinecone/images/6a3ea5a-pinecone-openai-overview.png) Let's get started... ### Environment Setup We start by installing the OpenAI and Pinecone clients, we will also need HuggingFace *Datasets* for downloading the TREC dataset that we will use in this guide. ```Bash Bash !pip install -qU \ pinecone-client[grpc]==3.0.2 \ openai==1.10.0 \ datasets==2.16.1 ``` #### Creating Embeddings To create embeddings we must first initialize our connection to OpenAI Embeddings, we sign up for an API key at [OpenAI](https://beta.openai.com/signup). ```Python Python from openai import OpenAI client = OpenAI( api_key="OPENAI_API_KEY" ) # get API key from platform.openai.com ``` We can now create embeddings with the OpenAI v3 small embedding model like so: ```Python Python MODEL = "text-embedding-3-small" res = client.embeddings.create( input=[ "Sample document text goes here", "there will be several phrases in each batch" ], model=MODEL ) ``` In `res` we should find a JSON-like object containing two 1536-dimensional embeddings, these are the vector representations of the two inputs provided above. To access the embeddings directly we can write: ```Python Python # we can extract embeddings to a list embeds = [record.embedding for record in res.data] len(embeds) ``` We will use this logic when creating our embeddings for the **T**ext **RE**trieval **C**onference (TREC) question classification dataset later. #### Initializing a Pinecone Index Next, we initialize an index to store the vector embeddings. For this we need a Pinecone API key, [sign up for one here](https://app.pinecone.io). ```Python Python import time from pinecone.grpc import PineconeGRPC as Pinecone from pinecone import ServerlessSpec pc = Pinecone(api_key="...") spec = ServerlessSpec(cloud="aws", region="us-east-1") index_name = 'semantic-search-openai' # check if index already exists (it shouldn't if this is your first run) if index_name not in pc.list_indexes().names(): # if does not exist, create index pc.create_index( index_name, dimension=len(embeds[0]), # dimensionality of text-embed-3-small metric='dotproduct', spec=spec ) # connect to index index = pc.Index(index_name) time.sleep(1) # view index stats index.describe_index_stats() ``` #### Populating the Index With both OpenAI and Pinecone connections initialized, we can move onto populating the index. For this, we need the TREC dataset. ```Python Python from datasets import load_dataset # load the first 1K rows of the TREC dataset trec = load_dataset('trec', split='train[:1000]') ``` Then we create a vector embedding for each question using OpenAI (as demonstrated earlier), and `upsert` the ID, vector embedding, and original text for each phrase to Pinecone. High-cardinality metadata values (like the unique text values we use here)\ can reduce the number of vectors that fit on a single pod. See\ [Known limitations](/reference/api/known-limitations) for more. ```Python Python from tqdm.auto import tqdm count = 0 # we'll use the count to create unique IDs batch_size = 32 # process everything in batches of 32 for i in tqdm(range(0, len(trec['text']), batch_size)): # set end position of batch i_end = min(i+batch_size, len(trec['text'])) # get batch of lines and IDs lines_batch = trec['text'][i: i+batch_size] ids_batch = [str(n) for n in range(i, i_end)] # create embeddings res = client.embeddings.create(input=lines_batch, model=MODEL) embeds = [record.embedding for record in res.data] # prep metadata and upsert batch meta = [{'text': line} for line in lines_batch] to_upsert = zip(ids_batch, embeds, meta) # upsert to Pinecone index.upsert(vectors=list(to_upsert)) ``` #### Querying With our data indexed, we're now ready to move onto performing searches. This follows a similar process to indexing. We start with a text `query`, that we would like to use to find similar sentences. As before we encode this with OpenAI's text similarity Babbage model to create a *query vector* `xq`. We then use `xq` to query the Pinecone index. ```Python Python query = "What caused the 1929 Great Depression?" xq = client.embeddings.create(input=query, model=MODEL).data[0].embedding ``` Now we query. ```Python Python res = index.query([xq], top_k=5, include_metadata=True) ``` The response from Pinecone includes our original text in the `metadata` field, let's print out the `top_k` most similar questions and their respective similarity scores. ```Python Python for match in res['matches']: print(f"{match['score']:.2f}: {match['metadata']['text']}") ``` ```[Out]: 0.75: Why did the world enter a global depression in 1929 ? 0.60: When was `` the Great Depression '' ? 0.37: What crop failure caused the Irish Famine ? 0.32: What were popular songs and types of songs in the 1920s ? 0.32: When did World War I start ? ``` Looks good, let's make it harder and replace *"depression"* with the incorrect term *"recession"*. ```Python Python query = "What was the cause of the major recession in the early 20th century?" # create the query embedding xq = client.embeddings.create(input=query, model=MODEL).data[0].embedding # query, returning the top 5 most similar results res = index.query(vector=[xq], top_k=5, include_metadata=True) for match in res['matches']: print(f"{match['score']:.2f}: {match['metadata']['text']}") ``` ```[Out]: 0.63: Why did the world enter a global depression in 1929 ? 0.55: When was `` the Great Depression '' ? 0.34: What were popular songs and types of songs in the 1920s ? 0.33: What crop failure caused the Irish Famine ? 0.29: What is considered the costliest disaster the insurance industry has ever faced ? ``` Let's perform one final search using the definition of depression rather than the word or related words. ```Python Python query = "Why was there a long-term economic downturn in the early 20th century?" # create the query embedding xq = client.embeddings.create(input=query, model=MODEL).data[0].embedding # query, returning the top 5 most similar results res = index.query(vector=[xq], top_k=5, include_metadata=True) for match in res['matches']: print(f"{match['score']:.2f}: {match['metadata']['text']}") ``` ```[Out]: 0.62: Why did the world enter a global depression in 1929 ? 0.54: When was `` the Great Depression '' ? 0.34: What were popular songs and types of songs in the 1920s ? 0.33: What crop failure caused the Irish Famine ? 0.32: What do economists do ? ``` It's clear from this example that the semantic search pipeline is clearly able to identify the meaning between each of our queries. Using these embeddings with Pinecone allows us to return the most semantically similar questions from the already indexed TREC dataset. Once we're finished with the index we delete it to save resources. ```Python Python pc.delete_index(name=index_name) ``` ## Related articles * [Generative Question-Answering with Long-Term Memory](https://www.pinecone.io/learn/openai-gen-qa) * [OpenAI's Text Embeddings v3](https://www.pinecone.io/learn/openai-embeddings-v3/) # Integrations Source: https://docs.pinecone.io/integrations/overview Pinecone integrations enable you to build and deploy AI applications faster and more efficiently. Integrate Pinecone with your favorite frameworks, data sources, and infrastructure providers. export const OpenAIIcon = () => ; export const SagemakerIcon = () => ; export const BedrockIcon = () => ;
Data Sources

Airbyte

Seamlessly integrate, transform, and load data into Pinecone from hundreds of systems, including databases, data warehouses, and SaasS products.
Frameworks

Amazon Bedrock

Integrate your enterprise data into Amazon Bedrock using Pinecone to build highly performant GenAI applications.
Frameworks

Amazon Sagemaker

Integrate machine learning models seamlessly with a fully-managed service that enables easy deployment and scalability.
Models

Anyscale

Focus on building applications powered by LLMs without the need to worry about the underlying infrastructure.
Data Sources

Apify

Integrate results from web scrapers or crawlers into a vector database to support RAG or semantic search over web content.
Data Sources

Aryn

Process complex, unstructured documents with a purpose-built ETL system for RAG and GenAI applications.
Infrastructure

AWS

Access Pinecone through our Amazon Web Services marketplace listing.
Infrastructure

Azure

Access Pinecone through our Microsoft Azure marketplace listing.
Data Sources

Box

Connect a Box account to a Pinecone vector database.
Frameworks

Cloudera AI

Vector embedding, RAG, and semantic search at scale.
Models

Cohere

Leverage cutting-edge natural language processing tools for enhanced text understanding and generation in your applications.
Data Sources

Confluent

Connect and process all of your data in real time with a cloud-native and complete data streaming platform.
Frameworks

Context Data

Create end-to-end data flows that connect data sources to Pinecone.
Data Sources

Databricks

Combine the power of a unified analytics platform with Pinecone for scalable data processing and AI insights.
Observability

Datadog

Monitor and secure your applications by integrating with a cloud-scale monitoring service that provides real-time analytics.
Data Sources

Datavolo

Source, transform, and enrich data in a continuous, composable and customizable manner.
Data Sources

Estuary

Source data from hundreds systems and push data to Pinecone, for an always up-to-date view.
Data Sources

FlowiseAI

Build customized LLM apps with an open source, low-code tool for developing orchestration flow & AI agents.
Data Sources

Fleak

Build, deploy, and manage complex workflows with a low-code platform for AI-assisted ML and LLM transformations.
Data Sources

Gathr

Build and operationalize data and AI-driven solutions at scale.
Infrastructure

GCP

Access Pinecone through our Google Cloud Marketplace listing.
Frameworks

Genkit

Build AI powered applications and agents.
Infrastructure

GitHub Copilot

Get personalized recommendations that enable you to retrieve relevant data and collaborate effectively with Copilot.
Frameworks

Haystack

Implement an end-to-end search pipeline for efficient retrieval and question answering over large datasets.
Observability

HoneyHive

Clearly visualize your execution traces and spans.
Frameworks

Hugging Face

Deploy state-of-the-art machine learning models on scalable infrastructure, streamlining the path from prototype to production.
Frameworks

Instill AI

Streamline AI development with a low-code full-stack infrastructure tool for data, model, and pipeline orchestration.
Models

Jina

Leverage powerful AI models to generate high-quality text embeddings, fine-tuned to both domain- and language-specific use cases.
Frameworks

LangChain

Combine language models with chain-of-thought reasoning for advanced problem solving and decision support.
Observability

Langtrace

Access rich and high cardinal tracing for Pinecone API calls, ingestible into your observability tool of choice.
Frameworks

Llama Index

Leverage Llama for indexing and retrieving information at scale, improving data access and analysis.
Data Sources

Matillion

Easily create and maintain data pipelines, build custom connectors for any source, and enjoy AI and high-code options to suit any need.
Observability

New Relic

Implement monitoring and integrate your Pinecone application with New Relic for performance analysis and insights.
Data Sources

Nexla

Ingest data from 500+ connectors with Nexla's low-code/no-code AI integration platform.
Frameworks

Nuclia

Nuclia RAG-as-a-Service automatically indexes files and documents from both internal and external sources.
Frameworks

OctoAI

Harness value from the latest AI innovations by delievering efficient, reliable, and customizable AI systems for your apps.
Models

OpenAI

Access powerful AI models like GPT for innovative applications and services, enhancing user experiences with AI capabilities.
Infrastructure

Pulumi

Manage your Pinecone collections and indexes using any language of Pulumi Infrastructure as Code.
Data Sources

Redpanda

Connect existing data sources to Pinecone with a Kafka-compatible streaming data platform built for data-intensive applications.
Data Sources

Snowflake

Run Pinecone with Snowpark Container Services, designed to deploy, manage, and scale containerized applications within the Snowflake ecosystem.
Data Sources

StreamNative

A scalable, resilient, and secure messaging and event streaming platform.
Infrastructure

Terraform

Manage your infrastructure using configuration files for a consistent workflow.
Observability

Traceloop

Produce traces and metrics that can be viewed in any OpenTelemetry-based platform.
Observability

TruLens

Gain insights into your machine learning models' decisions, improving interpretability and trustworthiness.
Models

Twelve Labs

Create high-quality multimodal embeddings that capture the rich context and interactions between different modalities in videos.
Data Source

Unstructured

Load data into Pinecone with a single click.
Infrastructure

Vercel

Use Pinecone as the long-term memory for your Vercel AI projects, and easily scale to support billions of data points.
Models

Voyage AI

Cutting-edge embedding models and rerankers for semantic search and RAG.

 {/* */} # Pulumi Source: https://docs.pinecone.io/integrations/pulumi export const PrimarySecondaryCTA = ({primaryLabel, primaryHref, primaryTarget, secondaryLabel, secondaryHref, secondaryTarget}) =>
{primaryLabel && primaryHref && } {secondaryLabel && secondaryHref && }
; Pulumi is an infrastructure as code platform that allows you to use familiar programming languages and tools to build, deploy, and manage cloud infrastructure. Pulumi is free, open source, and optionally pairs with the Pulumi Cloud to make managing infrastructure secure, reliable, and hassle-free. This Pulumi Pinecone Provider enables you to manage your Pinecone collections and indexes using any language of Pulumi Infrastructure as Code. # Redpanda Source: https://docs.pinecone.io/integrations/redpanda export const PrimarySecondaryCTA = ({primaryLabel, primaryHref, primaryTarget, secondaryLabel, secondaryHref, secondaryTarget}) =>
{primaryLabel && primaryHref && } {secondaryLabel && secondaryHref && }
; Redpanda Connect is a declarative data streaming service that solves a wide range of data engineering problems with simple, chained, stateless processing steps. It implements transaction based resiliency with back pressure, so when connecting to at-least-once sources and sinks, it's able to guarantee at-least-once delivery without needing to persist messages during transit. It's simple to deploy, comes with a wide range of connectors, and is totally data agnostic, making it easy to drop into your existing infrastructure. Redpanda Connect has functionality that overlaps with integration frameworks, log aggregators and ETL workflow engines, and can therefore be used to complement these traditional data engineering tools or act as a simpler alternative. The Pinecone connector for Redpanda provides a production-ready integration from many existing data sources, all in a few lines of YAML. # Snowflake Source: https://docs.pinecone.io/integrations/snowflake export const PrimarySecondaryCTA = ({primaryLabel, primaryHref, primaryTarget, secondaryLabel, secondaryHref, secondaryTarget}) =>
{primaryLabel && primaryHref && } {secondaryLabel && secondaryHref && }
; Deploy and run Pinecone with Snowpark Container Services. Snowpark Container Services is a fully managed container offering designed to facilitate the deployment, management, and scaling of containerized applications within the Snowflake ecosystem. This service enables users to run containerized workloads directly within Snowflake, ensuring that data does't need to be moved out of the Snowflake environment for processing. Unlike traditional container orchestration platforms like Docker or Kubernetes, Snowpark Container Services offers an OCI runtime execution environment specifically optimized for Snowflake. This integration allows for the seamless execution of OCI images, leveraging Snowflak's robust data platform. ## Related articles * [Snowpark Container Services: Securely Deploy and run Sophisticated Generative AI and full-stack apps in Snowflake](https://www.snowflake.com/blog/snowpark-container-services-deploy-genai-full-stack-apps/) # StreamNative Source: https://docs.pinecone.io/integrations/streamnative export const PrimarySecondaryCTA = ({primaryLabel, primaryHref, primaryTarget, secondaryLabel, secondaryHref, secondaryTarget}) =>
{primaryLabel && primaryHref && } {secondaryLabel && secondaryHref && }
; Founded by the original developers of Apache Pulsar and Apache BookKeeper, [StreamNative](https://streamnative.io) provides StreamNative Cloud, offering Apache Pulsar as a Service. The company also supports on-premise Pulsar deployments and related commercial support. StreamNative Cloud provides a scalable, resilient, and secure messaging and event streaming platform for enterprises. Additionally, StreamNative offers Kafka compatibility, enabling seamless integration with existing Kafka-based systems. The Pinecone integration with StreamNative allows access to pinecone.io with a Pulsar topic. The sink connector takes in messages and writes them if they are in a proper format to a Pinecone index. # Terraform Source: https://docs.pinecone.io/integrations/terraform Terraform is an infrastructure as code tool that lets you create, update, and version infrastructure by defining resources in configuration files. This allows for a repeated workflow for provisioning and managing your infrastructure. ## Setup guide This page shows you how to use the [Terraform Provider for Pinecone](https://registry.terraform.io/providers/pinecone-io/pinecone/latest/docs) to manage Pinecone indexes and collections. ### Requirements Ensure you have the following: * [Terraform](https://developer.hashicorp.com/terraform) >= v1.4.6 * [Go](https://go.dev/doc/install) >= v1.23.7 * A [Pinecone API key](https://app.pinecone.io/organizations/-/keys) ### Installation The provider is registered in the official [Terraform registry](https://registry.terraform.io/providers/pinecone-io/pinecone/latest). This enables the provider to be auto-installed when you run `terraform init`. You can also download the latest binary for your target platform from the [**Releases**](https://github.com/pinecone-io/terraform-provider-pinecone/releases) tab. ### Enable the provider To enable the provider in your Terraform configuration, set a `PINECONE_API_KEY` environment variable to your Pinecone API key and add the following to your Terraform configuration file: ```hcl terraform { required_providers { pinecone = { source = "pinecone-io/pinecone" } } } provider "pinecone" {} ``` If your API key was set as an [input variable](https://developer.hashicorp.com/terraform/language/values/variables), you can use that value in the declaration, for example: ```hcl provider "pinecone" { api_key = var.pinecone_api_key } ``` ### Resources The Pinecone Terraform Provider currently supports index and collection resources. #### Indexes The `pinecone_index` resource lets you create, delete, and update [indexes](/guides/index-data/indexing-overview). ```hcl // Create a dense index resource "pinecone_index" "dense-index-tf" { name = "tf-dense-index" dimension = 1536 metric = "cosine" spec = { serverless = { cloud = "aws" region = "us-west-2" } } deletion_protection = "disabled" tags = { environment = "development" } } // Create a sparse index resource "pinecone_index" "sparse-index-tf" { name = "tf-sparse-index" dimension = 10 metric = "dotproduct" vector_type = "sparse" spec = { serverless = { cloud = "aws" region = "us-west-2" } } deletion_protection = "disabled" tags = { environment = "development" } } // Create a dense index with integrated embedding resource "pinecone_index" "dense-index-integrated" { name = "tf-dense-index-with-integrated-embedding" spec = { serverless = { cloud = "aws" region = "us-west-2" } } embed = { model = "llama-text-embed-v2" field_map = { text = "chunk_text" } } } ``` #### Collections The `pinecone_collection` resource lets you create, delete, and update [collections](/guides/indexes/pods/understanding-collections) for pod-based indexes. ``` resource "pinecone_index" "test" { name = "tf-test-index" dimension = 10 spec = { pod = { environment = "us-west4-gcp" pod_type = "s1.x1" } } } resource "pinecone_collection" "example-collection" { name = "tf-example-collection" source = pinecone_index.test.name ``` ## Limitations The Terraform Provider for Pinecone does not currently support the following resources: * [Backups for serverless indexes](/guides/manage-data/backups-overview) * [Projects](/guides/projects/understanding-projects) * [API keys](/guides/projects/manage-api-keys) * [Service accounts](/guides/projects/manage-service-accounts) * [Private endpoints](/guides/production/connect-to-aws-privatelink) * [Assistants](/guides/assistant/overview) ## Additional resources * Documentation can be found on the [Terraform Registry](https://registry.terraform.io/providers/pinecone-io/pinecone/latest/docs). * See the [GitHub respository](https://github.com/pinecone-io/terraform-provider-pinecone/tree/main/examples) for additional usage examples. * For support requests, create an issue in the [GitHub repository](https://github.com/pinecone-io/terraform-provider-pinecone). # Traceloop Source: https://docs.pinecone.io/integrations/traceloop export const PrimarySecondaryCTA = ({primaryLabel, primaryHref, primaryTarget, secondaryLabel, secondaryHref, secondaryTarget}) =>
{primaryLabel && primaryHref && } {secondaryLabel && secondaryHref && }
; [Traceloop](https://www.traceloop.com/) provides observability for your LLM app using OpenTelemetry. Traceloop automatically monitors the quality of your LLM outputs. It helps you to debug and test changes to your models and prompts. The Pinecone integration with Traceloop produces traces and metrics that can be viewed in any OpenTelemetry-based platform like Datadog, Grafana, Traceloop, and others. # TruLens Source: https://docs.pinecone.io/integrations/trulens Using TruLens and Pinecone to evaluate grounded LLM applications export const PrimarySecondaryCTA = ({primaryLabel, primaryHref, primaryTarget, secondaryLabel, secondaryHref, secondaryTarget}) =>
{primaryLabel && primaryHref && } {secondaryLabel && secondaryHref && }
; TruLens is a powerful open source library for evaluating and tracking large language model-based applications. TruLens provides a set of tools for developing and monitoring neural nets, including large language models (LLMs). This includes both tools for evaluation of LLMs and LLM-based applications with TruLens-Eval and deep learning explainability with TruLens-Explain. To build an effective RAG-style LLM application, it is important to experiment with various configuration choices while setting up your Pinecone vector database, and study their impact on performance metrics. Tracking and evaluation with TruLens enables fast iteration of your application. ## Setup guide [TruLens](https://github.com/truera/trulens) is a powerful open source library for evaluating and tracking large language model-based applications. In this guide, we will show you how to use TruLens to evaluate applications built on top of a high performance Pinecone vector database. ### Why TruLens? Systematic evaluation is needed to support reliable, non-hallucinatory LLM-based applications. TruLens contains instrumentation and evaluation tools for large language model (LLM)-based applications. For evaluation, TruLens provides a set of feedback functions, analogous to labeling functions, to programmatically score the input, output and intermediate text of an LLM app. Each LLM application request can be scored on its question-answer relevance, context relevance and groundedness. These feedback functions provide evidence that your LLM-application is non-hallucinatory. ![diagram-1](https://mintlify.s3.us-west-1.amazonaws.com/pinecone/images/1.png) In addition to the above, feedback functions also support the evaluation of ground truth agreement, sentiment, model agreement, language match, toxicity, and a full suite of moderation evaluations, including hate, violence and more. TruLens implements feedback functions as an extensible framework that can evaluate your custom needs as well. During the development cycle, TruLens supports the iterative development of a wide range of LLM applications by wrapping your application to log cost, latency, key metadata and evaluations of each application run. This allows you to track and identify failure modes, pinpoint their root cause, and measure improvement across experiments. ![application-screenshot](https://mintlify.s3.us-west-1.amazonaws.com/pinecone/images/2.png) ### Why Pinecone? Large language models alone have a hallucination problem. Several decades of machine learning research have optimized models, including modern LLMs, for generalization, while actively penalizing memorization. However, many of today's applications require factual, grounded answers. LLMs are also expensive to train, and provided by third party APIs. This means the knowledge of an LLM is fixed. Retrieval-augmented generation (RAG) is a way to reliably ensure models are grounded, with Pinecone as the curated source of real world information, long term memory, application domain knowledge, or whitelisted data. In the RAG paradigm, rather than just passing a user question directly to a language model, the system retrieves any documents that could be relevant in answering the question from the knowledge base, and then passes those documents (along with the original question) to the language model to generate the final response. The most popular method for RAG involves chaining together LLMs with vector databases, such as the widely used Pinecone vector DB. In this process, a numerical vector (an embedding) is calculated for all documents, and those vectors are then stored in a database optimized for storing and querying vectors. Incoming queries are vectorized as well, typically using an encoder LLM to convert the query into an embedding. The query embedding is then matched via embedding similarity against the document embeddings in the vector database to retrieve the documents that are relevant to the query. ![diagram-2](https://mintlify.s3.us-west-1.amazonaws.com/pinecone/images/3.png) Pinecone makes it easy to build high-performance vector search applications, including retrieval-augmented question answering. Pinecone can easily handle very large scales of hundreds of millions and even billions of vector embeddings. Pinecone's large scale allows it to handle long term memory or a large corpus of rich external and domain-appropriate data so that the LLM component of RAG application can focus on tasks like summarization, inference and planning. This setup is optimal for developing a non-hallucinatory application.\ In addition, Pinecone is fully managed, so it is easy to change configurations and components. Combined with the tracking and evaluation with TruLens, this is a powerful combination that enables fast iteration of your application. ### Using Pinecone and TruLens to improve LLM performance and reduce hallucination To build an effective RAG-style LLM application, it is important to experiment with various configuration choices while setting up the vector database, and study their impact on performance metrics. In this example, we explore the downstream impact of some of these configuration choices on response quality, cost and latency with a sample LLM application built with Pinecone as the vector DB. The evaluation and experiment tracking is done with the [TruLens](https://www.trulens.org/) open source library. TruLens offers an extensible set of [feedback functions](https://truera.com/ai-quality-education/generative-ai-and-llms/whats-missing-to-evaluate-foundation-models-at-scale/) to evaluate LLM apps and enables developers to easily track their LLM app experiments. In each component of this application, different configuration choices can be made that can impact downstream performance. Some of these choices include the following: **Constructing the Vector DB** * Data preprocessing and selection * Chunk Size and Chunk Overlap * Index distance metric * Selection of embeddings **Retrieval** * Amount of context retrieved (top k) * Query planning **LLM** * Prompting * Model choice * Model parameters (size, temperature, frequency penalty, model retries, etc.) These configuration choices are useful to keep in mind when constructing your app. In general, there is no optimal choice for all use cases. Rather, we recommend that you experiment with and evaluate a variety of configurations to find the optimal selection as you are building your application. #### Creating the index in Pinecone Here we'll download a pre-embedded dataset from the `pinecone-datasets` library allowing us to skip the embedding and preprocessing steps. ```Python Python import pinecone_datasets dataset = pinecone_datasets.load_dataset('wikipedia-simple-text-embedding-ada-002-100K') dataset.head() ``` After downloading the data, we can initialize our pinecone environment and create our first index. Here, we have our first potentially important choice, by selecting the **distance metric** used for our index. Since all fields are currently indexed by default, we'll also pass in an additional empty metadata\_config parameter to avoid duplicative (and costly) indexing. ```Python Python pinecone.create_index( name=index_name_v1, metric='cosine', # We'll try each distance metric here. dimension=1536, # 1536 dim of text-embedding-ada-002, metadata_config={“indexed”:[]} # pass an empty list for names of indexed metadata fields ) ``` Then, we can upsert our documents into the index in batches. ```Python Python for batch in dataset.iter_documents(batch_size=100): index.upsert(batch) ``` #### Build the vector store Now that we've built our index, we can start using LangChain to initialize our vector store. ```Python Python embed = OpenAIEmbeddings( model='text-embedding-ada-002', openai_api_key=OPENAI_API_KEY ) from langchain.vectorstores import Pinecone text_field = "text" # Switch back to a normal index for LangChain. index = pinecone.Index(index_name_v1) vectorstore = Pinecone( index, embed.embed_query, text_field ) ``` In RAG, we take the query as a question that is to be answered by an LLM, but the LLM must answer the question based on the information it receives from the `vectorstore`. #### Initialize our RAG application To do this, we initialize a `RetrievalQA` as our app: ```Python Python from langchain.chat_models import ChatOpenAI from langchain.chains import RetrievalQA # completion llm llm = ChatOpenAI( model_name='gpt-3.5-turbo', temperature=0.0 ) qa = RetrievalQA.from_chain_type( llm=llm, chain_type="stuff", retriever=vectorstore.as_retriever() ) ``` #### TruLens for evaluation and tracking of LLM experiments Once we've set up our app, we should put together our [feedback functions](https://truera.com/ai-quality-education/generative-ai-and-llms/whats-missing-to-evaluate-foundation-models-at-scale/). As a reminder, feedback functions are an extensible method for evaluating LLMs. Here we'll set up two feedback functions: `qs_relevance` and `qa_relevance`. They're defined as follows: *QS Relevance: query-statement relevance is the average of relevance (0 to 1) for each context chunk returned by the semantic search.* *QA Relevance: question-answer relevance is the relevance (again, 0 to 1) of the final answer to the original question.* ```Python Python # Imports main tools for eval from trulens_eval import TruChain, Feedback, Tru, feedback, Select import numpy as np tru = Tru() # OpenAI as feedback provider openai = feedback.OpenAI() # Question/answer relevance between overall question and answer. qa_relevance = Feedback(openai.relevance).on_input_output() # Question/statement relevance between question and each context chunk. qs_relevance = Feedback(openai.qs_relevance). on_input() # See explanation below .on(Select.Record.app.combine_documents_chain._call.args.inputs.input_documents[:].page_content) .aggregate(np.mean) ``` Our use of selectors here also requires an explanation. QA Relevance is the simpler of the two. Here, we are using `.on_input_output()` to specify that the feedback function should be applied on both the input and output of the application. For QS Relevance, we use TruLens selectors to locate the context chunks retrieved by our application. Let's break it down into simple parts: 1. Argument Specification – The `on_input` which appears first is a convenient shorthand and states that the first argument to `qs_relevance` (the question) is to be the main input of the app. 2. Argument Specification – The `on(Select...)` line specifies where the statement argument to the implementation comes from. We want to evaluate the context chunks, which are an intermediate step of the LLM app. This form references the langchain app object call chain, which can be viewed from `tru.run_dashboard()`. This flexibility allows you to apply a feedback function to any intermediate step of your LLM app. Below is an example where TruLens displays how to select each piece of the context. ![subcomponents](https://mintlify.s3.us-west-1.amazonaws.com/pinecone/images/4.png) 3. Aggregation specification -- The last line aggregate (`np.mean`) specifies how feedback outputs are to be aggregated. This only applies to cases where the argument specification names more than one value for an input or output. The result of these lines is that `f_qs_relevance` can be now be run on apps/records and will automatically select the specified components of those apps/records To finish up, we just wrap our Retrieval QA app with TruLens along with a list of the feedback functions we will use for eval. ```Python Python # wrap with TruLens truchain = TruChain(qa, app_id='Chain1_WikipediaQA', feedbacks=[qa_relevance, qs_relevance]) truchain(“Which state is Washington D.C. in?”) ``` After submitting a number of queries to our application, we can track our experiment and evaluations with the TruLens dashboard. ```Python Python tru.run_dashboard() ``` Here is a view of our first experiment: ![trulens-dashboard-1](https://mintlify.s3.us-west-1.amazonaws.com/pinecone/images/5.png) #### Experiment with distance metrics Now that we've walked through the process of building our tracked RAG application using cosine as the distance metric, all we have to do for the next two experiments is to rebuild the index with `euclidean` or `dotproduct` as the metric and follow the rest of the steps above as is. Because we are using OpenAI embeddings, which are normalized to length 1, dot product and cosine distance are equivalent - and Euclidean will also yield the same ranking. See the OpenAI docs for more information. With the same document ranking, we should not expect a difference in response quality, but computation latency may vary across the metrics. Indeed, OpenAI advises that dot product computation may be a bit faster than cosine. We will be able to confirm this expected latency difference with TruLens. ```Python Python index_name_v2 = 'langchain-rag-euclidean' pinecone.create_index( name=index_name_v2, metric='euclidean', # metric='dotproduct', dimension=1536, # 1536 dim of text-embedding-ada-002 ) ``` After doing so, we can view our evaluations for all three LLM apps sitting on top of the different indexes. All three apps are struggling with query-statement relevance. In other words, the context retrieved is only somewhat relevant to the original query. **We can also see that both the Euclidean and dot-product metrics performed at a lower latency than cosine at roughly the same evaluation quality.** ![trulens-dashboard-2](https://mintlify.s3.us-west-1.amazonaws.com/pinecone/images/6.png) ### Problem: hallucination Digging deeper into the Query Statement Relevance, we notice one problem in particular with a question about famous dental floss brands. The app responds correctly, but is not backed up by the context retrieved, which does not mention any specific brands. ![trulens-dashboard-feedback-1](https://mintlify.s3.us-west-1.amazonaws.com/pinecone/images/7.png) #### Quickly evaluate app components with LangChain and TruLens Using a less powerful model is a common way to reduce hallucination for some applications. We'll evaluate ada-001 in our next experiment for this purpose. ![trulens-dashboard-3](https://mintlify.s3.us-west-1.amazonaws.com/pinecone/images/8.png) Changing different components of apps built with frameworks like LangChain is really easy. In this case we just need to call `text-ada-001` from the LangChain LLM store. Adding in easy evaluation with TruLens allows us to quickly iterate through different components to find our optimal app configuration. ```Python Python # completion llm from langchain.llms import OpenAI llm = OpenAI( model_name='text-ada-001', temperature=0 ) from langchain.chains import RetrievalQAWithSourcesChain qa_with_sources = RetrievalQA.from_chain_type( llm=llm, chain_type="stuff", retriever=vectorstore.as_retriever() ) # wrap with TruLens truchain = TruChain(qa_with_sources, app_id='Chain4_WikipediaQA', feedbacks=[qa_relevance, qs_relevance]) ``` **However, this configuration with a less powerful model struggles to return a relevant answer given the context provided.** ![trulens-dashboard-4](https://mintlify.s3.us-west-1.amazonaws.com/pinecone/images/9.png) For example, when asked “Which year was Hawaii's state song written?”, the app retrieves context that contains the correct answer but fails to respond with that answer, instead simply responding with the name of the song. ![trulens-dashboard-feedback-2](https://mintlify.s3.us-west-1.amazonaws.com/pinecone/images/10.png) While our relevance function is not doing a great job here in differentiating which context chunks are relevant, we can manually see that only the one (the 4th chunk) mentions the year the song was written. Narrowing our `top_k`, or the number of context chunks retrieved by the semantic search, may help. We can do so as follows: ```Python Python qa = RetrievalQA.from_chain_type( llm=llm, chain_type="stuff", retriever=vectorstore.as_retriever(top_k = 1) ) ``` The way the `top_k` is implemented in LangChain's RetrievalQA is that the documents are still retrieved by semantic search and only the `top_k` are passed to the LLM. Therefore, TruLens also captures all of the context chunks that are being retrieved. In order to calculate an accurate QS Relevance metric that matches what's being passed to the LLM, we only calculate the relevance of the top context chunk retrieved by slicing the `input_documents` passed into the TruLens Select function: ```Python Python qs_relevance = Feedback(openai.qs_relevance).on_input().on( Select.Record.app.combine_documents_chain._call.args.inputs.input_documents[:1].page_content ).aggregate(np.mean) ``` Once we've done so, our final application has much improved `qs_relevance`, `qa_relevance` and latency! ![trulens-dashboard-5](https://mintlify.s3.us-west-1.amazonaws.com/pinecone/images/11.png) With that change, our application is successfully retrieving the one piece of context it needs, and successfully forming an answer from that context. ![trulens-dashboard-feedback-3](https://mintlify.s3.us-west-1.amazonaws.com/pinecone/images/12.png) Even better, the application now knows what it doesn't know: ![trulens-dashboard-feedback-4](https://mintlify.s3.us-west-1.amazonaws.com/pinecone/images/13.png) ### Summary In conclusion, we note that exploring the downstream impact of some Pinecone configuration choices on response quality, cost and latency is an important part of the LLM app development process, ensuring that we make the choices that lead to the app performing the best. Overall, TruLens and Pinecone are the perfect combination for building reliable RAG-style applications. Pinecone provides a way to efficiently store and retrieve context used by LLM apps, and TruLens provides a way to track and evaluate each iteration of your application. # Twelve Labs Source: https://docs.pinecone.io/integrations/twelve-labs export const PrimarySecondaryCTA = ({primaryLabel, primaryHref, primaryTarget, secondaryLabel, secondaryHref, secondaryTarget}) =>
{primaryLabel && primaryHref && } {secondaryLabel && secondaryHref && }
; [Twelve Labs](https://twelvelabs.io) is an AI company that provides state-of-the-art video understanding capabilities through its easy-to-use APIs. Our newly released product is the Embed API, which enables developers to create high-quality multimodal embeddings that capture the rich context and interactions between different modalities in videos, such as visual expressions, body language, spoken words, and overall context. By integrating Twelve Labs' Embed API with Pinecone's vector database, developers can efficiently store, index, and retrieve these multimodal embeddings at scale. This integration empowers developers to build cutting-edge AI applications that leverage video data, such as video search, recommendation systems, content moderation, and more. Developers can seamlessly generate embeddings using Twelve Labs' API and store them in Pinecone for fast and accurate similarity search and retrieval. The integration of Twelve Labs and Pinecone offers developers a powerful toolkit to process and understand video content in a more human-like manner. By combining Twelve Labs' video-native approach with Pinecone's purpose-built vector search capabilities, developers can unlock new possibilities and build innovative applications across various industries, including media and entertainment, e-commerce, education, and beyond. ## Setup guide To integrate Twelve Labs' Embed API with Pinecone: 1. Sign up for a [Twelve Labs](https://twelvelabs.io) and obtain your API key. 2. Install the [Twelve Labs Python client library](https://github.com/twelvelabs-io/twelvelabs-python). 3. Sign up for a [Pinecone account](https://app.pinecone.io/) and [create an index](/guides/index-data/create-an-index). 4. Install the [Pinecone client library](/reference/pinecone-sdks). 5. Use the [Twelve Labs Embed API](https://docs.twelvelabs.io/docs/create-embeddings) to generate multimodal embeddings for your videos. 6. Connect to your Pinecone index and [upsert the embeddings](/guides/index-data/upsert-data). 7. [Query the Pinecone index](/guides/search/search-overview) to retrieve similar videos based on embeddings. For more detailed information and code examples, please see the [Twelve Labs documentation](https://docs.twelvelabs.io). # Unstructured Source: https://docs.pinecone.io/integrations/unstructured export const PrimarySecondaryCTA = ({primaryLabel, primaryHref, primaryTarget, secondaryLabel, secondaryHref, secondaryTarget}) =>
{primaryLabel && primaryHref && } {secondaryLabel && secondaryHref && }
; Unstructured builds ETL tools for LLMs, including an open source Python library, a SaaS API, and an ETL platform. Unstructured extracts content and metadata from 25+ document types, including PDFs, Word documents and PowerPoints. After extracting content and metadata, Unstructured performs additional preprocessing steps for LLMs such as chunking. Unstructured maintains upstream connections to data sources such as SharePoint and Google drive, and downstream connections to databases such as Pinecone. Integrating Pinecone with Unstructured enables developers to load data from an source or document type into Pinecone with a single click, accelerating the building of LLM apps that connect to organizational data. # Vercel Source: https://docs.pinecone.io/integrations/vercel export const PrimarySecondaryCTA = ({primaryLabel, primaryHref, primaryTarget, secondaryLabel, secondaryHref, secondaryTarget}) =>
{primaryLabel && primaryHref && } {secondaryLabel && secondaryHref && }
; Vercel is a platform for developers that provides the tools, workflows, and infrastructure you need to build and deploy your web apps faster, without the need for additional configuration. Vercel supports popular frontend frameworks out-of-the-box, and its scalable, secure infrastructure is globally distributed to serve content from data centers near your users for optimal speeds. Pinecone provides the long-term memory for your Vercel AI projects. Using Pinecone with Vercel enables you to quickly set up and authenticate a connection to your Pinecone data/indexes, and then easily scale to support billions of data points. The integration is designed to be self-serve with strong defaults for a smooth setup, with optional advanced settings. # Voyage AI Source: https://docs.pinecone.io/integrations/voyage Using Voyage AI and Pinecone to generate and index high-quality vector embeddings export const PrimarySecondaryCTA = ({primaryLabel, primaryHref, primaryTarget, secondaryLabel, secondaryHref, secondaryTarget}) =>
{primaryLabel && primaryHref && } {secondaryLabel && secondaryHref && }
; [Voyage AI](https://www.voyageai.com) provides cutting-edge embedding and rerankers. Voyage AI's generalist [embedding models](https://docs.voyageai.com/docs/embeddings) continually top the [MTEB leaderboard](https://huggingface.co/spaces/mteb/leaderboard), and the [domain-specific embeddings](https://blog.voyageai.com/2024/01/23/voyage-code-2-elevate-your-code-retrieval/) enhance the retrieval quality for enterprise use cases significantly. ## Setup guide In this guide, we use the [Voyage Embedding API endpoint](https://docs.voyageai.com/docs/embeddings) to generate text embeddings for terms of service and consumer contract documents, and then index those embeddings in the Pinecone vector database. This is a powerful and common combination for building retrieval-augmented generation (RAG), semantic search, question-answering, code assistants, and other applications that rely on NLP and search over a large corpus of text data. ### 1. Set up the environment Start by installing the Voyage and Pinecone clients and HuggingFace *Datasets* for downloading the *LegalBench: Consumer Contracts QA* ([`mteb/legalbench_consumer_contracts_qa`](https://huggingface.co/datasets/mteb/legalbench_consumer_contracts_qa)) dataset used in this guide: ```shell Shell pip install -U voyageai pinecone[grpc] datasets ``` ### 2. Create embeddings Sign up for an API key at [Voyage AI](https://dash.voyageai.com) and then use it to initialize your connection. ```Python Python import voyageai vc = voyageai.Client(api_key="") ``` Load the *LegalBench: Consumer Contracts QA* dataset, which contains 154 consumer contract documents and 396 labeled queries about these documents. ```Python Python from datasets import load_dataset # load the documents and queries of legalbench consumer contracts qa dataset documents = load_dataset('mteb/legalbench_consumer_contracts_qa', 'corpus', cache_dir = './', split='corpus') queries = load_dataset('mteb/legalbench_consumer_contracts_qa', 'queries', cache_dir = './', split='queries') ``` Each document in `mteb/legalbench_consumer_contracts_qa` contains a `text` field by which we will embed using the Voyage AI client. ```Python Python num_documents = len(documents['text']) voyageai_batch_size = 128 # Please check the restrictions of number of examples and number of tokens per request here https://docs.voyageai.com/docs/embeddings embeds = [] while len(embeds) < num_documents: embeds.extend(vc.embed( texts=documents['text'][len(embeds):len(embeds)+voyageai_batch_size], model='voyage-law-2', # Please check the available models here https://docs.voyageai.com/docs/embeddings input_type='document', truncation=True ).embeddings) ``` Check the dimensionality of the returned vectors. You will need to save the embedding dimensionality from this to be used when initializing your Pinecone index later. ```Python Python import numpy as np shape = np.array(embeds).shape print(shape) ``` ``` [Out]: (154, 1024) ``` In this example, you can see that for each of the `154` documents, we created a `1024`-dimensional embedding with the Voyage AI `voyage-law-2` model. ### 3. Store the Embeddings Now that you have your embeddings, you can move on to indexing them in the Pinecone vector database. For this, you need a Pinecone API key. [Sign up for one here](https://app.pinecone.io). You first initialize our connection to Pinecone and then create a new index called `voyageai-pinecone-legalbench` for storing the embeddings. When creating the index, you specify that you would like to use the cosine similarity metric to align with Voyage AI's embeddings, and also pass the embedding dimensionality of `1024`. ```Python Python from pinecone.grpc import PineconeGRPC as Pinecone from pinecone import ServerlessSpec # initialize connection to pinecone (get API key at app.pinecone.io) pc = Pinecone(api_key="") index_name = 'voyageai-pinecone-legalbench' # if the index does not exist, we create it if not pc.has_index(index_name): pc.create_index( index_name, dimension=shape[1], spec=ServerlessSpec( cloud='aws', region='us-east-1' ), metric='cosine' ) # connect to index index = pc.Index(index_name) ``` Now you can begin populating the index with your embeddings. Pinecone expects you to provide a list of tuples in the format (`id`, `vector`, `metadata`), where the `metadata` field is an optional extra field where you can store anything you want in a dictionary format. For this example, you will store the original text of the embeddings. While uploading your data, you will batch everything to avoid pushing too much data in one go. ```Python Python batch_size = 128 ids = [str(i) for i in range(shape[0])] # create list of metadata dictionaries meta = [{'text': text} for text in documents['text']] # create list of (id, vector, metadata) tuples to be upserted to_upsert = list(zip(ids, embeds, meta)) for i in range(0, shape[0], batch_size): i_end = min(i+batch_size, shape[0]) index.upsert(vectors=to_upsert[i:i_end]) # let's view the index statistics print(index.describe_index_stats()) ` `[Out]: {'dimension': 1024, 'index_fullness': 0.0, 'namespaces': {'': {'vector_count': 154}}, 'total_vector_count': 154} ``` You can see from `index.describe_index_stats` that you have a *1024-dimensionality* index populated with *154* embeddings. The `indexFullness` metric tells you how full your index is. At the moment, it is empty. Using the default value of one *p1* pod, you can fit around 750K embeddings before the `indexFullness` reaches capacity. The [Usage Estimator](https://www.pinecone.io/pricing/) can be used to identify the number of pods required for a given number of *n*-dimensional embeddings. ### 4. Semantic search Now that you have your indexed vectors, you can perform a few search queries. When searching, you will first embed your query using `voyage-law-2`, and then search using the returned vector in Pinecone. ```Python Python # get a sample query from the dataset, "Will Google help me if I think someone has taken and used content Ive created without my permission?" query = queries['text'][0] print(f"Query: {query}") # create the query embedding xq = vc.embed( texts=[query], model='voyage-law-2', input_type="query", truncation=True ).embeddings # query, returning the top 3 most similar results res = index.query(vector=xq, top_k=3, include_metadata=True) ``` The response from Pinecone includes your original text in the `metadata` field. Let's print out the `top_k` most similar questions and their respective similarity scores. ```Python Python for match in res['matches']: print(f"{match['score']:.2f}: {match['metadata']['text']}") ``` ```[Out]: 0.59: Your content Some of our services give you the opportunity to make your content publicly available for example, you might post a product or restaurant review that you wrote, or you might upload a blog post that you created. See the Permission to use your content section for more about your rights in your content, and how your content is used in our services See the Removing your content section to learn why and how we might remove user-generated content from our services If you think that someone is infringing your intellectual property rights, you can send us notice of the infringement and well take appropriate action. For example, we suspend or close the Google Accounts of repeat copyright infringers as described in our Copyright Help Centre. 0.47: Google content Some of our services include content that belongs to Google for example, many of the visual illustrations that you see in Google Maps. You may use Googles content as allowed by these terms and any service-specific additional terms, but we retain any intellectual property rights that we have in our content. Dont remove, obscure or alter any of our branding, logos or legal notices. If you want to use our branding or logos, please see the Google Brand Permissions page. Other content Finally, some of our services gives you access to content that belongs to other people or organisations for example, a store owners description of their own business, or a newspaper article displayed in Google News. You may not use this content without that person or organisations permission, or as otherwise allowed by law. The views expressed in the content of other people or organisations are their own, and dont necessarily reflect Googles views. 0.45: Taking action in case of problems Before taking action as described below, well provide you with advance notice when reasonably possible, describe the reason for our action and give you an opportunity to fix the problem, unless we reasonably believe that doing so would: cause harm or liability to a user, third party or Google violate the law or a legal enforcement authoritys order compromise an investigation compromise the operation, integrity or security of our services Removing your content If we reasonably believe that any of your content (1) breaches these terms, service-specific additional terms or policies, (2) violates applicable law, or (3) could harm our users, third parties or Google, then we reserve the right to take down some or all of that content in accordance with applicable law. Examples include child pornography, content that facilitates human trafficking or harassment, and content that infringes someone elses intellectual property rights. Suspending or terminating your access to Google services Google reserves the right to suspend or terminate your access to the services or delete your Google Account if any of these things happen: you materially or repeatedly breach these terms, service-specific additional terms or policies were required to do so to comply with a legal requirement or a court order we reasonably believe that your conduct causes harm or liability to a user, third party or Google for example, by hacking, phishing, harassing, spamming, misleading others or scraping content that doesnt belong to you If you believe that your Google Account has been suspended or terminated in error, you can appeal. Of course, youre always free to stop using our services at any time. If you do stop using a service, wed appreciate knowing why so that we can continue improving our services. ``` The semantic search pipeline with Voyage AI and Pinecone is able to identify the relevant consumer contract documents to answer the user query. # Model Gallery Source: https://docs.pinecone.io/models/overview Pinecone integrations enable you to build and deploy AI applications faster and more efficiently. Integrate Pinecone with your favorite frameworks, data sources, and infrastructure providers. export const ModelCard = ({logoSrc, company, modelName, description, task, modality, maxInputTokens, price, buttonLabel, buttonHref, buttonTarget, inferenceAPI, isPreviewCard, logoInvert}) => { const handleButtonClick = () => { if (inferenceAPI) { if (buttonTarget === "_blank") { window.open(buttonHref, buttonTarget); } else { window.location.href = buttonHref; } } else {} }; return
{`${company}

{modelName}

{company.toUpperCase()}

{description}

Task {task}
Modality {modality}
Max Input Tokens {maxInputTokens}
Price {price}
{isPreviewCard &&
Preview
}
; };
Inference

Build end-to-end faster with models hosted by Pinecone.

Request a model
{ const carousel = e.target; const prevButton = document.getElementById('prevButton'); const nextButton = document.getElementById('nextButton'); if (!prevButton || !nextButton) return; // Guard against null elements // Check if we're at the start or end const isAtStart = carousel.scrollLeft === 0; const isAtEnd = Math.abs(carousel.scrollLeft + carousel.offsetWidth - carousel.scrollWidth) < 1; // Update button visibility prevButton.style.visibility = isAtStart ? 'hidden' : 'visible'; nextButton.style.visibility = isAtEnd ? 'hidden' : 'visible'; }} >
{" "}