## Organization
An organization is a group of one or more [projects](#project) that use the same billing. Organizations allow one or more [users](#user) to control billing and permissions for all of the projects belonging to the organization.
For more information, see [Understanding organizations](/guides/organizations/understanding-organizations).
## Project
A project belongs to an [organization](#organization) and contains one or more [indexes](#index). Each project belongs to exactly one organization, but only [users](#user) who belong to the project can access the indexes in that project. [API keys](#api-key) and [Assistants](#assistant) are project-specific.
For more information, see [Understanding projects](/guides/projects/understanding-projects).
## Index
There are two types of [serverless indexes](/guides/index-data/indexing-overview), dense and sparse.
### Dense index
A dense index stores [dense vectors](#dense-vector), which are a series of numbers that represent the meaning and relationships of text, images, or other types of data. Each number in a dense vector corresponds to a point in a multidimensional space. Vectors that are closer together in that space are semantically similar.
When you query a dense index, Pinecone retrieves the dense vectors that are the most semantically similar to the query. This is often called **semantic search**, nearest neighbor search, similarity search, or just vector search.
### Sparse index
A sparse index stores [sparse vectors](#sparse-vector), which are a series of numbers that represent the words or phrases in a document. Sparse vectors have a very large number of dimensions, where only a small proportion of values are non-zero. The dimensions represent words from a dictionary, and the values represent the importance of these words in the document.
When you search a sparse index, Pinecone retrieves the sparse vectors that most exactly match the words or phrases in the query. Query terms are scored independently and then summed, with the most similar records scored highest. This is often called **lexical search** or **keyword search**.
## Namespace
A namespace is a partition within a [dense](#dense-index) or [sparse index](#sparse-index). It divides [records](#record) in an index into separate groups.
All [upserts](/guides/index-data/upsert-data), [queries](/guides/search/search-overview), and other [data operations](/reference/api/latest/data-plane) always target one namespace:
### API gateway
Every request to Pinecone includes an [API key](/guides/projects/manage-api-keys) that's assigned to a specific [project](/guides/projects/understanding-projects). The API gateway first validates your API key to make sure you have permission to access the project. Once validated, it routes your request to either the global control plane (for managing projects and indexes) or a regional data plane (for reading and writing data), depending on what you're trying to do.
### Control plane
The global control plane manages your organizational resources like projects and indexes. It uses a dedicated database to keep track of all these objects. The control plane also handles billing, user management, and coordinates operations across different regions.
### Data plane
The data plane handles all requests to write and read records in [indexes](/guides/index-data/indexing-overview) within a specific [cloud region](/guides/index-data/create-an-index#cloud-regions). Each index is divided into one or more logical [namespaces](/guides/index-data/indexing-overview#namespaces), and all your read and write requests target a specific namespace.
Pinecone separates write and read operations into different paths, with each scaling independently based on demand. This separation ensures that your queries never slow down your writes, and your writes never slow down your queries.
### Object storage
For each namespace in a serverless index, Pinecone organizes records into immutable files called slabs. These slabs are [optimized for fast querying](#index-builder) and stored in distributed object storage that provides virtually unlimited scalability and high availability.
## Write path
### Request log
When you send a write request (to add, update, or delete records), the [data plane](#data-plane) first logs the request details with a unique sequence number (LSN). This ensures all operations happen in the correct order and provides a way to track the state of the index.
Pinecone immediately returns a `200 OK` response, guaranteeing that your write is durable and won't be lost. The system then processes your write in the background.
### Index builder
The index builder stores your write data in an in-memory structure called a memtable. This includes your vector data, any metadata you've attached, and the sequence number. If you're updating or deleting a record, the system also tracks how to handle the old version during queries.
Periodically, the index builder moves data from the memtable to permanent storage. In [object storage](#object-storage), your data is organized into immutable files called slabs. These slabs are optimized for query performance. Smaller slabs use fast indexing techniques that provide good performance with minimal resource requirements. As slabs grow, the system merges them into larger slabs that use more sophisticated methods that provide better performance at scale. This adaptive process both optimizes query performance for each slab and amortizes the cost of more expensive indexing through the lifetime of the namespace.
### Query routers
When you send a search query, the [data plane](#data-plane) first validates your request and checks that it meets system limits like [rate and object limits](/reference/api/database-limits.mdx). The query router then identifies which slabs contain relevant data and routes your query to the appropriate executors. It also searches the memtable for any recent data that hasn't been moved to permanent storage yet.
### Query executors
Each query executor searches through its assigned slabs and returns the most relevant candidates to the query router. If your query includes metadata filters, the executors exclude records that don't match your criteria before finding the best matches.
Most of the time, the slabs are cached in memory or on local SSD, which provides very fast query performance. If a slab isn't cached (which happens when it's accessed for the first time or hasn't been used recently), the executor fetches it from object storage and caches it for future queries.
The query router then combines results from all executors, removes duplicates, merges them with results from the memtable, and returns the final set of best matches to you.
# Pinecone documentation
Source: https://docs.pinecone.io/guides/get-started/overview
Pinecone is the leading vector database for building accurate and performant AI applications at scale in production.
In Pinecone, an [index](/guides/index-data/indexing-overview) is the highest-level organizational unit of data, where you define the dimension of vectors to be stored in the index and the measure of similarity to be used when querying the index.
Within an index, records are stored in [namespaces](/guides/index-data/indexing-overview#namespaces), and all [upserts](/guides/index-data/upsert-data), [queries](/guides/search/search-overview), and other [data plane operations](/reference/api/latest/data-plane) always target one namespace.
This structure makes it easy to implement multitenancy. For example, for an AI-powered SaaS application where you need to isolate the data of each customer, you would assign each customer to a namespace and target their writes and queries to that namespace (diagram above).
In cases where you have different workload patterns (e.g., RAG and semantic search), you would use a different index for each workload, with one namespace per customer in each index:
## Vector embedding
[Dense vectors](/guides/get-started/concepts#dense-vector) and [sparse vectors](/guides/get-started/concepts#sparse-vector) are the basic units of data in Pinecone and what Pinecone was specially designed to store and work with. Dense vectors represents the semantics of data such as text, images, and audio recordings, while sparse vectors represent documents or queries in a way that captures keyword information.
To transform data into vector format, you use an embedding model. You can either use Pinecone's integrated embedding models to convert your source data to vectors automatically, or you can use an external embedding model and bring your own vectors to Pinecone.
### Integrated embedding
1. [Create an index](/guides/index-data/create-an-index) that is integrated with one of Pinecone's [hosted embedding models](/guides/index-data/create-an-index#embedding-models).
2. [Upsert](/guides/index-data/upsert-data) your source text. Pinecone uses the integrated model to convert the text to vectors automatically.
3. [Search](/guides/search/search-overview) with a query text. Again, Pinecone uses the integrated model to convert the text to a vector automatically.
96 records with text | | Max metadata size per record | 40 KB | | Max length for a record ID | 512 characters | | Max dimensionality for dense vectors | 20,000 | | Max non-zero values for sparse vectors | 2048 | | Max dimensionality for sparse vectors | 4.2 billion | # Back up a pod-based index Source: https://docs.pinecone.io/guides/indexes/pods/back-up-a-pod-based-index
When the new serverless index is ready, the status will change to green:
The BYOC architecture employs a split model:
* **Data plane**: The data plane is responsible for storing and processing your records, executing queries, and interacting with object storage for index data. In a BYOC deployment, the data plane is hosted in your own AWS or GCP account within a dedicated VPC, ensuring that all data is stored and processed locally and does not leave your organizational boundaries. You use a [private endpoint](#configure-a-private-endpoint) (AWS PrivateLink or GCP Private Service Connect) as an additional measure to secure requests to your indexes.
* **Control plane**: The control plane is responsible for managing the index lifecycle as well as region-agnostic services such as user management, authentication, and billing. The control plane does not hold or process any records. In a BYOC deployment, the control plane is managed by Pinecone and hosted globally. Communication between the data plane and control plane is encrypted using TLS and employs role-based access control (RBAC) with minimal IAM permissions.
## Onboarding
The onboarding process for BYOC in AWS or GCP involves the following general stages:
- Indexes: Permissions to query, import, fetch, add, update, and delete index data.
- Pinecone Assistant: Permissions to add, list, view, and delete files; chat with an assistant, and evaluate responses.
- Pinecone Inference: Permissions to generate embeddings and rerank documents.
- Indexes: Permissions to query, fetch, list ID, and view stats.
- Pinecone Assistant: Permissions to list and view files, chat with an assistant, and evaluate responses.
- Indexes: Permissions to query, import, fetch, add, update, and delete index data.
- Pinecone Assistant: Permissions to add, list, view, and delete files; chat with an assistant, and evaluate responses.
- Pinecone Inference: Permissions to generate embeddings and rerank documents.
- Indexes: Permissions to query, fetch, list ID, and view stats.
- Pinecone Assistant: Permissions to list and view files, chat with an assistant, and evaluate responses.
Traffic is also encrypted in transit between the Pinecone backend and cloud infrastructure services, such as S3 and GCS. For more information, see [Google Cloud Platform](https://cloud.google.com/docs/security/encryption-in-transit) and [AWS security documentation](https://docs.aws.amazon.com/AmazonS3/userguide/UsingEncryption.html).
## Network security
### Private Endpoints for AWS PrivateLink
Use [Private Endpoints to connect to Amazon Web Services (AWS) PrivateLink](/guides/production/connect-to-aws-privatelink). This establishes private connectivity between your Pinecone [serverless indexes](/guides/index-data/indexing-overview#serverless-indexes) and supported AWS services while keeping your VPC private from the public internet.
Private Endpoints are additive to other Pinecone security features: data is also [encrypted in transit](#encryption-in-transit), [encrypted at rest](#encryption-at-rest), and an [API key](#api-keys) is required to authenticate.
### Proxies
The following Pinecone SDKs support the use of proxies:
* [Python SDK](/reference/python-sdk#proxy-configuration)
* [Node.js SDK](/reference/node-sdk#proxy-configuration)
# Create a project
Source: https://docs.pinecone.io/guides/projects/create-a-project
This page shows you how to create a project.
If you are an [organization owner or user](/guides/organizations/understanding-organizations#organization-roles), you can create a project in your organization:
- You can start with dense for semantic search and add sparse for lexical search later.
- You can do sparse-only queries.
- You can rerank at multiple levels (for each index and for merged results).
- You can use integrated embedding and reranking.
- You need to manage and make requests to two separate indexes.
- You need to maintain the linkage between sparse and dense vectors in different indexes.
- You make requests to only a single index.
- The linkage between dense and sparse vectors is implicit.
- You can't do sparse-only queries.
- You can't use integrated embedding and reranking.
`END` truncates the input sequence at the input token limit. `NONE` returns an error when the input exceeds the input token limit. | `NONE` | | `rank_fields` | array of strings | Optional | The field to use for reranking. The model supports only a single rerank field. | `["text"]` |
`END` truncates the input sequence at the input token limit. `NONE` returns an error when the input exceeds the input token limit. | `END` | | `rank_fields` | array of strings | Optional | The field to use for reranking. The model supports only a single rerank field. | `["text"]` |
### Projects per organization | Starter plan | Standard plan | Enterprise plan | | ------------ | ------------- | --------------- | | 1 | 20 | 100 | When you reach this quota for an organization, trying to [create projects](/guides/projects/create-a-project) will fail and return a `403 - QUOTA_EXCEEDED` status with the following error: ``` Request failed. You've reached the max projects allowed in organization
96 records with text | | Max metadata size per record | 40 KB | | Max length for a record ID | 512 characters | | Max dimensionality for dense vectors | 20,000 | | Max non-zero values for sparse vectors | 2048 | | Max dimensionality for sparse vectors | 4.2 billion | ### Import limits | Metric | Limit | | :-------------------------------- | :-------------------------- | | Max size per import request | 2 TB or 200,000,000 records | | Max namespaces per import request | 10,000 | | Max files per import request | 100,000 | | Max size per file | 10 GB | ### Query limits | Metric | Limit | | :---------------- | :----- | | Max `top_k` value | 10,000 | | Max result size | 4MB | The query result size is affected by the dimension of the dense vectors and whether or not dense vector values and metadata are included in the result.
## Specify an API version
{title}
{text}
; }; export const ExampleCard = ({title, text, link, children, arrow, vectors, namespaces}) => { return{title}
{text}
{children &&{children}
}
{arrow && }
{(vectors || namespaces) &&
{vectors && {vectors} vectors}
{namespaces && {namespaces} namespaces}
}
;
};
export const Tag = ({text, icon}) => {
return
{icon && Search
Retrieval-augmented generation (RAG)
Miscellaneous
The official AWS Reference Architecture for high-scale systems using Pinecone.
[Documentation](https://github.com/pinecone-io/aws-reference-architecture-pulumi/blob/main/README.md) [Video tutorial](https://youtu.be/ySznARngHts) [Source code](https://github.com/pinecone-io/aws-reference-architecture-pulumi)
{title}
{text}
; }; export const ExampleCard = ({title, text, link, children, arrow, vectors, namespaces}) => { return{title}
{text}
{children &&{children}
}
{arrow && }
{(vectors || namespaces) &&
{vectors && {vectors} vectors}
{namespaces && {namespaces} namespaces}
}
;
};
More code examples
{primaryLabel && primaryHref && }
{secondaryLabel && secondaryHref && }
;
Airbyte offers a platform for creating ETL pipelines without writing integration code. It supports integrations with hundreds of systems, including databases, data warehouses, and SaaS products.
The Pinecone connector for Airbyte allows users to connect these systems directly into Pinecone. The connector fetches data from the connected source, embeds the data using an large language model (LLM), and then upserts it into Pinecone. From enhancing semantic search capabilities to building intelligent recommendation systems, the Pinecone Airbyte connector offers a versatile solution. By tapping into Airbyte's extensive array of source connectors, you can explore new ways to enrich your data-driven projects and achieve your specific goals.
{primaryLabel && primaryHref && }
{secondaryLabel && secondaryHref && }
;
Users can now select Pinecone as a Knowledge Base for [Amazon Bedrock](https://aws.amazon.com/bedrock/), a fully managed service from Amazon Web Services (AWS) for building GenAI applications.
The Pinecone vector database is a key component of the AI tech stack, helping companies solve one of the biggest challenges in deploying GenAI solutions — hallucinations — by allowing them to store, search, and find the most relevant and up-to-date information from company data and send that context to Large Language Models (LLMs) with every query. This workflow is called Retrieval Augmented Generation (RAG), and with Pinecone, it aids in providing relevant, accurate, and fast responses from search or GenAI applications to end users.
With the release of Knowledge Bases for Amazon Bedrock, developers can integrate their enterprise data into Amazon Bedrock using Pinecone as the fully-managed vector database to build GenAI applications that are:
* **Highly performant:** Speed through data in milliseconds. Leverage metadata filters and sparse-dense index support for top-notch relevance, ensuring quick, accurate, and grounded results across diverse search tasks.
* **Cost effective at scale:** Start for free on the starter plan and seamlessly scale usage with transparent usage-based pricing. Add or remove resources to meet your desired capacity and performance, upwards of billions of embeddings.
* **Enterprise ready:** Launch, use, and scale your AI solution without needing to maintain infrastructure, monitor services, or troubleshoot algorithms. Pinecone meets the security and operational requirements of enterprises.
## Setup guide
The process of using a Bedrock knowledge base with Pinecone works as follows:
3. Click **Next**.
4. Enter a **Secret name** and **Description**.
5. Click **Next** to save your key.
6. On the **Configure rotation** page, select all the default options in the next screen, and click **Next**.
7. Click **Store**.
8. Click on the new secret you created and save the secret ARN for a later step.
#### Set up S3
The knowledge base is going to draw on data saved in S3. For this example, we use a [sample of research papers](https://huggingface.co/datasets/jamescalam/ai-arxiv2-semantic-chunks) obtained from a dataset. This data will be embedded and then saved in Pinecone.
1. Create a new general purpose bucket in [Amazon S3](https://console.aws.amazon.com/s3/home).
2. Once the bucket is created, upload a CSV file.
By inspecting the trace, we can see what chunks were used by the Agent and diagnose issues with responses.
## Related articles
* [Pinecone as a Knowledge Base for Amazon Bedrock](https://www.pinecone.io/blog/amazon-bedrock-integration/)
# Amazon SageMaker
Source: https://docs.pinecone.io/integrations/amazon-sagemaker
export const PrimarySecondaryCTA = ({primaryLabel, primaryHref, primaryTarget, secondaryLabel, secondaryHref, secondaryTarget}) =>
{primaryLabel && primaryHref && }
{secondaryLabel && secondaryHref && }
;
Amazon SageMaker is a fully managed service that brings together a broad set of tools to enable high-performance, low-cost machine learning (ML) for any use case. With SageMaker, you can build, train and deploy ML models at scale-- all in one integrated development environment (IDE). SageMaker supports governance requirements with simplified access control and transparency over your ML projects. Amazon SageMaker offers access to hundreds of pretrained models, including publicly available foundational models (FMs), and you can build your own FMs with purpose-built tools to fine-tune, experiment, retrain, and deploy FMs.
Amazon SageMaker and Pinecone can be used together for high-performance, scalable, and reliable retrieval augmented generation (RAG) use cases. The integration uses Amazon SageMaker to compute and host models for large language Models (LLMs), and uses Pinecone as the knowledge base that keeps the LLMs up-to-date with the latest information, reducing the likelihood of hallucinations.
{primaryLabel && primaryHref && }
{secondaryLabel && secondaryHref && }
;
Anyscale Endpoints offers open-source large language models (LLMs) as fully managed API endpoints. This allows you to focus on building applications powered by LLMs without the need to worry about the underlying infrastructure.
Use Anyscale Endpoints with Canopy, an open source SDK that allows you to test, build, and package retrieval augmented generation (RAG) applications with Pinecone vector database.
{primaryLabel && primaryHref && }
{secondaryLabel && secondaryHref && }
;
[Apify](https://apify.com) is a web scraping and data extraction platform. It provides an app store with more than a thousand ready-made cloud tools called Actors. These tools are suitable for use cases including extracting structured data from e-commerce sites, social media, search engines, online maps, or any other website.
For example, the [Website Content Crawler](https://apify.com/apify/website-content-crawler) Actor can deeply crawl websites, clean their HTML by removing a cookies modal, footer, or navigation, and then transform the HTML into Markdown. This Markdown can then be used as training data for AI models or to feed LLM and generative AI applications with web content.
The Apify integration for Pinecone makes it easy to transfer results from Actors to the Pinecone vector database, enabling Retrieval-Augmented Generation (RAG) or semantic search over data extracted from the web.
{primaryLabel && primaryHref && }
{secondaryLabel && secondaryHref && }
;
Aryn is an AI-powered ETL system for complex, unstructured documents like PDFs, HTML, presentations, and more. It's purpose-built for building RAG and GenAI applications, providing up to 6x better accuracy in chunking and extracting information from documents. This can lead to 30% better recall and 2x improvement in answer accuracy for real-world use cases.
Aryn's ETL system has two components: Sycamore and the Aryn Partitioning Service. Sycamore is Aryn's open source document processing engine, available as a Python library. It contains a set of transforms for information extraction, LLM-powered enrichment, data cleaning, creating vector embeddings, and loading Pinecone indexes.
The Aryn Partitioning Service is used as a first step in a Sycamore data processing pipeline, and it identifies and extracts parts of documents, like text, tables, images, and more. It uses a state-of-the-art vision segmentation AI model, trained on hundreds of thousands of human-annotated documents.
The Pinecone integration with Aryn enables developers to easily chunk documents, create vector embeddings, and load Pinecone with high-quality data.
{primaryLabel && primaryHref && }
{secondaryLabel && secondaryHref && }
;
Access Pinecone through our Amazon Web Services (AWS) Marketplace listing. AWS Marketplace allows you to manage Pinecone and other third-party software from a centralized location, and simplifies software licensing and procurement with flexible pricing options and multiple deployment methods.
You can set up pay-as-you-go billing for a Pinecone organization through the AWS marketplace.
{primaryLabel && primaryHref && }
{secondaryLabel && secondaryHref && }
;
Access Pinecone through our Microsoft Azure marketplace listing. Azure Marketplace allows you to manage Pinecone and other third-party software from a centralized location, and simplifies software licensing and procurement with flexible pricing options and multiple deployment methods.
You can set up pay-as-you-go billing for a Pinecone organization through the Azure marketplace.
{primaryLabel && primaryHref && }
{secondaryLabel && secondaryHref && }
;
Connect a [Box](https://www.box.com/) account to a Pinecone vector database.
This integration allows embeddings generation based on content stored in a particular account and folder in Box. By default, the Pinecone Inference API is used, as well as OpenAI for the LLM. The integration can be used within a larger AI agent or workflow.
The `ConnectPopup` function can be called with either the JavaScript library or script. The JavaScript library is the most commonly used method, but the script can be used in instances where you cannot build and use a custom library, like within the constraints of a content management system (CMS).
The function includes the following **required** configuration option:
* `integrationId`: The slug assigned to the integration. If `integrationId` is not passed, the widget will not render.
[Once you have created your `integrationId`](#create-an-integration-id), you can embed the **Connect** widget multiple ways:
* [JavaScript](#javascript) library (`@pinecone-database/connect`) or script: Renders the widget in apps and websites.
* [Colab](#colab) (`pinecone-notebooks`): Renders the widget in Colab notebooks using Python.
Once you have created your integration, be sure to [attribute usage to your integration](/integrations/build-integration/attribute-usage-to-your-integration).
### JavaScript
To embed the **Connect to Pinecone** widget in your app or website using the [`@pinecone-database/connect` library](https://www.npmjs.com/package/@pinecone-database/connect), install the necessary dependencies:
```shell Shell
# Install dependencies
npm i -S @pinecone-database/connect
```
You can use the JavaScript library to render the **Connect to Pinecone** widget and obtain the API key with the [`connectToPinecone` function](#connecttopinecone-function). It displays the widget and calls the provided callback function with the Pinecone API key, once the user completes the flow.
The function includes the following **required** configuration options:
* `integrationId`: The slug assigned to the integration. If `integrationId` is not passed, the widget will not render.
{primaryLabel && primaryHref && }
{secondaryLabel && secondaryHref && }
;
[Cloudera AI](https://www.cloudera.com/) is an enterprise data cloud experience that provides scalable, secure, and agile machine learning and AI workflows. It leverages the power of Python, Apache Spark, R, and a host of other runtimes for distributed data processing, enabling the efficient creation, ingestion, and updating of vector embeddings at scale.
The primary advantage of Cloudera AI lies in its integration with the Cloudera ecosystem, which facilitates seamless data flow and processing across various stages of machine learning and AI pipelines. Cloudera AI offers interactive sessions, collaborative projects, model hosting capabilities, and application hosting features, all within a Python-centric development environment. This multifaceted approach enables users to efficiently develop, train, and deploy machine learning and AI models at scale.
Integrating Pinecone with Cloudera AI elevates the potential of Retrieval-Augmented Generation (RAG) models by providing a robust, scalable vector search platform. Pinecone's strength in handling vector embeddings — characterized by its ultra-low query latency, dynamic index updates, and scalability to billions of vector embeddings — make it the perfect match for the nuanced needs of RAG applications built on Cloudera AI.
Within the Cloudera AI ecosystem, Pinecone acts as a first-class citizen for RAG by efficiently retrieving relevant context from massive datasets, enhancing the generation capabilities of models with relevant, real-time data. This integration enables the development of sophisticated machine learning and AI applications that combine the predictive power of Cloudera AI's hosted models with the dynamic retrieval capabilities of Pinecone, offering unparalleled accuracy and relevance for generated outputs. By leveraging Cloudera AI's project and session management features, developers can prototype, develop, and deploy these complex systems more effectively, making advanced machine learning and AI applications more accessible and practical for enterprise use.
Cloudera's Accelerators for Machine Learning Projects (AMPs) drive efficient deployment of RAG architectures by doing the development work for you. This AMP serves as a prototype for fully integrating Pinecone into a RAG use case and illustrates semantic search with RAG at scale.
{primaryLabel && primaryHref && }
{secondaryLabel && secondaryHref && }
;
The Cohere platform builds natural language processing and generation into your product with a few lines of code. Cohere's large language models (LLMs) can solve a broad spectrum of natural language use cases, including classification, semantic search, paraphrasing, summarization, and content generation.
Use the Cohere Embed API endpoint to generate language embeddings, and then index those embeddings in the Pinecone vector database for fast and scalable vector search.
### Set up the environment
Start by installing the Cohere and Pinecone clients and HuggingFace *Datasets* for downloading the TREC dataset used in this guide:
```shell Shell
pip install -U cohere pinecone datasets
```
### Create embeddings
Sign up for an API key at [Cohere](https://dashboard.cohere.com/api-keys) and then use it to initialize your connection.
```Python Python
import cohere
co = cohere.Client("
{primaryLabel && primaryHref && }
{secondaryLabel && secondaryHref && }
;
Confluent allows you to connect and process all of your data in real time with a cloud-native and complete data streaming platform available everywhere you need it. Confluent's Kafka Connect is a tool for scalably and reliably streaming data between Apache Kafka and other data systems. It makes it simple to quickly define connectors that move large data sets in and out of Kafka.
Use the Pinecone Sink Connector, a Kafka Connect plugin for Pinecone, to take content from Confluent Cloud, convert it into vector embeddings using large language models (LLMs), and then store these embeddings in a Pinecone vector database.
{primaryLabel && primaryHref && }
{secondaryLabel && secondaryHref && }
;
Using [Context Data](https://contextdata.ai/), you can easily create end-to-end data flows by connecting to a myriad of data sources (PostgreSQL, MySQL, Amazon S3, Salesforce, etc.), seamlessly embedding and writing the results to Pinecone using Context Data's super simple no-code web interface. These flows can also be configured to be triggered and run on a user-defined schedule.
Additionally, Context Data provides the ability to create transformations like aggregations, joins, and feature engineering using SQL common table expressions before writing to Pinecone.
You also have the ability to "chat" with your Pinecone indexes directly from Context Data's privacy-focused Query Studio.
{primaryLabel && primaryHref && }
{secondaryLabel && secondaryHref && }
;
Databricks is a Unified Analytics Platform on top of Apache Spark. The primary advantage of using Spark is its ability to distribute workloads across a cluster of machines. By adding more machines or increasing the number of cores on each machine, it is easy to horizontally scale a cluster to handle computationally intensive tasks like vector embedding, where parallelization can save many hours of precious computation time and resources. Leveraging GPUs with Spark can produce even better results — enjoying the benefits of the fast computation of a GPU combined with parallelization will ensure optimal performance.
Efficiently create, ingest, and update vector embeddings at scale with Databricks and Pinecone.
{primaryLabel && primaryHref && }
{secondaryLabel && secondaryHref && }
;
{primaryLabel && primaryHref && }
{secondaryLabel && secondaryHref && }
;
[Datavolo](https://datavolo.io/) helps data teams build multimodal data pipelines to support their AI initiatives. Every organization has their own private data that they need to incorporate into their AI apps, and a predominant pattern to do so has emerged: retrieval augmented generation (RAG).
Datavolo sources, transforms, and enriches data in a continuous, composable and customizable manner, landing the data in Pinecone for retrieval. This ensures organizations can securely access their unstructured data.
{primaryLabel && primaryHref && }
{secondaryLabel && secondaryHref && }
;
[Estuary](https://estuary.dev/) builds real-time data pipelines that focus on moving data from sources to destinations with millisecond latency. It supports integrations with hundreds of systems including databases, warehouses, SaaS products and streaming solutions.
The Pinecone connector for Estuary enables users to source from these systems and push data to Pinecone, for an always up-to-date view. It incrementally updates source data to ensure that minimal credits are used when reaching out to get embeddings from providers prior to pushing them to Pinecone.
Estuary's Pinecone connector enables a variety of use cases like enabling LLM-based search across your organizations data and building intelligent recommendation systems. It can be set up in 5 minutes, without any engineering effort to maximize efficiency.
{primaryLabel && primaryHref && }
{secondaryLabel && secondaryHref && }
;
Fleak simplifies the process of building, deploying, and managing data workflows. As a low-code platform, Fleak lets users create and deploy complex workflows using SQL and pre-configured processing nodes. The platform facilitates seamless data processing, microservice interactions, and the inference of large language models (LLMs) within a single, intuitive environment. Fleak makes advanced technology accessible and manageable for Pinecone users without requiring extensive coding hours or infrastructure knowledge.
The platform provides serverless, autoscaling HTTP API endpoints, ensuring that workflows are robust, reliable, and scalable to meet the complex data needs of enterprises. This setup allows businesses to automate and enhance their operations, driving productivity and innovation through powerful, user-friendly tools.
By integrating Pinecone into Fleak's platform, users can access and leverage their vector data to enrich their workflows without additional engineering overhead, enabling seamless data-driven decision-making, advanced analytics, and the integration of AI-driven insights.
{primaryLabel && primaryHref && }
{secondaryLabel && secondaryHref && }
;
Flowise is a low-code LLM apps development platform. It supports integrations with dozens of systems, including databases and chat models.
The Pinecone integration with Flowise allows users to build RAG apps, including upserting and querying documents.
{primaryLabel && primaryHref && }
{secondaryLabel && secondaryHref && }
;
[Gathr](https://www.gathr.one/) is the world's first and only "data to outcome" platform. Leading enterprises use Gathr to build and operationalize data and AI-driven solutions at scale.
Gathr unifies data engineering, machine learning, generative AI, actionable analytics, and process automation on a single platform. With Gen AI capabilities and no-code rapid application development, Gathr significantly boosts productivity for all. The unified experience fosters seamless handoff and collaboration between teams, accelerating the journey from prototype to production.
Users have achieved success with Gathr, from ingesting petabyte-scale data in real time to orchestrating thousands of complex data processing pipelines in months and delivering actionable insights and xOps solutions to multiply business impact. Additionally, Gathr helps enterprises architect Gen AI solutions for use cases like document summarization, sentiment analysis, next best action, insider threat detection, predictive maintenance, custom chatbots, and more.
Gathr Gen AI Fabric is designed to build enterprise-grade Gen AI solutions end-to-end on a unified platform. It offers production-ready building blocks for creating Gen AI solutions, out-of-the-box Gen AI solution templates, and GathrIQ, a data-to-outcome copilot.
One of the building blocks is integration with Vector DB and Knowledge Graphs. Gathr supports reading and writing from Pinecone using a built-in, ready-to-use connector to support use cases requiring knowledge graphs.
{primaryLabel && primaryHref && }
{secondaryLabel && secondaryHref && }
;
Access Pinecone through our Google Cloud Marketplace listing. Google Cloud Marketplace allows you to manage Pinecone and other third-party software from a centralized location, and simplifies software licensing and procurement with flexible pricing options and multiple deployment methods.
You can set up pay-as-you-go billing for a Pinecone organization through the Google Cloud Marketplace.
{primaryLabel && primaryHref && }
{secondaryLabel && secondaryHref && }
;
The [Genkit](https://firebase.google.com/docs/genkit) Pinecone plugin empowers developers to reduce the complexity of integrating AI components through simple indexers, embedders and retrievers abstractions. Through the Genkit Pinecone plugin, developers can integrate AI models with their own custom logic and data to build AI features optimized for their businesses.
Additionally, developers can analyze unstructured text, generate creative content, select tasks, and send results back to their app as structured type-safe objects.
The plugin provides a common format for content that supports combinations of text, data, and other media. Developers can use Genkit for models that perform any generative task (such as image generation), not just LLMs.
{primaryLabel && primaryHref && }
{secondaryLabel && secondaryHref && }
;
Access the Pinecone Copilot Extension through our GitHub Marketplace listing. The Pinecone Copilot Extension serves as a seamless bridge between you and your Pinecone data-- providing product information, coding assistance, troubleshooting capabilities and streamlining the debugging process.
This extension offers personalized recommendations right to your fingertips, enabling you to swiftly retrieve relevant data and collaborate effectively with Copilot.
{primaryLabel && primaryHref && }
{secondaryLabel && secondaryHref && }
;
Haystack is the open source Python framework by Deepset for building custom apps with large language models (LLMs). It lets you quickly try out the latest models in natural language processing (NLP) while being flexible and easy to use. Their community of users and builders has helped shape Haystack into what it is today: a complete framework for building production-ready NLP apps.
Haystack and Pinecone integration can be used to keep your NLP-driven apps up-to-date with Haystack's indexing pipelines that help you prepare and maintain your data.
{primaryLabel && primaryHref && }
{secondaryLabel && secondaryHref && }
;
Hugging Face Inference Endpoints offers a secure production solution to easily deploy any Hugging Face Transformers, Sentence-Transformers and Diffusion models from the Hub on dedicated and autoscaling infrastructure managed by Hugging Face.
Coupled with Pinecone, you can use Hugging Face to generate and index high-quality vector embeddings with ease.
We click on **Create new endpoint**, choose a model repository (eg name of the model), endpoint name (this can be anything), and select a cloud environment. Before moving on it is *very important* that we set the **Task** to **Sentence Embeddings** (found within the *Advanced configuration* settings).
Other important options include the *Instance Type*, by default this uses CPU which is cheaper but also slower. For faster processing we need a GPU instance. And finally, we set our privacy setting near the end of the page.
After setting our options we can click **Create Endpoint** at the bottom of the page. This action should take use to the next page where we will see the current status of our endpoint.
Once the status has moved from **Building** to **Running** (this can take some time), we're ready to begin creating embeddings with it.
## Create embeddings
Each endpoint is given an **Endpoint URL**, it can be found on the endpoint **Overview** page. We need to assign this endpoint URL to the `endpoint_url` variable.
```Python Python
endpoint = "
```Python Python
api_org = "
{primaryLabel && primaryHref && }
{secondaryLabel && secondaryHref && }
;
Instill AI specializes in developing cutting-edge solutions for data, models, and pipeline orchestration. Their flagship source-available product, Instill Core, is a no-code/low-code platform designed to facilitate the development, deployment, and management of AI workflows. By simplifying the integration of various AI models and data sources, they enable businesses to harness the power of AI without requiring extensive technical expertise. Their solutions cater to a wide range of applications, from predictive analytics and autonomous AI agents to enterprise private knowledge bases, AI assistants, and beyond.
The Pinecone integration with Instill allows developers to incorporate its API for vector upsert and query tasks into AI pipelines. Developers configure their Pinecone component within Instill Core by providing the necessary API key and base URL. They can then perform tasks such as querying vector similarities to retrieve the most relevant results, complete with metadata and similarity scores, or upserting new vector data to keep their datasets up-to-date. This integration enables the addition of knowledge to LLMs via Retrieval Augmented Generation (RAG), significantly enhancing the capabilities of autonomous agents, chatbots, question-answering systems, and multi-agent systems.
{primaryLabel && primaryHref && }
{secondaryLabel && secondaryHref && }
;
Jina Embeddings leverage powerful models to generate high-quality text embeddings that can process inputs up to 8,000 tokens. Jina Embeddings are designed to be highly versatile, catering to both domain-specific use cases, such as e-commerce, and language-specific needs, including Chinese and German. By providing robust models and the expertise to fine-tune them for specific requirements, Jina AI empowers developers to enhance their search functionalities, improve natural language understanding, and drive more insightful data analysis.
By integrating Pinecone with Jina, you can add knowledge to LLMs via retrieval augmented generation (RAG), greatly enhancing LLM ability for autonomous agents, chatbots, question-answering, and multi-agent systems.
{primaryLabel && primaryHref && }
{secondaryLabel && secondaryHref && }
;
LangChain provides modules for managing and optimizing the use of large language models (LLMs) in applications. Its core philosophy is to facilitate data-aware applications where the language model interacts with other data sources and its environment. This framework consists of several parts that simplify the entire application lifecycle:
* Write your applications in LangChain/LangChain.js. Get started quickly by using Templates for reference.
* Use LangSmith to inspect, test, and monitor your chains to constantly improve and deploy with confidence.
* Turn any chain into an API with LangServe.
By integrating Pinecone with LangChain, you can add knowledge to LLMs via retrieval augmented generation (RAG), greatly enhancing LLM ability for autonomous agents, chatbots, question-answering, and multi-agent systems.
{primaryLabel && primaryHref && }
{secondaryLabel && secondaryHref && }
;
Scale3 Labs recently launched Langtrace AI, an open-source monitoring and evaluation platform for LLM-powered applications. Langtrace is built based on Open Telemetry(OTEL) standards and supports native tracing for the most popular LLM vendors, VectorDBs, and frameworks(like Langchain and LlamaIndex).
Langtrace AI supports tracing Pinecone natively, which means the Langtrace SDK can generate OTEL standard traces with automatic instrumentation in just 2 lines of code. These traces can be ingested by an observability tool that supports OTEL, such as Datadog, Grafana/Prometheus, SigNoz, Sentry, etc. Langtrace also has a visualization client that is optimized for visualizing the traces generated in an LLM stack.
By having a Pinecone integration, Pinecone users can get access to rich and high cardinal tracing for the Pinecone API calls using Langtrace, which they can ingest into their observability tool of choice. This helps customers gain insights into the DB calls and help with debugging and troubleshooting applications in case of incidents.
{primaryLabel && primaryHref && }
{secondaryLabel && secondaryHref && }
;
LlamaIndex is a framework for connecting data sources to LLMs, with its chief use case being the end-to-end development of retrieval augmented generation (RAG) applications. LlamaIndex provides the essential abstractions to more easily ingest, structure, and access private or domain-specific data in order to inject these safely and reliably into LLMs for more accurate text generation. It’s available in Python and Typescript.
Seamlessly integrate Pinecone vector database with LlamaIndex to build semantic search and RAG applications.
{primaryLabel && primaryHref && }
{secondaryLabel && secondaryHref && }
;
[Matillion Data Productivity Cloud](https://www.matillion.com/) is a unified platform that helps your team move faster with one central place to build and manage graphical, low-code data pipelines. It allows data teams to use structured, semi-structured, and unstructured data in analytics; build AI pipelines for new use cases; and be more productive.
Matillion Data Productivity Cloud and Pinecone can be used together for retrieval augmented generation (RAG) use cases, helping to contextualize business insights without code.
Matillion supports 150+ pre-built data source connectors, as well as the ability to build custom connectors to any REST API source system, making it easy to chunk unstructured datasets, create embeddings, and upsert to Pinecone.
Matillion's graphical AI Prompt Components integrate with large language models (LLM) running in OpenAI, Amazon Bedrock, Azure OpenAI, and Snowpark Container Services. They enable no-code lookup of external knowledge stored in Pinecone, enabling data engineers to enrich GenAI answers with contextualized and proprietary data.
{primaryLabel && primaryHref && }
{secondaryLabel && secondaryHref && }
;
New Relic is an all-in-one observability platform and provides the industry’s first APM solution for AI-powered applications. New Relic is pioneering AI observability with AIM to provide engineers unprecedented visibility and insights across the AI application stack, making it easier to troubleshoot and optimize their AI applications for performance, quality, cost, and responsible use of AI.
Implement monitoring and integrate your Pinecone application with New Relic for performance analysis and insights. The New Relic for Pinecone (Prometheus) quickstart contains one dashboard. These interactive visualizations let you easily explore your data, understand context, and resolve problems faster. It also includes three alerts to detect changes in key performance metrics. Integrate these alerts with your favorite tools (like Slack, PagerDuty, etc.) and New Relic will let you know when something needs your attention.
{primaryLabel && primaryHref && }
{secondaryLabel && secondaryHref && }
;
Nexla is a Data + AI Integration Platform that makes it easy for users to build data pipelines in a no-code/low-code manner.
The Pinecone integration with Nexla makes it easy for enterprise users to ingest data from systems like Sharepoint, OneDrive, Cloud Storage, Data Warehouses, and 500+ other connectors that Nexla supports natively.
{primaryLabel && primaryHref && }
{secondaryLabel && secondaryHref && }
;
[Nuclia](https://nuclia.com/) RAG-as-a-Service automatically indexes files and documents from both internal and external sources, powering diverse company use cases with large language models (LLMs). This comprehensive indexing capability ensures that organizations can leverage unstructured data effectively, transforming it into actionable insights. With Nuclia's modular Retrieval-Augmented Generation (RAG) system, you can deploy solutions tailored to various operational needs across different deployment options, enhancing flexibility and efficiency.
The modular RAG system from Nuclia is designed to fit specific use cases, allowing you to customize your RAG pipeline to meet your unique requirements. Whether it's defining your own retrieval and chunking strategies or choosing from various embedding models, Nuclia's RAG-as-a-Service makes it easy to bring your tailored solutions into production. This customization not only improves the value of your products but also helps you stay competitive by automating tasks and making your data smarter with LLMs, saving hundreds of hours in the process.
When you create a knowledge box at Nuclia, choose to store the index in Pinecone. This is especially useful for large datasets where full text search is not key on the retrieval phase.
{primaryLabel && primaryHref && }
{secondaryLabel && secondaryHref && }
;
Harness value from the latest AI innovations by delievering efficient, reliable, and customizable AI systems for your apps. Run your models or checkpoints on OctoAI's cost-effective API endpoints, or run OctoAI's optimized GenAI stack in your environment.
Choose from the best models that OctoAI has to offer, including GTE Large embedding model, the best foundational open source LLMs such as Mixtral-8x7B from Mistral AI, Llama2 from Meta, and highly capable model fine tunes like Nous Hermes 2 Pro Mistral from Nous Research.
As a fully open source solution, Pinecone Canopy and OctoAI is one of the fastest ways and more affordable ways to get started on your RAG journey. Canopy uses Pinecone vector database for storage and retrieval, which is free to use for up to 100k vectors (that's about 30k pages of text).
{primaryLabel && primaryHref && }
{secondaryLabel && secondaryHref && }
;
OpenAI's large language models (LLMs) enhance semantic search or “long-term memory” for LLMs. This combo utilizes LLMs' embedding and completion (or generation) endpoints alongside Pinecone's vector search capabilities for nuanced information retrieval.
By integrating OpenAI's LLMs with Pinecone, you can combine deep learning capabilities for embedding generation with efficient vector storage and retrieval. This approach surpasses traditional keyword-based search, offering contextually-aware, precise results.
Let's get started...
### Environment Setup
We start by installing the OpenAI and Pinecone clients, we will also need HuggingFace *Datasets* for downloading the TREC dataset that we will use in this guide.
```Bash Bash
!pip install -qU \
pinecone[grpc]==7.3.0 \
openai==1.93.0 \
datasets==3.6.0
```
#### Creating Embeddings
To create embeddings we must first initialize our connection to OpenAI Embeddings, we sign up for an API key at [OpenAI](https://beta.openai.com/signup).
```Python Python
from openai import OpenAI
client = OpenAI(
api_key="OPENAI_API_KEY"
) # get API key from platform.openai.com
```
We can now create embeddings with the OpenAI v3 small embedding model like so:
```Python Python
MODEL = "text-embedding-3-small"
res = client.embeddings.create(
input=[
"Sample document text goes here",
"there will be several phrases in each batch"
], model=MODEL
)
```
In `res` we should find a JSON-like object containing two 1536-dimensional embeddings, these are the vector representations of the two inputs provided above. To access the embeddings directly we can write:
```Python Python
# we can extract embeddings to a list
embeds = [record.embedding for record in res.data]
len(embeds)
```
We will use this logic when creating our embeddings for the **T**ext **RE**trieval **C**onference (TREC) question classification dataset later.
#### Initializing a Pinecone Index
Next, we initialize an index to store the vector embeddings. For this we need a Pinecone API key, [sign up for one here](https://app.pinecone.io).
```Python Python
import time
from pinecone.grpc import PineconeGRPC as Pinecone
from pinecone import ServerlessSpec
pc = Pinecone(api_key="...")
spec = ServerlessSpec(cloud="aws", region="us-east-1")
index_name = 'semantic-search-openai'
# check if index already exists (it shouldn't if this is your first run)
if index_name not in pc.list_indexes().names():
# if does not exist, create index
pc.create_index(
index_name,
dimension=len(embeds[0]), # dimensionality of text-embed-3-small
metric='dotproduct',
spec=spec
)
# connect to index
index = pc.Index(index_name)
time.sleep(1)
# view index stats
index.describe_index_stats()
```
#### Populating the Index
With both OpenAI and Pinecone connections initialized, we can move onto populating the index. For this, we need the TREC dataset.
```Python Python
from datasets import load_dataset
# load the first 1K rows of the TREC dataset
trec = load_dataset('trec', split='train[:1000]')
```
Then we create a vector embedding for each question using OpenAI (as demonstrated earlier), and `upsert` the ID, vector embedding, and original text for each phrase to Pinecone.
Airbyte
Seamlessly integrate, transform, and load data into Pinecone from hundreds of systems, including databases, data warehouses, and SaasS products.
Amazon Bedrock
Integrate your enterprise data into Amazon Bedrock using Pinecone to build highly performant GenAI applications.
Amazon Sagemaker
Integrate machine learning models seamlessly with a fully-managed service that enables easy deployment and scalability.
Anyscale
Focus on building applications powered by LLMs without the need to worry about the underlying infrastructure.
Apify
Integrate results from web scrapers or crawlers into a vector database to support RAG or semantic search over web content.
Aryn
Process complex, unstructured documents with a purpose-built ETL system for RAG and GenAI applications.
AWS
Access Pinecone through our Amazon Web Services marketplace listing.
Azure
Access Pinecone through our Microsoft Azure marketplace listing.
Box
Connect a Box account to a Pinecone vector database.
Cloudera AI
Vector embedding, RAG, and semantic search at scale.
Cohere
Leverage cutting-edge natural language processing tools for enhanced text understanding and generation in your applications.
Confluent
Connect and process all of your data in real time with a cloud-native and complete data streaming platform.
Context Data
Create end-to-end data flows that connect data sources to Pinecone.
Databricks
Combine the power of a unified analytics platform with Pinecone for scalable data processing and AI insights.
Datadog
Monitor and secure your applications by integrating with a cloud-scale monitoring service that provides real-time analytics.
Datavolo
Source, transform, and enrich data in a continuous, composable and customizable manner.
Estuary
Source data from hundreds systems and push data to Pinecone, for an always up-to-date view.
FlowiseAI
Build customized LLM apps with an open source, low-code tool for developing orchestration flow & AI agents.
Fleak
Build, deploy, and manage complex workflows with a low-code platform for AI-assisted ML and LLM transformations.
Gathr
Build and operationalize data and AI-driven solutions at scale.
GCP
Access Pinecone through our Google Cloud Marketplace listing.
Genkit
Build AI powered applications and agents.
GitHub Copilot
Get personalized recommendations that enable you to retrieve relevant data and collaborate effectively with Copilot.
Haystack
Implement an end-to-end search pipeline for efficient retrieval and question answering over large datasets.
HoneyHive
Clearly visualize your execution traces and spans.
Hugging Face
Deploy state-of-the-art machine learning models on scalable infrastructure, streamlining the path from prototype to production.
Instill AI
Streamline AI development with a low-code full-stack infrastructure tool for data, model, and pipeline orchestration.
Jina
Leverage powerful AI models to generate high-quality text embeddings, fine-tuned to both domain- and language-specific use cases.
LangChain
Combine language models with chain-of-thought reasoning for advanced problem solving and decision support.
Langtrace
Access rich and high cardinal tracing for Pinecone API calls, ingestible into your observability tool of choice.
Llama Index
Leverage Llama for indexing and retrieving information at scale, improving data access and analysis.
Matillion
Easily create and maintain data pipelines, build custom connectors for any source, and enjoy AI and high-code options to suit any need.
New Relic
Implement monitoring and integrate your Pinecone application with New Relic for performance analysis and insights.
Nexla
Ingest data from 500+ connectors with Nexla's low-code/no-code AI integration platform.
Nuclia
Nuclia RAG-as-a-Service automatically indexes files and documents from both internal and external sources.
OctoAI
Harness value from the latest AI innovations by delievering efficient, reliable, and customizable AI systems for your apps.
OpenAI
Access powerful AI models like GPT for innovative applications and services, enhancing user experiences with AI capabilities.
Pulumi
Manage your Pinecone collections and indexes using any language of Pulumi Infrastructure as Code.
Redpanda
Connect existing data sources to Pinecone with a Kafka-compatible streaming data platform built for data-intensive applications.
Snowflake
Run Pinecone with Snowpark Container Services, designed to deploy, manage, and scale containerized applications within the Snowflake ecosystem.
StreamNative
A scalable, resilient, and secure messaging and event streaming platform.
Terraform
Manage your infrastructure using configuration files for a consistent workflow.
Traceloop
Produce traces and metrics that can be viewed in any OpenTelemetry-based platform.
TruLens
Gain insights into your machine learning models' decisions, improving interpretability and trustworthiness.
Twelve Labs
Create high-quality multimodal embeddings that capture the rich context and interactions between different modalities in videos.
Unstructured
Load data into Pinecone with a single click.
Vercel
Use Pinecone as the long-term memory for your Vercel AI projects, and easily scale to support billions of data points.
VoltAgent
A TypeScript-based, AI-agent framework for building AI applications with retrieval-augmented generation (RAG) capabilities.
Voyage AI
Cutting-edge embedding models and rerankers for semantic search and RAG.
Zapier
Zapier connects Pinecone to thousands of apps to help you automate your work. No code required.
{primaryLabel && primaryHref && }
{secondaryLabel && secondaryHref && }
;
Pulumi is an infrastructure as code platform that allows you to use familiar programming languages and tools to build, deploy, and manage cloud infrastructure. Pulumi is free, open source, and optionally pairs with the Pulumi Cloud to make managing infrastructure secure, reliable, and hassle-free.
This Pulumi Pinecone Provider enables you to manage your Pinecone collections and indexes using any language of Pulumi Infrastructure as Code.
{primaryLabel && primaryHref && }
{secondaryLabel && secondaryHref && }
;
Redpanda Connect is a declarative data streaming service that solves a wide range of data engineering problems with simple, chained, stateless processing steps. It implements transaction based resiliency with back pressure, so when connecting to at-least-once sources and sinks, it's able to guarantee at-least-once delivery without needing to persist messages during transit.
It's simple to deploy, comes with a wide range of connectors, and is totally data agnostic, making it easy to drop into your existing infrastructure. Redpanda Connect has functionality that overlaps with integration frameworks, log aggregators and ETL workflow engines, and can therefore be used to complement these traditional data engineering tools or act as a simpler alternative.
The Pinecone connector for Redpanda provides a production-ready integration from many existing data sources, all in a few lines of YAML.
{primaryLabel && primaryHref && }
{secondaryLabel && secondaryHref && }
;
Deploy and run Pinecone with Snowpark Container Services. Snowpark Container Services is a fully managed container offering designed to facilitate the deployment, management, and scaling of containerized applications within the Snowflake ecosystem. This service enables users to run containerized workloads directly within Snowflake, ensuring that data does't need to be moved out of the Snowflake environment for processing. Unlike traditional container orchestration platforms like Docker or Kubernetes, Snowpark Container Services offers an OCI runtime execution environment specifically optimized for Snowflake. This integration allows for the seamless execution of OCI images, leveraging Snowflak's robust data platform.
{primaryLabel && primaryHref && }
{secondaryLabel && secondaryHref && }
;
Founded by the original developers of Apache Pulsar and Apache BookKeeper, [StreamNative](https://streamnative.io) provides StreamNative Cloud, offering Apache Pulsar as a Service. The company also supports on-premise Pulsar deployments and related commercial support. StreamNative Cloud provides a scalable, resilient, and secure messaging and event streaming platform for enterprises. Additionally, StreamNative offers Kafka compatibility, enabling seamless integration with existing Kafka-based systems.
The Pinecone integration with StreamNative allows access to pinecone.io with a Pulsar topic. The sink connector takes in messages and writes them if they are in a proper format to a Pinecone index.
{primaryLabel && primaryHref && }
{secondaryLabel && secondaryHref && }
;
[Traceloop](https://www.traceloop.com/) provides observability for your LLM app using OpenTelemetry. Traceloop automatically monitors the quality of your LLM outputs. It helps you to debug and test changes to your models and prompts.
The Pinecone integration with Traceloop produces traces and metrics that can be viewed in any OpenTelemetry-based platform like Datadog, Grafana, Traceloop, and others.
{primaryLabel && primaryHref && }
{secondaryLabel && secondaryHref && }
;
TruLens is a powerful open source library for evaluating and tracking large language model-based applications. TruLens provides a set of tools for developing and monitoring neural nets, including large language models (LLMs). This includes both tools for evaluation of LLMs and LLM-based applications with TruLens-Eval and deep learning explainability with TruLens-Explain.
To build an effective RAG-style LLM application, it is important to experiment with various configuration choices while setting up your Pinecone vector database, and study their impact on performance metrics. Tracking and evaluation with TruLens enables fast iteration of your application.
In addition to the above, feedback functions also support the evaluation of ground truth agreement, sentiment, model agreement, language match, toxicity, and a full suite of moderation evaluations, including hate, violence and more. TruLens implements feedback functions as an extensible framework that can evaluate your custom needs as well.
During the development cycle, TruLens supports the iterative development of a wide range of LLM applications by wrapping your application to log cost, latency, key metadata and evaluations of each application run. This allows you to track and identify failure modes, pinpoint their root cause, and measure improvement across experiments.
### Why Pinecone?
Large language models alone have a hallucination problem. Several decades of machine learning research have optimized models, including modern LLMs, for generalization, while actively penalizing memorization. However, many of today's applications require factual, grounded answers. LLMs are also expensive to train, and provided by third party APIs. This means the knowledge of an LLM is fixed. Retrieval-augmented generation (RAG) is a way to reliably ensure models are grounded, with Pinecone as the curated source of real world information, long term memory, application domain knowledge, or whitelisted data.
In the RAG paradigm, rather than just passing a user question directly to a language model, the system retrieves any documents that could be relevant in answering the question from the knowledge base, and then passes those documents (along with the original question) to the language model to generate the final response. The most popular method for RAG involves chaining together LLMs with vector databases, such as the widely used Pinecone vector DB.
In this process, a numerical vector (an embedding) is calculated for all documents, and those vectors are then stored in a database optimized for storing and querying vectors. Incoming queries are vectorized as well, typically using an encoder LLM to convert the query into an embedding. The query embedding is then matched via embedding similarity against the document embeddings in the vector database to retrieve the documents that are relevant to the query.
Pinecone makes it easy to build high-performance vector search applications, including retrieval-augmented question answering. Pinecone can easily handle very large scales of hundreds of millions and even billions of vector embeddings. Pinecone's large scale allows it to handle long term memory or a large corpus of rich external and domain-appropriate data so that the LLM component of RAG application can focus on tasks like summarization, inference and planning. This setup is optimal for developing a non-hallucinatory application.\
In addition, Pinecone is fully managed, so it is easy to change configurations and components. Combined with the tracking and evaluation with TruLens, this is a powerful combination that enables fast iteration of your application.
### Using Pinecone and TruLens to improve LLM performance and reduce hallucination
To build an effective RAG-style LLM application, it is important to experiment with various configuration choices while setting up the vector database, and study their impact on performance metrics.
In this example, we explore the downstream impact of some of these configuration choices on response quality, cost and latency with a sample LLM application built with Pinecone as the vector DB. The evaluation and experiment tracking is done with the [TruLens](https://www.trulens.org/) open source library. TruLens offers an extensible set of [feedback functions](https://truera.com/ai-quality-education/generative-ai-and-llms/whats-missing-to-evaluate-foundation-models-at-scale/) to evaluate LLM apps and enables developers to easily track their LLM app experiments.
In each component of this application, different configuration choices can be made that can impact downstream performance. Some of these choices include the following:
**Constructing the Vector DB**
* Data preprocessing and selection
* Chunk Size and Chunk Overlap
* Index distance metric
* Selection of embeddings
**Retrieval**
* Amount of context retrieved (top k)
* Query planning
**LLM**
* Prompting
* Model choice
* Model parameters (size, temperature, frequency penalty, model retries, etc.)
These configuration choices are useful to keep in mind when constructing your app. In general, there is no optimal choice for all use cases. Rather, we recommend that you experiment with and evaluate a variety of configurations to find the optimal selection as you are building your application.
#### Creating the index in Pinecone
Here we'll download a pre-embedded dataset from the `pinecone-datasets` library allowing us to skip the embedding and preprocessing steps.
```Python Python
import pinecone_datasets
dataset = pinecone_datasets.load_dataset('wikipedia-simple-text-embedding-ada-002-100K')
dataset.head()
```
After downloading the data, we can initialize our pinecone environment and create our first index. Here, we have our first potentially important choice, by selecting the **distance metric** used for our index.
```Python Python
pinecone.create_index(
name=index_name_v1,
metric='cosine', # We'll try each distance metric here.
dimension=1536 # 1536 dim of text-embedding-ada-002.
)
```
Then, we can upsert our documents into the index in batches.
```Python Python
for batch in dataset.iter_documents(batch_size=100):
index.upsert(batch)
```
#### Build the vector store
Now that we've built our index, we can start using LangChain to initialize our vector store.
```Python Python
embed = OpenAIEmbeddings(
model='text-embedding-ada-002',
openai_api_key=OPENAI_API_KEY
)
from langchain.vectorstores import Pinecone
text_field = "text"
# Switch back to a normal index for LangChain.
index = pinecone.Index(index_name_v1)
vectorstore = Pinecone(
index, embed.embed_query, text_field
)
```
In RAG, we take the query as a question that is to be answered by an LLM, but the LLM must answer the question based on the information it receives from the `vectorstore`.
#### Initialize our RAG application
To do this, we initialize a `RetrievalQA` as our app:
```Python Python
from langchain.chat_models import ChatOpenAI
from langchain.chains import RetrievalQA
# completion llm
llm = ChatOpenAI(
model_name='gpt-3.5-turbo',
temperature=0.0
)
qa = RetrievalQA.from_chain_type(
llm=llm,
chain_type="stuff",
retriever=vectorstore.as_retriever()
)
```
#### TruLens for evaluation and tracking of LLM experiments
Once we've set up our app, we should put together our [feedback functions](https://truera.com/ai-quality-education/generative-ai-and-llms/whats-missing-to-evaluate-foundation-models-at-scale/). As a reminder, feedback functions are an extensible method for evaluating LLMs. Here we'll set up two feedback functions: `qs_relevance` and `qa_relevance`. They're defined as follows:
*QS Relevance: query-statement relevance is the average of relevance (0 to 1) for each context chunk returned by the semantic search.*
*QA Relevance: question-answer relevance is the relevance (again, 0 to 1) of the final answer to the original question.*
```Python Python
# Imports main tools for eval
from trulens_eval import TruChain, Feedback, Tru, feedback, Select
import numpy as np
tru = Tru()
# OpenAI as feedback provider
openai = feedback.OpenAI()
# Question/answer relevance between overall question and answer.
qa_relevance = Feedback(openai.relevance).on_input_output()
# Question/statement relevance between question and each context chunk.
qs_relevance =
Feedback(openai.qs_relevance).
on_input()
# See explanation below
.on(Select.Record.app.combine_documents_chain._call.args.inputs.input_documents[:].page_content)
.aggregate(np.mean)
```
Our use of selectors here also requires an explanation.
QA Relevance is the simpler of the two. Here, we are using `.on_input_output()` to specify that the feedback function should be applied on both the input and output of the application.
For QS Relevance, we use TruLens selectors to locate the context chunks retrieved by our application. Let's break it down into simple parts:
1. Argument Specification – The `on_input` which appears first is a convenient shorthand and states that the first argument to `qs_relevance` (the question) is to be the main input of the app.
2. Argument Specification – The `on(Select...)` line specifies where the statement argument to the implementation comes from. We want to evaluate the context chunks, which are an intermediate step of the LLM app. This form references the langchain app object call chain, which can be viewed from `tru.run_dashboard()`. This flexibility allows you to apply a feedback function to any intermediate step of your LLM app. Below is an example where TruLens displays how to select each piece of the context.
3. Aggregation specification -- The last line aggregate (`np.mean`) specifies how feedback outputs are to be aggregated. This only applies to cases where the argument specification names more than one value for an input or output.
The result of these lines is that `f_qs_relevance` can be now be run on apps/records and will automatically select the specified components of those apps/records
To finish up, we just wrap our Retrieval QA app with TruLens along with a list of the feedback functions we will use for eval.
```Python Python
# wrap with TruLens
truchain = TruChain(qa,
app_id='Chain1_WikipediaQA',
feedbacks=[qa_relevance, qs_relevance])
truchain(“Which state is Washington D.C. in?”)
```
After submitting a number of queries to our application, we can track our experiment and evaluations with the TruLens dashboard.
```Python Python
tru.run_dashboard()
```
Here is a view of our first experiment:
#### Experiment with distance metrics
Now that we've walked through the process of building our tracked RAG application using cosine as the distance metric, all we have to do for the next two experiments is to rebuild the index with `euclidean` or `dotproduct` as the metric and follow the rest of the steps above as is.
Because we are using OpenAI embeddings, which are normalized to length 1, dot product and cosine distance are equivalent - and Euclidean will also yield the same ranking. See the OpenAI docs for more information. With the same document ranking, we should not expect a difference in response quality, but computation latency may vary across the metrics. Indeed, OpenAI advises that dot product computation may be a bit faster than cosine. We will be able to confirm this expected latency difference with TruLens.
```Python Python
index_name_v2 = 'langchain-rag-euclidean'
pinecone.create_index(
name=index_name_v2,
metric='euclidean', # metric='dotproduct',
dimension=1536, # 1536 dim of text-embedding-ada-002
)
```
After doing so, we can view our evaluations for all three LLM apps sitting on top of the different indexes. All three apps are struggling with query-statement relevance. In other words, the context retrieved is only somewhat relevant to the original query.
**We can also see that both the Euclidean and dot-product metrics performed at a lower latency than cosine at roughly the same evaluation quality.**
### Problem: hallucination
Digging deeper into the Query Statement Relevance, we notice one problem in particular with a question about famous dental floss brands. The app responds correctly, but is not backed up by the context retrieved, which does not mention any specific brands.
#### Quickly evaluate app components with LangChain and TruLens
Using a less powerful model is a common way to reduce hallucination for some applications. We'll evaluate ada-001 in our next experiment for this purpose.
Changing different components of apps built with frameworks like LangChain is really easy. In this case we just need to call `text-ada-001` from the LangChain LLM store. Adding in easy evaluation with TruLens allows us to quickly iterate through different components to find our optimal app configuration.
```Python Python
# completion llm
from langchain.llms import OpenAI
llm = OpenAI(
model_name='text-ada-001',
temperature=0
)
from langchain.chains import RetrievalQAWithSourcesChain
qa_with_sources = RetrievalQA.from_chain_type(
llm=llm,
chain_type="stuff",
retriever=vectorstore.as_retriever()
)
# wrap with TruLens
truchain = TruChain(qa_with_sources,
app_id='Chain4_WikipediaQA',
feedbacks=[qa_relevance, qs_relevance])
```
**However, this configuration with a less powerful model struggles to return a relevant answer given the context provided.**
For example, when asked “Which year was Hawaii's state song written?”, the app retrieves context that contains the correct answer but fails to respond with that answer, instead simply responding with the name of the song.
While our relevance function is not doing a great job here in differentiating which context chunks are relevant, we can manually see that only the one (the 4th chunk) mentions the year the song was written. Narrowing our `top_k`, or the number of context chunks retrieved by the semantic search, may help.
We can do so as follows:
```Python Python
qa = RetrievalQA.from_chain_type(
llm=llm,
chain_type="stuff",
retriever=vectorstore.as_retriever(top_k = 1)
)
```
The way the `top_k` is implemented in LangChain's RetrievalQA is that the documents are still retrieved by semantic search and only the `top_k` are passed to the LLM. Therefore, TruLens also captures all of the context chunks that are being retrieved. In order to calculate an accurate QS Relevance metric that matches what's being passed to the LLM, we only calculate the relevance of the top context chunk retrieved by slicing the `input_documents` passed into the TruLens Select function:
```Python Python
qs_relevance = Feedback(openai.qs_relevance).on_input().on(
Select.Record.app.combine_documents_chain._call.args.inputs.input_documents[:1].page_content
).aggregate(np.mean)
```
Once we've done so, our final application has much improved `qs_relevance`, `qa_relevance` and latency!
With that change, our application is successfully retrieving the one piece of context it needs, and successfully forming an answer from that context.
Even better, the application now knows what it doesn't know:
### Summary
In conclusion, we note that exploring the downstream impact of some Pinecone configuration choices on response quality, cost and latency is an important part of the LLM app development process, ensuring that we make the choices that lead to the app performing the best. Overall, TruLens and Pinecone are the perfect combination for building reliable RAG-style applications. Pinecone provides a way to efficiently store and retrieve context used by LLM apps, and TruLens provides a way to track and evaluate each iteration of your application.
# Twelve Labs
Source: https://docs.pinecone.io/integrations/twelve-labs
export const PrimarySecondaryCTA = ({primaryLabel, primaryHref, primaryTarget, secondaryLabel, secondaryHref, secondaryTarget}) =>
{primaryLabel && primaryHref && }
{secondaryLabel && secondaryHref && }
;
[Twelve Labs](https://twelvelabs.io) is an AI company that provides state-of-the-art video understanding capabilities through its easy-to-use APIs. Our newly released product is the Embed API, which enables developers to create high-quality multimodal embeddings that capture the rich context and interactions between different modalities in videos, such as visual expressions, body language, spoken words, and overall context.
By integrating Twelve Labs' Embed API with Pinecone's vector database, developers can efficiently store, index, and retrieve these multimodal embeddings at scale. This integration empowers developers to build cutting-edge AI applications that leverage video data, such as video search, recommendation systems, content moderation, and more. Developers can seamlessly generate embeddings using Twelve Labs' API and store them in Pinecone for fast and accurate similarity search and retrieval.
The integration of Twelve Labs and Pinecone offers developers a powerful toolkit to process and understand video content in a more human-like manner. By combining Twelve Labs' video-native approach with Pinecone's purpose-built vector search capabilities, developers can unlock new possibilities and build innovative applications across various industries, including media and entertainment, e-commerce, education, and beyond.
{primaryLabel && primaryHref && }
{secondaryLabel && secondaryHref && }
;
Unstructured builds ETL tools for LLMs, including an open source Python library, a SaaS API, and an ETL platform. Unstructured extracts content and metadata from 25+ document types, including PDFs, Word documents and PowerPoints. After extracting content and metadata, Unstructured performs additional preprocessing steps for LLMs such as chunking. Unstructured maintains upstream connections to data sources such as SharePoint and Google drive, and downstream connections to databases such as Pinecone.
Integrating Pinecone with Unstructured enables developers to load data from an source or document type into Pinecone with a single click, accelerating the building of LLM apps that connect to organizational data.
{primaryLabel && primaryHref && }
{secondaryLabel && secondaryHref && }
;
Vercel is a platform for developers that provides the tools, workflows, and infrastructure you need to build and deploy your web apps faster, without the need for additional configuration. Vercel supports popular frontend frameworks out-of-the-box, and its scalable, secure infrastructure is globally distributed to serve content from data centers near your users for optimal speeds.
Pinecone provides the long-term memory for your Vercel AI projects. Using Pinecone with Vercel enables you to quickly set up and authenticate a connection to your Pinecone data/indexes, and then easily scale to support billions of data points. The integration is designed to be self-serve with strong defaults for a smooth setup, with optional advanced settings.
{primaryLabel && primaryHref && }
{secondaryLabel && secondaryHref && }
;
[VoltAgent](https://voltagent.dev) is a TypeScript-based, AI-agent framework for building production-ready applications with retrieval-augmented generation (RAG) capabilities. It supports two retrieval patterns: automatic search on every interaction, or LLM-decides-when-to-search, with built-in observability and tracking.
This integration connects VoltAgent agents to Pinecone's managed vector database, automatically generating embeddings with OpenAI's API. It provides semantic search with similarity scoring, source tracking, metadata filtering, and namespace organization, and it supports serverless deployment with automatic scaling.
The integration provides:
* A complete RAG setup with sample data
* Two pre-configured agent types
* Automatic index creation and population
* Source references and similarity scores
* Production-ready architecture
Use this integration to quickly build AI agents that can intelligently search and retrieve information from Pinecone vector databases, while maintaining full observability and control over the retrieval process.
{primaryLabel && primaryHref && }
{secondaryLabel && secondaryHref && }
;
[Voyage AI](https://www.voyageai.com) provides cutting-edge embedding and rerankers. Voyage AI's generalist [embedding models](https://docs.voyageai.com/docs/embeddings) continually top the [MTEB leaderboard](https://huggingface.co/spaces/mteb/leaderboard), and the [domain-specific embeddings](https://blog.voyageai.com/2024/01/23/voyage-code-2-elevate-your-code-retrieval/) enhance the retrieval quality for enterprise use cases significantly.
{primaryLabel && primaryHref && }
{secondaryLabel && secondaryHref && }
;
[Zapier](https://zapier.com/) lets you connect Pinecone with thousands of the most popular apps, so you can automate your work and have more time for what matters most — no code required.
With this integration, Pinecone can trigger workflows when your index status changes, and other apps can command Pinecone to perform actions. This means you can automatically add or remove data, run searches, or manage indexes based on events happening in your other tools.
For example, you might use this integration to automatically add spreadsheet entries to your Pinecone index, or build a bot that searches for answers and posts them directly to your team chat. It's all about weaving Pinecone's powerful search and data capabilities into your existing workflows, making everything work together automatically.
{modelName}
{company.toUpperCase()}
{description}
| Task | {task} |
| Modality | {modality} |
| Max Input Tokens | {maxInputTokens} |
| Price | {price} |
Preview
}
Inference
Build end-to-end faster with models hosted by Pinecone.
Request a model {
const carousel = e.target;
const prevButton = document.getElementById('prevButton');
const nextButton = document.getElementById('nextButton');
if (!prevButton || !nextButton) return; // Guard against null elements
// Check if we're at the start or end
const isAtStart = carousel.scrollLeft === 0;
const isAtEnd = Math.abs(carousel.scrollLeft + carousel.offsetWidth - carousel.scrollWidth) < 1;
// Update button visibility
prevButton.style.visibility = isAtStart ? 'hidden' : 'visible';
nextButton.style.visibility = isAtEnd ? 'hidden' : 'visible';
}}
>
### Launch week: Pinecone Local
Pinecone now offers Pinecone Local, an in-memory database emulator available as a Docker image. You can use Pinecone Local to [develop your applications locally](/guides/operations/local-development), or to [test your applications in CI/CD](/guides/production/automated-testing), without connecting to your Pinecone account, affecting production data, or incurring any usage or storage fees. Pinecone Local is in [public preview](/release-notes/feature-availability).
### Launch week: Dark mode
Dark mode is now out for Pinecone's website, docs, and console. You can change your theme at the top right of each site.
Pinecone Assistant is generally available (GA) for all users.
[Read more](https://www.pinecone.io/blog/pinecone-assistant-generally-available) about the release on our blog.
{title}
{text}
; }; export const ExampleCard = ({title, text, link, children, arrow, vectors, namespaces}) => { return{title}
{text}
{children &&{children}
}
{arrow && }
{(vectors || namespaces) &&
{vectors && {vectors} vectors}
{namespaces && {namespaces} namespaces}
}
;
};
- Indexes: Permissions to query, import, fetch, add, update, and delete index data.
- Pinecone Assistant: Permissions to add, list, view, and delete files; chat with an assistant, and evaluate responses.
- Pinecone Inference: Permissions to generate embeddings and rerank documents.
- Indexes: Permissions to query, fetch, list ID, and view stats.
- Pinecone Assistant: Permissions to list and view files, chat with an assistant, and evaluate responses.
- Indexes: Permissions to query, import, fetch, add, update, and delete index data.
- Pinecone Assistant: Permissions to add, list, view, and delete files; chat with an assistant, and evaluate responses.
- Pinecone Inference: Permissions to generate embeddings and rerank documents.
- Indexes: Permissions to query, fetch, list ID, and view stats.
- Pinecone Assistant: Permissions to list and view files, chat with an assistant, and evaluate responses.
Traffic is also encrypted in transit between the Pinecone backend and cloud infrastructure services, such as S3 and GCS. For more information, see [Google Cloud Platform](https://cloud.google.com/docs/security/encryption-in-transit) and [AWS security documentation](https://docs.aws.amazon.com/AmazonS3/userguide/UsingEncryption.html).
## Network security
### Proxies
The following Pinecone SDKs support the use of proxies:
* [Python SDK](/reference/python-sdk#proxy-configuration)
* [Node.js SDK](/reference/node-sdk#proxy-configuration)
# Upgrade your plan
Source: https://docs.pinecone.io/guides/assistant/admin/upgrade-billing-plan
This page describes how to upgrade from the free Starter plan to the [Standard or Enterprise plan](https://www.pinecone.io/pricing/), paying either with a credit/debit card or through a supported cloud marketplace.
## Data ingestion
When a [document is uploaded](/guides/assistant/manage-files), the assistant processes the content by chunking it into smaller parts and generating [vector embeddings](https://www.pinecone.io/learn/vector-embeddings-for-developers/) for each chunk. These embeddings are stored in an [index](/guides/index-data/indexing-overview), making them ready for retrieval.
## Data retrieval
During a [chat](/guides/assistant/chat-with-assistant), the assistant processes the message to formulate relevant search queries, which are used to query the index and identify the most relevant chunks from the uploaded content.
## Response generation
After retrieving these chunks, the assistant performs a ranking step to determine which information is most relevant. This [context](/guides/assistant/context-snippets-overview), along with the chat history and [assistant instructions](/guides/assistant/manage-assistants#add-instructions-to-an-assistant), is then used by a large language model (LLM) to generate responses that are informed by your documents.
# 2022 releases
Source: https://docs.pinecone.io/release-notes/2022
## December 22, 2022
#### Pinecone is now available in GCP Marketplace
You can now [sign up for Pinecone billing through Google Cloud Platform Marketplace](/guides/organizations/manage-billing/upgrade-billing-plan).
## December 6, 2022
#### Organizations are generally available
Pinecone now features [organizations](/guides/organizations/understanding-organizations), which allow one or more users to control billing and project settings across multiple projects owned by the same organization.
#### p2 pod type is generally available
The [p2 pod type](/guides/index-data/indexing-overview#p2-pods) is now generally available and ready for production workloads. p2 pods are now available in the Starter plan and support the [dotproduct distance metric](/guides/index-data/create-an-index#dotproduct).
#### Performance improvements
* [Bulk vector\_deletes](/guides/index-data/upsert-data/#deleting-vectors) are now up to 10x faster in many circumstances.
* [Creating collections](/guides/manage-data/back-up-an-index) is now faster.
## October 31, 2022
#### Hybrid search (Early access)
Pinecone now supports keyword-aware semantic search with the new hybrid search indexes and endpoints. Hybrid search enables improved relevance for semantic search results by combining them with keyword search.
This is an **early access** feature and is available only by [signing up](https://www.pinecone.io/hybrid-search-early-access/).
## October 17, 2022
#### Status page
The new [Pinecone Status Page](https://status.pinecone.io/) displays information about the status of the Pinecone service, including the status of individual cloud regions and a log of recent incidents.
## September 16, 2022
#### Public collections
You can now create indexes from public collections, which are collections containing public data from real-world data sources. Currently, public collections include the Glue - SSTB collection, the TREC Question classification collection, and the SQuAD collection.
## August 16, 2022
#### Collections (Public preview)("Beta")
You can now \[make static copies of your index]\(/guides/manage-data/back-up-an-index using collections]\(/guides/manage-data/back-up-an-index#pod-based-index-backups-using-collections). After you create a collection from an index, you can create a new index from that collection. The new index can use any pod type and any number of pods. Collections only consume storage.
This is a **public preview** feature and is not appropriate for production workloads.
#### Vertical scaling
You can now [change the size of the pods](/guides/indexes/pods/scale-pod-based-indexes#increase-pod-size) for a live index to accommodate more vectors or queries without interrupting reads or writes. The p1 and s1 pod types are now available in [4 different sizes](/guides/index-data/indexing-overview/#pods-pod-types-and-pod-sizes): `1x`, `2x`, `4x`, and `8x`. Capacity and compute per pod double with each size increment.
#### p2 pod type (Public preview)("Beta")
The new [p2 pod type](/guides/index-data/indexing-overview/#p2-pods) provides search speeds of around 5ms and throughput of 200 queries per second per replica, or approximately 10x faster speeds and higher throughput than the p1 pod type, depending on your data and network conditions.
This is a **public preview** feature and is not appropriate for production workloads.
#### Improved p1 and s1 performance
The [s1](/guides/index-data/indexing-overview/#s1-pods) and [p1](/guides/index-data/indexing-overview/#p1-pods) pod types now offer approximately 50% higher query throughput and 50% lower latency, depending on your workload.
## July 26, 2022
You can now specify a [metadata filter](/guides/index-data/indexing-overview#metadata/) to get results for a subset of the vectors in your index by calling [describe\_index\_stats](/reference/api/2024-07/control-plane/describe_index) with a [filter](/reference/api/2024-07/control-plane/describe_index#!path=filter\&t=request) object.
The `describe_index_stats` operation now uses the `POST` HTTP request type. The `filter` parameter is only accepted by `describe_index_stats` calls using the `POST` request type. Calls to `describe_index_stats` using the `GET` request type are now deprecated.
## July 12, 2022
#### Pinecone Console Guided Tour
You can now choose to follow a guided tour in the [Pinecone console](https://app.pinecone.io). This interactive tutorial walks you through creating your first index, upserting vectors, and querying your data. The purpose of the tour is to show you all the steps you need to start your first project in Pinecone.
## June 24, 2022
#### Updated response codes
The [create\_index](/reference/api/2024-07/control-plane/create_index), [delete\_index](/reference/api/2024-07/control-plane/delete_index), and `scale_index` operations now use more specific HTTP response codes that describe the type of operation that succeeded.
## June 7, 2022
#### Selective metadata indexing
You can now store more metadata and more unique metadata values! [Select which metadata fields you want to index for filtering](/guides/indexes/pods/manage-pod-based-indexes#selective-metadata-indexing) and which fields you only wish to store and retrieve. When you index metadata fields, you can filter vector search queries using those fields. When you store metadata fields without indexing them, you keep memory utilization low, especially when you have many unique metadata values, and therefore can fit more vectors per pod.
#### Single-vector queries
You can now [specify a single query vector using the vector input](/reference/api/2024-07/data-plane/query/#!path=vector\&t=request). We now encourage all users to query using a single vector rather than a batch of vectors, because batching queries can lead to long response messages and query times, and single queries execute just as fast on the server side.
#### Query by ID
You can now [query your Pinecone index using only the ID for another vector](/reference/api/2024-07/data-plane/query/#!path=id\&t=request). This is useful when you want to search for the nearest neighbors of a vector that is already stored in Pinecone.
#### Improved index fullness accuracy
The index fullness metric in [describe\_index\_stats()](/reference/api/2024-07/control-plane/describe_index#!c=200\&path=indexFullness\&t=response) results is now more accurate.
## April 25, 2022
#### Partial updates (Public preview)
You can now perform a partial update by ID and individual value pairs. This allows you to update individual metadata fields without having to upsert a matching vector or update all metadata fields at once.
#### New metrics
Users on all plans can now see metrics for the past one (1) week in the Pinecone console. Users on the Enterprise plan now have access to the following metrics via the [Prometheus metrics endpoint](/guides/production/monitoring/):
* `pinecone_vector_count`
* `pinecone_request_count_total`
* `pinecone_request_error_count_total`
* `pinecone_request_latency_seconds`
* `pinecone_index_fullness` (Public preview)
**Note:** The accuracy of the `pinecone_index_fullness` metric is improved. This may result in changes from historic reported values. This metric is in public preview.
#### Spark Connector
Spark users who want to manage parallel upserts into Pinecone can now use the [official Spark connector for Pinecone](https://github.com/pinecone-io/spark-pinecone#readme) to upsert their data from a Spark dataframe.
#### Support for Boolean and float metadata in Pinecone indexes
You can now add `Boolean` and `float64` values to [metadata JSON objects associated with a Pinecone index.](/guides/index-data/indexing-overview#metadata)
#### New state field in describe\_index results
The [describe\_index](/reference/api/2024-07/control-plane/describe_index/) operation results now contain a value for `state`, which describes the state of the index. The possible values for `state` are `Initializing`, `ScalingUp`, `ScalingDown`, `Terminating`, and `Ready`.
##### Delete by metadata filter
The [Delete](/reference/api/2024-07/data-plane/delete/) operation now supports filtering my metadata.
# 2023 releases
Source: https://docs.pinecone.io/release-notes/2023
## December 2023
### Features
* The free Starter plan now supports up to 100 namespaces. [Namespaces](/guides/index-data/indexing-overview#namespaces) let you partition vectors within an index to speed up queries or comply with [multitenancy](/guides/index-data/implement-multitenancy) requirements.
## November 2023
### Features
* The new [Pinecone AWS Reference Architecture](https://github.com/pinecone-io/aws-reference-architecture-pulumi/tree/main) is an open-source, distributed system that performs vector-database-enabled semantic search over Postgres records. You can use it as a learning resource or as a starting point for high-scale use cases.
### SDKs
* [Canopy](https://github.com/pinecone-io/canopy/blob/main/README.md) is a new open-source Retrieval Augmented Generation (RAG) framework and context engine built on top of Pinecone. It enables you to start chatting with your documents or text data with a few simple commands.\
The latest version of the Canopy SDK (v0.2.0) adds support for OpenAI SDK v1.2.3. See the [release notes](https://github.com/pinecone-io/canopy/releases/tag/V0.2.0) in GitHub for more details.
### Billing
* Pinecone is now registered to collect Value Added Tax (VAT) or Goods and Services Tax (GST) for accounts based in various global regions. If applicable, add your VAT or GST number to your account under **Settings > Billing**.
### October 2023
### Features
* [Collections](/guides/manage-data/back-up-an-index#pod-based-index-backups-using-collections) are now generally available (GA).
### Regions
* Pinecone Azure support via the [‘eastus-azure\` region](/guides/projects/understanding-projects#project-environments) is now generally available (GA).
### SDKs
* The latest version of our Node SDK is v1.1.2. See the [release notes](https://github.com/pinecone-io/pinecone-ts-client/releases/tag/v1.1.2) in GitHub for more details.
### Console
* The Index Browser is now available in the console. This allows you to preview, query, and filter by metadata directly from the console. The Index Browser can be found within the index detail page.
* We’re improved the design of our metrics page to include new charts for record and error count plus additional latencies (p90, p99) to help triage and understand issues.
### Integrations
* Knowledge Base for Amazon Bedrock is now available in private preview. Integrate your enterprise data via retrieval augmented generation (RAG) when building search and GenAI applications. [Learn more](https://www.pinecone.io/blog/amazon-bedrock-integration/).
* Pinecone Sink Connector for Confluent is now available in public preview. Gain access to data streams from across your business to build a real-time knowledge base for your AI applications. [Learn more](https://www.pinecone.io/confluent-integration).
### Billing
* You can now [sign up for Pinecone billing through Azure Marketplace](/guides/organizations/manage-billing/upgrade-billing-plan).
### Privacy
* Pinecone is now HIPAA compliant across all of our cloud providers (AWS, Azure, and GCP).
## September 11, 2023
Pinecone Azure support via the [eastus-azure region](/guides/projects/understanding-projects#project-environments) is now generally available (GA).
## August 14, 2023
Pinecone now supports deploying projects to Azure using the new [eastus-azure region](/guides/projects/understanding-projects#project-environments). This is a public preview environment, so test thoroughly before deploying to production.
## June 21, 2023
The new `gcp-starter` region is now in public preview. This region has distinct limitations from other Starter Plan regions. `gcp-starter` is the default region for some new users.
## April 26, 2023
[Indexes in the starter plan](/guides/index-data/indexing-overview#starter-plan) now support approximately 100,000 1536-dimensional embeddings with metadata. Capacity is proportional for other dimensionalities.
## April 3, 2023
Pinecone now supports [new US and EU cloud regions](/guides/projects/understanding-projects#project-environments).
## March 21, 2023
Pinecone now supports SSO on some Enterprise plans. [Contact Support](https://app.pinecone.io/organizations/-/settings/support) to set up your integration.
## March 1, 2023
Pinecone now supports [40kb of metadata per vector](/guides/index-data/indexing-overview#metadata#supported-metadata-size).
## February 22, 2023
#### Sparse-dense embeddings are now in public preview.
Pinecone now supports [vectors with sparse and dense values](/guides/search/hybrid-search#use-a-single-hybrid-index). To use sparse-dense embeddings in Python, upgrade to Python SDK version 2.2.0.
#### Pinecone Python SDK version 2.2.0 is available
Python SDK version 2.2.0 with support for sparse-dense embeddings is now available on [GitHub](https://github.com/pinecone-io/pinecone-python-client) and [PYPI](https://pypi.org/project/pinecone-client/2.2.0/).
## February 15, 2023
#### New Node.js SDK is now available in public preview
You can now try out our new [Node.js SDK for Pinecone](https://sdk.pinecone.io/typescript/).
## February 14, 2023
#### New usage reports in the Pinecone console
You can now monitor your current and projected Pinecone usage with the [**Usage** dashboard](/guides/manage-cost/monitor-usage-and-costs).
## January 31, 2023
#### Pinecone is now available in AWS Marketplace
You can now [sign up for Pinecone billing through Amazon Web Services Marketplace](/guides/organizations/manage-billing/upgrade-billing-plan).
## January 3, 2023
#### Pinecone Python SDK version 2.1.0 is now available on GitHub.
The [latest release of the Python SDK](https://github.com/pinecone-io/pinecone-python-client/releases/tag/2.1.0) makes the following changes:
* Fixes "Connection Reset by peer" error after long idle periods
* Adds typing and explicit names for arguments in all client operations
* Adds docstrings to all client operations
* Adds Support for batch upserts by passing `batch_size` to the upsert method
* Improves gRPC query results parsing performance
# 2024 releases
Source: https://docs.pinecone.io/release-notes/2024
## December 2024
### Launch week: Pinecone Local
Pinecone now offers Pinecone Local, an in-memory database emulator available as a Docker image. You can use Pinecone Local to [develop your applications locally](/guides/operations/local-development), or to [test your applications in CI/CD](/guides/production/automated-testing), without connecting to your Pinecone account, affecting production data, or incurring any usage or storage fees. Pinecone Local is in [public preview](/release-notes/feature-availability).
### Launch week: Dark mode
Dark mode is now out for Pinecone's website, docs, and console. You can change your theme at the top right of each site.
Pinecone Assistant is generally available (GA) for all users.
[Read more](https://www.pinecone.io/blog/pinecone-assistant-generally-available) about the release on our blog.
# Error: Cannot import name 'Pinecone' from 'pinecone'
Source: https://docs.pinecone.io/troubleshooting/error-cannot-import-name-pinecone
## Problem
When using an older version of the [Python SDK](https://github.com/pinecone-io/pinecone-python-client/blob/main/README.md) (earlier than 3.0.0), trying to import the `Pinecone` class raises the following error:
```console console
ImportError: cannot import name 'Pinecone' from 'pinecone'
```
## Solution
Upgrade the SDK version and try again:
```Shell Shell
# If you're interacting with Pinecone via HTTP requests, use:
pip install pinecone --upgrade
```
```Shell Shell
# If you're interacting with Pinecone via gRPC, use:
pip install "pinecone[grpc]" --upgrade
```
# Error: Handshake read failed when connecting
Source: https://docs.pinecone.io/troubleshooting/error-handshake-read-failed
## Problem
When trying to connect to Pinecone server, some users may receive an error message that says `Handshake read failed` and their connection attempt fails. This error can prevent them from running queries against their Pinecone indexes.
## Solution
If you encounter this error message, it means that your computer is not properly connecting with the Pinecone server. The error is often due to a misconfiguration of your Pinecone client or API key. Here is a recommended solution:
1. Make sure your firewall is not blocking any traffic and your internet connection is working fine. If you are unsure about how to do this, please consult your IT team.
2. Check that you have set up the Pinecone client and API key correctly. Double-check that you have followed the instructions in our [documentation](/guides/get-started/quickstart) correctly.
3. If you are still having issues, try creating a new index on Pinecone and populating it with data by running another script on your computer. This will verify that your computer can access the Pinecone servers for some tasks.
4. If the error persists, you may need to check your code for any misconfigurations. Make sure you are setting up your Pinecone client correctly and passing the right parameters when running queries against your indexes.
5. If you are still unable to resolve the issue, you can reach out to Pinecone support for assistance. They will be able to help you diagnose and resolve the issue.
## Conclusion
If you encounter the `Handshake read failed` error when trying to connect to Pinecone server, there are several steps you can take to resolve the issue. First, double-check that you have set up the Pinecone client and API key correctly. Then, check for any misconfigurations in your code. If the error persists, [contact Pinecone Support](/troubleshooting/contact-support) for assistance.
# Export indexes
Source: https://docs.pinecone.io/troubleshooting/export-indexes
Pinecone does not support an export function. It is on our roadmap for the future, however.
In the meantime, we recommend keeping a copy of your source data in case you need to move from one project to another, in which case you'll need to reindex the data.
For backup purposes, we recommend that you take periodic backups. Please see [Back up indexes](/guides/manage-data/back-up-an-index) in our documentation for more details on doing so.
# Serverless index creation error - max serverless indexes
Source: https://docs.pinecone.io/troubleshooting/index-creation-error-max-serverless
## Problem
Each project is limited to 20 serverless indexes. Trying to create more than 20 serverless indexes in a project raises the following `403 (FORBIDDEN)` error:
```console console
This project already contains 20 serverless indexes, the maximum per project.
Delete any unused indexes and try again, or create a new project for more serverless indexes.
For additional help, please contact support@pinecone.io.
```
## Solution
[Delete any unused serverless indexes](/guides/manage-data/manage-indexes#delete-an-index) in the project and try again, or create a new project to hold additional serverless indexes.
Also consider using [namespaces](/guides/index-data/indexing-overview#namespaces) to partition vectors of the same dimensionality within a single index. Namespaces can help speed up queries as well as comply with [multitenancy](/guides/index-data/implement-multitenancy) requirements.
# Index creation error - missing spec parameter
Source: https://docs.pinecone.io/troubleshooting/index-creation-error-missing-spec
## Problem
Using the [new API](/reference/api), creating an index requires passing appropriate values into the `spec` parameter. Without this `spec` parameter, the `create_index` method raises the following error:
```console console
TypeError: Pinecone.create_index() missing 1 required positional argument: 'spec'
```
## Solution
Set the `spec` parameter. For guidance on how to set this parameter, see [Create an index](/guides/index-data/create-an-index#create-a-serverless-index).
# Keep customer data separate in Pinecone
Source: https://docs.pinecone.io/troubleshooting/keep-customer-data-separate
Some use cases require vectors to be segmented by their customers, either physically or logically. The table below describes three techniques to accomplish this and the pros and cons of considering each:
| **Techniques** | **Pros** | **Cons** |
| ----------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- |
| **Separate Indexes**Each customer would have a separate index | • Customer data is truly separated physically by indexes | • You cannot query across customers if you wish
• Cost and maintenance of several indexes | | **Namespaces**
You can isolate data within a single index using namespaces | • You can only query one namespace at a time, which would keep customer data separate logically
• Cheaper than #1 potentially by making more efficient use of index storage | • You cannot query across namespaces
• Customer data is only separated logically | | **Metadata Filtering**
All data is kept in a single index and logically separated by filtering at query time | • Most versatile if you wish to query across customers
• As with namespaces cheaper than #1 potentially | • Customer data is separated only by filtering at query time | # Limitations of querying by ID Source: https://docs.pinecone.io/troubleshooting/limitations-of-querying-by-id When [querying by record ID](/guides/search/semantic-search#search-with-a-record-id), even with a high `topK`, the response is not guaranteed to include the record with the specified ID. ## Approximate nearest neighbor (ANN) Approximate nearest neighbor algorithms are designed to quickly find the closest matches to a given data point within large datasets with reasonable accuracy rather than perfect precision. Depending on the data, ANN may have a slightly lower accuracy than Known Nearest Neighbor (KNN) algorithms, but will have significantly lower read costs and latency than KNN. This is one of the key features of ANN. ANN algorithms assess broad data clusters to find matches. Some of these clusters might be ignored even if they contain relevant records simply because their overall similarity to the query is lower, because the algorithm aims to optimize the search by focusing on areas with a higher density of potential matches. See our learning center for more information on [ANN algorithms](https://www.pinecone.io/learn/a-developers-guide-to-ann-algorithms/). ## Recommendations ### Perform a fetch instead of a query Results from [Fetch](/guides/manage-data/fetch-data) are guaranteed to include the record with the specified ID. ### Use metadata filtering A [metadata filter](/guides/index-data/indexing-overview#metadata) in an ANN search effectively narrows the dataset to a more relevant subset, fine-tuning the search process. By explicitly excluding less relevant clusters from the outset, the search is performed among a group of records more closely related to the query, thereby increasing the efficiency and accuracy of the search. # Minimize latencies Source: https://docs.pinecone.io/troubleshooting/minimize-latencies There are many aspects to consider to minimize latencies: ## Slow uploads or high latencies To minimize latency when accessing Pinecone: * Switch to a cloud environment. For example: EC2, GCE, [Google Colab](https://colab.research.google.com), [GCP AI Platform Notebook](https://cloud.google.com/ai-platform-notebooks), or [SageMaker Notebook](https://docs.aws.amazon.com/sagemaker/dg/nbi.html). If you experience slow uploads or high query latencies, it might be because you are accessing Pinecone from your home network. * Consider deploying your application in the same environment as your Pinecone service. * See [Decrease latency](/guides/optimize/decrease-latency) for more tips. ## High query latencies with batching If you're batching queries, try reducing the number of queries per call to 1 query vector. You can make these [calls in parallel](/troubleshooting/parallel-queries) and expect roughly the same performance as with batching. # Python AttributeError: module pinecone has no attribute init Source: https://docs.pinecone.io/troubleshooting/module-pinecone-has-no-attribute-init ## Problem If you are using Pinecone serverless and getting the error `"AttributeError: module 'pinecone' has no attribute 'init'`, first check that you are using the latest version of the Python SDK. You can check the version of the client by running: ```shell pip show pinecone ``` ## Solution Serverless requires a minimum version of 3.0. To upgrade to the latest version, run: ```shell # If you're interacting with Pinecone via HTTP: pip install pinecone --upgrade # If you're using gRPC: # pip install "pinecone[grpc]" --upgrade ``` If you're on the right version and getting this error, you just have to make some slight changes to your code to make use of serverless. Instead of calling: ```python import pinecone pinecone.init(api_key=api_key,environment=environment) ``` Use the following if you're interacting with Pinecone via HTTP requests: ```python from pinecone import Pinecone pc = Pinecone(api_key=api_key) ``` Or, use the following if you're using gRPC: ```python from pinecone.grpc import PineconeGRPC as Pinecone pc = Pinecone(api_key=api_key) ``` You no longer need to specify the cloud environment your index is hosted in; the API key is all you need. # Node.js Troubleshooting Source: https://docs.pinecone.io/troubleshooting/nodejs-troubleshooting There could be several reasons why a [Node.js application](/reference/node-sdk) works in development mode but not in deployment. In order to troubleshoot the issue, it's important to identify where the application is failing and compare the development and deployment environments to see what differences exist. It's also important to review any error messages or logs that are generated to help identify the issue. You may also reach out to our [community of Pinecone users](https://community.pinecone.io) for help. Here are a few aspects to troubleshoot: ## Dependency version mismatch Sometimes, different environments have different versions of dependencies installed. If the application was developed using a specific version of a dependency, and that version is not installed on the deployment environment, the application may not work as expected. ## Environment configuration The development environment may have different configurations from the deployment environment. For example, the development environment may have different environment variables set or different network settings. If the application relies on specific configuration settings that are not present in the deployment environment, it may not work. ## Permissions The application may require permissions to access certain resources that are only granted in the development environment. For example, if the application needs to write to a specific directory, the permissions to write to that directory may only be granted in the development environment. ## Database connection If the application relies on a database connection, it's possible that the connection settings are different in the deployment environment. For example, the database may have a different hostname or port number. ## Code optimization During development, the application may have been running on a development server that did not optimize the code. However, when deployed, the application may be running on a production server that is optimized for performance. If there are code issues or performance bottlenecks, they may only appear when the application is deployed. ## Install fetch It may be necessary to install the `fetch` Python library for compatibility with node.js. # Non-indexed field filter issues Source: https://docs.pinecone.io/troubleshooting/non-indexed-field-filter-issues