Skip to main content
Pinecone Context Engine is Pinecone’s next-generation solution for building scalable, high-quality retrieval systems for AI applications. It aims to simplify document ingestion and retrieval by treating JSON documents as the core primitive, eliminating the complexity of managing fragmented chunks across multiple systems.

​
Alpha release

Context Engine is currently in alpha, under active development. Features and behavior may change significantly before general availability. Intended for user feedback only.
The alpha release does not include the full capabilities of Context Engine. It focuses exclusively on the document repository functionality, providing:
  • Document-centric storage: Store and manage complete JSON documents with structured schemas.
  • Automatic chunking and embedding: Seamlessly chunk and embed your documents with hybrid embedding (dense and sparse).
  • Basic retrieval capabilities: Search across your documents using semantic search with filtering.
  • Document management: Full CRUD operations for documents and repositories.
The advanced query planning, intelligent query rewriting, and sophisticated context expansion strategies that define Context Engine’s full vision will be released later.

​
Key advantages

Context Engine addresses the core challenges developers face when building retrieval-augmented generation (RAG) and agentic AI applications:
  • Eliminates complex document management: Traditional approaches require maintaining relationships between chunks and their source documents across multiple systems. Context Engine stores and manages complete JSON documents in a single system.
  • Solves context expansion complexity: Balancing chunk size limitations with LLM context needs is handled automatically through built-in chunk expansion.
  • Simplifies infrastructure: Rather than managing separate embedding pipelines and synchronization between systems, everything operates at the logical document level on Pinecone’s vector database technology.
  • (Future functionality) Removes query formulation guesswork: Instead of struggling to determine the best query strategy, the system leverages repository schema to automatically determine optimal approaches.

​
Use cases

Context Engine is ideal for applications that work with structured JSON documents containing multiple searchable fields:
  • Product catalogs: Search across product descriptions, specifications, reviews, and metadata
  • Customer feedback analysis: Query support tickets, reviews, and feedback with both semantic and structured filtering
  • Document repositories: Maintain original document structure while enabling powerful search capabilities
  • Knowledge bases: Corporate documentation, policies, and procedures requiring both semantic and categorical search
  • Research databases: Academic papers, reports, and studies with rich metadata and content

​
How it works

Context Engine operates on three key components:

​
Repository with schema

A repository is a top-level container that stores your JSON documents and defines how they should be indexed. When you create a repository, you specify a schema that determines:
  • Which fields should be embedded (searchable via semantic similarity).
  • Which fields are not filterable (fields are filterable by default for exact matches and ranges).
  • Field types and constraints.
The schema uses sensible defaults:
  • All fields are filterable=true by default. Set filterable=false only for fields with large content (like binary data or very long text) to save metadata space and reduce costs.
  • Uses hybrid embedding with both dense (llama-text-embed-v2) and sparse (pinecone-sparse-english-v0) models.
Field names must be alphanumeric (A-Z, a-z, 0-9, _) and must not start with a number or underscore. Unicode field names may be supported in future releases.

​
Document ingestion and chunking

When you upsert a JSON document, Context Engine:
  1. Validates the document against your schema.
  2. Chunks fields according to the schema.
  3. Embeds each chunk using the configured embedding models.
  4. Indexes the chunks in the underlying vector database.
  5. Maintains relationships between chunks and their parent document.
Each chunk knows its position within the document and can reference adjacent chunks for context expansion.

​
Intelligent search and retrieval

The advanced retrieval functionality described here (query planning, intelligent query rewriting, and sophisticated context expansion strategies) will be released later. In the alpha release, hybrid semantic and lexical search is performed automatically, and the system uses a simple chunk expansion strategy.
When you search a repository, Context Engine:
  1. Analyzes your query to determine the best retrieval strategy.
  2. Executes semantic, filtered, or hybrid queries as appropriate.
  3. Retrieves matching chunks from the vector database.
  4. Expands context by including neighboring chunks when beneficial.
  5. Groups results by document and presents them with full context.
The search API returns complete document metadata along with the specific matching segments, making it easy to understand both what matched and why.
⌘I