OpenAI

OpenAI’s large language models (LLMs) enhance semantic search, or “long-term memory,” for LLMs. This combination uses OpenAI’s embedding and completion (or generation) endpoints alongside Pinecone’s vector search for information retrieval. By integrating OpenAI’s LLMs with Pinecone, you can combine deep learning capabilities for embedding generation with efficient vector storage and retrieval. This approach goes beyond traditional keyword-based search and returns context-aware results.

Setup guide

This guide shows how to integrate OpenAI’s LLMs with Pinecone, a combination referred to as the OP stack. LLMs like OpenAI’s text-embedding-3-small generate vector embeddings, i.e., numerical representations of text semantics. These embeddings enable semantic matches rather than literal text matches. Additionally, LLMs like gpt-4o-mini can predict text completions based on information provided from these contexts. Pinecone Database stores and queries high-dimensional vectors and provides fast semantic search over them. There are many ways to integrate these two tools, and Pinecone has several guides that focus on specific use cases. If you already know what you’d like to do, start with one of these:

Introduction to embeddings

At the core of the OP stack are embeddings, which come from the OpenAI Embedding API. You index those embeddings in Pinecone Database for fast, scalable retrieval augmentation of your LLMs and other information retrieval use cases. This example demonstrates the core OP stack. It’s the simplest workflow and is present in each of the other workflows, but it isn’t the only way to use the stack. For more advanced usage, see the guides listed earlier. The OP stack is built for semantic search, question-answering, threat-detection, and other applications that rely on language models and a large corpus of text data. The basic workflow looks like this:

Embed and index
- Use the OpenAI Embedding API to generate vector embeddings of your documents (or any text data).
- Upload those vector embeddings into Pinecone, which can store and index millions or billions of these vector embeddings and search through them at low latency.
Search
- Pass your query text or document through the OpenAI Embedding API again.
- Take the resulting vector embedding and send it as a query to Pinecone.
- Get back semantically similar documents, even if they don’t share any keywords with the query.

Set up the environment

Start by installing the OpenAI and Pinecone clients, along with Hugging Face Datasets for downloading the TREC dataset used in this guide.

Shell

pip install -qU \
    pinecone==10.0.0 \
    openai==3.26.0 \
    datasets==5.1.0

Create embeddings

To create embeddings, first initialize your connection to OpenAI. Sign up for an API key at OpenAI.

Python

from openai import OpenAI

client = OpenAI(
    api_key="OPENAI_API_KEY"
)  # get API key from platform.openai.com

You can now create embeddings with the OpenAI v3 small embedding model.

Python

MODEL = "text-embedding-3-small"

res = client.embeddings.create(
    input=[
        "Sample document text goes here",
        "there will be several phrases in each batch"
    ], model=MODEL
)

In res, you should find a JSON-like object containing two 1536-dimensional embeddings. These are the vector representations of the two inputs. To access the embeddings directly, extract them to a list.

Python

# we can extract embeddings to a list
embeds = [record.embedding for record in res.data]
len(embeds)

You’ll use this logic later when you create embeddings for the Text REtrieval Conference (TREC) question classification dataset.

Create an index

Next, initialize an index to store the vector embeddings. For this, you need a Pinecone API key. Sign up for one.

Python

import time
from pinecone.grpc import PineconeGRPC as Pinecone
from pinecone import ServerlessSpec

pc = Pinecone(api_key="...")

spec = ServerlessSpec(cloud="aws", region="us-east-1")

index_name = 'semantic-search-openai'

# check if index already exists (it shouldn't if this is your first run)
if not pc.has_index(index_name):
    # if does not exist, create index
    pc.create_index(
        index_name,
        dimension=len(embeds[0]),  # dimensionality of text-embedding-3-small
        metric='dotproduct',
        spec=spec
    )

# connect to index
index = pc.Index(index_name)
time.sleep(1)
# view index stats
index.describe_index_stats()

Store the embeddings

With both OpenAI and Pinecone connections initialized, you can populate the index. For this, you need the TREC dataset.

Python

from datasets import load_dataset

# load the first 1K rows of the TREC dataset
trec = load_dataset('CogComp/trec', revision='refs/convert/parquet', split='train[:1000]')

Then create a vector embedding for each question using OpenAI (as demonstrated earlier), and upsert the ID, vector embedding, and original text for each phrase to Pinecone.

Python

from tqdm.auto import tqdm

batch_size = 32  # process everything in batches of 32
for i in tqdm(range(0, len(trec['text']), batch_size)):
    # set end position of batch
    i_end = min(i+batch_size, len(trec['text']))
    # get batch of lines and IDs
    lines_batch = trec['text'][i: i+batch_size]
    ids_batch = [str(n) for n in range(i, i_end)]
    # create embeddings
    res = client.embeddings.create(input=lines_batch, model=MODEL)
    embeds = [record.embedding for record in res.data]
    # prep metadata and upsert batch
    meta = [{'text': line} for line in lines_batch]
    to_upsert = zip(ids_batch, embeds, meta)
    # upsert to Pinecone
    index.upsert(vectors=list(to_upsert))

Run a semantic search

With your data indexed, you’re ready to perform searches. This follows a similar process to indexing. Start with a text query that you want to use to find similar sentences. As before, encode it with text-embedding-3-small to create a query vector, xq. Then use xq to query the Pinecone index.

Python

query = "What caused the 1929 Great Depression?"

xq = client.embeddings.create(input=query, model=MODEL).data[0].embedding

Now query the index.

Python

res = index.query(vector=xq, top_k=5, include_metadata=True)

The response from Pinecone includes your original text in the metadata field. Print the top_k most similar questions and their similarity scores.

Python

for match in res['matches']:
    print(f"{match['score']:.2f}: {match['metadata']['text']}")

Response

75: Why did the world enter a global depression in 1929 ?
60: When was `` the Great Depression '' ?
37: What crop failure caused the Irish Famine ?
32: What were popular songs and types of songs in the 1920s ?
32: When did World War I start ?

The top results are relevant. To make the search harder, replace “depression” with the incorrect term “recession.”

Python

query = "What was the cause of the major recession in the early 20th century?"

# create the query embedding
xq = client.embeddings.create(input=query, model=MODEL).data[0].embedding

# query, returning the top 5 most similar results
res = index.query(vector=xq, top_k=5, include_metadata=True)

for match in res['matches']:
    print(f"{match['score']:.2f}: {match['metadata']['text']}")

Response

63: Why did the world enter a global depression in 1929 ?
55: When was `` the Great Depression '' ?
34: What were popular songs and types of songs in the 1920s ?
33: What crop failure caused the Irish Famine ?
29: What is considered the costliest disaster the insurance industry has ever faced ?

Finally, search using the definition of depression rather than the word or related words.

Python

query = "Why was there a long-term economic downturn in the early 20th century?"

# create the query embedding
xq = client.embeddings.create(input=query, model=MODEL).data[0].embedding

# query, returning the top 5 most similar results
res = index.query(vector=xq, top_k=5, include_metadata=True)

for match in res['matches']:
    print(f"{match['score']:.2f}: {match['metadata']['text']}")

Response

62: Why did the world enter a global depression in 1929 ?
54: When was `` the Great Depression '' ?
34: What were popular songs and types of songs in the 1920s ?
33: What crop failure caused the Irish Famine ?
32: What do economists do ?

This example shows that the semantic search pipeline can identify the meaning behind each of your queries. Using these embeddings with Pinecone lets you return the most semantically similar questions from the already indexed TREC dataset.

Clean up

When you’re finished with the index, delete it to save resources.

Python

pc.delete_index(name=index_name)

Connect an integration

Build an integration

Setup guide

Introduction to embeddings

Set up the environment

Create embeddings

Create an index

Store the embeddings

Run a semantic search

Clean up

Resources

​Setup guide

​Introduction to embeddings

Set up the environment

Create embeddings

Create an index

Store the embeddings

Run a semantic search

Clean up

​Resources

Setup guide

Introduction to embeddings

Resources