Using OpenAI and Pinecone to combine deep learning capabilities for embedding generation with efficient vector storage and retrieval
text-embedding-ada-002
generate vector embeddings, i.e., numerical representations of text semantics. These embeddings facilitate semantic-based rather than literal textual matches. Additionally, LLMs like gpt-4
or gpt-3.5-turbo
can predict text completions based on information provided from these contexts.
Pinecone is a vector database designed for storing and querying high-dimensional vectors. It provides fast, efficient semantic search over these vector embeddings.
By integrating OpenAI’s LLMs with Pinecone, we combine deep learning capabilities for embedding generation with efficient vector storage and retrieval. This approach surpasses traditional keyword-based search, offering contextually-aware, precise results.
There are many ways of integrating these two tools and we have several guides focusing on specific use-cases. If you already know what you’d like to do you can jump to these specific materials:
res
we should find a JSON-like object containing two 1536-dimensional embeddings, these are the vector representations of the two inputs provided above. To access the embeddings directly we can write:
upsert
the ID, vector embedding, and original text for each phrase to Pinecone.
query
, that we would like to use to find similar sentences. As before we encode this with OpenAI’s text similarity Babbage model to create a query vector xq
. We then use xq
to query the Pinecone index.
metadata
field, let’s print out the top_k
most similar questions and their respective similarity scores.