Now we need a place to store these embeddings and enable a efficient vector search through them all. To do that we use Pinecone, we can get a free API key and enter it below where we will initialize our connection to Pinecone and create a new index.

import os
import pinecone

# initialize connection to pinecone (get API key at app.pinecone.io)
api_key = os.getenv("PINECONE_API_KEY") or "PINECONE_API_KEY"
# find your environment next to the api key in pinecone console
env = os.getenv("PINECONE_ENVIRONMENT") or "PINECONE_ENVIRONMENT"

pinecone.init(api_key=api_key, enviroment=env)
pinecone.whoami()

/usr/local/lib/python3.10/dist-packages/pinecone/index.py:4: TqdmExperimentalWarning: Using `tqdm.autonotebook.tqdm` in notebook mode. Use `tqdm.tqdm` instead to force console mode (e.g. in jupyter console)
  from tqdm.autonotebook import tqdm





WhoAmIResponse(username='c78f2bd', user_label='default', projectname='9a4fbb6')

index_name = 'gen-qa-openai-fast'

# check if index already exists (it shouldn't if this is first time)
if index_name not in pinecone.list_indexes():
    # if does not exist, create index
    pinecone.create_index(
        index_name,
        dimension=1536,  # dimensionality of text-embedding-ada-002
        metric='cosine',
        metadata_config={'indexed': ['channel_id', 'published']}
    )
# connect to index
index = pinecone.GRPCIndex(index_name)
# view index stats
index.describe_index_stats()

{'dimension': 1536,
 'index_fullness': 0.0,
 'namespaces': {},
 'total_vector_count': 0}

We can see the index is currently empty with a total_vector_count of 0. We can begin populating it with OpenAI text-embedding-ada-002 built embeddings like so:

index.upsert_from_dataframe(dataset.documents, batch_size=100)

sending upsert requests:   0%|          | 0/38950 [00:00<?, ?it/s]



collecting async responses:   0%|          | 0/390 [00:00<?, ?it/s]





upserted_count: 38950

Now we've added all of our langchain docs to the index. With that we can move on to retrieval and then answer generation.

Retrieval

To search through our documents we first need to create a query vector xq. Using xq we will retrieve the most relevant chunks from the LangChain docs. To create that query vector we must initialize a text-embedding-ada-002 embedding model with OpenAI. For this, you need an OpenAI API key.

import openai

# get api key from platform.openai.com
openai.api_key = os.getenv('OPENAI_API_KEY') or 'OPENAI_API_KEY'

embed_model = "text-embedding-ada-002"

query = (
    "Which training method should I use for sentence transformers when " +
    "I only have pairs of related sentences?"
)

res = openai.Embedding.create(
    input=[query],
    engine=embed_model
)

# retrieve from Pinecone
xq = res['data'][0]['embedding']

# get relevant contexts (including the questions)
res = index.query(xq, top_k=2, include_metadata=True)

res

{'matches': [{'id': 'pNvujJ1XyeQ-t418.88',
              'metadata': {'channel_id': 'UCv83tO5cePwHMt1952IVVHw',
                           'end': 568.0,
                           'published': '2021-11-24 16:24:24 UTC',
                           'start': 418.0,
                           'text': 'pairs of related sentences you can go '
                                   'ahead and actually try training or '
                                   'fine-tuning using NLI with multiple '
                                   "negative ranking loss. If you don't have "
                                   'that fine. Another option is that you have '
                                   'a semantic textual similarity data set or '
                                   'STS and what this is is you have so you '
                                   'have sentence A here, sentence B here and '
                                   'then you have a score from from 0 to 1 '
                                   'that tells you the similarity between '
                                   'those two scores and you would train this '
                                   'using something like cosine similarity '
                                   "loss. Now if that's not an option and your "
                                   'focus or use case is on building a '
                                   'sentence transformer for another language '
                                   'where there is no current sentence '
                                   'transformer you can use multilingual '
                                   'parallel data. So what I mean by that is '
                                   'so parallel data just means translation '
                                   'pairs so if you have for example a English '
                                   'sentence and then you have another '
                                   'language here so it can it can be anything '
                                   "I'm just going to put XX and that XX is "
                                   'your target language you can fine-tune a '
                                   'model using something called multilingual '
                                   'knowledge distillation and what that does '
                                   'is takes a monolingual model for example '
                                   'in English and using those translation '
                                   'pairs it distills the knowledge the '
                                   'semantic similarity knowledge from that '
                                   'monolingual English model into a '
                                   'multilingual model which can handle both '
                                   'English and your target language. So '
                                   "they're three options quite popular very "
                                   'common that you can go for and as a '
                                   'supervised methods the chances are that '
                                   'probably going to outperform anything you '
                                   'do with unsupervised training at least for '
                                   'now. So if none of those sound like '
                                   'something',
                           'title': 'Today Unsupervised Sentence Transformers, '
                                    'Tomorrow Skynet (how TSDAE works)',
                           'url': 'https://youtu.be/pNvujJ1XyeQ'},
              'score': 0.8653299,
              'sparse_values': {'indices': [], 'values': []},
              'values': []},
             {'id': 'WS1uVMGhlWQ-t747.92',
              'metadata': {'channel_id': 'UCv83tO5cePwHMt1952IVVHw',
                           'end': 906.0,
                           'published': '2021-10-20 17:06:20 UTC',
                           'start': 747.0,
                           'text': "pooling approach. Or we can't use it in "
                                   'its current form. Now the solution to this '
                                   'problem was introduced by two people in '
                                   '2019 Nils Reimers and Irenia Gurevich. '
                                   'They introduced what is the first sentence '
                                   'transformer or sentence BERT. And it was '
                                   'found that sentence BERT or S BERT '
                                   'outformed all of the previous Save the Art '
                                   'models on pretty much all benchmarks. Not '
                                   'all of them but most of them. And it did '
                                   'it in a very quick time. So if we compare '
                                   'it to BERT, if we wanted to find the most '
                                   'similar sentence pair from 10,000 '
                                   'sentences in that 2019 paper they found '
                                   'that with BERT that took 65 hours. With S '
                                   'BERT embeddings they could create all the '
                                   'embeddings in just around five seconds. '
                                   'And then they could compare all those with '
                                   "cosine similarity in 0.01 seconds. So it's "
                                   'a lot faster. We go from 65 hours to just '
                                   'over five seconds which is I think pretty '
                                   "incredible. Now I think that's pretty much "
                                   'all the context we need behind sentence '
                                   'transformers. And what we do now is dive '
                                   'into a little bit of how they actually '
                                   'work. Now we said before we have the core '
                                   'transform models and what S BERT does is '
                                   'fine tunes on sentence pairs using what is '
                                   'called a Siamese architecture or Siamese '
                                   'network. What we mean by a Siamese network '
                                   'is that we have what we can see, what can '
                                   'view as two BERT models that are identical '
                                   'and the weights between those two models '
                                   'are tied. Now in reality when implementing '
                                   'this we just use a single BERT model. And '
                                   'what we do is we process one sentence, a '
                                   'sentence A through the model and then we '
                                   'process another sentence, sentence B '
                                   "through the model. And that's the sentence "
                                   'pair. So with our cross-linked we were '
                                   'processing the sentence pair together. We '
                                   'were putting them both together, '
                                   'processing them all at once. This time we '
                                   'process them separately. And during '
                                   'training what happens is the weights '
                                   'within BERT are optimized to reduce the '
                                   'difference between two vector embeddings '
                                   'or two sentence',
                           'title': 'Intro to Sentence Embeddings with '
                                    'Transformers',
                           'url': 'https://youtu.be/WS1uVMGhlWQ'},
              'score': 0.864513,
              'sparse_values': {'indices': [], 'values': []},
              'values': []}],
 'namespace': ''}

We write some functions to handle the retrieval and completion steps:

limit = 3750

def retrieve(query):
    res = openai.Embedding.create(
        input=[query],
        engine=embed_model
    )

    # retrieve from Pinecone
    xq = res['data'][0]['embedding']

    # get relevant contexts
    res = index.query(xq, top_k=3, include_metadata=True)
    contexts = [
        x['metadata']['text'] for x in res['matches']
    ]

    # build our prompt with the retrieved contexts included
    prompt_start = (
        "Answer the question based on the context below.\n\n"+
        "Context:\n"
    )
    prompt_end = (
        f"\n\nQuestion: {query}\nAnswer:"
    )
    # append contexts until hitting limit
    for i in range(1, len(contexts)):
        if len("\n\n---\n\n".join(contexts[:i])) >= limit:
            prompt = (
                prompt_start +
                "\n\n---\n\n".join(contexts[:i-1]) +
                prompt_end
            )
            break
        elif i == len(contexts)-1:
            prompt = (
                prompt_start +
                "\n\n---\n\n".join(contexts) +
                prompt_end
            )
    return prompt


def complete(prompt):
    # query text-davinci-003
    res = openai.Completion.create(
        engine='text-davinci-003',
        prompt=prompt,
        temperature=0,
        max_tokens=400,
        top_p=1,
        frequency_penalty=0,
        presence_penalty=0,
        stop=None
    )
    return res['choices'][0]['text'].strip()

# first we retrieve relevant items from Pinecone
query_with_contexts = retrieve(query)
query_with_contexts

"Answer the question based on the context below.\n\nContext:\npairs of related sentences you can go ahead and actually try training or fine-tuning using NLI with multiple negative ranking loss. If you don't have that fine. Another option is that you have a semantic textual similarity data set or STS and what this is is you have so you have sentence A here, sentence B here and then you have a score from from 0 to 1 that tells you the similarity between those two scores and you would train this using something like cosine similarity loss. Now if that's not an option and your focus or use case is on building a sentence transformer for another language where there is no current sentence transformer you can use multilingual parallel data. So what I mean by that is so parallel data just means translation pairs so if you have for example a English sentence and then you have another language here so it can it can be anything I'm just going to put XX and that XX is your target language you can fine-tune a model using something called multilingual knowledge distillation and what that does is takes a monolingual model for example in English and using those translation pairs it distills the knowledge the semantic similarity knowledge from that monolingual English model into a multilingual model which can handle both English and your target language. So they're three options quite popular very common that you can go for and as a supervised methods the chances are that probably going to outperform anything you do with unsupervised training at least for now. So if none of those sound like something\n\n---\n\npooling approach. Or we can't use it in its current form. Now the solution to this problem was introduced by two people in 2019 Nils Reimers and Irenia Gurevich. They introduced what is the first sentence transformer or sentence BERT. And it was found that sentence BERT or S BERT outformed all of the previous Save the Art models on pretty much all benchmarks. Not all of them but most of them. And it did it in a very quick time. So if we compare it to BERT, if we wanted to find the most similar sentence pair from 10,000 sentences in that 2019 paper they found that with BERT that took 65 hours. With S BERT embeddings they could create all the embeddings in just around five seconds. And then they could compare all those with cosine similarity in 0.01 seconds. So it's a lot faster. We go from 65 hours to just over five seconds which is I think pretty incredible. Now I think that's pretty much all the context we need behind sentence transformers. And what we do now is dive into a little bit of how they actually work. Now we said before we have the core transform models and what S BERT does is fine tunes on sentence pairs using what is called a Siamese architecture or Siamese network. What we mean by a Siamese network is that we have what we can see, what can view as two BERT models that are identical and the weights between those two models are tied. Now in reality when implementing this we just use a single BERT model. And what we do is we process one sentence, a sentence A through the model and then we process another sentence, sentence B through the model. And that's the sentence pair. So with our cross-linked we were processing the sentence pair together. We were putting them both together, processing them all at once. This time we process them separately. And during training what happens is the weights within BERT are optimized to reduce the difference between two vector embeddings or two sentence\n\n---\n\nstraight into it and take a look at where we might want to use this training approach and and how we can actually implement it. So the first question we need to ask is do we really need to resort to unsupervised training? Now what we're going to do here is just have a look at a few of the most popular training approaches and what sort of data we need for that. So the first one we're looking at here is Natural Language Inference or NLI and NLI requires that we have pairs of sentences that are labeled as either contradictory, neutral which means they're not necessarily related or as entailing or as inferring each other. So you have pairs that entail each other so they are both very similar pairs that are neutral and also pairs that are contradictory. And this is the traditional NLI data. Now using another version of fine-tuning with NLI called a multiple negatives ranking loss you can get by with only entailment pairs so pairs that are related to each other or positive pairs and it can also use contradictory pairs to improve the performance of training as well but you don't need it. So if you have positive pairs of related sentences you can go ahead and actually try training or fine-tuning using NLI with multiple negative ranking loss. If you don't have that fine. Another option is that you have a semantic textual similarity data set or STS and what this is is you have so you have sentence A here, sentence B here and then you have a score from from 0 to 1 that tells you the similarity\n\nQuestion: Which training method should I use for sentence transformers when I only have pairs of related sentences?\nAnswer:"

# then we complete the context-infused query
complete(query_with_contexts)

'NLI with multiple negative ranking loss.'

And we get a pretty great answer straight away, specifying to use multiple-rankings loss (also called multiple negatives ranking loss).

Once we're done with the index we delete it to save resources:

pinecone.delete_index(index_name)