## Built with
* Pinecone Serverless
* Voyage Embeddings
* Langchain
* Next.js + tailwind
* Node version 20 or higher
***
## Run the sample app
The fastest way to get started is to use the `create-pinecone-app` CLI tool to get up and running:
```bash theme={null}
npx -y create-pinecone-app@latest --template legal-semantic-search
```
### Get your API key
You need an API key to make API calls to your Pinecone project:
Then copy your generated key:
```
PINECONE_API_KEY="{{YOUR_API_KEY}}"
# This API key has ReadWrite access to all indexes in your project.
```
Alternatively, follow [these steps](/guides/projects/manage-api-keys#create-an-api-key):
1. Open the [Pinecone console](https://app.pinecone.io/).
2. Select your project.
3. Go to **API Keys**.
4. Create an API key.
5. Copy your API key.
### Get your Voyage AI API key
1. Create a new [Voyage AI](https://www.voyageai.com/) account [here](https://dash.voyageai.com/).
2. Create a [new API key](https://dashboard.voyageai.com/organization/api-keys).
3. Add your billing information to your Voyage AI account [here](https://dashboard.voyageai.com/organization/billing). This is required even to use the free tier.
4. Copy your API key.
### Create a Pinecone serverless index
Create a Pinecone index for this project.
The index should have the following properties:
* **dimension**: `1024`
The Voyage `voyage-law-2` embeddings model has 1024 dimensions.
* **metric**: `cosine`
* **region**: `us-east-1`
You can create the index [in the console](https://app.pinecone.io/organizations/-/projects/-/create-index/serverless),
or by following the instructions [here](https://docs.pinecone.io/guides/get-started/quickstart#4-create-a-serverless-index).
### Start the project
**Requires Node version 20+**
#### Dependency installation
From the project root directory, run the following command.
```bash theme={null}
cd legal-semantic-search && npm install
```
Make sure you have populated the client `.env` with relevant keys.
```bash theme={null}
# You must first activate a Billing Account here: https://www.voyageai.com/
# Then get your Voyage API Key here: https://dash.voyageai.com/
VOYAGE_API_KEY="your-api-key-here"
# Get your Pinecone API key here: https://app.pinecone.io/
PINECONE_API_KEY="your-api-key-here"
PINECONE_INDEX="legal-semantic-search"
```
Start the app.
```bash theme={null}
npm run dev
```
## Project structure
In this example we opted to use a standard Next.js application structure.
**Frontend Client**
The frontend uses Next.js, tailwind and custom React components to power the search experience. It also leverages API routes to make calls to the server to initiate bootstrapping of the Pinecone vector database as a knowledge store, and to fetch relevant document chunks for the UI.
**Backend Server**
This project uses Next.js API routes to handle file chunking, upsertion, and context provision etc. Learn more about the implementation details below.
***
### Simple semantic search
This project uses a basic semantic search architecture that achieves low latency natural language search across all embedded documents. When the app is loaded, it performs background checks to determine if the Pinecone vector database needs to be created and populated.
**Componentized suggested search interface**
To make it easier for you to clone this app as a starting point and quickly adopt it to your own purposes, we've
built the search interface as a component that accepts a list of suggested searches and renders them as a dropdown, helping the
user find things:
You can define your suggested searches in your parent component:
```typescript theme={null}
// For the purposes of our legal semantic search example, we pre-define some queries
// that we know will pull back interesting results for the user
const suggestedSearches = [
'Cases about personal freedoms being violated',
'Cases involving a US President',
'Cases involving guns',
'Cases where Nixon was the defendant',
'How much power does the commerce clause give Congress?',
'Cases about personal rights or congressional overreach?',
'Cases involving the ability to pay for an attorney',
...
];
// Then, we pass them into our SearchForm component:
{
handleSearch(query, setResults, setIsSearching);
setQuery(query);
}}
/>
```
This means you can pass in any suggested searches you wish given your specific use case.
The SearchForm component is exported from `src/components/SearchForm.tsx`. It handles:
* Displaying suggested searches
* Allowing the user to search, or clear the input
* Providing visual feedback to the user that the search is in progress
**Local document processing via a bootstrapping service**
We store several landmark legal cases as PDFs in the codebase, so that developers cloning and running the app locally can immediately build off the same experience being demonstrated by the legal semantic search app running on our Docs site.
We use Langchain to parse the PDFs, convert them into chunks, and embed them. We store the resulting vectors in the Pinecone vector database.
**Knowledge base bootstrapping**
This project demonstrates how to programmatically bootstrap a knowledge base backed by a Pinecone vector database using arbitrary PDF files
that are included in the codebase.
The sample app use case is focused on semantic search over legal documents, but this exact same technique and code can be applied to any content stored locally.
```typescript theme={null}
export const handleBootstrapping = async (targetIndex: string) => {
try {
console.log(`Running bootstrapping procedure against Pinecone index: ${targetIndex}`);
// If a Pinecone index with the target name doesn't exist, create it
// If it does exist, return while suppressing conflict errors
await createIndexIfNecessary(targetIndex);
// Short-circuit early if the index already exists and has vectors in it
const hasVectors = await pineconeIndexHasVectors(targetIndex);
if (hasVectors) {
console.log('Pinecone index already exists and has vectors in it - returning early without bootstrapping');
return NextResponse.json({ sucess: true }, { status: 200 });
}
if (!hasVectors) {
console.log('Pinecone index does not exist or has no vectors in it - bootstrapping');
}
// Load metadata from db.json
const metadata = await readMetadata();
// Form the local path to the PDFs documents
const docsPath = path.resolve(process.cwd(), 'docs/')
const loader = new DirectoryLoader(docsPath, {
'.pdf': (filePath: string) => new PDFLoader(filePath),
});
// Load all PDFs within the specified directory
const documents = await loader.load();
// Merge extracted metadata with documents based on filename
documents.forEach((doc, index) => {
const fileMetadata = metadata.find(meta => meta.filename === path.basename(doc.metadata.source));
if (fileMetadata) {
doc.metadata = { ...doc.metadata, ...fileMetadata, pageContent: doc.pageContent };
} else {
console.warn(`No metadata found for ${doc.metadata.source}`);
}
});
// Split text into chunks
const splitter = new RecursiveCharacterTextSplitter({ chunkSize: 1000 });
const splits = await splitter.splitDocuments(documents);
// Assign unique IDs to each split and flatten metadata
const castedSplits: Document[] = splits.map(split => ({
pageContent: split.pageContent,
metadata: {
...flattenMetadata(split.metadata as Document['metadata']),
id: uuidv4(),
pageContent: split.pageContent, // Ensure pageContent is included in metadata
},
}));
// Extract page contents
const pageContents = castedSplits.map(split => split.pageContent);
// Generate embeddings for each chunk
const voyageEmbeddings = new VoyageEmbeddings({
apiKey: process.env.VOYAGE_API_KEY,
inputType: "document",
modelName: "voyage-law-2",
});
const embeddings = await voyageEmbeddings.embedDocuments(pageContents);
// Combine embeddings with metadata and ensure IDs are defined
const vectors = castedSplits.map((split, index) => {
if (!split.metadata.id) {
throw new Error('Document chunk is missing an ID');
}
return {
id: split.metadata.id!,
values: embeddings[index],
metadata: split.metadata,
};
});
const pc = new Pinecone();
const index = pc.Index(process.env.PINECONE_INDEX as string);
// Batch upserts to stay within the 2MB request size limit
await batchUpserts(index, vectors);
console.log('Bootstrap procedure completed.');
return NextResponse.json({ success: true }, { status: 200 });
} catch (error) {
console.error('Error during bootstrap procedure:', error);
}
return true;
}
```
When a user access the app, it runs a check to determine if the bootsrapping procedure needs to be run.
If the Pinecone index does not already exist, or if it exists but does not yet contain vectors, the bootstrapping procedure is run.
The bootsrapping procedure:
* Creates the Pinecone index specified by the `PINECONE_INDEX` environment variable
* Loads metadata from the `docs/db.json` file
* Loads all PDFs in the `docs` directory
* Merges extracted metadata with documents based on filename
* Splits text into chunks
* Assigns unique IDs to each split and flattens metadata
* Upserts each chunk to the Pinecone vector database, in batches
**Domain-specific embeddings model**
This app uses Voyage AI's embeddings model, `voyage-law-2`, which is purpose-built for use with legal text. This app includes a small handfull of landmark U.S. cases from Justia.
During the bootstrapping phase, the case documents are chunked and passed to Voyage's embeddings model for embedding:
```typescript theme={null}
// Generate embeddings for each chunk
const voyageEmbeddings = new VoyageEmbeddings({
apiKey: process.env.VOYAGE_API_KEY,
inputType: "document",
modelName: "voyage-law-2",
});
```
When the user executes a search, their query is sent to the `/api/search` route, which also uses
Voyage's embeddings model to convert the user's query into query vectors:
```typescript theme={null}
// Initialize VoyageEmbeddings
const voyageEmbeddings = new VoyageEmbeddings({
apiKey: process.env.VOYAGE_API_KEY,
inputType: 'document',
modelName: "voyage-law-2",
});
// Initialize PineconeVectorStore
const vectorStore = new PineconeStore(voyageEmbeddings, {
pineconeIndex: pc.Index(process.env.PINECONE_INDEX as string),
});
const retrieved = await vectorStore.maxMarginalRelevanceSearch(query, { k: 20 });
```
***
## Troubleshooting
Experiencing any issues with the sample app?
[Submit an issue, create a PR](https://github.com/pinecone-io/sample-apps/), or post in our [community forum](https://community.pinecone.io)!