Multimodal search

Built with

Pinecone Database
Google Cloud Vertex AI multimodal embedding model
Next.js and Tailwind CSS
Vercel
Node.js

Run the sample app

To run Shop The Look in demo mode, use the create-pinecone-app CLI:

Terminal

npx create-pinecone-app@latest --template shop-the-look

Demo mode runs the frontend locally and connects it to a backend API that Pinecone hosts. The hosted backend already has a Pinecone index, Google Cloud Vertex AI, and more than 45,000 royalty-free images and videos set up, so you don’t need your own backend services or assets.

Full deployment

To customize Shop The Look with your own images and videos, deploy both the frontend and the backend. The backend uses Pinecone Database, Google Cloud Vertex AI, and Google Cloud Storage.The following steps are a summary. For complete instructions, see Full deployment in the Shop The Look README.

Get your Pinecone API key

You need an API key to make calls to your Pinecone project:

Then copy your generated key:

.env.development

PINECONE_API_KEY="{{YOUR_API_KEY}}"

You can also copy an existing key from the API keys page in the Pinecone console.

Get your Google Cloud credentials

The following steps are a summary. For complete instructions, see Google Cloud setup in the Shop The Look README.

If you don’t have a Google Cloud account, create one.
In the Google Cloud console, create a project.
Enable the Vertex AI API and the Cloud Storage API.
Create a service account with the Vertex AI User and Storage Object Viewer roles.
Generate a JSON key for the service account and download it.

Create a Pinecone index

Create a serverless index that stores dense vectors, with the following settings:

Dimension: 1408
Metric: cosine
Region: A region of your choice

You can create the index in the Pinecone console or with the API. For more information, see Create an index.

Install dependencies

The app requires Node.js 14 or later. From the project root directory, install the dependencies:

Terminal

cd shop-the-look && npm install

Set environment variables

In .env.development, set your Google Cloud and Pinecone values:

.env.development

GOOGLE_CLOUD_PROJECT_ID="YOUR_PROJECT_ID"
GOOGLE_CLOUD_PROJECT_LOCATION="YOUR_REGION"
GOOGLE_CLOUD_STORAGE_BUCKET_NAME="YOUR_BUCKET_NAME"
GOOGLE_CREDENTIALS_BASE64="YOUR_BASE64_ENCODED_CREDENTIALS"

PINECONE_API_KEY="YOUR_API_KEY"
PINECONE_INDEX_NAME="YOUR_INDEX_NAME"
PINECONE_TOP_K=20

Start the app

Start the development server:

Terminal

npm run dev

Project structure

Shop The Look architecture: a Next.js frontend calls a FastAPI backend, which generates embeddings with Vertex AI and searches Pinecone

Shop The Look has three parts:

Frontend: A Next.js app with Tailwind CSS and custom React components. Its API routes call the backend to generate embeddings and run searches.
Backend: A FastAPI server in Python that processes images and videos, generates embeddings, and runs Pinecone operations.
Utility scripts: Scripts that read a folder of images and videos from Google Cloud Storage, generate embeddings with Vertex AI, and upsert them to a Pinecone index.

How it works

The following pseudocode shows how the main parts of Shop The Look work together.

Store embeddings in Pinecone

Pinecone stores the multimodal embeddings that Vertex AI generates for each image and video:

Python

pc = Pinecone(api_key)
index = pc.Index(index_name)

vector = [
    {
        'id': embedding_id,
        'values': embedding_values,
        'metadata': {
            ...
        }
    }
]
index.upsert(vector)

Generate embeddings with Vertex AI

Vertex AI’s multimodal embedding model generates embeddings for text, images, and videos:

Python

embeddings = model.get_embeddings(
    asset=asset,
)

Search

When a user searches, the backend sends the query (text, an image, or a video) to the multimodal embedding model to convert it into an embedding:

Python

url, headers, data = get_embedding_request_data(access_token, asset, query)

response = requests.post(url, headers, data)

embedding_data = response.json()
vector = embedding_data['predictions'][0]['assetEmbedding']

The backend then uses the embedding to run a similarity search in Pinecone:

Python

query_response = index.query(
    vector=vector,
    top_k=top_k,
    include_metadata=True
)

matches = query_response['matches']

The frontend displays the results in page.tsx.

Troubleshooting

If you run into problems, see Troubleshooting in the Shop The Look README. If that doesn’t solve it, open an issue in the sample apps repository or ask in the Pinecone Discord community.

​Multimodal search

​Built with

​Run the sample app

​Full deployment

Get your Pinecone API key

Get your Google Cloud credentials

Create a Pinecone index

Install dependencies

Set environment variables

Start the app

​Project structure

​How it works

​Store embeddings in Pinecone

​Generate embeddings with Vertex AI

​Search

​Troubleshooting

Built with

Run the sample app

Full deployment

Project structure

How it works

Store embeddings in Pinecone

Generate embeddings with Vertex AI

Search

Troubleshooting