SAMPLE APP

​
Multimodal search

Multimodal search across text, images, and videos
$ npx create-pinecone-app@latest --template shop-the-look
The Shop The Look app demonstrates how to build a multimodal search engine for finding outfit inspiration using Pinecone Vector Database, Google Cloud Vertex AI’s Multimodal Embedding Model, and assets from Pexels. This application showcases how easy it is to use Pinecone in combining text, image, and video inputs to provide highly relevant outfit recommendations (or other multimodal use cases).
$ npx create-pinecone-app@latest --template shop-the-look
Github Open in a new window

​
Built with

  • Pinecone Serverless
  • Google Cloud Vertex AI Multimodal Embedding Model
  • Next.js + Tailwind CSS
  • Vercel
  • NodeJS

​
Run the sample app

The fastest way to get started is to use the create-pinecone-app CLI tool to run Shop The Look in demo mode.
npx create-pinecone-app@latest --template shop-the-look
Note: Demo mode is for developers who want to quickly deploy and test the Shop The Look application without setting up their own backend services or supply their own image/video assets. This demo deployment includes over 45,000 royalty-free images and videos, allowing you to deploy the front-end locally, while utilizing our hosted backend API (which we have set up with all assets, Pinecone Serverless index, and Google Cloud Vertex AI).

​
Full deployment

For developers who want to deploy a fully customizable Shop The Look application with their own images and videos, we offer a full deployment option. This method requires setting up both the frontend and backend components, including Pinecone Serverless, Google Cloud Vertex AI, and Google Cloud Storage, and uploading your own images and videos.A short version of the full deployment is listed below.For the complete documentation, click here for the Shop The Look Full Deployment instructions.

​
Get your Pinecone API key

You need an API key to make calls to your Pinecone project:
Then copy your generated key:
PINECONE_API_KEY="{{YOUR_API_KEY}}"
Alternatively, follow these steps:
  1. Open the Pinecone console.
  2. Select your project.
  3. Go to API Keys.
  4. Copy your API key.

​
Get your Google Cloud credentials

We recommend you follow the full docs for setting up your Google Cloud credentials. Below is a summary of the instructions.
  1. Create an account in Google Cloud if you don’t already have one.
  2. Create a new project in the Google Cloud Console.
  3. Enable the Vertex AI API and Cloud Storage API.
  4. Create a service account with Vertex AI User and Storage Object Viewer roles.
  5. Generate and download a JSON key for the service account.

​
Create a Pinecone serverless index

Create a Pinecone index for this project with the following properties:
  • dimension: 1408
  • metric: cosine
  • region: Choose your preferred region
You can create the index in the console, or by following the instructions here.

​
Start the project

Requires Node version 14+

​
Dependency installation

From the project root directory, run the following command:
cd shop-the-look && npm install
Make sure you have populated the .env.development with relevant keys:
GOOGLE_CLOUD_PROJECT_ID="your-project-id"
GOOGLE_CLOUD_PROJECT_LOCATION="your-region"
GOOGLE_CLOUD_STORAGE_BUCKET_NAME="your-bucket-name"
GOOGLE_CREDENTIALS_BASE64="your-base64-encoded-credentials"

PINECONE_API_KEY="your-api-key"
PINECONE_INDEX_NAME="your-index-name"
PINECONE_TOP_K=20
Start the app:
npm run dev

​
Project structure

titleShop The Look uses a NextJS frontend with a FastAPI Python backend. Utility scripts are included to process images and videos for embedding + upserting.Frontend ClientThe frontend uses Next.js, Tailwind CSS, and custom React components to power the search experience. It leverages custom API routes to make calls to the FastAPI backend for embedding and searching.Backend ServerThis project uses FastAPI to handle image/video processing, embedding generation, and Pinecone operations.Utility ScriptsThe image and video embedding processing scripts process a folder of images and videos from Google Cloud Storage, generate embeddings using Vertex AI, and upserts them to Pinecone Index.

​
Architecture

Shop The Look is built using Pinecone Vector Database, Google Cloud Vertex AI’s Multimodal Embedding Model, assets from Pexels, and is hosted on Vercel.Using pseudocode, we will explain how we built key components of Shop The Look.Pinecone ServerlessThis project uses Pinecone to store all the multimodal embeddings generated by Vertex AI’s Multimodal Embedding Model.
pc = Pinecone(api_key)
index = pc.Index(index_name)

vector = [
    {
        'id': embedding_id,
        'values': embedding_values,
        'metadata': {
            ...
        }
    }
]
index.upsert(vector)
Google Cloud Vertex AI integrationThis app uses Google Cloud Vertex AI’s Multimodal Embedding Model to generate embeddings for text, images, and videos:
embeddings = model.get_embeddings(
    asset=asset,
)
Search functionalityWhen the user executes a search, their query (text or image/video) is sent to the backend, which uses the Multimodal Embedding Model to convert the query into vector embeddings:
url, headers, data = get_embedding_request_data(access_token, asset, query)

response = requests.post(url, headers, data)

embedding_data = response.json()
vector = embedding_data['predictions'][0]['assetEmbedding']
These embeddings are then used to perform a similarity search in Pinecone:
query_response = index.query(
    vector=vector,
    top_k=top_k,
    include_metadata=True
)

matches = query_response['matches']
The results are then displayed in the React frontend page.tsx

​
Troubleshooting

Experiencing any issues with the sample app? Read the Troubleshooting section of the Shop The Look README. If issues still persist, submit an issue, create a PR, or post in our community forum!