This page shows you how to migrate a pod-based index to serverless using the Pinecone console.

If you have questions about this process, or run into any issues, contact Pinecone Support.

Creating a serverless index from a collection is in public preview and is currently limited to pod-based indexes on AWS and GCP with less than 10 million records and 20,000 namespaces. Support for larger indexes and indexes on Azure is coming soon.

Benefits

Migrating to serverless has many advantages over pod-based indexes, including:

Also note that the migration process is free; the standard costs of upserting records to a new serverless index are not applied.

How it works

Migrating a pod-based index to serverless is a 2-step process:

1

Save the pod-based index as a collection

2

Create a new serverless index from the collection

After migration, you will have both a new serverless index and the original pod-based index. Once you’ve switched your workload to the serverless index, you can delete the pod-based index to avoid paying for unused resources.

Creating a serverless index from a collection is in public preview. Migrating to serverless is currently limited to pod-based indexes with less than 10 million records and 20,000 namespaces. Support for larger indexes is coming soon.

1. Prepare for migration

Migrating a pod-based index to serverless can take anywhere from a few minutes to several hours, depending on the size of the index. During that time, you can continue reading from the pod-based index. However, all upserts, updates, and deletes to the pod-based index will not automatically be reflected in the new severless index, so be sure to prepare in one of the following ways:

  • Pause write traffic: If downtime is acceptable, pause traffic to the pod-based index before starting migration. After migration, you will start sending traffic to the serverless index.

  • Log your writes: If you need to continue reading from the pod-based index during migration, send read traffic to the pod-based index, but log your writes to a temporary location outside of Pinecone (e.g., S3). After migration, you will replay the logged writes to the new serverless index and start sending all traffic to the serverless index.

2. Start migration

  1. In the Pinecone console, go to your pod-based index and select Collection:

    save index as collection

  2. Enter a name for your collection and click Create Collection.

  3. Go to your collections list and check the status of the collection.

  4. When the collection is ready, click the Create Index action:

    create index from collection

  5. Enter a name for your new serverless index.

  6. Select the Serverless capacity mode.

  7. Select the cloud region where you want your new serverless index.

  8. Click Create index.

Depending on the size of the index, migration can take anywhere from a few minutes to several hours. While migration is in progress, you’ll see the yellow Initializing status:

create index from collection - initializing status

When the new serverless index is ready, the status will change to green:

create index from collection - ready status

3. Update clients

If you are using an older version of the Python, Node.js, or Java client, you must update the client to work with serverless indexes.

  1. Check your client version:

    pip show pinecone-client  
    
  2. If your client version is less than 3.0.0 for Python, 2.0.0 for Node.js, or 1.0.0 for Java, upgrade the client as follows:

    pip install pinecone-client[grpc] --upgrade  
    

4. Adapt existing code

You must make some minor code changes to work with serverless indexes.

  1. Change how you import the pinecone.grpc library and authenticate and initialize the client:

    from pinecone.grpc import PineconeGRPC as Pinecone
    from pinecone import ServerlessSpec, PodSpec  
    # ServerlessSpec and PodSpec are required only when  
    # creating serverless and pod-based indexes.  
    pc = Pinecone(api_key="YOUR_API_KEY")  
    
  2. Listing indexes now fetches a complete description of each index. If you were relying on the output of this operation, you’ll need to adapt your code.

    from pinecone.grpc import PineconeGRPC as Pinecone
    
    pc = Pinecone(api_key="YOUR_API_KEY")
    
    pc.list_indexes()
    
    # Response:
    # {'indexes': [{'dimension': 1536,
    #               'host': 'serverless-index-4zo0ijk.svc.us-west2-aws.pinecone.io',
    #               'metric': 'cosine',
    #               'name': 'serverless-index',
    #               'spec': {'serverless': {'cloud': 'aws', 'region': 'us-east-1'}},
    #               'status': {'ready': True, 'state': 'Ready'}},
    #              {'dimension': 1536,
    #               'host': 'pod-index-4zo0ijk.svc.us-west2-aws.pinecone.io',
    #               'metric': 'cosine',
    #               'name': 'pod-index',
    #               'spec': {'pod': {'environment': 'us-west2-aws',
    #                                'pod_type': 'p1.x1',
    #                                'pods': 1,
    #                                'replicas': 1,
    #                                'shards': 1}},
    #               'status': {'ready': True, 'state': 'Ready'}}]}
    
  3. Describing an index now returns a description of an index in a different format. It also returns the index endpoint needed to run data plane operations against the index. If you were relying on the output of this operation, you’ll need to adapt your code.

    from pinecone.grpc import PineconeGRPC as Pinecone
    
    pc = Pinecone(api_key="YOUR_API_KEY")
    
    pc.describe_index("serverless-index")
    
    # Response:
    # {'dimension': 1536,
    #  'host': 'serverless-index-4zo0ijk.svc.us-weset2-aws.pinecone.io',
    #  'metric': 'cosine',
    #  'name': 'serverless-index',
    #  'spec': {'serverless': {'cloud': 'aws', 'region': 'us-east-1'}},
    #  'status': {'ready': True, 'state': 'Ready'}}
    
  4. Serverless indexes do not support the following features. If you were relying on these features for your pod-based index, you’ll need to adapt your code.

Depending on your usage of the client, there may be other changes to make. For a comprehensive list, see the Python client v3 migration guide, Node.js client v2 migration guide, or Java client v1 migration guide.

5. Use your new index

When you’re ready to cutover to your new serverless index:

  1. Your new serverless index has a different name and unique endpoint than your pod-based index. Update your code to target the new serverless index:

    index = pc.Index("YOUR_SERVERLESS_INDEX_NAME")  
    
  2. Reinitialize your clients.

  3. If you logged writes to the pod-based index during migration, replay the logged writes to your serverless index.

  4. Delete the pod-based index to avoid paying for unused resources.

    It is not possible to save a serverless index as a collection, so if you want to retain the option to recreate your pod-based index, be sure to keep the collection you created earlier.

See also