Import data

This page shows you how to use the start_import, list_imports, describe_import, and cancel_import operations to import data into an index and interact with the import.

To learn how to format data for importing and other concepts related to imports, see Understanding imports. To run through this guide in your browser, use the Bulk import colab notebook.

​

Before you import

Before you can import data, ensure you have the following:

• An ID for your Amazon S3 integration (not needed for importing from a public bucket). The ID is found on the Storage integrations page of the Pinecone console.

• Data formatted in a Parquet file and uploaded to the Amazon S3 bucket.

• A serverless index on AWS to import records into.

  The index cannot have existing namespaces with the same name as the namespaces defined in your file directory structure.

​

Import records into an index

Use the start_import operation to start an asynchronous import of vectors from object storage into an index.

To import from a private bucket, specify the Integration ID (integration) of the Amazon S3 integration you created. The ID is found on the Storage integrations page of the Pinecone console. An ID is not needed to import from a public bucket.

The operation returns an id that you can use to check the status of the import.

{
   "operation_id": "101"
}

Once all the data is loaded, the index builder will index the records, which usually takes at least 10 minutes. During this indexing process, the expected job status is InProgress, but 100.0 percent complete. Once all the imported records are indexed and fully available for querying, the import operation will be set to Completed.

​

List recent and ongoing imports

Use the list_imports operation to list all of the recent and ongoing imports. By default, the operation returns up to 100 imports per page. If the limit parameter is passed, the operation returns up to that number of imports per page instead. For example, if limit=3, up to 3 imports are returned per page. Whenever there are additional imports to return, the response includes a pagination_token for fetching the next page of imports.

​

Implicit pagination

When using the Python SDK, list_import paginates automatically.

from pinecone import Pinecone, ImportErrorMode

pc = Pinecone(api_key="YOUR_API_KEY")

# To get the unique host for an index, 
# see https://docs.pinecone.io/guides/data/target-an-index
index = pc.Index(host="INDEX_HOST")

# List using a generator that handles pagination
for i in index.list_imports():
    print(f"id: {i.id} status: {i.status}")

# List using a generator that fetches all results at once
operations = list(index.list_imports())
print(operations)

{
  "data": [
    {
      "id": "1",
      "uri": "s3://BUCKET_NAME/PATH/TO/DIR",
      "status": "Pending",
      "started_at": "2024-08-19T20:49:00.754Z",
      "finished_at": "2024-08-19T20:49:00.754Z",
      "percent_complete": 42.2,
      "records_imported": 1000000
    }
  ],
  "pagination": {
    "next": "Tm90aGluZyB0byBzZWUgaGVyZQo="
  }
}

​

Manual pagination

When using the Node.js SDK, Go SDK, .NET SDK, or REST API to list recent and ongoing imports, you must manually fetch each page of results. To view the next page of results, include the paginationToken provided in the response of the list_imports / GET request.

​

Describe an import

Use the describe_import operation to get details about a specific import.

{
  "id": "101",
  "uri": "s3://BUCKET_NAME/PATH/TO/DIR",
  "status": "Pending",
  "created_at": "2024-08-19T20:49:00.754Z",
  "finished_at": "2024-08-19T20:49:00.754Z",
  "percent_complete": 42.2,
  "records_imported": 1000000
}

​

Cancel an ongoing import

The cancel_import operation cancels an import if it is not yet finished. It has no effect if the import is already complete.

{}