Import vectors into a legacy vector index

New projects should use the Documents API. To move an existing index, see Move to the Documents API.

This feature is available on Standard and Enterprise plans.

A legacy vector index imports Parquet files from object storage. Follow the steps in Import data to create an index, add a storage integration, lay out one directory per namespace, and start the import. This page covers the Parquet file format for each kind of legacy vector index. You can also import Parquet files into a legacy integrated index, but its records must contain vectors, not text. To add text that Pinecone embeds, use Upsert text.

Parquet file format

Dense vectors
Sparse vectors
Dense + sparse

To import into a namespace in an index of dense vectors, the Parquet file must contain the following columns:

Column name	Parquet type	Description
`id`	`STRING`	Required. The unique identifier for each record.
`values`	`LIST<FLOAT>`	Required. A list of floating-point values that make up the dense vector embedding.
`metadata`	`STRING`	Optional. Additional metadata for each record. To omit from specific rows, use `NULL`.

Additional columns in the Parquet file are silently ignored during import; only id, values, and metadata are processed.

For example:

id | values                   | metadata
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
1  | [ 3.82  2.48 -4.15 ... ] | {"year": 1984, "month": 6, "source": "source1", "title": "Example1", "text": "When ..."}
2  | [ 1.82  3.48 -2.15 ... ] | {"year": 1990, "month": 4, "source": "source2", "title": "Example2", "text": "Who ..."}

To import into a namespace in an index of sparse vectors, the Parquet file must contain the following columns:

Column name	Parquet type	Description
`id`	`STRING`	Required. The unique identifier for each record.
`sparse_values`	`STRUCT<indices: LIST<UINT_32>, values: LIST<FLOAT>>`	Required. A list of floating-point values (sparse values) and a list of integer values (sparse indices) that make up the sparse vector embedding.
`metadata`	`STRING`	Optional. Additional metadata for each record. To omit from specific rows, use `NULL`.

Additional columns in the Parquet file are silently ignored during import; only id, sparse_values, and metadata are processed.

For example:

id | sparse_values                                                                                       | metadata
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
1  | {"indices": [ 822745112 1009084850 1221765879 ... ], "values": [1.7958984 0.41577148 2.828125 ...]} | {"year": 1984, "month": 6, "source": "source1", "title": "Example1", "text": "When ..."}
2  | {"indices": [ 504939989 1293001993 3201939490 ... ], "values": [1.4383747 0.72849722 1.384775 ...]} | {"year": 1990, "month": 4, "source": "source2", "title": "Example2", "text": "Who ..."}

To import into a namespace in an index with both dense and sparse vectors, the Parquet file must contain the following columns:

Column name	Parquet type	Description
`id`	`STRING`	Required. The unique identifier for each record.
`values`	`LIST<FLOAT>`	Required. A list of floating-point values that make up the dense vector embedding.
`sparse_values`	`STRUCT<indices: LIST<UINT_32>, values: LIST<FLOAT>>`	Optional. A list of floating-point values that make up the sparse vector embedding. To omit from specific rows, use `NULL`.
`metadata`	`STRING`	Optional. Additional metadata for each record. To omit from specific rows, use `NULL`.

Additional columns in the Parquet file are silently ignored during import; only id, values, sparse_values, and metadata are processed.

For example:

id | values                   | sparse_values                                                                          | metadata
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
1  | [ 3.82  2.48 -4.15 ... ] | {"indices": [1082468256, 1009084850, 1221765879, ...], "values": [2.0, 3.0, 4.0, ...]} | {"year": 1984, "month": 6, "source": "source1", "title": "Example1", "text": "When ..."}
2  | [ 1.82  3.48 -2.15 ... ] | {"indices": [2225824123, 1293001993, 3201939490, ...], "values": [5.0, 2.0, 3.0, ...]} | {"year": 1990, "month": 4, "source": "source2", "title": "Example2", "text": "Who ..."}

Get started

Indexes

Data

Search

Optimize

Cost

Production

Agents

Legacy

Import vectors into a legacy vector index

Parquet file format

​Parquet file format

Parquet file format