README
¶
⛟ Migration → Qdrant
CLI tool for migrating data to Qdrant with support for resumable transfers in case of interruptions.
Supported Sources
- Chroma
- Pinecone
- Milvus
- Weaviate
- Redis
- MongoDB
- OpenSearch
- Elasticsearch
- Postgres (pgvector)
- S3 Vectors
- FAISS
- Apache Solr
- Another Qdrant instance
Installation
You can run this tool on any machine with connectivity to both the source and the Qdrant database. For best performance, use a machine with a fast network and minimal latency to both endpoints.
The tool is distributed as a container image, which can be run on any system with Docker, Podman or similar container runtimes. The following examples use Docker.
To get the latest container image run the following command:
docker pull registry.cloud.qdrant.io/library/qdrant-migration
How To Migrate?
Click each to expand
From Chroma
Migrate data from a Chroma database to Qdrant:
📥 Example
docker run --net=host --rm -it registry.cloud.qdrant.io/library/qdrant-migration chroma \
--chroma.url=http://source-hostname:8000
--chroma.collection 'collection-name' \
--qdrant.url 'https://example.cloud-region.cloud-provider.cloud.qdrant.io:6334' \
--qdrant.api-key 'optional-qdrant-api-key' \
--qdrant.collection 'target-collection' \
--migration.batch-size 64
Chroma Options
| Flag | Description |
|---|---|
--chroma.collection |
Chroma collection name. |
--chroma.url |
Chroma server URL Default: "http://localhost:8000" |
--chroma.tenant |
Chroma tenant. Optional. |
--chroma.auth-type |
Authentication type. "basic" or "token". Optional. |
--chroma.username |
Username for basic authentication. Optional. |
--chroma.password |
Password for basic authentication. Optional. |
--chroma.token |
Token for token authentication. Optional. |
--chroma.token-header |
Token header for authentication. Optional. |
--chroma.database |
Database name. Optional. |
Qdrant Options
| Flag | Description |
|---|---|
--qdrant.collection |
Target collection name. |
--qdrant.url |
Qdrant gRPC URL. Default: "http://localhost:6334" |
--qdrant.api-key |
Qdrant API key. Optional. |
--qdrant.id-field |
Field storing Chroma IDs in Qdrant. Default: "__id__" |
--qdrant.distance-metric |
Distance metric for the Qdrant collection. "cosine", "dot", "manhattan" or "euclid". Default: "euclid" |
--qdrant.document-field |
Field storing Chroma documents in Qdrant. Default: "document" |
- See Shared Migration Options for common migration parameters.
From Pinecone
Migrate data from a Pinecone database to Qdrant:
IMPORTANT ⚠️: Only Pinecone serverless indexes support listing all vectors for migration. Reference
📥 Example
docker run --net=host --rm -it registry.cloud.qdrant.io/library/qdrant-migration pinecone \
--pinecone.index-host 'https://example-index.svc.region.pinecone.io' \
--pinecone.index-name 'example-index' \
--pinecone.api-key 'optional-pinecone-api-key' \
--qdrant.url 'https://example.cloud-region.cloud-provider.cloud.qdrant.io:6334' \
--qdrant.api-key 'optional-qdrant-api-key' \
--qdrant.collection 'target-collection' \
--migration.batch-size 64
Pinecone Options
| Flag | Description |
|---|---|
--pinecone.index-name |
Pinecone index name. |
--pinecone.index-host |
Pinecone index host URL (e.g., https://your-pinecone-url) |
--pinecone.api-key |
Pinecone API key for authentication. |
--pinecone.namespace |
Namespace of the partition to migrate. Optional. |
--pinecone.service-host |
Pinecone service host URL. Optional. |
Qdrant Options
| Flag | Description |
|---|---|
--qdrant.collection |
Target collection name |
--qdrant.url |
Qdrant gRPC URL. Default: "http://localhost:6334" |
--qdrant.api-key |
Qdrant API key |
--qdrant.sparse-vector |
Name of the sparse vector in Qdrant. Default: "sparse_vector" |
--qdrant.id-field |
Field storing Pinecone IDs in Qdrant. Default: "__id__" |
- See Shared Migration Options for common migration parameters.
From Milvus
Migrate data from a Milvus database to Qdrant:
📥 Example
docker run --net=host --rm -it registry.cloud.qdrant.io/library/qdrant-migration milvus \
--milvus.url 'https://example.gcp-us-west1.cloud.zilliz.com' \
--milvus.enable-tls-auth \
--milvus.collection 'example-collection' \
--milvus.api-key 'optional-milvus-api-key' \
--qdrant.url 'https://example.cloud-region.cloud-provider.cloud.qdrant.io:6334' \
--qdrant.api-key 'optional-qdrant-api-key' \
--qdrant.collection 'target-collection' \
--migration.batch-size 64
Milvus Options
| Flag | Description |
|---|---|
--milvus.url |
Milvus URL (e.g. https://your-milvus-hostname) |
--milvus.collection |
Milvus collection name |
--milvus.api-key |
Milvus API key for authentication |
--milvus.enable-tls-auth |
Whether to enable TLS Auth |
--milvus.username |
Username for Milvus |
--milvus.password |
Password for Milvus |
--milvus.db-name |
Optional database name |
--milvus.server-version |
Milvus server version |
--milvus.partitions |
List of partition names |
Qdrant Options
| Flag | Description |
|---|---|
--qdrant.url |
Qdrant gRPC URL. Default: "http://localhost:6334" |
--qdrant.collection |
Target collection name |
--qdrant.api-key |
Qdrant API key |
--qdrant.distance-metric |
Map of vector names to distance metrics ("cosine","dot","euclid","manhattan"). Default: "cosine" |
- See Shared Migration Options for common migration parameters.
From Weaviate
Migrate data from a Weaviate database to Qdrant:
📥 Example
Important ⚠️: Weaviate does not expose vector dimensions and distance metric after a collection is created. Therefore, you must manually create a Qdrant collection before starting the migration. Ensure that the vector dimensions in Qdrant exactly match those used in Weaviate.
docker run --net=host --rm -it registry.cloud.qdrant.io/library/qdrant-migration weaviate \
--weaviate.host 'example.c0.asia-southeast1.gcp.weaviate.cloud' \
--weaviate.scheme 'https' \
--weaviate.auth-type 'apiKey' \
--weaviate.api-key 'optional-api-key' \
--weaviate.class-name 'ExampleClass' \
--qdrant.url 'http://target-hostname:6334' \
--qdrant.collection 'target-collection' \
--migration.batch-size 64
Weaviate Options
| Flag | Description |
|---|---|
--weaviate.host |
Host of the Weaviate instance (e.g. localhost:8080) (required) |
--weaviate.scheme |
Scheme of the Weaviate instance (e.g. http or https) (default: http) |
--weaviate.class-name |
Name of the Weaviate class to migrate (required) |
--weaviate.auth-type |
Authentication type (default: none). Options: none, apiKey, password, client, bearer |
--weaviate.api-key |
API key for authentication (when auth-type is apiKey) |
--weaviate.username |
Username for authentication (when auth-type is password) |
--weaviate.password |
Password for authentication (when auth-type is password) |
--weaviate.scopes |
Scopes for authentication (when auth-type is password or client) |
--weaviate.client-secret |
Client secret for authentication (when auth-type is client) |
--weaviate.token |
Bearer token for authentication (when auth-type is bearer) |
--weaviate.refresh-token |
Refresh token for authentication (when auth-type is bearer) |
--weaviate.expires-in |
Access token expiration time in seconds (when auth-type is bearer) |
--weaviate.tenant |
Objects belonging to which tenant to migrate |
Qdrant Options
| Flag | Description |
|---|---|
--qdrant.url |
Qdrant gRPC URL. Default: "http://localhost:6334" |
--qdrant.collection |
Target collection name |
--qdrant.api-key |
Qdrant API key |
- See Shared Migration Options for common migration parameters.
From Redis
Migrate data from a Redis database to Qdrant:
Important ⚠️: Redis does not expose vector configurations after an index is created. Therefore, you must manually create a Qdrant collection before starting the migration. Ensure that the vector names and dimensions in Qdrant exactly match those used in Redis.
📥 Example
docker run --net=host --rm -it registry.cloud.qdrant.io/library/qdrant-migration milvus \
--redis.index 'index_name' \
--redis.addr 'source-hostname:6379' \
--qdrant.url 'http://target-hostname:6334' \
--qdrant.collection 'target-collection' \
--migration.batch-size 100
Redis Options
| Flag | Description |
|---|---|
--redis.index |
Redis index name |
--redis.addr |
Redis address in the format host:port (default: localhost:6379) |
--redis.protocol |
Redis protocol version (default: 2) |
--redis.password |
Password to authenticate requests. Optional. |
--redis.username |
Username to authenticate requests. Optional. |
--redis.client-name |
Will execute the CLIENT SETNAME <NAME> for each connection. Optional. |
--redis.db |
Database to be selected after connecting to the server. Optional. |
--redis.network |
Redis network type (tcp or unix, default: tcp) |
Qdrant Options
| Flag | Description |
|---|---|
--qdrant.url |
Qdrant gRPC URL. Default: "http://localhost:6334" |
--qdrant.collection |
Target collection name |
--qdrant.api-key |
Qdrant API key |
--qdrant.id-field |
Field storing Redis IDs in Qdrant. Default: "__id__" |
- See Shared Migration Options for common migration parameters.
From MongoDB
Migrate data from a MongoDB database to Qdrant:
Important ⚠️: You must manually create a Qdrant collection before starting the migration. Ensure that the vector names and dimensions in Qdrant exactly match those used in MongoDB.
📥 Example
docker run --net=host --rm -it registry.cloud.qdrant.io/library/qdrant-migration mongodb \
--mongodb.url 'mongodb+srv://<username>:<password>@<cluster-url>/mydb' \
--mongodb.database 'mydb' \
--mongodb.collection 'mycollection' \
--mongodb.vector-fields 'my-first-vector,my-second-vector' \
--qdrant.url 'http://target-hostname:6334' \
--qdrant.collection 'target-collection' \
--migration.batch-size 64
MongoDB Options
| Flag | Description |
|---|---|
--mongodb.url |
MongoDB connection string |
--mongodb.database |
MongoDB database name |
--mongodb.collection |
MongoDB collection name |
--mongodb.vector-fields |
MongoDB vector fields, comma separated (required) |
Qdrant Options
| Flag | Description |
|---|---|
--qdrant.url |
Qdrant gRPC URL. Default: "http://localhost:6334" |
--qdrant.collection |
Target collection name |
--qdrant.api-key |
Qdrant API key (optional) |
--qdrant.id-field |
Field storing MongoDB IDs in Qdrant. Default: "__id__" |
See Shared Migration Options for common migration parameters.
From OpenSearch
Migrate data from an OpenSearch index to Qdrant:
📥 Example
docker run --net=host --rm -it registry.cloud.qdrant.io/library/qdrant-migration opensearch \
--opensearch.url 'http://source-hostname:9200' \
--opensearch.insecure-skip-verify \
--opensearch.index 'source-index' \
--opensearch.username 'username' \
--opensearch.password 'password' \
c \
--qdrant.api-key 'optional-qdrant-api-key' \
--qdrant.collection 'target-collection' \
--migration.batch-size 64
OpenSearch Options
| Flag | Description |
|---|---|
--opensearch.url |
OpenSearch URL (e.g. http://localhost:9200). |
--opensearch.index |
OpenSearch index name. |
--opensearch.username |
Username for basic authentication (optional) |
--opensearch.password |
Password for basic authentication (optional) |
--opensearch.api-key |
API key for authentication (optional) |
--opensearch.insecure-skip-verify |
Whether to skip TLS certificate verification (optional) |
Qdrant Options
| Flag | Description |
|---|---|
--qdrant.url |
Qdrant gRPC URL. Default: "http://localhost:6334" |
--qdrant.collection |
Target collection name |
--qdrant.api-key |
Qdrant API key (optional) |
--qdrant.id-field |
Field storing OpenSearch IDs in Qdrant. Default: "__id__" |
See Shared Migration Options for common migration parameters.
From Elasticsearch
Migrate data from an Elasticsearch index to Qdrant:
📥 Example
docker run --net=host --rm -it registry.cloud.qdrant.io/library/qdrant-migration elasticsearch \
--elasticsearch.url 'http://source-hostname:9200' \
--elasticsearch.insecure-skip-verify \
--elasticsearch.index 'source-index' \
--elasticsearch.username 'username' \
--elasticsearch.password 'password' \
--qdrant.url 'http://target-hostname:6334' \
--qdrant.api-key 'optional-qdrant-api-key' \
--qdrant.collection 'target-collection' \
--migration.batch-size 64
Elasticsearch Options
| Flag | Description |
|---|---|
--elasticsearch.url |
Elasticsearch URL (e.g. http://localhost:9200). |
--elasticsearch.index |
Elasticsearch index name. |
--elasticsearch.username |
Username for basic authentication (optional) |
--elasticsearch.password |
Password for basic authentication (optional) |
--elasticsearch.api-key |
API key for authentication (optional) |
--elasticsearch.insecure-skip-verify |
Whether to skip TLS certificate verification (optional) |
Qdrant Options
| Flag | Description |
|---|---|
--qdrant.url |
Qdrant gRPC URL. Default: "http://localhost:6334" |
--qdrant.collection |
Target collection name |
--qdrant.api-key |
Qdrant API key (optional) |
--qdrant.id-field |
Field storing Elasticsearch IDs in Qdrant. Default: "__id__" |
See Shared Migration Options for common migration parameters.
From Postgres
Migrate data from a Postgres database with pgvector to Qdrant:
📥 Example
docker run --net=host --rm -it registry.cloud.qdrant.io/library/qdrant-migration pg \
--pg.url 'postgres://user:password@source-hostname:5432/dbname' \
--pg.table 'your_table' \
--pg.key-column 'id' \
--qdrant.url 'http://target-hostname:6334' \
--qdrant.collection 'target-collection' \
--migration.batch-size 64
Postgres Options
| Flag | Description |
|---|---|
--pg.url |
Postgres connection string (e.g., postgres://user:pass@host:port/dbname). |
--pg.table |
Name of the table containing vector data. |
--pg.key-column |
Column with unique values to be hashed as point IDs in Qdrant. |
--pg.columns |
Columns to migrate. Must include the key column. Defaults to all columns. |
Qdrant Options
| Flag | Description |
|---|---|
--qdrant.collection |
Target collection name |
--qdrant.url |
Qdrant gRPC URL. Default: http://localhost:6334 |
--qdrant.api-key |
Qdrant API key (optional) |
--qdrant.distance-metric |
Map of vector names to distance metrics ("cosine", "dot", "euclid", "manhattan"). Default: "cosine" |
Additional Options
| Flag | Description |
|---|---|
--migration.num-workers |
Number of parallel workers to use. Default: Number of CPU cores |
- See Shared Migration Options for common migration parameters.
From S3 Vectors
Migrate data from an S3 Vectors index to Qdrant:
📥 Example
Important ⚠️: Set your AWS credentials using the AWS CLI's configure command or environment variables.
docker run --net=host --rm -it registry.cloud.qdrant.io/library/qdrant-migration s3 \
--s3.bucket 'your-bucket-name' \
--s3.index 'your-index-name' \
--qdrant.url 'http://target-hostname:6334' \
--qdrant.api-key 'optional-qdrant-api-key' \
--qdrant.collection 'target-collection' \
--migration.batch-size 64
S3 Vectors Options
| Flag | Description |
|---|---|
--s3.bucket |
S3 Vectors bucket name (required) |
--s3.index |
S3 Vectors index name (required) |
Qdrant Options
| Flag | Description |
|---|---|
--qdrant.url |
Qdrant gRPC URL. Default: "http://localhost:6334" |
--qdrant.collection |
Target collection name |
--qdrant.api-key |
Qdrant API key (optional) |
--qdrant.id-field |
Field storing S3 IDs in Qdrant. Default: "__id__" |
- See Shared Migration Options for common migration parameters.
From FAISS
Migrate data from a FAISS index file to Qdrant:
📥 Example
Important ⚠️: Vectors from quantized FAISS indexes cannot be migrated since they don't store the original vectors.
IndexFlatL2,IndexFlatIP,IndexHNSWFlat,IndexIVFFlatare supported.
docker run --net=host --rm -it \
-v /path/to/faiss_index:/mnt/index \ # Mount the FAISS index file into container
registry.cloud.qdrant.io/library/qdrant-migration faiss \
--faiss.index-path '/mnt/index' \
--qdrant.url 'http://target-hostname:6334' \
--qdrant.api-key 'optional-qdrant-api-key' \
--qdrant.collection 'target-collection' \
--migration.batch-size 64
FAISS Options
| Flag | Description |
|---|---|
--faiss.index-path |
Path to the FAISS index file (required) |
Qdrant Options
| Flag | Description |
|---|---|
--qdrant.url |
Qdrant gRPC URL. Default: "http://localhost:6334" |
--qdrant.collection |
Target collection name |
--qdrant.api-key |
Qdrant API key (optional) |
--qdrant.distance-metric |
Distance metric for the Qdrant collection. "cosine", "dot", "manhattan" or "euclid". Default: "cosine" |
- See Shared Migration Options for common migration parameters.
From Apache Solr
Migrate data from an Apache Solr collection to Qdrant:
📥 Example
Important ⚠️: Solr does not always expose vector dimensions and distance metrics in its schema API. Therefore, you must manually create a Qdrant collection before starting the migration. Ensure that the vector names and dimensions in Qdrant exactly match those used in Solr.
docker run --net=host --rm -it registry.cloud.qdrant.io/library/qdrant-migration solr \
--solr.url 'http://source-hostname:8983' \
--solr.collection 'my-collection' \
--qdrant.url 'http://target-hostname:6334' \
--qdrant.api-key 'optional-qdrant-api-key' \
--qdrant.collection 'target-collection' \
--migration.batch-size 100
Solr Options
| Flag | Description |
|---|---|
--solr.url |
Solr base URL (e.g., http://localhost:8983). |
--solr.collection |
Solr collection name. |
--solr.username |
Username for basic authentication (optional) |
--solr.password |
Password for basic authentication (optional) |
--solr.insecure-skip-verify |
Whether to skip TLS certificate verification (optional) |
Qdrant Options
| Flag | Description |
|---|---|
--qdrant.url |
Qdrant gRPC URL. Default: "http://localhost:6334" |
--qdrant.collection |
Target collection name |
--qdrant.api-key |
Qdrant API key (optional) |
--qdrant.id-field |
Field storing Solr IDs in Qdrant. Default: "__id__" |
- See Shared Migration Options for common migration parameters.
From Another Qdrant Instance
Migrate data from one Qdrant instance to another.
📥 Example
docker run --net=host --rm -it registry.cloud.qdrant.io/library/qdrant-migration qdrant \
--source.url 'http://source-hostname:6334' \
--source.collection 'source-collection' \
--target.url 'https://example.cloud-region.cloud-provider.cloud.qdrant.io:6334' \
--target.api-key 'qdrant-key' \
--target.collection 'target-collection' \
--migration.batch-size 64
NOTE: If the target collection already exists, its vector size and dimensions must match the source. Other settings like replication, shards can differ.
Source Qdrant Options
| Flag | Description |
|---|---|
--source.collection |
Source collection name |
--source.url |
Source gRPC URL. Default: "http://localhost:6334" |
--source.api-key |
API key for source instance |
--source.max-message-size |
Maximum gRPC message size in bytes (default: 33554432 = 32MB). Increase if you encounter ResourceExhausted errors with large batches. |
Target Qdrant Options
| Flag | Description |
|---|---|
--target.collection |
Target collection name |
--target.url |
Target gRPC URL. Default: "http://localhost:6334" |
--target.api-key |
API key for target instance |
--target.ensure-payload-indexes |
Ensure payload indexes exist. Default: true |
Additional Options
| Flag | Description |
|---|---|
--migration.num-workers |
Number of parallel workers to use. Default: Number of CPU cores |
See Shared Migration Options for shared parameters.
Global Options
These options apply globally to all commands.
| Flag | Description |
|---|---|
--debug |
Enable debug mode. Default: false |
--trace |
Enable trace mode. Default: false |
--skip-tls-verification |
Skip TLS verification. Default: false |
--version |
Print version information and quit |
Shared Migration Options
These options apply to all migrations, regardless of the source.
| Flag | Description |
|---|---|
--migration.batch-size |
Migration batch size. Default: 50 |
--migration.restart |
Restart migration without resuming from offset. Default: false |
--migration.create-collection |
Create the collection if it doesn't exist. Default: true |
--migration.offsets-collection |
Collection to store migration offset. Default: "_migration_offsets" |
--migration.batch-delay |
Upsert delay (in miliseconds) between batches |
TLS Verfication
By default, the migration tool verifies TLS certificates when connecting to source and target databases. This means that the TLS certificates need to be signed by a trusted certificate authority (CA) known to the container image of the mgiration tool, and the hostname in the certificate needs to match the hostname of the database URL. If either of these conditions is not met, the migration will fail with a TLS verification error.
If you want to skip verification for any reason, you can use the --skip-tls-verification flag.
If you signed the certificate with your own CA, that is not trusted by default, you can mount the CA certificate into the container and specify it using the SSL_CERT_FILE environment variable like this:
docker run --net=host --rm -it \
-v /local/path/to/ca.crt:/etc/ssl/certs/ca-certificates.crt \
-e SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt \
registry.cloud.qdrant.io/library/qdrant-migration qdrant \
--source.url 'https://source-hostname:6334' \
--source.collection 'test' \
--target.url 'https://target-hostname:6334' \
--target.collection 'target-collection' \
--migration.batch-size 64
Access to Source or Target on Localhost
By default, the migration tool runs in a container. Depending on your container runtime, you may not be able to access services running on localhost of the host machine. If that is the case, refer to the documentation of your container runtime for how to access host services from within the container. E.g. for Docker for Desktop, you can use host.docker.internal instead of localhost in your connection URLs (more information).
Documentation
¶
There is no documentation for this package.
Source Files
Directories