image-syncer

image-syncer

image-syncer is a docker registry tools. With image-syncer you can synchronize docker images from some source registries to target registries, which include most popular public docker registry services.

English | 简体中文

Features

• Support for many-to-many registry synchronization
• Supports docker registry services based on Docker Registry V2 (e.g., Alibaba Cloud Container Registry Service, Docker Hub, Quay.io, Harbor, etc.)
• Network & Memory Only, doesn't rely on any large disk storage, fast synchronization
• Incremental Synchronization, ignore unchanged images automatically
• BloB-Level Concurrent Synchronization, adjustable goroutine numbers
• Automatic Retries of Failed Sync Tasks, to resolve the network problems (rate limit, etc.) while synchronizing
• Doesn't rely on Docker daemon or other programs

Usage

GitHub Action

You can use image-sync-action to try image-syncer online without paying for any machine resources.

Install image-syncer

You can download the latest binary release here

Compile Manually

go get github.com/AliyunContainerService/image-syncer
cd $GOPATH/github.com/AliyunContainerService/image-syncer

# This will create a binary file named image-syncer
make

Example

# Get usage information
./image-syncer -h

# With this command, configure file path is "./config.json", default target registry is "registry.cn-beijing.aliyuncs.com",
# default target namespace is "ruohe", 6 of goroutine numbers, every failed task will be retried 3 times.
./image-syncer --proc=6 --auth=./auth.json --images=./images.json --registry=registry.cn-beijing.aliyuncs.com --retries=3

Configure Files

Image-syncer supports --auth and --images flag for passing authentication file and image sync configuration file, both of which supports YAML and JSON format. Seperate authentication information is more flexible to reuse it in different sync missions.

The older version (< v1.2.0) of configuration file is still supported via --config flag, you can find the example in config.yaml and config.json.

Authentication file

Authentication file holds all the authentication information for each registry. For each registry (or namespace), there is a object which contains username and password. For each images sync rule in image sync configuration file, image-syncer will try to find a match in all the authentication information and use the best(longest) fit one. Access will be anonymous if no authentication information is found.

You can find the example in auth.yaml and auth.json, here we use auth.yaml for explaination:

quay.io: # This "registry" or "registry/namespace" string should be the same as registry or registry/namespace used below in image sync rules. And if an url match multiple objects, the "registry/namespace" string will actually be used.
  username: xxx
  password: xxxxxxxxx
  insecure: true # Optional, "insecure" field needs to be true if this registry is a http service, default value is false.
registry.cn-beijing.aliyuncs.com:
  username: xxx # Optional, if the value string is a format of "${env}" or "$env", use the "env" environment variables as username.
  password: xxxxxxxxx # Optional, if the value string is a format of "${env}" or "$env", use the "env" environment variables as password.
docker.io:
  username: "${env}"
  password: "$env"
quay.io/coreos:
  username: abc
  password: xxxxxxxxx
  insecure: true

Image sync configuration file

Image sync configuration file defines all the image synch rules. Each rule is a key/value pair, of which the key refers to "the source images url" and the value refers to "the destination images url". The source/destination images url is mostly the same with the url we use in docker pull/push commands, but still something different in the "tags and digest" part:

1. The source images url cannot be empty.
2. If the source images url contains no tags or digest, all the tags of source repository will be synced.
3. The source images url can have more than one tags, which should be seperated by comma, only the specified tags will be synced.
4. The source images url can at most one digest, and the destination url should only have no digest or the same digest at the same time.
5. If the destination url has no digest or tags, it means the source images will keep the same tags or digest after being synced.
6. The destination url can have more than one tags, the number of which must be the same with the tags in the source images url, then all the source images' tags will be changed to a new one (correspond from left to right).
7. If the destination url is empty, all the source images will be synced to the "default registry" (by command line parameter) and in the same repository with the source images url.

You can find the example in images.yaml and images.json, here we use images.yaml for explaination:

quay.io/coreos/kube-rbac-proxy: quay.io/ruohe/kube-rbac-proxy
quay.io/coreos/kube-rbac-proxy:v1.0: quay.io/ruohe/kube-rbac-proxy
quay.io/coreos/kube-rbac-proxy:v1.0,v2.0: quay.io/ruohe/kube-rbac-proxy
quay.io/coreos/kube-rbac-proxy@sha256:14b267eb38aa85fd12d0e168fffa2d8a6187ac53a14a0212b0d4fce8d729598c: quay.io/ruohe/kube-rbac-proxy

Parameters

-h  --help       Usage information

    --config     Set the path of config file, this file need to be created before starting synchronization, default
                 config file is at "current/working/directory/config.json". (This flag can be replaced with flag --auth
                 and --images which for better orgnization.)

    --auth       Set the path of authentication file, this file need to be created before starting synchronization, default
                 config file is at "current/working/directory/auth.json". This flag need to be pair used with --images.

    --images     Set the path of image rules file, this file need to be created before starting synchronization, default
                 config file is at "current/working/directory/images.json". This flag need to be pair used with --auth.

    --log        Set the path of log file, logs will be printed to Stderr by default

    --registry   Set default-registry, default-registry can also be set by environment variable "DEFAULT_REGISTRY",
                 if they are both set at the same time, "DEFAULT_REGISTRY" will not work at this synchronization.

    --proc       Number of goroutines, default value is 5

    --retries    Times to retry failed tasks, default value is 2, the retries of failed tasks will start after all the tasks
                 are executed once, this can resolve most occasional network problems during synchronization

    --os         OS list to filter source tags, not works for docker v2 schema1 media, takes no effect if empty

    --arch       Architecture list to filter source tags, takes no effect if empty

    --force      Force update manifest whether the destination manifest exists

FAQs

Frequently asked questions are listed in FAQs