bbb

bbb

A Go fork of boostedblob — a fast, concurrent CLI for working with local files, Azure Blob Storage (az://), Amazon S3 (s3://), Google Cloud Storage (gs://), Hugging Face (hf://), and Azure Container Registry artifacts (acr://).

Why a fork of boostedblob

1. Need a single binary for multi-platform support
2. Add login user and network debug logs

Installation

Download the latest release from the Releases page, or build from source:

go install github.com/tg123/bbb@latest

Docker

Images are published to GitHub Container Registry for linux/amd64 and linux/arm64. They are also published to Docker Hub when the repository's Docker Hub credentials are configured:

docker pull ghcr.io/tg123/bbb:latest
docker pull farmer1992/bbb:latest

The image is based on Alpine and, besides bbb, ships a small set of tools commonly needed when scripting around it (bash, coreutils, findutils, curl, wget, sed, jq, tar, gzip, zstd, ca-certificates, tzdata), so it can be used as a job/sidecar image rather than a bare binary. Extra packages can be added with apk add in a derived image.

Run a command:

docker run --rm -v "$PWD:/data" ghcr.io/tg123/bbb:latest ls az://myaccount/mycontainer/

Run a script that uses bbb (override the entrypoint):

docker run --rm -v "$PWD:/data" --entrypoint bash ghcr.io/tg123/bbb:latest \
  -c 'bbb lsr az://myaccount/mycontainer/ | head -n 10'

Credentials are picked up from the environment as usual, e.g.:

docker run --rm -e BBB_AZBLOB_ACCOUNTKEY -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY \
  -v "$PWD:/data" ghcr.io/tg123/bbb:latest cp az://myaccount/mycontainer/file.bin /data/

To use Azure CLI / managed identity based login, mount the host credentials, e.g. -v "$HOME/.azure:/root/.azure".

Supported Path Types

acr:// paths

An acr:// path addresses an OCI artifact in an Azure Container Registry (or any registry implementing the OCI distribution spec):

acr://<registry>                           # lists the registry's repositories
acr://<registry>/<repository>              # lists the repository's tags
acr://<registry>/<repository>:<tag>[/<file>]
acr://<registry>/<repository>@<digest>[/<file>]

A registry name without a dot is expanded to <name>.azurecr.io, and paths are printed back in that short form. A path names one of three things, each a directory of the next:

$ bbb ls acr://myregistry                 # repositories
acr://myregistry/models/llama
acr://myregistry/orng

$ bbb ls acr://myregistry/orng            # tags
acr://myregistry/orng:0.1.2030838
acr://myregistry/orng:latest

$ bbb ls acr://myregistry/orng:latest     # the artifact's files
acr://myregistry/orng:latest/weights.bin

Repository names are reported exactly as the registry holds them, slashes included: models/llama is one repository, not a directory containing another. A repository has no implicit :latest — a tag that need not exist — so reading or publishing needs one named. Recursive commands likewise need a tag, since walking a repository would mean resolving every tag it holds.

The "files" of an artifact are its layers; each layer's name comes from the standard org.opencontainers.image.title annotation, falling back to its digest (e.g. sha256-abc...) when the annotation is missing. Because a file name follows the tag or digest, a file can only be addressed on a path that specifies one.

Layer names are validated lexically before use: absolute paths, .. traversal, backslashes, colons, characters Windows forbids (<>"|?* and control characters), Windows reserved device names (NUL, CON.txt, …) and segments ending in a dot or space are rejected. A name that is not already canonical (a/./b, a//b, dir/) is refused rather than repaired — the registry chooses these names, so a rewrite would put the bytes somewhere the manifest never declared; a single leading ./ is trimmed, since it names the same file. A name that is an ancestor of another (a alongside a/b) is rejected too, since no filesystem can hold a file and a directory at one path, as are names differing only in case (A.txt and a.txt), which alias on Windows and macOS. Recursive remote-to-local copies and syncs create files relative to the requested destination root and reject symlink traversal outside it. Explicit single-file destination paths still use normal local path semantics, including following pre-existing symlinks. Extract untrusted artifacts into a fresh directory.

Layer contents are verified against the digest recorded in the manifest, so a corrupt registry or proxy cannot silently return different bytes.

An acr:// destination must identify an artifact tag, not an individual file or digest. Copying or syncing one local file or directory uploads each file as an OCI layer and publishes the tag only after every layer succeeds:

bbb cp ./llama-artifact/ acr://myregistry/models/llama:v1
bbb cp ./weights.bin acr://myregistry/models/weights:v1
bbb sync ./llama-artifact/ acr://myregistry/models/llama:v1

Use -f with cp to replace an existing tag. Without it the manifest is written with If-None-Match: * and the published digest is checked afterwards, so a tag already holding different content is refused; a tag that already holds exactly the artifact being published is reported as success, which is what lets --retry-count finish a push whose response was lost. Note that a registry is not obliged to honour that condition and the distribution API offers no other way to claim a tag conditionally: against one that ignores it, two publishers racing for the same tag can both write, and the after-the-fact check reports an error to whichever lost rather than preventing the overwrite. Serialise publication elsewhere if that can happen. sync always replaces the destination tag; its exclude filter controls which files become layers.

Authentication is resolved in this order:

1. BBB_ACR_USERNAME / BBB_ACR_PASSWORD (registry credentials or a token), together with BBB_ACR_REGISTRY naming the host or hosts they belong to. All three are required: one invocation can address several registries, and an Azure suffix is not a scope — anyone can own a *.azurecr.io registry, so a taskfile touching two of them would otherwise send credentials meant for one to the other.
2. Entra ID (Azure AD) via DefaultAzureCredential — Azure CLI login, service principal, managed identity, workload identity — exchanged for a registry token
3. The Docker keychain (~/.docker/config.json and credential helpers), so a prior docker login works for any registry, including anonymous pull

Because the Entra step posts a live Azure access token to the registry, it is only attempted for Azure Container Registry hosts (*.azurecr.io, *.azurecr.cn, *.azurecr.us). Any other registry — a private ghcr.io repository, say — goes straight to the Docker keychain rather than being offered your Azure credential. Add custom-domain ACR hosts to BBB_ACR_ENTRA_HOSTS to opt them in. The Resource Manager audience and the sign-in authority both follow the registry's cloud, so sovereign endpoints authenticate against their own. The exchange stops at an ACR refresh token, which is presented per request so the registry's challenge names the exact repository scope — the same flow docker uses after az acr login.

Container images

A container image stores its files inside its layers rather than as layers, so the naming above describes nothing useful for one: image layers carry no title, leaving only digests. bbb bridges that in two ways.

A manifest reached through an index is prefixed with its platform, so the members of a multi-platform image are told apart by os/arch instead of by digest. A layer that is a filesystem tarball is presented as a directory of its entries:

$ bbb ls acr://myregistry/orng:latest
acr://myregistry/orng:latest/darwin
acr://myregistry/orng:latest/linux
acr://myregistry/orng:latest/windows

$ bbb ls acr://myregistry/orng:latest/linux/amd64
acr://myregistry/orng:latest/linux/amd64/orng

$ bbb cp acr://myregistry/orng:latest/linux/amd64/orng ./orng

Listing a layer means transferring it, so it happens only when a path reaches inside one — listing the root of a multi-platform image is answered from the index alone. Each file read reopens its layer, so pulling one or two files out of an image costs one layer read each, while copying a whole image directory re-reads the layer per file; extract what you need rather than mirroring an image. Within a manifest, layers overlay: a later layer replaces what an earlier one wrote and .wh. whiteouts remove it (including whole subtrees, and everything at the root for .wh..wh..opq), so the listing reflects the image's final filesystem. Only regular files are listed; directories, symlinks and device nodes are not files that can be streamed. Entry names are validated as written rather than repaired, so an entry that would need cleaning is refused instead of being admitted under a harmless-looking name, and the merged names are held to the same collision rules as layer titles — an image containing both A.txt and a.txt is rejected rather than silently collapsed on a case-insensitive destination.

Permissions are not carried across — like every other bbb transfer, files are written 0644, so an extracted binary needs chmod +x. Zstd-compressed layers (application/vnd.oci.image.layer.v1.tar+zstd) are left as opaque blobs rather than expanded, so an image built with zstd compression lists its layers by digest as before. Non-distributable (foreign) layers, which Windows base images use, are expanded like any other — the blob is read from the registry by digest and the descriptor's URLs are never followed, so if your registry does not hold the blob the error says so and names where the manifest claims it lives; mirror it in first. A manifest declaring more than 1024 filesystem layers is refused, since applying a layer costs a pass over the ones below it; real images are far below that (Docker's own ceiling is 127). Artifacts published by bbb are unaffected by any of this: their layers are not tarballs and are never expanded.

Registries in another tenant

If the registry is not in your credential's home tenant, ACR rejects the token with unknown tenantId. bbb then opens a browser so you can sign in with an account that does have access, retries, and reports the tenant it turned out to be:

  Your Azure sign-in is not valid for "myregistry.azurecr.io".
  Opening a browser — choose an account in the registry's tenant.
  Signed in to tenant 8b9ebe14-... Set BBB_ACR_TENANT_MYREGISTRY_AZURECR_IO=8b9ebe14-... to skip this prompt.

The prompt comes after the rejection because an ACR endpoint never reveals its tenant — every WWW-Authenticate challenge carries only realm, service and scope, with no authorization_uri, and a registry in another tenant is invisible to Resource Manager until you already hold a token for it. This is the one place acr:// cannot behave like az://, which discovers a storage account's tenant up front and so can prompt before making a request.

Naming the tenant skips the failed attempt entirely:

# Per registry (host form, or just the registry short name)
export BBB_ACR_TENANT_MYREGISTRY_AZURECR_IO=<tenant-id>
export BBB_ACR_TENANT_MYREGISTRY=<tenant-id>

# Or as a default for every acr:// registry
export BBB_ACR_TENANT=<tenant-id>

With a tenant set, bbb mirrors az://: it uses your Azure CLI login for that tenant, verifies the token really was issued by it — az can hold a token only for your home tenant and return that whatever you asked for — and otherwise signs you in. A browser opens, falling back to a device code when there is no browser to open, as over SSH or inside WSL. Only one sign-in is opened per tenant no matter how many transfers are running, and a tenant discovered by signing in is reused for the rest of the run.

The CLI login is read through az itself, so AZURE_CONFIG_DIR selects which profile is used. Being signed in to a tenant is not the same as holding a usable token for it — a profile whose session has lapsed reports Status_InteractionRequired for every resource — so when the CLI cannot answer, bbb says so before opening a prompt:

  The Azure CLI has no usable token for tenant 8b9ebe14-...; `az login --tenant 8b9ebe14-...` would avoid this prompt.

Signing in requires a terminal, so a pipeline never blocks on a prompt it cannot answer; without one the rejection is reported along with the variable to set. BBB_ACR_NO_LOGIN=1 suppresses the prompt for an interactive shell that should never show one.

The sign-in lasts for the life of the process, so prefer az login --tenant <tenant-id> for repeated use: that persists, and bbb then takes the CLI path with no prompt at all.

Writing requires credentials with push access to the target repository.

The registry protocol is handled by go-containerregistry, so manifest resolution, blob transfer, digest verification and auth challenges follow the same well-tested implementation used by crane and ko. Registries on localhost and loopback addresses are contacted over plain HTTP automatically, which is what makes a local registry:2 container usable without extra configuration. A private (RFC1918) IP literal would also be downgraded to HTTP, so it must be opted in with BBB_ACR_INSECURE; address the registry by a name that resolves over HTTPS instead.

Multi-manifest artifacts are supported: when a reference resolves to an image index, the layers of every child manifest are merged into one file listing.

Global Flags

Debug logging example — use --loglevel debug to inspect DNS resolution and the Azure AD token issuer (iss), which is useful for diagnosing connectivity or authentication problems:

bbb --loglevel debug ls az://myaccount/mycontainer/

Example debug output (sensitive fields redacted):

time=... level=DEBUG msg="DNS lookup" host=myaccount.blob.core.windows.net addrs=["198.51.100.1"]
time=... level=DEBUG msg="Decoded JWT payload" payload="{\"aud\":\"https://storage.azure.com\",\"iss\":\"https://sts.windows.net/<tenant-id>/\",…}"

The DNS lookup line shows the resolved IP addresses for the storage account, and the Decoded JWT payload line contains the full token claims including iss (the token issuer) and aud (audience), letting you verify the correct identity and tenant are being used.

Warning: Debug output may include personally identifiable information such as tenant IDs, object IDs, and other token claims. Do not share debug logs publicly or paste them into tickets without redacting sensitive fields.

Environment Variables

Non-Interactive Authentication (AZURE_* Env Vars)

By default, when bbb cannot find shared keys or role-specific credentials it discovers the storage account's tenant and authenticates via the Azure CLI, falling back to an interactive browser login. For CI/CD and other headless environments, set the standard AZURE_* environment variables to authenticate non-interactively:

• When AZURE_TENANT_ID, AZURE_CLIENT_ID, and AZURE_CLIENT_SECRET are all set, bbb uses a service principal credential.
• AZURE_SUBSCRIPTION_ID is not required for Blob Storage data-plane authentication and is ignored.

These take effect before the interactive CLI/browser flow, so no browser popup is opened when they are configured. Single-endpoint commands (ls, cat, rm, etc.) reuse the SRC role internally, so the unprefixed AZURE_* vars (and SRC_AZURE_*) are honored for them too. For per-account scoping across tenants, use the SRC_ / DST_ prefixed variables described below.

Multi-Tenant / Multi-Account Authentication (SRC_ / DST_ Env Vars)

When copying or syncing between Azure Storage accounts in different tenants (or using different credentials), prefix any standard Azure identity environment variable with SRC_ or DST_ to scope it to source or destination accounts respectively.

The unprefixed AZURE_* variables act as shared defaults: a plain AZURE_xxx is interpreted as if it were set for both SRC_AZURE_xxx and DST_AZURE_xxx, and the role-prefixed variant overrides it when present. This lets a single set of AZURE_* vars authenticate both sides while still allowing per-role overrides.

bbb uses DefaultAzureCredential under the hood, so all credential types are supported: service principal (secret or certificate), workload identity (OIDC / AKS), managed identity, and Azure CLI.

Supported env vars — prefix with SRC_ or DST_:

Example — service principal per tenant:

# Source tenant credentials
export SRC_AZURE_TENANT_ID=<tenant-a>
export SRC_AZURE_CLIENT_ID=<sp-a-id>
export SRC_AZURE_CLIENT_SECRET=<sp-a-secret>

# Destination tenant credentials
export DST_AZURE_TENANT_ID=<tenant-b>
export DST_AZURE_CLIENT_ID=<sp-b-id>
export DST_AZURE_CLIENT_SECRET=<sp-b-secret>

bbb cp az://src-account/container/ az://dst-account/container/

Example — shared key per account:

export SRC_BBB_AZBLOB_ACCOUNTKEY=<key-for-source>
export DST_BBB_AZBLOB_ACCOUNTKEY=<key-for-destination>

bbb sync az://src-account/data/ az://dst-account/data/

Credential resolution order (first match wins):

1. Shared key (SRC_BBB_AZBLOB_ACCOUNTKEY / DST_BBB_AZBLOB_ACCOUNTKEY, or BBB_AZBLOB_ACCOUNTKEY)
2. Role env credential via DefaultAzureCredential — SRC_AZURE_* / DST_AZURE_*, falling back to unprefixed AZURE_* defaults. Accounts without an explicit role (single-endpoint commands like ls/cat/rm) reuse the SRC role, so unprefixed AZURE_* (and SRC_AZURE_*) are honored for them.
3. Tenant-specific AzureCLI credential (auto-discovered from storage endpoint)
4. Interactive browser login (fallback)

Amazon S3 (s3://) Authentication and Configuration

S3 paths use the standard AWS SDK credential and region resolution chain, so any mechanism the AWS CLI/SDK understands works out of the box: environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN), shared config/credential files (~/.aws/credentials, AWS_PROFILE), SSO, and IAM instance/task roles.

Example using an S3-compatible server (MinIO):

export AWS_ACCESS_KEY_ID=minioadmin
export AWS_SECRET_ACCESS_KEY=minioadmin
export BBB_S3_ENDPOINT=http://127.0.0.1:9000
export BBB_S3_FORCE_PATH_STYLE=1

bbb cp ./data.bin s3://mybucket/data.bin
bbb ls s3://mybucket/

S3→S3 copies (within the same account/endpoint) use server-side CopyObject — with a multipart UploadPartCopy fallback for objects larger than 5 GiB — and never stream bytes through the client.

Cloudflare R2

Cloudflare R2 is accessed through the same s3:// paths. Set BBB_R2_ACCOUNT_ID (with an R2 API token's access key pair) and bbb configures the R2 endpoint, the auto region, and the R2-specific protocol quirks automatically:

export AWS_ACCESS_KEY_ID=<r2-access-key-id>
export AWS_SECRET_ACCESS_KEY=<r2-secret-access-key>
export BBB_R2_ACCOUNT_ID=<cloudflare-account-id>

bbb cp ./data.bin s3://mybucket/data.bin
bbb ls s3://mybucket/

Notes:

• BBB_S3_ENDPOINT still takes precedence, so jurisdiction-specific endpoints (e.g. https://<account-id>.eu.r2.cloudflarestorage.com) can be set directly; they are detected as R2 as well.
• R2 does not implement AWS flexible checksums, so bbb requests checksums only when required instead of the SDK default (when_supported).
• bbb s3 mkbucket omits the LocationConstraint on R2, which only accepts the pseudo-region auto for request signing.

Google Cloud Storage (gs://) Authentication and Configuration

GCS paths use Google's Application Default Credentials, so anything gcloud understands works out of the box: a service account key file pointed at by GOOGLE_APPLICATION_CREDENTIALS, gcloud auth application-default login credentials, or the metadata server / workload identity on GCE, GKE and Cloud Run.

Example using the fake-gcs-server emulator:

docker run -d -p 4443:4443 fsouza/fake-gcs-server \
  -scheme http -port 4443 -backend memory \
  -public-host localhost:4443 -external-url http://localhost:4443

export BBB_GS_ENDPOINT=http://localhost:4443

bbb gs mkbucket gs://mybucket
bbb cp ./data.bin gs://mybucket/data.bin
bbb ls gs://mybucket/

GCS→GCS copies use the server-side rewrite API — which handles objects of any size — and never stream bytes through the client.

Copy task streams to GCS share the --concurrency budget between source expansion and transfers. Once expansion finishes, transfers can use the full budget.

GCS tree copies and syncs reject overlapping source and destination prefixes in the same bucket before listing or copying objects. This includes equal prefixes and either prefix containing the other.

Downloads of gzip-encoded GCS objects use one full read instead of parallel ranges, preserving normal decompression without using the stored size to limit the decoded output.

Recursive remote-to-local copies and syncs collect and validate the selected file names before starting transfers. Nonportable names, case/Unicode aliases, and file/directory conflicts are rejected rather than overwriting the same local file. This preflight uses memory proportional to the listing size. GCS object names remain opaque for GCS destinations; copies to backends that normalize paths reject names they cannot preserve, such as ../file or a//file, rather than writing outside the requested prefix or renaming them.

BBB_DNS_SERVER

When set, bbb sends every DNS query to the given DNS server(s) instead of the system resolver configuration (/etc/resolv.conf, systemd-resolved, ...). This is useful when the host resolver is broken, slow, or returns endpoints you do not want to use.

BBB_DNS_SERVER=8.8.8.8 bbb cp ./data/ az://myaccount/mycontainer/data/

# with a fallback server and a custom port
BBB_DNS_SERVER=10.0.0.53,1.1.1.1:5353 bbb ls az://myaccount/mycontainer/

Servers must be given as IP addresses (an unresolvable name would need a resolver itself), optionally with a port — port 53 is assumed when omitted. IPv6 literals may be written bare (::1) or bracketed together with a port ([2001:db8::1]:5353). Servers are tried in the order given; the next one is used when a server cannot be reached.

The override applies to all DNS lookups in the bbb process (it replaces the process-wide resolver), including Azure SDK data-plane requests, OAuth token calls, Hugging Face, S3 and Google Cloud Storage traffic. It composes with BBB_DNS_CACHE / BBB_DNS_PIN, which cache or pin the results returned by the configured server.

Caveats:

• AzureCLICredential shells out to the az binary; DNS resolution in that child process uses the system resolver and is not affected.
• Setting BBB_DNS_SERVER implicitly switches bbb to Go's built-in DNS client (equivalent to GODEBUG=netdns=go), since the cgo resolver cannot be pointed at a custom server. You do not need to set GODEBUG yourself, and note that GODEBUG=netdns=go alone only picks the Go resolver — it cannot choose which DNS server to query.

BBB_DNS_CACHE

When enabled, bbb caches DNS resolution results in memory so that repeated connections to the same hostname (e.g. an Azure Storage endpoint) skip the DNS lookup. Cached entries expire after 5 minutes.

BBB_DNS_CACHE=1 bbb cp ./data/ az://myaccount/mycontainer/data/

The cache is installed on the shared HTTP transport used by all outbound traffic in the process, including the Azure SDK (data-plane requests, user-delegation-key / UDC acquisition, and OAuth token calls to login.microsoftonline.com) and Hugging Face API calls.

Caveats:

• DNS records that change during the TTL window (e.g. IP rotations) will not be picked up until the cached entry expires.
• Because cached addresses are dialled as IP literals, Go's standard Happy Eyeballs (RFC 6555) connection racing is bypassed. For Azure Blob Storage endpoints (typically single-stack) this has no practical impact.
• az login (when falling back to AzureCLICredential) shells out to the az binary; DNS resolution for that subprocess happens in the child and is not affected by these variables.

BBB_DNS_PIN

When enabled, bbb pins every hostname to a single IP address. If DNS returns multiple addresses, only the first reachable address is used for all connections to that host. This implicitly enables BBB_DNS_CACHE with unlimited TTL (BBB_DNS_CACHE_TTL is ignored). Pinning can help avoid 403 errors from services that tie authentication tokens to a specific endpoint IP.

BBB_DNS_PIN=1 bbb cp ./data/ az://myaccount/mycontainer/data/

The pin applies to all outbound HTTP traffic in the process, including Azure SDK data-plane requests, UDC acquisition, and OAuth token calls, as well as Hugging Face API calls. This matches the semantics of a /etc/hosts override but limited to the bbb process.

Caveats:

• Pinned entries never refresh. Long-lived processes will not pick up DNS or IP rotations and may require a restart (or disabling pinning) to recover if the pinned IP becomes unreachable.
• If the first resolved address is unreachable (e.g. an IPv6 address in an IPv4-only environment), bbb will try the remaining addresses and pin to the first one that successfully connects.
• AzureCLICredential shells out to the az binary; its DNS resolution runs in the child process and is not covered by the pin. MSAL-cached tokens on disk are also unaffected — occasional re-authentication may still occur.

Taskfile

A taskfile is a plain-text file with one src dst pair per line, separated by whitespace. Empty lines are ignored.

Note: Paths containing spaces are not supported in the taskfile format because fields are split on whitespace. Use cp or sync with positional arguments instead for such paths.

Example tasks.txt:

./data/model.bin   az://myaccount/mycontainer/models/model.bin
./data/config.json az://myaccount/mycontainer/models/config.json
./data/vocab.txt   az://myaccount/mycontainer/models/vocab.txt

Use --taskfile to pass the file to cp or sync. Use - to read from stdin. Pairs are consumed as a continuous stream, so a long-running producer piped into --taskfile - is processed incrementally instead of waiting for EOF. The exception is cp pairs with an acr:// source or destination: these are buffered until EOF so cross-task artifact conflicts can be rejected before any ACR task starts. Other pairs in the same stream still start as they arrive.

# Copy all pairs listed in the taskfile
bbb cp --taskfile tasks.txt

# Sync all pairs listed in the taskfile
bbb sync --taskfile tasks.txt

# Pipe pairs from another command
find ./models -name '*.bin' | awk '{print $0, "az://myaccount/mycontainer/"$0}' | bbb cp --taskfile -

State file

A state file tracks completed work so interrupted operations can be resumed. Pass --state to cp or sync and re-run the same command after a failure — already-finished items are skipped automatically.

# Start a large copy with crash recovery
bbb cp --taskfile tasks.txt --state copy.state

# If the process is interrupted, re-run the exact same command.
# Completed files are skipped; only remaining work is executed.
bbb cp --taskfile tasks.txt --state copy.state

--state also works without --taskfile:

bbb cp --state copy.state ./huge-dataset/ az://myaccount/mycontainer/dataset/

The state file is a plain-text append-only log. Each successfully copied file is recorded as src -> dst, and when all files in a taskfile pair are finished the pair is marked complete with a TASK\t prefix so the entire pair can be skipped on resume:

./data/model.bin -> az://myaccount/mycontainer/models/model.bin
./data/config.json -> az://myaccount/mycontainer/models/config.json
TASK	./data/model.bin -> az://myaccount/mycontainer/models/model.bin

Commands

ls — List directory contents

List files and directories at a given path.

bbb ls [flags] [path]

Examples:

# List local directory
bbb ls /tmp/data/

# Long listing of an Azure Blob container
bbb ls -l az://myaccount/mycontainer/

# List Hugging Face repo files with relative paths
bbb ls -s hf://meta-llama/Llama-2-7b/

# List the files of an OCI artifact in Azure Container Registry
bbb ls acr://myregistry/models/llama:v1

ll — Long listing (alias for ls -l)

bbb ll [flags] [path]

Example:

# Show sizes and timestamps of blobs in a container
bbb ll az://myaccount/mycontainer/models/

du — Estimate recursive file space usage

Recursively prints human-readable cumulative usage for each directory, followed by the root. For object stores, usage is based on apparent blob size.

bbb du -s --concurrency 32 az://myaccount/mycontainer/data/

lstree — Recursively list all files

Aliases: lsr

Recursively lists all files (not directories) under a path, with a summary of total count and size.

bbb lstree [flags] [path]

Examples:

# Recursively list all files under a local directory
bbb lstree /home/user/project/

# Machine-readable recursive listing of a blob container
bbb lstree --machine az://myaccount/mycontainer/data/

llr — Long recursive file list

Equivalent to lstree -l.

bbb llr [flags] [path]

Example:

bbb llr az://myaccount/mycontainer/

# Count a large folder without printing every file
bbb llr -s --concurrency 32 az://myaccount/mycontainer/

cat — Print file contents to stdout

bbb cat path [path ...]

Examples:

# Print a local file
bbb cat /tmp/config.yaml

# Print a blob from Azure
bbb cat az://myaccount/mycontainer/config.json

# Print multiple files
bbb cat file1.txt file2.txt

touch — Create or ensure file exists

Creates empty files if they don't exist. For local files, also updates the modification timestamp. For Azure blobs, creates an empty blob only when it doesn't already exist — it does not update the timestamp on existing blobs.

bbb touch path [path ...]

Examples:

# Create an empty local file
bbb touch /tmp/newfile.txt

# Touch a blob in Azure
bbb touch az://myaccount/mycontainer/marker.txt

cp — Copy files or directories

Aliases: cpr, cptree

Copy one or more source files/directories to a destination. Supports local and Azure Blob paths in any combination. Hugging Face (hf://) is source-only. Azure Container Registry (acr://) can be a source, or a destination when pushing one local file or directory as an OCI artifact.

bbb cp [flags] src [src ...] dst

Examples:

# Copy a local file to Azure Blob Storage
bbb cp ./model.bin az://myaccount/mycontainer/models/model.bin

# Copy an entire directory to Azure
bbb cp ./data/ az://myaccount/mycontainer/data/

# Download from Azure to local
bbb cp az://myaccount/mycontainer/results/ ./results/

# Server-side copy between Azure containers
bbb cp az://myaccount/src-container/data/ az://myaccount/dst-container/data/

# Download from Hugging Face (hf:// is source-only)
bbb cp hf://meta-llama/Llama-2-7b/ ./llama-model/

# Download an OCI artifact from Azure Container Registry
bbb cp acr://myregistry/models/llama:v1 ./llama-artifact/

# Download a single file from an artifact
bbb cp acr://myregistry/models/llama:v1/weights.bin ./weights.bin

# Push a local directory as one OCI artifact
bbb cp ./llama-artifact/ acr://myregistry/models/llama:v1

# Copy multiple sources to one destination
bbb cp file1.txt file2.txt az://myaccount/mycontainer/uploads/

# Copy with higher concurrency and retries
bbb cp --concurrency 16 --retry-count 3 ./big-dataset/ az://myaccount/mycontainer/dataset/

Taskfile Mode

Use --taskfile to provide a file of src dst pairs (one per line). See Taskfile for the file format.

# From a file
bbb cp --taskfile tasks.txt

# From stdin (pipe)
echo "local.txt az://myaccount/c/remote.txt" | bbb cp --taskfile -

Crash Recovery with State File

Use --state to resume interrupted copies. See State file for details.

# First run — starts copying and records progress
bbb cp --taskfile tasks.txt --state copy.state

# If interrupted, re-run the same command — already-copied files are skipped
bbb cp --taskfile tasks.txt --state copy.state

rm — Remove files

bbb rm [flags] path [path ...]

Examples:

# Remove a local file
bbb rm /tmp/old-file.txt

# Remove a blob from Azure
bbb rm az://myaccount/mycontainer/old-model.bin

# Force-remove (no error if missing)
bbb rm -f az://myaccount/mycontainer/maybe-exists.txt

rmtree — Remove a directory tree

Aliases: rmr

Recursively deletes an entire directory and all of its contents.

bbb rmtree [flags] path

Examples:

# Remove a local directory tree
bbb rmtree /tmp/scratch/

# Remove an Azure Blob virtual directory
bbb rmtree az://myaccount/mycontainer/old-experiment/

sync — Synchronise two directory trees

Unidirectional sync: copies new and updated files from source to destination.

bbb sync [flags] src dst

Server-side copy: For Azure→Azure and Hugging Face→Azure syncs, bbb instructs Azure Blob Storage to pull each file directly from the source (Az→Az) or from the Hugging Face CDN URL (HF→Az), so the data never transits the client. If a Hugging Face file can't be copied server-side, bbb automatically falls back to streaming it through the client.

--delete support: The delete phase is only implemented for local→local syncs. Any sync involving a remote backend on either side — including remote→local, such as az://… -> ./local — copies the source but keeps stale destination files, and prints a warning saying so. With an acr:// source, --delete is rejected outright. An acr:// destination needs no delete phase: the push replaces the tag with a manifest listing exactly the selected source files, so an empty or fully excluded source publishes an empty artifact rather than leaving the previous contents in place.

Examples:

# Sync a local directory to Azure
bbb sync ./data/ az://myaccount/mycontainer/data/

# Sync from Azure to local
bbb sync az://myaccount/mycontainer/data/ ./local-data/

# Mirror (delete extra files at destination; local→local only)
bbb sync --delete ./source/ ./dest/

# Preview changes without applying
bbb sync --dry-run ./data/ az://myaccount/mycontainer/data/

# Exclude certain file patterns
bbb sync --exclude '\.tmp$' ./project/ az://myaccount/mycontainer/project/

# Sync with taskfile and crash recovery (see Taskfile and State file sections above)
bbb sync --taskfile tasks.txt --state sync.state

md5sum — Compute MD5 checksums

bbb md5sum path [path ...]

Examples:

# Checksum a local file
bbb md5sum ./model.bin

# Checksum an Azure blob
bbb md5sum az://myaccount/mycontainer/model.bin

# Checksum multiple files
bbb md5sum file1.txt file2.txt file3.txt

share — Print browser-accessible link for a file

bbb share path

For Azure Blob paths, prints an Azure Portal link and a direct blob URL. For local files, prints a file:// URL.

Examples:

# Get a shareable link for an Azure blob
bbb share az://myaccount/mycontainer/report.pdf
# Output:
#   Azure Portal: https://portal.azure.com/#blade/...
#   Direct Blob (if public): https://myaccount.blob.core.windows.net/mycontainer/report.pdf

# Get a file:// link for a local file
bbb share ./report.pdf

edit — Open a file in your editor (local only)

Opens a local file in the editor specified by the $EDITOR environment variable (defaults to vi). Creates the file and parent directories if they don't exist. Remote paths (az://, hf://) are not supported.

bbb edit path

Example:

# Edit a local config file
bbb edit /etc/myapp/config.yaml

az mkcontainer — Create an Azure Blob container

bbb az mkcontainer az://account/container

Example:

# Create a new Azure Blob container
bbb az mkcontainer az://myaccount/newcontainer

s3 mkbucket — Create an S3 bucket

bbb s3 mkbucket s3://bucket

Example:

# Create a new S3 bucket
bbb s3 mkbucket s3://newbucket

gs mkbucket — Create a Google Cloud Storage bucket

bbb gs mkbucket gs://bucket

Bucket names are globally unique. An existing bucket returns a conflict, even when it is readable by the caller; read access does not prove project ownership.

Example:

# Create a new GCS bucket (needs BBB_GS_PROJECT or GOOGLE_CLOUD_PROJECT)
bbb gs mkbucket gs://newbucket

Benchmark

The Benchmark workflow compares bbb's single-file upload and download throughput against azcopy and boostedblob (the upstream Python bbb, referred to as py-bbb). It guards the parallel transfer paths introduced in #87 and #89 against regressions.

Because boostedblob hardcodes the https://{account}.blob.core.windows.net endpoint (no port or host override), the benchmark serves the Azurite emulator at that exact host. The whole benchmark runs inside Docker Compose (internal/benchmark), mirroring the E2E suite: Azurite runs as its own service (the official image), and the benchmark container generates a local CA (trusted inside the container), points {account}.blob.core.windows.net at the emulator over TLS on port 443, builds bbb, and runs all three tools against the same emulator. Because everything is containerised, the benchmark needs no host privileges, no secrets and no real Azure account, so it runs on every pull request.

The workflow runs on every pull request, on pushes to main, and on demand via Run workflow (workflow_dispatch), where you can set the test-file size, the number of runs, and a fail_factor that fails the job when bbb is slower than the fastest other tool by more than that factor (default 1.05, i.e. a 5% regression gate; set it blank for report-only). Results are written to the job summary as a table.

The emulator is CPU/loopback-bound, so the numbers reflect client-side overhead rather than real network throughput; fail_factor defaults to 1.05 (a 5% gate) and can be set blank to report only.

To run it locally you only need Docker:

cd internal/benchmark
docker compose up --build --abort-on-container-exit --exit-code-from benchmark
# Optional overrides: BENCH_SIZE_MB, BENCH_RUNS, BENCH_FAIL_FACTOR

Real Azure results

The same harness can be pointed at two real Azure Storage accounts:

export BENCH_SRC_ACCOUNT=<src-account>
export BENCH_DST_ACCOUNT=<dst-account>
export BENCH_CONTAINER=bench
export BENCH_SIZES_MB=10,100,500,1000,5000,10000   # comma-separated
export BENCH_RUNS=3
export BENCH_FAIL_FACTOR=99                         # disable the regression gate

go test -count=1 -timeout 0 -v -run TestBenchmark ./internal/benchmark/

Sample numbers (best of 3 runs, concurrency 32, seconds — lower is better).

Test environment:

• Client node: Standard_E32ads_v5 (32 vCPU AMD EPYC 7763, 256 GiB RAM), AKS in southcentralus.
• Source account: southcentralus (same region as the client, used for upload and download).
• Destination account: uksouth (S2S destination — cross-region copy).
• bbb commit at time of run: 55f4e44 (PR #99 head).
• Tool versions: bbb (this repo), boostedblob 10.0.0, azcopy 10.32.2.

Upload (s)

Download (s)

Server-to-server copy (s)

Network conditions vary, so treat these as representative rather than absolute; the Azurite-based CI run is the regression gate.

License

See LICENSE for details.