gantry

Gantry deployment artifacts

This directory carries the Helm chart and operator-facing rendered manifests needed to roll out the Gantry agent as a Kubernetes DaemonSet.

Files

The Helm chart under chart/ is the source of truth for resources shared by standalone and operator-managed installations. make gantry-manifests renders the internal operator profile into deploy/gantry/rendered/; the Unbounded operator embeds those files and applies its own image, ConfigMap, and Lease ownership semantics. The target also renders the standalone node configurator and examples used by development and benchmark tooling.

Installation paths

• Operator-managed clusters use the manifests embedded in the unbounded-operator binary. The operator never runs Helm.
• Clusters without the operator install the released OCI chart. The chart continuously reconciles its route under /etc/containerd/certs.d on every selected node. Containerd must already be configured to read that directory; the chart does not edit or restart containerd.

The paths are mutually exclusive. PriorityClass/gantry-low records the active manager, and both installers reject ownership by the other path.

helm upgrade --install gantry oci://ghcr.io/azure/charts/gantry \
   --version <release-without-v> \
   --namespace gantry-system \
   --create-namespace \
   --set image.digest=sha256:<gantry-image-digest> \
   --set gantry.upstreamRegistries[0].name=registry.example.com \
   --set gantry.upstreamRegistries[0].endpoint=https://registry.example.com

The container image is built from images/gantry/Containerfile via make image-gantry-local (or make image-gantry-push to push).

Chair sizing and upgrades

gantry.chairCount controls the maximum active chair pool and gantry.chairSeedCount controls the per-digest seed cohort at a full pool. Both default to 64 chairs and 8 seeds. The counts have no configured maximum, but both must be positive and the seed count cannot exceed the chair count. The active pool is also bounded by the Gantry DaemonSet's desired scheduled pod count because one agent can hold at most one chair. The previous chair_holder_count, GANTRY_CHAIR_HOLDER_COUNT, and --chair-holder-count names remain accepted as deprecated aliases.

For example:

helm upgrade gantry oci://ghcr.io/azure/charts/gantry \
   --version <release-without-v> \
   --namespace gantry-system \
   --reuse-values \
   --set gantry.chairCount=128 \
   --set gantry.chairSeedCount=100

Increasing the chair count preserves every existing Lease and holder. Agents create and claim the additional numbered Leases through the normal election path until the new target is reached. Decreasing it makes holders at IDs above the new count vacate; the empty Lease objects may remain and are ignored. Chair Leases carry Helm's keep resource policy so a reduction does not delete coordination state during a rolling update. This also leaves the Leases behind on uninstall; delete them explicitly only after every Gantry pod has stopped.

Releases that only understand the original 64-chair set reject higher-numbered Lease names. Upgrading from such a release therefore requires two stages:

1. Upgrade every Gantry pod to the count-aware image while keeping gantry.chairCount=64.
2. After that rollout completes, raise gantry.chairCount and gantry.chairSeedCount.

Do not combine these stages in one Helm upgrade. A new pod can create a higher-numbered Lease while old pods are still running, which makes the old pods reject their next chair snapshot.

Metrics

Each agent serves Prometheus metrics on port 9095 (/metrics). Set metricsService.enabled=true to render Service/gantry-metrics, a headless Service over the agents that also lists NotReady agents. Use metricsService.annotations for annotation-driven scrapers, for example a Datadog endpoints check that scrapes every agent from its own node:

metricsService:
  enabled: true
  annotations:
    ad.datadoghq.com/endpoints.checks: |
      {"openmetrics": {"init_config": {}, "instances": [{
        "openmetrics_endpoint": "http://%%host%%:9095/metrics",
        "namespace": "gantry", "metrics": [".*"]}]}}

## Agent rollout

`updateStrategy.rollingUpdate.maxUnavailable` sets the agent DaemonSet's
rolling-update step as a positive count or percentage. The default `50%`
replaces half of the agents at once; lower it on large clusters to bound how
many nodes lose their local mirror during a rollout:

```sh
--set-string updateStrategy.rollingUpdate.maxUnavailable=10%

Peer transfer tuning

gantry.peerFetchTimeout (default 15m) caps one complete peer blob request, including streaming the body. It is a safety ceiling, not a stall detector: the HTTP/2 transport separately probes connections after 10s without reads. A value much lower than a large layer's transfer time makes requesters abandon progressing transfers and switch providers.

gantry.transferMaxConcurrentServes (default 10) caps concurrent peer blob serves per agent. Requests over the cap receive 429 so the requester re-discovers another provider. 0 means unlimited.

Use gantry.config only when a setting has no dedicated value; it replaces the entire generated configuration.

Operator Profile

# Render the profile embedded by unbounded-operator.
make gantry-manifests

# Validate and package the standalone chart.
make gantry-chart-lint
make gantry-chart-package GANTRY_CHART_VERSION=0.1.0 GANTRY_CHART_APP_VERSION=v0.1.0

Building the image locally

# Single-arch into local container engine:
make image-gantry-local

# Build and push to $(CONTAINER_REGISTRY):
make image-gantry-push CONTAINER_REGISTRY=ghcr.io/your-org

# Explicit tag:
make image-gantry-local VERSION=v0.6.0

Per-node containerd setup

Nodes provisioned by unbounded-agent already have containerd configured to read /etc/containerd/certs.d and carry the managed default Gantry mirror entry in /etc/containerd/certs.d/_default/hosts.toml. On those nodes, install the Gantry DaemonSet normally; the mirror activates when the pod starts listening on 127.0.0.1:5000.

ACR attachment is separate from deployment and preserves existing authentication by default. For operator-managed AKS Flex clusters, see the ACR attachment guide for commands, managed-auth opt-in, compatible versions, and limitations.

Standalone Helm installations run DaemonSet/gantry-containerd-config. Its resident reconciler checks its hosts.toml files every five seconds and atomically restores chart-owned payloads when they are missing or different, including after a node upgrade resets host configuration. The reconciler saves the last managed contents under /etc/containerd/certs.d/.gantry-managed/ so registry removal, renaming, and routing-mode changes can remove stale files even after a pod restart. Stale-route cleanup and graceful shutdown remove a file only when it still matches those saved contents; modified and unowned files are preserved. Routes left by older versions without saved ownership records are not removed automatically. Set nodeConfig.enabled=false when another node-management system owns these files.

The default profile owns _default/hosts.toml. Artifact Streaming instead owns one registry-specific <registry-host>/hosts.toml for each configured upstream, leaving AKS's _default/hosts.toml unchanged.

Externally managed installations can instead drop a registry-specific hosts.toml at:

/etc/containerd/certs.d/<registry-host>/hosts.toml

derived from hosts.toml.template (substitute ${REGISTRY_SERVER} with the registry's https://... URL). containerd reloads certs.d on its own; no restart needed.

ACR Artifact Streaming

This integration consumes the existing ACR OverlayBD v0.1.0 artifact contract without running the OverlayBD daemon or snapshotter. Enable it on Linux nodes that support NBD, ext4, EROFS when required by the image, and shared mount propagation.

For a standalone Helm installation, enable the Gantry range endpoint and its host configurator together. Point Gantry's OCI upstream at the node-local AKS Artifact Streaming mirror and select only the streaming node pool:

nodeSelector:
   gantry-streaming: "true"

gantry:
   artifactStreaming:
      enabled: true
   upstreamRegistries:
      - name: <registry>.azurecr.io
        endpoint: http://127.0.0.1:8578?ns=<registry>.azurecr.io

overlaybdConfig:
   enabled: true
   nodeSelector:
      gantry-streaming: "true"

In this mode Gantry uses host networking so it can reach the loopback-only AKS mirror. Containerd resolves through Gantry on port 5000; Gantry forwards OCI requests to the AKS mirror on port 8578, preserving its ns query. The AKS mirror returns the OverlayBD manifest, which triggers Gantry's complete-layer chair pulls. The Gantry snapshotter reads sealed LSMT layers through the same Gantry listener under /blobs/, where the source order remains local, peer, then signed ACR origin.

The adapter accepts raw LSMT, raw ZFile, and tar/PAX-wrapped overlaybd.commit payloads. ZFile metadata and CRC32C checksums are validated before exposing the image; LZ4 and Zstandard blocks are fetched and decompressed on demand with bounded, coalesced range reads. Compression dictionaries, compressed indexes, and other compression algorithms are rejected explicitly. The adapter does not download or decompress the entire layer to mount it.

Gantry prefetches mount-critical head and tail regions when an ACR streaming manifest is served or committed by a background peer pull. Overlapping callbacks for the same registry, repository, and manifest share one bounded prefetch job. This also warms metadata when containerd reuses an already-cached manifest without requesting its body again. Prefetch completion logs report the manifest, duration, and number of cached regions; incomplete warmups are reported separately.

AKS initially selects the overlaybd snapshotter on these nodes. Gantry keeps that native plugin registered, adds a distinct gantry-overlaybd proxy, and switches CRI image unpacking to the new proxy. The adapter uses a private in-process overlayfs snapshotter under its state directory for ordinary OCI layers; only layers with the complete ACR OverlayBD contract use NBD. The fallback must not call back into containerd's snapshot service: the proxy's garbage collector would otherwise see independently managed host snapshots.

The registry-specific containerd file lists the AKS mirror after Gantry. If Gantry is unavailable, containerd continues through port 8578 with Artifact Streaming but without Gantry peer reuse.

The adapter DaemonSet runs gantry snapshotter beside the host configurator. The snapshotter owns both its remote metadata and its ordinary-image fallback state under /var/lib/gantry/snapshotter; it does not mount the host containerd socket or reuse the host's snapshot metadata. The adapter DaemonSet uses OnDelete: a Helm or operator image update does not replace its pods automatically because the snapshotter owns live NBD devices. For each affected node, cordon and drain all workloads using the Gantry snapshotter before deleting that node's adapter pod. Bootstrap its replacement with native overlayfs, verify the new adapter is ready, restore gantry-overlaybd for CRI image unpacking, and only then uncordon the node. The adapter cannot prepare its own replacement container through its socket while that adapter is stopped; pre-pulling the replacement image alone does not remove this dependency. The configurator waits for /artifact-streaming/readyz on the listener selected by overlaybdConfig.gantryEndpoint and for the host-visible snapshotter socket, loads the nbd module, and verifies device capacity. A custom socket filename must remain directly under the shared /run/gantry-overlaybd directory. It then snapshots the current containerd configuration under /var/lib/gantry/snapshotter-config, enables /etc/containerd/conf.d/*.toml through structured TOML, and writes a last-wins drop-in that registers proxy_plugins.gantry-overlaybd and selects it for CRI image unpacking. Containerd restarts when effective configuration changes or when a previous activation failed. A pending-activation marker survives apply or restore failures; retries clear it only after a successful restart and service health check. Unchanged configuration still requires an active containerd service before the configurator reports readiness. A second writer is never overwritten: if either managed file differs from the recorded bytes, apply or rollback preserves it and reports the conflict.

For an operator-managed installation, opt in through every participating Site using the same selector. This path assumes unbounded-agent or another node manager already routes containerd through Gantry. Use the standalone Helm chart on stock AKS nodes so the registry-specific route is installed alongside the snapshotter configuration.

apiVersion: unbounded-cloud.io/v1alpha3
kind: Site
metadata:
   name: <site-name>
spec:
   components:
      gantry:
         enabled: true
         artifactStreaming:
            enabled: true
            nodeSelector:
               kubernetes.azure.com/agentpool: <streaming-node-pool>

The operator rejects an empty selector, a Site that enables Artifact Streaming while disabling Gantry, or conflicting selectors across Sites.

Artifact Streaming rollback

Drain workloads using active Gantry snapshotter devices before disabling the operator setting. The operator issues deletion of DaemonSet/gantry-overlaybd-config before applying the non-streaming Gantry DaemonSet, and the configurator's preStop restores the original host configuration when it still owns the current value. Kubernetes deletion and pod termination are asynchronous, which is why workloads must be drained before this transition.

For standalone Helm, keep the Gantry endpoint available during restoration:

helm upgrade gantry oci://ghcr.io/azure/charts/gantry \
   --reuse-values \
   --set overlaybdConfig.enabled=false \
   --set gantry.artifactStreaming.enabled=true \
   --wait

kubectl -n gantry-system wait --for=delete \
   daemonset/gantry-overlaybd-config --timeout=5m

After the configurator is gone and streaming workloads are drained, disable gantry.artifactStreaming.enabled. If the host OverlayBD file was changed by another owner after Gantry configured it, rollback deliberately leaves that new value in place; inspect the configurator logs and resolve ownership before removing the saved state.

Artifact Streaming metrics

Labels are bounded: source is local, peer, or origin; outcome is success or error; reason is range or origin_url.

/artifact-streaming/readyz on the node-local mirror port reports endpoint readiness. It is loopback-only, so probe it from the node rather than Prometheus.

Artifact Streaming alerts

These separate the failure domains that need different responses. Tune the thresholds against a real workload before paging on them.

- alert: GantryStreamingErrors
  expr: |
    sum by (instance) (rate(gantry_streaming_requests_total{outcome="error"}[5m]))
      / sum by (instance) (rate(gantry_streaming_requests_total[5m])) > 0.05
  for: 10m
  annotations:
    summary: Gantry is failing OverlayBD range requests on {{ $labels.instance }}.

- alert: GantryStreamingOriginFailing
  expr: sum(rate(gantry_streaming_requests_total{source="origin",outcome="error"}[5m])) > 0
  for: 10m
  annotations:
    summary: Signed-origin range reads are failing; suspect ACR or the host allowlist.

- alert: GantryStreamingNoPeerReuse
  expr: |
    sum(rate(gantry_streaming_requests_total{source="peer",outcome="success"}[30m])) == 0
      and sum(rate(gantry_streaming_requests_total{source="origin",outcome="success"}[30m])) > 0
  for: 30m
  annotations:
    summary: Every range is coming from ACR; peer reuse is not happening.

- alert: GantryStreamingRejectingOriginURLs
  expr: sum(rate(gantry_streaming_rejected_total{reason="origin_url"}[5m])) > 0
  for: 5m
  annotations:
    summary: Gantry refuses the OverlayBD origin URL; check the host suffix allowlist.

Origin reads are capped at 32 in flight per node. A sustained gantry_streaming_inflight{source="origin"} at that ceiling means ranges are queueing behind ACR rather than failing.

Artifact Streaming incident response

Artifact Streaming compatibility

Gantry consumes ACR's existing OverlayBD artifact format but owns the runtime reader and snapshotter.

Every other containerd setting is preserved, and rollback restores the original main config and any pre-existing drop-in exactly.

What to verify after rollout

See docs/detailed-design.md §7.6 for the full metric catalog.

Hardening overlays

deploy/examples/ carries optional hardening manifests that are intentionally not part of the default kubectl apply workflow. Every overlay there contains at least one site-specific value (CIDR, endpoint, label) that no shipped manifest can guess correctly across arbitrary clusters, so applying them unedited will fail the cluster into a state that is hard to debug.

Production guidance: the default install leaves the mirror listener (5000) and transfer listener (5001) reachable from other pods on the cluster network at <podIP>:port. The hostIP: 127.0.0.1 binding on the DaemonSet's hostPort only restricts host-side reach; the listener inside the pod is still 0.0.0.0. Production installs should adopt examples/networkpolicy.yaml.tmpl (or an equivalent NetworkPolicy in their own overlay) to close that pod-network gap. The overlay is shipped as an example rather than a default because its allow-list depends on the cluster's node-CIDR range, which is site-specific (see the workflow below).

examples/networkpolicy.yaml

Locks transfer (5001), libp2p (4001), mirror (5000), and metrics (9095) to the minimum traffic each port needs. Holds the manifest shape required by §7.5 but defers four CIDR choices to the operator - apiserver endpoint, kubelet probe source, mirror DNAT source, registry egress. See the long "OPERATOR ACTION REQUIRED" block at the top of the file and the Production caveats table below.

Workflow:

1. Roll out the DaemonSet without the overlay and verify kubectl -n unbounded-system rollout status ds/gantry, p2p_cache_hit_total, and a successful workload pull.
2. Copy the overlay into your own repository (or a Kustomize / Helm chart), edit every ipBlock marked "OPERATOR ACTION REQUIRED", and review against your CNI's hostPort SNAT behavior (kubectl get nodes -o yaml | grep -A2 podCIDR, etc.).
3. Apply with kubectl apply -f your-overlay/networkpolicy.yaml. Watch /readyz and any in-flight mirror pulls for at least one full image pull cycle - a wrong CIDR will surface as dht routing table empty (no peer libp2p traffic) or as containerd connection refused on 5000 (wrong mirror source CIDR), not as a NetworkPolicy validation error.
4. Roll back with kubectl delete networkpolicy -n unbounded-system gantry-agent if anything regresses.

Future hardening overlays (Pod Security Standards, dedicated PriorityClass, alternative hostNetwork: true topology) will live in the same directory and follow the same "deferred to operator, not in default install" rule.

Production caveats

A few configuration knobs that need operator attention before going to production:

HEAD semantics on cache miss

GET /v2/<repo>/blobs/<digest> on a cache miss warms the cache as a side effect; HEAD on the same URL does NOT. This is intentional (see the comment block in internal/mirror/mirror.go at the HEAD return after writeBlobHeaders) - caching a multi-GB blob just because a client asked for its size would defeat the bandwidth amplification fix Gantry exists to provide. A subsequent GET for the same digest follows the cache-miss path normally and warms the cache then.

If your client emits HEAD-then-GET patterns where you'd prefer to amortize the origin metadata round-trip, raise the issue upstream (containerd's puller, BuildKit's resolver, etc.) - those clients generally have a one-shot resolve-and-pull mode that skips the HEAD entirely.