gantry

Gantry deployment artifacts

This directory carries the Helm chart and operator-facing rendered manifests needed to roll out the Gantry agent as a Kubernetes DaemonSet.

Files

The Helm chart under chart/ is the source of truth for resources shared by standalone and operator-managed installations. make gantry-manifests renders the internal operator profile into deploy/gantry/rendered/; the Unbounded operator embeds those files and applies its own image, ConfigMap, and Lease ownership semantics. The target also renders the standalone node configurator and examples used by development and benchmark tooling.

Installation paths

• Operator-managed clusters use the manifests embedded in the unbounded-operator binary. The operator never runs Helm.
• Clusters without the operator install the released OCI chart. The chart continuously reconciles /etc/containerd/certs.d/_default/hosts.toml on every selected node. Containerd must already be configured to read /etc/containerd/certs.d; the chart does not edit or restart containerd.

The paths are mutually exclusive. PriorityClass/gantry-low records the active manager, and both installers reject ownership by the other path.

helm upgrade --install gantry oci://ghcr.io/azure/charts/gantry \
   --version <release-without-v> \
   --namespace gantry-system \
   --create-namespace \
   --set image.digest=sha256:<gantry-image-digest> \
   --set gantry.upstreamRegistries[0].name=registry.example.com \
   --set gantry.upstreamRegistries[0].endpoint=https://registry.example.com

The container image is built from images/gantry/Containerfile via make image-gantry-local (or make image-gantry-push to push).

Chair sizing and upgrades

gantry.chairCount controls the maximum active chair pool and gantry.chairSeedCount controls the per-digest seed cohort at a full pool. Both default to 64 chairs and 8 seeds. The counts have no configured maximum, but both must be positive and the seed count cannot exceed the chair count. The active pool is also bounded by the Gantry DaemonSet's desired scheduled pod count because one agent can hold at most one chair. The previous chair_holder_count, GANTRY_CHAIR_HOLDER_COUNT, and --chair-holder-count names remain accepted as deprecated aliases.

For example:

helm upgrade gantry oci://ghcr.io/azure/charts/gantry \
   --version <release-without-v> \
   --namespace gantry-system \
   --reuse-values \
   --set gantry.chairCount=128 \
   --set gantry.chairSeedCount=100

Increasing the chair count preserves every existing Lease and holder. Agents create and claim the additional numbered Leases through the normal election path until the new target is reached. Decreasing it makes holders at IDs above the new count vacate; the empty Lease objects may remain and are ignored. Chair Leases carry Helm's keep resource policy so a reduction does not delete coordination state during a rolling update. This also leaves the Leases behind on uninstall; delete them explicitly only after every Gantry pod has stopped.

Releases that only understand the original 64-chair set reject higher-numbered Lease names. Upgrading from such a release therefore requires two stages:

1. Upgrade every Gantry pod to the count-aware image while keeping gantry.chairCount=64.
2. After that rollout completes, raise gantry.chairCount and gantry.chairSeedCount.

Do not combine these stages in one Helm upgrade. A new pod can create a higher-numbered Lease while old pods are still running, which makes the old pods reject their next chair snapshot.

Metrics

Each agent serves Prometheus metrics on port 9095 (/metrics). Set metricsService.enabled=true to render Service/gantry-metrics, a headless Service over the agents that also lists NotReady agents. Use metricsService.annotations for annotation-driven scrapers, for example a Datadog endpoints check that scrapes every agent from its own node:

metricsService:
  enabled: true
  annotations:
    ad.datadoghq.com/endpoints.checks: |
      {"openmetrics": {"init_config": {}, "instances": [{
        "openmetrics_endpoint": "http://%%host%%:9095/metrics",
        "namespace": "gantry", "metrics": [".*"]}]}}

## Agent rollout

`updateStrategy.rollingUpdate.maxUnavailable` sets the agent DaemonSet's
rolling-update step as a positive count or percentage. The default `50%`
replaces half of the agents at once; lower it on large clusters to bound how
many nodes lose their local mirror during a rollout:

```sh
--set-string updateStrategy.rollingUpdate.maxUnavailable=10%

Peer transfer tuning

gantry.peerFetchTimeout (default 15m) caps one complete peer blob request, including streaming the body. It is a safety ceiling, not a stall detector: the HTTP/2 transport separately probes connections after 10s without reads. A value much lower than a large layer's transfer time makes requesters abandon progressing transfers and switch providers.

gantry.transferMaxConcurrentServes (default 10) caps concurrent peer blob serves per agent. Requests over the cap receive 429 so the requester re-discovers another provider. 0 means unlimited.

Use gantry.config only when a setting has no dedicated value; it replaces the entire generated configuration.

Operator Profile

# Render the profile embedded by unbounded-operator.
make gantry-manifests

# Validate and package the standalone chart.
make gantry-chart-lint
make gantry-chart-package GANTRY_CHART_VERSION=0.1.0 GANTRY_CHART_APP_VERSION=v0.1.0

Building the image locally

# Single-arch into local container engine:
make image-gantry-local

# Build and push to $(CONTAINER_REGISTRY):
make image-gantry-push CONTAINER_REGISTRY=ghcr.io/your-org

# Explicit tag:
make image-gantry-local VERSION=v0.6.0

Per-node containerd setup

Nodes provisioned by unbounded-agent already have containerd configured to read /etc/containerd/certs.d and carry the managed default Gantry mirror entry in /etc/containerd/certs.d/_default/hosts.toml. On those nodes, install the Gantry DaemonSet normally; the mirror activates when the pod starts listening on 127.0.0.1:5000.

ACR attachment is separate from deployment and preserves existing authentication by default. For operator-managed AKS Flex clusters, see the ACR attachment guide for commands, managed-auth opt-in, compatible versions, and limitations.

Standalone Helm installations run DaemonSet/gantry-containerd-config. Its resident reconciler checks the default hosts.toml every five seconds and atomically restores the chart-owned payload when the file is missing or different, including after a node upgrade resets host configuration. Graceful shutdown removes the file only when it still matches the chart payload. Set nodeConfig.enabled=false when another node-management system owns this file.

Externally managed installations can instead drop a registry-specific hosts.toml at:

/etc/containerd/certs.d/<registry-host>/hosts.toml

derived from hosts.toml.template (substitute ${REGISTRY_SERVER} with the registry's https://... URL). containerd reloads certs.d on its own; no restart needed.

What to verify after rollout

See docs/detailed-design.md §7.6 for the full metric catalog.

Hardening overlays

deploy/examples/ carries optional hardening manifests that are intentionally not part of the default kubectl apply workflow. Every overlay there contains at least one site-specific value (CIDR, endpoint, label) that no shipped manifest can guess correctly across arbitrary clusters, so applying them unedited will fail the cluster into a state that is hard to debug.

Production guidance: the default install leaves the mirror listener (5000) and transfer listener (5001) reachable from other pods on the cluster network at <podIP>:port. The hostIP: 127.0.0.1 binding on the DaemonSet's hostPort only restricts host-side reach; the listener inside the pod is still 0.0.0.0. Production installs should adopt examples/networkpolicy.yaml.tmpl (or an equivalent NetworkPolicy in their own overlay) to close that pod-network gap. The overlay is shipped as an example rather than a default because its allow-list depends on the cluster's node-CIDR range, which is site-specific (see the workflow below).

examples/networkpolicy.yaml

Locks transfer (5001), libp2p (4001), mirror (5000), and metrics (9095) to the minimum traffic each port needs. Holds the manifest shape required by §7.5 but defers four CIDR choices to the operator - apiserver endpoint, kubelet probe source, mirror DNAT source, registry egress. See the long "OPERATOR ACTION REQUIRED" block at the top of the file and the Production caveats table below.

Workflow:

1. Roll out the DaemonSet without the overlay and verify kubectl -n unbounded-system rollout status ds/gantry, p2p_cache_hit_total, and a successful workload pull.
2. Copy the overlay into your own repository (or a Kustomize / Helm chart), edit every ipBlock marked "OPERATOR ACTION REQUIRED", and review against your CNI's hostPort SNAT behavior (kubectl get nodes -o yaml | grep -A2 podCIDR, etc.).
3. Apply with kubectl apply -f your-overlay/networkpolicy.yaml. Watch /readyz and any in-flight mirror pulls for at least one full image pull cycle - a wrong CIDR will surface as dht routing table empty (no peer libp2p traffic) or as containerd connection refused on 5000 (wrong mirror source CIDR), not as a NetworkPolicy validation error.
4. Roll back with kubectl delete networkpolicy -n unbounded-system gantry-agent if anything regresses.

Future hardening overlays (Pod Security Standards, dedicated PriorityClass, alternative hostNetwork: true topology) will live in the same directory and follow the same "deferred to operator, not in default install" rule.

Production caveats

A few configuration knobs that need operator attention before going to production:

HEAD semantics on cache miss

GET /v2/<repo>/blobs/<digest> on a cache miss warms the cache as a side effect; HEAD on the same URL does NOT. This is intentional (see the comment block in internal/mirror/mirror.go at the HEAD return after writeBlobHeaders) - caching a multi-GB blob just because a client asked for its size would defeat the bandwidth amplification fix Gantry exists to provide. A subsequent GET for the same digest follows the cache-miss path normally and warms the cache then.

If your client emits HEAD-then-GET patterns where you'd prefer to amortize the origin metadata round-trip, raise the issue upstream (containerd's puller, BuildKit's resolver, etc.) - those clients generally have a one-shot resolve-and-pull mode that skips the HEAD entirely.