Documentation
¶
Overview ¶
Package mirror is the loopback OCI registry mirror containerd talks to via hosts.toml .
endpoint contract (cited from architecture.md the API contract and the design doc):
GET /v2/ 200, {"api":"registry/2.0"}
GET /healthz 200, "ok"
GET /v2/<repo>/manifests/<tag> 503, empty body
GET /v2/<repo>/manifests/sha256:<hex> cache or origin
GET /v2/<repo>/blobs/sha256:<hex> cache or origin
The tag-manifests 503 is the "tag fallthrough" - hosts.toml lists origin as the next entry, so containerd retries against origin directly. Returning 503 (NOT 404) is load-bearing: hosts.toml only falls through on 5xx, NOT on 4xx. Returning the wrong code breaks tag-resolution.
?ns=<registry> routing (the design doc): containerd adds ?ns=<host> to every request when hosts.toml specifies `server=<origin>`. If exactly one upstream is configured, ?ns= is optional (and ignored if present). When more than one upstream is configured, ?ns= MUST match one of them or the request returns 404 - there is no safe default.
Index ¶
- Variables
- type AuthenticationChallenger
- type ColdStartResolution
- type ColdStartResolver
- type DirectOriginFallbackController
- type DirectOriginFallbackOptions
- type LayerPrefetcher
- type NegativeCacheRecorder
- type Option
- func WithColdStart(c ColdStartResolver) Option
- func WithDhtLookupMetric(onLookup func(outcome string, dur time.Duration)) Option
- func WithDhtStaleOnlyMetric(onStaleOnly func()) Option
- func WithDiscovery(d ifaces.DHT, peer ifaces.PeerDialer) Option
- func WithDownstreamFailureMetric(downstreamFailure func(kind, class string)) Option
- func WithLayerPrefetcher(p LayerPrefetcher) Option
- func WithLiveStreamCompletedHook(onCompleted func(d digest.Digest)) Option
- func WithLiveStreamThrough() Option
- func WithLogger(l *slog.Logger) Option
- func WithMetrics(cacheHit, cacheMiss func()) Option
- func WithNF5(c *DirectOriginFallbackController) Option
- func WithNegativeCacheRecorder(rec NegativeCacheRecorder) Option
- func WithOriginStreamMetrics(started, completed, failed func(kind string)) Option
- func WithOriginSuccessMetric(originSuccess func(kind string, bytes int64)) Option
- func WithPeerBudgets(lookup, fetch time.Duration, maxAttempts int) Option
- func WithPeerFetchLatencyMetric(onPeerFetchLatency func(outcome string, d time.Duration)) Option
- func WithPeerMetrics(peerFetchOutcome func(outcome string), peerDialResult func(success bool)) Option
- func WithProvideErrorMetric(onProvideErr func(op string)) Option
- func WithProviderFailureCacheTTL(staleTTL, unavailableTTL, suspiciousTTL time.Duration) Option
- func WithSelfNodeID(id ifaces.NodeID) Option
- func WithSelfPeerID(id ifaces.NodeID) Option
- func WithStaleProviderFilteredMetric(onFiltered func(n int)) Option
- func WithStartupReadinessGate() Option
- type Server
Constants ¶
This section is empty.
Variables ¶
var ErrColdStartExhausted = errors.New("mirror: cold-start cascade exhausted")
ErrColdStartExhausted is the boundary-stable sentinel that `ColdStartResolver.Resolve` returns when the cascade ran to its final fallthrough (rule-7 cold-start, top-2K expansion both failed). direct-origin-fallback fallback may attempt a direct origin pull only when this is the underlying error. Other cold-start errors (failure short-circuit, cooldown active) intentionally map to other errors so direct-origin-fallback cannot circumvent them.
Functions ¶
This section is empty.
Types ¶
type AuthenticationChallenger ¶ added in v0.1.22
type ColdStartResolution ¶
ColdStartResolution mirrors *coldstart.Resolution at this boundary so the mirror package does not import internal/coldstart (which would import internal/mirror by transitivity through wiring).
type ColdStartResolver ¶
type ColdStartResolver interface {
Resolve(ctx context.Context, d digest.Digest, kind ifaces.OriginRefKind, registry, repository string, expectedSize int64) (*ColdStartResolution, error)
}
ColdStartResolver is the subset of *coldstart.Resolver that mirror needs. Kept narrow for testability - production wires the concrete resolver via WithColdStart.
type DirectOriginFallbackController ¶
type DirectOriginFallbackController struct {
// contains filtered or unexported fields
}
DirectOriginFallbackController runs the gating sequence. Safe for concurrent use.
func NewDirectOriginFallback ¶
func NewDirectOriginFallback(opts DirectOriginFallbackOptions) *DirectOriginFallbackController
NewDirectOriginFallback builds a controller. Inflight must be non-nil; everything else receives reasonable defaults.
func (*DirectOriginFallbackController) Allow ¶
func (n *DirectOriginFallbackController) Allow(ctx context.Context, d digest.Digest, kind ifaces.OriginRefKind, expectedSize int64) (bool, func(), error)
Allow runs the direct-origin-fallback gating sequence for digest d. When it returns (true, release, nil), the caller MUST invoke `release` once the origin pull completes (success or failure) - this frees the in-flight slot. The token has already been consumed; releasing does not refund it.
When it returns (false, nil, nil), direct-origin-fallback has declined. The caller should respond 5xx (warm path exhausted).
On context cancellation (e.g. client disconnect during jitter), returns (false, nil, ctx.Err) and any in-flight handle is released.
kind and expectedSize are forwarded to inflight.Map.Start so the in-flight entry carries enough context for the design doc stall detection in case the direct-origin-fallback origin pull itself stalls.
type DirectOriginFallbackOptions ¶
type DirectOriginFallbackOptions struct {
// Logger is the structured logger. Required.
Logger *slog.Logger
// Now is the time source; defaults to time.Now.
Now func() time.Time
// JitterBase is the `nf5_jitter_base` (default 3s when ≤0).
// The jitter window is `[0, JitterBase × ln(ClusterSize))`.
JitterBase time.Duration
// JitterCap is a hard ceiling on the computed jitter window
// (`nf5_jitter_cap`). Zero means no cap. When set, the effective
// window is min(JitterBase*ln(N), JitterCap).
JitterCap time.Duration
// PerNodeRateLimit is the `nf5_per_node_rate_limit`
// (default 2 tokens/minute when ≤0). Refill is continuous
// (fractional tokens accrue at PerNodeRateLimit/60 per second).
PerNodeRateLimit int
// ClusterSize returns the current membership count; used as
// `N` in the `ln(N)` jitter scaling. Nil or returning ≤1
// disables the jitter component.
ClusterSize func() int
// InBootstrap reports whether the local DHT is still in the
// bootstrap-suppression window. direct-origin-fallback is forbidden while this
// returns true.
InBootstrap func() bool
// HealthyEnough reports whether the local DHT health is in a
// state where the empty-DHT answer can be trusted (i.e. not
// Unhealthy). When this returns false direct-origin-fallback declines.
HealthyEnough func() bool
// Inflight is the per-digest dedup map; direct-origin-fallback takes a handle so
// concurrent direct-origin-fallback calls for the same digest collapse to one.
Inflight *inflight.Map
// Recheck performs a final DHT + cache + peer probe at the end
// of the jitter window. Returns true if a provider materialised
// during jitter (direct-origin-fallback cancels and the caller retries the warm
// path).
Recheck func(context.Context, digest.Digest) bool
// OnFallback is invoked once per origin pull that direct-origin-fallback permits.
// Maps to the design doc metric `p2p_origin_fallback_total`.
OnFallback func()
// OnDecline reports the reason direct-origin-fallback declined a request. Useful
// for ops dashboards. reason ∈ {"bootstrap_window",
// "dht_unhealthy", "in_flight", "rate_limited", "recheck_hit",
// "context_cancelled"}. Optional.
OnDecline func(reason string)
}
DirectOriginFallbackOptions configures the last-resort fallback controller.
type LayerPrefetcher ¶
type LayerPrefetcher interface {
OnManifestServed(ctx context.Context, registry, repository string, manifestDigest digest.Digest)
}
LayerPrefetcher is the speculative wire-level optimisation hook (the design doc detailed-design.md L332 / architecture.md L180). After the mirror serves a manifest successfully the mirror invokes OnManifestServed in a goroutine so an implementation can fetch the just-cached manifest body, parse it, identify child layer/config digests, group them by HRW rank-0 puller, and issue batched please_pull RPCs to warm the cluster before containerd asks for the layers. The mirror never waits for the callback to return; failures are the prefetcher's to log.
type NegativeCacheRecorder ¶
type NegativeCacheRecorder interface {
RecordFailure(d digest.Digest, class ifaces.FailureClass)
RecordSuccess(d digest.Digest)
}
NegativeCacheRecorder is the negative-cache integration the mirror's direct-origin path uses to mirror what the coordinated puller-pump path (cmd/gantry/main.go's runOriginPull) already does: classify a terminal origin / downstream failure into an ifaces.FailureClass and seed the per-puller cooldown ladder so the next request for the same digest short-circuits via the same `recently_failed` propagation the please_pull path uses.
Why this exists (a prior review): before this hook, the mirror's direct-origin path - including the direct-origin-fallback fallback that fires after the cold-start cascade reports ErrColdStartExhausted - recorded the failure metric but did NOT enter a negative-cache cooldown. The next direct-origin-fallback-eligible request for the same digest could re-fire the direct-origin pull at the bottom of the next jitter window, even though the previous attempt had stalled mid-stream or digest-mismatched at commit. The puller-pump path correctly drops such retries on the recently_failed cooldown; the mirror direct path did not. That gap is a retry-amplification hardening hole, not a metrics bug - fireOriginDownstreamFailure was already wired by fixes.
Contract:
- RecordFailure is invoked once per terminal mirror-direct origin failure, BEFORE the response is finalized. The class is taken from *ifaces.OriginError when origin returns one; downstream failures (io.Copy / cw.Commit / directVerifier.Verify) are recorded as FailureTransient - the same classification the puller-pump path uses for those exact paths. - RecordSuccess is invoked exactly once per successful mirror-direct origin pull, AFTER cw.Commit (or the direct- stream digest verifier) passes. It clears any prior cooldown so the next failure restarts the ladder from Initial (the design doc "Self-healing"). Symmetric with the puller-pump path's neg.RecordSuccess(d) call after a successful Commit.
HEAD requests deliberately do NOT touch the negative cache: the coordinated path never issues HEAD, and HEAD does not warm the cache, so recording HEAD failures would diverge the two paths' cooldown semantics with no observability win.
type Option ¶
type Option func(*Server)
Option configures Server construction.
func WithColdStart ¶
func WithColdStart(c ColdStartResolver) Option
WithColdStart wires cold-start orchestration. When set, the orchestrator is consulted on the DHT-empty branch of the cache-miss path before falling through to origin.
func WithDhtLookupMetric ¶
WithDhtLookupMetric registers a hook that fires once per FindProviders call with the outcome label ("hit", "miss", "timeout", "error") and the observed lookup duration. Used to populate p2p_dht_lookup_total and p2p_dht_lookup_duration_seconds (the design doc).
func WithDhtStaleOnlyMetric ¶
func WithDhtStaleOnlyMetric(onStaleOnly func()) Option
WithDhtStaleOnlyMetric registers a hook that fires when a DHT lookup returned candidate providers but the local stale/suspicious/ unavailable/self filters removed every one before any peer fetch was attempted - treated as a "stale-only" outcome so dashboards can separate true empty DHT responses (counted under gantry_dht_lookup_total{outcome="miss"}) from "DHT had providers but they were all dead". Per "DHT stale-only" mitigation.
func WithDiscovery ¶
func WithDiscovery(d ifaces.DHT, peer ifaces.PeerDialer) Option
WithDiscovery wires P2P fetch: cache miss -> DHT FindProviders -> PeerDialer.FetchFromPeer (across up to 3 providers) -> origin fallback. Either argument nil disables P2P fallback entirely (behavior).
func WithDownstreamFailureMetric ¶
WithDownstreamFailureMetric registers a callback fired by the mirror's direct-origin path when the body has been received from origin but a DOWNSTREAM step (io.Copy stall, cw.Commit digest mismatch / cache I/O error, directVerifier mismatch) fails before the cluster has produced a usable artifact.
Why this is separate from the origin failure-hook (origin.WithMetrics' failure closure in cmd/gantry/main.go): - origin.WithMetrics' failure closure is the origin-side terminal counter - it bumps BOTH p2p_origin_pull_failure_total (operator dashboards) AND p2p_origin_failure_total (the "is origin sick?" alert). Origin-side failures are the ones where the origin pull never started, never returned 2xx, or returned a non-2xx body. Counting downstream failures (where origin DID return 2xx but the body stalled / corrupted en route to the cache) against the same closure would falsely accuse origin of being sick. - This hook bumps ONLY p2p_origin_pull_failure_total (per-(kind,class) detail) with class="transient", leaving p2p_origin_failure_total reserved for true origin-side failures. Operators see the failure detail without the alert false-positive.
Together with onOriginSuccess and the origin-side failure closure, this restores the per-pull arithmetic identity for the GET path:
p2p_origin_pull_total{kind} == p2p_origin_pull_success_total{kind}
+ p2p_origin_pull_failure_total{kind,class=any}
+ (in-flight at scrape time)
This constraint ensures the missing terminal counter for downstream failures as the second of the two reasons that identity drifted positive in production traces. (The first was HEAD, fixed by adding origin.Head.)
func WithLayerPrefetcher ¶
func WithLayerPrefetcher(p LayerPrefetcher) Option
WithLayerPrefetcher wires a speculative layer prefetcher. Nil-safe.
func WithLiveStreamCompletedHook ¶
WithLiveStreamCompletedHook registers a callback fired after any live stream-through response (peer or origin) fully completes and passes the final digest check. Callers use this to correlate the response with a later containerd inventory observation without forcing the mirror to ingest the bytes itself.
func WithLiveStreamThrough ¶
func WithLiveStreamThrough() Option
WithLiveStreamThrough enables the "Mode A: live mirror requests - stream-through" contract for cache misses handled on behalf of the local containerd mirror client. When enabled, the mirror no longer writes live peer/origin responses into the active store; it proxies them directly to the caller and relies on the caller's containerd to perform the final commit.
func WithLogger ¶
WithLogger plumbs a structured logger into the mirror handler.
func WithMetrics ¶
func WithMetrics(cacheHit, cacheMiss func()) Option
WithMetrics registers metric callbacks for cache hit and cache miss observed by the mirror. The origin pull-family counters are intentionally NOT plumbed here - they're split across origin and mirror to keep one source of truth per counter:
- p2p_origin_pull_total{kind} and p2p_origin_failure_total{class} belong to origin.WithMetrics in the origin Client. Origin is the single chokepoint that both the mirror direct-origin path and the coordinated please_pull / runOriginPull goroutine route through, so counting there means dashboards see one source of truth and the operator-facing "is origin sick?" alert (p2p_origin_failure_total) stays consistent across both paths and free of false positives from downstream failures. - p2p_origin_pull_success_total{kind} belongs to the mirror (WithOriginSuccessMetric) because origin can't know whether the caller actually committed bytes - see that option's doc. - p2p_origin_pull_failure_total{kind,class} is fed from BOTH halves: origin's failure hook bumps it on true origin-side failures (with double-bump of p2p_origin_failure_total), and the mirror's WithDownstreamFailureMetric bumps it on downstream failures (with class=transient, NO double-bump of p2p_origin_failure_total).
Counting any of these at the mirror's WithMetrics hook would silently undercount the please_pull-coordinated path (the bulk of pulls on a hot cluster) and break the started == success + failure + in-flight arithmetic identity that all three counters rely on.
func WithNF5 ¶
func WithNF5(c *DirectOriginFallbackController) Option
WithNF5 wires the direct-origin fallback controller. When non-nil and cold-start exits via ErrColdStartExhausted, the mirror runs the direct-origin-fallback gating sequence (jitter, token bucket, dedup, re-check) before falling through to a direct origin pull. When nil, cold-start exhaustion always returns 5xx.
func WithNegativeCacheRecorder ¶
func WithNegativeCacheRecorder(rec NegativeCacheRecorder) Option
WithNegativeCacheRecorder wires the design doc negative-cache integration into the mirror's direct-origin path. See NegativeCacheRecorder for the contract. Nil-safe: passing nil leaves the mirror behaving exactly as it did before this option existed (metric-only failure reporting; no cooldown propagation to subsequent direct-origin attempts on the same node).
func WithOriginStreamMetrics ¶
WithOriginStreamMetrics wires the the live-stream-through origin counters. Hooks fire only from the direct-origin stream-through path: start at the moment the mirror commits to the origin path, completed after the full body has been proxied and the final digest check passes, and failed on any terminal error before that completion point.
func WithOriginSuccessMetric ¶
WithOriginSuccessMetric registers a callback fired by the mirror's direct-origin path AFTER it has streamed the response body to completion AND committed the bytes to cache (or, when cache is unavailable, AFTER the direct-stream digest verifier confirms the served bytes match the requested digest). The kind label uses the design-doc Prometheus vocabulary (see ifaces.OriginRefKind.MetricLabel).
This hook is the mirror-side half of the origin-success contract: origin.Client.Pull no longer reports success itself because it has no way to know whether the caller actually drained and verified the stream. HEAD requests (which by design never read the body), io.Copy interruptions, and cache-commit failures all leave the response body Closed without a real success - so reporting success on Close inside origin.Client inflated p2p_origin_pull_success_total against operations that never produced a usable byte. The puller pump's runOriginPull owns the equivalent hook on the please_pull-coordinated path; together they're the two places that know what "the origin pull actually succeeded" means.
func WithPeerBudgets ¶
WithPeerBudgets overrides the default peer-path budgets. lookup ≤ 0 means "use default 2s"; fetch ≤ 0 means "use default 10s"; maxAttempts ≤ 0 means "use default 3".
func WithPeerFetchLatencyMetric ¶
WithPeerFetchLatencyMetric registers a hook that fires once per fetchOneProvider call with the terminal outcome label and the wall-clock time from the FetchFromPeer dial to either the cache commit (hit) or the failing-branch return. Used for the p2p_peer_fetch_duration_seconds{outcome} histogram so operators can see whether peer fetches are slow because of dial latency, body streaming, or commit-time digest verification.
func WithPeerMetrics ¶
func WithPeerMetrics(peerFetchOutcome func(outcome string), peerDialResult func(success bool)) Option
WithPeerMetrics registers peer-fallback metric callbacks. peerFetchOutcome labels include: "hit", "notfound", "unavailable", "auth_or_config", "server_error", "protocol_error", "digest_mismatch", "stall", and "local_error". peerDialResult is invoked per attempted dial.
func WithProvideErrorMetric ¶
WithProvideErrorMetric registers a hook that fires when the mirror's post-peer-fetch dht.Provide call fails. The hook receives a stable label string identifying the call site so a CounterVec keyed by `op` can distinguish mirror-internal Provide failures from other sites.
func WithProviderFailureCacheTTL ¶
WithProviderFailureCacheTTL configures TTLs used to suppress immediate retries against recently-failed providers.
func WithSelfNodeID ¶
WithSelfNodeID configures the local Kubernetes node identity used to filter stale self provider records after a local cache miss. Membership/cold-start providers use Kubernetes node names as NodeID values.
func WithSelfPeerID ¶ added in v0.1.17
WithSelfPeerID configures the local libp2p peer identity used to filter stale self provider records from DHT lookup results after a local cache miss. DHT providers use libp2p peer IDs as NodeID values.
func WithStaleProviderFilteredMetric ¶
WithStaleProviderFilteredMetric registers a hook that fires once per DHT lookup, reporting the total number of provider candidates removed by the local filter caches (stale + unavailable + suspicious + self). n may be zero. Used to size the local false-positive surface from the DHT layer per .
func WithStartupReadinessGate ¶
func WithStartupReadinessGate() Option
WithStartupReadinessGate opts the mirror into the the startup gate: until MarkReady is called, every /v2/ request returns 503 with reason "agent starting up". Production callers should pair this with a goroutine that polls the same conditions /readyz uses and calls MarkReady once they converge - see cmd/gantry/main.go's readyCheck-poller for the canonical wiring.
Without this option the Server is "ready immediately" so unit-test fixtures (which never call MarkReady) continue to behave as before. The shutdown drain (Drain / drainGuard) is independent of this gate and always installed.
type Server ¶
type Server struct {
// contains filtered or unexported fields
}
Server is the mirror HTTP handler.
func New ¶
func New(cfg *config.Config, store ifaces.LocalContentStore, origin ifaces.OriginPuller, opts ...Option) *Server
New builds a Server bound to the given local content store and origin.
func (*Server) Drain ¶
func (s *Server) Drain()
Drain flips the mirror into shutdown mode: new /v2/ requests return 503 immediately. Idempotent. Safe to call from a signal handler.
func (*Server) ListenAndServe ¶
ListenAndServe runs the mirror on the configured loopback address. The returned function stops the server gracefully.
func (*Server) MarkReady ¶
func (s *Server) MarkReady()
MarkReady flips the startup gate from "not yet ready" to "serving" for production deployments that opted into WithStartupReadinessGate. Sticky: subsequent /readyz flaps do NOT take the mirror back out of service - once we have decided to serve we stay serving until Drain. Safe to call multiple times; safe to call from any goroutine. No-op for Servers that did not opt into the startup gate.
Source Files