Documentation
¶
Overview ¶
Package containerdstore adapts a containerd content store to the gantry ifaces.LocalContentStore contract so the rest of the agent can read from and write into containerd as the single local source of truth for image content.
This package is the "containerd as source of truth" hop the plan introduces: instead of maintaining a parallel hostPath cache that drifts out of sync with what containerd actually has, every read goes through the live content store and every commit lands directly in the same store containerd shows to kubelet. Net effect: a successful Gantry pull is indistinguishable from a kubelet pull at rest.
All operations apply the configured containerd namespace to ctx before delegating. A single Store instance is bound to exactly one namespace (typically "k8s.io" for kubelet-managed pods).
Failure-mode discipline (per): - cerrdefs.ErrNotFound from the underlying store is downgraded to *ifaces.ErrNotFound so transfer-endpoint callers can distinguish "definitively missing" from "backend hiccup". - Any other underlying error surfaces verbatim so callers treat it as "containerd unavailable" and do not advertise stale positive availability via has_cached or DHT.Provide.
The wrapped content.Store interface (not a concrete client) keeps the package unit-testable on darwin against an in-memory fake; the production wiring constructs Store with the real containerd.Client.ContentStore in cmd/gantry/main.go.
Index ¶
- Constants
- Variables
- type LeaseGuard
- type LeaseManager
- type MetricsHooks
- type Option
- type Store
- func (s *Store) AttachLease(ctx context.Context, d gdigest.Digest, source, repository string) error
- func (s *Store) CleanupExpiredLeases(ctx context.Context) (int, error)
- func (s *Store) CreateLease(ctx context.Context, d gdigest.Digest, source, repository string) (*LeaseGuard, error)
- func (s *Store) Descriptor(ctx context.Context, d gdigest.Digest) (ocispec.Descriptor, error)
- func (s *Store) Has(ctx context.Context, d gdigest.Digest) (bool, error)
- func (s *Store) Inventory(ctx context.Context) ([]gdigest.Digest, error)
- func (s *Store) LeaseTTL() time.Duration
- func (s *Store) ListManagedLeases(ctx context.Context) ([]leases.Lease, error)
- func (s *Store) LookupMediaType(d gdigest.Digest) string
- func (s *Store) Open(ctx context.Context, d gdigest.Digest) (io.ReadCloser, int64, error)
- func (s *Store) Ping(ctx context.Context) error
- func (s *Store) RememberMediaType(d gdigest.Digest, mediaType string)
- func (s *Store) Writer(ctx context.Context, d gdigest.Digest) (ifaces.ContentWriter, error)
Constants ¶
const ( // LabelManaged identifies leases created by Gantry. Used as the // list filter for cleanup so we never touch leases owned by // kubelet or other system components. LabelManaged = "gantry.io/managed" // LabelSource records the upstream registry hostname the digest // was pulled from (e.g. "registry.example.com"). LabelSource = "gantry.io/source" // LabelRepository records the OCI repository the digest was // pulled from (e.g. "library/nginx"). LabelRepository = "gantry.io/repository" // LabelDigest records the digest the lease protects. Redundant // with the resource binding but useful for log/metric extraction // without an extra ListResources call. LabelDigest = "gantry.io/digest" // LabelCreated records the RFC3339 creation timestamp. Used by // CleanupExpiredLeases when the containerd gc.expire label is // missing (e.g. legacy leases from earlier agent versions). LabelCreated = "gantry.io/created" )
Plan-mandated label keys for Gantry-owned leases.
const DefaultLeaseTTL = 60 * time.Minute
DefaultLeaseTTL is the lease lifetime used when WithLeaseTTL is not specified. 60 minutes is the midpoint of the plan-specified 30–120 minute range and large enough to absorb pull-then-deploy latency on slow nodes without keeping unused content alive indefinitely.
const DefaultNamespace = "k8s.io"
DefaultNamespace is the containerd namespace kubelet places pod containers in when it uses containerd as its CRI runtime. Centralised here so multiple subsystems (cdsub, containerdstore, advertise) cannot drift on what namespace they bind to.
const DefaultRefPrefix = "gantry-"
DefaultRefPrefix is prepended to every ingest ref so concurrent gantry writes are namespaced apart from kubelet/containerd's own pulls (kubelet uses "default-" prefixed refs from its puller). The final ref is `<prefix><digest>` which is stable per-digest so a crashed/restarted ingest can be resumed by the same writer slot.
const LeasePrefix = "gantry-"
LeasePrefix is prepended to every lease ID so list-by-prefix filtering can identify gantry-owned leases without parsing labels.
Variables ¶
var ErrNoLeaseManager = fmt.Errorf("containerdstore: lease manager not configured")
ErrNoLeaseManager is returned by AttachLease and CleanupExpiredLeases when the Store was not constructed with WithLeaseManager.
Functions ¶
This section is empty.
Types ¶
type LeaseGuard ¶
type LeaseGuard struct {
// contains filtered or unexported fields
}
LeaseGuard is returned by CreateLease. Release deletes the lease; callers use it on failed ingest so an aborted background pull does not leave an empty Gantry-owned lease behind until TTL cleanup.
type LeaseManager ¶
type LeaseManager interface {
Create(ctx context.Context, opts ...leases.Opt) (leases.Lease, error)
Delete(ctx context.Context, l leases.Lease, opts ...leases.DeleteOpt) error
List(ctx context.Context, filters ...string) ([]leases.Lease, error)
AddResource(ctx context.Context, l leases.Lease, r leases.Resource) error
}
LeaseManager is the subset of containerd's leases.Manager interface the Store consumes. Defined here (instead of importing the concrete containerd type at the call site) so tests can substitute a fake.
type MetricsHooks ¶
type MetricsHooks struct {
// OnHit fires when Has returns (true, nil) or Open returns a reader.
OnHit func()
// OnMiss fires when Has returns (false, nil) or Open returns ErrNotFound.
OnMiss func()
// return ErrUnavailable (containerd unreachable or sick).
OnUnavailable func()
// OnOpenError fires only from Open's non-ErrNotFound path - kept
// separate so dashboards can tell "open returned anything else"
// from generic backend unavailability.
OnOpenError func()
}
MetricsHooks are optional callbacks invoked from Has/Open so the containerd-as-truth model is observable on the wire (// "gantry_containerd_*_total"). Any callback may be nil; nil callbacks are skipped. Hooks fire on every call, not just sampled ones, so keep them cheap (typical implementation: prometheus.Counter.Inc).
type Option ¶
type Option func(*Store)
Option configures a Store at construction.
func WithLeaseManager ¶
func WithLeaseManager(m LeaseManager) Option
WithLeaseManager wires the containerd lease manager into the Store so AttachLease and CleanupExpiredLeases can run. Without this option both methods return ErrNoLeaseManager - tests or dev-only wiring may omit it; production containerd-only wiring always sets it.
func WithLeaseTTL ¶
WithLeaseTTL overrides the default lease TTL. Zero or negative leaves the default unchanged.
func WithMetrics ¶
func WithMetrics(h MetricsHooks) Option
WithMetrics registers metric callbacks. The hooks struct is copied by value so subsequent mutations by the caller do not affect the Store.
func WithNamespace ¶
WithNamespace overrides the containerd namespace ctx is bound to before delegating. Empty string is ignored.
func WithRefPrefix ¶
WithRefPrefix overrides the ingest ref prefix used by Writer. Empty string is ignored.
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store wraps a containerd content.Store.
func New ¶
New builds a Store bound to cs and the supplied namespace. Panics on nil cs because constructing a Store without a backing content store is always a programming error, not a runtime one.
func (*Store) AttachLease ¶
AttachLease creates a containerd lease that keeps d alive for the configured TTL. The returned LeaseGuard is intentionally discarded because the caller does not need to release this lease early - it will expire via TTL. Idempotent at the digest level: a digest may be referenced by multiple leases without breaking anything, but gantry uses one lease per Commit to keep cleanup arithmetic simple.
func (*Store) CleanupExpiredLeases ¶
CleanupExpiredLeases removes every Gantry-managed lease whose creation timestamp + configured TTL is in the past, returning the number of leases deleted. Used by a periodic background task in cmd/gantry to keep the lease catalogue from growing unbounded.
We rely on our own LabelCreated rather than parsing containerd's containerd.io/gc.expire because the latter is internal-format and not exposed in the leases.Lease.Labels map after creation in some containerd versions. Lease.CreatedAt + Store.leaseTTL is the authoritative computation.
func (*Store) CreateLease ¶
func (s *Store) CreateLease(ctx context.Context, d gdigest.Digest, source, repository string) (*LeaseGuard, error)
CreateLease creates a containerd lease that keeps d alive for the configured TTL and returns a guard the caller can release on failed ingest. It is safe to call before content commit: the content resource binding names the digest that will be committed, so containerd GC sees the intended protection as soon as the content appears.
source is the upstream registry hostname; repository is the OCI repository path. Both are stamped as labels so the lease catalogue is self-describing without cross-referencing inflight state.
func (*Store) Descriptor ¶
Descriptor returns the OCI descriptor (mediatype unknown; size from the store) for d. Used by callers that need to construct an ocispec.Descriptor without opening a reader handle. Returns *ifaces.ErrNotFound for missing digests.
func (*Store) Has ¶
Has reports whether d is present AND openable in the wrapped content store. We probe via ReaderAt - opening and immediately closing the handle - rather than the cheaper Info lookup because coord PullIntent uses Has to drive HasCached on the wire: a peer that answers HasCached=true must be able to serve the bytes through the transfer endpoint, and Info-only Has would have lied for digests whose content file is missing or unreadable on disk (rare after disk issues but possible - and the resulting peer 404 would burn a wasted dial round-trip on every requester). This matches the openability semantics already used by Inventory and the advertiser's Notify pre-check.
(true, nil) -> present and openable (transfer endpoint will serve it). (false, nil) -> definitively absent (ErrNotFound from backend) OR present in Info but not openable (corrupt / partial / unreadable file). Either way callers should treat as miss. (false, err) -> backend failure (containerd unreachable, namespace misconfigured, etc.). Callers MUST NOT treat the false return as "definitively absent" in this case; coord.computeLocalIntent surfaces this distinctly via OnPullIntentStorageUnavailable.
func (*Store) Inventory ¶
Inventory enumerates every sha256 digest currently present and openable in the content store. Used by the advertiser as the periodic reconciliation source of truth so DHT provider records reflect content this node can actually serve right now. Non-sha256 digests are silently skipped because the rest of the agent only handles sha256 (per internal/digest). The ReaderAt probe is bounded to opening and closing the handle; it does not read content bytes.
func (*Store) LeaseTTL ¶
LeaseTTL returns the configured lease TTL. Used by cmd/gantry to schedule the cleanup interval as a function of the TTL.
func (*Store) ListManagedLeases ¶
ListManagedLeases returns the set of Gantry-managed leases currently live in containerd, filtered by the LabelManaged=true label. Used by the the gantry_containerd_lease_active gauge sampler. Returns ErrNoLeaseManager when this Store was built without WithLeaseManager.
This is a best-effort snapshot - the catalogue may have changed by the time the caller observes the slice. Callers that need exact counts should rely on the counters (created/released) instead.
func (*Store) LookupMediaType ¶
LookupMediaType returns the advisory descriptor-index media type for d. It satisfies transfer.Describer so peer manifest responses can carry the exact OCI/Docker media type when cdsub has walked the descriptor graph.
func (*Store) Open ¶
Open returns a streaming reader for d. Returns *ifaces.ErrNotFound when the digest is not present so the transfer endpoint and ContentWriter callers can distinguish miss from error.
func (*Store) Ping ¶
Ping checks that the configured containerd namespace/content store is reachable without mutating metrics. ErrNotFound from the impossible probe digest means the content API is healthy enough for readiness purposes.
func (*Store) RememberMediaType ¶
RememberMediaType records mediaType for d in the advisory descriptor index so a later Descriptor call can return the correct MediaType field without re-parsing manifest JSON. Empty mediaType values are ignored. Per "Descriptor index" the index is populated from:
1. containerd image target descriptors on startup/reconcile, 2. walked manifest/index child descriptors (cdsub.walk), 3. response headers observed during live stream-through, 4. JSON parse fallback (callers may compute and Remember).
The index is bounded; when the cap is reached the oldest entry (random one - we do not track LRU recency) is evicted. Callers MUST treat a missing entry as "unknown media type", not "absent".
func (*Store) Writer ¶
Writer returns a ContentWriter that streams bytes into containerd's content store. Commit verifies the streamed bytes against d (the content store enforces this on Writer.Commit; mismatch surfaces as a non-nil error). Abort cancels the in-progress ingest.
The ref is `<refPrefix><digest>` so concurrent calls for the same digest resume the same ingest slot rather than racing for distinct staging areas; this is the standard containerd pattern and matches what their own puller does.
If the digest is already committed in the store, Writer returns ErrAlreadyExists wrapped - callers who want "treat-as-committed" semantics should call Has first.
Source Files