source

package
v0.4.8 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 16, 2026 License: AGPL-3.0 Imports: 34 Imported by: 0

Documentation

Overview

Package source implements SourceController — reconciles GitRepository and OCIRepository resources by fetching the underlying artifact into a content-addressed on-disk cache and publishing a *store.GitArtifact or *store.OCIArtifact for downstream controllers.

Native libraries are used throughout:

  • go-git for git clone / checkout. No `git` subprocess invocation.
  • oras-go for OCI artifact pulls. No `crane` or `oras` subprocess.

The cache is keyed by SHA256(url + ref) so multiple revisions of the same upstream can co-exist on disk. A LocalRepository — a synthetic GitRepository pointing at the user's working directory — is also supported by the orchestrator.

Package source is the SDK adapter for Flux's source-controller CRs. It exposes per-kind Fetcher implementations (one per source type) plus a content-addressed disk Cache that all fetchers write into.

Adding a new source kind:

  1. Implement source.Fetcher for the new kind in a new file (or subpackage) under pkg/source.
  2. Register the fetcher with the orchestrator at construction time.

The source controller (pkg/controllers/source) does not know about individual kinds — it dispatches via the Fetchers map keyed by id.Kind, so adding a new kind is one registration call.

Index

Constants

View Source 
const SlotMetaFile = ".flate-meta.json"

SlotMetaFile is the single structured sidecar a fetched cache slot carries, replacing the family of per-fact .flate-* marker files (the git revision, the OCI digest). It is `.flate-`-prefixed so it never collides with a user file in the materialized source tree, and JSON so a new fact is a new field rather than a new file.

Variables

View Source 
var ResponseHeaderTimeout = 120 * time.Second

ResponseHeaderTimeout bounds how long an HTTP source fetch (git over HTTPS, OCI, bucket) waits for the FIRST response byte after writing the request — a liveness backstop, not a determinism knob. Consumers now wait for a fetch's OUTCOME (orchestrator quiescence) rather than a per-dep wall clock, so a host that completes TLS+dial then black-holes — connects but never sends response headers — would otherwise keep the task pool active forever and wedge the whole run. ResponseHeaderTimeout bounds only the header wait, NOT the streamed clone/pull/download body, so a large artifact on a slow-but- live link still completes; sized large enough never to trip a live repo. Mirrors helm's helmHTTPTimeout (helm's getter ignores ctx; this covers the ctx-aware git/OCI/bucket transports' header wait). A var so tests can shrink it — mutate only before a run starts to stay race-clean.

Functions

func ApplyIgnore 

func ApplyIgnore(root string, ignore *string) error

ApplyIgnore deletes every file under root that matches the source- controller ignore matcher: VCS + Default excludes (.git/, .github/, *.jpg/png/zip, .sops.yaml, .flux.yaml, ...) PLUS any in-tree .sourceignore files PLUS the user-supplied spec.ignore patterns when non-nil. Mirrors source-controller's GitRepository / OCIRepository artifact-build behavior.

nil spec.ignore is NOT a no-op — the VCS + Default patterns still apply, matching what real Flux ships in a Git/OCI artifact tarball.

Bucket sources use ApplyIgnoreNoDefaults instead — see that function for the rationale.

func ApplyIgnoreNoDefaults 

func ApplyIgnoreNoDefaults(root string, ignore *string) error

ApplyIgnoreNoDefaults is the Bucket-flavored variant: it skips the VCS + Default exclude patterns and applies ONLY the in-tree .sourceignore plus the user-supplied spec.ignore. Mirrors fluxcd/source-controller/internal/controller/bucket_controller.go which uses sourceignore.NewMatcher (no defaults) rather than the NewDefaultMatcher that GitRepository / OCIRepository use.

The rationale: Bucket has no VCS semantics. An object store can legitimately carry .git/-named keys, .jpg/.png images, .flux.yaml, etc., and source-controller delivers them as-is. Stripping them would diverge from what a cluster Bucket reconcile produces.

func AuthIdentity 

func AuthIdentity(parts ...string) string

AuthIdentity returns a deterministic, opaque tag identifying the auth context bound to a source fetch. It is appended to the cache key so two source CRs that target the same (URL, ref) but reference different SecretRefs do not collide on the same on-disk slot.

Inputs are usually `<ns>/<secretRef.Name>` strings; the helper trims empties and joins the rest with NULs so adding a new auth dimension later (cert secret, proxy secret, …) is a one-arg append.

Returns "" when every input is empty — the caller passes "" to Cache.Slot in that case, and slots match the legacy unauthenticated layout so existing caches survive.

Fetchers should prefer AuthIdentityFromRefs when their inputs are already *manifest.LocalObjectReference values; reach for this lower-level entry point only when a non-secret-ref dimension (e.g. a hashed cert thumbprint) needs to participate in the key.

func AuthIdentityFromRefs 

func AuthIdentityFromRefs(ns string, refs ...*manifest.LocalObjectReference) string

AuthIdentityFromRefs is the typed entry point fetchers use to build a cache auth identity from their namespaced SecretRefs. Each fetcher (git, oci, bucket) used to reimplement the same nil-check + format + AuthIdentity call sequence; collapsing them here keeps the format stable as new ref dimensions get added.

Refs are consumed in caller-defined order; nil entries slot in as "" so AuthIdentity strips them. ns is the owning CR's namespace (SecretRefs are LocalObjectReferences — they inherit it).

func BasicAuthFromSecret added in v0.4.0 

func BasicAuthFromSecret(sec *manifest.Secret, ownerKind, ns, name, secretRef string) (username, password string, err error)

BasicAuthFromSecret extracts the username/password pair every HTTP-auth fetcher reads from a Secret (git HTTPS, HelmRepository). It returns a uniform MissingSecretErr when either field is absent or PLACEHOLDER-wiped — empty covers both the missing-key and ExternalSecret-stub cases, so a caller's --allow-missing-secrets gate (errors.Is ErrMissingSecret) treats them alike. ownerKind/ns/name/secretRef only shape that error message.

func BuildTLSConfig 

func BuildTLSConfig(crt, key, ca string) (*tls.Config, error)

BuildTLSConfig assembles a *tls.Config from PEM-armored secret material. Any subset of (client cert + key) and (CA) is acceptable; crt/key must appear together if either is present. Returns an error when all inputs are empty so callers can distinguish "no TLS configured" from "malformed config".

MinVersion is pinned to TLS 1.2 to match source-controller's defaults. Callers that need TLS 1.3-only behavior can adjust the returned config.

Common Flux Secret keys feeding this helper: 

tls.crt / tls.key  → client certificate (mTLS)
ca.crt             → trust roots for the server

func CacheKeyHash 

func CacheKeyHash(v any, n int) (string, error)

CacheKeyHash marshals v to JSON, sha256-hashes the bytes, and returns the hex of the first n bytes — the shared cache-key fingerprint used by the per-kind fetchers. Exported so subpackages (git, oci) can reuse one implementation; each caller picks its own width (n) and decides how to treat a marshal error.

func ErrUnsupportedProvider 

func ErrUnsupportedProvider(kind, namespace, name, got, want, hint string) error

ErrUnsupportedProvider is the canonical "we only support X" error returned by every fetcher's provider gate. Centralizes the wording so a new provider added to one fetcher matches the others by default — and so an operator parsing logs sees one consistent shape across GitRepository / OCIRepository / Bucket failures.

hint is a short parenthetical that names what the supported provider expects (e.g. "SecretRef-based credentials" or "S3-compatible").

func MissingSecretErr 

func MissingSecretErr(kind, ns, name, secretRef, reason string) error

MissingSecretErr builds a *MissingSecretError. The signature is unchanged from the prior string-only form so every call site compiles untouched. The owner and secret share the namespace ns (a SecretRef is same-namespace).

func MutableCacheKey 

func MutableCacheKey(base string) string

MutableCacheKey returns a one-shot cache key for mutable source refs. Mutable refs must refresh instead of serving a stale slot, but a failed refresh must also leave any previously committed artifact intact.

func NewHTTPTransport 

func NewHTTPTransport(tlsCfg *tls.Config, proxy *ProxyConfig) (*http.Transport, error)

NewHTTPTransport composes an *http.Transport for HTTP source fetches: a clone of http.DefaultTransport (inheriting net/http's production-tuned connection pool / keep-alive / TLS-minimum settings) with ResponseHeaderTimeout applied as a liveness backstop, plus optional TLS / proxy customization.

It ALWAYS returns a non-nil transport so every caller is bounded — callers must NOT fall back to http.DefaultTransport (which has no header timeout). Cloning (not allocating fresh) is the idiomatic way to override exactly the dimensions we control without losing the rest of the defaults.

Replaces three near-identical implementations (git, oci, bucket) that had drifted on whether to clone or to allocate fresh.

func QualifiedName added in v0.3.4 

func QualifiedName(kind, namespace, name string) string

QualifiedName is the canonical "<Kind> <namespace>/<name>" identifier the per-kind fetchers prefix their error messages with — one helper so the "Kind ns/name" shape can't drift across fetchers (git, oci, bucket, and helmchart each wrap it as gitID/ociID/…). This is the slash form for human-readable errors, distinct from RepoName()'s "ns-name" Flux resource identifier used for cache slots and auth scopes.

func ResolveCertSecret 

func ResolveCertSecret(secrets SecretGetter, ns, ownerKind, ownerID string, ref *manifest.LocalObjectReference) (*tls.Config, error)

ResolveCertSecret reads ref's Secret and builds a *tls.Config from its tls.crt + tls.key (client cert) and/or ca.crt (server CA) keys — mirroring source-controller's certSecretRef schema.

Returns (nil, nil) when ref is nil. Loud errors when ref is set but SecretGetter is unwired or the Secret is missing/unparseable. ownerKind+ownerID are formatted into errors for diagnosability.

func StringFromSecret 

func StringFromSecret(sec *manifest.Secret, key string) string

StringFromSecret reads a key from a Secret, preferring StringData over Data. Data values are base64-decoded (per k8s Secret semantics) before being returned, so the same string surface holds regardless of whether the source manifest used `data:` or `stringData:`. PLACEHOLDER_-wiped values (the result of flate's secret wiping pre-processing) are reported as empty so callers surface a clear "missing keys" error rather than authenticating with the literal placeholder.

Used by per-kind Fetchers (git, oci, bucket) to resolve auth + TLS material from the Secret a SecretRef points at.

func UpdateSlotMeta added in v0.3.4 

func UpdateSlotMeta(slotDir string, mutate func(*SlotMeta)) error

UpdateSlotMeta read-modify-writes the slot's sidecar: read the current SlotMeta (preserving fields the caller leaves alone), apply mutate, write it back. Marker writers run under the slot lock (cache.Slot serializes per key), so the RMW has a single writer. Shared by the per-kind fetchers (oci digest, helmchart chart resolution, git revision) so each records its facts without clobbering the others.

func WriteSlotMeta added in v0.3.4 

func WriteSlotMeta(slotDir string, meta SlotMeta) error

WriteSlotMeta persists meta into slotDir atomically (temp file + fsync + rename + dir sync), so a crash mid-write can never leave a torn sidecar that a later run misreads as a live marker.

Types

type Cache 

type Cache struct {
	// contains filtered or unexported fields
}

Cache manages a content-addressed on-disk directory for fetched sources. Each (url, ref) tuple gets its own slot, so multiple revisions of the same upstream coexist without clobbering one another.

The cache is safe for concurrent use. Per-slot locks serialize the full fetch-write-read lifecycle on a single slot. Different slots proceed in parallel.

func NewCache 

func NewCache(layout cacheroot.Layout) *Cache

NewCache constructs a Cache backed by the supplied Layout. The caller-owned Layout is the single source of truth for cache paths; this Cache asks it for SourceSlot positions and never composes its own. When the Layout's Root is empty, a flate-cache subdirectory under os.TempDir() is used so embedders that don't wire CacheDir still work.

func (*Cache) BlobByDigest 

func (c *Cache) BlobByDigest(digest string) (string, bool)

BlobByDigest returns the populated blob directory for digest and true on a content-addressed cache hit, or ("", false) for an empty or absent digest.

func (*Cache) PutBytes 

func (c *Cache) PutBytes(ctx context.Context, content []byte, filename string) (dir, digest string, err error)

PutBytes installs content as a single file named filename in the shared content-addressed blob store, returning the populated blob directory and its sha256 digest. It's the immutable counterpart to Slot: Slot stages ref-keyed mutable artifacts (git trees, OCI layouts) that need a refresh/commit lifecycle, while PutBytes stores already-content-addressed bytes (e.g. an HTTP chart tarball pulled by digest) so identical content dedups across sources. Concurrent callers writing the same digest serialize internally.

func (*Cache) Slot 

func (c *Cache) Slot(ctx context.Context, url, ref, authID string) (*Slot, error)

Slot allocates a per-(url, ref) slot under the cache root with atomic-rename finalization. Holds the slot mutex until Release is called; serializes against other Slot acquisitions on the same key.

The returned Slot's Path is:

  • The final slot directory when Exists is true (cache hit — the fetcher reads from it directly).
  • A sibling staging directory when Exists is false (cache miss — the fetcher writes into it, then calls Commit to atomic-rename into the final slot).

The staging dir lives at `<final>.tmp.<rand>`, on the same filesystem as the final, so os.Rename is atomic per POSIX. If the fetcher returns without committing (any error path, any panic), Release removes the staging dir and the final slot is left absent — the next fetch starts clean. This atomicity replaces the older "write in place + .flate-* sentinels" pattern: the final slot is either complete or doesn't exist, never torn.

type ExistenceFetcher 

type ExistenceFetcher struct{}

ExistenceFetcher is the no-op Fetcher registered for kinds whose existence alone is enough to satisfy downstream waits — used today for HelmRepository (a registry/index base whose per-chart pull is a synthesized HelmChart). Returning nil artifact + nil error lands the resource in Ready without recording a SourceArtifact, so a HelmRelease that dependsOn a HelmRepository unblocks instantly without flate having to mirror the controllers' "did fetch succeed?" logic from outside the controller package. Embedders can also wire it for any kind they want present-but-not-fetched (e.g. OCIRepository in an offline embed).

func (ExistenceFetcher) Fetch 

func (ExistenceFetcher) Fetch(_ context.Context, _ manifest.BaseManifest) (*store.SourceArtifact, error)

Fetch implements source.Fetcher as a no-op — see ExistenceFetcher.

type Fetcher 

type Fetcher interface {
	Fetch(ctx context.Context, obj manifest.BaseManifest) (*store.SourceArtifact, error)
}

Fetcher resolves a single source CR into an on-disk artifact. The source controller stores Fetchers in a map keyed by Kind and dispatches via this untyped interface. Concrete implementations satisfy it through Wrap[T] — see TypedFetcher.

func WithRetry 

func WithRetry(f Fetcher, cfg RetryConfig) Fetcher

WithRetry wraps f so transient fetch failures are retried per cfg. When retry is disabled (Attempts <= 1) f is returned unwrapped, so the decorator adds zero overhead on the default-off path. Jitter is clamped to [0,1] here so a misconfigured flag can't produce negative backoff durations.

func Wrap 

func Wrap[T manifest.BaseManifest](kindLabel string, f TypedFetcher[T]) Fetcher

Wrap converts a TypedFetcher into the untyped Fetcher interface used by the source-controller dispatcher map. The single type-assertion site is here — a mismatched payload returns ErrInput rather than panicking. Embeds the Kind label so the error message names the responsible fetcher.

type MissingSecretError added in v0.3.4 

type MissingSecretError struct {
	// Owner is the source CR whose SecretRef couldn't resolve.
	Owner manifest.NamedResource
	// Secret is the missing Secret the SecretRef points at (Kind=Secret,
	// the source's namespace, the ref name).
	Secret manifest.NamedResource
	// Detail is the human reason ("not found", "missing username/password").
	Detail string
}

MissingSecretError reports that a source's auth SecretRef couldn't be resolved offline. It carries the missing Secret's identity so the source controller can consult the producer index (an in-repo ExternalSecret / SealedSecret declaring that Secret) and auto-skip without --allow-missing-secrets. Unwrap returns manifest.ErrMissingSecret so the existing errors.Is(err, ErrMissingSecret) gates (and the flag path) hold.

func (*MissingSecretError) Error added in v0.3.4 

func (e *MissingSecretError) Error() string

Error renders the same string the previous fmt.Errorf form produced, so the skip reason surfaced via base.go (SkippedPrefix + TrimSentinelPrefix) is byte-identical to before.

func (*MissingSecretError) Unwrap added in v0.3.4 

func (e *MissingSecretError) Unwrap() error

Unwrap keeps errors.Is(err, manifest.ErrMissingSecret) working.

type ProxyConfig 

type ProxyConfig struct {
	Address  string
	Username string
	Password string
}

ProxyConfig is the resolved view of a Flux spec.proxySecretRef. Matches source-controller's Secret schema: data.address (required), data.username + data.password (optional, basic-auth).

func ResolveProxy 

func ResolveProxy(secrets SecretGetter, ns, ownerKind, ownerID string, ref *manifest.LocalObjectReference) (*ProxyConfig, error)

ResolveProxy reads ref's Secret via secrets and decodes it into a ProxyConfig. Returns (nil, nil) when ref is nil — proxy is opt-in. Surfaces a loud error when ref is set but the Secret is missing or lacks the required address key, matching source-controller's fail-loud behavior.

func (*ProxyConfig) URL 

func (p *ProxyConfig) URL() (*url.URL, error)

URL parses Address into a *url.URL with optional basic-auth user info applied. The caller pre-validated non-empty by checking the return of ResolveProxy. Used by NewHTTPTransport when pinning a transport's Proxy function; the OCI / Bucket / HTTP-Git transports pick this up uniformly.

type RetryConfig 

type RetryConfig struct {
	// Attempts is the total tries per fetch, including the first.
	// <= 1 disables retry.
	Attempts int
	// MinWait is the backoff floor (and the first retry's base delay).
	MinWait time.Duration
	// MaxWait clamps the exponential backoff ceiling.
	MaxWait time.Duration
	// Jitter is the fraction ([0,1]) of randomized spread applied to
	// each backoff so a fleet of fetches don't retry in lockstep.
	Jitter float64
}

RetryConfig tunes the bounded, classified retry applied uniformly to every source Fetcher via WithRetry. It's the single retry layer in flate's fetch path: the per-kind HTTP clients (oras for OCI, go-git, minio for Bucket) are deliberately left non-retrying so retries happen exactly once, here, with one predictable policy across kinds.

Attempts is the TOTAL number of tries (the first call plus retries), so Attempts <= 1 disables retry entirely and WithRetry returns the wrapped Fetcher untouched. Backoff between tries is exponential (MinWait * 2^n) clamped to MaxWait, then spread by ±Jitter.

type SecretGetter 

type SecretGetter func(namespace, name string) *manifest.Secret

SecretGetter resolves a Secret CR by namespace + name. Fetchers that read authentication, TLS, or proxy material from a Flux spec.*SecretRef accept one of these so they don't need a back-reference to the Store. Today: GitRepository (auth + TLS), OCIRepository (auth + TLS), Bucket (auth + TLS). The orchestrator wires it to Store.GetByName at construction time.

type Slot 

type Slot struct {
	// Path is where the fetcher reads / writes:
	//   Exists == true  → Path is the final slot (read-only use).
	//   Exists == false → Path is the staging dir (write here, then
	//                     Commit to finalize).
	Path string
	// Exists reports whether the final slot was already populated by
	// a prior fetch. When true, the staging dance is skipped and
	// Path is the final slot directly.
	Exists bool
	// contains filtered or unexported fields
}

Slot is one allocated cache slot. Acquired by Cache.Slot, released by the caller's deferred Release. On a cache miss the fetcher writes into Path (a staging dir) and calls Commit to atomic-rename into the final slot; on a cache hit Path is already the final slot.

func (*Slot) Commit 

func (s *Slot) Commit() error

Commit finalizes a successful fetch: atomic-rename the staging dir over the final slot. No-op on a cache hit (Exists == true). Safe to call multiple times. After a successful commit, Path is updated to the final slot so subsequent reads work uniformly.

func (*Slot) PersistMeta added in v0.3.4 

func (s *Slot) PersistMeta(mutate func(*SlotMeta)) error

PersistMeta atomically records meta-only sidecar fields on the slot: the resolve-cache write path (OCI digest, helm chart resolution) whose only on-disk product is the .flate-meta.json sidecar, never a materialized tree. On a cache hit it stages a refresh (keeping the old slot readable until commit), applies mutate to the SlotMeta via the read-modify-write UpdateSlotMeta, then commits under the held slot lock. It is the meta-only sibling of Commit and folds the previously-duplicated stage→write→commit dance the OCI and helmchart resolve writers each carried. Callers keep their own no-op guard (a nil slot or an incomplete resolution) since the predicate differs per source kind.

func (*Slot) Refresh 

func (s *Slot) Refresh() error

Refresh wipes the final slot and allocates a fresh staging directory in one step. It is equivalent to calling Reset followed by Stage and is intended for callers that detect a stale immutable cache hit and need to re-fetch from scratch. The Reset+Stage sequence is preserved unchanged; this method exists only to avoid repeating the pair.

func (*Slot) Release 

func (s *Slot) Release()

Release drops the slot mutex AND, if Commit wasn't called, removes the orphan staging dir. MUST be deferred by every Cache.Slot caller. Safe to call multiple times — second+ calls are no-ops.

func (*Slot) Reset 

func (s *Slot) Reset() error

Reset wipes the final slot — used by callers that detected a stale cache hit (e.g. an OCI digest that no longer matches the resolved one). After Reset the slot looks like an Exists=false miss; the caller can write to a new staging via a fresh Cache.Slot call, OR can call Stage on this same Slot to allocate staging in place.

func (*Slot) Stage 

func (s *Slot) Stage() error

Stage allocates the staging dir for a Slot that was returned with Exists == true and then explicitly Reset by the caller. The new Path is the staging dir. No-op when staging is already allocated.

func (*Slot) StageRefresh 

func (s *Slot) StageRefresh() error

StageRefresh allocates a staging directory while preserving the current final slot until Commit. It is for interval-based refreshes: readers keep a usable old artifact if the fetch fails before commit, while a successful Commit replaces the final slot under the held slot lock.

type SlotMeta added in v0.3.4 

type SlotMeta struct {
	// Revision is the resolved git commit SHA (GitRepository slots).
	Revision string `json:"revision,omitempty"`
	// Digest is the resolved OCI content digest (OCIRepository slots).
	Digest string `json:"digest,omitempty"`

	// ChartVersion, ChartDigest, ChartURL record a HelmRepository HTTP chart
	// resolution (the concrete version, bare-hex tarball sha256, and absolute
	// .tgz URL the repo's index.yaml resolved a chart name+version to). They
	// live on a dedicated "helm-resolve:" slot so a warm run skips the live
	// index.yaml fetch within spec.interval. Named distinctly from the OCI
	// Digest so a slot never aliases the two. ChartDigest may be empty for a
	// digest-less (mutable) index entry.
	ChartVersion string `json:"chartVersion,omitempty"`
	ChartDigest  string `json:"chartDigest,omitempty"`
	ChartURL     string `json:"chartURL,omitempty"`
}

SlotMeta is the content of SlotMetaFile: the metadata a cache-hit needs to validate a slot and decide whether a refetch can be skipped. Fields are populated by whichever fetcher owns the slot — git sets Revision, OCI sets Digest.

func ReadSlotMeta added in v0.3.4 

func ReadSlotMeta(slotDir string) (meta SlotMeta, ok bool)

ReadSlotMeta returns the slot's sidecar. A missing, unreadable, or unparseable sidecar yields ok=false — all three mean "no usable marker", so the caller rebuilds. Atomic writes guarantee the file is either absent or complete, so a parse failure indicates a pre-meta.json slot or external tampering, not a torn write.

func ReadSlotMetaFresh added in v0.3.4 

func ReadSlotMetaFresh(slotDir string, maxAge time.Duration) (meta SlotMeta, ok bool)

ReadSlotMetaFresh returns the sidecar only when it exists and was written within maxAge — the freshness gate a mutable ref (branch/tag/semver/HEAD) uses to skip a network refetch within its reconcile interval. A single open (fstat for mtime, then read) avoids a stat/read TOCTOU on a fetcher-written file. maxAge <= 0 disables the gate (always stale).

type Suspendable 

type Suspendable interface {
	Suspended() bool
}

Suspendable is satisfied by source CR types that carry a spec.suspend bool. The source controller short-circuits suspended objects before invoking the Fetcher.

type SweepOpts 

type SweepOpts struct {
	// MaxAge marks an entry stale when its mtime is older than now-MaxAge.
	// Stale entries are removed unless DryRun. Zero disables age-based
	// pruning (only orphan refs are cleaned).
	MaxAge time.Duration

	// IncludeMirrors enables age-based pruning of bare git mirrors at
	// <root>/git-mirrors. Mirrors are otherwise preserved because re-
	// hydrating them is expensive (a full clone over the network).
	// Set true when running an explicit "wipe stale state" pass.
	IncludeMirrors bool

	// DryRun reports what would be removed without touching disk.
	DryRun bool
}

SweepOpts tunes Sweep's behavior.

type SweepResult 

type SweepResult struct {
	// Removed lists every entry that was deleted (or would be under
	// DryRun). Paths are absolute. Group by parent dir to recover the
	// "kind" of cache that lost the entry — sources/, baselines/, etc.
	Removed []string
	// Bytes is the cumulative size of removed entries before deletion.
	// Computed during the same walk that decides removal so the sweep
	// stays O(files) overall.
	Bytes int64
	// Errors aggregates per-entry I/O errors. Sweep continues past
	// individual failures and reports them here so a single permissions
	// error on one slot doesn't abort the whole sweep.
	Errors []error
}

SweepResult summarizes what Sweep did (or would have done under DryRun).

func Sweep 

func Sweep(layout cacheroot.Layout, opts SweepOpts) (SweepResult, error)

Sweep prunes stale entries from the cache root described by layout using a mark-sweep strategy:

  1. MARK — read every refs/<category>/<key> file, build the set of digests still referenced from disk.
  2. SWEEP — walk the layout-managed subtrees (sources, baselines, blobs, optionally git-mirrors) and remove entries older than MaxAge. Blobs whose digest is in the live set survive regardless of age — a fresh ref must always resolve.
  3. ORPHAN refs — drop refs whose digest no longer materializes a blob. Runs regardless of MaxAge; these are dead pointers and refs cost nothing on disk anyway.

Mirrors are preserved by default; pass IncludeMirrors to age-prune them too. Individual I/O errors land in Result.Errors rather than short-circuiting the sweep.

type TempFiles 

type TempFiles struct {
	// contains filtered or unexported fields
}

TempFiles collects temp files written from secret material so a fetcher can hand their paths to a downstream tool that expects on-disk files — helm's getter (WithTLSClientConfig takes paths) or docker's file-backed credential store — then remove them all once the fetch completes.

dir selects the parent passed to os.CreateTemp: "" uses the system temp dir; a non-empty dir scopes the files under a caller-owned location (e.g. helm's per-fetch tmpDir). Files land with os.CreateTemp's default 0o600 perms.

One TempFiles belongs to one in-flight fetch; it is not safe for concurrent use.

func NewTempFiles 

func NewTempFiles(dir string) *TempFiles

NewTempFiles returns a collector that writes into dir. An empty dir uses the system temp directory.

func (*TempFiles) Cleanup 

func (t *TempFiles) Cleanup()

Cleanup removes every file a successful Write registered. It is a no-op when nothing was written, so it is safe to defer immediately after construction.

func (*TempFiles) Write 

func (t *TempFiles) Write(pattern, content string) (string, error)

Write materializes content to a fresh temp file (os.CreateTemp pattern semantics — the last "*" is replaced by a random string) and returns its path, registering it for Cleanup.

Empty content writes no file and returns ("", nil): callers materializing an optional secret key (e.g. ca.crt with no key) rely on the empty path to mean "absent". A write that fails removes only the partial file it just created; files from earlier successful Writes stay registered so the caller's deferred Cleanup reaps them.

type TypedFetcher 

type TypedFetcher[T manifest.BaseManifest] interface {
	Fetch(ctx context.Context, obj T) (*store.SourceArtifact, error)
}

TypedFetcher is the kind-specific Fetcher each concrete source kind implements (e.g. TypedFetcher[*manifest.GitRepository] for git). The typed signature removes the per-implementation `obj, ok := obj.(*manifest.X)` boilerplate every fetcher previously opened with — a missing assertion would have panicked. Wrap[T] turns a TypedFetcher into the untyped Fetcher the source controller's map needs.

Source Files

View all Source files

Directories

Path Synopsis
Package atomic provides a stage-then-rename WriteFile primitive.
 Package atomic provides a stage-then-rename WriteFile primitive.
Package blob is a small content-addressed storage layer for flate's fetched artifacts.
 Package blob is a small content-addressed storage layer for flate's fetched artifacts.
Package bucket implements the source.Fetcher for KindBucket (S3-compatible object storage via minio-go).
 Package bucket implements the source.Fetcher for KindBucket (S3-compatible object storage via minio-go).
Package cacheroot owns the layout of flate's on-disk cache.
 Package cacheroot owns the layout of flate's on-disk cache.
Package external implements source.TypedFetcher for ExternalArtifact.
 Package external implements source.TypedFetcher for ExternalArtifact.
git
Package git implements the source.Fetcher for KindGitRepository.
 Package git implements the source.Fetcher for KindGitRepository.
internal/gittransport
Package gittransport carries the shared HTTPS-transport install lock serialized across git.Fetcher and the bare-mirror cache.
 Package gittransport carries the shared HTTPS-transport install lock serialized across git.Fetcher and the bare-mirror cache.
mirror
Package mirror implements the bare-clone object store shared across GitRepository fetches.
 Package mirror implements the bare-clone object store shared across GitRepository fetches.
Package gittree materializes a git commit's tree to disk, writing every blob as a regular file (or real symlink, where applicable), in parallel.
 Package gittree materializes a git commit's tree to disk, writing every blob as a regular file (or real symlink, where applicable), in parallel.
Package helmchart implements the source.Fetcher for KindHelmChart — the single authoritative path that fetches a Helm chart (by name + version) from its backing HelmRepository.
 Package helmchart implements the source.Fetcher for KindHelmChart — the single authoritative path that fetches a Helm chart (by name + version) from its backing HelmRepository.
Package oci implements the source.Fetcher for KindOCIRepository via oras-go.
 Package oci implements the source.Fetcher for KindOCIRepository via oras-go.
Package safepath provides a path-traversal guard used by the OCI and bucket source packages.
 Package safepath provides a path-traversal guard used by the OCI and bucket source packages.
Package sourceignore builds the file-exclusion matcher Flux's source-controller applies when it packages a GitRepository/OCIRepository artifact: its default patterns (.git/, .github/, *.jpg/png/zip, .sops.yaml, .flux.yaml, .goreleaser.yml, …) plus any in-tree .sourceignore files plus caller-supplied spec.ignore patterns.
 Package sourceignore builds the file-exclusion matcher Flux's source-controller applies when it packages a GitRepository/OCIRepository artifact: its default patterns (.git/, .github/, *.jpg/png/zip, .sops.yaml, .flux.yaml, .goreleaser.yml, …) plus any in-tree .sourceignore files plus caller-supplied spec.ignore patterns.
Package ssrfguard is an opt-in SSRF egress guard for flate's outbound source fetches.
 Package ssrfguard is an opt-in SSRF egress guard for flate's outbound source fetches.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.