Documentation
¶
Overview ¶
Package source implements SourceController — reconciles GitRepository and OCIRepository resources by fetching the underlying artifact into a content-addressed on-disk cache and publishing a *store.GitArtifact or *store.OCIArtifact for downstream controllers.
Native libraries are used throughout:
- go-git for git clone / checkout. No `git` subprocess invocation.
- oras-go for OCI artifact pulls. No `crane` or `oras` subprocess.
The cache is keyed by SHA256(url + ref) so multiple revisions of the same upstream can co-exist on disk. A LocalRepository — a synthetic GitRepository pointing at the user's working directory — is also supported by the orchestrator.
Package source is the SDK adapter for Flux's source-controller CRs. It exposes per-kind Fetcher implementations (one per source type) plus a content-addressed disk Cache that all fetchers write into.
Adding a new source kind:
- Implement source.Fetcher for the new kind in a new file (or subpackage) under pkg/source.
- Register the fetcher with the orchestrator at construction time.
The source controller (pkg/controllers/source) does not know about individual kinds — it dispatches via the Fetchers map keyed by id.Kind, so adding a new kind is one registration call.
Index ¶
- func ApplyIgnore(root string, ignore *string) error
- func ApplyIgnoreNoDefaults(root string, ignore *string) error
- func AuthIdentity(parts ...string) string
- func BuildTLSConfig(crt, key, ca string) (*tls.Config, error)
- func MissingSecretErr(kind, ns, name, secretRef, reason string) error
- func ResolveCertSecret(secrets SecretGetter, ns, ownerKind, ownerID string, ...) (*tls.Config, error)
- func SecretRefID(ns, name string) string
- func StringFromSecret(sec *manifest.Secret, key string) string
- type Cache
- type ExistenceFetcher
- type Fetcher
- type ProxyConfig
- type SecretGetter
- type Slot
- type Suspendable
- type SweepOpts
- type SweepResult
- type TypedFetcher
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func ApplyIgnore ¶
ApplyIgnore deletes every file under root that matches the source- controller ignore matcher: VCS + Default excludes (.git/, .github/, *.jpg/png/zip, .sops.yaml, .flux.yaml, ...) PLUS any in-tree .sourceignore files PLUS the user-supplied spec.ignore patterns when non-nil. Mirrors source-controller's GitRepository / OCIRepository artifact-build behavior.
nil spec.ignore is NOT a no-op — the VCS + Default patterns still apply, matching what real Flux ships in a Git/OCI artifact tarball.
Bucket sources use ApplyIgnoreNoDefaults instead — see that function for the rationale.
func ApplyIgnoreNoDefaults ¶
ApplyIgnoreNoDefaults is the Bucket-flavored variant: it skips the VCS + Default exclude patterns and applies ONLY the in-tree .sourceignore plus the user-supplied spec.ignore. Mirrors fluxcd/source-controller/internal/controller/bucket_controller.go which uses sourceignore.NewMatcher (no defaults) rather than the NewDefaultMatcher that GitRepository / OCIRepository use.
The rationale: Bucket has no VCS semantics. An object store can legitimately carry .git/-named keys, .jpg/.png images, .flux.yaml, etc., and source-controller delivers them as-is. Stripping them would diverge from what a cluster Bucket reconcile produces.
func AuthIdentity ¶
AuthIdentity returns a deterministic, opaque tag identifying the auth context bound to a source fetch. It is appended to the cache key so two source CRs that target the same (URL, ref) but reference different SecretRefs do not collide on the same on-disk slot.
Inputs are usually `<ns>/<secretRef.Name>` strings; the helper trims empties and joins the rest with NULs so adding a new auth dimension later (cert secret, proxy secret, …) is a one-arg append.
Returns "" when every input is empty — the caller passes "" to Cache.Slot in that case, and slots match the legacy unauthenticated layout so existing caches survive.
func BuildTLSConfig ¶
BuildTLSConfig assembles a *tls.Config from PEM-armored secret material. Any subset of (client cert + key) and (CA) is acceptable; crt/key must appear together if either is present. Returns an error when all inputs are empty so callers can distinguish "no TLS configured" from "malformed config".
MinVersion is pinned to TLS 1.2 to match source-controller's defaults. Callers that need TLS 1.3-only behavior can adjust the returned config.
Common Flux Secret keys feeding this helper:
tls.crt / tls.key → client certificate (mTLS) ca.crt → trust roots for the server
func MissingSecretErr ¶
MissingSecretErr wraps manifest.ErrMissingSecret so the source controller's --allow-missing-secrets path matches via errors.Is.
func ResolveCertSecret ¶
func ResolveCertSecret(secrets SecretGetter, ns, ownerKind, ownerID string, ref *manifest.LocalObjectReference) (*tls.Config, error)
ResolveCertSecret reads ref's Secret and builds a *tls.Config from its tls.crt + tls.key (client cert) and/or ca.crt (server CA) keys — mirroring source-controller's certSecretRef schema.
Returns (nil, nil) when ref is nil. Loud errors when ref is set but SecretGetter is unwired or the Secret is missing/unparseable. ownerKind+ownerID are formatted into errors for diagnosability.
func SecretRefID ¶
SecretRefID renders a Flux LocalObjectReference into the `<namespace>/<name>` shape AuthIdentity expects. ns is the owning CR's namespace (SecretRefs are LocalObjectReferences — they inherit it). Returns "" when name is empty so optional refs slot in as no-op zero values.
func StringFromSecret ¶
StringFromSecret reads a key from a Secret, preferring StringData over Data. Data values are base64-decoded (per k8s Secret semantics) before being returned, so the same string surface holds regardless of whether the source manifest used `data:` or `stringData:`. PLACEHOLDER_-wiped values (the result of flate's secret wiping pre-processing) are reported as empty so callers surface a clear "missing keys" error rather than authenticating with the literal placeholder.
Used by per-kind Fetchers (git, oci, bucket) and cosign verification to resolve auth + trust material from the Secret a SecretRef points at.
Types ¶
type Cache ¶
type Cache struct {
// contains filtered or unexported fields
}
Cache manages a content-addressed on-disk directory for fetched sources. Each (url, ref) tuple gets its own slot, so multiple revisions of the same upstream coexist without clobbering one another.
The cache is safe for concurrent use. A per-slot mutex serializes the full fetch-write-read lifecycle on a single slot — two distinct source CRs with the same (url, ref) hash to the same slot, and without per-slot locking one would observe the other mid-write (e.g. read an empty marker, call Reset, wipe the in-progress clone). Different slots proceed in parallel.
func NewCache ¶
NewCache constructs a Cache backed by the supplied Layout. The caller-owned Layout is the single source of truth for cache paths; this Cache asks it for SourceSlot positions and never composes its own. When the Layout's Root is empty, a flate-cache subdirectory under os.TempDir() is used so embedders that don't wire CacheDir still work.
func (*Cache) Slot ¶
Slot allocates a per-(url, ref) slot under the cache root with atomic-rename finalization. Holds the slot mutex until Release is called; serializes against other Slot acquisitions on the same key.
The returned Slot's Path is:
- The final slot directory when Exists is true (cache hit — the fetcher reads from it directly).
- A sibling staging directory when Exists is false (cache miss — the fetcher writes into it, then calls Commit to atomic-rename into the final slot).
The staging dir lives at `<final>.tmp.<rand>`, on the same filesystem as the final, so os.Rename is atomic per POSIX. If the fetcher returns without committing (any error path, any panic), Release removes the staging dir and the final slot is left absent — the next fetch starts clean. This atomicity replaces the older "write in place + .flate-* sentinels" pattern: the final slot is either complete or doesn't exist, never torn.
type ExistenceFetcher ¶
type ExistenceFetcher struct{}
ExistenceFetcher is the no-op Fetcher registered for kinds whose existence alone is enough to satisfy downstream waits — used today for HelmRepository (always) and OCIRepository when EnableOCI is false. Returning nil artifact + nil error lands the resource in Ready without recording a SourceArtifact, so a HelmRelease that dependsOn a HelmRepository unblocks instantly without flate having to mirror the controllers' "did fetch succeed?" logic from outside the controller package.
func (ExistenceFetcher) Fetch ¶
func (ExistenceFetcher) Fetch(_ context.Context, _ manifest.BaseManifest) (*store.SourceArtifact, error)
Fetch implements source.Fetcher as a no-op — see ExistenceFetcher.
type Fetcher ¶
type Fetcher interface {
Fetch(ctx context.Context, obj manifest.BaseManifest) (*store.SourceArtifact, error)
}
Fetcher resolves a single source CR into an on-disk artifact. The source controller stores Fetchers in a map keyed by Kind and dispatches via this untyped interface. Concrete implementations satisfy it through Wrap[T] — see TypedFetcher.
func Wrap ¶
func Wrap[T manifest.BaseManifest](kindLabel string, f TypedFetcher[T]) Fetcher
Wrap converts a TypedFetcher into the untyped Fetcher interface used by the source-controller dispatcher map. The single type-assertion site is here — a mismatched payload returns ErrInput rather than panicking. Embeds the Kind label so the error message names the responsible fetcher.
type ProxyConfig ¶
ProxyConfig is the resolved view of a Flux spec.proxySecretRef. Matches source-controller's Secret schema: data.address (required), data.username + data.password (optional, basic-auth).
func ResolveProxy ¶
func ResolveProxy(secrets SecretGetter, ns, ownerKind, ownerID string, ref *manifest.LocalObjectReference) (*ProxyConfig, error)
ResolveProxy reads ref's Secret via secrets and decodes it into a ProxyConfig. Returns (nil, nil) when ref is nil — proxy is opt-in. Surfaces a loud error when ref is set but the Secret is missing or lacks the required address key, matching source-controller's fail-loud behavior.
func (*ProxyConfig) HTTPProxyFunc ¶
HTTPProxyFunc returns a net/http.Transport.Proxy function pinned to this proxy. Use when configuring an http.Transport for the OCI, Bucket, or HTTP-Git transports.
type SecretGetter ¶
SecretGetter resolves a Secret CR by namespace + name. Fetchers that read authentication, TLS, proxy, or cosign-verify material from a Flux spec.*SecretRef accept one of these so they don't need a back-reference to the Store. Today: GitRepository (auth + TLS), OCIRepository (auth + TLS + cosign verify), Bucket (auth + TLS). The orchestrator wires it to Store.GetByName at construction time.
type Slot ¶
type Slot struct {
// Path is where the fetcher reads / writes:
// Exists == true → Path is the final slot (read-only use).
// Exists == false → Path is the staging dir (write here, then
// Commit to finalize).
Path string
// Exists reports whether the final slot was already populated by
// a prior fetch. When true, the staging dance is skipped and
// Path is the final slot directly.
Exists bool
// contains filtered or unexported fields
}
Slot is one allocated cache slot. Acquired by Cache.Slot, released by the caller's deferred Release. On a cache miss the fetcher writes into Path (a staging dir) and calls Commit to atomic-rename into the final slot; on a cache hit Path is already the final slot.
func (*Slot) Commit ¶
Commit finalizes a successful fetch: atomic-rename the staging dir over the final slot. No-op on a cache hit (Exists == true). Safe to call multiple times. After a successful commit, Path is updated to the final slot so subsequent reads work uniformly.
func (*Slot) Release ¶
func (s *Slot) Release()
Release drops the slot mutex AND, if Commit wasn't called, removes the orphan staging dir. MUST be deferred by every Cache.Slot caller. Safe to call multiple times — second+ calls are no-ops.
func (*Slot) Reset ¶
Reset wipes the final slot — used by callers that detected a stale cache hit (e.g. cosign signature changed against the cached digest). After Reset the slot looks like an Exists=false miss; the caller can write to a new staging via a fresh Cache.Slot call, OR can call Stage on this same Slot to allocate staging in place.
type Suspendable ¶
type Suspendable interface {
Suspended() bool
}
Suspendable is satisfied by source CR types that carry a spec.suspend bool. The source controller short-circuits suspended objects before invoking the Fetcher.
type SweepOpts ¶
type SweepOpts struct {
// MaxAge marks an entry stale when its mtime is older than now-MaxAge.
// Stale entries are removed unless DryRun. Zero disables age-based
// pruning (only orphan refs are cleaned).
MaxAge time.Duration
// IncludeMirrors enables age-based pruning of bare git mirrors at
// <root>/git-mirrors. Mirrors are otherwise preserved because re-
// hydrating them is expensive (a full clone over the network).
// Set true when running an explicit "wipe stale state" pass.
IncludeMirrors bool
// DryRun reports what would be removed without touching disk.
DryRun bool
}
SweepOpts tunes Sweep's behavior.
type SweepResult ¶
type SweepResult struct {
// Removed lists every entry that was deleted (or would be under
// DryRun). Paths are absolute. Group by parent dir to recover the
// "kind" of cache that lost the entry — sources/, baselines/, etc.
Removed []string
// Bytes is the cumulative size of removed entries before deletion.
// Computed during the same walk that decides removal so the sweep
// stays O(files) overall.
Bytes int64
// Errors aggregates per-entry I/O errors. Sweep continues past
// individual failures and reports them here so a single permissions
// error on one slot doesn't abort the whole sweep.
Errors []error
}
SweepResult summarizes what Sweep did (or would have done under DryRun).
func Sweep ¶
func Sweep(layout cacheroot.Layout, opts SweepOpts) (SweepResult, error)
Sweep prunes stale entries from the cache root described by layout using a mark-sweep strategy:
- MARK — read every refs/<category>/<key> file, build the set of digests still referenced from disk.
- SWEEP — walk the layout-managed subtrees (sources, baselines, blobs, optionally git-mirrors) and remove entries older than MaxAge. Blobs whose digest is in the live set survive regardless of age — a fresh ref must always resolve.
- ORPHAN refs — drop refs whose digest no longer materializes a blob. Runs regardless of MaxAge; these are dead pointers and refs cost nothing on disk anyway.
Mirrors are preserved by default; pass IncludeMirrors to age-prune them too. Individual I/O errors land in Result.Errors rather than short-circuiting the sweep.
func (*SweepResult) Log ¶
func (res *SweepResult) Log()
Log emits a structured summary of res at slog.LevelInfo. Convenience for CLI / orchestrator callers that don't want to format the result inline.
type TypedFetcher ¶
type TypedFetcher[T manifest.BaseManifest] interface { Fetch(ctx context.Context, obj T) (*store.SourceArtifact, error) }
TypedFetcher is the kind-specific Fetcher each concrete source kind implements (e.g. TypedFetcher[*manifest.GitRepository] for git). The typed signature removes the per-implementation `obj, ok := obj.(*manifest.X)` boilerplate every fetcher previously opened with — a missing assertion would have panicked. Wrap[T] turns a TypedFetcher into the untyped Fetcher the source controller's map needs.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
Package atomic carries the small "stage-then-rename" file write helper every flate cache writer used to reinvent (helm.writeAtomic, oci.writeCachedDigest, oci.writeVerifyMarker, blob.Refs.Put).
|
Package atomic carries the small "stage-then-rename" file write helper every flate cache writer used to reinvent (helm.writeAtomic, oci.writeCachedDigest, oci.writeVerifyMarker, blob.Refs.Put). |
|
Package blob is a small content-addressed storage layer for flate's fetched artifacts.
|
Package blob is a small content-addressed storage layer for flate's fetched artifacts. |
|
Package bucket implements the source.Fetcher for KindBucket (S3-compatible object storage via minio-go).
|
Package bucket implements the source.Fetcher for KindBucket (S3-compatible object storage via minio-go). |
|
Package cacheroot owns the layout of flate's on-disk cache.
|
Package cacheroot owns the layout of flate's on-disk cache. |
|
Package external implements the source.Fetcher for KindExternalArtifact (third-party-published artifacts under Flux's source-controller schema).
|
Package external implements the source.Fetcher for KindExternalArtifact (third-party-published artifacts under Flux's source-controller schema). |
|
Package git implements the source.Fetcher for KindGitRepository.
|
Package git implements the source.Fetcher for KindGitRepository. |
|
Package gittree materializes a git commit's tree to disk, writing every blob as a regular file (or real symlink, where applicable), in parallel.
|
Package gittree materializes a git commit's tree to disk, writing every blob as a regular file (or real symlink, where applicable), in parallel. |
|
Package oci implements the source.Fetcher for KindOCIRepository via oras-go.
|
Package oci implements the source.Fetcher for KindOCIRepository via oras-go. |