loader

package
v0.3.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 7, 2026 License: AGPL-3.0 Imports: 22 Imported by: 0

Documentation

Overview

Package loader implements Loader — the entry point that scans a local filesystem path, decodes every YAML document found, and adds the recognized Flux objects to a Store.

Loader honors a `.krmignore`-style ignore file at the scan root. Unrecognized objects (Pods, Deployments, etc.) are silently skipped to match flux-local's pre-filtering: flate derives them later from the rendered output of Kustomizations and HelmReleases.

Package loader hydrates a Store from on-disk Flux manifests.

The loader's discovery model mirrors `kustomize build` (and flux-local): the kustomize resource graph is the source of truth. A directory with a kustomization.yaml defines a kustomize package — the loader follows its `resources:` entries (files load, directories recurse) and ignores everything else in the directory. Files outside the resource graph are invisible by construction; there is no "tree walk + filter" post-pass, no orphan-skip rule, no reachability set computed up front.

Entry points without a kustomization.yaml use a fall-back tree walk that loads every YAML it finds and switches into graph-walk mode when it encounters a subdirectory that IS a kustomize package. This keeps the bootstrap-style "bare directory of CRs" shape working without forcing every user to wrap their entry point in a kustomization.yaml.

Each loader.Load call is one independent graph root. The orchestrator's iterative discovery — a Flux KS's spec.path triggers another Load — composes naturally: each spec.path is its own graph root with its own resource graph.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func ApplyDefaultNamespaces 

func ApplyDefaultNamespaces(s *store.Store, sourceFiles map[manifest.NamedResource]string)

ApplyDefaultNamespaces preserves flate's flux-system fallback for top-level Flux Operator and HelmChart source objects after namespace inheritance has had a chance to project kustomize/Flux namespaces.

func ApplyNamespaceInheritance 

func ApplyNamespaceInheritance(s *store.Store, sourceFiles map[manifest.NamedResource]string, repoRoot string)

ApplyNamespaceInheritance is ApplyNamespaceInheritanceWithRefs without a consumer→source ref map (sourceRefs resolution skipped).

func ApplyNamespaceInheritanceWithRefs 

func ApplyNamespaceInheritanceWithRefs(s *store.Store, sourceFiles map[manifest.NamedResource]string, sourceRefs map[manifest.NamedResource][]manifest.NamedResource, repoRoot string)

ApplyNamespaceInheritanceWithRefs fills empty metadata.namespace fields on loaded resources from the nearest enclosing namespace directive — either a Flux Kustomization's spec.targetNamespace or a kustomization.yaml `namespace:` field. This is the load-time analog of kustomize-controller's apply-time behavior, without which the store ends up with two copies of the same resource (one with the inherited namespace, one with namespace=""). repoRoot anchors the kustomization.yaml lookups; sourceFiles is mutated as ids are rewritten.

sourceRefs (the discovery-supplied consumer→source ref map; nil to skip) is resolved in step with each consumer's inherited namespace: recordSourceRefs captures those edges at PARSE time, so a chartRef/sourceRef that omits its namespace carries an empty-namespace target. Render-driven HelmReleases are never in the Store, so the Store-object pass below can't fix them — resolving the target to the consumer's inherited namespace here (Flux semantics: an omitted ref namespace tracks the consumer's namespace) keeps changed-only mode's keep-set walk from missing a changed HelmRelease's chart source.

func BuildParentIndexForKind 

func BuildParentIndexForKind(s *store.Store, repoRoot string, sourceFiles map[manifest.NamedResource]string, childKind string) map[manifest.NamedResource]manifest.NamedResource

BuildParentIndexForKind maps each childKind resource to its enclosing Flux Kustomization — the KS whose spec.path or component directory is the deepest strict ancestor of the child's source file. Excludes self-matches.

Real Flux's reconcile chain enforces this naturally: a parent Kustomization renders and applies its children, then the downstream controller reconciles each. flate's controllers fire on AddObject and would otherwise race the parent's render — the child controllers use this index to gate reconcile on the parent's Ready, so any parent-render-time spec mutations (`replacements:` injecting spec.targetNamespace, `patches:` rewriting HelmRelease driftDetection) are visible to the child's first reconcile. Without the gate the file-loaded child renders once with stale spec, the parent re-emits a mutated copy, and the child renders again — twice the helm template / kustomize build work for one logical resource.

sourceFiles is the orchestrator's NamedResource → repo-relative source-file map; entries without a recorded file are skipped.

childKind=KindKustomization for the KS→KS parent map; pass KindHelmRelease for the HR→KS map. The orchestrator builds both (see discovery.Run → mergeParents).

repoRoot is the filesystem root used to read each KS's kustomization.yaml when folding `components:` into the prefix set; pass the orchestrator's --path. An empty repoRoot means "no on-disk component lookup", which still gives a correct (just slightly less-precise) index built from spec.path + spec.components alone.

func BuildParentIndexForKindWithCache 

func BuildParentIndexForKindWithCache(s *store.Store, repoRoot string, sourceFiles map[manifest.NamedResource]string, childKind string, cache *manifest.ComponentCache) map[manifest.NamedResource]manifest.NamedResource

BuildParentIndexForKindWithCache is BuildParentIndexForKind with a shared *manifest.ComponentCache threaded into the KSPathPrefixes call. Used by discovery so the KS-parent-map build and the HR-parent-map build share component-file reads across the two passes — without sharing, each invocation walks the same KS list and re-reads every kustomization.yaml's `components:` independently.

func BuildParentIndexFromPrefixes 

func BuildParentIndexFromPrefixes(prefixes []KSPathPrefix, s *store.Store, sourceFiles map[manifest.NamedResource]string, childKind string) map[manifest.NamedResource]manifest.NamedResource

BuildParentIndexFromPrefixes is BuildParentIndexForKindWithCache with the KS path-prefix list passed in precomputed. discovery.Run derives the prefixes once (an O(KS) walk + sort + component reads) and reuses them for the KS-parent index, the HR-parent index, AND orphan promotion — three consumers that previously each rebuilt the identical list. Standalone callers use BuildParentIndexForKind(WithCache), which compute the prefixes for a single use.

func LongestParent 

func LongestParent(prefixes []KSPathPrefix, file string, self manifest.NamedResource) (manifest.NamedResource, bool)

LongestParent returns the deepest KS whose spec.path covers file (slash-normalized repo-relative path), excluding self. The second return reports whether a parent was found. prefixes is expected to be the sorted output of KSPathPrefixes.

func NormalizePrefix 

func NormalizePrefix(p string) string

NormalizePrefix turns a Kustomization spec.path into a slash- terminated repo-relative prefix suitable for HasPrefix matching. Applies filepath.ToSlash first so Windows-style spec.path values (rare, but possible since the Flux CRD doesn't constrain it) normalize to the same shape as loader.SourceFiles entries.

func ResolveDependsOnSubstitutions 

func ResolveDependsOnSubstitutions(s *store.Store)

ResolveDependsOnSubstitutions resolves bare ${VAR} references in Kustomization spec.dependsOn names/namespaces using the cluster's postBuild substitute values, mirroring how kustomize-controller substitutes a child KS's text (via its parent's postBuild) before the dependency is ever matched. flate builds its dependency graph from the raw file-parsed KS — where `dependsOn: 0-${CLUSTER_NAME}-config` (no default) survives ResolveEnvsubstDefaults and never matches the real `0-biohazard-config` — so without this pass the dependency is reported "not found".

${VAR:=default} forms are already collapsed at parse time (ResolveEnvsubstDefaults), so this pass only ever resolves *bare* vars, drawing values from the union of every discovered KS's spec.postBuild.substitute map. Two guards keep it from manufacturing a false dependency match:

  • a var declared with conflicting values across Kustomizations (e.g. per-cluster CLUSTER_NAME in a multi-cluster repo) is dropped from the union and left literal;
  • a var with no value leaves envsubst.Eval in its error path, so the original name is kept verbatim rather than collapsed to empty.

Run once, after the full KS set is discovered (so the union is complete and the conflict check is sound) and before the dependency graph is built. Idempotent: a KS's own name carries no var (templated names are skipped at load), so its store id is invariant, and a resolved name contains no ${ left to re-resolve.

func StampTransformerTargetNamespaces 

func StampTransformerTargetNamespaces(s *store.Store, sourceFiles map[manifest.NamedResource]string, repoRoot string)

StampTransformerTargetNamespaces fills empty spec.targetNamespace on file-loaded Flux Kustomizations from a builtin NamespaceTransformer that an enclosing kustomize overlay applies.

flatops-style repos keep resources namespace-less "for DRYness" and inject the namespace via a shared NamespaceTransformer that sets spec.targetNamespace on every Kustomization (issue #528). That injection only happens when the overlay is kustomize-built — and when the overlay is rendered by a further-up, itself render-emitted parent, the injection lands *after* flate has already fired the leaf KS's first reconcile. The leaf then renders its children with an empty targetNamespace, producing an empty-namespace copy of each HelmRelease that lingers in the store and later fails to render (no namespace to resolve its chart's HelmRepository against).

Resolving the namespace here, at load time, lets the leaf KS render into the right namespace on its first pass — the load-time analog of the same NamespaceTransformer kustomize would apply, mirroring how ApplyNamespaceInheritance front-runs kustomize-controller's apply-time namespace defaulting.

Only spec.targetNamespace is set (never metadata.namespace), so the KS's store id stays stable. The value reaches rendered children via ks.Contents (RenderFlux feeds it to kustomize) and reaches store-resident namespace-less resources under spec.path via ApplyNamespaceInheritance's existing projection, which now sees a populated targetNamespace.

Runs before ApplyNamespaceInheritance (see discovery.applyNamespaces).

Types

type ExistenceIndex 

type ExistenceIndex struct {
	// contains filtered or unexported fields
}

ExistenceIndex records "object with this id was parsed from this file" without committing it to the store. Populated by the loader under DiscoveryOnly: HelmReleases, sources, ConfigMaps, Secrets, and other reconcilable kinds skip AddObject and land here instead.

Three consumers:

  • depwait's missing-dep fallback. A KS that names a CM via substituteFrom blocks on existence; if the CM lives in a sibling KS's spec.path the file is in the index even though the store doesn't have it yet. Lazy-loading from the indexed path unblocks the depwait without forcing the rendered version to materialize first — which is impossible for self-rendering patterns like bjw-s where the parent KS's substituteFrom CM is rendered by that same parent KS.

  • orchestrator orphan promotion. After discovery, any index entry whose file path is not under any KS's spec.path is promoted to the store via AddObject. Standalone CRs (e.g. a loose HR file at repo root with no enclosing KS) keep working through `flate build`.

  • change.Detect sees the file path through the loader's SourceFiles map regardless of whether the object reached the store. The index doesn't duplicate that bookkeeping.

Thread-safe: the loader walks files concurrently from one goroutine today, but lazy-promotion from depwait can be invoked from reconcile-worker goroutines. Cheap RW mutex keeps the contract clear if either side grows concurrency later.

func NewExistenceIndex 

func NewExistenceIndex() *ExistenceIndex

NewExistenceIndex returns an empty index ready for Record / Get.

func (*ExistenceIndex) All 

func (i *ExistenceIndex) All() map[manifest.NamedResource]string

All yields a snapshot of every recorded {id, path}. The returned map is a fresh copy so callers can iterate without holding the lock or worrying about concurrent Record.

func (*ExistenceIndex) Get 

func (i *ExistenceIndex) Get(id manifest.NamedResource) (string, bool)

Get returns the indexed file path for id and whether the index has an entry. The returned path is absolute (whatever Record was called with).

func (*ExistenceIndex) Promote 

func (i *ExistenceIndex) Promote(st *store.Store, id manifest.NamedResource, wipeSecrets bool) bool

Promote materializes id into st by re-parsing the file the ExistenceIndex recorded for it and AddObject'ing every Flux CR in the file. Returns true on success; false if the id is unknown to the index or its file can no longer be parsed.

The whole file is parsed (not just id) because YAML multi-doc files often pack a CM + its Secret + a HelmRelease together — promoting one frequently means callers will need a sibling next. Parsing once and AddObject'ing every doc avoids re-opening the same file repeatedly when several lazy lookups land on it.

PreferExisting semantics are honored: if an id already lives in the store (already promoted, or render-emitted by a KS), the existing object stays put. wipeSecrets matches the loader's: callers should pass through whatever DiscoveryOnly used at file- load time so SOPS Secrets stay wiped on promotion the same way they were skipped at load.

func (*ExistenceIndex) Record 

func (i *ExistenceIndex) Record(id manifest.NamedResource, absPath string)

Record associates id with the absolute file path it was parsed from. Subsequent Record calls for the same id overwrite — the loader's PreferExisting branch handles ordering across multiple scan roots before reaching here.

type KSPathPrefix 

type KSPathPrefix struct {
	ID     manifest.NamedResource
	Prefix string
}

KSPathPrefix pairs a Kustomization id with one of its slash-terminated, repo-relative claimed-path prefixes. A KS may produce multiple prefixes — one for spec.path plus one per spec.components entry, plus any on-disk `components:` referenced from the kustomization.yaml living at spec.path. Sharing the same ID across multiple prefixes is intentional: parent-index lookup returns the longest-matching entry, and a child file inside a component directory is correctly attributed to the parent that includes that component.

func KSPathPrefixes 

func KSPathPrefixes(s *store.Store, repoRoot string) []KSPathPrefix

KSPathPrefixes returns one or more entries per loaded Kustomization with a non-empty spec.path. Each KS contributes:

  1. Its spec.path (always).
  2. Each spec.components entry (when present, resolved against spec.path).
  3. Each entry from `components:` declared in the kustomization.yaml at spec.path (when readable from repoRoot; missing or malformed files are silently skipped — pure best-effort, the spec.path entry is enough to keep the index sound).

Entries are sorted by prefix length descending so the first HasPrefix match on a given file is the deepest claimant — a child file under a parent's component dir wins over the parent's spec.path. Previously this function only emitted (1); the new (2)+(3) bring loader's parent index in line with change/ownership's already-richer attribution, eliminating the false-orphan class where a child KS lives inside a parent's component subtree.

repoRoot is the filesystem root the kustomization-file reads resolve relative to. Pass "" to skip on-disk component lookup entirely (only spec.path + spec.components are recorded).

On-disk component reads route through a local cache so a single call doesn't re-read the same kustomization.yaml across KSes that share a spec.path. Cross-call sharing is the orchestrator's job — see KSPathPrefixesWithCache.

func KSPathPrefixesWithCache 

func KSPathPrefixesWithCache(s *store.Store, repoRoot string, cache *manifest.ComponentCache) []KSPathPrefix

KSPathPrefixesWithCache is KSPathPrefixes with a shared component cache threaded in. The orchestrator instantiates one cache per Bootstrap and passes it to every consumer (discovery's orphan promotion, BuildParentIndexForKind, the orchestrator's finalize detectOrphans, change.buildOwnership) so the kustomization.yaml at each spec.path is read once per Bootstrap instead of once per consumer. Pass nil to fall back to the per-call cache.

type Loader 

type Loader struct {
	Store   *store.Store
	Options Options

	// SourceRoot, when non-empty, is the directory used as the
	// reference point for SourceFiles. Paths recorded there are
	// slash-separated and relative to this root, which matches the
	// shape change.Detect produces.
	SourceRoot string

	// SourceFiles is populated as each manifest is added. Keyed by
	// the parsed resource's NamedResource. Nil disables tracking.
	SourceFiles map[manifest.NamedResource]string

	// SourceRefs maps each loaded consumer (HelmRelease / Kustomization)
	// to the source resources it references — an HR's chart source, a
	// KS's spec.sourceRef. Captured at parse time because under
	// DiscoveryOnly an HR never reaches the Store, yet the change
	// filter's reverse edge needs to know which HelmReleases a changed
	// source feeds (so bumping a centralized OCIRepository's tag
	// re-renders its consumers). Nil disables tracking. Keyed by the
	// consumer's NamedResource; mirrors SourceFiles' lifecycle.
	SourceRefs map[manifest.NamedResource][]manifest.NamedResource

	// PreferExisting suppresses overwrites of resources already in
	// the store (and their SourceFiles entries). Used by the
	// orchestrator's recursive spec.path discovery so the initial
	// --path scan's data wins over downstream paths that may point
	// into a different tree.
	PreferExisting bool

	// Existence captures every file-loaded object that DiscoveryOnly
	// keeps out of the Store. Nil disables the bookkeeping (the
	// default for non-DiscoveryOnly callers).
	Existence *ExistenceIndex

	// ComponentCache, when non-nil, is the shared component-file
	// cache used by FinalizeGenerators' KSPathPrefixes call. Wired
	// by the orchestrator at Bootstrap so the loader, discovery's
	// parent-index passes, the orphan-promotion pass, the orchestrator's
	// finalize detectOrphans, and change.buildOwnership all read each
	// kustomization.yaml's `components:` field at most once per
	// Bootstrap. nil falls back to per-call caches with no cross-call
	// sharing (the pre-1.A behavior).
	ComponentCache *manifest.ComponentCache
	// contains filtered or unexported fields
}

Loader walks a directory tree and adds Flux objects to a Store.

func New 

func New(s *store.Store) *Loader

New returns a Loader configured to wipe secrets.

func (*Loader) FinalizeGenerators 

func (l *Loader) FinalizeGenerators(repoRoot string)

FinalizeGenerators materializes every harvested configMapGenerator/ secretGenerator entry into the store. Effective namespace is resolved per kustomize precedence: the entry's own namespace wins, then the kustomization.yaml's namespace, then the enclosing Flux Kustomization's namespace (looked up via KSPathPrefixes against the source file).

Synthesized objects honor PreferExisting: if an explicit CM/Secret with the same id already came from a real on-disk YAML, we don't overwrite it with the generated placeholder.

repoRoot is the filesystem root used to compute slash-relative paths for the parent lookup; pass the same root the change filter is keyed against (the Flux KS spec.path prefixes are repoRoot-relative).

func (*Loader) Load 

func (l *Loader) Load(ctx context.Context, root string) (int, error)

Load discovers Flux objects under root by walking the kustomize resource graph. When root has a kustomization.yaml it's treated as a kustomize package and only files reachable through `resources:` are loaded; when it has none, a recursive walk finds and enters kustomize packages it encounters, loading every YAML otherwise.

Honors ctx cancellation; visited-set protects against cycles.

type Options 

type Options struct {
	// WipeSecrets controls Secret cleartext replacement. Default true.
	WipeSecrets bool

	// DiscoveryOnly restricts file-loaded kinds that reach the Store
	// to the discovery-meta set: Kustomization, ResourceSet, and
	// ResourceSetInputProvider. Every other Flux CR (HelmRelease,
	// sources, ConfigMap, Secret) is recorded in Existence instead of
	// AddObject'd, matching real Flux's render-driven discovery
	// model where only the bootstrap KS is static and the rest of
	// the cluster materializes through KS reconciles. depwait
	// consults the existence index on missing deps; orchestrator
	// orphan-promotes any index entry not under a KS's spec.path
	// before reconcile starts.
	//
	// Why RS + RSIP stay in-scope: the discovery loop renders
	// ResourceSets to discover further KSes (RSIPs feed selectors,
	// RSes produce KSes/RSIPs). There is no ResourceSet controller
	// yet, so render-emitted RSes would never be processed; keeping
	// them file-loaded preserves the meta-discovery fixed point.
	DiscoveryOnly bool
}

Options tunes the Loader.

type SelfProduceIndex 

type SelfProduceIndex struct {
	// contains filtered or unexported fields
}

SelfProduceIndex maps a ConfigMap id (kind + resolved namespace + name) to the Flux Kustomization(s) whose OWN render subtree emits it.

It exists for one job: let collectDeps tell a self-substitute apart from a cross-KS substituteFrom. A Kustomization that renders the very ConfigMap it lists in postBuild.substituteFrom (the bjw-s/onedr0p `cluster-apps` → `cluster-settings` pattern: a bare-dir spec.path whose subdir bases pull in a component defining the CM) must NOT hard-wait on that CM — no other reconcile can produce it, so depwait would deadlock the consumer against its own render. A CM produced by a DIFFERENT KS, or genuinely absent, is left out of this index so its dependency edge stays and still fails loudly.

Built once per Bootstrap (in discovery.Run) and read-only afterwards.

func BuildSelfProduceIndex 

func BuildSelfProduceIndex(s *store.Store, repoRoot string) *SelfProduceIndex

BuildSelfProduceIndex resolves, per Flux Kustomization with a spec.path, which ConfigMaps its own kustomize render emits and in which namespace — by walking the render graph the loader's discovery pass does not attribute to a producer: bare-dir base generation (each immediate subdir holding a kustomization file becomes a base, mirroring Flux's own generator), recursion through `resources:` (files + nested bases) and `components:`, and propagation of each layer's `namespace:` transformer. repoRoot anchors the on-disk reads; an empty repoRoot yields an empty (but usable) index.

func (*SelfProduceIndex) ProducedBy 

func (i *SelfProduceIndex) ProducedBy(cm manifest.NamedResource) []manifest.NamedResource

ProducedBy returns the Kustomizations whose render subtree emits cm. Nil-safe: a nil index (no repoRoot, stripped-down tests) produces no matches, so collectDeps falls back to the always-add edge behavior.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.