orchestrator

package
v0.4.7 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 14, 2026 License: AGPL-3.0 Imports: 36 Imported by: 0

Documentation

Overview

Package orchestrator wires the controllers together and runs the reconcile loop. It owns:

  • A central Store.
  • A task.Service tracking active reconciliations.
  • The Source, Kustomization, and HelmRelease controllers.
  • The loader that primes the Store with on-disk objects.

The lifecycle is:

o := orchestrator.New(orchestrator.Config{...})
if err := o.Bootstrap(ctx); err != nil { ... }
if err := o.Run(ctx); err != nil { ... }       // blocks until done
o.Stop()

Bootstrap loads YAML manifests from the configured path AND publishes a synthetic GitRepository pointing at the local working tree, so Kustomizations whose sourceRef resolves to the bootstrap repo can reconcile immediately.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Config

type Config struct {
	// Path is the directory to scan for Flux objects (the scan entry
	// point — a cluster's Flux entry, which may be a subdir of RepoRoot).
	Path string
	// RepoRoot is the source root that Kustomization spec.path values
	// resolve against. Supplied explicitly by SDK consumers rendering
	// extracted trees that have no .git; the CLI defaults it to the .git
	// ancestor of Path. Empty ⇒ the .git walk (FindRepoRoot) runs,
	// preserving local behavior.
	RepoRoot string
	// SelfURLs are the remote URL(s) this tree represents, for
	// self-referential GitRepository aliasing (a cluster pulling itself).
	// Supplied explicitly by SDK consumers rendering extracted trees with
	// no .git/config; empty ⇒ the working tree's .git remotes are read.
	SelfURLs []string
	// PathOrig, when non-empty, switches every command into
	// changed-only mode: only resources whose source files differ
	// (plus the sources they reference) get reconciled.
	PathOrig string

	// HelmOptions tunes templating (skip CRDs/secrets/tests, kube
	// version, etc.).
	HelmOptions helm.Options
	// WipeSecrets controls Secret cleartext placeholders.
	WipeSecrets bool
	// AllowMissingSecrets converts source auth-secret-not-found errors
	// into skips and omits HelmRelease valuesFrom Secret/ConfigMap refs
	// that cannot materialize in the offline tree. Skipped source
	// resources mark Ready with a "skipped:" reason; omitted valuesFrom
	// refs let HelmReleases render with the remaining values.
	AllowMissingSecrets bool

	// RestrictEgress is the untrusted-render switch. It (1) enables the SSRF
	// egress guard for ALL source fetches (kustomize remote resources/bases +
	// Git/OCI/Helm/Bucket sources) — outbound dials to loopback/RFC1918/link-
	// local/cloud-metadata addresses are refused — and (2) rejects GitRepository
	// URLs whose transport is not https/ssh (file://, git://, http://, bare
	// paths), so a fork-PR file:// source can't disclose an arbitrary in-pod
	// repo (#743). Off by default — flate normally renders TRUSTED repos that
	// legitimately reach private/LAN hosts and use local file:// self-sources;
	// a consumer rendering UNTRUSTED input (e.g. konflate fork mode) sets this.
	// The egress half is process-level (see pkg/source/ssrfguard): go-git
	// installs its HTTPS transport process-wide, so a per-render git egress
	// policy is impossible.
	RestrictEgress bool

	// RegistryConfig is the docker config.json used for OCI auth.
	RegistryConfig string

	// CacheDir overrides the default on-disk cache root. The default
	// follows os.UserCacheDir (XDG_CACHE_HOME on Linux, Library/Caches
	// on macOS, %LocalAppData% on Windows) with a "flate" subdir, so
	// the cache survives reboots and OS tmpfs cleanups. Falls back to
	// os.TempDir()/flate-cache only when UserCacheDir errors.
	CacheDir string
	// SourceCache, when non-nil, is shared across orchestrators. The
	// `flate diff` flow constructs two orchestrators that point at the
	// same on-disk source-cache root; they MUST share one *Cache so the
	// internal mutex serializes concurrent slot allocation. When nil a
	// per-orchestrator cache is constructed (fine for single-orchestrator
	// commands like `build` / `get`).
	SourceCache *source.Cache
	// ExternalChanges, when non-nil, supplies the file-level diff so
	// the orchestrator skips its built-in change.Detect step. The
	// filter is still built from this set + the loaded SourceFiles
	// during Bootstrap.
	ExternalChanges *change.Set

	// Concurrency caps the number of active reconcile bodies running
	// in parallel. <= 0 means unbounded (every Kustomization / HR
	// reconciles on its own goroutine). Background watch loops are
	// unaffected. Sensible default for I/O-bound work is
	// runtime.NumCPU() * 4.
	Concurrency int

	// SourceRetry tunes the bounded, classified retry applied to every
	// source fetch (Git/OCI/Bucket) on transient network failures —
	// connection resets, refused connections, dial/IO timeouts. Permanent
	// errors (bad path, missing secret, not-found) fail fast. Attempts
	// <= 1 disables retry; the CLI defaults it to 3. See
	// source.RetryConfig / source.WithRetry.
	SourceRetry source.RetryConfig

	// GitDepth caps the shallow-clone history depth for GitRepository
	// sources (both the bare mirror and the legacy clone). 0 clones full
	// history; the CLI defaults it to 1 (opt-out via --git-depth=0).
	// Commit-pinned refs always full-clone regardless. See
	// git.Fetcher.Depth.
	GitDepth int

	// HelmTemplateCacheBytes caps the in-memory helm template-output
	// cache. Repeat HRs with identical effective inputs (chart
	// fingerprint, resolved values, render options) hit the cache and
	// skip action.Install.RunWithContext — the single largest CPU +
	// allocation consumer in the codebase. <= 0 disables the cache.
	// The CLI flag `--helm-template-cache-mb` exposes this in MB
	// units; embedders pass bytes directly.
	HelmTemplateCacheBytes int64

	// HelmRenderCacheBytes caps the persistent on-disk helm template-
	// output cache (Phase 3.4a). Cross-process reuse: repeat `flate
	// build` / `flate diff` invocations against the same checkout
	// short-circuit the helm render entirely when the chart
	// fingerprint + resolved values + opts tuple hits the disk layer.
	// <= 0 disables disk caching; the in-memory layer (sized by
	// HelmTemplateCacheBytes) continues to operate independently.
	// The CLI flag `--helm-render-cache-mb` exposes this in MB
	// units; embedders pass bytes directly.
	HelmRenderCacheBytes int64

	// KustomizeRenderCacheBytes caps the persistent on-disk kustomize
	// render cache. Cross-process reuse: a Kustomization whose recorded
	// disk inputs are unchanged since a prior run skips krusty entirely
	// (see pkg/kustomize/render_cache.go). <= 0 disables it. The CLI flag
	// `--kustomize-render-cache-mb` exposes this in MB; embedders pass bytes.
	KustomizeRenderCacheBytes int64
}

Config carries everything the orchestrator needs.

type FailuresError added in v0.4.3

type FailuresError struct{ Message string }

FailuresError aggregates per-resource reconcile failures into one error. Its identity (recovered with errors.As) — not its message text — is how the CLI tells the resource-failure aggregate apart from incidental run errors (a panic, a cancellation) when it re-scopes failures to a namespace filter and re-renders them.

func (*FailuresError) Error added in v0.4.3

func (e *FailuresError) Error() string

type Orchestrator

type Orchestrator struct {
	// contains filtered or unexported fields
}

Orchestrator wires controllers and drives reconciliation.

func New

func New(cfg Config) (*Orchestrator, error)

New constructs an Orchestrator. It allocates the Store and TaskService but does not yet start any reconciliation — call Bootstrap then Run.

func (*Orchestrator) Bootstrap

func (o *Orchestrator) Bootstrap(ctx context.Context) error

Bootstrap discovers manifests, applies namespace inheritance, primes existence-only sources Ready, and prepares the change filter. Delegates the load / expand / alias phase to pkg/discovery; the remainder is dependency validation + change-filter construction.

Idempotent: a second call returns nil without re-running discovery. Bootstrap mutates orchestrator state (sourceFiles, parentOf, existence, depGraph, componentCache, filter); replaying it would rebuild the change.Filter from scratch, dropping any OnAdd hook already wired into the store's listener set. Centralizing the guard here means Render, embedders, and test harnesses all get the invariant for free. A partial-failure path that returns before flipping bootstrapped=true leaves the orchestrator eligible for a clean retry on the next call.

func (*Orchestrator) Filter

func (o *Orchestrator) Filter() *change.Filter

Filter returns the change filter (may be nil-but-non-active).

func (*Orchestrator) Render

func (o *Orchestrator) Render(ctx context.Context) (*Result, error)

Render is the structured embed-friendly entry point: Bootstrap + Run + collect everything an external caller needs to consume the reconcile result. CLI / Run() ergonomics remain unchanged; callers who want a single function that returns a typed Result use this.

The returned Result is non-nil even when err is non-nil — failures during reconcile populate Result.Failed without aborting collection, so the caller sees both the partial output and the failure list. An error from Bootstrap (the load phase) is fatal and returns (nil, err); errors from Run yield (result, err).

On any returned error, render defers Stop so embedders that receive (nil, err) — Bootstrap failure paths in particular — don't leak the staging cache tempdir and helm client until process exit. Stop is sync.Once-guarded so the deferred call composes safely with Run's own deferred Stop and any explicit caller Stop.

Idempotent: a second call returns the cached Result/err pair from the first. The controllers' Configure hooks panic if invoked after Start (reconcile-shaping config is frozen once dispatch begins), so a re-Run would panic. Caching at this boundary lets embedders retry Render without restarting controllers — pair with Bootstrap's same guarantee guard above.

func (*Orchestrator) Run

func (o *Orchestrator) Run(ctx context.Context) error

Run starts every controller, blocks until the task service drains, then aggregates and returns any failures. The post-drain reporting + error-string assembly lives in finalize so Run reads as a clean start → drain → finalize sequence.

func (*Orchestrator) Stop

func (o *Orchestrator) Stop()

Stop shuts the controllers down in reverse-construction order. Safe to call multiple times: each controller's Close is idempotent (drains the unsub slice once and nils it). Wrapped in sync.Once so the bookkeeping reads cleanly even if a caller's defer runs after Run's defer.

The kustomize render cache is in-memory and holds no OS resources, so there is nothing to release here — it is freed with the Orchestrator.

func (*Orchestrator) Store

func (o *Orchestrator) Store() *store.Store

Store returns the underlying object store.

func (*Orchestrator) WithFetcher

func (o *Orchestrator) WithFetcher(kind string, f source.Fetcher) *Orchestrator

WithFetcher installs (or replaces) a per-kind source.Fetcher on the internal source controller. Call BEFORE Bootstrap. Returns the orchestrator for chaining. Use this to embed flate as a library with a custom fetcher (in-memory test fixtures, additional source kinds, alternate verification logic) without forking the New() construction.

Passing a nil fetcher unregisters the kind — useful for stripping a default registration in tests.

Panics if called after Bootstrap: by then discovery has already run and any source CR discovered between New() and Bootstrap was reconciled by whichever fetcher was registered at that moment. A later swap silently misses those reconciles and the embedder gets inconsistent behavior across source CRs of the same kind. Bootstrap is the natural commit point for the controller wiring.

type Rendered added in v0.3.2

type Rendered struct {
	*Orchestrator
	Result *Result
	Err    error
}

Rendered is one side of a RenderTrees comparison: the Orchestrator (already Stopped — its background reconcile loops are released, but its Store stays readable for diff doc gathering) paired with its render Result and that side's render error. Result is nil only on a fatal (Bootstrap) error; a per-resource render failure keeps Result non-nil (the failures are in Result.Failed) and is reported in Err.

func RenderTrees added in v0.3.2

func RenderTrees(ctx context.Context, baseTree, headTree Tree, cfg Config) (base, head Rendered, err error)

RenderTrees renders two cluster directories for comparison — the engine behind `flate diff`, exposed so an SDK consumer (a PR-diff bot, a CI gate) doesn't re-implement the two-orchestrator dance.

Each side reconciles in CHANGED-ONLY mode against the other (its PathOrig is set to the opposite tree's RepoRoot), so only resources whose source files differ between the two trees — plus their dependency closure — are rendered, not the whole cluster. Pairing the two scoped outputs (e.g. via diff.Changes) yields the same diff a full render would: an unchanged resource renders on neither side.

Both sides share one source.Cache so a chart / OCI layer / git source fetched for one side is reused by the other (and concurrent slot allocation is serialized), and they run concurrently. cfg supplies the render tuning; cfg.Path / RepoRoot / PathOrig / SelfURLs are set from baseTree/headTree (any values on cfg are ignored), and cfg.SourceCache, when nil, is created from cfg.CacheDir (falling back to the OS tempdir). A consumer that renders many comparisons (one per PR) should pass its own long-lived cfg.SourceCache so artifacts persist across calls.

Both orchestrators are Stopped before return; their Stores remain readable. err is non-nil if either side had any failure; callers that still want the partial diff check Rendered.Result != nil (nil only on a fatal Bootstrap error) and treat err as advisory. baseTree is the left/old side and headTree the right/new — matching diff.Changes(left, right). Each side reconciles changed-only against the OTHER tree's RepoRoot.

type Result

type Result struct {
	Manifests map[manifest.NamedResource][]map[string]any
	Failed    map[manifest.NamedResource]store.StatusInfo
	Orphans   map[manifest.NamedResource]string
	// DependsOn is the declared spec.dependsOn graph: each Kustomization or
	// HelmRelease that names dependencies maps to the sorted ids it depends on.
	// Same-kind only (KS→KS, HR→HR per the Flux spec); implicit coupling
	// (healthChecks, shared CRDs, service DNS) is not represented. Nodes with no
	// declared dependencies are omitted, and the map is nil (not empty) when
	// nothing declares a dependsOn. Consumers invert this into a dependents
	// adjacency list for impact / "blast radius" analysis.
	//
	// It is the declared graph verbatim: cycle members and self-edges (A → A) are
	// retained, not pruned (cycles also surface as render Failures) — a consumer
	// that walks it must tolerate them. It is also independent of the --skip-*
	// render filters, so an id here may have no entry in Manifests when its kind
	// was skipped.
	DependsOn map[manifest.NamedResource][]manifest.NamedResource
	// Blocked maps a failed resource to the immediate dependencies that blocked
	// it — present only for failures that are purely derived (the resource's own
	// reconcile never ran because a dependency failed or was missing), absent for
	// primary failures (a real render/template/build error). A consumer groups
	// derived failures under their root cause by walking these edges to a primary
	// failure or a missing id, instead of surfacing every cascaded failure.
	Blocked map[manifest.NamedResource][]manifest.NamedResource
	// Warnings are non-fatal render advisories — the render succeeded, but the
	// operator probably wants to look at these (e.g. a HelmRelease pinning
	// values the chart's schema no longer defines, an empty --path scan). Each
	// carries a stable Category code a consumer filters on rather than parsing
	// the message, an optional Resource (zero = render-global), and optional
	// structured Detail. Sorted deterministically; nil (not empty) when none.
	// Distinct from Failed (which blocks a resource) and from operational logs
	// (submodule skips, cache resets) which stay on stderr. A consumer like
	// konflate surfaces these alongside the diff.
	Warnings []manifest.Warning
}

Result is the structured output of Orchestrator.Render: the rendered manifests keyed by the originating Kustomization / HelmRelease, the set of resources that failed reconcile, and the orphans flate detected (sources sitting under a parent KS's spec.path but never emitted by that parent's render — real Flux would not reconcile them either).

Manifests is empty for an HR that had nothing to render or a KS whose render produced zero docs; Failed/Orphans are empty when everything reconciled cleanly.

type Tree added in v0.3.3

type Tree struct {
	RepoRoot string
	Path     string
	SelfURLs []string
}

Tree is one cluster directory to render for comparison. RepoRoot is the source root that Kustomization spec.path values resolve against (the GitRepository artifact root); Path is the scan entry point and defaults to RepoRoot when empty (a cluster whose Flux entry is a subdir passes the subdir as Path and the root as RepoRoot). SelfURLs are the remote URL(s) the tree represents, for self-referential GitRepository aliasing. A consumer rendering extracted trees with no .git supplies all three explicitly; nothing here is inferred from a .git ancestor.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL