reconcile

package
v0.0.6 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 16, 2026 License: MIT Imports: 23 Imported by: 0

Documentation

Overview

Package reconcile turns the executors section of a config file into registered, preflighted executors — once, in one place, for every entry point that hosts a control plane (Task 20170).

Why this package exists

Before it, three bootstraps disagreed. `cloop ui` and `cloop serve` registered only the localprocess driver from their own package-local bootstrap; the container and Kubernetes drivers were registered from cmd/root.go's PersistentPreRunE, keyed off os.Getwd() rather than the directory the server was actually given, and their failures were printed to stderr and dropped. A hub started from the Helm chart or the docker-compose stack with executors.allow_host_process: false could therefore come up with the host driver refused (correctly) and no isolating driver registered (silently) — zero usable executors, /readyz green, and every run failing at dispatch with a 409. A Kubernetes rollout would report success.

So this package does three things the old split could not:

  1. One reconciliation for all entry points, taking the directory explicitly instead of inferring it from the process's cwd;
  2. A per-driver preflight whose result is *recorded* — a structured Diagnostic with an id, kind, status and remediation string — rather than a warning on stderr that nothing can read back; and
  3. A readiness verdict (see ready.go) that the /readyz probes consult, so a hub with nothing to dispatch to fails its readiness gate loudly instead of accepting traffic it cannot serve.

Why it is not in pkg/executor

The natural home would be executor.ReconcileFromConfig. It cannot live there: pkg/config imports pkg/executor/container and pkg/executor/kubernetes for their Options types, and both import pkg/executor. A pkg/executor that imported pkg/config would close that loop. Keeping reconciliation one level down leaves pkg/executor free of driver and config dependencies, which is what lets any package import it.

Degraded is not failed

A driver whose preflight finds a fatal problem stays registered and is marked degraded. Preflight is a point-in-time probe of a remote system: a cluster that is briefly unreachable while the hub boots would otherwise lose its executor until someone restarted the process, which is a worse outcome than a hub that reports the problem and lets the next dispatch retry. Only a driver that could not be *constructed* — no container runtime on PATH, no credential source for the cluster — is failed, because there is nothing to register.

Index

Constants

View Source
const DefaultPreflightTimeout = 45 * time.Second

DefaultPreflightTimeout bounds one driver's preflight. Generous enough for a cold container runtime and a cross-region API server, short enough that a hub with an unreachable cluster still finishes booting promptly.

View Source
const DefaultSweepInterval = 15 * time.Minute

DefaultSweepInterval is how often the orphan sweep repeats when a deployment does not choose.

Fifteen minutes is chosen against the cost of being wrong in each direction. Too long and an evicted Pod burns a node's CPU and a ResourceQuota slot for that much longer; too short and every hub adds steady list traffic to its cluster's API server for a condition that is rare. A sweep is two list calls per executor, so four an hour is negligible, and fifteen minutes bounds the lifetime of a leaked workload at something an operator would describe as "shortly" rather than "eventually".

View Source
const MinSweepInterval = time.Minute

MinSweepInterval floors a configured interval.

A misconfigured `orphan_sweep_interval_minutes: 0.1` equivalent — or a unit confusion that produced seconds — would turn a background tidy-up into a hot loop against the cluster's API server, which is a much more damaging failure than never sweeping at all. Zero still means disabled; this floors only values that asked for something positive and unreasonable.

Variables

This section is empty.

Functions

func ForgetNetworkPolicyVerdict added in v0.0.2

func ForgetNetworkPolicyVerdict(db *statedb.DB, executorID string) error

ForgetNetworkPolicyVerdict removes a recorded verdict.

It exists so that an operator who has changed CNI can invalidate the record without waiting for it to expire, and so that a test can return the hub to the unverified state without knowing how the record is stored.

func LoadNetworkPolicyVerdict added in v0.0.2

func LoadNetworkPolicyVerdict(db *statedb.DB, executorID string) (kubernetes.ProbeVerdict, error)

LoadNetworkPolicyVerdict reads the recorded verdict for one executor.

A missing record is not an error — it is the state every deployment starts in, and it resolves to "unverified", which fails closed. A record that will not parse is also not an error: it is treated as missing, for the same reason. The alternative is a hub that refuses to start because a field it uses to *deny* a capability is corrupt, and denying is what a zero value already does.

func LoadNetworkPolicyVerdictFromDir added in v0.0.2

func LoadNetworkPolicyVerdictFromDir(dir, executorID string) (kubernetes.ProbeVerdict, error)

LoadNetworkPolicyVerdictFromDir opens dir's control-plane database, reads one executor's verdict and closes it again.

Its own short-lived handle rather than one borrowed from the credential source, because the two have different lifetimes and different owners: the source's database is held for the process's life by the driver, and an in-cluster hub has no broker at all — reading through it would silently lose the verdict on exactly the deployment most likely to have one.

A database that cannot be opened yields the zero verdict and no error. The field's only effect is to grant a capability, so an unreadable record costs a refusal that names the probe command; failing startup over it would turn a narrow degradation into an outage.

func NetworkPolicyVerdictKey added in v0.0.2

func NetworkPolicyVerdictKey(executorID string) string

NetworkPolicyVerdictKey is the hub-meta key holding one executor's verdict.

func PublishForTest

func PublishForTest(r Report)

PublishForTest installs a report without running a reconciliation, for tests in other packages that need a specific diagnostic to be visible.

It exists because the interesting case to render — a driver that failed to build — otherwise requires a broken container runtime on PATH. What those tests check is the plumbing from report to HTTP response; that a missing runtime produces this diagnostic is pinned down in this package's own tests. Only tests should call it.

func Ready

func Ready() error

Ready reports whether this control plane can dispatch a workload, against the default registry.

It is computed live rather than read off the last report on purpose. A hub may legitimately start with nothing isolating registered and become usable when a remote agent enrolls minutes later; a readiness verdict frozen at startup would keep that hub out of service until someone restarted it. The last report is consulted only to explain a failure, never to decide one.

func ReadyIn

func ReadyIn(reg *executor.Registry) error

ReadyIn is Ready against a specific registry, for tests and embedders that keep their own.

func ResetForTest

func ResetForTest()

ResetForTest clears the published report, for tests in other packages.

Exported because the report is process-global: pkg/ui asserts that a failed driver reaches GET /api/executors, and without a way to clear the report afterwards that fixture would leak into every later test's readiness verdict. Only tests should call it.

func SaveNetworkPolicyVerdict added in v0.0.2

func SaveNetworkPolicyVerdict(db *statedb.DB, executorID string, v kubernetes.ProbeVerdict) error

SaveNetworkPolicyVerdict records a probe result for one executor.

It refuses a verdict with no timestamp. ProbeVerdict.Recorded() is what tells every reader "a probe actually ran", so writing a zero-valued one would store a record that reads as evidence and is not — and, because an unrecorded verdict resolves to unverified, it would also be indistinguishable from having written nothing while looking like a successful save.

func StartPeriodicSweep added in v0.0.2

func StartPeriodicSweep(ctx context.Context, opts Options)

StartPeriodicSweep runs SweepOrphansOnce every interval until ctx is done.

It replaces any sweeper already running in this process: a hub reconciles more than once — Bootstrap does two passes by design, and tests construct many servers — and two tickers would double the API traffic while racing to delete the same objects. Calling it with a disabled interval stops the current sweeper and starts nothing, so toggling the setting off takes effect without a restart of anything but the reconciliation.

The first pass is deliberately one interval away, not immediate: the startup sweep has just run or is running, and repeating it in the same second would only race it.

func StopPeriodicSweep added in v0.0.2

func StopPeriodicSweep()

StopPeriodicSweep halts the process's sweeper. Safe to call when none runs, and safe to call twice — a server shutting down does both.

func Sweep

func Sweep(ctx context.Context, dir string, opts Options)

Sweep reconciles the durable residue of a control plane that died mid-run.

It is safe to call on every start and does nothing a second time: closed session rows stay closed, reaped containers stay reaped, and a pruned worktree is gone. It never returns an error — a hub must come up whether or not it could tidy up — and reports what it did through opts.Logf.

Types

type CredentialsFunc

type CredentialsFunc func(dir string, cfg *config.Config, execID string) (Identity, func(), error)

CredentialsFunc builds the identity the Kubernetes driver authenticates with, returning a cleanup to release whatever backs it.

The cleanup exists because the broker source owns an open state database that the executor keeps for the process's lifetime. On the success path it is deliberately never called; on the path where the source was built but registration then failed, it is the only way to avoid leaking a SQLite handle and its WAL lock on every reconciliation pass. It may be nil.

type Diagnostic

type Diagnostic struct {
	// ID is the executor ID, which is also the registry key.
	ID string `json:"id"`
	// Kind is the executor.Kind* the driver reports.
	Kind string `json:"kind"`
	// Status is the reconciliation outcome.
	Status Status `json:"status"`
	// Registered reports whether the executor ended up in the registry.
	// Degraded executors are registered; failed ones are not.
	Registered bool `json:"registered"`
	// Isolating reports whether this driver puts a boundary between the
	// workload and the control-plane host. It is what strict mode counts.
	Isolating bool `json:"isolating"`
	// Message is the one-line summary an operator reads first.
	Message string `json:"message"`
	// Remediation is the concrete next action. Empty only when Status is OK.
	Remediation string `json:"remediation,omitempty"`
	// Findings is the preflight checklist, when one was run.
	Findings []Finding `json:"findings,omitempty"`
	// CheckedAt is when this diagnostic was produced.
	CheckedAt time.Time `json:"checked_at"`
}

Diagnostic is the per-driver record this package exists to produce: what was configured, whether it came up, and what to do when it did not.

func (Diagnostic) OK

func (d Diagnostic) OK() bool

OK reports whether this driver is usable.

type ExecutorSweep added in v0.0.2

type ExecutorSweep struct {
	ID   string `json:"id"`
	Kind string `json:"kind"`
	// Removed counts the objects collected. For Kubernetes that is Pods plus
	// the NetworkPolicies that governed them; for the container driver it is
	// sandbox containers.
	Removed int `json:"removed"`
	// Objects names what went, newest-format "namespace/name", capped so a
	// pathological sweep cannot put an unbounded slice into an HTTP response.
	Objects []string `json:"objects,omitempty"`
	// Error is the driver's failure, if the pass could not complete. A sweep
	// that failed is reported rather than hidden: an operator whose RBAC lost
	// the list verb should find out from the fleet panel, not from a namespace
	// slowly filling with Pods.
	Error string `json:"error,omitempty"`
}

ExecutorSweep is one driver's result from one pass.

type Finding

type Finding struct {
	Name string `json:"name"`
	// Level is "ok", "warn" or "fail", matching both drivers' vocabulary.
	Level   string `json:"level"`
	Message string `json:"message"`
	Fix     string `json:"fix,omitempty"`
}

Finding is one preflight observation, normalised across drivers. The container and Kubernetes drivers each define their own identically-shaped Finding; flattening them here means the API and the panel render one type.

type Identity

type Identity struct {
	// Cluster authenticates to the Kubernetes API server. Required.
	Cluster kubernetes.CredentialSource
	// Workspace leases the credential a private-repository fetch needs. nil is
	// legitimate and means "public repositories only": an in-cluster hub with
	// no sealing key can still schedule Pods, and the driver's preflight says
	// so rather than failing a deployment that never needed it.
	Workspace executor.WorkspaceCredentialSource
}

Identity is everything the Kubernetes driver authenticates with: one credential for the cluster it schedules into, and one for the repositories it fetches project source from.

They travel together, and are built together, because both are backed by the same secret broker and therefore the same open state database. Building them through two independent hooks would open two SQLite handles for one executor and hold both for the process's lifetime — see the comment on ensureKubernetes for why that lifetime makes a second handle a leak rather than an inefficiency.

func BrokerCredentials

func BrokerCredentials(dir string, cfg *config.Config, execID string) (Identity, func(), error)

BrokerCredentials is the default Kubernetes identity source: the hub Pod's own ServiceAccount when in_cluster is set, otherwise a kubeconfig leased from the secret broker in dir's state database.

There is no fallback between the two. An executor that quietly authenticated as something other than what config named would make the audit trail lie about who did what in the cluster.

On the success path the returned cleanup is deliberately NOT called by the caller that registers the executor: the driver holds the broker for as long as the process runs and needs it on every Start. Process exit closes it, which for both a CLI invocation and a long-running hub is exactly the intended lifetime. The cleanup exists for the one path where the source was built and registration then failed.

type NotReadyError

type NotReadyError struct {
	// Reason is the one-line cause.
	Reason string
	// Remediation is the concrete next action.
	Remediation string
	// Diagnostics are the failed or degraded drivers from the last
	// reconciliation, when there was one. Empty means nothing was even
	// configured — a materially different and more urgent situation than a
	// driver that tried and failed.
	Diagnostics []Diagnostic
}

NotReadyError reports that no workload can be dispatched, and why.

It carries the remediation as a separate field rather than only inside the message so a probe body, a startup log line and the Executors panel can render the cause and the fix in whatever shape each needs.

func (*NotReadyError) Error

func (e *NotReadyError) Error() string

Error implements error.

type Options

type Options struct {
	// Registry receives the executors. nil means executor.DefaultRegistry.
	Registry *executor.Registry

	// SkipPreflight registers without probing. Preflight costs container
	// runtime calls and Kubernetes API round-trips, which is the right price
	// for a long-running hub and the wrong one for a short CLI command that
	// will never dispatch a workload.
	SkipPreflight bool

	// PreflightTimeout bounds each driver's probe. Zero uses
	// DefaultPreflightTimeout. A cluster that cannot answer in this long is
	// one the operator needs told about, not one worth waiting for.
	PreflightTimeout time.Duration

	// ReconcileOrphans asks for the full restart sweep: orphaned Pods and
	// orphaned containers from the drivers, stale `running` session rows, and
	// leaked task worktrees. It costs API calls and runtime round-trips, so
	// only the entry points a control plane actually restarts as should ask
	// for it.
	//
	// It was Kubernetes-only until Task 20191, which is why the name says
	// "orphans" rather than "restart". The three additions all answer the
	// same question — "what did the process I am replacing leave behind?" —
	// and splitting them across four flags would only create combinations
	// with no operator meaning. See startup.go.
	ReconcileOrphans bool

	// HandleStore persists driver handle identity so a control plane that
	// restarts reattaches to workloads that are still running instead of
	// losing them (Task 20191).
	//
	// Nil means "open dir's state database", which is what a real deployment
	// wants and what makes ReconcileOrphans able to tell a workload this hub
	// dispatched from an orphan. Set DisableHandleStore to opt out entirely.
	HandleStore executor.HandleStore

	// DisableHandleStore suppresses handle persistence. Tests reconciling
	// into a private registry set it so a pass over a temp directory does not
	// create a state database as a side effect.
	//
	// It is a separate bool rather than a nil HandleStore because nil already
	// means "use the default", the same shape Publish needed and for the same
	// reason: the zero value has to be the production behaviour.
	DisableHandleStore bool

	// WorktreeMinAge bounds the leaked-worktree sweep. Zero uses
	// worktree.DefaultMinAge. It is the single most safety-critical knob in
	// the sweep — see pruneWorktrees.
	WorktreeMinAge time.Duration

	// SweepInterval is how often the orphan sweep repeats after startup. Zero
	// uses DefaultSweepInterval; negative disables it entirely.
	//
	// Negative-means-off rather than zero-means-off because zero is what an
	// unset field is, and the default has to be the behaviour a deployment
	// that never heard of this option should get. An operator turning it off
	// says so explicitly; see ExecutorsConfig.OrphanSweepIntervalMinutes,
	// where the operator-facing 0 is translated to a negative here.
	//
	// Only honoured alongside ReconcileOrphans: the sweep is the same work,
	// and an entry point that does not want it at startup does not want it
	// every fifteen minutes either. See periodic.go.
	SweepInterval time.Duration

	// KubernetesCredentials overrides how the Kubernetes driver's identity is
	// obtained. nil uses BrokerCredentials, which reads the secret broker out
	// of dir's state database. Tests and embedders that already hold a
	// credential source inject it here.
	KubernetesCredentials CredentialsFunc

	// WrapWorkspaceSource decorates the credential source a driver is given,
	// per executor. nil means no decoration.
	//
	// It exists so the hub can interpose the git interception proxy
	// (pkg/executor/gitproxycreds) without this package learning about it:
	// reconciliation decides *which* executors exist and what authenticates
	// them, while whether a sandbox reaches the forge directly or through a
	// proxy is a property of the deployment. A nil source stays nil — there is
	// nothing to route when no credential is leased in the first place.
	WrapWorkspaceSource func(execID string, src executor.WorkspaceCredentialSource) executor.WorkspaceCredentialSource

	// Logf receives one line per diagnostic. nil logs to stderr; supply a
	// no-op to silence a pass entirely.
	Logf func(format string, args ...any)

	// QuietWhenHealthy logs only problems. A CLI invocation wants this — a
	// line reading "executor container: ready" on every `cloop status` is
	// noise, and a caller that prefixes its output with "warning:" would
	// otherwise emit "warning: ... ready". A long-running server wants the
	// healthy lines, because "which executors did this hub come up with" is
	// the first question asked of a startup log.
	QuietWhenHealthy bool

	// Publish records the report where LastReport and Ready can see it.
	// Defaults to true; tests reconciling into a private registry set it
	// false so they do not disturb process-global state.
	//
	// It is a pointer so the zero value can mean "yes" while still letting a
	// caller say "no" explicitly.
	Publish *bool

	// ControlPlaneDir is the hub's own directory, handed to the drivers that
	// bind or fork in a project's directory so each refuses one that contains
	// it (Task 20418, projectconfine.Backstop). Empty — a CLI command
	// reconciling for itself — checks nothing there; the hub always sets it.
	ControlPlaneDir string

	// SweepGate, when set, is asked before the startup sweep (periodic false)
	// and before each periodic orphan sweep (periodic true); a pass it refuses
	// is skipped. Nil runs every pass, which is right for a control plane
	// served by one process.
	//
	// For one served by several (Task 20354): the startup sweep closes
	// session rows whose workload this process cannot see and prunes the
	// worktrees of tasks not among them — both true of every live peer's work
	// — so it may only run while no peer is serving. The periodic orphan sweep
	// spares peers' workloads by itself (see executor.TrackedElsewhere) and is
	// gated to one process only so that it is not done N times.
	SweepGate func(periodic bool) bool
}

Options tunes a reconciliation pass. The zero value is what the hub entry points want: the default registry, preflight on, no orphan sweep.

type Report

type Report struct {
	// Dir is the directory whose config was reconciled.
	Dir string `json:"dir"`
	// Diagnostics is one entry per configured driver, ordered by ID.
	Diagnostics []Diagnostic `json:"diagnostics"`
	// StrictMode is the host-execution policy in force when the pass ran.
	StrictMode bool `json:"strict_mode"`
	// ReconciledAt is when the pass completed.
	ReconciledAt time.Time `json:"reconciled_at"`
}

Report is the result of one reconciliation pass.

func Bootstrap

func Bootstrap(dir string, cfg *config.Config, opts Options) Report

Bootstrap is what a long-running control plane calls: it registers synchronously and preflights in the background, returning the registration pass's report.

The split exists because the two halves have opposite latency budgets. Registration is cheap — a PATH lookup for the container runtime, a struct for the cluster — and must complete before the server accepts its first request, or /readyz would report a hub with no executors as ready purely because bootstrap had not got there yet. Preflight is a container runtime round-trip and a Kubernetes API call, up to PreflightTimeout each; paying that before the listener opens would make `cloop ui` look hung on a host whose docker daemon is wedged.

Running both is safe precisely because FromConfig is idempotent: the second pass finds every executor already registered, reuses it, and only adds the probe results. The published report is upgraded in place, so /readyz and GET /api/executors go from "registered, not yet probed" to the full checklist without either endpoint knowing a second pass happened.

func FromConfig

func FromConfig(ctx context.Context, dir string, cfg *config.Config, opts Options) Report

FromConfig registers every executor the config enables, preflights each one, and records a diagnostic for it.

It is idempotent. A driver already present in the registry is reused rather than rebuilt — which matters beyond tidiness for the Kubernetes driver, whose construction opens a state database that is intentionally never closed. Rebuilding it on every pass would leak a handle per call.

It never returns an error and never aborts on one driver's account: a hub with a broken container runtime and a working cluster must come up with the cluster. What the failure buys the operator is the diagnostic, and — when strict mode leaves nothing to dispatch to — a red /readyz.

func LastReport

func LastReport() (Report, bool)

LastReport returns a deep copy of the most recent published reconciliation, and whether one has happened. The copy is deliberate: callers marshal it into HTTP responses from other goroutines while a later pass may be running, and some hand the Findings slice on to a view struct.

func (Report) IsolatingRegistered

func (r Report) IsolatingRegistered() bool

IsolatingRegistered reports whether at least one isolating executor came up during this pass. It is deliberately *not* what readiness is computed from — see Ready — because a remote agent can enroll after startup.

func (Report) Problems

func (r Report) Problems() []Diagnostic

Problems returns the diagnostics that are not OK, for logging and for the readiness message.

type Status

type Status string

Status is the outcome of reconciling one configured driver.

const (
	// StatusOK: registered, and preflight found nothing fatal.
	StatusOK Status = "ok"
	// StatusDegraded: registered, but preflight found a fatal problem.
	// Workloads placed here will probably fail until it is fixed.
	StatusDegraded Status = "degraded"
	// StatusFailed: not registered. The driver could not be built at all.
	StatusFailed Status = "failed"
)

type SweepReport added in v0.0.2

type SweepReport struct {
	At        time.Time       `json:"at"`
	Removed   int             `json:"removed"`
	Executors []ExecutorSweep `json:"executors,omitempty"`
}

SweepReport is the result of one periodic pass across every driver.

func LastSweep added in v0.0.2

func LastSweep() (SweepReport, bool)

LastSweep returns the most recent periodic sweep and whether one has run.

The copy is deliberate, for the reason LastReport's is: callers marshal it into HTTP responses from other goroutines while a later pass may be running.

func SweepOrphansOnce added in v0.0.2

func SweepOrphansOnce(ctx context.Context, opts Options) SweepReport

SweepOrphansOnce runs one pass across every driver in the registry that can collect its own orphans, records it, and returns it.

Exported because it is also the honest implementation of "sweep now" for an operator who does not want to wait out the interval, and because a test that cannot trigger a pass deterministically would have to sleep.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL