reconcile

package
v0.0.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 13, 2026 License: MIT Imports: 20 Imported by: 0

Documentation

Overview

Package reconcile turns the executors section of a config file into registered, preflighted executors — once, in one place, for every entry point that hosts a control plane (Task 20170).

Why this package exists

Before it, three bootstraps disagreed. `cloop ui` and `cloop serve` registered only the localprocess driver from their own package-local bootstrap; the container and Kubernetes drivers were registered from cmd/root.go's PersistentPreRunE, keyed off os.Getwd() rather than the directory the server was actually given, and their failures were printed to stderr and dropped. A hub started from the Helm chart or the docker-compose stack with executors.allow_host_process: false could therefore come up with the host driver refused (correctly) and no isolating driver registered (silently) — zero usable executors, /readyz green, and every run failing at dispatch with a 409. A Kubernetes rollout would report success.

So this package does three things the old split could not:

  1. One reconciliation for all entry points, taking the directory explicitly instead of inferring it from the process's cwd;
  2. A per-driver preflight whose result is *recorded* — a structured Diagnostic with an id, kind, status and remediation string — rather than a warning on stderr that nothing can read back; and
  3. A readiness verdict (see ready.go) that the /readyz probes consult, so a hub with nothing to dispatch to fails its readiness gate loudly instead of accepting traffic it cannot serve.

Why it is not in pkg/executor

The natural home would be executor.ReconcileFromConfig. It cannot live there: pkg/config imports pkg/executor/container and pkg/executor/kubernetes for their Options types, and both import pkg/executor. A pkg/executor that imported pkg/config would close that loop. Keeping reconciliation one level down leaves pkg/executor free of driver and config dependencies, which is what lets any package import it.

Degraded is not failed

A driver whose preflight finds a fatal problem stays registered and is marked degraded. Preflight is a point-in-time probe of a remote system: a cluster that is briefly unreachable while the hub boots would otherwise lose its executor until someone restarted the process, which is a worse outcome than a hub that reports the problem and lets the next dispatch retry. Only a driver that could not be *constructed* — no container runtime on PATH, no credential source for the cluster — is failed, because there is nothing to register.

Index

Constants

View Source
const DefaultPreflightTimeout = 45 * time.Second

DefaultPreflightTimeout bounds one driver's preflight. Generous enough for a cold container runtime and a cross-region API server, short enough that a hub with an unreachable cluster still finishes booting promptly.

Variables

This section is empty.

Functions

func PublishForTest

func PublishForTest(r Report)

PublishForTest installs a report without running a reconciliation, for tests in other packages that need a specific diagnostic to be visible.

It exists because the interesting case to render — a driver that failed to build — otherwise requires a broken container runtime on PATH. What those tests check is the plumbing from report to HTTP response; that a missing runtime produces this diagnostic is pinned down in this package's own tests. Only tests should call it.

func Ready

func Ready() error

Ready reports whether this control plane can dispatch a workload, against the default registry.

It is computed live rather than read off the last report on purpose. A hub may legitimately start with nothing isolating registered and become usable when a remote agent enrolls minutes later; a readiness verdict frozen at startup would keep that hub out of service until someone restarted it. The last report is consulted only to explain a failure, never to decide one.

func ReadyIn

func ReadyIn(reg *executor.Registry) error

ReadyIn is Ready against a specific registry, for tests and embedders that keep their own.

func ResetForTest

func ResetForTest()

ResetForTest clears the published report, for tests in other packages.

Exported because the report is process-global: pkg/ui asserts that a failed driver reaches GET /api/executors, and without a way to clear the report afterwards that fixture would leak into every later test's readiness verdict. Only tests should call it.

func Sweep

func Sweep(ctx context.Context, dir string, opts Options)

Sweep reconciles the durable residue of a control plane that died mid-run.

It is safe to call on every start and does nothing a second time: closed session rows stay closed, reaped containers stay reaped, and a pruned worktree is gone. It never returns an error — a hub must come up whether or not it could tidy up — and reports what it did through opts.Logf.

Types

type CredentialsFunc

type CredentialsFunc func(dir string, cfg *config.Config, execID string) (Identity, func(), error)

CredentialsFunc builds the identity the Kubernetes driver authenticates with, returning a cleanup to release whatever backs it.

The cleanup exists because the broker source owns an open state database that the executor keeps for the process's lifetime. On the success path it is deliberately never called; on the path where the source was built but registration then failed, it is the only way to avoid leaking a SQLite handle and its WAL lock on every reconciliation pass. It may be nil.

type Diagnostic

type Diagnostic struct {
	// ID is the executor ID, which is also the registry key.
	ID string `json:"id"`
	// Kind is the executor.Kind* the driver reports.
	Kind string `json:"kind"`
	// Status is the reconciliation outcome.
	Status Status `json:"status"`
	// Registered reports whether the executor ended up in the registry.
	// Degraded executors are registered; failed ones are not.
	Registered bool `json:"registered"`
	// Isolating reports whether this driver puts a boundary between the
	// workload and the control-plane host. It is what strict mode counts.
	Isolating bool `json:"isolating"`
	// Message is the one-line summary an operator reads first.
	Message string `json:"message"`
	// Remediation is the concrete next action. Empty only when Status is OK.
	Remediation string `json:"remediation,omitempty"`
	// Findings is the preflight checklist, when one was run.
	Findings []Finding `json:"findings,omitempty"`
	// CheckedAt is when this diagnostic was produced.
	CheckedAt time.Time `json:"checked_at"`
}

Diagnostic is the per-driver record this package exists to produce: what was configured, whether it came up, and what to do when it did not.

func (Diagnostic) OK

func (d Diagnostic) OK() bool

OK reports whether this driver is usable.

type Finding

type Finding struct {
	Name string `json:"name"`
	// Level is "ok", "warn" or "fail", matching both drivers' vocabulary.
	Level   string `json:"level"`
	Message string `json:"message"`
	Fix     string `json:"fix,omitempty"`
}

Finding is one preflight observation, normalised across drivers. The container and Kubernetes drivers each define their own identically-shaped Finding; flattening them here means the API and the panel render one type.

type Identity

type Identity struct {
	// Cluster authenticates to the Kubernetes API server. Required.
	Cluster kubernetes.CredentialSource
	// Workspace leases the credential a private-repository fetch needs. nil is
	// legitimate and means "public repositories only": an in-cluster hub with
	// no sealing key can still schedule Pods, and the driver's preflight says
	// so rather than failing a deployment that never needed it.
	Workspace executor.WorkspaceCredentialSource
}

Identity is everything the Kubernetes driver authenticates with: one credential for the cluster it schedules into, and one for the repositories it fetches project source from.

They travel together, and are built together, because both are backed by the same secret broker and therefore the same open state database. Building them through two independent hooks would open two SQLite handles for one executor and hold both for the process's lifetime — see the comment on ensureKubernetes for why that lifetime makes a second handle a leak rather than an inefficiency.

func BrokerCredentials

func BrokerCredentials(dir string, cfg *config.Config, execID string) (Identity, func(), error)

BrokerCredentials is the default Kubernetes identity source: the hub Pod's own ServiceAccount when in_cluster is set, otherwise a kubeconfig leased from the secret broker in dir's state database.

There is no fallback between the two. An executor that quietly authenticated as something other than what config named would make the audit trail lie about who did what in the cluster.

On the success path the returned cleanup is deliberately NOT called by the caller that registers the executor: the driver holds the broker for as long as the process runs and needs it on every Start. Process exit closes it, which for both a CLI invocation and a long-running hub is exactly the intended lifetime. The cleanup exists for the one path where the source was built and registration then failed.

type NotReadyError

type NotReadyError struct {
	// Reason is the one-line cause.
	Reason string
	// Remediation is the concrete next action.
	Remediation string
	// Diagnostics are the failed or degraded drivers from the last
	// reconciliation, when there was one. Empty means nothing was even
	// configured — a materially different and more urgent situation than a
	// driver that tried and failed.
	Diagnostics []Diagnostic
}

NotReadyError reports that no workload can be dispatched, and why.

It carries the remediation as a separate field rather than only inside the message so a probe body, a startup log line and the Executors panel can render the cause and the fix in whatever shape each needs.

func (*NotReadyError) Error

func (e *NotReadyError) Error() string

Error implements error.

type Options

type Options struct {
	// Registry receives the executors. nil means executor.DefaultRegistry.
	Registry *executor.Registry

	// SkipPreflight registers without probing. Preflight costs container
	// runtime calls and Kubernetes API round-trips, which is the right price
	// for a long-running hub and the wrong one for a short CLI command that
	// will never dispatch a workload.
	SkipPreflight bool

	// PreflightTimeout bounds each driver's probe. Zero uses
	// DefaultPreflightTimeout. A cluster that cannot answer in this long is
	// one the operator needs told about, not one worth waiting for.
	PreflightTimeout time.Duration

	// ReconcileOrphans asks for the full restart sweep: orphaned Pods and
	// orphaned containers from the drivers, stale `running` session rows, and
	// leaked task worktrees. It costs API calls and runtime round-trips, so
	// only the entry points a control plane actually restarts as should ask
	// for it.
	//
	// It was Kubernetes-only until Task 20191, which is why the name says
	// "orphans" rather than "restart". The three additions all answer the
	// same question — "what did the process I am replacing leave behind?" —
	// and splitting them across four flags would only create combinations
	// with no operator meaning. See startup.go.
	ReconcileOrphans bool

	// HandleStore persists driver handle identity so a control plane that
	// restarts reattaches to workloads that are still running instead of
	// losing them (Task 20191).
	//
	// Nil means "open dir's state database", which is what a real deployment
	// wants and what makes ReconcileOrphans able to tell a workload this hub
	// dispatched from an orphan. Set DisableHandleStore to opt out entirely.
	HandleStore executor.HandleStore

	// DisableHandleStore suppresses handle persistence. Tests reconciling
	// into a private registry set it so a pass over a temp directory does not
	// create a state database as a side effect.
	//
	// It is a separate bool rather than a nil HandleStore because nil already
	// means "use the default", the same shape Publish needed and for the same
	// reason: the zero value has to be the production behaviour.
	DisableHandleStore bool

	// WorktreeMinAge bounds the leaked-worktree sweep. Zero uses
	// worktree.DefaultMinAge. It is the single most safety-critical knob in
	// the sweep — see pruneWorktrees.
	WorktreeMinAge time.Duration

	// KubernetesCredentials overrides how the Kubernetes driver's identity is
	// obtained. nil uses BrokerCredentials, which reads the secret broker out
	// of dir's state database. Tests and embedders that already hold a
	// credential source inject it here.
	KubernetesCredentials CredentialsFunc

	// WrapWorkspaceSource decorates the credential source a driver is given,
	// per executor. nil means no decoration.
	//
	// It exists so the hub can interpose the git interception proxy
	// (pkg/executor/gitproxycreds) without this package learning about it:
	// reconciliation decides *which* executors exist and what authenticates
	// them, while whether a sandbox reaches the forge directly or through a
	// proxy is a property of the deployment. A nil source stays nil — there is
	// nothing to route when no credential is leased in the first place.
	WrapWorkspaceSource func(execID string, src executor.WorkspaceCredentialSource) executor.WorkspaceCredentialSource

	// Logf receives one line per diagnostic. nil logs to stderr; supply a
	// no-op to silence a pass entirely.
	Logf func(format string, args ...any)

	// QuietWhenHealthy logs only problems. A CLI invocation wants this — a
	// line reading "executor container: ready" on every `cloop status` is
	// noise, and a caller that prefixes its output with "warning:" would
	// otherwise emit "warning: ... ready". A long-running server wants the
	// healthy lines, because "which executors did this hub come up with" is
	// the first question asked of a startup log.
	QuietWhenHealthy bool

	// Publish records the report where LastReport and Ready can see it.
	// Defaults to true; tests reconciling into a private registry set it
	// false so they do not disturb process-global state.
	//
	// It is a pointer so the zero value can mean "yes" while still letting a
	// caller say "no" explicitly.
	Publish *bool
}

Options tunes a reconciliation pass. The zero value is what the hub entry points want: the default registry, preflight on, no orphan sweep.

type Report

type Report struct {
	// Dir is the directory whose config was reconciled.
	Dir string `json:"dir"`
	// Diagnostics is one entry per configured driver, ordered by ID.
	Diagnostics []Diagnostic `json:"diagnostics"`
	// StrictMode is the host-execution policy in force when the pass ran.
	StrictMode bool `json:"strict_mode"`
	// ReconciledAt is when the pass completed.
	ReconciledAt time.Time `json:"reconciled_at"`
}

Report is the result of one reconciliation pass.

func Bootstrap

func Bootstrap(dir string, cfg *config.Config, opts Options) Report

Bootstrap is what a long-running control plane calls: it registers synchronously and preflights in the background, returning the registration pass's report.

The split exists because the two halves have opposite latency budgets. Registration is cheap — a PATH lookup for the container runtime, a struct for the cluster — and must complete before the server accepts its first request, or /readyz would report a hub with no executors as ready purely because bootstrap had not got there yet. Preflight is a container runtime round-trip and a Kubernetes API call, up to PreflightTimeout each; paying that before the listener opens would make `cloop ui` look hung on a host whose docker daemon is wedged.

Running both is safe precisely because FromConfig is idempotent: the second pass finds every executor already registered, reuses it, and only adds the probe results. The published report is upgraded in place, so /readyz and GET /api/executors go from "registered, not yet probed" to the full checklist without either endpoint knowing a second pass happened.

func FromConfig

func FromConfig(ctx context.Context, dir string, cfg *config.Config, opts Options) Report

FromConfig registers every executor the config enables, preflights each one, and records a diagnostic for it.

It is idempotent. A driver already present in the registry is reused rather than rebuilt — which matters beyond tidiness for the Kubernetes driver, whose construction opens a state database that is intentionally never closed. Rebuilding it on every pass would leak a handle per call.

It never returns an error and never aborts on one driver's account: a hub with a broken container runtime and a working cluster must come up with the cluster. What the failure buys the operator is the diagnostic, and — when strict mode leaves nothing to dispatch to — a red /readyz.

func LastReport

func LastReport() (Report, bool)

LastReport returns a deep copy of the most recent published reconciliation, and whether one has happened. The copy is deliberate: callers marshal it into HTTP responses from other goroutines while a later pass may be running, and some hand the Findings slice on to a view struct.

func (Report) IsolatingRegistered

func (r Report) IsolatingRegistered() bool

IsolatingRegistered reports whether at least one isolating executor came up during this pass. It is deliberately *not* what readiness is computed from — see Ready — because a remote agent can enroll after startup.

func (Report) Problems

func (r Report) Problems() []Diagnostic

Problems returns the diagnostics that are not OK, for logging and for the readiness message.

type Status

type Status string

Status is the outcome of reconciling one configured driver.

const (
	// StatusOK: registered, and preflight found nothing fatal.
	StatusOK Status = "ok"
	// StatusDegraded: registered, but preflight found a fatal problem.
	// Workloads placed here will probably fail until it is fixed.
	StatusDegraded Status = "degraded"
	// StatusFailed: not registered. The driver could not be built at all.
	StatusFailed Status = "failed"
)

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL