Help us shape the future of Go! Take the 2026 Go Developer Survey

reconciler

package
v0.5.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 13, 2026 License: AGPL-3.0 Imports: 23 Imported by: 0

Documentation

Overview

Package reconciler runs the loop that converges observed state toward pinned specs.

The dividing line: the reconciler may create and start things; it may not destroy anything a human may have wanted (R-148, R-028). A failed app stays failed — R-151 is true because there is no code path here that touches the failed state, not because a flag is checked. See design 05.

Package reconciler runs the loop that converges observed state toward pinned specs.

The dividing line: the reconciler may create and start things; it may not destroy anything a human may have wanted (R-148, R-028). A failed app stays failed — R-151 is true because there is no code path here that touches the failed state, not because a flag is checked. See design 05.

Index

Constants

View Source
const (
	BranchPollInterval  = 5 * time.Minute
	ReleasePollInterval = 15 * time.Minute
)

Auto-deploy polling intervals. [P], design 05 §5.

View Source
const (
	DeliveryPoll    = "poll"
	DeliveryWebhook = "webhook"
)

How a check came about: the poll, or a webhook from the git host (R-142).

View Source
const (
	DriftWorkloadMissing  = "workload missing"
	DriftWorkloadStopped  = "workload stopped"
	DriftWrongImage       = "wrong image"
	DriftStaleEnvironment = "stale environment"
	DriftVolumeMissing    = "volume missing"
	DriftVolumeLostData   = "volume that held data is missing"
	DriftUnknownWorkload  = "unrecognized workload"
	DriftRouteMissing     = "route missing"
)

Kinds of difference.

View Source
const (
	// Interval between ticks. Short enough that a killed container comes back
	// while someone is still looking at the page, long enough that a hundred
	// apps is not a hundred Observe calls a second.
	Interval = 15 * time.Second

	// Concurrency is how many apps are reconciled at once.
	Concurrency = 8

	// DefaultFailureThreshold and DefaultFailureWindow are R-150's give-up rule.
	DefaultFailureThreshold = 10
	DefaultFailureWindow    = 30 * time.Minute

	// DefaultLeaseDuration is how long a claim on an app lasts unless the
	// pass holding it extends it, which it does every third of this. It is
	// how long an app held by a replica that died waits to be visited again.
	DefaultLeaseDuration = 2 * time.Minute

	// DefaultMinRevisit is how recently visited an app must be for a pass to
	// leave it for the next one. Half an interval: with several replicas
	// ticking out of phase, an app one of them has just looked at is not
	// looked at again by the next a few seconds later, and on one replica a
	// pass that finishes within half an interval still visits every app on
	// every tick.
	DefaultMinRevisit = Interval / 2
)

Tunables, all [P] from design 05 §2.

View Source
const (
	// DefaultSettledRevisit is how long a settled app on a runtime whose
	// events are followed waits for its next visit: the sweep that catches
	// anything an event did not report. Twenty times the fast cadence, which
	// takes 20,000 settled apps from about 1,300 Observe calls a second to
	// about 67.
	DefaultSettledRevisit = 5 * time.Minute
)

Tunables, [P] (design 05 §2.3).

View Source
const GCInterval = time.Hour

GCInterval is how often garbage collection runs. [P], design 05 §6.

View Source
const MinProductionCap = 30 * time.Second

MinProductionCap is the smallest last-step backoff that is not obviously a test setting.

Not enforced — a floor would make the schedule untestable end to end, which is the problem this configurability exists to solve. It is the threshold for saying so loudly at startup instead: an install retrying a broken app every two seconds forever is a real way to melt a host, and it should not be something an operator can do without being told.

View Source
const RestartSettleWindow = 30 * time.Second

RestartSettleWindow is how long a workload that has restarted must stay up before it counts as recovered rather than looping. [P].

View Source
const TeardownBatch = 20

TeardownBatch bounds how many bundles one pass destroys.

Bounded because this is destructive and runs unattended. An install upgrading with two hundred long-deleted apps should reclaim them over a few hours rather than issue two hundred destroys in a burst against a runtime that is also serving live traffic.

Variables

View Source
var DefaultBackoff = []time.Duration{0, 5 * time.Second, 15 * time.Second, 60 * time.Second, 5 * time.Minute}

DefaultBackoff is R-149, capped at five minutes.

Indexed by consecutive failures, so the first correction is immediate: a container killed once should come back now, not in five seconds. The cap matters more than the curve — an app that cannot start must not be retried forever at speed, and must still be retried.

Functions

func EnvHash

func EnvHash(s *spec.AppSpec, versions map[string]int) string

EnvHash is the fingerprint of the environment an app should be running with.

func PlanShape

func PlanShape(s *spec.AppSpec, image string, perWorkload map[string]state.WorkloadImage, rules api.EgressRules) api.BundlePlan

PlanShape builds what should be running, without resolving any secret.

The reconciler compares shape — which workloads, which images, which ports and mounts — and that comparison runs every fifteen seconds for every app. It must never be a reason to decrypt anything, so environment is deliberately absent here and drift in it is detected by fingerprint instead (R-193).

Secrets are resolved only when there is something to apply, which is the uncommon case.

func TriggerOf

func TriggerOf(ad spec.AutoDeploy) spec.AutoDeployTrigger

TriggerOf is the trigger an app's settings use; empty is the branch.

Types

type AuditEvent

type AuditEvent struct {
	Action string
	AppID  string
	Detail map[string]any
}

AuditEvent is what the reconciler records.

type AuditWriter

type AuditWriter interface {
	Write(ctx context.Context, e audit.Event) error
}

AuditWriter appends to the audit log.

type Auditor

type Auditor interface {
	Write(ctx context.Context, e AuditEvent) error
}

Auditor writes the events a reconciliation produces.

type AutoDeploy

type AutoDeploy struct {
	Apps        *state.Apps
	Deployments *state.Deployments
	Checks      *state.AutoDeployChecks
	Resolver    RefResolver

	// Deployer starts a deploy as a person's would (approval.Service).
	Deployer Deployer

	// Audit records each auto-deploy, with the commit it moves from and to,
	// before anything is created (R-227, R-228).
	Audit AuditWriter

	Logger *zap.Logger

	// Concurrency is how many apps are checked at once. Eight when zero
	// [P]: each check is a `git ls-remote` against somebody's git host, and
	// in series a poll over a few thousand apps outlasted its own interval.
	Concurrency int

	// Policy is host policy as it is now. An app whose deploys need approval
	// does not auto-deploy (R-158); nil reads as no policy, so nothing is
	// skipped for it.
	Policy PolicyLoader
}

AutoDeploy watches tracked refs and deploys what they point at (R-141).

A separate scheduled job, deliberately not part of the reconciler. It never modifies a running app: it creates a spec revision carrying the new commit and deploys it through the same service a person's deploy goes through, so every plan-time check, the capacity hold and the audit event are the same. The reconciler converges what is pinned; this decides what should be pinned. Conflating them would make an unrelated drift correction able to ship new code.

Off by default (R-141). Nothing here runs for an app that did not ask.

func (*AutoDeploy) CheckApp

func (a *AutoDeploy) CheckApp(ctx context.Context, appID, delivery string) error

CheckApp checks one app now, as a poll would: what a webhook from the git host asks for (R-142). The same path as the poll, so a webhook can start nothing a poll could not — it only makes the check sooner.

func (*AutoDeploy) Poll

func (a *AutoDeploy) Poll(ctx context.Context)

Poll checks every app that auto-deploys, whichever trigger it uses.

func (*AutoDeploy) PollTrigger

func (a *AutoDeploy) PollTrigger(ctx context.Context, trigger spec.AutoDeployTrigger)

PollTrigger checks the apps that auto-deploy on trigger, and no others: what one tick of Run does.

func (*AutoDeploy) Run

func (a *AutoDeploy) Run(ctx context.Context)

Run polls until the context is canceled: apps following a branch every BranchPollInterval, apps following releases every ReleasePollInterval. Releases are rarer than pushes, and a release check lists every tag.

type BackupRunner

type BackupRunner interface {
	CreateForApp(ctx context.Context, id string, req backup.AppCreateRequest) (backup.Created, error)
	Discard(ctx context.Context, adapterRef, objectName string) error
}

BackupRunner takes and prunes per-app backups.

An interface rather than *backup.Service so the collector is testable without a destination, and so this package cannot reach anything else the service exposes — restore in particular, which the janitor has no business calling.

type BuildCaches

type BuildCaches interface {
	Forget(ctx context.Context, builderRef, appID string) error
}

BuildCaches has the builder that built an app forget its cache.

type Deployer

type Deployer interface {
	Deploy(ctx context.Context, p authz.Principal, app state.App, rev state.Revision, trigger string) (state.Deployment, error)
}

Deployer starts a deploy of rev on behalf of p.

type Desired

type Desired interface {
	SetDesiredState(ctx context.Context, appID, desired string) error
}

Desired sets an app's desired state, which is how an app is stopped and started (design 09 §4.2: never a delete, never `failed`).

type Difference

type Difference struct {
	Workload string
	Kind     string
	Detail   string
}

Difference is one thing that does not match.

type Drift

type Drift struct {
	// Reconcilable are differences the reconciler may correct. Every one of
	// them is a thing to create or start.
	Reconcilable []Difference

	// ReportOnly are differences it must not. Every one of them would mean
	// destroying or overwriting something a person may have put there.
	ReportOnly []Difference
}

Drift is the difference between what a spec says and what is running.

func Classify

func Classify(want api.BundlePlan, observed api.ObservedBundle, in Inputs) Drift

Classify compares what should be running against what is.

The dividing line, and the rule to hold when new cases come up (design 05 §2.1): **the reconciler may create and start things; it may not destroy anything a human may have wanted.**

Everything in Reconcilable is a creation or a start. Everything in ReportOnly is something where the only way to "fix" it would be to throw away state or a container that someone put there deliberately — and being wrong about that is unrecoverable, while being wrong about reporting it costs a notification.

func (Drift) Actionable

func (d Drift) Actionable() bool

Actionable reports whether there is anything the reconciler may fix.

func (Drift) Describe

func (d Drift) Describe() string

Describe renders the drift for a notification or a log line.

func (Drift) None

func (d Drift) None() bool

None reports whether anything differs.

type Environments

type Environments interface {
	Environments(ctx context.Context, s *spec.AppSpec) (map[string]map[string]secret.Value, error)
}

Environments resolves the environment each workload runs with, keyed by workload name. Asked only when a correction is about to be applied, which is the uncommon case: comparing never decrypts anything (R-193).

type GC

type GC struct {
	Apps   *state.Apps
	Logger *zap.Logger

	// Registry resolves the adapters that hold a deleted app's bundle. Nil
	// disables teardown rather than failing: an install with no runtime
	// configured has nothing to tear down.
	Registry Registry

	// Auditor records teardowns. Destroying a bundle is destruction, and
	// R-227's rule does not have an exception for the janitor.
	Auditor Auditor

	// Backups gates reclaiming a deleted app's storage on a backup holding
	// its data. Nil disables that rather than failing. Taking and expiring
	// rolling backups is RollingBackups, a job of its own (issue #72).
	Backups *state.Backups

	// The security pass (R-315, R-316). All four are nil on an installation
	// with no scanner, where nothing is scored and the pass does nothing.
	Security      SecurityScores
	SecurityState SecurityState
	PolicyStore   PolicyStore
	Desired       Desired
	Notifier      OwnerNotifier

	// Clock is here so retention is testable without waiting a day.
	Clock clock.Clock

	// Interval overrides GCInterval. Zero means the default.
	//
	// Configurable for the same reason the retry backoff is: the acceptance
	// test for bundle teardown would otherwise have to wait an hour to watch a
	// janitor run. Nothing here is urgent, so a short interval is wasteful
	// rather than dangerous — which is why this one gets no startup warning.
	Interval time.Duration

	// TeardownNow starts a teardown pass between ticks. Deleting an app sends
	// on it, so the app's containers stop when it is deleted rather than at the
	// next pass — which was up to an hour later, while a force-deleted app kept
	// running and holding its share of the host (issue #55). Nil means ticks
	// only.
	//
	// Served by a loop of its own, so a delete is acted on while the rest of
	// a pass — a security pass over every app, reclaiming storage — is still
	// running, rather than after it (issue #72).
	TeardownNow <-chan struct{}

	// TeardownEvery is how often teardown runs when nothing asks: a delete
	// whose signal reached another replica, or one whose runtime was away.
	// Ten seconds when zero.
	TeardownEvery time.Duration

	// BuildCaches and DiscardUpload remove a deleted app's build cache and
	// uploaded source at teardown. Nil skips each.
	BuildCaches   BuildCaches
	DiscardUpload func(appID string) error

	// DiscardCredential removes a deleted app's registry credential (issue
	// #41). Apps soft-delete, so the row's cascade never fires; a credential
	// for an app nobody can deploy is a key kept for no one. Nil skips it.
	DiscardCredential func(ctx context.Context, appID string) error

	// RegistryImages deletes a deleted app's builds from the install's image
	// registry (issue #72 PR 5, R-224). Nil when the install has none.
	RegistryImages RegistryImages
	// contains filtered or unexported fields
}

GC reclaims what retention says should be gone.

A separate job from the reconciler, on a much slower clock, because nothing here is urgent and all of it is destructive. The reconciler's rule — never destroy anything a human may have wanted — applies here too, and is the reason each of these deletions has an explicit exemption attached.

func (*GC) Collect

func (g *GC) Collect(ctx context.Context)

Collect runs one pass.

Spec revision pruning only, for now. Log retention (R-222–R-224) is not here because there is nothing for it to act on: Pando does not hold app logs, it streams them from the runtime, and the bytes live wherever that runtime put them. Enforcing a per-app cap means the log driver's own options at container creation; enforcing the aggregate would mean recreating every container when one app turns chatty, which is destruction on a schedule and not something the reconciler may do. Recorded as O-16 rather than guessed at.

Rolling backups and their expiry (R-211) are RollingBackups, a job of its own, and teardown also runs on a loop of its own in Run (issue #72); Collect tears down too, so one call is a whole pass.

func (*GC) Run

func (g *GC) Run(ctx context.Context)

Run collects until the context is canceled.

func (*GC) TearDownDeleted

func (g *GC) TearDownDeleted(ctx context.Context)

tearDownDeletedBundles destroys the bundles of apps that have been deleted.

Nothing used to do this. Deleting an app archived the row and left its containers running on a private network nobody reclaimed — and Pando takes one network per app against a Docker pool that holds about thirty, so an install that adds and removes apps eventually cannot start one. The failure arrives as a message about subnets, on an unrelated deploy, long after the deletion that caused it.

Here rather than in the delete handler because convergence is the reconciler's job (design 05): a delete that destroyed inline would leave a bundle standing forever if the runtime happened to be unreachable at that moment, and nothing would ever come back for it.

Volumes are destroyed only when the delete said so. R-204 has a delete settle the app's storage — keep a final backup or discard it — and a delete of an app with storage is refused until it does; both answers end with the volumes gone, the second after a backup holds the data. That decision is recorded on the app (TeardownTarget.DiscardStorage) and is the only thing that sets KeepVolumes false here: this is the one place a bug would silently destroy data, so nothing is inferred. TearDownDeleted tears down the bundles of deleted apps now, outside the GC's own loop: the replica that took a delete calls it whether or not it leads (issue #72). teardownMu keeps it from overlapping this replica's own pass, and teardown is idempotent, so meeting the leader's pass on one app is harmless.

type Inputs

type Inputs struct {
	// ExpectedDigests is what each workload ran, by name. A workload with no
	// entry is not checked: unknown is not drift.
	//
	// Per workload because an app's parts can be built separately. One digest
	// compared against every workload made every service but one look wrong
	// forever, and "correcting" that replaced them with the primary's image.
	ExpectedDigests map[string]string

	// ExpectedDigest is the image the last successful deployment ran, which is
	// the primary workload's. Kept for the single-image case and for the
	// report text; the per-workload map decides.
	ExpectedDigest string

	// AppliedEnvHash and CurrentEnvHash stand in for environment observation
	// (R-193). Either being empty means the comparison cannot be made, and an
	// unanswerable question is not drift.
	AppliedEnvHash string
	CurrentEnvHash string

	// VolumesThatHeldData is the difference between a volume Pando may recreate
	// and one it must only report.
	VolumesThatHeldData map[string]bool

	RouteExpected bool
	RoutePresent  bool
}

Inputs are the facts Classify needs that are not in the bundle plan.

type Notifier

type Notifier interface {
	Notify(ctx context.Context, n api.Notification) error
}

Notifier tells someone an app needs attention.

type OwnerNotifier

type OwnerNotifier interface {
	Notify(ctx context.Context, userID, appID, subject, body string)
}

OwnerNotifier tells an app's owner what is about to happen to it (R-315).

type PolicyLoader

type PolicyLoader interface {
	Load(ctx context.Context) (policy.Document, error)
}

PolicyLoader reads host policy, per poll rather than cached (R-274): an administrator who starts requiring approval stops the next poll.

type PolicyStore

type PolicyStore interface {
	Document(ctx context.Context) (policy.Document, error)
}

PolicyStore reads host policy.

type Reconciler

type Reconciler struct {
	Apps       *state.Apps
	Reconciles *state.Reconciles
	Secrets    *state.Secrets

	// Services adds provisioned services to what should be running (R-131).
	// Nil on an install with no provisioner, where every app's shape is already
	// complete.
	Services ServiceShapes

	// Environments supplies what each workload runs with when a correction
	// is applied (R-148). Nil applies the shape as it is, which is complete
	// only for an app whose workloads take no environment.
	Environments Environments

	Volumes  *state.Volumes
	Registry Registry
	Auditor  Auditor
	Notifier Notifier
	Logger   *zap.Logger
	Clock    clock.Clock

	// BuiltImageAuth is the install registry's credential for an image Pando
	// pushed there, nil for any other (issue #72, PR 5). A workload restored
	// on a runtime that pulls needs it for the pull, as the deploy did. Nil
	// means no install registry.
	BuiltImageAuth func(ctx context.Context, ref string) *api.RegistryAuth

	// Backoff, FailureThreshold and FailureWindow override R-149 and R-150's
	// defaults. Zero values mean the defaults, so a caller that does not care
	// sets nothing.
	//
	// Configurable because the acceptance test for R-151 — a crash-looping app
	// reaches failed and stays there — has to wait out the real schedule, and
	// at the production numbers that is forty minutes of a forty-three minute
	// suite. The test is asserting the state machine, not the durations, and it
	// was paying for the durations.
	//
	// The state machine is what stays under test either way: compressing the
	// schedule changes how long each step waits and nothing about which step
	// comes next.
	Backoff          []time.Duration
	FailureThreshold int
	FailureWindow    time.Duration

	// MinRevisit is how recently visited an app may be and still be left
	// for a later pass (DefaultMinRevisit in production). Zero visits every
	// app on every pass, which is what a test calling Tick in a row wants.
	MinRevisit time.Duration

	// LeaseDuration overrides DefaultLeaseDuration. Zero means the default.
	LeaseDuration time.Duration

	// SettledRevisit is how long an app found as it should be is left before
	// it is looked at again, when its runtime reports events (O-52, events.go).
	// Zero means DefaultSettledRevisit.
	SettledRevisit time.Duration

	// Runtimes names the runtime adapters whose events Run follows (O-52).
	// Asked again on every tick, so a runtime added in the console is
	// followed without a restart. Nil follows none, and every app is visited
	// on every pass, as before events.
	Runtimes func() []string

	// ProxyUpstream is where routes point. Every route points at Pando's proxy
	// and never at a workload (R-023) — the reconciler re-ensuring a route must
	// not be the one place that forgets.
	ProxyUpstream string
	// contains filtered or unexported fields
}

Reconciler converges running apps toward their pinned specs.

func (*Reconciler) Run

func (r *Reconciler) Run(ctx context.Context)

Run ticks until the context is canceled.

Every replica runs this, and every replica follows every runtime's events (O-52, events.go): the lease gives no app an owning replica, so the replica that hears about an app is not necessarily the one that will visit it. An event makes the app due in the database, where any replica's next claim finds it, and wakes this replica's loop so it does not wait for the tick.

func (*Reconciler) Tick

func (r *Reconciler) Tick(ctx context.Context)

Tick reconciles every app that is due.

Apps are claimed in small batches under a lease (state.Lease), least recently visited first, until none is left that was last visited before the pass began. The batches keep the number held but not yet started small, and the lease is what keeps two passes — on this replica or another — off one app. Nothing is held on a database connection while an app is reconciled, however long its Apply takes.

type RefResolver

type RefResolver interface {
	Resolve(ctx context.Context, src spec.Source, ad spec.AutoDeploy) (source.Tracked, error)
}

RefResolver finds what an app's auto-deploy trigger points at now.

Listing a remote's refs rather than cloning: this runs every few minutes for every app that auto-deploys, and cloning each time to learn a SHA that has usually not changed would be most of Pando's network traffic.

type Registry

type Registry interface {
	Runtime(ref string) (api.RuntimeAdapter, bool)
	Routing(ref string) (api.RoutingAdapter, bool)
}

Registry resolves adapters by reference.

type RegistryImages

type RegistryImages interface {
	DeleteApp(ctx context.Context, appID string) error
}

RegistryImages removes a deleted app's images from the install's registry, by manifest. The layers are freed by the registry's own scheduled garbage collection (O-38).

type RollingBackups

type RollingBackups struct {
	Apps         *state.Apps
	Backups      *state.Backups
	Backup       BackupRunner
	BundleSource *state.BundleSource
	Auditor      Auditor
	Logger       *zap.Logger
	Clock        clock.Clock

	// Every is how often to look for apps due a backup. Five minutes when
	// zero [P]: a backup is due once a day, so this is how late one can be.
	Every time.Duration

	// Concurrency is how many backups run at once. Two when zero [P]: a
	// backup copies a volume, and the disk it reads is the one the apps use.
	Concurrency int

	// RetryAfter is how long after an attempt that was skipped or failed the
	// app is tried again. An hour when zero, which is what the attempt's
	// remedy tells a person ("within the hour").
	RetryAfter time.Duration

	// Batch is the most apps one pass takes. 200 when zero.
	Batch int
}

RollingBackups takes rolling backups for apps that are due one, and removes backups whose retention has passed (R-211). A leader job of its own (issue #72), not part of the GC's hourly pass: that walked every app with storage in series, so one slow destination held up every other app's backup and the rest of the GC with it.

Due-driven: each pass asks the database for a page of apps due a backup and takes them a few at a time, so how long a pass takes depends on how many are due, not on how many apps there are.

func (*RollingBackups) Pass

func (b *RollingBackups) Pass(ctx context.Context)

Pass takes every due backup it is given a page of, then expires old ones.

func (*RollingBackups) Run

func (b *RollingBackups) Run(ctx context.Context)

Run takes backups until ctx ends.

type SecurityScores

type SecurityScores interface {
	Configured() (string, bool)
	Place(ctx context.Context, scores map[string]security.Scores) (map[string]security.Placed, error)
}

SecurityScores is what this pass needs from the security service.

Place, the batched path, rather than a Report per app: a report is a policy load, a scan read and a state read for each app, which over twenty thousand apps is sixty thousand queries an hour for an answer one query and one policy load give (issue #72). The scores come from LiveSecurityState.

type SecurityState

type SecurityState interface {
	LiveSecurityState(ctx context.Context, t state.SecurityThreshold) ([]state.SecurityState, error)
	MarkInsecure(ctx context.Context, appID string, at time.Time) error
	ClearInsecure(ctx context.Context, appID string) error
	SetStoppedForSecurity(ctx context.Context, appID string, stopped bool) error
}

SecurityState is the store half.

type ServiceShapes

type ServiceShapes interface {
	ServiceShapes(ctx context.Context, s *spec.AppSpec) (api.BundlePlan, error)
}

ServiceShapes adds an app's provisioned services to what should be running.

An interface rather than the Runner so a Reconciler can be built without a deploy Runner, which the unit tests do. Nil means an install with no provisioner, where every app's shape is already complete.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL