reconcile

type Builder ¶

type Builder struct {
	// contains filtered or unexported fields
}

Builder is the sole public construction entry point for a Loop. Use New(reconciler, tenancy).With*(...).Build() to create a Loop.

Design: Builder fields are all unexported; Build() constructs a *Loop with private fields set from the Builder's configuration and returns it. A Loop with no configured Trigger never receives external work and cannot converge — Build() returns an error in that case.

Missing-required validation in Build:

• reconciler must be non-nil (validation.IsNilInterface guard).
• tenancy must be a declared stance (Tenancy.IsUnset rejected): a reconciler emits under a tenantless system principal, so it must consciously declare whether "_notenant" Claimer keys are correct for it (#1954).
• trigger must be provided (missing trigger → Loop never gets work).
• a FencedRepo requires a Leader: fencing is only meaningful with a leadership epoch source; WithFencedRepo without WithLeader would silently bind Epoch 0 (no fencing), so Build rejects it.

All other runtime fail-fasts (typed-nil Leader/FencedRepo, backoff inversion, reconcilerID charset) are preserved in preStartValidate (Loop.Start) — not duplicated in Build.

There is no WithClock option by design: control-plane scheduling (timers, startup probe, lease renew) uses a sealed real-only clock (see Loop.controlPlaneClock); Reconciler implementations source their own domain clock.

There is no WithLogger option by design either: the Loop logs through slog.Default(), which every production entry point seals to the redacting process-global sink (runtime/observability/logging). A per-loop raw logger would bypass that fail-closed redaction. (So the With* set is the faithful superset of every CONSUMER-configurable field — logger and the control-plane clock are deliberately framework-owned, not Builder inputs.)

ref: kubernetes-sigs/controller-runtime pkg/builder/controller.go (TypedBuilder.Build — fields unexported, fluent With* return *Builder, terminal Build validates and constructs the controller).

func New(reconciler Reconciler, tenancy Tenancy) *Builder

func (b *Builder) Build() (*Loop, error)

func (b *Builder) WithBackoff(base, max time.Duration) *Builder

func (b *Builder) WithConcurrency(n int) *Builder

func (b *Builder) WithFencedRepo(repo FencedRepository) *Builder

func (b *Builder) WithInterval(d time.Duration) *Builder

func (b *Builder) WithLeader(leader LeaderElector) *Builder

func (b *Builder) WithMetrics(m Metrics) *Builder

func (b *Builder) WithName(name string) *Builder

func (b *Builder) WithReconcilerID(id string) *Builder

func (b *Builder) WithRenewInterval(d time.Duration) *Builder

func (b *Builder) WithTrigger(trigger Trigger) *Builder

func (b *Builder) WithoutDefaultRequeue() *Builder

type FencedRepository ¶

type FencedRepository interface {
	ApplyFenced(ctx context.Context, entityID string, epoch uint64, mutation any) (accepted bool, err error)
}

FencedRepository is the consumer-implemented, epoch-aware persistence seam that closes the cross-replica correctness gap leader election alone cannot (see LeaderElector — "leader election is NOT fencing"). A consumer that performs device writes / command emission from inside Reconcile wires its repository as a FencedRepository on the Loop; the Loop then injects a per-Reconcile epoch-bound FencedWriter into ctx, and that writer is the reconciler's ONLY write surface.

ApplyFenced MUST implement a monotonic-epoch compare-and-swap: the resource row records the highest epoch it has ever seen, and the write is applied only when epoch >= row.last_epoch (and last_epoch advanced to epoch). It returns accepted=false (NOT an error) when epoch is stale (< highest-seen) so a zombie leader's late write is rejected structurally rather than corrupting state. This is Kleppmann (DDIA §8.4) monotonic fencing, NOT the kernel/outbox UUID identity-fencing (which only rejects "not equal to current"): a device write may land after several L1→L2→L3 handoffs and only "older than highest-seen" can reject an out-of-order late write.

A type-assertion failure on mutation is a programmer error (the wrong mutation type was passed to the writer). The implementation SHOULD return a non-nil error (transient or wrapped with reconcile.PermanentError to dead-letter) rather than panic, so the Loop can classify and log it correctly.

The implementation MUST forward ctx to all I/O operations. Ignoring ctx cancellation defeats the lost-lease interrupt (ADR §4.2): a lease loss cancels the ctx so the in-flight Reconcile terminates promptly; an implementation that ignores the ctx will proceed with a write after the lease is gone, which is the split-brain scenario fencing exists to prevent.

mutation is `any` deliberately. The reconciler reaches FencedWriter through a non-generic context value (Reconciler.Reconcile is frozen single-method and takes no extra param), so a generic FencedRepository[M] would have to be boxed to `any` at the ctx seam anyway — the type parameter buys nothing while making the Loop (which mints the writer without knowing M) impossible to keep monomorphic. The consumer's ApplyFenced does the single type-assert at its own write boundary. The fencing guarantee depends on the epoch being unforgeable and the CAS being applied — not on the payload being statically typed.

type FencedWriter ¶

type FencedWriter struct {
	// contains filtered or unexported fields
}

FencedWriter is the reconciler's intended write surface for one Reconcile call. It is a STRUCT with unexported fields and an unexported constructor (newFencedWriter), so a consumer in another package can RECEIVE one (from FencedWriterFrom) but can never COMPOSE one with an attacker-chosen epoch — reconcile.FencedWriter{epoch: 999} is a compile error outside this package. The epoch a FencedWriter carries is therefore always the Loop's live-lease value; forging it is unrepresentable in Go's type system (type-system Hard).

Honest scope (PR-A6 review C3): this seal closes "forge a writer with a chosen epoch", NOT "a consumer cannot emit an unfenced write at all". A consumer's own FencedRepository.ApplyFenced(ctx, id, epoch, mut) takes epoch as a caller param, so a direct call bypasses this writer — that vector is closed DOWNSTREAM by the RECONCILE-FENCED-WRITE-FUNNEL-01 ApplyFenced caller-allowlist (archtest, not the type system). See that archtest's godoc for the full three-vector grade.

func FencedWriterFrom(ctx context.Context) (FencedWriter, bool)

func (w FencedWriter) Epoch() uint64

func (w FencedWriter) Write(ctx context.Context, entityID string, mutation any) error

type LeaderElector ¶

type LeaderElector interface {
	// AcquireLease attempts to become the leader for reconcilerID. On success it
	// returns a LeaseToken whose ExpiresAt is now+LeaseDuration and whose Epoch is
	// the monotonic fencing token (incremented on every holder CHANGE, see
	// LeaseToken.Epoch). On contention (another holder owns a live lease) it
	// returns a non-nil error; the Loop treats any acquire error as fail-closed
	// (it does NOT dispatch) and retries after a backoff.
	AcquireLease(ctx context.Context, reconcilerID string) (LeaseToken, error)
	// ReleaseLease relinquishes the lease for a graceful handoff (the next
	// AcquireLease by any holder takes over immediately rather than waiting for
	// TTL expiry). Idempotent; a release of an already-lost lease is not an error.
	ReleaseLease(ctx context.Context, token LeaseToken) error
	// RenewLease extends the lease's TTL while keeping its Epoch unchanged. It
	// returns ErrReconcileLeaseLost (errors.Is-matchable) when the lease is no
	// longer owned by this holder (expired or taken over) — the Loop cancels its
	// lease-scoped ctx the instant this happens to interrupt the in-flight
	// Reconcile. Other (I/O) errors are transient and also drive a re-acquire.
	RenewLease(ctx context.Context, token LeaseToken) error
}

LeaderElector is the leader-election seam a multi-replica reconcile deployment wires into a Loop. The interface is declared in kernel/reconcile and implemented in adapters (adapters/redis SETNX+EXPIRE + epoch key, adapters/postgres row-TTL CAS in reconcile_leases) — the kernel-declares / adapter-implements layering that mirrors outbox.Emitter and persistence.CellTxManager.

CRITICAL — leader election is NOT fencing. client-go's own tools/leaderelection documents: "This implementation does not guarantee that only one client is acting as a leader (a.k.a. fencing)." STW GC pauses, clock skew, and renew/acquire races all leave a residual dual-execution window: an old leader paused mid-Reconcile, its lease expired and taken over, then it wakes and finishes its write. The Loop uses the lease only to NARROW that window (single dispatcher in steady state + lost-lease ctx cancel). Cross-replica correctness is the job of the monotonic LeaseToken.Epoch fed into a FencedWriter (see fenced.go) plus consumer idempotency — never the lease. Adapter choice matters for the epoch provenance: Redis keeps the epoch in an evictable key and is best-effort under allkeys-* eviction, while Postgres keeps the epoch in a durable row and is the strong-fencing adapter when eviction residuals are unacceptable.

A nil LeaderElector on a Loop means single-process mode: the Loop is always the leader, Epoch is 0, and no fencing is applied (the single-replica deployment has no zombie-leader window to fence).

INVARIANT: RECONCILE-LEADER-INTERFACE-FROZEN-01 — the method set is frozen to exactly AcquireLease / ReleaseLease / RenewLease with the signatures below, and LeaseToken's field set is frozen (in particular the monotonic Epoch uint64). A reflect golden assertion in CI rejects any added method or changed field; it is a public-contract change that must be made together with the design ADR.

ref: kubernetes/client-go tools/leaderelection/leaderelection.go (lease/renew model + the explicit "not fencing" disclaimer).

type LeaseTTL ¶

type LeaseTTL struct {
	// contains filtered or unexported fields
}

LeaseTTL is a validated lease-window value — a sealed value type whose only constructor, NewLeaseTTL, rejects sub-millisecond durations so the truncation-to- zero defect is UNREPRESENTABLE for any constructed value rather than guarded at each adapter call site.

The lease window reaches the Redis PX / Postgres `interval '1 millisecond'` lease commands as an integer-millisecond quantity (Milliseconds). A time.Duration finer than that wire unit truncates to 0 there, yielding a lease that expires the instant it is written — so a follower acquires immediately and multiple replicas run as leader at once, breaking the cross-replica mutual exclusion the monotonic LeaseToken.Epoch fencing relies on. NewLeaseTTL fail-closes that window at the boundary; no elector can be built with a sub-millisecond lease.

Sealed construction: the unexported ms field means no package outside reconcile can mint a non-zero LeaseTTL via a struct literal; the only escape is the zero value LeaseTTL{} (ms == 0), which IsZero reports so elector constructors reject it fail-fast — a loud forgot-to-construct error, never the silent sub-ms truncation.

INVARIANT: any LeaseTTL returned by NewLeaseTTL has ms >= 1, so Milliseconds() never truncates to 0 and Duration() is never sub-millisecond. Enforced by the unexported field (compile-time sealing) + NewLeaseTTL's guard; covered by leasettl_test.go. ref: kubernetes/client-go tools/leaderelection (LeaseDuration is a positive, usable window validated before election starts).

func NewLeaseTTL(d time.Duration) (LeaseTTL, error)

func (t LeaseTTL) Duration() time.Duration

func (t LeaseTTL) IsZero() bool

func (t LeaseTTL) Milliseconds() int64

type LeaseToken ¶

type LeaseToken struct {
	// ReconcilerID is the leader-election key (one lease per reconciler identity).
	ReconcilerID string
	// HolderID identifies the replica that holds this lease — minted once per
	// elector instance so two replicas competing for the same ReconcilerID are
	// distinguishable (renew ownership CAS, handoff epoch bump).
	HolderID string
	// Epoch is the monotonic fencing token. The lease store increments it on
	// every holder CHANGE (a successful AcquireLease that takes the lease from
	// free/expired/another-holder) and keeps it UNCHANGED across RenewLease. The
	// Loop binds it to a FencedWriter so a zombie leader's late write — carrying a
	// strictly lower Epoch — is rejected by the resource's write-path CAS. This is
	// monotonic (Kleppmann) fencing: reject ALL epoch < highest-seen, not the
	// outbox UUID identity-fencing (equal-or-reject), because a device write may
	// arrive after several L1→L2→L3 handoffs and only "older than highest" — not
	// "not equal to current" — can reject an out-of-order late write.
	Epoch uint64
	// AcquiredAt is when this holder acquired (or re-acquired) the lease.
	AcquiredAt time.Time
	// ExpiresAt is when the lease lapses absent a successful RenewLease.
	ExpiresAt time.Time
}

LeaseToken is the opaque, value-typed proof of leadership an adapter mints and the Loop threads back into ReleaseLease / RenewLease. It is NOT sealed-construction: adapters in different packages must build it, so its fields are exported. The frozen field set is enforced by RECONCILE-LEADER-INTERFACE-FROZEN-01.

type Loop ¶

type Loop struct {
	// contains filtered or unexported fields
}

Loop is the reconcile scheduling environment: it pulls Requests from a source, dispatches each to the Reconciler across a bounded worker pool (serializing per EntityID), and requeues per the returned Result. Its lifecycle skeleton (Start fast-return + startup probe, owner-ctx derivation, graceful Stop) is the scheduling loop's own control shell; the per-entity worker dispatch and requeue are reconcile-specific.

Construct via the Builder (reconcile.New(r, tenancy).With*().Build()) in production. All configuration fields are unexported; the Builder is the sole public construction entry point. Loop config fields are private so that &reconcile.Loop{Field: ...} from outside the package is a compile error — the funnel is enforced by the type system (Hard upstream).

Defaulting is handled by a single applyDefaults() called once at the end of preStartValidate(), before the metric preflight, so start() sees all config fields at their final values.

Owner ctx (controller-runtime Runnable.Start semantics): Start receives the long-lived owner ctx and derives the workers' runCtx from it, so assembly shutdown (ownerCancel) drains the Loop even without an explicit Stop.

Start must return promptly (spawn pool + fast probe, then return) — it is usable directly as a cell.LifecycleHook.OnStart; Stop as OnStop.

ref: kubernetes-sigs/controller-runtime pkg/builder/controller.go

func (l *Loop) Start(ownerCtx context.Context) error

func (l *Loop) Stop(ctx context.Context) error

type Metrics ¶

type Metrics struct {
	// Total counts reconcile outcomes, labels {reconciler, result}.
	Total kernelmetrics.CounterVec
	// Duration observes reconcile wall-clock seconds, labels {reconciler}.
	Duration kernelmetrics.HistogramVec
	// InFlight tracks concurrently-running reconciles, labels {reconciler}.
	InFlight kernelmetrics.GaugeVec
	// Leader is 1 when this instance owns the reconcile lease, else 0,
	// labels {reconciler}. In the single-process Loop it is set to 1 at Start;
	// real lease gating is the leader-election PR's concern.
	Leader kernelmetrics.GaugeVec
}

Metrics holds the optional pre-bound instruments the Loop records to. A nil field disables that instrument (the Loop nil-checks before every record). Build via RegisterMetrics at the composition root and inject; the Loop preflights label sets at Start (same construction-time label preflight pattern).

func RegisterMetrics(p kernelmetrics.Provider) (Metrics, error)

type Reconciler ¶

type Reconciler interface {
	Reconcile(ctx context.Context, req Request) (Result, error)
}

Reconciler is the single method a consumer implements to converge ONE entity toward its desired state. The Loop calls it once per scheduled Request.

Contract:

• Return (Result{RequeueAfter: d>0}, nil) to re-observe this entity after d.
• Return (Result{}, nil) (RequeueAfter == 0) to re-observe at the default tick interval.
• Return (Result{}, err) with a transient err to have the Loop back off and retry (Result is ignored when err != nil).
• Return (Result{}, PermanentError(err)) when retrying cannot help; the Loop records a dead-letter metric and stops scheduling this entity until a fresh trigger re-observes it.

Idempotency contract (P1): Reconcile MUST be idempotent. The framework may invoke it multiple times for the same entity without an intervening state change: on periodic resync (Interval tick), on F5 dirty re-run after coalesced duplicate triggers, and on transient-error retry. This is the level-triggered convergence contract — the reconciler observes current state and drives toward desired state, regardless of how many times it is called.

Reconcile MUST honor ctx cancellation (the Loop cancels ctx on shutdown / StopTimeout).

INVARIANT: RECONCILE-INTERFACE-FROZEN-01 — the method set is frozen to exactly Reconcile(context.Context, Request) (Result, error). Adding a method, or changing this signature, trips an exact-set reflect assertion in CI; it is a public-contract change that must be made together with the design ADR.

ref: kubernetes-sigs/controller-runtime pkg/reconcile/reconcile.go (Reconciler)

type Request ¶

type Request struct {
	// EntityID identifies the single entity to reconcile. It is opaque to the
	// Loop and meaningful only to the Reconciler (typically a primary key in the
	// cell's own table).
	//
	// An empty EntityID is the "resync-all" sentinel emitted by TickerTrigger
	// (a context-free interval ticker has no entity to name): a Reconciler wired
	// to a TickerTrigger MUST treat req.EntityID == "" as a "re-observe every
	// entity you own" sweep, fanning out to its own entity set.
	//
	// Bounded-set contract (S1): EntityID MUST come from a bounded set (e.g.
	// primary keys of a cell-local table). The Loop retains a small per-entity
	// backoff counter in memory for every entity that has experienced at least one
	// transient error; the counter is released on success, permanent failure, or
	// Loop restart (mirroring controller-runtime workqueue). An unbounded or
	// high-cardinality EntityID space (e.g. user input strings) would grow this
	// in-memory map without bound. The dirty/processing maps obey the same
	// constraint: they hold at most one entry per in-flight entity and are cleared
	// on completion or Loop restart.
	EntityID string
}

Request is the minimal locator the Loop hands a Reconciler. Unlike controller-runtime's reconcile.Request (which carries a types.NamespacedName), a GoCell entity lives in a single cell-local table and is addressed by one opaque EntityID — there is no namespace dimension.

INVARIANT: RECONCILE-REQUEST-FIELDS-FROZEN-01 — the field set is frozen to exactly { EntityID string }. Adding a field (e.g. a Namespace or Priority) trips an exact-set reflect assertion in CI.

type Result ¶

type Result struct {
	// RequeueAfter, when > 0, asks the Loop to re-observe this entity after the
	// given duration. When 0, the entity is re-observed at the Loop's default
	// tick interval. It is ignored when Reconcile also returns a non-nil error
	// (the error path drives backoff instead).
	RequeueAfter time.Duration
}

Result is the scheduling hint a Reconciler returns to the Loop. Unlike controller-runtime's reconcile.Result { Requeue bool; RequeueAfter Duration }, GoCell carries only RequeueAfter: "requeue immediately / at the default tick" is expressed as RequeueAfter == 0, so the Requeue bool is redundant and dropped. There is no Priority dimension.

INVARIANT: RECONCILE-RESULT-FIELDS-FROZEN-01 — the field set is frozen to exactly { RequeueAfter time.Duration }. Re-introducing Requeue bool or a Priority int (controller-runtime / K8s-scheduler residue) trips an exact-set reflect assertion in CI.

type Tenancy ¶

type Tenancy struct {
	// contains filtered or unexported fields
}

Tenancy is a reconciler's declared tenant stance — a sealed, closed-set value a reconcile Loop MUST declare at construction (the required second argument to reconcile.New). It exists so "didn't think about the tenant dimension of the emitted commands' Claimer dedup key" is unrepresentable at construction (#1954).

Why a required declaration ¶

A reconcile Loop runs under a positively-installed, TENANTLESS system producer identity (installSystemProducerIdentity): actor/subject="system", tenant cleared. So every command a reconciler emits lands under the "_notenant" Claimer-key namespace BY CONSTRUCTION (idemkey.DeriveCommandKey with an empty tenant). That is correct for a single-tenant archetype where entity ids are globally unique (e.g. device UUIDs), but a multi-tenant reconciler MUST encode the tenant in its OWN command-id derivation — otherwise two tenants sharing an entity id collide on one "_notenant" key and one tenant's work silently suppresses the other's. Before #1954 that was a pure godoc MUST (Soft, forbidden by ai-robust.md §分级). Making Tenancy a required positional parameter turns the silent omission into a conscious, type-checked, auditable choice.

Sealed construction (closed value set) ¶

The single field `mode` is unexported and the only minters are SingleTenant() and TenantScoped(). Outside this package there is NO way to construct a NON-ZERO Tenancy: reconcile.Tenancy{mode: …} is a compile error (unexported field). The zero value reconcile.Tenancy{} IS constructible (mode == tenancyUnset) but is invalid — IsUnset() reports it and Build() rejects it fail-fast (the same accepted residual as authz.Permission{} / reconcile.LeaseTTL{}). The two stances are exposed as ACCESSOR FUNCTIONS (not exported vars) that construct the value INLINE: no package-level Tenancy var exists to reassign (in OR out of package), and a func declaration cannot be reassigned, so the value is immutable everywhere (an exported var would leave the F6 gap that authz.SingleTenant = Tenancy{} could zero). The frozen membership of the minter set is the external witness RECONCILE-TENANCY-DECLARED-01 (tools/archtest).

AI-robust rating (four axes; full statement in the archtest godoc) ¶

• Forging a non-zero Tenancy → type-system HARD (sealed construction).
• Omitting the stance → compile HARD (positional param: reconcile.New(r) does not compile).
• Passing the zero Tenancy{} → runtime fail-fast (Build rejects it).
• The minter VALUE SET → MEDIUM (RECONCILE-TENANCY-DECLARED-01 archtest freeze).

Residual blind spot: the guard forces a declaration but cannot verify a TenantScoped reconciler's body actually encodes the tenant, nor cross-check a SingleTenant declaration against an actual multi-tenant cell (no cell-level tenancy signal exists — multi-tenancy is a runtime ctx property). See the archtest godoc and ADR 202606101200-1821.

func SingleTenant() Tenancy

func TenantScoped() Tenancy

func (t Tenancy) IsUnset() bool

type Trigger ¶

type Trigger interface {
	Start(ctx context.Context, queue chan<- Request) error
}

Trigger is the source of reconcile work: Start launches a producer that feeds Requests into the Loop's work queue until ctx is canceled. It is GoCell's minimal analog of controller-runtime's source.Source — the K8s informer / predicate machinery is dropped (GoCell has no control plane to watch), so the sink is a plain chan<- Request rather than a rate-limiting workqueue (deduplication / backoff live in the Loop, not the Trigger).

Contract:

• Start MUST be non-blocking: spawn the producer goroutine and return. It is invoked once by the Loop (or the Builder, PR-A7) with the Loop's internal queue. (Mirrors source.Source's "Start must be non-blocking".)
• The producer MUST honor ctx: exit promptly on ctx.Done(), and never block a queue send past cancellation (no goroutine leak on shutdown).
• The producer forwards with backpressure — it blocks on a full queue rather than dropping (level-triggered: a coalesced signal is re-observed on the next trigger / requeue, never silently lost).
• Start's error return is for genuine runtime startup failures. The two minimal triggers below validate their configuration at CONSTRUCTION (a misconfigured trigger is a programmer error, surfaced via panic — the clock funnel + panic taxonomy convention), so their Start never fails; the return is kept for source.Source parity and future triggers (e.g. one that dials a broker) that can fail at startup.

The Loop's internal WaitGroup does NOT track the producer goroutine started from a Trigger; it is solely the Trigger's responsibility to exit on ctx.Done(). A Trigger that leaks a goroutine after ctx cancellation cannot be detected by Loop.Stop's drain. The two built-in triggers (TickerTrigger, ChannelTrigger) and the reconciletest.FakeTrigger comply with this contract.

INVARIANT: RECONCILE-TRIGGER-INTERFACE-FROZEN-01 — the method set is frozen to exactly Start(context.Context, chan<- Request) error.

ref: kubernetes-sigs/controller-runtime pkg/source/source.go

func ChannelTrigger(in <-chan Request) Trigger

func TickerTrigger(clk clock.Clock, interval time.Duration) Trigger