runctl

package
v1.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 6, 2026 License: MIT Imports: 11 Imported by: 0

Documentation

Overview

Package runctl owns the measurement run lifecycle: a single process-wide Controller decides when a run starts, when its boundaries are frozen, when its immutable snapshot may be published, and when it is aborted.

The package exists because measurement boundaries are the one thing an ISUCON toolkit cannot get "mostly right". A run whose opening moments are missing, whose closing moments leak into the next run, or whose data is published by a worker belonging to an already-abandoned run is worse than no measurement at all: it looks authoritative while being wrong. Everything here is therefore built around two invariants.

  1. Epoch fencing. Every run owns an Epoch. The Controller's current epoch advances on every successful StartRun and on every AbortRun. Background workers never touch the Controller state directly; they go through [Controller.commit] and [Controller.publish], which reject any epoch that is no longer current with ErrStaleEpoch. A worker belonging to an aborted run is therefore structurally incapable of publishing data or of dragging the run back to "finished", even when the abort could not join it in time.

  2. Fail-open measurement. Collector failures never surface as Go errors from StartRun/FinishRun; they downgrade the run's Validity instead. An error return means the Controller itself refused or could not perform the operation. Callers must always inspect Validity, never only err.

Boundaries are treated as intervals, not instants: generation collectors are switched sequentially and baseline collectors are sampled in parallel, and the measured width of both is recorded in a BoundaryWindow so that a run whose boundary was too smeared can be marked partial or invalid rather than silently trusted.

The Controller never blocks on collectors or workers while holding its mutex, because ResetNow is expected to be callable from inside an instrumented HTTP handler.

Index

Constants

View Source
const (
	// StartRunBudget bounds the whole synchronous part of StartRun.
	StartRunBudget = 6 * time.Second
	// FinishSyncBudget bounds the whole synchronous part of FinishRun
	// (freeze + final sampling). Draining and snapshot building happen after
	// the response, outside this budget.
	FinishSyncBudget = 6 * time.Second

	// PhaseStartBoundaryBudget bounds BeginBoundary across all generation
	// collectors. Generation swaps are pointer swaps, so this is generous.
	PhaseStartBoundaryBudget = 500 * time.Millisecond
	// PhaseStartBaselineBudget bounds CaptureBaseline across all baseline
	// collectors. Baseline sampling does bounded I/O (procfs, DB), hence the
	// order-of-magnitude difference from the boundary phase.
	PhaseStartBaselineBudget = 5 * time.Second
	// PhaseFinishFreezeBudget bounds Freeze across all generation collectors.
	PhaseFinishFreezeBudget = 500 * time.Millisecond
	// PhaseFinishFinalBudget bounds CaptureFinal across all baseline collectors.
	PhaseFinishFinalBudget = 5 * time.Second

	// PerCollectorGenerationBudget bounds one boundary operation of one
	// generation collector.
	PerCollectorGenerationBudget = 100 * time.Millisecond
	// PerCollectorBaselineBudget bounds one sampling operation of one baseline
	// collector, including all of the DB targets it fans out to.
	PerCollectorBaselineBudget = 3500 * time.Millisecond
	// PerTargetBudget bounds one collector's call against one DB target.
	PerTargetBudget = 1 * time.Second

	// DrainBudget bounds draining every handle of one run. Background work.
	DrainBudget = 10 * time.Second
	// DrainCancelGrace is how long a collector's Drain may take to return
	// after its context is done. Collectors must wait on a per-generation done
	// channel rather than sync.Cond, because sync.Cond.Wait cannot be
	// interrupted by a context and would make this contract unimplementable.
	DrainCancelGrace = 1 * time.Second
	// SnapshotBuildBudget bounds Collect plus immutable snapshot construction.
	SnapshotBuildBudget = 5 * time.Second
	// EnrichBudget bounds post-freeze enrichment such as EXPLAIN capture.
	EnrichBudget = 2 * time.Second
	// AbortJoinBudget bounds how long AbortRun waits for the run's worker.
	// Exceeding it is safe but noisy: the worker is detached and fenced.
	AbortJoinBudget = 2 * time.Second
	// PreemptTotalBudget bounds the whole preempt path.
	PreemptTotalBudget = AbortJoinBudget + StartRunBudget
	// DetachedReapBudget bounds how long the reaper watches a detached worker
	// before giving up on reclaiming its handles.
	DetachedReapBudget = 60 * time.Second
	// InitializeGuardBudget bounds how long SerializeInitialize waits for the
	// process-wide initialize guard.
	InitializeGuardBudget = 30 * time.Second
)

Hierarchical time budgets. This package is the single authority for these numbers: downstream collectors cite them instead of inventing their own, so that "why did my collector get cut off?" always has one answer.

The hierarchy is strict — a run budget bounds a phase budget, which bounds a per-collector budget, which bounds a per-target budget. TestBudgetHierarchy pins the inequalities so a future tweak cannot quietly invert them.

View Source
const (
	// FinishLease is how long one phase of a finish may live before the
	// watchdog force-aborts the run. It is armed twice: FinishRun arms it over
	// the synchronous freeze (hence FinishSyncBudget < FinishLease), and the
	// background worker re-arms it when it takes over (hence
	// DrainBudget+SnapshotBuildBudget+EnrichBudget < FinishLease). Both halves
	// are therefore bounded by it, and neither is charged for the other's time.
	FinishLease = 20 * time.Second
	// StartedTTL reclaims runs that nobody ever finished.
	StartedTTL = 30 * time.Minute
	// FinishedTTL is how long a finished or acknowledged snapshot is retained.
	FinishedTTL = 10 * time.Minute
	// TombstoneTTL is how long aborted and expired records are retained.
	TombstoneTTL = 10 * time.Minute
	// NonceTTL bounds the StartRun idempotency cache.
	NonceTTL = 10 * time.Minute
	// WatchdogInterval is how often leases and TTLs are examined.
	WatchdogInterval = 1 * time.Second
)

Lease and TTL defaults. All of them are injectable through Budgets so tests never wait out a real 30 minute TTL.

View Source
const (
	// BaselineConcurrency caps parallel baseline sampling. Parallelism is what
	// keeps the boundary window narrow; the cap keeps the measured application
	// from being hit by a thundering herd of collectors.
	BaselineConcurrency = 8
	// NonceHistoryMax bounds the nonce idempotency cache.
	NonceHistoryMax = 64
	// RetainedRuns is how many run records the Controller keeps, *including*
	// the in-flight one. Two generations back is therefore always a 404.
	RetainedRuns = 2
)

Structural limits.

View Source
const (
	// SpreadLimitGeneration caps the measured width of the generation swap
	// window. Expected in practice: well under a millisecond.
	SpreadLimitGeneration = 50 * time.Millisecond
	// SpreadLimitBoundary caps the measured width of the whole boundary
	// window, generation and baseline collectors together. Expected in
	// practice: under 200ms with parallel sampling.
	SpreadLimitBoundary = 1500 * time.Millisecond
)

Boundary spread limits. These are deliberately unrelated to the budgets above: a budget is a forced cutoff that protects the application, a spread limit is a quality threshold that decides whether the boundary is usable as a boundary. "Inside budget but over spread" is a normal, expected verdict.

View Source
const (
	KindGeneration = "generation"
	KindBaseline   = "baseline"
)

Collector kinds as they appear in CollectorBoundary.Kind.

View Source
const (
	// CodeNotCaptured means the collector was never sampled because the phase
	// budget ran out.
	CodeNotCaptured = "not-captured"
	// CodeDrainTimeout means in-flight work did not settle within DrainBudget.
	// Partial data still exists and is kept.
	CodeDrainTimeout = "drain-timeout"
	// CodeCollectFailed means the interval value could not be derived.
	CodeCollectFailed = "collect-failed"
	// CodeBoundaryFailed means the boundary operation itself returned an error.
	CodeBoundaryFailed = "boundary-failed"
	// CodeSpreadExceeded marks a collector whose measured time sits outside the
	// allowed boundary window.
	CodeSpreadExceeded = "spread-exceeded"
	// CodeContractViolation means the collector reported success without
	// committing, which is a collector bug rather than a runtime condition.
	CodeContractViolation = "contract-violation"
)

Machine-readable failure codes. An empty code means the step succeeded. The set is closed: transports and advisors switch on these values.

View Source
const (
	// AckedByExplicit is a direct Ack call.
	AckedByExplicit = "explicit"
	// AckedBySave is the implicit acknowledgement performed by POST /save.
	AckedBySave = "save"
	// AckedByPreempt is the implicit acknowledgement of a finished run whose
	// successor started with Preempt. The snapshot is retained.
	AckedByPreempt = "preempt"
	// AckedByHub is an acknowledgement driven by a multi-host hub.
	AckedByHub = "hub"
	// AckedByLease is a self-acknowledgement after a peer ack lease expired, so
	// a vanished hub cannot wedge the system in "finished".
	AckedByLease = "lease"
)

Stable AckedBy values. This package owns the set; multi-host transports copy the field one-to-one and must not invent values.

View Source
const (
	// ReasonExplicit is an operator- or API-driven abort.
	ReasonExplicit = "explicit"
	// ReasonRequiredFailed is a required collector failing the opening boundary.
	ReasonRequiredFailed = "required-failed"
	// ReasonPreemptedBy is the prefix for "preempted-by:<runID>".
	ReasonPreemptedBy = "preempted-by:"
	// ReasonFinishLeaseExpired is the watchdog reclaiming a stuck worker.
	ReasonFinishLeaseExpired = "finish-lease-expired"
	// ReasonStartedTTL is the watchdog reclaiming a run nobody finished.
	ReasonStartedTTL = "started-ttl"
	// ReasonHubAbort is a multi-host hub aborting a peer's run.
	ReasonHubAbort = "hub-abort"
)

Stable AbortResult.Reason values.

View Source
const (
	// HealthBoundarySpread reports a boundary window wider than its limit.
	HealthBoundarySpread = "runctl-boundary-spread"
	// HealthContractViolation reports a collector returning success without
	// committing its boundary.
	HealthContractViolation = "runctl-contract-violation"
	// HealthWorkerDetached reports an abort that could not join its worker.
	HealthWorkerDetached = "runctl-worker-detached"
	// HealthLeaseExpired reports a finishing worker killed by FinishLease.
	HealthLeaseExpired = "runctl-lease-expired"
)

Health keys this package reports. The set is fixed at four so that a health snapshot stays readable; new conditions reuse these keys with a different message rather than adding keys.

Variables

View Source
var (
	// ErrRunActive is returned when another run occupies the Controller and
	// the caller did not ask to preempt it. Callers that must win (an
	// initialize handler, for example) retry with StartRunOptions.Preempt.
	// Maps to HTTP 409.
	ErrRunActive = errors.New("runctl: another run is active")

	// ErrRunTransitioning is returned while a run is mid-transition
	// (starting or aborting) and the requested operation cannot be ordered
	// against it. Retrying after the transition settles is safe.
	// Maps to HTTP 409.
	ErrRunTransitioning = errors.New("runctl: run is transitioning")

	// ErrRunAborted is returned for operations on a run that was abandoned.
	// The run's data is gone for good, so this is deliberately distinct from
	// ErrUnknownRun: the caller learns the run existed and failed rather than
	// that it never existed. Maps to HTTP 410.
	ErrRunAborted = errors.New("runctl: run was aborted")

	// ErrUnknownRun is returned for a run the Controller does not retain:
	// never started, evicted by RetainedRuns, or expired. Maps to HTTP 404.
	ErrUnknownRun = errors.New("runctl: unknown run")

	// ErrStaleEpoch rejects a state change or a snapshot publication coming
	// from a worker whose run is no longer current. It is an internal fence
	// signal and is not expected to reach an API caller.
	ErrStaleEpoch = errors.New("runctl: stale epoch")

	// ErrBudgetInversion reports a configuration in which a child budget is
	// not strictly smaller than its parent. Such a configuration cannot
	// honour the hierarchy, so it is rejected at construction/registration
	// time instead of producing unexplainable timeouts at runtime.
	ErrBudgetInversion = errors.New("runctl: child budget >= parent budget")

	// ErrInitializeBusy reports that SerializeInitialize could not acquire the
	// process-wide initialize guard within InitializeGuardBudget.
	ErrInitializeBusy = errors.New("isutools: initialize guard busy")

	// ErrCollectorRegistered rejects a duplicate collector name. Names index
	// snapshot sections, so collisions would silently merge two collectors.
	ErrCollectorRegistered = errors.New("runctl: collector already registered")

	// ErrInvalidRegistration reports a registration with no name or no
	// collector.
	ErrInvalidRegistration = errors.New("runctl: invalid registration")
)

Sentinel errors. Transport layers map these one-to-one onto HTTP status codes, so their identity is part of the package contract: wrap them with %w rather than replacing them, and never invent a new sentinel for a condition that is already covered here.

Functions

func HasInitializeGuard

func HasInitializeGuard(ctx context.Context) bool

HasInitializeGuard reports whether ctx came from inside SerializeInitialize. The public wrapper uses it to flag an initialize-triggered run that bypassed the guard: such a run can still succeed, but it may have been polluted by a concurrent rebuild, and silently trusting it is exactly the failure mode the guard exists to expose.

The check is context-based on purpose. Goroutine-local state would not survive the handler spawning its own goroutines, and would let the marker leak across unrelated requests.

func SerializeInitialize

func SerializeInitialize(ctx context.Context, fn func(context.Context) error) error

SerializeInitialize runs fn as the only initialize in this process.

Starting a run only serializes the instant the boundary is taken. If one initialize takes its boundary and a second then rebuilds the database, the first run is polluted by the second one's load; preemption makes that visible by invalidating the run, but it cannot prevent it. The only real fix is to serialize the whole initialize handler, which is what this guard is for.

The context passed to fn carries the guard marker, so a run started inside fn can be told apart from one started outside the guard.

func SerializeInitializeWithBudget

func SerializeInitializeWithBudget(ctx context.Context, budget time.Duration, fn func(context.Context) error) error

SerializeInitializeWithBudget is SerializeInitialize with an explicit acquisition timeout. It exists so tests can exercise the busy path without waiting out InitializeGuardBudget.

Types

type AbortResult

type AbortResult struct {
	RunID string
	Epoch Epoch
	// Reason is one of the stable Reason* values.
	Reason string
	// Detached reports that the worker outlived AbortJoinBudget. Correctness is
	// unaffected because the run is already fenced; only resource release is
	// delayed.
	Detached  bool
	AbortedAt time.Time
	// Partial lists collectors that had already switched when the run died.
	Partial []string
}

AbortResult is the immutable record of an abort.

type BaselineCollector

type BaselineCollector interface {
	// Name identifies the snapshot section this collector fills.
	Name() string

	// CaptureBaseline samples the opening boundary and returns an immutable
	// handle carrying the sampled values.
	CaptureBaseline(ctx context.Context, runID string, ep Epoch) (SampleResult, error)

	// CaptureFinal samples the closing boundary.
	CaptureFinal(ctx context.Context, runID string, ep Epoch) (SampleResult, error)

	// Collect derives the interval value from two frozen samples. The only
	// legal inputs are base.Sample() and final.Sample(); it must not touch the
	// collector's own fields, the database, or /proc. A type mismatch must be
	// returned as an error, never a panic — measurement may not break the
	// measured application.
	Collect(base, final BaselineHandle) (any, error)

	// Release frees whatever the handle pins. Idempotent.
	Release(h BaselineHandle)
}

BaselineCollector is a collector that measures a delta between two samples: process stats, table row counts, DB pool stats, network counters, host stats. Sampling does bounded I/O, so it gets a much larger per-collector budget than a generation swap and is executed in parallel to keep the boundary window narrow.

type BaselineHandle

type BaselineHandle struct {
	RunID     string
	Epoch     Epoch
	Collector string
	Phase     Phase
	SampledAt time.Time
	// contains filtered or unexported fields
}

BaselineHandle is an immutable sample taken at a boundary. The sample is carried inside the handle rather than left in the collector, because Collect(base, final) must be able to build an interval from fixed values alone — reading the collector's live state at snapshot time is exactly the bug this design removes.

func NewBaselineHandle

func NewBaselineHandle(runID string, ep Epoch, collector string, phase Phase, sampledAt time.Time, sample any) BaselineHandle

NewBaselineHandle builds a handle around an already-copied sample. The caller must never mutate the value it passes in afterwards: handles are copied and shared, so a later mutation would be observed by every holder and would silently change an interval that was supposed to be frozen. Prefer a value type over a pointer for exactly that reason.

func (BaselineHandle) Sample

func (h BaselineHandle) Sample() any

Sample returns the frozen sample this handle carries. It is the only official way for a BaselineCollector to reach the values it needs inside Collect(base, final); reaching into the collector's own fields instead would violate the "fixed values only" contract even when it happens to work.

The returned value must be treated as read-only.

func (BaselineHandle) Zero

func (h BaselineHandle) Zero() bool

Zero reports whether the handle carries no sample.

type BoundaryResult

type BoundaryResult struct {
	Handle GenerationHandle
	// At is the measured moment of the swap or freeze.
	At time.Time
	// Committed states whether the switch is in effect for this run ID. It is
	// a state predicate, not a "did this call do it" flag, so a retry of the
	// same (runID, epoch) returns the same value.
	Committed bool
}

BoundaryResult is what a generation collector returns from a boundary operation. It is returned even on error, never as a zero value, because the Controller must know whether the switch took effect before it decides whether the collector's data can still be used.

type BoundaryWindow

type BoundaryWindow struct {
	Min    time.Time     `json:"min"`
	Max    time.Time     `json:"max"`
	Spread time.Duration `json:"spread"`
}

BoundaryWindow is the measured width of a boundary. Boundaries are intervals rather than instants, so the width is recorded and judged instead of assumed away.

type BudgetAware

type BudgetAware interface {
	Budget() time.Duration
}

BudgetAware is the optional interface a collector implements to declare the per-operation budget it needs. Registration rejects a collector asking for more than its per-collector budget allows, so the mismatch is reported where it can be fixed instead of showing up as a truncated measurement.

type Budgets

type Budgets struct {
	StartRun      time.Duration
	FinishSync    time.Duration
	PhaseBoundary time.Duration // BeginBoundary phase (start side)
	PhaseBaseline time.Duration // CaptureBaseline phase (start side)
	PhaseFreeze   time.Duration // Freeze phase (finish side)
	PhaseFinal    time.Duration // CaptureFinal phase (finish side)

	PerCollectorGeneration time.Duration
	PerCollectorBaseline   time.Duration

	Drain         time.Duration
	SnapshotBuild time.Duration
	Enrich        time.Duration
	AbortJoin     time.Duration
	DetachedReap  time.Duration

	FinishLease  time.Duration
	StartedTTL   time.Duration
	FinishedTTL  time.Duration
	TombstoneTTL time.Duration
	NonceTTL     time.Duration
	Watchdog     time.Duration

	SpreadGeneration time.Duration
	SpreadBoundary   time.Duration
}

Budgets is the injectable form of the constants above. A zero field falls back to its package constant, so callers override only what they care about and tests can shrink the whole table without waiting out real leases.

func (Budgets) Validate

func (b Budgets) Validate() error

Validate enforces the budget hierarchy on an already-defaulted table. Violations are rejected at construction time because an inverted budget produces timeouts that are impossible to explain from a snapshot alone.

type CollectorBoundary

type CollectorBoundary struct {
	Name      string    `json:"name"`
	Kind      string    `json:"kind"`
	Required  bool      `json:"required"`
	Phase     Phase     `json:"phase"`
	At        time.Time `json:"at"`
	Committed bool      `json:"committed"`
	// Code is a stable machine-readable code; empty means success.
	Code string `json:"code,omitempty"`
	// Err is the human-readable original message.
	Err string `json:"err,omitempty"`
	// Dropped marks the section as excluded from the snapshot.
	Dropped bool `json:"dropped,omitempty"`
}

CollectorBoundary is one collector's record of one boundary step.

type Controller

type Controller struct {
	// contains filtered or unexported fields
}

Controller owns the run lifecycle for one process.

func Default

func Default() *Controller

Default returns the process-wide Controller. The lifecycle is a property of the process being measured, so there is exactly one of these.

func New

func New(o Options) (*Controller, error)

New builds a Controller. It fails only on an inverted budget table, which is a configuration bug worth refusing at startup rather than debugging later from truncated measurements.

func (*Controller) AbortRun

func (c *Controller) AbortRun(ctx context.Context, runID, reason string) (AbortResult, error)

AbortRun abandons a run. The ordering below is the specification, not an implementation detail:

  1. under the mutex, move to aborting and advance the Controller epoch, which fences every worker still belonging to this run;
  2. release the mutex, then cancel the run's context — waiting for a collector while holding the mutex would deadlock a reset called from inside an instrumented handler;
  3. join the worker, bounded by the abort join budget;
  4. release the handles, or leave that to the detached worker's own cleanup;
  5. settle in aborted so the next run can start.

Step 1 is what makes step 3's timeout harmless: a detached worker's commit and publish are already rejected, and its handles only reach the generation it was given, so it can neither publish this run's data nor corrupt the next run's. A join timeout costs delayed cleanup and nothing else.

Aborting a run the Controller does not know is a successful no-op. Stopping something that is already stopped is always satisfied, and callers retrying an abort after a lost response must not be told the world is inconsistent.

func (*Controller) Ack

func (c *Controller) Ack(runID string) error

Ack marks a finished run's snapshot as handed over.

func (*Controller) AckBy

func (c *Controller) AckBy(runID, by string) error

AckBy is Ack with an explicit provenance. The provenance matters because a self-acknowledgement driven by an expired lease is not the same evidence as an operator collecting a result, and multi-host debugging depends on telling them apart.

func (*Controller) Await

func (c *Controller) Await(ctx context.Context, runID string) (RunStatus, error)

Await blocks until the run reaches a state that answers the caller's question: a starting run until it has started, a finishing run until its snapshot exists or the run died.

func (*Controller) Close

func (c *Controller) Close()

Close stops the watchdog. It is idempotent.

func (*Controller) FinishRun

func (c *Controller) FinishRun(ctx context.Context, runID string) (FinishAccepted, error)

FinishRun fixes a run's closing boundary and returns as soon as that boundary exists. Draining, collecting and snapshot building continue in the background, because the caller is usually a benchmark driver that must be released the instant measurement has stopped — making it wait for the snapshot would put snapshot-building time inside the measured window of whatever runs next.

func (*Controller) PublishedSnapshots

func (c *Controller) PublishedSnapshots() uint64

PublishedSnapshots reports how many snapshots were accepted.

func (*Controller) RegisterBaseline

func (c *Controller) RegisterBaseline(r Registration, b BaselineCollector) error

RegisterBaseline adds a baseline collector.

func (*Controller) RegisterGeneration

func (c *Controller) RegisterGeneration(r Registration, g GenerationCollector) error

RegisterGeneration adds a generation collector.

func (*Controller) SnapshotOf

func (c *Controller) SnapshotOf(runID string) (*Snapshot, error)

SnapshotOf returns a finished run's immutable snapshot.

func (*Controller) StaleRejections

func (c *Controller) StaleRejections() uint64

StaleRejections reports how many commit and publish attempts the epoch fence turned away. Tests assert it is non-zero to prove the fence actually ran rather than the race merely not happening.

func (*Controller) StartRun

func (c *Controller) StartRun(ctx context.Context, o StartRunOptions) (StartResult, error)

StartRun opens a measurement run: it switches every generation collector to a fresh generation and samples every baseline collector, then returns an immutable record of that boundary.

A collector failure is not an error return. It downgrades StartResult Validity instead, because the boundary itself did happen and the caller needs the record. Callers must inspect Validity. An error return means the Controller refused (ErrRunActive) or could not act at all.

func (*Controller) Status

func (c *Controller) Status(runID string) (RunStatus, bool)

Status returns a run's current state. The bool is false for runs the Controller does not retain.

func (*Controller) Sweep

func (c *Controller) Sweep()

Sweep applies lease and TTL expiry once. It is exported so tests can drive expiry from an injected clock instead of waiting out a twenty second lease.

type Epoch

type Epoch uint64

Epoch is the Controller's monotonic fencing token. It advances on every successful StartRun and on every AbortRun, which is what makes a worker belonging to an abandoned run structurally unable to publish.

type FinishAccepted

type FinishAccepted struct {
	RunID            string
	Epoch            Epoch
	Validity         Validity
	Collectors       []CollectorBoundary
	GenerationWindow BoundaryWindow
	BoundaryWindow   BoundaryWindow
	AcceptedAt       time.Time
}

FinishAccepted is the immutable record of a closing boundary. It is returned as soon as the boundary is fixed; draining and snapshot building continue in the background.

type GenerationCollector

type GenerationCollector interface {
	// Name identifies the snapshot section this collector fills.
	Name() string

	// BeginBoundary swaps in a fresh generation and returns a handle to the
	// generation it just closed. Fast and non-blocking.
	BeginBoundary(ctx context.Context, runID string, ep Epoch) (BoundaryResult, error)

	// Freeze seals the current generation and returns its handle. Observations
	// made after Freeze belong to the next generation, outside the run.
	Freeze(ctx context.Context, runID string, ep Epoch) (BoundaryResult, error)

	// Drain settles in-flight work pinned to the handle's generation only. It
	// must return within DrainCancelGrace of ctx being done and must leave no
	// goroutine that will later modify that generation.
	//
	// Implementations must wait on a per-generation done channel. sync.Cond
	// cannot be interrupted by a context, so a cond-based wait makes this
	// contract impossible to honour when a request never returns.
	Drain(ctx context.Context, h GenerationHandle) error

	// Collect reads the drained generation's fixed data. It must not read the
	// collector's mutable current state.
	Collect(h GenerationHandle) (any, error)

	// Release frees whatever the handle pins. Idempotent: a second Release,
	// or a Release racing the owner's own cleanup, is a no-op.
	Release(h GenerationHandle)
}

GenerationCollector is a collector that accumulates into a swappable generation: HTTP stats, SQL stats, access log offsets, counters. Boundaries are pointer swaps, so they are expected to be non-blocking and to finish inside PerCollectorGenerationBudget.

Every method takes or returns a handle rather than exposing the collector's current state, so that a run's data is fixed the instant its boundary is taken and cannot drift while the snapshot is being built.

type GenerationHandle

type GenerationHandle struct {
	RunID     string
	Epoch     Epoch
	Collector string
	Gen       uint64
	// contains filtered or unexported fields
}

GenerationHandle is an immutable reference to a closed or frozen generation. Holding one gives access to fixed data only; the collector's mutable current generation is never reachable through it.

func NewGenerationHandle

func NewGenerationHandle(runID string, ep Epoch, collector string, gen uint64, token any) GenerationHandle

NewGenerationHandle builds a handle. Collectors live in other packages, so this constructor is the only way to populate the unexported token; that keeps the token opaque to everyone except the collector that created it.

func (GenerationHandle) Token

func (h GenerationHandle) Token() any

Token returns the collector-internal generation reference. Only the collector that created the handle may interpret it.

func (GenerationHandle) Zero

func (h GenerationHandle) Zero() bool

Zero reports whether the handle refers to nothing.

type HealthRecorder

type HealthRecorder interface {
	Set(collector string, status health.Status, message string)
}

HealthRecorder is the sink for the four runctl-* health keys. It is the narrow subset of internal/health.Registry this package needs, so a Controller can be built in tests without a registry and so measurement never depends on health reporting succeeding.

type Options

type Options struct {
	// Budgets overrides the timing table. Zero fields keep their constants.
	Budgets Budgets
	// Now overrides the clock. Tests inject a controllable clock so leases and
	// TTLs can be exercised without waiting them out.
	Now func() time.Time
	// Health receives the runctl-* keys. Nil disables health reporting.
	Health HealthRecorder
	// Enrich runs after Collect and before the snapshot is published, bounded
	// by the enrich budget. It is how post-freeze extras such as EXPLAIN
	// output are attached without widening the freeze boundary.
	Enrich func(ctx context.Context, s *Snapshot) error
	// DisableWatchdog stops the Controller from starting its lease/TTL sweeper
	// goroutine. Tests that drive Sweep by hand set this.
	DisableWatchdog bool
}

Options configures a Controller. The zero value is usable: every field falls back to a package default.

type Phase

type Phase string

Phase names one step of a boundary. It appears in CollectorBoundary so a failure can be attributed to the exact step that produced it.

const (
	// PhaseStartBoundary is BeginBoundary on generation collectors.
	PhaseStartBoundary Phase = "start-boundary"
	// PhaseStartBaseline is CaptureBaseline on baseline collectors.
	PhaseStartBaseline Phase = "start-baseline"
	// PhaseFinishFreeze is Freeze on generation collectors.
	PhaseFinishFreeze Phase = "finish-freeze"
	// PhaseFinishFinal is CaptureFinal on baseline collectors.
	PhaseFinishFinal Phase = "finish-final"
	// PhaseCollect is the background Drain then Collect step.
	PhaseCollect Phase = "collect"
)

type Registration

type Registration struct {
	// Name identifies the snapshot section this collector fills.
	Name string
	// Required marks a collector whose failure invalidates the run rather than
	// merely degrading it.
	Required bool
	// SerialOnly excludes the collector from the parallel baseline group. It
	// widens the boundary window, so it is only for collectors that are
	// genuinely unsafe to sample concurrently.
	SerialOnly bool
}

Registration describes how a collector participates in runs.

type RunState

type RunState string

RunState is a run's position in the lifecycle state machine. Transport layers copy these strings onto the wire verbatim, so the values are part of the contract.

const (
	// StateIdle means no run exists. It is never stored on a run record; it
	// describes the Controller when nothing is retained.
	StateIdle RunState = "idle"
	// StateStarting means the opening boundary is being taken. Owner: the
	// StartRun caller.
	StateStarting RunState = "starting"
	// StateStarted means measurement is in progress. No owner goroutine once
	// the previous generations have been drained.
	StateStarted RunState = "started"
	// StateFinishing means the closing boundary is fixed and a background
	// worker is draining, collecting and building the snapshot.
	StateFinishing RunState = "finishing"
	// StateFinished means an immutable snapshot exists.
	StateFinished RunState = "finished"
	// StateAcknowledged means the snapshot was handed over. Terminal.
	StateAcknowledged RunState = "acknowledged"
	// StateAborting means the run is fenced and its worker is being joined.
	StateAborting RunState = "aborting"
	// StateAborted means the run was abandoned and holds no snapshot. Terminal.
	StateAborted RunState = "aborted"
	// StateExpired means the snapshot was released by TTL. Terminal tombstone.
	StateExpired RunState = "expired"
)

type RunStatus

type RunStatus struct {
	RunID    string   `json:"run_id"`
	Epoch    Epoch    `json:"epoch"`
	State    RunState `json:"state"`
	Validity Validity `json:"validity"`
	Reason   string   `json:"reason,omitempty"`
	// AckedBy is one of the stable AckedBy* values.
	AckedBy  string    `json:"acked_by,omitempty"`
	Detached bool      `json:"detached,omitempty"`
	Since    time.Time `json:"since"`
}

RunStatus is the queryable state of a run.

type SampleResult

type SampleResult struct {
	Handle BaselineHandle
	// At is the measured moment of sampling and equals Handle.SampledAt.
	At time.Time
	// Committed states whether the sample is fixed for this run ID.
	Committed bool
}

SampleResult is what a baseline collector returns from a sampling operation. Like BoundaryResult it is returned even on error.

type Snapshot

type Snapshot struct {
	RunID            string
	Epoch            Epoch
	Validity         Validity
	Trigger          string
	Sections         map[string]any
	Collectors       []CollectorBoundary
	GenerationWindow BoundaryWindow
	BoundaryWindow   BoundaryWindow
	StartedAt        time.Time
	FinishedAt       time.Time
}

Snapshot is a run's immutable result. Sections holds each collector's interval value keyed by collector name; the concrete types belong to the collectors, so the transport layer decides how to serialize them.

type StartResult

type StartResult struct {
	RunID            string
	Nonce            string
	Epoch            Epoch
	State            RunState
	Validity         Validity
	Collectors       []CollectorBoundary
	GenerationWindow BoundaryWindow
	BoundaryWindow   BoundaryWindow
	// PreemptedRunID names the run this one displaced, if any.
	PreemptedRunID string
	StartedAt      time.Time
}

StartResult is the immutable record of an opening boundary. err == nil with a degraded Validity is the normal way collector failures are reported; callers must inspect Validity rather than only err.

type StartRunOptions

type StartRunOptions struct {
	// Nonce makes the call idempotent. Empty means the Controller mints one.
	Nonce string
	// Preempt aborts an in-flight run instead of failing with ErrRunActive.
	// This is how "the last initialize wins" is made deterministic.
	Preempt bool
	// Reason records who asked: "api", "initialize", "http", "hub".
	Reason string
	// Trigger is recorded on the run as its reset trigger.
	Trigger string
}

StartRunOptions configures StartRun.

type Validity

type Validity string

Validity is the data-quality axis, orthogonal to RunState. A run can be perfectly "finished" and completely untrustworthy; callers that compare runs must filter on this, not on the state.

const (
	// ValidityValid means every registered collector contributed a complete
	// interval within its boundary window.
	ValidityValid Validity = "valid"
	// ValidityPartial means some optional sections are missing but the
	// interval itself is usable.
	ValidityPartial Validity = "partial"
	// ValidityInvalid means the interval cannot be trusted. Advisors and diffs
	// must exclude it.
	ValidityInvalid Validity = "invalid"
)

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL