Documentation
¶
Overview ¶
Package runctl owns the measurement run lifecycle: a single process-wide Controller decides when a run starts, when its boundaries are frozen, when its immutable snapshot may be published, and when it is aborted.
The package exists because measurement boundaries are the one thing an ISUCON toolkit cannot get "mostly right". A run whose opening moments are missing, whose closing moments leak into the next run, or whose data is published by a worker belonging to an already-abandoned run is worse than no measurement at all: it looks authoritative while being wrong. Everything here is therefore built around two invariants.
Epoch fencing. Every run owns an Epoch. The Controller's current epoch advances on every successful StartRun and on every AbortRun. Background workers never touch the Controller state directly; they go through [Controller.commit] and [Controller.publish], which reject any epoch that is no longer current with ErrStaleEpoch. A worker belonging to an aborted run is therefore structurally incapable of publishing data or of dragging the run back to "finished", even when the abort could not join it in time.
Fail-open measurement. Collector failures never surface as Go errors from StartRun/FinishRun; they downgrade the run's Validity instead. An error return means the Controller itself refused or could not perform the operation. Callers must always inspect Validity, never only err.
Boundaries are treated as intervals, not instants: generation collectors are switched sequentially and baseline collectors are sampled in parallel, and the measured width of both is recorded in a BoundaryWindow so that a run whose boundary was too smeared can be marked partial or invalid rather than silently trusted.
The Controller never blocks on collectors or workers while holding its mutex, because ResetNow is expected to be callable from inside an instrumented HTTP handler.
Index ¶
- Constants
- Variables
- func HasInitializeGuard(ctx context.Context) bool
- func SerializeInitialize(ctx context.Context, fn func(context.Context) error) error
- func SerializeInitializeWithBudget(ctx context.Context, budget time.Duration, fn func(context.Context) error) error
- type AbortResult
- type BaselineCollector
- type BaselineHandle
- type BoundaryResult
- type BoundaryWindow
- type BudgetAware
- type Budgets
- type CollectorBoundary
- type Controller
- func (c *Controller) AbortRun(ctx context.Context, runID, reason string) (AbortResult, error)
- func (c *Controller) Ack(runID string) error
- func (c *Controller) AckBy(runID, by string) error
- func (c *Controller) Await(ctx context.Context, runID string) (RunStatus, error)
- func (c *Controller) Close()
- func (c *Controller) FinishRun(ctx context.Context, runID string) (FinishAccepted, error)
- func (c *Controller) PublishedSnapshots() uint64
- func (c *Controller) RegisterBaseline(r Registration, b BaselineCollector) error
- func (c *Controller) RegisterGeneration(r Registration, g GenerationCollector) error
- func (c *Controller) SnapshotOf(runID string) (*Snapshot, error)
- func (c *Controller) StaleRejections() uint64
- func (c *Controller) StartRun(ctx context.Context, o StartRunOptions) (StartResult, error)
- func (c *Controller) Status(runID string) (RunStatus, bool)
- func (c *Controller) Sweep()
- type Epoch
- type FinishAccepted
- type GenerationCollector
- type GenerationHandle
- type HealthRecorder
- type Options
- type Phase
- type Registration
- type RunState
- type RunStatus
- type SampleResult
- type Snapshot
- type StartResult
- type StartRunOptions
- type Validity
Constants ¶
const ( // StartRunBudget bounds the whole synchronous part of StartRun. StartRunBudget = 6 * time.Second // FinishSyncBudget bounds the whole synchronous part of FinishRun // (freeze + final sampling). Draining and snapshot building happen after // the response, outside this budget. FinishSyncBudget = 6 * time.Second // PhaseStartBoundaryBudget bounds BeginBoundary across all generation // collectors. Generation swaps are pointer swaps, so this is generous. PhaseStartBoundaryBudget = 500 * time.Millisecond // PhaseStartBaselineBudget bounds CaptureBaseline across all baseline // collectors. Baseline sampling does bounded I/O (procfs, DB), hence the // order-of-magnitude difference from the boundary phase. PhaseStartBaselineBudget = 5 * time.Second // PhaseFinishFreezeBudget bounds Freeze across all generation collectors. PhaseFinishFreezeBudget = 500 * time.Millisecond // PhaseFinishFinalBudget bounds CaptureFinal across all baseline collectors. PhaseFinishFinalBudget = 5 * time.Second // PerCollectorGenerationBudget bounds one boundary operation of one // generation collector. PerCollectorGenerationBudget = 100 * time.Millisecond // PerCollectorBaselineBudget bounds one sampling operation of one baseline // collector, including all of the DB targets it fans out to. PerCollectorBaselineBudget = 3500 * time.Millisecond // PerTargetBudget bounds one collector's call against one DB target. PerTargetBudget = 1 * time.Second // DrainBudget bounds draining every handle of one run. Background work. DrainBudget = 10 * time.Second // DrainCancelGrace is how long a collector's Drain may take to return // after its context is done. Collectors must wait on a per-generation done // channel rather than sync.Cond, because sync.Cond.Wait cannot be // interrupted by a context and would make this contract unimplementable. DrainCancelGrace = 1 * time.Second // SnapshotBuildBudget bounds Collect plus immutable snapshot construction. SnapshotBuildBudget = 5 * time.Second // EnrichBudget bounds post-freeze enrichment such as EXPLAIN capture. EnrichBudget = 2 * time.Second // AbortJoinBudget bounds how long AbortRun waits for the run's worker. // Exceeding it is safe but noisy: the worker is detached and fenced. AbortJoinBudget = 2 * time.Second // PreemptTotalBudget bounds the whole preempt path. PreemptTotalBudget = AbortJoinBudget + StartRunBudget // DetachedReapBudget bounds how long the reaper watches a detached worker // before giving up on reclaiming its handles. DetachedReapBudget = 60 * time.Second // InitializeGuardBudget bounds how long SerializeInitialize waits for the // process-wide initialize guard. InitializeGuardBudget = 30 * time.Second )
Hierarchical time budgets. This package is the single authority for these numbers: downstream collectors cite them instead of inventing their own, so that "why did my collector get cut off?" always has one answer.
The hierarchy is strict — a run budget bounds a phase budget, which bounds a per-collector budget, which bounds a per-target budget. TestBudgetHierarchy pins the inequalities so a future tweak cannot quietly invert them.
const ( // FinishLease is how long one phase of a finish may live before the // watchdog force-aborts the run. It is armed twice: FinishRun arms it over // the synchronous freeze (hence FinishSyncBudget < FinishLease), and the // background worker re-arms it when it takes over (hence // DrainBudget+SnapshotBuildBudget+EnrichBudget < FinishLease). Both halves // are therefore bounded by it, and neither is charged for the other's time. FinishLease = 20 * time.Second // StartedTTL reclaims runs that nobody ever finished. StartedTTL = 30 * time.Minute // FinishedTTL is how long a finished or acknowledged snapshot is retained. FinishedTTL = 10 * time.Minute // TombstoneTTL is how long aborted and expired records are retained. TombstoneTTL = 10 * time.Minute // NonceTTL bounds the StartRun idempotency cache. NonceTTL = 10 * time.Minute // WatchdogInterval is how often leases and TTLs are examined. WatchdogInterval = 1 * time.Second )
Lease and TTL defaults. All of them are injectable through Budgets so tests never wait out a real 30 minute TTL.
const ( // BaselineConcurrency caps parallel baseline sampling. Parallelism is what // keeps the boundary window narrow; the cap keeps the measured application // from being hit by a thundering herd of collectors. BaselineConcurrency = 8 // NonceHistoryMax bounds the nonce idempotency cache. NonceHistoryMax = 64 // RetainedRuns is how many run records the Controller keeps, *including* // the in-flight one. Two generations back is therefore always a 404. RetainedRuns = 2 )
Structural limits.
const ( // SpreadLimitGeneration caps the measured width of the generation swap // window. Expected in practice: well under a millisecond. SpreadLimitGeneration = 50 * time.Millisecond // SpreadLimitBoundary caps the measured width of the whole boundary // window, generation and baseline collectors together. Expected in // practice: under 200ms with parallel sampling. SpreadLimitBoundary = 1500 * time.Millisecond )
Boundary spread limits. These are deliberately unrelated to the budgets above: a budget is a forced cutoff that protects the application, a spread limit is a quality threshold that decides whether the boundary is usable as a boundary. "Inside budget but over spread" is a normal, expected verdict.
const ( KindGeneration = "generation" KindBaseline = "baseline" )
Collector kinds as they appear in CollectorBoundary.Kind.
const ( // CodeNotCaptured means the collector was never sampled because the phase // budget ran out. CodeNotCaptured = "not-captured" // CodeDrainTimeout means in-flight work did not settle within DrainBudget. // Partial data still exists and is kept. CodeDrainTimeout = "drain-timeout" // CodeCollectFailed means the interval value could not be derived. CodeCollectFailed = "collect-failed" // CodeBoundaryFailed means the boundary operation itself returned an error. CodeBoundaryFailed = "boundary-failed" // CodeSpreadExceeded marks a collector whose measured time sits outside the // allowed boundary window. CodeSpreadExceeded = "spread-exceeded" // CodeContractViolation means the collector reported success without // committing, which is a collector bug rather than a runtime condition. CodeContractViolation = "contract-violation" )
Machine-readable failure codes. An empty code means the step succeeded. The set is closed: transports and advisors switch on these values.
const ( // AckedByExplicit is a direct Ack call. AckedByExplicit = "explicit" // AckedBySave is the implicit acknowledgement performed by POST /save. AckedBySave = "save" // AckedByPreempt is the implicit acknowledgement of a finished run whose // successor started with Preempt. The snapshot is retained. AckedByPreempt = "preempt" // AckedByHub is an acknowledgement driven by a multi-host hub. AckedByHub = "hub" // AckedByLease is a self-acknowledgement after a peer ack lease expired, so // a vanished hub cannot wedge the system in "finished". AckedByLease = "lease" )
Stable AckedBy values. This package owns the set; multi-host transports copy the field one-to-one and must not invent values.
const ( // ReasonExplicit is an operator- or API-driven abort. ReasonExplicit = "explicit" // ReasonRequiredFailed is a required collector failing the opening boundary. ReasonRequiredFailed = "required-failed" // ReasonPreemptedBy is the prefix for "preempted-by:<runID>". ReasonPreemptedBy = "preempted-by:" // ReasonFinishLeaseExpired is the watchdog reclaiming a stuck worker. ReasonFinishLeaseExpired = "finish-lease-expired" // ReasonStartedTTL is the watchdog reclaiming a run nobody finished. ReasonStartedTTL = "started-ttl" // ReasonHubAbort is a multi-host hub aborting a peer's run. ReasonHubAbort = "hub-abort" )
Stable AbortResult.Reason values.
const ( // HealthBoundarySpread reports a boundary window wider than its limit. HealthBoundarySpread = "runctl-boundary-spread" // HealthContractViolation reports a collector returning success without // committing its boundary. HealthContractViolation = "runctl-contract-violation" // HealthWorkerDetached reports an abort that could not join its worker. HealthWorkerDetached = "runctl-worker-detached" // HealthLeaseExpired reports a finishing worker killed by FinishLease. HealthLeaseExpired = "runctl-lease-expired" )
Health keys this package reports. The set is fixed at four so that a health snapshot stays readable; new conditions reuse these keys with a different message rather than adding keys.
Variables ¶
var ( // ErrRunActive is returned when another run occupies the Controller and // the caller did not ask to preempt it. Callers that must win (an // initialize handler, for example) retry with StartRunOptions.Preempt. // Maps to HTTP 409. ErrRunActive = errors.New("runctl: another run is active") // ErrRunTransitioning is returned while a run is mid-transition // (starting or aborting) and the requested operation cannot be ordered // against it. Retrying after the transition settles is safe. // Maps to HTTP 409. ErrRunTransitioning = errors.New("runctl: run is transitioning") // ErrRunAborted is returned for operations on a run that was abandoned. // The run's data is gone for good, so this is deliberately distinct from // ErrUnknownRun: the caller learns the run existed and failed rather than // that it never existed. Maps to HTTP 410. ErrRunAborted = errors.New("runctl: run was aborted") // ErrUnknownRun is returned for a run the Controller does not retain: // never started, evicted by RetainedRuns, or expired. Maps to HTTP 404. ErrUnknownRun = errors.New("runctl: unknown run") // ErrStaleEpoch rejects a state change or a snapshot publication coming // from a worker whose run is no longer current. It is an internal fence // signal and is not expected to reach an API caller. ErrStaleEpoch = errors.New("runctl: stale epoch") // ErrBudgetInversion reports a configuration in which a child budget is // not strictly smaller than its parent. Such a configuration cannot // honour the hierarchy, so it is rejected at construction/registration // time instead of producing unexplainable timeouts at runtime. ErrBudgetInversion = errors.New("runctl: child budget >= parent budget") // ErrInitializeBusy reports that SerializeInitialize could not acquire the // process-wide initialize guard within InitializeGuardBudget. ErrInitializeBusy = errors.New("isutools: initialize guard busy") // ErrCollectorRegistered rejects a duplicate collector name. Names index // snapshot sections, so collisions would silently merge two collectors. ErrCollectorRegistered = errors.New("runctl: collector already registered") // ErrInvalidRegistration reports a registration with no name or no // collector. ErrInvalidRegistration = errors.New("runctl: invalid registration") )
Sentinel errors. Transport layers map these one-to-one onto HTTP status codes, so their identity is part of the package contract: wrap them with %w rather than replacing them, and never invent a new sentinel for a condition that is already covered here.
Functions ¶
func HasInitializeGuard ¶
HasInitializeGuard reports whether ctx came from inside SerializeInitialize. The public wrapper uses it to flag an initialize-triggered run that bypassed the guard: such a run can still succeed, but it may have been polluted by a concurrent rebuild, and silently trusting it is exactly the failure mode the guard exists to expose.
The check is context-based on purpose. Goroutine-local state would not survive the handler spawning its own goroutines, and would let the marker leak across unrelated requests.
func SerializeInitialize ¶
SerializeInitialize runs fn as the only initialize in this process.
Starting a run only serializes the instant the boundary is taken. If one initialize takes its boundary and a second then rebuilds the database, the first run is polluted by the second one's load; preemption makes that visible by invalidating the run, but it cannot prevent it. The only real fix is to serialize the whole initialize handler, which is what this guard is for.
The context passed to fn carries the guard marker, so a run started inside fn can be told apart from one started outside the guard.
func SerializeInitializeWithBudget ¶
func SerializeInitializeWithBudget(ctx context.Context, budget time.Duration, fn func(context.Context) error) error
SerializeInitializeWithBudget is SerializeInitialize with an explicit acquisition timeout. It exists so tests can exercise the busy path without waiting out InitializeGuardBudget.
Types ¶
type AbortResult ¶
type AbortResult struct {
RunID string
Epoch Epoch
// Reason is one of the stable Reason* values.
Reason string
// Detached reports that the worker outlived AbortJoinBudget. Correctness is
// unaffected because the run is already fenced; only resource release is
// delayed.
Detached bool
AbortedAt time.Time
// Partial lists collectors that had already switched when the run died.
Partial []string
}
AbortResult is the immutable record of an abort.
type BaselineCollector ¶
type BaselineCollector interface {
// Name identifies the snapshot section this collector fills.
Name() string
// CaptureBaseline samples the opening boundary and returns an immutable
// handle carrying the sampled values.
CaptureBaseline(ctx context.Context, runID string, ep Epoch) (SampleResult, error)
// CaptureFinal samples the closing boundary.
CaptureFinal(ctx context.Context, runID string, ep Epoch) (SampleResult, error)
// Collect derives the interval value from two frozen samples. The only
// legal inputs are base.Sample() and final.Sample(); it must not touch the
// collector's own fields, the database, or /proc. A type mismatch must be
// returned as an error, never a panic — measurement may not break the
// measured application.
Collect(base, final BaselineHandle) (any, error)
// Release frees whatever the handle pins. Idempotent.
Release(h BaselineHandle)
}
BaselineCollector is a collector that measures a delta between two samples: process stats, table row counts, DB pool stats, network counters, host stats. Sampling does bounded I/O, so it gets a much larger per-collector budget than a generation swap and is executed in parallel to keep the boundary window narrow.
type BaselineHandle ¶
type BaselineHandle struct {
RunID string
Epoch Epoch
Collector string
Phase Phase
SampledAt time.Time
// contains filtered or unexported fields
}
BaselineHandle is an immutable sample taken at a boundary. The sample is carried inside the handle rather than left in the collector, because Collect(base, final) must be able to build an interval from fixed values alone — reading the collector's live state at snapshot time is exactly the bug this design removes.
func NewBaselineHandle ¶
func NewBaselineHandle(runID string, ep Epoch, collector string, phase Phase, sampledAt time.Time, sample any) BaselineHandle
NewBaselineHandle builds a handle around an already-copied sample. The caller must never mutate the value it passes in afterwards: handles are copied and shared, so a later mutation would be observed by every holder and would silently change an interval that was supposed to be frozen. Prefer a value type over a pointer for exactly that reason.
func (BaselineHandle) Sample ¶
func (h BaselineHandle) Sample() any
Sample returns the frozen sample this handle carries. It is the only official way for a BaselineCollector to reach the values it needs inside Collect(base, final); reaching into the collector's own fields instead would violate the "fixed values only" contract even when it happens to work.
The returned value must be treated as read-only.
func (BaselineHandle) Zero ¶
func (h BaselineHandle) Zero() bool
Zero reports whether the handle carries no sample.
type BoundaryResult ¶
type BoundaryResult struct {
Handle GenerationHandle
// At is the measured moment of the swap or freeze.
At time.Time
// Committed states whether the switch is in effect for this run ID. It is
// a state predicate, not a "did this call do it" flag, so a retry of the
// same (runID, epoch) returns the same value.
Committed bool
}
BoundaryResult is what a generation collector returns from a boundary operation. It is returned even on error, never as a zero value, because the Controller must know whether the switch took effect before it decides whether the collector's data can still be used.
type BoundaryWindow ¶
type BoundaryWindow struct {
Min time.Time `json:"min"`
Max time.Time `json:"max"`
Spread time.Duration `json:"spread"`
}
BoundaryWindow is the measured width of a boundary. Boundaries are intervals rather than instants, so the width is recorded and judged instead of assumed away.
type BudgetAware ¶
BudgetAware is the optional interface a collector implements to declare the per-operation budget it needs. Registration rejects a collector asking for more than its per-collector budget allows, so the mismatch is reported where it can be fixed instead of showing up as a truncated measurement.
type Budgets ¶
type Budgets struct {
StartRun time.Duration
FinishSync time.Duration
PhaseBoundary time.Duration // BeginBoundary phase (start side)
PhaseBaseline time.Duration // CaptureBaseline phase (start side)
PhaseFreeze time.Duration // Freeze phase (finish side)
PhaseFinal time.Duration // CaptureFinal phase (finish side)
PerCollectorGeneration time.Duration
PerCollectorBaseline time.Duration
Drain time.Duration
SnapshotBuild time.Duration
Enrich time.Duration
AbortJoin time.Duration
DetachedReap time.Duration
FinishLease time.Duration
StartedTTL time.Duration
FinishedTTL time.Duration
TombstoneTTL time.Duration
NonceTTL time.Duration
Watchdog time.Duration
SpreadGeneration time.Duration
SpreadBoundary time.Duration
}
Budgets is the injectable form of the constants above. A zero field falls back to its package constant, so callers override only what they care about and tests can shrink the whole table without waiting out real leases.
type CollectorBoundary ¶
type CollectorBoundary struct {
Name string `json:"name"`
Kind string `json:"kind"`
Required bool `json:"required"`
Phase Phase `json:"phase"`
At time.Time `json:"at"`
Committed bool `json:"committed"`
// Code is a stable machine-readable code; empty means success.
Code string `json:"code,omitempty"`
// Err is the human-readable original message.
Err string `json:"err,omitempty"`
// Dropped marks the section as excluded from the snapshot.
Dropped bool `json:"dropped,omitempty"`
}
CollectorBoundary is one collector's record of one boundary step.
type Controller ¶
type Controller struct {
// contains filtered or unexported fields
}
Controller owns the run lifecycle for one process.
func Default ¶
func Default() *Controller
Default returns the process-wide Controller. The lifecycle is a property of the process being measured, so there is exactly one of these.
func New ¶
func New(o Options) (*Controller, error)
New builds a Controller. It fails only on an inverted budget table, which is a configuration bug worth refusing at startup rather than debugging later from truncated measurements.
func (*Controller) AbortRun ¶
func (c *Controller) AbortRun(ctx context.Context, runID, reason string) (AbortResult, error)
AbortRun abandons a run. The ordering below is the specification, not an implementation detail:
- under the mutex, move to aborting and advance the Controller epoch, which fences every worker still belonging to this run;
- release the mutex, then cancel the run's context — waiting for a collector while holding the mutex would deadlock a reset called from inside an instrumented handler;
- join the worker, bounded by the abort join budget;
- release the handles, or leave that to the detached worker's own cleanup;
- settle in aborted so the next run can start.
Step 1 is what makes step 3's timeout harmless: a detached worker's commit and publish are already rejected, and its handles only reach the generation it was given, so it can neither publish this run's data nor corrupt the next run's. A join timeout costs delayed cleanup and nothing else.
Aborting a run the Controller does not know is a successful no-op. Stopping something that is already stopped is always satisfied, and callers retrying an abort after a lost response must not be told the world is inconsistent.
func (*Controller) Ack ¶
func (c *Controller) Ack(runID string) error
Ack marks a finished run's snapshot as handed over.
func (*Controller) AckBy ¶
func (c *Controller) AckBy(runID, by string) error
AckBy is Ack with an explicit provenance. The provenance matters because a self-acknowledgement driven by an expired lease is not the same evidence as an operator collecting a result, and multi-host debugging depends on telling them apart.
func (*Controller) Await ¶
Await blocks until the run reaches a state that answers the caller's question: a starting run until it has started, a finishing run until its snapshot exists or the run died.
func (*Controller) FinishRun ¶
func (c *Controller) FinishRun(ctx context.Context, runID string) (FinishAccepted, error)
FinishRun fixes a run's closing boundary and returns as soon as that boundary exists. Draining, collecting and snapshot building continue in the background, because the caller is usually a benchmark driver that must be released the instant measurement has stopped — making it wait for the snapshot would put snapshot-building time inside the measured window of whatever runs next.
func (*Controller) PublishedSnapshots ¶
func (c *Controller) PublishedSnapshots() uint64
PublishedSnapshots reports how many snapshots were accepted.
func (*Controller) RegisterBaseline ¶
func (c *Controller) RegisterBaseline(r Registration, b BaselineCollector) error
RegisterBaseline adds a baseline collector.
func (*Controller) RegisterGeneration ¶
func (c *Controller) RegisterGeneration(r Registration, g GenerationCollector) error
RegisterGeneration adds a generation collector.
func (*Controller) SnapshotOf ¶
func (c *Controller) SnapshotOf(runID string) (*Snapshot, error)
SnapshotOf returns a finished run's immutable snapshot.
func (*Controller) StaleRejections ¶
func (c *Controller) StaleRejections() uint64
StaleRejections reports how many commit and publish attempts the epoch fence turned away. Tests assert it is non-zero to prove the fence actually ran rather than the race merely not happening.
func (*Controller) StartRun ¶
func (c *Controller) StartRun(ctx context.Context, o StartRunOptions) (StartResult, error)
StartRun opens a measurement run: it switches every generation collector to a fresh generation and samples every baseline collector, then returns an immutable record of that boundary.
A collector failure is not an error return. It downgrades StartResult Validity instead, because the boundary itself did happen and the caller needs the record. Callers must inspect Validity. An error return means the Controller refused (ErrRunActive) or could not act at all.
func (*Controller) Status ¶
func (c *Controller) Status(runID string) (RunStatus, bool)
Status returns a run's current state. The bool is false for runs the Controller does not retain.
func (*Controller) Sweep ¶
func (c *Controller) Sweep()
Sweep applies lease and TTL expiry once. It is exported so tests can drive expiry from an injected clock instead of waiting out a twenty second lease.
type Epoch ¶
type Epoch uint64
Epoch is the Controller's monotonic fencing token. It advances on every successful StartRun and on every AbortRun, which is what makes a worker belonging to an abandoned run structurally unable to publish.
type FinishAccepted ¶
type FinishAccepted struct {
RunID string
Epoch Epoch
Validity Validity
Collectors []CollectorBoundary
GenerationWindow BoundaryWindow
BoundaryWindow BoundaryWindow
AcceptedAt time.Time
}
FinishAccepted is the immutable record of a closing boundary. It is returned as soon as the boundary is fixed; draining and snapshot building continue in the background.
type GenerationCollector ¶
type GenerationCollector interface {
// Name identifies the snapshot section this collector fills.
Name() string
// BeginBoundary swaps in a fresh generation and returns a handle to the
// generation it just closed. Fast and non-blocking.
BeginBoundary(ctx context.Context, runID string, ep Epoch) (BoundaryResult, error)
// Freeze seals the current generation and returns its handle. Observations
// made after Freeze belong to the next generation, outside the run.
Freeze(ctx context.Context, runID string, ep Epoch) (BoundaryResult, error)
// Drain settles in-flight work pinned to the handle's generation only. It
// must return within DrainCancelGrace of ctx being done and must leave no
// goroutine that will later modify that generation.
//
// Implementations must wait on a per-generation done channel. sync.Cond
// cannot be interrupted by a context, so a cond-based wait makes this
// contract impossible to honour when a request never returns.
Drain(ctx context.Context, h GenerationHandle) error
// Collect reads the drained generation's fixed data. It must not read the
// collector's mutable current state.
Collect(h GenerationHandle) (any, error)
// Release frees whatever the handle pins. Idempotent: a second Release,
// or a Release racing the owner's own cleanup, is a no-op.
Release(h GenerationHandle)
}
GenerationCollector is a collector that accumulates into a swappable generation: HTTP stats, SQL stats, access log offsets, counters. Boundaries are pointer swaps, so they are expected to be non-blocking and to finish inside PerCollectorGenerationBudget.
Every method takes or returns a handle rather than exposing the collector's current state, so that a run's data is fixed the instant its boundary is taken and cannot drift while the snapshot is being built.
type GenerationHandle ¶
type GenerationHandle struct {
RunID string
Epoch Epoch
Collector string
Gen uint64
// contains filtered or unexported fields
}
GenerationHandle is an immutable reference to a closed or frozen generation. Holding one gives access to fixed data only; the collector's mutable current generation is never reachable through it.
func NewGenerationHandle ¶
func NewGenerationHandle(runID string, ep Epoch, collector string, gen uint64, token any) GenerationHandle
NewGenerationHandle builds a handle. Collectors live in other packages, so this constructor is the only way to populate the unexported token; that keeps the token opaque to everyone except the collector that created it.
func (GenerationHandle) Token ¶
func (h GenerationHandle) Token() any
Token returns the collector-internal generation reference. Only the collector that created the handle may interpret it.
func (GenerationHandle) Zero ¶
func (h GenerationHandle) Zero() bool
Zero reports whether the handle refers to nothing.
type HealthRecorder ¶
HealthRecorder is the sink for the four runctl-* health keys. It is the narrow subset of internal/health.Registry this package needs, so a Controller can be built in tests without a registry and so measurement never depends on health reporting succeeding.
type Options ¶
type Options struct {
// Budgets overrides the timing table. Zero fields keep their constants.
Budgets Budgets
// Now overrides the clock. Tests inject a controllable clock so leases and
// TTLs can be exercised without waiting them out.
Now func() time.Time
// Health receives the runctl-* keys. Nil disables health reporting.
Health HealthRecorder
// Enrich runs after Collect and before the snapshot is published, bounded
// by the enrich budget. It is how post-freeze extras such as EXPLAIN
// output are attached without widening the freeze boundary.
Enrich func(ctx context.Context, s *Snapshot) error
// DisableWatchdog stops the Controller from starting its lease/TTL sweeper
// goroutine. Tests that drive Sweep by hand set this.
DisableWatchdog bool
}
Options configures a Controller. The zero value is usable: every field falls back to a package default.
type Phase ¶
type Phase string
Phase names one step of a boundary. It appears in CollectorBoundary so a failure can be attributed to the exact step that produced it.
const ( // PhaseStartBoundary is BeginBoundary on generation collectors. PhaseStartBoundary Phase = "start-boundary" // PhaseStartBaseline is CaptureBaseline on baseline collectors. PhaseStartBaseline Phase = "start-baseline" // PhaseFinishFreeze is Freeze on generation collectors. PhaseFinishFreeze Phase = "finish-freeze" // PhaseFinishFinal is CaptureFinal on baseline collectors. PhaseFinishFinal Phase = "finish-final" // PhaseCollect is the background Drain then Collect step. PhaseCollect Phase = "collect" )
type Registration ¶
type Registration struct {
// Name identifies the snapshot section this collector fills.
Name string
// Required marks a collector whose failure invalidates the run rather than
// merely degrading it.
Required bool
// SerialOnly excludes the collector from the parallel baseline group. It
// widens the boundary window, so it is only for collectors that are
// genuinely unsafe to sample concurrently.
SerialOnly bool
}
Registration describes how a collector participates in runs.
type RunState ¶
type RunState string
RunState is a run's position in the lifecycle state machine. Transport layers copy these strings onto the wire verbatim, so the values are part of the contract.
const ( // StateIdle means no run exists. It is never stored on a run record; it // describes the Controller when nothing is retained. StateIdle RunState = "idle" // StateStarting means the opening boundary is being taken. Owner: the // StartRun caller. StateStarting RunState = "starting" // StateStarted means measurement is in progress. No owner goroutine once // the previous generations have been drained. StateStarted RunState = "started" // StateFinishing means the closing boundary is fixed and a background // worker is draining, collecting and building the snapshot. StateFinishing RunState = "finishing" // StateFinished means an immutable snapshot exists. StateFinished RunState = "finished" // StateAcknowledged means the snapshot was handed over. Terminal. StateAcknowledged RunState = "acknowledged" // StateAborting means the run is fenced and its worker is being joined. StateAborting RunState = "aborting" // StateAborted means the run was abandoned and holds no snapshot. Terminal. StateAborted RunState = "aborted" // StateExpired means the snapshot was released by TTL. Terminal tombstone. StateExpired RunState = "expired" )
type RunStatus ¶
type RunStatus struct {
RunID string `json:"run_id"`
Epoch Epoch `json:"epoch"`
State RunState `json:"state"`
Validity Validity `json:"validity"`
Reason string `json:"reason,omitempty"`
// AckedBy is one of the stable AckedBy* values.
AckedBy string `json:"acked_by,omitempty"`
Detached bool `json:"detached,omitempty"`
Since time.Time `json:"since"`
}
RunStatus is the queryable state of a run.
type SampleResult ¶
type SampleResult struct {
Handle BaselineHandle
// At is the measured moment of sampling and equals Handle.SampledAt.
At time.Time
// Committed states whether the sample is fixed for this run ID.
Committed bool
}
SampleResult is what a baseline collector returns from a sampling operation. Like BoundaryResult it is returned even on error.
type Snapshot ¶
type Snapshot struct {
RunID string
Epoch Epoch
Validity Validity
Trigger string
Sections map[string]any
Collectors []CollectorBoundary
GenerationWindow BoundaryWindow
BoundaryWindow BoundaryWindow
StartedAt time.Time
FinishedAt time.Time
}
Snapshot is a run's immutable result. Sections holds each collector's interval value keyed by collector name; the concrete types belong to the collectors, so the transport layer decides how to serialize them.
type StartResult ¶
type StartResult struct {
RunID string
Nonce string
Epoch Epoch
State RunState
Validity Validity
Collectors []CollectorBoundary
GenerationWindow BoundaryWindow
BoundaryWindow BoundaryWindow
// PreemptedRunID names the run this one displaced, if any.
PreemptedRunID string
StartedAt time.Time
}
StartResult is the immutable record of an opening boundary. err == nil with a degraded Validity is the normal way collector failures are reported; callers must inspect Validity rather than only err.
type StartRunOptions ¶
type StartRunOptions struct {
// Nonce makes the call idempotent. Empty means the Controller mints one.
Nonce string
// Preempt aborts an in-flight run instead of failing with ErrRunActive.
// This is how "the last initialize wins" is made deterministic.
Preempt bool
// Reason records who asked: "api", "initialize", "http", "hub".
Reason string
// Trigger is recorded on the run as its reset trigger.
Trigger string
}
StartRunOptions configures StartRun.
type Validity ¶
type Validity string
Validity is the data-quality axis, orthogonal to RunState. A run can be perfectly "finished" and completely untrustworthy; callers that compare runs must filter on this, not on the state.
const ( // ValidityValid means every registered collector contributed a complete // interval within its boundary window. ValidityValid Validity = "valid" // ValidityPartial means some optional sections are missing but the // interval itself is usable. ValidityPartial Validity = "partial" // ValidityInvalid means the interval cannot be trusted. Advisors and diffs // must exclude it. ValidityInvalid Validity = "invalid" )