Documentation
¶
Overview ¶
Package watch implements the dead-man state machine: it tracks when each configured beat last pinged, declares a beat missing once its deadline of silence passes, and notifies on the missing and recovered transitions.
The deadline clock for every beat starts at the process-start baseline supplied to New, so a beat that never pings still alerts one deadline after process start even when startup work delays watcher construction. That deliberately closes the classic dead-man blind spot where a receiver restart silently disarms the switch until the first ping re-arms it.
Notifications are split by whether the incident is still open. A live outage gets the present-tense missing notice and, when the beat returns, its recovered notice. Outages that were already over by the time anything about them could be sent are reported once as history, so a resolved incident is never announced as a new live failure. A notice is late for one of three reasons — a delivery attempt failed, this observer's own scheduling deferred the notice before anything was attempted, or the outage ended between two sweeps and no alert was ever due — and each record carries which one (LateReason), because the operator's next step differs.
Every log line reporting a delivery failure or a notice abandoned for good carries a retryable attribute: true when the record survives and the next sweep sends it again (the missing and history send failures), false when nothing is left to attempt and no notice for that transition will ever arrive (a discarded record, a failed recovered notice, the notices abandoned at shutdown). The LEVEL is deliberately not that signal — a retried send stays at Error rather than spamming Warn every sweep — so this attribute is what a log rule keys on to tell "wait for it" from "reconstruct the window".
Index ¶
Constants ¶
const DefaultTick = 15 * time.Second
DefaultTick is the watch loop's check cadence. Deadlines are minutes to hours, so a fixed 15s sweep bounds alert latency without configuration.
const MaxHistoryBatch = 8
MaxHistoryBatch is the largest number of ended outages one BeatOutageHistory notice can carry: the per-beat queue bound, and therefore the batch size a notifier's rendering budget must cover.
Variables ¶
This section is empty.
Functions ¶
func LongestOutage ¶ added in v1.1.0
LongestOutage returns the longest span among outages, or zero for an empty slice. It lives here, next to the type, so the notifier's summary and the sender's log line report the same figure by construction.
Types ¶
type Beat ¶ added in v1.1.0
Beat is one watched beat as the state machine needs it: an id and the silence deadline that declares it missing. It is the watch package's own input type, so the state machine and its tests do not depend on how (or from where) the composition root parsed the configuration.
type BeatOutcome ¶ added in v1.1.0
type BeatOutcome uint8
BeatOutcome is what recording a ping resulted in: the three states the beat endpoint answers for. A single value rather than a pair of booleans, so the combination the state machine never produces ("recorded, but admission is closed") cannot be constructed or mishandled by a caller.
const ( // BeatClosed means shutdown has closed admission for the rest of the // process's life; nothing was recorded. It is deliberately the zero value: // an omitted outcome fails closed. BeatClosed BeatOutcome = iota // BeatUnknown means id is not a configured beat; nothing was recorded and no // metric series was minted for the id (it is a label). BeatUnknown // BeatRecorded means the ping was accepted and the beat's state was updated. BeatRecorded )
type LateReason ¶ added in v1.1.0
type LateReason uint8
LateReason names WHY an outage was already over by the time anything about it could be sent — why its notice is past tense rather than a live alarm. All three values route identically (see collectDue) and all three render in the past tense; they differ in what the operator should DO, which is the one thing a past-tense notice cannot state correctly without being told.
The three form a one-way ORDER, weakest claim about delivery first:
LateEndedBeforeDetection (delivery was never involved) LateSchedulerDeferred (this observer held the notice; nothing was tried) LateUndelivered (a delivery attempt was made and failed)
A record's reason only ever moves ALONG that order, never back, because a notice that blames delivery must have an attempt to point at. Which of the two weaker reasons a record starts on is decided once, where the record is queued (collectBeatDue for an open one, lateReasonForUnqueuedOutage for a closed one), and blameDelivery is the ONLY writer after that: a confirmed delivery failure outranks both. The NUMERIC values are deliberately NOT that order — LateUndelivered is the zero value, for the reason its own comment gives — so a comparison operator is never the right way to ask about it.
const ( // LateUndelivered means a delivery ATTEMPT for this outage was made and // FAILED: the live missing notice a sweep raised for it did not get through // (sendMissing blames delivery for the record it left queued), or the // past-tense notice reporting it was refused and is being retried // (sendHistory does the same for every record that notice covered). Either // way delivery is demonstrably behind, so the notice points the operator at // the webhook. // // It is the strongest claim in the order above, and the zero value // deliberately: a future producer that queues a record without naming a // reason then points at delivery instead of vouching for it, and a delivery // path that is quietly behind is the one thing a dead-man switch must never // claim is healthy. Nothing ever moves a record back off it. LateUndelivered LateReason = iota // LateEndedBeforeDetection means no sweep ever saw this outage. It crossed // its deadline and was ended by a ping inside the gap between two sweeps // (DefaultTick), so no live notice was ever due and delivery was never // involved. Reporting it as a delivery problem would send an operator // hunting through a webhook that was working. // // It is the weakest claim in the order above, and the only thing that ever // moves a record off it is a failed send (blameDelivery, to // LateUndelivered): nothing rewrites it into a deferral, because a record // carries the reason it was queued with. LateEndedBeforeDetection // LateSchedulerDeferred means this observer never STARTED sending anything // for this outage before the beat came back: the notice was deferred by // knell's own scheduling, not refused by the webhook. Three producers, and // not one of them is evidence about delivery: // // - sweepSendBudget cut the sweep short before this beat's turn came, so the // open record collectBeatDue had already queued with this reason is what // the sealed outage reports (a full-fleet storm spends the 5s budget // against a perfectly healthy webhook, so the cut diagnoses nothing); // - the beat was held behind a queued or in-flight recovered notice, so // collectBeatDue started no missing notice for it (st.recovering); // - the beat's pending queue was full when a sweep detected the outage, so // there was no record and no notice for it at all until a slot freed // (lateReasonForUnqueuedOutage via overflowAccounted). // // It exists because the reason an operator READS decides where they look, // and all three of these used to render as "check the webhook" — sending // them to hunt through a webhook that had never been asked to deliver // anything. It is the reason every record queued by collectBeatDue starts // with, so a record only ever claims a failed delivery once one has // actually failed. LateSchedulerDeferred )
type Notifier ¶
type Notifier interface {
// BeatMissing reports that id has been silent since live.Started and was
// still silent when the sweep observed it at live.Observed (its deadline
// has passed), so the outage is still open. live.DownFor() is the silence
// to report.
BeatMissing(ctx context.Context, id string, live Transition) error
// BeatRecovered reports that id pinged again at live.Observed after having
// been declared missing, ending the silence that began at live.Started
// (its last accepted ping). live.DownFor() is how long it was down.
BeatRecovered(ctx context.Context, id string, live Transition) error
// BeatOutageHistory reports outages of id that were already over by the
// time anything about them could be sent, collapsed into a single
// past-tense notice. outages is chronological and never empty, every entry
// carries its own recovery point and every entry has a start its silence
// can be measured from, so the implementation must not present them as
// ongoing. Each entry also carries its own LateReason, which the
// implementation must report rather than assume: one batch can hold all
// three.
//
// Those guarantees are the CALLER's, and this package holds them before it
// calls: non-emptiness by construction (collectDue collapses a run only
// under `len(run) > 0`; assertSealedRun iterates the run, so it would pass
// an empty one vacuously) and the per-record properties by assertion
// (assertSealedRun). So an implementation must not re-check them: it has no
// way to report a producer bug. Every error it returns is read as a
// delivery failure, so a refusal would keep the malformed records queued
// and re-offered every sweep, hold that beat's live notices behind them,
// and leave /healthz reporting a healthy observer (Meyer's non-redundancy
// principle, with a concrete price attached).
BeatOutageHistory(ctx context.Context, id string, outages []Outage) error
}
Notifier delivers the transition notifications. Implementations are expected to retry transient failures internally and return the final outcome.
Every method's returned error is logged VERBATIM by this package (sendMissing, sendHistory and sendRecovered log it as the "error" attribute), so an implementation must return only log-safe errors: the webhook endpoint knell posts to carries its credential in its own URL path, and an error that wraps that URL would publish the credential into the log stream. internal/notify reduces every transport error through httpx.LogSafeError for exactly this reason; a second implementation has to do the same.
The first two methods report a LIVE incident as it happens; the third reports outages that were already over by the time their notice could be sent, so an implementation must render it in the past tense. Nothing else distinguishes the two: a live outage never reaches BeatOutageHistory, and a history notice is never followed by BeatRecovered calls for the outages it covers.
Every method carries one of this package's own value types — a Transition for the two live notices, Outage records for history — and each of those derives its own span through DownFor. So the tense lives in the method NAME while "how long was the beat down" has a single home, and an implementation cannot measure the live case one way and the history case another.
type Outage ¶ added in v1.1.0
type Outage struct {
// Started is the last accepted ping before the outage: the instant the
// beat's silence began (Transition.Started is the same instant).
Started time.Time
// Recovered is the first accepted ping after the outage: the instant it
// ended. Always after Started.
Recovered time.Time
// LateReason says why this outage is reported after the fact instead of
// as a live incident. The renderer must not guess it: the three reasons
// lead an operator to different next steps.
LateReason LateReason
}
Outage is one already-ended outage as the state machine observed it, the unit BeatOutageHistory reports. It is the watch package's own output type, so the notifier renders the outage from the state machine's shape without the state machine depending on how (or where) it is rendered. Transition is its live counterpart: same Started anchor, same DownFor measurement, plus the two facts only an ended outage has (its recovery point and LateReason).
func (Outage) DownFor ¶ added in v1.1.0
DownFor is the outage's full span: from the last ping before it to the first ping after it. It is the figure a past-tense notice reports ("was missing for ..."), derived through Transition.DownFor — the one home of the measurement — so every renderer, live or history, measures it the same way.
func (Outage) Ended ¶ added in v1.1.0
Ended reports whether the outage carries a usable recovery point: one that is set and no earlier than Started. It is the executable form of the invariant the Recovered field documents, and it lives here, next to the type, so the assertion that ENFORCES that invariant where the run is built (assertSealedRun) and the field that states it cannot come to disagree.
type Transition ¶ added in v1.1.0
type Transition struct {
// Started is the last accepted ping before the outage: the instant the
// beat's silence began, or the process-start baseline for a beat that has
// never pinged. Outage.Started is the same instant, for an outage that has
// already ended.
Started time.Time
// Observed is the instant this notice speaks for: for a missing notice the
// sweep that observed the beat still silent, for a recovered notice the
// ping that ended the outage. At or after Started.
Observed time.Time
}
Transition is the live incident a missing or a recovered notice reports: the beat's silence as the two instants the state machine measured it between, rather than a duration the call site computed. The notifier therefore derives the figure it renders exactly the way it derives an ended outage's (Outage.DownFor), and a new rendering fact about a live notice is a field here instead of a new interface parameter.
Both instants are always set: a Transition never carries an empty field, and WHICH observation Observed is comes from the method carrying it, never from a zero value. Nothing here says whether the outage is over — BeatMissing and BeatRecovered do, by being different methods.
func (Transition) DownFor ¶ added in v1.1.0
func (t Transition) DownFor() time.Duration
DownFor is how long the beat had been unseen at Observed, the figure a live notice reports ("silent for ...", "after ... of silence"). It is the same measurement Outage.DownFor makes for an outage that has ended — last accepted ping to the instant being reported — and Outage.DownFor derives it through this method, so the live and history notices cannot report the same span two different ways.
type Watcher ¶
type Watcher struct {
// contains filtered or unexported fields
}
Watcher tracks beat freshness and drives transition notifications. Beat is safe for concurrent use; Run is the single background sender so notify calls never hold the lock.
func New ¶
New builds a Watcher for the given beats. start is the process-start baseline every beat's first deadline counts from (the caller captures it at process entry, before any startup work that could delay wiring); now is the ongoing clock, pass time.Now in production. The pending recovered-transition queue is sized from the beat count: each beat can hold at most one pending recovery, so that bound is exactly enough for a ping on every beat to queue without blocking.
func (*Watcher) Beat ¶
func (w *Watcher) Beat(id string) BeatOutcome
Beat records a ping for id and reports which of the three outcomes it was: BeatUnknown when id is not a configured beat (the caller answers 404 and nothing is recorded), BeatClosed once shutdown has closed admission (the caller answers 503 and nothing is recorded), BeatRecorded otherwise. A ping on an alerted beat queues the recovered notification for the Run loop, so this never blocks on the webhook.
Admission is decided HERE, under the same mutex as the state mutation and the recovered enqueue, because that is the only way the decision can be atomic with respect to shutdown: any check the CALLER made instead could pass and then be descheduled while Run takes its undelivered-work snapshot and returns, so the ping would land behind a tally that has already been reported. That is why the HTTP handler keeps no lifecycle check of its own and simply reports this outcome. Since StopAccepting takes this mutex too, every ping either completes before admission closes — and is therefore visible to logUndelivered — or observes accepting=false and records nothing at all.
func (*Watcher) Run ¶
Run drives the watch loop until ctx is cancelled: a sweep every tick plus immediate delivery of queued recovered transitions. It is the only goroutine that calls the notifier.
func (*Watcher) StopAccepting ¶ added in v1.1.0
func (w *Watcher) StopAccepting()
StopAccepting closes beat admission for the rest of the process's life. It takes the mutex Beat mutates under, so once it returns no ping can still be between its admission check and its state change: that ordering is what makes logUndelivered's tally complete rather than merely narrow.
The watcher is the single owner of this decision, and it is exported because the shutdown SEQUENCE is the composition root's: main closes admission from webhttp's pre-drain hook (and from the serve-exit path, which skips pre-drain), so the beat endpoint stops accepting before the HTTP drain begins instead of only when Run reaches its own ctx.Done arm. Run still calls it there, so the ordering above holds whether or not anything called it first.
Calling it more than once is safe and expected: it assigns a state, it does not transition one, so the second call is a no-op that changes nothing the tally or the recovery drain then reads.