incident

package
v0.3.4 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 11, 2026 License: MIT Imports: 17 Imported by: 0

Documentation

Overview

Package incident turns production error reports into journaled facts, folds those facts into incidents, and reacts by minting agent fix tasks (ADR-0021) — the command/event spine of error-driven auto-fix:

  • Command: Report (from a web-client beacon or server panic/5xx middleware) arrives via the API and is recorded ONCE as an error.observed fact on the synthetic incident:<fingerprint> stream.
  • Projection: the fold derives incident state (occurrences, status, minted tasks, regressions) purely from journal facts — never a mirrored column.
  • Policy: the watermark-cursor sweep (the sweeper seam shared with review/dlqfix/status, ADR-0009 exact-consumer semantics) reacts to each new fact: first occurrence mints a hot-band agent fix task, occurrences on an open incident only count, an error observed after the fix task completed is a regression and mints at machine band.

Storm safety: 1,000 identical reports in one day are 1,000 facts but at most ONE task per incident lifecycle stage; replayed pages (crash between consumption and checkpoint) converge on task dedup keys instead of minting duplicates — the same idempotency contract as every sweeper.

Index

Constants

View Source
const (
	MaxMessage     = 4096
	MaxStack       = 8192
	MaxBreadcrumbs = 20
	MaxCrumb       = 256
	MaxShort       = 256
)

Field caps enforced by Clip before anything is journaled — the journal is append-only and the dashboard renders payloads, so reports are bounded at the boundary, not trusted from the wire.

View Source
const (
	// TaskDedupPrefix namespaces minted fix-task dedup keys.
	TaskDedupPrefix = "err:"

	// DefaultFirstPriority mints a fresh incident's fix task in the ADR-0015
	// hot band (100-149): production errors jump the backlog.
	DefaultFirstPriority = 120
	// DefaultRegressionPriority mints a regression (the previous fix did
	// not hold) in the machine band (150+): operational work.
	DefaultRegressionPriority = 150
	// DefaultMaxAttempts bounds burned failures on a minted fix task — a
	// broken repo must not drain the attempt budget.
	DefaultMaxAttempts = 2
)
View Source
const ConsumerKey = "incident-policy"

ConsumerKey is the policy's identity in the watermarks table: the persisted cursor shared by every policy run over the same database, so error facts observed while no policy was running still mint their fix tasks on the next pool start.

View Source
const IDPrefix = "incident:"

IDPrefix marks synthetic incident-stream fact TaskIDs (the session:* precedent: an identity no enqueue/claim machinery can ever pick up).

Variables

This section is empty.

Functions

func Fingerprint

func Fingerprint(r Report) string

Fingerprint groups reports that share one bug: project + kind + the normalized message + the top stack frame (numbers, quoted strings, and whitespace collapsed). The release is deliberately EXCLUDED so the same bug groups across releases — a recurrence after a fix is the regression signal, not a new release's fresh incident. Returns 16 hex chars.

func FingerprintOfID

func FingerprintOfID(id string) string

FingerprintOfID extracts the fingerprint from an incident stream ID ("" when the ID does not carry the prefix).

func IncidentID

func IncidentID(fingerprint string) string

IncidentID is the synthetic journal-stream identity for one fingerprint: the fact TaskID every error.observed and incident.task-minted fact of this incident carries.

func TaskDedupKey

func TaskDedupKey(fingerprint string, seq int64) string

TaskDedupKey identifies one mint: fingerprint + the observed fact's Seq. Replayed deliveries of that fact converge on the stored task; a NEW observation (regression) carries a new Seq and mints afresh.

func TopFrame

func TopFrame(stack string) string

TopFrame extracts the first stack line (the crashing frame), stripped of common prefixes ("at ", goroutine markers). Empty stack yields "".

Types

type FactSink

type FactSink interface {
	AppendFact(ctx context.Context, f journal.Fact) error
}

FactSink appends NON-task journal facts (Store.AppendFact satisfies it; the session-bridge seam precedent — facts ride the same append-only log as task facts, in their own transaction).

type Incident

type Incident struct {
	Fingerprint string
	Project     string
	Kind        Kind
	Message     string
	FirstSeen   time.Time
	LastSeen    time.Time
	Occurrences int
	Regressions int
	Status      Status
	Mints       []Mint
}

Incident is one fingerprint's folded state — the read model of the error.observed / incident.task-minted fact family.

type Kind

type Kind string

Kind names the production surface an error was observed on.

const (
	// KindClient is an uncaught browser exception or rejection (beacon).
	KindClient Kind = "client"
	// KindServer is a panic or unexpected 5xx from server middleware.
	KindServer Kind = "server"
)

type Mint

type Mint struct {
	TaskID     string `json:"taskID"`
	SourceSeq  int64  `json:"sourceSeq"`
	Regression bool   `json:"regression"`
	Priority   int    `json:"priority"`
	// Outcome is the minted task's terminal state folded from its own
	// lifecycle facts: "" while running, "completed", or "dead".
	Outcome string `json:"outcome"`
	// DoneSeq is the Seq of the terminal fact (ordering evidence for the
	// regression rule).
	DoneSeq int64 `json:"doneSeq,omitempty"`
}

Mint records one fix task minted for an incident, as journaled by the incident.task-minted fact.

func (Mint) Terminal

func (m Mint) Terminal() bool

Terminal reports whether the minted fix task reached a terminal state.

type Policy

type Policy struct {
	// contains filtered or unexported fields
}

Policy is the reaction half of the pipeline: a watermark-cursor sweeper (the seam shared with review/dlqfix/status; ADR-0009 exact-consumer semantics) that folds every incident-family fact into the State and mints agent fix tasks for incidents that need one. Safe for concurrent use (the cursor serializes sweeps).

art-dupl:accept mirrored sweeper shell: config fields and fact validation differ per sweeper; the shared pump already lives in watermark.Cursor.

func NewPolicy

func NewPolicy(ctx context.Context, store queue.Store, sink FactSink, cfg PolicyConfig) (*Policy, error)

NewPolicy returns a policy over store (facts, enqueue, watermarks) and sink (non-task fact appends; the same object satisfies both in production). The cursor resumes from the persisted checkpoint — error facts observed while no policy was running mint on the next start. A FIRST run deliberately replays the whole journal (see the checkpoint note in the constructor): the family is new, so nothing predates it.

The full journal is folded into the state so the projection is current; only facts after the persisted cursor are REACTED to — the seq guard makes the overlap idempotent.

func (*Policy) State

func (p *Policy) State() *State

State exposes the folded read model (for `tq incidents`).

func (*Policy) Sweep

func (p *Policy) Sweep(ctx context.Context) (PolicyStats, error)

Sweep consumes new facts since the last pass and mints fix tasks for incidents that need one. Idempotent: a replayed page converges on task dedup keys and the fold's seq guard instead of duplicating work.

type PolicyConfig

type PolicyConfig struct {
	// Log receives mint failures and skips. nil logs nothing.
	Log *slog.Logger
	// PageSize bounds one fact-stream page; 0 selects the default.
	PageSize int
	// FirstPriority mints first-occurrence fix tasks; 0 selects the default.
	FirstPriority int
	// RegressionPriority mints regression fix tasks; 0 selects the default.
	RegressionPriority int
	// MaxAttempts bounds minted fix tasks; 0 selects the default.
	MaxAttempts int
}

PolicyConfig controls one Policy.

type PolicyStats

type PolicyStats struct {
	// Facts is the number of facts consumed.
	Facts int
	// TasksMinted counts fix tasks enqueued (dedup converges replays).
	TasksMinted int
	// Occurrences counts error.observed facts delivered (storms included).
	Occurrences int
	// Regressions counts observations that reopened a terminal incident.
	Regressions int
	// Known counts deliveries whose fix task was already minted (replay or
	// in-flight mint).
	Known int
	// Skipped counts facts that could not yield a mint (bad detail, enqueue
	// failure) — the incident stays open and the next occurrence re-arms.
	Skipped int
}

PolicyStats summarizes one sweep pass.

type RecordResult

type RecordResult struct {
	// Incident is the incident stream identity ("incident:<fingerprint>").
	Incident string `json:"incident"`
}

RecordResult reports where the report landed.

type Recorder

type Recorder struct {
	// contains filtered or unexported fields
}

Recorder is the ReportError command handler: validate, cap, fingerprint, and append ONE error.observed fact to the journal. It reacts to nothing — the policy sweeper owns the reactions, so recording stays cheap enough for the error path of a production service.

func NewRecorder

func NewRecorder(sink FactSink) *Recorder

NewRecorder returns a Recorder appending through sink.

func (*Recorder) Record

func (rec *Recorder) Record(ctx context.Context, rep Report) (RecordResult, error)

Record executes the ReportError command. The report is clipped and validated BEFORE anything is journaled; an invalid report appends nothing.

type Report

type Report struct {
	// Project is the repo (or logical owner) the error belongs to and the
	// unit the minted fix task's Project/Repo derive from. Required.
	Project string `json:"project"`
	// Kind selects client or server. Empty defaults to server.
	Kind Kind `json:"kind"`
	// Message is the error message (exception text or panic value).
	Message string `json:"message"`
	// Stack is the raw stack (minified frames are fine; the agent resolves
	// source maps from the pinned Release in-repo).
	Stack string `json:"stack,omitempty"`
	// Release is the commit SHA (or version) the error was observed on.
	Release string `json:"release,omitempty"`
	// Route is the route pattern (server) or path (client), not the raw
	// URL — query strings never enter the journal.
	Route string `json:"route,omitempty"`
	// Method is the HTTP method, server errors only.
	Method string `json:"method,omitempty"`
	// TraceURL is a deep link to the full trace in the observability stack
	// (SigNoz) — the query surface; this pipeline is the action surface.
	TraceURL string `json:"traceUrl,omitempty"`
	// SessionID correlates reports from one browser session.
	SessionID string `json:"sessionId,omitempty"`
	// Breadcrumbs are the last user actions/requests before the error.
	Breadcrumbs []string `json:"breadcrumbs,omitempty"`
	// OccurredAt is the observation time (zero = recording time).
	OccurredAt time.Time `json:"occurredAt,omitempty"`
}

Report is the ReportError command payload. Secrets are unrepresentable by construction: the type has no place to carry headers, cookies, or request bodies — only the fields a fix needs. Free-text fields are length-capped by Clip.

func (Report) Clip

func (r Report) Clip() Report

Clip bounds every free-text field and defaults Kind. Reports are capped before they touch the journal, so a hostile or buggy producer cannot bloat the append-only fact log.

func (Report) Validate

func (r Report) Validate() error

Validate checks the required fields with actionable errors.

type State

type State struct {
	// contains filtered or unexported fields
}

State is the fold over the journal: incidents keyed by fingerprint, first-appearance order preserved, plus the task→incident mint index. A replayed fact (Seq at or below the folded high-water mark) is a no-op, so at-least-once delivery never double-counts.

func FoldAll

func FoldAll(ctx context.Context, store queue.Store) (*State, error)

FoldAll builds the incident read model from the whole journal without touching watermarks — the read-only fold behind `tq incidents`. Unlike NewPolicy it reacts to nothing and persists nothing, so CLI reads never move a consumer cursor.

func NewState

func NewState() *State

NewState returns an empty fold.

func (*State) Apply

func (s *State) Apply(f journal.Fact) bool

Apply folds one fact. Returns false when the fact was already folded (Seq guard) or does not belong to a known family — both are no-ops.

func (*State) FingerprintOfTask

func (s *State) FingerprintOfTask(taskID string) string

FingerprintOfTask resolves which incident a task was minted for ("" when the task is not a minted fix task).

func (*State) Get

func (s *State) Get(fingerprint string) (Incident, bool)

Get returns a copy of one incident's folded state.

func (*State) Incidents

func (s *State) Incidents() []Incident

Incidents returns every folded incident in first-appearance order.

func (*State) LastSeq

func (s *State) LastSeq() int64

LastSeq returns the highest folded fact Seq.

type Status

type Status string

Status is an incident's derived lifecycle state. It is NEVER stored — the fold recomputes it from facts, so "resolved" means "the latest minted fix task completed", by construction.

const (
	// StatusOpen: observed, no fix task minted yet (or regressed after a
	// terminal fix task — the regression path reopens the incident).
	StatusOpen Status = "open"
	// StatusFixDispatched: a fix task is minted and not terminal.
	StatusFixDispatched Status = "fix-dispatched"
	// StatusResolved: the latest minted fix task completed.
	StatusResolved Status = "resolved"
	// StatusFixFailed: the latest minted fix task dead-lettered.
	StatusFixFailed Status = "fix-failed"
)

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL