Documentation
¶
Overview ¶
Package errortracking forwards Agent error logs to the internal agent telemetry intake. It exposes:
ErrorLog: the value-type that crosses the boundary from pkg/util/log into comp/core/agenttelemetry. Keeping a plain struct on the boundary lets the foundational logger subtree stay free of comp/core dependencies and lets agenttelemetry's public method avoid leaking log/slog into every consumer.
Submitter: the function type that consumers register to receive ErrorLog values. Implementations MUST be non-blocking and safe for concurrent use; the agenttelemetry component owns an internal bounded channel and flushes asynchronously.
Handler: the slog.Handler installed in the Agent's logger chain at construction. It captures records at level >= Error, builds an ErrorLog, and calls the currently registered Submitter (atomically loaded on every record). When no Submitter is registered, Handle is a silent no-op so the chain works whether or not errortracking is opted in.
Bouncer: an optional per-PC first-sighting deduplicator with a sliding time window. When attached via Handler.WithBouncerLoader, duplicate sightings of the same call site inside the window are suppressed; the suppressed-duplicate count rides on the next non-suppressed sighting via ErrorLog.Count.
Why the slot-and-loader indirection (atomic.Pointer-backed): the slog chain is built at logger setup time, very early in agent startup — before the Fx graph has constructed the agenttelemetry component that owns the Submitter and the Bouncer. The foundational logger subtree (pkg/util/log/*) cannot import comp/* (layering rule + import cycle). Atomic-pointer slots let the producer publish a value and the consumer read it lock-free on every Handle call, without the producer ever taking a static dependency on the consumer's package. The same pattern can be reused for any future late-bound handler (flare upload, remote config, …) — see pkg/util/log/setup/log.go's "late-binding handlers" section comment.
Wiring: pkg/util/log/setup installs the Handler at logger build time with closures that atomically load the package-global Submitter and Bouncer slots. The Fx graph in cmd/agent/subcommands/run/ calls RegisterErrortrackingSubmitter and RegisterErrortrackingBouncer exactly once at OnStart, pointing at agenttelemetry's SubmitErrorLog method and a freshly-constructed Bouncer. agenttelemetry.stop() clears both slots before its own cancel/drain so producers stop reaching the channel before the final flush begins.
Index ¶
- Constants
- type Bouncer
- type ErrorLog
- type Handler
- func (h *Handler) Enabled(_ context.Context, level slog.Level) bool
- func (h *Handler) Handle(_ context.Context, r slog.Record) error
- func (h *Handler) WithAttrs(_ []slog.Attr) slog.Handler
- func (h *Handler) WithBouncerLoader(loadBouncer func() *Bouncer) *Handler
- func (h *Handler) WithGroup(_ string) slog.Handler
- type Submitter
- type SyncCapture
Constants ¶
const MaxStackFrames = 16
MaxStackFrames is the upper bound on captured stack PCs per record. 16 is empirically deep enough to cover user code and its immediate callers from any agent error site and shallow enough that the resulting StackTrace string stays well under typical intake-side per-record limits.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Bouncer ¶
type Bouncer struct {
// contains filtered or unexported fields
}
Bouncer is a per-key first-sighting deduplicator with a sliding time window. The first time a given key is observed inside a window, Observe returns suppressed=false with count=1 and the observation time as firstSeen. Subsequent observations of the same key inside the same window are suppressed (return suppressed=true) and the count is incremented. After the window elapses, the next observation returns suppressed=false carrying the suppressed count of the prior window (priorTotal-1, since the first sighting was already delivered), then resets the entry to a fresh count=1. If no sightings were suppressed (priorTotal==1), the rollover is silent and the next observation is treated as a fresh first sighting (count=1).
The key is an opaque uint64 — the caller is responsible for choosing it. Today's caller (Handler.Handle) hashes the captured stack PCs with FNV-1a so two distinct stacks reaching the same terminal function are NOT collapsed into the same bouncer entry.
The 15-minute default at the agent call site (agent_telemetry.errortracking.bouncer_window_seconds) was chosen so a hot bug path collapses to one record per quarter-hour — long enough to avoid flooding the wire, short enough that operators see new error patterns promptly. A hot bug path with N sightings per window ships Count=1 at first sighting, then Count=N-1 on rollover (the suppressed portion). Summing both gives N without double-counting the first sighting.
Bouncer is purpose-built rather than reusing the global rate.Sometimes wrapper in pkg/util/log/log_limit.go — that primitive is keyless (one Limit per call site), whereas we need per-key state and need to expose the per-key count. The implementation is a small mutex-protected map with a periodic prune to bound memory; total entries are capped so a pathological input cannot blow up memory.
The zero value is NOT usable; construct via NewBouncer. Observe is safe for concurrent use.
func NewBouncer ¶
NewBouncer returns a Bouncer with the given sliding-window duration and a soft cap on tracked entries. A non-positive window disables dedup (Observe always returns suppressed=false with count=1); a non-positive maxEntries falls back to a sane default (4096).
func (*Bouncer) Observe ¶
func (b *Bouncer) Observe(key uint64, now time.Time) (suppressed bool, count uint32, firstSeen time.Time)
Observe records a sighting of the given key at now and returns whether the caller should suppress the record, the count to attach to the next delivered record (≥ 1), and the firstSeen time of the current window.
The first sighting of a key in a window returns suppressed=false, count=1, firstSeen=now. Subsequent sightings inside the same window return suppressed=true with an incrementing count and the original firstSeen. When the window elapses since firstSeen, the next sighting returns suppressed=false with count=priorSuppressed (the number of sightings that were suppressed in the elapsed window, i.e. priorTotal-1), then resets the entry to a fresh count=1. If no sightings were suppressed (priorTotal==1), the rollover is silent: Observe returns suppressed=false, count=1, as though it were a fresh first sighting, avoiding a Count=0 delivery.
This design ensures consumers can sum Count fields without double-counting: the first delivery carries Count=1, and the rollover carries the remainder.
When window is non-positive, Observe is a pass-through: returns suppressed=false, count=1, firstSeen=now.
type ErrorLog ¶
type ErrorLog struct {
// Time is the wall-clock instant the record was emitted.
Time time.Time
// PC is the program counter of the call site that emitted the
// record. PCs[0] is the same value when the handler captured a full
// stack; PC is retained as a convenience for consumers that only
// need the immediate caller (e.g. the Bouncer key).
PC uintptr
// PCs is a bounded stack capture starting at the immediate caller
// and walking up. PCsLen records the number of valid entries (may
// be less than len(PCs) when the stack is shallow). Captured at
// log-time by the handler anchored at r.PC so slog and
// pkg/util/log wrapper frames are excluded — see Handler.Handle.
PCs [MaxStackFrames]uintptr
PCsLen int
// Count is the number of same-PC sightings the Bouncer collapsed
// into this record (≥ 1; 1 means "first or only sighting in the
// current bouncer window"). Propagated to the wire Log.Count.
Count uint32
// ErrorKind is the reflect type name of the first error-typed slog
// attribute found in the record (e.g. "*net.OpError"). Empty when the
// log call carried no error attribute. The type name is not
// user-controlled — it is determined by the code that creates the
// error — so it is safe to ship unlike the error message itself.
ErrorKind string
}
ErrorLog is a value-typed snapshot of an error log record. It crosses the pkg/util/log -> comp/core boundary, keeping the foundational logger subtree free of comp/core dependencies and the agenttelemetry component free of log/slog leakage on its public interface.
Producers (slog handlers under pkg/util/log) build an ErrorLog and pass it to a Submitter. Consumers (the agenttelemetry component) accept ErrorLog on their public method and translate it internally into the dd-go wire schema before sending.
The handler captures only the wire-relevant fields (Time, PC, stack PCs, Count); message text and attrs are not captured because they are potentially user-controlled.
type Handler ¶
type Handler struct {
// contains filtered or unexported fields
}
Handler is an slog.Handler that captures records at level >= Error and forwards them to the currently registered Submitter as an ErrorLog value.
The handler holds no transport, no buffer and no goroutines. Each Handle call atomically loads the current Submitter via the load closure supplied at construction. When load returns nil the record is dropped silently; this is the steady state before the agenttelemetry component registers its Submitter during Fx startup, and again during test cleanup.
The handler optionally also late-binds a per-stack Bouncer (see bouncer.go) via loadBouncer; when loadBouncer is set, the bouncer is consulted on every Handle and may suppress the record. The bouncer key is a FNV-1a hash of the captured stack PCs — two distinct stacks reaching the same terminal function are NOT collapsed into the same bouncer entry. The running count of suppressed dupes ships on the next non-suppressed sighting via ErrorLog.Count. Late-binding mirrors the Submitter pattern so the Bouncer's lifecycle (Fx start/stop) can be managed by the agenttelemetry component without restructuring the foundational logger build.
The Submitter contract requires non-blocking submission (the consumer owns a bounded channel and flushes asynchronously), so Handle is non-blocking by construction. The handler is safe for concurrent use.
func NewHandler ¶
NewHandler returns a Handler whose Handle method atomically loads the current Submitter via load on every record. load MUST be safe for concurrent use and MUST return nil to indicate "no submitter registered"; nil records are dropped silently rather than panicking the logger chain.
The returned Handler will DROP all records until a Bouncer is wired via WithBouncerLoader — a Bouncer is mandatory for submission.
func (*Handler) Enabled ¶
Enabled reports whether the Handler will forward records at the given level. It returns true only when level >= slog.LevelError AND a Submitter is currently registered; an unregistered handler short-circuits the parent multi-handler so non-error formatting work is not wasted.
The "is a Submitter currently registered?" check is also the runtime gate for the agent_telemetry.errortracking.enabled config knob. installErrortrackingHandler (cmd/agent/subcommands/run) only calls pkg/util/log/setup.RegisterErrortrackingSubmitter when the gate is true — when the feature is off, the slot stays nil and Enabled returns false here, taking the steady-state path with no allocation.
func (*Handler) Handle ¶
Handle builds an ErrorLog from r and submits it. Records below Error are dropped (defensive: slog calls Enabled first, but direct callers might not). If no Submitter is registered the record is dropped silently. Handle always returns nil - errortracking must never break the rest of the logger chain.
The handler captures only the wire-relevant fields (Time, PC, stack PCs, Count); message text and attrs are not captured because they are potentially user-controlled.
Flow: level gate → submitter gate → capture stack PCs → (optional) bouncer check keyed by FNV-1a hash of the captured PCs → build ErrorLog → submit. When loadBouncer is set but returns nil the record is dropped (bouncer temporarily unavailable). The bouncer-key-is-a-hash-of-the-full-stack choice means two distinct stacks reaching the same terminal function each get their own dedup window.
func (*Handler) WithAttrs ¶
WithAttrs is a required slog.Handler interface method. Attrs are not shipped to the wire; this method is a required interface no-op.
func (*Handler) WithBouncerLoader ¶
WithBouncerLoader returns a Handler that consults loadBouncer on every Handle to decide whether to suppress the current record. The closure MUST be safe for concurrent use. Returning nil causes the record to be dropped — this is the safe behaviour when the bouncer is temporarily unavailable during Fx lifecycle transitions (startup / shutdown). Passing nil to WithBouncerLoader itself clears the late-binder; since a Bouncer is mandatory for submission, a nil loader causes all records to be dropped.
func (*Handler) WithGroup ¶
WithGroup returns a new Handler instance with the same Submitter loader and Bouncer loader. The group name is intentionally discarded — the wire payload is flat and does not distinguish nested groups from top-level attrs, and we don't ship attrs to the wire anyway. Returning a NEW instance (rather than the receiver) matches the canonical shape of pkg/util/log/slog/handlers/multi.go::WithGroup and async.go:: WithGroup; a no-op-receiver pattern can subtly break parent multi-handlers that expect each child to materialize a fresh instance per group context.
type Submitter ¶
type Submitter func(ErrorLog)
Submitter is the registration target for sending an ErrorLog to a consumer. The slog handler under pkg/util/log calls the currently installed Submitter (atomically loaded) on each error record.
Why a function-pointer slot rather than a constructor-injected dependency: the slog handler chain is built at logger setup time — very early in agent startup, before the Fx graph has constructed any component. The eventual consumer (the agenttelemetry component) does not exist yet at that point, and pkg/util/log/* cannot import comp/* (layering rule + import cycle). The atomic-pointer indirection is the only shape that satisfies (a) the layering constraint, (b) the "consumer is built later than the producer" lifecycle gap, and (c) the lock-free hot path requirement.
Implementations MUST be non-blocking on the hot path — the consumer is expected to enqueue into a bounded buffer and flush asynchronously. Implementations MUST be safe for concurrent calls.
A nil Submitter means "errortracking is not yet configured"; callers must guard against this and drop the record silently. This is also the gate the agenttelemetry component uses to enable/disable the feature: when agent_telemetry.errortracking.enabled=false (or gov/FIPS excludes the parent agent_telemetry feature), the component never calls pkg/util/log/setup.RegisterErrortrackingSubmitter and the slot stays nil — the handler short-circuits via Enabled() == false.
type SyncCapture ¶
type SyncCapture struct {
// contains filtered or unexported fields
}
SyncCapture wraps any slog.Handler and pre-captures the goroutine's full call stack in the emitting goroutine before forwarding the record to the inner handler. This solves the async-boundary problem: when the inner handler is dispatched by an async worker goroutine, runtime.Callers can no longer see the original caller's frames. SyncCapture must be placed in the synchronous layer of the logger chain (before handlers.NewAsync or any other async wrapper) so that Handle is called in the same goroutine that emitted the log record.
Handler.Handle reads the pre-captured PCs from the stackPCsAttrKey slog attribute added by SyncCapture.Handle, bypassing its own runtime.Callers call when the attr is present.
func NewSyncCapture ¶
func NewSyncCapture(inner slog.Handler) *SyncCapture
NewSyncCapture returns a SyncCapture that wraps inner. Install the returned handler in the synchronous layer of the logger chain so it is always called from the emitting goroutine.
func (*SyncCapture) Handle ¶
Handle captures the current goroutine's call stack anchored at r.PC, attaches the PCs as a stackPCsAttrKey slog attribute, then forwards to the inner handler. Must be called from the goroutine that emitted r.
Source Files