investigate

package
v0.8.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 10, 2026 License: Apache-2.0 Imports: 22 Imported by: 0

Documentation

Overview

Package investigate routes triggers (incident alerts, GitOps failures) into a single async investigation queue and runs them. The investigation itself is pluggable via Investigator: LoopInvestigator (loop.go) is the real implementation — a ReAct loop that drives a ModelProvider with tools, feeds tool results back, and delivers a providers.Investigation when the model calls submit_findings — while LogInvestigator remains the read-only fallback used when no model is configured.

Index

Constants

View Source
const ReinvestigateLabel = "reinvestigate"

ReinvestigateLabel is the label a human adds to a curated KB issue to ask RunLore to run the investigation again (e.g. after more has happened, or to get a fresh look). RunLore polls for it — it receives no inbound GitHub webhooks.

Variables

This section is empty.

Functions

func IsCascadeFailure added in v0.2.0

func IsCascadeFailure(fe providers.FailureEvent) bool

IsCascadeFailure reports whether the event is a downstream cascade symptom rather than a root-cause failure worth investigating. The gitops watcher source adapter drops these so only root failures reach the pipeline.

func WithObservedResources added in v0.8.0

func WithObservedResources(ctx context.Context, seed ...providers.Workload) context.Context

WithObservedResources derives a context carrying an observed-resource collector seeded with the given workloads (typically the originating alert/failure workload — the trigger's subject is by definition a legitimate thing to act on, so it ALWAYS counts as observed even when no tool re-reads it). Read tools that confirm resources server-side call recordObserved; reviewActions consults the set via guardUnobservedTargets.

Types

type CloudResourceHealthTool

type CloudResourceHealthTool struct {
	Cloud providers.CloudProvider
}

CloudResourceHealthTool exposes AWS-side resource health (EC2/ASG/EKS) to the model.

func (CloudResourceHealthTool) Call

Call renders cloud resource health.

func (CloudResourceHealthTool) Description

func (t CloudResourceHealthTool) Description() string

Description returns the tool description.

func (CloudResourceHealthTool) Name

Name returns the tool name.

func (CloudResourceHealthTool) Schema

func (t CloudResourceHealthTool) Schema() string

Schema returns the JSON schema for the arguments.

type CloudWhatChangedTool

type CloudWhatChangedTool struct {
	Cloud providers.CloudProvider
}

CloudWhatChangedTool exposes recent mutating AWS control-plane events (CloudTrail) as the AWS-layer "what changed" lens — infra/manual changes invisible to GitOps.

func (CloudWhatChangedTool) Call

func (t CloudWhatChangedTool) Call(ctx context.Context, args string) (string, error)

Call lists cloud changes over the window and renders them.

func (CloudWhatChangedTool) Description

func (t CloudWhatChangedTool) Description() string

Description returns the tool description.

func (CloudWhatChangedTool) Name

func (t CloudWhatChangedTool) Name() string

Name returns the tool name.

func (CloudWhatChangedTool) Schema

func (t CloudWhatChangedTool) Schema() string

Schema returns the JSON schema for the arguments.

type ControllerLogsTool

type ControllerLogsTool struct {
	Logs providers.LogReader
}

ControllerLogsTool reads recent logs from a Flux controller, optionally filtered to a resource name — surfacing WHY a source/object failed to reconcile (auth, reference not found, build error) when resource status alone isn't enough.

func (ControllerLogsTool) Call

func (t ControllerLogsTool) Call(ctx context.Context, args string) (string, error)

Call fetches and renders the controller's recent logs.

func (ControllerLogsTool) Description

func (t ControllerLogsTool) Description() string

Description returns the tool description.

func (ControllerLogsTool) Name

func (t ControllerLogsTool) Name() string

Name returns the tool name.

func (ControllerLogsTool) Schema

func (t ControllerLogsTool) Schema() string

Schema returns the JSON schema for the arguments.

type Debouncer

type Debouncer struct {
	// contains filtered or unexported fields
}

Debouncer delays a GitOps-failure investigation until the failure has PERSISTED for a short window, then re-reads the resource's current Ready status and enqueues only if it is still failing. This filters reconcile-churn transients (e.g. a brief HealthCheckCanceled or an ArtifactFailed while a new artifact is mid-checkout) that clear on their own — the kind of misleading status that previously produced a confident-but-wrong root cause.

A zero window disables the wait and re-check entirely: Debounce enqueues immediately, preserving the original fire-on-every-failure behavior.

func NewDebouncer

func NewDebouncer(window time.Duration, check StillFailing, log *slog.Logger) *Debouncer

NewDebouncer builds a Debouncer. A zero window disables debouncing (immediate enqueue). check is consulted only when window > 0.

func (*Debouncer) Debounce

func (d *Debouncer) Debounce(ctx context.Context, r Request, q Enqueuer)

Debounce decides whether to enqueue r. With window 0 it enqueues immediately. Otherwise it waits window, re-checks the workload's current Ready status, and enqueues only if it is still failing — dropping (debug-logged) any failure that cleared or flipped to a different transient within the window. It blocks for the wait; callers run it in a goroutine per failure event.

func (*Debouncer) WithMetrics

func (d *Debouncer) WithMetrics(m *telemetry.Metrics) *Debouncer

WithMetrics installs the (nil-safe) metric set and returns the Debouncer for chaining. A dropped transient increments GitOpsFailuresDebounced.

type Enqueuer

type Enqueuer interface {
	Enqueue(r Request)
}

Enqueuer accepts investigation requests.

type GitOpsStatusTool

type GitOpsStatusTool struct {
	Inspector providers.GitOpsInspector
}

GitOpsStatusTool exposes a GitOps/Kubernetes resource's status (a Flux Ready condition or an Argo CD Application's health/sync), key spec refs, and recent Events — so the model can learn WHY a resource is failing, not just that it is.

func (GitOpsStatusTool) Call

func (t GitOpsStatusTool) Call(ctx context.Context, args string) (string, error)

Call fetches the resource status and renders it.

func (GitOpsStatusTool) Description

func (t GitOpsStatusTool) Description() string

Description returns the tool description.

func (GitOpsStatusTool) Name

func (t GitOpsStatusTool) Name() string

Name returns the tool name.

func (GitOpsStatusTool) Schema

func (t GitOpsStatusTool) Schema() string

Schema returns the JSON schema for the arguments.

type GitOpsTreeTool

type GitOpsTreeTool struct {
	Inspector providers.GitOpsInspector
}

GitOpsTreeTool walks a resource's dependency graph (Flux dependsOn/sourceRef, or an Argo CD Application's managed-resource tree) and renders it, so the model can find the ROOT failing resource behind a cascade (the not-Ready/Degraded or missing node), not just the downstream symptom.

func (GitOpsTreeTool) Call

func (t GitOpsTreeTool) Call(ctx context.Context, args string) (string, error)

Call walks the dependency tree and renders it.

func (GitOpsTreeTool) Description

func (t GitOpsTreeTool) Description() string

Description returns the tool description.

func (GitOpsTreeTool) Name

func (t GitOpsTreeTool) Name() string

Name returns the tool name.

func (GitOpsTreeTool) Schema

func (t GitOpsTreeTool) Schema() string

Schema returns the JSON schema for the arguments.

type Investigator

type Investigator interface {
	Investigate(ctx context.Context, r Request) error
}

Investigator runs an investigation for a Request.

type KBSearchTool

type KBSearchTool struct {
	Catalog catalog.Searcher
	// contains filtered or unexported fields
}

KBSearchTool lets the model search the OKF knowledge catalog (runbooks, past incidents) to ground its reasoning.

func (KBSearchTool) Call

func (t KBSearchTool) Call(_ context.Context, args string) (string, error)

Call searches the catalog and renders the top matches. When the catalog exposes relevance scores (catalog.ScoredSearcher — the production *catalog.Catalog does), the scored search is preferred so the loop can capture the strongest clear-match hit for the notification; a plain Searcher falls back to the unscored path.

func (KBSearchTool) Description

func (t KBSearchTool) Description() string

Description returns the tool description.

func (KBSearchTool) Name

func (t KBSearchTool) Name() string

Name returns the tool name.

func (KBSearchTool) Schema

func (t KBSearchTool) Schema() string

Schema returns the JSON schema for the arguments.

type KubeEventsTool

type KubeEventsTool struct {
	Kube providers.KubeReader
}

KubeEventsTool surfaces causes that live in the Kubernetes event stream rather than logs or status — FailedScheduling (Insufficient cpu/memory), FailedMount, FailedAttachVolume, BackOff, etc.

func (KubeEventsTool) Call

func (t KubeEventsTool) Call(ctx context.Context, args string) (string, error)

Call lists events and renders them.

func (KubeEventsTool) Description

func (t KubeEventsTool) Description() string

Description returns the tool description.

func (KubeEventsTool) Name

func (t KubeEventsTool) Name() string

Name returns the tool name.

func (KubeEventsTool) Schema

func (t KubeEventsTool) Schema() string

Schema returns the JSON schema for the arguments.

type LogInvestigator

type LogInvestigator struct {
	Log *slog.Logger
}

LogInvestigator is the read-only fallback used when no model is configured: it logs the request it would investigate and does nothing else. The real investigation is LoopInvestigator.

func (LogInvestigator) Investigate

func (l LogInvestigator) Investigate(_ context.Context, r Request) error

Investigate logs the request.

type LoopInvestigator

type LoopInvestigator struct {
	Model      providers.ModelProvider
	Tools      []Tool
	Log        *slog.Logger
	MaxSteps   int
	OnComplete func(providers.Investigation) // delivery hook (Slack/Matrix later)
	Actions    *action.Policy                // autonomy ladder; nil/off = read-only findings only
	Recall     *Recall                       // optional: short-circuit on a high-confidence catalog hit
	Recurrence *RecurrenceGate               // optional: suppress re-investigating a just-answered trigger
	Verify     bool                          // run an adversarial review of root causes before delivery

	// OnRecall, when set, receives one RecallDecision per investigation whenever a
	// Recall is configured and consulted — reporting whether instant recall fired,
	// which entry it matched, and whether the recalled answer short-circuited the loop
	// or was withdrawn by the verify pass. It is telemetry only (nil-safe, no effect on
	// the investigation); the eval harness uses it to assert recall behaviour
	// mechanically rather than inferring it from the final finding.
	OnRecall func(RecallDecision)

	// VerifyModel optionally routes the adversarial verify pass to a cheaper/faster
	// model. nil ⇒ the verify pass reuses Model. Verify itself always runs.
	VerifyModel providers.ModelProvider

	// Timeout bounds a single investigation end-to-end (recall + every model/tool
	// call, including a hung git clone/patch). 0 disables it. On expiry the loop
	// delivers a synthetic timeout result rather than starving the queue worker.
	Timeout time.Duration

	// ToolTimeout bounds a SINGLE tool call so one hung/slow provider (a stuck git
	// clone, an unresponsive metrics/logs endpoint) can't consume the whole
	// per-investigation budget. On expiry runTool returns a clear, NON-fatal "timed
	// out" string that becomes the tool result and the loop continues. 0 disables it
	// (tool calls then share only the per-investigation Timeout). Defaulted at
	// construction (see internal/app/investigate.go).
	ToolTimeout time.Duration

	// Cost controls (0 means disabled/unlimited):
	MaxToolOutputBytes        int // truncate tool results larger than this before adding to history
	MaxTokensPerInvestigation int // inject a budget-nudge message when the estimated token count exceeds this

	// Compaction selects how mid-loop history compaction treats the tool outputs it
	// elides: "" / "elide" (default) drops their bodies for markers; "summarize" first
	// asks a model for one compact factual digest of the batch and keeps that in place
	// of the markers, falling back to plain elision on any summarizer failure. When
	// "summarize", the digest call is routed to VerifyModel if set, else Model.
	Compaction string

	// Observability — nil-safe; no-op when telemetry is disabled.
	Metrics       *telemetry.Metrics
	ModelProvider string // label for model_requests/model_request_duration metrics (e.g. "anthropic")

	// OnProgress, when set, receives an interim progress snapshot every
	// ProgressEverySteps steps of a long investigation. It is nil-safe and
	// default-off (nil ⇒ never called; zero extra model calls). The callback must
	// never fail the investigation: the app wires it to a best-effort delivery that
	// logs and swallows its own errors. The interim text passed is already
	// secret-redacted at this egress boundary.
	OnProgress func(providers.ProgressUpdate)
	// ProgressEverySteps is the cadence for OnProgress. <= 0 disables progress
	// pings even when OnProgress is set.
	ProgressEverySteps int

	// Pricing (optional) estimates a per-investigation cost from the accumulated
	// token totals. nil ⇒ token totals are still reported but no cost is shown.
	// VerifyPricing prices the adversarial verify pass's tokens; nil ⇒ it inherits
	// Pricing (so a cheaper verify model is costed correctly when configured).
	Pricing       *Pricing
	VerifyPricing *Pricing

	// KBMatchScore is the BM25 bar the per-investigation kb_search hit tracker uses to
	// decide a hit is a "clear match" worth surfacing on the notification
	// (Investigation.MatchedKnowledge). It is threaded from the operator's CONFIGURED
	// recall SoloFloor (see BuildInvestigator) so the visibility bar tracks the same
	// corpus/query-dependent BM25 score regime kb_search runs in: a cluster that tunes
	// solo_floor DOWN for its sub-1.0 alert-query scores gets a correspondingly low bar
	// instead of the feature silently no-opping. 0 (recall disabled/unconfigured) ⇒ the
	// tracker falls back to the historical 4.0 default (kbClearMatchScoreDefault).
	KBMatchScore float64
}

LoopInvestigator is the ReAct investigation loop: it drives a ModelProvider with tools, feeds tool results back, and finishes when the model calls submit_findings (or MaxSteps is reached). The completed investigation is handed to OnComplete.

func (*LoopInvestigator) Investigate

func (li *LoopInvestigator) Investigate(ctx context.Context, req Request) error

Investigate runs the loop for a request. It implements Investigator.

type NetworkDropsTool

type NetworkDropsTool struct {
	Network providers.NetworkProvider
}

NetworkDropsTool lets the model list recently denied/dropped network flows from the configured (pluggable, CNI-agnostic) network-flow source — surfacing NetworkPolicy denials, firewall/security-group rejects, and connectivity failures.

func (NetworkDropsTool) Call

func (t NetworkDropsTool) Call(ctx context.Context, args string) (string, error)

Call lists dropped flows over [now-since, now] and renders them.

func (NetworkDropsTool) Description

func (t NetworkDropsTool) Description() string

Description returns the tool description.

func (NetworkDropsTool) Name

func (t NetworkDropsTool) Name() string

Name returns the tool name.

func (NetworkDropsTool) Schema

func (t NetworkDropsTool) Schema() string

Schema returns the JSON schema for the arguments.

type OutcomeStats

type OutcomeStats interface {
	OpenCounts() (map[string]outcome.Aggregate, error)
}

OutcomeStats reports per-entry recall outcomes for confidence decay. *outcome.Ledger satisfies it.

type PodLogsTool added in v0.2.0

type PodLogsTool struct {
	Logs providers.LogReader

	// IncidentNamespace is the namespace of the workload under investigation; it is
	// always permitted. Set per-investigation by the loop from req.Workload.Namespace.
	IncidentNamespace string

	// AllowedNamespaces is the operator-configured allowlist of extra namespaces
	// pod_logs may read (config.investigation.pod_log_namespaces). Empty ⇒ the
	// incident namespace is the only permitted namespace (secure by default).
	AllowedNamespaces []string
}

PodLogsTool reads recent logs from a workload's pods over the Kubernetes API, optionally the PREVIOUS (last-terminated) container — the crash output of a CrashLoopBackOff pod that the freshly-restarted container no longer has. This is the general workload-log reader (controller_logs is Flux-controller-only).

Because pod logs carry secrets/PII and are streamed to the external LLM, the model is constrained at the application layer to a fixed allowed set of namespaces — the incident's own namespace plus an operator-configured allowlist — rather than trusting Kubernetes RBAC alone. A request for any other namespace is rejected before the cluster is queried.

func (PodLogsTool) Call added in v0.2.0

func (t PodLogsTool) Call(ctx context.Context, args string) (string, error)

Call fetches and renders the workload's recent pod logs.

func (PodLogsTool) Description added in v0.2.0

func (t PodLogsTool) Description() string

Description returns the tool description.

func (PodLogsTool) Name added in v0.2.0

func (t PodLogsTool) Name() string

Name returns the tool name.

func (PodLogsTool) Schema added in v0.2.0

func (t PodLogsTool) Schema() string

Schema returns the JSON schema for the arguments.

type PodStatusTool

type PodStatusTool struct {
	Kube providers.KubeReader
}

PodStatusTool surfaces pod-level failures (CreateContainerConfigError, ImagePullBackOff, CrashLoopBackOff, …) that never reach logs because the container never started — the gap the GitOps/logs/metrics tools can't fill.

func (PodStatusTool) Call

func (t PodStatusTool) Call(ctx context.Context, args string) (string, error)

Call lists pod statuses and renders them.

func (PodStatusTool) Description

func (t PodStatusTool) Description() string

Description returns the tool description.

func (PodStatusTool) Name

func (t PodStatusTool) Name() string

Name returns the tool name.

func (PodStatusTool) Schema

func (t PodStatusTool) Schema() string

Schema returns the JSON schema for the arguments.

type Pricing added in v0.3.0

type Pricing struct {
	InputUSDPerMTok       float64
	OutputUSDPerMTok      float64
	CachedInputUSDPerMTok float64
}

Pricing is per-model token pricing in USD per MILLION tokens, used to estimate a per-investigation cost. A nil *Pricing means unpriced (cost is not surfaced); the zero value prices everything at $0.

type QueryLogsTool

type QueryLogsTool struct {
	Logs providers.LogsProvider
}

QueryLogsTool lets the model query logs (LogsQL) over a recent window.

func (QueryLogsTool) Call

func (t QueryLogsTool) Call(ctx context.Context, args string) (string, error)

Call runs the logs query over [now-since, now] and renders the lines.

func (QueryLogsTool) Description

func (t QueryLogsTool) Description() string

Description returns the tool description.

func (QueryLogsTool) Name

func (t QueryLogsTool) Name() string

Name returns the tool name.

func (QueryLogsTool) Schema

func (t QueryLogsTool) Schema() string

Schema returns the JSON schema for the arguments.

type QueryMetricsRangeTool added in v0.2.0

type QueryMetricsRangeTool struct {
	Metrics providers.MetricsProvider
}

QueryMetricsRangeTool lets the model run a PromQL RANGE query and see how a metric trends over a recent window — the time dimension an instant query lacks, which is what reveals when a problem started (rising / spiking / recovering).

func (QueryMetricsRangeTool) Call added in v0.2.0

func (t QueryMetricsRangeTool) Call(ctx context.Context, args string) (string, error)

Call runs the range query over [now-since, now] and renders a per-series trend.

func (QueryMetricsRangeTool) Description added in v0.2.0

func (t QueryMetricsRangeTool) Description() string

Description returns the tool description.

func (QueryMetricsRangeTool) Name added in v0.2.0

func (t QueryMetricsRangeTool) Name() string

Name returns the tool name.

func (QueryMetricsRangeTool) Schema added in v0.2.0

func (t QueryMetricsRangeTool) Schema() string

Schema returns the JSON schema for the arguments.

type QueryMetricsTool

type QueryMetricsTool struct {
	Metrics providers.MetricsProvider
}

QueryMetricsTool lets the model run PromQL instant queries (saturation, error rates, health) against the metrics backend.

func (QueryMetricsTool) Call

func (t QueryMetricsTool) Call(ctx context.Context, args string) (string, error)

Call runs the instant query and renders the series.

func (QueryMetricsTool) Description

func (t QueryMetricsTool) Description() string

Description returns the tool description.

func (QueryMetricsTool) Name

func (t QueryMetricsTool) Name() string

Name returns the tool name.

func (QueryMetricsTool) Schema

func (t QueryMetricsTool) Schema() string

Schema returns the JSON schema for the arguments.

type Queue

type Queue struct {
	// contains filtered or unexported fields
}

Queue is a rate-limiting investigation queue: duplicate triggers coalesce, and failed investigations are retried with exponential backoff. A fresh workqueue is built per Run (leadership term), so the queue recovers after a lost-then- re-acquired leadership instead of staying permanently shut down.

func NewQueue

func NewQueue(inv Investigator, log *slog.Logger) *Queue

NewQueue builds an investigation queue. The workqueue itself is created per Run.

func (*Queue) CancelByFingerprint added in v0.8.0

func (q *Queue) CancelByFingerprint(fp string) bool

CancelByFingerprint drops a QUEUED — accepted but not yet started — investigation whose sole alert fingerprint matches, and reports whether one was cancelled. It is called from the pipeline's resolved-webhook path when triggers.incidents.cancel_queued_on_resolve is enabled, extending the debounce idea past the hold window: without it, a fire→resolve sequence whose firing already passed into the queue still burns a full paid investigation.

Deliberate boundaries:

  • Only pending SINGLE-fingerprint requests cancel (see cancellableFingerprint); a coalesced multi-alert batch survives one member's resolve.
  • An IN-FLIGHT investigation is never cancelled (see the inflight field): it completes and delivers as usual.

Cancellation is just the payload delete: the already-queued workqueue item for k stays queued and no-ops when it surfaces — process() reads q.reqs[k] first and its `if !ok { Forget; return }` guard already tolerates a missing key.

func (*Queue) ConfigureRateLimit

func (q *Queue) ConfigureRateLimit(starts *ratelimit.Window, maxRequeues int, m *telemetry.Metrics)

ConfigureRateLimit installs a sliding-window start budget and wires metric counters into the Queue. Call before Run; nil starts = unlimited. The once-per-window throttle-log guard is derived from starts internally.

func (*Queue) Drain

func (q *Queue) Drain(ctx context.Context)

Drain stops the current term's queue from starting new work and waits for an in-flight investigation to finish, up to ctx's deadline. It is the graceful counterpart to a workCtx cancel (which aborts immediately — used on LOST leadership): on SIGTERM the leader keeps its work context alive and calls Drain so the in-flight investigation can COMPLETE (record its outcome + deliver) before the process exits, instead of being killed mid-flight. A no-op when not running (no leadership / between terms). If the deadline fires first it returns, and the caller's subsequent workCtx cancel aborts the straggler.

func (*Queue) Enqueue

func (q *Queue) Enqueue(r Request)

Enqueue submits a request. Re-enqueuing the same key before it is processed coalesces (latest payload wins). Requests that arrive between terms are held and replayed when the next Run starts.

func (*Queue) Run

func (q *Queue) Run(ctx context.Context)

Run builds a fresh workqueue for this leadership term, replays any pending requests, and consumes until ctx is done — then shuts that queue down. Safe to call again on re-acquired leadership.

type Recall

type Recall struct {
	Catalog              catalog.ScoredSearcher
	MinScore             float64 // similarity floor for the top hit
	MarginGap            float64 // top hit must beat the runner-up by at least this
	SoloFloor            float64 // confident bar when there is only one hit
	RequireWorkloadMatch bool    // true = exact namespace+workload (also disables scopeless matching); false = namespace-level agreement is enough

	// Hybrid, when non-nil AND it has vectors, switches recall to fused BM25+embedding
	// retrieval gated on COSINE similarity (HybridMinScore / HybridMarginGap) instead
	// of the BM25 thresholds above. nil (or no vectors) ⇒ BM25 recall, unchanged.
	Hybrid          catalog.HybridSearcher
	HybridMinScore  float64 // cosine floor for the top hit (hybrid mode)
	HybridMarginGap float64 // cosine margin over the runner-up (hybrid mode)

	// Rerank, when non-nil, REPLACES the BM25-magnitude fire gate (SoloFloor/MarginGap)
	// with an LLM reranker: it ranks the top-K structurally-agreeing candidates and
	// short-circuits only on the reranker's CALIBRATED, corpus-INDEPENDENT match
	// confidence. nil ⇒ the BM25-magnitude gate is used, byte-for-byte unchanged. The
	// structural pre-filter, outcome decay, confirm and verify steps are identical
	// either way — the reranker only changes WHICH signal decides the fire. Opt-in
	// (catalog.instant_recall.rerank). See the Reranker doc for the full rationale.
	Rerank *Reranker

	Outcome      OutcomeStats // optional; nil ⇒ no outcome decay
	OutcomePrior float64      // k — Beta prior strength for decay (e.g. 2.0)
	OutcomeFloor float64      // reject the recall when the outcome factor drops below this (e.g. 0.5)

	Metrics *telemetry.Metrics // optional; nil-safe — instruments are no-op when provider is unset
	Log     *slog.Logger       // optional; nil-safe — log line omitted when unset
}

Recall short-circuits an investigation when the knowledge catalog already has a trustworthy answer for the symptom — skipping the slow, paid ReAct loop. From a wider candidate set it keeps only entries whose stored resource structurally agrees with the alert's workload (a workload-less request agrees only with resource-less entries — the scopeless tier), then requires a clear margin over the runner-up among those agreeing candidates, plus (in the loop) the adversarial verify pass. Confidence is derived from those signals, never asserted — scores are corpus-dependent and a stale hit must not silently replace an investigation.

type RecallDecision added in v0.7.0

type RecallDecision struct {
	// Fired is true when the catalog lookup cleared all three recall gates
	// (structural, margin, outcome-decay) and produced a recalled answer to verify.
	Fired bool
	// Entry is the matched catalog entry path when Fired; empty otherwise.
	Entry string
	// ShortCircuited is true when the recalled answer survived the verify pass and was
	// delivered, skipping the full ReAct loop. When Fired && !ShortCircuited the
	// recalled answer was WITHDRAWN (verify rejected it) and the loop fell through to a
	// full investigation.
	ShortCircuited bool
}

RecallDecision reports what instant recall did on an investigation, so callers (the eval harness, future telemetry) can assert recall behaviour mechanically rather than inferring it from the final finding — which cannot distinguish a recall that was withdrawn by verify from one that never fired. It is emitted at most once per investigation, only when a Recall is configured and consulted.

type RecurrenceGate added in v0.8.0

type RecurrenceGate struct {
	Outcome  RecurrenceStats
	Cooldown time.Duration // 0 disables the gate (default: off, opt-in)
	Log      *slog.Logger  // optional; nil-safe
}

RecurrenceGate suppresses re-investigating a trigger that was conclusively investigated moments ago. Without it, nothing keys on TriggerKey before the paid loop: Alertmanager re-sends a still-firing alert every repeat_interval and a persistently-failing GitOps resource re-emits every informer resync (~10m), each re-running a full investigation that re-delivers the same answer as fresh noise — the recall short-circuit only helps once the KB PR is MERGED, so the human-review window is exactly when the repetition is worst.

The gate is deliberately human-deferential, in both directions:

  • it only suppresses a CONCLUSIVE prior answer (verdict no_action / action_suggested / action_required) — an inconclusive or pre-verdict prior never suppresses, because there is no answer worth not repeating;
  • a standing 👎 on the trigger breaks the cooldown immediately — a human saying "that diagnosis is wrong" re-arms the very next occurrence.

A suppressed occurrence makes no model call, sends no notification, and records no ledger open. (It does still consume a workqueue turn and a rate-limiter slot — the gate runs below Queue.process like its sibling, the recall short-circuit; the same accepted tradeoff for both.) Not recording the open is load-bearing — an open would slide the byTrigger newest-open timestamp and the cooldown would never lapse while the incident keeps firing. Anchored on the last REAL investigation, a persistent failure is re-investigated once per cooldown (with its recurrence count intact) instead of once per resync. The flip side: the ledger keeps no durable record of suppressed firings (only the recurrence_suppressed metric and a log line see them) — a future consumer needing raw firing frequency is the moment to promote suppression to a first-class event kind, not before.

type RecurrenceStats added in v0.8.0

type RecurrenceStats interface {
	Recurrence(triggerKey string) outcome.TriggerRecurrence
}

RecurrenceStats is the per-TriggerKey ledger snapshot the suppression gate reads. *outcome.Ledger satisfies it.

type Reinvestigator

type Reinvestigator struct {
	Forge providers.ReinvestForge
	// Run executes a fresh investigation for the request and returns its findings.
	Run func(ctx context.Context, req Request) (providers.Investigation, error)
	Log *slog.Logger
}

Reinvestigator polls the forge for issues labelled ReinvestigateLabel, re-runs the investigation, posts the fresh findings as a comment, and flips the label to "investigating". This is the outbound-poll re-trigger path that fits RunLore's no-inbound-webhook design.

func (*Reinvestigator) Poll

func (r *Reinvestigator) Poll(ctx context.Context, interval time.Duration)

Poll re-investigates flagged issues on each tick until ctx is done.

type Request

type Request struct {
	Source       Source
	Title        string
	Workload     providers.Workload // optional; zero for alerts without a workload
	Reason       string
	Severity     string // alert severity (alert-like sources); shapes prompt + notification
	Environment  string // deployment environment (prod/staging/…)
	Message      string
	Labels       map[string]string
	Annotations  map[string]string // alert annotations (runbook_url, dashboards, …); surfaced in the seed prompt
	GroupKey     string            // Alertmanager group identity (shared by all alerts in one webhook POST)
	At           time.Time
	Fingerprint  string   // Alertmanager fingerprint (stable firing↔resolved); for outcome attribution
	Fingerprints []string // coalesced batch fingerprints; one open is recorded per entry so every constituent alert's resolve matches
	TriggerKey   string   // deterministic incident identity (alert fingerprint, or failing resource+condition) set at trigger time; threaded to Investigation.TriggerKey for stable dedup across reworded re-investigations (#137)
}

Request is a normalized investigation trigger.

func FromFailureEvent

func FromFailureEvent(fe providers.FailureEvent) Request

FromFailureEvent builds a Request from a GitOps failure.

type Reranker added in v0.8.0

type Reranker struct {
	// Model ranks the candidates. Routed to the verify tier (cheaper/faster) when
	// model.verify is configured, else the main model — mirroring verifyFindings.
	Model providers.ModelProvider
	// Threshold is the calibrated match-confidence bar to short-circuit. Unlike
	// SoloFloor it is corpus-INDEPENDENT (a probability), so the default (0.7) needs no
	// per-cluster tuning.
	Threshold float64
	// K bounds how many structurally-agreeing candidates are ranked in the one call.
	K int
	// MinScore is the trivial retrieval-score floor the cost guard applies before
	// spending the call — read by Recall.lookup, kept here so the whole reranker
	// configuration lives in one place.
	MinScore float64

	Metrics *telemetry.Metrics // optional; nil-safe
	Log     *slog.Logger       // optional; nil-safe
}

Reranker is the opt-in LLM reranking stage of instant recall.

WHY it exists. Query enrichment fixed retrieval RANKING (the correct runbook now ranks #1 on real BM25), but the short-circuit still gated on the ABSOLUTE BM25 magnitude (SoloFloor): an enriched real-corpus top score is ~0.1–1.2, an order of magnitude below the default SoloFloor 4.0, so instant recall only fires where the operator hand-tuned solo_floor down to their corpus's score regime. That gate is fragile — it does not "just work" at the default across clusters. Reranking is the single most impactful retrieval step precisely because it produces CALIBRATED scores: a 0.0–1.0 match confidence that is corpus-INDEPENDENT. So when a Reranker is wired, the fire is gated on that calibrated confidence (RerankThreshold, a stable default) and the BM25 score is demoted to retrieval-ranking-only (pick the top-K).

WHERE it sits. It runs AFTER structural pre-filtering, over only the top-K structurally-agreeing candidates, and BEFORE the "free" short-circuit — so it is the one paid call that decides "which candidate, and confident enough to skip the investigation?". Everything downstream is unchanged: a fired recall still goes through confirmRecall (live-state confrontation) and the adversarial verifyFindings pass. This is a retrieval-time decision, NOT a second verify.

COST discipline. It is one cheap call (~1–2k tokens vs the ~100k a full investigation costs) and only ever runs when retrieval already surfaced a plausible candidate (Recall.lookup applies the MinScore cost guard before calling rank).

FALSE-RECALL guard. A reranker that hallucinates a match is worse than no recall, so rank fails SAFE: a model error, a "no match", or an entry_id outside the candidate set all yield ok=false (fall through to a full investigation), never a wrong recall.

type Source

type Source string

Source identifies what triggered an investigation.

const (
	// SourceAlert means the investigation was triggered by an incident alert.
	SourceAlert Source = "alert"
	// SourceGitOpsFailure means the investigation was triggered by a GitOps failure.
	SourceGitOpsFailure Source = "gitops-failure"
	// SourcePagerDuty means the investigation was triggered by a PagerDuty incident.
	SourcePagerDuty Source = "pagerduty"
)

type StillFailing

type StillFailing func(ctx context.Context, w providers.Workload) (bool, error)

StillFailing reports whether a workload is STILL Ready=False right now. It backs the debounce re-check; the Flux/ArgoCD implementation reads the resource's current Ready condition via GitOpsInspector.ResourceStatus.

type Tool

type Tool interface {
	Name() string
	Description() string
	Schema() string // JSON Schema for the arguments
	Call(ctx context.Context, args string) (string, error)
}

Tool is a model-callable capability used during an investigation.

type WhatChangedTool

type WhatChangedTool struct {
	GitOps providers.GitOpsProvider
}

WhatChangedTool exposes the GitOps "what changed" lens to the model: the change timeline for a namespace/workload, each with its diff.

func (WhatChangedTool) Call

func (t WhatChangedTool) Call(ctx context.Context, args string) (string, error)

Call lists changes for the selector and renders each with its diff.

func (WhatChangedTool) Description

func (t WhatChangedTool) Description() string

Description returns the human-readable tool description advertised to the model.

func (WhatChangedTool) Name

func (t WhatChangedTool) Name() string

Name returns the tool name registered with the model.

func (WhatChangedTool) Schema

func (t WhatChangedTool) Schema() string

Schema returns the JSON Schema for the tool's arguments.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL