Documentation
¶
Overview ¶
Package investigate routes triggers (incident alerts, GitOps failures) into a single async investigation queue. The investigation itself is pluggable via Investigator; LogInvestigator is the read-only placeholder until the ReAct loop lands.
Index ¶
- Constants
- func DrainFailures(ctx context.Context, src <-chan providers.FailureEvent, q Enqueuer, ...)
- type CloudResourceHealthTool
- type CloudWhatChangedTool
- type ControllerLogsTool
- type Debouncer
- type Enqueuer
- type GitOpsStatusTool
- type GitOpsTreeTool
- type Investigator
- type KBSearchTool
- type KubeEventsTool
- type LogInvestigator
- type LoopInvestigator
- type NetworkDropsTool
- type OutcomeStats
- type PodStatusTool
- type QueryLogsTool
- type QueryMetricsTool
- type Queue
- type Recall
- type Reinvestigator
- type Request
- type Source
- type StillFailing
- type Tool
- type WhatChangedTool
Constants ¶
const ReinvestigateLabel = "reinvestigate"
ReinvestigateLabel is the label a human adds to a curated KB issue to ask RunLore to run the investigation again (e.g. after more has happened, or to get a fresh look). RunLore polls for it — it receives no inbound GitHub webhooks.
Variables ¶
This section is empty.
Functions ¶
func DrainFailures ¶
func DrainFailures(ctx context.Context, src <-chan providers.FailureEvent, q Enqueuer, dedup *trigger.Deduper, deb *Debouncer, log *slog.Logger)
DrainFailures forwards GitOps FailureEvents into the queue as investigation requests. It drops dependency-cascade symptoms (so only root failures are investigated) and dedups by workload (a Ready=False resource emits repeated events). A nil dedup disables dedup.
When deb is non-nil with a positive window, each surviving failure is debounced: the investigation is enqueued only after the failure has PERSISTED (re-checked still-Ready=False after the window), filtering reconcile-churn transients. A nil deb (or a zero-window one) enqueues immediately — today's behavior. It returns when src closes or ctx is done.
Types ¶
type CloudResourceHealthTool ¶
type CloudResourceHealthTool struct {
Cloud providers.CloudProvider
}
CloudResourceHealthTool exposes AWS-side resource health (EC2/ASG/EKS) to the model.
func (CloudResourceHealthTool) Description ¶
func (t CloudResourceHealthTool) Description() string
Description returns the tool description.
func (CloudResourceHealthTool) Name ¶
func (t CloudResourceHealthTool) Name() string
Name returns the tool name.
func (CloudResourceHealthTool) Schema ¶
func (t CloudResourceHealthTool) Schema() string
Schema returns the JSON schema for the arguments.
type CloudWhatChangedTool ¶
type CloudWhatChangedTool struct {
Cloud providers.CloudProvider
}
CloudWhatChangedTool exposes recent mutating AWS control-plane events (CloudTrail) as the AWS-layer "what changed" lens — infra/manual changes invisible to GitOps.
func (CloudWhatChangedTool) Description ¶
func (t CloudWhatChangedTool) Description() string
Description returns the tool description.
func (CloudWhatChangedTool) Name ¶
func (t CloudWhatChangedTool) Name() string
Name returns the tool name.
func (CloudWhatChangedTool) Schema ¶
func (t CloudWhatChangedTool) Schema() string
Schema returns the JSON schema for the arguments.
type ControllerLogsTool ¶
ControllerLogsTool reads recent logs from a Flux controller, optionally filtered to a resource name — surfacing WHY a source/object failed to reconcile (auth, reference not found, build error) when resource status alone isn't enough.
func (ControllerLogsTool) Description ¶
func (t ControllerLogsTool) Description() string
Description returns the tool description.
func (ControllerLogsTool) Name ¶
func (t ControllerLogsTool) Name() string
Name returns the tool name.
func (ControllerLogsTool) Schema ¶
func (t ControllerLogsTool) Schema() string
Schema returns the JSON schema for the arguments.
type Debouncer ¶
type Debouncer struct {
// contains filtered or unexported fields
}
Debouncer delays a GitOps-failure investigation until the failure has PERSISTED for a short window, then re-reads the resource's current Ready status and enqueues only if it is still failing. This filters reconcile-churn transients (e.g. a brief HealthCheckCanceled or an ArtifactFailed while a new artifact is mid-checkout) that clear on their own — the kind of misleading status that previously produced a confident-but-wrong root cause.
A zero window disables the wait and re-check entirely: Debounce enqueues immediately, preserving the original fire-on-every-failure behavior.
func NewDebouncer ¶
NewDebouncer builds a Debouncer. A zero window disables debouncing (immediate enqueue). check is consulted only when window > 0.
func (*Debouncer) Debounce ¶
Debounce decides whether to enqueue r. With window 0 it enqueues immediately. Otherwise it waits window, re-checks the workload's current Ready status, and enqueues only if it is still failing — dropping (debug-logged) any failure that cleared or flipped to a different transient within the window. It blocks for the wait; callers run it in a goroutine per failure event.
type Enqueuer ¶
type Enqueuer interface {
Enqueue(r Request)
}
Enqueuer accepts investigation requests.
type GitOpsStatusTool ¶
type GitOpsStatusTool struct {
Inspector providers.GitOpsInspector
}
GitOpsStatusTool exposes a GitOps/Kubernetes resource's status (a Flux Ready condition or an Argo CD Application's health/sync), key spec refs, and recent Events — so the model can learn WHY a resource is failing, not just that it is.
func (GitOpsStatusTool) Description ¶
func (t GitOpsStatusTool) Description() string
Description returns the tool description.
func (GitOpsStatusTool) Schema ¶
func (t GitOpsStatusTool) Schema() string
Schema returns the JSON schema for the arguments.
type GitOpsTreeTool ¶
type GitOpsTreeTool struct {
Inspector providers.GitOpsInspector
}
GitOpsTreeTool walks a resource's dependency graph (Flux dependsOn/sourceRef, or an Argo CD Application's managed-resource tree) and renders it, so the model can find the ROOT failing resource behind a cascade (the not-Ready/Degraded or missing node), not just the downstream symptom.
func (GitOpsTreeTool) Description ¶
func (t GitOpsTreeTool) Description() string
Description returns the tool description.
func (GitOpsTreeTool) Schema ¶
func (t GitOpsTreeTool) Schema() string
Schema returns the JSON schema for the arguments.
type Investigator ¶
Investigator runs an investigation for a Request.
type KBSearchTool ¶
KBSearchTool lets the model search the OKF knowledge catalog (runbooks, past incidents) to ground its reasoning.
func (KBSearchTool) Description ¶
func (t KBSearchTool) Description() string
Description returns the tool description.
func (KBSearchTool) Schema ¶
func (t KBSearchTool) Schema() string
Schema returns the JSON schema for the arguments.
type KubeEventsTool ¶
type KubeEventsTool struct {
Kube providers.KubeReader
}
KubeEventsTool surfaces causes that live in the Kubernetes event stream rather than logs or status — FailedScheduling (Insufficient cpu/memory), FailedMount, FailedAttachVolume, BackOff, etc.
func (KubeEventsTool) Description ¶
func (t KubeEventsTool) Description() string
Description returns the tool description.
func (KubeEventsTool) Schema ¶
func (t KubeEventsTool) Schema() string
Schema returns the JSON schema for the arguments.
type LogInvestigator ¶
LogInvestigator is the read-only placeholder: it logs the request it would investigate. Replaced by the ReAct loop in a later phase.
func (LogInvestigator) Investigate ¶
func (l LogInvestigator) Investigate(_ context.Context, r Request) error
Investigate logs the request.
type LoopInvestigator ¶
type LoopInvestigator struct {
Model providers.ModelProvider
Tools []Tool
Log *slog.Logger
MaxSteps int
OnComplete func(providers.Investigation) // delivery hook (Slack/Matrix later)
Actions *action.Policy // autonomy ladder; nil/off = read-only findings only
Recall *Recall // optional: short-circuit on a high-confidence catalog hit
Verify bool // run an adversarial review of root causes before delivery
// VerifyModel optionally routes the adversarial verify pass to a cheaper/faster
// model. nil ⇒ the verify pass reuses Model. Verify itself always runs.
VerifyModel providers.ModelProvider
// Timeout bounds a single investigation end-to-end (recall + every model/tool
// call, including a hung git clone/patch). 0 disables it. On expiry the loop
// delivers a synthetic timeout result rather than starving the queue worker.
Timeout time.Duration
// Cost controls (0 means disabled/unlimited):
MaxToolOutputBytes int // truncate tool results larger than this before adding to history
MaxTokensPerInvestigation int // inject a budget-nudge message when the estimated token count exceeds this
// Observability — nil-safe; no-op when telemetry is disabled.
Metrics *telemetry.Metrics
ModelProvider string // label for model_requests/model_request_duration metrics (e.g. "anthropic")
}
LoopInvestigator is the ReAct investigation loop: it drives a ModelProvider with tools, feeds tool results back, and finishes when the model calls submit_findings (or MaxSteps is reached). The completed investigation is handed to OnComplete.
func (*LoopInvestigator) Investigate ¶
func (li *LoopInvestigator) Investigate(ctx context.Context, req Request) error
Investigate runs the loop for a request. It implements Investigator.
type NetworkDropsTool ¶
type NetworkDropsTool struct {
Network providers.NetworkProvider
}
NetworkDropsTool lets the model list recently denied/dropped network flows from the configured (pluggable, CNI-agnostic) network-flow source — surfacing NetworkPolicy denials, firewall/security-group rejects, and connectivity failures.
func (NetworkDropsTool) Description ¶
func (t NetworkDropsTool) Description() string
Description returns the tool description.
func (NetworkDropsTool) Schema ¶
func (t NetworkDropsTool) Schema() string
Schema returns the JSON schema for the arguments.
type OutcomeStats ¶
OutcomeStats reports per-entry recall outcomes for confidence decay. *outcome.Ledger satisfies it.
type PodStatusTool ¶
type PodStatusTool struct {
Kube providers.KubeReader
}
PodStatusTool surfaces pod-level failures (CreateContainerConfigError, ImagePullBackOff, CrashLoopBackOff, …) that never reach logs because the container never started — the gap the GitOps/logs/metrics tools can't fill.
func (PodStatusTool) Description ¶
func (t PodStatusTool) Description() string
Description returns the tool description.
func (PodStatusTool) Schema ¶
func (t PodStatusTool) Schema() string
Schema returns the JSON schema for the arguments.
type QueryLogsTool ¶
type QueryLogsTool struct {
Logs providers.LogsProvider
}
QueryLogsTool lets the model query logs (LogsQL) over a recent window.
func (QueryLogsTool) Description ¶
func (t QueryLogsTool) Description() string
Description returns the tool description.
func (QueryLogsTool) Schema ¶
func (t QueryLogsTool) Schema() string
Schema returns the JSON schema for the arguments.
type QueryMetricsTool ¶
type QueryMetricsTool struct {
Metrics providers.MetricsProvider
}
QueryMetricsTool lets the model run PromQL instant queries (saturation, error rates, health) against the metrics backend.
func (QueryMetricsTool) Description ¶
func (t QueryMetricsTool) Description() string
Description returns the tool description.
func (QueryMetricsTool) Schema ¶
func (t QueryMetricsTool) Schema() string
Schema returns the JSON schema for the arguments.
type Queue ¶
type Queue struct {
// contains filtered or unexported fields
}
Queue is a rate-limiting investigation queue: duplicate triggers coalesce, and failed investigations are retried with exponential backoff. A fresh workqueue is built per Run (leadership term), so the queue recovers after a lost-then- re-acquired leadership instead of staying permanently shut down.
func NewQueue ¶
func NewQueue(inv Investigator, log *slog.Logger) *Queue
NewQueue builds an investigation queue. The workqueue itself is created per Run.
func (*Queue) ConfigureRateLimit ¶
ConfigureRateLimit installs a sliding-window start budget and wires metric counters into the Queue. Call before Run; nil starts = unlimited. The once-per-window throttle-log guard is derived from starts internally.
func (*Queue) Drain ¶
Drain stops the current term's queue from starting new work and waits for an in-flight investigation to finish, up to ctx's deadline. It is the graceful counterpart to a workCtx cancel (which aborts immediately — used on LOST leadership): on SIGTERM the leader keeps its work context alive and calls Drain so the in-flight investigation can COMPLETE (record its outcome + deliver) before the process exits, instead of being killed mid-flight. A no-op when not running (no leadership / between terms). If the deadline fires first it returns, and the caller's subsequent workCtx cancel aborts the straggler.
type Recall ¶
type Recall struct {
Catalog catalog.ScoredSearcher
MinScore float64 // similarity floor for the top hit
MarginGap float64 // top hit must beat the runner-up by at least this
SoloFloor float64 // confident bar when there is only one hit
RequireWorkloadMatch bool // true = exact namespace+workload; false = namespace-level agreement is enough
// Hybrid, when non-nil AND it has vectors, switches recall to fused BM25+embedding
// retrieval gated on COSINE similarity (HybridMinScore / HybridMarginGap) instead
// of the BM25 thresholds above. nil (or no vectors) ⇒ BM25 recall, unchanged.
Hybrid catalog.HybridSearcher
HybridMinScore float64 // cosine floor for the top hit (hybrid mode)
HybridMarginGap float64 // cosine margin over the runner-up (hybrid mode)
Outcome OutcomeStats // optional; nil ⇒ no outcome decay
OutcomePrior float64 // k — Beta prior strength for decay (e.g. 2.0)
OutcomeFloor float64 // reject the recall when the outcome factor drops below this (e.g. 0.5)
Metrics *telemetry.Metrics // optional; nil-safe — instruments are no-op when provider is unset
Log *slog.Logger // optional; nil-safe — log line omitted when unset
}
Recall short-circuits an investigation when the knowledge catalog already has a trustworthy answer for the symptom — skipping the slow, paid ReAct loop. From a wider candidate set it keeps only entries whose stored resource structurally agrees with the alert's workload, then requires a clear margin over the runner-up among those agreeing candidates, plus (in the loop) the adversarial verify pass. Confidence is derived from those signals, never asserted — scores are corpus-dependent and a stale hit must not silently replace an investigation.
type Reinvestigator ¶
type Reinvestigator struct {
Forge providers.ReinvestForge
// Run executes a fresh investigation for the request and returns its findings.
Run func(ctx context.Context, req Request) (providers.Investigation, error)
Log *slog.Logger
}
Reinvestigator polls the forge for issues labelled ReinvestigateLabel, re-runs the investigation, posts the fresh findings as a comment, and flips the label to "investigating". This is the outbound-poll re-trigger path that fits RunLore's no-inbound-webhook design.
type Request ¶
type Request struct {
Source Source
Title string
Workload providers.Workload // optional; zero for alerts without a workload
Reason string
Message string
Labels map[string]string
At time.Time
Fingerprint string // Alertmanager fingerprint (stable firing↔resolved); for outcome attribution
Fingerprints []string // coalesced batch fingerprints; one open is recorded per entry so every constituent alert's resolve matches
TriggerKey string // deterministic incident identity (alert fingerprint, or failing resource+condition) set at trigger time; threaded to Investigation.TriggerKey for stable dedup across reworded re-investigations (#137)
}
Request is a normalized investigation trigger.
func FromFailureEvent ¶
func FromFailureEvent(fe providers.FailureEvent) Request
FromFailureEvent builds a Request from a GitOps failure.
func FromIncident ¶
FromIncident builds a Request from a matched incident alert.
type StillFailing ¶
StillFailing reports whether a workload is STILL Ready=False right now. It backs the debounce re-check; the Flux/ArgoCD implementation reads the resource's current Ready condition via GitOpsInspector.ResourceStatus.
type Tool ¶
type Tool interface {
Name() string
Description() string
Schema() string // JSON Schema for the arguments
Call(ctx context.Context, args string) (string, error)
}
Tool is a model-callable capability used during an investigation.
type WhatChangedTool ¶
type WhatChangedTool struct {
GitOps providers.GitOpsProvider
}
WhatChangedTool exposes the GitOps "what changed" lens to the model: the change timeline for a namespace/workload, each with its diff.
func (WhatChangedTool) Description ¶
func (t WhatChangedTool) Description() string
Description returns the human-readable tool description advertised to the model.
func (WhatChangedTool) Name ¶
func (t WhatChangedTool) Name() string
Name returns the tool name registered with the model.
func (WhatChangedTool) Schema ¶
func (t WhatChangedTool) Schema() string
Schema returns the JSON Schema for the tool's arguments.
Source Files