investigate

package
v1.26.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 13, 2026 License: Apache-2.0 Imports: 7 Imported by: 0

Documentation

Overview

Package investigate provides types and helpers for correlating Prometheus metrics, Jaeger traces, and code symbols into a ranked list of root-cause hypotheses for the debug_investigate MCP tool.

Pure package: no HTTP, no database, no external state. Ranking is delegated to go-kit/rerank (LinearMinMax fusion). The MCP tool layer (cmd/go-code/tool_debug_investigate.go) wires it to live data sources and may inject LLM-derived Confidence labels before passing through RankHypotheses.

internal/investigate/fleet_report.go

internal/investigate/lifecycle.go

internal/investigate/prompt.go

Index

Constants

View Source 
const (
	ConfidenceLow    = score.ConfidenceLow
	ConfidenceMedium = score.ConfidenceMedium
	ConfidenceHigh   = score.ConfidenceHigh
)

Re-exported confidence labels — identical to the score package consts.

View Source 
const (
	BlastRadiusNone   = "none"
	BlastRadiusLow    = "low"
	BlastRadiusMedium = "medium"
	BlastRadiusHigh   = "high"
)

BlastRadius enum values for ImpactInfo.BlastRadius.

View Source 
const (
	HypothesisSourceSpan      = "span"
	HypothesisSourceHintMatch = "hint_match"
	HypothesisSourceAlert     = "alert"
	// HypothesisSourceUpstream marks a hypothesis generated by the upstream
	// callgraph walk (Sprint B2). These hypotheses are callers of a Tier-3
	// span-attributed hypothesis and represent root-cause candidates upstream
	// of the observed metric attribution site.
	HypothesisSourceUpstream = "upstream_caller"
	// HypothesisSourceDownstream marks a hypothesis generated by the downstream
	// callgraph walk (Sprint B4). These hypotheses are callees of the top-1
	// span-attributed hypothesis and represent root-cause candidates downstream
	// of the observed metric attribution site (e.g. an event-loop dispatcher
	// that calls the handler that actually contains the bug).
	HypothesisSourceDownstream = "downstream_callee"
)

Source values for Hypothesis.Source (γ.C.3).

Variables

This section is empty.

Functions

func BuildSystemPrompt 

func BuildSystemPrompt(c PromptContext) string

BuildSystemPrompt assembles the LLM correlation prompt with hard constraints against hallucination. Layout: role + ground truth + reasoning rules + output schema.

func OperationToFuncName 

func OperationToFuncName(op string) string

OperationToFuncName extracts a Go-friendly function name from a Jaeger span operation name. Handles three shapes:

  • gRPC: "/pkg.Service/Method" → "Method"
  • HTTP: "GET /api/v1/users" → "users" (last non-empty path segment)
  • Plain: "ProcessMessage" or "(*Server).Handle" → "ProcessMessage" / "Handle"

Returns empty string if no meaningful identifier can be extracted. The output is the symbol name to feed into compound.FindSymbol — best-effort, not guaranteed to match an existing function.

Types

type AlertViolation 

type AlertViolation struct {
	AlertName   string `json:"alert_name"`
	Severity    string `json:"severity"`
	Service     string `json:"service"`
	Summary     string `json:"summary"`
	Description string `json:"description,omitempty"`
	Runbook     string `json:"runbook,omitempty"`
	ActiveAt    string `json:"active_at,omitempty"`
}

AlertViolation captures a firing Prometheus alert that matches the investigated service. These represent constant-state invariant violations that Phase 4 spike detection misses because there is no delta — the ratio never changes, it is just always wrong.

type ConfidenceLevel 

type ConfidenceLevel = score.ConfidenceLevel

ConfidenceLevel is re-exported from go-kit/score so callers of the investigate package don't need to import score directly. The underlying type is identical — investigate.ConfidenceLevel == score.ConfidenceLevel.

func ConfidenceFromScore 

func ConfidenceFromScore(s float64) ConfidenceLevel

ConfidenceFromScore is re-exported as a thin wrapper around score.ConfidenceFromScore. Defaults: <0.2 → low, <0.7 → medium, ≥0.7 → high.

type Diagnostics 

type Diagnostics struct {
	MetricsQueried          int      `json:"metrics_queried"`
	TracesFetched           int      `json:"traces_fetched"`
	SpansAnalyzed           int      `json:"spans_analyzed"`
	SymbolsTouched          int      `json:"symbols_touched"`
	AlertsQueried           int      `json:"alerts_queried,omitempty"`
	LogsFetched             int      `json:"logs_fetched,omitempty"`
	HypothesesDroppedAsDead int      `json:"hypotheses_dropped_as_dead,omitempty"`
	LearningsPersisted      bool     `json:"learnings_persisted,omitempty"`
	LLMCacheHit             bool     `json:"llm_cache_hit,omitempty"`
	LLMSkippedReason        string   `json:"llm_skipped_reason,omitempty"`
	Warnings                []string `json:"warnings,omitempty"`
}

Diagnostics records counters from the investigation run for transparency.

type FleetDiffRow 

type FleetDiffRow struct {
	Image            string    `json:"image"`
	Status           string    `json:"status"` // "Match" / "TagDrift" / ... (from fleet.DiffStatus)
	PinnedTag        string    `json:"pinned_tag,omitempty"`
	RuntimeTag       string    `json:"runtime_tag,omitempty"`
	PinnedDigest     string    `json:"pinned_digest,omitempty"`
	RuntimeDigest    string    `json:"runtime_digest,omitempty"`
	Source           string    `json:"source,omitempty"`    // pinned source file
	Container        string    `json:"container,omitempty"` // runtime container name
	Service          string    `json:"service,omitempty"`   // compose service (either side)
	State            string    `json:"state,omitempty"`     // runtime state
	StartedAt        time.Time `json:"started_at,omitempty"`
	Explanation      string    `json:"explanation,omitempty"`
	Unresolved       string    `json:"unresolved,omitempty"`        // pinned-side parse caveat
	ChangelogSummary string    `json:"changelog_summary,omitempty"` // first commit subject from upstream changelog, if available
}

type FleetReport 

type FleetReport struct {
	// Single-host fields (backward compat; also populated in multi-host).
	Target  string         `json:"target"`
	Diffs   []FleetDiffRow `json:"diffs"`
	Error   string         `json:"error,omitempty"`
	Summary string         `json:"summary,omitempty"` // top-20-ish line for LLM

	// Multi-host fields (populated when ≥2 hosts probed).
	Targets       []FleetTargetRow       `json:"targets,omitempty"`
	SiblingDrifts []FleetSiblingDriftRow `json:"sibling_drifts,omitempty"`
}

FleetReport carries the output of runFleetVersionsPhase for an investigation. Shape mirrors cmd/go-code's FleetVersionsOutput but lives in this package to keep internal/fleet from depending on cmd/.

Note: FleetReport does NOT directly reference internal/fleet types — fields are flattened to plain strings/ints so this package stays import-free of /fleet. Translation happens in runFleetVersionsPhase.

Multi-host layout: Targets holds one FleetTargetRow per probed host; SiblingDrifts carries cross-host drift rows (populated when ≥2 hosts). Single-host layout (backward compat): Target/Diffs/Error/Summary fields are also populated so existing callers continue to work.

type FleetSiblingDriftRow 

type FleetSiblingDriftRow struct {
	Image    string                `json:"image"`
	Variants []FleetSiblingVariant `json:"variants"`
}

FleetSiblingDriftRow is the flattened mirror of fleet.SiblingDriftRow. Lives here so internal/investigate stays import-free of internal/fleet.

type FleetSiblingVariant 

type FleetSiblingVariant struct {
	Target    string `json:"target"`
	Tag       string `json:"tag"`
	Digest    string `json:"digest,omitempty"`
	Container string `json:"container,omitempty"`
	State     string `json:"state,omitempty"`
}

FleetSiblingVariant is one host's view of a drifting image.

type FleetTargetRow 

type FleetTargetRow struct {
	Target string         `json:"target"`
	Diffs  []FleetDiffRow `json:"diffs"`
	Error  string         `json:"error,omitempty"`
}

FleetTargetRow is one host's result in a multi-host FleetReport.

type HistoricalIncident 

type HistoricalIncident struct {
	Repo      string `json:"repo"`
	Symbol    string `json:"symbol"`
	RiskLevel string `json:"risk_level,omitempty"`
	Flag      string `json:"flag,omitempty"`
	Note      string `json:"note,omitempty"`
}

HistoricalIncident is a past investigation record retrieved from the learnings store.

type Hypothesis 

type Hypothesis struct {
	Subject string `json:"subject"`
	File    string `json:"file,omitempty"`
	Line    int    `json:"line,omitempty"`
	// EndLine is the last line of the function body (1-based, inclusive).
	// Populated from parser.Symbol.EndLine in Tier-3 callgraph path.
	// For Tier-1 (OTEL code.* tags) it equals Line (single attribution point).
	EndLine int `json:"end_line,omitempty"`

	SpanCount    int     `json:"span_count"`
	AnomalyScore float64 `json:"anomaly_score"`

	// Confidence is a bucketed label (low/medium/high). Populated by
	// RankHypotheses ONLY if caller left it empty — caller-set values
	// (typically by the LLM correlate step which weighs evidence holistically)
	// are preserved. To force the heuristic, leave this zero on input.
	Confidence ConfidenceLevel `json:"confidence"`

	Impact     *ImpactInfo     `json:"impact,omitempty"`
	SymbolBody *SymbolBodyInfo `json:"symbol_body,omitempty"`

	EvidenceLinks []string    `json:"evidence_links,omitempty"`
	NextChecks    []NextCheck `json:"next_checks,omitempty"`

	// Source tracks the origin of this hypothesis: span | hint_match | alert | "" (backwards compat).
	Source string `json:"source,omitempty"`

	// FusedScore is the multi-signal combined score (Phase γ.D). Set after
	// all enrichment phases. Sortable. Higher = more likely root cause.
	FusedScore float64 `json:"fused_score,omitempty"`

	// SignalBreakdown shows individual normalized signal values that fed
	// into FusedScore. Useful for operator triage and debugging the rank.
	SignalBreakdown map[string]float64 `json:"signal_breakdown,omitempty"`

	// RecentChange embeds a recent git diff for the hypothesis file (Phase γ.D).
	// Set for top-1 hypothesis only when a repo path is available.
	RecentChange *RecentChange `json:"recent_change,omitempty"`

	// BodySource is the raw source text of the function body (Sprint B1).
	// Populated for top-3 hypotheses after FusionRank. Enables LLM to
	// reason about the code rather than just its metadata.
	// Empty when file read fails or File is empty (no symbol resolved).
	BodySource string `json:"body_source,omitempty"`
}

Hypothesis is one candidate root-cause site.

func RankHypotheses 

func RankHypotheses(h []Hypothesis) []Hypothesis

RankHypotheses returns a copy of h sorted by fused score (descending). Fusion: rerank.LinearMinMax with equal weights on SpanCount and AnomalyScore signals. Both signals are MinMax-normalised to [0,1] within the input set, summed (weights 1:1), resulting score is in [0,2].

Confidence policy: caller-set values are PRESERVED; empty Confidence is filled via ConfidenceFromScore on the fused score. The LLM correlate step (Task 13) populates Confidence based on holistic evidence and survives.

Stable: equal scores preserve input order (rerank.LinearMinMax guarantees).

Empty input → nil. Single-element input → that element with Confidence filled from heuristic on fused=0 (degenerate; LLM step typically produces meaningful labels in real flows).

type ImpactInfo 

type ImpactInfo struct {
	DirectCallers int     `json:"direct_callers"`
	TotalAffected int     `json:"total_affected"`
	BlastRadius   string  `json:"blast_radius,omitempty"` // BlastRadiusNone | BlastRadiusLow | BlastRadiusMedium | BlastRadiusHigh
	RiskScore     float64 `json:"risk_score,omitempty"`
}

ImpactInfo captures blast-radius data from impact.Analyze for a hypothesis.

type InvestigationResult 

type InvestigationResult struct {
	Service             string               `json:"service"`
	Range               TimeRange            `json:"range"`
	StartedAt           time.Time            `json:"started_at"`
	FinishedAt          time.Time            `json:"finished_at"`
	Hypotheses          []Hypothesis         `json:"hypotheses"`
	LLMSummary          string               `json:"llm_summary,omitempty"`
	MetricSpikes        []MetricSpike        `json:"metric_spikes,omitempty"`
	AlertViolations     []AlertViolation     `json:"alert_violations,omitempty"`
	HintKind            string               `json:"hint_kind,omitempty"`
	LogExcerpts         []LogExcerpt         `json:"log_excerpts,omitempty"`
	HistoricalIncidents []HistoricalIncident `json:"historical_incidents,omitempty"`
	Diagnostics         Diagnostics          `json:"diagnostics"`

	// RuntimeVersions, when non-nil, carries the Phase 7 deployed-image
	// diff against pinned source. Nil = phase skipped or host not provided.
	RuntimeVersions *FleetReport `json:"runtime_versions,omitempty"`
}

InvestigationResult is the final tool output.

Time fields:

  • StartedAt / FinishedAt: wall-clock — when this investigation started and finished executing.
  • Range: the *data* window the investigation analysed (Prometheus query range, Jaeger trace search range). Independent of wall-clock.

type InvestigationStore 

type InvestigationStore struct {
	// contains filtered or unexported fields
}

InvestigationStore deduplicates concurrent debug_investigate calls and stores results for polling. Key: service + range + repo. Thread-safe.

func NewInvestigationStore 

func NewInvestigationStore() *InvestigationStore

NewInvestigationStore builds an empty store.

func (*InvestigationStore) Fail 

func (s *InvestigationStore) Fail(service string, start, end time.Time, repo string, errMsg string)

Fail marks the investigation failed with an error message. Last writer wins when called concurrently.

func (*InvestigationStore) Finish 

func (s *InvestigationStore) Finish(service string, start, end time.Time, repo string, res *InvestigationResult)

Finish marks the investigation done and stores the result. Last writer wins when called concurrently.

func (*InvestigationStore) Get 

func (s *InvestigationStore) Get(service string, start, end time.Time, repo string) (*State, bool)

Get returns the State for (service, range, repo) or (nil, false) if absent.

func (*InvestigationStore) Start 

func (s *InvestigationStore) Start(service string, start, end time.Time, repo string) (*State, bool)

Start either creates a new running investigation or returns the existing one. fresh=true on first call for this (service, range, repo), false if already running or completed (dedup).

type LogExcerpt 

type LogExcerpt struct {
	Ts    string `json:"ts,omitempty"`
	Level string `json:"level,omitempty"`
	Msg   string `json:"msg,omitempty"`
	Raw   string `json:"raw,omitempty"`
}

LogExcerpt is a single log line from the dozor sidecar, attached to the investigation result when dozor is configured.

type MetricSpike 

type MetricSpike struct {
	Kind       string  `json:"kind"`        // failure | latency | saturation
	MetricName string  `json:"metric_name"` // full Prometheus metric name
	Labels     string  `json:"labels"`      // label-set rendered for human reading
	Ratio      float64 `json:"ratio"`       // window_max / baseline_max
	Score      float64 `json:"score"`       // bucketed anomaly score 0..1
}

MetricSpike captures a single metric (failure / latency / saturation) showing anomaly above baseline.

type NextCheck 

type NextCheck struct {
	Tool string            `json:"tool"`           // Tool must be non-empty (the MCP tool name to invoke). e.g. "understand", "code_health"
	Args map[string]string `json:"args,omitempty"` // named arguments for the tool
}

NextCheck is a structured follow-up recommendation for the operator or agent. It names the tool to call and the arguments to pass.

type OperationInfo 

type OperationInfo struct {
	Operation     string // span operation name (key)
	Count         int    // CUMULATIVE — total spans for this op in window
	HTTPRoute     string // FIRST-SEEN — http.route tag (axum MatchedPath)
	HTTPMethod    string // FIRST-SEEN — http.method tag
	CodeFilepath  string // FIRST-SEEN — code.filepath OR code.file.path (absolute path inside container)
	CodeLineno    int    // FIRST-SEEN — code.lineno OR code.line.number
	CodeNamespace string // FIRST-SEEN — code.namespace OR code.module.name (e.g. Rust module path)
	CodeFunction  string // FIRST-SEEN — code.function OR code.function.name (e.g. Go method like (*T).Method)
}

OperationInfo aggregates per-operation context from a service's spans. Used by Phase 3 (runSymbolsPhase) to feed symbol resolution with both span counts and OTEL semantic-convention attributes. When code.* tags are present, Phase 3 resolves directly to file:line without needing a callgraph.

type PromptContext 

type PromptContext struct {
	Service           string
	AvailableMetrics  []string // truncated to first 80 if longer
	AvailableServices []string
	OperationsSeen    []string // top operations from traces
	FiringAlerts      []string // alert names currently firing for this service
}

PromptContext is the ground-truth payload injected into the LLM system prompt. Pattern source: Zagalin Grafana plugin — list real metric names and trace services so the LLM cannot hallucinate names.

type RecentChange 

type RecentChange struct {
	File  string `json:"file"`
	Since string `json:"since"`          // e.g. "2026-04-09" — 30 days ago
	Diff  string `json:"diff,omitempty"` // unified diff, capped at maxLines
}

RecentChange captures a recent git diff for a hypothesis file.

type State 

type State struct {
	// contains filtered or unexported fields
}

State is one investigation's transient state. All field access goes through Lock/RLock — direct field reads from outside this package are not safe.

func (*State) Error 

func (s *State) Error() string

Error returns the failure message (only valid when Status() == StatusFailed).

func (*State) Result 

func (s *State) Result() *InvestigationResult

Result returns the stored result (only valid when Status() == StatusDone).

func (*State) StartedAt 

func (s *State) StartedAt() time.Time

StartedAt returns when this investigation was created.

func (*State) Status 

func (s *State) Status() Status

Status returns the current lifecycle status.

func (*State) UpdatedAt 

func (s *State) UpdatedAt() time.Time

UpdatedAt returns when this state was last mutated.

type Status 

type Status string

Status represents the lifecycle of an investigation. Only the three exported consts (StatusRunning/Done/Failed) are valid; callers must not construct Status values directly. 

const (
	StatusRunning Status = "running"
	StatusDone    Status = "done"
	StatusFailed  Status = "failed"
)

type SymbolBodyInfo 

type SymbolBodyInfo struct {
	ErrorExits      int  `json:"error_exits"`
	HasDeferCleanup bool `json:"has_defer_cleanup,omitempty"`
	HasTODO         bool `json:"has_todo,omitempty"`
}

SymbolBodyInfo captures structural analysis from compound.AnalyzeBody for a hypothesis.

type TimeRange 

type TimeRange struct {
	Start time.Time `json:"start"`
	End   time.Time `json:"end"`
}

TimeRange is the [Start, End] window the investigation covered.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.