agent

type AIBundle ¶ added in v1.4.0

type AIBundle struct {
	Router      *router.Router
	Detect      core.AIAgent // kind=AITaskDetect
	Analyze     core.AIAgent // kind=AITaskAnalyze, built when AI.Enable is true
	Cache       *ai.ResultCache
	Rate        *ai.RateLimiter
	AnalyzeRate *ai.RateLimiter // separate hourly cap for analyze
	// Runbooks is the runbook corpus manager shared by the find_runbook
	// read path and the admin runbooks UI (upload/list/delete). Nil when
	// storage is unavailable. Present even without embeddings so operators
	// can manage the corpus before configuring an embedding model.
	Runbooks *runbook.Manager
}

AIBundle bundles every AI-side dependency. All fields are nil-safe: when AI is disabled the worker accepts a zero bundle and falls back to "dry detect" (classify, log, do not emit).

Router exposes the typed task dispatcher to non-worker consumers (admin endpoints, future analyze controller). The worker keeps using Detect + Cache + Rate directly so its per-outcome logging (dry / cache / quota / ai_error / emitted) stays explicit.

func BuildAIs(cfg config.AgentConfig, catalog *Catalog, store storage.Provider, httpClient *http.Client) AIBundle

type AIProviderResolver ¶ added in v1.4.4

type AIProviderResolver interface {
	EffectiveProvider(ctx context.Context) (provider string, ok bool)
}

AIProviderResolver is an OPTIONAL extension of AISettingsResolver: a registered resolver that ALSO implements it can override the model PROVIDER (openai | deepseek | qwen | ollama | claude | gemini) at runtime, so an operator's provider change rebuilds the agent's model on its next run without a process restart. ok=false ⇒ no opinion (use the configured provider). A resolver that does not implement this interface simply has no provider opinion — composition is additive, so the existing X27 key/enabled resolver keeps working unchanged. OSS registers nothing.

type AISettingsResolver ¶ added in v1.4.4

type AISettingsResolver interface {
	EffectiveKey(ctx context.Context) (key string, ok bool)
	EffectiveEnabled(ctx context.Context) (enabled bool, ok bool)
}

AISettingsResolver resolves effective AI settings at runtime. ok=false on a method means "no opinion" -> caller uses the static YAML ai config.

type BrainFactory ¶ added in v1.4.4

type BrainFactory func(name string, options map[string]any) (core.SignalLearner, core.SignalDetector, error)

BrainFactory builds the Learner + Detector pair for one configured source instance from its instance name and the generic per-source options block (the decoded YAML under the source's `options:` key, see config.AgentSourceConfig.Options). A factory decodes the map into whatever concrete config it owns. Returning an error makes the worker fall back to the log brain for that source (logged, non-fatal).

type Catalog ¶

type Catalog struct {
	// contains filtered or unexported fields
}

Catalog is the in-memory + on-disk pattern store.

All public methods are safe for concurrent use. Disk persistence is debounced — calls to MarkDirty() set a flag that the agent worker flushes at most once per `persist_interval`.

func LoadCatalog(store storage.Provider) (*Catalog, error)

func (c *Catalog) All() []*Pattern

func (c *Catalog) AllServices() map[string]ServiceInfo

func (c *Catalog) Delete(patternID string) bool

func (c *Catalog) Dirty() bool

func (c *Catalog) EndServiceGrace(name string) bool

func (c *Catalog) Get(id string) *Pattern

func (c *Catalog) IsServiceInGrace(name string, graceDuration time.Duration) bool

func (c *Catalog) Label(patternID, verdict string, tags []string) bool

func (c *Catalog) Len() int

func (c *Catalog) MarkKnown(patternID string) bool

func (c *Catalog) Persist() error

func (c *Catalog) RegisterService(name string) bool

func (c *Catalog) RestartServiceGrace(name string) bool

func (c *Catalog) Upsert(patternID, template, source string, tickCount int, alpha float64, ruleName, service string) *Pattern

type CatalogEdit ¶ added in v1.4.4

type CatalogEdit struct {
	Kind CatalogEditKind
	// PatternID identifies the pattern for the label/delete/mark-known edits.
	PatternID string
	// Verdict is the operator verdict for a label edit ("" leaves it
	// unchanged, matching Catalog.Label).
	Verdict string
	// Tags are the operator tags for a label edit (nil leaves them unchanged,
	// matching Catalog.Label).
	Tags []string
	// Service identifies the service for the grace edits.
	Service string
}

CatalogEdit is one operator mutation, modelled on the Catalog curation method set so a CatalogStore can apply it without importing the controllers. Kind selects which fields are meaningful:

• CatalogEditLabel → PatternID, Verdict, Tags
• CatalogEditDelete → PatternID
• CatalogEditMarkKnown → PatternID
• CatalogEditEndServiceGrace → Service
• CatalogEditRestartServiceGrace→ Service

type CatalogEditKind ¶ added in v1.4.4

type CatalogEditKind string

CatalogEditKind discriminates which operator mutation a CatalogEdit carries. The set mirrors the Catalog curation method set one-for-one.

const (
	// CatalogEditLabel carries Label(PatternID, Verdict, Tags).
	CatalogEditLabel CatalogEditKind = "label"
	// CatalogEditDelete carries Delete(PatternID).
	CatalogEditDelete CatalogEditKind = "delete"
	// CatalogEditMarkKnown carries MarkKnown(PatternID).
	CatalogEditMarkKnown CatalogEditKind = "mark_known"
	// CatalogEditEndServiceGrace carries EndServiceGrace(Service).
	CatalogEditEndServiceGrace CatalogEditKind = "end_service_grace"
	// CatalogEditRestartServiceGrace carries RestartServiceGrace(Service).
	CatalogEditRestartServiceGrace CatalogEditKind = "restart_service_grace"
)

type CatalogStore ¶ added in v1.4.4

type CatalogStore interface {
	// Load returns the working-set this consumer wants the Catalog to hold in
	// memory after boot. Either returned map may be nil to mean "empty".
	Load() (patterns map[string]*Pattern, services map[string]*ServiceInfo, err error)

	// Persist writes the Catalog's current in-memory working set.
	Persist(patterns map[string]*Pattern, services map[string]*ServiceInfo) error

	// Snapshot returns the unified read view for the bulk/admin list reads
	// (Catalog.All / Catalog.AllServices) and the miner seed.
	Snapshot() (patterns []*Pattern, services map[string]ServiceInfo, err error)

	// Curate persists ONE operator edit (label/verdict/tags, delete,
	// mark-known, end/restart service grace) so it is visible to the read
	// view. It carries exactly the mutations the Catalog's own curation
	// methods perform inline.
	Curate(edit CatalogEdit) error
}

CatalogStore is the OPTIONAL load/persist/read strategy for the pattern catalog. nil (the default) ⇒ the Catalog runs its current inline whole-blob path unchanged. A consumer installs a strategy via SetCatalogStore; OSS installs nothing.

Implementations must not retain the map arguments passed to Persist beyond the call (the Catalog passes its live maps under its own lock; the store is expected to serialize them synchronously, exactly as the inline path marshals under lock).

type CursorStore ¶

type CursorStore struct {
	// contains filtered or unexported fields
}

CursorStore persists the last-seen timestamp for each SignalSource. Redis is preferred so cursors survive crashes and replica restarts; the worker falls back to in-memory storage when Redis is unavailable so that development setups don't require Redis just to try training mode.

func NewCursorStore(rdb *redis.Client) *CursorStore

func (s *CursorStore) Get(ctx context.Context, source string) (time.Time, bool)

func (s *CursorStore) Set(ctx context.Context, source string, t time.Time) error

type DetectEvent ¶ added in v1.4.0

type DetectEvent struct {
	ID        string    `json:"id"`        // 16-byte hex; stable over restarts
	Timestamp time.Time `json:"timestamp"` // worker decision time

	// Pattern context
	Source    string   `json:"source"`
	PatternID string   `json:"pattern_id"`
	Template  string   `json:"template"`
	Service   string   `json:"service,omitempty"`
	Verdict   string   `json:"verdict"` // unknown | spike
	Frequency int      `json:"frequency"`
	Baseline  float64  `json:"baseline"`
	Samples   []string `json:"samples,omitempty"` // up to 3, redacted
	// RuleSeverity is the operator-declared severity floor carried from the
	// source signals (empty for auto-discovered signals). Persisted so the
	// declared-severity-honoured path is auditable/testable.
	RuleSeverity string `json:"rule_severity,omitempty"`

	// AI call (empty when outcome != "emitted" — cached/dry/quota/etc.
	// did not invoke the model; ai_error fills RawResponse only when
	// available).
	Model       string `json:"model,omitempty"`
	UserPrompt  string `json:"user_prompt,omitempty"`
	RawResponse string `json:"raw_response,omitempty"`
	DurationMs  int64  `json:"duration_ms,omitempty"`

	// Final structured finding (nil when no finding was produced).
	Finding *core.AIFinding `json:"finding,omitempty"`

	// Outcome label — see Worker.emitDetect: emitted | cached | dry |
	// quota | ai_error | send_error.
	Outcome string `json:"outcome"`
	// Error message when Outcome is ai_error / send_error.
	Error string `json:"error,omitempty"`
}

DetectEvent is the audit record for a single detect-mode handling of a pattern: what was sent to the model, what came back, what the final structured finding looked like, and what the worker did with it. Surfaced to the UI so operators can validate AI behavior end to end.

One DetectEvent is recorded per worker decision — including "cached", "quota", "ai_error", and "send_error" outcomes — so the log doubles as a debugging aid when alerts disappear.

type DetectLog ¶ added in v1.4.0

type DetectLog struct {
	// contains filtered or unexported fields
}

DetectLog is the in-memory + on-disk store of detect-mode AI calls. Bounded ring of the most-recent N events; older entries are evicted FIFO by Timestamp.

All public methods are safe for concurrent use. Disk persistence is debounced — Record sets a dirty flag that the worker flushes at most once per persist tick.

func LoadDetectLog(store storage.Provider, max int) (*DetectLog, error)

func (d *DetectLog) All() []*DetectEvent

func (d *DetectLog) Clear() int

func (d *DetectLog) Dirty() bool

func (d *DetectLog) Get(id string) *DetectEvent

func (d *DetectLog) Len() int

func (d *DetectLog) Persist() error

func (d *DetectLog) Record(e *DetectEvent)

func (d *DetectLog) Stats() map[string]int

type Emitter ¶ added in v1.4.0

type Emitter func(f *core.AIFinding, r core.AgentResult, source, service string) error

Emitter delivers an AI finding to the rest of the system. In production this is services.CreateIncidentFromFinding; tests inject a capturing stub. A nil Emitter disables emission (worker logs the would-be call and moves on).

type LearnExclusion ¶ added in v1.4.5

type LearnExclusion interface {
	ExcludeFromLearning(ctx context.Context, service, signal string) bool
}

LearnExclusion decides whether one signal should be kept out of the learning pipeline. ExcludeFromLearning returns true to DROP the signal (it is neither learned nor surfaced), false to keep it on the normal path. service is the discovered service name ("_unknown" when none could be resolved); signal is the logical signal name carried in Signal.Fields[core.FieldSignal] (empty for plain logs). Implementations must be safe for concurrent use — ticks run one goroutine per source.

type MatchResult ¶

type MatchResult struct {
	RuleName string // empty when no rule matched
	Default  bool   // true when only the default pattern matched
}

MatchResult is the (possibly empty) tag returned by Match.

func (m MatchResult) Matched() bool

type Miner ¶

type Miner struct {
	// contains filtered or unexported fields
}

Miner is a small, dependency-free, Drain-style log clusterer.

The classic Drain algorithm uses a fixed-depth prefix tree where the first few tokens index into buckets, and each leaf holds a list of templates. To classify an incoming log line:

1. Tokenize the message and replace tokens that look like variables (numbers, IPs, UUIDs, hex, IDs) with the wildcard `<*>` so that "user_id=42" and "user_id=99" share a structural shape.
2. Walk the tree using `len(tokens)` as the first bucket key (so messages of different lengths never collide), then the first N non-wildcard tokens as inner bucket keys.
3. At the leaf, pick the existing template with the highest token-by-token similarity. If similarity ≥ threshold, merge (replace differing tokens with `<*>`) and return that template's ID. Otherwise, register a new template.

This implementation is intentionally simpler than full Drain3 (no parameter masking learning, no parser tree compaction) but covers the 95% case for production logs. ADR-0001 tracks the longer-term plan.

func NewMiner(similarityThreshold float64, depth, maxChildren int) *Miner

func (m *Miner) AddCluster(id string, template string, size int)

func (m *Miner) Cluster(message string) (id string, template string, isNew bool)

func (m *Miner) Snapshot() []MinerCluster

type MinerCluster ¶

type MinerCluster struct {
	ID     string   // "p-<sha1[:12]>"
	Tokens []string // current template tokens (with `<*>` for variables)
	Size   int      // total observations matched into this cluster
}

MinerCluster is one learned log template plus its observation count.

type ModeResolver ¶ added in v1.4.4

type ModeResolver interface {
	Mode(ctx context.Context, agent string) (mode string, ok bool)
}

ModeResolver resolves the effective lifecycle mode at runtime. ok=false means "no opinion" -> the worker uses the static YAML cfg.Mode.

type Pattern ¶

type Pattern struct {
	ID string `json:"id"`
	// OrgID scopes the pattern to one organization. Defaults to
	// storage.DefaultOrgID ("default") so single-tenant OSS catalogs are
	// unaffected; enterprise multi-tenant routing reads it to isolate
	// per-org catalogs.
	OrgID     string    `json:"org_id,omitempty"`
	Template  string    `json:"template"`
	FirstSeen time.Time `json:"first_seen"`
	LastSeen  time.Time `json:"last_seen"`
	Count     int       `json:"count"`
	// BaselineFrequency is the EWMA of per-tick counts. Computed during
	// training; consumed by the spike detector in detect mode.
	BaselineFrequency float64 `json:"baseline_frequency"`
	// Verdict is the agent's classification of this pattern: "known" once
	// it is part of baseline (auto-promoted by count or set explicitly via
	// the admin API), otherwise empty. Operators flip a pattern to
	// "known" by POSTing {"verdict":"known"} to /api/agent/patterns/:id.
	Verdict string `json:"verdict"`
	// RuleName is the regex tag attached on first sighting ("default" when
	// only the default pattern matched, or the named rule otherwise).
	RuleName string `json:"rule_name"`
	// Source is the SignalSource name where the pattern was first observed.
	Source string `json:"source"`
	// Service is the service name extracted from the pattern's first
	// matching log message (via the agent.service_pattern regex). Empty
	// when service detection is disabled or the regex did not match. The
	// agent uses this to gate detect-mode AI analysis behind the
	// new-service grace window.
	Service string `json:"service,omitempty"`
	// Tags are arbitrary operator-supplied markers.
	Tags []string `json:"tags,omitempty"`
}

Pattern is one entry in the on-disk catalog (`patterns.json`).

The catalog is the agent's long-term memory. During training we add patterns; during shadow / detect we look them up to decide whether a signal is "known". Operators curate it via the admin REST endpoints.

type Redactor ¶

type Redactor struct {
	// contains filtered or unexported fields
}

Redactor scrubs sensitive substrings before any other component sees them. It is intentionally regex-based (not a full parser) — the goal is to make it operationally reasonable to send log content to an external LLM, not to be a perfect DLP solution.

func NewRedactor(redactIPs bool, extra []string) (*Redactor, []error)

func (r *Redactor) Scrub(s string) string

func (r *Redactor) ScrubFields(fields map[string]interface{}) map[string]interface{}

type RegexMatcher ¶

type RegexMatcher struct {
	// contains filtered or unexported fields
}

RegexMatcher is a small helper around a list of RegexRule plus an optional "default" pattern. It acts as the agent's pre-filter: only signals whose message matches at least one rule (named or default) are forwarded to the pattern miner / catalog. Set `regex.default_pattern: ".*"` to learn from every line.

func NewRegexMatcher(cfg config.AgentRegexConfig) (*RegexMatcher, []error)

func (m *RegexMatcher) Match(message string) MatchResult

type RegexRule ¶

type RegexRule struct {
	Name    string
	Pattern *regexp.Regexp
}

RegexRule is a compiled user-defined rule.

type ServiceInfo ¶

type ServiceInfo struct {
	// OrgID scopes the service entry to one organization. Defaults to
	// storage.DefaultOrgID ("default"); see Pattern.OrgID.
	OrgID     string    `json:"org_id,omitempty"`
	FirstSeen time.Time `json:"first_seen"`
}

ServiceInfo tracks when a service was first seen by the agent. Stored in the same patterns.json file alongside patterns — one data store, no Redis dependency for this feature.

type ServiceMatcher ¶

type ServiceMatcher struct {
	// contains filtered or unexported fields
}

ServiceMatcher extracts a service name from a log message using an ordered list of regexes. The first pattern that matches wins; the FIRST capture group is returned. A nil/empty matcher returns "" — service detection is off and every signal is attributed to "_unknown" in the worker. There is no built-in default list: operators who want service detection MUST configure `agent.service_patterns` (or set `AGENT_SERVICE_PATTERNS`).

func NewServiceMatcher(patterns []string) (*ServiceMatcher, []error)

func (m *ServiceMatcher) Extract(message string) string

type ShadowEvent ¶

type ShadowEvent struct {
	PatternID     string    `json:"pattern_id"`
	Template      string    `json:"template"`
	Source        string    `json:"source"`
	RuleName      string    `json:"rule_name,omitempty"`
	Verdict       string    `json:"verdict"` // "unknown" | "spike"
	SampleMessage string    `json:"sample_message"`
	Count         int       `json:"count"`       // total signals across all ticks
	Occurrences   int       `json:"occurrences"` // number of ticks that flagged this
	FirstSeen     time.Time `json:"first_seen"`
	LastSeen      time.Time `json:"last_seen"`
}

ShadowEvent is one "we would have alerted on this" record produced by the worker while running in shadow mode. Events are coalesced per (source, pattern_id): repeat hits update Count / LastSeen / Occurrences rather than appending new rows, so the on-disk log stays small and operator-reviewable.

type ShadowLog ¶

type ShadowLog struct {
	// contains filtered or unexported fields
}

ShadowLog is the in-memory + on-disk store of shadow-mode verdicts.

All public methods are safe for concurrent use. Disk persistence is debounced — Record sets a dirty flag that the worker flushes at most once per `persist_interval`.

func LoadShadowLog(store storage.Provider, max int) (*ShadowLog, error)

func (s *ShadowLog) All() []*ShadowEvent

func (s *ShadowLog) Clear() int

func (s *ShadowLog) Dirty() bool

func (s *ShadowLog) Len() int

func (s *ShadowLog) Persist() error

func (s *ShadowLog) Record(source, patternID, template, sample, rule, verdict string, freq int)

func (s *ShadowLog) Stats() map[string]int

type Worker ¶

type Worker struct {
	// contains filtered or unexported fields
}

Worker runs the agent loop:

Pull (per source) → Redact → Cluster → Catalog upsert → (mode-specific tail)

Training mode is fully end-to-end. Shadow and detect modes share the same classification path; shadow records would-have-alerted events to the shadow log, while detect calls the AI SRE and emits an incident through the existing services.CreateIncident pipeline.

func NewWorker(opt WorkerOptions) (*Worker, error)

func (w *Worker) Run(ctx context.Context)

type WorkerOptions ¶

type WorkerOptions struct {
	Cfg      config.AgentConfig
	Sources  []core.SignalSource
	Cursors  *CursorStore // optional; pass nil for in-memory cursors
	Redactor *Redactor
	Matcher  *RegexMatcher
	Miner    *Miner
	Catalog  *Catalog
	Shadow   *ShadowLog      // optional; pass nil to disable shadow recording
	Detect   *DetectLog      // optional; pass nil to disable detect-call audit log
	Services *ServiceMatcher // optional; pass nil to disable service detection

	// AI is the detect-mode bundle (analyzer + cache + rate limiter).
	// Pass a zero-value AIBundle to disable AI emission.
	AI AIBundle
	// Emitter is invoked for each finding in detect mode. nil disables
	// emission (worker still calls AI and caches the result, but the
	// finding does not flow through to channels).
	Emitter Emitter
}

WorkerOptions bundles the dependencies a Worker needs. Construction does not connect to anything; the worker dials lazily inside Run.