agent

package
v1.4.15 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 24, 2026 License: MIT Imports: 34 Imported by: 0

Documentation

Overview

Package agent contains the AI agent mode pipeline: 

SignalSource → Redact → Detector pipeline (Regex → Miner → Catalog →
Frequency) → (training: persist | shadow: log | detect: AI → Incident)

Training mode is end-to-end. Shadow and detect honor the same classification path; the AI analyzer call in detect mode is wired up separately from the worker.

Index

Constants

View Source 
const (
	OverrideSourceLog    = "log"
	OverrideSourceMetric = "metric"
	OverrideSourceTrace  = "trace"
)

Override source types. A rule only matches an input of the SAME type, so a metric override can never re-label a log and vice-versa.

View Source 
const (
	// SampleRingCap is the maximum number of redacted examples kept per
	// pattern/signal (drop-oldest past the cap). The ring stores oldest→newest,
	// so the LATEST example is ring[len(ring)-1].
	SampleRingCap = 10
	// SampleMaxLen caps each stored line (in bytes) so the ring can never bloat
	// the whole-blob catalog. Worst case per pattern is
	// SampleRingCap × SampleMaxLen ≈ 5 KB. It is the one knob to turn if a
	// deployment proves the ring heavy — no schema change.
	SampleMaxLen = 512
)
View Source 
const SpikeSettingsBlobName = "models/settings/spike-baseline"

SpikeSettingsBlobName is the single blob the spike baseline-mode setting lives under, in the models/settings/ namespace (a sibling of the report settings blob and the model-state artifacts).

Variables

View Source 
var (
	// ErrServiceExists is returned when creating/renaming to a name that is
	// already tracked (auto-discovered or manual).
	ErrServiceExists = fmt.Errorf("service already exists")
	// ErrServiceNotFound is returned when renaming/deleting a name that is not
	// tracked.
	ErrServiceNotFound = fmt.Errorf("service not found")
)

Sentinel errors for manual-service CRUD, so the admin controller can map them to precise HTTP statuses without string matching.

View Source 
var ErrSpikeNoStorage = errors.New("spike settings: storage not configured")

ErrSpikeNoStorage is returned by SaveSpikeSettings when no backend is configured, so the API can map it to a 503.

Functions

func BuildSources 

func BuildSources(cfg config.AgentConfig) ([]core.SignalSource, []error)

BuildSources constructs every enabled SignalSource from AgentConfig. Sources with type set to an unknown value are skipped (with a non-fatal error in the returned slice) so a config typo cannot stop the agent from starting.

func ContextWithOverrideOrg added in v1.4.7 

func ContextWithOverrideOrg(ctx context.Context, org string) context.Context

ContextWithOverrideOrg returns a copy of ctx carrying org for the worker's Run context. ResolveService reads it; absent, it resolves the default org.

func EffectiveModeForOrg added in v1.4.4 

func EffectiveModeForOrg(org, yamlFloor string) string

EffectiveModeForOrg returns the runtime-effective lifecycle mode for org, for read-only display surfaces such as the admin config endpoint and the dashboard top bar. It consults the registered ModeResolver's optional org-scoped getter and falls back to yamlFloor when there is no resolver, no override, or an invalid stored value. Community OSS registers no resolver, so it returns yamlFloor byte-for-byte unchanged.

func KnownBaselineMode added in v1.4.9 

func KnownBaselineMode(mode string) bool

KnownBaselineMode reports whether mode is exactly one of the three recognized baseline modes. The API uses it to reject an unknown mode with a 400 rather than silently folding it to the default, which is the store's job for a hand-edited or legacy blob.

func LogReadiness added in v1.4.7 

func LogReadiness(p *Pattern, autoPromoteAfter int, pollInterval time.Duration) core.Readiness

LogReadiness computes the readiness of one log pattern as a generic core.Readiness — the read-side view of "how close is this pattern to being treated as routine baseline (known)". It is a pure function that does NOT mutate the pattern: it reads the pattern's own counters (the same ones the gate compares) and the catalog's AutoPromoteAfter threshold.

  • Seen = the pattern's sighting count (the gate's own counter).
  • Needed = the effective auto-promotion threshold — always a positive gate (a non-positive configured value is resolved to the default), so there is no indeterminate/manual-only case.
  • Ready = isLogKnown(...), so an already-"known"/"spike"-marked or count-promoted pattern reports Ready.
  • RatePerMin = the pattern's learned per-second sighting rate (BaselineFrequency) rendered as sightings/minute (rate × 60), used to derive an ETA. 0 is the stalled/unknown sentinel (no rate yet, no flow, or no poll interval wired) — the UI then shows no ETA. It is only computed while the pattern is not yet Ready, since a Ready pattern has no countdown.

func NewMetricReader added in v1.4.4 

func NewMetricReader(q *signalsources.PrometheusQuerier) analyzetools.MetricReader

NewMetricReader wraps an already-constructed shared PrometheusQuerier as an analyze-tool MetricReader. It is the programmatic seam an out-of-tree module (e.g. the enterprise metric data source) uses to point the query_metrics tool at its own backend client without re-deriving the adapter: 

q, _ := signalsources.NewPrometheusQuerier(addr, auth, insecure)
tool := analyzetools.QueryMetrics{Reader: agent.NewMetricReader(q)}

A nil querier yields a nil reader so the caller can pass it straight to analyzetools.Default (which omits the tool when the reader is nil).

func NewTraceReader added in v1.4.4 

func NewTraceReader(q *signalsources.TempoQuerier) analyzetools.TraceReader

NewTraceReader wraps an already-constructed shared TempoQuerier as an analyze-tool TraceReader. It is the programmatic seam an out-of-tree module (e.g. the enterprise trace data source) uses to point the query_traces tool at its own backend client: 

q, _ := signalsources.NewTempoQuerier(addr, auth, insecure)
tool := analyzetools.QueryTraces{Reader: agent.NewTraceReader(q), Redactor: red}

A nil querier yields a nil reader so the caller can pass it straight to analyzetools.Default (which omits the tool when the reader is nil).

func PushSample added in v1.4.8 

func PushSample(ring []string, sample string, scrub core.Scrubber) []string

PushSample scrubs sample (defence-in-depth), one-lines + truncates it to SampleMaxLen, appends it to ring, and trims ring to the last SampleRingCap entries (drop-oldest, order oldest→newest).

scrub is applied FIRST so it sees the whole line — a planted secret can never survive to storage even if a future refactor composed the sample from a not-yet-scrubbed source — and the truncate runs AFTER so SampleMaxLen stays a hard ceiling even when a placeholder (e.g. <REDACTED:basic_auth>) is longer than the secret it replaced. scrub MAY be nil (input already redacted); when non-nil it is ALWAYS applied. An empty result is not appended.

This is the generic seam the enterprise metric/trace brains import to keep their per-signal ring identical in shape to the OSS log ring.

func RegisterTypedBrain added in v1.4.4 

func RegisterTypedBrain(sourceType string, mk BrainFactory)

RegisterTypedBrain makes a per-type brain constructible by the worker for a given source type. It is intended to be called from an init() in a module that provides additional signal types (e.g. Versus Enterprise). Registering the same type twice, or with a nil factory, panics — both indicate a wiring bug, not a runtime condition.

func RegisteredTypedBrains added in v1.4.4 

func RegisteredTypedBrains() []string

RegisteredTypedBrains returns the sorted list of source types that have a registered brain. Useful for diagnostics and admin surfaces. OSS returns an empty slice (the log brain is the un-registered default).

func ResolveServiceOverride added in v1.4.7 

func ResolveServiceOverride(ctx context.Context, in ServiceOverrideInput) string

ResolveServiceOverride applies the installed manual-attribution override to a detected attribution, returning the operator-chosen service when a rule matches, or in.Service unchanged otherwise. It is the single chokepoint the OSS log brain and the enterprise metric/trace brains funnel through, so a manual override wins over regex detection identically for all three source types. A nil resolver (no override wired) returns in.Service — one nil-check, no allocation. A resolver that returns ("", true) or a blank service is ignored (never blanks a real attribution).

func SaveSpikeSettings added in v1.4.9 

func SaveSpikeSettings(st storage.Provider, s SpikeSettings) error

SaveSpikeSettings persists the settings blob after sanitizing it. It returns ErrSpikeNoStorage when no backend is configured so the API can map it to 503.

func SetAISettingsResolver added in v1.4.4 

func SetAISettingsResolver(r AISettingsResolver)

SetAISettingsResolver registers the resolver used to override the effective AI settings at runtime. Last-wins: a second call replaces the first. Passing nil clears the slot (back to the YAML floor). OSS ships none, so the worker and transport use the static ai config unchanged. This is the entry point the enterprise hooks.Register attaches to (mirror of SetModeResolver). Call at boot, before the worker starts.

func SetCatalogStore added in v1.4.4 

func SetCatalogStore(s CatalogStore)

SetCatalogStore installs the process-wide catalog load/persist/read strategy. Last-wins: a second call replaces the first. Passing nil clears the slot (back to the inline whole-blob path). Call at boot, before the worker starts and before LoadCatalog. OSS ships none, so the Catalog runs its current inline code unchanged. This is the entry point a consumer (e.g. the enterprise boot path) attaches to — mirror of scheduler.SetOwnership / SetModeResolver.

func SetLearnExclusion added in v1.4.5 

func SetLearnExclusion(x LearnExclusion)

SetLearnExclusion installs the process-wide learn-exclusion policy. Last-wins: a second call replaces the first. Passing nil clears the slot (back to the inert "learn everything" path). Call at boot, before the worker starts. OSS ships none, so the worker runs its current pipeline unchanged.

func SetModeResolver added in v1.4.4 

func SetModeResolver(r ModeResolver)

SetModeResolver registers the resolver used to override the effective lifecycle mode at runtime. Last-wins: a second call replaces the first. Passing nil clears the slot (back to the YAML floor). OSS ships none, so the worker uses cfg.Mode unchanged. This is the entry point the enterprise hooks.Register attaches to (mirror of middleware.SetOrgResolver). Call at boot, before the worker starts.

func SetServiceOverride added in v1.4.7 

func SetServiceOverride(o ServiceOverride)

SetServiceOverride installs the process-wide manual-attribution resolver. Last-wins: a second call replaces the first. Passing nil clears the slot (back to "keep the detected service"). Call at boot, before the worker starts. OSS installs its ServiceOverrideStore here; an unwired build leaves it nil and attribution is byte-for-byte unchanged.

Types

type AIBundle added in v1.4.0 

type AIBundle struct {
	Router      *router.Router
	Detect      core.AIAgent // kind=AITaskDetect
	Analyze     core.AIAgent // kind=AITaskAnalyze, built when AI.Enable is true
	Cache       *ai.ResultCache
	Rate        *ai.RateLimiter
	AnalyzeRate *ai.RateLimiter // separate hourly cap for analyze
	// Runbooks is the runbook corpus manager shared by the find_runbook
	// read path and the admin runbooks UI (upload/list/delete). Nil when
	// storage is unavailable. Present even without embeddings so operators
	// can manage the corpus before configuring an embedding model.
	Runbooks *runbook.Manager
}

AIBundle bundles every AI-side dependency. All fields are nil-safe: when AI is disabled the worker accepts a zero bundle and emits a deterministic templated alert instead of enriching via AI.

Router exposes the typed task dispatcher to non-worker consumers (admin endpoints, future analyze controller). The worker keeps using Detect + Cache + Rate directly so its per-outcome logging (emitted / cached / emitted_basic*) stays explicit.

func BuildAIs added in v1.4.3 

func BuildAIs(cfg config.AgentConfig, catalog *Catalog, store storage.Provider, httpClient *http.Client) AIBundle

BuildAIs constructs every AI dependency (router, detect agent, optional analyze agent with its tool catalog, per-task cache, per- task rate limiter) from the agent config.

Returns a zero AIBundle when cfg.AI.Enable is false so callers can pass the result straight to NewWorker without nil checks.

httpClient may be nil — a default *http.Client is used by the chat model. store may be nil — caches degrade to in-memory only; the analyze agent's tool registry will also be smaller.

type AIProviderResolver added in v1.4.4 

type AIProviderResolver interface {
	EffectiveProvider(ctx context.Context) (provider string, ok bool)
}

AIProviderResolver is an OPTIONAL extension of AISettingsResolver: a registered resolver that ALSO implements it can override the model PROVIDER (openai | deepseek | qwen | ollama | claude | gemini) at runtime, so an operator's provider change rebuilds the agent's model on its next run without a process restart. ok=false ⇒ no opinion (use the configured provider). A resolver that does not implement this interface simply has no provider opinion — composition is additive, so the existing key/enabled resolver keeps working unchanged. OSS registers nothing.

type AISettingsResolver added in v1.4.4 

type AISettingsResolver interface {
	EffectiveKey(ctx context.Context) (key string, ok bool)
	EffectiveEnabled(ctx context.Context) (enabled bool, ok bool)
}

AISettingsResolver resolves effective AI settings at runtime. ok=false on a method means "no opinion" -> caller uses the static YAML ai config.

type BaselineFold added in v1.4.9 

type BaselineFold struct {
	// PollSeconds is the tick duration; the folded value is tickCount /
	// PollSeconds, a per-second rate. <= 0 folds the raw per-tick count (legacy).
	PollSeconds float64
	// RejectZ holds out a tick whose rate sits >= this many σ from the pre-fold
	// baseline once the estimator is confident, so a burst can't drag the mean.
	// <= 0 disables the hold-out (every tick folds).
	RejectZ float64
	// SeasonalPeriod selects the seasonal buckets: 0 = off (global only),
	// 24 = hour-of-day, 168 = hour-of-week.
	SeasonalPeriod int
	// MinBaselineCount is the confidence gate for the global outlier hold-out:
	// the estimator is confident once the pattern's total sighting count reaches
	// this. <= 0 leaves the global fold always cold (never rejects).
	MinBaselineCount int
	// MinBucketSamples is the confidence gate for a seasonal bucket's own
	// hold-out (mirrors the detector's bucket-fallback threshold).
	MinBucketSamples int
}

BaselineFold configures how Catalog.Upsert folds a tick into a pattern's baseline. The zero value is the legacy mean-only fold of the raw per-tick count (so a catalog with no fold wired — every unit test, and any pre-rate caller — behaves exactly as before). The agent worker wires the real config from the poll interval + spike settings via SetBaselineFold.

type BrainFactory added in v1.4.4 

type BrainFactory func(name string, options map[string]any) (core.SignalLearner, core.SignalDetector, error)

BrainFactory builds the Learner + Detector pair for one configured source instance from its instance name and the generic per-source options block (the decoded YAML under the source's `options:` key, see config.AgentSourceConfig.Options). A factory decodes the map into whatever concrete config it owns. Returning an error makes the worker fall back to the log brain for that source (logged, non-fatal).

type Catalog 

type Catalog struct {
	// contains filtered or unexported fields
}

Catalog is the in-memory + on-disk pattern store.

All public methods are safe for concurrent use. Disk persistence is debounced — calls to MarkDirty() set a flag that the agent worker flushes at most once per `persist_interval`.

func LoadCatalog 

func LoadCatalog(store storage.Provider) (*Catalog, error)

LoadCatalog opens an existing patterns blob from the storage provider or returns an empty catalog if none exists. The blob name is config.CatalogBlobName ("patterns").

func (*Catalog) All 

func (c *Catalog) All() []*Pattern

func (*Catalog) AllServices 

func (c *Catalog) AllServices() map[string]ServiceInfo

AllServices returns a snapshot of every tracked service, sorted by FirstSeen ascending.

func (*Catalog) CreateService added in v1.4.7 

func (c *Catalog) CreateService(name string) error

CreateService records an operator-created (manual) service so it is selectable as an override target before any signal is attributed to it. The caller (admin controller) validates non-existence first — this is the write. It routes through the CatalogStore when one is installed so the manual service is fleet-visible; otherwise it writes the in-memory + blob path.

func (*Catalog) Delete 

func (c *Catalog) Delete(patternID string) bool

Delete removes a pattern (e.g. operator marks a false-positive cluster).

func (*Catalog) DeleteService added in v1.4.7 

func (c *Catalog) DeleteService(name string) bool

DeleteService removes a tracked service entry. Returns false when it does not exist. The admin controller gates this on the service being manual and on having no override rules that target it, so a delete never orphans an override.

func (*Catalog) Dirty 

func (c *Catalog) Dirty() bool

Dirty reports whether there are unflushed changes.

func (*Catalog) EndServiceGrace 

func (c *Catalog) EndServiceGrace(name string) bool

EndServiceGrace forces a service out of its grace period by setting FirstSeen to the zero time. Returns false when the service doesn't exist.

func (*Catalog) Get 

func (c *Catalog) Get(id string) *Pattern

Get returns a deep copy of the pattern keyed by id, or nil when not found. Returning a copy (rather than the live pointer) prevents callers from observing torn reads while a concurrent Upsert mutates the same struct.

func (*Catalog) IsServiceInGrace 

func (c *Catalog) IsServiceInGrace(name string, graceDuration time.Duration) bool

IsServiceInGrace reports whether the named service is still inside its new-service grace window. A zero graceDuration means grace is disabled (always returns false). An unknown service is registered on the spot and enters grace.

func (*Catalog) Label 

func (c *Catalog) Label(patternID string, verdict *string, tags []string) bool

Label updates operator-curated metadata for a pattern.

verdict is a tri-state pointer that lets the admin API distinguish "verdict absent (leave it)" from "verdict present and empty (clear it)":

  • nil → leave the stored verdict unchanged (a tags-only update)
  • non-nil, "" → CLEAR the verdict (operator "Clear verdict")
  • non-nil, "..." → SET the verdict to that value

tags nil leaves tags unchanged. Returns false when the pattern doesn't exist.

func (*Catalog) Len 

func (c *Catalog) Len() int

Len returns the number of patterns currently in the catalog.

func (*Catalog) MarkKnown 

func (c *Catalog) MarkKnown(patternID string) bool

MarkKnown stamps a pattern as auto-promoted ("known") in the catalog.

func (*Catalog) Persist 

func (c *Catalog) Persist() error

Persist flushes the in-memory catalog to the storage backend. Safe to call concurrently with Upsert/Label/Delete.

func (*Catalog) RecordSample added in v1.4.8 

func (c *Catalog) RecordSample(patternID, sample string, scrub core.Scrubber)

RecordSample appends a redacted example log line to a pattern's bounded sample ring (drop-oldest past SampleRingCap) and marks the catalog dirty, so the ring rides the existing debounced Persist path (and, under an installed CatalogStore, the same per-partition Persist) with NO new persistence wiring — exactly like the Service re-point Upsert already carries. It is the log brain's post-Upsert companion: the brain holds the pipeline's redactor and passes the representative POST-REDACTION Observation message; Signal.Raw is never a source. scrub re-scrubs the line at the storage boundary (defence-in-depth) and MAY be nil. A missing pattern or empty sample is a no-op. It stays OFF the store hot path (never calls Curate), so community and single-instance behaviour is unchanged.

func (*Catalog) RegisterService 

func (c *Catalog) RegisterService(name string) bool

RegisterService records a service name the first time it is seen. Returns true when the service was newly registered (first sighting), false when it was already known. The caller uses this to decide whether to log a "new service discovered" message.

func (*Catalog) RenameService added in v1.4.7 

func (c *Catalog) RenameService(oldName, newName string) error

RenameService moves a service entry from oldName to newName, preserving its FirstSeen and manual flag. The caller validates that oldName exists and newName does not. Pattern attribution is not bulk-rewritten (a pattern's Service is a historical label; future signals re-attribute); the admin controller repoints override rules that target the old name so none dangle.

func (*Catalog) RepointService added in v1.4.7 

func (c *Catalog) RepointService(patternID, service string) bool

RepointService immediately sets an EXISTING pattern's Service to service — the retroactive half of an operator "Reassign to service" correction. Unlike Upsert's re-observation re-point (which only lands when a fresh matching log line re-clusters the pattern on a later tick), this takes effect on the very next catalog read, so reassigning a pattern that is not currently receiving traffic is reflected immediately instead of after an unbounded delay.

service MUST be a real service (non-"" and not the "_unknown" sentinel); a blank/"_unknown" target is rejected so a reassignment can never blank out or unknown-out a pattern's attribution. The caller (createServiceOverride) also validates the target exists in the catalog first.

Returns true when a pattern's Service was changed; false when the guard fails, the pattern does not exist (e.g. the override match was a message substring, not a pattern id — the lazy re-observation path still applies it), or the pattern already points at service.

Store-aware: when a CatalogStore is installed the re-point routes through Curate(CatalogEditRepointService) so a fleet-wide read view (the enterprise partition store) re-points too; otherwise it mutates the in-memory pattern under lock and marks the catalog dirty for the next Persist flush.

func (*Catalog) ResetPatterns added in v1.4.7 

func (c *Catalog) ResetPatterns() (patterns int, err error)

ResetPatterns wipes every learned log pattern — the whole `patterns` map — and persists the empty pattern set so log training restarts from scratch on the next tick. Discovered/manual services are LEFT INTACT. It returns the number of patterns that were removed (the pre-reset view: fleet-wide when a CatalogStore is installed, otherwise this instance's in-memory set).

This is the pattern-half counterpart to ResetServices: when a CatalogStore is installed the empty pattern state routes through it (so a fleet-wide read view is cleared, not just this instance's working set); otherwise the inline whole-blob path rewrites the "patterns" blob from the in-memory maps, which still carry the untouched services.

func (*Catalog) ResetServices added in v1.4.7 

func (c *Catalog) ResetServices() (services int, err error)

ResetServices wipes every discovered/manual service — the whole `services` map — and persists the empty service set so service discovery restarts from scratch on the next tick. Learned log patterns are LEFT INTACT. It returns the number of services that were removed (the pre-reset view: fleet-wide when a CatalogStore is installed, otherwise this instance's in-memory set).

This is the service-half counterpart to ResetPatterns: when a CatalogStore is installed the empty service state routes through it (so a fleet-wide read view is cleared, not just this instance's working set); otherwise the inline whole-blob path rewrites the "patterns" blob from the in-memory maps, which still carry the untouched patterns.

func (*Catalog) RestartServiceGrace 

func (c *Catalog) RestartServiceGrace(name string) bool

RestartServiceGrace resets a service's FirstSeen to now, restarting the grace window. Returns false when the service doesn't exist.

func (*Catalog) Service added in v1.4.7 

func (c *Catalog) Service(name string) (ServiceInfo, bool)

Service returns a copy of one tracked service's info and whether it exists. It reads the same unified view AllServices() serves (store-aware when a CatalogStore is installed), so a manual service created on any instance is visible fleet-wide.

func (*Catalog) SetBaselineFold added in v1.4.9 

func (c *Catalog) SetBaselineFold(f BaselineFold)

SetBaselineFold wires the baseline-fold configuration the agent worker derives from its poll interval + spike settings. It is set once at construction; the zero value (never set) keeps the legacy mean-only per-tick-count fold.

func (*Catalog) Upsert 

func (c *Catalog) Upsert(patternID, template, source string, tickCount int, alpha float64, ruleName, service string) *Pattern

Upsert records an observation against patternID. If the pattern is new it is created with FirstSeen=now; otherwise Count is incremented and LastSeen is updated. tickCount is the number of matches observed in the current worker tick — used to update the EWMA baseline.

service is the service name extracted from the log via agent.service_patterns, then run through the manual-attribution override resolver by the caller. It is stamped on a new pattern verbatim; on an existing pattern it is REFRESHED whenever the incoming attribution is real (non-empty and not "_unknown"), so an operator override resolved on a later tick re-points an already-learned pattern's Service instead of leaving the stale first-sighting label. A "" or "_unknown" attribution on a later tick never clobbers a good stored Service, so a tick where detection/override yields nothing keeps the last real attribution. Pass "" when service detection is disabled or the pattern's regexes did not match.

ruleName comes from the regex pre-filter and is applied:

  • on first-seen: always
  • subsequently: only when a non-default named rule supersedes a previous default tag, or when the previous tag was empty

type CatalogEdit added in v1.4.4 

type CatalogEdit struct {
	Kind CatalogEditKind
	// PatternID identifies the pattern for the label/delete/mark-known edits.
	PatternID string
	// Verdict is the operator verdict for a label edit — a tri-state pointer
	// matching Catalog.Label: nil leaves the stored verdict unchanged (a
	// tags-only edit), a non-nil pointer SETS it, and a non-nil pointer to the
	// empty string CLEARS it. A store MUST honour the &"" case (clear) — a
	// plain empty string can never clear, which is exactly the "Clear verdict"
	// no-op bug this seam change fixes.
	Verdict *string
	// Tags are the operator tags for a label edit (nil leaves them unchanged,
	// matching Catalog.Label).
	Tags []string
	// Service identifies the service for the grace and manual-service edits
	// (the OLD name for a rename) and the NEW target service for a
	// CatalogEditRepointService.
	Service string
	// NewService is the target name for a rename edit; unused otherwise.
	NewService string
}

CatalogEdit is one operator mutation, modelled on the Catalog curation method set so a CatalogStore can apply it without importing the controllers. Kind selects which fields are meaningful:

  • CatalogEditLabel → PatternID, Verdict, Tags
  • CatalogEditDelete → PatternID
  • CatalogEditMarkKnown → PatternID
  • CatalogEditEndServiceGrace → Service
  • CatalogEditRestartServiceGrace→ Service
  • CatalogEditResetPatterns → (no fields — clears every pattern)
  • CatalogEditResetServices → (no fields — clears every service)
  • CatalogEditCreateService → Service (manual service name)
  • CatalogEditRenameService → Service (old), NewService (new)
  • CatalogEditDeleteService → Service (manual service name)

type CatalogEditKind added in v1.4.4 

type CatalogEditKind string

CatalogEditKind discriminates which operator mutation a CatalogEdit carries. The set mirrors the Catalog curation method set one-for-one. 

const (
	// CatalogEditLabel carries Label(PatternID, Verdict, Tags).
	CatalogEditLabel CatalogEditKind = "label"
	// CatalogEditDelete carries Delete(PatternID).
	CatalogEditDelete CatalogEditKind = "delete"
	// CatalogEditMarkKnown carries MarkKnown(PatternID).
	CatalogEditMarkKnown CatalogEditKind = "mark_known"
	// CatalogEditRepointService carries RepointService(PatternID, Service): set
	// an EXISTING pattern's Service immediately — the retroactive half of an
	// operator "Reassign to service" correction. Unlike a Persist-carried
	// re-observation re-point (which only lands when a fresh matching log line
	// re-clusters the pattern via Upsert), this takes effect on the very next
	// read view, so a reassignment of a pattern that is not currently receiving
	// traffic is reflected immediately. It carries PatternID + Service (Service
	// is the NEW target, guaranteed by the caller to be a real, non-"_unknown"
	// service). A store MUST treat a missing PatternID as a no-op (the log
	// override match can be a message substring, not a pattern id) rather than
	// an error, exactly as the in-memory path does.
	CatalogEditRepointService CatalogEditKind = "repoint_service"
	// CatalogEditEndServiceGrace carries EndServiceGrace(Service).
	CatalogEditEndServiceGrace CatalogEditKind = "end_service_grace"
	// CatalogEditRestartServiceGrace carries RestartServiceGrace(Service).
	CatalogEditRestartServiceGrace CatalogEditKind = "restart_service_grace"
	// CatalogEditResetPatterns carries ResetPatterns(): wipe EVERY learned +
	// curated log pattern and persist the empty pattern set, LEAVING services
	// untouched. It carries no PatternID/Service — it targets the whole pattern
	// half, not one entry — so a store implementation empties only its pattern
	// read view.
	CatalogEditResetPatterns CatalogEditKind = "reset_patterns"
	// CatalogEditResetServices carries ResetServices(): wipe EVERY discovered +
	// manual service and persist the empty service set, LEAVING patterns
	// untouched. It carries no PatternID/Service — it targets the whole service
	// half, not one entry — so a store implementation empties only its service
	// read view.
	CatalogEditResetServices CatalogEditKind = "reset_services"
	// CatalogEditCreateService carries CreateService(Service): record an
	// operator-created (manual) service so it is selectable before any signal.
	CatalogEditCreateService CatalogEditKind = "create_service"
	// CatalogEditRenameService carries RenameService(Service, NewService): move
	// a manual service entry to a new name, preserving FirstSeen + manual flag.
	CatalogEditRenameService CatalogEditKind = "rename_service"
	// CatalogEditDelRepointService     → PatternID, Service (new target)
	//   - CatalogEditeteService carries DeleteService(Service): remove a manual
	// service entry.
	CatalogEditDeleteService CatalogEditKind = "delete_service"
)

type CatalogStore added in v1.4.4 

type CatalogStore interface {
	// Load returns the working-set this consumer wants the Catalog to hold in
	// memory after boot. Either returned map may be nil to mean "empty".
	Load() (patterns map[string]*Pattern, services map[string]*ServiceInfo, err error)

	// Persist writes the Catalog's current in-memory working set.
	//
	// Persist carries re-observation updates as well as fresh patterns: an
	// existing pattern's Service is REFRESHED in the working set whenever a real
	// attribution (an operator override or regex detection) resolves for it on a
	// tick (see Catalog.Upsert). A store MUST apply upsert semantics for the
	// Service field — overwrite the stored Service from the working-set value
	// rather than treating an existing pattern as immutable — or an operator's
	// "Reassign to service" never re-points an already-learned pattern in the
	// read view. Upsert stays off the store hot path (it never calls Curate);
	// this Service change rides Persist like every other working-set mutation.
	Persist(patterns map[string]*Pattern, services map[string]*ServiceInfo) error

	// Snapshot returns the unified read view for the bulk/admin list reads
	// (Catalog.All / Catalog.AllServices) and the miner seed.
	Snapshot() (patterns []*Pattern, services map[string]ServiceInfo, err error)

	// Curate persists ONE operator edit (label/verdict/tags, delete,
	// mark-known, end/restart service grace, or a whole-catalog reset) so it
	// is visible to the read view. It carries exactly the mutations the
	// Catalog's own curation methods perform inline.
	Curate(edit CatalogEdit) error
}

CatalogStore is the OPTIONAL load/persist/read strategy for the pattern catalog. nil (the default) ⇒ the Catalog runs its current inline whole-blob path unchanged. A consumer installs a strategy via SetCatalogStore; OSS installs nothing.

Implementations must not retain the map arguments passed to Persist beyond the call (the Catalog passes its live maps under its own lock; the store is expected to serialize them synchronously, exactly as the inline path marshals under lock).

func NewPostgresCatalogStore added in v1.4.8 

func NewPostgresCatalogStore(db *sql.DB, orgID string, instanceIndex int) CatalogStore

NewPostgresCatalogStore builds the OSS Postgres catalog store over the typed signal tables. db is obtained via storage.SQLAccessor.DB(); orgID is the boot-pinned deployment org (storage.DefaultOrgID for single-tenant OSS); instanceIndex is the write-shard ordinal (0 for OSS single-instance; the enterprise HA install passes the cluster ordinal). Install it via agent.SetCatalogStore at boot, before the worker starts and before LoadCatalog. This is the seam the Enterprise Engineer rides: the enterprise intel store adds vs_metrics/vs_traces on the SAME vs_patterns root, and the enterprise HA install constructs THIS store with the ordinal.

type CursorStore 

type CursorStore struct {
	// contains filtered or unexported fields
}

CursorStore persists the last-seen timestamp for each SignalSource. Redis is preferred so cursors survive crashes and replica restarts; the worker falls back to in-memory storage when Redis is unavailable so that development setups don't require Redis just to try training mode.

func NewCursorStore 

func NewCursorStore(rdb redis.UniversalClient) *CursorStore

NewCursorStore returns a CursorStore. Pass `rdb=nil` for in-memory only.

func (*CursorStore) Get 

func (s *CursorStore) Get(ctx context.Context, source string) (time.Time, bool)

Get returns the cursor for a source. The bool is false when no cursor has been recorded yet (caller should fall back to AgentConfig.Lookback).

func (*CursorStore) Reset added in v1.4.7 

func (s *CursorStore) Reset(ctx context.Context) error

Reset forgets every recorded cursor so the next Pull for each source starts from the worker's lookback window again (loadCursor falls back to now-lookback when no cursor exists). It is the in-place equivalent of a fresh process start with empty in-memory cursors: after an operator wipes the learned catalog, the SAME running worker re-reads the available history and relearns immediately, instead of sitting idle until brand-new-timestamp signals arrive because its cursor is still pinned past the consumed window.

The in-memory map is always cleared; when Redis backs the store every persisted cursor key under the prefix is deleted too. Redis errors are returned so the caller can decide whether the rewind fully succeeded.

func (*CursorStore) Set 

func (s *CursorStore) Set(ctx context.Context, source string, t time.Time) error

Set records a cursor. Redis errors are returned to the caller (the worker logs them and continues — mem cache always succeeds).

type DetectEvent added in v1.4.0 

type DetectEvent struct {
	ID        string    `json:"id"`        // 16-byte hex; stable over restarts
	Timestamp time.Time `json:"timestamp"` // worker decision time

	// Pattern context
	Source    string  `json:"source"`
	PatternID string  `json:"pattern_id"`
	Template  string  `json:"template"`
	Service   string  `json:"service,omitempty"`
	Verdict   string  `json:"verdict"` // unknown | spike
	Frequency int     `json:"frequency"`
	Baseline  float64 `json:"baseline"`
	// BaselineStd is the learned standard deviation the spike z-score was
	// computed against (0 when not a spike or no dispersion learned yet).
	// Together with Baseline (the mean) and Score (the z-score) it makes every
	// spike verdict reconstructable from the stored numbers.
	BaselineStd float64 `json:"baseline_std,omitempty"`
	// Score is the spike z-score: how many standard deviations the tick's
	// per-second rate sat above the learned baseline.
	Score float64 `json:"score,omitempty"`
	// Explanation is the deterministic human-readable spike math, e.g.
	// "47.3/s = 4.2σ above 38.4/s ± 3.1".
	Explanation string   `json:"explanation,omitempty"`
	Samples     []string `json:"samples,omitempty"` // up to 3, redacted
	// RuleSeverity is the operator-declared severity floor carried from the
	// source signals (empty for auto-discovered signals). Persisted so the
	// declared-severity-honoured path is auditable/testable.
	RuleSeverity string `json:"rule_severity,omitempty"`

	// AI call (empty when the model was not invoked — cached and the
	// "emitted_basic*" templated-alert outcomes carry no model call;
	// Model is "heuristic" for the deterministic templated alerts).
	Model       string `json:"model,omitempty"`
	UserPrompt  string `json:"user_prompt,omitempty"`
	RawResponse string `json:"raw_response,omitempty"`
	DurationMs  int64  `json:"duration_ms,omitempty"`

	// Final structured finding (nil when no finding was produced).
	Finding *core.AIFinding `json:"finding,omitempty"`

	// Outcome label — see Worker.emitDetect: emitted | cached |
	// emitted_basic | emitted_basic_quota | emitted_basic_error |
	// send_error.
	Outcome string `json:"outcome"`
	// Error message when Outcome is emitted_basic_error / send_error.
	Error string `json:"error,omitempty"`
}

DetectEvent is the audit record for a single detect-mode handling of a pattern: what was sent to the model, what came back, what the final structured finding looked like, and what the worker did with it. Surfaced to the UI so operators can validate AI behavior end to end.

One DetectEvent is recorded per worker decision — including "cached", the "emitted_basic*" templated-alert outcomes, and "send_error" — so the log doubles as a debugging aid when alerts disappear.

type DetectLog added in v1.4.0 

type DetectLog struct {
	// contains filtered or unexported fields
}

DetectLog is the in-memory + on-disk store of detect-mode AI calls. Bounded ring of the most-recent N events; older entries are evicted FIFO by Timestamp.

All public methods are safe for concurrent use. Disk persistence is debounced — Record sets a dirty flag that the worker flushes at most once per persist tick.

func LoadDetectLog added in v1.4.0 

func LoadDetectLog(store storage.Provider, max int) (*DetectLog, error)

LoadDetectLog opens an existing detect log from the storage provider or returns an empty one when no blob is present. `max` caps retained events; pass 0 for the default. A nil store disables persistence (in-memory only).

func (*DetectLog) All added in v1.4.0 

func (d *DetectLog) All() []*DetectEvent

All returns a snapshot of every event, sorted by Timestamp descending (newest first). Returned events are shallow copies; the embedded *AIFinding pointer is shared (read-only by convention).

func (*DetectLog) Clear added in v1.4.0 

func (d *DetectLog) Clear() int

Clear removes every event.

func (*DetectLog) Dirty added in v1.4.0 

func (d *DetectLog) Dirty() bool

Dirty reports whether there are unflushed changes.

func (*DetectLog) Get added in v1.4.0 

func (d *DetectLog) Get(id string) *DetectEvent

Get returns a copy of the event with the given ID, or nil.

func (*DetectLog) Len added in v1.4.0 

func (d *DetectLog) Len() int

Len returns the number of stored events.

func (*DetectLog) Persist added in v1.4.0 

func (d *DetectLog) Persist() error

Persist atomically writes the detect log via the storage backend. No-op when store is nil or there are no unflushed changes.

func (*DetectLog) Record added in v1.4.0 

func (d *DetectLog) Record(e *DetectEvent)

Record appends a DetectEvent. The caller passes a partially-filled event; ID and Timestamp are assigned here if zero.

Field-size caps are applied defensively so a misbehaving model can't bloat the on-disk file.

func (*DetectLog) Stats added in v1.4.0 

func (d *DetectLog) Stats() map[string]int

Stats returns aggregate counts useful for /api/agent/detect/stats.

type Emitter added in v1.4.0 

type Emitter func(f *core.AIFinding, r core.AgentResult, source, service string) error

Emitter delivers an AI finding to the rest of the system. In production this is services.CreateIncidentFromFinding; tests inject a capturing stub. A nil Emitter disables emission (worker logs the would-be call and moves on).

type LearnExclusion added in v1.4.5 

type LearnExclusion interface {
	ExcludeFromLearning(ctx context.Context, service, signal string) bool
	ExcludeLogPattern(ctx context.Context, service, patternKey string) bool
}

LearnExclusion decides whether a signal should be kept out of the learning pipeline. It answers at two grains, consulted at two points in the tick:

  • ExcludeFromLearning is the PRE-Group chokepoint (whole-service and metric/trace per-signal exclusion). It returns true to DROP the signal (neither learned nor surfaced), false to keep it. service is the discovered service name ("_unknown" when none could be resolved); signal is the logical signal name carried in Signal.Fields[core.FieldSignal] (empty for plain logs — which is exactly why a single LOG PATTERN cannot be singled out here, see ExcludeLogPattern).

  • ExcludeLogPattern is the POST-Group per-LOG-PATTERN grain. A plain log line carries no signal identity before grouping, so the pre-Group hook can only exclude a log pattern's whole SERVICE. Once the log brain's Group has assigned each observation its stable pattern Key/id, the worker consults this method once per observation to drop just the excluded PATTERNS before Learn — the log analogue of the metric/trace per-signal exclusion. service is the observation's attributed service; patternKey is the pattern's stable Key/id (the miner cluster id — the same identity shown in the patterns list and used by relabel/reassign). It is consulted for LOG brains only; metric/trace signal exclusion stays at the pre-Group hook. A nil resolver (community/OSS) is never installed, so neither method is ever called there.

Implementations must be safe for concurrent use — ticks run one goroutine per source.

type MatchResult 

type MatchResult struct {
	RuleName string // empty when no rule matched
	Default  bool   // true when only the default pattern matched
}

MatchResult is the (possibly empty) tag returned by Match.

func (MatchResult) Matched 

func (m MatchResult) Matched() bool

Matched reports whether at least one rule (named or default) hit. The worker uses this to decide whether a signal is interesting enough to learn from. To train on every line, set `regex.default_pattern: ".*"`.

type Miner 

type Miner struct {
	// contains filtered or unexported fields
}

Miner is a small, dependency-free, Drain-style log clusterer.

The classic Drain algorithm uses a fixed-depth prefix tree where the first few tokens index into buckets, and each leaf holds a list of templates. To classify an incoming log line:

  1. Tokenize the message and replace tokens that look like variables (numbers, IPs, UUIDs, hex, IDs) with the wildcard `<*>` so that "user_id=42" and "user_id=99" share a structural shape.
  2. Walk the tree using `len(tokens)` as the first bucket key (so messages of different lengths never collide), then the first N non-wildcard tokens as inner bucket keys.
  3. At the leaf, pick the existing template with the highest token-by-token similarity. If similarity ≥ threshold, merge (replace differing tokens with `<*>`) and return that template's ID. Otherwise, register a new template.

This implementation is intentionally simpler than full Drain3 (no parameter masking learning, no parser tree compaction) but covers the 95% case for production logs. ADR-0001 tracks the longer-term plan.

func NewMiner 

func NewMiner(similarityThreshold float64, depth, maxChildren int) *Miner

NewMiner builds a Miner from config. Sensible defaults are applied when values are zero.

func (*Miner) AddCluster 

func (m *Miner) AddCluster(id string, template string, size int)

AddCluster pre-registers a known cluster (used when loading a catalog from disk so mining can resume against existing patterns).

func (*Miner) Cluster 

func (m *Miner) Cluster(message string) (id string, template string, isNew bool)

Cluster classifies a single message and returns the matched (or newly created) cluster ID, the (post-merge) template tokens joined by space, and a flag indicating whether this was the first time we saw the pattern.

func (*Miner) Reset added in v1.4.7 

func (m *Miner) Reset()

Reset forgets every learned drain cluster and the whole bucket tree so the next Cluster call starts from an empty model. It pairs with the catalog "clear all" admin action: clearing the catalog alone leaves the miner remembering pre-reset templates (it would report recurring lines as already known), so mining only truly restarts from scratch once the miner is reset too. Config (thresholds, depth) is preserved.

func (*Miner) Snapshot 

func (m *Miner) Snapshot() []MinerCluster

Snapshot returns a copy of every learned cluster — used for catalog persistence and admin endpoints.

type MinerCluster 

type MinerCluster struct {
	ID     string   // "p-<sha1[:12]>"
	Tokens []string // current template tokens (with `<*>` for variables)
	Size   int      // total observations matched into this cluster
}

MinerCluster is one learned log template plus its observation count.

type ModeResolver added in v1.4.4 

type ModeResolver interface {
	Mode(ctx context.Context, agent string) (mode string, ok bool)
}

ModeResolver resolves the effective lifecycle mode at runtime. ok=false means "no opinion" -> the worker uses the static YAML cfg.Mode.

type OverrideRule added in v1.4.7 

type OverrideRule struct {
	// ID is a stable, server-assigned identifier used for delete addressing.
	ID string `json:"id"`
	// OrgID scopes the rule to one organization. Defaults to
	// storage.DefaultOrgID for single-tenant OSS.
	OrgID string `json:"org_id,omitempty"`
	// SourceType is one of OverrideSourceLog / OverrideSourceMetric /
	// OverrideSourceTrace.
	SourceType string `json:"source_type"`
	// Match is the source-appropriate match key: a log pattern id / message
	// substring, or a metric/trace signal name (exact or `*`/`?` glob).
	Match string `json:"match"`
	// Service is the operator-chosen attribution this rule forces.
	Service string `json:"service"`
	// CreatedAt is when the rule was created.
	CreatedAt time.Time `json:"created_at"`
	// CreatedBy is a best-effort actor label for the audit seam (empty in OSS).
	CreatedBy string `json:"created_by,omitempty"`
}

OverrideRule maps a signal to an operator-chosen service. It is one durable correction: the match is a source-appropriate key (a log Pattern identity or message substring; a metric/trace signal name — exact or `*`/`?` glob) and Service is the attribution it forces.

type Pattern 

type Pattern struct {
	ID string `json:"id"`
	// OrgID scopes the pattern to one organization. Defaults to
	// storage.DefaultOrgID ("default") so single-tenant OSS catalogs are
	// unaffected; enterprise multi-tenant routing reads it to isolate
	// per-org catalogs.
	OrgID     string    `json:"org_id,omitempty"`
	Template  string    `json:"template"`
	FirstSeen time.Time `json:"first_seen"`
	LastSeen  time.Time `json:"last_seen"`
	Count     int       `json:"count"`
	// BaselineFrequency is the EWMA of the pattern's per-second match RATE
	// (tick matches ÷ poll seconds). Folding a rate rather than the raw per-tick
	// count makes the number intuitive (~38/s, not ~1151/30s-tick) and
	// independent of the poll interval. Computed during training; consumed by
	// the spike detector in detect mode. (When no fold config is wired — the
	// pre-rate default, exercised by unit tests — it folds the raw per-tick
	// count, unchanged.)
	BaselineFrequency float64 `json:"baseline_frequency"`
	// BaselineVariance is the EWMA variance folded alongside BaselineFrequency
	// (West's incremental form), giving the spike detector the dispersion its
	// z-score needs. 0 until the pattern re-learns after the variance migration
	// (`omitempty` keeps a pre-feature catalog byte-identical on read-back).
	BaselineVariance float64 `json:"baseline_variance,omitempty"`
	// BaselineAvg is the cumulative arithmetic mean of the per-second match RATE
	// (total ÷ number of folded ticks), updated incrementally each fold. It is
	// the CENTER the "average" baseline mode scores against; that mode reuses
	// the global EWMA spread (BaselineVariance) as its scale, so there is one
	// dispersion source and only the center differs from the default mode. 0
	// until the pattern re-learns after the baseline-mode migration
	// (`omitempty` keeps a pre-feature catalog byte-identical on read-back).
	BaselineAvg float64 `json:"baseline_avg,omitempty"`
	// Seasonal is the per-time-bucket EWMA rate (hour-of-day = 24 buckets, or
	// hour-of-week = 168, UTC), letting the detector score a tick against the
	// normal-for-this-hour rate. Empty unless seasonal spike detection is
	// enabled; length is the configured period. `omitempty` keeps a pre-feature
	// catalog byte-identical on read-back.
	Seasonal []stats.EWMA `json:"seasonal,omitempty"`
	// SpikeBaselineMode is the operator's per-pattern override of which learned
	// baseline the spike z-score is scored against ("default" | "average" |
	// "time_of_day"). Empty means inherit the agent.catalog.spike_baseline_mode
	// config default. `omitempty` keeps a pre-feature catalog byte-identical on
	// read-back.
	SpikeBaselineMode string `json:"spike_baseline_mode,omitempty"`
	// Verdict is the agent's classification of this pattern: "known" once
	// it is part of baseline (auto-promoted by count or set explicitly via
	// the admin API), otherwise empty. Operators flip a pattern to
	// "known" by POSTing {"verdict":"known"} to /api/agent/patterns/:id.
	Verdict string `json:"verdict"`
	// RuleName is the regex tag attached on first sighting ("default" when
	// only the default pattern matched, or the named rule otherwise).
	RuleName string `json:"rule_name"`
	// Source is the SignalSource name where the pattern was first observed.
	Source string `json:"source"`
	// Service is the service name extracted from the pattern's first
	// matching log message (via the agent.service_pattern regex). Empty
	// when service detection is disabled or the regex did not match. The
	// agent uses this to gate detect-mode AI analysis behind the
	// new-service grace window.
	Service string `json:"service,omitempty"`
	// Tags are arbitrary operator-supplied markers.
	Tags []string `json:"tags,omitempty"`
	// Samples is a bounded ring (≤ SampleRingCap) of the most recent
	// POST-REDACTION example log lines this pattern was learned from, ordered
	// oldest→newest so the LATEST is Samples[len-1]. Populated inside
	// Catalog.RecordSample via PushSample (re-scrubbed at the storage boundary);
	// Signal.Raw is NEVER a source — the log brain passes the already-redacted
	// Observation sample message. It rides the whole-blob catalog Persist with
	// no new persistence wiring, is STRIPPED from the /api/agent/patterns list
	// rows (surfaced only on the pattern detail read), and is `omitempty` so a
	// pre-feature catalog reads back byte-identical.
	Samples []string `json:"samples,omitempty"`
}

Pattern is one entry in the on-disk catalog (`patterns.json`).

The catalog is the agent's long-term memory. During training we add patterns; during shadow / detect we look them up to decide whether a signal is "known". Operators curate it via the admin REST endpoints.

type Redactor 

type Redactor struct {
	// contains filtered or unexported fields
}

Redactor scrubs sensitive substrings before any other component sees them. It is intentionally regex-based (not a full parser) — the goal is to make it operationally reasonable to send log content to an external LLM, not to be a perfect DLP solution.

func NewRedactor 

func NewRedactor(redactIPs bool, extra []string) (*Redactor, []error)

NewRedactor builds a Redactor from defaults + user-supplied extra patterns. Invalid extra patterns are skipped (with their compile error returned in the slice for the caller to log).

func (*Redactor) Scrub 

func (r *Redactor) Scrub(s string) string

Scrub returns s with every match of every rule replaced by `<REDACTED:<rule>>` (or the rule's own replacement template, when set, so a rule like basic_auth can keep the delimiters it only matched for context).

func (*Redactor) ScrubFields 

func (r *Redactor) ScrubFields(fields map[string]interface{}) map[string]interface{}

ScrubFields recursively scrubs string values inside a fields map. Maps and slices are walked; non-string scalars are returned untouched.

type RegexMatcher 

type RegexMatcher struct {
	// contains filtered or unexported fields
}

RegexMatcher is a small helper around a list of RegexRule plus an optional "default" pattern. It acts as the agent's pre-filter: only signals whose message matches at least one rule (named or default) are forwarded to the pattern miner / catalog. Set `regex.default_pattern: ".*"` to learn from every line.

func NewRegexMatcher 

func NewRegexMatcher(cfg config.AgentRegexConfig) (*RegexMatcher, []error)

NewRegexMatcher compiles user-supplied rules. Compilation errors are reported but the matcher is still returned with the rules that did compile, so a single bad rule cannot disable the entire pipeline.

func (*RegexMatcher) Match 

func (m *RegexMatcher) Match(message string) MatchResult

Match runs explicit rules first (in declaration order — first hit wins), then falls back to the default pattern.

type RegexRule 

type RegexRule struct {
	Name    string
	Pattern *regexp.Regexp
}

RegexRule is a compiled user-defined rule.

type ServiceInfo 

type ServiceInfo struct {
	// OrgID scopes the service entry to one organization. Defaults to
	// storage.DefaultOrgID ("default"); see Pattern.OrgID.
	OrgID     string    `json:"org_id,omitempty"`
	FirstSeen time.Time `json:"first_seen"`
	// Manual is true when an operator created the service by hand via the admin
	// API (so it is selectable as an override target BEFORE any signal is
	// attributed to it). Auto-discovered services leave it false. A manual
	// service coexists with auto-discovery: RegisterService never clobbers an
	// existing entry, so a name seen in a real signal keeps its manual flag.
	//
	// Serialized WITHOUT omitempty so the origin is ALWAYS determinable in the
	// services API: an auto-discovered row returns "manual":false explicitly
	// rather than dropping the key, letting the UI render an "Auto vs Manual"
	// origin column for every service.
	Manual bool `json:"manual"`
}

ServiceInfo tracks when a service was first seen by the agent. Stored in the same patterns.json file alongside patterns — one data store, no Redis dependency for this feature.

type ServiceMatcher 

type ServiceMatcher struct {
	// contains filtered or unexported fields
}

ServiceMatcher extracts a service name from a log message using an ordered list of regexes. The first pattern that matches wins; the FIRST capture group is returned. A nil/empty matcher returns "" — service detection is off and every signal is attributed to "_unknown" in the worker. There is no built-in default list: operators who want service detection MUST configure `agent.service_patterns` (or set `AGENT_SERVICE_PATTERNS`).

func NewServiceMatcher 

func NewServiceMatcher(patterns []string) (*ServiceMatcher, []error)

NewServiceMatcher compiles the supplied regexes. Bad patterns are reported in the returned error slice but do not prevent the matcher from being built with whatever did compile, so a single typo cannot disable the entire pipeline. An empty/nil `patterns` list yields a matcher whose Extract always returns "" (service detection disabled).

func (*ServiceMatcher) Extract 

func (m *ServiceMatcher) Extract(message string) string

Extract returns the first capture group of the first matching pattern, or "" when nothing matches.

Four generic correctness guards run here so they benefit every configured pattern at once:

  • ANSI escape sequences are stripped from the message before matching, so a colour-wrapped token (e.g. Spring Boot's "\x1b[34mlead-service\x1b[m") is matchable by ordinary patterns.
  • A bare log-LEVEL word (TRACE/DEBUG/INFO/WARN/WARNING/ERROR/FATAL) is never returned as a service. If a pattern's first group captures exactly a level token, we skip it and continue to the next pattern, so a greedy positional pattern cannot misattribute a signal to "DEBUG".
  • A purely-numeric/separator token (no ASCII letter — e.g. "1210", "8080", "10.0.0.1") is never returned as a service. The bracket/syslog patterns capture with [A-Za-z0-9._-]+, so a bracketed thread id / PID / port like "[1210]" would otherwise surface as the service; the number guard skips it and continues to the next pattern. A name that merely CONTAINS digits (has a letter) — "s3", "api-v2" — is kept.
  • A JVM / servlet-container / reactor THREAD name (e.g. "nio-8080-exec-8", "pool-2-thread-1", "reactor-http-nio-4") is never returned as a service. Those names contain letters, so the number guard does not catch them; a bracket rule would otherwise file a signal under the Tomcat worker thread instead of the service. The thread-name guard skips such a capture and continues to the next pattern (the logger / real service).

type ServiceOverride added in v1.4.7 

type ServiceOverride interface {
	ResolveService(ctx context.Context, in ServiceOverrideInput) (service string, ok bool)
}

ServiceOverride is the OPTIONAL per-org manual-attribution resolver. It returns the operator-chosen service for a signal, or ("", false) when no override applies (keep the auto-detected service). Implementations must be safe for concurrent use — ticks run one goroutine per source.

type ServiceOverrideInput added in v1.4.7 

type ServiceOverrideInput struct {
	// SourceType is one of OverrideSourceLog / OverrideSourceMetric /
	// OverrideSourceTrace. A rule only matches an input of the SAME type.
	SourceType string
	// Service is the auto-detected attribution BEFORE the override runs —
	// the regex ServiceMatcher.Extract result, "_unknown" when none matched.
	// It is returned unchanged when no override applies.
	Service string
	// Signal is the logical signal / series name (a metric golden-signal or a
	// trace operation label). Empty for plain logs.
	Signal string
	// Pattern is the mined pattern / template identity the log signal folded
	// into (the durable log match key). Empty for metrics/traces.
	Pattern string
	// Message is the raw (redacted) log message, used for substring matching
	// when a log rule keys on a matched substring rather than a pattern id.
	// Empty for metrics/traces.
	Message string
}

ServiceOverrideInput carries everything a resolver needs to decide the manual attribution for one signal across all three source types. The match key is source-appropriate: logs match on the mined Pattern identity or a Message substring; metrics/traces match on the Signal (series) name (exact or glob).

type ServiceOverrideStore added in v1.4.7 

type ServiceOverrideStore struct {
	// contains filtered or unexported fields
}

ServiceOverrideStore is the process-wide, per-org override rule store. It implements the ServiceOverride seam (ResolveService) AND the CRUD the admin API drives (List / Put / Delete). All methods are safe for concurrent use.

func LoadServiceOverrideStore added in v1.4.7 

func LoadServiceOverrideStore(store storage.Provider) (*ServiceOverrideStore, error)

LoadServiceOverrideStore opens the override blob from the provider (or starts empty when none exists / store is nil). A parse error returns the empty store plus the error so boot can log-and-continue, exactly like LoadCatalog.

func (*ServiceOverrideStore) CountForService added in v1.4.7 

func (s *ServiceOverrideStore) CountForService(org, service string) int

CountForService reports how many of org's rules target the named service. The service-delete path uses it to refuse orphaning overrides.

func (*ServiceOverrideStore) Delete added in v1.4.7 

func (s *ServiceOverrideStore) Delete(org, id string) (bool, error)

Delete removes one rule by id within org. Returns false when the rule does not exist (or belongs to another org).

func (*ServiceOverrideStore) List added in v1.4.7 

func (s *ServiceOverrideStore) List(org string) []OverrideRule

List returns a stable, sorted snapshot of one org's rules (newest first).

func (*ServiceOverrideStore) Put added in v1.4.7 

func (s *ServiceOverrideStore) Put(org string, in OverrideRule) (OverrideRule, error)

Put validates and persists one new override rule for org, assigning a stable ID. It rejects an unknown source type, a blank match/service, an over-long entry, and an over-full rule set. A rule with the SAME (source_type, match) as an existing one REPLACES it (last correction wins) so a re-reassign does not stack duplicates. The stored rule (with its assigned ID) is returned.

func (*ServiceOverrideStore) RepointService added in v1.4.7 

func (s *ServiceOverrideStore) RepointService(org, oldService, newService string) (int, error)

RepointService moves every rule targeting oldService to newService within org (used when a manual service is renamed so its overrides never dangle). Returns the number of rules repointed.

func (*ServiceOverrideStore) ResolveService added in v1.4.7 

func (s *ServiceOverrideStore) ResolveService(ctx context.Context, in ServiceOverrideInput) (string, bool)

ResolveService implements ServiceOverride. It returns the operator-chosen service for the first rule that matches in.SourceType + match key, or ("", false) when none applies. HOT PATH: an RLock read of the org's rule set (no storage I/O) + a match scan. The org is deployment-supplied via ctx (ContextWithOverrideOrg), never caller-supplied.

type ShadowEvent 

type ShadowEvent struct {
	PatternID     string    `json:"pattern_id"`
	Template      string    `json:"template"`
	Source        string    `json:"source"`
	Service       string    `json:"service,omitempty"` // attributed service (may be blank/_unknown)
	RuleName      string    `json:"rule_name,omitempty"`
	Verdict       string    `json:"verdict"` // "unknown" | "spike"
	SampleMessage string    `json:"sample_message"`
	Count         int       `json:"count"`       // total signals across all ticks
	Occurrences   int       `json:"occurrences"` // number of ticks that flagged this
	FirstSeen     time.Time `json:"first_seen"`
	LastSeen      time.Time `json:"last_seen"`
}

ShadowEvent is one "we would have alerted on this" record produced by the worker while running in shadow mode. Events are coalesced per (source, pattern_id): repeat hits update Count / LastSeen / Occurrences rather than appending new rows, so the on-disk log stays small and operator-reviewable.

type ShadowLog 

type ShadowLog struct {
	// contains filtered or unexported fields
}

ShadowLog is the in-memory + on-disk store of shadow-mode verdicts.

All public methods are safe for concurrent use. Disk persistence is debounced — Record sets a dirty flag that the worker flushes at most once per `persist_interval`.

func LoadShadowLog 

func LoadShadowLog(store storage.Provider, max int) (*ShadowLog, error)

LoadShadowLog opens an existing shadow log from the storage provider or returns an empty one when no blob is present. `max` caps distinct events; pass 0 for the default. A nil store disables persistence (in-memory only).

func (*ShadowLog) All 

func (s *ShadowLog) All() []*ShadowEvent

All returns a snapshot of every shadow event, sorted by LastSeen descending (most recent first). Returned events are copies — callers cannot mutate the log.

func (*ShadowLog) Clear 

func (s *ShadowLog) Clear() int

Clear removes every event. The change is persisted on the next Persist.

func (*ShadowLog) Dirty 

func (s *ShadowLog) Dirty() bool

Dirty reports whether there are unflushed changes.

func (*ShadowLog) Len 

func (s *ShadowLog) Len() int

Len returns the number of distinct (source, pattern) events.

func (*ShadowLog) Persist 

func (s *ShadowLog) Persist() error

Persist atomically writes the shadow log via the storage backend. Safe to call concurrently with Record / Clear. No-op when store is nil.

func (*ShadowLog) Record 

func (s *ShadowLog) Record(source, patternID, service, template, sample, rule, verdict string, freq int)

Record merges a shadow-mode hit into the log. The record is coalesced per (source, pattern_id); repeat hits bump Count and Occurrences instead of appending a new entry. `freq` is the number of signals observed in the current worker tick.

func (*ShadowLog) Stats 

func (s *ShadowLog) Stats() map[string]int

Stats returns aggregate counts useful for /api/agent/shadow/stats.

type SpikeSettings added in v1.4.9 

type SpikeSettings struct {
	// BaselineMode is the GLOBAL default baseline the spike z-score is measured
	// against for any pattern that does not carry its own per-pattern override:
	// "default" (global rate baseline), "average" (cumulative mean of the rate,
	// never decays), or "time_of_day" (hour-of-day baseline).
	BaselineMode string `json:"baseline_mode"`
}

SpikeSettings is the non-secret runtime configuration for the spike detector's global default baseline mode. It is the JSON shape persisted in the settings blob and the shape the admin GET/PUT endpoints exchange.

func DefaultSpikeSettings added in v1.4.9 

func DefaultSpikeSettings() SpikeSettings

DefaultSpikeSettings is the built-in floor applied when the store holds no value: the global rate baseline. A fresh install therefore scores spikes against the global baseline until an operator picks another mode in the UI.

func LoadSpikeSettings added in v1.4.9 

func LoadSpikeSettings(st storage.Provider) SpikeSettings

LoadSpikeSettings returns the effective spike settings: the stored blob merged over the built-in default, sanitized. A nil store or an absent/empty/corrupt blob yields the built-in default — never an error, mirroring the ReadBlob "fresh start" contract. Callers get a fresh value each time, so there is no shared mutable state to guard.

type Worker 

type Worker struct {
	// contains filtered or unexported fields
}

Worker runs the agent loop: 

Pull (per source) → Redact → Cluster → Catalog upsert → (mode-specific tail)

Training mode is fully end-to-end. Shadow and detect modes share the same classification path; shadow records would-have-alerted events to the shadow log, while detect calls the AI SRE and emits an incident through the existing services.CreateIncident pipeline.

func NewWorker 

func NewWorker(opt WorkerOptions) (*Worker, error)

NewWorker validates options and applies defaults.

func (*Worker) Run 

func (w *Worker) Run(ctx context.Context)

Run drives the worker until ctx is canceled. It is intended to be called in a goroutine from cmd/main.go.

type WorkerOptions 

type WorkerOptions struct {
	Cfg      config.AgentConfig
	Sources  []core.SignalSource
	Cursors  *CursorStore // optional; pass nil for in-memory cursors
	Redactor *Redactor
	Matcher  *RegexMatcher
	Miner    *Miner
	Catalog  *Catalog
	Shadow   *ShadowLog      // optional; pass nil to disable shadow recording
	Detect   *DetectLog      // optional; pass nil to disable detect-call audit log
	Services *ServiceMatcher // optional; pass nil to disable service detection

	// AI is the detect-mode bundle (analyzer + cache + rate limiter).
	// Pass a zero-value AIBundle to disable AI emission.
	AI AIBundle
	// Emitter is invoked for each finding in detect mode. nil disables
	// emission (worker still calls AI and caches the result, but the
	// finding does not flow through to channels).
	Emitter Emitter
}

WorkerOptions bundles the dependencies a Worker needs. Construction does not connect to anything; the worker dials lazily inside Run.

Source Files

View all Source files

Directories

Path Synopsis
ai
analyze
Package analyze contains the analyze-kind AI agent: operator- triggered, tool-using, single-incident investigation.
 Package analyze contains the analyze-kind AI agent: operator- triggered, tool-using, single-incident investigation.
analyze/tools
Package tools holds the read-only tool catalog exposed to the analyze-kind AI agent.
 Package tools holds the read-only tool catalog exposed to the analyze-kind AI agent.
detect
Package detect contains the detect-kind AI agent: cheap, tool-free, single-call classification of unknown / spiking log patterns.
 Package detect contains the detect-kind AI agent: cheap, tool-free, single-call classification of unknown / spiking log patterns.
eino
Package eino contains the Eino-backed chat model wrapper used by all AI agents in versus-incident.
 Package eino contains the Eino-backed chat model wrapper used by all AI agents in versus-incident.
prompt
Package prompt is a content-free helper for assembling multi-file system prompts.
 Package prompt is a content-free helper for assembling multi-file system prompts.
router
Package router dispatches AITasks to the right AIAgent and wraps each call with per-kind caching and per-kind rate limiting.
 Package router dispatches AITasks to the right AIAgent and wraps each call with per-kind caching and per-kind rate limiting.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.