engine

package
v1.10.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 25, 2026 License: MIT Imports: 34 Imported by: 0

Documentation

Overview

Package engine provides the central Sentinel evaluation coordinator.

Index

Constants

View Source 
const (
	MetricRunDuration    = "sentinel_run_duration_seconds"
	MetricRunsInFlight   = "sentinel_runs_in_flight"
	MetricCasesEvaluated = "sentinel_cases_evaluated_total"
	MetricTargetDuration = "sentinel_target_call_duration_seconds"
	MetricTargetTokens   = "sentinel_target_tokens_total"
	MetricScorerDuration = "sentinel_scorer_duration_seconds"
	MetricRegressions    = "sentinel_regressions_detected_total"
)

Metric names. A Forge app exports them with its namespace in front, so sentinel_run_duration_seconds scrapes as forge_sentinel_run_duration_seconds.

View Source 
const InstrumentationName = "github.com/xraph/sentinel"

InstrumentationName is the tracer name on every span sentinel starts. It is the module path, so a trace backend shows which library a span came from.

Variables

This section is empty.

Functions

This section is empty.

Types

type Engine 

type Engine struct {
	// contains filtered or unexported fields
}

Engine is the central coordinator for the Sentinel evaluation pipeline.

func New 

func New(opts ...Option) (*Engine, error)

New creates a new Engine with the given options.

func (*Engine) CancelRun 

func (e *Engine) CancelRun(ctx context.Context, runID id.EvalRunID) error

CancelRun asks a running run to stop. The runner stops scheduling cases at its next check; cases already in flight finish and are stored. It works from any replica because the signal lives in the store.

func (*Engine) Config 

func (e *Engine) Config() sentinel.Config

Config returns a copy of the engine's configuration.

func (*Engine) CountCases 

func (e *Engine) CountCases(ctx context.Context, suiteID id.SuiteID) (int64, error)

CountCases returns the number of test cases for a suite.

func (*Engine) CreateCase 

func (e *Engine) CreateCase(ctx context.Context, tc *testcase.Case) error

CreateCase creates a new test case.

func (*Engine) CreateCaseBatch 

func (e *Engine) CreateCaseBatch(ctx context.Context, cases []*testcase.Case) error

CreateCaseBatch creates multiple test cases.

func (*Engine) CreatePromptVersion 

func (e *Engine) CreatePromptVersion(ctx context.Context, pv *promptversion.PromptVersion) error

CreatePromptVersion creates the next version of a suite's prompt. The engine assigns Version as the suite's highest plus one, and retries once when a concurrent create took the same number. IsCurrent on the input makes the new version current, through SetCurrentPromptVersion, so a suite never has two current versions.

func (*Engine) CreateSuite 

func (e *Engine) CreateSuite(ctx context.Context, s *suite.Suite) error

CreateSuite creates a new evaluation suite.

func (*Engine) DeleteBaseline 

func (e *Engine) DeleteBaseline(ctx context.Context, baselineID id.BaselineID) error

DeleteBaseline removes a baseline.

func (*Engine) DeleteCase 

func (e *Engine) DeleteCase(ctx context.Context, caseID id.CaseID) error

DeleteCase removes a test case.

func (*Engine) DeleteSuite 

func (e *Engine) DeleteSuite(ctx context.Context, suiteID id.SuiteID) error

DeleteSuite removes a suite.

func (*Engine) EnqueueRecorded 

func (e *Engine) EnqueueRecorded(ctx context.Context, req RecordedRunRequest) (*RecordedJob, error)

EnqueueRecorded acknowledges only persisted, exactly scoped captured evidence.

func (*Engine) EvaluateRecorded 

func (e *Engine) EvaluateRecorded(ctx context.Context, req RecordedRunRequest) (*RunResult, error)

EvaluateRecorded scores captured outputs and reuses durable completed results. Caller cancellation returns the persisted partial result with ErrRunCancelled.

func (*Engine) Extensions 

func (e *Engine) Extensions() *plugin.Registry

Extensions returns the extension registry.

func (*Engine) FreezeEvaluation 

func (e *Engine) FreezeEvaluation(ctx context.Context, req FreezeRequest) (*EvaluationPlan, error)

FreezeEvaluation resolves case, prompt and scorer definitions before execution.

func (*Engine) GenerateRedTeam 

func (e *Engine) GenerateRedTeam(ctx context.Context, suiteID id.SuiteID, types []redteam.AttackType, count int) ([]*testcase.Case, error)

GenerateRedTeam writes attack cases for each type into a suite, up to redteam.MaxPerType per type. It validates every type before generating anything, so a refusal writes nothing.

Leakage and injection cases carry the suite's effective prompt as the substring their not_contains scorer looks for. With no prompt that scorer falls back to the case's Expected text and checks the wrong thing, so both are refused for a suite without one.

func (*Engine) GetBaseline 

func (e *Engine) GetBaseline(ctx context.Context, baselineID id.BaselineID) (*baseline.Baseline, error)

GetBaseline retrieves a baseline by ID.

func (*Engine) GetCase 

func (e *Engine) GetCase(ctx context.Context, caseID id.CaseID) (*testcase.Case, error)

GetCase retrieves a test case by ID.

func (*Engine) GetCurrentPromptVersion 

func (e *Engine) GetCurrentPromptVersion(ctx context.Context, suiteID id.SuiteID) (*promptversion.PromptVersion, error)

GetCurrentPromptVersion returns the current prompt version for a suite.

func (*Engine) GetLatestBaseline 

func (e *Engine) GetLatestBaseline(ctx context.Context, suiteID id.SuiteID) (*baseline.Baseline, error)

GetLatestBaseline retrieves the latest baseline for a suite.

func (*Engine) GetPromptVersion 

func (e *Engine) GetPromptVersion(ctx context.Context, pvID id.PromptVersionID) (*promptversion.PromptVersion, error)

GetPromptVersion retrieves a prompt version by ID.

func (*Engine) GetResultStats 

func (e *Engine) GetResultStats(ctx context.Context, runID id.EvalRunID) (*evalrun.ResultStats, error)

GetResultStats returns aggregate statistics for a run.

func (*Engine) GetRun 

func (e *Engine) GetRun(ctx context.Context, runID id.EvalRunID) (*evalrun.Run, error)

GetRun retrieves an evaluation run by ID.

func (*Engine) GetSuite 

func (e *Engine) GetSuite(ctx context.Context, suiteID id.SuiteID) (*suite.Suite, error)

GetSuite retrieves a suite by ID.

func (*Engine) GetSuiteByName 

func (e *Engine) GetSuiteByName(ctx context.Context, appID, name string) (*suite.Suite, error)

GetSuiteByName retrieves a suite by app and name.

func (*Engine) Health 

func (e *Engine) Health(ctx context.Context) error

Health checks the health of the engine by pinging its store.

func (*Engine) ImportCases 

func (e *Engine) ImportCases(ctx context.Context, suiteID id.SuiteID, format string, data []byte) (int64, error)

ImportCases parses cases in json, csv or jsonl and writes them to a suite. The store's ImportCases was a stub in every backend and is no longer called.

func (*Engine) ListBaselines 

func (e *Engine) ListBaselines(ctx context.Context, suiteID id.SuiteID) ([]*baseline.Baseline, error)

ListBaselines returns all baselines for a suite.

func (*Engine) ListCases 

func (e *Engine) ListCases(ctx context.Context, suiteID id.SuiteID) ([]*testcase.Case, error)

ListCases returns all test cases for a suite.

func (*Engine) ListPromptVersions 

func (e *Engine) ListPromptVersions(ctx context.Context, suiteID id.SuiteID) ([]*promptversion.PromptVersion, error)

ListPromptVersions returns all prompt versions for a suite.

func (*Engine) ListResults 

func (e *Engine) ListResults(ctx context.Context, runID id.EvalRunID) ([]*evalrun.Result, error)

ListResults returns all results for an evaluation run.

func (*Engine) ListRuns 

func (e *Engine) ListRuns(ctx context.Context, filter *evalrun.ListFilter) ([]*evalrun.Run, error)

ListRuns returns evaluation runs matching the given filter.

func (*Engine) ListRunsBySuite 

func (e *Engine) ListRunsBySuite(ctx context.Context, suiteID id.SuiteID) ([]*evalrun.Run, error)

ListRunsBySuite returns all evaluation runs for a suite.

func (*Engine) ListSuites 

func (e *Engine) ListSuites(ctx context.Context, filter *suite.ListFilter) ([]*suite.Suite, error)

ListSuites returns suites matching the given filter.

func (*Engine) Logger 

func (e *Engine) Logger() log.Logger

Logger returns the engine's logger.

func (*Engine) RecordedStatus 

func (e *Engine) RecordedStatus(ctx context.Context) (RecordedWorkerStatus, error)

RecordedStatus is conservative: readiness requires a persistent store and a running worker in this process for the caller's exact app and tenant.

func (*Engine) RunEval 

func (e *Engine) RunEval(ctx context.Context, cfg *RunConfig) (*RunResult, error)

RunEval runs a suite synchronously with target and scorer values, for Go and CI callers. Results are stored as each case finishes, exactly as for StartRun, and cancelling ctx cancels the run. A run that ends cancelled, by ctx or by CancelRun, returns its partial result together with an error that wraps sentinel.ErrRunCancelled (and ctx's error when ctx ended).

func (*Engine) SaveBaseline 

func (e *Engine) SaveBaseline(ctx context.Context, b *baseline.Baseline) error

SaveBaseline saves a baseline.

func (*Engine) Scorers 

func (e *Engine) Scorers() *scorer.Registry

Scorers returns the scorer registry: the built-ins plus whatever the application registered with WithScorer.

func (*Engine) SetCurrentPromptVersion 

func (e *Engine) SetCurrentPromptVersion(ctx context.Context, suiteID id.SuiteID, pvID id.PromptVersionID) error

SetCurrentPromptVersion makes a version current for its suite. A version from another suite is refused with ErrPromptVersionNotFound before anything changes.

func (*Engine) Start 

func (e *Engine) Start(_ context.Context) error

Start initialises the engine. Reserved for future background processes.

func (*Engine) StartRun 

func (e *Engine) StartRun(ctx context.Context, cfg *StartConfig) (*evalrun.Run, error)

StartRun validates a run, creates it and returns at once. Evaluation continues on the engine's own context, so the request that started it can return; poll the run and its results to watch it, and CancelRun to stop it. Every refusal happens before anything is written.

func (*Engine) Stop 

func (e *Engine) Stop(ctx context.Context) error

Stop cancels runs this process is evaluating, waits up to ShutdownTimeout for them to record their cancellation, then shuts extensions and the store down. A run cancelled here ends cancelled, with the results it stored.

func (*Engine) Store 

func (e *Engine) Store() store.Store

Store returns the engine's store.

func (*Engine) Target 

func (e *Engine) Target(name string) (RegisteredTarget, bool)

Target returns a registered target by name.

func (*Engine) Targets 

func (e *Engine) Targets() []RegisteredTarget

Targets lists the registered targets, sorted by name.

func (*Engine) UpdateCase 

func (e *Engine) UpdateCase(ctx context.Context, tc *testcase.Case) error

UpdateCase updates an existing test case.

func (*Engine) UpdateSuite 

func (e *Engine) UpdateSuite(ctx context.Context, s *suite.Suite) error

UpdateSuite updates an existing suite.

func (*Engine) WorkRecorded 

func (e *Engine) WorkRecorded(ctx context.Context) error

WorkRecorded runs explicitly scoped durable work until cancellation or Stop. Hosts call it for each authenticated app/tenant binding. It never scans across scopes and never invokes a target. Memory backends support tests only.

type EvaluationPlan 

type EvaluationPlan = evalplan.Plan

EvaluationPlan is an immutable evaluation definition persisted by Sentinel.

type FreezeRequest 

type FreezeRequest struct {
	SuiteID id.SuiteID
	CaseIDs []id.CaseID
	Scorers []evalplan.Criterion
}

FreezeRequest selects cases and additional criteria. Empty CaseIDs selects all.

type Option 

type Option func(*Engine) error

Option configures the Engine.

func WithConfig 

func WithConfig(cfg sentinel.Config) Option

WithConfig sets the engine configuration.

func WithExtension 

func WithExtension(extension plugin.Extension) Option

WithExtension registers an extension with the engine.

func WithLogger 

func WithLogger(l log.Logger) Option

WithLogger sets the structured logger.

func WithMetrics 

func WithMetrics(m gometrics.Metrics) Option

WithMetrics records the engine's metrics on m: run, case, target and scorer timings, tokens, runs in flight and regressions. In a Forge app pass app.Metrics(), and the series land on /_/metrics with the forge_ prefix. sentinel's Forge extension does this for you. A nil m records nothing, which is the default.

Spans need no option: the engine starts them on the global tracer provider, which forge.New installs.

func WithScorer 

func WithScorer(d scorer.Descriptor, f scorer.Factory) Option

WithScorer registers a scorer factory with a descriptor. This is how an application makes the LLM scorers available, since only it holds an LLMClient.

func WithStore 

func WithStore(s store.Store) Option

WithStore sets the evaluation store.

func WithTarget 

func WithTarget(name, description string, t target.Target) Option

WithTarget registers a named target, so a run can be started by name.

type RecordedCase 

type RecordedCase = recorded.Case

RecordedCase is one captured case and repetition.

type RecordedJob 

type RecordedJob = recorded.Job

RecordedJob is durable captured-evaluation work and its result checkpoints.

type RecordedRunRequest 

type RecordedRunRequest = recorded.Request

RecordedRunRequest carries captured evidence without an executable target.

type RecordedWorkerStatus 

type RecordedWorkerStatus struct {
	Durable       bool `json:"durable"`
	WorkerRunning bool `json:"worker_running"`
	Ready         bool `json:"ready"`
}

RecordedWorkerStatus describes local worker and backend availability.

type RegisteredTarget 

type RegisteredTarget struct {
	Name        string
	Description string
	Target      target.Target
}

RegisteredTarget is a target an application named, so a run can be started by name from somewhere that cannot hold a Go value, such as the dashboard.

type RunConfig 

type RunConfig struct {
	SuiteID     id.SuiteID
	Model       string
	Target      target.Target
	Scorers     []scorer.Scorer
	PersonaRef  string
	Tags        []string
	Concurrency int
}

RunConfig configures a single evaluation run.

type RunResult 

type RunResult struct {
	Recorded *recorded.Job // Non-nil for captured execution evidence, including usage quality.
	Run      *evalrun.Run
	Results  []*evalrun.Result
	Stats    *evalrun.ResultStats
}

RunResult holds the outcome of an evaluation run.

type StartConfig 

type StartConfig struct {
	SuiteID id.SuiteID
	Target  string
	Scorers []string
	Model   string
}

StartConfig starts a run by the names an application registered with WithTarget and WithScorer.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.