evalqueue

package
v0.21.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 10, 2026 License: Apache-2.0 Imports: 24 Imported by: 0

Documentation

Overview

Package evalqueue is the eval job queue, backed by Postgres claim/lease/heartbeat.

Index

Constants

View Source
const (
	ProviderAnthropic        = "anthropic"
	ProviderOpenAI           = "openai"
	ProviderOpenRouter       = "openrouter"
	ProviderOpenAICompatible = "openai-compatible"
)

Provider types.

View Source
const (
	EventPhase        = "phase"
	EventHealth       = "health"
	EventPlan         = "plan"
	EventSessionStart = "session_start"
	EventSessionEnd   = "session_end"
	EventGrade        = "grade"
	EventCancelled    = "cancelled"
	EventFailed       = "failed"
)

Job event kinds a worker posts, plus cancelled, which the server records.

View Source
const EventRetention = 30 * 24 * time.Hour

EventRetention is how long a finished job keeps its events.

View Source
const StalledAfter = 3 * time.Minute

StalledAfter is how long a running job may go without a job event before it shows as stalled.

Variables

View Source
var ErrLeaseLost = errors.New("evalqueue: lease lost")

ErrLeaseLost is returned when a heartbeat or fail targets a job that is no longer this worker's — a cancelled job, or one another worker reclaimed.

View Source
var ErrNotCancellable = errors.New("evalqueue: job is not queued or running")

ErrNotCancellable is returned when Cancel targets a job that has already finished (done, failed, or cancelled) or does not exist.

ProviderTypes lists every provider type, in the order the UI offers them.

Functions

func Explain added in v0.11.0

func Explain(raw string) string

Explain reduces a worker's error chain to one sentence. The raw chain is what the worker reports and what an operator needs; this is what a reader sees first. An unrecognised failure passes through unchanged rather than becoming a vague placeholder that hides it.

func RegisterRoutes

func RegisterRoutes(api huma.API, q *PoolExecutor, qual *quality.Store, skills *skill.Store, suites *evalsuite.Registry, opts RouteOptions)

RegisterRoutes wires up the eval job queue HTTP endpoints: claim, heartbeat, report ingestion, fail, status lookup, and re-run.

func Stalled added in v0.20.0

func Stalled(status string, started, lastEvent *time.Time, now time.Time) bool

Stalled reports whether a running job has gone StalledAfter without a job event, counting from its start when it has none.

Types

type Candidate added in v0.14.0

type Candidate struct {
	SkillID   string `json:"skill_id"`
	SkillName string `json:"skill_name"`
	Version   int    `json:"version"`
	Label     string `json:"label"`
	// ReusedReport is a same-day report of this exact candidate from another
	// contest, named by ReusedFrom. The worker runs the candidate anyway when
	// the report's conditions differ from a fresh one.
	ReusedReport json.RawMessage `json:"reused_report,omitempty"`
	ReusedFrom   string          `json:"reused_from,omitempty"`
	// PRHeadID names a stored PR head the worker fetches instead of a
	// version. Version is then 0.
	PRHeadID string `json:"pr_head_id,omitempty"`
}

Candidate is one entry in a contest job.

type ContestLookup added in v0.14.0

type ContestLookup interface {
	CandidatesForClaim(ctx context.Context, jobID string) ([]Candidate, int, error)
}

ContestLookup returns the candidates a contest job must run, in order, with any same-day report the claim may reuse attached.

type DBExecutor

type DBExecutor interface {
	Exec(ctx context.Context, sql string, args ...any) (pgconn.CommandTag, error)
	QueryRow(ctx context.Context, sql string, args ...any) pgx.Row
	Query(ctx context.Context, sql string, args ...any) (pgx.Rows, error)
}

Executor is the subset of *pgxpool.Pool that PoolExecutor's queries need. It is satisfied by both *pgxpool.Pool and pgx.Tx, so a caller can run queue writes inside a transaction shared with another store (see WithExecutor) instead of each store committing its own write independently.

type EvalBackend added in v0.19.0

type EvalBackend struct {
	Provider   string     `json:"provider"`
	Type       string     `json:"type"`
	BaseURL    string     `json:"base_url,omitempty"`
	AuthHeader string     `json:"auth_header,omitempty"`
	Credential string     `json:"credential,omitempty"`
	Model      string     `json:"model"`
	Cost       *ModelCost `json:"cost,omitempty"`
}

EvalBackend is one provider and model, as a worker needs them. Credential is the provider's key: the server sends it only inside a job claim.

func (EvalBackend) Label added in v0.19.0

func (b EvalBackend) Label() string

Label names the backend in a report's model_panel: empty for Anthropic's own API, otherwise the type and the endpoint's host. It ignores the provider's name, so a rename does not split a score trend.

type EvalConfig added in v0.19.0

type EvalConfig struct {
	Judge   EvalBackend   `json:"judge"`
	Members []EvalBackend `json:"members"`
}

EvalConfig is the judge and the panel the server chose. The first member is the primary member.

type EvalJobEvent added in v0.20.0

type EvalJobEvent struct {
	ID           int64     `json:"id"`
	At           time.Time `json:"at"`
	Kind         string    `json:"kind" maxLength:"32"`
	Phase        string    `json:"phase,omitempty" maxLength:"64"`
	Total        int       `json:"total,omitempty"`
	Task         string    `json:"task,omitempty" maxLength:"200"`
	Condition    string    `json:"condition,omitempty" maxLength:"32"`
	Model        string    `json:"model,omitempty" maxLength:"200"`
	Attempt      int       `json:"attempt,omitempty"`
	Status       string    `json:"status,omitempty" maxLength:"32"`
	OK           *bool     `json:"ok,omitempty"`
	DurationMS   int64     `json:"duration_ms,omitempty"`
	InputTokens  int64     `json:"input_tokens,omitempty"`
	OutputTokens int64     `json:"output_tokens,omitempty"`
	CostUSD      float64   `json:"cost_usd,omitempty"`
	Error        string    `json:"error,omitempty" maxLength:"4000"`
	Actor        string    `json:"actor,omitempty"`
}

EvalJobEvent is one progress record of an eval job.

type EvalJobEventsBody added in v0.20.0

type EvalJobEventsBody struct {
	Events []EvalJobEvent `json:"events" maxItems:"500"`
}

EvalJobEventsBody is a batch of job events a worker posts.

type EvalJobEventsOutputBody added in v0.20.0

type EvalJobEventsOutputBody struct {
	Job    *EvalJobSummary `json:"job"`
	Events []EvalJobEvent  `json:"events"`
}

EvalJobEventsOutputBody is one job and its events.

type EvalJobSummary added in v0.20.0

type EvalJobSummary struct {
	ID          string     `json:"id"`
	Skill       string     `json:"skill"`
	Version     int        `json:"version"`
	Tier        string     `json:"tier"`
	ContestID   string     `json:"contest_id,omitempty"`
	Status      string     `json:"status"`
	WorkerID    string     `json:"worker_id"`
	Attempts    int        `json:"attempts"`
	LastError   string     `json:"last_error"`
	RequestedBy string     `json:"requested_by"`
	CreatedAt   time.Time  `json:"created_at"`
	StartedAt   *time.Time `json:"started_at,omitempty"`
	FinishedAt  *time.Time `json:"finished_at,omitempty"`
	Phase       string     `json:"phase"`
	Total       int        `json:"sessions_total"`
	Done        int        `json:"sessions_done"`
	Failed      int        `json:"sessions_failed"`
	CostUSD     float64    `json:"cost_usd"`
	LastEventAt *time.Time `json:"last_event_at,omitempty"`
	Stalled     bool       `json:"stalled"`
}

EvalJobSummary is one job on the Jobs page.

type EvalJobsBody added in v0.20.0

type EvalJobsBody struct {
	Jobs []EvalJobSummary `json:"jobs"`
}

EvalJobsBody is the Jobs page.

type Executor

type Executor interface {
	Submit(ctx context.Context, j Job) (JobID, error)
	Cancel(ctx context.Context, id JobID) error
}

Executor submits and cancels eval jobs.

type Job

type Job struct {
	ID             JobID
	SkillID        string
	SkillName      string
	Version        int
	SuiteRef       string
	Tier           string
	Panel          Panel
	Status         Status
	Attempts       int
	MaxAttempts    int
	WorkerID       string
	LeaseExpiresAt *time.Time
	LeaseSeconds   int
	LastError      string
	RequestedBy    string
	// ContestID is set when this job runs a contest. The job's SkillID,
	// SkillName and Version then hold the first candidate, so every existing
	// query on eval_jobs stays correct, and Candidates carries all of them.
	// One job, because two jobs can reach two workers with different agent CLI
	// versions on different days — the drift a contest exists to remove.
	ContestID  string
	Candidates []Candidate
	// ContestAttempts is the attempts per task per candidate the contest
	// asked for. Attempts above counts job retries.
	ContestAttempts int
	CreatedAt       time.Time
	// Eval is the server's judge and panel at claim time. It is not stored:
	// a retried job takes the configuration current at its new claim.
	Eval *EvalConfig
	// StartedAt is set once on the first claim and never moved, so elapsed
	// time is measured from the start of work, not the latest retry.
	StartedAt *time.Time
}

Job is a unit of eval work against a specific skill version.

type JobID

type JobID string

JobID identifies an eval job.

type ModelCost added in v0.20.0

type ModelCost struct {
	Input      float64 `json:"input"`
	Output     float64 `json:"output"`
	CacheRead  float64 `json:"cache_read,omitempty"`
	CacheWrite float64 `json:"cache_write,omitempty"`
}

ModelCost is a model's price in US dollars per million tokens.

type Panel

type Panel struct {
	Agents []string `json:"agents,omitempty"`
	Models []string `json:"models,omitempty"`
	// Providers names the provider of each model, by id, in the same order.
	// A model without one runs on the configured primary member's provider.
	Providers []string `json:"providers,omitempty"`
}

Panel is the requested agent/model panel for a run. Empty means the worker's default panel.

type PoolExecutor

type PoolExecutor struct {
	// contains filtered or unexported fields
}

PoolExecutor is a Postgres-backed Executor. The table itself is the queue.

func NewPool

func NewPool(db *pgxpool.Pool) *PoolExecutor

NewPool builds a PoolExecutor over db.

func (*PoolExecutor) AppendEvents added in v0.20.0

func (p *PoolExecutor) AppendEvents(ctx context.Context, id JobID, events []EvalJobEvent) error

AppendEvents stores events a worker posted for a job it holds.

func (*PoolExecutor) Cancel

func (p *PoolExecutor) Cancel(ctx context.Context, id JobID) error

Cancel stops a job that has not finished. A running job is cancelled too: the worker sees the status on its next heartbeat and abandons the run.

func (*PoolExecutor) Claim

func (p *PoolExecutor) Claim(ctx context.Context, workerID string, lease time.Duration) (*Job, string, bool, error)

Claim takes one job under SKIP LOCKED so concurrent workers never contend on the same row and never block each other. Claimability covers two cases: a queued job, and a running job whose lease lapsed — the latter is how a worker that died mid-run gives its job back without anyone reaping it.

func (*PoolExecutor) Complete

func (p *PoolExecutor) Complete(ctx context.Context, id JobID, workerID string) error

Complete marks a job done and clears its claim.

func (*PoolExecutor) Events added in v0.20.0

func (p *PoolExecutor) Events(ctx context.Context, id JobID, after int64) ([]EvalJobEvent, error)

Events returns a job's events after the given id, oldest first.

func (*PoolExecutor) Fail

func (p *PoolExecutor) Fail(ctx context.Context, id JobID, workerID, cause string) error

Fail returns the job to the pool while attempts remain and marks it failed once they do not. The cause is always recorded: a job that failed three times with no reason recorded is a support ticket.

func (*PoolExecutor) Get

func (p *PoolExecutor) Get(ctx context.Context, id JobID) (*Job, error)

Get fetches a job by ID. A missing row is not an error: it returns (nil, nil), matching the repo convention (see auth.UserStore.GetByEmail).

func (*PoolExecutor) Heartbeat

func (p *PoolExecutor) Heartbeat(ctx context.Context, id JobID, workerID string) error

Heartbeat extends the lease only while this worker still owns the job and it is still running. It re-applies the lease duration persisted at claim time (lease_seconds) rather than a fixed default — otherwise a worker that claimed a long lease would have it truncated on every beat, making the lease expire (and the job become reclaimable by a second worker) long before the original grant runs out. A cancelled job stops a worker at its next beat.

func (*PoolExecutor) JobSummaries added in v0.20.0

func (p *PoolExecutor) JobSummaries(ctx context.Context, now time.Time) ([]EvalJobSummary, error)

JobSummaries lists every queued and running job, and jobs that finished in the last week, newest first, with their progress.

func (*PoolExecutor) ListBySkill

func (p *PoolExecutor) ListBySkill(ctx context.Context, skillID string) ([]Job, error)

ListBySkill returns all jobs for a skill, most recent first.

func (*PoolExecutor) Pool

func (p *PoolExecutor) Pool() *pgxpool.Pool

Pool returns the underlying connection pool, so a caller can Begin a transaction shared with another store (see WithExecutor).

func (*PoolExecutor) PruneEvents added in v0.20.0

func (p *PoolExecutor) PruneEvents(ctx context.Context) (int64, error)

PruneEvents deletes the events of jobs that finished before EventRetention.

func (*PoolExecutor) QueuePosition

func (p *PoolExecutor) QueuePosition(ctx context.Context, id JobID) (int, error)

QueuePosition reports how many queued jobs sit ahead of id, so the UI can say "position 3" instead of showing a spinner for the 45-90 minutes an eval takes. It returns 0 for a job that is not queued - running, done, failed or absent - because a position is only meaningful while waiting.

It is a hint, not a promise: concurrency and retries move it.

func (*PoolExecutor) ReapExpired

func (p *PoolExecutor) ReapExpired(ctx context.Context) (int, error)

ReapExpired marks jobs whose lease lapsed and whose attempts are exhausted as failed. Claim recovers a lapsed lease while retries remain; without this sweep a worker that dies on its final attempt leaves the job running forever, neither retried nor reported. Returns the number reaped.

func (*PoolExecutor) RecordDerivedSuite added in v0.11.0

func (p *PoolExecutor) RecordDerivedSuite(ctx context.Context, q evalsuite.Queryer, jobID, ref string) error

RecordDerivedSuite is SetSuiteRef through a caller-supplied executor, so the job's suite_ref lands in the same transaction as the suite row it names.

func (*PoolExecutor) SetSuiteRef added in v0.11.0

func (p *PoolExecutor) SetSuiteRef(ctx context.Context, id JobID, ref string) error

SetSuiteRef records the ref a derive job's worker actually measured against. Only a job submitted with no suite_ref should ever call this — the report handler enforces that, not this query.

func (*PoolExecutor) Submit

func (p *PoolExecutor) Submit(ctx context.Context, j Job) (JobID, error)

Submit enqueues a job. It does not deduplicate: two publishes of the same version are two measurements, and the caller decides whether that is wanted.

func (*PoolExecutor) VerifyClaim

func (p *PoolExecutor) VerifyClaim(ctx context.Context, id JobID, token string) (*Job, bool, error)

VerifyClaim compares in constant time: the claim token is a bearer credential and a byte-at-a-time comparison leaks it. Status and lease liveness are checked in the same query as the hash, against the database's clock — the same one Claim, Heartbeat, and ReapExpired use — so a lapsed lease or a cancelled job (Cancel clears claim_token_hash) can never verify due to skew between the app server's clock and the database's, even if nobody has polled to notice yet.

func (*PoolExecutor) VerifyDerivePush added in v0.11.0

func (p *PoolExecutor) VerifyDerivePush(ctx context.Context, jobID, token, skillName string) (bool, error)

VerifyDerivePush answers the suite-upload route's one question: is this push coming from the worker that currently holds jobID's claim, for a job that was submitted with no suite of its own and names skillName? Anything else — a forged token, a lapsed lease, a job that already names a suite, a job for another skill — is false, so a push can never talk the server into attributing a suite to a job it does not belong to.

func (*PoolExecutor) WithExecutor

func (p *PoolExecutor) WithExecutor(e DBExecutor) *PoolExecutor

WithExecutor returns a PoolExecutor whose queries run against e (typically a pgx.Tx) instead of the pool directly, so a caller can compose a write with another store's write in one transaction.

type RouteOptions

type RouteOptions struct {
	// Releaser re-decides a held version once its score lands. Nil disables
	// the re-decision entirely: reports are still ingested, held versions
	// simply stay held.
	Releaser *skill.Releaser
	// QualityFloor is the minimum headline score a verified report must
	// reach to clear a held version. It comes from
	// the quality_floor setting, the same value publish decides with. Nil
	// means 0.
	QualityFloor func() float64
	// Contests supplies a contest job's candidates at claim time. A local
	// interface, because internal/contest imports this package's Job type.
	Contests ContestLookup
	// EvalConfig returns the judge and panel for a job's requested panel, keys
	// included, or nil when none is configured.
	EvalConfig func(ctx context.Context, requested Panel) (*EvalConfig, error)
}

RouteOptions carries the collaborators the report handler needs to act on a score beyond storing it. Each is optional: an operator running without the gate wired must not hit a nil dereference.

type Status

type Status string

Status is the lifecycle state of a job.

const (
	StatusQueued    Status = "queued"
	StatusRunning   Status = "running"
	StatusDone      Status = "done"
	StatusFailed    Status = "failed"
	StatusCancelled Status = "cancelled"
)

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL