Documentation
¶
Overview ¶
Package evalqueue is the eval job queue, backed by Postgres claim/lease/heartbeat.
Index ¶
- Constants
- Variables
- func Explain(raw string) string
- func RegisterRoutes(api huma.API, q *PoolExecutor, qual *quality.Store, skills *skill.Store, ...)
- type Candidate
- type ContestLookup
- type DBExecutor
- type EvalBackend
- type EvalConfig
- type Executor
- type Job
- type JobID
- type Panel
- type PoolExecutor
- func (p *PoolExecutor) Cancel(ctx context.Context, id JobID) error
- func (p *PoolExecutor) Claim(ctx context.Context, workerID string, lease time.Duration) (*Job, string, bool, error)
- func (p *PoolExecutor) Complete(ctx context.Context, id JobID, workerID string) error
- func (p *PoolExecutor) Fail(ctx context.Context, id JobID, workerID, cause string) error
- func (p *PoolExecutor) Get(ctx context.Context, id JobID) (*Job, error)
- func (p *PoolExecutor) Heartbeat(ctx context.Context, id JobID, workerID string) error
- func (p *PoolExecutor) ListBySkill(ctx context.Context, skillID string) ([]Job, error)
- func (p *PoolExecutor) Pool() *pgxpool.Pool
- func (p *PoolExecutor) QueuePosition(ctx context.Context, id JobID) (int, error)
- func (p *PoolExecutor) ReapExpired(ctx context.Context) (int, error)
- func (p *PoolExecutor) RecordDerivedSuite(ctx context.Context, q evalsuite.Queryer, jobID, ref string) error
- func (p *PoolExecutor) SetSuiteRef(ctx context.Context, id JobID, ref string) error
- func (p *PoolExecutor) Submit(ctx context.Context, j Job) (JobID, error)
- func (p *PoolExecutor) VerifyClaim(ctx context.Context, id JobID, token string) (*Job, bool, error)
- func (p *PoolExecutor) VerifyDerivePush(ctx context.Context, jobID, token, skillName string) (bool, error)
- func (p *PoolExecutor) WithExecutor(e DBExecutor) *PoolExecutor
- type RouteOptions
- type Status
Constants ¶
const ( AdapterAnthropic = "anthropic" AdapterGateway = "gateway" AdapterSubscription = "subscription" )
Provider adapters.
Variables ¶
var ErrLeaseLost = errors.New("evalqueue: lease lost")
ErrLeaseLost is returned when a heartbeat or fail targets a job that is no longer this worker's — a cancelled job, or one another worker reclaimed.
var ErrNotCancellable = errors.New("evalqueue: job is not queued or running")
ErrNotCancellable is returned when Cancel targets a job that has already finished (done, failed, or cancelled) or does not exist.
Functions ¶
func Explain ¶ added in v0.11.0
Explain reduces a worker's error chain to one sentence. The raw chain is what the worker reports and what an operator needs; this is what a reader sees first. An unrecognised failure passes through unchanged rather than becoming a vague placeholder that hides it.
func RegisterRoutes ¶
func RegisterRoutes(api huma.API, q *PoolExecutor, qual *quality.Store, skills *skill.Store, suites *evalsuite.Registry, opts RouteOptions)
RegisterRoutes wires up the eval job queue HTTP endpoints: claim, heartbeat, report ingestion, fail, status lookup, and re-run.
Types ¶
type Candidate ¶ added in v0.14.0
type Candidate struct {
SkillID string `json:"skill_id"`
SkillName string `json:"skill_name"`
Version int `json:"version"`
Label string `json:"label"`
// ReusedReport is a same-day report of this exact candidate from another
// contest, named by ReusedFrom. The worker runs the candidate anyway when
// the report's conditions differ from a fresh one.
ReusedReport json.RawMessage `json:"reused_report,omitempty"`
ReusedFrom string `json:"reused_from,omitempty"`
// PRHeadID names a stored PR head the worker fetches instead of a
// version. Version is then 0.
PRHeadID string `json:"pr_head_id,omitempty"`
}
Candidate is one entry in a contest job.
type ContestLookup ¶ added in v0.14.0
type ContestLookup interface {
CandidatesForClaim(ctx context.Context, jobID string) ([]Candidate, int, error)
}
ContestLookup returns the candidates a contest job must run, in order, with any same-day report the claim may reuse attached.
type DBExecutor ¶
type DBExecutor interface {
Exec(ctx context.Context, sql string, args ...any) (pgconn.CommandTag, error)
QueryRow(ctx context.Context, sql string, args ...any) pgx.Row
Query(ctx context.Context, sql string, args ...any) (pgx.Rows, error)
}
Executor is the subset of *pgxpool.Pool that PoolExecutor's queries need. It is satisfied by both *pgxpool.Pool and pgx.Tx, so a caller can run queue writes inside a transaction shared with another store (see WithExecutor) instead of each store committing its own write independently.
type EvalBackend ¶ added in v0.19.0
type EvalBackend struct {
Provider string `json:"provider"`
Adapter string `json:"adapter"`
BaseURL string `json:"base_url,omitempty"`
AuthStyle string `json:"auth_style,omitempty"`
CredentialEnv string `json:"credential_env"`
Model string `json:"model"`
}
EvalBackend is one provider and model, as a worker needs them: where to send calls and which of its own env vars holds the credential.
func (EvalBackend) Label ¶ added in v0.19.0
func (b EvalBackend) Label() string
Label names the backend in a report's model_panel: empty for Anthropic's own API, "subscription", or "gateway:<host>". It ignores the provider's name, so a rename does not split a score trend.
type EvalConfig ¶ added in v0.19.0
type EvalConfig struct {
Judge EvalBackend `json:"judge"`
Members []EvalBackend `json:"members"`
}
EvalConfig is the judge and the panel the server chose. The first member is the primary member.
type Executor ¶
type Executor interface {
Submit(ctx context.Context, j Job) (JobID, error)
Cancel(ctx context.Context, id JobID) error
}
Executor submits and cancels eval jobs.
type Job ¶
type Job struct {
ID JobID
SkillID string
SkillName string
Version int
SuiteRef string
Tier string
Panel Panel
Status Status
Attempts int
MaxAttempts int
WorkerID string
LeaseExpiresAt *time.Time
LeaseSeconds int
LastError string
RequestedBy string
// ContestID is set when this job runs a contest. The job's SkillID,
// SkillName and Version then hold the first candidate, so every existing
// query on eval_jobs stays correct, and Candidates carries all of them.
// One job, because two jobs can reach two workers with different agent CLI
// versions on different days — the drift a contest exists to remove.
ContestID string
Candidates []Candidate
// ContestAttempts is the attempts per task per candidate the contest
// asked for. Attempts above counts job retries.
ContestAttempts int
CreatedAt time.Time
// Eval is the server's judge and panel at claim time. It is not stored:
// a retried job takes the configuration current at its new claim.
Eval *EvalConfig
// StartedAt is set once on the first claim and never moved, so elapsed
// time is measured from the start of work, not the latest retry.
StartedAt *time.Time
}
Job is a unit of eval work against a specific skill version.
type Panel ¶
type Panel struct {
Agents []string `json:"agents,omitempty"`
Models []string `json:"models,omitempty"`
}
Panel is the requested agent/model panel for a run. Empty means the worker's default panel.
type PoolExecutor ¶
type PoolExecutor struct {
// contains filtered or unexported fields
}
PoolExecutor is a Postgres-backed Executor. The table itself is the queue.
func (*PoolExecutor) Cancel ¶
func (p *PoolExecutor) Cancel(ctx context.Context, id JobID) error
Cancel stops a job that has not finished. A running job is cancelled too: the worker sees the status on its next heartbeat and abandons the run.
func (*PoolExecutor) Claim ¶
func (p *PoolExecutor) Claim(ctx context.Context, workerID string, lease time.Duration) (*Job, string, bool, error)
Claim takes one job under SKIP LOCKED so concurrent workers never contend on the same row and never block each other. Claimability covers two cases: a queued job, and a running job whose lease lapsed — the latter is how a worker that died mid-run gives its job back without anyone reaping it.
func (*PoolExecutor) Fail ¶
Fail returns the job to the pool while attempts remain and marks it failed once they do not. The cause is always recorded: a job that failed three times with no reason recorded is a support ticket.
func (*PoolExecutor) Get ¶
Get fetches a job by ID. A missing row is not an error: it returns (nil, nil), matching the repo convention (see auth.UserStore.GetByEmail).
func (*PoolExecutor) Heartbeat ¶
Heartbeat extends the lease only while this worker still owns the job and it is still running. It re-applies the lease duration persisted at claim time (lease_seconds) rather than a fixed default — otherwise a worker that claimed a long lease would have it truncated on every beat, making the lease expire (and the job become reclaimable by a second worker) long before the original grant runs out. A cancelled job stops a worker at its next beat.
func (*PoolExecutor) ListBySkill ¶
ListBySkill returns all jobs for a skill, most recent first.
func (*PoolExecutor) Pool ¶
func (p *PoolExecutor) Pool() *pgxpool.Pool
Pool returns the underlying connection pool, so a caller can Begin a transaction shared with another store (see WithExecutor).
func (*PoolExecutor) QueuePosition ¶
QueuePosition reports how many queued jobs sit ahead of id, so the UI can say "position 3" instead of showing a spinner for the 45-90 minutes an eval takes. It returns 0 for a job that is not queued - running, done, failed or absent - because a position is only meaningful while waiting.
It is a hint, not a promise: concurrency and retries move it.
func (*PoolExecutor) ReapExpired ¶
func (p *PoolExecutor) ReapExpired(ctx context.Context) (int, error)
ReapExpired marks jobs whose lease lapsed and whose attempts are exhausted as failed. Claim recovers a lapsed lease while retries remain; without this sweep a worker that dies on its final attempt leaves the job running forever, neither retried nor reported. Returns the number reaped.
func (*PoolExecutor) RecordDerivedSuite ¶ added in v0.11.0
func (p *PoolExecutor) RecordDerivedSuite(ctx context.Context, q evalsuite.Queryer, jobID, ref string) error
RecordDerivedSuite is SetSuiteRef through a caller-supplied executor, so the job's suite_ref lands in the same transaction as the suite row it names.
func (*PoolExecutor) SetSuiteRef ¶ added in v0.11.0
SetSuiteRef records the ref a derive job's worker actually measured against. Only a job submitted with no suite_ref should ever call this — the report handler enforces that, not this query.
func (*PoolExecutor) Submit ¶
Submit enqueues a job. It does not deduplicate: two publishes of the same version are two measurements, and the caller decides whether that is wanted.
func (*PoolExecutor) VerifyClaim ¶
VerifyClaim compares in constant time: the claim token is a bearer credential and a byte-at-a-time comparison leaks it. Status and lease liveness are checked in the same query as the hash, against the database's clock — the same one Claim, Heartbeat, and ReapExpired use — so a lapsed lease or a cancelled job (Cancel clears claim_token_hash) can never verify due to skew between the app server's clock and the database's, even if nobody has polled to notice yet.
func (*PoolExecutor) VerifyDerivePush ¶ added in v0.11.0
func (p *PoolExecutor) VerifyDerivePush(ctx context.Context, jobID, token, skillName string) (bool, error)
VerifyDerivePush answers the suite-upload route's one question: is this push coming from the worker that currently holds jobID's claim, for a job that was submitted with no suite of its own and names skillName? Anything else — a forged token, a lapsed lease, a job that already names a suite, a job for another skill — is false, so a push can never talk the server into attributing a suite to a job it does not belong to.
func (*PoolExecutor) WithExecutor ¶
func (p *PoolExecutor) WithExecutor(e DBExecutor) *PoolExecutor
WithExecutor returns a PoolExecutor whose queries run against e (typically a pgx.Tx) instead of the pool directly, so a caller can compose a write with another store's write in one transaction.
type RouteOptions ¶
type RouteOptions struct {
// Releaser re-decides a held version once its score lands. Nil disables
// the re-decision entirely: reports are still ingested, held versions
// simply stay held.
Releaser *skill.Releaser
// QualityFloor is the minimum headline score a verified report must
// reach to clear a held version. It comes from
// the quality_floor setting, the same value publish decides with. Nil
// means 0.
QualityFloor func() float64
// Contests supplies a contest job's candidates at claim time. A local
// interface, because internal/contest imports this package's Job type.
Contests ContestLookup
// EvalConfig returns the configured judge and panel, or nil.
EvalConfig func(ctx context.Context) (*EvalConfig, error)
}
RouteOptions carries the collaborators the report handler needs to act on a score beyond storing it. Each is optional: an operator running without the gate wired must not hit a nil dereference.