evalqueue

package
v0.17.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 2, 2026 License: Apache-2.0 Imports: 23 Imported by: 0

Documentation

Overview

Package evalqueue is the eval job queue, backed by Postgres claim/lease/heartbeat.

Index

Constants

This section is empty.

Variables

View Source
var ErrLeaseLost = errors.New("evalqueue: lease lost")

ErrLeaseLost is returned when a heartbeat or fail targets a job that is no longer this worker's — a cancelled job, or one another worker reclaimed.

View Source
var ErrNotCancellable = errors.New("evalqueue: job is not queued or running")

ErrNotCancellable is returned when Cancel targets a job that has already finished (done, failed, or cancelled) or does not exist.

Functions

func Explain added in v0.11.0

func Explain(raw string) string

Explain reduces a worker's error chain to one sentence. The raw chain is what the worker reports and what an operator needs; this is what a reader sees first. An unrecognised failure passes through unchanged rather than becoming a vague placeholder that hides it.

func RegisterRoutes

func RegisterRoutes(api huma.API, q *PoolExecutor, qual *quality.Store, skills *skill.Store, suites *evalsuite.Registry, opts RouteOptions)

RegisterRoutes wires up the eval job queue HTTP endpoints: claim, heartbeat, report ingestion, fail, status lookup, and re-run.

Types

type Candidate added in v0.14.0

type Candidate struct {
	SkillID   string `json:"skill_id"`
	SkillName string `json:"skill_name"`
	Version   int    `json:"version"`
	Label     string `json:"label"`
}

Candidate is one entry in a contest job.

type ContestLookup added in v0.14.0

type ContestLookup interface {
	CandidatesForJob(ctx context.Context, jobID string) ([]Candidate, int, error)
}

ContestLookup returns the candidates a contest job must run, in order.

type DBExecutor

type DBExecutor interface {
	Exec(ctx context.Context, sql string, args ...any) (pgconn.CommandTag, error)
	QueryRow(ctx context.Context, sql string, args ...any) pgx.Row
	Query(ctx context.Context, sql string, args ...any) (pgx.Rows, error)
}

Executor is the subset of *pgxpool.Pool that PoolExecutor's queries need. It is satisfied by both *pgxpool.Pool and pgx.Tx, so a caller can run queue writes inside a transaction shared with another store (see WithExecutor) instead of each store committing its own write independently.

type Executor

type Executor interface {
	Submit(ctx context.Context, j Job) (JobID, error)
	Cancel(ctx context.Context, id JobID) error
}

Executor submits and cancels eval jobs.

type Job

type Job struct {
	ID             JobID
	SkillID        string
	SkillName      string
	Version        int
	SuiteRef       string
	Tier           string
	Panel          Panel
	Status         Status
	Attempts       int
	MaxAttempts    int
	WorkerID       string
	LeaseExpiresAt *time.Time
	LeaseSeconds   int
	LastError      string
	RequestedBy    string
	// ContestID is set when this job runs a contest. The job's SkillID,
	// SkillName and Version then hold the first candidate, so every existing
	// query on eval_jobs stays correct, and Candidates carries all of them.
	// One job, because two jobs can reach two workers with different agent CLI
	// versions on different days — the drift a contest exists to remove.
	ContestID  string
	Candidates []Candidate
	CreatedAt  time.Time
	// StartedAt is set once on the first claim and never moved, so elapsed
	// time is measured from the start of work, not the latest retry.
	StartedAt *time.Time
}

Job is a unit of eval work against a specific skill version.

type JobID

type JobID string

JobID identifies an eval job.

type Panel

type Panel struct {
	Agents []string `json:"agents,omitempty"`
	Models []string `json:"models,omitempty"`
}

Panel is the requested agent/model panel for a run. Empty means the worker's default panel.

type PoolExecutor

type PoolExecutor struct {
	// contains filtered or unexported fields
}

PoolExecutor is a Postgres-backed Executor. The table itself is the queue.

func NewPool

func NewPool(db *pgxpool.Pool) *PoolExecutor

NewPool builds a PoolExecutor over db.

func (*PoolExecutor) Cancel

func (p *PoolExecutor) Cancel(ctx context.Context, id JobID) error

Cancel stops a job that has not finished. A running job is cancelled too: the worker sees the status on its next heartbeat and abandons the run.

func (*PoolExecutor) Claim

func (p *PoolExecutor) Claim(ctx context.Context, workerID string, lease time.Duration) (*Job, string, bool, error)

Claim takes one job under SKIP LOCKED so concurrent workers never contend on the same row and never block each other. Claimability covers two cases: a queued job, and a running job whose lease lapsed — the latter is how a worker that died mid-run gives its job back without anyone reaping it.

func (*PoolExecutor) Complete

func (p *PoolExecutor) Complete(ctx context.Context, id JobID, workerID string) error

Complete marks a job done and clears its claim.

func (*PoolExecutor) Fail

func (p *PoolExecutor) Fail(ctx context.Context, id JobID, workerID, cause string) error

Fail returns the job to the pool while attempts remain and marks it failed once they do not. The cause is always recorded: a job that failed three times with no reason recorded is a support ticket.

func (*PoolExecutor) Get

func (p *PoolExecutor) Get(ctx context.Context, id JobID) (*Job, error)

Get fetches a job by ID. A missing row is not an error: it returns (nil, nil), matching the repo convention (see auth.UserStore.GetByEmail).

func (*PoolExecutor) Heartbeat

func (p *PoolExecutor) Heartbeat(ctx context.Context, id JobID, workerID string) error

Heartbeat extends the lease only while this worker still owns the job and it is still running. It re-applies the lease duration persisted at claim time (lease_seconds) rather than a fixed default — otherwise a worker that claimed a long lease would have it truncated on every beat, making the lease expire (and the job become reclaimable by a second worker) long before the original grant runs out. A cancelled job stops a worker at its next beat.

func (*PoolExecutor) ListBySkill

func (p *PoolExecutor) ListBySkill(ctx context.Context, skillID string) ([]Job, error)

ListBySkill returns all jobs for a skill, most recent first.

func (*PoolExecutor) Pool

func (p *PoolExecutor) Pool() *pgxpool.Pool

Pool returns the underlying connection pool, so a caller can Begin a transaction shared with another store (see WithExecutor).

func (*PoolExecutor) QueuePosition

func (p *PoolExecutor) QueuePosition(ctx context.Context, id JobID) (int, error)

QueuePosition reports how many queued jobs sit ahead of id, so the UI can say "position 3" instead of showing a spinner for the 45-90 minutes an eval takes. It returns 0 for a job that is not queued - running, done, failed or absent - because a position is only meaningful while waiting.

It is a hint, not a promise: concurrency and retries move it.

func (*PoolExecutor) ReapExpired

func (p *PoolExecutor) ReapExpired(ctx context.Context) (int, error)

ReapExpired marks jobs whose lease lapsed and whose attempts are exhausted as failed. Claim recovers a lapsed lease while retries remain; without this sweep a worker that dies on its final attempt leaves the job running forever, neither retried nor reported. Returns the number reaped.

func (*PoolExecutor) RecordDerivedSuite added in v0.11.0

func (p *PoolExecutor) RecordDerivedSuite(ctx context.Context, q evalsuite.Queryer, jobID, ref string) error

RecordDerivedSuite is SetSuiteRef through a caller-supplied executor, so the job's suite_ref lands in the same transaction as the suite row it names.

func (*PoolExecutor) SetSuiteRef added in v0.11.0

func (p *PoolExecutor) SetSuiteRef(ctx context.Context, id JobID, ref string) error

SetSuiteRef records the ref a derive job's worker actually measured against. Only a job submitted with no suite_ref should ever call this — the report handler enforces that, not this query.

func (*PoolExecutor) Submit

func (p *PoolExecutor) Submit(ctx context.Context, j Job) (JobID, error)

Submit enqueues a job. It does not deduplicate: two publishes of the same version are two measurements, and the caller decides whether that is wanted.

func (*PoolExecutor) VerifyClaim

func (p *PoolExecutor) VerifyClaim(ctx context.Context, id JobID, token string) (*Job, bool, error)

VerifyClaim compares in constant time: the claim token is a bearer credential and a byte-at-a-time comparison leaks it. Status and lease liveness are checked in the same query as the hash, against the database's clock — the same one Claim, Heartbeat, and ReapExpired use — so a lapsed lease or a cancelled job (Cancel clears claim_token_hash) can never verify due to skew between the app server's clock and the database's, even if nobody has polled to notice yet.

func (*PoolExecutor) VerifyDerivePush added in v0.11.0

func (p *PoolExecutor) VerifyDerivePush(ctx context.Context, jobID, token, skillName string) (bool, error)

VerifyDerivePush answers the suite-upload route's one question: is this push coming from the worker that currently holds jobID's claim, for a job that was submitted with no suite of its own and names skillName? Anything else — a forged token, a lapsed lease, a job that already names a suite, a job for another skill — is false, so a push can never talk the server into attributing a suite to a job it does not belong to.

func (*PoolExecutor) WithExecutor

func (p *PoolExecutor) WithExecutor(e DBExecutor) *PoolExecutor

WithExecutor returns a PoolExecutor whose queries run against e (typically a pgx.Tx) instead of the pool directly, so a caller can compose a write with another store's write in one transaction.

type RouteOptions

type RouteOptions struct {
	// Releaser re-decides a held version once its score lands. Nil disables
	// the re-decision entirely: reports are still ingested, held versions
	// simply stay held.
	Releaser *skill.Releaser
	// QualityFloor is the minimum headline score a verified report must
	// reach to clear a held version. It comes from
	// platform.Config.QualityFloor, the same value publish decides with.
	QualityFloor float64
	// Contests supplies a contest job's candidates at claim time. A local
	// interface, because internal/contest imports this package's Job type.
	Contests ContestLookup
}

RouteOptions carries the collaborators the report handler needs to act on a score beyond storing it. Each is optional: an operator running without the gate wired must not hit a nil dereference.

type Status

type Status string

Status is the lifecycle state of a job.

const (
	StatusQueued    Status = "queued"
	StatusRunning   Status = "running"
	StatusDone      Status = "done"
	StatusFailed    Status = "failed"
	StatusCancelled Status = "cancelled"
)

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL