evalqueue

package
v0.10.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 6, 2026 License: Apache-2.0 Imports: 21 Imported by: 0

Documentation

Overview

Package evalqueue is the eval job queue. Postgres is the queue: workers claim with SELECT … FOR UPDATE SKIP LOCKED, hold a lease with a heartbeat, and a lapsed lease returns the job to the pool. No new infrastructure — a worker is one more container in the compose file.

Index

Constants

This section is empty.

Variables

View Source
var ErrLeaseLost = errors.New("evalqueue: lease lost")

ErrLeaseLost is returned when a heartbeat or fail targets a job that is no longer this worker's — a cancelled job, or one another worker reclaimed.

View Source
var ErrNotCancellable = errors.New("evalqueue: job is not queued or running")

ErrNotCancellable is returned when Cancel targets a job that has already finished (done, failed, or cancelled) or does not exist.

Functions

func RegisterRoutes

func RegisterRoutes(api huma.API, q *PoolExecutor, qual *quality.Store, skills *skill.Store, suites *evalsuite.Registry, opts RouteOptions)

RegisterRoutes wires up the eval job queue HTTP endpoints: claim, heartbeat, report ingestion, fail, status lookup, and re-run.

Types

type DBExecutor

type DBExecutor interface {
	Exec(ctx context.Context, sql string, args ...any) (pgconn.CommandTag, error)
	QueryRow(ctx context.Context, sql string, args ...any) pgx.Row
	Query(ctx context.Context, sql string, args ...any) (pgx.Rows, error)
}

Executor is the subset of *pgxpool.Pool that PoolExecutor's queries need. It is satisfied by both *pgxpool.Pool and pgx.Tx, so a caller can run queue writes inside a transaction shared with another store (see WithExecutor) instead of each store committing its own write independently.

type Executor

type Executor interface {
	Submit(ctx context.Context, j Job) (JobID, error)
	Cancel(ctx context.Context, id JobID) error
}

Executor submits and cancels eval jobs.

type Job

type Job struct {
	ID             JobID
	SkillID        string
	SkillName      string
	Version        int
	SuiteRef       string
	Tier           string
	Panel          Panel
	Status         Status
	Attempts       int
	MaxAttempts    int
	WorkerID       string
	LeaseExpiresAt *time.Time
	// LeaseSeconds is the lease duration granted at claim time, persisted so
	// a heartbeat can re-apply it without the worker resending it and
	// without the server guessing at a fixed value.
	LeaseSeconds int
	LastError    string
	RequestedBy  string
	CreatedAt    time.Time
	// StartedAt is when the job first entered `running`. It is set once and
	// never moved: a retry re-claims the same job and elapsed time is
	// measured from when the work began, not from the latest attempt.
	// Elapsed cannot be derived from LeaseExpiresAt, which every heartbeat
	// pushes forward.
	StartedAt *time.Time
}

Job is a unit of eval work against a specific skill version.

type JobID

type JobID string

JobID identifies an eval job.

type Panel

type Panel struct {
	Agents []string `json:"agents,omitempty"`
	Models []string `json:"models,omitempty"`
}

Panel is the requested agent/model panel for a run. Empty means the worker's default panel.

type PoolExecutor

type PoolExecutor struct {
	// contains filtered or unexported fields
}

PoolExecutor is a Postgres-backed Executor. The table itself is the queue.

func NewPool

func NewPool(db *pgxpool.Pool) *PoolExecutor

NewPool builds a PoolExecutor over db.

func (*PoolExecutor) Cancel

func (p *PoolExecutor) Cancel(ctx context.Context, id JobID) error

Cancel stops a job that has not finished. A running job is cancelled too: the worker sees the status on its next heartbeat and abandons the run.

func (*PoolExecutor) Claim

func (p *PoolExecutor) Claim(ctx context.Context, workerID string, lease time.Duration) (*Job, string, bool, error)

Claim takes one job under SKIP LOCKED so concurrent workers never contend on the same row and never block each other. Claimability covers two cases: a queued job, and a running job whose lease lapsed — the latter is how a worker that died mid-run gives its job back without anyone reaping it.

func (*PoolExecutor) Complete

func (p *PoolExecutor) Complete(ctx context.Context, id JobID, workerID string) error

Complete marks a job done and clears its claim.

func (*PoolExecutor) Fail

func (p *PoolExecutor) Fail(ctx context.Context, id JobID, workerID, cause string) error

Fail returns the job to the pool while attempts remain and marks it failed once they do not. The cause is always recorded: a job that failed three times with no reason recorded is a support ticket.

func (*PoolExecutor) Get

func (p *PoolExecutor) Get(ctx context.Context, id JobID) (*Job, error)

Get fetches a job by ID. A missing row is not an error: it returns (nil, nil), matching the repo convention (see auth.UserStore.GetByEmail).

func (*PoolExecutor) Heartbeat

func (p *PoolExecutor) Heartbeat(ctx context.Context, id JobID, workerID string) error

Heartbeat extends the lease only while this worker still owns the job and it is still running. It re-applies the lease duration persisted at claim time (lease_seconds) rather than a fixed default — otherwise a worker that claimed a long lease would have it truncated on every beat, making the lease expire (and the job become reclaimable by a second worker) long before the original grant runs out. A cancelled job stops a worker at its next beat.

func (*PoolExecutor) ListBySkill

func (p *PoolExecutor) ListBySkill(ctx context.Context, skillID string) ([]Job, error)

ListBySkill returns all jobs for a skill, most recent first.

func (*PoolExecutor) Pool

func (p *PoolExecutor) Pool() *pgxpool.Pool

Pool returns the underlying connection pool, so a caller can Begin a transaction shared with another store (see WithExecutor).

func (*PoolExecutor) QueuePosition

func (p *PoolExecutor) QueuePosition(ctx context.Context, id JobID) (int, error)

QueuePosition reports how many queued jobs sit ahead of id, so the UI can say "position 3" instead of showing a spinner for the 45-90 minutes an eval takes. It returns 0 for a job that is not queued - running, done, failed or absent - because a position is only meaningful while waiting.

It is a hint, not a promise: concurrency and retries move it.

func (*PoolExecutor) ReapExpired

func (p *PoolExecutor) ReapExpired(ctx context.Context) (int, error)

ReapExpired marks jobs whose lease lapsed and whose attempts are exhausted as failed. Claim recovers a lapsed lease while retries remain; without this sweep a worker that dies on its final attempt leaves the job running forever, neither retried nor reported. Returns the number reaped.

func (*PoolExecutor) Submit

func (p *PoolExecutor) Submit(ctx context.Context, j Job) (JobID, error)

Submit enqueues a job. It does not deduplicate: two publishes of the same version are two measurements, and the caller decides whether that is wanted.

func (*PoolExecutor) VerifyClaim

func (p *PoolExecutor) VerifyClaim(ctx context.Context, id JobID, token string) (*Job, bool, error)

VerifyClaim compares in constant time: the claim token is a bearer credential and a byte-at-a-time comparison leaks it. Status and lease liveness are checked in the same query as the hash, against the database's clock — the same one Claim, Heartbeat, and ReapExpired use — so a lapsed lease or a cancelled job (Cancel clears claim_token_hash) can never verify due to skew between the app server's clock and the database's, even if nobody has polled to notice yet.

func (*PoolExecutor) WithExecutor

func (p *PoolExecutor) WithExecutor(e DBExecutor) *PoolExecutor

WithExecutor returns a PoolExecutor whose queries run against e (typically a pgx.Tx) instead of the pool directly, so a caller can compose a write with another store's write in one transaction.

type RouteOptions

type RouteOptions struct {
	// Releaser re-decides a held version once its score lands. Nil disables
	// the re-decision entirely: reports are still ingested, held versions
	// simply stay held.
	Releaser *skill.Releaser
	// QualityFloor is the minimum headline score a verified report must
	// reach to clear a held version. It comes from
	// platform.Config.QualityFloor, the same value publish decides with.
	QualityFloor float64
}

RouteOptions carries the collaborators the report handler needs to act on a score beyond storing it. Each is optional: an operator running without the gate wired must not hit a nil dereference.

type Status

type Status string

Status is the lifecycle state of a job.

const (
	StatusQueued    Status = "queued"
	StatusRunning   Status = "running"
	StatusDone      Status = "done"
	StatusFailed    Status = "failed"
	StatusCancelled Status = "cancelled"
)

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL