Documentation
¶
Overview ¶
Package evalqueue is the eval job queue. Postgres is the queue: workers claim with SELECT … FOR UPDATE SKIP LOCKED, hold a lease with a heartbeat, and a lapsed lease returns the job to the pool. No new infrastructure — a worker is one more container in the compose file.
Index ¶
- Variables
- func RegisterRoutes(api huma.API, q *PoolExecutor, qual *quality.Store, skills *skill.Store, ...)
- type DBExecutor
- type Executor
- type Job
- type JobID
- type Panel
- type PoolExecutor
- func (p *PoolExecutor) Cancel(ctx context.Context, id JobID) error
- func (p *PoolExecutor) Claim(ctx context.Context, workerID string, lease time.Duration) (*Job, string, bool, error)
- func (p *PoolExecutor) Complete(ctx context.Context, id JobID, workerID string) error
- func (p *PoolExecutor) Fail(ctx context.Context, id JobID, workerID, cause string) error
- func (p *PoolExecutor) Get(ctx context.Context, id JobID) (*Job, error)
- func (p *PoolExecutor) Heartbeat(ctx context.Context, id JobID, workerID string) error
- func (p *PoolExecutor) ListBySkill(ctx context.Context, skillID string) ([]Job, error)
- func (p *PoolExecutor) Pool() *pgxpool.Pool
- func (p *PoolExecutor) QueuePosition(ctx context.Context, id JobID) (int, error)
- func (p *PoolExecutor) ReapExpired(ctx context.Context) (int, error)
- func (p *PoolExecutor) Submit(ctx context.Context, j Job) (JobID, error)
- func (p *PoolExecutor) VerifyClaim(ctx context.Context, id JobID, token string) (*Job, bool, error)
- func (p *PoolExecutor) WithExecutor(e DBExecutor) *PoolExecutor
- type RouteOptions
- type Status
Constants ¶
This section is empty.
Variables ¶
var ErrLeaseLost = errors.New("evalqueue: lease lost")
ErrLeaseLost is returned when a heartbeat or fail targets a job that is no longer this worker's — a cancelled job, or one another worker reclaimed.
var ErrNotCancellable = errors.New("evalqueue: job is not queued or running")
ErrNotCancellable is returned when Cancel targets a job that has already finished (done, failed, or cancelled) or does not exist.
Functions ¶
func RegisterRoutes ¶
func RegisterRoutes(api huma.API, q *PoolExecutor, qual *quality.Store, skills *skill.Store, suites *evalsuite.Registry, opts RouteOptions)
RegisterRoutes wires up the eval job queue HTTP endpoints: claim, heartbeat, report ingestion, fail, status lookup, and re-run.
Types ¶
type DBExecutor ¶
type DBExecutor interface {
Exec(ctx context.Context, sql string, args ...any) (pgconn.CommandTag, error)
QueryRow(ctx context.Context, sql string, args ...any) pgx.Row
Query(ctx context.Context, sql string, args ...any) (pgx.Rows, error)
}
Executor is the subset of *pgxpool.Pool that PoolExecutor's queries need. It is satisfied by both *pgxpool.Pool and pgx.Tx, so a caller can run queue writes inside a transaction shared with another store (see WithExecutor) instead of each store committing its own write independently.
type Executor ¶
type Executor interface {
Submit(ctx context.Context, j Job) (JobID, error)
Cancel(ctx context.Context, id JobID) error
}
Executor submits and cancels eval jobs.
type Job ¶
type Job struct {
ID JobID
SkillID string
SkillName string
Version int
SuiteRef string
Tier string
Panel Panel
Status Status
Attempts int
MaxAttempts int
WorkerID string
LeaseExpiresAt *time.Time
// LeaseSeconds is the lease duration granted at claim time, persisted so
// a heartbeat can re-apply it without the worker resending it and
// without the server guessing at a fixed value.
LeaseSeconds int
LastError string
RequestedBy string
CreatedAt time.Time
// StartedAt is when the job first entered `running`. It is set once and
// never moved: a retry re-claims the same job and elapsed time is
// measured from when the work began, not from the latest attempt.
// Elapsed cannot be derived from LeaseExpiresAt, which every heartbeat
// pushes forward.
StartedAt *time.Time
}
Job is a unit of eval work against a specific skill version.
type Panel ¶
type Panel struct {
Agents []string `json:"agents,omitempty"`
Models []string `json:"models,omitempty"`
}
Panel is the requested agent/model panel for a run. Empty means the worker's default panel.
type PoolExecutor ¶
type PoolExecutor struct {
// contains filtered or unexported fields
}
PoolExecutor is a Postgres-backed Executor. The table itself is the queue.
func (*PoolExecutor) Cancel ¶
func (p *PoolExecutor) Cancel(ctx context.Context, id JobID) error
Cancel stops a job that has not finished. A running job is cancelled too: the worker sees the status on its next heartbeat and abandons the run.
func (*PoolExecutor) Claim ¶
func (p *PoolExecutor) Claim(ctx context.Context, workerID string, lease time.Duration) (*Job, string, bool, error)
Claim takes one job under SKIP LOCKED so concurrent workers never contend on the same row and never block each other. Claimability covers two cases: a queued job, and a running job whose lease lapsed — the latter is how a worker that died mid-run gives its job back without anyone reaping it.
func (*PoolExecutor) Fail ¶
Fail returns the job to the pool while attempts remain and marks it failed once they do not. The cause is always recorded: a job that failed three times with no reason recorded is a support ticket.
func (*PoolExecutor) Get ¶
Get fetches a job by ID. A missing row is not an error: it returns (nil, nil), matching the repo convention (see auth.UserStore.GetByEmail).
func (*PoolExecutor) Heartbeat ¶
Heartbeat extends the lease only while this worker still owns the job and it is still running. It re-applies the lease duration persisted at claim time (lease_seconds) rather than a fixed default — otherwise a worker that claimed a long lease would have it truncated on every beat, making the lease expire (and the job become reclaimable by a second worker) long before the original grant runs out. A cancelled job stops a worker at its next beat.
func (*PoolExecutor) ListBySkill ¶
ListBySkill returns all jobs for a skill, most recent first.
func (*PoolExecutor) Pool ¶
func (p *PoolExecutor) Pool() *pgxpool.Pool
Pool returns the underlying connection pool, so a caller can Begin a transaction shared with another store (see WithExecutor).
func (*PoolExecutor) QueuePosition ¶
QueuePosition reports how many queued jobs sit ahead of id, so the UI can say "position 3" instead of showing a spinner for the 45-90 minutes an eval takes. It returns 0 for a job that is not queued - running, done, failed or absent - because a position is only meaningful while waiting.
It is a hint, not a promise: concurrency and retries move it.
func (*PoolExecutor) ReapExpired ¶
func (p *PoolExecutor) ReapExpired(ctx context.Context) (int, error)
ReapExpired marks jobs whose lease lapsed and whose attempts are exhausted as failed. Claim recovers a lapsed lease while retries remain; without this sweep a worker that dies on its final attempt leaves the job running forever, neither retried nor reported. Returns the number reaped.
func (*PoolExecutor) Submit ¶
Submit enqueues a job. It does not deduplicate: two publishes of the same version are two measurements, and the caller decides whether that is wanted.
func (*PoolExecutor) VerifyClaim ¶
VerifyClaim compares in constant time: the claim token is a bearer credential and a byte-at-a-time comparison leaks it. Status and lease liveness are checked in the same query as the hash, against the database's clock — the same one Claim, Heartbeat, and ReapExpired use — so a lapsed lease or a cancelled job (Cancel clears claim_token_hash) can never verify due to skew between the app server's clock and the database's, even if nobody has polled to notice yet.
func (*PoolExecutor) WithExecutor ¶
func (p *PoolExecutor) WithExecutor(e DBExecutor) *PoolExecutor
WithExecutor returns a PoolExecutor whose queries run against e (typically a pgx.Tx) instead of the pool directly, so a caller can compose a write with another store's write in one transaction.
type RouteOptions ¶
type RouteOptions struct {
// Releaser re-decides a held version once its score lands. Nil disables
// the re-decision entirely: reports are still ingested, held versions
// simply stay held.
Releaser *skill.Releaser
// QualityFloor is the minimum headline score a verified report must
// reach to clear a held version. It comes from
// platform.Config.QualityFloor, the same value publish decides with.
QualityFloor float64
}
RouteOptions carries the collaborators the report handler needs to act on a score beyond storing it. Each is optional: an operator running without the gate wired must not hit a nil dereference.