Documentation
¶
Overview ¶
Package work runs the persistent work queue (decision 0033): a pool of workers that lease due items, run the handler registered for each item's kind, and complete, retry or park it. Work is asked for with Enqueue inside the transaction of the change that needs it; readers that must see the work done call EnsureFresh first.
The package also declares the store port the queue persists through: WorkQueue, the queue table, with its rows and bounds, and LeaseStore, the named leases that keep work from running twice at once.
Role ¶
platform layer: the persistent work queue and its store port. Background work belongs to no module, so the queue sits in the platform. The composition root builds one per node and runs it; the item service registers each format's derived work, the reindex kinds and cache maintenance on it; the server's reindex and Debian snapshot operations enqueue through it; and the remote repositories run their Smart Remote and Maven companion work on queues of their own over the same table. The port and its rows are declared here, by the package that persists through them, and New takes Store, the two methods the queue calls, so that the package imports no store (decision 0044, "Narrowing"); the metadata store satisfies it. The platform's store adapters (platform/store/sql and platform/store/inmem) implement WorkQueue and LeaseStore, and what must not run twice at once, such as the blob mirror's repair and the NuGet and Nix publication locks, takes a Lease through LeaseStore. The package imports the kernel, the OpenTelemetry metric API and the standard library.
Inputs ¶
- New: a Store, whose WorkQueue() is the queue table and whose AfterCommit runs a function once the caller's transaction commits; and Options: Workers (4), Lease (two minutes), MaxAttempts (10), Poll (one second), Holder (the host name, process id and a random suffix), Backoff (Backoff), Now, Log and Meters (no metrics when nil). The composition root sets the first three from work.workers, work.lease and work.max_attempts.
- The handlers, registered per kind before Run: Handle for work that runs as soon as it is due, HandleEvery for periodic work that waits for a timer.
- Enqueue's kind, key and payload (a small JSON document), called with the context of the caller's transaction; EnsureFresh's kind and key.
Outputs ¶
- Queue, with Handle, HandleEvery, Enqueue, Run, RunDue, EnsureFresh and WaitIdle; Handler, Options, Backoff and the defaults; the sentinels ErrPermanent, ErrParked, ErrNoHandler, ErrOptions, ErrDuplicateHandler and ErrPanic.
- The port: Store; WorkQueue, with WorkLease, WorkFailure, the bounds MaxWorkKind, MaxWorkKey and MaxWorkPayload, and ValidateWork and ValidateLease, which every adapter applies; LeaseStore.
- The rows: WorkItem, WorkState with WorkPending, WorkLeased and WorkParked, WorkStats, and Lease with Held.
- Metrics through Options.Meters: artifactweave.work.items by kind and state, artifactweave.work.oldest_pending by kind, in seconds, and artifactweave.work.runs by kind and outcome (completed, failed, parked, released, lost).
Invariants ¶
- Work is enqueued inside the caller's transaction, so it is asked for exactly when the change that needs it commits and never after a rollback; this node's workers are woken once it has committed.
- One item per kind and key: work asked for again while the item waits joins it, and work asked for during a run runs once more after it.
- A run holds its item by a lease and a generation, and extends the lease every half lease. A run whose lease is lost is cancelled and recorded as lost, and its generation can no longer complete or fail the item.
- A failed run is retried after Backoff, and parked once the item has used MaxAttempts or the handler returns ErrPermanent, until the key is enqueued again; a panic counts as a failure. A run cut short because the queue stops is handed back without counting the attempt and runs again, so handlers must be idempotent.
- Kinds are code constants, so the kind label of every metric stays bounded (decision 0016).
- The meter scope is the package's path after promotion, github.com/weaveplatform/artifactweave/internal/platform/work, set once in P9d (decision 0044), so dashboards and alert rules that select on it change once.
- Every port method is given the time it acts at, so no adapter reads a clock.
References ¶
- Decision 0033: docs/research/decisions/0033-persistent-work-queue.md (the queue, leasing, retries and parking, periodic kinds, its configuration and its metrics).
- Decision 0016: docs/research/decisions/0016-observability-and-health.md, line 75 (the queue's OpenTelemetry meter, with no-op defaults and recorder tests).
- Decision 0044: docs/research/decisions/0044-modular-monolith-migration.md ("Narrowing"; the meter scope, set once to the final path).
- Research 15: docs/research/15-data-model.md, lines 697 and 698 (the leases and work_items tables).
- Target layout: docs/research/architecture-options/5-modular-monolith.md, line 144 (platform/work) and line 1039 (the mapping row).
- Plan: docs/implementation/plan.md#p2-stores, #p7-npm-go-maven-locals and #p9d-modular-monolith-migration, epic PLT (https://github.com/weaveplatform/artifactweave/issues/938).
- https://github.com/weaveplatform/artifactweave/issues/297
- https://github.com/weaveplatform/artifactweave/issues/576
- https://github.com/weaveplatform/artifactweave/issues/985
Status ¶
Implemented P2: LeaseStore and Lease, the named leases on background work (pull request 3). P7: the persistent work queue with its port and rows, EnsureFresh and the metrics (issue 297, pull request 299), and periodic kinds, HandleEvery and RunDue, for maxUniqueSnapshots (pull request 512). P8: a pass holds an item it meets again until the pass ends (issue 576, pull request 577). P9: ErrPermanent, for the Maven remotes' durable companion work (pull request 657). Moved here in P9d: New takes Store, and the meter scope is the final path.
Testing ¶
- work_test.go: Backoff and New's options; runs that complete once their transaction commits, work asked for twice that runs once, and nothing after a rollback; retries and parking, a panic, a run handed back when the queue stops, heartbeats and a lost lease, EnsureFresh, WaitIdle's deadline, Run without handlers, a failing store and a permanent refusal; TestUnitMeterName: every artifactweave.work.* metric is reported under the final scope.
- periodic_test.go: periodic kinds, their timer and RunDue, with items asked for again during their run, ties on a clock that stands still, a pass cut short, a held item whose lease runs out, and a hand-back or a store that fails.
- types_test.go: TestUnitLeaseHeld.
The queue's tests run over the platform's in-memory store. The platform's work and lease contracts (the Work* subtests of platform/store/storetest: validation, lifecycle, coalescing, fencing, failures, LeaseKey, transactions and concurrency; and the Leases subtest) run on the platform's in-memory and SQLite stores and on the metadata store's in-memory, SQLite and PostgreSQL stores.
Index ¶
- Constants
- Variables
- func Backoff(attempt int) time.Duration
- func ValidateLease(req WorkLease) error
- func ValidateWork(kind, key string, payload []byte) error
- type Handler
- type Lease
- type LeaseStore
- type Options
- type Queue
- func (q *Queue) Enqueue(ctx context.Context, kind, key string, payload []byte) error
- func (q *Queue) EnsureFresh(ctx context.Context, kind, key string) error
- func (q *Queue) Handle(kind string, h Handler) error
- func (q *Queue) HandleEvery(kind string, every time.Duration, h Handler) error
- func (q *Queue) Run(ctx context.Context) error
- func (q *Queue) RunDue(ctx context.Context, kind string) (int, error)
- func (q *Queue) WaitIdle(ctx context.Context) error
- type Store
- type WorkFailure
- type WorkItem
- type WorkLease
- type WorkQueue
- type WorkState
- type WorkStats
Constants ¶
const ( MaxWorkKind = 64 MaxWorkKey = 1024 MaxWorkPayload = 64 << 10 )
Bounds on a work item, so the queue stays a queue of references to work rather than a store of its data (decision 0033).
const ( DefaultWorkers = 4 DefaultLease = 2 * time.Minute DefaultMaxAttempts = 10 DefaultPoll = time.Second )
Defaults for Options left zero.
Variables ¶
var ( // ErrPermanent marks a refusal that cannot succeed through automatic retries. // The failed operation remains parked and visible until explicitly enqueued. ErrPermanent = errors.New("work: permanent refusal") // ErrParked reports that the work a reader waits for keeps failing and // is parked until it is enqueued again. ErrParked = errors.New("work: parked") // ErrNoHandler reports a kind with no registered handler. ErrNoHandler = errors.New("work: no handler") // ErrOptions reports options New cannot use. ErrOptions = errors.New("work: invalid options") // ErrDuplicateHandler reports a second handler for one kind. ErrDuplicateHandler = errors.New("work: kind already has a handler") // ErrPanic reports a handler that panicked. ErrPanic = errors.New("work: handler panicked") )
Functions ¶
func ValidateLease ¶
ValidateLease checks a lease request. It returns an error wrapping kernel.ErrInvalid unless the request names a holder and an expiry after its Now.
func ValidateWork ¶
ValidateWork checks an item to enqueue against the queue's bounds. It returns an error wrapping kernel.ErrInvalid for an empty or overlong kind or key, or a payload over MaxWorkPayload or not JSON.
Types ¶
type Handler ¶
Handler does the work of one item. It must be idempotent: an item can run more than once, for example when a worker stops after the work but before recording it. A cancelled ctx means the queue is stopping or the lease was lost; the handler should return promptly.
type Lease ¶
Lease is a named lock on background work (cleanup runs, replication, GC, archive runs). The scheduler takes one before a run so a job never overlaps itself, and the locks endpoint reports the table.
type LeaseStore ¶
type LeaseStore interface {
// Acquire takes the lease l.Name for l.Holder until l.Expires when it is
// free, expired at now, or already held by l.Holder (a renewal), and
// reports whether it did. The stored lease is l as given, so a renewal
// that wants to keep its acquisition time passes it back.
// kernel.ErrInvalid when the name, holder or expiry is missing.
Acquire(ctx context.Context, l Lease, now time.Time) (bool, error)
// Release gives up a lease held by holder; releasing a lease someone
// else holds, or nobody does, changes nothing.
Release(ctx context.Context, name, holder string) error
// List returns every lease ordered by name.
List(ctx context.Context) ([]Lease, error)
}
LeaseStore persists named leases on background work.
type Options ¶
type Options struct {
// Workers is how many items run at once on this node.
Workers int
// Lease is how long a run holds its item before another worker may
// take it over; a running item's lease is extended every half lease.
Lease time.Duration
// MaxAttempts is how many runs an item gets before it is parked.
MaxAttempts int
// Poll is how long an idle worker waits before asking again, when no
// enqueue on this node wakes it sooner.
Poll time.Duration
// Holder names this node's leases; empty uses the host name, the
// process id and a random suffix.
Holder string
// Backoff is the wait before an item's next run after its attempt'th
// failed; nil doubles from a second up to ten minutes.
Backoff func(attempt int) time.Duration
Now func() time.Time
Log *slog.Logger
// Meters exports the queue's metrics (decision 0016); nil exports none.
Meters metric.MeterProvider
}
Options configures a Queue.
type Queue ¶
type Queue struct {
// contains filtered or unexported fields
}
Queue runs the persistent work queue for one node.
func (*Queue) Enqueue ¶
Enqueue asks for kind's work on key. Call it with the context of the transaction that makes the change needing the work, so the work is asked for exactly when the change commits; this node's workers are woken once it has.
func (*Queue) EnsureFresh ¶
EnsureFresh brings the work owed for kind and key up to date before the caller reads what it derives: it runs a waiting item now, in the caller's context, or waits for a run already in progress elsewhere. It returns nil when nothing is owed, the handler's error when the run it made fails, and ErrParked when the item is parked.
func (*Queue) Handle ¶
Handle registers h for kind. Register every handler before Run: Run leases only the kinds registered when it starts. A second handler for one kind is an error.
func (*Queue) HandleEvery ¶
HandleEvery registers h for kind as periodic work, for work that Artifactory does on a timer rather than as it is asked for, such as its snapshot cleanup every five minutes. An item of kind is not run as soon as it is due: it waits in the queue, so a restart keeps it, for the next tick of a timer every interval on each node, which runs every item of kind then due (RunDue). EnsureFresh still runs one at once. Register it before Run, as Handle.
func (*Queue) Run ¶
Run leases and runs items until ctx ends, then hands back the leases of runs it cuts short and returns once every run has settled. Periodic kinds run on their own timers, outside the workers.
func (*Queue) RunDue ¶
RunDue runs every item of kind that is due now, one after another, and returns how many runs it made. A periodic kind's timer calls it; so may a caller that need not wait for the timer, such as a test. A run that fails is retried after its backoff, never in the same pass, and an item asked for again during its run waits for the next pass.
func (*Queue) WaitIdle ¶
WaitIdle returns once no item of a registered kind is pending or leased and no run is in progress on this node, for tests. Parked items do not count; an item that keeps failing keeps it waiting until ctx ends. A pending item of a periodic kind does not count either, since it waits for its timer by design; RunDue runs it.
type Store ¶
Store is what the queue needs from persistence: the queue table, and a hook that runs fn once the caller's transaction commits (Enqueue wakes the workers through it).
type WorkFailure ¶
type WorkFailure struct {
Error string
// RetryAt is when the item is due again unless parked.
RetryAt time.Time
// Park stops retrying until the key is enqueued again.
Park bool
Now time.Time
}
WorkFailure is how a run ended unsuccessfully.
type WorkItem ¶
type WorkItem struct {
Kind string
Key string
// Payload is a small JSON document from the latest enqueue.
Payload []byte
State WorkState
// Attempts counts the runs started since the item was last enqueued
// afresh; it is what parks a key that keeps failing.
Attempts int
// Generation increases with every lease, so a worker whose lease ran
// out cannot complete or fail the run another worker took over.
Generation int64
// NotBefore is when a pending item becomes due.
NotBefore time.Time
LeasedBy string
LeaseUntil time.Time
// DirtySince is when work was asked for again during the current run,
// which then runs once more; zero when it was not.
DirtySince time.Time
LastError string
// Enqueued is when the work the item now owes was first asked for.
Enqueued time.Time
Updated time.Time
}
WorkItem is one unit of asynchronous derived work (decision 0033). There is one item per Kind and Key, so work asked for again while an item waits coalesces into the run it is waiting for.
type WorkLease ¶
type WorkLease struct {
// Kinds limits Lease to these kinds; empty takes any.
Kinds []string
Holder string
// Limit is the most items Lease takes; LeaseKey ignores it.
Limit int
// Until is when the lease runs out unless extended; Now is when it is
// taken.
Until, Now time.Time
}
WorkLease is what a worker asks the queue for.
type WorkQueue ¶
type WorkQueue interface {
// Enqueue asks for the work kind names for key, with payload (a small
// JSON document, or empty). Called inside the caller's transaction, it
// commits or rolls back with the change that needs the work.
// - A key with no item gets a pending one, due now.
// - A pending key keeps its schedule and takes the new payload.
// - A leased key takes the new payload and is marked to run once more
// when its current run ends.
// - A parked key is pending again from now, its attempts reset.
// kernel.ErrInvalid for an empty or overlong kind or key, or a payload
// that is not JSON or exceeds MaxWorkPayload.
Enqueue(ctx context.Context, kind, key string, payload []byte, now time.Time) error
// Lease takes up to req.Limit due items of req.Kinds (every kind when
// empty) for req.Holder until req.Until: pending items from their
// NotBefore on, and leased items whose lease ran out, the longest due
// first. Each taken item's Generation and Attempts go up by one and its
// DirtySince clears, since the new run covers that work. Concurrent
// callers never take the same item.
Lease(ctx context.Context, req WorkLease) ([]WorkItem, error)
// LeaseKey takes the item of kind and key, as Lease would, when it is
// pending (whatever its NotBefore) or its lease ran out, so a reader
// can bring derived data up to date before using it. It returns the
// item as it stands and whether it was taken. kernel.ErrNotFound when
// there is no item.
LeaseKey(ctx context.Context, kind, key string, req WorkLease) (WorkItem, bool, error)
// Extend moves the lease of the run with generation gen to until.
// kernel.ErrConflict when that run no longer holds the item.
Extend(ctx context.Context, kind, key string, gen int64, until, now time.Time) error
// Complete ends the run with generation gen successfully. The item is
// deleted, or, when work was asked for again during the run, pending
// again from now with its attempts reset. kernel.ErrConflict when the
// run no longer holds the item.
Complete(ctx context.Context, kind, key string, gen int64, now time.Time) error
// Fail ends the run with generation gen unsuccessfully: pending again
// from f.RetryAt, or parked when f.Park and no work was asked for during
// the run. kernel.ErrConflict when the run no longer holds the item.
Fail(ctx context.Context, kind, key string, gen int64, f WorkFailure) error
// Release hands back the run with generation gen unfinished, as a
// stopping worker does: pending again from now, without counting the
// attempt. kernel.ErrConflict when the run no longer holds the item.
Release(ctx context.Context, kind, key string, gen int64, now time.Time) error
// Get returns the item of kind and key; kernel.ErrNotFound when there is
// none.
Get(ctx context.Context, kind, key string) (WorkItem, error)
// Stats counts each kind's items by state, ordered by kind.
Stats(ctx context.Context) ([]WorkStats, error)
}
WorkQueue persists asynchronous derived work (decision 0033): one item per kind and key, run by one worker at a time. Every method is given the time it acts at, so no backend reads a clock.