work

package
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 16, 2026 License: MIT Imports: 18 Imported by: 0

Documentation

Overview

Package work runs the persistent work queue (decision 0033): a pool of workers that lease due items, run the handler registered for each item's kind, and complete, retry or park it. Work is asked for with Enqueue inside the transaction of the change that needs it; readers that must see the work done call EnsureFresh first.

The package also declares the store port the queue persists through: WorkQueue, the queue table, with its rows and bounds, and LeaseStore, the named leases that keep work from running twice at once.

Role

platform layer: the persistent work queue and its store port. Background work belongs to no module, so the queue sits in the platform. The composition root builds one per node and runs it; the item service registers each format's derived work, the reindex kinds and cache maintenance on it; the server's reindex and Debian snapshot operations enqueue through it; and the remote repositories run their Smart Remote and Maven companion work on queues of their own over the same table. The port and its rows are declared here, by the package that persists through them, and New takes Store, the two methods the queue calls, so that the package imports no store (decision 0044, "Narrowing"); the metadata store satisfies it. The platform's store adapters (platform/store/sql and platform/store/inmem) implement WorkQueue and LeaseStore, and what must not run twice at once, such as the blob mirror's repair and the NuGet and Nix publication locks, takes a Lease through LeaseStore. The package imports the kernel, the OpenTelemetry metric API and the standard library.

Inputs

  • New: a Store, whose WorkQueue() is the queue table and whose AfterCommit runs a function once the caller's transaction commits; and Options: Workers (4), Lease (two minutes), MaxAttempts (10), Poll (one second), Holder (the host name, process id and a random suffix), Backoff (Backoff), Now, Log and Meters (no metrics when nil). The composition root sets the first three from work.workers, work.lease and work.max_attempts.
  • The handlers, registered per kind before Run: Handle for work that runs as soon as it is due, HandleEvery for periodic work that waits for a timer.
  • Enqueue's kind, key and payload (a small JSON document), called with the context of the caller's transaction; EnsureFresh's kind and key.

Outputs

  • Queue, with Handle, HandleEvery, Enqueue, Run, RunDue, EnsureFresh and WaitIdle; Handler, Options, Backoff and the defaults; the sentinels ErrPermanent, ErrParked, ErrNoHandler, ErrOptions, ErrDuplicateHandler and ErrPanic.
  • The port: Store; WorkQueue, with WorkLease, WorkFailure, the bounds MaxWorkKind, MaxWorkKey and MaxWorkPayload, and ValidateWork and ValidateLease, which every adapter applies; LeaseStore.
  • The rows: WorkItem, WorkState with WorkPending, WorkLeased and WorkParked, WorkStats, and Lease with Held.
  • Metrics through Options.Meters: artifactweave.work.items by kind and state, artifactweave.work.oldest_pending by kind, in seconds, and artifactweave.work.runs by kind and outcome (completed, failed, parked, released, lost).

Invariants

  • Work is enqueued inside the caller's transaction, so it is asked for exactly when the change that needs it commits and never after a rollback; this node's workers are woken once it has committed.
  • One item per kind and key: work asked for again while the item waits joins it, and work asked for during a run runs once more after it.
  • A run holds its item by a lease and a generation, and extends the lease every half lease. A run whose lease is lost is cancelled and recorded as lost, and its generation can no longer complete or fail the item.
  • A failed run is retried after Backoff, and parked once the item has used MaxAttempts or the handler returns ErrPermanent, until the key is enqueued again; a panic counts as a failure. A run cut short because the queue stops is handed back without counting the attempt and runs again, so handlers must be idempotent.
  • Kinds are code constants, so the kind label of every metric stays bounded (decision 0016).
  • The meter scope is the package's path after promotion, github.com/weaveplatform/artifactweave/internal/platform/work, set once in P9d (decision 0044), so dashboards and alert rules that select on it change once.
  • Every port method is given the time it acts at, so no adapter reads a clock.

References

Status

Implemented P2: LeaseStore and Lease, the named leases on background work (pull request 3). P7: the persistent work queue with its port and rows, EnsureFresh and the metrics (issue 297, pull request 299), and periodic kinds, HandleEvery and RunDue, for maxUniqueSnapshots (pull request 512). P8: a pass holds an item it meets again until the pass ends (issue 576, pull request 577). P9: ErrPermanent, for the Maven remotes' durable companion work (pull request 657). Moved here in P9d: New takes Store, and the meter scope is the final path.

Testing

  • work_test.go: Backoff and New's options; runs that complete once their transaction commits, work asked for twice that runs once, and nothing after a rollback; retries and parking, a panic, a run handed back when the queue stops, heartbeats and a lost lease, EnsureFresh, WaitIdle's deadline, Run without handlers, a failing store and a permanent refusal; TestUnitMeterName: every artifactweave.work.* metric is reported under the final scope.
  • periodic_test.go: periodic kinds, their timer and RunDue, with items asked for again during their run, ties on a clock that stands still, a pass cut short, a held item whose lease runs out, and a hand-back or a store that fails.
  • types_test.go: TestUnitLeaseHeld.

The queue's tests run over the platform's in-memory store. The platform's work and lease contracts (the Work* subtests of platform/store/storetest: validation, lifecycle, coalescing, fencing, failures, LeaseKey, transactions and concurrency; and the Leases subtest) run on the platform's in-memory and SQLite stores and on the metadata store's in-memory, SQLite and PostgreSQL stores.

Index

Constants

View Source
const (
	MaxWorkKind    = 64
	MaxWorkKey     = 1024
	MaxWorkPayload = 64 << 10
)

Bounds on a work item, so the queue stays a queue of references to work rather than a store of its data (decision 0033).

View Source
const (
	DefaultWorkers     = 4
	DefaultLease       = 2 * time.Minute
	DefaultMaxAttempts = 10
	DefaultPoll        = time.Second
)

Defaults for Options left zero.

Variables

View Source
var (
	// ErrPermanent marks a refusal that cannot succeed through automatic retries.
	// The failed operation remains parked and visible until explicitly enqueued.
	ErrPermanent = errors.New("work: permanent refusal")
	// ErrParked reports that the work a reader waits for keeps failing and
	// is parked until it is enqueued again.
	ErrParked = errors.New("work: parked")
	// ErrNoHandler reports a kind with no registered handler.
	ErrNoHandler = errors.New("work: no handler")
	// ErrOptions reports options New cannot use.
	ErrOptions = errors.New("work: invalid options")
	// ErrDuplicateHandler reports a second handler for one kind.
	ErrDuplicateHandler = errors.New("work: kind already has a handler")
	// ErrPanic reports a handler that panicked.
	ErrPanic = errors.New("work: handler panicked")
)

Functions

func Backoff

func Backoff(attempt int) time.Duration

Backoff doubles from one second after the first failure up to ten minutes.

func ValidateLease

func ValidateLease(req WorkLease) error

ValidateLease checks a lease request. It returns an error wrapping kernel.ErrInvalid unless the request names a holder and an expiry after its Now.

func ValidateWork

func ValidateWork(kind, key string, payload []byte) error

ValidateWork checks an item to enqueue against the queue's bounds. It returns an error wrapping kernel.ErrInvalid for an empty or overlong kind or key, or a payload over MaxWorkPayload or not JSON.

Types

type Handler

type Handler func(ctx context.Context, item WorkItem) error

Handler does the work of one item. It must be idempotent: an item can run more than once, for example when a worker stops after the work but before recording it. A cancelled ctx means the queue is stopping or the lease was lost; the handler should return promptly.

type Lease

type Lease struct {
	Name     string
	Holder   string
	Task     string
	Acquired time.Time
	Expires  time.Time
}

Lease is a named lock on background work (cleanup runs, replication, GC, archive runs). The scheduler takes one before a run so a job never overlaps itself, and the locks endpoint reports the table.

func (Lease) Held

func (l Lease) Held(now time.Time) bool

Held reports whether the lease is still in force at now.

type LeaseStore

type LeaseStore interface {
	// Acquire takes the lease l.Name for l.Holder until l.Expires when it is
	// free, expired at now, or already held by l.Holder (a renewal), and
	// reports whether it did. The stored lease is l as given, so a renewal
	// that wants to keep its acquisition time passes it back.
	// kernel.ErrInvalid when the name, holder or expiry is missing.
	Acquire(ctx context.Context, l Lease, now time.Time) (bool, error)
	// Release gives up a lease held by holder; releasing a lease someone
	// else holds, or nobody does, changes nothing.
	Release(ctx context.Context, name, holder string) error
	// List returns every lease ordered by name.
	List(ctx context.Context) ([]Lease, error)
}

LeaseStore persists named leases on background work.

type Options

type Options struct {
	// Workers is how many items run at once on this node.
	Workers int
	// Lease is how long a run holds its item before another worker may
	// take it over; a running item's lease is extended every half lease.
	Lease time.Duration
	// MaxAttempts is how many runs an item gets before it is parked.
	MaxAttempts int
	// Poll is how long an idle worker waits before asking again, when no
	// enqueue on this node wakes it sooner.
	Poll time.Duration
	// Holder names this node's leases; empty uses the host name, the
	// process id and a random suffix.
	Holder string
	// Backoff is the wait before an item's next run after its attempt'th
	// failed; nil doubles from a second up to ten minutes.
	Backoff func(attempt int) time.Duration
	Now     func() time.Time
	Log     *slog.Logger
	// Meters exports the queue's metrics (decision 0016); nil exports none.
	Meters metric.MeterProvider
}

Options configures a Queue.

type Queue

type Queue struct {
	// contains filtered or unexported fields
}

Queue runs the persistent work queue for one node.

func New

func New(st Store, o Options) (*Queue, error)

New returns a queue over st; register handlers with Handle before Run.

func (*Queue) Enqueue

func (q *Queue) Enqueue(ctx context.Context, kind, key string, payload []byte) error

Enqueue asks for kind's work on key. Call it with the context of the transaction that makes the change needing the work, so the work is asked for exactly when the change commits; this node's workers are woken once it has.

func (*Queue) EnsureFresh

func (q *Queue) EnsureFresh(ctx context.Context, kind, key string) error

EnsureFresh brings the work owed for kind and key up to date before the caller reads what it derives: it runs a waiting item now, in the caller's context, or waits for a run already in progress elsewhere. It returns nil when nothing is owed, the handler's error when the run it made fails, and ErrParked when the item is parked.

func (*Queue) Handle

func (q *Queue) Handle(kind string, h Handler) error

Handle registers h for kind. Register every handler before Run: Run leases only the kinds registered when it starts. A second handler for one kind is an error.

func (*Queue) HandleEvery

func (q *Queue) HandleEvery(kind string, every time.Duration, h Handler) error

HandleEvery registers h for kind as periodic work, for work that Artifactory does on a timer rather than as it is asked for, such as its snapshot cleanup every five minutes. An item of kind is not run as soon as it is due: it waits in the queue, so a restart keeps it, for the next tick of a timer every interval on each node, which runs every item of kind then due (RunDue). EnsureFresh still runs one at once. Register it before Run, as Handle.

func (*Queue) Run

func (q *Queue) Run(ctx context.Context) error

Run leases and runs items until ctx ends, then hands back the leases of runs it cuts short and returns once every run has settled. Periodic kinds run on their own timers, outside the workers.

func (*Queue) RunDue

func (q *Queue) RunDue(ctx context.Context, kind string) (int, error)

RunDue runs every item of kind that is due now, one after another, and returns how many runs it made. A periodic kind's timer calls it; so may a caller that need not wait for the timer, such as a test. A run that fails is retried after its backoff, never in the same pass, and an item asked for again during its run waits for the next pass.

func (*Queue) WaitIdle

func (q *Queue) WaitIdle(ctx context.Context) error

WaitIdle returns once no item of a registered kind is pending or leased and no run is in progress on this node, for tests. Parked items do not count; an item that keeps failing keeps it waiting until ctx ends. A pending item of a periodic kind does not count either, since it waits for its timer by design; RunDue runs it.

type Store

type Store interface {
	WorkQueue() WorkQueue
	AfterCommit(ctx context.Context, fn func())
}

Store is what the queue needs from persistence: the queue table, and a hook that runs fn once the caller's transaction commits (Enqueue wakes the workers through it).

type WorkFailure

type WorkFailure struct {
	Error string
	// RetryAt is when the item is due again unless parked.
	RetryAt time.Time
	// Park stops retrying until the key is enqueued again.
	Park bool
	Now  time.Time
}

WorkFailure is how a run ended unsuccessfully.

type WorkItem

type WorkItem struct {
	Kind string
	Key  string
	// Payload is a small JSON document from the latest enqueue.
	Payload []byte
	State   WorkState
	// Attempts counts the runs started since the item was last enqueued
	// afresh; it is what parks a key that keeps failing.
	Attempts int
	// Generation increases with every lease, so a worker whose lease ran
	// out cannot complete or fail the run another worker took over.
	Generation int64
	// NotBefore is when a pending item becomes due.
	NotBefore  time.Time
	LeasedBy   string
	LeaseUntil time.Time
	// DirtySince is when work was asked for again during the current run,
	// which then runs once more; zero when it was not.
	DirtySince time.Time
	LastError  string
	// Enqueued is when the work the item now owes was first asked for.
	Enqueued time.Time
	Updated  time.Time
}

WorkItem is one unit of asynchronous derived work (decision 0033). There is one item per Kind and Key, so work asked for again while an item waits coalesces into the run it is waiting for.

type WorkLease

type WorkLease struct {
	// Kinds limits Lease to these kinds; empty takes any.
	Kinds  []string
	Holder string
	// Limit is the most items Lease takes; LeaseKey ignores it.
	Limit int
	// Until is when the lease runs out unless extended; Now is when it is
	// taken.
	Until, Now time.Time
}

WorkLease is what a worker asks the queue for.

type WorkQueue

type WorkQueue interface {
	// Enqueue asks for the work kind names for key, with payload (a small
	// JSON document, or empty). Called inside the caller's transaction, it
	// commits or rolls back with the change that needs the work.
	//   - A key with no item gets a pending one, due now.
	//   - A pending key keeps its schedule and takes the new payload.
	//   - A leased key takes the new payload and is marked to run once more
	//     when its current run ends.
	//   - A parked key is pending again from now, its attempts reset.
	// kernel.ErrInvalid for an empty or overlong kind or key, or a payload
	// that is not JSON or exceeds MaxWorkPayload.
	Enqueue(ctx context.Context, kind, key string, payload []byte, now time.Time) error
	// Lease takes up to req.Limit due items of req.Kinds (every kind when
	// empty) for req.Holder until req.Until: pending items from their
	// NotBefore on, and leased items whose lease ran out, the longest due
	// first. Each taken item's Generation and Attempts go up by one and its
	// DirtySince clears, since the new run covers that work. Concurrent
	// callers never take the same item.
	Lease(ctx context.Context, req WorkLease) ([]WorkItem, error)
	// LeaseKey takes the item of kind and key, as Lease would, when it is
	// pending (whatever its NotBefore) or its lease ran out, so a reader
	// can bring derived data up to date before using it. It returns the
	// item as it stands and whether it was taken. kernel.ErrNotFound when
	// there is no item.
	LeaseKey(ctx context.Context, kind, key string, req WorkLease) (WorkItem, bool, error)
	// Extend moves the lease of the run with generation gen to until.
	// kernel.ErrConflict when that run no longer holds the item.
	Extend(ctx context.Context, kind, key string, gen int64, until, now time.Time) error
	// Complete ends the run with generation gen successfully. The item is
	// deleted, or, when work was asked for again during the run, pending
	// again from now with its attempts reset. kernel.ErrConflict when the
	// run no longer holds the item.
	Complete(ctx context.Context, kind, key string, gen int64, now time.Time) error
	// Fail ends the run with generation gen unsuccessfully: pending again
	// from f.RetryAt, or parked when f.Park and no work was asked for during
	// the run. kernel.ErrConflict when the run no longer holds the item.
	Fail(ctx context.Context, kind, key string, gen int64, f WorkFailure) error
	// Release hands back the run with generation gen unfinished, as a
	// stopping worker does: pending again from now, without counting the
	// attempt. kernel.ErrConflict when the run no longer holds the item.
	Release(ctx context.Context, kind, key string, gen int64, now time.Time) error
	// Get returns the item of kind and key; kernel.ErrNotFound when there is
	// none.
	Get(ctx context.Context, kind, key string) (WorkItem, error)
	// Stats counts each kind's items by state, ordered by kind.
	Stats(ctx context.Context) ([]WorkStats, error)
}

WorkQueue persists asynchronous derived work (decision 0033): one item per kind and key, run by one worker at a time. Every method is given the time it acts at, so no backend reads a clock.

type WorkState

type WorkState string

WorkState is where an item of the persistent work queue stands (decision 0033).

const (
	// WorkPending waits for a worker, from NotBefore on.
	WorkPending WorkState = "pending"
	// WorkLeased is being run by LeasedBy until LeaseUntil.
	WorkLeased WorkState = "leased"
	// WorkParked failed too often and waits for a fresh enqueue.
	WorkParked WorkState = "parked"
)

Work item states.

type WorkStats

type WorkStats struct {
	Kind                    string
	Pending, Leased, Parked int64
	// OldestPending is the earliest Enqueued of a pending item; zero when
	// none is pending.
	OldestPending time.Time
}

WorkStats summarises the queue for one kind, for metrics.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL