go-taskqueue

go-taskqueue

Projects-aware task work queue / worker pool for Go — with a self-managing agent pool built in. Embedded SQLite journal, lease-based claims with crash reclaim, DAG dependencies, exponential backoff retries with a dead-letter queue, and pluggable executors: shell commands, HTTP webhooks, or headless AI coding agents. Zero external services — the whole system is one Go binary and one file.

Built on the semantics proven in go-cqrs-lite (facts/journal projection style) and PapDashboard (worker pools over durable queues). Not a wrapper around either — a standalone library with its own small core, designed to be embeddable and observable.

Library consumers: import the public facade modules (github.com/larsartmann/go-taskqueue/queue, .../queue/sqlite, .../queue/postgres, .../task, .../journal, .../executor, .../worker — see Embedding it in Go). The internal/… modules behind them are implementation detail (ADR-0016).

Priority system (ADR-0015)

One 0-100 priority scale with three bands — backlog 0-99 (TODO_LIST — P[1-4] markers, per-repo importance via --priority-from importance, keyword bumps, cached AI scores), hot 100-149 (same-session promotion), machine 150+ (operational tasks). Claim order is stored priority plus aging: a waiting task climbs +1 per 3 days, capped at +10, so nothing starves forever. tq reprioritize re-resolves pending tasks from the current truth; --max-pending-per-repo turns the queue into a per-repo working set; a dependency completing bumps its unblocked dependents; an optional AI batch scorer (--prioritize, default off) caches verdicts per item and re-ranks the backlog under marker > AI > keyword precedence.

Why

I have too many projects. Cross-project work (builds, releases, scrapes, checks, agent tasks) ends up scattered across cron jobs, shell scripts, and memory. I want one durable queue where tasks know which project they belong to, retry themselves, dead-letter when broken, and show me everything in one place.

The core idea: facts first. Every state change is an immutable fact in an append-only journal; the queue, retries, dead letters, and dashboard are all projections of those facts. Nothing is ever deleted, so "what exactly happened to this task?" always has an answer.

Contents

Quickstart · Live dashboard · Agent pool · Bootstrap · Running unattended · Command map · Concepts · Embedding in Go · Distribution · Development

Quickstart

# the library (v0.3.0+) is proxy-installable per module — see Embedding below.
# The tq BINARY is its own replace-free module (ADR-0017): `go install` the tag
# (proxy-verified at publish; check-cleanroom-install.sh gates every release),
# build from a clone, or use nix:
go install github.com/larsartmann/go-taskqueue/cmd/tq@v0.3.4
git clone https://github.com/LarsArtmann/go-taskqueue && cd go-taskqueue
./scripts/build-tq.sh ~/.local/bin/tq    # or: nix run github:LarsArtmann/go-taskqueue

tq enqueue --type sh --project demo --payload 'echo hello from $(uname -s)'

# or as JSON
tq enqueue --type sh --project demo --payload '{"cmd":"go test ./..."}'

# run a worker (Ctrl-C for graceful drain; --once drains and exits — cron-friendly)
tq worker

# watch it work
tq stats
tq tail -f

Tasks land in ./tasks.db (override with $TQ_DB or --db PATH on any command). Enqueueing is idempotent: give a task a DedupKey and re-enqueueing returns the stored task instead of duplicating work.

The live dashboard

tq serve          # http://127.0.0.1:8090 (read-only)
tq serve --addr 0.0.0.0:8090 --auth-token "$(openssl rand -hex 16)"   # LAN

One tab shows the whole system updating live: status cards, the task table (or a kanban board view at /?view=board — one column per lifecycle status), the dead-letter queue, per-project progress, and the fact feed — pushed by SSE as server-rendered fragments, reconnect-safe, filterable and searchable (?project=demo&status=running&q=flake), paginated, with a detail page per task at /task/{id} that renders the task's payload as content — the work item, prompt contract, and verify gate for agent tasks, the shell line for commands — plus a retry trail summarizing why a task keeps coming back. A journal browser pages through the full fact history. Press / to search, 1–4 to jump between sections.

The dashboard is a pure projection of the journal: it cannot mutate the queue, and the worst failure is a stale page. Opt into operator actions — cancel (including cooperative stop of running tasks) and rescue, CSRF-guarded forms with a failed-attempt lockout — via tq serve --allow-writes, off by default.

Serving beyond localhost: the dashboard renders every task payload and error tail, so tq serve refuses to bind a non-loopback address (including :port / 0.0.0.0) without --auth-token / $TQ_SERVE_TOKEN. With a token set, every route — pages, API, SSE, static — answers 401 unless the request carries it as Authorization: Bearer <token> or ?token=<token> (the browser EventSource client cannot set headers, so it forwards the query param; --verbose access logs redact the token). Plain HTTP: use a token only on a trusted LAN, or front it with TLS.

The agent pool

One process turns every repo's TODO_LIST.md backlog into a self-burning fire:

# feed the queue from every repo's TODO_LIST.md (idempotent, dedup-keyed)
tq harvest --projects-dir ~/projects --dry-run   # preview
tq harvest --projects-dir ~/projects

# run the whole loop: harvest every 5m + a pool of headless AI agents
# (--project-exclusive guarantees one agent per repo across ALL pools
# sharing the DB; the model is pinned per-repo by tq bootstrap — see below)
tq agent-pool --projects-dir ~/projects --yolo --concurrency 2 --interval 5m \
  --project-exclusive

# cron/timer-friendly: one harvest tick, drain, exit
# (pairs well with --daily-budget, the cost ceiling)
tq agent-pool --projects-dir ~/projects --once --daily-budget 20

# optional: every completed agent task gets ONE review by a second agent;
# --review-autofix turns request_changes findings into deduped fix tasks
# (verdicts land in the task's facts — inspect with `tq show`)
tq agent-pool --projects-dir ~/projects --yolo --review --review-autofix \
  --daily-budget 40

# optional: every N completions per project mint one done-prompt report —
# an agent writes docs/status/<ts>_<name>.md and appends next items back
# into TODO_LIST.md, closing the loop (the pool feeds itself)
tq agent-pool --projects-dir ~/projects --yolo --status-every 10

# optional: Code-Quality-Agent findings become fix tasks each tick
tq agent-pool --projects-dir ~/projects --yolo --cqa-url http://localhost:8080 --cqa-owner $CQA_OWNER_ID

Each TODO item becomes one agent task: a headless crush run in that repo with a strict contract (read AGENTS.md, smallest correct change, tick the checkbox, commit, never push). The executor enforces the safety rails:

• Opt-in autonomy — agents only run under --agents/tq agent-pool; a plain tq worker never spawns one. --yolo is the operator's autonomy request — crush's headless mode has no yolo flag, so autonomy is actually granted by the repo itself: a project-local .crushrc declaring which tools its agents may use. The pool cannot over-grant what a repo never offered; a --yolo task on a repo without such a config fails fast with remediation guidance instead of burning agent attempts.

  # in each repo that wants agents (committed, reviewable, per-repo scope):
  echo 'permissions allow view ls grep glob edit multiedit write bash fetch download todos' > .crushrc
  

• Clean tree required — agents refuse repos with uncommitted changes (the pool never tramples human WIP; --allow-dirty opts out).

• Verify enforced — a task only completes when the repo still builds and tests pass. The verify command is resolved in priority order: the repo's own .tq-verify file wins (committed, reviewable — the repo decides how it is proven), then the payload's verify, then auto-detection (Go → go build ./... && go test ./... -count=1, package.json → npm test --silent, Makefile → make test, flake.nix → nix build && nix flake check, Cargo.toml → cargo test --quiet).

  # in the repo: pin exactly how agents must prove their work
  echo 'go vet ./... && go test ./... -count=1' > .tq-verify
  

• Fail fast, not fail often — input mistakes (bad payload, missing repo, unknown type) are dead-lettered after ONE attempt instead of burning the retry budget; environment problems (dirty tree, missing autonomy config) requeue the task without burning an attempt, so the pool picks it up once the human commits or adds the config.

• Cost ceilings — --max-per-tick bounds new tasks per harvest run, --daily-budget caps enqueues per calendar day, and --budget-cmd lets your own accounting veto every tick (exit non-zero = skip the tick). Poisoned repos (recent work all dead) pause harvesting for a cooldown (--dlq-backoff, together with --repo-interval for per-repo pacing).

• Paced & pruned — at most one in-flight backlog item per repo, and --prune-stale (on by default for pools) cancels pending tasks whose item is now [x], reworded, or gone from the file — work that no longer has a reason to run never runs.

• Machine-wide agent cap — --max-concurrent-agents N bounds agent processes across every tq pool on the host via slot files (this is what tq bootstrap --agents N sets).

• Full output sidecars — --log-dir DIR (or $TQ_LOG_DIR, or a log-dir = line in the pool config) writes each task's complete agent + verify output to DIR/<task-id>.log (0600). Without it, only a tail lands in the result detail — turn this on for daemon pools where you cannot watch the terminal. Retention: --log-dir-max-age 168h sweeps logs older than the age, --log-dir-max-bytes 5368709120 caps the directory's total size (oldest deleted first) — combine both so a long-running pool's output directory is bounded by time AND bytes.

• Durable — lease claims with heartbeats, exponential backoff, DLQ on exhaustion (tq dlq --rescue to retry), and the whole lifecycle replayable via tq facts.

Model selection rides the repo .crushrc, not the pool: bootstrap pins the model + reasoning effort into a managed .crushrc block per repo. A payload-level model would make the executor pass crush run -m, which empirically RESETS the reasoning effort to the provider default (crush debug telemetry, 2026-09-08). The .crushrc model large <provider/model> --reasoning-effort <effort> slot is the only mechanism that carries effort — it applies to both agent runs and interactive crush sessions in that repo. (tq agent-pool --model is retired — the pool refuses it, because a payload-level model resets the reasoning effort; bootstrap deliberately never composes it either.)

One command from zero: tq bootstrap

Everything above, wired in one idempotent command — per repo it validates the checkout, pins the verify contract into .tq-verify (auto-detected, or forced via --verify name=cmd), grants agent autonomy in a managed .crushrc block (plus the model + reasoning effort when --model is set), commits exactly those files (never pushes), previews the harvest, then starts the pool:

tq bootstrap CV,SystemNix --agents 2                        # named repos under --projects-dir
tq bootstrap CV SystemNix --model zai/glm-5.3-flash --once  # explicit model, one supervised tick
tq bootstrap CV --verify 'CV=templ generate && bash scripts/go-change-gate.sh' --dry-run
tq bootstrap --install                                      # systemd user unit + pool.conf, then exit

--agents N sets pool concurrency and the machine-wide agent cap; --reasoning defaults to xhigh (max possible; applied when --model is set). Full agent output sidecars default ON (~/.local/state/tq/logs/<task-id>.log; --log-dir "" turns them off) — a daemon you cannot watch needs its logs. Re-running is safe: the managed block is replaced in place, existing user config untouched.

Where the state lives

One embedded sqlite journal holds everything (facts, queue, read-model projection). Point tq at it with --db <path> (or $TQ_DB, default ./tasks.db). The four long-running surfaces (serve, api, worker, agent-pool) also accept --store, the unified frontend: a bare path is a sqlite file, a postgres:// DSN selects the shared-server driver (wired but not yet gate-verified — refused with a pointer until then).

Every sqlite home reads its IO durability tier from TQ_SQLITE_SYNC (full | normal | off, default normal: fsync at WAL checkpoints only — the queue tolerates a lost tail commit). The projection home sits next to the journal as <db>.readmodel.db.

Running unattended

The pool is meant to outlive your terminal. A hardened user-level systemd unit ships in deploy/systemd/tq-agent-pool.service (graceful 45-minute drain on stop so in-flight agents finish and record their outcome, NoNewPrivileges + read-only system paths, restart on failure):

mkdir -p ~/.config/systemd/user ~/.config/tq
cp deploy/systemd/tq-agent-pool.service ~/.config/systemd/user/
cat > ~/.config/tq/pool.conf <<'CONF'
# key=value, same names as the flags; flag > env > file precedence
projects-dir = /home/you/projects
yolo = true
project-exclusive = true
concurrency = 2
interval = 5m
daily-budget = 40
CONF
systemctl --user daemon-reload
systemctl --user enable --now tq-agent-pool.service
loginctl enable-linger $USER   # start at boot without a login session
journalctl --user -u tq-agent-pool -f

tq agent-pool --config <file> (or $TQ_POOL_CONFIG) reads flat key=value settings — same names as the flags — applied to every flag you did not pass explicitly. Precedence: flag > environment > config file > built-in default. Unknown keys are an error, so a typo in the file fails the pool loudly instead of silently running with defaults. A few flags read their default from the environment at startup (cqa-url, cqa-owner, cqa-token, log-dir, redact); for those the precedence is flag > env > file, so a live TQ_LOG_DIR beats a pool.conf line — the full key list lives in docs/planning/2026-10-10_agent-pool-flag-manifest.md.

Prefer cron or a systemd timer? tq agent-pool --once runs exactly one harvest tick, drains the queue, and exits — tasks owned by other pools or scheduled for later are left alone.

On NixOS, a module ships with the flake (nixosModules.default, declaring services.tq-agent-pool with the drain invariants and an explicit agent toolchain PATH baked in) — see deploy/nixos/tq-agent-pool.nix.

Where do logs live

Full agent output (stdout + the verify run) is written per task as a sidecar: ~/.local/state/tq/logs/<task-id>.log — the XDG state dir on purpose: logs are state, deletable without breaking anything. Files are created 0600, provider-token-shaped secrets are redacted on write, but repo paths and prompt content remain plaintext, so treat the directory as sensitive. Overrides and retention:

tq agent-pool --log-dir /var/log/tq            # or TQ_LOG_DIR; --log-dir "" turns sidecars off
tq agent-pool --log-dir-max-age 168h           # TQ_LOG_DIR_MAX_AGE: sweep sidecars older than 7d (0 = keep forever)
tq agent-pool --log-dir-max-bytes 5368709120   # TQ_LOG_DIR_MAX_BYTES: cap total size, oldest deleted first (0 = uncapped)

tq show <id> still carries the result tail and the sidecar path; the webui renders it on the task detail page. tq audit --journal reports SECRET EVIDENCE if secrets ever reached the stored journal. Note: a count is distinct secret LOCATIONS (overlapping pattern spans merge); SECRET EVIDENCE counts quoted in status reports dated before the 2026-09-16 span-merge fix (4ba81d8) counted raw pattern matches and are NOT comparable with fresh tq audit --journal output.

Every command at a glance

Session triggers: tq crush

tq session close needs a trigger that fires when an interactive session ends. The shipped one is the wrapper: run the session through it and the close-out happens on process exit — clean or crashed.

tq crush                                       # wraps the interactive crush session
tq crush --repo ~/projects/CV -- run -m "…"    # everything after the flags is the crush invocation

The child runs with inherited stdio (you watch everything live) while the wrapper tees the output to resolve the session id — --id, $CRUSH_SESSION_ID, or a scan of the child output — then runs the same replay-safe tq session close (one review + one status task over the session's Crush-Session: footer commits). The child's exit code is preserved and SIGINT/SIGTERM are forwarded; no id derivable means nothing is closed and the database is never touched. Remaining flags: --project, --summary, --allow-dirty, --db, --bin. Two hook-based triggers exist alongside it: the PreToolUse session registry (tq session ping/sweep, scripts/hook-session-registry.sh) and crush's upstream SessionEnd hook (crush #3146, unreleased).

Concepts

• Task — unit of work: type (executor key), project, JSON payload, priority, deps (task IDs that must complete first), maxAttempts, notBefore (delayed tasks).
• Fact — immutable journal record of everything that happens: task.enqueued, claimed, heartbeat, completed, failed, requeued, released, dead-lettered, cancelled, orphaned (DLQ rescue records task.enqueued again with a rescue detail; failed attempts carry structured failure evidence — stage, exit code, output tail — so a failure is debuggable from the journal alone). tq facts / tq tail -f replay the entire history; tq show TASK_ID renders one task with its complete fact trail — for agent tasks that includes the crush session id and the verify output tail.
• Claim / Lease — a worker claims a due task exclusively; the lease has a TTL renewed by heartbeat. If the worker dies, the lease expires and another worker reclaims the task. At-least-once execution, no stuck tasks.
• Deps — DAG ordering between tasks: a task with un-completed deps is not claimable. Completion of A unlocks B.
• DLQ — a task exhausting maxAttempts lands in Dead status; tq dlq --rescue re-queues it with a fresh attempt budget.
• Watermark — a journal consumer's persisted read cursor. Bridges and sweepers checkpoint through it, so restarts resume instead of replaying or skipping; tq watermarks show/set is the ops escape hatch.
• Executor — pluggable execution: sh (command), http (POST to URL), agent (headless AI coding agent with an enforced verify gate), or your own Go func. Workers look executors up by task type.
• Repo-side claim marker — the queue lease dedupes WORKERS, not two dispatches racing on the same task in the target repo. The consumer-side convention (a pre-flight tq-claim.sh claim + already_done / blocked_owner / blocked_claimed verdict shapes) is adopted by CV; the task-dispatch contract lives in the LarsArtmann/CV repo, not here.

Status codes

Embedding it in Go

The library core is importable. Public facade modules re-export each internal implementation module via type aliases (ADR-0016), so the names you import are stable while the implementations stay free to refactor:

(All under github.com/larsartmann/go-taskqueue/…. Each facade is an independently tagged Go module; the internal/… modules behind them are implementation detail.)

Both store drivers implement the same queue.Store contract with byte-compatible facts, so the choice is one import line:

import "github.com/larsartmann/go-taskqueue/queue/sqlite"

store, err := sqlite.Open("tq.db", sqlite.WithProjectExclusivity())
// or
import "github.com/larsartmann/go-taskqueue/queue/postgres"

store, err := postgres.Open(ctx, "postgres://…", 4)
// already holding a *pgxpool.Pool? reuse it instead of a second pool:
store, err := postgres.OpenWithPool(ctx, myPool) // pool stays yours; Store.Close will not close it

SQLite serializes writers through one connection (WAL + busy_timeout); Postgres uses SELECT … FOR UPDATE SKIP LOCKED so competing workers lock disjoint rows. The CLI itself wires the SQLite driver today; Postgres CLI wiring (--store postgres://…) is on the ROADMAP. Executors are pluggable in Go (register funcs on an executor.Registry and hand a worker.Pool any queue.Store).

If you only need one job type, see the minimal queue profile — the right-sized semantics this library's store contract implements — before deciding to embed or to build your own ~500-LOC fork. A runnable version of the embed story on public imports lives at examples/embed (go run ./examples/embed, --backend postgres --dsn … for the OpenWithPool path).

Distribution

Single-node SQLite by default. The Store interface is the distribution seam — and the first two slices of it already ship: a Postgres store (SELECT … FOR UPDATE SKIP LOCKED) lets multiple tq worker processes on different machines share the same queue with the same semantics, and tq api serves non-Go producers over HTTP with token auth. Outbound integrations ship today: a PapDashboard bridge turns dead letters into alerts — and later resolves them when the task is rescued and completes (tq worker --alert-url) — and a Code-Quality-Agent bridge turns scan findings into fix tasks (tq agent-pool --cqa-url). See ROADMAP for the remaining distribution work (CLI store wiring, consumer-group fencing tokens).

Development

nix run .#test        # full multi-module suite (root + every library module)
./scripts/ci-local.sh # the pre-push gate: full CI replicant (vet/build/race/smokes/nix)
go build ./... && go vet ./... && go test ./... -race   # root-module suite
./scripts/smoke/multi-repo.sh   # live smoke: 3 repos, 2 pools, 1 shared DB
./scripts/smoke/webui.sh        # live smoke: worker + tq serve + HTTP/SSE assertions

CI gates every push on vet, build, tests with -race, gofmt, a nix build, flake check, the live smokes, and a doc ghost-reference check. The e2e suite drives the real tq binary as a subprocess with a stub agent, so the full agent loop is tested in CI.

For contributors and agents: read AGENTS.md first — it carries the architecture invariants (single serialized SQLite writer, facts in the same transaction as state, task contexts that survive shutdown). Feature status lives in FEATURES.md, upcoming work in TODO_LIST.md, long-term direction in ROADMAP.md, design decisions in docs/adr/, and the domain vocabulary in docs/DOMAIN_LANGUAGE.md.

examples/api (enqueue endpoint, Prometheus /metrics, live stats page) and examples/sse (fact stream with Last-Event-ID resume) are proofs-of-concept — examples/ stays experimental by design and tq serve is the production dashboard path. Those two PoC HTTP servers bind loopback only and have no authentication; do not expose them. examples/fullcore is the full library embed demo in one file: queue + executors + worker pool, with the sqlite/postgres backend picked at run time.

License

MIT.