harvest

package
v0.4.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 8, 2026 License: MIT Imports: 29 Imported by: 0

Documentation

Overview

This file owns EFFORT ESTIMATION for harvest batch sizing: it predicts per-item work effort (minutes) so batches are sized by predicted work instead of raw item count (effort-aware batching, 2026-11-03). The harvester itself stays OFFLINE unless an estimator is wired via Config.EffortEstimator — the network client below only runs behind that flag.

The split follows the TypeSafe integration craft rules: Jev (the System One model behind api.typesafe.ai) supplies the JUDGMENT — one score question per item over a five-level effort rubric, answered as a calibrated expected score — and code owns everything mechanical: the rubric-to-minutes interpolation, the clamps, the [10, 240] bound, the cache, and the fallback. A failed or absent estimate is the flat per-item default, never an error: estimates are SCHEDULING SIGNALS (the ADR-0015 §signals rule verbatim) — they size runs and raise timeouts, never lower them, and never touch trust or priority.

PRIVACY: enabling the Jev estimator sends each item's text, its section heading, the repo name and the repo's stated purpose to the configured TypeSafe endpoint (default https://api.typesafe.ai). The pool flag help says so; the endpoint only sees that state, nothing else from the repo.

Estimates are cached (JSON sidecar, default <tq-db>.effort.json) keyed by the item key — which hashes the text, so an edited item re-keys and re-enters unscored — and stamped with the answering model: a model swap invalidates every cached verdict (the prioritize scorerSource lesson), so fresh work is never ranked or sized against a retired model's judgment.

Package harvest turns repository backlogs into queue tasks.

It is the ingestion half of a self-managing agent pool: every repo's TODO_LIST.md is the shared, human-readable work backlog; the harvester repeatedly scans it and enqueues one agent task per open item, paced to at most one in-flight task per repo. Agents (see executor.AgentExecutor) do the item and mark it done in the same file, which closes the loop: done items are never re-enqueued, edited items become new work.

Viewpoint — Consumer: the pool orchestrator pacing one in-flight task per repo; Concern: faithful backlog→task conversion (dedup keys, batched items, priority markers, drift audit, prune-stale); Non-goals: running the tasks (the agent executor does that) and writing the backlog (agents mark their items done).

Cross-package map (lifecycle × concern): docs/architecture-understanding/2026-10-23_lifecycle-concern-grid.md

Index

Constants

View Source
const (
	// DefaultJevBaseURL is the TypeSafe System One endpoint.
	DefaultJevBaseURL = "https://api.typesafe.ai"

	// DefaultJevModel is the Jev alias used when none is configured.
	DefaultJevModel = "jev-latest"

	// MinMinutes is the floor for any single-item estimate: smaller work
	// still pays a session start, and a near-zero estimate would pack an
	// unbounded context into one run.
	MinEffortMinutes = 10

	// MaxMinutes is the ceiling for any single-item estimate: a monster
	// item ships as its own batch, and the pool's scaled --task-timeout
	// default never needs to exceed it while effort mode is on.
	MaxEffortMinutes = 240

	// DefaultJevRequestTimeout bounds one System One request.
	DefaultJevRequestTimeout = 30 * time.Second
)
View Source
const (
	// RedispatchClassLive: a PENDING or RUNNING task whose row is already
	// closed — it will burn a window (or die) proving done work. The mint
	// gate refuses NEW mints of this shape; this is the residue already
	// inside the queue (tq cancel / prune-stale territory).
	RedispatchClassLive = "live-task-on-closed-row"
	// RedispatchClassChurn: a TERMINAL task on a closed row that burned
	// more than one attempt — the paid verify-churn census (the row-110
	// class: five close-out doc commits for an already-[x] row).
	RedispatchClassChurn = "spend-after-close"
)

Redispatch finding classes: what the row-vs-queue join means for the operator reading `tq audit --redispatch`.

View Source
const BatchKeyPrefix = "batch:"

BatchKeyPrefix marks a batched task's dedup key (harvest --batch-items).

View Source
const CatchupKeyPrefix = "catchup:"

CatchupKeyPrefix marks a catch-up task's dedup key. A catch-up task asks an agent to tick a checkbox whose work is already done; the prefix keeps it distinct from the original item's key, so it can never mask the drift it exists to close.

View Source
const DefaultBatchPromptTemplate = `` /* 1730-byte string literal not displayed */

DefaultBatchPromptTemplate is the agent contract for BATCHED work items (harvest --batch-items > 1): one task carries a section's open items (chunked to the batch cap), worked in order by ONE session. Outcome-shaped like DefaultPromptTemplate (2026-10-11 redesign; mechanical rails only). Placeholders: {{REPO_ABS}}, {{REPO}}, {{REPO_PURPOSE}}, {{HEADING}}, {{COUNT}}, {{ITEMS}} (numbered list) and {{TASK_ID}} (resolved at EXECUTION time, one footer for every commit of the batch — derived outcomes attribute them all to this task).

View Source
const DefaultBatchTimeoutMinutes = 30

DefaultBatchTimeoutMinutes is the per-item ceiling a batch scales from when the repo timeout ladder has no entry (executor default: 30). It is also the flat per-item EFFORT estimate used when no estimator is configured, and the unit the derived effort budget (BatchItems x this) counts in — one number, three roles, never forked (2026-11-03 single-sourcing: the pool's scaled --task-timeout default derives from it too).

View Source
const DefaultCatchupPrompt = `` /* 1155-byte string literal not displayed */

DefaultCatchupPrompt is the contract for catch-up tasks: close the loop in the file, change nothing else.

View Source
const DefaultDiscoveryTimeout = 5 * time.Second

DefaultDiscoveryTimeout bounds one daemon discovery RPC: an unreachable or wedged daemon must never stall a harvest tick (the local scan is the fallback and costs milliseconds).

View Source
const DefaultImportance = 50

DefaultImportance is the effective importance of a repo without a .config/metadata.yaml (project-meta file contract, ADR-0015 §7).

View Source
const DefaultMaxPerTick = 10

DefaultMaxPerTick bounds how many new agent tasks one harvest run may enqueue across all repos — the cost throttle for unattended pools.

View Source
const DefaultPromptTemplate = `` /* 1515-byte string literal not displayed */

DefaultPromptTemplate is the agent contract handed to Crush for each item. Outcome-shaped since 2026-10-11 (docs/problems/major-prompting-and-batching.md): the trust budget goes to verification, not prose — AGENTS.md and the verify gate own the rules; this template carries only goal, context pointer, the mechanical rails (re-dispatch no-op, footer, verify, safety lines, TODO loop-closing, backlog routing) and nothing else. Placeholders: {{REPO_ABS}}, {{REPO}}, {{REPO_PURPOSE}}, {{HEADING}}, {{ITEM}}, and {{TASK_ID}} (substituted at EXECUTION time by the agent executor — the queue task ID does not exist when the harvester renders this template; it exists so commits can carry a Task-Queue-ID footer for git log ↔ tq facts cross-reference).

View Source
const DefaultTodoFile = "TODO_LIST.md"

DefaultTodoFile is the backlog file scanned in each repository.

View Source
const DefaultType = "agent"

DefaultType is the task type harvest enqueues: the headless agent executor (executor.AgentExecutor, registered as "agent" by the tq CLI).

View Source
const MaxMarkerLevel = 4

MaxMarkerLevel is the highest valid marker level (P4).

View Source
const ReasonScanFailed = "scan failed"

ReasonScanFailed prefixes every repo-level scan error in Skipped.Reason (the detail after the colon is the underlying error).

Variables

View Source
var ErrNoRepos = errors.New("harvest: no repos and no projects dir configured")

ErrNoRepos is returned when neither --repos nor a --projects-dir is configured: there is nothing to harvest. One sentinel (previously the same message was inlined in Run and Audit independently, so callers could not match it).

View Source
var ErrRedispatchRefused = errors.New(
	"harvest: re-dispatch refused (work already done; force the re-dispatch to verify anyway)",
)

ErrRedispatchRefused is the mint-time done-check refusal (O4 ruling, 2026-10-05): the candidate describes work that repo-side signals prove already landed, so minting it buys a paid verify-only lap over done work. The escape hatch is --force-redispatch (tq enqueue) / Config.ForceRedispatch.

Functions

func DiscoverRepos

func DiscoverRepos(dir, todoFile string) ([]string, error)

DiscoverRepos returns the depth-1 subdirectories of dir that contain the todo file, sorted by name.

func DiscoverReposDaemon added in v0.2.0

func DiscoverReposDaemon(ctx context.Context, addr, projectsDir, todoFile string) ([]string, error)

DiscoverReposDaemon enumerates candidate repos on a project-discovery-daemon (POST /v1/discover over the HTTP endpoint at addr, usually a unix socket path such as /run/project-discovery/daemon.sock) and maps the result to harvestable repos — absolute project paths that contain the todo file, deduplicated and sorted, exactly the contract DiscoverRepos upholds for the local scan. Addr forms: /path/to/sock and unix:///path/to/sock dial a unix socket, anything else is treated as a host:port TCP address (what httptest servers and networked daemons use).

func DiscoverReposFor added in v0.2.0

func DiscoverReposFor(ctx context.Context, addr, projectsDir, todoFile string, log *slog.Logger) ([]string, error)

DiscoverReposFor resolves the candidate repos for one harvest tick: with a daemon addr it asks the daemon and, when that fails (socket unreachable, non-200, bad payload), logs ONE warning and falls back to the local scan — daemon mode is additive and must never fail the tick. An empty addr is the plain scan (the zero-external-services default).

func ItemKey

func ItemKey(repoName, text string) string

ItemKey derives the stable dedup key for an item: the hash of repo name and item text. Item text is whitespace-collapsed so reflowing a line does not create duplicate work.

func JevSourceFor added in v0.4.2

func JevSourceFor(model string) string

JevSourceFor stamps an estimate with the model that produced it.

func KeywordBump added in v0.3.0

func KeywordBump(text string) int

KeywordBump reports the strongest keyword bump for an item text, 0 when none matches. Case-insensitive.

func MarkerPriority added in v0.3.0

func MarkerPriority(level int) int

MarkerPriority returns the backlog priority of a marker level — P1 through P4. 0 ("no marker") maps to 0.

func MinutesFromScore added in v0.4.2

func MinutesFromScore(score float64) int

MinutesFromScore converts a rubric score (0..len(rubric)-1, fractional expected values allowed) into estimated minutes: piecewise-linear between level minutes, rounded UP to the next 5-minute step (never under-ask on a fraction), clamped to [MinEffortMinutes, MaxMinutes].

func ReadImportance added in v0.3.0

func ReadImportance(repo string) (int, error)

ReadImportance reads a repo's importance from its .config/metadata.yaml (the project-meta flat file contract). Absent file: DefaultImportance. Present but malformed (unparseable importance, or out of 0–100): an error the caller reports as a repo skip reason — an importance the owner DID set must never silently degrade to a default.

This is a deliberate strict-subset reader: project-meta is proprietary (ADR-0015 §7), so tq parses the two bytes it needs instead of requiring the module. Only a top-level `importance:` key counts; indented keys belong to nested mappings and are ignored.

func RepoPurpose added in v0.4.2

func RepoPurpose(repo string) string

RepoPurpose reads a repo's one-line purpose from .config/metadata.yaml's top-level `purpose:` scalar — the goal-ancestry rung above the item and its section heading: an agent that knows WHY the repo exists stops asking whether its work item even matters (M23). Absent file, absent key, or parse trouble: the honest fallback names the README, never an invented purpose.

func RepriMutable added in v0.3.0

func RepriMutable(current int) bool

RepriMutable reports whether a re-resolution pass (tq reprioritize, the prioritize sweeper) may rewrite a task currently stored at current: hot (100–149) and machine (150+) priorities are explicit human/flag acts and are never overwritten by automated re-resolution — the band protection half of the ADR-0015 §3 precedence.

func RubricSituations added in v0.4.2

func RubricSituations() []string

RubricSituations returns the ordered rubric level descriptions (the score question criteria).

func SplitMarker added in v0.3.0

func SplitMarker(text string) (int, string, bool)

SplitMarker splits a trailing priority marker off an item line, returning the marker level (1–4) and the marker-free text. ok is false when the text carries no marker — including a marker-only line (there is no item left to prioritize).

Types

type Config

type Config struct {
	// ProjectsDir is scanned (depth 1) for repos containing TodoFile.
	ProjectsDir string
	// DiscoveryAddr points at a project-discovery-daemon endpoint (unix
	// socket path, unix:// prefixed, or host:port) used INSTEAD of the
	// depth-1 ProjectsDir scan to enumerate candidate repos. Additive by
	// contract: an unreachable daemon logs one warning per tick and the
	// scan takes over. Empty keeps the zero-external-services default.
	DiscoveryAddr string
	// Log receives the daemon-discovery degradation warning; nil discards.
	Log *slog.Logger
	// Repos lists explicit repo directories; when set, ProjectsDir is ignored.
	Repos []string
	// Type is the task type enqueued (must match a registered executor).
	Type string
	// TodoFile is the backlog file name inside each repo.
	TodoFile string
	// MaxPerTick caps new enqueues per run across all repos.
	MaxPerTick int
	// MaxAttempts is the attempt budget for harvested tasks (0 = default).
	MaxAttempts int
	// Priority for harvested tasks.
	Priority int
	// SameSessionPriority promotes items whose text references /tmp paths
	// to this priority at enqueue time ("hot": work the same session, the
	// files will not survive). Zero disables the promotion.
	SameSessionPriority int
	// UseImportance resolves harvested priorities from the repo's
	// .config/metadata.yaml importance (default 50) plus keyword bumps,
	// clamped to the backlog band (ADR-0015 §3). Markers and hot promotion
	// apply regardless of this flag; a malformed metadata file skips the
	// repo's items with a reason instead of guessing.
	UseImportance bool
	// MaxPendingPerRepo caps how many PENDING tasks one repo may hold in
	// the queue (0 = unlimited): the queue is the WORKING SET, TODO_LIST.md
	// is the warehouse — prompts stay fresh and AI cost stays O(working
	// set). Admission resumes as claims drain the slot; running tasks do
	// not count (the repo-busy rule owns those).
	MaxPendingPerRepo int
	// PromptTemplate overrides DefaultPromptTemplate.
	PromptTemplate string
	// BatchPromptTemplate overrides DefaultBatchPromptTemplate (used only
	// when BatchItems > 1).
	BatchPromptTemplate string
	// BatchItems groups up to this many open items from the same
	// TODO_LIST.md section into ONE agent task (one session works them
	// in order): fewer cold sessions, more done per provider window.
	// Grouping is SECTION-scoped, not adjacency-scoped: a done, blocked
	// or already-known item between fresh ones does not fork the batch
	// (the section-goal form of
	// docs/problems/major-prompting-and-batching.md, 2026-11-03).
	// 0/1 = off (one item per task). The pool CLI default is 5 since
	// 2026-11-03 (3 from 2026-10-11). A batch is ONE task against every
	// gate (--max-per-tick, daily budget, repo pacing) — the per-item
	// cost is amortized, so raise the gates consciously.
	BatchItems int
	// BatchEffortMinutes caps the SUMMED per-item effort estimates of
	// one batched run (effort-aware batching, 2026-11-03): a 3-hour item
	// ships alone while ten 10-minute items ride together, all under
	// the same context cap. 0 = BatchItems x DefaultBatchTimeoutMinutes,
	// which under flat estimates is exactly the count cap (zero behavior
	// change without an estimator). Estimates are SCHEDULING SIGNALS
	// (ADR-0015 §signals): they size runs and raise payload timeouts,
	// never lower them, and never touch trust or priority.
	BatchEffortMinutes int
	// EffortEstimator, when set, predicts per-item work minutes (keyed
	// by Item.Key) for run sizing and timeout budgets. nil keeps the
	// flat per-item default (count-equivalent grouping). Called once per
	// repo with the admissible candidates; a missing key or non-positive
	// value falls back to the flat default, so a failed estimation can
	// never drop or stall work.
	EffortEstimator EffortEstimator
	// Model overrides the crush model ("provider/model") in every harvested
	// agent payload. Empty = the agent binary's default model.
	Model string
	// RepoIntervals sets a minimum gap between new enqueues per repo (by
	// repo name): after any enqueue for a repo, further items wait until
	// the interval has passed. Empty or zero entries = only the default
	// one-per-tick pacing applies.
	RepoIntervals map[string]time.Duration
	// RepoTimeouts sets a per-repo agent-task timeout ladder (by repo
	// name), pinned into each harvested payload's TimeoutMinutes: big
	// repos get long ceilings, quick ones stay tight. Repos without an
	// entry keep the executor's 30-minute default.
	RepoTimeouts map[string]time.Duration
	// DLQBackoff pauses harvesting of a repo whose recent work all went to
	// the dead-letter queue (dead tasks present, none completed, newest
	// dead within the window): the repo is poisoned until a human fixes or
	// rescues it. Zero disables the guard. Default 0 (set it, e.g. 30m).
	DLQBackoff time.Duration
	// RequireClean passes the clean-tree policy through to agent payloads:
	// nil = executor default (require a clean git tree), false lets agents
	// run on dirty repos (scratch/fixtures only), true forces the check.
	RequireClean *bool
	// DryRun reports what a real run would enqueue, without writing to the
	// queue. Result.Enqueued entries then carry an empty TaskID.
	DryRun bool
	// ForceRedispatch disables the mint-time done-check (the O4 escape
	// hatch): candidates whose work repo signals prove already landed are
	// minted anyway — a forced window is then expected to produce the stop
	// artifact (re-verify annotation + footer commit), not redo work.
	ForceRedispatch bool
	// RedispatchCooldown is the window a live repo-side claim marker
	// (.tq/claimed/<id>, tq-claim.sh) machine-blocks re-dispatch of the
	// same work at mint time (owner ruling fam.12.3, 2026-10-17): a
	// marker younger than the window means another dispatched session
	// holds the item. Zero = 6h (the tq-claim.sh TQ_STALE_AFTER default);
	// the claim sweep releasing stale/footer-committed markers is what
	// ends the block, never this gate alone. The sibling verdict marker
	// (.tq/verdicts/<id>, already_done terminal state) blocks while
	// PRESENT regardless of this window — it expires only via the marker
	// TTL sweep (TQ_VERDICT_TTL). Both markers are read by the mint-time
	// RedispatchCheck; ForceRedispatch bypasses them with the rest of the
	// gate.
	RedispatchCooldown time.Duration
}

Config controls one harvester.

type Drift added in v0.2.0

type Drift struct {
	Kind       DriftKind
	Item       Item
	TaskID     task.ID
	TaskStatus task.Status
}

Drift is one file-vs-queue disagreement found by Audit.

type DriftKind added in v0.2.0

type DriftKind string

DriftKind names which side of the file-vs-queue contract broke.

const (
	// DriftStaleOpen: the queue says the work completed, the checkbox is
	// still open — the loop was never closed in the file (or was reverted).
	DriftStaleOpen DriftKind = "stale-open"
	// DriftStaleDone: the checkbox is ticked but the task never completed
	// (pending, running or dead) — someone did the work by hand or ticked
	// prematurely; the queue task is now pointless. Report-only: cancelling
	// human-visible work is an operator decision (`tq cancel`).
	DriftStaleDone DriftKind = "stale-done"
)

type DriftResult added in v0.2.0

type DriftResult struct {
	Repos int
	// StaleOpen lists completed-work-open-checkbox drift; with DryRun unset,
	// each entry here also got a catch-up task enqueued (see Enqueued).
	StaleOpen []Drift
	// StaleDone lists ticked-checkbox-unfinished-task drift (report-only).
	StaleDone []Drift
	// Enqueued lists the catch-up tasks actually created this pass.
	Enqueued []Enqueued
	// ScanFailures lists repos that could not be read during the audit;
	// the audit keeps going past them.
	ScanFailures []ScanFailure
}

DriftResult summarizes one audit pass over all repos.

type EffortCache added in v0.4.2

type EffortCache struct {
	// contains filtered or unexported fields
}

EffortCache is the file-backed estimate store: JSON sidecar, single writer (the pool process), advisory in every direction — a lost or corrupt cache costs one re-estimate pass, nothing else.

func NewEffortCache added in v0.4.2

func NewEffortCache() *EffortCache

NewEffortCache returns an empty estimate cache.

func (*EffortCache) Get added in v0.4.2

func (c *EffortCache) Get(key, model string) (EffortEntry, bool)

Get returns the cached estimate when it exists AND its source stamp matches the given model (a model swap invalidates everything — never size fresh work against a retired model's judgment).

func (*EffortCache) Len added in v0.4.2

func (c *EffortCache) Len() int

Len reports the entry count (tests).

func (*EffortCache) Load added in v0.4.2

func (c *EffortCache) Load(path string) error

Load reads the sidecar, pruning entries older than cacheMaxAge.

func (*EffortCache) Put added in v0.4.2

func (c *EffortCache) Put(key string, minutes int, model string, confidence float64)

Put records an estimate under the given model's stamp.

func (*EffortCache) Save added in v0.4.2

func (c *EffortCache) Save(path string) error

Save writes the sidecar best-effort (direct write: single writer, a torn write self-heals as an unreadable cache).

type EffortEntry added in v0.4.2

type EffortEntry struct {
	Minutes    int       `json:"minutes"`
	Source     string    `json:"source"` // "jev:<model>"
	Confidence float64   `json:"confidence"`
	At         time.Time `json:"at"`
}

EffortEntry is one cached estimate.

type EffortEstimator added in v0.4.2

type EffortEstimator interface {
	// EstimateEffort returns predicted minutes keyed by Item.Key.
	// Missing keys and non-positive values use the flat default.
	EstimateEffort(ctx context.Context, items []Item) map[string]int
}

EffortEstimator predicts per-item work effort (minutes) so batched runs are sized by predicted work, not raw item count. Estimates are advisory scheduling signals, never trust or priority (ADR-0015 §signals): implementations must treat failures as "no estimate" (omit the key) rather than errors — the harvester falls back to the flat per-item default for anything missing.

type Enqueued

type Enqueued struct {
	Item   Item
	TaskID task.ID
	Fresh  bool // true when this run actually created the task
	Hot    bool // text referenced /tmp: enqueued at SameSessionPriority
}

Enqueued records a task created (or already present) for an item.

type Harvester

type Harvester struct {
	// contains filtered or unexported fields
}

Harvester scans repos and enqueues agent tasks for open todo items.

func New

func New(q *queue.Queue, cfg Config) *Harvester

New creates a Harvester over a queue.

func (*Harvester) Audit added in v0.2.0

func (h *Harvester) Audit(ctx context.Context) (DriftResult, error)

Audit compares every repo's checkbox states against the queue's terminal task states and repairs one direction of drift: when a task COMPLETED but its checkbox is still open, a catch-up task (dedup-keyed CatchupKeyPrefix + item key) is enqueued to close the loop in the file. The store's unique dedup index makes that enqueue-once, so a repeating audit never piles up duplicate catch-ups even if the first one also fails to tick the box. The opposite direction (ticked checkbox, unfinished task) is reported only. DryRun reports everything without enqueuing.

func (*Harvester) DonePreflight added in v0.3.3

func (h *Harvester) DonePreflight(ctx context.Context, t task.Task) (bool, string)

DonePreflight is the claim-time done gate (docs/planning/2026-10-06_18-50 _dispatch-done-preflight-gate.md): it decides whether a claimed task's work is provably already done, so the worker can complete it with zero agent spend instead of re-running a full session on landed work — the re-fire class (one task: enqueued once, claimed 15×, every claim a paid no-op over 21 footer commits).

Signals, strongest first, first hit wins (the reason names the signal):

  1. Footer commits: at least one commit on any ref carries this task's Task-Queue-ID footer (works for harvested AND fix tasks). For a MULTI-member batch the footer proves only the members whose work landed in that commit — the signal completes only when every member key is also ticked-or-gone (the 2026-10-14 04:17 reclaim of task 000001a1382a1765 completed 1/3 members zero-cost on the footer alone; a partial footer now falls through to the remaining signals, and a not-done verdict re-runs the agent over the open members).
  2. Fix-ticket disposition: payload.RejectedSHA exists AND a commit cites it (superseding fix), or the SHA is gone AND the anchor text has zero hits in the working tree (finding cured by the rebase).
  3. Harvest item state (the prune-stale rules applied per claim): the item's dedup key is now `[x]`, or gone from the file entirely.
  4. Closeout report: docs/status/tasks/*_task-<id>* exists in the repo (legacy docs/status/*_task-<id>* reports count too).

Foreign payload shapes (no Repo/Prompt), unresolvable repos, git/file errors and timeouts all evaluate to NOT done: the gate never guesses and never blocks dispatch on its own failure.

func (*Harvester) PruneStale added in v0.2.0

func (h *Harvester) PruneStale(ctx context.Context) (PruneResult, error)

PruneStale cancels pending queue tasks whose TODO_LIST item is now `[x]` (dedup-key match), or whose item text is gone from the file entirely, so a pool relaunch never inherits zombies: harvesting while no pool runs creates pending tasks, and ticking — or deleting — an item later never withdraws them (21:40 report §d1/§e1 — six stale tasks sat 5–24 h). Only PENDING tasks are cancelled; running ones are reported (a cooperative stop mid-execution is an operator decision), dead ones are terminal. Only tasks carrying a harvest dedup key in their payload are ever touched — external work is invisible to the sweep. DryRun reports without cancelling.

func (*Harvester) RedispatchAudit added in v0.3.3

func (h *Harvester) RedispatchAudit(ctx context.Context) (RedispatchResult, error)

RedispatchAudit surfaces re-dispatch exposure as a queried fact (row 272): every task whose TODO row is already closed — live tasks that will spend a window proving done work, and terminal tasks that burned attempts on closed work. Report-only: cancelling live residue is an operator decision (prune-stale owns pending withdrawals at pool start).

func (*Harvester) RedispatchCheck added in v0.3.3

func (h *Harvester) RedispatchCheck(ctx context.Context, n task.New) error

RedispatchCheck is the mint-time done gate (row 115, O4): it judges a MINT CANDIDATE — before the task exists, before any spend — against the repo-side signals the store's dedup index cannot see. The claim-time DonePreflight answers "is this CLAIM a re-fire?"; this answers "should this MINT happen at all?". Signals, first hit wins:

  1. Fix-ticket cured: a review-fix candidate whose rejected-SHA disposition already reads cured (superseded, rebased away, or — the four-paid-lap 2026-10-07 class — the SHA lives but the anchor text is gone from HEAD). The dispatch footer-join dedup three reports demanded.
  2. Item closed: the candidate's harvest item keys resolve to rows now ticked or absent (the check-off race row 164: a row can close between the harvest scan and the mint — re-read the file at mint time).
  3. Closeout indexed: the prompt cites a task ID whose closeout report already exists under docs/status — a re-verification dispatch of closed work (the row-110 class: five close-out doc commits for an already-[x] row).
  4. Coordination markers: a prior dispatch of the same work left a .tq/verdicts/<id> already_done marker (terminal state, fam.12.2) or a live .tq/claimed/<id> claim (another session holds it, the Config.RedispatchCooldown window, fam.12.3) — found by prompt citation or by the item-key join through the queue (markerBlocked).

Foreign payload shapes, unresolvable repos, git/file errors and timeouts all pass: the gate refuses only on positive proof, never on its own failure.

func (*Harvester) Reprioritize added in v0.3.0

func (h *Harvester) Reprioritize(ctx context.Context, dryRun bool) ([]RepriChange, []string)

Reprioritize re-resolves the priorities of PENDING tasks from the CURRENT truth in the todo files (markers) and, with UseImportance, each repo's metadata importance — so an owner's marker edit or importance change reaches tasks already sitting in the queue (ADR-0015 §5).

Rules: only PENDING tasks are touched; hot/machine tasks are protected (RepriMutable); same-value writes are no-ops (the store appends no fact); dryRun reports without writing. Per-repo scan failures are returned in failures, never as errors — a broken repo must not block the others.

func (*Harvester) Run

func (h *Harvester) Run(ctx context.Context) (Result, error)

Run performs one scan-and-enqueue pass. It never fails on individual repos; repo-level errors are reported as Skipped entries with the error as reason.

type Item

type Item struct {
	Repo     string // absolute repo path
	RepoName string // project name (repo dir base name)
	Heading  string // nearest markdown heading above the item
	Text     string // the checkbox text, trimmed, priority marker stripped
	Key      string // stable dedup key: hash of repo name + MARKER-STRIPPED text
	Done     bool   // true when the checkbox is ticked ([x])
	// MarkerLevel is the parsed trailing `— P[1-4]` marker (1–4); 0 = none.
	// Stripped from Text before hashing so editing a marker never forks a
	// task (ADR-0015 §2). omitempty keeps legacy JSON consumers and golden
	// fixtures byte-stable for unmarked items.
	MarkerLevel int `json:"markerLevel,omitempty"`
}

Item is one checkbox found in a repo's todo file.

func ParseRepo

func ParseRepo(repo, todoFile string) ([]Item, error)

ParseRepo extracts the open checkbox items from a repo's todo file.

Semantics: a line matching `- [ ]`/`* [ ]` (leading whitespace allowed) is an item; `- [x]`/`- [X]` are done and ignored (ParseRepoAll returns them with Done set). The nearest heading above the item (any level, text without the #s) is its Heading. Lines inside fenced code blocks are ignored, so TODO examples in documentation are never harvested. The Key changes when the item text changes, so editing an item re-arms it even if a previous task for the old wording exists.

func ParseRepoAll added in v0.2.0

func ParseRepoAll(repo, todoFile string) ([]Item, error)

ParseRepoAll returns every checkbox item (open and done) from a repo's todo file; Item.Done distinguishes them. It backs the drift auditor, which compares BOTH checkbox states against terminal task states.

type JevConfig added in v0.4.2

type JevConfig struct {
	// APIKey is the TypeSafe bearer token (required — a missing key at
	// startup is a configuration error, not a runtime fallback).
	APIKey string
	// Model overrides DefaultModel.
	Model string
	// BaseURL overrides DefaultBaseURL (tests point it at httptest).
	BaseURL string
	// Timeout bounds one request; 0 = DefaultTimeout.
	Timeout time.Duration
	// CachePath persists estimates as a JSON sidecar; empty disables
	// persistence (every pass re-estimates).
	CachePath string
	// Log receives one WARN line per degraded pass; nil discards.
	Log *slog.Logger
}

JevConfig configures one JevEstimator.

type JevEstimator added in v0.4.2

type JevEstimator struct {
	// contains filtered or unexported fields
}

JevEstimator implements harvest.EffortEstimator with Jev: cache-first per item key, one batched System One request per pass for the misses, flat-default on any failure (missing keys — the harvest falls back itself). It is safe for sequential harvest use (the pool loop calls it once per repo per tick).

func NewJevEstimator added in v0.4.2

func NewJevEstimator(cfg JevConfig) (*JevEstimator, error)

NewJevEstimator loads the estimate cache (a missing or corrupt sidecar starts empty with a warning — the cache is advisory) and returns the estimator. An empty API key is refused: the pool should fail fast on a misconfigured key, not silently run estimator-less.

func (*JevEstimator) EstimateEffort added in v0.4.2

func (e *JevEstimator) EstimateEffort(ctx context.Context, items []Item) map[string]int

EstimateEffort implements harvest.EffortEstimator.

type PayloadItem added in v0.3.0

type PayloadItem struct {
	// Repo is the payload's AgentPayload.Repo — the resolution-ready
	// reference the agent executor itself uses.
	Repo string
	// Text is the marker-stripped item text.
	Text string
	// Key is the item's dedup key (ItemKey derivation) — the score-cache
	// key.
	Key string
	// MarkerLevel is 0 (no marker) or 1-4 when the item carried a
	// `— P[1-4]` marker at harvest time.
	MarkerLevel int
}

PayloadItem is the harvested backlog-item identity carried by a task payload: what was harvested, from where, under which dedup key.

func PayloadItemOf added in v0.3.0

func PayloadItemOf(t task.Task) (PayloadItem, bool)

PayloadItemOf reads one task's payload back into its harvested item identity. ok is false for non-harvest payloads (foreign agent tasks, review/status/dlqfix mints): only harvest-minted tasks carry the "dedup" key.

type PriorityScorer added in v0.3.1

type PriorityScorer interface {
	PriorityScore(ctx context.Context, itemKey string) (queue.PriorityScore, bool, error)
}

PriorityScorer is the score-cache read the priority story needs: both the sqlite store behind `tq show` and the webui server's queue.Store satisfy it.

type PrioritySource added in v0.3.0

type PrioritySource string

PrioritySource names what produced an effective priority (ADR-0015 §3).

const (
	// PrioritySourceHot: same-session promotion.
	PrioritySourceHot PrioritySource = "hot"
	// PrioritySourceMarker: trailing `— P[1-4]` human marker.
	PrioritySourceMarker PrioritySource = "marker"
	// PrioritySourceAI: cached priority_scores verdict.
	PrioritySourceAI PrioritySource = "ai"
	// PrioritySourceKeyword: importance + keyword bump.
	PrioritySourceKeyword PrioritySource = "keyword"
	// PrioritySourceImportance: bare repo importance.
	PrioritySourceImportance PrioritySource = "importance"
	// PrioritySourceDefault: flat config priority (legacy behavior).
	PrioritySourceDefault PrioritySource = "default"
)

func ResolvePriority added in v0.3.0

func ResolvePriority(input ResolveInput) (int, PrioritySource)

ResolvePriority applies the precedence ladder (ADR-0015 §3) and returns the effective priority with the source that won. Backlog computations clamp to [0, queue.BacklogMax]; only HotPriority may cross upward.

type Provenance added in v0.3.1

type Provenance struct {
	Current     int
	Band        string
	ItemKey     string
	MarkerLevel int
	Score       *queue.PriorityScore
	History     []ProvenanceEvent
}

Provenance is a task's priority story (ADR-0015): what the task's priority is, which band it sits in, the harvested item identity with its cached AI verdict (when the item was ever scored), and the reprioritization history — the answer to "why is this task ranked here?". Renderers project it into their own view shapes (`tq show`'s JSON section, the webui detail rows).

func BuildProvenance added in v0.3.1

func BuildProvenance(ctx context.Context, scorer PriorityScorer, t task.Task, trail []journal.Fact) Provenance

BuildProvenance reads a task's priority story off the score cache and its fact trail. Best-effort by design: a score-cache or evidence-parse miss simply omits the field rather than failing the caller.

type ProvenanceEvent added in v0.3.1

type ProvenanceEvent struct {
	At     time.Time
	Old    int
	New    int
	Source string
	Reason string
}

ProvenanceEvent is one distilled task.reprioritized fact.

type PruneResult added in v0.2.0

type PruneResult struct {
	Repos int
	// Cancelled lists pending tasks withdrawn because their TODO_LIST item
	// is now ticked — the zombies a pool relaunch would otherwise execute —
	// or because the item text is gone from the file entirely.
	Cancelled []PrunedTask
	// Running lists stale-item tasks currently executing: a cooperative
	// stop is an operator decision (tq cancel), not a sweep default.
	Running []PrunedTask
	// Dead lists stale-item tasks already dead-lettered: terminal, listed
	// because the dedup key still suppresses re-enqueue if the item is
	// ever un-ticked with unchanged text.
	Dead []PrunedTask
	// ScanFailures lists repos that could not be read; the prune keeps
	// going past them (same contract as Audit).
	ScanFailures []ScanFailure
}

PruneResult summarizes one stale-task prune pass over all repos.

type PruneWhy added in v0.2.0

type PruneWhy string

PruneWhy records which stale rule withdrew (or reported) a task.

const (
	// PruneTicked: the item is still in the file and now `[x]`.
	PruneTicked PruneWhy = "ticked"
	// PruneAbsent: the item text is gone from the file entirely.
	PruneAbsent PruneWhy = "absent"
)

type PrunedTask added in v0.2.0

type PrunedTask struct {
	Item   Item
	TaskID task.ID
	Why    PruneWhy
}

PrunedTask pairs a withdrawn/declined task with the item text that doomed it and the rule that applied. For absent items the text is gone by definition; Item carries the repo and the dedup key instead.

type RedispatchFinding added in v0.3.3

type RedispatchFinding struct {
	Repo     string
	TaskID   task.ID
	Status   task.Status
	Attempts int
	ItemKey  string
	ItemText string
	Class    string
}

RedispatchFinding is one row-vs-queue disagreement the re-dispatch surface reports: a task whose TODO row is closed (ticked or gone) while the task is still live, or burned multiple attempts on closed work.

type RedispatchRefusalError added in v0.3.3

type RedispatchRefusalError struct {
	Reason string
}

RedispatchRefusalError is one refusal with its named signal, so callers can render the reason without parsing the sentinel out of the error string.

func (*RedispatchRefusalError) Error added in v0.3.3

func (r *RedispatchRefusalError) Error() string

func (*RedispatchRefusalError) Unwrap added in v0.3.3

func (r *RedispatchRefusalError) Unwrap() error

type RedispatchResult added in v0.3.3

type RedispatchResult struct {
	Repos        int
	Findings     []RedispatchFinding
	ScanFailures []ScanFailure
}

RedispatchResult summarizes one re-dispatch audit pass over all repos.

type RepriChange added in v0.3.0

type RepriChange struct {
	TaskID      task.ID        `json:"task_id"`
	ItemText    string         `json:"item_text"`
	OldPriority int            `json:"old_priority"`
	NewPriority int            `json:"new_priority"`
	Source      PrioritySource `json:"source"`
}

RepriChange records one would-be or applied priority change from a Reprioritize pass.

type ResolveInput added in v0.3.0

type ResolveInput struct {
	Text              string
	MarkerLevel       int
	HotPriority       int  // --same-session-priority; <= 0 disables hot
	FlatPriority      int  // --priority fallback when nothing else applies
	Importance        int  // repo importance, already defaulted to 50
	ImportanceEnabled bool // importance + keyword composite on
	AIScore           *int // cached AI score; nil = no verdict yet
}

ResolveInput carries every signal the precedence resolver considers. Text is the MARKER-STRIPPED item text; MarkerLevel is 0 (absent) or 1–4.

type Result

type Result struct {
	Repos    int
	Items    int
	Enqueued []Enqueued
	Skipped  []Skipped
}

Result summarizes one harvest run.

type ScanFailure added in v0.2.0

type ScanFailure struct {
	Repo   string
	Reason string
}

ScanFailure records one repo the audit could not scan.

type Skipped

type Skipped struct {
	Item   Item
	Reason string
}

Skipped records an item deliberately not enqueued, and why. Reasons are stable strings meant for humans and tests.

type WatchConfig added in v0.2.0

type WatchConfig struct {
	// Addr is the daemon endpoint, same forms as Config.DiscoveryAddr
	// (unix socket path, unix:// prefixed, or host:port).
	Addr string
	// ProjectsDir scopes triggers to repos under this directory; events for
	// projects elsewhere on the daemon never fire a tick. Ignored when
	// Repos is set.
	ProjectsDir string
	// Repos, when set, restricts triggers to exactly these repos (the
	// pool's --repos mode) and overrides the ProjectsDir scope.
	Repos []string
	// RepoIntervals debounces triggers per repo, keyed by repo dir base
	// name (the same key the Harvester paces enqueues by): after a trigger
	// for a repo, further watch events for it wait out the repo's gap.
	// Repos without an entry have no explicit gap — the coalescing trigger
	// channel and the harvester's own pacing still apply.
	RepoIntervals map[string]time.Duration
	// Log receives connection and drop warnings; nil discards.
	Log *slog.Logger
	// InitialBackoff is the first reconnect delay after a dropped stream;
	// it doubles up to MaxBackoff. Zero uses the default (500ms).
	InitialBackoff time.Duration
	// MaxBackoff caps the reconnect backoff ladder. Zero uses the default
	// (30s).
	MaxBackoff time.Duration
}

WatchConfig controls the watch-driven harvest trigger.

type Watcher added in v0.2.0

type Watcher struct {
	// contains filtered or unexported fields
}

Watcher subscribes to the daemon's GET /v1/watch SSE stream and converts project events into coalesced harvest-tick triggers. It is purely additive: a dead or missing watch stream never stalls the pool — the interval tick stays the fallback heartbeat, and the trigger channel simply stays silent.

func NewWatcher added in v0.2.0

func NewWatcher(cfg WatchConfig) *Watcher

NewWatcher creates a Watcher. Signals are delivered on a coalescing channel of capacity one: while a tick runs, further events collapse into at most one pending tick.

func (*Watcher) Run added in v0.2.0

func (w *Watcher) Run(ctx context.Context) error

Run subscribes to the watch stream and reconnects with exponential backoff until ctx is cancelled. It never returns an error for a dead daemon: each failed connect or dropped stream logs one warning and retries — degradation to interval-only harvesting, never a stalled pool.

func (*Watcher) Triggers added in v0.2.0

func (w *Watcher) Triggers() <-chan struct{}

Triggers yields harvest-tick signals. The channel is never closed; when the daemon is unreachable it simply stays quiet (the interval tick owns progress). A nil channel (watch off) blocks its select case forever.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL