Documentation
¶
Overview ¶
Package taskrecover repairs tasks left in_progress by a run that died.
A run process can disappear between the moment its agent finishes speaking and the moment that outcome reaches the database. The window is small but it is not rare: the control plane is killed and the orphaned run inherits a closed stdout pipe (Go makes SIGPIPE fatal on fd 1 and 2, so the next status line is lethal), the host OOMs, the machine reboots. Whatever the cause, the process is gone and the row still says in_progress.
Both recovery paths used to answer that the same way: reset the task to pending and run it again. That is right when the agent never finished, and destructive when it did. Task 63 of the liebid project is the case that motivated this package — an hour of work, a full result, `TASK_DONE` on the last line, and the only surviving copy sitting in the live artifact while the plan prepared to do all of it a second time. Re-running a task whose work is already committed is not a neutral retry; it is an agent rediscovering its own finished changes and deciding what to do about them.
Why the live artifact is trustworthy evidence ¶
The orchestrator streams every token into .cloop/artifacts/<id>_output.txt as it arrives, so the file is complete the instant the agent stops talking — strictly before the bookkeeping this package exists to replace. It is opened O_TRUNC at the start of each execution, so it always describes the newest attempt and never an older one.
Adoption then applies exactly the contract the live path applies: pm.CheckTaskSignal over the last five lines. This package cannot invent a completion the orchestrator would not itself have recognised, because it is asking the same function the same question about the same bytes. What it adds is one guard the live path does not need — that the artifact is no older than the execution being recovered — so a leftover file from a previous attempt can never be read as this attempt's result.
The rule ¶
The orchestrator's verdict first (verdict.go): where it had decided the execution's outcome before the run ended, that decision stands, whatever the agent said — a review gate's rejection, a failed verification, abandoned background work. Where it had not, adopt the agent's terminal signal, unless the outcome was still waiting on the review gate and the agent claims done; re-queue everything else. Silence is not success: an artifact that stops mid-sentence means the agent was interrupted, and that task genuinely has to run again.
Callers must guarantee no process owns the project directory. Reconcile is a repair for the dead, and running it against a live run would reset a task out from under the process executing it.
Index ¶
- Constants
- Variables
- func ClearVerdict(workDir string, taskID int) error
- func EventFor(oc Outcome) (row state.EventRow, details map[string]any, ok bool)
- func LogOutcome(workDir string, oc Outcome)
- func VerdictPath(workDir string, taskID int) string
- func WriteVerdict(workDir string, v Verdict) error
- type Action
- type Outcome
- type PruneResult
- type Stop
- type Verdict
Constants ¶
const ( // SourceAgent: the agent's own signal, accepted as it stood. SourceAgent = "agent" // SourceReviewGate: the review gate (Task 20357) — a block, a publish // that failed, an approval, or a review still pending. SourceReviewGate = "review_gate" // SourceVerify and SourceScriptVerify: --verify and --script-verify. SourceVerify = "verify" SourceScriptVerify = "script_verify" // SourceBackground: work the agent left running (Task 20205). SourceBackground = "background" // SourceClarification: the agent asked questions instead of working. SourceClarification = "clarification" // SourceAbort: the execution produced no finished work — an unfinished // turn, a refusal, empty output. The task goes back to pending. SourceAbort = "abort" // SourceInterrupted: the run stopped under the task (Task 20348). SourceInterrupted = "interrupted" // SourceBudget: the token budget or cost limit ran out after the task. SourceBudget = "budget" // SourceTimeout: the task's time budget expired. SourceTimeout = "timeout" // SourceProvider: the provider failed the call. SourceProvider = "provider_error" // SourceOperator: a person chose the outcome. SourceOperator = "operator" // SourceFailover: the hub's executor failover (Task 20391) — the run's // node went unreachable after executors.failover.max_attempts was used // up, or the task is a suspected node killer. SourceFailover = "failover" // SourceOrchestrator: anything else the orchestrator decides on its own — // a risk check, a pre-task hook, a dry run, an implicit completion. SourceOrchestrator = "orchestrator" )
Sources name what decided a task's outcome. They are recorded in the verdict and repeated in the recovered task's annotation and journal row, so an operator can tell "the review gate failed this" from "the agent gave up".
const ReasonReviewPending = "review_pending"
ReasonReviewPending marks the verdict written while a task's outcome waits on the review gate: from the start of a gated execution until the gate decides. Such a verdict decides nothing itself, but it stops recovery from adopting a TASK_DONE the reviewer never approved — the work behind it was never published, since held pushes die with the run.
const VerdictFormat = 1
VerdictFormat is the version of the sidecar this build writes. Readers accept any format at or above 1 and read the fields they know: every field but the status is optional, so a later format that adds one is still a valid format-1 document.
Variables ¶
var ErrNoVerdict = errors.New("taskrecover: no verdict")
ErrNoVerdict reports that a task has no verdict sidecar.
Functions ¶
func ClearVerdict ¶ added in v0.0.5
ClearVerdict removes taskID's sidecar. A sidecar that is not there is not an error.
func EventFor ¶ added in v0.0.5
EventFor renders one reconciliation as the journal row LogOutcome writes, with its structured details — nil when the recovery followed the agent's report rather than a verdict. ok is false for an outcome that is not journalled. A remote device uses it to send the row back with the run's own (pkg/executor/projectseed), so a recovery reads the same on the hub whether it happened there or on the device.
func LogOutcome ¶
LogOutcome appends one reconciliation to the project's event journal.
An adopted task is recorded as its real outcome — task_done, task_failed, task_skipped — rather than as some recovery-specific event type, because that is what it is: the completion the dying run failed to write. Recording it as anything else would leave the timeline showing a task that never finished. The message carries the provenance so nobody reads it as work this run performed, and the timestamp is when the agent actually stopped, not when we got around to noticing.
Best-effort, like every other event write: observability must never be the reason a repair fails.
func VerdictPath ¶ added in v0.0.5
VerdictPath is where a task's verdict sidecar lives: next to its live artifact.
func WriteVerdict ¶ added in v0.0.5
WriteVerdict writes v as taskID's sidecar, atomically. WrittenAt is set to now unless the caller set it; every free-text field is bounded.
Types ¶
type Action ¶
type Action string
Action says what reconciliation did with a task.
const ( // ActionAdopted means the agent had finished and its outcome was taken // from the live artifact instead of being thrown away. ActionAdopted Action = "adopted" // ActionRequeued means no terminal signal was found, so the task was // reset to pending to be executed again. ActionRequeued Action = "requeued" )
type Outcome ¶
type Outcome struct {
TaskID int
Title string
Action Action
// Status is the status the task now has: the adopted signal, or pending.
Status pm.TaskStatus
// CompletedAt is when the agent actually stopped writing, for adopted
// tasks only. Taken from the artifact's mtime rather than from the clock,
// because reconciliation may happen days after the fact and "completed
// when we noticed" would be a lie in the timeline.
CompletedAt time.Time
// Reason explains a re-queue in words a UI can show.
Reason string
// Verdict is the orchestrator's decision the outcome followed (Task
// 20365), when one was found for this execution; nil when the agent's own
// report decided it.
Verdict *Verdict
// Stop is set on a re-queue caused by a stop from outside the run (Task
// 20405).
Stop *pm.TaskStop
// contains filtered or unexported fields
}
Outcome is what happened to one recovered task. Callers use it to log, broadcast, and decide whether the plan changed enough to save.
func Reconcile ¶
Reconcile repairs every task in the plan left in_progress by a run that no longer exists, and returns one Outcome per task it touched.
It mutates the plan in place and never persists: the caller owns the state handle and decides when to save, which keeps this package free of pkg/state and testable with nothing but a temp dir.
The caller must have established that no process is executing this project.
func ReconcileStopped ¶ added in v0.0.5
ReconcileStopped is Reconcile for a run stopped from outside. What the orchestrator decided and what the agent finished stand exactly as Reconcile has them; a task the run had not finished goes back to pending for stop's reason, noted by cloop rather than inferred from its transcript. A nil stop is Reconcile.
func SettleFromVerdicts ¶ added in v0.0.5
SettleFromVerdicts applies the fresh verdicts in workDir to the tasks plan left in progress, and leaves every other in-progress task as it is. It reads the verdict sidecars and nothing else, and writes nothing: it mutates plan in place and returns one Outcome per task it settled.
It is the half of Reconcile that is safe on a remote device. Once a seeded run has exited, the device reads back what it left (pkg/executor/projectseed) and runs this first, so a run that died — or stopped on a failed write — after the orchestrator decided an outcome reports that decision rather than a task still in progress. What the orchestrator had not decided stays in progress, for the hub's dead-run recovery to settle: the hub cannot read the device's live artifact, so it re-queues.
A sidecar that is present but unusable is noted on the task, which stays in progress. The caller must guarantee the run has exited.
type PruneResult ¶ added in v0.0.5
type PruneResult struct {
Deleted int
Bytes int64
// Kept counts sidecars left in place: their task is in progress, or they
// are newer than the cutoff.
Kept int
}
PruneResult is what PruneVerdicts removed, or would have.
func PruneVerdicts ¶ added in v0.0.5
func PruneVerdicts(workDir string, inProgress func(taskID int) bool, cutoff time.Time, dryRun bool) (PruneResult, error)
PruneVerdicts removes the sidecars nothing will read again: those older than cutoff whose task is not in progress, including tasks no longer in the plan.
A sidecar is read only to recover a task left in progress, so one whose task is in any other state is redundant — the outcome it decided is stored. The cutoff keeps the pass away from a run that is writing one right now: a task that a concurrent run has just started would read as not in progress to a caller holding an older plan.
inProgress reports whether a task is in progress. Nothing outside .cloop/artifacts is touched, and a symbolic link there is never followed.
type Stop ¶ added in v0.0.5
type Stop struct {
// Reason is the clause a re-queued task's note and journal row give in
// place of "the agent was interrupted before it reported an outcome".
Reason string
// Task is set on each task re-queued for it, for the save that records
// the re-queue: its task.finish row carries the cause as fields.
Task pm.TaskStop
}
Stop names what ended a run from outside it, when the party settling the run knows (Task 20405) — its executor stopped it at its disk limit.
type Verdict ¶ added in v0.0.5
type Verdict struct {
// Format is the document version; see VerdictFormat.
Format int `json:"format"`
// TaskID names the task. A sidecar whose ID does not match its file name
// is refused as corrupt.
TaskID int `json:"task_id"`
// Status is what was decided: done, failed, skipped or timed_out is the
// outcome; pending means the task runs again; in_progress means nothing
// was decided yet (see ReasonReviewPending).
Status pm.TaskStatus `json:"status"`
// Reason is a machine-readable code for why — "review_blocked",
// "background_abandoned", "verify_failed", an abort class.
Reason string `json:"reason,omitempty"`
// Source is what decided; one of the Source constants.
Source string `json:"source,omitempty"`
// Detail is one sentence for the operator.
Detail string `json:"detail,omitempty"`
// Summary is the task's result summary when the decision was made, so a
// recovery that cannot read the live artifact still has one.
Summary string `json:"summary,omitempty"`
// Diagnosis is the task's failure diagnosis when the decision was made:
// the reviewer's findings, the background work that never finished.
Diagnosis string `json:"diagnosis,omitempty"`
// Review is the review gate's record, when the gate took part.
Review *pm.TaskReview `json:"review,omitempty"`
// Background is the background-work record the decision was made on.
Background *pm.BackgroundWork `json:"background,omitempty"`
// StartedAt is the start of the execution this verdict decided.
StartedAt *time.Time `json:"started_at,omitempty"`
// WrittenAt is when the decision was written down.
WrittenAt time.Time `json:"written_at"`
}
Verdict is the orchestrator's decision about one execution of one task.
func ReadVerdict ¶ added in v0.0.5
ReadVerdict reads taskID's sidecar. It returns ErrNoVerdict when there is none, and any other error when there is one that cannot be used — which the caller reports, because a sidecar that exists says a decision was made.
Symbolic links are refused, in the file and in the directory holding it. On a remote device the reader is the agent, which may be able to read files the workload that wrote this directory cannot, and a link is how the workload would ask it to.
func (*Verdict) AwaitingReview ¶ added in v0.0.5
AwaitingReview reports whether the verdict records an execution whose outcome was waiting on the review gate.
func (*Verdict) Decided ¶ added in v0.0.5
Decided reports whether the verdict settles the task: an outcome, or a return to pending. A verdict that is still waiting on something does not.
Source Files