suspension

package
v1.9.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 27, 2026 License: MIT Imports: 18 Imported by: 0

Documentation

Overview

Package suspension defines a paused run and everything needed to continue it.

Index

Constants

View Source
const MaxResumeInputBytes = 8 << 20

MaxResumeInputBytes bounds retained encoded input, including JSON escaping.

Variables

View Source
var (
	ErrResumeAdmissionUnavailable = errors.New("cortex: resume admission unavailable")
	ErrResumeAdmissionNotFound    = errors.New("cortex: resume admission not found")
	ErrResumeAdmissionConflict    = errors.New("cortex: resume admission conflict")
	ErrResumeAlreadyStarted       = errors.New("cortex: resume execution may have started")
)

Resume admission errors distinguish missing evidence from uncertain execution.

View Source
var ErrVersionClaimUnavailable = errors.New("cortex: suspension version claim unavailable")

ErrVersionClaimUnavailable means storage cannot atomically claim a pinned pause.

Functions

This section is empty.

Types

type ClaimVersion

type ClaimVersion struct {
	RunID        id.AgentRunID
	SuspensionID id.SuspensionID
	RunRevision  int64
}

ClaimVersion identifies the pause and run revision validated before resume. A later suspension of the same run must never satisfy this claim.

type Continuation

type Continuation struct {
	ToolResults   []verification.ToolResult `json:"tool_results,omitempty"`
	SchemaVersion int                       `json:"schema_version,omitempty"`
	Prepared      []byte                    `json:"prepared,omitempty"`
	Provenance    run.ExecutionProvenance   `json:"provenance,omitempty"`
	Usage         execution.Usage           `json:"usage,omitempty"`
	Messages      []llm.Message             `json:"messages"`
	SystemPrompt  string                    `json:"system_prompt"`
	StepIndex     int                       `json:"step_index"`
	TokensUsed    int                       `json:"tokens_used"`

	// NewMessagesFrom is the index in Messages where this run's own
	// messages begin. Everything before it is conversation history the
	// run loaded at startup and must never save again.
	//
	// Without it a resume has no way to tell the two apart, and its
	// first SaveConversation writes the whole history back as new rows.
	// That is not hypothetical: it shipped in v1.8.0, and agents went
	// deaf after about six runs because the duplicated rows filled the
	// fixed read window until it held nothing recent. The engine tracks
	// this boundary as newMessagesFrom in both react loops; this field
	// is the same number, carried across the pause.
	NewMessagesFrom int `json:"new_messages_from"`

	// SessionID is the session the run was loading from and saving to. A
	// resume cannot re-derive it: resolveSession lazily creates a
	// default session under a create race, so asking it again after a
	// pause can land on a different session than the one the messages
	// above came from.
	SessionID id.SessionID `json:"session_id,omitempty"`

	// Config is the resolved configuration the run was executing under.
	// See RunConfig: a resume builds its config from this and never
	// re-merges the agent's, so the run's own overrides survive the
	// pause.
	Config RunConfig `json:"config"`
}

Continuation is everything the loop needs to pick up where it stopped. It is stored in typed columns rather than an untyped metadata map, so a malformed continuation is a scan error at the boundary instead of a half-restored run several steps later.

SystemPrompt holds the ASSEMBLED string, not the sections it came from. That is deliberate and it matters once v1.12.0 turns prompt assembly into a pipeline over sections and overlays: a run suspended before an overlay changed, and resumed after, continues with the prompt it began with. Consistency within one run beats picking up a mid-run change to the agent's identity.

type PendingCall

type PendingCall struct {
	// ReplyWith binds an agent reply to the question created by this pause.
	// A tool-call ID alone may be reused by a later model response.
	ReplyWith string `json:"reply_with,omitempty"`
	ID        string `json:"id"`
	Name      string `json:"name"`
	// Arguments is the JSON-encoded argument object, verbatim from the
	// model's tool call.
	Arguments string `json:"arguments"`
}

PendingCall is one tool call awaiting a result from outside the engine.

Arguments is carried here rather than left to be recovered from the continuation's last assistant message. A caller resuming an external tool has to actually execute the call, and making it cross-reference the message history by call id to find out with what would be a needless puzzle for the one consumer this type exists to serve.

type ReplyDiscoveryStore

type ReplyDiscoveryStore interface {
	ListAgentReplyRuns(context.Context, id.AgentRunID, int) ([]id.AgentRunID, error)
}

ReplyDiscoveryStore lists only run identities with retained agent-reply suspensions or running resume admissions in the exact context scope. Results are ordered by run ID after the exclusive cursor. Limit must be between 1 and 256. Discovery includes expired and already-admitted suspensions, plus running admissions after their pause was deleted, so hosts can reconcile interrupted execution.

type ResumeAdmission

type ResumeAdmission struct {
	SchemaVersion    int                    `json:"schema_version"`
	Request          ResumeAdmissionRequest `json:"request"`
	RequestDigest    string                 `json:"request_digest"`
	SuspensionDigest string                 `json:"suspension_digest"`
	ClaimedRevision  int64                  `json:"claimed_revision"`
	AdmittedAt       time.Time              `json:"admitted_at"`
	Fresh            bool                   `json:"-"`
}

ResumeAdmission records the atomic pause claim and its retained input. Fresh describes the acknowledgment and never grants execution permission.

type ResumeAdmissionPage

type ResumeAdmissionPage struct {
	Admissions []ResumeAdmission
	Next       string
}

ResumeAdmissionPage is exact-scope history with an opaque bounded cursor.

type ResumeAdmissionRequest

type ResumeAdmissionRequest struct {
	Scope   cortex.Scope `json:"scope"`
	Version ClaimVersion `json:"version"`
	Input   []byte       `json:"input"`
}

ResumeAdmissionRequest binds the exact agent-reply input to a validated pause. Input is encoded bytes so storage cannot normalize its JSON representation.

type ResumeAdmissionStore

ResumeAdmissionStore separates claiming an agent-reply pause from permission to execute its continuation. Errors return no execution authority, including ambiguous commits. Host recovery must recheck current authority before Start.

type ResumeCloser

type ResumeCloser interface {
	Close(context.Context, ResumeClosureRequest) (ResumeClosure, error)
}

ResumeCloser terminates uncertain execution as incomplete under host guards. It neither retries work nor accepts successful results. Errors can follow a commit; retry identical contents to recover the original durable receipt.

func NewResumeCloser

func NewResumeCloser(storage ResumeClosureStore, fence ResumeClosureFence, authorizer ResumeClosureAuthorizer) (ResumeCloser, error)

NewResumeCloser requires explicit host proof authentication and fencing. Each operation is bounded to thirty seconds. Host ports must honor cancellation.

type ResumeClosure

type ResumeClosure struct {
	Request       ResumeClosureRequest `json:"request"`
	RequestDigest string               `json:"request_digest"`
	EventID       id.ID                `json:"event_id"`
	RunRevision   int64                `json:"run_revision"`
	ClosedAt      time.Time            `json:"closed_at"`
	Fresh         bool                 `json:"-"`
}

ResumeClosure is immutable evidence of an incomplete terminal transition. Its metadata event is committed in the same transaction as the run state.

func (ResumeClosure) Validate

func (r ResumeClosure) Validate() error

Validate checks receipt integrity without granting access or new execution.

type ResumeClosureAuthorizer

type ResumeClosureAuthorizer interface {
	AuthorizeResumeClosure(context.Context, ResumeClosureRequest) error
}

ResumeClosureAuthorizer authenticates the actor, proof issuer/revision and proof contents bound to the exact request, then checks current run/team access. Original worker identities must come from retained dispatch evidence. Current agent configuration cannot replace them. Recheck access and original proof on historical retries; a stored receipt or a hash is not permission.

type ResumeClosureFence

type ResumeClosureFence interface {
	WithResumeClosure(context.Context, ResumeClosureRequest, func(ResumeClosureRequest) error) error
}

ResumeClosureFence holds current run/team grants and permanent quiescence of every original worker and tool through one synchronous callback. Include remote workers, pending callbacks and delayed external actions. Stopping one process is sufficient only when the host proves it owned all possible execution.

type ResumeClosureRequest

type ResumeClosureRequest struct {
	SchemaVersion   int          `json:"schema_version"`
	Scope           cortex.Scope `json:"scope"`
	Version         ClaimVersion `json:"version"`
	AdmissionDigest string       `json:"admission_digest"`
	StartDigest     string       `json:"start_digest"`
	RunRevision     int64        `json:"run_revision"`
	Actor           string       `json:"actor"`
	Issuer          string       `json:"issuer"`
	IssuerRevision  string       `json:"issuer_revision"`
	ProofRef        string       `json:"proof_ref"`
	ProofDigest     string       `json:"proof_digest"`
	ObservedAt      time.Time    `json:"observed_at"`
	ExpiresAt       time.Time    `json:"expires_at"`
}

ResumeClosureRequest closes an uncertain start as incomplete. The host must authenticate permanent worker and tool quiescence, including delayed actions, for this exact admission, start and current run revision. A deadline, a stopped local process or a digest alone does not establish that external work stopped. Closure never accepts output, refunds usage, clears effects or permits replay.

func (ResumeClosureRequest) Freeze

Freeze checks structure and detaches the request. It does not authenticate proof.

type ResumeClosureStore

type ResumeClosureStore interface {
	CloseResume(context.Context, ResumeClosureRequest) (ResumeClosure, error)
	GetResumeClosure(context.Context, id.AgentRunID, id.SuspensionID) (ResumeClosure, error)
}

ResumeClosureStore requires current host proof and authority before writes. Closure serializes with run writers, binds the latest admission and start, and rejects changed requests. Retry the identical request after a lost ack. Historical retries return the original receipt even after proof expiry.

type ResumeExpiry

type ResumeExpiry struct {
	SchemaVersion int          `json:"schema_version"`
	Version       ClaimVersion `json:"version"`
	Scope         cortex.Scope `json:"scope"`
	RequestDigest string       `json:"request_digest"`
	EventID       id.ID        `json:"event_id"`
	RunRevision   int64        `json:"run_revision"`
	ExpiredAt     time.Time    `json:"expired_at"`
	Fresh         bool         `json:"-"`
}

ResumeExpiry proves an admitted continuation expired before execution started. It grants no execution or retry authority and does not release reservations.

type ResumeRecoveryStore

type ResumeRecoveryStore interface {
	LatestResumeAdmission(context.Context, id.AgentRunID) (ResumeAdmission, error)
	ExpireResumeAdmission(context.Context, id.AgentRunID, id.SuspensionID, string) (ResumeExpiry, error)
	GetResumeExpiry(context.Context, id.AgentRunID, id.SuspensionID) (ResumeExpiry, error)
}

ResumeRecoveryStore retains exact-scope recovery evidence independently of the live pause. LatestResumeAdmission orders by suspension ID. Expiry and execution start must share the run lock; an existing start refuses expiry. Hosts check current authority before every operation.

type ResumeStart

type ResumeStart struct {
	SchemaVersion int          `json:"schema_version"`
	Version       ClaimVersion `json:"version"`
	Scope         cortex.Scope `json:"scope"`
	RequestDigest string       `json:"request_digest"`
	StartedAt     time.Time    `json:"started_at"`
	Fresh         bool         `json:"-"`
	Suspension    *Suspension  `json:"-"`
}

ResumeStart records a single execution admission. Only a fresh acknowledged start includes a suspension. Observing or retrying a start never grants it.

func (ResumeStart) Digest

func (s ResumeStart) Digest() (string, error)

Digest binds a retained start without its transient execution authority.

type RunConfig

type RunConfig struct {
	Context            *cognitive.ContextPolicy `json:"context,omitempty"`
	Publication        safety.Policy            `json:"publication"`
	Budget             execution.Budget         `json:"budget,omitempty"`
	AllToolsRestricted bool                     `json:"all_tools_restricted,omitempty"`
	Model              string                   `json:"model,omitempty"`
	Temperature        *float64                 `json:"temperature,omitempty"`
	MaxSteps           int                      `json:"max_steps,omitempty"`
	MaxTokens          int                      `json:"max_tokens,omitempty"`
	ReasoningLoop      string                   `json:"reasoning_loop,omitempty"`
	Tools              []string                 `json:"tools,omitempty"`
	PersonaRef         string                   `json:"persona_ref,omitempty"`

	// ToolsRestricted says Tools is an explicit allowlist even when it
	// is empty, which is what an overlay that withdrew the agent's last
	// tool produces. It travels with the continuation because a resumed
	// run rebuilds its config from here and from nowhere else: without
	// it, an empty list would read as "every registered tool" and the
	// resume would hand back exactly what the overlay took away.
	//
	// It is absent from rows written before this field existed, and
	// decodes there as false, which is the behavior those runs had.
	// The flag can therefore be LOST across an upgrade, but it can
	// never be spuriously SET. That is the safe direction here: losing
	// it resumes an old run under the tool rules it was suspended
	// under, where inventing it would strip tools from a run nobody
	// restricted.
	ToolsRestricted bool `json:"tools_restricted,omitempty"`
}

RunConfig is the resolved configuration a run was executing under: the merge of engine defaults, agent config and that run's overrides, already flattened.

It is stored for the same reason SystemPrompt is stored assembled rather than as the sections it came from. A run's overrides live only in the call that started it, so a resume that rebuilt the config from the agent would silently drop them: a run deliberately started with a narrowed Tools list would come back with the agent's full tool set, which is authority widening across a pause, and a run started with a raised MaxSteps would come back on the agent's smaller budget and refuse to continue at all.

Storing the resolved values also pins what the run was actually executing, so an agent-config edit landing during the pause cannot shift a conversation already in flight.

type Store

type Store interface {
	// CreateSuspension writes a new suspension row for a run that just
	// paused.
	CreateSuspension(ctx context.Context, s *Suspension) error

	// GetSuspension reads the suspension for a run without changing
	// anything. Callers that intend to resume the run must use
	// ClaimSuspension instead: a Get followed by a separate write is the
	// race ClaimSuspension exists to close.
	GetSuspension(ctx context.Context, runID id.AgentRunID) (*Suspension, error)

	// ClaimSuspension is the concurrency primitive for resume.
	// Implementations must perform the run's paused-to-running
	// transition and the suspension read as ONE atomic operation, not a
	// read followed by a separate write: that gap is exactly the race
	// this method exists to close, since two concurrent resume calls
	// must not both observe a paused run and proceed. The caller that
	// loses the race gets cortex.ErrNotSuspended, not the suspension it
	// lost the race for.
	ClaimSuspension(ctx context.Context, runID id.AgentRunID) (*Suspension, error)

	// DeleteSuspension removes a suspension row. Callers use this once a
	// resume has fully completed, or as part of the expiry sweep once an
	// expired suspension has been handled.
	DeleteSuspension(ctx context.Context, runID id.AgentRunID) error

	// ListExpired returns suspensions whose ExpiresAt is at or before
	// now, bounded to at most limit rows, within the caller's scope.
	ListExpired(ctx context.Context, now time.Time, limit int) ([]*Suspension, error)

	// ListExpiredAcrossScopes is ListExpired with the scope filter
	// DELIBERATELY BYPASSED: it returns expired suspensions from every
	// scope in the database, and it is the one read in this interface
	// that does.
	//
	// It exists for the engine's expiry sweeper and nothing else. A
	// sweeper is process infrastructure started by Engine.Start, so
	// there is no request scope on its context and no way to enumerate
	// the scopes that have rows; ListExpired would sweep whichever scope
	// the sweeper's context happened to carry, which in practice is
	// none. Hence a second method rather than a flag or a sentinel
	// scope: a cross-scope read has to be visible as one at the call
	// site, and no handler can reach this by passing a zero value.
	//
	// Crossing scopes to FIND the work does not license doing the work
	// unscoped. Callers must rebind their context to each returned
	// suspension's own Scope before touching that run, exactly as Resume
	// does.
	ListExpiredAcrossScopes(ctx context.Context, now time.Time, limit int) ([]*Suspension, error)

	// ClaimExpiredSuspension is ClaimSuspension's mirror for the expiry
	// sweeper: the same atomic paused-to-running transition, conditioned
	// on the suspension being PAST its deadline rather than short of it.
	//
	// The two predicates partition the same row, which is what makes a
	// resume and a sweep exclusive rather than merely unlikely to
	// collide. A resume that beats the deadline claims the run and the
	// sweeper can no longer take it; a sweep that finds the deadline
	// passed takes the run and no resume can claim it afterwards. Both
	// answer against the same stored ExpiresAt, so there is no window
	// where the two agree the row is theirs.
	//
	// Zero rows matched means somebody else got there first, and it
	// returns cortex.ErrNotSuspended. The sweeper skips such a run
	// rather than failing it: it is either gone, already resumed, or no
	// longer paused.
	ClaimExpiredSuspension(ctx context.Context, runID id.AgentRunID, now time.Time) (*Suspension, error)
}

Store persists suspensions and provides the concurrency primitive that resuming a run depends on.

type SuspendReason

type SuspendReason string

SuspendReason says why a run stopped. Both reasons share one mechanism because they are the same problem: the loop cannot proceed until something outside it returns.

const (
	// ReasonApproval means a human must grant or deny before the run
	// continues.
	ReasonApproval SuspendReason = "approval"
	// ReasonExternalTool means the caller, not the engine, executes the
	// pending tool call and reports the result back.
	ReasonExternalTool SuspendReason = "external_tool"
	// ReasonAgentReply means the run is waiting on another agent's answer.
	//
	// It is not ReasonExternalTool even though both wait on something
	// outside the loop, because the two say different things about who
	// acts next. External says the CALLER executes the call and reports
	// back. Agent-reply says cortex itself is waiting on a peer, and a
	// caller answering it would be forging a message that peer never
	// sent. The messaging bus resumes these, off its own correlation
	// ledger; nobody else may.
	ReasonAgentReply SuspendReason = "agent_reply"
)

type Suspension

type Suspension struct {
	// ClaimedRun is the snapshot returned by an atomic claim, including its
	// new revision. It is transient and is never saved as suspension content.
	ClaimedRun *run.Run `json:"-"`
	cortex.Entity
	ID    id.SuspensionID `json:"id"`
	RunID id.AgentRunID   `json:"run_id"`

	// Scope is stored so Resume re-derives authorization from the scope
	// the run STARTED under, rather than trusting the scope of whoever
	// happens to call Resume. A resume arriving with different authority
	// must not quietly continue with it.
	Scope cortex.Scope `json:"scope"`

	Reason    SuspendReason `json:"reason"`
	Pending   []PendingCall `json:"pending,omitempty"`
	Cont      Continuation  `json:"continuation"`
	ExpiresAt *time.Time    `json:"expires_at,omitempty"`
}

Suspension is a paused run.

type VersionedStore

type VersionedStore interface {
	ClaimSuspensionVersion(context.Context, ClaimVersion) (*Suspension, error)
}

VersionedStore fences the suspension identity and run revision in the same transaction as the paused-to-running transition. Failed or ambiguous commits return no suspension. Retrying cannot claim a later pause in the same run. This primitive does not retain a resume payload or grant recovery authority.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL