Documentation
¶
Overview ¶
Package runs implements per-run worker templates and the run supervisor behind `claw runs serve` (issue #372).
A template is compiled by `claw up` from a pod service declared with x-claw.lifecycle: per-run. The supervisor instantiates it once per run as a separate Compose project; Compose remains the only lifecycle writer.
Index ¶
- Constants
- Variables
- func CheckSocketDir(dir string, uid uint32) error
- func EnsurePrivateDir(path string) error
- func HasLiveRuns(podDir string) (bool, error)
- func ListTemplates(podDir string) ([]string, error)
- func LoadTemplate(podDir, name string) (*Template, *ContextTemplate, error)
- func PodNamespace(podDir string) string
- func RunPath(k Key, suffix string) string
- func TemplateDir(podDir string) string
- func WriteTemplates(podDir string, templates []*Template, contexts map[string]*ContextTemplate) error
- type APIError
- type Callbacks
- type Client
- type CllamaRevoker
- type Config
- type Container
- type ContextTemplate
- type DockerExecutor
- func (d *DockerExecutor) Close() error
- func (d *DockerExecutor) Containers(ctx context.Context, keyHash string) ([]Container, error)
- func (d *DockerExecutor) Down(ctx context.Context, project, file string) error
- func (d *DockerExecutor) RunWorkers(ctx context.Context, podNS string) ([]Container, error)
- func (d *DockerExecutor) Stdout(ctx context.Context, containerID string, limit int) ([]byte, bool, error)
- func (d *DockerExecutor) Up(ctx context.Context, project, file string, env map[string]string) error
- type Executor
- type ExitInfo
- type HTTPNotifier
- type Key
- type LaunchRequest
- type Ledger
- func (l *Ledger) All() ([]*Record, error)
- func (l *Ledger) DeleteSecret(k Key, kind string) error
- func (l *Ledger) Get(k Key) (*Record, error)
- func (l *Ledger) Put(rec *Record) error
- func (l *Ledger) PutSecret(k Key, kind, value string) error
- func (l *Ledger) PutTombstone(k Key, generation uint64) error
- func (l *Ledger) Secret(k Key, kind string) (string, bool, error)
- func (l *Ledger) Tombstone(k Key) (uint64, bool, error)
- type Limits
- type Notifier
- type PodLock
- type Record
- type ResultInfo
- type Revoker
- type Server
- type Spec
- type State
- type StatusView
- type Supervisor
- func (s *Supervisor) Advance(ctx context.Context, k Key)
- func (s *Supervisor) AdvanceAll(ctx context.Context)
- func (s *Supervisor) Cancel(ctx context.Context, k Key, generation uint64) (*Record, error)
- func (s *Supervisor) CheckRevocationSupport(ctx context.Context) error
- func (s *Supervisor) FinalizeWithoutProxy(ctx context.Context, k Key, proxyGone func(context.Context) (bool, error)) error
- func (s *Supervisor) Get(k Key) (*Record, error)
- func (s *Supervisor) Launch(ctx context.Context, req LaunchRequest) (*Record, bool, error)
- func (s *Supervisor) LiveKeys() ([]Key, error)
- func (s *Supervisor) Orphans(ctx context.Context) ([]Container, error)
- func (s *Supervisor) ReadCaptureEpoch() (string, error)
- func (s *Supervisor) RemoveOrphan(ctx context.Context, c Container) error
- func (s *Supervisor) ReplaceCredential(ctx context.Context, k Key, generation uint64, credential string, ...) (*Record, error)
- func (s *Supervisor) Result(k Key) ([]byte, *Record, error)
- func (s *Supervisor) Run(ctx context.Context, interval time.Duration)
- func (s *Supervisor) Wait(ctx context.Context, k Key)
- type Template
- type Timestamps
- type WorkerLister
Constants ¶
const ( ReasonExitNonZero = "exit_nonzero" ReasonTimeout = "timeout" ReasonAuthorityExpired = "authority_expired" ReasonSpawnError = "spawn_error" ReasonContainerMissing = "container_missing" ReasonDuplicate = "duplicate" ReasonMismatch = "mismatch" ReasonResultMissing = "result_missing" )
Failure reasons.
const ( // ProofCllamaAck: cllama durably revoked the identity and drained it. ProofCllamaAck = "cllama_ack" // ProofProxyStopped: the whole pod (proxy included) was removed and the // run's context deleted, so no process can hold or regain its authority. ProofProxyStopped = "proxy_stopped" )
Revocation proofs. A run is revoked only with one of these.
const ( CredentialNone = "none" CredentialHeld = "held" CredentialMissing = "missing" )
Credential custody states reported to the orchestrator.
const ( CallbackAcked = "acked" CallbackRejected = "rejected" // 401/403: the orchestrator's authority ended CallbackAbandoned = "abandoned" // undeliverable before authority expired CallbackSkipped = "skipped" // no credential held, or the run was cancelled // End aliases kept for readability at end-callback call sites. EndAcked = CallbackAcked EndRejected = CallbackRejected EndAbandoned = CallbackAbandoned EndSkipped = CallbackSkipped )
Callback outcomes. Only CallbackAcked means the orchestrator acknowledged.
const ( LabelPod = "claw.pod" LabelRole = "claw.role" LabelRunKey = "claw.run.key" LabelInstall = "claw.run.installation" LabelRunID = "claw.run.id" LabelGeneration = "claw.run.generation" LabelTemplate = "claw.run.template" LabelTemplateID = "claw.run.template-id" LabelDigest = "claw.run.spec-digest" LabelPodNS = "claw.run.pod-ns" RoleRunWorker = "run-worker" )
Worker container labels. The key hash is the reconciliation handle; the others let reconcile verify a container against its ledger record.
const ( // StateDirName is the pod-relative directory holding templates, the ledger, // rendered projects and the pod lock. claw up never rotates it. StateDirName = ".claw-runs" // RevocationDirName is the pod-relative proxy-owned revocation ledger dir. RevocationDirName = ".claw-revocations" )
const ( // DefaultMaxRuntime applies only if a template carries no runtime limit. DefaultMaxRuntime = 30 * time.Minute )
Advancement bounds. Every external operation inherits a deadline.
const ProtocolVersion = 1
ProtocolVersion is the run supervisor wire protocol version.
const (
SocketName = "claw-runs.sock"
)
Variables ¶
var ErrAuthority = errors.New("orchestrator authority rejected")
ErrAuthority means the orchestrator rejected the held credential (401/403).
var ErrPodLocked = errors.New("pod run lock is held by another process")
ErrPodLocked means another process (normally `claw runs serve`) holds the pod's run lock.
Functions ¶
func CheckSocketDir ¶
CheckSocketDir verifies the socket directory is a real directory (not a symlink) owned by uid with mode 0700 or 0750. Any ancestor writable by group or others must carry the sticky bit.
func EnsurePrivateDir ¶
EnsurePrivateDir creates path as a 0700 directory, or verifies an existing one is a real directory (not a symlink) owned by the current user, and tightens its mode to 0700. Durable fences (the run ledger, cllama's revocation ledger) must not be writable by other host users.
func HasLiveRuns ¶
HasLiveRuns reports whether any record is non-terminal. A missing ledger has no live runs.
func ListTemplates ¶
ListTemplates returns the names of compiled templates.
func LoadTemplate ¶
func LoadTemplate(podDir, name string) (*Template, *ContextTemplate, error)
LoadTemplate reads and verifies one compiled template.
func PodNamespace ¶
PodNamespace is the server-owned identity of a pod on this host, derived from its absolute directory. Two pods with the same name (or the same run keys) never share a Compose project, label value or worker network.
func TemplateDir ¶
TemplateDir returns the pod's template directory.
func WriteTemplates ¶
func WriteTemplates(podDir string, templates []*Template, contexts map[string]*ContextTemplate) error
WriteTemplates replaces the pod's compiled templates atomically. Callers must hold the pod lock.
Types ¶
type APIError ¶
type APIError struct {
Status int `json:"-"`
Code string `json:"code"`
Message string `json:"message"`
}
APIError is a client-visible error. Messages never carry secrets or spec content.
type Callbacks ¶
type Callbacks struct {
Start string `json:"start,omitempty"`
End string `json:"end,omitempty"`
LastHeartbeat *time.Time `json:"last_heartbeat,omitempty"`
}
Callbacks records lifecycle callback delivery. Start is always resolved before End for a worker that spawned.
type Client ¶
type Client struct {
// contains filtered or unexported fields
}
Client talks to a run supervisor. It authenticates the server in two ways: the socket and its directory must be owned by the expected UID and not writable by others, and the peer credential of every connection must be that UID (path checks alone race with rename).
type CllamaRevoker ¶
CllamaRevoker calls cllama's operator revoke endpoint (cllama#54) on the published UI listener with the pod's CLLAMA_UI_TOKEN.
type Config ¶
type Config struct {
PodDir string
Executor Executor
Revoker Revoker
Notifier Notifier // nil disables credential custody
HeartbeatInterval time.Duration
Now func() time.Time
Logf func(format string, args ...any)
}
Config configures a Supervisor.
type Container ¶
type Container struct {
ID string
Running bool
Exited bool
ExitCode int
ImageID string
Labels map[string]string
}
Container is the read-only view of a worker container.
type ContextTemplate ¶
type ContextTemplate struct {
Input cllama.AgentContextInput `json:"input"`
}
ContextTemplate is the cllama context compiled for a template. It carries managed tool credentials, so it is stored 0600 and never mounted into a worker. Each run receives a copy with its own identity and allowlist.
type DockerExecutor ¶
type DockerExecutor struct {
Logf func(format string, args ...any)
// Hold, when set, is the pod lock file. Every lifecycle subprocess
// inherits it, so a SIGKILLed supervisor's in-flight compose command keeps
// writer authority until it exits.
Hold *os.File
// contains filtered or unexported fields
}
DockerExecutor runs `docker compose` for lifecycle (ADR-002) and uses the Docker SDK for reads only.
func NewDockerExecutor ¶
func NewDockerExecutor() (*DockerExecutor, error)
func (*DockerExecutor) Close ¶
func (d *DockerExecutor) Close() error
func (*DockerExecutor) Containers ¶
Containers lists every container, running or not, labelled with the key.
func (*DockerExecutor) Down ¶
func (d *DockerExecutor) Down(ctx context.Context, project, file string) error
Down removes a run's project and orphans. It never relies on ambient Compose discovery: with the recorded file it uses that file; if the file is gone it passes a minimal empty project on stdin, which removes only the deterministic project's containers. The project directory is pinned so no stray .env file is read.
func (*DockerExecutor) RunWorkers ¶
RunWorkers lists every run-worker container labelled for this pod's namespace.
func (*DockerExecutor) Stdout ¶
func (d *DockerExecutor) Stdout(ctx context.Context, containerID string, limit int) ([]byte, bool, error)
Stdout returns at most limit bytes of the container's stdout, and whether the stream held more.
type Executor ¶
type Executor interface {
Up(ctx context.Context, project, file string, env map[string]string) error
Down(ctx context.Context, project, file string) error
Containers(ctx context.Context, keyHash string) ([]Container, error)
// Stdout returns at most limit bytes of stdout, and whether more existed.
Stdout(ctx context.Context, containerID string, limit int) ([]byte, bool, error)
}
Executor performs container lifecycle through Docker Compose and reads container state through the Docker SDK. It never receives the orchestrator credential.
type HTTPNotifier ¶
HTTPNotifier delivers the Clawdapus run lifecycle callback contract:
POST {origin}{path}/start {installation, run_id, generation}
POST {origin}{path}/heartbeat {installation, run_id, generation}
POST {origin}{path}/end {installation, run_id, generation, outcome, reason, summary}
path comes from supervisor configuration with "{run_id}" substituted. The origin is fixed by configuration, never by the caller, redirects are not followed, and the held credential is sent only as a bearer header.
func NewHTTPNotifier ¶
func NewHTTPNotifier(origin, pathTemplate string) (*HTTPNotifier, error)
NewHTTPNotifier validates the configured origin.
type Key ¶
Key identifies a run.
func (Key) Hash ¶
Hash is the stable, label-safe digest of a key. Project names, file names and container labels derive from it, never from caller strings directly.
type LaunchRequest ¶
type LaunchRequest struct {
Key Key
Generation uint64
SpecDigest string
Spec json.RawMessage
Credential string
AuthorityExpiresAt time.Time
}
LaunchRequest is a validated-at-the-edge launch.
type Ledger ¶
type Ledger struct {
// contains filtered or unexported fields
}
Ledger is the supervisor's durable store: one record file per run, plus separate 0600 secret files that are never mounted anywhere.
func OpenLedger ¶
OpenLedger opens (creating if absent) the ledger under podDir.
func (*Ledger) DeleteSecret ¶
DeleteSecret removes a stored secret.
func (*Ledger) PutTombstone ¶
PutTombstone durably records a cancel. It never lowers a recorded generation. Tombstones are written without the run's key lock so a cancel is never delayed behind a slow lifecycle operation.
type Limits ¶
type Limits struct {
MaxRuntimeSeconds int64 `json:"max_runtime_seconds"`
MaxAuthoritySeconds int64 `json:"max_authority_seconds"`
MaxTaskBytes int `json:"max_task_bytes"`
MaxResultBytes int `json:"max_result_bytes"`
Memory string `json:"memory"`
Pids int `json:"pids"`
}
Limits bound every run instantiated from a template.
func (Limits) MaxAuthority ¶
func (Limits) MaxRuntime ¶
type Notifier ¶
type Notifier interface {
Started(ctx context.Context, rec *Record, credential string) error
Heartbeat(ctx context.Context, rec *Record, credential string) error
Ended(ctx context.Context, rec *Record, credential string, outcome, reason string, summary []byte) error
}
Notifier delivers lifecycle callbacks with the held orchestrator credential.
type PodLock ¶
type PodLock struct {
// contains filtered or unexported fields
}
PodLock is the single-writer fence shared by `claw runs serve`, `claw up` and `claw down`. The supervisor holds it for its whole lifetime.
func AcquirePodLock ¶
AcquirePodLock takes the pod's exclusive run lock without blocking.
type Record ¶
type Record struct {
Version int `json:"version"`
Key Key `json:"key"`
Generation uint64 `json:"generation"`
SpecDigest string `json:"spec_digest,omitempty"`
Spec *Spec `json:"spec,omitempty"`
TemplateID string `json:"template_id,omitempty"`
ImageID string `json:"image_id,omitempty"`
RunAgentID string `json:"run_agent_id,omitempty"`
RunKey string `json:"run_key,omitempty"` // pod-scoped container label value
Project string `json:"project,omitempty"`
State State `json:"state"`
Cancelled bool `json:"cancelled"`
Revoked bool `json:"revoked"`
RevocationProof string `json:"revocation_proof,omitempty"`
SpawnAttempted bool `json:"spawn_attempted"`
Exit *ExitInfo `json:"exit_observed,omitempty"`
Failure string `json:"failure,omitempty"`
AuthorityExpiresAt time.Time `json:"authority_expires_at"`
Deadline *time.Time `json:"deadline,omitempty"`
Credential string `json:"credential"`
Callbacks Callbacks `json:"callbacks"`
Result *ResultInfo `json:"result,omitempty"`
Timestamps Timestamps `json:"timestamps"`
LastError string `json:"last_error,omitempty"`
// The trace capture epoch cllama had last written when the worker was
// started. Observed once at spawn; it does not prove the live process.
CaptureEpoch string `json:"capture_epoch,omitempty"`
CaptureEpochObservedAt *time.Time `json:"capture_epoch_observed_at,omitempty"`
}
Record is one run's durable ledger entry.
type ResultInfo ¶
ResultInfo describes the stored result.
type Revoker ¶
Revoker closes a run's cllama identity (cllama#54). It must return nil only once cllama has durably revoked the agent and drained its dispatches.
type Server ¶
type Server struct {
Sup *Supervisor
Pod string
AllowedUID []uint32
Logf func(string, ...any)
}
Server exposes a Supervisor over a Unix socket. Trust model: any process running as an allowed UID (or root) holds every template and installation grant of this pod. Peer-UID checks authenticate a Unix account, not a program.
type Spec ¶
type Spec struct {
Template string `json:"template"`
Task string `json:"task"`
Tools []string `json:"tools"`
MaxRuntimeSeconds int64 `json:"max_runtime_seconds,omitempty"`
}
Spec is the immutable, digest-covered launch request body.
func CanonicalSpec ¶
CanonicalSpec validates a raw launch spec and returns its canonical JSON form (sorted keys, no insignificant whitespace, no HTML escaping), the parsed Spec, and the digest "sha256:<hex>" over the canonical bytes. Only strings, integers and string arrays are accepted, so the canonical form is unambiguous across languages.
type StatusView ¶
type StatusView struct {
Installation string `json:"installation"`
RunID string `json:"run_id"`
Generation uint64 `json:"generation"`
SpecDigest string `json:"spec_digest,omitempty"`
Template string `json:"template,omitempty"`
RunAgentID string `json:"run_agent_id,omitempty"`
State State `json:"state"`
Cancelled bool `json:"cancelled"`
Revoked bool `json:"revoked"`
RevocationProof string `json:"revocation_proof,omitempty"`
ExitCode *int `json:"exit_code"`
Failure string `json:"failure,omitempty"`
Result *ResultInfo `json:"result,omitempty"`
Credential string `json:"credential"`
Callbacks Callbacks `json:"callbacks"`
AuthorityExpiresAt *time.Time `json:"authority_expires_at,omitempty"`
Deadline *time.Time `json:"deadline,omitempty"`
Timestamps Timestamps `json:"timestamps"`
Pending string `json:"pending,omitempty"`
CaptureEpoch string `json:"capture_epoch,omitempty"`
CaptureEpochObservedAt *time.Time `json:"capture_epoch_observed_at,omitempty"`
}
StatusView is the public run status.
type Supervisor ¶
type Supervisor struct {
// contains filtered or unexported fields
}
Supervisor owns per-run worker lifecycle for one pod.
Each run key has two locks. The short admit lock covers only ledger reads and writes of the record and tombstone, so launch, cancel and credential replacement linearize without waiting on Docker, cllama or callbacks. The long step lock serializes advancement, which performs those external operations. Orchestrator-owned fields (generation, credential custody, authority expiry) are only written under the admit lock and are re-read and merged whenever advancement persists.
func NewSupervisor ¶
func NewSupervisor(cfg Config) (*Supervisor, error)
NewSupervisor opens the ledger. Callers must hold the pod lock.
func (*Supervisor) Advance ¶
func (s *Supervisor) Advance(ctx context.Context, k Key)
Advance drives one run's state machine as far as it can go now. Every step is idempotent, so restart reconciliation is the same code path.
func (*Supervisor) AdvanceAll ¶
func (s *Supervisor) AdvanceAll(ctx context.Context)
AdvanceAll advances every unfinished run concurrently, bounded, so one slow run cannot delay another run's cancel, expiry or teardown. A run whose step lock is busy is skipped this round.
func (*Supervisor) Cancel ¶
Cancel tombstones a run and returns immediately; advancement revokes and removes it. The tombstone is written under the admit lock, which Launch holds across its final tombstone check and accepted write, so a cancel always beats a launch that has not been accepted yet. A key with no record is terminal at once.
func (*Supervisor) CheckRevocationSupport ¶
func (s *Supervisor) CheckRevocationSupport(ctx context.Context) error
CheckRevocationSupport proves the running proxy acknowledges durable revocation (cllama#54) before any worker is started. An older or misconfigured proxy (no endpoint, no revocation dir) fails this check.
func (*Supervisor) FinalizeWithoutProxy ¶
func (s *Supervisor) FinalizeWithoutProxy(ctx context.Context, k Key, proxyGone func(context.Context) (bool, error)) error
FinalizeWithoutProxy ends a run whose cllama acknowledgement is unavailable during a full pod teardown. The caller must have removed the pod (proxy included) and verified it is gone; proxyGone re-checks that. With no worker container and no proxy process, nothing can hold the run's authority, and deleting its context means a restarted proxy cannot authenticate it. Remote effects that were already dispatched are not undone.
func (*Supervisor) Get ¶
func (s *Supervisor) Get(k Key) (*Record, error)
Get returns a run's status view, or nil when it was never accepted or cancelled.
func (*Supervisor) Launch ¶
func (s *Supervisor) Launch(ctx context.Context, req LaunchRequest) (*Record, bool, error)
Launch accepts a run idempotently. It persists intent, including the run's identity and secrets, before any side effect, then advances asynchronously.
func (*Supervisor) LiveKeys ¶
func (s *Supervisor) LiveKeys() ([]Key, error)
LiveKeys returns every unfinished run.
func (*Supervisor) Orphans ¶
func (s *Supervisor) Orphans(ctx context.Context) ([]Container, error)
Orphans returns labelled run workers with no ledger record or tombstone. They can only exist after ledger loss, because a run's project is recorded before `compose create`; the supervisor refuses to serve while any exist.
func (*Supervisor) ReadCaptureEpoch ¶ added in v0.31.0
func (s *Supervisor) ReadCaptureEpoch() (string, error)
ReadCaptureEpoch returns the capture epoch cllama last wrote. It reports what the file says, not which proxy process is live. An empty epoch with a nil error means there is no epoch file; a malformed record yields errCaptureEpochInvalid. No value is ever guessed.
func (*Supervisor) RemoveOrphan ¶
func (s *Supervisor) RemoveOrphan(ctx context.Context, c Container) error
RemoveOrphan removes one orphaned run worker through Compose. The project must be a run project of this pod's namespace (clawrun-<ns>-<key>); the executor passes an empty project on stdin, never an ambient file.
func (*Supervisor) ReplaceCredential ¶
func (s *Supervisor) ReplaceCredential(ctx context.Context, k Key, generation uint64, credential string, expires *time.Time) (*Record, error)
ReplaceCredential installs a reissued orchestrator credential and, optionally, a later authority expiry. It is persisted before the ACK.
func (*Supervisor) Result ¶
func (s *Supervisor) Result(k Key) ([]byte, *Record, error)
Result returns a stopped run's stored answer.
type Template ¶
type Template struct {
Version int `json:"version"`
ID string `json:"template_id"`
Name string `json:"template"`
Pod string `json:"pod"`
ClawType string `json:"claw_type"`
// ImageRef is the reference declared in the pod; ImageID pins the exact
// image resolved at claw up.
ImageRef string `json:"image_ref"`
ImageID string `json:"image_id"`
RunNetwork string `json:"run_network"`
CllamaURL string `json:"cllama_url"`
CllamaAPIURL string `json:"cllama_api_url"`
// ToolsGrant is the sorted set of managed tool names a run may enable.
ToolsGrant []string `json:"tools_grant"`
// ContextDigest binds the cllama context (tools, models, policy, rules,
// with credentials redacted) into the template identity.
ContextDigest string `json:"context_digest"`
Limits Limits `json:"limits"`
Worker driver.RunWorkerSpec `json:"worker"`
}
Template is the compiled, claw up-owned description of a per-run worker. The caller of a launch only selects within it.