Documentation
¶
Overview ¶
Dialer-side helpers in package agent: CP-side outbound mTLS dial logic for the CP→clawkerd Session channel.
Single entry point: Dialer.DialAgent(ctx, containerID). The same function is invoked at CP boot (over the result of listAgentIDs) and from the typed-event subscriber on dockerevents.DockerEvent (filtered to container start/restart/unpause for purpose=agent containers) when an agent container reaches running state at runtime — so two callers, one dial path.
DialAgent is fire-and-forget: it spawns a goroutine that owns the dial, the Session stream, and the lifetime drain loop. All failures are logged at the Error level — callers don't need to handle errors. A failed dial leaves no resources behind; a successful dial is held open until ctx cancels (CP shutdown) or the peer closes.
Asymmetric trust model — load-bearing ¶
CP is the overlord. The dial NEVER aborts on cert / identity grounds. Cert chain verification and registry thumbprint classification outcomes are captured on the establishResult and surfaced through the typed event surface — SessionConnected carries flat PeerAgentFullName/PeerThumbprint fields (purely diagnostic, never a gate); AgentRegistered/AgentUntrusted carry the policy outcomes. Subscribers consume those events to enact policy (containment, alerting, eviction); the dialer holds no policy itself.
Why permissive: CP must always be able to reach clawkerd to issue containment commands (iptables lock, network detach, container kill). A compromised clawkerd presenting a bad cert is exactly when the channel must be UP so CP can react. Aborting on cert grounds would strand CP exactly at the moment governance is most needed.
The asymmetric counterpart lives in clawkerd/listener.go: the clawkerd-side listener is STRICT — CP CN pin + Client-Auth EKU + CA chain enforced at TLS layer. clawkerd refuses any peer that isn't CP. This pairs with the dialer's permissive client posture.
Connection establishment failures still happen on connectivity grounds (TCP timeout, container gone, retry exhausted, ctx cancelled) — those drive establishOutcome and SessionFailed.
FD-leak ceiling: a successful dial whose conn.Close() repeatedly fails would accumulate file descriptors and gRPC keepalive goroutines indefinitely. After closeErrCeiling consecutive close failures the dial loop bails for the target with a SessionFailed event (Reason carries the "fd-leak-ceiling" classification) so operators see the outcome instead of a silent leak. A successful close anywhere in the loop resets the counter.
Identity resolution flow for AgentService RPCs:
- The CP's agent listener enforces mTLS at the TLS layer (server cert + ClientAuth EKU + chain to the CLI CA).
- AuthInterceptor verifies the bearer token + per-method scope.
- IdentityInterceptor (this package) runs a universal three-stage gate on EVERY RPC including Register: (a) Subject.CommonName == consts.ContainerClawkerd (constant-time). The cert CN is the deterministic clawkerd binary identity; mismatch means the peer is not presenting a CLI-minted agent cert. (b) Resolve the kernel-attested peer IP to the purpose=agent container that owns it on the clawker network (via ContainerByPeerIP), reading the project/agent labels as the authoritative identity source — Docker is independent ground truth. (c) Compose the label-derived AgentFullName and constant-time compare it against the cert's urn:clawker:agent: URI SAN. The cert's SAN claim is VERIFIED against the IP-grounded label truth — never the basis of lookup.
- The resolved (containerID, project, agentName) is attached to ctx via WithResolvedContainer; downstream handlers read it via ResolvedContainerFromContext to avoid re-inspecting the container.
All rejections return codes.PermissionDenied with a generic envelope — no leak about which check failed.
Package agent owns the CP-side agent surface: the persisted identity registry, the AgentService.Register handler, the per-RPC IdentityInterceptor, the CP→clawkerd Session dialer, and the AgentRegistered/AgentUntrusted event types.
Registry rows record the (mTLS cert thumbprint, container_id, project, agent_name) tuple. Reads are issued by:
- the Register handler (idempotency / replay-protection at row write time, keyed by container_id)
- the dialer at Hello time (LookupByContainerID → drives AgentRegistered / AgentUntrusted publication via thumbprint compare against the live peer cert)
- AdminService.ListAgents (Snapshot)
Writes are CP-only: the Register handler captures the live mTLS peer's thumbprint and writes the row. Eviction: startup orphan-row reap (in agent.Start), plus dockerevents container/destroy (subscribed in agent.Start). Stop/die/kill do NOT evict because a stopped container can be `docker start`-ed back into life.
Identity is channel-bound: the registry key is `(thumbprint, container_id)` (both UNIQUE in sqlite). Agent-full-name composition from `(project, agent_name)` is no longer persisted — the IdentityInterceptor establishes the trust anchor via the kernel-attested peer IP and cross-checks the cert SAN against the label-derived AgentFullName at the gRPC boundary, so the registry holds only the per-row identity tuple.
Index ¶
- Constants
- Variables
- func DialEvent(ev dockerevents.DockerEvent) bool
- func EnsureSchema(dbPath string, log *logger.Logger) error
- func IdentityInterceptor(peerLookup ContainerByPeerIP, log *logger.Logger) (grpc.UnaryServerInterceptor, grpc.StreamServerInterceptor, error)
- func Publish(topic *pubsub.Topic[AgentEvent], ev AgentEvent) bool
- func ReapOrphans(ctx context.Context, reg Registry, lister ContainerListFunc, ...) (int, error)
- func ShouldReconnect(ctx context.Context, drain DrainResult) bool
- func Start(ctx context.Context, deps StartDeps) (func(), error)
- func SubscribeEvict(ctx context.Context, topic *pubsub.Topic[dockerevents.DockerEvent], ...)
- func SubscribeSessionCancel(ctx context.Context, topic *pubsub.Topic[dockerevents.DockerEvent], ...)
- func WithResolvedContainer(ctx context.Context, resolved ResolvedContainer) context.Context
- type Action
- type Agent
- type AgentEvent
- type AgentEventState
- type AgentInitializedStep
- type AgentReadyStep
- type AgentStore
- type AgentWatcher
- type AgentWatcherOptions
- type ContainerByPeerIP
- type ContainerListFunc
- type ContainerLister
- type Dialer
- func (d *Dialer) CancelDial(containerID string)
- func (d *Dialer) CapturePeer(rawCerts [][]byte, peer *PeerInfo)
- func (d *Dialer) ClassifyRegistry(peerThumbprint [sha256.Size]byte, containerID string) (RegistryOutcome, string)
- func (d *Dialer) CloseAndCheckLeak(c closeable, count *int, log *logger.Logger) bool
- func (d *Dialer) DialAgent(ctx context.Context, containerID string)
- func (d *Dialer) DialAllRunning(ctx context.Context, lister *ContainerLister, opts ListOpts)
- func (d *Dialer) DispatchAgentEvents(ctx context.Context, containerID string, res EstablishResult, ...)
- func (d *Dialer) DriveRegister(ctx context.Context, containerID string, res EstablishResult, ...)
- func (d *Dialer) PublishConnected(ctx context.Context, containerID, agent, project, addr string, attempt int, ...)
- type DrainResult
- type Entry
- type EstablishOutcome
- type EstablishResult
- type EventType
- type ExecTarget
- type Executor
- type ExecutorEventState
- func (i ExecutorEventState) Complete(at time.Time) ExecutorEventState
- func (i ExecutorEventState) CompletedAt() time.Time
- func (i ExecutorEventState) Fail(at time.Time, detail string) ExecutorEventState
- func (i ExecutorEventState) LastError() string
- func (i ExecutorEventState) StartedAt() time.Time
- func (i ExecutorEventState) Status() Status
- func (i ExecutorEventState) StepCount() int
- func (i ExecutorEventState) StepIndex() int
- func (i ExecutorEventState) StepName() string
- func (i ExecutorEventState) WithStep(stepName string, stepIndex int) ExecutorEventState
- func (i ExecutorEventState) WithStepError(detail string) ExecutorEventState
- type Handler
- type ListOpts
- type Message
- type MobyPeerLookup
- type PeerInfo
- type Reason
- type Registry
- type RegistryOutcome
- type Repository
- type ResolvedContainer
- type ShellStep
- type StartDeps
- type Status
- type Step
- type Trust
Constants ¶
const ( // DrainGracefulEOF peer (clawkerd) closed the Session cleanly. DrainGracefulEOF drainOutcome = iota // DrainCtxCanceled parent ctx cancelled (CP shutdown). // publishBroken is suppressed for this outcome — the bus is on // its way down anyway. DrainCtxCanceled // DrainStreamErr Recv returned a non-EOF error and ctx is still // live. Treat as transient (re-establish on next cycle). DrainStreamErr )
const ( ConfigSeedScript = `` /* 733-byte string literal not displayed */ // GitconfigFilterTemplate strips [credential] sections from the // host-mounted gitconfig before placing it under the unprivileged // user's home. %q is replaced with consts.HostGitConfigStagingPath // (Go-quoted so the bash literal never drifts from the workspace // mount target). // // Three branches: // - awk succeeds with non-empty output → move into place // - awk succeeds with empty output (host gitconfig had ONLY // [credential] blocks) → discard the empty tmp; copying the // unfiltered file would leak credentials. // - awk syscall failed → discard tmp and bail; same rationale. GitconfigFilterTemplate = `` /* 329-byte string literal not displayed */ GitCredentialsScript = `[ -n "$` + consts.EnvHostProxy + `" ] || exit 0 [ "$` + consts.EnvGitHTTPS + `" = "true" ] || exit 0 git config --global credential.helper clawker ` // SshKnownHostsScript reads the host blob from stdin (PipeStage's // initial_stdin) and appends only lines not already present. // Idempotent across container restarts. SshKnownHostsScript = `` /* 252-byte string literal not displayed */ // PostInitScript runs the user's one-time post_init hook. Contract: // attempted at most once per container lifecycle. DONE-first short // circuits an already-attempted container; a missing script writes the // marker and exits (nothing to do); a present script runs once. On // success the marker is written so a restart never re-runs it; on // failure the marker is NOT written and the exit code propagates — the // Step is fatal (plan halts, agent-ready never sent). A failed container // is torn down rather than restarted: the Step carries exit_on_non_zero // so clawkerd self-exits with the mirrored code, and CP enforces a // grace-then-SIGKILL backstop (killAfterGrace). A manually-restarted // container re-runs this hook (no marker was written). // // The `[ -x … ] || { …; }` brace group is load-bearing: `|| touch && exit` // without braces binds as `([ -x ] || touch) && exit`, which exits 0 when // the script EXISTS and never runs it. Do not remove the braces. PostInitScript = `POST="$HOME/` + consts.DotClawkerDir + `/` + consts.HookPostInit + `.sh" DONE="$HOME/.claude/post-initialized" [ -f "$DONE" ] && exit 0 [ -x "$POST" ] || { touch "$DONE"; exit 0; } if "$POST"; then touch "$DONE" else exit $? fi ` // PreRunScript runs the every-start pre_run hook. No marker (runs every // start) and no log/ready-file. The `[ -x … ] || exit 0` guard is a // defensive regression net: with always-deliver the file is present, but // if it ever goes missing the Step no-ops instead of failing the plan. // `[ -x … ] && …` would exit 1 when absent (fail the Step); `&& … || true` // would swallow a real failure (break the fatal contract) — this two-line // form no-ops when absent AND propagates the exit code when present. The // file carries #!/bin/bash + set -e from PrepareHookTar. PreRunScript = `[ -x "$HOME/` + consts.DotClawkerDir + `/` + consts.HookPreRun + `.sh" ] || exit 0 "$HOME/` + consts.DotClawkerDir + `/` + consts.HookPreRun + `.sh" ` )
Inline self-gating shell scripts. Each script's gating predicates ([ -d INIT_DIR ], [ -f $HOST_GITCONFIG ], CLAWKER_GIT_HTTPS, etc.) keep the dispatch CP-feature-flag-free: CP doesn't need to know which optional features a given container has wired.
const CloseErrCeiling = 5
CloseErrCeiling bounds the number of consecutive conn.Close() failures the dial loop will tolerate before giving up on the target. A non-zero close error usually means a transport-level state machine the gRPC client can't unwind cleanly (broken keepalive, half-closed socket); each one leaks a file descriptor + a couple of background goroutines. Five gives a transient hiccup room to recover without letting a structurally-broken peer accumulate resources without bound. Successful Close anywhere in the loop resets the counter.
const RegisterRequiredTimeout = 30 * time.Second
RegisterRequiredTimeout caps how long the dispatch path waits for clawkerd's RegisterDone Response after sending RegisterRequired on the Session stream. Hydra exchange + mTLS dial + Register handler is the chain on the agent side; 30s is comfortably more than the chained latency under normal conditions.
Variables ¶
var ErrAmbiguousPeerIP = errors.New("multiple purpose=agent containers match peer IP")
ErrAmbiguousPeerIP is returned when two or more `purpose=agent` containers on the clawker network advertise endpoints with the same peer IP. Docker can transiently leave stale endpoints in NetworkSettings.Networks during restart cycles, and grounding the trust anchor on the first match would create a race window. Fail closed instead — operators see a distinct event and the trust gate rejects the RPC until the daemon state converges.
var ErrContainerStopped = errors.New("container not running")
ErrContainerStopped is the sentinel resolveAgent returns when the inspect succeeded but State.Running is false. Distinct from a generic inspect error: a stopped container is terminal (no point retrying), a generic inspect error is transient (retry within the connect-total budget).
var ErrInvalidAgentLabel = errors.New("agent container has invalid identity label")
ErrInvalidAgentLabel is returned when the container matched by peer IP carries a missing or malformed dev.clawker.agent label. A missing dev.clawker.project label is NOT an error here — it is the legitimate global-scope-agent signal (no project namespace, producing 2-segment naming clawker.<agent>); auth.NewProjectSlug("") returns a zero-value slug with nil err. Distinguishing this from a clean no-match lets the trust gate emit a daemon-state diagnostic instead of a generic auth reject.
var ErrMalformedEntry = errors.New("agentregistry: malformed registry row")
ErrMalformedEntry is returned by sqlite reads when a row's agent_name / project / thumbprint fails re-validation (a value that landed pre-typed-boundary, a hand-edited DB, or a corrupted column). The Register handler treats this sentinel as "evict + re-write": the row's identity is unusable so it gets purged and replaced by the typed identity the middleware just resolved off the live peer IP + cert. Wrapped via fmt.Errorf so errors.Is on the wrapped error works for callers that don't unwrap themselves.
var ErrNoContainerForPeerIP = errors.New("no purpose=agent container with matching " + consts.Network + " IP")
ErrNoContainerForPeerIP is returned when no `purpose=agent` container on the clawker network has an endpoint IP matching the requested peer IP. Callers MUST treat this as a hard authentication failure.
var ErrUnknownAgent = errors.New("agentregistry: unknown agent")
ErrUnknownAgent is returned by LookupByContainerID when no entry matches. Thumbprint compare against the live peer cert is the dialer's job, not the registry's.
var ErrWatcherMisconfig = errors.New("controlplane: AgentWatcher misconfigured")
ErrWatcherMisconfig is returned by NewAgentWatcher when a required callback is nil or an option is negative. The CP serve path never panics on misconfig — the orchestrator surfaces this as a structured event=agent_watcher_unavailable line and degrades.
Functions ¶
func DialEvent ¶ added in v0.12.9
func DialEvent(ev dockerevents.DockerEvent) bool
DialEvent reports whether a docker event should trigger a dial. Dial actions: Start, Restart, UnPause — every transition that takes a container into running state. ActionCreate is intentionally excluded: a created-but-not-started container has no clawkerd listener, so dialing always fails; the next ActionStart re-fires. Scope: only purpose=agent containers (CP itself, host proxy never run clawkerd).
func EnsureSchema ¶
EnsureSchema opens the database in writer mode (creating the file if missing), applies the schema, and closes. Called by the CP daemon at startup before NewSQLiteWriter so the schema apply path is observable independent of the long-lived writer handle.
func IdentityInterceptor ¶
func IdentityInterceptor(peerLookup ContainerByPeerIP, log *logger.Logger) (grpc.UnaryServerInterceptor, grpc.StreamServerInterceptor, error)
IdentityInterceptor returns paired unary and stream interceptors that enforce the universal identity gate documented at the top of handler.go. Wired AFTER AuthInterceptor on the agent listener so token validation runs first.
peerLookup is required — a nil resolver would silently disable the trust gate and admit every RPC. Returns a non-nil error rather than panicking: this constructor runs post-eBPF-load in clawkercp.go, so a panic here would strand pinned eBPF programs with no supervisor. clawkercp.go logs the error structurally (event=agent_identity_unavailable) and refuses to bring up the agent listener, degrading the AgentService surface while CP + firewall + admin listener stay up.
log defaults to logger.Nop() if nil so the panic-recovery path of grpc-go never sees a nil deref.
Every rejection returns codes.PermissionDenied with the same generic envelope ("registration rejected") — attackers must not learn which check failed. The structured log carries the classification via a unique event= field per stage.
func Publish ¶ added in v0.12.9
func Publish(topic *pubsub.Topic[AgentEvent], ev AgentEvent) bool
Publish wraps an AgentEvent in the pubsub envelope and offers it to the topic. It is the single producer-side seam: dialer, exec, and start all Publish through here so the envelope (ID, Timestamp, Source) is stamped one way. A nil topic is a no-op — a degraded CP that failed to construct the topic must not NPE a producer (the orchestrator logs the construction failure as event=<subsystem>_unavailable and wires nil). Publish is non-blocking and returns false on a full or closed topic; the caller does not block.
func ReapOrphans ¶ added in v0.12.9
func ReapOrphans(ctx context.Context, reg Registry, lister ContainerListFunc, log *logger.Logger) (int, error)
ReapOrphans drops every registry row whose container_id is not present in the lister's snapshot.
func ShouldReconnect ¶ added in v0.12.9
func ShouldReconnect(ctx context.Context, drain DrainResult) bool
ShouldReconnect classifies the post-drain decision: re-enter establishWithRetry (true) or return from runDial (false). False on intentional teardown — parent ctx cancelled (CP shutdown) or drain reported the same outcome — so neither SessionBroken nor a fresh establish cycle is published while the bus is on its way down. Pure function; unit-testable independent of the dial path.
func Start ¶
Start gathers the full agent worldview at CP boot and wires every ongoing agent-axis subscription onto the dockerevents topic. Called once from the orchestrator after the topics are up. Returns a cleanup func; the per-subscriber delivery goroutines are owned by the topic itself (the orchestrator's topic.Close() tears them down), so the returned cleanup is a no-op kept for call-site symmetry.
Steps, in order:
- Reap orphan registry rows. List every purpose=agent container (All:true, includes stopped). For each registry row whose container_id is missing from docker, evict. Heals the registry against `docker rm`s that landed while CP was down.
- Subscribe to the dockerevents topic for the evict path — predicate container/destroy folded into the handler; consumer evicts the registry row.
- Subscribe to the dockerevents topic for the session-cancel path — predicate container/{die,stop,kill,oom,destroy}; consumer calls dialer.CancelDial so the in-flight Session tears down immediately.
- Subscribe to the dockerevents topic for the dial path — predicate container/start|restart|unpause with purpose=agent; consumer calls dialer.DialAgent.
func SubscribeEvict ¶ added in v0.12.9
func SubscribeEvict(ctx context.Context, topic *pubsub.Topic[dockerevents.DockerEvent], reg Registry, log *logger.Logger)
SubscribeEvict wires the registry's evict path to dockerevents container/destroy envelopes. moby fires `destroy` for every `docker rm` on a container; `remove` is image-only and never fires for containers.
Stop/die/kill/oom do NOT evict: the container still exists in the daemon and may be `docker start`-ed back into life. The registry row survives so a subsequent restart finds its existing identity.
consecutiveFailures is closed over by the handler so sustained evict failures escalate to a single Error line; the pipe's serialized, single-goroutine-per-subscriber delivery makes the closure-local counter safe without a mutex.
func SubscribeSessionCancel ¶ added in v0.12.9
func SubscribeSessionCancel(ctx context.Context, topic *pubsub.Topic[dockerevents.DockerEvent], dialer sessionCanceller, log *logger.Logger)
SubscribeSessionCancel wires CancelDial to docker container lifecycle events. The docker event is the source of truth for "clawkerd is no longer serving"; canceling at this layer keeps the dialer independent of registry state. A nil dialer is a no-op (no subscription).
func WithResolvedContainer ¶
func WithResolvedContainer(ctx context.Context, resolved ResolvedContainer) context.Context
WithResolvedContainer attaches the peer-IP-resolved container identity to ctx for downstream handlers. Exposed so test code and future identity-augmenting interceptors can attach a resolved container directly; production code never needs to call this (the interceptor does it on success).
A zero-value ResolvedContainer (empty ContainerID) is silently dropped — ctx is returned unchanged so downstream ResolvedContainerFromContext sees ok=false. The CP must not panic on the serving path (root CLAUDE.md security contract); the read-side defensive ok=false is the floor against the silent-identity-vacuum hazard.
Types ¶
type Action ¶
type Action string
Action is the per-EventType verb. The (Type, Action) pair is the full discriminator the projection Stores switch on to mutate their own worldview.
const ( ActionConnecting Action = "connecting" ActionConnected Action = "connected" ActionFailed Action = "failed" ActionBroken Action = "broken" )
Session-axis actions (DialerEventType).
const ( ActionExecStarted Action = "started" ActionExecStepStarted Action = "step_started" ActionExecStepCompleted Action = "step_completed" ActionExecStepFailed Action = "step_failed" ActionExecFailed Action = "exec_failed" ActionExecCompleted Action = "completed" )
Exec-axis actions (ExecutorEventType). The plan dispatch lifecycle: the plan starts, each step starts/completes/fails, and the plan reaches a terminal completed/failed state.
type Agent ¶
type Agent struct {
ContainerID string `json:"container_id"`
ContainerName string `json:"container_name"`
AgentName string `json:"agent_name"`
Project string `json:"project"`
}
Agent is the stable identity triple every AgentEvent carries so a subscriber sees consistent (container, agent, project) fields without re-deriving them from the registry.
type AgentEvent ¶
AgentEvent is the single unified payload on the agent Topic. It rides inside pubsub.Event[AgentEvent]; the pipe never inspects it. The (Type, Action) discriminator on Message is the routing identity, and the projection Stores read Message fields natively to build their own worldview.
func (AgentEvent) MarshalZerologObject ¶
func (e AgentEvent) MarshalZerologObject(z *zerolog.Event)
MarshalZerologObject surfaces the (Type, Action) discriminator plus the classification + identity fields so the single generic audit hook in controlplane/pubsub can log per-event detail without reflection. Zero-value fields are omitted to keep lines compact.
type AgentEventState ¶
type AgentEventState struct {
ContainerID string
AgentName string
Project string
Address string
SessionStatus Status
Registered bool
Trust Trust
Thumbprint [sha256.Size]byte
PeerAgentFullName string
Attempts int
LastError string
UpdatedAt time.Time
Executor ExecutorEventState
}
AgentEventState is the agent domain's own in-memory worldview of one clawker-managed agent, held in this domain's AgentStore. Three axes — session (SessionStatus, Address, Attempts, LastError, Thumbprint), identity (Registered, Trust), and exec (ExecutorEventState) — held as a single entity. The agentregistry sqlite store remains the durable truth source for identity rows; this struct is the observed-now view derived from the AgentEvent stream.
LastError is the SESSION-axis last error (dial failures, broken streams). ExecutorEventState-axis failures land in ExecutorEventState.LastError. Trust zero value is "trusted with no reason" (see Trust).
type AgentInitializedStep ¶ added in v0.12.9
type AgentInitializedStep struct {
Name string
}
func (AgentInitializedStep) Command ¶ added in v0.12.9
func (s AgentInitializedStep) Command(id string) (*clawkerdv1.Command, bool)
func (AgentInitializedStep) IsStep ¶ added in v0.12.9
func (AgentInitializedStep) IsStep()
func (AgentInitializedStep) StepName ¶ added in v0.12.9
func (s AgentInitializedStep) StepName() string
type AgentReadyStep ¶ added in v0.12.9
type AgentReadyStep struct {
Name string
}
func (AgentReadyStep) Command ¶ added in v0.12.9
func (s AgentReadyStep) Command(id string) (*clawkerdv1.Command, bool)
func (AgentReadyStep) IsStep ¶ added in v0.12.9
func (AgentReadyStep) IsStep()
func (AgentReadyStep) StepName ¶ added in v0.12.9
func (s AgentReadyStep) StepName() string
type AgentStore ¶
type AgentStore struct {
// contains filtered or unexported fields
}
AgentStore is this domain's own worldview projection: a thread-safe map of container_id -> AgentEventState, mutated ONLY by its own subscriber callbacks. The agent domain owns its observed worldview rather than writing into a shared central store. Cross-domain consumers never read this directly; if another domain wants agent state it subscribes to the agent Topic and builds its own projection.
Distinct from the agentregistry sqlite rows (the durable, attested identity axis): this store is the observed-now axis derived from the live AgentEvent stream and is cleared on CP restart.
func (*AgentStore) Get ¶
func (s *AgentStore) Get(containerID string) (AgentEventState, bool)
Get returns a copy of the projected state for a container and whether it exists. The returned value is a copy (AgentEventState is a value type with value-type sub-structs), so callers cannot mutate the store.
func (*AgentStore) Len ¶
func (s *AgentStore) Len() int
Len reports how many agents the store currently tracks.
func (*AgentStore) Subscribe ¶
func (s *AgentStore) Subscribe(topic *pubsub.Topic[AgentEvent])
Subscribe wires this store to the agent Topic. Every AgentEvent is projected into the store via project. A nil topic is a no-op so a degraded orchestrator (topic construction failed) does not NPE.
func (*AgentStore) SubscribeDockerEvents ¶
func (s *AgentStore) SubscribeDockerEvents(topic *pubsub.Topic[dockerevents.DockerEvent])
SubscribeDockerEvents wires this store's eviction path to the dockerevents topic so the observed-now worldview stays bounded. The container/destroy predicate is folded into the handler (the pipe has no SubscribeFiltered): on docker rm of a container, the store drops its projected entry. Only destroy evicts — die/stop/kill leave the container docker start-able, and the worldview re-projects from the next session event, so evicting on a mere exit would churn the map for a container that is coming back. This matches the registry's destroy-only evict semantics (see subscribeEvict). A nil topic is a no-op so a degraded orchestrator does not NPE.
type AgentWatcher ¶
type AgentWatcher struct {
// contains filtered or unexported fields
}
AgentWatcher polls the Docker daemon for clawker-managed agent containers and invokes a drain-to-zero callback once the agent count has been zero across MissedThreshold consecutive polls past the grace period. The CP uses this to self-terminate when no agents remain — there is no reason to keep the control plane (Envoy + CoreDNS + CP daemon) alive after the last agent drains.
INV-B2-007: strict drain-to-zero ordering is enforced by the caller via the onDrainToZero callback. The watcher guarantees only that the callback fires at most once, before Run returns.
func NewAgentWatcher ¶
func NewAgentWatcher( log *logger.Logger, listAgents func(context.Context) (int, error), onDrainToZero func(context.Context) error, opts AgentWatcherOptions, ) (*AgentWatcher, error)
NewAgentWatcher constructs an AgentWatcher. A nil callback or a negative option value returns (nil, ErrWatcherMisconfig) — misconfig must fail loudly but NEVER via panic, because a panic on the CP serve path kills PID 1 and strands eBPF programs unsupervised (see the resilience contract). The orchestrator degrades the subsystem on error instead.
func (*AgentWatcher) Run ¶
func (w *AgentWatcher) Run(ctx context.Context) error
Run blocks until ctx is cancelled, the drain-to-zero condition fires, or consecutive list errors exceed ListErrCeiling. On drain, Run invokes onDrainToZero synchronously and returns its error. On ctx cancel, returns ctx.Err(). On error ceiling, returns a wrapped error surfacing the last list failure.
The grace period is measured on wall-clock time from Run entry — any zero-count polls during the grace window count toward the miss streak, but the streak cannot reach the threshold before grace expires. This prevents a race where the CP starts up before any agents have been enrolled and immediately drains.
Run must be called at most once per watcher; a second call returns an error rather than spinning up a second poll loop.
type AgentWatcherOptions ¶
type AgentWatcherOptions struct {
PollInterval time.Duration
MissedThreshold int
GracePeriod time.Duration
// ListErrCeiling bounds how many consecutive listAgents errors the
// watcher tolerates before returning an error from Run. Prevents
// "Docker is wedged" from leaving the CP blind and permanent.
ListErrCeiling int
}
AgentWatcherOptions overrides default watcher tuning. Zero values select defaults (PollInterval 30s, MissedThreshold 2, GracePeriod 60s, ListErrCeiling 20). Negative values are a construction error — silently snapping to a default would hide caller misconfig.
type ContainerByPeerIP ¶
type ContainerByPeerIP interface {
LookupByIP(ctx context.Context, ip netip.Addr) (ResolvedContainer, error)
}
ContainerByPeerIP resolves a live mTLS peer IP to the `purpose=agent` container owning that IP on the clawker network. Returns ErrNoContainerForPeerIP when nothing matches, ErrInvalidAgentLabel when the matching container's dev.clawker.agent label is missing or malformed (an absent dev.clawker.project label is a legitimate global-scope-agent signal, not an error), ErrAmbiguousPeerIP when two or more containers share the peer IP (Docker restart-race window — fails closed), or a wrapped daemon error.
type ContainerListFunc ¶
ContainerListFunc enumerates every `purpose=agent` container ID currently known to the docker daemon. The implementation MUST include stopped/exited containers — a stopped container can be `docker start`-ed back into life, and its registry row should survive that transition. Only `docker rm` (destroy) means the container is genuinely gone and the row is orphaned.
type ContainerLister ¶
type ContainerLister struct {
// contains filtered or unexported fields
}
ContainerLister enumerates managed `purpose=agent` container IDs from the docker daemon. It is the production source behind every agent-axis container listing in the CP orchestrator:
- Firewall handler + AgentWatcher list running-only (ListOpts{}).
- Reaper / initial-dial reconciler list running + stopped (ListOpts{All: true}) — a stopped container can be `docker start`-ed back into life, and its registry row should survive that transition.
The label filter is non-overridable: every List call narrows to BOTH the managed label AND purpose=agent, so a caller can't accidentally widen scope past purpose=agent. Construct once at CP startup; share across consumers.
func NewContainerLister ¶
func NewContainerLister(dc mobyclient.APIClient, cfg config.Config) *ContainerLister
NewContainerLister wraps a moby APIClient as the managed-agent container enumerator. The label filter keys/values are read from cfg so the lister stays consistent with the rest of the CP's resource filtering.
type Dialer ¶
type Dialer struct {
Log *logger.Logger
Docker mobyclient.APIClient
Topic *pubsub.Topic[AgentEvent]
// Agents is the CP-owned agentregistry (read+write). The dialer
// reads it at handshake time to classify the peer cert against
// the registered row: match (registered), miss (drives Register
// handshake), thumbprint mismatch (untrusted). Connection stays
// open in all cases. The Register flow re-reads after RegisterDone
// to confirm the row landed. Required (non-nil) — wiring bug if
// unset.
Agents Registry
// Executor dispatches the CP-driven init plan after
// dispatchAgentEvents and before drainStream. nil disables init
// (entrypoint hangs on its fifo until timeout) — runInit logs a
// warning so the misconfiguration is observable. Set at
// construction; immutable after Start.
Executor *Executor
CpClientCert tls.Certificate
CaPool *x509.CertPool
Dialing map[string]context.CancelFunc
// contains filtered or unexported fields
}
Dialer captures the CP-side material every dial needs. Construct once at CP startup; share across all agent dials.
dialing is the dedup + cancel map: containerIDs currently being dialed (or already-Session-established) mapped to the cancel func for their per-dial ctx. Initial poll and the dockerevents subscriber both call DialAgent for the same running container; the dedup keeps the second call from spinning a duplicate goroutine against an already-open Session. Membership lasts the lifetime of the dial goroutine — after the Session closes (peer drop, ctx cancel, retry timeout), the entry is removed and a future event for the same containerID dials fresh.
CancelDial uses the stored cancel func to tear down a Session synchronously with a registry-evict (container/destroy) — without it, the dialer's runDial loop only notices the disappearance on the next reconnect attempt via outcomeContainerGone, leaving a doomed stream open during the interval.
func NewDialer ¶ added in v0.12.9
func NewDialer(log *logger.Logger, docker mobyclient.APIClient, topic *pubsub.Topic[AgentEvent], agents Registry, certPath, keyPath string, caPool *x509.CertPool, executor *Executor) (*Dialer, error)
NewDialer constructs a Dialer. Returns an error if the CP client cert / key cannot be loaded — better to fail at CP startup than to defer the failure to the first dial. Returns (nil, error) on any nil required dependency; never panics, per the CP serve-path contract.
topic is required: the dialer publishes the unified AgentEvent (session connecting/connected/failed/broken, registry registered, trust untrusted) so other CP domains can subscribe to the agent axis without coupling to the dialer directly. Pass a real *pubsub.Topic[AgentEvent]; tests can construct one cheaply.
agents is required: every successful dial cross-checks the peer cert against the registry row keyed by container_id and dispatches the registered / untrusted AgentEvents accordingly. nil agents would strand worldview consumers without a registration signal.
func (*Dialer) CancelDial ¶
CancelDial synchronously cancels the in-flight Session for containerID, if any. Called by the registry-evict subscriber on container/destroy so the dialer tears down the doomed stream immediately rather than waiting for the next reconnect to classify outcomeContainerGone. Safe to call when no dial is in flight (no-op) and concurrent-safe (mu-guarded). The goroutine's own cleanup runs the deferred delete after runDial returns.
func (*Dialer) CapturePeer ¶ added in v0.12.9
CapturePeer populates PeerAgentFullName / PeerThumbprint / ChainVerified on peer from the peer's TLS handshake material. Permissive: every code path returns without error; outcomes flow into peer fields (and peer.CaptureReason for unusual cases). Extracted from the dial() callback so tests can drive it directly without standing up a TLS server.
func (*Dialer) ClassifyRegistry ¶ added in v0.12.9
func (d *Dialer) ClassifyRegistry(peerThumbprint [sha256.Size]byte, containerID string) (RegistryOutcome, string)
ClassifyRegistry cross-checks the captured peer cert thumbprint against the agentregistry row keyed by container_id and returns the typed outcome plus any diagnostic detail. The dial flow uses the outcome to drive event publication (Match → SessionConnected only; Miss → drives Register handshake; ThumbprintMismatch → AgentUntrusted).
Identity comparison is between (a) the live peer cert thumbprint and (b) the registry row's Thumbprint. The cert SAN AgentFullName vs label-derived AgentFullName check lives upstream in the IdentityInterceptor — the dialer is the OUTBOUND CP→clawkerd path, where trust on the agent side comes from clawkerd's listener pinning CP's CN, not from CP re-deriving an AgentFullName from a registry row. The row IS the identity once written; later label edits cannot drift identity without invalidating the cert.
Connection NEVER aborts here. Lookup errors that are NOT "no such row" return OutcomeRegistryNotQueried and a non-empty detail string — these indicate a sqlite/IO regression visible to operators even though the connection proceeds.
ErrMalformedEntry (a stored row that no longer re-validates as a typed identity) is classified as OutcomeRegistryMiss so the dial drives the Register handshake — the Register handler will evict the malformed row and re-write it from the middleware-resolved identity. Without this, a malformed row self-perpetuates: the dialer would Publish AgentUntrusted on every reconnect and never trigger the cleanup path.
func (*Dialer) CloseAndCheckLeak ¶ added in v0.12.9
CloseAndCheckLeak closes the conn and tracks consecutive close failures against CloseErrCeiling. Returns true iff the caller should bail the dial loop (ceiling reached). A successful close resets the counter so a transient hiccup does not poison the ledger across the lifetime of the cycle. Extracted for testability — production callers always pass a *grpc.ClientConn, but the closeable interface lets unit tests inject a controlled error sequence without standing up a real gRPC channel.
func (*Dialer) DialAgent ¶
DialAgent opens a Session stream to the clawkerd listener inside the given agent container, sends Hello, awaits HelloAck, and holds the stream open until ctx cancels or the peer closes. Returns immediately — the dial + lifetime drain run on a background goroutine. All failures are logged.
Dedup: a no-op if a dial for the same containerID is already in flight. Initial poll + dockerevents subscriber both reach this function with overlapping IDs at CP startup; dedup keeps a second call from spinning a duplicate goroutine.
Retry: the dial goroutine retries connection establishment with exponential backoff + full jitter (cap connectMaxBackoff) until either the Session is established (Hello + HelloAck) or connectTotalTimeout elapses. Once established, no retry from the drain — a stream break ends the goroutine and removes the dedup entry, so a subsequent restart event re-dials.
func (*Dialer) DialAllRunning ¶
func (d *Dialer) DialAllRunning(ctx context.Context, lister *ContainerLister, opts ListOpts)
DialAllRunning dials every already-running agent container at CP boot. It spawns its own goroutine and returns immediately — it MUST NOT block CP readiness, and MUST NOT fail CP if the list call errors.
The list is retried with bounded exponential backoff (initialDialMaxAttempts attempts from initialDialListBackoff) to absorb the transient docker-daemon hiccup that dominates boot-time failures. Each resolved container ID is handed to the fire-and-forget DialAgent; the dialing dedup map keeps a later runtime ContainerStarted event from spinning a duplicate dial against an already-open Session.
The goroutine recovers: a panic deep in DialAgent (or the list path) must not strand the initial-poll dispatch silently or take down PID 1 and freeze eBPF — runtime ContainerStarted handlers are unaffected.
func (*Dialer) DispatchAgentEvents ¶ added in v0.12.9
func (d *Dialer) DispatchAgentEvents(ctx context.Context, containerID string, res EstablishResult, log *logger.Logger)
DispatchAgentEvents drives agent-axis event publication after the Session is up. Branches on the registry classification:
- Match → no extra event (SessionConnected already updated the trusted state; the durable Registered=true flag persists because the row exists)
- Miss → drive the Register handshake (send RegisterRequired, wait for RegisterDone, re-lookup), publish AgentRegistered and AgentUntrusted{ReasonRegisterFailed} on failure
- ThumbprintMismatch → publish AgentUntrusted{ReasonThumbprintMismatch}
- NotQueried (lookup error) → publish AgentUntrusted with detail
Caller must have already called publishConnected — the agent state projection is populated by the time we evaluate trust outcomes.
func (*Dialer) DriveRegister ¶ added in v0.12.9
func (d *Dialer) DriveRegister(ctx context.Context, containerID string, res EstablishResult, log *logger.Logger)
DriveRegister sends RegisterRequired on the Session stream and waits for the matching RegisterDone Response. After the agent reports completion, re-looks up the registry to confirm the row landed and publishes AgentRegistered (success or failure) plus AgentUntrusted on failure.
Concurrent-Recv safety: gRPC streams are NOT safe for concurrent stream.Recv. DriveRegister is called BEFORE drainStream and must guarantee that its inner Recv goroutine has fully exited before it returns — otherwise drainStream's Recv would race the leftover goroutine on the same stream. On timeout we cancel the stream-scoped ctx (res.StreamCancel) which unblocks the parked Recv with a ctx error; we then wait on `done` to confirm the goroutine exited. On the timeout path the stream is intentionally torn down — runDial will reconnect on the next cycle (a clawkerd that didn't respond to RegisterRequired in registerRequiredTimeout is unhealthy enough that re-establishing is the right reaction). Asymmetric trust is still preserved: we don't abort on cert grounds, only on protocol-level liveness.
func (*Dialer) PublishConnected ¶ added in v0.12.9
func (d *Dialer) PublishConnected(ctx context.Context, containerID, agent, project, addr string, attempt int, peer PeerInfo)
PublishConnected records that the Session handshake succeeded (mTLS + Hello + HelloAck) on the given attempt. peer carries the captured cert identity (PeerAgentFullName, PeerThumbprint) flat on the event. Trust/registration outcomes are published via separate events (AgentRegistered, AgentUntrusted).
type DrainResult ¶ added in v0.12.9
type DrainResult struct {
Outcome drainOutcome
Reason string
}
DrainResult is the typed return of drainStream. Reason carries a short classification string for the SessionBroken event when the outcome is drainGracefulEOF or drainStreamErr.
type Entry ¶
type Entry struct {
// AgentName is the user-typed short name. Typed (auth.AgentName)
// purely for compile-time discipline so the registry can't hold a
// raw string where an AgentName is expected. auth.NewAgentName
// rejects only the empty case today; charset/length/form
// constraints are enforced downstream (Docker container/volume
// create, x509 URI SAN encoding, IdentityInterceptor's symmetric
// SAN-vs-label compare). User-typed input is normalized upstream
// by cmdutil.ProjectSlugify before it crosses into auth.
// Constructed by the Register handler at the wire boundary via
// auth.NewAgentName; re-validated via auth.NewAgentName during
// sqlite Snapshot reads — rows with an empty agent_name column
// are skipped, never panicked on.
AgentName auth.AgentName
// Project is the clawker project slug under which the agent
// registered. The zero value (auth.ProjectSlug{}) signals a
// global-scope agent (no project namespace), matching the
// 2-segment docker.ContainerName shape. Typed for the same reason
// as AgentName; auth.NewProjectSlug accepts any input including
// empty.
Project auth.ProjectSlug
ContainerID string
Thumbprint [sha256.Size]byte
RegisteredAt time.Time
LastSeen time.Time
}
Entry is one registered agent. Created CP-side at Register handler entry carrying the ContainerID, AgentName, Project, and the SHA-256 over the peer cert DER (Thumbprint, captured live from the mTLS handshake). LastSeen currently equals RegisteredAt; future per-agent RPCs will refresh LastSeen at their own boundary.
type EstablishOutcome ¶ added in v0.12.9
type EstablishOutcome int
EstablishOutcome classifies the terminal state of one establishWithRetry call. Replaces the previous (gone bool, ok bool) pair where (ok=true, gone=true) was structurally illegal but compiled.
const ( // OutcomeSuccess: Hello + HelloAck completed. Conn / Stream are // non-nil; Attempt records the number of tries. OutcomeSuccess EstablishOutcome = iota )
type EstablishResult ¶ added in v0.12.9
type EstablishResult struct {
HelloAck *clawkerdv1.HelloAck
Conn *grpc.ClientConn
Stream clawkerdv1.ClawkerdService_SessionClient
StreamCancel context.CancelFunc
Agent string
Project string
Addr string
Attempt int
Outcome EstablishOutcome
PeerInfo PeerInfo
}
EstablishResult is the typed return of establishWithRetry. Conn / Stream are populated only when Outcome is outcomeSuccess. Agent / Project / Addr carry the latest inspect's view; Attempt records the cycle's attempt count (for both success and failure publishing). PeerInfo captures the connection-time identity outcomes (chain verify, CN pin, registry cross-check) populated during the TLS handshake + post-handshake registry lookup.
StreamCancel cancels the gRPC stream's underlying context. The dial flow uses ONE Recv consumer at a time: helloHandshake then either drainStream alone (Match) or driveRegister followed by drainStream (Miss / mismatch). When driveRegister times out it must cancel the stream to unblock its own Recv goroutine BEFORE returning — that avoids two concurrent stream.Recv() callers (gRPC streams are not safe for concurrent Recv). runDial defers StreamCancel at end of cycle so the stream is always torn down even on early returns.
type EventType ¶
type EventType string
EventType is the top-level discriminator for an AgentEvent: which CP-side subsystem produced it. Combined with Action it is the (Type, Action) discriminator the projection Stores switch on.
const ( DialerEventType EventType = "session" ExecutorEventType EventType = "exec" RegistryEventType EventType = "registry" )
Known event types. Each maps to one producer surface in this package: the dialer (session lifecycle), the executor (CP-driven plan dispatch), and the registry/trust axis (provenance + trust verdicts).
type ExecTarget ¶
ExecTarget identifies the agent the Executor is executing against. Threaded through every event so subscribers see consistent identity fields without re-deriving from the registry.
type Executor ¶
type Executor struct {
// contains filtered or unexported fields
}
Executor dispatches a static CP-driven plan (init or boot) against an open Session stream. Owns Recv during Run; the dialer's drainStream takes over after Run returns.
Plan idempotency contract: every Session establish runs the full plan, including reconnects after CP restart. Each step is idempotent (file-presence gates, append-if-missing, marker-file post-init), and AgentReady is no-op success when clawkerd already spawned the user CMD (spawnState's CAS rejects re-fork; handler replies Done{0}). Trade: a small volume of shell commands fires on every reconnect; gain: no per-container completed flag. Executor is shared across all containers — the dialer constructs one at CP boot and calls Run from a goroutine per DialAgent (one per agent container). Run holds no Executor-scoped mutable state: every call gets its own (ctx, stream, target) and drives its own stream's Recv loop in a single goroutine. The Run-owns-Recv invariant is per-stream (one Run, one stream, one Recv-driving goroutine), not per-Executor — concurrent Runs across different containers must not be serialized.
func NewExecutor ¶
func NewExecutor(topic *pubsub.Topic[AgentEvent], dockerCli *docker.Client, log *logger.Logger) (*Executor, error)
NewExecutor constructs an Executor. nil log is replaced with logger.Nop(). topic is required — Run publishes exec events unconditionally, so a nil topic is a wiring bug the caller must catch at construction. Returning an error lets the caller (the orchestrator) log the wiring bug to the structured log surface and degrade gracefully (executor = nil → dialer logs agent_<plan>_executor_unset per dial) instead of crashing CP and stranding the failure on os.Stderr where only `docker logs` sees it. Matches the nil-topic contract on agent.NewDialer for the dialer.
func (*Executor) KillAfterGrace ¶ added in v0.12.9
func (e *Executor) KillAfterGrace(ctx context.Context, containerID string, log *logger.Logger) error
KillAfterGrace ensures the agent container is torn down after a fatal command. A command carrying exit_on_non_zero makes a healthy clawkerd echo the output and self-exit PID 1 with the mirrored code, so CP waits consts.CPAgentKillGrace for that clean self-exit — keyed on real container liveness via ContainerWait — and escalates to SIGKILL only if the container is still running past the grace (a wedged clawkerd, or a timeout where clawkerd never self-exits). Generic to the CP→clawkerd command service; returns an error only when the SIGKILL itself fails, so the caller can report that the doomed container could not be torn down.
ctx is the CP-lifetime context (NOT the dial ctx — that one is cancelled by the very container/die this awaits). The wait rides ctx, so a CP shutdown during the grace abandons the wait and proceeds straight to the kill: a doomed container must not outlive CP. The kill itself runs on context.Background(), because if ctx is what woke the wait (shutdown), the moby client rejects a request on a cancelled ctx before it reaches the daemon — the SIGKILL would never issue and the container would leak.
func (*Executor) Run ¶
func (e *Executor) Run(ctx context.Context, stream clawkerdv1.ClawkerdService_SessionClient, target ExecTarget, plan []Step, label string) (runErr error)
type ExecutorEventState ¶
type ExecutorEventState struct {
// contains filtered or unexported fields
}
ExecutorEventState bundles the CP-driven exec phase fields for an Agent. Zero value means no exec phase has been observed (Status == ExecStatusUnknown); producers transition by emitting ExecutorEventState* events whose ApplyTo methods project here. Held as a sub-struct so the exec axis is isolated from the session axis (Session*, Address, Attempts) and the identity axis (Trust, Registered).
func ExecRunning ¶
func ExecRunning(stepCount int, at time.Time) ExecutorEventState
ExecRunning resets the substruct to an active phase: Status becomes Running, StartedAt records the phase boundary, StepCount is captured for streaming subscribers ("1 of N" rendering). Any stale step / completion / failure carried by a previous phase is dropped — a reconnect that re-runs the plan should not surface the prior terminal state. A negative stepCount is clamped to zero.
func (ExecutorEventState) Complete ¶
func (i ExecutorEventState) Complete(at time.Time) ExecutorEventState
Complete returns a terminal ExecutorEventState in Completed state, clearing LastError so a subscriber switching on Status sees a coherent success snapshot. CompletedAt is forced to be at least StartedAt so the (CompletedAt < StartedAt) projection bug is unrepresentable.
func (ExecutorEventState) CompletedAt ¶
func (i ExecutorEventState) CompletedAt() time.Time
func (ExecutorEventState) Fail ¶
func (i ExecutorEventState) Fail(at time.Time, detail string) ExecutorEventState
Fail returns a terminal ExecutorEventState in Failed state with detail. Same CompletedAt floor as Complete.
func (ExecutorEventState) LastError ¶
func (i ExecutorEventState) LastError() string
LastError carries the most recent exec-axis failure detail. Cleared on Complete; populated by WithStepError and Fail. Distinct from the session-axis Agent.LastError so the two failure surfaces don't overwrite each other.
func (ExecutorEventState) StartedAt ¶
func (i ExecutorEventState) StartedAt() time.Time
func (ExecutorEventState) Status ¶
func (i ExecutorEventState) Status() Status
func (ExecutorEventState) StepCount ¶
func (i ExecutorEventState) StepCount() int
func (ExecutorEventState) StepIndex ¶
func (i ExecutorEventState) StepIndex() int
func (ExecutorEventState) StepName ¶
func (i ExecutorEventState) StepName() string
StepName is the most recently started step's human-readable label. Empty until the first WithStep transition fires.
func (ExecutorEventState) WithStep ¶
func (i ExecutorEventState) WithStep(stepName string, stepIndex int) ExecutorEventState
WithStep returns a copy of i with StepName / StepIndex updated. StepIndex is clamped into [0, StepCount-1] when StepCount > 0; the (name, index) pair stays internally consistent because both are written together from the same event payload, so a clamp affects only the index — not the human-readable name subscribers display.
func (ExecutorEventState) WithStepError ¶
func (i ExecutorEventState) WithStepError(detail string) ExecutorEventState
WithStepError returns a copy of i with LastError set. Status is untouched — ExecStepFailed is mid-phase, the terminal transition is Fail.
type Handler ¶
type Handler struct {
agentv1.UnimplementedAgentServiceServer
Registry Registry
Log *logger.Logger
Clock func() time.Time
}
Handler serves the Register RPC on the CP's clawker network agent listener and is the SOLE writer of the agentregistry sqlite DB. IdentityInterceptor has already grounded the peer in the daemon-attested container identity (peer IP → purpose=agent container → labels → cross-checked cert SAN AgentFullName) and attached the resolved (containerID, project, agentName) to ctx. The handler reads that, captures the cert thumbprint at the gate that persists it, cross-checks the cert's container_id SAN + request fields against the resolved truth, and writes the registry row using label-derived (authoritative) values.
Trust ordering: daemon labels > cert claim > request claim. Persisting `resolved.*` keeps the registry aligned with the daemon view; the request body is treated as a client claim that must agree but never as the source of truth.
All identity-rejection paths return codes.PermissionDenied with a generic envelope; the structured log line carries the specific failure classification for operator triage. A missing resolved container in ctx is a wiring bug (interceptor not chained) and surfaces as codes.Internal.
func NewHandler ¶
NewHandler constructs a Register handler. registry MUST be non-nil (NPE on first Register otherwise — fail at construction so the regression surfaces at CP startup, not at the first agent boot). log defaults to logger.Nop() when nil. clock defaults to time.Now.
func (*Handler) Register ¶
func (h *Handler) Register(ctx context.Context, req *agentv1.RegisterRequest) (*agentv1.Welcome, error)
Register is the one-time-per-container handshake CP triggers via a RegisterRequired Command on the Session bidi stream. clawkerd presents a CLI-CA-signed cert and calls Register; the universal IdentityInterceptor has already pinned CN=ContainerClawkerd, resolved the peer IP to a purpose=agent container, and verified that the cert's urn:clawker:agent: SAN matches the label-derived AgentFullName. The handler captures the live mTLS peer's cert thumbprint, cross-checks the cert's urn:clawker:container: SAN and the request fields against the middleware-resolved identity, then writes the agentregistry row using the label-derived (authoritative) values.
Rejection paths (all return PermissionDenied unless noted; the classification is logged at Warn for operator triage):
- resolved container missing from ctx — middleware did not run on this RPC (wiring regression). Returns Internal.
- peer cert missing from ctx post-resolve — defense-in-depth; the interceptor pre-validates the cert, so reaching this branch means ctx was tampered between interceptor and handler.
- request agent_name or project malformed (InvalidArgument)
- request fields disagree with middleware-resolved labels (the client lying about its own identity in the RPC body, even though cert+labels agree)
- cert urn:clawker:container: SAN missing or doesn't match the middleware-resolved container_id
- existing row for this container_id has a different thumbprint (cert replay; CLI is the only legit cert source)
Idempotent retry: an existing row whose thumbprint matches the captured thumbprint returns Welcome silently — Session retries after the row was already written.
type ListOpts ¶
type ListOpts struct {
All bool
}
ListOpts controls the scope of a ContainerLister.List call. All=false (the zero value) restricts to running containers; All=true includes stopped/exited containers.
type Message ¶
type Message struct {
Type EventType `json:"type"`
Action Action `json:"action"`
Time int64 `json:"time"`
TimeNano int64 `json:"timeNano"`
// Classification shared across axes. Reason is the typed verdict;
// Detail is the human-readable specifics.
Reason Reason `json:"reason,omitempty"`
Detail string `json:"detail,omitempty"`
// Session-axis fields (DialerEventType).
Address string `json:"address,omitempty"`
Attempts int `json:"attempts,omitempty"`
PeerAgentFullName string `json:"peer_agent_full_name,omitempty"`
PeerThumbprint [sha256.Size]byte `json:"-"`
// Registry-axis fields (RegistryEventType).
RegisterOk bool `json:"register_ok,omitempty"`
// Exec-axis fields (ExecutorEventType).
StepName string `json:"step_name,omitempty"`
StepIndex int `json:"step_index,omitempty"`
StepCount int `json:"step_count,omitempty"`
ExitCode int32 `json:"exit_code,omitempty"`
Duration time.Duration `json:"duration,omitempty"`
}
Message is the unified, discriminated agent-event body. The (Type, Action) pair routes; the remaining fields are the union of every signal the legacy per-event structs carried (session address / attempts, peer identity, register outcome, exec step coordinates, classified Reason + Detail). A producer fills only the fields its (Type, Action) defines; readers tolerate zero values for the rest.
type MobyPeerLookup ¶
type MobyPeerLookup struct {
// contains filtered or unexported fields
}
MobyPeerLookup is the production ContainerByPeerIP backed by the Docker daemon.
func NewMobyPeerLookup ¶
func NewMobyPeerLookup(cli mobyclient.APIClient, log *logger.Logger) *MobyPeerLookup
NewMobyPeerLookup wraps a moby APIClient as a ContainerByPeerIP resolver. log defaults to logger.Nop() when nil.
func (*MobyPeerLookup) LookupByIP ¶
func (m *MobyPeerLookup) LookupByIP(ctx context.Context, ip netip.Addr) (ResolvedContainer, error)
LookupByIP walks every `purpose=agent` container and returns the one whose clawker network endpoint IP matches ip. The walk is exhaustive: ambiguous-IP advertisements (multiple containers with overlapping endpoint state during restart cycles) return ErrAmbiguousPeerIP rather than picking the first match. A transient per-container inspect failure is logged and iteration continues. A daemon-error wrap is returned only when NO candidate could be inspected — a real no-match is reported as ErrNoContainerForPeerIP even when one inspect along the way failed, so the peer_lookup_no_match audit signal stays useful.
type PeerInfo ¶ added in v0.12.9
type PeerInfo struct {
PeerAgentFullName string
PeerThumbprint [sha256.Size]byte
ChainVerified bool
// CaptureReason is set when capturePeer hit an unusual case (no
// peer certs, leaf parse failed, chain verify failed). Empty on
// the happy path. Stays purely diagnostic — the dialer never
// aborts on cert grounds.
CaptureReason string
}
PeerInfo is the connection-time identity capture from the TLS handshake. Registry-outcome state lives in the local RegistryOutcome enum, not on this struct, since the dial flow drives event publication directly off the outcome rather than threading a unified payload.
PeerAgentFullName is the AgentFullName ("clawker.<project>.<agent>") read from the peer's URI SAN (urn:clawker:agent:<agent_full_name>) — NOT from Subject.CommonName, which is the deterministic consts.ContainerClawkerd literal and carries no per-agent information.
type Reason ¶
type Reason string
Reason is the unified failure/trust classification. It collapses the legacy split between the dialer's UntrustedReason and the executor's ExecFailureReason into one vocabulary so a single Reason field on the AgentEvent message carries every classified outcome. Subscribers switch on it to enact policy (containment, alerting, eviction) and to distinguish the dialer-permissive asymmetric-trust verdicts (thumbprint mismatch / cert invalid / register failed) from exec failures. Empty string is the zero value: no classified failure.
const ( ReasonNone Reason = "" // Trust-axis reasons. Keep the dialer-permissive classifications // distinct so thumbprint mismatch / cert invalid / register failed // stay distinguishable to subscribers enacting trust policy. ReasonThumbprintMismatch Reason = "cert_thumbprint_mismatch" ReasonCertInvalid Reason = "cert_invalid" ReasonRegisterFailed Reason = "register_failed" // ReasonInitBypassed: the user CMD is running while init never ran — // a structurally-impossible lifecycle state (CP bug or out-of-band CMD // fork). See agentInitBypassed. ReasonInitBypassed Reason = "init_bypassed" // Exec-axis reasons: classified failure modes for a dispatched plan. ReasonExitCode Reason = "exit_code" ReasonTimeout Reason = "timeout" ReasonSpawnFailed Reason = "spawn_failed" ReasonIOError Reason = "io_error" ReasonTransportError Reason = "transport_error" ReasonProtocolError Reason = "protocol_error" // ReasonFailed is the session-axis generic dial failure (retry // exhausted, container gone, addr invalid, panic) — the dialer // records the human-readable specifics in Message.Detail. ReasonFailed Reason = "failed" // ReasonUnknown is the exec-axis fallback for an unrecognized // clawkerd ErrorCode. ReasonUnknown Reason = "unknown" )
type Registry ¶
type Registry interface {
// Add inserts an entry keyed by (Entry.Thumbprint, Entry.ContainerID).
// Container restart produces a new cert and a new thumbprint, so
// re-registration creates a new entry; the dockerevents subscription
// is responsible for evicting the stale one by container ID.
//
// Returns an error when the persistence layer (sqlite) rejects the
// write — disk full, schema corruption, UNIQUE collision against a
// stale row that hasn't been evicted yet. Callers translate the
// error into the appropriate gRPC status.
//
// Add returns an error on programming-error invariants (zero
// thumbprint, empty ContainerID, zero AgentName, zero
// RegisteredAt). Both Registry implementations (in-memory and
// sqlite) propagate the error from `validateEntry` rather than
// panicking — Add lives on a gRPC handler goroutine reachable
// post-SetReady from Register, and a panic on that path would
// strand eBPF programs with no supervisor (see root CLAUDE.md).
// Register maps the error to codes.Internal: every field
// validateEntry checks is server-derived (thumbprint from the
// live peer cert, ContainerID/AgentName/Project from
// IdentityInterceptor's ResolvedContainer, RegisteredAt from
// h.clock()), so a failure here is a CP wiring bug, not bad
// client input. User-controlled identity strings are validated
// upstream at the wire boundary (auth.NewProjectSlug /
// auth.NewAgentName → codes.InvalidArgument).
Add(entry Entry) error
// LookupByContainerID returns the entry whose ContainerID matches.
// Used by the Register handler (idempotency / replay-protection)
// and by the dialer at Hello time to drive registry classification
// (Match / Miss / ThumbprintMismatch) and decide whether to send
// RegisterRequired or publish AgentUntrusted. The dialer performs
// the thumbprint comparison against the returned entry itself;
// this read intentionally does not gate on thumbprint so a
// mismatch surfaces as a typed local outcome rather than as
// ErrUnknownAgent.
//
// Returns (nil, ErrUnknownAgent) when no entry matches.
LookupByContainerID(containerID string) (*Entry, error)
// EvictByContainerID removes any entry whose ContainerID matches.
// Returns the underlying persistence error so callers can decide
// whether to retry, log, or abort. The dockerevents-driven and
// reaper-driven callers log-and-proceed because a transient sqlite
// failure must not stall the eviction pipeline; CLI-side `clawker
// container remove` similarly logs at debug since registry hiccups
// must not surface as remove failures (the row gets pruned later
// by the dockerevents subscription).
EvictByContainerID(containerID string) error
// Snapshot returns a copy of every live entry, sorted by
// (Project, AgentName) for deterministic output. Project is the
// primary sort key because the same short AgentName can be reused
// across different projects (the composite identity is
// (project, agent)). Used by AdminService.ListAgents and the
// `clawker controlplane agents` CLI; both rely on stable ordering
// for diffability.
//
// Returns a non-nil error on persistence failure (sqlite db.Query
// or rows.Err non-nil). Callers must NOT treat an empty result as
// authoritative when err != nil — reapOrphans would otherwise
// evict every registered agent on a transient query failure;
// ListAgents would surface "no agents" to operators while the
// registry is intact but unreadable.
Snapshot() ([]Entry, error)
}
Registry is the consumer-facing contract.
func NewRegistry ¶
NewRegistry constructs an empty in-memory registry. Logger is required (use logger.Nop() in tests) so audit-trail messages on Add and Evict are captured even when production logging is otherwise disabled.
func NewSQLiteWriter ¶
NewSQLiteWriter opens (or creates) the sqlite database at dbPath, applies the schema, and returns a registry suitable for the writer process. CP is the SOLE writer in this design (the CLI no longer opens the registry DB), so a separate reader-mode constructor is not provided — every consumer that previously read from sqlite now goes through `f.AdminClient(ctx).ListAgents`.
The parent directory of dbPath must already exist.
type RegistryOutcome ¶ added in v0.12.9
type RegistryOutcome int
RegistryOutcome classifies the result of cross-checking the peer cert against the agentregistry row keyed by container_id. Internal to the dialer — drives event publication; not exposed on any public event type.
const ( // OutcomeRegistryNotQueried is the zero value. Set when the // registry could not be queried at all (lookup error). OutcomeRegistryNotQueried RegistryOutcome = iota // OutcomeRegistryMatch — row exists and thumbprint agrees with // the peer cert. Trusted, registered. OutcomeRegistryMatch // OutcomeRegistryMiss — no row for this container_id. Drives the // Register handshake (RegisterRequired Command on the Session // stream). OutcomeRegistryMiss // OutcomeRegistryThumbprintMismatch — row exists but thumbprint // disagrees with the live peer cert. Untrusted; AgentUntrusted // fires with ReasonThumbprintMismatch. OutcomeRegistryThumbprintMismatch )
type Repository ¶
type Repository struct {
Agents *AgentStore
}
Repository is the agent bounded context's storage repository. It aggregates the domain's Stores (currently just AgentStore) behind one Subscribe call the orchestrator wires. Future agent-axis stores (a per-project rollup, a trust-event log) join here.
func NewRepository ¶
func NewRepository() *Repository
NewRepository constructs the agent repository with empty stores.
func (*Repository) Subscribe ¶
func (r *Repository) Subscribe(topic *pubsub.Topic[AgentEvent])
Subscribe wires every store in the repository to the agent Topic.
func (*Repository) SubscribeDockerEvents ¶
func (r *Repository) SubscribeDockerEvents(topic *pubsub.Topic[dockerevents.DockerEvent])
SubscribeDockerEvents wires every store's eviction path to the dockerevents Topic so a destroyed container's projected worldview is reclaimed rather than leaked. The orchestrator calls this alongside Subscribe so the worldview is both written (agent events) and bounded (docker destroy).
type ResolvedContainer ¶
type ResolvedContainer struct {
ContainerID string
Project auth.ProjectSlug
AgentName auth.AgentName
}
ResolvedContainer is the authoritative identity of an agent container as resolved from its live mTLS peer IP. Project and AgentName are typed so a zero-value instance cannot carry empty strings into a downstream identity comparison; the resolver constructs these via auth.NewProjectSlug / auth.NewAgentName and rejects malformed labels at the seam.
func ResolvedContainerFromContext ¶
func ResolvedContainerFromContext(ctx context.Context) (ResolvedContainer, bool)
ResolvedContainerFromContext returns the peer-IP-resolved container identity IdentityInterceptor attached to ctx. ok=false means the interceptor did not run on this RPC — a wiring bug for any AgentService handler that consumes resolved identity. Defensive against a zero-value resolved container (empty ContainerID): treats it as ok=false so handlers can rely on ok=true meaning a non-empty resolved container is available.
type ShellStep ¶ added in v0.12.9
type ShellStep struct {
Name string
Shell *clawkerdv1.ShellCommand
}
type StartDeps ¶
type StartDeps struct {
Registry Registry
DockerLister ContainerListFunc
PeerLookup ContainerByPeerIP
Dialer *Dialer
DockerTopic *pubsub.Topic[dockerevents.DockerEvent]
AgentTopic *pubsub.Topic[AgentEvent]
Log *logger.Logger
}
StartDeps bundles the dependencies the umbrella `Start` procedure needs. The topics + Registry + Dialer + Docker lister + peer-IP resolver are all owned by the CP orchestrator; passing them as a single struct keeps the call site a single function call. All fields are required.
DockerTopic is the dockerevents domain's Topic[DockerEvent]: the evict, session-cancel, and dial subscribers attach to it with their predicates folded into the handler closures. AgentTopic is this domain's own Topic[AgentEvent], used by the reap-degraded notification (the only AgentEvent Start itself publishes).
type Status ¶
type Status string
Status is the projected session/exec lifecycle state a Store derives from the (Type, Action) of an AgentEvent. Disjoint vocabulary from Action — Action is the wire verb, Status is the worldview state.
const ( StatusUnknown Status = "unknown" StatusUntrusted Status = "untrusted" StatusConnecting Status = "connecting" StatusConnected Status = "connected" StatusBroken Status = "broken" StatusFailed Status = "failed" StatusDegraded Status = "degraded" StatusHealthy Status = "healthy" StatusRunning Status = "running" StatusCompleted Status = "completed" )
type Step ¶ added in v0.12.9
type Step interface {
StepName() string
// Command builds the wire payload for this Step under commandID.
// followCloseStdin reports whether runStep should follow with a
// CloseStdin frame (true for shell steps that don't consume
// stdin; false for AgentReady which has no stdin pipe).
Command(commandID string) (cmd *clawkerdv1.Command, followCloseStdin bool)
// IsStep is the unexported sealing marker. A third implementer
// outside this package is rejected at compile time; package-
// internal additions still need a paired runStep / plan() update
// by convention.
IsStep()
}
Step is one entry in an Executor plan (init or boot). Sealed sum: ShellStep, AgentReadyStep, or AgentInitializedStep. Adding a new Step kind is a compile-time change (implement Step) — runStep's type switch loses its runtime "unknown Step kind" branch entirely.
type Trust ¶
type Trust struct {
// contains filtered or unexported fields
}
Trust bundles the trust verdict for an Agent so the "is the agent trusted?" and "if not, why?" questions are a single invariant-respecting value rather than a (bool, enum) pair where the zero value is structurally illegal ("untrusted with no reason" is not a thing the worldview should be able to represent).
Zero value = trusted with no reason. That's the right default: an Agent struct freshly inserted into State.Agents has not yet been proven untrustworthy, and the consumer-facing API (IsTrusted, Reason) returns sensible answers immediately. Producers transition to untrusted via Untrust(reason); that's the only path that flips the verdict.
func Untrust ¶
Untrust constructs a Trust value carrying the classification. The reason MUST be non-empty; an empty reason is rejected by returning a Trust whose IsTrusted is still true (no producer should call Untrust(UntrustedReasonNone) — that's a logic bug).