agent

package
v0.12.8 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 20, 2026 License: AGPL-3.0 Imports: 47 Imported by: 0

Documentation

Overview

Dialer-side helpers in package agent: CP-side outbound mTLS dial logic for the CP→clawkerd Session channel.

Single entry point: Dialer.DialAgent(ctx, containerID). The same function is invoked at CP boot (over the result of listAgentIDs) and from the typed-event subscriber on dockerevents.DockerEvent (filtered to container start/restart/unpause for purpose=agent containers) when an agent container reaches running state at runtime — so two callers, one dial path.

DialAgent is fire-and-forget: it spawns a goroutine that owns the dial, the Session stream, and the lifetime drain loop. All failures are logged at the Error level — callers don't need to handle errors. A failed dial leaves no resources behind; a successful dial is held open until ctx cancels (CP shutdown) or the peer closes.

Asymmetric trust model — load-bearing

CP is the overlord. The dial NEVER aborts on cert / identity grounds. Cert chain verification and registry thumbprint classification outcomes are captured on the establishResult and surfaced through the typed event surface — SessionConnected carries flat PeerAgentFullName/PeerThumbprint fields (purely diagnostic, never a gate); AgentRegistered/AgentUntrusted carry the policy outcomes. Subscribers consume those events to enact policy (containment, alerting, eviction); the dialer holds no policy itself.

Why permissive: CP must always be able to reach clawkerd to issue containment commands (iptables lock, network detach, container kill). A compromised clawkerd presenting a bad cert is exactly when the channel must be UP so CP can react. Aborting on cert grounds would strand CP exactly at the moment governance is most needed.

The asymmetric counterpart lives in clawkerd/listener.go: the clawkerd-side listener is STRICT — CP CN pin + Client-Auth EKU + CA chain enforced at TLS layer. clawkerd refuses any peer that isn't CP. This pairs with the dialer's permissive client posture.

Connection establishment failures still happen on connectivity grounds (TCP timeout, container gone, retry exhausted, ctx cancelled) — those drive establishOutcome and SessionFailed.

FD-leak ceiling: a successful dial whose conn.Close() repeatedly fails would accumulate file descriptors and gRPC keepalive goroutines indefinitely. After closeErrCeiling consecutive close failures the dial loop bails for the target with a SessionFailed event (Reason carries the "fd-leak-ceiling" classification) so operators see the outcome instead of a silent leak. A successful close anywhere in the loop resets the counter.

Identity resolution flow for AgentService RPCs:

  1. The CP's agent listener enforces mTLS at the TLS layer (server cert + ClientAuth EKU + chain to the CLI CA).
  2. AuthInterceptor verifies the bearer token + per-method scope.
  3. IdentityInterceptor (this package) runs a universal three-stage gate on EVERY RPC including Register: (a) Subject.CommonName == consts.ContainerClawkerd (constant-time). The cert CN is the deterministic clawkerd binary identity; mismatch means the peer is not presenting a CLI-minted agent cert. (b) Resolve the kernel-attested peer IP to the purpose=agent container that owns it on the clawker network (via ContainerByPeerIP), reading the project/agent labels as the authoritative identity source — Docker is independent ground truth. (c) Compose the label-derived AgentFullName and constant-time compare it against the cert's urn:clawker:agent: URI SAN. The cert's SAN claim is VERIFIED against the IP-grounded label truth — never the basis of lookup.
  4. The resolved (containerID, project, agentName) is attached to ctx via WithResolvedContainer; downstream handlers read it via ResolvedContainerFromContext to avoid re-inspecting the container.

All rejections return codes.PermissionDenied with a generic envelope — no leak about which check failed.

Package agent owns the CP-side agent surface: the persisted identity registry, the AgentService.Register handler, the per-RPC IdentityInterceptor, the CP→clawkerd Session dialer, and the AgentRegistered/AgentUntrusted event types.

Registry rows record the (mTLS cert thumbprint, container_id, project, agent_name) tuple. Reads are issued by:

  • the Register handler (idempotency / replay-protection at row write time, keyed by container_id)
  • the dialer at Hello time (LookupByContainerID → drives AgentRegistered / AgentUntrusted publication via thumbprint compare against the live peer cert)
  • AdminService.ListAgents (Snapshot)

Writes are CP-only: the Register handler captures the live mTLS peer's thumbprint and writes the row. Eviction: startup orphan-row reap (in agent.Start), plus dockerevents container/destroy (subscribed in agent.Start). Stop/die/kill do NOT evict because a stopped container can be `docker start`-ed back into life.

Identity is channel-bound: the registry key is `(thumbprint, container_id)` (both UNIQUE in sqlite). Agent-full-name composition from `(project, agent_name)` is no longer persisted — the IdentityInterceptor establishes the trust anchor via the kernel-attested peer IP and cross-checks the cert SAN against the label-derived AgentFullName at the gRPC boundary, so the registry holds only the per-row identity tuple.

Index

Constants

This section is empty.

Variables

View Source
var ErrAmbiguousPeerIP = errors.New("multiple purpose=agent containers match peer IP")

ErrAmbiguousPeerIP is returned when two or more `purpose=agent` containers on the clawker network advertise endpoints with the same peer IP. Docker can transiently leave stale endpoints in NetworkSettings.Networks during restart cycles, and grounding the trust anchor on the first match would create a race window. Fail closed instead — operators see a distinct event and the trust gate rejects the RPC until the daemon state converges.

View Source
var ErrInvalidAgentLabel = errors.New("agent container has invalid identity label")

ErrInvalidAgentLabel is returned when the container matched by peer IP carries a missing or malformed dev.clawker.agent label. A missing dev.clawker.project label is NOT an error here — it is the legitimate global-scope-agent signal (no project namespace, producing 2-segment naming clawker.<agent>); auth.NewProjectSlug("") returns a zero-value slug with nil err. Distinguishing this from a clean no-match lets the trust gate emit a daemon-state diagnostic instead of a generic auth reject.

View Source
var ErrMalformedEntry = errors.New("agentregistry: malformed registry row")

ErrMalformedEntry is returned by sqlite reads when a row's agent_name / project / thumbprint fails re-validation (a value that landed pre-typed-boundary, a hand-edited DB, or a corrupted column). The Register handler treats this sentinel as "evict + re-write": the row's identity is unusable so it gets purged and replaced by the typed identity the middleware just resolved off the live peer IP + cert. Wrapped via fmt.Errorf so errors.Is on the wrapped error works for callers that don't unwrap themselves.

View Source
var ErrNoContainerForPeerIP = errors.New("no purpose=agent container with matching " + consts.Network + " IP")

ErrNoContainerForPeerIP is returned when no `purpose=agent` container on the clawker network has an endpoint IP matching the requested peer IP. Callers MUST treat this as a hard authentication failure.

View Source
var ErrUnknownAgent = errors.New("agentregistry: unknown agent")

ErrUnknownAgent is returned by LookupByContainerID when no entry matches. Thumbprint compare against the live peer cert is the dialer's job, not the registry's.

View Source
var ErrWatcherMisconfig = errors.New("controlplane: AgentWatcher misconfigured")

ErrWatcherMisconfig is returned by NewAgentWatcher when a required callback is nil or an option is negative. The CP serve path never panics on misconfig — the orchestrator surfaces this as a structured event=agent_watcher_unavailable line and degrades.

Functions

func EnsureSchema

func EnsureSchema(dbPath string, log *logger.Logger) error

EnsureSchema opens the database in writer mode (creating the file if missing), applies the schema, and closes. Called by the CP daemon at startup before NewSQLiteWriter so the schema apply path is observable independent of the long-lived writer handle.

func IdentityInterceptor

IdentityInterceptor returns paired unary and stream interceptors that enforce the universal identity gate documented at the top of handler.go. Wired AFTER AuthInterceptor on the agent listener so token validation runs first.

peerLookup is required — a nil resolver would silently disable the trust gate and admit every RPC. Returns a non-nil error rather than panicking: this constructor runs post-eBPF-load in clawkercp.go, so a panic here would strand pinned eBPF programs with no supervisor. clawkercp.go logs the error structurally (event=agent_identity_unavailable) and refuses to bring up the agent listener, degrading the AgentService surface while CP + firewall + admin listener stay up.

log defaults to logger.Nop() if nil so the panic-recovery path of grpc-go never sees a nil deref.

Every rejection returns codes.PermissionDenied with the same generic envelope ("registration rejected") — attackers must not learn which check failed. The structured log carries the classification via a unique event= field per stage.

func Start

func Start(ctx context.Context, deps StartDeps) (func(), error)

Start gathers the full agent worldview at CP boot and wires every ongoing agent-axis subscription onto the dockerevents topic. Called once from the orchestrator after the topics are up. Returns a cleanup func; the per-subscriber delivery goroutines are owned by the topic itself (the orchestrator's topic.Close() tears them down), so the returned cleanup is a no-op kept for call-site symmetry.

Steps, in order:

  1. Reap orphan registry rows. List every purpose=agent container (All:true, includes stopped). For each registry row whose container_id is missing from docker, evict. Heals the registry against `docker rm`s that landed while CP was down.
  2. Subscribe to the dockerevents topic for the evict path — predicate container/destroy folded into the handler; consumer evicts the registry row.
  3. Subscribe to the dockerevents topic for the session-cancel path — predicate container/{die,stop,kill,oom,destroy}; consumer calls dialer.CancelDial so the in-flight Session tears down immediately.
  4. Subscribe to the dockerevents topic for the dial path — predicate container/start|restart|unpause with purpose=agent; consumer calls dialer.DialAgent.

func WithResolvedContainer

func WithResolvedContainer(ctx context.Context, resolved ResolvedContainer) context.Context

WithResolvedContainer attaches the peer-IP-resolved container identity to ctx for downstream handlers. Exposed so test code and future identity-augmenting interceptors can attach a resolved container directly; production code never needs to call this (the interceptor does it on success).

A zero-value ResolvedContainer (empty ContainerID) is silently dropped — ctx is returned unchanged so downstream ResolvedContainerFromContext sees ok=false. The CP must not panic on the serving path (root CLAUDE.md security contract); the read-side defensive ok=false is the floor against the silent-identity-vacuum hazard.

Types

type Action

type Action string

Action is the per-EventType verb. The (Type, Action) pair is the full discriminator the projection Stores switch on to mutate their own worldview.

const (
	ActionConnecting Action = "connecting"
	ActionConnected  Action = "connected"
	ActionFailed     Action = "failed"
	ActionBroken     Action = "broken"
)

Session-axis actions (DialerEventType).

const (
	ActionExecStarted       Action = "started"
	ActionExecStepStarted   Action = "step_started"
	ActionExecStepCompleted Action = "step_completed"
	ActionExecStepFailed    Action = "step_failed"
	ActionExecFailed        Action = "exec_failed"
	ActionExecCompleted     Action = "completed"
)

Exec-axis actions (ExecutorEventType). The plan dispatch lifecycle: the plan starts, each step starts/completes/fails, and the plan reaches a terminal completed/failed state.

const (
	ActionRegistered Action = "registered"
	ActionUntrusted  Action = "untrusted"
	ActionReap       Action = "reap_degraded"
)

Registry/trust-axis actions (RegistryEventType).

type Agent

type Agent struct {
	ContainerID   string `json:"container_id"`
	ContainerName string `json:"container_name"`
	AgentName     string `json:"agent_name"`
	Project       string `json:"project"`
}

Agent is the stable identity triple every AgentEvent carries so a subscriber sees consistent (container, agent, project) fields without re-deriving them from the registry.

type AgentEvent

type AgentEvent struct {
	Agent   Agent
	Message Message
}

AgentEvent is the single unified payload on the agent Topic. It rides inside pubsub.Event[AgentEvent]; the pipe never inspects it. The (Type, Action) discriminator on Message is the routing identity, and the projection Stores read Message fields natively to build their own worldview.

func (AgentEvent) MarshalZerologObject

func (e AgentEvent) MarshalZerologObject(z *zerolog.Event)

MarshalZerologObject surfaces the (Type, Action) discriminator plus the classification + identity fields so the single generic audit hook in controlplane/pubsub can log per-event detail without reflection. Zero-value fields are omitted to keep lines compact.

type AgentEventState

type AgentEventState struct {
	ContainerID       string
	AgentName         string
	Project           string
	Address           string
	SessionStatus     Status
	Registered        bool
	Trust             Trust
	Thumbprint        [sha256.Size]byte
	PeerAgentFullName string
	Attempts          int
	LastError         string
	UpdatedAt         time.Time
	Executor          ExecutorEventState
}

AgentEventState is the agent domain's own in-memory worldview of one clawker-managed agent, held in this domain's AgentStore. Three axes — session (SessionStatus, Address, Attempts, LastError, Thumbprint), identity (Registered, Trust), and exec (ExecutorEventState) — held as a single entity. The agentregistry sqlite store remains the durable truth source for identity rows; this struct is the observed-now view derived from the AgentEvent stream.

LastError is the SESSION-axis last error (dial failures, broken streams). ExecutorEventState-axis failures land in ExecutorEventState.LastError. Trust zero value is "trusted with no reason" (see Trust).

type AgentStore

type AgentStore struct {
	// contains filtered or unexported fields
}

AgentStore is this domain's own worldview projection: a thread-safe map of container_id -> AgentEventState, mutated ONLY by its own subscriber callbacks. The agent domain owns its observed worldview rather than writing into a shared central store. Cross-domain consumers never read this directly; if another domain wants agent state it subscribes to the agent Topic and builds its own projection.

Distinct from the agentregistry sqlite rows (the durable, attested identity axis): this store is the observed-now axis derived from the live AgentEvent stream and is cleared on CP restart.

func NewAgentStore

func NewAgentStore() *AgentStore

NewAgentStore constructs an empty store.

func (*AgentStore) Get

func (s *AgentStore) Get(containerID string) (AgentEventState, bool)

Get returns a copy of the projected state for a container and whether it exists. The returned value is a copy (AgentEventState is a value type with value-type sub-structs), so callers cannot mutate the store.

func (*AgentStore) Len

func (s *AgentStore) Len() int

Len reports how many agents the store currently tracks.

func (*AgentStore) Subscribe

func (s *AgentStore) Subscribe(topic *pubsub.Topic[AgentEvent])

Subscribe wires this store to the agent Topic. Every AgentEvent is projected into the store via project. A nil topic is a no-op so a degraded orchestrator (topic construction failed) does not NPE.

func (*AgentStore) SubscribeDockerEvents

func (s *AgentStore) SubscribeDockerEvents(topic *pubsub.Topic[dockerevents.DockerEvent])

SubscribeDockerEvents wires this store's eviction path to the dockerevents topic so the observed-now worldview stays bounded. The container/destroy predicate is folded into the handler (the pipe has no SubscribeFiltered): on docker rm of a container, the store drops its projected entry. Only destroy evicts — die/stop/kill leave the container docker start-able, and the worldview re-projects from the next session event, so evicting on a mere exit would churn the map for a container that is coming back. This matches the registry's destroy-only evict semantics (see subscribeEvict). A nil topic is a no-op so a degraded orchestrator does not NPE.

type AgentWatcher

type AgentWatcher struct {
	// contains filtered or unexported fields
}

AgentWatcher polls the Docker daemon for clawker-managed agent containers and invokes a drain-to-zero callback once the agent count has been zero across MissedThreshold consecutive polls past the grace period. The CP uses this to self-terminate when no agents remain — there is no reason to keep the control plane (Envoy + CoreDNS + CP daemon) alive after the last agent drains.

INV-B2-007: strict drain-to-zero ordering is enforced by the caller via the onDrainToZero callback. The watcher guarantees only that the callback fires at most once, before Run returns.

func NewAgentWatcher

func NewAgentWatcher(
	log *logger.Logger,
	listAgents func(context.Context) (int, error),
	onDrainToZero func(context.Context) error,
	opts AgentWatcherOptions,
) (*AgentWatcher, error)

NewAgentWatcher constructs an AgentWatcher. A nil callback or a negative option value returns (nil, ErrWatcherMisconfig) — misconfig must fail loudly but NEVER via panic, because a panic on the CP serve path kills PID 1 and strands eBPF programs unsupervised (see the resilience contract). The orchestrator degrades the subsystem on error instead.

func (*AgentWatcher) Run

func (w *AgentWatcher) Run(ctx context.Context) error

Run blocks until ctx is cancelled, the drain-to-zero condition fires, or consecutive list errors exceed ListErrCeiling. On drain, Run invokes onDrainToZero synchronously and returns its error. On ctx cancel, returns ctx.Err(). On error ceiling, returns a wrapped error surfacing the last list failure.

The grace period is measured on wall-clock time from Run entry — any zero-count polls during the grace window count toward the miss streak, but the streak cannot reach the threshold before grace expires. This prevents a race where the CP starts up before any agents have been enrolled and immediately drains.

Run must be called at most once per watcher; a second call returns an error rather than spinning up a second poll loop.

type AgentWatcherOptions

type AgentWatcherOptions struct {
	PollInterval    time.Duration
	MissedThreshold int
	GracePeriod     time.Duration
	// ListErrCeiling bounds how many consecutive listAgents errors the
	// watcher tolerates before returning an error from Run. Prevents
	// "Docker is wedged" from leaving the CP blind and permanent.
	ListErrCeiling int
}

AgentWatcherOptions overrides default watcher tuning. Zero values select defaults (PollInterval 30s, MissedThreshold 2, GracePeriod 60s, ListErrCeiling 20). Negative values are a construction error — silently snapping to a default would hide caller misconfig.

type ContainerByPeerIP

type ContainerByPeerIP interface {
	LookupByIP(ctx context.Context, ip netip.Addr) (ResolvedContainer, error)
}

ContainerByPeerIP resolves a live mTLS peer IP to the `purpose=agent` container owning that IP on the clawker network. Returns ErrNoContainerForPeerIP when nothing matches, ErrInvalidAgentLabel when the matching container's dev.clawker.agent label is missing or malformed (an absent dev.clawker.project label is a legitimate global-scope-agent signal, not an error), ErrAmbiguousPeerIP when two or more containers share the peer IP (Docker restart-race window — fails closed), or a wrapped daemon error.

type ContainerListFunc

type ContainerListFunc func(ctx context.Context) ([]string, error)

ContainerListFunc enumerates every `purpose=agent` container ID currently known to the docker daemon. The implementation MUST include stopped/exited containers — a stopped container can be `docker start`-ed back into life, and its registry row should survive that transition. Only `docker rm` (destroy) means the container is genuinely gone and the row is orphaned.

type ContainerLister

type ContainerLister struct {
	// contains filtered or unexported fields
}

ContainerLister enumerates managed `purpose=agent` container IDs from the docker daemon. It is the production source behind every agent-axis container listing in the CP orchestrator:

  • Firewall handler + AgentWatcher list running-only (ListOpts{}).
  • Reaper / initial-dial reconciler list running + stopped (ListOpts{All: true}) — a stopped container can be `docker start`-ed back into life, and its registry row should survive that transition.

The label filter is non-overridable: every List call narrows to BOTH the managed label AND purpose=agent, so a caller can't accidentally widen scope past purpose=agent. Construct once at CP startup; share across consumers.

func NewContainerLister

func NewContainerLister(dc mobyclient.APIClient, cfg config.Config) *ContainerLister

NewContainerLister wraps a moby APIClient as the managed-agent container enumerator. The label filter keys/values are read from cfg so the lister stays consistent with the rest of the CP's resource filtering.

func (*ContainerLister) List

func (l *ContainerLister) List(ctx context.Context, opts ListOpts) ([]string, error)

List returns the container IDs of every managed purpose=agent container matching opts. The label filter is non-overridable — both the managed label and purpose=agent are always applied so the scope can never widen past purpose=agent.

type Dialer

type Dialer struct {
	// contains filtered or unexported fields
}

Dialer captures the CP-side material every dial needs. Construct once at CP startup; share across all agent dials.

dialing is the dedup + cancel map: containerIDs currently being dialed (or already-Session-established) mapped to the cancel func for their per-dial ctx. Initial poll and the dockerevents subscriber both call DialAgent for the same running container; the dedup keeps the second call from spinning a duplicate goroutine against an already-open Session. Membership lasts the lifetime of the dial goroutine — after the Session closes (peer drop, ctx cancel, retry timeout), the entry is removed and a future event for the same containerID dials fresh.

CancelDial uses the stored cancel func to tear down a Session synchronously with a registry-evict (container/destroy) — without it, the dialer's runDial loop only notices the disappearance on the next reconnect attempt via outcomeContainerGone, leaving a doomed stream open during the interval.

func New

func New(log *logger.Logger, docker mobyclient.APIClient, topic *pubsub.Topic[AgentEvent], agents Registry, certPath, keyPath string, caPool *x509.CertPool, initExec *Executor) (*Dialer, error)

New constructs a Dialer. Returns an error if the CP client cert / key cannot be loaded — better to fail at CP startup than to defer the failure to the first dial. Returns (nil, error) on any nil required dependency; never panics, per the CP serve-path contract.

topic is required: the dialer publishes the unified AgentEvent (session connecting/connected/failed/broken, registry registered, trust untrusted) so other CP domains can subscribe to the agent axis without coupling to the dialer directly. Pass a real *pubsub.Topic[AgentEvent]; tests can construct one cheaply.

agents is required: every successful dial cross-checks the peer cert against the registry row keyed by container_id and dispatches the registered / untrusted AgentEvents accordingly. nil agents would strand worldview consumers without a registration signal.

func (*Dialer) CancelDial

func (d *Dialer) CancelDial(containerID string)

CancelDial synchronously cancels the in-flight Session for containerID, if any. Called by the registry-evict subscriber on container/destroy so the dialer tears down the doomed stream immediately rather than waiting for the next reconnect to classify outcomeContainerGone. Safe to call when no dial is in flight (no-op) and concurrent-safe (mu-guarded). The goroutine's own cleanup runs the deferred delete after runDial returns.

func (*Dialer) DialAgent

func (d *Dialer) DialAgent(ctx context.Context, containerID string)

DialAgent opens a Session stream to the clawkerd listener inside the given agent container, sends Hello, awaits HelloAck, and holds the stream open until ctx cancels or the peer closes. Returns immediately — the dial + lifetime drain run on a background goroutine. All failures are logged.

Dedup: a no-op if a dial for the same containerID is already in flight. Initial poll + dockerevents subscriber both reach this function with overlapping IDs at CP startup; dedup keeps a second call from spinning a duplicate goroutine.

Retry: the dial goroutine retries connection establishment with exponential backoff + full jitter (cap connectMaxBackoff) until either the Session is established (Hello + HelloAck) or connectTotalTimeout elapses. Once established, no retry from the drain — a stream break ends the goroutine and removes the dedup entry, so a subsequent restart event re-dials.

func (*Dialer) DialAllRunning

func (d *Dialer) DialAllRunning(ctx context.Context, lister *ContainerLister, opts ListOpts)

DialAllRunning dials every already-running agent container at CP boot. It spawns its own goroutine and returns immediately — it MUST NOT block CP readiness, and MUST NOT fail CP if the list call errors.

The list is retried with bounded exponential backoff (initialDialMaxAttempts attempts from initialDialListBackoff) to absorb the transient docker-daemon hiccup that dominates boot-time failures. Each resolved container ID is handed to the fire-and-forget DialAgent; the dialing dedup map keeps a later runtime ContainerStarted event from spinning a duplicate dial against an already-open Session.

The goroutine recovers: a panic deep in DialAgent (or the list path) must not strand the initial-poll dispatch silently or take down PID 1 and freeze eBPF — runtime ContainerStarted handlers are unaffected.

type Entry

type Entry struct {
	// AgentName is the user-typed short name. Typed (auth.AgentName)
	// purely for compile-time discipline so the registry can't hold a
	// raw string where an AgentName is expected. auth.NewAgentName
	// rejects only the empty case today; charset/length/form
	// constraints are enforced downstream (Docker container/volume
	// create, x509 URI SAN encoding, IdentityInterceptor's symmetric
	// SAN-vs-label compare). User-typed input is normalized upstream
	// by cmdutil.ProjectSlugify before it crosses into auth.
	// Constructed by the Register handler at the wire boundary via
	// auth.NewAgentName; re-validated via auth.NewAgentName during
	// sqlite Snapshot reads — rows with an empty agent_name column
	// are skipped, never panicked on.
	AgentName auth.AgentName
	// Project is the clawker project slug under which the agent
	// registered. The zero value (auth.ProjectSlug{}) signals a
	// global-scope agent (no project namespace), matching the
	// 2-segment docker.ContainerName shape. Typed for the same reason
	// as AgentName; auth.NewProjectSlug accepts any input including
	// empty.
	Project      auth.ProjectSlug
	ContainerID  string
	Thumbprint   [sha256.Size]byte
	RegisteredAt time.Time
	LastSeen     time.Time
}

Entry is one registered agent. Created CP-side at Register handler entry carrying the ContainerID, AgentName, Project, and the SHA-256 over the peer cert DER (Thumbprint, captured live from the mTLS handshake). LastSeen currently equals RegisteredAt; future per-agent RPCs will refresh LastSeen at their own boundary.

type EventType

type EventType string

EventType is the top-level discriminator for an AgentEvent: which CP-side subsystem produced it. Combined with Action it is the (Type, Action) discriminator the projection Stores switch on.

const (
	DialerEventType   EventType = "session"
	ExecutorEventType EventType = "exec"
	RegistryEventType EventType = "registry"
)

Known event types. Each maps to one producer surface in this package: the dialer (session lifecycle), the executor (CP-driven plan dispatch), and the registry/trust axis (provenance + trust verdicts).

type ExecTarget

type ExecTarget struct {
	ContainerID string
	AgentName   string
	Project     string
}

ExecTarget identifies the agent the Executor is executing against. Threaded through every event so subscribers see consistent identity fields without re-deriving from the registry.

type Executor

type Executor struct {
	// contains filtered or unexported fields
}

Executor dispatches the static CP-driven init plan against an open Session stream. Owns Recv during Run; the dialer's drainStream takes over after Run returns.

Plan idempotency contract: every Session establish runs the full plan, including reconnects after CP restart. Each step is idempotent (file-presence gates, append-if-missing, marker-file post-init), and AgentReady is no-op success when clawkerd already spawned the user CMD (spawnState's CAS rejects re-fork; handler replies Done{0}). Trade: a small volume of shell commands fires on every reconnect; gain: no per-container completed flag. Executor is shared across all containers — the dialer constructs one at CP boot and calls Run from a goroutine per DialAgent (one per agent container). Run holds no Executor-scoped mutable state: every call gets its own (ctx, stream, target) and drives its own stream's Recv loop in a single goroutine. The Run-owns-Recv invariant is per-stream (one Run, one stream, one Recv-driving goroutine), not per-Executor — concurrent Runs across different containers must not be serialized.

func NewExecutor

func NewExecutor(topic *pubsub.Topic[AgentEvent], dockerCli *docker.Client, log *logger.Logger) (*Executor, error)

NewExecutor constructs an Executor. nil log is replaced with logger.Nop(). topic is required — Run publishes exec events unconditionally, so a nil topic is a wiring bug the caller must catch at construction. Returning an error lets the caller (the orchestrator) log the wiring bug to the structured log surface and degrade gracefully (initExec = nil → dialer logs agent_init_executor_unset per dial) instead of crashing CP and stranding the failure on os.Stderr where only `docker logs` sees it. Matches the nil-topic contract on agent.New for the dialer.

func (*Executor) Run

func (e *Executor) Run(ctx context.Context, stream clawkerdv1.ClawkerdService_SessionClient, target ExecTarget, plan []step, label string) (runErr error)

type ExecutorEventState

type ExecutorEventState struct {
	// contains filtered or unexported fields
}

ExecutorEventState bundles the CP-driven exec phase fields for an Agent. Zero value means no exec phase has been observed (Status == ExecStatusUnknown); producers transition by emitting ExecutorEventState* events whose ApplyTo methods project here. Held as a sub-struct so the exec axis is isolated from the session axis (Session*, Address, Attempts) and the identity axis (Trust, Registered).

func ExecRunning

func ExecRunning(stepCount int, at time.Time) ExecutorEventState

ExecRunning resets the substruct to an active phase: Status becomes Running, StartedAt records the phase boundary, StepCount is captured for streaming subscribers ("1 of N" rendering). Any stale step / completion / failure carried by a previous phase is dropped — a reconnect that re-runs the plan should not surface the prior terminal state. A negative stepCount is clamped to zero.

func (ExecutorEventState) Complete

Complete returns a terminal ExecutorEventState in Completed state, clearing LastError so a subscriber switching on Status sees a coherent success snapshot. CompletedAt is forced to be at least StartedAt so the (CompletedAt < StartedAt) projection bug is unrepresentable.

func (ExecutorEventState) CompletedAt

func (i ExecutorEventState) CompletedAt() time.Time

func (ExecutorEventState) Fail

Fail returns a terminal ExecutorEventState in Failed state with detail. Same CompletedAt floor as Complete.

func (ExecutorEventState) LastError

func (i ExecutorEventState) LastError() string

LastError carries the most recent exec-axis failure detail. Cleared on Complete; populated by WithStepError and Fail. Distinct from the session-axis Agent.LastError so the two failure surfaces don't overwrite each other.

func (ExecutorEventState) StartedAt

func (i ExecutorEventState) StartedAt() time.Time

func (ExecutorEventState) Status

func (i ExecutorEventState) Status() Status

func (ExecutorEventState) StepCount

func (i ExecutorEventState) StepCount() int

func (ExecutorEventState) StepIndex

func (i ExecutorEventState) StepIndex() int

func (ExecutorEventState) StepName

func (i ExecutorEventState) StepName() string

StepName is the most recently started step's human-readable label. Empty until the first WithStep transition fires.

func (ExecutorEventState) WithStep

func (i ExecutorEventState) WithStep(stepName string, stepIndex int) ExecutorEventState

WithStep returns a copy of i with StepName / StepIndex updated. StepIndex is clamped into [0, StepCount-1] when StepCount > 0; the (name, index) pair stays internally consistent because both are written together from the same event payload, so a clamp affects only the index — not the human-readable name subscribers display.

func (ExecutorEventState) WithStepError

func (i ExecutorEventState) WithStepError(detail string) ExecutorEventState

WithStepError returns a copy of i with LastError set. Status is untouched — ExecStepFailed is mid-phase, the terminal transition is Fail.

type Handler

type Handler struct {
	agentv1.UnimplementedAgentServiceServer
	// contains filtered or unexported fields
}

Handler serves the Register RPC on the CP's clawker network agent listener and is the SOLE writer of the agentregistry sqlite DB. IdentityInterceptor has already grounded the peer in the daemon-attested container identity (peer IP → purpose=agent container → labels → cross-checked cert SAN AgentFullName) and attached the resolved (containerID, project, agentName) to ctx. The handler reads that, captures the cert thumbprint at the gate that persists it, cross-checks the cert's container_id SAN + request fields against the resolved truth, and writes the registry row using label-derived (authoritative) values.

Trust ordering: daemon labels > cert claim > request claim. Persisting `resolved.*` keeps the registry aligned with the daemon view; the request body is treated as a client claim that must agree but never as the source of truth.

All identity-rejection paths return codes.PermissionDenied with a generic envelope; the structured log line carries the specific failure classification for operator triage. A missing resolved container in ctx is a wiring bug (interceptor not chained) and surfaces as codes.Internal.

func NewHandler

func NewHandler(registry Registry, log *logger.Logger) (*Handler, error)

NewHandler constructs a Register handler. registry MUST be non-nil (NPE on first Register otherwise — fail at construction so the regression surfaces at CP startup, not at the first agent boot). log defaults to logger.Nop() when nil. clock defaults to time.Now.

func (*Handler) Register

func (h *Handler) Register(ctx context.Context, req *agentv1.RegisterRequest) (*agentv1.Welcome, error)

Register is the one-time-per-container handshake CP triggers via a RegisterRequired Command on the Session bidi stream. clawkerd presents a CLI-CA-signed cert and calls Register; the universal IdentityInterceptor has already pinned CN=ContainerClawkerd, resolved the peer IP to a purpose=agent container, and verified that the cert's urn:clawker:agent: SAN matches the label-derived AgentFullName. The handler captures the live mTLS peer's cert thumbprint, cross-checks the cert's urn:clawker:container: SAN and the request fields against the middleware-resolved identity, then writes the agentregistry row using the label-derived (authoritative) values.

Rejection paths (all return PermissionDenied unless noted; the classification is logged at Warn for operator triage):

  • resolved container missing from ctx — middleware did not run on this RPC (wiring regression). Returns Internal.
  • peer cert missing from ctx post-resolve — defense-in-depth; the interceptor pre-validates the cert, so reaching this branch means ctx was tampered between interceptor and handler.
  • request agent_name or project malformed (InvalidArgument)
  • request fields disagree with middleware-resolved labels (the client lying about its own identity in the RPC body, even though cert+labels agree)
  • cert urn:clawker:container: SAN missing or doesn't match the middleware-resolved container_id
  • existing row for this container_id has a different thumbprint (cert replay; CLI is the only legit cert source)

Idempotent retry: an existing row whose thumbprint matches the captured thumbprint returns Welcome silently — Session retries after the row was already written.

type ListOpts

type ListOpts struct {
	All bool
}

ListOpts controls the scope of a ContainerLister.List call. All=false (the zero value) restricts to running containers; All=true includes stopped/exited containers.

type Message

type Message struct {
	Type     EventType `json:"type"`
	Action   Action    `json:"action"`
	Time     int64     `json:"time"`
	TimeNano int64     `json:"timeNano"`

	// Classification shared across axes. Reason is the typed verdict;
	// Detail is the human-readable specifics.
	Reason Reason `json:"reason,omitempty"`
	Detail string `json:"detail,omitempty"`

	// Session-axis fields (DialerEventType).
	Address           string            `json:"address,omitempty"`
	Attempts          int               `json:"attempts,omitempty"`
	PeerAgentFullName string            `json:"peer_agent_full_name,omitempty"`
	PeerThumbprint    [sha256.Size]byte `json:"-"`

	// Registry-axis fields (RegistryEventType).
	RegisterOk bool `json:"register_ok,omitempty"`

	// Exec-axis fields (ExecutorEventType).
	StepName  string        `json:"step_name,omitempty"`
	StepIndex int           `json:"step_index,omitempty"`
	StepCount int           `json:"step_count,omitempty"`
	ExitCode  int32         `json:"exit_code,omitempty"`
	Duration  time.Duration `json:"duration,omitempty"`
}

Message is the unified, discriminated agent-event body. The (Type, Action) pair routes; the remaining fields are the union of every signal the legacy per-event structs carried (session address / attempts, peer identity, register outcome, exec step coordinates, classified Reason + Detail). A producer fills only the fields its (Type, Action) defines; readers tolerate zero values for the rest.

type MobyPeerLookup

type MobyPeerLookup struct {
	// contains filtered or unexported fields
}

MobyPeerLookup is the production ContainerByPeerIP backed by the Docker daemon.

func NewMobyPeerLookup

func NewMobyPeerLookup(cli mobyclient.APIClient, log *logger.Logger) *MobyPeerLookup

NewMobyPeerLookup wraps a moby APIClient as a ContainerByPeerIP resolver. log defaults to logger.Nop() when nil.

func (*MobyPeerLookup) LookupByIP

func (m *MobyPeerLookup) LookupByIP(ctx context.Context, ip netip.Addr) (ResolvedContainer, error)

LookupByIP walks every `purpose=agent` container and returns the one whose clawker network endpoint IP matches ip. The walk is exhaustive: ambiguous-IP advertisements (multiple containers with overlapping endpoint state during restart cycles) return ErrAmbiguousPeerIP rather than picking the first match. A transient per-container inspect failure is logged and iteration continues. A daemon-error wrap is returned only when NO candidate could be inspected — a real no-match is reported as ErrNoContainerForPeerIP even when one inspect along the way failed, so the peer_lookup_no_match audit signal stays useful.

type Reason

type Reason string

Reason is the unified failure/trust classification. It collapses the legacy split between the dialer's UntrustedReason and the executor's ExecFailureReason into one vocabulary so a single Reason field on the AgentEvent message carries every classified outcome. Subscribers switch on it to enact policy (containment, alerting, eviction) and to distinguish the dialer-permissive asymmetric-trust verdicts (thumbprint mismatch / cert invalid / register failed) from exec failures. Empty string is the zero value: no classified failure.

const (
	ReasonNone Reason = ""

	// Trust-axis reasons. Keep the dialer-permissive classifications
	// distinct so thumbprint mismatch / cert invalid / register failed
	// stay distinguishable to subscribers enacting trust policy.
	ReasonThumbprintMismatch Reason = "cert_thumbprint_mismatch"
	ReasonCertInvalid        Reason = "cert_invalid"
	ReasonRegisterFailed     Reason = "register_failed"
	// ReasonInitBypassed: the user CMD is running while init never ran —
	// a structurally-impossible lifecycle state (CP bug or out-of-band CMD
	// fork). See agentInitBypassed.
	ReasonInitBypassed Reason = "init_bypassed"

	// Exec-axis reasons: classified failure modes for a dispatched plan.
	ReasonExitCode       Reason = "exit_code"
	ReasonTimeout        Reason = "timeout"
	ReasonSpawnFailed    Reason = "spawn_failed"
	ReasonIOError        Reason = "io_error"
	ReasonTransportError Reason = "transport_error"
	ReasonProtocolError  Reason = "protocol_error"

	// ReasonFailed is the session-axis generic dial failure (retry
	// exhausted, container gone, addr invalid, panic) — the dialer
	// records the human-readable specifics in Message.Detail.
	ReasonFailed Reason = "failed"
	// ReasonUnknown is the exec-axis fallback for an unrecognized
	// clawkerd ErrorCode.
	ReasonUnknown Reason = "unknown"
)

type Registry

type Registry interface {
	// Add inserts an entry keyed by (Entry.Thumbprint, Entry.ContainerID).
	// Container restart produces a new cert and a new thumbprint, so
	// re-registration creates a new entry; the dockerevents subscription
	// is responsible for evicting the stale one by container ID.
	//
	// Returns an error when the persistence layer (sqlite) rejects the
	// write — disk full, schema corruption, UNIQUE collision against a
	// stale row that hasn't been evicted yet. Callers translate the
	// error into the appropriate gRPC status.
	//
	// Add returns an error on programming-error invariants (zero
	// thumbprint, empty ContainerID, zero AgentName, zero
	// RegisteredAt). Both Registry implementations (in-memory and
	// sqlite) propagate the error from `validateEntry` rather than
	// panicking — Add lives on a gRPC handler goroutine reachable
	// post-SetReady from Register, and a panic on that path would
	// strand eBPF programs with no supervisor (see root CLAUDE.md).
	// Register maps the error to codes.Internal: every field
	// validateEntry checks is server-derived (thumbprint from the
	// live peer cert, ContainerID/AgentName/Project from
	// IdentityInterceptor's ResolvedContainer, RegisteredAt from
	// h.clock()), so a failure here is a CP wiring bug, not bad
	// client input. User-controlled identity strings are validated
	// upstream at the wire boundary (auth.NewProjectSlug /
	// auth.NewAgentName → codes.InvalidArgument).
	Add(entry Entry) error
	// LookupByContainerID returns the entry whose ContainerID matches.
	// Used by the Register handler (idempotency / replay-protection)
	// and by the dialer at Hello time to drive registry classification
	// (Match / Miss / ThumbprintMismatch) and decide whether to send
	// RegisterRequired or publish AgentUntrusted. The dialer performs
	// the thumbprint comparison against the returned entry itself;
	// this read intentionally does not gate on thumbprint so a
	// mismatch surfaces as a typed local outcome rather than as
	// ErrUnknownAgent.
	//
	// Returns (nil, ErrUnknownAgent) when no entry matches.
	LookupByContainerID(containerID string) (*Entry, error)
	// EvictByContainerID removes any entry whose ContainerID matches.
	// Returns the underlying persistence error so callers can decide
	// whether to retry, log, or abort. The dockerevents-driven and
	// reaper-driven callers log-and-proceed because a transient sqlite
	// failure must not stall the eviction pipeline; CLI-side `clawker
	// container remove` similarly logs at debug since registry hiccups
	// must not surface as remove failures (the row gets pruned later
	// by the dockerevents subscription).
	EvictByContainerID(containerID string) error
	// Snapshot returns a copy of every live entry, sorted by
	// (Project, AgentName) for deterministic output. Project is the
	// primary sort key because the same short AgentName can be reused
	// across different projects (the composite identity is
	// (project, agent)). Used by AdminService.ListAgents and the
	// `clawker controlplane agents` CLI; both rely on stable ordering
	// for diffability.
	//
	// Returns a non-nil error on persistence failure (sqlite db.Query
	// or rows.Err non-nil). Callers must NOT treat an empty result as
	// authoritative when err != nil — reapOrphans would otherwise
	// evict every registered agent on a transient query failure;
	// ListAgents would surface "no agents" to operators while the
	// registry is intact but unreadable.
	Snapshot() ([]Entry, error)
}

Registry is the consumer-facing contract.

func NewRegistry

func NewRegistry(log *logger.Logger) Registry

NewRegistry constructs an empty in-memory registry. Logger is required (use logger.Nop() in tests) so audit-trail messages on Add and Evict are captured even when production logging is otherwise disabled.

func NewSQLiteWriter

func NewSQLiteWriter(dbPath string, log *logger.Logger) (Registry, error)

NewSQLiteWriter opens (or creates) the sqlite database at dbPath, applies the schema, and returns a registry suitable for the writer process. CP is the SOLE writer in this design (the CLI no longer opens the registry DB), so a separate reader-mode constructor is not provided — every consumer that previously read from sqlite now goes through `f.AdminClient(ctx).ListAgents`.

The parent directory of dbPath must already exist.

type Repository

type Repository struct {
	Agents *AgentStore
}

Repository is the agent bounded context's storage repository. It aggregates the domain's Stores (currently just AgentStore) behind one Subscribe call the orchestrator wires. Future agent-axis stores (a per-project rollup, a trust-event log) join here.

func NewRepository

func NewRepository() *Repository

NewRepository constructs the agent repository with empty stores.

func (*Repository) Subscribe

func (r *Repository) Subscribe(topic *pubsub.Topic[AgentEvent])

Subscribe wires every store in the repository to the agent Topic.

func (*Repository) SubscribeDockerEvents

func (r *Repository) SubscribeDockerEvents(topic *pubsub.Topic[dockerevents.DockerEvent])

SubscribeDockerEvents wires every store's eviction path to the dockerevents Topic so a destroyed container's projected worldview is reclaimed rather than leaked. The orchestrator calls this alongside Subscribe so the worldview is both written (agent events) and bounded (docker destroy).

type ResolvedContainer

type ResolvedContainer struct {
	ContainerID string
	Project     auth.ProjectSlug
	AgentName   auth.AgentName
}

ResolvedContainer is the authoritative identity of an agent container as resolved from its live mTLS peer IP. Project and AgentName are typed so a zero-value instance cannot carry empty strings into a downstream identity comparison; the resolver constructs these via auth.NewProjectSlug / auth.NewAgentName and rejects malformed labels at the seam.

func ResolvedContainerFromContext

func ResolvedContainerFromContext(ctx context.Context) (ResolvedContainer, bool)

ResolvedContainerFromContext returns the peer-IP-resolved container identity IdentityInterceptor attached to ctx. ok=false means the interceptor did not run on this RPC — a wiring bug for any AgentService handler that consumes resolved identity. Defensive against a zero-value resolved container (empty ContainerID): treats it as ok=false so handlers can rely on ok=true meaning a non-empty resolved container is available.

type StartDeps

type StartDeps struct {
	Registry     Registry
	DockerLister ContainerListFunc
	PeerLookup   ContainerByPeerIP
	Dialer       *Dialer
	DockerTopic  *pubsub.Topic[dockerevents.DockerEvent]
	AgentTopic   *pubsub.Topic[AgentEvent]
	Log          *logger.Logger
}

StartDeps bundles the dependencies the umbrella `Start` procedure needs. The topics + Registry + Dialer + Docker lister + peer-IP resolver are all owned by the CP orchestrator; passing them as a single struct keeps the call site a single function call. All fields are required.

DockerTopic is the dockerevents domain's Topic[DockerEvent]: the evict, session-cancel, and dial subscribers attach to it with their predicates folded into the handler closures. AgentTopic is this domain's own Topic[AgentEvent], used by the reap-degraded notification (the only AgentEvent Start itself publishes).

type Status

type Status string

Status is the projected session/exec lifecycle state a Store derives from the (Type, Action) of an AgentEvent. Disjoint vocabulary from Action — Action is the wire verb, Status is the worldview state.

const (
	StatusUnknown    Status = "unknown"
	StatusUntrusted  Status = "untrusted"
	StatusConnecting Status = "connecting"
	StatusConnected  Status = "connected"
	StatusBroken     Status = "broken"
	StatusFailed     Status = "failed"
	StatusDegraded   Status = "degraded"
	StatusHealthy    Status = "healthy"
	StatusRunning    Status = "running"
	StatusCompleted  Status = "completed"
)

type Trust

type Trust struct {
	// contains filtered or unexported fields
}

Trust bundles the trust verdict for an Agent so the "is the agent trusted?" and "if not, why?" questions are a single invariant-respecting value rather than a (bool, enum) pair where the zero value is structurally illegal ("untrusted with no reason" is not a thing the worldview should be able to represent).

Zero value = trusted with no reason. That's the right default: an Agent struct freshly inserted into State.Agents has not yet been proven untrustworthy, and the consumer-facing API (IsTrusted, Reason) returns sensible answers immediately. Producers transition to untrusted via Untrust(reason); that's the only path that flips the verdict.

func Untrust

func Untrust(reason Reason) Trust

Untrust constructs a Trust value carrying the classification. The reason MUST be non-empty; an empty reason is rejected by returning a Trust whose IsTrusted is still true (no producer should call Untrust(UntrustedReasonNone) — that's a logic bug).

func (Trust) IsTrusted

func (t Trust) IsTrusted() bool

IsTrusted reports whether the agent is currently trusted by CP. Zero-value Trust is trusted.

func (Trust) Reason

func (t Trust) Reason() Reason

Reason returns the classification when IsTrusted is false. ReasonNone otherwise.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL