runtime-controller

Runtime Controller

Runtime Controller owns the platform lifecycle of one logical Runtime Environment per Agent. Agent Controller issues explicit Initialize, Update, Disable, Enable, and Delete commands; Runtime Controller realizes them as private Docker compute and workspace resources and reports a platform-neutral result. It is written in Go.

The cross-service identity is agent_id plus an opaque runtime_revision. Physical generations, deployment digests, container IDs, and volume names stay private to this service. Docker is the only implemented platform adapter; a Kubernetes adapter is planned and would be added in-process, not as a separate service.

Workload authentication and configured Registry transport use the shared Go module, with the caller-context forwarding policy. RC retains its Controller-only route grants, Docker boundary and separate Runtime instance credential lifecycle.

Startup rejects published PostgreSQL passwords under the development secret policy. The exact ANTNEST_ALLOW_PUBLIC_DEV_SECRETS=true opt-in emits a variable-only WARN; it never restores the retired Registry token.

Responsibilities

• Issue and atomically seal distinct RC/ACP Runtime tokens with each private compute generation. Deliver only caller hashes through a verified root-only, read-only receiver volume; exact recovery retains credentials. Privately resolve ACP's current connection for Controller; ordinary projections remain secret-free. See the instance connection contract.

• Idempotently Initialize, Update, Disable, Enable, Inspect, and Delete one Agent Runtime Environment.

• Allocate internal immutable compute generations and expose only an opaque Runtime revision to callers.

• Map one language-neutral Runtime configuration to deterministic Docker compute and workspace resources, and bind each generation permanently to one deployment digest.

• Resolve an installed image reference to an immutable image ID through a read-only platform query; never build or implicitly pull images.

• Complete creation after confirmed platform create/start, without waiting for health (see creation and observation).

• Report the current Docker IPv4 from the configured management-network attachment as runtime_endpoint in Inspect/List (contract revision 19). Controller reads this address before binding Egress traffic; it is independent of MCP URLs and execution readiness. Controller forwards the fresh address for Egress attachment opens and restart rebinding; see peer binding. Missing/invalid management IPv4 is an individual unknown-health inspection with runtime_peer_unavailable; it does not fail the whole inventory.

• Consume platform health and List/Watch events, verify Runtime /status on Healthy observations and explicit reads, and normalize the facts into an ordered, time-retained observation journal.

• Create and retain the Agent workspace as part of lifecycle commands; workspace operations are never exposed as a cross-service API.

• Prepare per-Agent system Skill volumes from exact Skill Registry versions and mount them read-only into Runtime containers.

• Freeze the Runtime Skill maintenance verifier keys into each accepted lifecycle operation. Key IDs follow the shared RuntimeSpec grammar and fixtures; invalid operator configuration fails at startup before creating Runtimes.

• Serialize all mutations for one Agent across Controller replicas, and elect one platform-Watch consumer.

• Emit structured logs, traces, and low-cardinality metrics.

Non-responsibilities

• It does not decide when an Agent is initialized, updated, disabled, enabled, or deleted.
• It does not own Agent desired state, active binding, or Run admission.
• It does not dispatch Runs or proxy MCP Tool calls.
• It does not allocate Tunnel IPs, persist Egress policy, or call Runtime Egress. Agent Controller supplies the Egress attachment in the Runtime configuration.
• It does not run a Runtime reverse-connection server.
• It does not authenticate end users or expose a public API.

Runtime instance bootstrap

ANTNEST_RUNTIME_INSTANCE_KEY_FILE is required: a private regular mode-0600 file containing exactly 32 raw CSPRNG bytes, mounted read-only by deployment. Retain it with RC's journal across restarts. It is separate from Controller's credential key and the workload-token directory; losing or changing it makes existing instance authority unavailable. There is no automatic reissue during recovery. Development has no legacy data to migrate; rebuild a fresh test stack.

The first instance receiver supports token mode with the explicit ANTNEST_SERVICE_AUTH_ALLOW_INSECURE_TRANSPORT=true opt-in on its isolated internal network. Native Runtime TLS/mTLS is unsupported and unsupported profiles fail RC startup; no opt-in is synthesized. The control-service TLS library still validates TLS when configured. Production transport expansion is separate work. The token-profile Runtime, Controller relay and ACP consumers are implemented.

• It does not read another service's database.
• It does not hold the Skill maintenance signing key; that belongs to Agent ACP Service.

Authenticated tunnel generation (#111)

RC also seals independent WireGuard endpoint keys and a 32-byte PSK with every Initialize/Update/Enable operation. Exact recovery keeps the accepted key ID. The same root-only private volume now contains tunnel.json; both file digests, mode 0600 and actual read-only mount ownership are checked before start. RuntimeSpec.authentication.tunnel carries only ID, fixed path and digest. Inspect/List reports the live generation's tunnel_key_id; compute completion receipts do not claim a currently verified peer or execution identity.

ANTNEST_RUNTIME_EGRESS_URL and the workload sender file for runtime-egress are required. RC privately registers Egress's endpoint material before any platform effect. Dependency outages return retryable tunnel_registration_unavailable while the admitted operation stays running; retry the same request ID. Deterministic rejection preserves an existing source Runtime. Keys never enter HTTP content capture. Runtime, Egress and Controller adoption plus deployment wiring remain subsequent #111 batches; this producer alone is not a tunnel. See the shared design.

Interfaces

The Runtime status reader recognizes the compiled test_features array while retaining strict unknown-field decoding. Deploy this reader before upgrading Runtime images to the required-field status contract. It also accepts the previous status shape during rollout. This change does not alter readiness criteria or reject test-feature images; image admission is enforced by the operator repository/digest policy in #29.

Configuration

Standard OTEL_* variables configure OTLP http/protobuf export. Selected OTEL_* keys, optionally overridden by ANTNEST_RUNTIME_OTEL_*, are forwarded to managed Runtimes. See operations for details.

Transient observation leadership/readiness query and reconciliation failures retry without ending the process. Failed initial reconciliation releases leadership before retrying; readiness is announced only after reconciliation and the Watch handshake succeed. Retries start at 1s, double to the configured limit, include up to 20% positive jitter within that limit, and reset after Watch readiness. Explicit permanent configuration/schema/programming errors still return to startup supervision. The standard Compose service uses restart: unless-stopped as a fallback for process failures.

Control contract revision 14 adds a required monitor_ready field to /status. Monitor retries and Watch reconnection return HTTP 503 with live: true; a ready Watch or ready-leader probe restores HTTP 200. The route reads cached state and does not call Docker or scan Runtimes. A Watch-only disconnect can leave lifecycle calls working; a Docker API outage makes actual platform calls fail. Brief reconnects have a 503 window handled by existing probe failure thresholds. See operations.

Dependencies

• PostgreSQL: the private runtime_controller schema. Startup fails if the database is unreachable or carries an unknown future migration.
• Docker Engine: required for lifecycle, Watch, and image resolution. The management network and system Skill volume are checked during lifecycle calls, not at startup; /status never calls Docker.
• Managed Runtime /status endpoints on the management network.
• Skill Registry: optional. Without it, Skill set preparation is unavailable, but lifecycle commands that do not reference a prepared set still work.
• A prebuilt Antnest Runtime image installed on the Docker host.

Build and test

All control routes require verified Controller workload identity; RC never calls its own API. The workload boundary specifies exact token/mTLS, JSON, listening addresses, Registry transport and Docker socket limitations. The retired ANTNEST_SKILL_REGISTRY_API_TOKEN must be removed; any nonempty value fails startup. Deployment wiring and complete token-profile cross-service E2E passed the final integration batch, recorded in the rollout ledger.

Service-local checks, from services/runtime-controller:

make fmt-check
make lint
make test

make lint uses the repository's checked-in .golangci.yml.

From the repository root:

make test-go                           # unit tests with root integration overlays
make test-runtime-controller-postgres  # repository tests against disposable PostgreSQL
node tests/e2e/service-authentication/runtime-controller/run.mjs # isolated native authentication/network/image/lifecycle gate
make e2e-runtime-controller            # builds images and runs the lifecycle E2E
make e2e-runtime-controller-observation-retry # isolated Docker socket outage/recovery
make integration-stage4-skill-prepare  # Registry to Runtime Controller Skill preparation
docker build -f services/runtime-controller/Dockerfile -t antnest/runtime-controller:local .

Unit tests live alongside the service packages. PostgreSQL and Docker integration sources live in tests/integration/go/runtime-controller, and the deployed lifecycle scenario in tests/e2e/runtime-controller/run.sh. The root Go runner overlays those sources into the owning packages so they can reach package-private details. The lifecycle E2E covers initialization from an empty environment, Controller restart recovery, Runtime restart observation, Update, Disable workspace retention, Enable, and Delete cleanup.

The observation retry E2E builds a separately tagged candidate Controller and uses a private Unix-socket proxy to simulate unavailable Docker at startup and a later socket outage. It verifies HTTP 200/503/200, readiness leases, zero process restarts, and successful Runtime Initialize/Delete. It separately disconnects only Watch and proves a lifecycle call still works while monitor_ready is false, then removes its owned containers, volumes, networks and candidate image.

Opt-in checks:

• Installed image resolution without creating resources:

  ANTNEST_RUNTIME_CONTROLLER_TEST_DOCKER_SOCKET=/var/run/docker.sock \
  ANTNEST_RUNTIME_CONTROLLER_TEST_IMAGE_TAG=antnest/antnest-runtime:local \
  node tests/integration/go/run.mjs runtime-controller --package internal/platform/docker -- \
    -run '^TestInstalledImageResolution$' -count=1
  

• Build metadata smoke against an existing development stack with Jaeger: node tests/e2e/runtime-controller/build-image-smoke.mjs --project <compose-project>.

• Creation and observation E2E: make e2e-observation from the service directory (see creation and observation).

• Update process-crash recovery: make test-crash-recovery from the service directory (see the crash recovery contract).

Test-only variables: ANTNEST_RUNTIME_CONTROLLER_TEST_DATABASE_URL, ANTNEST_RUNTIME_CONTROLLER_TEST_DOCKER_SOCKET, ANTNEST_RUNTIME_CONTROLLER_TEST_IMAGE_TAG, ANTNEST_RUNTIME_CONTROLLER_TEST_MOVED_IMAGE_TAG, ANTNEST_RUNTIME_CONTROLLER_TEST_URL, ANTNEST_RUNTIME_TEST_CONFIGURATION, ANTNEST_RUNTIME_TEST_AGENT_ID, ANTNEST_RUNTIME_CONTROLLER_CRASH_TEST, ANTNEST_RUNTIME_CONTROLLER_CRASH_IMAGE, ANTNEST_RUNTIME_CRASH_EVIDENCE, and ANTNEST_TEST_REAL_RUNTIME_IMAGE.

Documentation

• Architecture - domain model, lifecycle workflows, persistence, observation semantics, Skill delivery, and invariants.
• Operations - deployment, configuration, readiness, health, and failure diagnosis.
• Creation and observation - separation of command completion from Runtime readiness.
• Observability - trace, SQL, and RPC content boundaries.
• Inspect absence contract - telemetry classification of an expected missing container.
• Crash recovery contract - opt-in Update process-crash component test.
• Control API and API directory - owned RPC contract and schemas.
• Skill Registry Runtime delivery contract
  • Skill set preparation and lifecycle consumption.
• Stage 1 Runtime - cross-service Runtime contract.
• Skill learning design - maintenance verifier keys and rotation.
• Service layout - repository ownership rules.

Managed MCP secret bootstrap uses optional ANTNEST_AGENT_CONTROLLER_URL and the existing authenticated outbound transport. It is required for deployments with secret_env descriptors, not for configurations without managed secrets. Controller owns encrypted values; RC owns generation-private read-only delivery and cleanup. See the delivery contract.