Fenced

module
v1.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 20, 2026 License: Apache-2.0

README

Fenced

The OS kernel for AI agents. Every agent runs fenced.

Fenced (formerly AgentOS) manages the execution around your agent code: it stores task state in PostgreSQL, places work on runtime pools, recovers after worker lease loss, and rejects writes from stale workers. Gateways authorize and meter tool and model calls against task budgets.

Use it when your agent tasks need to survive worker failures, coordinate through durable mailboxes, or run with explicit execution budgets.

Try locally · Recovery and fencing · Architecture · Current limits · 中文使用指南

Apache 2.0 · Source version: 1.4.0 · v1.2 public contract freeze

Try locally

Start with a dependency-free Python agent and the Runtime Interface conformance suite. You need Go 1.26.x and Python 3.11+. This trial needs no Docker, model account, API key, or framework installation.

Naming note: Fenced was previously named AgentOS. The repository, CLI binaries, SDK package names, Go module path, and protocol identifiers have all been renamed to Fenced / fenced. Historical documents and released artifacts may still reference the former name; see docs/NAMING.md for the complete migration map and compatibility notes.

git clone https://github.com/CloudEdgeCore/Fenced.git Fenced
cd Fenced

PowerShell:

$env:PYTHONPATH = "./sdk/python"
go run ./cmd/fenced-conformance -cmd "python examples/agents/python_remote/server.py --port 0" -timeout 30s

Bash / zsh:

PYTHONPATH=./sdk/python go run ./cmd/fenced-conformance -cmd "python3 examples/agents/python_remote/server.py --port 0" -timeout 30s

The command starts a local adapter, discovers its port, checks the protocol, and stops the adapter when finished. The first Go build downloads dependencies.

Expected output includes:

Adapter:  python-remote
Protocol: fenced.runtime.interface/v1
...
Fenced Compatible = PASS

What this checks: startup, idempotency, events, checkpoint/restore, result, stop, and capability denial at the Runtime Interface boundary.

Scope: the example uses an EchoAgent and in-memory checkpoints. This is a protocol smoke test; it does not exercise the PostgreSQL task kernel, real model calls, durable recovery, or sandbox isolation.

To run the complete local platform, follow the control-plane setup.

Recovery and fencing

When recovery retries a task after worker lease loss, it creates a replacement attempt with a higher fencing token. Calls carrying the expired identity are rejected before they can update durable task state.

Reproduce the takeover check with Go and a disposable PostgreSQL instance; no model or API key is needed. It deliberately expires a lease, invokes recovery, and verifies that:

  1. The replacement receives a higher fencing token.
  2. The old identity cannot heartbeat, commit a checkpoint, change phase, or read its assignment.
  3. No stale checkpoint reaches PostgreSQL.

The check uses real storage and the Runtime Control service. It injects lease expiry directly; it does not kill a deployed worker process.

For a broader deterministic workload, the research workflow takeover report describes worker loss and replacement across runtime pools, with commands and the boundaries of that experiment.

How it works

flowchart LR
    A["Your agent + Runtime Interface adapter"] --> B["Control API"]
    B --> C["PostgreSQL: Task / Run / Attempt"]
    C --> D["Admission: policy + budgets"]
    D --> E["Scheduler + recovery"]
    E --> F["Worker: fenced lease"]
    F --> G["Tool / Model / Memory gateways"]
    G --> C
    F --> H["Checkpoint + result"]
    H --> C

Publish an immutable agent version, submit a task, and let admission and the scheduler select a runtime pool. Workers execute through Runtime Protocol v1; gateways enforce authorization and usage accounting on calls routed through them. Replacement workers can resume compatible checkpoints.

See the architecture and framework integration guide for the full design.

What is available

Capability What it does
Task kernel Durable Task / Run / Attempt state machines, immutable agent versions, retries, checkpoints, and audited transitions
Scheduling and recovery Capacity reservations, ranked placement, worker leases, monotonic fencing, and recovery after lease expiry
Execution governance Token, cost, and tool-call budgets, deadlines, and runtime-specific resource limits
Model gateway OpenAI-compatible and Anthropic Messages protocols, with exact provider-reported usage settlement, streaming sessions, and per-provider circuit breaking
Durable IPC At-least-once mailbox delivery, receiver deduplication receipts, and tenant-scoped access
Task-backed services Supervised replicas, restart policies, rolling version convergence, and cooperative drain
Runtime integration Reference and HTTP adapter workers, Wasmtime and OCI/gVisor providers, and Go/Python/TypeScript SDKs

Framework adapters include LangGraph, AutoGen, CrewAI, OpenAI Agents, and custom agents. Start with the ecosystem guide; use the conformance suite to check an adapter's Runtime Interface compatibility.

For services, supply a published spec.agentVersionRef and spec.workloadSpec (fenced service create -spec task-spec.json), and apply migration 000037. Each replica consumes a real worker slot and retains Task budgets and timeouts. See the service guide.

Current limits

  • CLI-first operation. The web workbench is an internal, loopback-only preview; a production administration console and managed hosting are pending.
  • Infrastructure you operate. The full platform uses PostgreSQL and NATS. Production paths require configured HTTPS/OIDC and SPIFFE mTLS identities; model, tool, embedding, and secret services need their own configuration.
  • Fencing protects ownership. It rejects stale kernel operations. Physical isolation depends on the runtime, such as Wasmtime or OCI/gVisor; the reference provider is development infrastructure.
  • Cancellation is cooperative; attempts are non-preemptible. Workers must acknowledge cancellation. Checkpoint recovery requires compatible agent versions, runtime ABIs, and state schemas.
  • Resource controls use different mechanisms. Gateway budgets, scheduling capacity, and sandbox limits have different enforcement points. Calls that bypass the gateways are outside their usage accounting.
  • Frozen contracts have implementation boundaries. Syscall ABI 1.0.0 defines 21 calls across eight groups. Production gateway entry points currently register Tool, Model, Memory, and IPC services; unified Syscall and Effect endpoints are not yet wired into those entry points.

See the feature status matrix and production setup before deployment.

Evidence and compatibility

Evidence Scope
Runtime conformance checks Black-box Runtime Interface compatibility checks
Recovery and fencing test PostgreSQL lease takeover and rejection of stale identities
TLA+ model Finite-model checks of v0.1 core safety and liveness invariants
100K report and 1M report Fixed-host control-plane pipeline measurements with raw logs; not real-agent or production throughput figures

Reproducible raw 72h/7d production soak evidence is not yet published. The existing soak report does not include the raw logs and run metadata needed to substantiate those duration claims.

The v1.2 contract freeze documents stable Runtime, Gateway, Control API, Service, IPC, Effect, and Syscall contracts. Breaking changes require a new contract version. Legacy v1alpha1 compatibility will not close before 2027-02-17; see the compatibility policy.

Documentation

Try it and share feedback

Run the local trial and open an issue with the first step that blocks you: your OS, the command, and the error output. If you already run agent workloads, describe the failure or budget-control problem you would want Fenced to handle.

Directories

Path Synopsis
cmd
agent command
agentos command
Command agentos is a compatibility shim for the AgentOS → Fenced rename.
Command agentos is a compatibility shim for the AgentOS → Fenced rename.
fenced command
Command fenced is the stable developer and operator workflow.
Command fenced is the stable developer and operator workflow.
fenced-conformance command
Command fenced-conformance certifies a running third-party adapter through the public Runtime Interface only.
Command fenced-conformance certifies a running third-party adapter through the public Runtime Interface only.
fenced-control command
fenced-gateway command
fenced-migrate command
fenced-orchestrator command
Command fenced-orchestrator runs the durable workflow orchestrator: it decides which workflow steps execute when and creates ordinary Tasks for them.
Command fenced-orchestrator runs the durable workflow orchestrator: it decides which workflow steps execute when and creates ordinary Tasks for them.
fenced-outbox command
fenced-pkg command
Command fenced-pkg is the CI-facing Agent Package tool (ADR-010): it generates package-signing keys, signs canonical manifests, and verifies signed packages against a trust registry.
Command fenced-pkg is the CI-facing Agent Package tool (ADR-010): it generates package-signing keys, signs canonical manifests, and verifies signed packages against a trust registry.
fenced-projector command
Command fenced-projector consumes canonical store events from JetStream and applies them to OpenSearch search projections.
Command fenced-projector consumes canonical store events from JetStream and applies them to OpenSearch search projections.
fenced-slo command
fenced-svid command
Command fenced-svid generates the X.509-SVID identity material for the Secure Runtime boundary (ADR-011): a self-signed SPIFFE CA, a control plane SVID (with loopback IP SAN), and one worker SVID per runtime instance.
Command fenced-svid generates the X.509-SVID identity material for the Secure Runtime boundary (ADR-011): a self-signed SPIFFE CA, a control plane SVID (with loopback IP SAN), and one worker SVID per runtime instance.
mtlsutil
Package mtlsutil shares the mTLS client credential construction between the runtime worker commands (ADR-011 identity boundary).
Package mtlsutil shares the mTLS client credential construction between the runtime worker commands (ADR-011 identity boundary).
deploy
oci/agent-runtime command
Minimal agent runtime for the OCI/gVisor isolation drill.
Minimal agent runtime for the OCI/gVisor isolation drill.
examples
devops-workflow/runtime
Package devops implements the six agent roles of the DevOps reference workload as one Fenced-native runtime.
Package devops implements the six agent roles of the DevOps reference workload as one Fenced-native runtime.
devops-workflow/tools/cluster
Package cluster implements the DevOps reference workload's deterministic tools as a webhook-backed tool endpoint behind the Fenced Tool Gateway:
Package cluster implements the DevOps reference workload's deterministic tools as a webhook-backed tool endpoint behind the Fenced Tool Gateway:
research-workflow/app/api
Package api exposes the §13 application-layer research REST API.
Package api exposes the §13 application-layer research REST API.
research-workflow/app/cmd/research-api command
Command research-api is the application-layer Research REST API of the multi-agent research workflow (design doc §13).
Command research-api is the application-layer Research REST API of the multi-agent research workflow (design doc §13).
research-workflow/app/domain
Package domain holds the research domain model of the reference application (design doc §5).
Package domain holds the research domain model of the reference application (design doc §5).
research-workflow/app/report
Package report renders the final research report (design doc §14 Phase 3: "Markdown Renderer", "Citation Coverage", "Artifact Store").
Package report renders the final research report (design doc §14 Phase 3: "Markdown Renderer", "Citation Coverage", "Artifact Store").
research-workflow/app/repository
Package repository materializes the research domain model from kernel surfaces.
Package repository materializes the research domain model from kernel surfaces.
research-workflow/runtime
Package research implements the seven agent roles of the multi-agent research workflow as one Fenced-native runtime.
Package research implements the seven agent roles of the multi-agent research workflow as one Fenced-native runtime.
research-workflow/runtime/cmd command
research-runtime serves the seven research workflow roles behind the Fenced Runtime Interface (HTTP).
research-runtime serves the seven research workflow roles behind the Fenced Runtime Interface (HTTP).
research-workflow/tools/cmd/webtools command
webtools serves the research workflow's two tenant tools over HTTPS for standalone (non-test) deployments.
webtools serves the research workflow's two tenant tools over HTTPS for standalone (non-test) deployments.
research-workflow/tools/webtools
Package webtools implements the research tools of the reference workflow as webhook-backed tool endpoints behind the Fenced Tool Gateway:
Package webtools implements the research tools of the reference workflow as webhook-backed tool endpoints behind the Fenced Tool Gateway:
runtimes/docker-runtime command
Docker Runtime: An official Fenced runtime adapter that isolates agent tasks inside container boundaries, maps checkpoints to persistent volumes, and enforces the fenced.runtime.interface/v1 specification.
Docker Runtime: An official Fenced runtime adapter that isolates agent tasks inside container boundaries, maps checkpoints to persistent volumes, and enforces the fenced.runtime.interface/v1 specification.
runtimes/remote-http-runtime command
Remote HTTP Runtime: An official Fenced runtime adapter that bridges remote HTTP/REST microservice agents to the Fenced Runtime Interface v1 specification.
Remote HTTP Runtime: An official Fenced runtime adapter that bridges remote HTTP/REST microservice agents to the Fenced Runtime Interface v1 specification.
third-party/hello-agent
Package hello is a minimal third-party agent.
Package hello is a minimal third-party agent.
gen
internal
control/api
The agent version endpoints: publication and observation of an immutable publication.
The agent version endpoints: publication and observation of an immutable publication.
control/auth
Package auth defines the authenticated identity carried by control-plane requests.
Package auth defines the authenticated identity carried by control-plane requests.
gateway
Package gateway implements the fenced Tool Gateway boundary that runtime workers call on behalf of an Attempt.
Package gateway implements the fenced Tool Gateway boundary that runtime workers call on behalf of an Attempt.
gateway/bao
Package bao implements the reference Secret Broker backed by OpenBao (ADR-012).
Package bao implements the reference Secret Broker backed by OpenBao (ADR-012).
kernel/admission
Package admission evaluates deterministic pre-scheduling requirements.
Package admission evaluates deterministic pre-scheduling requirements.
kernel/agentpkg
Package agentpkg defines the signed Agent Package manifest (ADR-010) and its admission verification.
Package agentpkg defines the signed Agent Package manifest (ADR-010) and its admission verification.
kernel/agentversion
Package agentversion defines the bounded AgentVersion vocabulary of the Agent OS kernel: the canonical reference grammar, the immutable published spec, and the deterministic digest over that spec.
Package agentversion defines the bounded AgentVersion vocabulary of the Agent OS kernel: the canonical reference grammar, the immutable published spec, and the deterministic digest over that spec.
kernel/capability
Package capability enforces immutable AgentVersion grants at Gateway boundaries without coupling the Agent manifest vocabulary to persistence.
Package capability enforces immutable AgentVersion grants at Gateway boundaries without coupling the Agent manifest vocabulary to persistence.
kernel/domain
Package domain defines the stable lifecycle vocabulary enforced by the Fenced kernel.
Package domain defines the stable lifecycle vocabulary enforced by the Fenced kernel.
kernel/memory
Package memory implements the v0.1 Memory decision chain (ADR-009): the canonical store lives in PostgreSQL (FTS + pg_trgm + pgvector share the transaction system), embeddings are produced by a pluggable Embedder, and retrieval filters tenant/sensitivity before scoring.
Package memory implements the v0.1 Memory decision chain (ADR-009): the canonical store lives in PostgreSQL (FTS + pg_trgm + pgvector share the transaction system), embeddings are produced by a pluggable Embedder, and retrieval filters tenant/sensitivity before scoring.
kernel/model
Package model implements the Model Gateway decision chain of the Fenced kernel: versioned model descriptors with price tables, policy checks, incremental token/cost settlement against the task budget ledger, and the hard stop when a reservation is exhausted (tech baseline §12.1; invariant 12).
Package model implements the Model Gateway decision chain of the Fenced kernel: versioned model descriptors with price tables, policy checks, incremental token/cost settlement against the task budget ledger, and the hard stop when a reservation is exhausted (tech baseline §12.1; invariant 12).
kernel/model/provider
Package provider implements the real model execution layer behind the Model Gateway decision chain: an HTTP executor covering two wire dialects.
Package provider implements the real model execution layer behind the Model Gateway decision chain: an HTTP executor covering two wire dialects.
kernel/model/tokens
Package tokens provides tokenizer-aware token estimation for model budget reservations.
Package tokens provides tokenizer-aware token estimation for model budget reservations.
kernel/money
Package money provides the fixed-point representation used by every accounting ledger.
Package money provides the fixed-point representation used by every accounting ledger.
kernel/observability
Package observability provides the read-side surface for the Phase 7 observability contract: every execution exposes a full correlation chain (Workflow → Step → Task → Run → Attempt → Model/Tool Call → Memory → Result) plus the core metrics the platform guarantees.
Package observability provides the read-side surface for the Phase 7 observability contract: every execution exposes a full correlation chain (Workflow → Step → Task → Run → Attempt → Model/Tool Call → Memory → Result) plus the core metrics the platform guarantees.
kernel/policy
Package policy evaluates default-deny Rego policies inside the kernel process.
Package policy evaluates default-deny Rego policies inside the kernel process.
kernel/recovery
Package recovery reconciles expired runtime leases into fenced retries.
Package recovery reconciles expired runtime leases into fenced retries.
kernel/scheduler
Package scheduler provides deterministic, explainable runtime placement.
Package scheduler provides deterministic, explainable runtime placement.
kernel/store
Package store defines the transactional persistence contract used by kernel controllers.
Package store defines the transactional persistence contract used by kernel controllers.
kernel/store/postgres
Package postgres implements the Fenced kernel persistence contract on PostgreSQL.
Package postgres implements the Fenced kernel persistence contract on PostgreSQL.
kernel/tool
Package tool implements the Tool Gateway decision chain of the Fenced kernel: versioned descriptors, canonical argument normalization, policy evaluation, human approval binding, budget hard-stop and durable side-effect receipts (architecture §9.2, §12.2; invariants 6 and 9).
Package tool implements the Tool Gateway decision chain of the Fenced kernel: versioned descriptors, canonical argument normalization, policy evaluation, human approval binding, budget hard-stop and durable side-effect receipts (architecture §9.2, §12.2; invariants 6 and 9).
kernel/workflow
The orchestrator controller.
The orchestrator controller.
kernel/workload
Package workload defines the bounded task specification understood by admission and placement.
Package workload defines the bounded task specification understood by admission and placement.
mcp
Brokered system tools of the Agent Runtime MCP endpoint: the sandboxed Agent reaches the Model execution layer and the Memory store through the same MCP surface it uses for tenant tools, with the fenced attempt identity and capability grants injected by the runtime worker.
Brokered system tools of the Agent Runtime MCP endpoint: the sandboxed Agent reaches the Model execution layer and the Memory store through the same MCP surface it uses for tenant tools, with the fenced attempt identity and capability grants injected by the runtime worker.
platform/agentmetrics
Package agentmetrics owns Fenced operational metrics.
Package agentmetrics owns Fenced operational metrics.
platform/artifact
Package artifact provides content-addressed Artifact storage adapters.
Package artifact provides content-addressed Artifact storage adapters.
platform/compat
Package compat provides backward-compatibility shims for the AgentOS → Fenced rename.
Package compat provides backward-compatibility shims for the AgentOS → Fenced rename.
platform/errorcode
Package errorcode is the canonical registry for stable public Fenced error codes.
Package errorcode is the canonical registry for stable public Fenced error codes.
platform/grpcx
Package grpcx centralizes the transport options every fenced gRPC boundary uses: message size ceilings above the largest payloads (assignments embed the full workload spec and receipt lists), keepalive so long-lived worker connections detect dead peers promptly instead of hanging on a half-open transport, bounded connect attempts so a down endpoint surfaces quickly instead of wedging the first call, and OpenTelemetry interceptors so every boundary participates in the reference observability stack.
Package grpcx centralizes the transport options every fenced gRPC boundary uses: message size ceilings above the largest payloads (assignments embed the full workload spec and receipt lists), keepalive so long-lived worker connections detect dead peers promptly instead of hanging on a half-open transport, bounded connect attempts so a down endpoint surfaces quickly instead of wedging the first call, and OpenTelemetry interceptors so every boundary participates in the reference observability stack.
platform/migrate
Package migrate applies ordered, checksum-protected SQL migrations.
Package migrate applies ordered, checksum-protected SQL migrations.
platform/opensearch
Package opensearch is the minimal REST client for the OpenSearch projection target (ADR-013).
Package opensearch is the minimal REST client for the OpenSearch projection target (ADR-013).
platform/otel
Package otel wires the reference observability stack (tech baseline §15): OTLP trace/metric/log exporters configured through the standard OTEL_* environment variables, plus a bounded async slog → OTel log bridge.
Package otel wires the reference observability stack (tech baseline §15): OTLP trace/metric/log exporters configured through the standard OTEL_* environment variables, plus a bounded async slog → OTel log bridge.
platform/outbox
Package outbox delivers PostgreSQL outbox events to a durable publisher.
Package outbox delivers PostgreSQL outbox events to a durable publisher.
platform/projector
Package projector applies canonical store events to the OpenSearch search projection (ADR-013).
Package projector applies canonical store events to the OpenSearch search projection (ADR-013).
platform/redact
Package redact contains cross-cutting trust-boundary sanitizers.
Package redact contains cross-cutting trust-boundary sanitizers.
platform/slo
Package slo evaluates immutable v1 GA service objectives from a measured observation window.
Package slo evaluates immutable v1 GA service objectives from a measured observation window.
platform/spiffe
Package spiffe implements the X.509-SVID identity boundary of the Secure Runtime (ADR-011): every fenced gRPC peer presents a certificate whose SPIFFE ID names its exact principal, and every boundary verifies the chain against the trust bundle and matches the presented identity against the claims of the request.
Package spiffe implements the X.509-SVID identity boundary of the Secure Runtime (ADR-011): every fenced gRPC peer presents a certificate whose SPIFFE ID names its exact principal, and every boundary verifies the chain against the trust bundle and matches the presented identity against the claims of the request.
runtime/adapter
Package adapter implements the Runtime Protocol provider that delegates an assignment to a language/framework-neutral Agent Runtime Interface endpoint.
Package adapter implements the Runtime Protocol provider that delegates an assignment to a language/framework-neutral Agent Runtime Interface endpoint.
runtime/attemptstate
Package attemptstate helps Runtime Protocol workers converge after an optimistic-concurrency conflict without weakening fencing.
Package attemptstate helps Runtime Protocol workers converge after an optimistic-concurrency conflict without weakening fencing.
runtime/bindings
Package bindings resolves Runtime Interface endpoints and the transport policy that guards them.
Package bindings resolves Runtime Interface endpoints and the transport policy that guards them.
runtime/broker
gRPC broker clients: the adapter worker's loopback MCP endpoint reaches the Model execution layer and the Memory store through the same fenced gateway services every other runtime uses.
gRPC broker clients: the adapter worker's loopback MCP endpoint reaches the Model execution layer and the Memory store through the same fenced gateway services every other runtime uses.
runtime/control
Package control implements the fenced Kernel-to-Runtime gRPC boundary.
Package control implements the fenced Kernel-to-Runtime gRPC boundary.
runtime/leasekeeper
Package leasekeeper keeps a Runtime Protocol lease alive while a worker executes a workload, and stops the execution when the lease can no longer be renewed.
Package leasekeeper keeps a Runtime Protocol lease alive while a worker executes a workload, and stops the execution when the lease can no longer be renewed.
runtime/oci
Package oci provides the OCI + gVisor (runsc) Runtime Provider worker.
Package oci provides the OCI + gVisor (runsc) Runtime Provider worker.
runtime/reference
Package reference provides a deterministic Runtime Protocol conformance worker.
Package reference provides a deterministic Runtime Protocol conformance worker.
security
Package security is the v0.3 negative test suite: it proves that the four security boundaries hold under attack, not just under happy paths:
Package security is the v0.3 negative test suite: it proves that the four security boundaries hold under attack, not just under happy paths:
version
Package version is the single source of truth for product and public contract versions.
Package version is the single source of truth for product and public contract versions.
sdk
agent
Package agent defines the provider-neutral Agent Runtime Interface used by native SDKs and framework adapters.
Package agent defines the provider-neutral Agent Runtime Interface used by native SDKs and framework adapters.
conformance
Package conformance provides the black-box certification suite every Agent Runtime Interface adapter must pass, independent of language or framework.
Package conformance provides the black-box certification suite every Agent Runtime Interface adapter must pass, independent of language or framework.
provider
Package provider defines the unified Fenced Provider Plugin Protocol.
Package provider defines the unified Fenced Provider Plugin Protocol.
runtimesdk
Package runtimesdk provides the developer kit for building third-party Fenced Runtimes conforming to fenced.runtime.interface/v1.
Package runtimesdk provides the developer kit for building third-party Fenced Runtimes conforming to fenced.runtime.interface/v1.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL