host-api

Host API Example

This example exposes the host-side runtime skeleton over HTTP. It is not a new core module; it shows how a host application can compose:

• workflowkit for workflow lifecycle and approval resume.
• goagent for the agent run.
• llmkit for model routing, model stats, and provider health.
• artifactkit for durable payload refs.
• runkit for durable agent run audit records and events.

Optional governed project memory

Project memory is an optional injection-based composition. When Config.Memory is nil, the existing Host behavior and route set are unchanged. When enabled, the caller must inject one complete, fail-closed runtime: a caller-owned memorykit.Store, a real project authorizer, content validator, explicit limits/policies/clocks, recall adapters, durable extraction queue, and embedding/extraction workers. The Host does not create a default authorizer or an allow-all fallback, and Server.Close does not take ownership of the memory Store.

An enabled workflow requires project_id. memory_write_intent: true requests the separate memory.write_explicit capability; it does not grant that capability. The verified tenant/user identity and project are persisted as trusted Host metadata so restart, requeue, and approval resume retain the same Scope. V1 constructs only project Scope and rejects caller-supplied tenant, subject, or subject_type=user.

Automatic recall projects a bounded active/effective project view before the model call. For a production workflow, the Host verifies the trusted workflow ID, expected input Artifact ref, and current ref-only user message, then reads the UTF-8 Artifact body transiently as the recall query. The body is not copied into Agent metadata/checkpoints and the ref-only user message is unchanged. Blank, NUL-containing, malformed, or over-policy query bodies fail closed and are never truncated. The always-available search_memory and read_memory tools provide on-demand recall; remember_project_memory appears only for authorized explicit write intent. Model-derived extraction writes candidates, never active memory. Recoverable vector failure keeps exact/full-text recall, while invalid Scope or integrity fails closed.

When memory is enabled, these governed routes are registered:

• GET /projects/{projectID}/memories
• GET /projects/{projectID}/memories/{memoryID}
• GET /projects/{projectID}/memories/{memoryID}/revisions
• POST /projects/{projectID}/memories
• POST /projects/{projectID}/memories/{memoryID}/activate
• POST /projects/{projectID}/memories/{memoryID}/dismiss
• POST /projects/{projectID}/memories/{memoryID}/correct
• POST /projects/{projectID}/memories/{memoryID}/forget
• POST /projects/{projectID}/memories/{memoryID}/erase

Read, explicit write, candidate review, and privacy erase are separate capabilities. Bodies use strict JSON and optimistic expected_version confirmation; they never accept Scope fields. See memorykit/README.md for pgvector setup, recall and degradation semantics, lifecycle differences, and the required PostgreSQL gate.

Local MVP acceptance

The shortest path that needs no real credentials for external or cloud providers is the real-process black-box smoke. From this directory, first run the default package tests, then the complete MVP acceptance suite:

go test ./... -count=1
go test -v -tags hostapisystemsmoke \
  -run '^TestHostAPIProcessMVPBlackBoxClosure$' \
  -count=1 ./...

The tagged suite requires macOS, CGO, an interactive login session, and an unlocked login Keychain. It starts loopback-only OIDC and OpenAI-compatible test providers, uses a synthetic API key, and keeps each scenario in an isolated temporary runtime home. It never loads a real Qwen or cloud-provider credential.

Each scenario uses a separate Keychain service/account pair whose service contains .smoke.. Cleanup deletes only that exact pair and refuses non-smoke services. The suite never accesses or deletes the production-default item. That item uses service goagents.host-api.approvals and account approval-data-key:local-v1.

The three subtests cover approval plus Skill restart recovery, Provider 503 followed by HTTP requeue, and fail-closed handling of an unregistered tool. All three must report PASS. A SKIP means the environment is blocked and is not a successful MVP acceptance result.

Real Provider acceptance

The external Provider gate is separate from the deterministic local smoke. It uses a caller-supplied OpenAI-compatible endpoint and verifies a text response, one typed-tool observation round trip, structured-output success, fail-closed schema validation, and stable authentication/timeout error classification:

export OPENAI_COMPAT_BASE_URL=https://provider.example/v1
export OPENAI_COMPAT_MODEL=provider-model
export OPENAI_COMPAT_API_KEY=... # supply through the local shell or secret store

go test -v -tags provideracceptance \
  -run '^TestRealProviderMVPAcceptance$' \
  -count=1 ./...

On success, the test logs only subtest status. Failure diagnostics are limited to non-sensitive tool metadata and stable error type/class/code. It does not log the endpoint, model identifier, credential, prompt, model response, random observation nonce, or raw Provider error. Missing configuration produces SKIP, which is an environment blocker rather than a passing acceptance result. The deliberately invalid credential used by the negative subtest is a fixed synthetic value.

Real Host local trial

The opt-in composition gate combines a real OpenAI-compatible Provider with the actual Host binary, SQLite runtime state, loopback OIDC verification, and a temporary .smoke. Keychain item. It creates a workflow, rejects an invalid approval token, restarts at the persisted final-approval boundary, completes approval with a valid token, restarts again, and checks the workflow, AgentRun, route, and event evidence:

export OPENAI_COMPAT_BASE_URL=https://provider.example/v1
export OPENAI_COMPAT_MODEL=provider-model
export OPENAI_COMPAT_API_KEY=... # supply through the local shell or secret store

go test -v -tags 'hostapisystemsmoke provideracceptance' \
  -run '^TestHostAPIProcessRealProviderLocalTrial$' \
  -count=1 ./...

This gate needs an interactive macOS login session with an unlocked Keychain. It does not enter default CI and it does not replace a production IdP test: the Provider is real, while OIDC discovery/JWKS and the bearer token are local and synthetic. Missing Provider configuration or Keychain access produces SKIP, which is an environment blocker rather than a passing trial.

Single-host stability gate

The stability gate targets an Apple M1 Pro with 10 CPU cores, 16 GiB RAM, and macOS 26.5.1. It runs three waves of 50 queued workflows with 10 concurrent HTTP submitters, performs OIDC final approval, restarts the real Host process, and checks SQLite convergence plus RSS/file-descriptor bounds:

go test -tags hostapisystemsmoke \
  -run '^(TestWaitForStabilityWorkflowsRejectsMatchAfterDeadline|TestOpenStabilityDatabaseReadOnly)$' \
  -count=1 ./...

go test -v -tags hostapisystemsmoke \
  -run '^TestHostAPIProcessMVPStability$' \
  -count=1 ./...

The first command is a fast negative-regression gate for wall-clock convergence deadlines and the final SQLite read-only connection. The second command runs the full process workload.

This is a correctness and leak-regression gate for the current single-process, single-slot worker. It is not a production throughput SLO. A SKIP means the required macOS, Keychain, ps, or lsof environment is unavailable and is not a passing acceptance result. The test uses only loopback Provider/OIDC servers, synthetic credentials, temporary runtime state, and its exact .smoke. Keychain service/account pair.

Shutdown boundary

The process handles SIGINT and SIGTERM in two stages.

• The first signal closes HTTP intake, stops new queued claims and janitor scans, and lets already accepted workflows and approvals drain. Configure the positive Go duration with HOST_API_SHUTDOWN_TIMEOUT; it defaults to 30s.
• If drain completes, stores close and the process exits 0. Pending workflows that were not claimed remain available when a process starts with the same HOST_RUNTIME_HOME; completed or stable waiting_approval workflows remain unchanged.
• A second signal skips the remaining drain wait. A drain timeout does the same. Both paths cancel active execution, conditionally fail unfinished state with host_shutdown_timeout, clear owned leases, and use one fixed 5s budget for force stop plus close. A completed force path exits 5 with shutdown_timeout; exhausting cleanup exits 5 with shutdown_cleanup_timeout.

New long-running workflow or approval operations that reach the registration boundary after drain begins receive 503 host_draining. Short queued creation and requeue transactions remain request-scoped.

A workflow force-closed with host_shutdown_timeout is not automatically executed after restart. After checking whether an external effect already committed, an operator may explicitly call POST /workflows/{id}/requeue. The retry may issue the external request again. Irreversible write tools must use an idempotent external API or deduplicate by stable ToolCallID.

Graceful shutdown narrows controlled-signal failure windows; it does not make an external side effect and the local checkpoint one transaction, and it does not guarantee exactly-once behavior at arbitrary crash points.

By default, the example creates a temporary runtime directory. Set HOST_RUNTIME_HOME to make workflow state, agent run audit, artifacts, and llmkit audit survive process restarts:

$HOST_RUNTIME_HOME/
  workflow.db
  agent-runs.db
  artifacts/
  .llmkit/

LLM Configuration

If $LLMKIT_HOME/config.yaml exists, host-api loads candidates and OpenAI-compatible providers from that file. If LLMKIT_HOME is unset, it defaults to $HOST_RUNTIME_HOME/.llmkit.

If no config.yaml exists, the example falls back to deterministic demo models:

• local-free: local, free, simple, fast.
• cloud-advanced: cloud, high price, advanced, normal latency.

If config.yaml exists but references an unset api_key_env, startup fails. This is intentional: configured providers should fail closed instead of making unauthenticated provider calls.

Minimal config:

accounts:
  - alias: local-dev
    provider: openai_compatible
    base_url: http://127.0.0.1:11434/v1
  - alias: cloud-prod
    provider: openai_compatible
    base_url: https://api.example.com/v1
    api_key_env: CLOUD_PROD_API_KEY

models:
  - alias: local-free
    model: qwen2.5:7b
    provider: openai_compatible
    account_alias: local-dev
    is_local: true
    capability_level: simple
    context_window_class: medium
    price_class: free
    latency_class: fast
    estimated_cents: 0
  - alias: cloud-advanced
    model: advanced-model
    provider: openai_compatible
    account_alias: cloud-prod
    capability_level: advanced
    context_window_class: long
    price_class: high
    latency_class: normal
    estimated_cents: 8

Host-api refreshes model-stats.json from llmkit audit files before route decisions and when serving /llmkit/models. That makes new outcomes from a long-running process visible without waiting for restart.

Approval Authentication

Workflow approval requires an OIDC bearer token. The CLI refuses to start unless both variables are configured:

export HOST_API_OIDC_ISSUER=https://id.example.com
export HOST_API_OIDC_AUDIENCE=goagents-host-api

At startup host-api discovers the issuer and verifies approval tokens through its JWKS. Both POST /workflows/{id}/agent-approve and POST /workflows/{id}/approve accept Authorization: Bearer <OIDC JWT> and record the verified sub claim as the approver. They never accept an approver identity from the request body or persist the bearer token. Missing or invalid tokens return 401 unauthorized.

Endpoints

Endpoints:

• POST /workflows
• GET /skills
• GET /workflows
• GET /workflows/{id}
• GET /workflows/{id}/events
• POST /workflows/{id}/requeue
• POST /workflows/{id}/agent-approve
• POST /workflows/{id}/approve
• GET /workflows/{id}/llm-routes
• GET /agent-runs/{id}
• GET /llmkit/models
• GET /workers/queued

Host-owned skills

GET /skills exposes a safe catalog view for a host-provided skill snapshot. Each item contains only its name, description, immutable digest, scope, availability state, and stable availability reason codes/subjects. It never returns skill manifests, instruction or resource bodies, package/root paths, or other host-local catalog details.

The default CLI discovers no Skills unless HOST_API_SKILL_ROOT names one absolute local directory. Setting the variable explicitly trusts that directory as the user-scope root for the lifetime of the process:

export HOST_API_SKILL_ROOT="$PWD/skills"

The catalog is rebuilt only at process startup. The default CLI gate supplies the current OS but no host features or allowed tools, so instruction-only Skills can activate while capability-requiring Skills remain unavailable. HTTP callers cannot add roots, change trust, or refresh the catalog.

POST /workflows accepts an optional skill_refs array. Each element supplies name and may also supply digest:

{
  "id": "wf-review-1",
  "input": "Review this high-risk policy.",
  "skill_refs": [
    {"name": "workflow-review"},
    {"name": "checklist", "digest": "sha256:..."}
  ]
}

The host resolves every requested name against that fixed catalog, evaluates it with the fixed gate context, and persists only complete name/digest pairs in workflow metadata. Workflow creation and workflow responses therefore always return the complete resolved digest, even if the request omitted it. The digest is the immutable selection: startup recovery, process restart, requeue, and agent-approval resume reactivate the same name@digest from stored metadata; they never silently reselect the current skill by name.

This boundary is fail-closed. Catalog absence, an unknown skill, a missing or changed digest, malformed stored references, or a gate failure rejects workflow creation or prevents the resumed agent from running. In particular, a workflow with persisted skill references cannot resume against a newer same-named skill.

This slice activates only skill instructions through the existing agent skill provider. It does not execute skill scripts, automatically grant permissions, or project skill-declared tools into the agent tool registry. Tool projection is an explicit later host-policy decision.

GET /workflows/{id}/llm-routes returns the sanitized llmkit routing audit for that workflow: effective task profile, selected model/account aliases, provider, reason, score breakdown, candidate aliases, full candidate-level scores or filter reasons, and outcome metadata such as success, latency, tokens, and estimated cents. It does not return prompts, responses, headers, or API keys.

GET /workflows returns a bounded operational list. It accepts optional status, run_mode, order, and limit query parameters, for example GET /workflows?status=pending&run_mode=queued&order=desc&limit=50. Results are ordered by workflow creation time, then workflow id; order=desc reverses both.

GET /workflows/{id}/events returns a workflow operation timeline. Step records are returned as type: "step" events with step name, status, attempt, refs, errors, and timestamps. Manual requeues are returned as type: "workflow_requeued" events.

POST /workflows accepts optional run_mode. sync is the default and runs the workflow during the HTTP request. queued is an in-process proof: it writes the input artifact and pending workflow, returns immediately, then a background worker loop claims a workflowkit.QueueLeaseStore lease. Each Server has one execution slot: create and requeue only send a bounded wake signal, and the worker drains SQLite serially until no runnable workflow remains. It advances each workflow until waiting_approval or terminal, extends the lease while the workflow is running, and releases the lease. StartQueuedWorker starts at most one loop during a Server lifetime. Without that call, queued workflows remain pending; the CLI calls it on boot, so restarting with the same HOST_RUNTIME_HOME can recover pending or expired-lease workflows. It is still not a distributed worker model; worker crash supervision and multi-worker scheduling are intentionally not implemented.

POST /workflows/{id}/requeue is an explicit operator action for failed or cancelled workflows. It moves the existing workflow back to pending, preserves step history, records run_mode: "queued", and lets the queued worker retry the unfinished work. It also records a workflow_requeued event in workflow metadata. It does not create a new workflow id.

GET /workers/queued returns in-process worker observability: whether the worker loop has been started, the worker id, claim/completion/idle/error counts, lease extension and heartbeat error counts, the last workflow id, and the latest error. These counters are process-local diagnostics, not durable metrics.

POST /workflows also accepts optional task_profile_preset and task_profile so a host can describe the task before routing:

{
  "id": "wf-review-1",
  "input": "Review this high-risk policy.",
  "task_profile_preset": "high_success",
  "task_profile": {
    "task_type": "policy_review",
    "needs_reasoning": true,
    "max_estimated_cents": 10
  }
}

Available presets:

• simple_local: simple, low failure cost, local-preferred.
• balanced: medium complexity, medium failure cost, cloud allowed.
• high_success: hard, high failure cost, cloud allowed, reasoning required.
• local_only: simple, low failure cost, local-only.

If both fields are present, task_profile_preset provides the base profile and task_profile patches it. Missing fields inherit the preset/default profile. Present string fields must be non-empty. Present boolean fields explicitly override the base profile, including false. Invalid or unroutable profiles return invalid_task_profile; for example, local_only plus complexity: hard fails when no local advanced model exists. The routing decision is visible through GET /workflows/{id}/llm-routes.

Set task_profile.needs_tools=true only when the selected model declares tool support. The built-in deterministic demonstration models do so and request one local record_review write tool. It only writes artifact:<workflow-id>:review-action; it never publishes or performs network side effects. Before that write, the agent returns agent_approval with an opaque checkpoint ID plus tool call identity. Resolve it through POST /workflows/{id}/agent-approve using the exact index, tool_call_id, tool, and allowed fields. The endpoint rejects unknown request fields, including free-form reasons. An allowed decision resumes the agent once but still requires the existing POST /workflows/{id}/approve final-output step. A denied decision decrypts nothing, runs no tool, and cancels the workflow.

For a real local macOS tool pause, the host lazily creates a 32-byte data key and persists only a versioned AES-GCM envelope in agent-runs.db. Production defaults to Keychain service goagents.host-api.approvals and key ID local-v1. Hosts may set HOST_API_AGENT_APPROVAL_KEYCHAIN_SERVICE and HOST_API_AGENT_APPROVAL_KEY_ID together to select another host-owned item; these environment variables are lookup identifiers, never key material. When both raw environment values are exactly empty (""), whether unset or explicitly set empty, the production defaults apply. In every other case, both trimmed values must be non-empty: partial configuration or any whitespace-only value, including two whitespace-only values, fails startup. There is still no file, environment-variable, or SQLite key fallback. Default unit and integration tests inject a cipher and do not access the machine Keychain.

On an interactive macOS login session, run the optional real-process smoke to exercise the actual binary, local OIDC discovery/JWKS verification, SQLite restart recovery, and Keychain-backed tool approval:

go test -tags hostapisystemsmoke -run TestHostAPIProcessToolApprovalSurvivesRestart -count=1 -v

It starts a loopback-only OIDC issuer and uses a temporary runtime directory. This hostapisystemsmoke test is the exception to the default tests above: it uses a test-only Keychain item whose unique service has the fixed prefix goagents.host-api.approvals.smoke.. Both host subprocesses reuse that same test item across the restart. Before deletion, cleanup verifies the fixed prefix and then deletes only the exact service/account pair created by this test. It never accesses or deletes the production-default goagents.host-api.approvals/local-v1 item, and it never prints or exports key material. It skips when the current process cannot access an unlocked login Keychain, and therefore a skip is not evidence of a passed smoke. Default go test ./... and bash ../../scripts/verify-all.sh do not run it.

An agent tool approval expires one hour after the pause. The host process starts an in-process janitor that defaults to a one-minute sweep interval; set HOST_API_AGENT_APPROVAL_SWEEP_INTERVAL to another positive Go duration such as 30s. On expiry the checkpoint remains fail-closed, the correlated agent run and workflow become failed with agent tool approval expired, and the tool is never replayed. A temporary local persistence failure leaves the workflow waiting so the next sweep retries reconciliation.

GET /llmkit/models returns:

• models: current routable model aliases and coarse capability metadata.
• health: in-memory provider health snapshot.
• stats: generated model statistics grouped by task type, account, model, and provider.

POST /workflows/{id}/approve records a business outcome signal on the selected LLM route: business_outcome=success and success_signal=human_accepted. Its JSON body accepts only an optional note; the approver identity comes from the verified OIDC bearer token. Provider-level success remains available in the same route outcome.

The agent review step uses strict host-owned persistence for the critical audit path. If writing the agent output artifact or terminal run summary fails, the workflow step fails instead of moving to waiting_approval with broken refs.

See ../../docs/host-api-contract.md for the prose contract and openapi.yaml for the machine-readable HTTP contract.

Run the long-lived service

go run . is not a zero-configuration command. The service fails closed unless it can discover a real OIDC issuer and verify approval bearer tokens through that issuer's JWKS:

export HOST_API_OIDC_ISSUER=https://id.example.com
export HOST_API_OIDC_AUDIENCE=goagents-host-api
go run .

Set HOST_RUNTIME_HOME to retain state across restarts and HOST_API_ADDR to choose the listen address. If LLMKIT_HOME is unset, it defaults to $HOST_RUNTIME_HOME/.llmkit. Set HOST_API_QUEUED_LEASE_DURATION to tune the in-process queued worker lease duration; it accepts Go durations such as 30s or 2m and defaults to 1m. Set HOST_API_SHUTDOWN_TIMEOUT to a positive Go duration to change the graceful drain timeout from its 30s default; the force-stop and close budget remains fixed at 5s.

There is intentionally no switch that disables OIDC or accepts an approver identity from the request body. For a local proof without real external or cloud-provider credentials, use the MVP acceptance smoke above.