podman-api

podman-api

A small REST wrapper around Podman's libpod REST API that lets a CMS (or any orchestrator) deploy and manage pods described by YAML templates across a fleet of Podman hosts.

It is opinionated, single-binary, and deliberately narrow: one host group, bearer-token auth, audit log to stdout, Prometheus metrics on a separate listener, kubernetes-style secrets, idempotent applies. Use it when Kubernetes is too much and ad-hoc podman run over SSH is too little.

Documentation

This README is the quick reference. The wiki is the operator's handbook:

• Building — why the build needs tags, make targets, static/cross builds.
• Provisioning a Podman Host — turn a fresh box into an SSH-drivable target.
• Deploying — install the daemon: user, config tree, systemd, TLS.
• Operating — key rotation, audit-log shipping, metrics, health checks.
• Troubleshooting — common failures and their fixes.

What it does

• Lists, applies, upgrades, starts/stops, and deletes pod instances on remote Podman hosts via SSH-tunneled podman.sock.
• Renders parameterised templates (Go text/template over Kubernetes-style YAML) into Pod manifests, then plays them with podman play kube.
• Manages per-host and per-instance secrets as Kubernetes Secret resources (so secretKeyRef works).
• Pre-pulls every container image before play, so a bad tag fails fast with a clear registry error instead of a 30-second timeout.
• Streams logs as plain text or SSE.
• Migrates and evacuates instances across hosts as async jobs. Movement is cold-copy (stop → copy volumes → re-apply → verify → reap source) with rollback if the destination doesn't come up.
• Backs up and restores instance volumes on demand: POST .../backup snapshots every volume; POST /backups/{id}/restore restores in-place with content verification. See Backing up and Restoring.
• Exposes Prometheus metrics on a separate, opt-in listener so they aren't world-readable on the public port.

What it doesn't do

• No multi-replica HA; one daemon, one config tree.
• No image registry, no scheduler, no rolling deploy primitives. (Upgrade is single-pod replace-in-place.)
• No webhook callbacks; callers poll (including the async migrate/evacuate jobs).
• No daemon-side placement/scheduler — cross-host moves work, but the client chooses destinations (the daemon executes the move).
• No live migration — cross-host moves are cold (stop → copy volumes → re-apply → verify → reap the source).

Architecture

   ┌────────────┐  HTTPS    ┌────────────────┐  SSH-tunneled  ┌────────────┐
   │ CMS / curl │ ───────▶  │  podman-api    │ ─────────────▶ │ podman.sock│
   │            │           │  (this binary) │   libpod REST  │  on hostN  │
   └────────────┘           └────────────────┘                └────────────┘
                              │            │
                              │            └─▶  /metrics  (separate addr, optional)
                              │
                              └─▶  audit log (stdout, JSON lines)

Hosts are declared once in hosts/*.yaml. Templates live in the store. Bearer keys live in auth/keys.yaml and reload without restart via SIGHUP.

Build

make build          # -> bin/podman-api

The podman v5 bindings require build tags to exclude CGO graph-driver headers. make build carries them. See Building for cross-compile and CI details.

Configure and run

See Deploying for the full setup guide (installer, systemd unit, TLS, bearer keys). The short version:

podman-api \
  -addr=127.0.0.1:8080 \
  -metrics-addr=127.0.0.1:9090 \
  -hosts-dir=/etc/podman-api/hosts \
  -keys-file=/etc/podman-api/keys.yaml \
  -state-db=/var/lib/podman-api/state.db \
  -spec-key-file=/etc/podman-api/spec.key

A contrib/install.sh script creates a dedicated user, installs the binary, and enables the service.

When the background inventory poller is enabled (-inventory-refresh-interval, default 30s), /metrics additionally exposes per-container state rendered from the warm inventory cache:

The last two are not optional decoration. When a host becomes unreachable the cache keeps serving last-known-good data, so podman_api_container_running stays at 1 for the duration of the outage. Any alert built on the container metrics should be gated on podman_api_host_reachable == 1, with a separate alert on staleness.

Resource metrics (#209). With the poller enabled, the same listener also exports per-container resource usage:

Container stats are sampled with one containers.Stats call per host on each inventory tick — but only when that tick's inventory refresh for the host succeeded. A host whose refresh fails is not also charged a second timeout for stats; instead its cached samples are dropped, so its container series go absent on the next scrape rather than freezing at their last value, exactly like the liveness metrics above. Disable the sampler with -container-stats=false (default true — an existing deployment picks up one extra call per host per tick on upgrade, with no flag change needed).

The sample gets its own budget, -container-stats-timeout (default 5s), independent of -inventory-refresh-timeout. It briefly shared the refresh's budget, and on the fleet's busiest host that meant no resource metrics at all: the inventory sweep consumed nearly the full 20s, the refresh still returned success, and the stats call — under 100ms of work even for 115 containers — was cancelled the instant it started. If a host logs container stats unavailable: … context canceled while its liveness series are fine, check that it is running a build with this flag. Repeated container stats unavailable / container stats available again now points at the sampler itself: raise -container-stats-timeout, not -inventory-refresh-timeout. The metrics stay honest either way — a missed sample renders absent, never stale.

Two independent budgets means a host's worst-case cost for one tick is -inventory-refresh-timeout + -container-stats-timeout, and a tick blocks the next one, so keep that sum under -inventory-refresh-interval (the shipped defaults, 20s + 5s < 30s, satisfy it comfortably). Raising a timeout past that line does not refuse to start — it logs a warning naming all three flags, and the consequence is a stretched poll cadence fleet-wide, visible as a rising podman_api_inventory_age_seconds.

The cumulative series (_total suffix) reset when a pod is recreated, exactly as podman_api_container_restarts_total does; graph them with rate() or increase(), not as raw counters. Podman's own CPU/memory percentages are not exported: for a non-streaming stats call they're averaged against the container's start time, so a container busy at boot and idle since would read as permanently hot — the raw counters and rate() avoid that.

Volume usage runs on its own, much slower loop, independent of the inventory tick, on -volume-usage-interval (default 1h, 0 disables it) with -volume-usage-timeout (default 5m) — podman system df walks the whole store and can take minutes, so it must never delay an inventory refresh. It also does an immediate first walk at startup, so on a large store that walk can run for minutes concurrently with boot. A failed walk leaves the previous sizing in place; podman_api_volume_usage_age_seconds climbing is how a wedged or slow walk becomes visible.

Attributing a size to an instance needs the volume in the inventory, so the inventory sweep now populates every volume the template declares, without inspecting it — the sweep must make no podman call per volume or it would cost one round trip per volume per host per tick. That changes what the list route reports: GET /hosts/{host}/instances lists a template's declared volumes, including any that do not exist on the host yet, always with size_bytes 0, whereas GET /hosts/{host}/instances/{template}/{slug} still inspects and so lists only volumes that really exist, with their real sizes. Treat the list route's volumes as names, and the single-instance route as the authority on existence and size.

Neither sampler can affect podman_api_host_reachable: the Grafana Infrastructure Alerts rules gate on it, so a slow system df or a stats timeout must never be able to silence them.

Admin UI

An embedded, server-rendered admin UI (HTMX + PureCSS) is served at /ui. Disabled unless -operator-file <path> is set. Pass -ui-secure-cookie when serving over HTTPS.

Generate a password hash:

podman-api hash-token <your-password>

API reference

A complete OpenAPI 3.0 spec lives at api/openapi.yaml and is served by the binary at GET /openapi.yaml (no auth).

GET    /healthz
GET    /metrics                                                      separate listener

GET    /hosts
GET    /hosts/{host}
GET    /hosts/{host}/healthz
GET    /hosts/{host}/ports-in-use
POST   /hosts/{host}/rename                     body: {"new_id": "..."}

GET    /templates
GET    /templates/{id}
GET    /templates/{id}/render?<params>
POST   /templates                            body: {id, body, ...}
PUT    /templates/{id}                       body: {body, ...}
DELETE /templates/{id}?force=
POST   /templates/{id}/clone                 body: {new_id}

GET    /hosts/{host}/secrets
PUT    /hosts/{host}/secrets/{name}          body: {"value": "..."}
DELETE /hosts/{host}/secrets/{name}

GET    /hosts/{host}/instances?template=<id>
GET    /hosts/{host}/instances/{template}/{slug}
POST   /hosts/{host}/instances
PUT    /hosts/{host}/instances/{template}/{slug}
DELETE /hosts/{host}/instances/{template}/{slug}?prune_volumes=&prune_secrets=

POST   /hosts/{host}/instances/{template}/{slug}/start
POST   /hosts/{host}/instances/{template}/{slug}/stop
POST   /hosts/{host}/instances/{template}/{slug}/restart
POST   /hosts/{host}/instances/{template}/{slug}/upgrade  body: {"image": "..."}
PATCH  /hosts/{host}/instances/{template}/{slug}/parameters  body: {"parameters": {...}}
PATCH  /hosts/{host}/instances/{template}/{slug}/secrets     body: {"secrets": {...}}

GET    /hosts/{host}/instances/{template}/{slug}/logs?container=&tail=&follow=
GET    /hosts/{host}/instances/{template}/{slug}/volumes
DELETE /hosts/{host}/volumes/{name}

POST   /migrate    body: {from_host, to_host, template, slug, parameters?}
POST   /evacuate   body: {from_host, map:{slug: to_host}}

POST   /hosts/{host}/instances/{template}/{slug}/backup
GET    /hosts/{host}/instances/{template}/{slug}/backups?limit=
POST   /backups/{id}/restore
DELETE /backups/{id}

GET    /jobs?state=&kind=&parent_id=
GET    /jobs/{id}

Notes:

• POST /migrate and POST /evacuate require -state-db (else 501), validate synchronously, and return 202 {job_id}. Poll GET /jobs/{id} for progress.
• POST /hosts/{host}/rename returns 409 for two distinct reasons, told apart by the error code: host_already_exists (another host, or stale store state, already holds new_id) and host_has_backups (the host has backup rows; their blob keys are not re-keyed, so renaming would make existing backups unreachable — delete them first). 501 not_implemented means the server was built/wired without the host-rename dependency.
• ?skip_pull=true on POST/PUT skips the pre-pull step.
• The two PATCH routes change one instance in place and reuse its sealed per-instance secrets, so they work on an instance whose secret plaintext nobody can read back (a full PUT requires every declared secret). Both merge — omitted parameters/secrets keep their stored value, and neither can delete one. Both replace the pod.
• On DELETE, prune_volumes and prune_secrets default to false. Pass both as true to reap volumes and secrets. DELETE is idempotent — a prune-requested delete on an already-gone pod still removes orphaned volumes/secrets and returns 204.