exitmesh-agent

module

v0.16.0 Latest Latest Go to latest Published: Oct 4, 2026 License: Apache-2.0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/cloud-exit/exitmesh-agent

Links

Open Source Insights

README ¶

ExitMesh Telemetry Agent

The open source ExitMesh Telemetry Agent. One Go binary, exitmesh-agent, runs in the three roles shown above.

In a Kubernetes cluster

Node agent (DaemonSet, one per node): scrapes the kubelet and cAdvisor into a local TSDB, tails pod logs, evaluates PromQL and LogQL rules against bounded local windows, and tracks findings. Findings and per-resource metric facts wait in a durable local queue until the coordinator acknowledges them.
Node agent to coordinator: node agents submit over HTTPS on the coordinator ClusterIP Service (port 8443), authenticated with their projected ServiceAccount token, and long-poll the coordinator for the current rule bundle and investigation tasks. Nothing outside the cluster network is involved.
Coordinator (StatefulSet, one replica): watches cluster state through informers, keeps a compact normalized model of it, records every meaningful change as a restorable delta, evaluates CEL state rules and cluster PromQL rules, and merges node records into one hash-chained spool on its volume.

On a Linux host

Host agent (systemd service, single process): runs the node and coordinator roles together, with no Kubernetes components and no node API hop. It collects host state, node_* metrics, the journal, and log files.

To ExitMesh

Coordinator or host agent to ExitMesh: the only connection that leaves your network is an outbound WebSocket tunnel to your ExitMesh endpoint. Checkpoint, delta, range, and finding records flow out over the open History Protocol (EMHP v1).
ExitMesh back to the agent: over that same tunnel, ExitMesh commits records by sequence and chain hash (the spool keeps everything until then), delivers signed rule bundles, and calls investigation tools. ExitMesh owns the rule bundle trust roots; the agent only verifies bundles against the public half.

The agent is licensed under the Apache License 2.0.

Read-only and outbound-only

Secret inventory and privacy. Secrets are collected with their identity, ownership, type, immutable flag, and data key names. Every value in data and stringData is replaced unconditionally with <redacted> before caching, storage, or transit to ExitMesh. Secret annotations are dropped because they can contain copied values. Helm release names, revisions, and statuses come from storage-object labels; release payloads remain redacted and are never decoded. ExternalSecret conditions, target Secret names, and workload references expose failed synchronization and affected workloads.
Custom-resource inventory. Custom resources are discovered from the CRD watch by default and exported as bounded identity and condition projections. Independent discovery controls, include/exclude filters, and kind/scope limits keep collection bounded without delaying initial synchronization.
Inventory permissions. Automatic coverage grants the coordinator read-only list and watch across API groups and resources within the selected scope. Secret responses are redacted locally before entering the inventory pipeline. No write verbs are granted.
Read-only. Kubernetes access is get, list, and watch only. No writes of any kind, no exported Secret values, no tokenreviews or subjectaccessreviews, no pods/exec, pods/attach, pods/portforward, nodes/proxy, or nodes/log. The agent never executes commands and never self-updates.
One disclosed host directory. Node agents mount /var/log/pods read-only and write only to /var/lib/exitmesh (mode 0700, capped at 1 GiB by the agent). Hosts run as the unprivileged exitmesh user under a hardened systemd unit whose only writable path is /var/lib/exitmesh.
Outbound-only. The coordinator (or host agent) dials out to your ExitMesh endpoint over a WebSocket tunnel. Nothing listens outside the cluster network; node agents reach the coordinator over a ClusterIP Service. Shipped NetworkPolicies make default-deny namespaces work unchanged.
Durable store and forward. Every checkpoint, delta, metric fact, and finding is spooled before it counts as emitted and is resent byte-identical until committed, so outages leave no gaps.
Inspectable rules. Rule bundles are signed for integrity, not secrecy. Every rule that runs on your infrastructure can be read on the coordinator volume or under /var/lib/exitmesh.

Secret collection retains structure like this, then exports it as normalized inventory fields:

apiVersion: v1
kind: Secret
metadata:
  name: sh.helm.release.v1.hcloud-csi.v4
  namespace: kube-system
  labels:
    owner: helm
    name: hcloud-csi
    status: deployed
    version: "4"
type: helm.sh/release.v1
data:
  release: <redacted>

Quickstart: Kubernetes

Your ExitMesh workspace needs an agent trust root first: a workspace administrator generates it once in ExitMesh under Settings > General > Encryption > Agent trust root (Generate trust root). Its private key stays in the ExitMesh Vault, and agent connections cannot be created until it exists. Then copy the endpoint, enrollment token, and trust root (the public half of that key, which signs the workspace's rule bundles) from the ExitMesh connector page:

helm install exitmesh-agent oci://ghcr.io/cloud-exit/charts/exitmesh-agent \
  --version 0.1.0 \
  --namespace default \
  --set endpoint=https://<endpoint from the connector page> \
  --set enrollment.token=<enrollment token> \
  --set-json 'trust.roots=["<root id>:<root public key>"]'

The chart creates two namespaces: exitmesh-node (Pod Security privileged, required for the hostPath volumes) and exitmesh (Pod Security restricted). The release record lives in the namespace passed with --namespace. Profiles, GitOps installs, and pre-created namespaces are covered in docs/install-kubernetes.md.

Quickstart: Linux hosts

# Debian and Ubuntu
sudo apt install ./exitmesh-agent_0.1.0_amd64.deb
# RHEL, Fedora, Rocky, Alma
sudo dnf install ./exitmesh-agent-0.1.0-1.x86_64.rpm

sudo sed -i 's#^endpoint: ""#endpoint: "https://<endpoint from the connector page>"#' /etc/exitmesh/agent.yaml
sudo sed -i 's#^  roots: \[\]#  roots: ["<root id>:<root public key>"]#' /etc/exitmesh/agent.yaml
printf '%s\n' '<enrollment token>' | sudo install -m 0640 -o root -g exitmesh /dev/stdin /etc/exitmesh/enrollment-token
sudo systemctl start exitmesh-agent

Groups, golden images, and the tarball install are covered in docs/install-host.md.

Documentation

Topic	Document
Kubernetes install, profiles, GitOps	docs/install-kubernetes.md
Host install	docs/install-host.md
Configuration reference	docs/configuration.md
Security model	docs/security.md
Operations runbook	docs/operations.md
Upgrades	docs/upgrades.md
Air-gap profile	docs/airgap.md
Uninstall and removal	docs/uninstall.md
Architecture and internal contracts	docs/architecture.md
History Protocol specification	protocol/SPEC.md
Command line	docs/cli.md
Rule bundle format and signing	docs/bundle-format.md
State rules (CEL)	docs/state-rules.md
PromQL rules	docs/promql-rules.md
LogQL subset	docs/logql-subset.md
Default rules and insight coverage	docs/insight-coverage.md
Investigation tools and lookback	docs/investigation.md
Kubernetes field catalog	docs/field-catalog.md
Published kube_* series	docs/kube-series.md
Host facts and availability	docs/host-facts.md
Log tailing contract	docs/log-contract.md

Building

make build        # static binary in bin/, CGO_ENABLED=0, -trimpath
make test         # unit tests
make chart-test   # helm lint and chart assertions

See CONTRIBUTING.md for every quality gate, the DCO sign-off, and dependency rules. Report vulnerabilities as described in SECURITY.md.

The mounted enrollment-token Secret is authoritative at coordinator startup. Rotation within the same target refreshes the credential and preserves queued history. Starting with v0.15.0, a token for a different target is validated first, then the old spool is retained as an archive and a fresh writer starts for the new target. Update the Secret and upgrade/restart the coordinator; do not delete the PVC. See token rotation for upgrade behavior and recovery limits.

Directories ¶

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL

Path	Synopsis
cmd
exitmesh-agent command Command exitmesh-agent runs the ExitMesh Telemetry Agent in the node, coordinator, or host role and provides its administration subcommands (docs/cli.md).	Command exitmesh-agent runs the ExitMesh Telemetry Agent in the node, coordinator, or host role and provides its administration subcommands (docs/cli.md).
exitmesh-bundle command Command exitmesh-bundle builds, signs, verifies, and inspects rule bundles (docs/bundle-format.md).	Command exitmesh-bundle builds, signs, verifies, and inspects rule bundles (docs/bundle-format.md).
exitmesh-refcp command Command exitmesh-refcp serves the reference control plane over TLS with a generated self-signed CA for local testing.	Command exitmesh-refcp serves the reference control plane over TLS with a generated self-signed CA for local testing.
internal
admin Package admin is the local unix-socket administration API used by the CLI while the agent holds its lock.	Package admin is the local unix-socket administration API used by the CLI while the agent holds its lock.
config Package config loads and validates the agent configuration file (docs/configuration.md).	Package config loads and validates the agent configuration file (docs/configuration.md).
coordinator Package coordinator wires the coordinator role: state collection, the spool and writer session, cluster rule evaluation, findings, the node agent API, and the local administration socket.	Package coordinator wires the coordinator role: state collection, the spool and writer session, cluster rule evaluation, findings, the node agent API, and the local administration socket.
deploytest Package deploytest renders the Helm chart and checks the repository's deployment, packaging, and CI assets.	Package deploytest renders the Helm chart and checks the repository's deployment, packaging, and CI assets.
deploytest/releaseinfo command Command releaseinfo prints the protocol, schema, and rule engine versions for release notes as KEY=VALUE lines.	Command releaseinfo prints the protocol, schema, and rule engine versions for release notes as KEY=VALUE lines.
findings Package findings turns rule and query observations into finding episodes and records (PRD 7.7).	Package findings turns rule and query observations into finding episodes and records (PRD 7.7).
host Package host runs the host role: node agent and coordinator in one process on a Linux host without Kubernetes.	Package host runs the host role: node agent and coordinator in one process on a Linux host without Kubernetes.
hostfacts Package hostfacts collects normalized Linux host state, reports per-fact availability, and tracks it as protocol ops.	Package hostfacts collects normalized Linux host state, reports per-fact availability, and tracks it as protocol ops.
hostfacts/journal Package journal is a pure-Go, read-only reader of the systemd journal file format.	Package journal is a pure-Go, read-only reader of the systemd journal file format.
hostfacts/nodemetrics Package nodemetrics runs upstream node_exporter collectors in process so node_* series and upstream rules work unchanged.	Package nodemetrics runs upstream node_exporter collectors in process so node_* series and upstream rules work unchanged.
hostfacts/sqlitedb Package sqlitedb reads rowid tables of SQLite files (with committed WAL frames) in pure Go and serves them to go-rpmdb as a database/sql driver.	Package sqlitedb reads rowid tables of SQLite files (with committed WAL frames) in pure Go and serves them to go-rpmdb as a database/sql driver.
investigate Package investigate serves bounded live and lookback investigation tools with AST scope injection (docs/investigation.md).	Package investigate serves bounded live and lookback investigation tools with AST scope injection (docs/investigation.md).
kv Package kv is the small durable key-value store used for agent metadata: alert state, log offsets, bundle state, key manifest sequence, cursors.	Package kv is the small durable key-value store used for agent metadata: alert state, log offsets, bundle state, key manifest sequence, cursors.
node Package node wires the node agent role: scrape, logs, rules, findings, and delivery to the coordinator.	Package node wires the node agent role: scrape, logs, rules, findings, and delivery to the coordinator.
nodeapi Package nodeapi implements the in-cluster HTTPS API between node agents and the coordinator (docs/architecture.md).	Package nodeapi implements the in-cluster HTTPS API between node agents and the coordinator (docs/architecture.md).
privdrop Package privdrop re-executes the process as an unprivileged user that keeps selected capabilities as ambient capabilities.	Package privdrop re-executes the process as an unprivileged user that keeps selected capabilities as ambient capabilities.
redact Package redact removes secret-looking values from text before it reaches evidence rings, spools, transmission, or diagnostics (PRD L6, 10).	Package redact removes secret-looking values from text before it reaches evidence rings, spools, transmission, or diagnostics (PRD L6, 10).
rules/bundle Package bundle defines the rule bundle format (docs/bundle-format.md): parsing, validation, signature and key manifest verification.	Package bundle defines the rule bundle format (docs/bundle-format.md): parsing, validation, signature and key manifest verification.
rules/engine Package engine evaluates CEL state rules and PromQL and LogQL alerting rules (docs/promql-rules.md).	Package engine evaluates CEL state rules and PromQL and LogQL alerting rules (docs/promql-rules.md).
rules/logql Package logql implements the LogQL subset published in docs/logql-subset.md without Loki code.	Package logql implements the LogQL subset published in docs/logql-subset.md without Loki code.
rules/validators Package validators builds the bundle validators backed by the real rule engines.	Package validators builds the bundle validators backed by the real rule engines.
rulesdefault Package rulesdefault serves the default ExitMesh rule bundle sources embedded from rules/ and builds their archives.	Package rulesdefault serves the default ExitMesh rule bundle sources embedded from rules/ and builds their archives.
spool Package spool is the writer's durable record store, the node agent queue, and directory locks.	Package spool is the writer's durable record store, the node agent queue, and directory locks.
state Package state normalizes Kubernetes objects into protocol state, edges, scopes, and kube_* series.	Package state normalizes Kubernetes objects into protocol state, edges, scopes, and kube_* series.
telemetry/disk Package disk enforces the state directory cap by shrinking the TSDB first and reporting pressure.	Package disk enforces the state directory cap by shrinking the TSDB first and reporting pressure.
telemetry/evidence Package evidence holds redacted matched lines under a per-node byte ceiling split into rule shares.	Package evidence holds redacted matched lines under a per-node byte ceiling split into rule shares.
telemetry/logs Package logs tails container and host log files per the contract in docs/log-contract.md.	Package logs tails container and host log files per the contract in docs/log-contract.md.
telemetry/metricfacts Package metricfacts summarizes local series into per-resource facts and change thresholds.	Package metricfacts summarizes local series into per-resource facts and change thresholds.
telemetry/scrape Package scrape is the agent's own scrape loop with per-node budgets, staleness, and coverage status.	Package scrape is the agent's own scrape loop with per-node budgets, staleness, and coverage status.
telemetry/tsdb Package tsdb wraps the Prometheus TSDB with rule-derived retention, a size ceiling, and pressure shrinking.	Package tsdb wraps the Prometheus TSDB with rule-derived retention, a size ceiling, and pressure shrinking.
tunnel Package tunnel implements the WebSocket tunnel binding (SPEC 9) and the enrollment client.	Package tunnel implements the WebSocket tunnel binding (SPEC 9) and the enrollment client.
pkg
protocol Package protocol implements the ExitMesh History Protocol v1 described in protocol/SPEC.md.	Package protocol implements the ExitMesh History Protocol v1 described in protocol/SPEC.md.
protocol/client Package client implements the transport-agnostic writer session of the History Protocol (protocol/SPEC.md sections 8 and 9): hello, resume and drain, windowed replay, acknowledgements, divergence handling with rebaseline, and the reverse MCP channel.	Package client implements the transport-agnostic writer session of the History Protocol (protocol/SPEC.md sections 8 and 9): hello, resume and drain, windowed replay, acknowledgements, divergence handling with rebaseline, and the reverse MCP channel.
protocol/client/clienttest Package clienttest provides an in-memory writer model implementing client.Hooks over a client.MemStore, for tests.	Package clienttest provides an in-memory writer model implementing client.Hooks over a client.MemStore, for tests.
protocol/refcp Package refcp is an in-memory reference control plane for contract and end-to-end tests.	Package refcp is an in-memory reference control plane for contract and end-to-end tests.
protocol
genvectors command Command genvectors deterministically writes the language-neutral vectors under protocol/vectors.	Command genvectors deterministically writes the language-neutral vectors under protocol/vectors.
rules Package rules embeds the default ExitMesh rule bundle sources, one directory per target type.	Package rules embeds the default ExitMesh rule bundle sources, one directory per target type.