agent

tare-agent (proof of concept)

tare decides whether a container is starved of CPU from PSI (container_pressure_cpu_waiting_seconds_total). Many clusters do not have PSI: cgroup v1 nodes, RHEL 8/9/10 kernels (PSI ships switched off), kubelets older than 1.33. tare-agent measures the same thing another way, so it can be compared with PSI where PSI exists, and used where it does not.

It measures CPU wait per container: the seconds the container's tasks spent ready to run but waiting for a CPU, summed over all of its tasks, including tasks that have already exited. On every scrape it reads the scheduler counters of every live thread (/proc/<pid>/task/<tid>/schedstat); in between, the kernel sends it the final counters of every task that exits (taskstats exit notifications), so short-lived processes are counted too. To know which container an exited task belonged to, it also notes the container of every task that starts a program (process connector exec events): exec probes are started from outside the container and are usually gone before their exit can be traced back.

This is a proof of concept. tare does not need it, does not install it, and does not read the agent's metrics yet.

Before you deploy it: it runs privileged, with hostPID and hostNetwork, on every node including the control plane, and serves /metrics on port 9105 of each node over plain HTTP with no authentication. NetworkPolicy does not apply to host-network pods, so restrict 9105 to your Prometheus at the node firewall or security group. The image is ghcr.io/2listic/tare-agent (linux/amd64, arm64).

To see how its numbers compare with PSI today, use compare-pressure (below), which reads Prometheus and judges the agent's data the way tare judges PSI.

What gets deployed

The agent does not talk to the Kubernetes API (no ServiceAccount token, no RBAC).

Requirements:

• Linux nodes, kernel 4.18 or newer, cgroup v1 or v2, containerd or CRI-O.
• A Prometheus that already scrapes the kubelet's cAdvisor endpoint (/metrics/cadvisor), with the id label kept on those series. The agent labels its series only with id, the container's cgroup path. The kubelet's series carry the same id, and that is how namespace, pod and container names are attached.
• kube-state-metrics, optional: with it, CPU requests and limits are taken into account.
• Permission to run a privileged pod with hostPID and hostNetwork (see below).

1. The image

deploy/agent/agent.yaml pulls ghcr.io/2listic/tare-agent:main, rebuilt by CI on every change to agent/ on main. Every release also has :vX.Y.Z (and :latest), and every main build :sha-<commit>; those never move, so pin one of them for anything you keep, and set imagePullPolicy: IfNotPresent. Nothing else is needed if the nodes can pull from ghcr.io; otherwise build it yourself (2).

2. Build it yourself (optional)

From the repository root:

docker build -t tare-agent:poc agent/

The build runs inside Docker (Go is not needed on your machine). The result is a static binary on a distroless base, about 12 MB.

To build or test the binary itself (Go 1.27):

cd agent
go test ./...
go build -o tare-agent .

Then either push it to a registry the cluster can pull from, or import it into every node's container runtime.

Registry:

docker tag tare-agent:poc registry.example.com/tare-agent:poc
docker push registry.example.com/tare-agent:poc

Then in deploy/agent/agent.yaml set image: registry.example.com/tare-agent:poc and imagePullPolicy: IfNotPresent.

Import into each node (no registry): set image: tare-agent:poc and imagePullPolicy: Never in deploy/agent/agent.yaml, then

# containerd
docker save tare-agent:poc | ssh <node> 'sudo ctr -n k8s.io images import -'
# CRI-O
docker save tare-agent:poc | ssh <node> 'cat > /tmp/tare-agent.tar &&
  sudo skopeo copy docker-archive:/tmp/tare-agent.tar containers-storage:docker.io/library/tare-agent:poc'

ctr may live in /usr/local/bin, which sudo leaves out of its PATH on RHEL-family nodes: use the full path (sudo /usr/local/bin/ctr ...).

3. Deploy

kubectl apply -f deploy/agent/agent.yaml
kubectl -n tare-agent rollout status ds/tare-agent
kubectl -n tare-agent logs ds/tare-agent

The log should show the cgroup version, registration sent for cpus ... and process connector: listening for execs, and the pods should become ready. If the exec listener cannot start, the agent keeps serving what it has and logs why. If the exit listener cannot register or stops (no CAP_NET_ADMIN, not in the host network namespace), the agent logs why, keeps serving the live threads, reports tare_agent_exit_listener_up 0, and fails its readiness probe: without exit records CPU wait misses every task that exited between scrapes.

Why the pod is privileged:

The namespace carries pod-security.kubernetes.io/enforce: privileged. On OpenShift the pods also need a privileged SCC.

Agent flags (rarely needed): -listen (default :9105), -cgroup (default /sys/fs/cgroup), -proc (default /proc).

4. Have Prometheus scrape it

The agent publishes its numbers on a web page on every node (http://<node>:9105/metrics). That alone stores nothing: Prometheus has to be told to read ("scrape") those pages every 15 seconds and keep the values. Prometheus does not find new things to scrape on its own, so you add one piece of configuration. Which one depends on how your Prometheus was installed.

Check which case you are in:

kubectl get prometheus -A

If this lists something, your Prometheus is managed by the Prometheus Operator (for example the kube-prometheus-stack Helm chart): case A. If the command says the resource type does not exist, you configure Prometheus yourself through its prometheus.yml: case B.

Case A: Prometheus Operator

You do not edit Prometheus' configuration; the operator writes it from Kubernetes objects. Create one object, a PodMonitor, that says "scrape the pods labelled app=tare-agent, on their port named metrics".

1. Find the label your Prometheus requires on PodMonitors:

   kubectl get prometheus -A -o jsonpath='{range .items[*]}{.metadata.name}: {.spec.podMonitorSelector}{"\n"}{end}'
   

   With kube-prometheus-stack this is usually {"matchLabels":{"release":"<your Helm release>"}}. An empty selector ({}) means every PodMonitor is picked up.

2. Save this as tare-agent-podmonitor.yaml, with that label under metadata.labels:

   apiVersion: monitoring.coreos.com/v1
   kind: PodMonitor
   metadata:
     name: tare-agent
     namespace: tare-agent
     labels:
       release: kube-prometheus-stack   # the label found in step 1
   spec:
     selector:
       matchLabels: {app: tare-agent}
     podMetricsEndpoints:
     - port: metrics
       interval: 15s
   

3. Apply it:

   kubectl apply -f tare-agent-podmonitor.yaml
   

   The Prometheus Operator may also limit which namespaces it reads PodMonitors from (spec.podMonitorNamespaceSelector of the prometheus object); tare-agent must be one of them.

Case B: plain Prometheus

Add this job under scrape_configs: in prometheus.yml, then reload Prometheus (restart it, or curl -X POST http://<prometheus>/-/reload if it runs with --web.enable-lifecycle):

scrape_configs:
- job_name: tare-agent
  scrape_interval: 15s
  kubernetes_sd_configs:
  - role: pod
    namespaces: {names: [tare-agent]}
  relabel_configs:
  - {source_labels: [__meta_kubernetes_pod_label_app], regex: tare-agent, action: keep}
  - {source_labels: [__meta_kubernetes_pod_container_port_name], regex: metrics, action: keep}
  - {source_labels: [__meta_kubernetes_pod_node_name], target_label: node}

It tells Prometheus to look up the tare-agent pods through the Kubernetes API (Prometheus' service account needs list on pods, which it already has if it discovers the kubelet that way) and to scrape their metrics port.

Check

In Prometheus' web UI, Status → Targets must list the agent with one target per node, all UP. Or run this query in Prometheus: it must return one series per node with value 1.

up{job=~".*tare-agent.*"}

(With a PodMonitor the job is usually called tare-agent/tare-agent or tare-agent.)

5. Check that it works

kubectl -n tare-agent port-forward ds/tare-agent 9105 &
curl -s localhost:9105/metrics | grep '^tare_'

In Prometheus, this must return your containers' names; if it returns nothing, the id join is not working (see Troubleshooting):

sum by (namespace, pod, container) (
  rate(tare_cpu_wait_seconds_total[5m])
  * on (id) group_left (namespace, pod, container)
    max by (id, namespace, pod, container) (container_cpu_usage_seconds_total{container!="",container!="POD"} * 0 + 1))

6. Compare with PSI: compare-pressure

agent/compare-pressure is a standalone script (Python 3.9+, standard library only). It reads Prometheus and, per container, judges the agent's CPU wait the way tare judges PSI. Let the agent run for at least a day of normal traffic first: each container's "normal" is learned from its own history in the window.

agent/compare-pressure --prometheus http://localhost:9090 --window 1d
agent/compare-pressure --prometheus http://localhost:9090 --window 7d --namespace shop,payments
agent/compare-pressure --prometheus http://localhost:9090 --window 1d --all --stretches stretches.csv

If Prometheus is not reachable directly: kubectl -n <monitoring namespace> port-forward svc/<prometheus service> 9090.

How it decides

Per container, the window is cut into 5-minute stretches. In each stretch:

• wait: the agent's CPU wait (s/s);
• usage: CPU usage from the kubelet (cores);
• slowdown = wait ÷ usage: seconds of waiting per second of running;
• psi: PSI waiting from the kubelet (s/s), when the cluster has it.

A stretch is stalled under each rule:

Then each rule gives each container the verdict tare's pressure check gives:

tare's other verdicts (LEAKING, IDLE, WASTE, BLOCKED, UNSET, CONFLICT, NO-DATA) come from memory and usage, not from pressure. The script does not reproduce them.

Output

using http://127.0.0.1:9191: tare-agent on 2 nodes, PSI present, 1h ending 2026-10-21T16:35Z
# section=findings
PSI        RELATIVE-SLOWDOWN  FIXED-SLOWDOWN  ABSOLUTE-WAIT  NAMESPACE  WORKLOAD     CONTAINER    STALLED          NORMAL  COVER  WHY
SATURATED  SATURATED          SATURATED       SATURATED      reallab    postgres-db  postgres-db  69%/38%/38%/85%  0.131   0.86   CPU_PRESSURE,CPU_WAIT_RELATIVE,+2
NOT-STARVED  NOT-STARVED        NOT-STARVED     NOT-STARVED    reallab    load-pg      pgbench      0%/0%/0%/0%      0.216   0.98   -

7 containers (7 listed) over 1h: SATURATED by psi 6, relative-slowdown 6, fixed-slowdown 6, absolute-wait 6
agreement  relative-slowdown: same verdict as psi on 7 of 7 containers; same call on 76.9% of 91 stretches (16 psi only, 5 relative-slowdown only)

On a cluster without PSI the PSI column and the agreement lines are left out, and the agent's rules are shown on their own.

--stretches writes one row per container and stretch: namespace, pod, container, time, wait, usage, slowdown, psi, normal, and 0/1 per rule.

Troubleshooting

Remove

kubectl delete -f deploy/agent/agent.yaml     # and your PodMonitor / scrape config

Limits of the proof of concept

• There is no Helm chart.
• The pod runs privileged; the capability-only profile is not done yet.
• tare does not read the agent yet: compare-pressure is the way to look at it.
• Measured cost: about 0.002–0.05 cores and 15–20 MiB per node, growing with the rate of process exits (0.05 cores at about 280 exits per second).
• Exercised on Ubuntu 24.04 (kernel 6.8), Debian 12 (6.1), AlmaLinux 9 (5.14), AlmaLinux 10 (6.12), Debian 11 with cgroup v1 (5.10), CRI-O 1.36, Rocky Linux 8.9 (4.18, without Kubernetes) and arm64 (emulated, without Kubernetes).