gitlab-haproxy-agent

gitlab-haproxy-agent

The agent-check responder for the HAProxy control plane: haproxy asks it about every backend server on a short interval, and it answers with the desired admin state held in consul KV. Writing one KV key drains, un-drains, or reweights a server fleet-wide within an agent-check interval, with no haproxy reload.

Design, rationale, contracts, rollout, and alternatives live in the control plane work item.

How it works

Two components around an in-memory copy of the desired state, so answers never wait on consul:

• a consul watcher: a blocking-query loop on a KV prefix, swapping in a full copy of the state on every change
• a TCP responder: per connection, read the newline-terminated agent-send line carrying the server identity (by convention backend/server), look it up, write one reply line, close

haproxy side, per server:

server srv1 10.0.0.1:443 weight 100 check agent-check agent-addr <agent-host> agent-port 9777 agent-send "be_web/srv1\n" agent-inter 2s

KV contract

Keys live under a prefix (default gitlab-haproxy-agent/state). The part below the prefix is the agent-send identity, the value is the agent-check reply:

An absent key means ready: operators only write state to divert from the default, and deleting the key restores it. Values that don't parse as a valid reply are dropped (logged and counted), as if the key were absent.

Use maint for take-out-and-return work and drain for instant removal from new-connection balancing: leaving maintenance triggers the configured slowstart ramp, leaving drain does not.

Health states (up, down, stopped, fail) are deliberately not accepted: server health belongs to haproxy's own checks, the control plane only sets admin state.

Failure behavior

The agent never guesses. In every degraded situation it either answers from the last known state or answers nothing. Per the agent-check protocol, a connection without a parseable reply line leaves the server's state unchanged ("Ignoring incomplete line from agent").

Running it

gitlab-haproxy-agent \
  -listen-addr :9777 \
  -monitoring-addr :9778 \
  -pprof-addr 127.0.0.1:9779 \
  -consul-addr 127.0.0.1:8500 \
  -kv-prefix gitlab-haproxy-agent/state

All flags are optional and also available as environment variables (GITLAB_HAPROXY_AGENT_LISTEN_ADDR etc.). -consul-addr falls back to the consul client defaults (CONSUL_HTTP_ADDR etc.). Configuration stays flags-only until options outgrow flags. Then it becomes a labkit v2 config file, not before.

Observability

-monitoring-addr serves /-/metrics, /-/liveness, and /-/readiness. -pprof-addr serves /debug/pprof. Logs are JSON on stderr (GITLAB_LOG_LEVEL, GITLAB_LOG_FORMAT).

Packaging

Both delivery forms come from goreleaser:

• linux binaries as archives attached to GitLab releases, for the Chef VMs (remote_file in the gitlab-haproxy cookbook, same pattern as hatop)
• a container image pushed to the registry, for the k8s sidecar

Both are also published on ops.gitlab.net for HA: the repo push-mirrors to ops and the mirrored tag pipeline publishes archives and images there. The agent must stay buildable and installable during a gitlab.com outage. That is when it gets used.

Development

1. Follow the developer setup guide at https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/blob/main/docs/developer-setup.md.
2. Clone this project
3. Run scripts/prepare-dev-env.sh

Tests

mise run test:unit
mise run test:integration   # needs the haproxy binary (brew install haproxy), consul comes from mise

The integration test is a real haproxy and a real consul around ordinary Go code: consul in dev mode, haproxy with one agent-checked backend server (a net.Listener in the test), the agent in-process. The test writes KV and asserts through show stat on the haproxy stats socket. At agent-inter 100ms the suite runs in seconds. Covered beyond the happy path: startup convergence, responder-down state freeze, garbage replies, weight percentages.

In CI the same test runs in an in-repo image (.gitlab/ci/test-image/Dockerfile) that pins the haproxy minor production runs, plus the Go and consul versions from .tool-versions. The image rebuilds automatically when any pinned version changes the tag.

Weight stepping

The agent has none, deliberately. It stays a stateless lookup: ramp state, restart semantics, and divergence between KV and what haproxy sees are not worth carrying for something the rest of the system already provides.

• Undrain ramps come from haproxy's own slowstart on the server line. Leaving agent-driven maintenance ramps the effective weight natively, per node (covered by TestSlowstartRampsAfterMaint).
• Explicit trajectories (canary shifting, gradual drain-down), should they ever be needed, belong in the KV writer: the contract accepts arbitrary N% values, so a control-plane tool can implement any curve by writing intermediate weights. Every step is visible in KV and in the metrics.

What's next

• the gstg rollout, via the gitlab-haproxy cookbook and the k8s sidecar