Documentation
¶
Overview ¶
Package agent embeds in every Nucleus framework process and connects to a standalone admin server (see ../server) to ship observability events.
The module is implemented: the agent loop (Run), the per-kind drop-oldest ring buffers, the endpoint-failover dialer with exponential backoff, the bidi stream lifecycle, and the admin_agent_* Prometheus collectors live in the sub-packages listed in README.md.
Hard architectural invariants for the agent:
- The framework's hot path must NEVER block on, fail because of, or wait for the agent. The agent is a strictly opt-in observer.
- When no operator is subscribed, the agent's per-event-type atomic counter is zero and the agent is a no-op. Constructing an event must be gated on that counter at the call site.
- The agent dials the admin server, never the other way around. This makes NAT/firewall traversal symmetric between on-prem and cloud deployments.
- The connection list (ExtensionConfig.Endpoints) is tried in order with health-check; only after every endpoint fails does the agent enter exponential backoff (cap 30s, jitter).
- Events that cannot be shipped are dropped (drop-oldest per type) and reported via the admin_agent_events_dropped_total Prometheus counter. The framework never persists telemetry; that is OpenTelemetry's job.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ErrDisabled = errors.New("admin agent: no endpoints configured (disabled)")
ErrDisabled is returned by New when no admin endpoints are configured. Callers should treat this as "the agent is disabled" rather than as an error; it lets fail-open wiring in pkg/app skip the agent without noise.
Functions ¶
func NewExtension ¶
func NewExtension(adminCfg ExtensionConfig, stateDir, version string) app.Extension
NewExtension adapts the agent into an app.Extension so callers can wire it through pkg/app.WithExtensions. The extension is fail-open: when adminCfg.Endpoints is empty, Attach returns nil and the framework starts without an agent.
Example wiring (typically in cmd/server/main.go):
a, err := app.New(cfg,
app.WithExtensions(
agent.NewExtension(agent.ExtensionConfig{Endpoints: []string{"https://admin:8443"}}, stateDir, "v0.7.0"),
),
)
The agent's lifecycle is bound to App.Shutdown: graceful Goodbye and drain happen when the framework shuts down, no extra wiring needed.
Types ¶
type Agent ¶
type Agent struct {
// contains filtered or unexported fields
}
Agent is the long-lived top-level type. Its Run method blocks until ctx is cancelled and returns nil on graceful shutdown.
func (*Agent) Connected ¶
func (a *Agent) Connected() <-chan struct{}
Connected returns a channel that is closed the first time an admin server accepts a stream from this agent — the first frame received from the server under auth (stream.Config.OnAccepted) — NOT on the first successful dial: the dial's /healthz probe is auth-exempt, so reachability proves nothing about the token (OR6-1). Subsequent disconnects/reconnects do NOT re-open the channel.
It is the integration point for the require_connection path: callers that need the framework to fail boot when no admin accepts the stream select on this channel against a timeout, and abort if the timeout fires first.
type Config ¶
type Config struct {
// Endpoints is the ordered list of admin server URLs (admin.endpoints
// in nucleus.yml). At least one is required for the agent to start;
// passing an empty list returns ErrDisabled from New.
Endpoints []string
// DB, when non-nil, lets the heartbeat report the framework database
// pool stats (in-use / idle / max) alongside the host metrics sample.
DB *sql.DB
// Token is the shared bearer token sent on every Connect-RPC call.
// May be empty when the admin server authenticates agents by client
// certificate instead (see TLS).
Token string
// TLS is applied to every https:// endpoint. Nil means the system
// trust store. Set RootCAs for an admin server signed by a private
// CA, and Certificates when the server's agent listener requires a
// client certificate (--agent-client-ca). Ignored for http:// (h2c)
// endpoints.
TLS *tls.Config
// StateDir is the path under which node_id is persisted (the new
// top-level state_dir key in nucleus.yml). Empty means "use ephemeral
// NodeID with WARN", consistent with decision 15.
StateDir string
// NodeIDOverride pins the NodeID instead of resolving from StateDir.
// Empty means resolve via identity.Resolver.
NodeIDOverride string
// Version, Labels, StartedAt are forwarded to NodeRegistration.
Version string
Labels map[string]string
StartedAt time.Time
// Bus is the in-process observability bus the agent subscribes to. If
// nil, the agent runs but no events flow.
Bus *observability.Bus
// HeartbeatInterval is the cadence between Heartbeat frames sent to
// the admin server. Default 10s.
HeartbeatInterval time.Duration
// DrainTimeout is the maximum time spent flushing the ring buffer to
// the stream during graceful shutdown. Default 2s.
DrainTimeout time.Duration
// HTTPBufferSize / SQLBufferSize / SessionBufferSize / CustomBufferSize
// configure the per-kind drop-oldest ring buffers that absorb
// backpressure while a stream is open (defaults 256/256/64/64).
HTTPBufferSize int
SQLBufferSize int
SessionBufferSize int
CustomBufferSize int
// MetricsAddr, when non-empty, starts a /metrics + /healthz HTTP
// server on this address.
MetricsAddr string
// Registry is the framework's model registry. Required for Data
// Studio support; nil disables the Data Studio path on this agent.
Registry *model.Registry
// Databases are the framework's DB handles keyed by alias. The
// agent's Data Studio handler uses them to execute model.CRUD
// operations on behalf of UI requests routed through the admin
// server. Empty disables the Data Studio path.
Databases map[string]*db.DB
// Authorizer is a read-only view of the framework's RBAC state (the
// *authz.Enforcer satisfies it). Required for the Access control
// screen of the fleet UI; nil disables the RBAC snapshot path on
// this agent.
Authorizer rbac.PolicySource
// DefaultDatabaseAlias is the alias used when a Data Studio request
// arrives with an empty database_alias. Falls back to "default" if
// unset.
DefaultDatabaseAlias string
// Logger receives WARN/INFO/DEBUG diagnostics. Pass nil for
// slog.Default.
Logger *slog.Logger
}
Config bundles every dependency the agent needs. The framework's app.New constructs this from nucleus.yml plus the in-process bus and passes it to New.
type ExtensionConfig ¶
type ExtensionConfig struct {
// Endpoints is the ordered list of admin server URLs the agent will
// try to connect to. Each URL may be http:// (h2c, dev), https://
// (production), or any other Connect-RPC compatible scheme. Failover
// happens left-to-right; once every endpoint has failed, the agent
// enters exponential backoff (cap 30s).
Endpoints []string `koanf:"endpoints"`
// Token is the shared bearer token sent on every Connect-RPC call.
// In production pair it with an https:// endpoint (or let the server
// authenticate the agent by client certificate via TLS); in dev a
// plain token over http:// suffices.
Token string `koanf:"token"`
// TLS is applied to every https:// endpoint. Nil uses the system
// trust store. Set RootCAs for a server signed by a private CA and
// Certificates to present a client certificate when the admin
// server's agent listener requires one (--agent-client-ca). It is
// not bound from a config file (koanf:"-"): build it in code from
// the PEM files your deployment ships.
TLS *tls.Config `koanf:"-"`
// HeartbeatInterval defines the cadence of Heartbeat frames the agent
// sends to the server. Default 10s.
HeartbeatInterval time.Duration `koanf:"heartbeat_interval"`
// DrainTimeout caps the time the agent spends flushing buffered
// events to the stream during graceful shutdown. Default 2s.
DrainTimeout time.Duration `koanf:"drain_timeout"`
// MetricsAddr, when non-empty, runs a Prometheus /metrics + /healthz
// HTTP server on this address. Format: "[host]:port", e.g.
// "127.0.0.1:9101". Empty disables the standalone server.
MetricsAddr string `koanf:"metrics_addr"`
// HTTPBufferSize, SQLBufferSize, SessionBufferSize, CustomBufferSize
// configure the per-event-kind drop-oldest ring buffer that absorbs
// bursts while the stream is open (events queue when the stream's
// send path is busy and drain when it catches up). Events that occur
// while the agent has no stream at all are dropped, not buffered.
// Defaults: 256, 256, 64, 64.
HTTPBufferSize int `koanf:"http_buffer_size"`
SQLBufferSize int `koanf:"sql_buffer_size"`
SessionBufferSize int `koanf:"session_buffer_size"`
CustomBufferSize int `koanf:"custom_buffer_size"`
// NodeIDOverride pins the NodeID the agent reports in
// NodeRegistration. Empty means "resolve from
// ${state_dir}/node_id" (UUIDv4 persisted at first run).
NodeIDOverride string `koanf:"node_id"`
// Labels are arbitrary key/value pairs forwarded with NodeRegistration
// and shown in the admin UI's node topology view.
Labels map[string]string `koanf:"labels"`
// DefaultDatabaseAlias is the alias the agent's Data Studio handler
// uses when a request arrives with an empty database_alias. Falls
// back to "default" if unset.
DefaultDatabaseAlias string `koanf:"default_database_alias"`
// RequireConnection, when true, makes the framework fail to boot if
// the agent does not establish a stream to any admin endpoint within
// RequireConnectionTimeout. Default: false (fail-open). Operators in
// compliance-sensitive environments can set this to true so that the
// application refuses to serve traffic when its observability lifeline
// is missing.
RequireConnection bool `koanf:"require_connection"`
// RequireConnectionTimeout caps the wait when RequireConnection is
// true. Default 10s. Ignored when RequireConnection is false.
RequireConnectionTimeout time.Duration `koanf:"require_connection_timeout"`
}
ExtensionConfig is the framework-facing configuration for the admin observability agent, consumed by NewExtension and mapped into the agent's internal Config. It used to live in pkg/app as app.AdminAgentConfig (bound from the application config's `admin:` subtree), but moved here when the admin panel was extracted from the framework core (nucleus ADR-019): the framework no longer carries admin-specific configuration, so the agent owns its own config type. Callers populate it directly (e.g. from their own config file) and pass it to NewExtension.
Directories
¶
| Path | Synopsis |
|---|---|
|
Package buffer holds the per-event-kind drop-oldest ring buffer the agent uses to absorb backpressure on an OPEN stream.
|
Package buffer holds the per-event-kind drop-oldest ring buffer the agent uses to absorb backpressure on an OPEN stream. |
|
Package connection establishes and maintains the agent's transport to an admin server.
|
Package connection establishes and maintains the agent's transport to an admin server. |
|
Package convert translates the framework's in-process observability events into proto messages on the wire.
|
Package convert translates the framework's in-process observability events into proto messages on the wire. |
|
Package datastudio is the agent-side handler for DataStudioRequest frames sent by the admin server.
|
Package datastudio is the agent-side handler for DataStudioRequest frames sent by the admin server. |
|
examples
|
|
|
fleet-app
command
Command fleet-app is a minimal Nucleus host process wired with the Orbit cluster agent (orbit/agent).
|
Command fleet-app is a minimal Nucleus host process wired with the Orbit cluster agent (orbit/agent). |
|
Package hostmetrics samples the agent process's runtime health for the Heartbeat frame: CPU share, memory, goroutines, GC pauses, and the framework database pool.
|
Package hostmetrics samples the agent process's runtime health for the Heartbeat frame: CPU share, memory, goroutines, GC pauses, and the framework database pool. |
|
Package identity resolves the NodeID this agent reports to the admin server.
|
Package identity resolves the NodeID this agent reports to the admin server. |
|
internal
|
|
|
testserver
Package testserver provides an in-process fake admin server used in agent integration tests.
|
Package testserver provides an in-process fake admin server used in agent integration tests. |
|
Package metrics exposes the agent's Prometheus collectors and a small HTTP server that serves /metrics.
|
Package metrics exposes the agent's Prometheus collectors and a small HTTP server that serves /metrics. |
|
Package rbac is the agent-side handler for RbacRequest frames sent by the admin server.
|
Package rbac is the agent-side handler for RbacRequest frames sent by the admin server. |
|
Package sampler implements per-kind sampling and post-bus filtering for the agent.
|
Package sampler implements per-kind sampling and post-bus filtering for the agent. |
|
Package stream owns the bidi stream lifecycle between the agent and an admin server: the registration handshake, heartbeats, command dispatch (Subscribe / Unsubscribe / SnapshotRequest / Goodbye), event egress from the in-process bus, and graceful drain on shutdown.
|
Package stream owns the bidi stream lifecycle between the agent and an admin server: the registration handshake, heartbeats, command dispatch (Subscribe / Unsubscribe / SnapshotRequest / Goodbye), event egress from the in-process bus, and graceful drain on shutdown. |