agent

package module
v0.6.19 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 11, 2026 License: Apache-2.0 Imports: 22 Imported by: 0

README

orbit/agent

The Nucleus admin observability agent. Embeds in every framework process and ships events to a standalone admin server (../server) over a single Connect-RPC bidi stream.

Framework floor. This module builds against Nucleus and moves in lockstep with the certified suite set. The exact version it requires is in this module's go.mod — that file is the source of truth, so this note cannot go stale.

Wiring into a Nucleus app

The agent owns its configuration type (ExtensionConfig) — the framework carries no admin-specific config. Populate it directly and pass it to NewExtension with the framework's state directory and your app's version string:

import (
    "context"
    "log"
    "os"

    "github.com/jcsvwinston/nucleus/pkg/app"
    "github.com/jcsvwinston/orbit/agent"
)

func main() {
    cfg, err := app.LoadConfig("nucleus.yml")
    if err != nil {
        log.Fatal(err)
    }
    a, err := app.New(cfg,
        app.WithExtensions(
            agent.NewExtension(agent.ExtensionConfig{
                Endpoints: []string{"https://admin.internal:9090"},
                Token:     os.Getenv("NUCLEUS_ADMIN_TOKEN"),
            }, cfg.StateDir, "v1.2.3"), // your app's version string
        ),
    )
    if err != nil {
        log.Fatal(err)
    }
    if err := a.Run(context.Background()); err != nil {
        log.Fatal(err)
    }
}

When ExtensionConfig.Endpoints is empty, the extension is a no-op and the framework runs unchanged. When it is set, the agent starts in parallel with the framework's Run; observability events flow through the framework's pkg/observability bus into the bidi stream.

The full configuration surface is the godoc of ExtensionConfig (extension_config.go).

Layered structure

Sub-package Responsibility
identity Resolves the persistent NodeID (UUIDv4 in ${state_dir}/node_id) with hostname-derived ephemeral fallback.
convert pkg/observability events → proto events; Filter proto → in-process Filter.
sampler Per-kind sampling rate + HTTP/SQL filter from server-side Subscribe commands.
buffer Per-event-kind drop-oldest ring buffer for bridging brief disconnects.
connection Endpoint failover dialer with exponential backoff (cap 30s) and rate-limited disconnect WARN (1/min).
stream Bidi stream lifecycle: registration, three-goroutine recv/send/heartbeat loop, command dispatch, replay on reconnect. Events ship with the agent's registered NodeID (the in-process bus NodeID is host-local and does not correlate with the fleet registry). Snapshot providers: GO_RUNTIME and REGISTERED_MODELS; other types answer with a per-type error.
metrics admin_agent_* Prometheus collectors + standalone /metrics + /healthz server.
rbac Read-only Casbin snapshot handler for the fleet UI's Access control screen (wired from app.Authorizer).
internal/testserver In-process h2c admin server fake for integration tests. (Internal; do not import.)

The top-level Agent (agent.go) composes everything and exposes:

  • New(cfg) — constructor; ErrDisabled when no endpoints.
  • Run(ctx) — blocks until ctx cancels; reconnect loop with backoff.
  • NodeID() — the resolved identifier.
  • Connected() — channel closed on the first frame the admin server accepts under auth (stream.Config.OnAccepted), not on dial success: the dial's /healthz probe is auth-exempt, so reachability proves nothing about the token. Used by NewExtension(...).Attach when ExtensionConfig.RequireConnection is true (boot blocks until a stream is accepted or RequireConnectionTimeout expires).
  • Metrics() — the Prometheus registry, in case the host wants to serve /metrics from its own port.

Hot-path invariants

The agent never blocks the framework's request thread. Every public producer-side path (the HTTP middleware, the SQL observer) starts with a single atomic load on pkg/observability.Bus.HasSubscribers(kind) and short-circuits when nobody is watching.

Tests

cd agent && go test -race ./...

Integration tests live in agent_test.go and extension_test.go. The in-process testserver fakes the admin server with enough fidelity to exercise reconnect, subscribe/unsubscribe, drain, and Goodbye paths.

Distribution

Released as its own Go module with component tags (agent/vX.Y.Z, via release-please). Consumer apps add it with:

go get github.com/jcsvwinston/orbit/agent

Pre-1.0: the module version signals honestly that the agent's public surface may still change before its own v1.0.

Documentation

Overview

Package agent embeds in every Nucleus framework process and connects to a standalone admin server (see ../server) to ship observability events.

The module is implemented: the agent loop (Run), the per-kind drop-oldest ring buffers, the endpoint-failover dialer with exponential backoff, the bidi stream lifecycle, and the admin_agent_* Prometheus collectors live in the sub-packages listed in README.md.

Hard architectural invariants for the agent:

  • The framework's hot path must NEVER block on, fail because of, or wait for the agent. The agent is a strictly opt-in observer.
  • When no operator is subscribed, the agent's per-event-type atomic counter is zero and the agent is a no-op. Constructing an event must be gated on that counter at the call site.
  • The agent dials the admin server, never the other way around. This makes NAT/firewall traversal symmetric between on-prem and cloud deployments.
  • The connection list (ExtensionConfig.Endpoints) is tried in order with health-check; only after every endpoint fails does the agent enter exponential backoff (cap 30s, jitter).
  • Events that cannot be shipped are dropped (drop-oldest per type) and reported via the admin_agent_events_dropped_total Prometheus counter. The framework never persists telemetry; that is OpenTelemetry's job.

Index

Constants

This section is empty.

Variables

View Source
var ErrDisabled = errors.New("admin agent: no endpoints configured (disabled)")

ErrDisabled is returned by New when no admin endpoints are configured. Callers should treat this as "the agent is disabled" rather than as an error; it lets fail-open wiring in pkg/app skip the agent without noise.

Functions

func NewExtension

func NewExtension(adminCfg ExtensionConfig, stateDir, version string) app.Extension

NewExtension adapts the agent into an app.Extension so callers can wire it through pkg/app.WithExtensions. The extension is fail-open: when adminCfg.Endpoints is empty, Attach returns nil and the framework starts without an agent.

Example wiring (typically in cmd/server/main.go):

a, err := app.New(cfg,
    app.WithExtensions(
        agent.NewExtension(agent.ExtensionConfig{Endpoints: []string{"https://admin:8443"}}, stateDir, "v0.7.0"),
    ),
)

The agent's lifecycle is bound to App.Shutdown: graceful Goodbye and drain happen when the framework shuts down, no extra wiring needed.

Types

type Agent

type Agent struct {
	// contains filtered or unexported fields
}

Agent is the long-lived top-level type. Its Run method blocks until ctx is cancelled and returns nil on graceful shutdown.

func New

func New(cfg Config) (*Agent, error)

New constructs an Agent. Returns ErrDisabled when no endpoints are configured.

func (*Agent) Connected

func (a *Agent) Connected() <-chan struct{}

Connected returns a channel that is closed the first time an admin server accepts a stream from this agent — the first frame received from the server under auth (stream.Config.OnAccepted) — NOT on the first successful dial: the dial's /healthz probe is auth-exempt, so reachability proves nothing about the token (OR6-1). Subsequent disconnects/reconnects do NOT re-open the channel.

It is the integration point for the require_connection path: callers that need the framework to fail boot when no admin accepts the stream select on this channel against a timeout, and abort if the timeout fires first.

func (*Agent) Metrics

func (a *Agent) Metrics() *metrics.Metrics

Metrics returns the agent's Prometheus metrics. Useful when the host app wants to expose them through its own /metrics rather than via the agent's stand-alone metrics server.

func (*Agent) NodeID

func (a *Agent) NodeID() string

NodeID returns the resolved node identifier.

func (*Agent) Run

func (a *Agent) Run(ctx context.Context) error

Run drives the agent until ctx is cancelled. It blocks. On reconnect errors it sleeps according to the dialer backoff and retries. Returns nil on graceful shutdown.

type Config

type Config struct {
	// Endpoints is the ordered list of admin server URLs (admin.endpoints
	// in nucleus.yml). At least one is required for the agent to start;
	// passing an empty list returns ErrDisabled from New.
	Endpoints []string

	// DB, when non-nil, lets the heartbeat report the framework database
	// pool stats (in-use / idle / max) alongside the host metrics sample.
	DB *sql.DB

	// Token is the shared bearer token sent on every Connect-RPC call.
	// May be empty when the admin server authenticates agents by client
	// certificate instead (see TLS).
	Token string

	// TLS is applied to every https:// endpoint. Nil means the system
	// trust store. Set RootCAs for an admin server signed by a private
	// CA, and Certificates when the server's agent listener requires a
	// client certificate (--agent-client-ca). Ignored for http:// (h2c)
	// endpoints.
	TLS *tls.Config

	// StateDir is the path under which node_id is persisted (the new
	// top-level state_dir key in nucleus.yml). Empty means "use ephemeral
	// NodeID with WARN", consistent with decision 15.
	StateDir string

	// NodeIDOverride pins the NodeID instead of resolving from StateDir.
	// Empty means resolve via identity.Resolver.
	NodeIDOverride string

	// Version, Labels, StartedAt are forwarded to NodeRegistration.
	Version   string
	Labels    map[string]string
	StartedAt time.Time

	// Bus is the in-process observability bus the agent subscribes to. If
	// nil, the agent runs but no events flow.
	Bus *observability.Bus

	// HeartbeatInterval is the cadence between Heartbeat frames sent to
	// the admin server. Default 10s.
	HeartbeatInterval time.Duration

	// DrainTimeout is the maximum time spent flushing the ring buffer to
	// the stream during graceful shutdown. Default 2s.
	DrainTimeout time.Duration

	// HTTPBufferSize / SQLBufferSize / SessionBufferSize / CustomBufferSize
	// configure the per-kind drop-oldest ring buffers that absorb
	// backpressure while a stream is open (defaults 256/256/64/64).
	HTTPBufferSize    int
	SQLBufferSize     int
	SessionBufferSize int
	CustomBufferSize  int

	// MetricsAddr, when non-empty, starts a /metrics + /healthz HTTP
	// server on this address.
	MetricsAddr string

	// Registry is the framework's model registry. Required for Data
	// Studio support; nil disables the Data Studio path on this agent.
	Registry *model.Registry

	// Databases are the framework's DB handles keyed by alias. The
	// agent's Data Studio handler uses them to execute model.CRUD
	// operations on behalf of UI requests routed through the admin
	// server. Empty disables the Data Studio path.
	Databases map[string]*db.DB

	// Authorizer is a read-only view of the framework's RBAC state (the
	// *authz.Enforcer satisfies it). Required for the Access control
	// screen of the fleet UI; nil disables the RBAC snapshot path on
	// this agent.
	Authorizer rbac.PolicySource

	// DefaultDatabaseAlias is the alias used when a Data Studio request
	// arrives with an empty database_alias. Falls back to "default" if
	// unset.
	DefaultDatabaseAlias string

	// Logger receives WARN/INFO/DEBUG diagnostics. Pass nil for
	// slog.Default.
	Logger *slog.Logger
}

Config bundles every dependency the agent needs. The framework's app.New constructs this from nucleus.yml plus the in-process bus and passes it to New.

type ExtensionConfig

type ExtensionConfig struct {
	// Endpoints is the ordered list of admin server URLs the agent will
	// try to connect to. Each URL may be http:// (h2c, dev), https://
	// (production), or any other Connect-RPC compatible scheme. Failover
	// happens left-to-right; once every endpoint has failed, the agent
	// enters exponential backoff (cap 30s).
	Endpoints []string `koanf:"endpoints"`

	// Token is the shared bearer token sent on every Connect-RPC call.
	// In production pair it with an https:// endpoint (or let the server
	// authenticate the agent by client certificate via TLS); in dev a
	// plain token over http:// suffices.
	Token string `koanf:"token"`

	// TLS is applied to every https:// endpoint. Nil uses the system
	// trust store. Set RootCAs for a server signed by a private CA and
	// Certificates to present a client certificate when the admin
	// server's agent listener requires one (--agent-client-ca). It is
	// not bound from a config file (koanf:"-"): build it in code from
	// the PEM files your deployment ships.
	TLS *tls.Config `koanf:"-"`

	// HeartbeatInterval defines the cadence of Heartbeat frames the agent
	// sends to the server. Default 10s.
	HeartbeatInterval time.Duration `koanf:"heartbeat_interval"`

	// DrainTimeout caps the time the agent spends flushing buffered
	// events to the stream during graceful shutdown. Default 2s.
	DrainTimeout time.Duration `koanf:"drain_timeout"`

	// MetricsAddr, when non-empty, runs a Prometheus /metrics + /healthz
	// HTTP server on this address. Format: "[host]:port", e.g.
	// "127.0.0.1:9101". Empty disables the standalone server.
	MetricsAddr string `koanf:"metrics_addr"`

	// HTTPBufferSize, SQLBufferSize, SessionBufferSize, CustomBufferSize
	// configure the per-event-kind drop-oldest ring buffer that absorbs
	// bursts while the stream is open (events queue when the stream's
	// send path is busy and drain when it catches up). Events that occur
	// while the agent has no stream at all are dropped, not buffered.
	// Defaults: 256, 256, 64, 64.
	HTTPBufferSize    int `koanf:"http_buffer_size"`
	SQLBufferSize     int `koanf:"sql_buffer_size"`
	SessionBufferSize int `koanf:"session_buffer_size"`
	CustomBufferSize  int `koanf:"custom_buffer_size"`

	// NodeIDOverride pins the NodeID the agent reports in
	// NodeRegistration. Empty means "resolve from
	// ${state_dir}/node_id" (UUIDv4 persisted at first run).
	NodeIDOverride string `koanf:"node_id"`

	// Labels are arbitrary key/value pairs forwarded with NodeRegistration
	// and shown in the admin UI's node topology view.
	Labels map[string]string `koanf:"labels"`

	// DefaultDatabaseAlias is the alias the agent's Data Studio handler
	// uses when a request arrives with an empty database_alias. Falls
	// back to "default" if unset.
	DefaultDatabaseAlias string `koanf:"default_database_alias"`

	// RequireConnection, when true, makes the framework fail to boot if
	// the agent does not establish a stream to any admin endpoint within
	// RequireConnectionTimeout. Default: false (fail-open). Operators in
	// compliance-sensitive environments can set this to true so that the
	// application refuses to serve traffic when its observability lifeline
	// is missing.
	RequireConnection bool `koanf:"require_connection"`

	// RequireConnectionTimeout caps the wait when RequireConnection is
	// true. Default 10s. Ignored when RequireConnection is false.
	RequireConnectionTimeout time.Duration `koanf:"require_connection_timeout"`
}

ExtensionConfig is the framework-facing configuration for the admin observability agent, consumed by NewExtension and mapped into the agent's internal Config. It used to live in pkg/app as app.AdminAgentConfig (bound from the application config's `admin:` subtree), but moved here when the admin panel was extracted from the framework core (nucleus ADR-019): the framework no longer carries admin-specific configuration, so the agent owns its own config type. Callers populate it directly (e.g. from their own config file) and pass it to NewExtension.

Directories

Path Synopsis
Package buffer holds the per-event-kind drop-oldest ring buffer the agent uses to absorb backpressure on an OPEN stream.
Package buffer holds the per-event-kind drop-oldest ring buffer the agent uses to absorb backpressure on an OPEN stream.
Package connection establishes and maintains the agent's transport to an admin server.
Package connection establishes and maintains the agent's transport to an admin server.
Package convert translates the framework's in-process observability events into proto messages on the wire.
Package convert translates the framework's in-process observability events into proto messages on the wire.
Package datastudio is the agent-side handler for DataStudioRequest frames sent by the admin server.
Package datastudio is the agent-side handler for DataStudioRequest frames sent by the admin server.
examples
fleet-app command
Command fleet-app is a minimal Nucleus host process wired with the Orbit cluster agent (orbit/agent).
Command fleet-app is a minimal Nucleus host process wired with the Orbit cluster agent (orbit/agent).
Package hostmetrics samples the agent process's runtime health for the Heartbeat frame: CPU share, memory, goroutines, GC pauses, and the framework database pool.
Package hostmetrics samples the agent process's runtime health for the Heartbeat frame: CPU share, memory, goroutines, GC pauses, and the framework database pool.
Package identity resolves the NodeID this agent reports to the admin server.
Package identity resolves the NodeID this agent reports to the admin server.
internal
testserver
Package testserver provides an in-process fake admin server used in agent integration tests.
Package testserver provides an in-process fake admin server used in agent integration tests.
Package metrics exposes the agent's Prometheus collectors and a small HTTP server that serves /metrics.
Package metrics exposes the agent's Prometheus collectors and a small HTTP server that serves /metrics.
Package rbac is the agent-side handler for RbacRequest frames sent by the admin server.
Package rbac is the agent-side handler for RbacRequest frames sent by the admin server.
Package sampler implements per-kind sampling and post-bus filtering for the agent.
Package sampler implements per-kind sampling and post-bus filtering for the agent.
Package stream owns the bidi stream lifecycle between the agent and an admin server: the registration handshake, heartbeats, command dispatch (Subscribe / Unsubscribe / SnapshotRequest / Goodbye), event egress from the in-process bus, and graceful drain on shutdown.
Package stream owns the bidi stream lifecycle between the agent and an admin server: the registration handshake, heartbeats, command dispatch (Subscribe / Unsubscribe / SnapshotRequest / Goodbye), event egress from the in-process bus, and graceful drain on shutdown.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL