runnertelemetry

package
v0.18.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 16, 2026 License: Apache-2.0 Imports: 12 Imported by: 0

Documentation

Overview

Package runnertelemetry accepts the metrics and logs a distributed runner ships, and forwards them to the cluster's VictoriaMetrics and VictoriaLogs.

It exists so that neither of those ever has to listen anywhere a runner can reach. Open-source VictoriaMetrics and VictoriaLogs have no authentication of their own, so historically the only thing standing between a runner's network and unauthenticated write access to both was a firewall rule. Runners already hold a certificate from Join and already talk to the coordinator over an authenticated listener, so routing telemetry through that listener lets both stay bound to loopback permanently.

Index

Constants

View Source
const (
	// Audience scopes a telemetry token to this service. A system workload may
	// legitimately call several services, so the token it presents here must
	// name this one; sharing an audience with, say, the registry would make a
	// token minted for one replayable against the other.
	Audience = "miren-telemetry"

	// TokenHeader carries the runner's system workload token.
	//
	// Deliberately not Authorization. On a cloud-registered cluster the RPC
	// listener's authenticator tries the cloud JWT validator first whenever an
	// Authorization header is present, and a validation failure there returns a
	// hard error rather than falling through to the certificate check below it
	// (see pkg/cloudauth/rpc_authenticator.go). A cluster-issued workload token
	// is not a cloud JWT and would fail that validator, so putting it in
	// Authorization would turn an otherwise-valid mTLS request into a 401. A
	// header the authenticator ignores keeps the two credentials independent.
	TokenHeader = "Miren-Workload-Token"

	// StreamHeader marks a metrics batch as the runner's own operational
	// series (process memory and identity, host usage, controller queues)
	// rather than per-sandbox usage. Operational series join the
	// coordinator's, so they reach the embedded store and also leave the
	// cluster through the managed-metrics shipping path; sandbox series stay
	// in the embedded store, the same as the coordinator's own.
	//
	// A header on the existing route rather than a route of its own is what
	// makes version skew free. A coordinator that predates it ignores the
	// header and forwards the batch as it always has, so a runner upgraded
	// first loses nothing, where a new route would answer 404 and leave the
	// runner's writer retrying the batch until the coordinator caught up.
	StreamHeader = "Miren-Telemetry-Stream"

	// StreamOperational is the StreamHeader value for operational series.
	StreamOperational = "operational"

	// MetricsBasePath and LogsBasePath are what a runner points its writers at.
	// Each writer appends its own backend-native suffix, so the bytes on the
	// wire are exactly what VictoriaMetrics and VictoriaLogs already accept and
	// this package never has to understand the payloads.
	MetricsBasePath = "/_telemetry/metrics"
	LogsBasePath    = "/_telemetry/logs"
)

Variables

View Source
var (
	MetricsPattern = http.MethodPost + " " + MetricsBasePath + metricsImportPath
	LogsPattern    = http.MethodPost + " " + LogsBasePath + logsInsertPath
)

MetricsPattern and LogsPattern are the ServeMux patterns these handlers mount on.

They pin an exact path and method rather than proxying everything beneath a prefix. A prefix would hand a runner the rest of both APIs, including reads and VictoriaMetrics' delete-series admin endpoint, which would make scoping the token pointless: the credential would say "may write telemetry" while the route said "may do anything."

Functions

func LogsURL

func LogsURL(coordinatorAddress string) string

func MetricsURL

func MetricsURL(coordinatorAddress string) string

MetricsURL and LogsURL are what a runner points its writers at. Each writer appends its own backend-native suffix.

func NewLogsHandler

func NewLogsHandler(log *slog.Logger, verifier Verifier, address string) http.Handler

NewLogsHandler forwards accepted batches to VictoriaLogs' JSON-lines insert endpoint. address is the backend's host:port, normally loopback.

func NewMetricsHandler

func NewMetricsHandler(log *slog.Logger, verifier Verifier, address string, operational metrics.PointWriter) http.Handler

NewMetricsHandler forwards accepted batches to VictoriaMetrics' Prometheus import endpoint. address is the backend's host:port, normally loopback.

Batches marked as operational go to operational instead, which is the coordinator's own operational fanout: it already writes to the same embedded store, and it is where the shipping sink attaches. A nil operational forwards those batches to the backend like any other.

func NewTokenSource added in v0.17.0

func NewTokenSource() *workloadidentity.SystemTokenSource

NewTokenSource returns the source a runner's telemetry client mints through: telemetry writer tokens scoped to the coordinator's ingest. Its issuer is set once the runner has connected.

Types

type Client

type Client struct {
	// HTTP is what the telemetry writers are constructed with.
	HTTP *http.Client
	// contains filtered or unexported fields
}

Client is the HTTP client a runner's telemetry writers send through, paired with the QUIC transport underneath it.

The transport is held rather than left anonymous so it can be closed. The writers only ever need HTTP, but something has to own the QUIC connections, and without a handle on them they stay open until the process exits.

func NewClient

func NewClient(cfg ClientConfig) (*Client, error)

NewClient builds the client a runner's telemetry writers send through: HTTP/3 to the coordinator, authenticated by the runner's certificate, with a scoped workload token on every request.

HTTP/3 is not a preference. The coordinator's authenticated listener is QUIC only, so a plain net/http client cannot reach it at all.

func (*Client) Close

func (c *Client) Close() error

Close tears down the underlying QUIC connections.

func (*Client) Operational added in v0.17.0

func (c *Client) Operational() *http.Client

Operational returns a client for the runner's operational metrics writer. It shares this client's transport and credential and marks every request with StreamOperational, so the coordinator can tell those batches from the per-sandbox series sent through HTTP.

type ClientConfig

type ClientConfig struct {
	// ClientCertPEM and ClientKeyPEM are the runner's certificate from Join.
	// The listener requires one, so telemetry rides the same mutual TLS as the
	// rest of the runner's traffic and the token narrows what that identity may
	// do rather than replacing it.
	ClientCertPEM []byte
	ClientKeyPEM  []byte

	// CACertPEM verifies the coordinator.
	CACertPEM []byte

	// TokenSource supplies the system workload token.
	TokenSource workloadidentity.TokenSource

	// Timeout bounds a single telemetry request.
	Timeout time.Duration
}

ClientConfig describes how a runner reaches its coordinator's ingest endpoints.

type Verifier

type Verifier interface {
	VerifySystemWorkloadToken(token, audience string, workload workloadidentity.SystemWorkload) (*workloadidentity.WorkloadClaims, error)
}

Verifier checks that a presented token really identifies the telemetry writer of a runner in this cluster. It is satisfied by *workloadidentity.Issuer, which verifies in-process against the signing keys it already holds.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL