Help us shape the future of Go! Take the 2026 Go Developer Survey

setupmetrics

package
v1.3.104 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 31, 2026 License: AGPL-3.0 Imports: 12 Imported by: 0

Documentation

Overview

Package setupmetrics pkg/router/setupmetrics/empty.go c2-net-routing

Package setupmetrics pkg/router/setupmetrics/kinds.go c2-net-routing

Which SETUP PATH each request took, and what batching bought.

A route can reach a setup node three ways today and the /stats snapshot could not tell them apart: the legacy single DialRouteGroup (what production runs — routing.enable_cascade_route_setup defaults off), the batched DialRouteGroupBatch added alongside it, and the source-driven cascade, where the node only SIGNS and the source injects the cascade down its own transports. "total_requests 54" said nothing about which of the three, so an operator could not see whether a deployed client had negotiated the batch form at all.

RequestsByKind answers that, RoutesPerBatch shows the batch sizes actually arriving, and PerHopRPCsSaved is the concrete coalescing win: the per-hop RPCs the batched path did NOT issue because one id reservation and one intermediary install covered several routes.

Package setupmetrics pkg/router/setupmetrics/own_view.go c2-net-routing

The caller-scoped view of the route-setup snapshot.

/stats splits in two: the aggregate half is public (anyone depends on this shared layer and deserves to see whether it works), the topology half — top_destinations, top_failed_destinations, recent_failures — is gated on the survey whitelist, because the setup node participates in EVERY route setup and its record of who sets up routes to whom is traffic-analysis material.

The gap that left: an ordinary visor asking its own setup node "which destination is failing FOR ME" got `"top_destinations": null` and `"recent_failures": null`, so `pk`, `src_pk` and `dst_pk` were empty for every operator not on the whitelist — which is every operator, since the whitelist holds seven deployment keys. The structured fields exist and are filled in correctly; they were simply never sent.

A caller does not need privilege to be told about its OWN route setups: it already knows which destinations it dialed and which of them failed. So the non-whitelisted view is no longer empty — it is the caller's own rows, and nobody else's. The whitelisted view is unchanged (every row, every key).

Package setupmetrics pkg/router/setupmetrics/stats.go c2-net-routing requests. Complements victoria_metrics.go (which publishes Prometheus gauges for external scrape) with a structured snapshot the visor can expose over RPC and the CLI can pretty-print for quick triage.

The collector is safe for concurrent use. It categorizes each failure into a small set of well-known reasons so operators can answer "which part of the setup path is broken?" without correlating raw log lines.

Package setupmetrics pkg/router/setupmetrics/victoria_metrics.go c2-net-routing

Index

Constants

View Source 
const (
	// CircuitBreakerEnabledDefault is OFF.
	//
	// The breaker is a per-destination lockout owned by whichever route
	// setup node served the request, and that node cannot see what the
	// SOURCE visor already has: on 2026-09-18 a visor holding 32 live
	// route groups to one exit was refused every further setup to it for
	// minutes, and a fresh proxy start got 0 route groups for 180s. The
	// deployment RSN's own counters at the time read 481 failures of
	// which 457 were circuit_open — the breaker's refusals, not route
	// setups that were tried and failed, and they had evicted the real
	// failures out of the diagnostic ring. A setup that is attempted and
	// fails costs one id-reservation timeout; a breaker that opens on a
	// misattributed failure costs every caller of that destination. Off
	// is the safer default.
	CircuitBreakerEnabledDefault = false
	// CircuitFailureThresholdDefault is the number of consecutive
	// failures to a destination that trips the breaker to OPEN.
	//
	// Lowered from 5 → 3 after the 1.3.59-era observation that
	// dmsg-error-202 (intermediate's stale dmsg session) consistently
	// produces 3+ failures back-to-back; tolerating 5 wastes the next
	// two setup attempts on doomed routes. With the open duration
	// tuned to dmsg-session-refresh cadence (below), tripping earlier
	// is the right balance.
	CircuitFailureThresholdDefault = 3
	// CircuitOpenDurationDefault is how long the breaker stays OPEN
	// before transitioning to HALF_OPEN to allow a probe setup.
	//
	// Bumped from 60s → 5min to align with dmsg's own session-refresh
	// cadence (~5min). dmsg-error-202 means a visor is registered in
	// dmsg-disc but its actual delegated-server session is stale; the
	// session re-publishes itself on a ~5min cycle, so retrying inside
	// that window is doomed to hit the same stale state. 60s let
	// half-open probes burn through the same failure on every cycle.
	CircuitOpenDurationDefault = 5 * time.Minute
	// CircuitMaxOpenDurationDefault is the maximum time a breaker can
	// stay in the open/half_open cycle before being force-reset to
	// closed. This prevents permanent lockout when the RSN's own DMSG
	// sessions are stale but the destination is actually alive — fresh
	// traffic will establish new DMSG paths.
	//
	// Bumped from 10min → 30min in concert with the longer open
	// duration; with 5min between half-open probes, 10min is only two
	// retry windows which is tight for a genuinely-down peer waiting
	// to come back. 30min gives ~6 half-open probes before force-reset.
	CircuitMaxOpenDurationDefault = 30 * time.Minute
	// CircuitFailureWindowDefault bounds how long consecutive failures
	// must occur within to count toward the threshold. Failures older
	// than this window are considered stale and the consecutive counter
	// is reset on the next record.
	CircuitFailureWindowDefault = 5 * time.Minute
)

Circuit breaker defaults. These are what the setup.circuit_* knobs in the router settings catalog ship with; the live values are read through the accessors below, so an operator can retune the breaker — or turn it back on — without a restart.

Variables

This section is empty.

Functions

This section is empty.

Types

type BatchStats 

type BatchStats struct {
	// Batches is how many batched requests were handled.
	Batches uint64 `json:"batches"`
	// Routes is the total number of routes those batches carried.
	Routes uint64 `json:"routes"`
	// Installed is how many of those routes came back installed — a batch
	// returns per-route results and partial success is normal.
	Installed uint64 `json:"installed"`
	// RoutesPerBatch is a histogram of batch sizes: routes-in-the-batch →
	// number of batches of that size.
	RoutesPerBatch map[int]uint64 `json:"routes_per_batch"`
	// PerHopRPCsSaved is the running total of per-hop RPCs the coalescing
	// removed — the id-reservation and intermediary-install calls the batch
	// did NOT have to issue because one call covered several routes.
	PerHopRPCsSaved uint64 `json:"per_hop_rpcs_saved"`
}

BatchStats is the batched-path summary in a StatsSnapshot. It is present even when no batch has arrived (all zeroes), so an operator can tell "no client has negotiated the batch form" from "this field does not exist on this build".

type BreakerState 

type BreakerState struct {
	State            CircuitState `json:"state"`
	OpenedAt         time.Time    `json:"opened_at"`
	ConsecutiveFails int          `json:"consecutive_fails"`
	ProbeInFlight    bool         `json:"probe_in_flight"`
}

BreakerState is the JSON view of one circuit breaker that is not closed. Snapshot only emits non-closed entries, so an empty "breakers" object means route setup is short-circuiting nobody — and a lingering half_open entry with probe_in_flight=true is immediately visible to an operator instead of only showing up as "probe in flight" strings in the caller's error.

type CircuitState 

type CircuitState string

CircuitState is the per-destination breaker state. "closed" = normal, "open" = fast-fail new setups, "half_open" = allow one probe to test recovery. 

const (
	// CircuitClosed means setups to this destination proceed normally.
	CircuitClosed CircuitState = "closed"
	// CircuitOpen means setups to this destination are short-circuited.
	CircuitOpen CircuitState = "open"
	// CircuitHalfOpen means one probe setup is allowed to test recovery.
	CircuitHalfOpen CircuitState = "half_open"
)

type Collector 

type Collector struct {
	// contains filtered or unexported fields
}

Collector is a thread-safe aggregator of route-setup attempt metadata. It implements Metrics so it can be wired anywhere the existing Victoria Metrics struct is used (the two are independent — wiring the Collector does not remove Prometheus output).

func NewCollector 

func NewCollector(cfg CollectorConfig) *Collector

NewCollector returns a ready-to-use Collector.

func (*Collector) AllowDestination 

func (c *Collector) AllowDestination(dstPK cipher.PubKey, held *ProbeHolder) (bool, string)

AllowDestination reports whether a new route setup to dst should proceed. Returns (true, "") when the breaker is closed or half-open, (false, reason) when it is open. Call this from the request handler BEFORE doing any dial work — rejecting early is the whole point.

The half-open transition is driven here: when we find an open breaker that has been open for circuitOpenDuration, we flip it to half-open and allow the current probe through. A subsequent success will close the breaker (via finish); a failure re-opens it and resets the per-cycle timer.

held records the probe against the calling request; pass the same *ProbeHolder that was given to RecordRouteContextProbes so finish() can release the slot however the request ends. A nil holder is accepted for callers that only inspect the breaker and never run a request through finish() (tests, diagnostics) — such a caller never takes the probe slot.

AllowIntermediate is the per-intermediate sibling — see below.

func (*Collector) AllowIntermediate 

func (c *Collector) AllowIntermediate(interPK cipher.PubKey, held *ProbeHolder) (bool, string)

AllowIntermediate reports whether a route candidate whose path traverses interPK should be allowed to proceed. Intermediate breakers are populated by id_reservation failures attributed to the intermediate (see the default branch in finish()). Callers that have already computed a candidate route's intermediates (typically the route-setup-node, or a router that wants to pre-filter route-finder output) can call this for each hop and reject the route early if any intermediate's breaker is open — avoiding a ~10s id-reservation timeout per known-bad hop.

Uses the same allowPK helper as AllowDestination; intermediate breakers live in the same map as destination breakers (keyed by PK string), which keeps the bookkeeping uniform — including the half-open probe slot, which is why held matters here just as much as it does for the destination.

func (*Collector) RecordBatch 

func (c *Collector) RecordBatch(routes, installed, rpcsSaved int)

RecordBatch records one completed batch: how many routes it carried, how many installed, and how many per-hop RPCs the coalescing saved.

func (*Collector) RecordConcurrencyDrop 

func (c *Collector) RecordConcurrencyDrop()

RecordConcurrencyDrop is called when a request is refused at the accept-loop backpressure check (never reaches the handler).

func (*Collector) RecordRequest 

func (c *Collector) RecordRequest() func(*routing.EdgeRules, *error)

RecordRequest satisfies setupmetrics.Metrics — it bumps the active request counter up-front and decrements it when the deferred closure runs. Per-request success/failure counts are deliberately kept in RecordRoute so the two don't double-count when both are called (which is what the standalone setup node does today).

func (*Collector) RecordRoute 

func (c *Collector) RecordRoute() func(*error)

RecordRoute satisfies setupmetrics.Metrics. When the collector is wired through the normal path this records the attempt under its "unknown" labels — callers who want richer data should use RecordRouteContext instead.

func (*Collector) RecordRouteContext 

func (c *Collector) RecordRouteContext(ctx context.Context, srcPK, dstPK cipher.PubKey, hopCount int) func(*error)

RecordRouteContext is the richer variant of RecordRoute that captures enough context to fill a FailureEvent. Call it once per route attempt at the top of the setup handler and defer the returned closure. 

defer collector.RecordRouteContext(ctx, src, dst, hopCount)(&err)

func (*Collector) RecordRouteContextProbes 

func (c *Collector) RecordRouteContextProbes(ctx context.Context, srcPK, dstPK cipher.PubKey, hopCount int, held *ProbeHolder) func(*error)

RecordRouteContextProbes is RecordRouteContext plus the half-open probe bookkeeping: held is the same *ProbeHolder the request passes to AllowDestination / AllowIntermediate, and every probe it holds is resolved when the returned closure runs. Callers that consult the breakers MUST use this variant — a probe taken by a request that never releases it blocks the breaker's only recovery slot until the 30-minute force-reset. 

var probes setupmetrics.ProbeHolder
defer collector.RecordRouteContextProbes(ctx, src, dst, hopCount, &probes)(&err)

func (*Collector) RecordSetupKind 

func (c *Collector) RecordSetupKind(kind SetupKind, routes int)

RecordSetupKind counts one request against a setup path. routes is how many routes the request carried (1 for single and cascade-sign), which is what makes "requests by kind" comparable with "routes by kind".

func (*Collector) Reset 

func (c *Collector) Reset()

Reset clears all counters and ring buffers. Useful before starting a diagnostic capture window.

func (*Collector) Snapshot 

func (c *Collector) Snapshot() StatsSnapshot

Snapshot returns a deep copy of the current state. Safe to marshal.

func (*Collector) SnapshotFor 

func (c *Collector) SnapshotFor(caller cipher.PubKey, whitelisted bool) StatsSnapshot

SnapshotFor returns the snapshot a specific caller may see. A whitelisted caller gets the full snapshot; everyone else gets the aggregate plus their OWN rows — the destinations they themselves set up routes to, and their own failures, error strings included (the keys in those strings are the caller's own peers).

A null caller PK — an unparseable RemoteAddr — gets the aggregate alone.

What a non-whitelisted caller still never sees is the who-talks-to-whom half for anyone else: the other visors this node set up routes for, the destinations they used, and the FailureEvent.Error strings that embed the same keys in prose. The RSN participates in EVERY route setup, so its view of which visors route to which is unusually complete, and publishing that is traffic-analysis material a routing overlay must not hand out.

type CollectorConfig 

type CollectorConfig struct {
	LatencyRingSize int // number of recent successful latencies to keep for percentile calc
	FailureRingSize int // number of recent failures to keep in RecentFailures
	DestCapacity    int // max number of distinct destinations to track
}

CollectorConfig configures the ring buffer / destination cap sizes. Zero values mean "use the default".

type DestStat 

type DestStat struct {
	PK      string `json:"pk"`
	Total   uint64 `json:"total"`
	Failed  uint64 `json:"failed"`
	Circuit string `json:"circuit,omitempty"`
}

DestStat is a per-destination counter used to surface "hot" visors (targets of many requests) and "troublesome" visors (high failure rate). Circuit state is the CircuitState (closed/open/half_open) as of the last update.

type Empty 

type Empty struct{}

Empty is a `Metrics` implementation which does nothing.

func NewEmpty 

func NewEmpty() Empty

NewEmpty creates a new metrics implementation that does nothing.

func (Empty) RecordRequest 

func (Empty) RecordRequest() func(*routing.EdgeRules, *error)

RecordRequest implements `Metrics`.

func (Empty) RecordRoute 

func (Empty) RecordRoute() func(*error)

RecordRoute implements `Metrics`.

type FailureEvent 

type FailureEvent struct {
	Timestamp  time.Time     `json:"timestamp"`
	SrcPK      string        `json:"src_pk"`
	DstPK      string        `json:"dst_pk"`
	HopCount   int           `json:"hop_count"`
	Reason     FailureReason `json:"reason"`
	Error      string        `json:"error"`
	DurationMs int64         `json:"duration_ms"`
}

FailureEvent captures a single failed setup attempt with enough context to start an investigation without a log trawl.

type FailureReason 

type FailureReason string

FailureReason is a short stable identifier for why a route setup attempt failed. Keep values lowercase-snake so they render cleanly in JSON / CLI tables. 

const (
	ReasonInvalidRoute            FailureReason = "invalid_route"            // BidirectionalRoute.Check failed
	ReasonRuleGeneration          FailureReason = "rule_generation"          // GenerateRules returned ErrNoKey etc.
	ReasonIDReservation           FailureReason = "id_reservation"           // ReserveRouteIDs failed dialing the destination
	ReasonSourceUnreachable       FailureReason = "source_unreachable"       // ReserveRouteIDs could not reach the SOURCE visor (not the dst's fault)
	ReasonIntermediateUnreachable FailureReason = "intermediate_unreachable" // ReserveRouteIDs could not reach an INTERMEDIATE visor (not the dst's fault)
	ReasonIntermediaryRules       FailureReason = "intermediary_rules"       // BroadcastIntermediaryRules failed
	ReasonDestinationRules        FailureReason = "destination_rules"        // AddEdgeRules on destination router failed
	ReasonContextDeadline         FailureReason = "context_deadline"         // overall request timed out
	ReasonContextCanceled         FailureReason = "context_canceled"         // caller canceled before completion
	ReasonConcurrencyLimit        FailureReason = "concurrency_limit"        // dropped at the accept loop backpressure check
	ReasonCircuitOpen             FailureReason = "circuit_open"             // per-destination breaker short-circuited the setup
	ReasonUnknown                 FailureReason = "unknown"                  // failure classifier could not decide
)

Failure reason categories. The set intentionally stays small so operators can reason about them; unknown / unclassified failures fall through to ReasonUnknown and the raw error message is still captured in the RecentFailures ring buffer for investigation.

type LatencyStats 

type LatencyStats struct {
	Count uint64 `json:"count"`
	Min   int64  `json:"min_ms"`
	Max   int64  `json:"max_ms"`
	Mean  int64  `json:"mean_ms"`
	P50   int64  `json:"p50_ms"`
	P95   int64  `json:"p95_ms"`
	P99   int64  `json:"p99_ms"`
}

LatencyStats are percentile summaries (milliseconds) computed over the collector's latency ring on demand.

type Metrics 

type Metrics interface {
	RecordRequest() func(*routing.EdgeRules, *error)
	RecordRoute() func(*error)
}

Metrics collects metrics in prometheus format.

type ProbeHolder 

type ProbeHolder struct {
	// contains filtered or unexported fields
}

ProbeHolder records which PKs a single request was admitted through as the half-open probe, so finish() can resolve every one of them when that request ends. Admission and resolution MUST be symmetric: a probe left in flight past the end of the request that took it locks the breaker's half-open slot until circuitMaxOpenDuration force-resets it (30 minutes of refusing everybody), which is exactly the failure observed live on 2026-09-16 for an intermediate that was perfectly reachable.

The zero value is ready to use. Create one per request, pass it to every Allow* call that request makes, and hand the same holder to RecordRouteContextProbes.

type SetupKind 

type SetupKind string

SetupKind names a route-setup path. Values are lowercase-snake so they render as JSON object keys. 

const (
	// SetupKindSingle is one BidirectionalRoute per DialRouteGroup request —
	// the legacy path, and what the live deployment uses today.
	SetupKindSingle SetupKind = "single"
	// SetupKindBatch is one DialRouteGroupBatch carrying N routes to one
	// destination, with the per-hop work coalesced.
	SetupKindBatch SetupKind = "batch"
	// SetupKindCascadeSign is the source-driven cascade: the node signs the
	// reserve/install cascades and the SOURCE injects them down its own
	// transports. Counted once per signing request pair.
	SetupKindCascadeSign SetupKind = "cascade_sign"
)

The setup paths a request can take.

type StatsSnapshot 

type StatsSnapshot struct {
	// Version is the build of the setup node that answered.
	Version   string    `json:"version,omitempty"`
	StartedAt time.Time `json:"started_at"`
	UptimeSec int64     `json:"uptime_sec"`

	TotalRequests    uint64 `json:"total_requests"`
	Successful       uint64 `json:"successful"`
	Failed           uint64 `json:"failed"`
	ConcurrencyDrops uint64 `json:"concurrency_drops"`
	ActiveRequests   int    `json:"active_requests"`

	SuccessRatePct float64 `json:"success_rate_pct"`

	FailuresByReason map[FailureReason]uint64 `json:"failures_by_reason"`

	LatencyMs LatencyStats `json:"latency_ms"`

	// RouteLengthHist maps hop count → number of successful setups.
	RouteLengthHist map[int]uint64 `json:"route_length_hist"`

	// RequestsByKind / RoutesByKind split the request count by the setup PATH
	// it took — single, batch, cascade_sign (see kinds.go). Aggregate, so both
	// are part of the PUBLIC view: they name no visor.
	RequestsByKind map[SetupKind]uint64 `json:"requests_by_kind"`
	RoutesByKind   map[SetupKind]uint64 `json:"routes_by_kind"`

	// Batch is the batched-path summary: sizes seen and per-hop RPCs the
	// coalescing saved. Also aggregate, also public.
	Batch BatchStats `json:"batch"`

	// Top destinations sorted by total request count (descending).
	// Capped at the collector's top-N config.
	TopDestinations []DestStat `json:"top_destinations"`

	// Destinations ranked by failure count (descending). Same cap.
	TopFailedDestinations []DestStat `json:"top_failed_destinations"`

	// Most recent N failures, newest first.
	RecentFailures []FailureEvent `json:"recent_failures"`

	// Breakers holds every circuit breaker that is not closed, keyed
	// by PK — destinations and intermediates alike. Omitted when all
	// breakers are closed.
	Breakers map[string]BreakerState `json:"breakers,omitempty"`

	LastSuccessAt *time.Time `json:"last_success_at,omitempty"`
	LastFailureAt *time.Time `json:"last_failure_at,omitempty"`
}

StatsSnapshot is the public, JSON-friendly view exposed over RPC/CLI. Every field is a point-in-time copy — the underlying collector holds its own mutex during Snapshot so the struct is safe to marshal and return.

type VictoriaMetrics 

type VictoriaMetrics struct {
	// contains filtered or unexported fields
}

VictoriaMetrics implements `Metrics` using Victoria Metrics.

func NewVictoriaMetrics 

func NewVictoriaMetrics() *VictoriaMetrics

NewVictoriaMetrics returns the Victoria Metrics implementation of Metrics.

func (*VictoriaMetrics) RecordRequest 

func (m *VictoriaMetrics) RecordRequest() func(rules *routing.EdgeRules, err *error)

RecordRequest implements `Metrics`.

func (*VictoriaMetrics) RecordRoute 

func (m *VictoriaMetrics) RecordRoute() func(err *error)

RecordRoute implements `Metrics`.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.