Documentation
¶
Overview ¶
Package store pkg/deployment/tpd/store/all_transports_cache.go c4-net-discovery
Package store pkg/deployment/tpd/store/bandwidth_day_index.go c4-net-discovery
Package store pkg/deployment/tpd/store/cxo_metrics_layout.go c4-net-discovery
Layout of TPD's transport-metrics CXO feed, plus the pivot/merge pair that converts between it and the []TransportMetric window shape every caller already expects.
The feed publishes ONE LEAF PER CALENDAR DAY at
metrics/day/<YYYY-MM-DD>
rather than one leaf per day-window. A window (1, 7, 30 days) is assembled reader-side from the N newest day leaves. The point is that a past day cannot change, so its leaf hashes the same on every republish and CXO ships nothing for it — whereas one leaf per window meant a 30-day body was a new object every time any single byte moved, and the whole multi-megabyte window went back over the wire even though 29 of its 30 days were settled.
Two fields are current-state, not day-scoped: Live (a transport's registration can expire at any moment) and Latency (a rolling min/max/avg that moves constantly). Carrying them in every day leaf would make every past day mutable again and defeat the whole exercise — across tens of thousands of transports, at least one liveness flip between cycles is a certainty, so every leaf would be a new object every time. They therefore live ONLY in the newest day's leaf, which is rewritten every cycle anyway. Older leaves carry the day-scoped facts and the transport's stable identity (ID, Type, Edges). MergeDailyMetrics walks the days newest-first, so the newest leaf's values win and the merged record has the shape it always had.
The rule this implies, stated plainly: the current day's leaf holds exactly what a 1-day query returns — every transport with a latency measurement or bandwidth today — and a transport that appears in NO current-day record reads back Live=false with no latency. That is the one behavioral difference from the old per-window bodies, where a 30-day window reported current liveness for a transport last seen three weeks ago. In practice a registered transport has a latency record (lat:<id> outlives registration by 35 days), so it is in the current leaf regardless of whether it moved bytes today.
These live in the store package rather than next to the publisher because they are the contract BETWEEN the publisher and its readers (pkg/visor, pkg/tpviz), and TransportMetric — the thing being pivoted — is defined right here.
Package store pkg/deployment/tpd/store/edge_entries_cache.go c4-net-discovery
Package store pkg/deployment/tpd/store/expired_entries_cache.go c4-net-discovery
Package store pkg/deployment/tpd/store/memory_store.go c4-net-discovery
Package store pkg/deployment/tpd/store/pubkey_cache.go c4-net-discovery
Package store pkg/deployment/tpd/store/redis_bandwidth.go c4-net-discovery
Package store pkg/deployment/tpd/store/redis_metrics.go c4-net-discovery
Package store pkg/deployment/tpd/store/redis_store.go c4-net-discovery
Package store pkg/deployment/tpd/store/redis_transport.go c4-net-discovery
Package store pkg/deployment/tpd/store/redis_uptime.go c4-net-discovery
Package store pkg/deployment/tpd/store/store.go c4-net-discovery
Index ¶
- Constants
- Variables
- func MetricsDayDate(path string) (string, bool)
- func MetricsDayPartPath(date string, part int) string
- func MetricsDayPath(date string) string
- func MetricsWindowDates(now time.Time, days int) []string
- func PivotDailyMetrics(metrics []TransportMetric, dates []string) map[string][]TransportMetric
- type BandwidthAggregation
- type CumulativeAggregate
- type DailyAggregate
- type DailyBandwidthEntry
- type DailyEdgeBandwidth
- type DailyVisorAggregate
- type EdgeBandwidth
- type LatencyRecord
- type MetricsQuery
- type NetworkMetricResponse
- type Store
- type ThroughputRecord
- type TransportData
- type TransportLatency
- type TransportMetric
- type TransportStore
- type TransportSummary
- type TransportUptimeSummary
- type TypeMetricAggregate
- type VisorBandwidthAggregate
- type VisorCumulativeAggregate
- type VisorMetricResponse
- type VisorSummary
Constants ¶
const MetricsDateFormat = "2006-01-02"
MetricsDateFormat is the calendar-date layout used in a day leaf's path. It matches the format the bw:daily:<id>:<date> Redis keys already use, so a leaf's name and its source rows agree.
const MetricsDayPrefix = "metrics/day/"
MetricsDayPrefix is the sub-tree the per-day leaves live under. It is also what a subscriber's path prefix must cover.
Variables ¶
var ( // ErrNotEnoughACKs means that we're still waiting for a visor to confirm the transport registration ErrNotEnoughACKs = errors.New("not enough ACKs") // ErrAlreadyRegistered indicates that transport ID is already in use. ErrAlreadyRegistered = errors.New("ID already registered") // ErrTransportNotFound indicates that requested transport is not registered. ErrTransportNotFound = errors.New("transport not found") // ErrBadEntry is returned when entry is malformed. ErrBadEntry = errors.New("bad entry format") // ErrUnknownStoreType means that store type is unknown. ErrUnknownStoreType = errors.New("unknown store type") )
Functions ¶
func MetricsDayDate ¶
MetricsDayDate extracts the calendar date from a day-leaf path or from one of its part paths. ok is false for any path that is not under MetricsDayPrefix — including the legacy metrics/days/<n> window leaves, which a reader must not mistake for a day.
func MetricsDayPartPath ¶
MetricsDayPartPath returns the path of one part of a day leaf that did not fit in a single CXO object. Zero-padded so a lexical sort of the paths is the publication order.
func MetricsDayPath ¶
MetricsDayPath returns the leaf path for one calendar day. Dates in this format sort lexically into chronological order, so a reader recovers "the N newest days" from a plain string sort.
func MetricsWindowDates ¶
MetricsWindowDates returns the calendar dates of a `days`-long window ending at now, NEWEST FIRST. The stepping matches buildTransportMetrics exactly (UTC, one AddDate per day back) so a pivoted row always finds its date in the window.
func PivotDailyMetrics ¶
func PivotDailyMetrics(metrics []TransportMetric, dates []string) map[string][]TransportMetric
PivotDailyMetrics splits a window's metrics into one slice per calendar day. dates is the window newest-first (as returned by MetricsWindowDates); every date gets an entry, empty days included, so the published path set is a deterministic function of the window rather than of the data.
dates[0] — the current day — is the only leaf that carries Live and Latency, and it carries a record for every transport that has latency at all, not just those with bandwidth today. That keeps pkg/tpviz's latency graph (which reads a single day) seeing exactly the transports the old 1-day window showed it.
A record with neither latency nor a row on the current day is omitted from the current leaf, mirroring the store's own rule that a transport with no metrics data is not reported.
Types ¶
type BandwidthAggregation ¶
type BandwidthAggregation struct {
TransportID string `json:"transport_id"`
Period string `json:"period"` // "daily", "weekly", "monthly"
PeriodKey string `json:"period_key"` // e.g., "2026-02-10", "2026-W06", "2026-02"
Bandwidth uint64 `json:"bandwidth"` // total bytes (sent + recv)
UpdatedAt int64 `json:"updated_at"`
}
BandwidthAggregation stores aggregated bandwidth for a time period.
type CumulativeAggregate ¶
type CumulativeAggregate struct {
Bandwidth uint64 `json:"bandwidth,omitempty"`
Latency float64 `json:"latency,omitempty"`
ByType map[string]*TypeMetricAggregate `json:"by_type,omitempty"`
}
CumulativeAggregate contains cumulative totals across all days.
type DailyAggregate ¶
type DailyAggregate struct {
Date string `json:"date"`
Bandwidth uint64 `json:"bandwidth,omitempty"`
Latency float64 `json:"latency,omitempty"`
ByType map[string]*TypeMetricAggregate `json:"by_type,omitempty"`
}
DailyAggregate contains network-wide aggregate for a single day.
type DailyBandwidthEntry ¶
type DailyBandwidthEntry struct {
Date string `json:"date"` // "2006-01-02"
Bandwidth uint64 `json:"bw"` // total bytes (sent + recv)
}
DailyBandwidthEntry stores bandwidth total for a single day.
type DailyEdgeBandwidth ¶
type DailyEdgeBandwidth struct {
Date string `json:"date"`
A *EdgeBandwidth `json:"a,omitempty"`
B *EdgeBandwidth `json:"b,omitempty"`
}
DailyEdgeBandwidth contains per-day bandwidth with edge A and B data.
type DailyVisorAggregate ¶
type DailyVisorAggregate struct {
Date string `json:"date"`
Bandwidth *VisorBandwidthAggregate `json:"bandwidth,omitempty"`
Latency float64 `json:"latency,omitempty"`
}
DailyVisorAggregate contains per-day visor aggregate data.
type EdgeBandwidth ¶
type EdgeBandwidth struct {
Sent uint64 `json:"sent"` // Cumulative bytes sent
Recv uint64 `json:"recv"` // Cumulative bytes received
}
EdgeBandwidth contains bandwidth data from one edge's perspective.
type LatencyRecord ¶
type LatencyRecord struct {
Min int64 `json:"min"` // microseconds
Max int64 `json:"max"` // microseconds
Avg int64 `json:"avg"` // microseconds
UpdatedAt int64 `json:"updated_at"`
}
LatencyRecord is the durable per-transport latency snapshot persisted at transport-discovery:lat:<id>. It lives independently of the registration blob (TransportData) so a 5-min entry-timeout cycle doesn't erase latency the way it did when the values were stored only inside the tp: key. Bandwidth gets the same independence via bw:daily:* keys; this gives latency the equivalent durability so /metrics doesn't show a transport with bandwidth-today but no latency just because the visor's re-registration window briefly lapsed.
Last-writer-wins on the per-transport key: latency is round-trip and both edges observe the same RTT modulo measurement noise, so no merge or per-edge tracking. Min/Max/Avg are stored in microseconds to match the existing TransportData layout. UpdatedAt is informational (lets readers gauge staleness) and not currently surfaced to the API.
type MetricsQuery ¶
type MetricsQuery struct {
Days int // 0 = all available (max 35)
Type string // Filter by transport type (stcpr, sudph, dmsg, stcp)
Live string // Filter by liveness: "true", "false", "all" (default: "all")
Edges bool // Include actual public keys in response
Bandwidth bool // Include bandwidth data (default: true)
Latency bool // Include latency data (default: true)
}
MetricsQuery contains query parameters for metrics endpoints.
type NetworkMetricResponse ¶
type NetworkMetricResponse struct {
Daily []DailyAggregate `json:"daily"`
Cumulative *CumulativeAggregate `json:"cumulative"`
}
NetworkMetricResponse is the response for GET /metric.
type Store ¶
type Store interface {
TransportStore
}
Store stores Transport metadata and generated nonce values.
type ThroughputRecord ¶
ThroughputRecord is the durable per-transport PEAK-goodput snapshot persisted at transport-discovery:tput:<id>. Bps is the passively observed peak (an EWMA tracked on the visor and pushed on the sharded telemetry feed); UpdateThroughput keeps the MAX across the two edges' reports so a momentarily-idle edge can't erase the other's peak. Shares latency's retention window and last-writer-wins independence from the registration blob. UpdatedAt is informational (staleness gauge).
type TransportData ¶
type TransportData struct {
ID string `json:"id"`
EdgeA string `json:"edge_a"`
EdgeB string `json:"edge_b"`
Type string `json:"type"`
Label string `json:"label"`
LatencyMin int64 `json:"lat_min"` // Minimum latency in microseconds
LatencyMax int64 `json:"lat_max"` // Maximum latency in microseconds
LatencyAvg int64 `json:"lat_avg"` // Average latency in microseconds
Bandwidth uint64 `json:"bandwidth"` // Total bytes (sent + recv)
LastUpdate int64 `json:"last_update"` // Unix timestamp of last update
}
type TransportLatency ¶
type TransportLatency struct {
Min int64 `json:"min"` // Minimum observed latency in microseconds
Max int64 `json:"max"` // Maximum observed latency in microseconds
Avg int64 `json:"avg"` // Average latency in microseconds
}
TransportLatency contains latency statistics for a transport in microseconds. Latency is at the transport level (not per-edge) since it's round-trip.
type TransportMetric ¶
type TransportMetric struct {
ID string `json:"id"`
Type string `json:"type"`
Live bool `json:"live"`
Edges []string `json:"edges,omitempty"` // Only included if query.Edges=true
Latency *TransportLatency `json:"latency,omitempty"` // Transport-level latency stats
Daily []DailyEdgeBandwidth `json:"daily"` // Per-day bandwidth by edge
}
TransportMetric contains per-transport metrics with latency at transport level and daily bandwidth breakdowns per edge.
func MergeDailyMetrics ¶
func MergeDailyMetrics(days [][]TransportMetric) []TransportMetric
MergeDailyMetrics rebuilds the []TransportMetric window shape from day slices given NEWEST FIRST. Records are joined on transport ID; daily rows accumulate in the order the days were supplied (newest first, matching buildTransportMetrics), and the first day that mentions a transport supplies its Live/Latency/Type/Edges.
The result is sorted by ID so a window assembled from the same leaves is byte-identical every time.
type TransportStore ¶
type TransportStore interface {
// RegisterTransport registers/refreshes a transport. reporter is the
// authenticated registering edge: only that edge's per-edge index TTL is
// refreshed, so an offline edge's index expires even while its live peer
// keeps re-registering the shared transport. A zero reporter (or one that
// is not an edge) falls back to refreshing both edges (legacy behavior).
RegisterTransport(context.Context, cipher.PubKey, *transport.SignedEntry) error
RegisterTransportsBatch(context.Context, cipher.PubKey, []*transport.SignedEntry) error
DeregisterTransport(context.Context, uuid.UUID) error
GetTransportByID(context.Context, uuid.UUID) (*transport.Entry, error)
GetTransportsByEdge(context.Context, cipher.PubKey) ([]*transport.Entry, error)
// GetTransportsByEdgeNoLatency is the cheap variant for callers
// (DHT mirror, transport-count stats) that don't need the
// durable latency overlay. Skips the per-call MGET on lat:<id>
// keys and the JSON decode that follows.
GetTransportsByEdgeNoLatency(context.Context, cipher.PubKey) ([]*transport.Entry, error)
GetNumberOfTransports(context.Context) (map[types.Type]int, error)
// GetTransportSummary returns network-wide aggregate counts (total,
// per-type, unique visors) computed in a single index+MGET pass that
// never materializes []*transport.Entry — the cheap path behind GET
// /all-transports/stats, so a client counting transports doesn't have
// to fetch every entry. selfTransports=false excludes loopback
// self-transports (Edges[0]==Edges[1]), matching GetAllTransports.
GetTransportSummary(ctx context.Context, selfTransports bool) (*TransportSummary, error)
GetAllTransports(context.Context, bool) ([]*transport.Entry, error)
// Bandwidth ingest (called by the CXO aggregator with the
// reporter's cumulative counters; deltas are computed
// internally against the per-reporter previous snapshot).
UpdateBandwidth(ctx context.Context, transportID string, reporterPK cipher.PubKey, sent, recv uint64) error
// Latency ingest (called by the CXO aggregator with the
// reporter's window min/max/avg in milliseconds). RTT-symmetric,
// so last-writer-wins across the two edges is acceptable.
UpdateLatency(ctx context.Context, transportID string, minMS, maxMS, avgMS float64) error
// Throughput ingest (called by the CXO aggregator with the
// reporter's passively-observed PEAK goodput estimate in bytes/sec,
// carried on the sharded telemetry feed). Recorded per-transport with
// a peak-preserving max so the higher of the two edges' observations
// survives; overlaid onto Entry.ThroughputBps on read.
UpdateThroughput(ctx context.Context, transportID string, reporterPK cipher.PubKey, bps float64) error
// Bandwidth query methods (legacy)
GetTransportBandwidth(ctx context.Context, tpID uuid.UUID, period string, limit int) ([]BandwidthAggregation, error)
GetVisorBandwidth(ctx context.Context, pk cipher.PubKey, period string, limit int) ([]BandwidthAggregation, error)
GetAllVisorSummaries(ctx context.Context, v2 bool, timeline bool) ([]VisorSummary, error)
RecordHeartbeat(ctx context.Context, pk cipher.PubKey, version string) error
GetDailyTimeline(ctx context.Context, pkHex string, now time.Time) map[string]string
// Transport uptime tracking (stcpr/sudph only).
// at is the time the heartbeat was *observed* on the visor side
// (snap.SampledAt for the CXO path, time.Now() for the HTTP
// register path); the slot bit and date bucket derive from it,
// so leaf-arrival skew that crosses a 5-minute boundary still
// credits the correct slot. Zero time falls back to time.Now().
RecordTransportHeartbeat(ctx context.Context, tpID uuid.UUID, tpType string, at time.Time) error
// IngestTransportTimeline OR-merges a visor-supplied per-transport
// uptime bitmap into the persistent timeline. Used by the CXO
// aggregator to absorb the visor's locally-tracked bitmap, which
// captures slots TPD's heartbeat path never observed (e.g.
// during TPD downtime). bitmap must be 36 bytes; date is the
// UTC YYYY-MM-DD the bitmap covers.
IngestTransportTimeline(ctx context.Context, tpID uuid.UUID, date string, bitmap []byte) error
GetTransportUptimeSummaries(ctx context.Context, tpIDs []uuid.UUID, v2 bool, timeline bool) ([]TransportUptimeSummary, error)
GetTransportUptimeByVisor(ctx context.Context, pk cipher.PubKey, v2 bool, timeline bool) ([]TransportUptimeSummary, error)
GetTransportDailyTimeline(ctx context.Context, tpID string, now time.Time) map[string]string
BackupAndCleanOldBandwidth(ctx context.Context, backupPath string) error
// New metrics methods
GetNetworkMetrics(ctx context.Context, query MetricsQuery) (*NetworkMetricResponse, error)
GetVisorAggregateMetrics(ctx context.Context, pks []cipher.PubKey, query MetricsQuery) (map[string]*VisorMetricResponse, error)
GetAllTransportMetrics(ctx context.Context, query MetricsQuery) ([]TransportMetric, error)
GetTransportMetricsByIDs(ctx context.Context, ids []uuid.UUID, query MetricsQuery) ([]TransportMetric, error)
GetTransportMetricsByVisors(ctx context.Context, pks []cipher.PubKey, query MetricsQuery) ([]TransportMetric, error)
Close()
}
TransportStore stores Transport metadata.
type TransportSummary ¶
type TransportSummary struct {
Total int `json:"total_transports"`
ByType map[string]int `json:"by_type"`
UniqueVisors int `json:"unique_visors"`
// Partial reports that at least one batch of the underlying read failed,
// so Total and ByType undercount by an unknown amount.
//
// This exists because the counts are assembled from batched MGETs over the
// transport index, and a failed batch used to be skipped with a bare
// `continue` — silently removing up to mgetBatch transports from the
// answer while still returning a plausible-looking number. A consumer had
// no way to tell "there are N transports" from "N is what I managed to
// read", which is indistinguishable from the network shrinking.
//
// Callers that publish or chart these numbers must treat Partial as
// "do not trust this sample" rather than as data.
Partial bool `json:"partial,omitempty"`
// MissingBatches counts the failed batches, for logging and for judging
// how far off Total might be.
MissingBatches int `json:"missing_batches,omitempty"`
}
TransportSummary is a network-wide aggregate of registered transports, computed server-side so a client can obtain the counts without fetching (and counting) the full transport list. Marshals to the same JSON shape as GET /all-transports/stats.
type TransportUptimeSummary ¶
type TransportUptimeSummary struct {
ID uuid.UUID `json:"id"`
Online bool `json:"on"`
Type string `json:"type,omitempty"`
EdgeA string `json:"edge_a,omitempty"`
EdgeB string `json:"edge_b,omitempty"`
Daily map[string]string `json:"daily,omitempty"`
Timeline map[string]string `json:"timeline,omitempty"`
}
TransportUptimeSummary holds uptime data for a single transport. Edges are omitted by default — include via ?edges=true query param.
type TypeMetricAggregate ¶
type TypeMetricAggregate struct {
Bandwidth uint64 `json:"bandwidth,omitempty"`
Latency float64 `json:"latency,omitempty"`
}
TypeMetricAggregate contains aggregate metrics for a transport type.
type VisorBandwidthAggregate ¶
type VisorBandwidthAggregate struct {
Sent uint64 `json:"sent"`
Recv uint64 `json:"recv"`
Total uint64 `json:"total"`
}
VisorBandwidthAggregate contains bandwidth totals for a visor.
type VisorCumulativeAggregate ¶
type VisorCumulativeAggregate struct {
Bandwidth *VisorBandwidthAggregate `json:"bandwidth,omitempty"`
Latency float64 `json:"latency,omitempty"`
}
VisorCumulativeAggregate contains cumulative visor totals.
type VisorMetricResponse ¶
type VisorMetricResponse struct {
Daily []DailyVisorAggregate `json:"daily"`
Cumulative *VisorCumulativeAggregate `json:"cumulative"`
}
VisorMetricResponse is the response for a single visor in GET /metric/visor/{pks}.
type VisorSummary ¶
type VisorSummary struct {
PK cipher.PubKey `json:"pk"`
Online bool `json:"on"`
Version string `json:"version,omitempty"`
Daily map[string]string `json:"daily,omitempty"`
Timeline map[string]string `json:"timeline,omitempty"`
}
VisorSummary holds a visor's uptime data. This format matches the Uptime Tracker's response format. v3 adds Timeline — a per-day string of 288 chars (one per 5-minute slot) where '.' = heartbeat received and ' ' = missed.