Documentation
¶
Overview ¶
Package redmetrics turns socket observations into per-container, per-minute RED measurements for the containers running on this node.
The shape of the thing is fixed by where it ends up. Rows land in a ClickHouse SummingMergeTree keyed by (cluster_id, timestamp, container_id), which means merges ADD rows that share a key. Three consequences run through this package:
- Every value emitted is a DELTA for one minute, never a running total.
- A minute bucket closes when the wall clock passes it and is never reopened, so a bucket is emitted exactly once.
- Nothing may retry. A re-sent item is indistinguishable from real extra traffic, and no amount of care downstream can undo it.
See docs/features/red-metrics/INVARIANTS.md.
The aggregation is keyed by {minute, container} and deliberately NOT by peer. That is what keeps the in-memory footprint bounded by the number of containers the kubelet is running rather than by how many distinct callers reach them.
Index ¶
Constants ¶
const DurationBucketCount = 14
DurationBucketCount is the number of cumulative latency buckets carried by one emitted item. The server REJECTS THE WHOLE REQUEST when an item carries any other number, on the grounds that a length mismatch means the two sides no longer agree about what the buckets mean.
const PayloadCap = 2048
PayloadCap is the most bytes a single socket observation carries into userspace. The probe copies at most this much per call and always reports the call's true length separately, so a decoder can advance its stream position through a body it never saw.
It is the reason a truncated message boundary costs a connection its decodability while a truncated body does not: a body is walked past by count, but a header block has to be read, and 2KB is the budget for reading one.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Aggregator ¶
type Aggregator struct {
// contains filtered or unexported fields
}
Aggregator turns socket observations into per-container, per-minute items.
Its output is deltas. A bucket closes when the wall clock passes the minute it covers, is emitted exactly once, and is never reopened -- an observation arriving for a closed minute is dropped rather than filed late, because a late row is indistinguishable from real extra traffic once the SummingMergeTree has merged it. For the same reason nothing that consumes Flush may retry.
func NewAggregator ¶
NewAggregator builds an aggregator. A zero-valued Config field takes the default, so callers can override one knob without restating the others.
func (*Aggregator) Flush ¶
func (a *Aggregator) Flush() []*gen.ContainerRedMetricsItem
Flush closes every minute the wall clock has passed and returns them as items, holding back the minute in progress. Buckets are removed as they are emitted, so a second call returns nothing new -- and a caller that drops the result loses those measurements, which is the correct trade against re-sending them.
func (*Aggregator) Observe ¶
func (a *Aggregator) Observe(ev SocketEvent)
Observe files one socket observation. It is safe for concurrent use and is on the probe's hot path, so the expensive work -- pod lookup and port attribution -- happens once per connection and never again.
type Clock ¶
Clock is this package's only source of wall clock time. Minute buckets are stamped by the agent, so tests need to move time deliberately rather than wait for it.
type Config ¶
type Config struct {
// ConnTTL is how long a connection's state survives without an observation.
//
// It is a leak backstop rather than the mechanism that reclaims connections. The
// probe attaches to tcp_close, so a socket going away says so and its state is
// released at that point; this bounds only the entries whose close was never
// seen -- an event lost to a full ring buffer, or a socket already open when the
// probe attached.
//
// Which is why it can be generous, and why it should be. When idleness was the
// only available death signal this had to be short enough to bound the table,
// and a short one is actively destructive: a gRPC stream or an HTTP keep-alive
// connection quiet for longer than the TTL was swept, then decoded again from
// the middle when it resumed -- where nothing parses, so it landed opaque -- or,
// if its next syscall was a send, classified outbound and ignored for the rest
// of its life. Neither is a small loss on a node serving long-lived connections.
//
// Keep it at or above RequestTTL. Below it, a connection can be reclaimed while
// it still holds requests the sweep has not yet had cause to file, and those
// requests are lost rather than counted unanswered.
ConnTTL time.Duration
// MaxConnections caps concurrently tracked connections. Beyond it, new
// connections are dropped rather than tracked -- the alternative is an
// unbounded map on a DaemonSet that shares a node with the workloads it
// measures.
//
// Outbound connections count against it too, even though nothing is measured
// about them. The cap bounds memory, and an outbound entry occupies the table
// exactly as an inbound one does; excluding them would make the number stop
// describing the thing it exists to bound. On a node whose outbound churn is
// large enough to fill the table this does cost inbound coverage, which the
// one-shot capacity log is there to make visible.
//
// What the cap holds changed with the close probe. It used to hold every
// connection seen within ConnTTL, which on a node churning short-lived
// connections is overwhelmingly sockets the kernel had already freed -- so the
// table filled with corpses and refused live connections at a fraction of the
// offered rate. It now holds connections that are actually open, which is a
// number set by concurrency rather than by rate.
MaxConnections int
// RequestTTL is how long a request may stay outstanding before it is filed as
// unanswered: counted in request_count, in no latency bucket, and not in
// error_count.
//
// It is a deadline on the agent's patience, not a statement about the service.
// A request answered after it has been swept is not filed twice -- the response
// finds nothing outstanding and is dropped -- so raising this trades memory for
// the latency of genuinely slow requests being measured rather than merely
// counted, and lowering it trades the other way.
//
// It applies only to a request left outstanding on a connection that stays open.
// A connection that closes settles everything it was still holding at the close,
// so this does not have to be long enough to cover a client that hangs up.
RequestTTL time.Duration
// MaxInFlightRequests caps outstanding requests across all connections. A
// request arriving with no room takes its connection opaque; see
// startRequestLocked for why the connection cannot simply skip it.
MaxInFlightRequests int
// MaxBufferedBytes caps the bytes held over across socket events, across all
// connections, for units that arrived split. It exists because the per-buffer
// cap alone does not bound the agent: MaxConnections connections each holding
// two halves of one legitimately-split frame is a ceiling in the gigabytes, on
// a DaemonSet that shares a node with the workloads it measures.
//
// A connection that cannot get room takes itself opaque rather than resuming
// from the wrong offset, so exhausting this degrades coverage into conn_count
// instead of degrading the node.
MaxBufferedBytes int
}
Config tunes the aggregator's memory ceilings and how long it waits for an answer. None of them changes what is measured, only how much of it the agent is willing to hold and how long it holds it.
type ContainerKey ¶
ContainerKey identifies the container a measurement belongs to, in the only terms this agent can see. The DevZero container, pod and node UUIDs are resolved server side from exactly these fields, by the same lookup the existing bucketed container metrics path uses -- which is what makes the two tables agree about which container a measurement belongs to.
type PodLookup ¶
PodLookup resolves a local pod by its IP.
It is declared here rather than imported because the concrete cache lives in the parent package, which already depends on this one. The one method is satisfied structurally, so no wiring is needed at the call site.
type SocketEvent ¶
type SocketEvent struct {
// SocketCookie identifies the socket the bytes crossed. It is opaque and only
// unique among sockets alive at the same instant: the kernel recycles the
// structure it derives from, so it must be paired with the connection's
// four-tuple before two observations can be called the same connection.
SocketCookie uint64
// Path is the syscall the bytes crossed on.
Path SyscallPath
// LocalAddr and LocalPort are this node's side of the connection. For an
// inbound service request these are the served pod and the port it listens on,
// which is what attribution keys off.
LocalAddr netip.Addr
LocalPort uint16
// RemoteAddr and RemotePort are the peer's side of the connection. They are
// recorded but never aggregated by: keying on the caller is exactly what would
// make the in-memory map grow with caller cardinality.
RemoteAddr netip.Addr
RemotePort uint16
// TrueLen is the length the syscall actually carried, whatever prefix of it the
// probe copied. The decoder walks by this rather than by len(Payload), which is
// what lets a body larger than the copy limit be stepped over without losing the
// stream position.
TrueLen uint32
// Payload is the prefix of those bytes the probe copied, at most PayloadCap of
// them and often fewer. It is a copy rather than a view of the ring buffer
// sample, which the kernel reuses the moment the record is released.
//
// It may be shorter than TrueLen and may be empty. A decoder must treat a
// boundary that falls beyond it as unreadable rather than as absent.
Payload []byte
// Timestamp is when the kernel saw the call, already mapped onto wall clock.
Timestamp time.Time
}
SocketEvent is one observation of bytes crossing a TCP socket, read off the socket structure rather than parsed out of packet headers. Reading the socket rather than the wire is what removes both header parsing and NAT ambiguity: the addresses are the ones the kernel itself associates with the connection.
type SyscallPath ¶
type SyscallPath uint8
SyscallPath says which socket call the event came from. Direction falls out of the two that carry bytes: the first bytes seen on a connection arriving on the receive path mean this node is answering a request, and on the send path mean it is making one.
const ( // SyscallSend is tcp_sendmsg: bytes leaving this node. SyscallSend SyscallPath = iota // SyscallRecv is tcp_recvmsg: bytes arriving at this node. SyscallRecv // SyscallClose is tcp_close: the socket is going away. It carries no bytes -- // TrueLen is zero and Payload is nil -- and says only that this connection will // never be seen again, which is what lets its state be released at the moment it // stops being worth anything instead of when an idle timer runs out. SyscallClose )
The socket calls the probe attaches to.
func (SyscallPath) String ¶
func (p SyscallPath) String() string
String renders the syscall path for logs.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
Package socketprobe attaches the socket probes that feed container RED metric aggregation and turns their ring buffer records into redmetrics.SocketEvent values.
|
Package socketprobe attaches the socket probes that feed container RED metric aggregation and turns their ring buffer records into redmetrics.SocketEvent values. |