Documentation
¶
Overview ¶
Package denylog reads what a sandbox's firewall refused and reports it per run (Task 20437).
A sandbox's IP firewall (pkg/netfilter) used to end in a silent drop, so a blocked connection looked like an outage: the workload waited out a TCP timeout, the harness retried for minutes, and nothing said which address had been refused by which rule. The ruleset now refuses instead, and every deny path logs — rate-limited, to an NFLOG group — under a prefix naming the rule set's fingerprint and the rule. This package is the other end of that log:
- nflog_linux.go binds the group over a netlink socket in this process. It needs CAP_NET_ADMIN and AF_NETLINK, which is exactly what installing the ruleset needs (the agent's --packet-filter grant), and no child process: the capability never leaves the process that holds it (pkg/caps).
- parse.go and headers.go read the kernel's messages: the prefix, the input interface, and the IPv4/IPv6 and TCP/UDP headers of the packet. Addresses and ports are all that survive; the rule's snaplen means the kernel copies little more, and nothing past the ports is read.
- monitor.go attributes each packet to its run by the sandbox's bridge and source address, and aggregate.go counts it, per run, by destination, port, protocol and rule — capped, so a hostile workload cannot make the process do unbounded work.
What becomes of a batch is the process's business, through its Sink: an executor agent sends it to the hub over its session, and the hub's own container executor hands it to the hub directly.
Index ¶
- Constants
- Variables
- func ParseMessages(buf []byte, fn func(Packet)) error
- func ParsePrefix(s string) (fingerprint, rule string, ok bool)
- func RuleKind(rule string) string
- func RulePrefix(rule string) netip.Prefix
- func Running() (bool, string)
- func SetSink(s Sink) (restore func())
- func ValidRule(s string) bool
- type Batch
- type Entry
- type Header
- type Monitor
- func (m *Monitor) Close() error
- func (m *Monitor) FlushFinal(ctx context.Context, runID string) error
- func (m *Monitor) Group() uint16
- func (m *Monitor) Release(runID string)
- func (m *Monitor) Stats() Stats
- func (m *Monitor) Track(run Run, fingerprint string, ifindex uint32, addrs []netip.Addr) error
- func (m *Monitor) Tracked(runID string) bool
- type Packet
- type Proto
- type Run
- type Sink
- type Stats
Constants ¶
const ( // MaxEntriesPerRun bounds the distinct (destination, port, protocol, // rule) a run's aggregate itemises. Past it, the entry seen least — the // smallest count, then the longest ago — makes room, and what it had not // yet reported is counted as unitemised. MaxEntriesPerRun = 64 // MaxRuns bounds the runs one process tracks at once: far more sandboxes // than a host runs, and a ceiling on the memory a confused caller could // make the monitor hold. MaxRuns = 4096 )
Caps on what one process holds. Every one of them is reached only by a workload doing something no developer's task does — spraying addresses, ports or protocols at its firewall — and each turns that into a count.
const (
// EnvBaseGroup overrides the first group tried.
EnvBaseGroup = "CLOOP_DENIAL_LOG_GROUP"
)
Group selection. A group is one socket's: a second process binding the same number gets EBUSY. So the base is configurable for a host that already uses NFLOG, and each process takes the first free group of a small range — an agent and a hub, or two agents, on one host each get their own.
Variables ¶
var ErrNoSink = errors.New("denylog: no sink is installed in this process")
ErrNoSink reports that this process has nowhere to send refusals, so there is nothing to read the group for.
var ( // FlushInterval is how often batches are reported while a run lives: // soon enough that a developer watching the run's history sees a refusal // while it still explains what the harness is doing. FlushInterval = 2 * time.Second )
Timing. Variables so tests can shorten them.
var ProtoNames = []string{"tcp", "udp", "icmp", "icmpv6", "sctp", "udplite", "other"}
ProtoNames lists every value Entry.Proto may take.
Functions ¶
func ParseMessages ¶
ParseMessages calls fn for every NFLOG packet message in buf and skips every other message. It stops at the first malformed one and reports it: what came before it was delivered, and nothing after it can be trusted to be aligned.
func ParsePrefix ¶
ParsePrefix splits a log prefix "cloop:<fingerprint>:<tag>".
func RulePrefix ¶
RulePrefix is the range a tag names, or the zero prefix for "default".
func Running ¶
Running reports whether this process reads a group, and if not, why — the sentence an operator is shown in place of a list of refusals.
Types ¶
type Batch ¶
type Batch struct {
// Fingerprint is the rule set the run's sandbox was filtered by, as the
// prefixes named it.
Fingerprint string `json:"fingerprint,omitempty"`
// Entries are the itemised refusals since the last report.
Entries []Entry `json:"entries,omitempty"`
// Unitemised counts refusals since the last report that no entry holds:
// those of entries that made room for others before they were reported.
Unitemised uint64 `json:"unitemised,omitempty"`
// Final says the run is over and nothing more will be reported for it.
Final bool `json:"final,omitempty"`
}
Batch is what a flush reports for one run.
type Entry ¶
type Entry struct {
// Dst is the address the sandbox tried to reach.
Dst netip.Addr `json:"dst"`
// Port is its destination port; 0 for a protocol without ports, or a
// packet whose transport header did not fit in what the kernel copied.
Port uint16 `json:"port,omitempty"`
// Proto is one of ProtoNames.
Proto string `json:"proto"`
// Rule is the deny path's tag (pkg/netfilter: "default", "deny:<prefix>",
// "meta:<prefix>", "block:<prefix>", "xlat:<prefix>").
Rule string `json:"rule"`
// Count is how many packets this report adds — a delta, so a batch that
// arrives twice cannot be mistaken for one that arrived once, and a
// report that never arrives loses only its own share.
Count uint64 `json:"count"`
// First and Last bound when they were seen.
First time.Time `json:"first"`
Last time.Time `json:"last"`
}
Entry is one aggregated refusal: every packet a run sent to one destination and port, over one protocol, that one rule denied.
type Header ¶
Header is what a refusal records of one packet.
func ParseHeader ¶
ParseHeader reads an IPv4 or IPv6 packet's header, and the destination port of a TCP, UDP, SCTP or UDP-Lite packet behind it.
type Monitor ¶
type Monitor struct {
// contains filtered or unexported fields
}
Monitor reads one NFLOG group and reports refusals per run.
func Shared ¶
Shared returns this process's monitor, binding the NFLOG group on first use. It fails with ErrNoSink in a process that installed no sink, and with the reason the group could not be bound — no CAP_NET_ADMIN, no AF_NETLINK, no free group — otherwise; a failure is remembered for a minute.
func (*Monitor) FlushFinal ¶
FlushFinal reports what is left of run with Final set and forgets it. It waits — at most settle — for log lines the kernel may still hold, so it is called once the sandbox has exited and before the run's end is reported. A run the monitor does not know is nothing to report. An error means the report did not arrive: the run is kept, and a later FlushFinal tries again.
func (*Monitor) Release ¶
Release marks run's sandbox gone. Its addresses stay attributed for linger, for the log lines still queued in the kernel; its refusals are reported by FlushFinal, or by the monitor itself once forgetAfter has passed.
func (*Monitor) Track ¶
Track attributes packets from addrs on the interface ifindex to run, whose sandbox is filtered by the rule set fingerprint. A later Track of the same address takes it over: an engine reuses a departed container's address, and the newest claim is the live one.
With no addresses the run claims the bridge instead: a sandbox that exited before the driver could read its address back has none any more, and its refusals — already in the kernel's queue, or held as unclaimed — are its own as long as it is the only run on that bridge. With another run there they are nobody's provably, and stay unattributed.
type Packet ¶
type Packet struct {
// Group is the NFLOG group it was logged to.
Group uint16
// Family is the nfgenmsg family: AF_INET (2) or AF_INET6 (10).
Family uint8
// Hook is the netfilter hook it was logged in.
Hook uint8
// InDev and OutDev are interface indexes, 0 when absent.
InDev, OutDev uint32
// Prefix is the log statement's prefix.
Prefix string
// Time is the kernel's timestamp for the packet, zero when it sent none.
Time time.Time
// Payload is the packet from its network header, as far as the copy
// went. It aliases the receive buffer and is valid only during the
// callback.
Payload []byte
}
Packet is one logged packet, as the kernel reported it.
type Proto ¶
type Proto uint8
Proto names an IP protocol the way an entry carries it.
const ( ProtoICMP Proto = 1 ProtoTCP Proto = 6 ProtoUDP Proto = 17 ProtoICMPv6 Proto = 58 ProtoSCTP Proto = 132 ProtoUDPLite Proto = 136 )
The protocols an entry names. Anything else is "other": the number of an exotic protocol is not worth an unbounded label.
type Run ¶
type Run struct {
// ID is the handle of the workload, in the executor that started it.
ID string
// Executor is that executor's ID.
Executor string
}
Run identifies one sandbox's refusals to a Sink.
type Sink ¶
Sink receives a run's batch. An error means it was not delivered; the batch is kept and reported again.
type Stats ¶
type Stats struct {
// Packets is every logged packet read from the group.
Packets uint64
// Foreign is packets under a prefix this package did not write.
Foreign uint64
// Unattributed is packets no run claimed within pendingTTL.
Unattributed uint64
// Malformed is packets whose header could not be read.
Malformed uint64
// Overruns is how often the socket's buffer overflowed and the kernel
// discarded log lines.
Overruns uint64
}
Stats are a monitor's counters, for diagnostics.
Source Files