Documentation
¶
Overview ¶
Package xsk is an in-repo AF_XDP (XSK) socket implementation, written to replace github.com/slavc/xdp v0.3.4. It exists because the upstream binding has structural correctness defects that cannot be patched without a redesign, and because icx wants shared-UMEM zero-copy forwarding, which the upstream UMEM-per-socket model does not support.
Why a rewrite (vs forking slavc/xdp) ¶
icx uses only a thin slice of the upstream API: the Socket datapath (GetDescs/GetFrame/Fill/Receive/Transmit/Complete/Num*) from the forwarder, and xdp.Program purely as a struct of {*ebpf.Program, qidconf map, xsks map} with Attach/Detach/Register glue (filter.go builds it by hand from a cilium collection). The upstream NewProgram eBPF-asm program is unused. The defects that matter are all structural:
- ring producer/consumer indices accessed via plain (non-atomic) *uint32 dereferences of kernel-shared mmap memory, with every memory fence commented out -> works by luck on x86-64 TSO, data corruption on ARM64, and a compiler hoist/cache hazard on every architecture;
- UMEM owned per-socket + an RX/TX "half partition" freelist, which forces a full per-frame copy between the phy and virt UMEMs and blocks zero-copy;
- Transmit panics the process on an unexpected sendto errno;
- Close never munmaps the four ring mappings (leak);
- GetFrame returns a slice whose cap runs to the end of the whole UMEM.
Fixing the memory ordering and the UMEM model is a redesign of the exact pieces a fork would carry, so we rewrite.
Memory-ordering contract (the whole point) ¶
Each ring is single-producer/single-consumer with the KERNEL as the counterparty on the other end, running on a different CPU. The Go race detector cannot observe the kernel, so these accesses MUST be explicitly ordered. This package follows the libbpf xsk.h discipline exactly, using sync/atomic (which on Go gives acquire/release and also defeats compiler caching/hoisting, unlike re-enabling an arch-specific asm fence):
- producer reserve: load-ACQUIRE the consumer index to compute free space;
- producer submit: write descriptors, THEN store-RELEASE the producer index;
- consumer peek: load-ACQUIRE the producer index, THEN read descriptors;
- consumer release: read descriptors, THEN store-RELEASE the consumer index.
See ring.go; this is the foundation everything else is built on.
Zero-copy model (shared UMEM) ¶
The forwarder splices frames between a physical and a virtual interface. With slavc/xdp each socket had its own UMEM, so every forwarded frame was copied between the two. Here a single UMEM (NewUMEM) is shared by both sockets (NewSocket(umem, ...)): a frame received on one socket's RX ring is placed directly onto the other socket's TX ring by handing over its descriptor — same UMEM addr, no copy — and reclaimed via that socket's COMPLETION ring back into the one shared free-frame pool. An in-place transform (e.g. Geneve decap) rewrites the frame within its own chunk and adjusts the descriptor.
Cross-netdev XDP_SHARED_UMEM was validated empirically against the kernel: the UMEM is registered on an fd (regFD); the first socket reuses regFD and binds normally, and each later socket has its OWN FILL/COMPLETION/RX/TX rings and binds XDP_SHARED_UMEM + sxdp_shared_umem_fd = regFD, even on a different netdev/queue. (A dedicated never-bound UMEM fd does NOT work — the kernel requires the referenced fd to be already bound, returning EBADF otherwise.) So the rings live on Socket and the UMEM is shared memory + allocator + regFD.
Note: this gives USERSPACE zero-copy (no memcpy between two UMEMs). The driver-level XDP_ZEROCOPY flag (NIC DMA straight to/from UMEM, no kernel copy) is independent, needs driver support, and is opt-in via Options.ZeroCopy; on veth (tests) it is unavailable and the kernel uses copy mode, which still benefits from the shared UMEM.
Status ¶
Linux-only. The ring atomics (ring.go), the allocator (umem.go), and the setup/bind wiring (setup_linux.go) are validated on a real aarch64 kernel by afxdp_linux_test.go (run via `dagger call test`): NewUMEM/NewSocket, the shared cross-device bind, Fill, and Transmit->Complete all pass. The RX path (an XDP redirect program — xsks_map — steering packets into a socket) is wired via the filter package and exercised end-to-end by the forwarder over veth (forwarder RX-headroom and crypto round-trip tests), which now runs entirely on this package.
Index ¶
- Constants
- Variables
- type Desc
- type Options
- type Socket
- func (s *Socket) Close() error
- func (s *Socket) Complete(n int) int
- func (s *Socket) FD() int
- func (s *Socket) Fill(n int) int
- func (s *Socket) Kick() error
- func (s *Socket) NeedsRxWakeup() bool
- func (s *Socket) NumCompleted() int
- func (s *Socket) NumFilled() int
- func (s *Socket) NumFreeFillSlots() int
- func (s *Socket) NumFreeTxSlots() int
- func (s *Socket) NumReceived() int
- func (s *Socket) NumTransmitted() int
- func (s *Socket) Receive(dst []Desc, n int) []Desc
- func (s *Socket) ReleaseRX(descs []Desc)
- func (s *Socket) Transmit(descs []Desc) (int, error)
- func (s *Socket) UMEM() *UMEM
- func (s *Socket) XStats() unix.XDPStatistics
- type Stats
- type UMEM
Constants ¶
const DefaultBusyPollBudget = 64
DefaultBusyPollBudget is the SO_BUSY_POLL_BUDGET used when BusyPoll is enabled but BusyPollBudget is left zero. It mirrors NAPI_POLL_WEIGHT (the per-poll packet budget the kernel uses for ordinary softirq NAPI) and the value in the kernel's Documentation/networking/af_xdp.rst busy-poll example.
Variables ¶
var DefaultOptions = Options{
NumFrames: 8192,
FrameSize: 2048,
FillRingNumDescs: 4096,
CompletionRingNumDescs: 4096,
RxRingNumDescs: 4096,
TxRingNumDescs: 4096,
}
DefaultOptions mirror sane high-throughput defaults (icx currently uses NumFrames=8192, FrameSize=2048, all rings=4096).
Functions ¶
This section is empty.
Types ¶
type Desc ¶
Desc is an XDP Rx/Tx descriptor: {Addr uint64; Len uint32; Options uint32}. It is an alias for the kernel ABI struct so callers can set .Len/.Addr directly and slices of Desc map straight onto the ring memory.
type Options ¶
type Options struct {
// NumFrames is the number of frames in the UMEM (frame pool size).
NumFrames int
// FrameSize is the size of each frame in bytes; must be a power of two
// (chunk size). 2048 or 4096 are typical.
FrameSize int
FillRingNumDescs int
CompletionRingNumDescs int
RxRingNumDescs int
TxRingNumDescs int
// UseNeedWakeup binds with XDP_USE_NEED_WAKEUP so the kernel can tell us
// when a poll()/sendto() wakeup is actually required, eliding syscalls.
UseNeedWakeup bool
// ZeroCopy requests XDP_ZEROCOPY at bind. If the driver does not support
// it, NewSocket fails rather than silently falling back; callers that want
// a fallback should retry with ZeroCopy=false (XDP_COPY).
ZeroCopy bool
// ForceCopy requests XDP_COPY at bind, pinning the socket to copy mode even
// on a zero-copy-capable driver. The shared-UMEM datapath needs the FIRST
// (phy) socket pinned to copy so the kernel does not ZC-DMA-map the UMEM to
// one netdev, which would make the second (shared) bind fail EOPNOTSUPP. Set
// it on the phy socket only — XDP_COPY on the shared bind itself is EINVAL.
ForceCopy bool
// BusyPoll, when > 0, enables socket busy polling on the socket (Linux >=
// 5.11): a poll()/recvmsg() on this fd drives the bound netdev's NAPI inline
// on the calling core, instead of waiting for the NIC IRQ's RX softirq to run
// on its own (IRQ-affined) core. It is the AF_XDP analogue of a DPDK poll-mode
// driver and removes the IRQ-core contention a pinned datapath thread
// otherwise hits (APO-670). The value is the SO_BUSY_POLL timeout in
// microseconds (a value around 20 is typical). The per-netdev
// napi_defer_hard_irqs/gro_flush_timeout knobs that make the IRQ deferral
// actually engage are set out of band (the forwarder does it); without them
// busy poll still runs but the hard IRQ keeps firing in parallel.
BusyPoll int
// BusyPollBudget caps how many packets one busy-poll NAPI pass processes
// (SO_BUSY_POLL_BUDGET). Zero uses DefaultBusyPollBudget when BusyPoll > 0,
// and is ignored entirely when BusyPoll == 0.
BusyPollBudget int
}
Options configure a UMEM and its sockets. All ring sizes must be powers of two; NumFrames must be >= the sum of frames that can be in flight across the FILL/RX/TX/COMPLETION rings.
type Socket ¶
type Socket struct {
// contains filtered or unexported fields
}
Socket is an AF_XDP socket bound to one (ifindex, queueID), backed by a shared UMEM. It owns all four rings — FILL, COMPLETION, RX, TX — because under XDP_SHARED_UMEM across different netdevs/queues each socket needs its own (see UMEM doc). The frame memory and the free-frame allocator live in the shared UMEM; the rings here are private to this socket.
A Socket is single-threaded: one goroutine drives Fill/Receive/Transmit/ Complete for it, matching the SPSC ring contract (the kernel is the other party on each ring). The forwarder pins one goroutine per queue (runtime.LockOSThread). The four rings are therefore lock-free; only the shared UMEM free-frame pool is mutex-guarded, since sibling sockets draw from it too.
func NewSocket ¶
NewSocket binds a socket on (ifindex, queueID) sharing umem's frame pool and returns it. The returned Socket does not own the UMEM; Close it before umem.Close().
Shared-UMEM model (libbpf): the FIRST socket on a UMEM reuses the UMEM registration fd and binds NORMALLY; every later socket gets its own fd and binds XDP_SHARED_UMEM referencing the registration fd. The kernel rejects a shared bind whose referenced fd is not itself already bound (EBADF), so the registration fd must become a real bound socket — the first one. Each socket (first or later) has its OWN FILL/COMPLETION/RX/TX rings, which is required when sharing a UMEM across different netdevs/queues (the forwarder's case).
For zero-copy forwarding, create one UMEM and bind both the phy and virt sockets to it: a frame received on one can be transmitted on the other with no copy.
func (*Socket) Close ¶
Close munmaps this socket's four rings and closes its fd — unless this is the first socket on the UMEM, which reuses the UMEM registration fd (UMEM.Close owns and closes that). It does NOT close the shared UMEM; the caller owns that and must Close it after all its sockets are closed. Errors are joined.
func (*Socket) Complete ¶
Complete reclaims up to n frames the kernel finished transmitting on this socket's COMPLETION ring back into the shared free pool, returning how many were reclaimed.
func (*Socket) FD ¶
FD returns the socket file descriptor (for poll and registration in an xsks_map so the XDP program can redirect RX traffic to this socket).
func (*Socket) Fill ¶
Fill hands up to n free frames to the kernel on this socket's FILL ring so it can receive into them, returning how many were queued (bounded by free FILL-ring slots AND free frames in the shared pool). submit publishes exactly the count written, so a short pool fills fewer slots and never exposes an unwritten slot to the kernel.
func (*Socket) Kick ¶
Kick wakes the kernel to process this socket's TX ring WITHOUT queueing any new descriptors — the copy-mode TX drain. In copy (generic) mode the kernel pulls at most TX_BATCH_SIZE (32) descriptors off the TX ring per sendto and never pulls on its own, so after a Transmit larger than 32 the tail of the batch is still queued; the producer goroutine MUST keep kicking until the ring drains or those descriptors (and the UMEM frames they reference) are stranded — the pool bleeds out and the datapath wedges after the first burst (APO-801). Returns the same errno class as the implicit kick inside Transmit. When the socket is bound with NEED_WAKEUP and the kernel is already draining, it elides to a no-op.
func (*Socket) NeedsRxWakeup ¶
NeedsRxWakeup reports whether the kernel wants a poll() wakeup to make RX/FILL progress (XDP_USE_NEED_WAKEUP). When false, the forwarder can skip the poll.
func (*Socket) NumCompleted ¶
NumCompleted reports how many frames are waiting on the COMPLETION ring to be reclaimed via Complete.
func (*Socket) NumFilled ¶
NumFilled reports how many frames are queued on the FILL ring that the kernel has not yet consumed (i.e. outstanding RX buffers handed to the kernel). It is the FILL-ring analogue of NumTransmitted: derived from the ring indices (size - free), so it cannot drift the way slavc/xdp's hand-maintained numFilled counter did. The forwarder uses it to decide how many more frames to hand the kernel via Fill and whether to arm POLLIN.
func (*Socket) NumFreeFillSlots ¶
NumFreeFillSlots reports how many FILL-ring slots are free to produce into.
func (*Socket) NumFreeTxSlots ¶
NumFreeTxSlots reports how many TX-ring slots are free to produce into.
func (*Socket) NumReceived ¶
NumReceived reports how many RX descriptors are available to consume.
func (*Socket) NumTransmitted ¶
NumTransmitted reports how many descriptors are queued on the TX ring that the kernel has not yet consumed (in-flight), for POLLOUT gating. Derived from the ring indices, so it cannot drift the way slavc/xdp's hand-maintained counter did.
func (*Socket) Receive ¶
Receive consumes up to n frames the kernel produced on the RX ring, appending the descriptors to dst and returning it. Returns into a caller-owned slice (no shared scratch buffer aliasing footgun). The frames referenced by the returned descriptors are owned by the caller until either handed to a TX ring (Transmit) or released back to the shared pool (ReleaseRX).
func (*Socket) ReleaseRX ¶
ReleaseRX returns RX frames to the shared free pool without transmitting them (the drop path), so they re-enter circulation instead of leaking. Thin alias for UMEM.Free, kept on Socket for symmetry with Receive.
func (*Socket) Transmit ¶
Transmit queues descs on the TX ring and kicks the kernel if needed. It returns the number actually queued (bounded by free TX-ring slots) and an error. Unlike slavc/xdp it NEVER panics: an unexpected sendto errno is returned so the caller can treat link-down conditions as a graceful shutdown.
The descriptors' frames must be owned by the caller (freshly received or Alloc'd). The queued descs[:n] are handed to the kernel and reclaimed later via Complete. IMPORTANT: when n < len(descs) (TX ring full), the tail descs[n:] was NOT queued and its frames are still owned by the caller — the caller MUST UMEM.Free(descs[n:]) (or retry it) or those frames leak. This is the explicit ownership rule that replaces slavc/xdp's silently-dropped tail.
func (*Socket) XStats ¶
func (s *Socket) XStats() unix.XDPStatistics
XStats returns the kernel-side XDP_STATISTICS for this socket (rx_dropped, rx_invalid_descs, tx_invalid_descs, rx_ring_full, rx_fill_ring_empty_descs, tx_ring_empty_descs) — the authoritative place to see where the kernel drops.
type Stats ¶
type Stats struct {
Filled uint64
Received uint64
Transmitted uint64
Completed uint64
KernelStats unix.XDPStatistics
}
Stats reports ring progress counters plus the kernel-side XDP statistics.
type UMEM ¶
type UMEM struct {
// contains filtered or unexported fields
}
UMEM is the shared frame memory region and the frame-ownership allocator. It deliberately does NOT hold the FILL/COMPLETION rings: under XDP_SHARED_UMEM across different netdevs/queues (the forwarder's zero-copy case), each socket needs its OWN fill/comp/rx/tx rings, all drawing from this one frame pool. (Confirmed empirically: the FIRST socket REUSES regFD and binds normally; each later socket binds XDP_SHARED_UMEM + sxdp_shared_umem_fd=regFD with its own rings. A never-bound fd as the shared ref fails with EBADF — the kernel needs the referenced fd already bound.) So the rings live on Socket; the UMEM is just memory + allocator + the registration fd that sockets reference.
Zero-copy forwarding: a frame received on one socket's RX ring is transmitted on another socket's TX ring by handing over the SAME descriptor — same UMEM addr, no copy — because both sockets share this frame pool. The frame is reclaimed via the transmitting socket's COMPLETION ring back into the pool.
Frame ownership is a single LIFO free-frame stack: a frame is "free" iff it sits on the stack. There is exactly one ownership structure (unlike slavc/xdp's two independent freeRX/freeTX arrays that could double-allocate a frame), so a frame can never be believed free by two paths at once. mu guards it because multiple socket goroutines allocate/free from the one shared pool.
func NewUMEM ¶
NewUMEM allocates the shared frame area and registers it on an AF_XDP fd (regFD), also setting that fd's FILL/COMPLETION ring sizes (libbpf order: REG, then FILL, then COMP). regFD is NOT bound here — the first NewSocket on this UMEM reuses it and binds it (see NewSocket), so its FILL/COMPLETION rings are the ones whose sizes are set here. Sockets created with NewSocket(umem, ...) share this frame pool; Close the UMEM only after all its sockets are closed.
func (*UMEM) Alloc ¶
Alloc pops up to n free frames and appends them to dst as descriptors with Addr set and Len = FrameSize, ready to fill and Transmit. It may append fewer than n (down to zero) when the pool is short; callers MUST handle a short return — this is the contract slavc/xdp callers violated by panicking on a short GetDescs. Used for the scheduled/keepalive TX and any non-zero-copy path; the zero-copy path instead hands an RX descriptor straight to Transmit.
func (*UMEM) Close ¶
Close munmaps the frame area and closes the registration fd. Call it only after every Socket bound to this UMEM has been Closed (the kernel refcounts the UMEM, but closing the regFD while sockets still reference it is closing out from under them). Errors are joined so one failure does not skip the rest.
func (*UMEM) Frame ¶
Frame returns the byte slice for descriptor d, bounded to a SINGLE frame, or nil if d.Addr is outside the UMEM. The three-index slice caps capacity at the frame boundary so a handler that re-slices to cap cannot scribble into the adjacent frame (the slavc/xdp GetFrame footgun, where cap ran to the end of the whole UMEM). Both the lower bound (Addr in range) and the upper bound (len clamped to the frame boundary) are checked, so a malformed/foreign descriptor — a bad Addr, an oversized Len, or an in-frame headroom offset — returns nil or a clamped frame rather than panicking the datapath. This matters on the RX / zero-copy-handoff path, where d.Addr is not necessarily one this process produced.
func (*UMEM) FrameFull ¶
FrameFull returns the full frame chunk for the frame containing d.Addr (len == cap == FrameSize) for producers that need to write a fresh frame, or nil if d.Addr is outside the UMEM.
func (*UMEM) Free ¶
Free returns the frames referenced by descs to the pool. Use it for RX frames the handler dropped and for TX frames Alloc'd but not transmitted, so they re-enter circulation instead of leaking. Each frame must be Free'd exactly once (or reclaimed via a socket's Complete for TX'd frames), never both.
func (*UMEM) NumFreeFrames ¶
NumFreeFrames reports how many frames are currently allocatable.
Source Files