Documentation
¶
Index ¶
- func MTU(pathMTU int) int
- type Clock
- type Handler
- func (h *Handler) AddVirtualNetwork(vni uint, remoteAddr *tcpip.FullAddress, allowedRoutes []Route) error
- func (h *Handler) GetVirtualNetwork(vni uint) (*VirtualNetwork, bool)
- func (h *Handler) IsLayer3() bool
- func (h *Handler) ListVirtualNetworks() []*VirtualNetwork
- func (h *Handler) PhyToVirt(phyFrame, virtFrame []byte) int
- func (h *Handler) PhyToVirtInPlace(buf []byte, off, length int) (int, int)
- func (h *Handler) RemoveVirtualNetwork(vni uint) error
- func (h *Handler) ToPhy(phyFrame []byte) int
- func (h *Handler) ToPhyInPlace(buf []byte, off int) (int, int)
- func (h *Handler) UpdateVirtualNetworkRoutes(vni uint, allowedRoutes []Route) error
- func (h *Handler) UpdateVirtualNetworkSecret(vni uint, master [32]byte, rxSPI, txSPI uint32, expiresAt time.Time) error
- func (h *Handler) VirtToPhy(virtFrame, phyFrame []byte) (int, bool)
- func (h *Handler) VirtToPhyInPlace(buf []byte, off, length int) (int, int, bool)
- type HandlerOption
- func WithClock(c Clock) HandlerOption
- func WithKeepAliveInterval(interval time.Duration) HandlerOption
- func WithLayer3VirtFrames() HandlerOption
- func WithLocalAddr(a *tcpip.FullAddress) HandlerOption
- func WithOuterSrcValidation() HandlerOption
- func WithOuterUDPChecksum() HandlerOption
- func WithRXRateLimit(pps int) HandlerOption
- func WithSourceLearning() HandlerOption
- func WithSourceMAC(mac tcpip.LinkAddress) HandlerOption
- func WithSourcePortHashing() HandlerOption
- func WithVirtMAC(mac tcpip.LinkAddress) HandlerOption
- type Route
- type Statistics
- type VirtualNetwork
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type Handler ¶
type Handler struct {
// contains filtered or unexported fields
}
Handler processes encapsulated Geneve traffic for one or more virtual networks. It performs encryption/decryption, replay protection, address validation, and translation between physical and virtual frame formats.
func NewHandler ¶
func NewHandler(opts ...HandlerOption) (*Handler, error)
NewHandler returns a new Handler configured with the given options. It validates required parameters and allocates internal state for managing virtual networks and packet processing.
func (*Handler) AddVirtualNetwork ¶
func (h *Handler) AddVirtualNetwork(vni uint, remoteAddr *tcpip.FullAddress, allowedRoutes []Route) error
AddVirtualNetwork adds a new network with the given VNI and remote address. Under WithSourceLearning remoteAddr may be nil: keys can be installed before any endpoint is known, TX fails closed, and the first authenticated inbound packet populates the remote (APO-740).
func (*Handler) GetVirtualNetwork ¶ added in v0.8.0
func (h *Handler) GetVirtualNetwork(vni uint) (*VirtualNetwork, bool)
GetVirtualNetwork retrieves a virtual network by its VNI.
func (*Handler) IsLayer3 ¶ added in v0.18.0
IsLayer3 reports whether the handler emits and expects raw L3 (IP) virtual frames (WithLayer3VirtFrames), as opposed to the default L2 (Ethernet) frames.
The AF_XDP forwarder uses this to reject an L3 handler at construction: its virtual interface is an L2 device (a veth), so an in-place L3 decap would write a raw IP packet onto the veth — which the veth silently drops. An L3 peer (e.g. the userspace vtep/tun datapath, which drives a real TUN device) interoperates with the forwarder on the wire regardless, because both modes encrypt the same inner IP packet; only the local virtual framing differs.
func (*Handler) ListVirtualNetworks ¶ added in v0.8.0
func (h *Handler) ListVirtualNetworks() []*VirtualNetwork
ListVirtualNetworks returns a snapshot of all configured virtual networks.
func (*Handler) PhyToVirt ¶
PhyToVirt converts a physical frame to a virtual frame typically by performing decapsulation. Returns the length of the resulting virtual frame.
func (*Handler) PhyToVirtInPlace ¶ added in v0.18.0
PhyToVirtInPlace performs decapsulation IN PLACE within buf.
On entry, buf[off:off+length] holds the physical frame ([outer Eth/IP/UDP][Geneve][ciphertext+tag]). On success it returns the (offset, length) window within buf where the resulting virtual frame lives:
- layer3: the raw inner IP packet (plaintext written over the ciphertext).
- layer2: a freshly written 14-byte Ethernet header immediately followed by the inner IP packet, placed in the headroom that previously held the consumed outer/Geneve headers.
On any drop (bad UDP, bad Geneve, unknown VNI, no nonce, no/expired key, replay, decrypt failure, out-of-band, invalid src) it returns a zero-length window (length 0), matching every early-return in PhyToVirt. The crypto and validation logic is identical to PhyToVirt; only dst aliasing differs.
func (*Handler) RemoveVirtualNetwork ¶
RemoveVirtualNetwork removes a network by its VNI.
func (*Handler) ToPhy ¶ added in v0.12.0
ToPhy is called periodically to allow the handler to send scheduled frames to the physical interface, e.g. keep-alive packets. Returns the length of the resulting physical frame.
func (*Handler) ToPhyInPlace ¶ added in v0.18.0
ToPhyInPlace performs the keep-alive encapsulation IN PLACE within buf.
Unlike VirtToPhyInPlace there is no inner payload: the AEAD is computed over an empty plaintext, so the ciphertext is just the 16-byte tag. The caller provides a buffer with at least PayloadOffset + 32 (Geneve) + 16 (tag) bytes available starting at off. The keep-alive frame is built starting at off and the returned window is (off, frameLen).
It returns a zero length when there is nothing due to send or on any error, matching ToPhy. This is the in-place analogue of ToPhy.
func (*Handler) UpdateVirtualNetworkRoutes ¶ added in v0.15.0
UpdateVirtualNetworkRoutes updates the allowed routes for a virtual network.
func (*Handler) UpdateVirtualNetworkSecret ¶ added in v0.18.0
func (h *Handler) UpdateVirtualNetworkSecret(vni uint, master [32]byte, rxSPI, txSPI uint32, expiresAt time.Time) error
UpdateVirtualNetworkSecret installs/rotates a virtual network's pair of simplex security associations (PSP model). The handler derives each direction's AES-GCM-128 key itself — data_key = KDF(master, SPI) via the PSP SP 800-108/AES-CMAC KDF (psp.DeriveSAKey) — so callers hand it a 32-byte master secret and two SPIs, never raw AEAD keys. Because the SPI is a KDF input, advancing it under an unchanged master yields a cryptographically fresh key, which is what makes the nonce-uniqueness invariant — no (key, nonce=SPI‖counter) pair ever repeats — structural rather than a caller obligation.
rxSPI and txSPI are the per-direction 32-bit SPIs that select the receive and transmit SAs. Each is carried in the Geneve key-epoch option and bound into the high 4 bytes of its direction's AES-GCM nonce (nonce = SPI‖counter):
- rxSPI is OUR receive SPI — the one we allocated and the peer encrypts to. We store the RX cipher under it and look inbound frames up by it. Inbound frames carry rxSPI in their key-epoch option (== the sender's txSPI).
- txSPI is the PEER's receive SPI — the one we encrypt to. We stamp it into the key-epoch option and nonce[:4] of every outbound frame.
The two SPIs must be distinct: equal SPIs under one master derive equal keys for the two directions. The QUIC control plane partitions the SPI space by role (psp.MakeSPI); symmetric-epoch callers get mirrored role-partitioned pairs from psp.EpochSPIs.
Nonce-uniqueness rests on two independent freshness sources, either of which suffices per install:
- a FRESH master secret (the control plane: every QUIC session is a fresh ECDHE handshake — no 0-RTT, no resumption, enforced in control/transport.go; the relay path: a fresh random master per connection). Any SPI is then safe, even one that collides with or regresses below the live SA's — its from-zero counter pairs with a key that has never sealed a packet. This is what makes a one-sided restart recover seamlessly with no persisted state.
- a strictly ADVANCED txSPI under a reused master (rotation within a connection): a new KDF context is a fresh key.
The guards fail closed on everything else: reserved SPIs (low 31 bits zero, per the PSP spec and control/sa.go), rxSPI == txSPI, and a TX nonce-safety check that rejects a txSPI at or below the live SA's when the master is UNCHANGED (same SHA-256 fingerprint) — the only in-process action that would reset a live counter into an already-used (key, nonce) space. There is deliberately no RX monotonicity guard — the receive side never emits a nonce, so a reused receive SPI is harmless (its per-SA replay filter is rebuilt with the fresh key); rxEpoch is tracked only to grace-clamp the previous receive cipher (see installKeys).
Callers must serialize installs per VNI; the guard→install sequence is not internally locked (the control plane is single-threaded per Tunnel).
func (*Handler) VirtToPhy ¶
VirtToPhy converts a virtual frame to a physical frame typically by performing encapsulation. Returns the length of the resulting physical frame.
func (*Handler) VirtToPhyInPlace ¶ added in v0.18.0
VirtToPhyInPlace performs encapsulation IN PLACE within buf.
On entry, buf[off:off+length] holds the virtual frame:
- layer3: a raw inner IP packet.
- layer2: an Ethernet frame; the 14-byte Ethernet header is stripped before encryption. ARP requests and IPv6 Neighbor Solicitations are answered locally via the proxy helpers (which write into the headroom in front of the inner IP packet) and returned with handled=true.
The buffer MUST provide headroom in front of the inner IP packet for the outer headers (PayloadOffset = 42 for an IPv4 underlay / 62 for IPv6) plus the 32-byte Geneve header, and at least 16 bytes of tailroom after the inner IP packet for the GCM tag. The forwarder's UMEM frame (FrameSize 2048) and the test corpus both satisfy this.
On success it returns the (offset, length) window of the resulting physical frame within buf and handled=false. ARP/ND immediate replies return (window, true). Drops return a zero-length window (length 0) with handled=false, matching every early-return in VirtToPhy. Crypto/validation logic is identical to VirtToPhy; only src aliasing differs.
type HandlerOption ¶ added in v0.7.0
type HandlerOption func(*handlerOptions) error
func WithClock ¶ added in v0.12.1
func WithClock(c Clock) HandlerOption
WithClock overrides the time source used by the handler (useful for tests).
func WithKeepAliveInterval ¶ added in v0.10.0
func WithKeepAliveInterval(interval time.Duration) HandlerOption
WithKeepAliveInterval configures the handler to send keep-alive packets on each virtual network at the given interval. If nil or zero, no keep-alives are sent. A value of between 10 and 30s is recommended to keep NAT mappings alive.
func WithLayer3VirtFrames ¶ added in v0.7.0
func WithLayer3VirtFrames() HandlerOption
WithLayer3VirtFrames configures the handler for L3 mode, where virtual frames are raw IP packets (no Ethernet header). Default is L2 mode (Ethernet frames).
func WithLocalAddr ¶ added in v0.7.0
func WithLocalAddr(a *tcpip.FullAddress) HandlerOption
WithLocalAddr sets the local UDP endpoint used as the source for encapsulated packets. This option is required. If multiple addresses are provided, the best one is chosen per packet based on the remote address.
If WithSourcePortHashing is enabled, the Port field of this address is overridden per packet with a hash of the inner flow. Otherwise, the Port specified here is used as-is.
func WithOuterSrcValidation ¶ added in v0.18.0
func WithOuterSrcValidation() HandlerOption
WithOuterSrcValidation makes the RX path verify that each frame's outer underlay source IP matches the receiving virtual network's configured peer (RemoteAddr) and drop it otherwise, before any per-packet crypto or replay work (APO-650). Only the IP is checked, not the UDP source port, so it is compatible with source-port hashing (which rewrites the port per packet).
This suits the single-static-peer deployment the CLI ships. It is left off by default in the library because a peer behind asymmetric routing/NAT (whose packets arrive from an address other than the one we send to) would otherwise be silently dropped; enable it when the peer's source address is stable.
func WithOuterUDPChecksum ¶ added in v0.18.0
func WithOuterUDPChecksum() HandlerOption
WithOuterUDPChecksum forces the TX path to compute the outer UDP checksum on an IPv4 underlay. By default it is skipped: the encapsulated payload is already AES-GCM-authenticated with the full Geneve header as AAD, the ICX RX path does not validate the outer UDP checksum (udp.Decode runs with skipChecksumValidation), and a zero UDP checksum is legal on IPv4 (RFC 768) — so the per-packet software checksum over the whole ciphertext is pure overhead (APO-668, ~25% of VirtToPhy CPU on an MTU-class frame). Enable this only if a middlebox on the underlay drops or mishandles zero-checksum UDP. On an IPv6 underlay a zero UDP checksum is illegal, so the checksum is always computed regardless of this option.
func WithRXRateLimit ¶ added in v0.18.0
func WithRXRateLimit(pps int) HandlerOption
WithRXRateLimit caps how many frames per second per virtual network may reach the AES-GCM Open on the RX path, bounding the CPU an off-path flood of forgeable VNI/epoch frames can burn (APO-655). pps <= 0 disables the limiter (the default), preserving the zero-overhead datapath; a positive value should be set comfortably above the tunnel's expected legitimate peak, since legitimate and attacker traffic share the per-network budget.
func WithSourceLearning ¶ added in v0.19.0
func WithSourceLearning() HandlerOption
WithSourceLearning makes the RX path learn each virtual network's remote endpoint from the outer underlay source {addr, port} of inbound packets that pass BOTH AEAD authentication and the anti-replay check (WireGuard-roaming semantics, APO-740). An unauthenticated packet can never move the endpoint.
When enabled, AddVirtualNetwork accepts a nil remote address: keys can be installed before any endpoint is known and the first authenticated packet populates it (TX fails closed until then). A non-nil remote passed to AddVirtualNetwork is merely the initial endpoint and is updated when the peer roams. Changes to an established endpoint are damped to one per remoteLearnDampInterval per network.
The outer source PORT is part of the learned endpoint (a NAT rebinding moves the port and return traffic must follow it), which constrains the peer's TX configuration: a peer whose driver puts per-flow hashed source ports on the wire (the AF_XDP forwarder under WithSourcePortHashing) would teach a learning receiver an ephemeral port that accepts no return traffic. Peers of a learning handler must transmit from their stable bound port — the tun and netstack drivers do so inherently (their UDP socket's bound port overrides the engine's hashed port), as do keep-alives on every driver.
Mutually exclusive with WithOuterSrcValidation, which pins the peer's source instead of following it; NewHandler rejects the combination.
func WithSourceMAC ¶ added in v0.7.0
func WithSourceMAC(mac tcpip.LinkAddress) HandlerOption
WithSourceMAC overrides the synthetic source MAC used for L2 frames and for ProxyARP replies. By default, a random MAC is generated at handler creation. Ignored when L3 mode is enabled.
func WithSourcePortHashing ¶ added in v0.7.0
func WithSourcePortHashing() HandlerOption
WithSourcePortHashing enables per-packet UDP source-port selection based on a hash of the inner IP flow. This improves ECMP distribution in the underlay. When enabled, it overrides the Port from WithLocalAddr for each packet.
func WithVirtMAC ¶ added in v0.7.0
func WithVirtMAC(mac tcpip.LinkAddress) HandlerOption
WithVirtMAC sets the MAC address used for the virtual interface in L2 mode. This is required when not running in L3 mode (see WithLayer3VirtFrames). Ignored when L3 mode is enabled.
type Route ¶ added in v0.16.0
type Route struct {
// Src is the source address prefix.
Src netip.Prefix
// Dst is the destination address prefix.
Dst netip.Prefix
}
Route represents a source/destination address prefix pair allowed for a virtual network.
type Statistics ¶ added in v0.12.0
type Statistics struct {
// RXPackets is the number of received packets.
RXPackets atomic.Uint64
// RXBytes is the number of bytes received.
RXBytes atomic.Uint64
// LastRXUnixNano is the timestamp of the last received packet.
LastRXUnixNano atomic.Int64
// TXPackets is the number of transmitted packets.
TXPackets atomic.Uint64
// TXBytes is the number of bytes transmitted.
TXBytes atomic.Uint64
// LastTXUnixNano is the timestamp of the last transmitted packet.
LastTXUnixNano atomic.Int64
// RXDropsNoKey is the number of received packets dropped due to a missing key.
RXDropsNoKey atomic.Uint64
// RXDropsExpiredKey is the number of received packets dropped due to an expired key.
RXDropsExpiredKey atomic.Uint64
// RXReplayDrops is the number of received packets dropped due to a potential replay attack.
RXReplayDrops atomic.Uint64
// RXDecryptErrors is the number of received packets that failed decryption.
RXDecryptErrors atomic.Uint64
// RXDropsSPIMismatch is the number of received packets dropped because the
// SPI bound into the AEAD nonce (nonce[:4]) did not match the key epoch the
// frame selected — a malformed or tampered frame (APO-644).
RXDropsSPIMismatch atomic.Uint64
// RXInvalidSrc is the number of received packets with an invalid source address.
RXInvalidSrc atomic.Uint64
// RXInvalidDst is the number of received packets dropped because the decrypted
// inner destination address fell outside every allowed route.Src prefix — the
// destination-side half of cryptokey routing that RX previously skipped (APO-649).
RXInvalidDst atomic.Uint64
// RXDropsBadPeer is the number of received packets dropped because the outer
// underlay source IP did not match the configured peer (APO-650). Only counted
// when outer-source validation is enabled (WithOuterSrcValidation).
RXDropsBadPeer atomic.Uint64
// RXRateLimitDrops is the number of received packets dropped before AES-GCM Open
// by the per-network RX rate limiter (APO-655). Only counted when a limit is
// configured (WithRXRateLimit).
RXRateLimitDrops atomic.Uint64
// TXErrors is the number of transmission errors.
TXErrors atomic.Uint64
// TXDropsExpiredKey is the number of outbound frames dropped because the transmit
// SA's key had expired (APO-656). RX enforces key expiry; this makes TX fail closed
// symmetrically instead of sealing indefinitely under a stale key.
TXDropsExpiredKey atomic.Uint64
// TXDropsNoRemote is the number of outbound frames dropped because the virtual
// network has no remote endpoint yet: under source learning (APO-740) a network
// may be installed with keys but no static remote, and TX fails closed until
// the first authenticated inbound packet populates it.
TXDropsNoRemote atomic.Uint64
// RXLearnedRemotes is the number of times the remote endpoint was learned or
// re-learned from an authenticated inbound packet (WithSourceLearning, APO-740).
RXLearnedRemotes atomic.Uint64
// RXLearnsDamped is the number of authenticated inbound packets whose differing
// outer source was NOT adopted because a remote change had already been accepted
// within the damping interval (WithSourceLearning, APO-740).
RXLearnsDamped atomic.Uint64
// LastKeepAliveUnixNano is the timestamp of the last transmitted keep-alive packet.
LastKeepAliveUnixNano atomic.Int64
// contains filtered or unexported fields
}
type VirtualNetwork ¶ added in v0.8.0
type VirtualNetwork struct {
// ID is the virtual network identifier.
ID uint
// Statistics associated with this virtual network.
Stats Statistics
// contains filtered or unexported fields
}
The state associated with each virtual network.
func (*VirtualNetwork) AllowedRoutes ¶ added in v0.15.0
func (v *VirtualNetwork) AllowedRoutes() []Route
AllowedRoutes returns a snapshot of the virtual network's allowed routes. The slice is published atomically (APO-652) and must be treated as read-only: a concurrent UpdateVirtualNetworkRoutes replaces the whole slice rather than mutating it in place, so a caller keeps a consistent (if possibly older) view.
func (*VirtualNetwork) RemoteAddr ¶ added in v0.8.0
func (v *VirtualNetwork) RemoteAddr() *tcpip.FullAddress
RemoteAddr returns the virtual network's current remote endpoint, or nil when none is configured or learned yet. The pointer is published atomically (APO-740): under source learning the RX path may replace it at any time, so the pointee must be treated as read-only — a caller keeps a consistent (if possibly older) endpoint.
Directories
¶
| Path | Synopsis |
|---|---|
|
_examples
|
|
|
keyexchange
command
Command keyexchange is a runnable demonstration of the ICX control plane: two peers establish a forward-secret, mutually-authenticated QUIC/mTLS session over loopback, derive PSP master keys from the TLS exporter, and negotiate per-direction Security Associations whose AES-GCM keys feed the Geneve/AF_XDP data plane.
|
Command keyexchange is a runnable demonstration of the ICX control plane: two peers establish a forward-secret, mutually-authenticated QUIC/mTLS session over loopback, derive PSP master keys from the TLS exporter, and negotiate per-direction Security Associations whose AES-GCM keys feed the Geneve/AF_XDP data plane. |
|
cmd
|
|
|
icxbench
command
Command icxbench runs the AF_XDP forwarder over a real NIC with a STATIC-keyed handler (bypassing the QUIC control plane) so a single box can drive the datapath under load.
|
Command icxbench runs the AF_XDP forwarder over a real NIC with a STATIC-keyed handler (bypassing the QUIC control plane) so a single box can drive the datapath under load. |
|
icxbench-tun
command
Command icxbench-tun runs the ICX engine over the USERSPACE vtep/tun datapath (a kernel /dev/net/tun device + a plain UDP underlay socket), as opposed to the zero-copy AF_XDP datapath that cmd/icxbench drives.
|
Command icxbench-tun runs the ICX engine over the USERSPACE vtep/tun datapath (a kernel /dev/net/tun device + a plain UDP underlay socket), as opposed to the zero-copy AF_XDP datapath that cmd/icxbench drives. |
|
Package control implements ICX's key-establishment control plane (a QUIC/mTLS channel) and the PSP-model key derivation that turns an authenticated, forward-secret session into per-Security-Association AEAD keys for the existing Geneve/AF_XDP data plane.
|
Package control implements ICX's key-establishment control plane (a QUIC/mTLS channel) and the PSP-model key derivation that turns an authenticated, forward-secret session into per-Security-Association AEAD keys for the existing Geneve/AF_XDP data plane. |
|
internal
|
|
|
xsk
Package xsk is an in-repo AF_XDP (XSK) socket implementation, written to replace github.com/slavc/xdp v0.3.4.
|
Package xsk is an in-repo AF_XDP (XSK) socket implementation, written to replace github.com/slavc/xdp v0.3.4. |
|
Package psp implements the PSP-model cryptographic primitives shared by the icx data plane (package icx) and the key-establishment control plane (package control): the NIST SP 800-108 / AES-CMAC key-derivation function from the PSP Architecture Specification, and the SPI bit layout that partitions the derivation space by master key and allocating role.
|
Package psp implements the PSP-model cryptographic primitives shared by the icx data plane (package icx) and the key-establishment control plane (package control): the NIST SP 800-108 / AES-CMAC key-derivation function from the PSP Architecture Specification, and the SPI bit layout that partitions the derivation space by master key and allocating role. |
|
Package replay implements an efficient anti-replay algorithm as specified in RFC 6479.
|
Package replay implements an efficient anti-replay algorithm as specified in RFC 6479. |
|
Package vtep defines the seam between the ICX engine (encap/decap + crypto, implemented by *icx.Handler) and the I/O driver that moves frames between the engine and the outside world.
|
Package vtep defines the seam between the ICX engine (encap/decap + crypto, implemented by *icx.Handler) and the I/O driver that moves frames between the engine and the outside world. |
|
afxdp
Package afxdp adapts the zero-copy AF_XDP forwarder.Forwarder to the vtep.Datapath seam.
|
Package afxdp adapts the zero-copy AF_XDP forwarder.Forwarder to the vtep.Datapath seam. |
|
netstack
Package netstack implements the netstack VTEP datapath driver: it splices a gVisor channel.Endpoint (the overlay-side, userspace L3 link) to the ICX engine, moving encap'd frames over an injected underlay transport.
|
Package netstack implements the netstack VTEP datapath driver: it splices a gVisor channel.Endpoint (the overlay-side, userspace L3 link) to the ICX engine, moving encap'd frames over an injected underlay transport. |
|
tun
Package tun implements the tun VTEP datapath driver: it splices a kernel /dev/net/tun device (the overlay-side, L3 link the consumer routes to) to the ICX engine, moving encap'd frames over a UDP-socket underlay.
|
Package tun implements the tun VTEP datapath driver: it splices a kernel /dev/net/tun device (the overlay-side, L3 link the consumer routes to) to the ICX engine, moving encap'd frames over a UDP-socket underlay. |