Documentation
¶
Overview ¶
Package queue holds the frame-ownership logic of the DPDK backend, with no cgo and no DPDK in sight.
Everything that decides who owns a frame lives here: what Alloc may hand out, what Transmit accepts, when a frame goes back on the free list, and which free list it goes back to. That is the part of a backend where the bugs are -- a frame in two places at once is invisible in every counter -- so it is the part written against a fake driver and tested without hardware.
The driver appears as the PMD interface. In production it is one cgo call per batch; in the tests it is a Go type that behaves like a real poll-mode driver, including the awkward parts: taking a prefix of a burst rather than all of it, freeing transmitted buffers only when it feels like it, and pulling receive buffers out of a pool in all-or-nothing bulks.
Index ¶
- type Config
- type PMD
- type Ring
- type Rx
- func (b *Rx) Close()
- func (b *Rx) Closed() bool
- func (b *Rx) Dead() error
- func (q *Rx) DrainReturned() int
- func (q *Rx) Fill(n int) int
- func (q *Rx) NumFreeFillSlots() int
- func (b *Rx) NumFreeFrames() int
- func (q *Rx) NumOutstanding() int
- func (b *Rx) Owns(addr uint64) bool
- func (b *Rx) PoolRejected() uint64
- func (q *Rx) Receive(max int) []packetio.Desc
- func (q *Rx) Recycle(descs []packetio.Desc)
- func (b *Rx) Region() []byte
- type RxStats
- type Tx
- func (t *Tx) Alloc(n int) []packetio.Desc
- func (b *Tx) Close()
- func (b *Tx) Closed() bool
- func (t *Tx) Complete(max int) int
- func (b *Tx) Dead() error
- func (t *Tx) Free(descs []packetio.Desc)
- func (t *Tx) NumCompleted() int
- func (b *Tx) NumFreeFrames() int
- func (t *Tx) NumFreeSlots() int
- func (t *Tx) NumInFlight() int
- func (b *Tx) Owns(addr uint64) bool
- func (b *Tx) PoolRejected() uint64
- func (t *Tx) Reclaim(max int, out []packetio.Desc) []packetio.Desc
- func (b *Tx) Region() []byte
- func (t *Tx) Transmit(descs []packetio.Desc) int
- func (t *Tx) TransmitOffload(descs []packetio.Desc, offs []packetio.Offload) (int, error)
- type TxStats
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Config ¶
type Config struct {
// Region is the frame memory, and RegionVA the address the same memory is
// known by to the driver and to the mempool. Subtracting one from the other
// is how an address the driver returns becomes an offset this package can
// index -- the only pointer arithmetic in the backend, done once per batch.
Region []byte
RegionVA uint64
// Layout says where the mbuf and the packet sit inside a frame.
Layout mbuf.Layout
// PoolVA is this queue's rte_mempool, which every frame it hands to the
// driver is stamped with so that the driver frees it back here.
PoolVA uint64
// FirstFrame and Frames are this queue's slice of the region, in frames.
// Queues never share a frame: that is what lets their free lists run
// without a lock.
FirstFrame int
Frames int
// Depth is the driver's descriptor ring depth, which bounds how many
// frames can be outstanding at once.
Depth int
// Supply is the ring the driver draws receive buffers from, and Returned
// the ring it frees finished buffers into. A transmit queue uses only
// Returned; a receive queue uses both.
Supply *Ring
Returned *Ring
// Checksums says the device took transmit checksum offload, so plain
// Transmit should ask the NIC to compute them, and TSO that it took
// segmentation. Both are what the device actually accepted at configure
// time, never what was asked for.
Checksums bool
TSO bool
PMD PMD
}
Config describes one queue's share of a device.
type PMD ¶
type PMD interface {
// TxBurst hands mbuf addresses to the driver and returns how many it
// accepted, always a prefix. The driver owns those buffers until it frees
// them, which it does into the mempool each mbuf names.
TxBurst(mbufs []uint64) int
// RxBurst fills out with the addresses of received mbufs and returns how
// many. The driver takes the buffers it needs from the supply ring.
RxBurst(out []uint64) int
// Poke asks the driver to look at its transmit completions now.
//
// It exists because DPDK has no way to say "give me back what you have
// finished with". A PMD processes completions inside its own transmit
// burst and nowhere else, so a queue that has stopped sending never gets
// its last frames back. On mlx5, rte_eth_tx_done_cleanup is not
// implemented at all and rte_eth_tx_descriptor_status is what runs the
// completion handler; either way it is one call, on the idle path only.
Poke()
}
PMD is the driver, as this package needs it: three calls, each a single cgo crossing in production and none of them per packet.
Measured on a ConnectX-6 Dx, a crossing costs about 36 ns. Over a batch of 64 that is under a nanosecond a packet, which is why the whole design is built around keeping the crossings per batch rather than per frame.
type Ring ¶
type Ring struct {
// contains filtered or unexported fields
}
Ring is one of the two arrays that stand between the PMD and this backend's free list: the supply, which Go fills and the driver drains, and the returned, which the driver fills and Go drains. Each holds mbuf addresses.
It is deliberately not synchronised. In production each ring has exactly one producer and one consumer, and both run on the queue's own goroutine: the driver only ever touches a ring from inside a burst call that this goroutine made, so the two never run at the same time. Atomics here would cost a locked instruction per frame to order accesses that cannot race.
In the real backend the backing array is C memory the mempool ops write to, viewed from Go as a slice, and so are the indices, which is why they are pointers. Nothing in this type cares which it is.
The indices are one word each because in production they live in C memory, beside the array, where the mempool ops read and write them. Each index has exactly one writer -- Go moves the supply's tail and the returned ring's head, the driver moves the other two -- and both sides run on the queue's own thread, so plain loads and stores are enough and no atomic is needed.
func NewRing ¶
NewRing builds a ring holding capacity addresses, which must be a power of two so that wrapping is a mask. Sizing is the caller's problem and is not a tuning knob: a ring too small to hold everything the driver may hand back loses frames, so the backend sizes both from the queue depth at Open and never grows them on the packet path.
func NewRingOver ¶
NewRingOver builds a ring over memory and indices the caller already has, which is how the real backend shares one with the mempool ops in C.
func (*Ring) Pop ¶
Pop removes up to n addresses, appending them to dst, and returns the grown slice. It allocates nothing when dst has the capacity.
func (*Ring) PopBulk ¶
PopBulk removes exactly n addresses into dst, or none at all, and reports whether it did.
All or nothing, because that is what rte_mempool_get_bulk promises the driver and what the driver relies on: mlx5's vectorised receive path asks for a whole replenishment batch and treats a short answer as no answer, dropping packets rather than taking what there is. A supply that hands out a partial bulk would be a subtly different pool from the one DPDK expects.
func (*Ring) Push ¶
Push adds addresses and returns how many it took, always a prefix. A short return means the ring is full, which for the returned ring would mean losing a frame -- so the backend sizes it so that cannot happen and counts it if it does.
type Rx ¶
type Rx struct {
Stats RxStats
// contains filtered or unexported fields
}
Rx is one receive queue.
func (*Rx) Close ¶
func (b *Rx) Close()
Close stops the queue. Every method afterwards returns nothing rather than touching memory the device may have taken away.
func (*Rx) Dead ¶
func (b *Rx) Dead() error
Dead is the latched fatal error, or nil. Safe from any goroutine.
func (*Rx) DrainReturned ¶
DrainReturned takes back frames the driver freed on this queue's pool rather than delivering them, which is what happens to the buffers still in its ring when a port is stopped. It reports how many came back.
func (*Rx) Fill ¶
Fill makes up to n frames available for the driver to receive into and reports how many it made available.
This is where the DPDK receive model differs from every other backend's, and the difference is worth stating: nothing is posted to a ring here. The driver takes buffers from the supply when it wants them, in bulks it chooses, and a bulk it cannot satisfy in full is one it does not take at all. So a receiver should keep the supply comfortably full rather than topping it up a frame at a time.
func (*Rx) NumFreeFillSlots ¶
NumFreeFillSlots is how many more frames the supply will accept.
func (*Rx) NumFreeFrames ¶
func (b *Rx) NumFreeFrames() int
NumFreeFrames is how many frames are on this queue's free list.
func (*Rx) NumOutstanding ¶
NumOutstanding is how many frames the driver currently holds.
func (*Rx) Owns ¶
Owns reports whether an address lies inside this queue's slice of the region -- the frames its pool covers. A wrapper that fans several hardware queues out behind one logical queue uses it to route a descriptor back to the queue whose pool the frame belongs to.
func (*Rx) PoolRejected ¶
func (b *Rx) PoolRejected() uint64
PoolRejected is how many frames the pool refused: foreign, misaligned, or already free. Anything but zero is a bug above this layer.
func (*Rx) Receive ¶
Receive takes up to max received packets. The frames belong to the caller until Recycle.
A packet spanning more than one mbuf is refused rather than delivered: this backend keeps the MTU inside one frame precisely so that cannot happen, and a chained packet would mean the queue was configured differently from what this code assumes. Its frames go straight back.
type RxStats ¶
type RxStats struct {
Packets atomic.Uint64
Bytes atomic.Uint64
Filled atomic.Uint64
Batches atomic.Uint64
PoolEmpty atomic.Uint64
Chained atomic.Uint64 // packets spanning several mbufs, which this backend refuses
BadLen atomic.Uint64 // lengths the driver reported that do not fit their frame
Foreign atomic.Uint64
}
RxStats counts what a receive queue has done. Atomic for the same reason as TxStats: read by a monitoring goroutine, added to once per batch.
type Tx ¶
type Tx struct {
Stats TxStats
// contains filtered or unexported fields
}
Tx is one transmit queue.
func (*Tx) Alloc ¶
Alloc takes up to n frames from the free list and returns descriptors naming the packet area of each, with the headroom already reserved.
It never returns more than the following Transmit could accept, so a caller that transmits everything it allocated cannot leak a frame.
func (*Tx) Close ¶
func (b *Tx) Close()
Close stops the queue. Every method afterwards returns nothing rather than touching memory the device may have taken away.
func (*Tx) Complete ¶
Complete returns up to max finished frames to this queue's free list and reports how many.
func (*Tx) Dead ¶
func (b *Tx) Dead() error
Dead is the latched fatal error, or nil. Safe from any goroutine.
func (*Tx) NumCompleted ¶
NumCompleted is how many frames are waiting to be handed back right now, without asking the driver for more.
func (*Tx) NumFreeFrames ¶
func (b *Tx) NumFreeFrames() int
NumFreeFrames is how many frames are on this queue's free list.
func (*Tx) NumFreeSlots ¶
NumFreeSlots is how many more frames the driver's ring can accept.
func (*Tx) NumInFlight ¶
NumInFlight is how many frames the driver currently owns.
func (*Tx) Owns ¶
Owns reports whether an address lies inside this queue's slice of the region -- the frames its pool covers. A wrapper that fans several hardware queues out behind one logical queue uses it to route a descriptor back to the queue whose pool the frame belongs to.
func (*Tx) PoolRejected ¶
func (b *Tx) PoolRejected() uint64
PoolRejected is how many frames the pool refused: foreign, misaligned, or already free. Anything but zero is a bug above this layer.
func (*Tx) Reclaim ¶
Reclaim is Complete for frames that belong somewhere else: it hands them to the caller instead of returning them here.
This is what a forwarder uses. The frame came from a receive queue's pool, so putting it on this queue's free list would starve that receive queue and alias the frame into the wrong pool -- the one bug in a forwarder that no counter on the wire shows.
func (*Tx) Transmit ¶
Transmit hands descriptors to the driver and returns how many it took, always a prefix.
Frames in the prefix belong to the driver until it frees them; frames in the rest still belong to the caller. The mbuf of each is written first -- where its packet starts, how long it is, and which pool to free it into -- and then the whole batch goes across in one call.
Where the device was opened with checksum offload, each packet's headers are read here and the NIC is asked to compute its checksums. A packet this package cannot parse is sent as it stands, with whatever checksums the caller wrote: that is the behaviour of every other backend, and refusing it would make an ARP frame an error.
func (*Tx) TransmitOffload ¶
TransmitOffload is Transmit with segmentation and checksum metadata for each frame, so a super-frame is cut up by the NIC rather than here.
It returns the accepted prefix and an error naming the first descriptor whose Offload it refused. A zero Offload sends an ordinary frame. What the device cannot do is ErrUnsupported rather than something quietly weaker: a caller asking for segmentation and getting a 64 KB frame on the wire has no way to notice until the far end does.
type TxStats ¶
type TxStats struct {
Packets atomic.Uint64
Bytes atomic.Uint64
Completed atomic.Uint64
Batches atomic.Uint64
RingFull atomic.Uint64
PoolEmpty atomic.Uint64
BadDesc atomic.Uint64
// BadOffload counts Offloads refused: metadata that does not describe the
// packet it came with, or asks for work this device does not do.
BadOffload atomic.Uint64
Pokes atomic.Uint64
Foreign atomic.Uint64 // addresses returned that this region does not contain
}
TxStats counts what a transmit queue has done.
The fields are atomics so that a monitoring goroutine may read them while the owner drives the queue. The owner adds once per batch, never per packet, so the cost on the packet path is a handful of uncontended atomic adds per burst -- the same arrangement the mlx5 backend runs at line rate.
Source Files