Documentation
¶
Overview ¶
Package shape implements tc HTB hierarchy persistence and the `network shape` CLI verb. It renders the boot-replay script (solo-provisioner-bandwidth-shaper.sh) with bandwidth-parameterised class configurations, manages the solo-provisioner-bandwidth-shaper.service oneshot unit, and applies live tc class changes to the $EGRESS physical NIC.
The $VETH HTB is deliberately NOT persisted here: the veth interface does not survive reboot (Cilium recreates it on pod start), so persisting its qdisc would be meaningless. The daemon pod-lifecycle watcher reinstalls the $VETH HTB on the next pod create event by reading the class configs stored under ClassConfigDir.
Index ¶
- Constants
- func ApplyTcEgressScript(ctx context.Context) error
- func DetectEgressInterface() (string, error)
- func EnsureTcEgressUnit(ctx context.Context) error
- func FormatSpeedHint(mbit int) string
- func ParseSpeedMbit(s string) (int, bool)
- func ProvisionDefaultEgressShape(ctx context.Context, nicName, trunkRate string, ...) error
- func ProvisionDefaultIngressShape(ctx context.Context, nicName, trunkRate string, ...) error
- func ReadLinkSpeedMbit(nic string) (int, bool)
- func RemoveTcEgressUnit(ctx context.Context) error
- func ValidateClassOverride(name string, o ClassOverride) error
- type ClassConfig
- type ClassDelta
- type ClassOverride
- type ClassStat
- type Config
- type DeviceConfig
- type Manager
- func (m *Manager) ApplyIngressVeth(ctx context.Context, veth string) error
- func (m *Manager) CreateClass(ctx context.Context, cls *ClassConfig, force bool) (bool, error)
- func (m *Manager) CreateDevice(ctx context.Context, dev *DeviceConfig, force bool) (bool, error)
- func (m *Manager) DeleteClass(ctx context.Context, name string) error
- func (m *Manager) DeleteDevice(ctx context.Context, dir string) error
- func (m *Manager) ProvisionDefaultEgress(ctx context.Context, nicName, trunkRate string, ...) error
- func (m *Manager) ProvisionDefaultIngress(_ context.Context, trunkRate string, overrides map[string]ClassOverride) error
- func (m *Manager) RemoveIngressVeth(ctx context.Context, veth string) error
- func (m *Manager) SetClass(ctx context.Context, name string, rate, ceil *string, prio *int) error
- func (m *Manager) ShowAll() (string, error)
- func (m *Manager) ShowClass(name string) (string, error)
- func (m *Manager) ShowDevice(dir string) (string, error)
- func (m *Manager) TeardownEgress(ctx context.Context) error
- func (m *Manager) TeardownIngress(_ context.Context) error
- func (m *Manager) WatchClasses(ctx context.Context, spec WatchSpec, w io.Writer) error
- type TCRunner
- type WatchSpec
Constants ¶
const ( DirIngress = "ingress" // $VETH (pod traffic, applied by the daemon pod-lifecycle watcher) DirEgress = "egress" // $EGRESS physical NIC )
Direction constants for tc device/class mapping.
const ( // TcEgressScriptPath is the shell script that replays the $EGRESS HTB // hierarchy at boot. Lives under /usr/local/sbin (root-executable tools). TcEgressScriptPath = "/usr/local/sbin/solo-provisioner-bandwidth-shaper.sh" // TcEgressService is the systemd oneshot unit that executes TcEgressScriptPath // at boot, before solo-provisioner-daemon.service starts. TcEgressService = "solo-provisioner-bandwidth-shaper.service" // TcEgressServiceUnitPath is the absolute path where the unit file is // installed so systemd can discover it. TcEgressServiceUnitPath = "/usr/lib/systemd/system/" + TcEgressService // ShapeConfigDir is the root of the shape configuration tree persisted by // the `network shape` CLI verb. ShapeConfigDir = "/etc/solo-provisioner/network/shape" // DeviceConfigDir holds one JSON file per configured tc device ("ingress" // or "egress"), describing the root qdisc rate and default class. DeviceConfigDir = ShapeConfigDir + "/devices" // ClassConfigDir holds one JSON file per configured tc class, describing its // bandwidth parameters (rate, ceil, prio). The daemon pod-lifecycle watcher // reads this directory to reinstall $VETH classes on each pod create event. ClassConfigDir = ShapeConfigDir + "/classes" // ShapeLockDir is the directory containing the tc apply lock (on tmpfs so // it is auto-cleared on reboot). ShapeLockDir = "/run/solo-provisioner/network" // ShapeLockPath is the flock acquired (LOCK_EX) for the duration of any // tc mutating verb, preventing concurrent modifications. ShapeLockPath = ShapeLockDir + "/.tc-applying" )
const DefaultLinkSpeedMbit = 1000
DefaultLinkSpeedMbit is the link speed assumed when sysfs cannot report one (common on virtual NICs and at early boot, where /sys/class/net/<nic>/speed reads -1). It mirrors the fallback baked into the bandwidth-shaper boot script's sysfs-detection block; keep the two in sync.
Variables ¶
This section is empty.
Functions ¶
func ApplyTcEgressScript ¶
ApplyTcEgressScript ensures the bandwidth-shaper systemd unit is installed, then restarts it so the kernel picks up the HTB hierarchy immediately without waiting for a reboot. EnsureTcEgressUnit is idempotent (SHA-256 gated) so repeated calls are cheap. RestartService resets a previously-failed unit before running it, so a corrected script takes effect without a manual reset-failed. Using restart (rather than executing the script directly) keeps unit state visible: on success the unit shows active (exited), on failure failed — matching what operators see after a reboot.
func DetectEgressInterface ¶
DetectEgressInterface returns the name of the interface that carries the default route — the physical $EGRESS NIC the HTB hierarchy should be attached to. On multi-NIC hosts the desired interface must be specified explicitly via --egress-interface.
Fails with an actionable error when no default route is found (e.g. the routing table is not yet populated or the host has no default gateway).
func EnsureTcEgressUnit ¶
EnsureTcEgressUnit installs (or updates) the solo-provisioner-bandwidth-shaper.service unit file, daemon-reloads systemd, and enables the unit for boot. SHA-256 comparison is used so the write, reload, and enable are skipped when the on-disk content already matches the embedded template — making repeated installs cheap while ensuring a template change (e.g. ordering fix) is applied automatically on the next `block node install`.
func FormatSpeedHint ¶ added in v0.23.0
FormatSpeedHint converts a Mbit/s value into a tc-style bandwidth string suitable for operator-facing prompts (e.g. 1000 → "1gbit", 10000 → "10gbit", 100 → "100mbit").
func ParseSpeedMbit ¶ added in v0.23.0
ParseSpeedMbit parses a tc-style bandwidth string (e.g. "1gbit", "100mbit") into Mbit/s. Returns (0, false) for empty input or unrecognised formats.
func ProvisionDefaultEgressShape ¶ added in v0.23.0
func ProvisionDefaultEgressShape(ctx context.Context, nicName, trunkRate string, overrides map[string]ClassOverride) error
ProvisionDefaultEgressShape configures the egress shape registry with the three default HTB classes at proportions derived from trunkRate (with any per-class --shape overrides merged in), then renders and applies the boot script. Convenience wrapper over NewManager().ProvisionDefaultEgress.
func ProvisionDefaultIngressShape ¶ added in v0.25.0
func ProvisionDefaultIngressShape(ctx context.Context, nicName, trunkRate string, overrides map[string]ClassOverride) error
ProvisionDefaultIngressShape records the ingress shape registry with the three default HTB classes at proportions derived from trunkRate. Convenience wrapper over ProvisionDefaultIngress. When nicName is non-empty it pins the resolution of a "auto" trunkRate to the operator-chosen NIC (parity with the egress path on multi-NIC hosts); ingress bandwidth defaults to egress.
func ReadLinkSpeedMbit ¶ added in v0.23.0
ReadLinkSpeedMbit reads the link speed of nic from the kernel sysfs entry /sys/class/net/<nic>/speed and returns it in Mbit/s.
Returns (0, false) when the speed is unavailable: the file is missing (virtual NIC, tunnel), the value is non-positive (kernel reports -1 for unknown/down links), or the content is non-numeric. Callers treat false as "no hint available" and must never block an install on this.
func RemoveTcEgressUnit ¶ added in v0.28.0
RemoveTcEgressUnit is the teardown counterpart to EnsureTcEgressUnit: it stops and disables solo-provisioner-bandwidth-shaper.service, removes the unit file and the boot script it executes, then daemon-reloads. Callers must have torn the egress hierarchy down first — this only removes the boot-replay machinery, not the live tc state.
Idempotent: an already-absent unit or script is not an error, and a stop that fails on an inactive oneshot is logged and ignored.
func ValidateClassOverride ¶ added in v0.25.0
func ValidateClassOverride(name string, o ClassOverride) error
ValidateClassOverride checks that name is a known class and that each set field is individually valid (rate/ceil parseable, ceil >= rate when both are set, prio in [0,7]). It deliberately does NOT check the sum-of-rates constraint — that is enforced against the full merged class set when the shape is provisioned (see validateProvisionedClasses), because the sum depends on the profile defaults the override merges into.
Types ¶
type ClassConfig ¶ added in v0.23.0
type ClassConfig struct {
Name string `json:"name"`
Rate string `json:"rate"`
Ceil string `json:"ceil,omitempty"` // defaults to Rate when empty
Prio int `json:"prio"`
CreatedAt time.Time `json:"created_at"`
}
ClassConfig is the persisted bandwidth configuration for one named tc class. One JSON file per class under ClassConfigDir.
type ClassDelta ¶ added in v0.27.0
type ClassDelta struct {
Name string // class name ("partner") when the classid is known, else the raw classid
ClassID string // tc handle, e.g. "1:40"
RateBitsPerSec float64 // byte delta * 8 / interval seconds
BytesDelta uint64
OverlimitsDelta uint64
DropsDelta uint64
}
ClassDelta is the change in one class's counters between two samples, plus the throughput implied by the byte delta over the sampling interval. It is what `network shape watch` prints per tick: rate-over-time, not the cumulative totals (which `network shape show` does not expose and dashboards cover via the Prometheus counters).
type ClassOverride ¶ added in v0.25.0
ClassOverride carries operator-supplied overrides for one HTB class's bandwidth fields, parsed from `block node install --shape <class>=rate=...,ceil=...,prio=...`. A zero-value field (empty Rate/Ceil, nil Prio) means "keep the profile default", so an operator can override just one field (e.g. only --shape publisher=ceil=1gbit) without restating the others.
type ClassStat ¶ added in v0.27.0
type ClassStat struct {
ClassID string // tc handle, e.g. "1:40"
Bytes uint64
Packets uint64
Drops uint64
Overlimits uint64
}
ClassStat is a point-in-time snapshot of one tc HTB class's cumulative counters, as read from `tc -s class show dev <device>`. The byte/packet/drop/ overlimit counters are monotonic since the class (qdisc) was installed, so a throughput reading is the delta between two snapshots over the elapsed time.
type Config ¶ added in v0.23.0
type Config struct {
ScriptPath string
LockPath string
NICDetect func() (string, error)
SpeedDetect func(nic string) (int, bool)
ApplyEgress func(ctx context.Context) error
TCRunner TCRunner
}
Config customises a Manager. The zero value is not useful; prefer NewManager.
type DeviceConfig ¶ added in v0.23.0
type DeviceConfig struct {
Dir string `json:"dir"` // "ingress" or "egress"
Rate string `json:"rate"` // root HTB trunk class rate
DefaultClass string `json:"default_class"` // class name; unmatched traffic falls here
CreatedAt time.Time `json:"created_at"`
}
DeviceConfig is the persisted root-level tc configuration for one traffic direction. One JSON file per device under DeviceConfigDir (named by Dir).
Dir "egress" targets the $EGRESS physical NIC and drives the re-rendered bandwidth-shaper.sh boot script. Dir "ingress" targets $VETH and is consumed by the daemon pod-lifecycle watcher; no script is rendered for it.
type Manager ¶ added in v0.23.0
type Manager struct {
// contains filtered or unexported fields
}
Manager implements the `network shape` verb: create, set, show, and delete operations for tc HTB device roots and per-class bandwidth configurations.
Egress mutations (Dir "egress") re-render TcEgressScriptPath and restart the bandwidth-shaper.service oneshot so the kernel picks up changes immediately. Ingress mutations (Dir "ingress") only write config; the daemon pod-lifecycle watcher reads them and applies them to each new veth interface.
func NewManager ¶ added in v0.23.0
func NewManager() *Manager
NewManager returns a Manager wired to the live kernel and production paths.
func NewManagerWithConfig ¶ added in v0.23.0
NewManagerWithConfig returns a Manager, filling unset Config fields with their production defaults.
func (*Manager) ApplyIngressVeth ¶ added in v0.25.0
ApplyIngressVeth installs the $VETH ingress HTB hierarchy (design §5.1) on the given host-side veth, using the ingress device root and per-class budgets recorded by `network shape` (under DeviceConfigDir / ClassConfigDir). It is the privileged operation the daemon's pod-lifecycle watcher delegates via `block node tc-attach` on each BN pod create.
The hierarchy is: root HTB qdisc (default → the ingress default class), a trunk class 1:1 at the device root rate, one leaf class per recorded ingress class (1:10 / 1:20 / 1:30) with an fq_codel qdisc, and no tc filters — HTB classifies natively on skb->priority set by the nft classification rules.
The apply is idempotent: the root qdisc is torn down first (cascading to all classes and leaf qdiscs), so a rebind on a recycled veth name starts clean.
func (*Manager) CreateClass ¶ added in v0.23.0
CreateClass creates (or replaces with --force) a per-class bandwidth configuration. The device for the class direction must exist before adding classes. For egress classes: re-renders TcEgressScriptPath and restarts bandwidth-shaper.service. For ingress classes: writes config only (daemon pod-lifecycle watcher handles VETH apply).
Returns true if the config was created or replaced, false if it already existed and force was not set.
func (*Manager) CreateDevice ¶ added in v0.23.0
CreateDevice creates (or replaces with --force) the root device configuration. For "egress": re-renders TcEgressScriptPath and restarts bandwidth-shaper.service. For "ingress": writes config only (daemon pod-lifecycle watcher handles VETH apply).
Returns true if the config was created or replaced, false if it already existed and force was not set.
func (*Manager) DeleteClass ¶ added in v0.23.0
DeleteClass removes a class configuration. Fails if the class is referenced as the device's default class or by any policy's --stamp/--reply-stamp. For egress classes: re-renders TcEgressScriptPath and restarts bandwidth-shaper.service.
func (*Manager) DeleteDevice ¶ added in v0.23.0
DeleteDevice removes a device configuration. Fails if any classes are still configured for this device (delete classes first).
func (*Manager) ProvisionDefaultEgress ¶ added in v0.23.0
func (m *Manager) ProvisionDefaultEgress(ctx context.Context, nicName, trunkRate string, overrides map[string]ClassOverride) error
ProvisionDefaultEgress configures the egress device root and three default HTB classes at proportions derived from trunkRate (partner 40%/70%, public 30%/70%, reserve-egress 30%/100%), then renders and applies the boot script. trunkRate may be "auto", which is resolved to the detected link speed at create time (see resolveAutoRateString). Called by block node install so the shape registry is the single source of truth from first install, and re-run by reconfigure/upgrade — a re-run against an unchanged trunk rate keeps the recorded per-class values rather than resetting them to the proportions above (see mergeExistingConfig).
func (*Manager) ProvisionDefaultIngress ¶ added in v0.25.0
func (m *Manager) ProvisionDefaultIngress(_ context.Context, trunkRate string, overrides map[string]ClassOverride) error
ProvisionDefaultIngress records the ingress ($VETH) device root and three default HTB classes at proportions derived from trunkRate (publisher 80%, backfill-response 10%, reserve-ingress 10%; all ceil 100%). trunkRate may be "auto", resolved to the detected link speed at record time (see resolveAutoRateString). Unlike ProvisionDefaultEgress this writes config only and renders NO boot script: the $VETH HTB is deliberately not persisted across reboot — the daemon pod-lifecycle watcher replays it on each pod create from the stored class configs. Called by block node install so ApplyIngressVeth finds concrete ingress config on the first pod create (the per-pod replay has no sysfs fallback, so the recorded rates must always be concrete).
func (*Manager) RemoveIngressVeth ¶ added in v0.25.0
RemoveIngressVeth tears down the $VETH ingress HTB hierarchy on the given veth. It is best-effort: the kernel auto-removes veth-attached qdiscs when the veth disappears on pod delete, so this is mostly for the proactive (re)attach-time cleanup path and never fails on an already-absent qdisc.
func (*Manager) SetClass ¶ added in v0.23.0
SetClass atomically updates one or more bandwidth parameters of an existing class. Only non-nil pointer fields are changed; nil means "keep current value". For egress classes: runs `tc class change` on the live kernel and re-renders the boot script for reboot persistence. For ingress classes: updates config only.
func (*Manager) ShowAll ¶ added in v0.23.0
ShowAll returns a human-readable summary of all configured devices and classes.
func (*Manager) ShowClass ¶ added in v0.23.0
ShowClass returns a human-readable summary of the named class config.
func (*Manager) ShowDevice ¶ added in v0.23.0
ShowDevice returns a human-readable summary of the named device config.
func (*Manager) TeardownEgress ¶ added in v0.27.0
TeardownEgress removes the entire egress tc shape configuration — every egress class and the egress device root — then re-renders the boot script to its empty default and applies it, dropping the live HTB hierarchy on the physical NIC.
Unlike DeleteClass/DeleteDevice (the operator-facing single-object verbs, which deliberately refuse to delete a device's default class or a still-referenced class), this is the wholesale teardown used when traffic shaping is disabled on an existing block node: the caller has already removed the BN network policies that reference these classes, so the per-object safety guards no longer apply. It is idempotent — with no egress config present it re-renders the empty script and returns nil.
func (*Manager) TeardownIngress ¶ added in v0.28.0
TeardownIngress removes the entire ingress tc shape configuration — every ingress class and the ingress device root — from the shape registry. Unlike TeardownEgress there is no boot script to re-render: the $VETH HTB is ephemeral (Cilium recreates the veth per pod) and was never persisted for boot replay. Idempotent — with no ingress config present it returns nil.
func (*Manager) WatchClasses ¶ added in v0.27.0
WatchClasses samples the live tc class counters on the resolved interface every spec.Interval and writes a per-class delta row (rate, bytes sent, Δoverlimits, Δdrops) each tick to w. It is read-only: no locks, no mutations. It returns when ctx is cancelled (e.g. the operator hits Ctrl-C) or, when spec.Count > 0, after that many delta rows have been printed.
type TCRunner ¶ added in v0.23.0
type TCRunner interface {
// ClassChange runs `tc class change dev <nic> parent 1:1 classid 1:<minor>
// htb rate <rate> ceil <ceil> prio <prio>` on the live kernel.
ClassChange(ctx context.Context, nic, minor, rate, ceil string, prio int) error
// QdiscDelRoot runs `tc qdisc del dev <nic> root`, tearing down any existing
// hierarchy (cascading to all classes and leaf qdiscs). It is best-effort: a
// missing root qdisc (a fresh veth) is not an error, so the caller can
// unconditionally rebuild — matching the bandwidth-shaper boot script's `|| true`.
QdiscDelRoot(ctx context.Context, nic string) error
// QdiscAddRoot runs `tc qdisc add dev <nic> root handle 1: htb default
// <defaultMinor>`, installing the root HTB qdisc whose unmatched traffic
// falls to class 1:<defaultMinor>.
QdiscAddRoot(ctx context.Context, nic, defaultMinor string) error
// ClassAddRoot runs `tc class add dev <nic> parent 1: classid 1:1 htb rate
// <rate> ceil <ceil>`, the trunk class every per-class leaf attaches to.
ClassAddRoot(ctx context.Context, nic, rate, ceil string) error
// ClassAdd runs `tc class add dev <nic> parent 1:1 classid 1:<minor> htb
// rate <rate> ceil <ceil> prio <prio>`, a per-class leaf under the trunk.
ClassAdd(ctx context.Context, nic, minor, rate, ceil string, prio int) error
// QdiscAddFqCodel runs `tc qdisc add dev <nic> parent 1:<minor> handle
// <handle>: fq_codel`, the leaf qdisc for a class.
QdiscAddFqCodel(ctx context.Context, nic, minor, handle string) error
// ClassStats runs `tc -s -j class show dev <dev>` and returns each class's
// cumulative counters keyed by tc handle (e.g. "1:40"). It is the read
// counterpart to the write verbs above, backing `network shape watch`.
ClassStats(ctx context.Context, dev string) (map[string]ClassStat, error)
}
TCRunner abstracts live kernel tc qdisc/class operations for testability. The egress path drives ClassChange (live tuning of an already-installed hierarchy); the ingress path drives the qdisc/class add verbs to install the per-veth HTB hierarchy from scratch on each BN pod create.
The interface lives in this build-tag-free file so both the Linux (execTCRunner) and non-Linux (noopTCRunner) implementations, and the platform-agnostic Manager that consumes it, share a single declaration.
type WatchSpec ¶ added in v0.27.0
type WatchSpec struct {
Device string // "egress" or "ingress" — required; selects the class set
Iface string // interface to sample — required
Class string // optional: narrow to one class in Device's direction
Interval time.Duration // sampling interval
Count int // number of delta samples to print then stop; 0 = until ctx is cancelled
}
WatchSpec parameterises Manager.WatchClasses. Both Device (the traffic direction, which selects the class set) and Iface (the interface to sample) are required and operator-supplied: `network shape watch` performs no environment probing — no NIC or veth auto-detection — so it never depends on a block node running or on Kubernetes. Class optionally narrows the watch to one class, which must belong to Device's direction.
Source Files