Documentation
¶
Overview ¶
Package shape implements tc HTB hierarchy persistence and the `network shape` CLI verb. It renders the boot-replay script (solo-provisioner-tc-egress.sh) with bandwidth-parameterised class configurations, manages the solo-provisioner-tc-egress.service oneshot unit, and applies live tc class changes to the $EGRESS physical NIC.
The $VETH HTB is deliberately NOT persisted here: the veth interface does not survive reboot (Cilium recreates it on pod start), so persisting its qdisc would be meaningless. The daemon pod-lifecycle watcher reinstalls the $VETH HTB on the next pod create event by reading the class configs stored under ClassConfigDir.
Index ¶
- Constants
- func ApplyTcEgressScript(ctx context.Context) error
- func ClassDirection(name string) (string, error)
- func DetectEgressInterface() (string, error)
- func EnsureTcEgressUnit(ctx context.Context) error
- func FormatSpeedHint(mbit int) string
- func ParseSpeedMbit(s string) (int, bool)
- func ProvisionDefaultEgressShape(ctx context.Context, nicName, trunkRate string, ...) error
- func ProvisionDefaultIngressShape(ctx context.Context, nicName, trunkRate string, ...) error
- func ReadLinkSpeedMbit(nic string) (int, bool)
- func RenderAndApplyDefaultEgress(ctx context.Context, nic string) error
- func ValidateClassOverride(name string, o ClassOverride) error
- type ClassConfig
- type ClassOverride
- type Config
- type DeviceConfig
- type Manager
- func (m *Manager) ApplyIngressVeth(ctx context.Context, veth string) error
- func (m *Manager) CreateClass(ctx context.Context, cls *ClassConfig, force bool) (bool, error)
- func (m *Manager) CreateDevice(ctx context.Context, dev *DeviceConfig, force bool) (bool, error)
- func (m *Manager) DeleteClass(ctx context.Context, name string) error
- func (m *Manager) DeleteDevice(ctx context.Context, dir string) error
- func (m *Manager) HasEgressConfig() (bool, error)
- func (m *Manager) ProvisionDefaultEgress(ctx context.Context, nicName, trunkRate string, ...) error
- func (m *Manager) ProvisionDefaultIngress(_ context.Context, trunkRate string, overrides map[string]ClassOverride) error
- func (m *Manager) RemoveIngressVeth(ctx context.Context, veth string) error
- func (m *Manager) RenderAndApplyEgress(ctx context.Context, nic string) error
- func (m *Manager) SetClass(ctx context.Context, name string, rate, ceil *string, prio *int) error
- func (m *Manager) ShowAll() (string, error)
- func (m *Manager) ShowClass(name string) (string, error)
- func (m *Manager) ShowDevice(dir string) (string, error)
- type TCRunner
Constants ¶
const ( DirIngress = "ingress" // $VETH (pod traffic, applied by the daemon pod-lifecycle watcher) DirEgress = "egress" // $EGRESS physical NIC )
Direction constants for tc device/class mapping.
const ( // TcEgressScriptPath is the shell script that replays the $EGRESS HTB // hierarchy at boot. Lives under /usr/local/sbin (root-executable tools). TcEgressScriptPath = "/usr/local/sbin/solo-provisioner-tc-egress.sh" // TcEgressService is the systemd oneshot unit that executes TcEgressScriptPath // at boot, before solo-provisioner-daemon.service starts. TcEgressService = "solo-provisioner-tc-egress.service" // TcEgressServiceUnitPath is the absolute path where the unit file is // installed so systemd can discover it. TcEgressServiceUnitPath = "/usr/lib/systemd/system/" + TcEgressService // ShapeConfigDir is the root of the shape configuration tree persisted by // the `network shape` CLI verb. ShapeConfigDir = "/etc/solo-provisioner/network/shape" // DeviceConfigDir holds one JSON file per configured tc device ("ingress" // or "egress"), describing the root qdisc rate and default class. DeviceConfigDir = ShapeConfigDir + "/devices" // ClassConfigDir holds one JSON file per configured tc class, describing its // bandwidth parameters (rate, ceil, prio). The daemon pod-lifecycle watcher // reads this directory to reinstall $VETH classes on each pod create event. ClassConfigDir = ShapeConfigDir + "/classes" // ShapeLockDir is the directory containing the tc apply lock (on tmpfs so // it is auto-cleared on reboot). ShapeLockDir = "/run/solo-provisioner/network" // ShapeLockPath is the flock acquired (LOCK_EX) for the duration of any // tc mutating verb, preventing concurrent modifications. ShapeLockPath = ShapeLockDir + "/.tc-applying" )
const DefaultLinkSpeedMbit = 1000
DefaultLinkSpeedMbit is the link speed assumed when sysfs cannot report one (common on virtual NICs and at early boot, where /sys/class/net/<nic>/speed reads -1). It mirrors the fallback baked into the tc-egress boot script's sysfs-detection block; keep the two in sync.
Variables ¶
This section is empty.
Functions ¶
func ApplyTcEgressScript ¶
ApplyTcEgressScript ensures the tc-egress systemd unit is installed, then restarts it so the kernel picks up the HTB hierarchy immediately without waiting for a reboot. EnsureTcEgressUnit is idempotent (SHA-256 gated) so repeated calls are cheap. RestartService resets a previously-failed unit before running it, so a corrected script takes effect without a manual reset-failed. Using restart (rather than executing the script directly) keeps unit state visible: on success the unit shows active (exited), on failure failed — matching what operators see after a reboot.
func ClassDirection ¶ added in v0.25.0
ClassDirection returns the tc device direction ("ingress" or "egress") a class belongs to, so callers can route a --shape override to the device it applies to. Unknown class names return an error naming the known classes.
func DetectEgressInterface ¶
DetectEgressInterface returns the name of the interface that carries the default route — the physical $EGRESS NIC the HTB hierarchy should be attached to. On multi-NIC hosts the desired interface must be specified explicitly via --egress-interface.
Fails with an actionable error when no default route is found (e.g. the routing table is not yet populated or the host has no default gateway).
func EnsureTcEgressUnit ¶
EnsureTcEgressUnit installs (or updates) the solo-provisioner-tc-egress.service unit file, daemon-reloads systemd, and enables the unit for boot. SHA-256 comparison is used so the write, reload, and enable are skipped when the on-disk content already matches the embedded template — making repeated installs cheap while ensuring a template change (e.g. ordering fix) is applied automatically on the next `block node install`.
func FormatSpeedHint ¶ added in v0.23.0
FormatSpeedHint converts a Mbit/s value into a tc-style bandwidth string suitable for operator-facing prompts (e.g. 1000 → "1gbit", 10000 → "10gbit", 100 → "100mbit").
func ParseSpeedMbit ¶ added in v0.23.0
ParseSpeedMbit parses a tc-style bandwidth string (e.g. "1gbit", "100mbit") into Mbit/s. Returns (0, false) for empty input or unrecognised formats.
func ProvisionDefaultEgressShape ¶ added in v0.23.0
func ProvisionDefaultEgressShape(ctx context.Context, nicName, trunkRate string, overrides map[string]ClassOverride) error
ProvisionDefaultEgressShape configures the egress shape registry with the three default HTB classes at proportions derived from trunkRate (with any per-class --shape overrides merged in), then renders and applies the boot script. Convenience wrapper over NewManager().ProvisionDefaultEgress.
func ProvisionDefaultIngressShape ¶ added in v0.25.0
func ProvisionDefaultIngressShape(ctx context.Context, nicName, trunkRate string, overrides map[string]ClassOverride) error
ProvisionDefaultIngressShape records the ingress shape registry with the three default HTB classes at proportions derived from trunkRate. Convenience wrapper over ProvisionDefaultIngress. When nicName is non-empty it pins the resolution of a "auto" trunkRate to the operator-chosen NIC (parity with the egress path on multi-NIC hosts); ingress bandwidth defaults to egress.
func ReadLinkSpeedMbit ¶ added in v0.23.0
ReadLinkSpeedMbit reads the link speed of nic from the kernel sysfs entry /sys/class/net/<nic>/speed and returns it in Mbit/s.
Returns (0, false) when the speed is unavailable: the file is missing (virtual NIC, tunnel), the value is non-positive (kernel reports -1 for unknown/down links), or the content is non-numeric. Callers treat false as "no hint available" and must never block an install on this.
func RenderAndApplyDefaultEgress ¶ added in v0.23.0
RenderAndApplyDefaultEgress renders the tc-egress script for nic from the shape registry (or sysfs fallback when no config exists) and applies it. Used when no trunk rate is supplied (e.g. block node reconfigure without --link-rate).
func ValidateClassOverride ¶ added in v0.25.0
func ValidateClassOverride(name string, o ClassOverride) error
ValidateClassOverride checks that name is a known class and that each set field is individually valid (rate/ceil parseable, ceil >= rate when both are set, prio in [0,7]). It deliberately does NOT check the sum-of-rates constraint — that is enforced against the full merged class set when the shape is provisioned (see validateProvisionedClasses), because the sum depends on the profile defaults the override merges into.
Types ¶
type ClassConfig ¶ added in v0.23.0
type ClassConfig struct {
Name string `json:"name"`
Rate string `json:"rate"`
Ceil string `json:"ceil,omitempty"` // defaults to Rate when empty
Prio int `json:"prio"`
CreatedAt time.Time `json:"created_at"`
}
ClassConfig is the persisted bandwidth configuration for one named tc class. One JSON file per class under ClassConfigDir.
type ClassOverride ¶ added in v0.25.0
ClassOverride carries operator-supplied overrides for one HTB class's bandwidth fields, parsed from `block node install --shape <class>=rate=...,ceil=...,prio=...`. A zero-value field (empty Rate/Ceil, nil Prio) means "keep the profile default", so an operator can override just one field (e.g. only --shape publisher=ceil=1gbit) without restating the others.
type Config ¶ added in v0.23.0
type Config struct {
ScriptPath string
LockPath string
NICDetect func() (string, error)
SpeedDetect func(nic string) (int, bool)
ApplyEgress func(ctx context.Context) error
TCRunner TCRunner
}
Config customises a Manager. The zero value is not useful; prefer NewManager.
type DeviceConfig ¶ added in v0.23.0
type DeviceConfig struct {
Dir string `json:"dir"` // "ingress" or "egress"
Rate string `json:"rate"` // root HTB trunk class rate
DefaultClass string `json:"default_class"` // class name; unmatched traffic falls here
CreatedAt time.Time `json:"created_at"`
}
DeviceConfig is the persisted root-level tc configuration for one traffic direction. One JSON file per device under DeviceConfigDir (named by Dir).
Dir "egress" targets the $EGRESS physical NIC and drives the re-rendered tc-egress.sh boot script. Dir "ingress" targets $VETH and is consumed by the daemon pod-lifecycle watcher; no script is rendered for it.
type Manager ¶ added in v0.23.0
type Manager struct {
// contains filtered or unexported fields
}
Manager implements the `network shape` verb: create, set, show, and delete operations for tc HTB device roots and per-class bandwidth configurations.
Egress mutations (Dir "egress") re-render TcEgressScriptPath and restart the tc-egress.service oneshot so the kernel picks up changes immediately. Ingress mutations (Dir "ingress") only write config; the daemon pod-lifecycle watcher reads them and applies them to each new veth interface.
func NewManager ¶ added in v0.23.0
func NewManager() *Manager
NewManager returns a Manager wired to the live kernel and production paths.
func NewManagerWithConfig ¶ added in v0.23.0
NewManagerWithConfig returns a Manager, filling unset Config fields with their production defaults.
func (*Manager) ApplyIngressVeth ¶ added in v0.25.0
ApplyIngressVeth installs the $VETH ingress HTB hierarchy (design §5.1) on the given host-side veth, using the ingress device root and per-class budgets recorded by `network shape` (under DeviceConfigDir / ClassConfigDir). It is the privileged operation the daemon's pod-lifecycle watcher delegates via `block node tc-attach` on each BN pod create.
The hierarchy is: root HTB qdisc (default → the ingress default class), a trunk class 1:1 at the device root rate, one leaf class per recorded ingress class (1:10 / 1:20 / 1:30) with an fq_codel qdisc, and no tc filters — HTB classifies natively on skb->priority set by the nft classification rules.
The apply is idempotent: the root qdisc is torn down first (cascading to all classes and leaf qdiscs), so a rebind on a recycled veth name starts clean.
func (*Manager) CreateClass ¶ added in v0.23.0
CreateClass creates (or replaces with --force) a per-class bandwidth configuration. The device for the class direction must exist before adding classes. For egress classes: re-renders TcEgressScriptPath and restarts tc-egress.service. For ingress classes: writes config only (daemon pod-lifecycle watcher handles VETH apply).
Returns true if the config was created or replaced, false if it already existed and force was not set.
func (*Manager) CreateDevice ¶ added in v0.23.0
CreateDevice creates (or replaces with --force) the root device configuration. For "egress": re-renders TcEgressScriptPath and restarts tc-egress.service. For "ingress": writes config only (daemon pod-lifecycle watcher handles VETH apply).
Returns true if the config was created or replaced, false if it already existed and force was not set.
func (*Manager) DeleteClass ¶ added in v0.23.0
DeleteClass removes a class configuration. Fails if the class is referenced as the device's default class or by any policy's --stamp/--reply-stamp. For egress classes: re-renders TcEgressScriptPath and restarts tc-egress.service.
func (*Manager) DeleteDevice ¶ added in v0.23.0
DeleteDevice removes a device configuration. Fails if any classes are still configured for this device (delete classes first).
func (*Manager) HasEgressConfig ¶ added in v0.25.0
HasEgressConfig reports whether the egress device already has a recorded shape config (device + classes) in the registry — i.e. whether a prior ProvisionDefaultEgress (via `block node install`/`network shape create`) has already run. Used by the install-time TcEgressPersist step to decide between provisioning fresh defaults (nothing recorded yet: matches TcIngressRecord's unconditional behavior) and re-applying from what's already there (an existing install: never clobber operator-adjusted `network shape set` values just because --link-rate was omitted).
func (*Manager) ProvisionDefaultEgress ¶ added in v0.23.0
func (m *Manager) ProvisionDefaultEgress(ctx context.Context, nicName, trunkRate string, overrides map[string]ClassOverride) error
ProvisionDefaultEgress configures the egress device root and three default HTB classes at proportions derived from trunkRate (partner 40%/70%, public 30%/70%, reserve-egress 30%/100%), then renders and applies the boot script. trunkRate may be "auto", which is resolved to the detected link speed at create time (see resolveAutoRateString). Existing configs are always replaced. Called by block node install so the shape registry is the single source of truth from first install.
func (*Manager) ProvisionDefaultIngress ¶ added in v0.25.0
func (m *Manager) ProvisionDefaultIngress(_ context.Context, trunkRate string, overrides map[string]ClassOverride) error
ProvisionDefaultIngress records the ingress ($VETH) device root and three default HTB classes at proportions derived from trunkRate (publisher 80%, backfill-response 10%, reserve-ingress 10%; all ceil 100%). trunkRate may be "auto", resolved to the detected link speed at record time (see resolveAutoRateString). Unlike ProvisionDefaultEgress this writes config only and renders NO boot script: the $VETH HTB is deliberately not persisted across reboot — the daemon pod-lifecycle watcher replays it on each pod create from the stored class configs. Called by block node install so ApplyIngressVeth finds concrete ingress config on the first pod create (the per-pod replay has no sysfs fallback, so the recorded rates must always be concrete).
func (*Manager) RemoveIngressVeth ¶ added in v0.25.0
RemoveIngressVeth tears down the $VETH ingress HTB hierarchy on the given veth. It is best-effort: the kernel auto-removes veth-attached qdiscs when the veth disappears on pod delete, so this is mostly for the proactive (re)attach-time cleanup path and never fails on an already-absent qdisc.
func (*Manager) RenderAndApplyEgress ¶ added in v0.23.0
RenderAndApplyEgress renders the tc-egress boot script for nic from stored shape config (if available) or the sysfs auto-detect default, then installs and restarts tc-egress.service. Idempotent.
func (*Manager) SetClass ¶ added in v0.23.0
SetClass atomically updates one or more bandwidth parameters of an existing class. Only non-nil pointer fields are changed; nil means "keep current value". For egress classes: runs `tc class change` on the live kernel and re-renders the boot script for reboot persistence. For ingress classes: updates config only.
func (*Manager) ShowAll ¶ added in v0.23.0
ShowAll returns a human-readable summary of all configured devices and classes.
type TCRunner ¶ added in v0.23.0
type TCRunner interface {
// ClassChange runs `tc class change dev <nic> parent 1:1 classid 1:<minor>
// htb rate <rate> ceil <ceil> prio <prio>` on the live kernel.
ClassChange(ctx context.Context, nic, minor, rate, ceil string, prio int) error
// QdiscDelRoot runs `tc qdisc del dev <nic> root`, tearing down any existing
// hierarchy (cascading to all classes and leaf qdiscs). It is best-effort: a
// missing root qdisc (a fresh veth) is not an error, so the caller can
// unconditionally rebuild — matching the tc-egress boot script's `|| true`.
QdiscDelRoot(ctx context.Context, nic string) error
// QdiscAddRoot runs `tc qdisc add dev <nic> root handle 1: htb default
// <defaultMinor>`, installing the root HTB qdisc whose unmatched traffic
// falls to class 1:<defaultMinor>.
QdiscAddRoot(ctx context.Context, nic, defaultMinor string) error
// ClassAddRoot runs `tc class add dev <nic> parent 1: classid 1:1 htb rate
// <rate> ceil <ceil>`, the trunk class every per-class leaf attaches to.
ClassAddRoot(ctx context.Context, nic, rate, ceil string) error
// ClassAdd runs `tc class add dev <nic> parent 1:1 classid 1:<minor> htb
// rate <rate> ceil <ceil> prio <prio>`, a per-class leaf under the trunk.
ClassAdd(ctx context.Context, nic, minor, rate, ceil string, prio int) error
// QdiscAddFqCodel runs `tc qdisc add dev <nic> parent 1:<minor> handle
// <handle>: fq_codel`, the leaf qdisc for a class.
QdiscAddFqCodel(ctx context.Context, nic, minor, handle string) error
}
TCRunner abstracts live kernel tc qdisc/class operations for testability. The egress path drives ClassChange (live tuning of an already-installed hierarchy); the ingress path drives the qdisc/class add verbs to install the per-veth HTB hierarchy from scratch on each BN pod create.
Source Files