bandwidth

package
v1.10.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 21, 2026 License: Apache-2.0 Imports: 14 Imported by: 0

Documentation

Overview

Package bandwidth is the bandwidth protection of the laboratory: how much a node can carry, how that is divided between the entry (demux and L7 proxy), the internet (the gateways) and the Geneve fabric, and the token buckets the pods use to keep to it.

The numbers: U = the bandwidth of the node (label) - the system reserve - the reserves of the platform pods on that node. Of U the entry gets EntryShare percent, the internet InternetShare, Geneve GeneveShare. Inside a class everyone gets a fair share, never more than the cap of its kind (a participant, a lab's internet, a lab's Geneve). Nothing inside a node is limited.

The operator computes the limits when something changes and writes them as annotations; the pods apply them (nothing here polls).

Index

Constants

View Source
const (
	AnnotationsDir  = "/etc/podinfo"
	AnnotationsFile = AnnotationsDir + "/annotations"
)

AnnotationsDir and AnnotationsFile are where the pod's own annotations are mounted (a downward API volume).

View Source
const ConfigMapDir = "/etc/bandwidth"

ConfigMapDir is where the chart mounts the ConfigMap of the entry limits (names.BandwidthConfigMap) into the demux and the L7 proxy.

View Source
const DefaultMaxKeys = 65536

DefaultMaxKeys bounds the participants a Throttle tracks. The keys come from sessions and clients the caller has accepted, but a bound is still what keeps the memory and the rescans of a throttle from growing with whatever reaches it; the participants past it share one bucket.

View Source
const Version = 1

Version is the format of the limits annotation. A reader ignores a version it does not know (no limits), never guesses.

Variables

This section is empty.

Functions

func FairShare

func FairShare(classKbps, capKbps int64, active int) int64

FairShare is what each of n active users of a class of classKbps gets: an equal part, never more than capKbps. A class or cap of 0 is no limit of that kind (class 0 and cap 0 together: 0, which means unlimited to the callers).

func FairShareFloor added in v1.10.0

func FairShareFloor(classKbps, capKbps, floorKbps int64, active int) int64

FairShareFloor is FairShare with a lower bound for one participant: a share below floorKbps becomes floorKbps, but never more than capKbps (the ceiling wins over the floor). A floor of 0 is FairShare; an unlimited result (class 0 and cap 0) stays unlimited. With a floor the shares of all the active participants together may pass the class.

func LabGeneveRate

func LabGeneveRate(classKbps, labCapKbps int64, splitLabs, vteps int) int64

LabGeneveRate is LabGeneveKbps from the class and the cap in kbit/s, which is all the node-agent is given.

func NodeGeneveKbps

func NodeGeneveKbps(n NodeLimits, maxVTEPs int) int64

NodeGeneveKbps is the rate of the node meter, the one every split lab goes through: the class divided by the most remote nodes any lab floods to.

func NodeMbit

func NodeMbit(node *corev1.Node) int64

NodeMbit is the bandwidth a node's label says, in Mbit/s; 0 when it has none or it cannot be read.

func ParseAnnotations

func ParseAnnotations(data string) map[string]string

ParseAnnotations reads the downward API file: one `key="value"` per line, the value in Go quoting.

func ParseMbit

func ParseMbit(s string) (int64, bool)

ParseMbit reads a bandwidth: a number of Mbit/s with an optional trailing M, or G for Gbit/s ("1562", "1562M", "10G"). ok is false for an empty or unreadable value, a zero or a negative one.

func ReserveMbit

func ReserveMbit(pod *corev1.Pod) int64

ReserveMbit is what a platform pod keeps for itself, in Mbit/s (0 when it declares nothing or the value is not a number).

func Watch

func Watch(ctx context.Context, path string, apply func(Limits), onError func(error)) error

Watch calls apply with the limits at the start and again each time they change, for a pod that reads them from its own annotations (the file AnnotationsFile): the gateway. A bad value calls onError and keeps the last good limits. It returns when ctx ends.

func WatchNode

func WatchNode(ctx context.Context, dir, node string, apply func(Limits), onError func(error)) error

WatchNode is Watch for a pod that reads the limits of its own node from the mounted ConfigMap: the file dir/node.

Types

type Bucket

type Bucket struct {
	// contains filtered or unexported fields
}

Bucket is a token bucket counted in bytes whose rate can change at any time. The zero value is unlimited.

func (*Bucket) Allow

func (b *Bucket) Allow(now time.Time, n int) bool

Allow takes n bytes if they are there and says whether it did (a packet that does not fit is dropped).

func (*Bucket) Limited

func (b *Bucket) Limited() bool

Limited says the bucket has a rate.

func (*Bucket) Prime

func (b *Bucket) Prime(now time.Time, tokens float64)

Prime sets the tokens a bucket starts with (a newcomer gets a small burst, not a full one, so a flood of fresh keys cannot each spend a full burst).

func (*Bucket) Reserve

func (b *Bucket) Reserve(now time.Time, n int) time.Duration

Reserve takes n bytes and says how long the caller has to wait before it may send them (a stream is paced, not dropped). The bucket may go into debt, so one write larger than the burst is still paced to the rate.

func (*Bucket) SetRate

func (b *Bucket) SetRate(kbps int64)

SetRate changes the rate; the tokens kept stay (cut to the new burst). kbps 0 = unlimited.

type Config

type Config struct {
	// DefaultMbit is the bandwidth of a node without the label.
	DefaultMbit int64
	// SystemReserveMbit is taken off every node: the kubelet, the control plane traffic, image pulls.
	SystemReserveMbit int64
	// EntryShare, InternetShare and GeneveShare are the percent of U each class gets (their sum is at most 100).
	EntryShare, InternetShare, GeneveShare int64
	// ParticipantCapMbit, LabInternetCapMbit and LabGeneveCapMbit are the most one participant, one lab's internet and one lab's
	// Geneve traffic may take, whatever is free.
	ParticipantCapMbit, LabInternetCapMbit, LabGeneveCapMbit int64
	// ParticipantMinKbps is the lower bound of one participant's share (0 = none): with very many active participants a person still gets
	// this much, so a console session keeps working. It never lifts a share above the ceiling (a floor above the ceiling is clamped to it),
	// so the sum of the shares may pass the entry class. It does not apply to the lab internet or Geneve shares.
	ParticipantMinKbps int64
	// LabUploadKbps is the most one lab may send to the internet (payload; the ACKs of what it downloads are not counted). UploadTotalMbit is the
	// most all the labs of the cluster together may send; a lab's upload is scaled down when they would pass it (0 = no total).
	LabUploadKbps, UploadTotalMbit int64
	// GeneveOverhead is the share of the Geneve class an inner frame may use: the OVS meter counts the frame before the
	// encapsulation, and the outer headers (about 50 bytes) come on top. 0.85 leaves room for packets down to a few hundred bytes.
	GeneveOverhead float64
	// EntryOverhead is the share of the entry class that payload may use on the way to a VPN pod: the VXLAN of the cluster network adds
	// about 50 bytes to every packet the demux and the L7 proxy forward to another node.
	EntryOverhead float64
}

Config is the chart's bandwidth section.

func DefaultConfig

func DefaultConfig() Config

DefaultConfig mirrors the chart values.

func (Config) Compute

func (c Config) Compute(bandwidthMbit int64, reservesMbit []int64) NodeLimits

Compute divides a node: bandwidth is the label's value in Mbit/s (0 = the default), reserves the reserves of the platform pods that run on it in Mbit/s.

func (Config) EntryClassKbps

func (c Config) EntryClassKbps(minEntryKbps int64, senders int) int64

EntryClassKbps is the entry class one demux (or L7 proxy) may use, given the smallest entry class of the nodes it can send to and the number of nodes that can send. The demux of every lab node may forward to a VPN pod on one node, so each gets a 1/senders share of the half of the receiving node's entry class that is for the demux, scaled by the overhead factor: all of them sending to one node together stay within what that node was given. senders < 1 counts as one.

func (Config) ForNode

func (c Config) ForNode(node *corev1.Node, pods []corev1.Pod) NodeLimits

ForNode computes a node: its label, and the reserves of the pods that run on it (pods of other nodes are ignored, and so are pods that are over).

func (Config) LabGeneveKbps

func (c Config) LabGeneveKbps(n NodeLimits, splitLabs, vteps int) int64

LabGeneveKbps is the rate of the OVS meter of one lab that is split between nodes, when splitLabs such labs are on the node and the lab's frames go to vteps remote nodes (a flooded frame is copied to each, the meter counts it once, so the rate is divided).

func (Config) LabInternetKbps

func (c Config) LabInternetKbps(n NodeLimits, labs int) int64

LabInternetKbps is the ceiling of one lab's internet traffic when labs labs share the internet class of the node.

func (Config) LabUploadKbpsFor added in v1.9.0

func (c Config) LabUploadKbpsFor(labs int) int64

LabUploadKbpsFor is the upload ceiling of one lab when labs labs of the cluster use the internet: the lab cap, or the total divided by the labs when that is less (never below 64 kbit/s: a lab always keeps a trickle). 0 means no limit.

func (Config) NodeGeneveLimits

func (c Config) NodeGeneveLimits(n NodeLimits) Limits

NodeGeneveLimits are the limits the node-agent applies for the Geneve traffic of a node.

func (Config) Validate

func (c Config) Validate() error

Validate checks the shares and caps.

type Limits

type Limits struct {
	V          int   `json:"v"`
	ClassKbps  int64 `json:"classKbps,omitempty"`
	CapKbps    int64 `json:"capKbps,omitempty"`
	MinKbps    int64 `json:"minKbps,omitempty"`
	GeneveKbps int64 `json:"geneveKbps,omitempty"`
	LabCapKbps int64 `json:"labCapKbps,omitempty"`
	UpKbps     int64 `json:"upKbps,omitempty"`
}

Limits is the content of the limits annotation (names.AnnotationBandwidthLimits). Which fields count depends on the object:

node:                       GeneveKbps (the class, already scaled), LabCapKbps (the ceiling of one lab's Geneve traffic)
demux, L7 proxy pod:        ClassKbps (the part of the entry class of this component), CapKbps (the ceiling of one participant), MinKbps (its floor)
gateway pod:                CapKbps (the ceiling of one lab's internet traffic, the fair share is already in it), UpKbps (the ceiling of what
                            one lab sends to the internet: the lab cap, scaled down when the labs of the cluster together would pass the total)

func Decode

func Decode(s string) (Limits, error)

Decode reads an annotation value. An empty value is no limits and no error; a value of another version is an error.

func ReadLimits

func ReadLimits(path string) (Limits, error)

ReadLimits reads the limits annotation of the pod from the file; a missing file or annotation is no limits.

func ReadLimitsFile

func ReadLimitsFile(path string) (Limits, error)

ReadLimitsFile reads limits written as JSON in a file (a key of the mounted ConfigMap); a missing file is no limits.

func (Limits) Encode

func (l Limits) Encode() string

Encode writes the annotation value.

func (Limits) Zero

func (l Limits) Zero() bool

Zero says there is nothing to apply.

type NodeLimits

type NodeLimits struct {
	// UKbps is the bandwidth left for labs and participants on the node.
	UKbps int64
	// EntryKbps, InternetKbps and GeneveKbps are the classes; GeneveKbps is already scaled by the overhead factor.
	EntryKbps, InternetKbps, GeneveKbps int64
}

NodeLimits are the classes of one node in kbit/s, for each direction (the same number both ways).

type Settings

type Settings struct {
	// Enabled turns the whole protection on. Off: nothing is limited and nothing is written.
	Enabled            bool    `env:"BANDWIDTH_ENABLED" envDefault:"false"`
	DefaultMbit        int64   `env:"BANDWIDTH_DEFAULT_MBIT" envDefault:"1000"`
	SystemReserveMbit  int64   `env:"BANDWIDTH_SYSTEM_RESERVE_MBIT" envDefault:"100"`
	EntryShare         int64   `env:"BANDWIDTH_ENTRY_SHARE" envDefault:"50"`
	InternetShare      int64   `env:"BANDWIDTH_INTERNET_SHARE" envDefault:"25"`
	GeneveShare        int64   `env:"BANDWIDTH_GENEVE_SHARE" envDefault:"25"`
	ParticipantCapMbit int64   `env:"BANDWIDTH_PARTICIPANT_CAP_MBIT" envDefault:"20"`
	ParticipantMinKbps int64   `env:"BANDWIDTH_PARTICIPANT_MIN_KBPS" envDefault:"1000"`
	LabInternetCapMbit int64   `env:"BANDWIDTH_LAB_INTERNET_CAP_MBIT" envDefault:"50"`
	LabGeneveCapMbit   int64   `env:"BANDWIDTH_LAB_GENEVE_CAP_MBIT" envDefault:"100"`
	LabUploadKbps      int64   `env:"BANDWIDTH_LAB_UPLOAD_KBPS" envDefault:"2048"`
	UploadTotalMbit    int64   `env:"BANDWIDTH_UPLOAD_TOTAL_MBIT" envDefault:"100"`
	GeneveOverhead     float64 `env:"BANDWIDTH_GENEVE_OVERHEAD_FACTOR" envDefault:"0.85"`
	EntryOverhead      float64 `env:"BANDWIDTH_ENTRY_OVERHEAD_FACTOR" envDefault:"0.85"`
}

Settings are the chart's bandwidth values as the operator and the node-agent read them from the environment.

func (Settings) Config

func (s Settings) Config() Config

Config is the Settings as a Config.

type Throttle

type Throttle[K comparable] struct {
	// contains filtered or unexported fields
}

Throttle keeps participants to their fair share of a class: every participant has a bucket of its own, all of them also take from one bucket of the class, and the rate of each is the class divided by the number of participants that were active lately, never more than the cap. Participants are named by a key (a session, a client); one that sends nothing for the window stops counting and is forgotten later.

func NewThrottle

func NewThrottle[K comparable](window time.Duration) *Throttle[K]

NewThrottle makes a throttle with no limits; SetLimits turns it on. window is how long a silent participant still counts as active (5 s when zero).

func (*Throttle[K]) Active

func (t *Throttle[K]) Active() int

Active is how many participants counted as active at the last scan.

func (*Throttle[K]) Allow

func (t *Throttle[K]) Allow(key K, n int) bool

Allow takes n bytes of a participant: false when its bucket or the class is empty (the packet is dropped). Nothing is taken from the class when the participant's own bucket refuses.

func (*Throttle[K]) Delay

func (t *Throttle[K]) Delay(key K, n int) time.Duration

Delay takes n bytes of a participant and returns how long the sender has to wait (the stream is paced). The longer of the participant's and the class's wait counts.

func (*Throttle[K]) Enabled

func (t *Throttle[K]) Enabled() bool

Enabled says some limit is set.

func (*Throttle[K]) Rate

func (t *Throttle[K]) Rate() int64

Rate is the rate of one participant now, in kbit/s (0 = unlimited).

func (*Throttle[K]) SetClock

func (t *Throttle[K]) SetClock(now func() time.Time)

SetClock replaces the clock (tests move it by hand).

func (*Throttle[K]) SetFloor added in v1.10.0

func (t *Throttle[K]) SetFloor(minKbps int64)

SetFloor sets the lower bound of one participant's share in kbit/s (0 = none); call it before SetLimits, which applies it.

func (*Throttle[K]) SetLimits

func (t *Throttle[K]) SetLimits(classKbps, capKbps int64)

SetLimits sets the class and the cap of one participant in kbit/s; 0 is no limit of that kind. Both 0 turns the throttle off.

func (*Throttle[K]) SetMaxKeys

func (t *Throttle[K]) SetMaxKeys(n int)

SetMaxKeys changes the bound on the participants tracked (the rest share one bucket).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL