Documentation
¶
Overview ¶
Package bandwidth is the bandwidth protection of the laboratory: how much a node can carry, how that is divided between the entry (demux and L7 proxy), the internet (the gateways) and the Geneve fabric, and the token buckets the pods use to keep to it.
The numbers: U = the bandwidth of the node (label) - the system reserve - the reserves of the platform pods on that node. Of U the entry gets EntryShare percent, the internet InternetShare, Geneve GeneveShare. Inside a class everyone gets a fair share, never more than the cap of its kind (a participant, a lab's internet, a lab's Geneve). Nothing inside a node is limited.
The operator computes the limits when something changes and writes them as annotations; the pods apply them (nothing here polls).
Index ¶
- Constants
- func FairShare(classKbps, capKbps int64, active int) int64
- func LabGeneveRate(classKbps, labCapKbps int64, splitLabs, vteps int) int64
- func NodeGeneveKbps(n NodeLimits, maxVTEPs int) int64
- func NodeMbit(node *corev1.Node) int64
- func ParseAnnotations(data string) map[string]string
- func ParseMbit(s string) (int64, bool)
- func ReserveMbit(pod *corev1.Pod) int64
- func Watch(ctx context.Context, path string, apply func(Limits), onError func(error)) error
- func WatchNode(ctx context.Context, dir, node string, apply func(Limits), onError func(error)) error
- type Bucket
- type Config
- func (c Config) Compute(bandwidthMbit int64, reservesMbit []int64) NodeLimits
- func (c Config) EntryClassKbps(minEntryKbps int64, senders int) int64
- func (c Config) ForNode(node *corev1.Node, pods []corev1.Pod) NodeLimits
- func (c Config) LabGeneveKbps(n NodeLimits, splitLabs, vteps int) int64
- func (c Config) LabInternetKbps(n NodeLimits, labs int) int64
- func (c Config) LabUploadKbpsFor(labs int) int64
- func (c Config) NodeGeneveLimits(n NodeLimits) Limits
- func (c Config) Validate() error
- type Limits
- type NodeLimits
- type Settings
- type Throttle
- func (t *Throttle[K]) Active() int
- func (t *Throttle[K]) Allow(key K, n int) bool
- func (t *Throttle[K]) Delay(key K, n int) time.Duration
- func (t *Throttle[K]) Enabled() bool
- func (t *Throttle[K]) Rate() int64
- func (t *Throttle[K]) SetClock(now func() time.Time)
- func (t *Throttle[K]) SetLimits(classKbps, capKbps int64)
- func (t *Throttle[K]) SetMaxKeys(n int)
Constants ¶
const ( AnnotationsDir = "/etc/podinfo" AnnotationsFile = AnnotationsDir + "/annotations" )
AnnotationsDir and AnnotationsFile are where the pod's own annotations are mounted (a downward API volume).
const ConfigMapDir = "/etc/bandwidth"
ConfigMapDir is where the chart mounts the ConfigMap of the entry limits (names.BandwidthConfigMap) into the demux and the L7 proxy.
const DefaultMaxKeys = 65536
DefaultMaxKeys bounds the participants a Throttle tracks. The keys come from sessions and clients the caller has accepted, but a bound is still what keeps the memory and the rescans of a throttle from growing with whatever reaches it; the participants past it share one bucket.
const Version = 1
Version is the format of the limits annotation. A reader ignores a version it does not know (no limits), never guesses.
Variables ¶
This section is empty.
Functions ¶
func FairShare ¶
FairShare is what each of n active users of a class of classKbps gets: an equal part, never more than capKbps. A class or cap of 0 is no limit of that kind (class 0 and cap 0 together: 0, which means unlimited to the callers).
func LabGeneveRate ¶
LabGeneveRate is LabGeneveKbps from the class and the cap in kbit/s, which is all the node-agent is given.
func NodeGeneveKbps ¶
func NodeGeneveKbps(n NodeLimits, maxVTEPs int) int64
NodeGeneveKbps is the rate of the node meter, the one every split lab goes through: the class divided by the most remote nodes any lab floods to.
func NodeMbit ¶
NodeMbit is the bandwidth a node's label says, in Mbit/s; 0 when it has none or it cannot be read.
func ParseAnnotations ¶
ParseAnnotations reads the downward API file: one `key="value"` per line, the value in Go quoting.
func ParseMbit ¶
ParseMbit reads a bandwidth: a number of Mbit/s with an optional trailing M, or G for Gbit/s ("1562", "1562M", "10G"). ok is false for an empty or unreadable value, a zero or a negative one.
func ReserveMbit ¶
ReserveMbit is what a platform pod keeps for itself, in Mbit/s (0 when it declares nothing or the value is not a number).
Types ¶
type Bucket ¶
type Bucket struct {
// contains filtered or unexported fields
}
Bucket is a token bucket counted in bytes whose rate can change at any time. The zero value is unlimited.
func (*Bucket) Allow ¶
Allow takes n bytes if they are there and says whether it did (a packet that does not fit is dropped).
func (*Bucket) Prime ¶
Prime sets the tokens a bucket starts with (a newcomer gets a small burst, not a full one, so a flood of fresh keys cannot each spend a full burst).
type Config ¶
type Config struct {
// DefaultMbit is the bandwidth of a node without the label.
DefaultMbit int64
// SystemReserveMbit is taken off every node: the kubelet, the control plane traffic, image pulls.
SystemReserveMbit int64
Config is the chart's bandwidth section.
func (Config) Compute ¶
func (c Config) Compute(bandwidthMbit int64, reservesMbit []int64) NodeLimits
Compute divides a node: bandwidth is the label's value in Mbit/s (0 = the default), reserves the reserves of the platform pods that run on it in Mbit/s.
func (Config) EntryClassKbps ¶
EntryClassKbps is the entry class one demux (or L7 proxy) may use, given the smallest entry class of the nodes it can send to and the number of nodes that can send. The demux of every lab node may forward to a VPN pod on one node, so each gets a 1/senders share of the half of the receiving node's entry class that is for the demux, scaled by the overhead factor: all of them sending to one node together stay within what that node was given. senders < 1 counts as one.
func (Config) ForNode ¶
ForNode computes a node: its label, and the reserves of the pods that run on it (pods of other nodes are ignored, and so are pods that are over).
func (Config) LabGeneveKbps ¶
func (c Config) LabGeneveKbps(n NodeLimits, splitLabs, vteps int) int64
LabGeneveKbps is the rate of the OVS meter of one lab that is split between nodes, when splitLabs such labs are on the node and the lab's frames go to vteps remote nodes (a flooded frame is copied to each, the meter counts it once, so the rate is divided).
func (Config) LabInternetKbps ¶
func (c Config) LabInternetKbps(n NodeLimits, labs int) int64
LabInternetKbps is the ceiling of one lab's internet traffic when labs labs share the internet class of the node.
func (Config) LabUploadKbpsFor ¶ added in v1.9.0
LabUploadKbpsFor is the upload ceiling of one lab when labs labs of the cluster use the internet: the lab cap, or the total divided by the labs when that is less (never below 64 kbit/s: a lab always keeps a trickle). 0 means no limit.
func (Config) NodeGeneveLimits ¶
func (c Config) NodeGeneveLimits(n NodeLimits) Limits
NodeGeneveLimits are the limits the node-agent applies for the Geneve traffic of a node.
type Limits ¶
type Limits struct {
V int `json:"v"`
ClassKbps int64 `json:"classKbps,omitempty"`
CapKbps int64 `json:"capKbps,omitempty"`
GeneveKbps int64 `json:"geneveKbps,omitempty"`
LabCapKbps int64 `json:"labCapKbps,omitempty"`
UpKbps int64 `json:"upKbps,omitempty"`
}
Limits is the content of the limits annotation (names.AnnotationBandwidthLimits). Which fields count depends on the object:
node: GeneveKbps (the class, already scaled), LabCapKbps (the ceiling of one lab's Geneve traffic)
demux, L7 proxy pod: ClassKbps (the part of the entry class of this component), CapKbps (the ceiling of one participant)
gateway pod: CapKbps (the ceiling of one lab's internet traffic, the fair share is already in it), UpKbps (the ceiling of what
one lab sends to the internet: the lab cap, scaled down when the labs of the cluster together would pass the total)
func Decode ¶
Decode reads an annotation value. An empty value is no limits and no error; a value of another version is an error.
func ReadLimits ¶
ReadLimits reads the limits annotation of the pod from the file; a missing file or annotation is no limits.
func ReadLimitsFile ¶
ReadLimitsFile reads limits written as JSON in a file (a key of the mounted ConfigMap); a missing file is no limits.
type NodeLimits ¶
type NodeLimits struct {
// UKbps is the bandwidth left for labs and participants on the node.
UKbps int64
// EntryKbps, InternetKbps and GeneveKbps are the classes; GeneveKbps is already scaled by the overhead factor.
EntryKbps, InternetKbps, GeneveKbps int64
}
NodeLimits are the classes of one node in kbit/s, for each direction (the same number both ways).
type Settings ¶
type Settings struct {
// Enabled turns the whole protection on. Off: nothing is limited and nothing is written.
Enabled bool `env:"BANDWIDTH_ENABLED" envDefault:"false"`
DefaultMbit int64 `env:"BANDWIDTH_DEFAULT_MBIT" envDefault:"1000"`
SystemReserveMbit int64 `env:"BANDWIDTH_SYSTEM_RESERVE_MBIT" envDefault:"100"`
ParticipantCapMbit int64 `env:"BANDWIDTH_PARTICIPANT_CAP_MBIT" envDefault:"50"`
LabInternetCapMbit int64 `env:"BANDWIDTH_LAB_INTERNET_CAP_MBIT" envDefault:"50"`
LabGeneveCapMbit int64 `env:"BANDWIDTH_LAB_GENEVE_CAP_MBIT" envDefault:"100"`
LabUploadKbps int64 `env:"BANDWIDTH_LAB_UPLOAD_KBPS" envDefault:"2048"`
UploadTotalMbit int64 `env:"BANDWIDTH_UPLOAD_TOTAL_MBIT" envDefault:"100"`
GeneveOverhead float64 `env:"BANDWIDTH_GENEVE_OVERHEAD_FACTOR" envDefault:"0.85"`
EntryOverhead float64 `env:"BANDWIDTH_ENTRY_OVERHEAD_FACTOR" envDefault:"0.85"`
}
Settings are the chart's bandwidth values as the operator and the node-agent read them from the environment.
type Throttle ¶
type Throttle[K comparable] struct { // contains filtered or unexported fields }
Throttle keeps participants to their fair share of a class: every participant has a bucket of its own, all of them also take from one bucket of the class, and the rate of each is the class divided by the number of participants that were active lately, never more than the cap. Participants are named by a key (a session, a client); one that sends nothing for the window stops counting and is forgotten later.
func NewThrottle ¶
func NewThrottle[K comparable](window time.Duration) *Throttle[K]
NewThrottle makes a throttle with no limits; SetLimits turns it on. window is how long a silent participant still counts as active (5 s when zero).
func (*Throttle[K]) Allow ¶
Allow takes n bytes of a participant: false when its bucket or the class is empty (the packet is dropped). Nothing is taken from the class when the participant's own bucket refuses.
func (*Throttle[K]) Delay ¶
Delay takes n bytes of a participant and returns how long the sender has to wait (the stream is paced). The longer of the participant's and the class's wait counts.
func (*Throttle[K]) SetLimits ¶
SetLimits sets the class and the cap of one participant in kbit/s; 0 is no limit of that kind. Both 0 turns the throttle off.
func (*Throttle[K]) SetMaxKeys ¶
SetMaxKeys changes the bound on the participants tracked (the rest share one bucket).