Documentation
¶
Overview ¶
Package throughput is the process-wide answer to one question: "how fast is this storage right now?"
It exists because two features need the same number and must not each grow their own: the operator metrics (`filex_storage_throughput_bytes_per_second`) and internal/filecache, which decides whether an object is worth caching because the storage behind it is slow. A cache that measured slowness its own way would disagree with the dashboard the operator is looking at, and the first "the graph says it is fine but the cache keeps kicking in" bug would cost a day.
The API filecache should use ¶
throughput.Observe(storageID, throughput.Read, n, elapsed) // record
bps, ok := throughput.Rate(storageID, throughput.Read) // read back
if throughput.IsSlow(storageID, throughput.Read) { … } // the decision
filecache does not in fact use IsSlow: its policy is more than one comparison (it lets a fast measurement overrule an operator's `slow: true` flag, and it refuses to act on thin evidence), so it asks StatAbove for the rate AND the sample count and applies that policy itself. The division is the point: the MEASUREMENT lives here and has one implementation; the POLICY lives in filecache, where it is not duplicated.
`ok == false` means "no measurement yet". That is deliberately NOT folded into IsSlow's answer: an unmeasured storage is reported as not-slow, because treating silence as slowness would make every fresh boot behave as if every backend were a NAS over a phone line. A caller that wants to distinguish the two asks Rate.
What it measures ¶
A rolling window (Window, default 2 min / MaxSamples samples) of (bytes, duration) observations per storage and direction; the rate is sum(bytes)/sum(duration) over the window, i.e. a byte-weighted average, so a handful of tiny reads cannot drag the number around the way an average-of-rates would.
Samples below MinSample bytes are ignored: the elapsed time of a 200-byte read is dominated by round-trip latency and says nothing about throughput. A caller that needs a stronger bar than MinSample passes its own floor to StatAbove rather than keeping a second set of samples.
A read is timed by the time spent inside Read, not by wall clock across the transfer — see CountingReader for why that distinction decides whether the number describes the backend or the person downloading from it.
Index ¶
- Constants
- Variables
- func CountingReader(rc io.ReadCloser, storageID int64, dir Direction) io.ReadCloser
- func IsSlow(storageID int64, dir Direction) bool
- func IsSlowerThan(storageID int64, dir Direction, bytesPerSec float64) bool
- func Observe(storageID int64, dir Direction, bytes int64, d time.Duration)
- func Rate(storageID int64, dir Direction) (float64, bool)
- func Subscribe(o Observer)
- type Direction
- type Meter
- func (m *Meter) IsSlow(storageID int64, dir Direction) bool
- func (m *Meter) IsSlowerThan(storageID int64, dir Direction, bytesPerSec float64) bool
- func (m *Meter) Observe(storageID int64, dir Direction, bytes int64, d time.Duration)
- func (m *Meter) Rate(storageID int64, dir Direction) (float64, bool)
- func (m *Meter) Snapshot() []Stat
- func (m *Meter) StatAbove(storageID int64, dir Direction, minBytes int64) (Stat, bool)
- type Observer
- type Stat
Constants ¶
const ( // Window is how far back a sample still counts. Window = 2 * time.Minute // MaxSamples caps memory per storage+direction. MaxSamples = 64 // MinSample is the smallest transfer that says anything about throughput. MinSample = 64 << 10 // 64 KiB // SlowBytesPerSec is the default "this storage is slow" line: 8 MB/s, // roughly where a 100 MB file stops being instant and starts being a // progress bar. filecache may pass its own threshold to IsSlowerThan. SlowBytesPerSec = 8 << 20 )
Tuning constants. Exported so a caller can reason about them (and so a test can state which one it is exercising) rather than guess.
Variables ¶
var Default = New()
Default is the process-wide meter. Every caller shares it on purpose.
Functions ¶
func CountingReader ¶
func CountingReader(rc io.ReadCloser, storageID int64, dir Direction) io.ReadCloser
CountingReader wraps rc so every byte read is observed against storageID. The sample is recorded on Close — one observation per transfer rather than one per Read call, which is what makes the rate byte-weighted.
This is how the download path is measured: filebody.Source hands back the wrapped reader, so every surface that streams a file (browser, WebDAV, share link, ShareX, the desktop app) feeds the same meter without knowing.
⚠ The duration recorded is the time spent INSIDE rc.Read, not wall-clock across the transfer, and that difference is the whole validity of the measurement. A body being copied to a client is paced by the client: wall clock would measure the slower of {backend, browser}, so one user on hotel wifi would mark a fast bucket slow for everybody — and internal/filecache would then answer the next person "preparing…" for a file that would have streamed instantly. Where read-time is still wrong it is wrong in the safe direction: buffered bytes return instantly, so the estimate reads FAST, which means fewer files qualify for the cache, never more.
func IsSlow ¶
IsSlow asks the process-wide meter. This is the call internal/filecache should make; do not re-measure.
func IsSlowerThan ¶
IsSlowerThan asks the process-wide meter with a caller-chosen threshold.
func Observe ¶
Observe records one transfer on the process-wide meter and notifies every subscriber.
⚠ The MinSample rule applies to the RATE, not to the notification. The meter drops a small transfer because its elapsed time is round-trip latency rather than throughput; a bytes-total that quietly ignored every small file would simply be wrong, and "we moved 4 GB today" is a question the counter has to be able to answer.
func Subscribe ¶
func Subscribe(o Observer)
Subscribe registers a sink for every transfer recorded through the package-level Observe (and therefore through CountingReader).
This exists so ONE call at the transfer site feeds both consumers: the rolling rate here, and the Prometheus counters in internal/metrics — which subscribes at init. Two calls at every site would eventually be one call at some site, and then the graph and the cache would be measuring different things. The dependency runs metrics → throughput; the hook is what keeps it from having to run both ways.
Types ¶
type Direction ¶
type Direction string
Direction distinguishes reads from writes — a backend can be fast one way and slow the other (S3 uploads vs. downloads over an asymmetric link).
type Meter ¶
type Meter struct {
// contains filtered or unexported fields
}
Meter holds the rolling samples. The zero value is not usable — use New, or the package-level Default.
func (*Meter) IsSlow ¶
IsSlow reports whether the storage is below SlowBytesPerSec. An unmeasured storage is NOT slow — see the package comment.
func (*Meter) IsSlowerThan ¶
IsSlowerThan is IsSlow with the caller's own threshold in bytes/sec.
func (*Meter) Rate ¶
Rate returns the rolling bytes/sec for one storage and direction. ok is false when there is nothing in the window — "unknown", never "zero".
func (*Meter) Snapshot ¶
Snapshot returns every measured storage+direction — the metrics exporter's input, and a readable thing to log when someone asks why a copy is crawling.
func (*Meter) StatAbove ¶
StatAbove is Rate plus the evidence behind it — how many samples, how many bytes, when the last one landed — computed from the samples of at least minBytes.
The floor is a parameter rather than a constant because a caller may need a stronger bar than the meter's own MinSample without growing a second meter to get it. internal/filecache is that caller: it will not let a measurement overrule an operator, or promote an unflagged storage, on the strength of a handful of small reads, so it asks with its own (much larger) floor. Same samples, same window, same arithmetic — one source, parametrised.
minBytes <= 0 means "every sample the meter kept". ok is false when nothing clears the floor: unknown, never zero.
Source Files