Documentation
¶
Overview ¶
Package storage watches the disk under the data directory and keeps the server from running it full (ADR-0481).
A full disk is where an event-sourced engine is most fragile. The WAL can be repaired — a write it could not finish is cut off again — but the state store cannot: Pebble ends the process when it fails to write its manifest, and a checkpoint, which is how WAL space is given back, needs room of its own to be taken. So the guard is a reserve, not a reaction. While less than the reserve is available, the server takes no new writes (the HTTP layer answers 507) and starts no work of its own; what is already running may finish, and reads go on. Once the reserve and a margin above it are free again, it resumes by itself, with nothing to restart.
The Monitor reads the file system on a fixed cadence, off the run loop: a reading is a single statfs or GetDiskFreeSpaceExW call whose cost does not depend on the data (ADR-0382). It also forecasts when the reserve will be reached, without keeping a history: each file system carries two numbers, a smoothed consumption rate and how long it has been observed. Prometheus users get the same readings as gauges and can ask predict_linear for a forecast over as long a window as their retention holds.
Index ¶
Constants ¶
const DefaultInterval = 5 * time.Second
DefaultInterval is how often the disk is read.
const DefaultReserve uint64 = 1 << 30
DefaultReserve is the room the server keeps free for itself: 1 GiB.
It is sized from what still has to fit after the server stops taking writes. A state-store flush writes out its two 16 MiB memtables; a compaction writes its output before deleting its input; the manifest Pebble rewrites can reach 128 MiB; a checkpoint copies that manifest and Pebble's log beside the files it links; the WAL may roll a fresh segment; and the requests already admitted when the reading was taken still land. That adds up to roughly half a gibibyte. The default is twice that, because the cost of an overestimate is a server that pauses early and the cost of an underestimate is Pebble ending the process. It is an estimate from those sizes, not a measurement.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Config ¶
type Config struct {
// Dirs are the directories the server writes to. Several on one file system are
// read as one.
Dirs []string
// Reserve is the space kept free; 0 turns the freeze off and leaves the warnings.
Reserve uint64
// Read reads a file system; diskspace.Of unless a test supplies one.
Read func(dir string) (diskspace.Usage, error)
// Now is the clock readings are timed by; time.Now unless a test supplies one.
Now func() time.Time
// Halted reports why the engine stopped, or nil while it runs.
Halted func() error
// Footprint reports what Atlas holds on disk. Called on every reading, off the
// run loop, so it must be safe from any goroutine.
Footprint func() Footprint
}
Config is what a Monitor is built from. Only Dirs is required.
type FileSystem ¶
type FileSystem struct {
// Paths are the directories of the server's that live on it.
Paths []string `json:"paths"`
TotalBytes uint64 `json:"totalBytes"`
UsedBytes uint64 `json:"usedBytes"`
AvailableBytes uint64 `json:"availableBytes"`
// UsedPercent is df's Use%: used / (used + available), to one decimal.
UsedPercent float64 `json:"usedPercent"`
// WarnBelowBytes is the available space under which this file system warns: the
// reserve plus the larger of the reserve and a tenth of used + available.
WarnBelowBytes uint64 `json:"warnBelowBytes"`
// Decides marks the file system with the least room — the one that freezes the
// server first.
Decides bool `json:"decides"`
Forecast *Forecast `json:"forecast,omitempty"`
// contains filtered or unexported fields
}
FileSystem is one file system the server writes to.
type Footprint ¶
type Footprint struct {
WALBytes int64 `json:"walBytes"`
WALSegments int `json:"walSegments"`
StateBytes uint64 `json:"stateBytes"`
}
Footprint is what Atlas itself holds on disk, as far as it can tell without walking a directory: the WAL's segment sizes and the state store's own account.
type Forecast ¶
type Forecast struct {
// BytesPerSecond is the smoothed net consumption: positive while the disk fills,
// negative while something — retention, compaction — gives back more than is
// written.
BytesPerSecond float64 `json:"bytesPerSecond"`
BytesPerDay float64 `json:"bytesPerDay"`
// SecondsUntilFreeze is how long until less than the reserve is available at
// that rate; absent while nothing grows.
SecondsUntilFreeze *float64 `json:"secondsUntilFreeze,omitempty"`
// BasisSeconds is how much observation the rate rests on. It starts over when the
// server does: nothing of it is stored.
BasisSeconds float64 `json:"basisSeconds"`
}
Forecast is the consumption the monitor has measured and where it leads.
type Monitor ¶
type Monitor struct {
// contains filtered or unexported fields
}
Monitor reads the disk and decides whether the server may take writes.
func (*Monitor) Frozen ¶
Frozen reports whether new writes are refused. Cheap enough for every request.
func (*Monitor) HandleGet ¶
func (m *Monitor) HandleGet(w http.ResponseWriter, _ *http.Request)
HandleGet serves GET /api/v1/storage: the last reading, what it means, and the forecast. It reads nothing itself — the monitor's loop does — so it costs the same however full the disk is.
func (*Monitor) Run ¶
Run reads the disk every interval until quit is closed, starting at once so the first request already has a reading.
type Snapshot ¶
type Snapshot struct {
State State `json:"state"`
Reason string `json:"reason"`
// SampledAt is when the file systems were last read successfully; null before the
// first reading.
SampledAt *time.Time `json:"sampledAt"`
ReserveBytes uint64 `json:"reserveBytes"`
ResumeBytes uint64 `json:"resumeBytes"`
FileSystems []FileSystem `json:"fileSystems"`
// Forecast is the soonest any file system reaches the reserve.
Forecast *Forecast `json:"forecast,omitempty"`
Atlas Footprint `json:"atlas"`
EngineHalted string `json:"engineHalted,omitempty"`
Error string `json:"error,omitempty"`
}
Snapshot is the monitor's last conclusion, as GET /api/v1/storage serves it.
type State ¶
type State string
State is what the monitor concludes about the disk.
const ( // StateOK is plenty of room and no forecast of trouble within a week. StateOK State = "ok" // StateWarn is room running low, or a forecast that reaches the reserve within a // week. Nothing is refused yet. StateWarn State = "warn" // StateFrozen is less than the reserve available: new writes are refused. StateFrozen State = "frozen" // StateHalted is the engine stopped after a batch could not be made durable. It // needs a restart, which is why it outranks every other state. StateHalted State = "halted" // StateUnknown is no usable reading: none taken yet, or the last one failed. StateUnknown State = "unknown" )
Source Files