Documentation
¶
Overview ¶
Package execopt carries the execution-time knobs that are not planning decisions: how much memory one query may hold, and where it is allowed to spill.
Why a ledger rather than a counter ¶
Seven operators in ursus buffer, and every one of them retains the SOURCE's batches rather than copies of them. Columns share payloads on top of that — Rename shares one, so does a sliced String column's character buffer. So a counter that adds up what each operator was handed double-counts, and a budget that double-counts fires early and unpredictably, which is worse for a user than having no budget at all.
The ledger below therefore counts ALLOCATIONS, keyed by data.BufferID, with one reference count per allocation across every account. An allocation two sinks hold is one entry, and it is released when the last account lets go.
What it deliberately does not do ¶
It does not hook the allocator. arrowx pins memory.NewGoAllocator, whose Free is a no-op and which reports nothing, so there is no allocation-site seam to hook — and design/physical.md's proposed kernel.Ctx, which would have carried a reservation into every kernel, was never built. Accounting therefore happens where memory is RETAINED, not where it is allocated. That is exactly right for the buffering operators, which are the only things that hold memory across batches, and it is silent about a single kernel's transient output.
An account keeps alive what it holds ¶
A data.BufferID is a pointer to an allocation's first byte, and the ledger is keyed by it, so an allocation stays reachable while any account holds it. An operator that lets go of a buffer it will not read again, to free it, must release it from its account first: released afterwards, the account keeps it alive until then. A group-by assembling its answer and Collect assembling its result each held their input twice that way (step 131).
A result is held apart ¶
Collect holds its result in an account of its own, opened with Result, and only under the default budget. The ledger keeps the bytes only a result holds apart from what the operators hold, and the limit, Over and Peak see only the operators'. Charged with them, a result would put a spilled group-by's or join's replay over budget from its first partition, though it cannot be spilled: see Result.
Index ¶
- func Bytes(n int64) string
- func Ceiling() int64
- func DefaultLimit() int64
- type Account
- type Budget
- func (b *Budget) Account(op string) *Account
- func (b *Budget) Holding() int64
- func (b *Budget) IsDefault() bool
- func (b *Budget) Limit() int64
- func (b *Budget) MarkDefault()
- func (b *Budget) NoteSpill()
- func (b *Budget) OnAccount(fn func(*Account))
- func (b *Budget) Over() bool
- func (b *Budget) Peak() int64
- func (b *Budget) Result() *Account
- func (b *Budget) ResultLimit() int64
- func (b *Budget) SpillDir() string
- func (b *Budget) Spills() int64
- func (b *Budget) Used() int64
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Ceiling ¶ added in v0.4.0
func Ceiling() int64
Ceiling is the memory this process may use: the smaller of its cgroup's limit and the machine's RAM, or 0 where neither can be read. Detected once per process.
func DefaultLimit ¶ added in v0.3.1
func DefaultLimit() int64
DefaultLimit is the memory budget a query gets when its caller sets none: half of the smaller of the process's cgroup memory limit and the machine's RAM, or 0 — unlimited — where neither can be read.
Why a default at all ¶
Spilling is decided by the budget, and with no budget nothing spills. A cgroup's limit is enforced by the kernel from outside, and ursus never saw it: the v0.3.0 benchmark ran each query under an 8 GB cgroup, and ursus's h2o gb10 over CSV — a group-by that spills perfectly well under WithMemoryLimit — grew past it and was OOM-killed instead. Every container with a memory limit had the same exposure. DuckDB defaults its limit to 80% of RAM for the same reason.
Why half, and not DuckDB's 80% ¶
The budget counts live bytes, and ursus's buffers live on the Go heap, which with GOGC=100 grows to about twice its live size before collecting. Half the ceiling keeps the process's true peak under it. Set GOMEMLIMIT, or WithMemoryLimit, to choose differently; WithMemoryLimit(0) turns the budget off.
Detected once per process. On Linux from /proc and the cgroup files; on macOS and Windows the machine's RAM, from sysctl's hw.memsize and GlobalMemoryStatusEx (step 181, scope item R3: both had been unbudgeted, so nothing spilled unless the caller set a limit). Elsewhere nothing is readable, and the default stays unlimited.
Types ¶
type Account ¶
type Account struct {
// contains filtered or unexported fields
}
Account is one operator's share of a budget.
It holds each allocation at most once — retaining the same buffer twice does not hold more memory — so Retain is idempotent per allocation and Release is exact.
A nil *Account is a working no-op, which is what lets an operator written against it work unchanged when no budget was configured.
func (*Account) Check ¶
Check returns a resource error naming this operator when the query is over budget. An operator that can spill calls Over instead and spills.
func (*Account) CheckResult ¶ added in v0.4.0
CheckResult returns a resource error naming collect when the query holds more, its result included, than ResultLimit. It is Check for the account Result opens.
func (*Account) Over ¶
Over reports whether the QUERY is over its ceiling.
Deliberately query-wide rather than per-account: the budget is shared, so an operator that can spill should do so when the query is under pressure, not only when its own share happens to be the large one.
func (*Account) Peak ¶ added in v0.6.8
Peak returns the most this account has held at once, as Used counts it: the memory a profile reports for its operator (step 369).
func (*Account) RetainBytes ¶
RetainBytes records plain-Go state that has no BufferID.
This is not a rounding detail. Three of the largest retentions in the engine are not batches at all — windowSink's one int32 per row PER DISTINCT PARTITIONING, joinBuildSink's one int32 per build row, and quantileAcc's []float64 per group — so a walker over batches alone would report a window sink at half its true size.
type Budget ¶
type Budget struct {
// contains filtered or unexported fields
}
Budget is one query's memory ceiling and spill location.
A nil *Budget means unlimited and untracked; every method is nil-safe so that operators need no branch of their own.
func NewBudget ¶
NewBudget returns a budget with the given limit in bytes; a limit <= 0 means unlimited, and the ledger still tracks the peak so a caller can measure what a query actually held.
func (*Budget) Account ¶
Account opens an operator's view of the budget.
op is the user-facing operator name — "sort", "group_by", "join" — because the only useful thing a memory error can say is which operator could not fit.
func (*Budget) Holding ¶ added in v0.4.0
Holding returns everything the query holds now: what its operators hold, and the result Collect is assembling, each allocation once.
func (*Budget) IsDefault ¶ added in v0.4.0
IsDefault reports whether the limit is the default rather than the caller's.
func (*Budget) MarkDefault ¶ added in v0.4.0
func (b *Budget) MarkDefault()
MarkDefault records that the limit is the default (DefaultLimit), not one the caller asked for. Only a parallel group-by reads it: under a limit the caller set, it stays serial, as it always was, so its row order is the same on every run (step 92).
func (*Budget) OnAccount ¶ added in v0.6.8
OnAccount has fn told of each account opened from now on, on the goroutine that opens it: LazyFrame.Profile's, which attributes an account to the plan node being planned (step 369). Set before the query runs.
func (*Budget) Over ¶
Over reports whether the query is above its ceiling. Always false when unlimited.
func (*Budget) Peak ¶
Peak returns the highest retained total the query reached.
This is the number that demonstrates bounded memory. A test that asserts a query completed proves only that it did not run out; asserting the peak proves it stayed under the ceiling.
func (*Budget) Result ¶ added in v0.4.0
Result opens the account Collect holds its result in, or nil, a working no-op, when ResultLimit is 0.
What only a result holds is kept off the operators' total, so it moves neither Over nor Peak. An operator that can spill decides to by Over, and a result cannot be spilled: a spilled group-by or join replays its partitions while the answers of the earlier ones sit in Collect, and charged with them each partition's sub-sink would start over budget, partition again and again, and reach maxSpillDepth on data that is not skewed at all. That is the rebasing hashAggSink.releaseState does, for the same reason.
An allocation an operator holds too, such as a group-by's answer the result holds slices of, stays the operator's until the operator lets go.
func (*Budget) ResultLimit ¶ added in v0.4.0
ResultLimit is what a query may hold, its result included, before Collect refuses it, or 0 when Collect is not checked.
It is three quarters of the memory ceiling: the default budget, half the ceiling, and half as much again. Under the default budget ursus sets the Go soft limit at nine tenths of the ceiling, and a query holds up to about a fifth more than the ledger counts, measured by internal/memcheck; three quarters, and a fifth more, is that soft limit. Past it the collector cannot keep the heap under the soft limit, and a process under a container's limit is killed.
Only under the default budget. A limit the caller gave bounds what the operators hold, as it always has: Collect then holds whatever it is asked to.
func (*Budget) SpillDir ¶
SpillDir returns the directory spill files are created under, or "" for the system temporary directory.
Source Files