db

package module
v0.42.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 17, 2026 License: Apache-2.0 Imports: 36 Imported by: 14

Documentation

Overview

Package db is an embedded ClassAd log: a persistent key->ClassAd store with optimistic multi-writer transactions, mirroring HTCondor's ClassAdLog (src/condor_utils/classad_log.h). It is the Go core that the cgo layer (package capi) exposes as C symbols for a C++ interface to sit on top of, and that the client/server module serves over CEDAR.

It maps directly onto the collections store: the key->ClassAd table is a Collection, and each transaction is a collections.Txn (snapshot isolation, write-write conflicts, per-ad commit). Unlike classad_log.h -- which allows only one active transaction -- this supports any number of independent concurrent transactions, each a distinct *Txn.

Example (UserDefinedAggregate)

Example_userDefinedAggregate registers the aggregate above and runs a GROUP BY with it over a live table, exactly as a query naming SUM would run.

package main

import (
	"fmt"
	"math"
	"sort"

	"github.com/PelicanPlatform/classad/classad"
	"github.com/PelicanPlatform/classad/db"
)

// stdDev is a user-defined aggregate: the population standard deviation of an attribute.
//
// It is the shape a user-defined aggregate is FOR. The built-in SUM and AVG retain every value in
// the group so that the final answer can be computed by the reference implementation, which makes
// their state O(group size); this one keeps three floats whatever the group holds.
type stdDev struct{}

// NewAccumulator returns the identity -- the answer for a group that contributed no rows.
func (stdDev) NewAccumulator() db.Accumulator { return &stdDevAcc{} }

// stdDevAcc accumulates by Welford's method, which is numerically stable and, more to the point
// here, has a MERGE: Chan's formula combines two partial states without either one seeing the
// other's values. An aggregate whose state cannot be merged cannot be fanned out.
type stdDevAcc struct {
	n    float64
	mean float64
	m2   float64
}

// Add folds one row's value, from the record scan. It is handed EVERY row in the group, including
// rows whose attribute is absent, a string, or an expression that evaluated to an error -- so
// skipping non-numbers is this aggregate's own decision, and it is the same one SUM makes.
func (a *stdDevAcc) Add(v classad.Value) {
	x, err := v.NumberValue()
	if err != nil {
		return
	}
	a.addNumber(x)
}

// AddNum folds one value read straight from a numeric column. Declaring it (see ColumnAccumulator)
// is a promise about TWO things no signature can check, and both have to be true of this aggregate
// before the method may be written:
//
//   - The answer must not depend on the order values arrive in. The column pass walks segments and
//     blocks and skips whole blocks a zone map rules out, so the order is not the scan's and not
//     insertion order. Welford is order-dependent in its last bits -- see the tolerance in
//     ExampleCheckAggregateEqual below -- but not in its answer.
//   - It must not count what it was never offered. The pass yields only the column's NUMERIC
//     values, so a row whose attribute is absent or a string is not offered at all. This aggregate
//     drops those in Add too, so the two rungs count the same rows.
func (a *stdDevAcc) AddNum(v db.NumValue) {
	switch v.Kind {
	case db.NumInt, db.NumBool:
		a.addNumber(float64(v.I))
	default:
		a.addNumber(v.F)
	}
}

func (a *stdDevAcc) addNumber(x float64) {
	a.n++
	d := x - a.mean
	a.mean += d / a.n
	a.m2 += d * (x - a.mean)
}

// Merge folds another accumulator of the same registration into this one. It must be ASSOCIATIVE
// AND COMMUTATIVE: the engine fans the columnar pass out across segment windows, and which worker
// draws which segment is a race. An order-sensitive merge is correct in every test its author
// writes and wrong in production.
func (a *stdDevAcc) Merge(other db.Accumulator) {
	b := other.(*stdDevAcc)
	if b.n == 0 {
		return
	}
	if a.n == 0 {
		*a = *b
		return
	}
	n := a.n + b.n
	d := b.mean - a.mean
	a.m2 += b.m2 + d*d*a.n*b.n/n
	a.mean += d * b.n / n
	a.n = n
}

// Result renders the group's answer. It may be called more than once and must not mutate the
// state, so it reads and returns rather than finalizing in place.
func (a *stdDevAcc) Result() classad.Value {
	if a.n == 0 {
		return classad.NewUndefinedValue()
	}
	return classad.NewRealValue(math.Sqrt(a.m2 / a.n))
}

func main() {
	if err := db.RegisterAggregate("stddev", stdDev{}); err != nil {
		panic(err)
	}

	d, err := db.OpenConfig(db.Config{})
	if err != nil {
		panic(err)
	}
	defer d.Close()

	tx := d.Begin()
	for i, src := range []string{
		`[ Pool = "cm1"; Memory = 1024 ]`,
		`[ Pool = "cm1"; Memory = 2048 ]`,
		`[ Pool = "cm1"; Memory = 4096 ]`,
		`[ Pool = "cm2"; Memory = 8192 ]`,
		`[ Pool = "cm2"; Memory = 8192 ]`,
		`[ Pool = "cm2"; Memory = "lots" ]`, // not a number: no aggregate counts it
	} {
		ad, perr := classad.Parse(src)
		if perr != nil {
			panic(perr)
		}
		tx.NewClassAd(fmt.Sprintf("k%d", i), ad)
	}
	if cerr := tx.Commit(); cerr != nil {
		panic(cerr)
	}

	// AggUser plus the registered name is how a request names it. Everything else about the
	// request -- the constraint, the group columns, the other aggregates -- is unchanged.
	rows, err := d.AggregateColsStats("true",
		[]db.GroupCol{{Attr: "Pool"}},
		[]db.AggSpec{
			{Func: db.AggCount, Arg: "*"},
			{Func: db.AggUser, UserFunc: "stddev", Arg: "Memory"},
		}, nil, nil)
	if err != nil {
		panic(err)
	}
	sort.Slice(rows, func(i, j int) bool { return rows[i].Group[0] < rows[j].Group[0] })
	for _, r := range rows {
		fmt.Printf("%s count=%s stddev=%s\n", r.Group[0], r.Values[0], r.Values[1])
	}

}
Output:
"cm1" count=3 stddev=1277.1523880188386
"cm2" count=3 stddev=0

Index

Examples

Constants

View Source
const (
	NumInt  = collections.NumInt
	NumReal = collections.NumReal
	NumBool = collections.NumBool
)

The column value kinds, re-exported so an aggregate implementing ColumnAccumulator does not have to import the storage package to read the one field that decides how to fold a value.

View Source
const (
	NDVExact     = collections.NDVExact
	NDVBound     = collections.NDVBound
	NDVEstimated = collections.NDVEstimated
)

Accuracy classes, re-exported so a planner can branch on them without importing collections.

View Source
const (
	NDVErrNone   = collections.NDVErrNone
	NDVErrLow    = collections.NDVErrLow
	NDVErrHigh   = collections.NDVErrHigh
	NDVErrEither = collections.NDVErrEither
)

Error directions, re-exported alongside the classes.

View Source
const (
	WatchResetNone          = collections.WatchResetNone
	WatchResetInitial       = collections.WatchResetInitial
	WatchResetEpoch         = collections.WatchResetEpoch
	WatchResetRetention     = collections.WatchResetRetention
	WatchResetAhead         = collections.WatchResetAhead
	WatchResetCursorInvalid = collections.WatchResetCursorInvalid
)

The reasons, re-exported so a db caller need not import collections.

View Source
const ApproxDistinctRelErr = 1.04 / 128.0
View Source
const DefaultArchiveIndexBackfillBytes = 256 << 20

DefaultArchiveIndexBackfillBytes is the per-pass index backfill budget when MaintainOptions.ArchiveIndexBackfillBytes is zero: 32 segments at the default 8 MiB segment size, small enough that a pass does not hold up the maintenance that follows it.

View Source
const DefaultArchiveRetrainInterval = 24 * time.Hour

DefaultArchiveRetrainInterval is the recommended cadence for MaintainOptions.ArchiveRetrainInterval, which is off unless set.

A day, because the thing it tracks moves on that scale: an archive's records drift as the workload does, and unlike a mutable table nothing else ever revisits its dictionary -- no compaction, no rewrite -- so without this a history table keeps whatever dictionary it happened to be given early in life.

WHAT IT COSTS, which is two different things. A retrain on an append-only table does not re-compress sealed segments the way it does on a mutable one: it trains, registers the dictionary and points new writes at it, and old segments keep the dictionary they were written under, which recovery reconstructs. But it also interns any sealed segment still in inline form, and interning reseals that segment. On an archive that has never been interned -- the default, since nothing here sets InternAtSeal -- the first retrain therefore rewrites the whole history. Interning is one-way per segment, so later retrains skip what is already done and cost the sample and the registration alone.

That asymmetry is why this is opt-in rather than on by default, and why Rewrite stays out of the maintenance pass altogether: Rewrite re-encodes every segment on EVERY call.

View Source
const DefaultArchiveSampleMax = 2000

DefaultArchiveSampleMax is the ad sample an archive's maintenance pass takes when MaintainOptions.SampleMax is unset. Lower than the mutable default because sampling an archive decompresses records.

View Source
const DefaultDeltaMax = 16

Config opens a DB with indexing and ordered-index configuration. Dir empty is in-memory; a non-empty Dir is persistent. DefaultDeltaMax is the delta-record chain bound applied when Config.DeltaMax is left at zero, which is to say: delta records are ON by default. Storing only what a write changed is the cheaper way to hold a queue that is updated far more often than it is created -- measured at -42% ingest CPU and -32% allocation replaying a real schedd's job_queue.log -- and a feature that ships switched off is a feature that rots.

It is NOT reversible for data already written. A store that has written one delta record replays them for the rest of its life (the on-disk marker says so); setting DeltaMaxOff later stops new deltas and leaves the existing ones readable, it does not convert them back.

View Source
const DeltaMaxOff = -1

DeltaMaxOff disables delta records for a table. Any negative Config.DeltaMax means off; this spelling exists because zero means "use the default", so there has to be a way to say "none".

Variables

View Source
var (
	// ErrAggregateExists means the name is already registered.
	ErrAggregateExists = errors.New("db: aggregate already registered")
	// ErrBadAggregate means the registration itself is malformed: a nil aggregate, or a name
	// no query could refer to.
	ErrBadAggregate = errors.New("db: invalid aggregate registration")
	// ErrNoSuchAggregate means a query named an aggregate that is not registered here. It is
	// returned rather than answered as undefined: a query asking for p95 and receiving a
	// column of "undefined" has been answered WRONGLY, where a query that fails says what
	// happened.
	ErrNoSuchAggregate = errors.New("db: no such aggregate")
)

Registration failures a caller can act on differently, as sentinels rather than only as messages.

View Source
var ErrColumnsUnsupported = errors.New("classad-db: table cannot serve a column scan for this query")

ErrColumnsUnsupported reports that a table cannot serve the column relay for this query -- it has no columnar accelerator, or the constraint is one the columns cannot evaluate. Callers fall back to a row scan, which every table can serve.

Returned before any batch, always. A column scan that yielded nothing would be indistinguishable from a query that matched nothing, which is the failure this exists to prevent rather than one it is allowed to cause.

View Source
var ErrJoinWireRedactNeedsProjection = errors.New(
	"db: a redacted wire-form join needs a projection on each side")

ErrJoinWireRedactNeedsProjection reports that a redacted wire-form join was asked for over a side whose whole ad was requested.

A whole-ad member has no projection to filter, and the private names are inside the member rather than at the top of the row, so there is nothing here that can withhold them. It refuses rather than shipping the member intact -- the caller falls back to the text row stream, where the render withholds them. A refusal costs the fallback; guessing costs the private attributes of every row.

View Source
var ErrListAggregateNeedsBound = errors.New(
	"a list-valued aggregate needs a bound on its list, which this request has no field for")

ErrListAggregateNeedsBound reports a list-valued aggregate asked for where no bound on it can be stated. It is not a statement that the aggregate is unavailable -- see AggList -- but that this path has nowhere to put the cap that makes it safe.

View Source
var ErrRawWireUnsupported = errors.New("classad-db: table does not support wire-form rows")

ErrRawWireUnsupported reports that a table cannot serve the wire-form relay scan -- today, that it is in-memory rather than persistent. Callers fall back to a text row stream, which every table can serve.

View Source
var ErrSeqCursorExpired = collections.ErrSeqCursorExpired

ErrSeqCursorExpired is collections.ErrSeqCursorExpired: a page's Err wraps it when the cursor's snapshot has been reclaimed, and the listing must restart from the zero cursor.

ErrTooDeep is collections.ErrTooDeep (and wire.ErrTooDeep), the same error value: a write refused because a value nests deeper than any reader would read back. SetAttribute returns it at once, naming the attribute; a whole-ad write (NewClassAd, Put) is refused at Commit as an *UnappliedError that unwraps to it, leaving the key as it was; UpdateOld and an archive Append return it. It is permanent -- the same write is refused every time -- so a caller drops or changes the value and moves on, and must never retry it.

Functions

func AggFilterAttrs added in v0.23.1

func AggFilterAttrs(filter string) []string

AggFilterAttrs returns the attributes a per-aggregate filter reads. Callers use it to hold a filter to the same rules as the rest of the request -- notably the RPC layer's refusal to let an unprivileged reader touch a private attribute, which a filter could otherwise turn into an oracle (`COUNT(*) FILTER (WHERE Secret == "guess")` leaks by its count).

func AggProjection added in v0.16.8

func AggProjection(groupCols []GroupCol, aggs []AggSpec) (attrs []string, groupCol, aggCol []int)

AggProjection builds the deduplicated list of attributes the aggregation reads (group columns then non-"*" aggregate arguments) and the index of each group column / aggregate argument within that list. An aggregate whose argument is "*" (COUNT(*)) gets index -1. A caller projects a scan to attrs and feeds the resulting value rows, groupCol, and aggCol to AggregateValues.

func AppendGroupValueText added in v0.42.0

func AppendGroupValueText(dst []byte, v classad.Value) []byte

AppendGroupValueText appends exactly what GroupValueText returns, without building the string.

It is for a caller assembling a group KEY in a buffer it reuses -- a scan folding a record per iteration, where GroupValueText's return is allocated, hashed and dropped. The numeric kinds append in place; anything else falls back to GroupValueText, because those kinds are rare in a group column and an append form of each would be a second rendering to keep in step with this one.

The two MUST agree: a key assembled here and a key assembled there name the same group, and a scan that used both would otherwise split one group in two. TestAppendGroupValueTextMatches holds them together.

func BuildInsertAds added in v0.41.0

func BuildInsertAds(rows []AggRow, spec InsertSpec) ([]*classad.ClassAd, error)

BuildInsertAds turns aggregate result rows into the ads to be written.

The group half and the aggregate half are read back DIFFERENTLY, and the asymmetry is not an oversight -- it is what the two texts can support:

  • A group's text is a faithful ClassAd literal, and ParseGroupValue inverts every one of them exactly. So a row whose producer supplied GroupValues is read from those, and one whose producer did not is read back from the label with no loss.
  • An aggregate's text is a RENDERING with no inverse: ValueText drops a string's quotes, so "4" and 4 arrive identically. There is nothing to invert, so the typed value is REQUIRED and a row without one is refused rather than guessed at.

Refusing is the right failure because the alternative is silent and permanent: a count guessed back as the string "12" would be stored as text and would compare and sort as text for the rest of that table's life, with nothing to say it happened. Every in-process aggregate path fills TypedValues; a row lacking it came from somewhere that does not, and that is a bug to fix at the producer rather than paper over here.

func CheckAggregate added in v0.37.0

func CheckAggregate(agg Aggregate, samples []classad.Value) error

CheckAggregate tests one Aggregate against the laws the reduction relies on, over the sample values given.

IT SHIPS IN THE PACKAGE RATHER THAN IN A TEST because the thing it checks is not this package's code -- it is a caller's, written somewhere this package's tests cannot see. The alternative is a comment saying "merge must be associative and commutative", and a comment is not a check. An aggregate whose merge is order-sensitive passes every test an author would think to write, because an author tests the order their own code produces; it fails in production, where segment and zone visit order is not the author's to choose.

THE SAMPLES ARE THE CALLER'S because only the caller knows what its aggregate eats. A generic generator here would produce integers for an aggregate over host names and prove nothing. Pass values of the shape the aggregate will really see, and include the awkward ones -- undefined, an error, a string where a number is expected, duplicates, a value appearing once -- since those are where merge and identity usually part company.

WHAT IT CANNOT CHECK. These laws are universally quantified and this runs them over one finite sample, so a pass is evidence rather than proof; an aggregate that is commutative except on a value the caller did not pass still passes here. It is a filter for the mistakes people actually make, not a verification.

func CheckAggregateEqual added in v0.37.0

func CheckAggregateEqual(agg Aggregate, samples []classad.Value, equal func(a, b classad.Value) bool) error

CheckAggregateEqual is CheckAggregate with the caller's own notion of two answers being the same. A nil equal compares the rendered values exactly, which is what CheckAggregate does.

THE EXACT COMPARISON IS WRONG FOR AN AGGREGATE THAT DOES FLOATING-POINT ARITHMETIC, and that is not a weakness of the implementation being checked. Floating-point addition is not associative, so an aggregate computing a variance -- the obvious first thing anyone writes here, since its state is bounded where the built-ins' is O(group size) -- gives answers that differ in the last bits depending on how the rows were divided into partial states. That IS the order dependence the law is about; it is simply not avoidable, and a checker demanding exact equality would reject every correct implementation of it while passing an integer aggregate that is genuinely broken.

So the comparison is the caller's, and an aggregate needing a tolerance has thereby SAID that it needs one, in its own tests, where a reader can see the size of the slack and judge it. The alternative -- a tolerance built in here -- would apply silently to aggregates that should have been exact.

Example

ExampleCheckAggregateEqual tests an aggregate against the laws the reduction relies on, over a sample of the caller's own choosing.

The comparison is the caller's because this aggregate does floating-point arithmetic, and float addition is not associative: the same values divided into partial states a different way give answers differing in the last bits. That IS the order dependence the law is about, and it cannot be removed -- so an aggregate that needs slack says so here, in its own tests, where a reader can see how much.

package main

import (
	"fmt"
	"math"

	"github.com/PelicanPlatform/classad/classad"
	"github.com/PelicanPlatform/classad/db"
)

// stdDev is a user-defined aggregate: the population standard deviation of an attribute.
//
// It is the shape a user-defined aggregate is FOR. The built-in SUM and AVG retain every value in
// the group so that the final answer can be computed by the reference implementation, which makes
// their state O(group size); this one keeps three floats whatever the group holds.
type stdDev struct{}

// NewAccumulator returns the identity -- the answer for a group that contributed no rows.
func (stdDev) NewAccumulator() db.Accumulator { return &stdDevAcc{} }

// stdDevAcc accumulates by Welford's method, which is numerically stable and, more to the point
// here, has a MERGE: Chan's formula combines two partial states without either one seeing the
// other's values. An aggregate whose state cannot be merged cannot be fanned out.
type stdDevAcc struct {
	n    float64
	mean float64
	m2   float64
}

// Add folds one row's value, from the record scan. It is handed EVERY row in the group, including
// rows whose attribute is absent, a string, or an expression that evaluated to an error -- so
// skipping non-numbers is this aggregate's own decision, and it is the same one SUM makes.
func (a *stdDevAcc) Add(v classad.Value) {
	x, err := v.NumberValue()
	if err != nil {
		return
	}
	a.addNumber(x)
}

// AddNum folds one value read straight from a numeric column. Declaring it (see ColumnAccumulator)
// is a promise about TWO things no signature can check, and both have to be true of this aggregate
// before the method may be written:
//
//   - The answer must not depend on the order values arrive in. The column pass walks segments and
//     blocks and skips whole blocks a zone map rules out, so the order is not the scan's and not
//     insertion order. Welford is order-dependent in its last bits -- see the tolerance in
//     ExampleCheckAggregateEqual below -- but not in its answer.
//   - It must not count what it was never offered. The pass yields only the column's NUMERIC
//     values, so a row whose attribute is absent or a string is not offered at all. This aggregate
//     drops those in Add too, so the two rungs count the same rows.
func (a *stdDevAcc) AddNum(v db.NumValue) {
	switch v.Kind {
	case db.NumInt, db.NumBool:
		a.addNumber(float64(v.I))
	default:
		a.addNumber(v.F)
	}
}

func (a *stdDevAcc) addNumber(x float64) {
	a.n++
	d := x - a.mean
	a.mean += d / a.n
	a.m2 += d * (x - a.mean)
}

// Merge folds another accumulator of the same registration into this one. It must be ASSOCIATIVE
// AND COMMUTATIVE: the engine fans the columnar pass out across segment windows, and which worker
// draws which segment is a race. An order-sensitive merge is correct in every test its author
// writes and wrong in production.
func (a *stdDevAcc) Merge(other db.Accumulator) {
	b := other.(*stdDevAcc)
	if b.n == 0 {
		return
	}
	if a.n == 0 {
		*a = *b
		return
	}
	n := a.n + b.n
	d := b.mean - a.mean
	a.m2 += b.m2 + d*d*a.n*b.n/n
	a.mean += d * b.n / n
	a.n = n
}

// Result renders the group's answer. It may be called more than once and must not mutate the
// state, so it reads and returns rather than finalizing in place.
func (a *stdDevAcc) Result() classad.Value {
	if a.n == 0 {
		return classad.NewUndefinedValue()
	}
	return classad.NewRealValue(math.Sqrt(a.m2 / a.n))
}

func main() {
	samples := []classad.Value{
		classad.NewIntValue(1),
		classad.NewIntValue(2),
		classad.NewRealValue(3.5),
		classad.NewIntValue(100),
		classad.NewStringValue("not a number"), // a value the aggregate must ignore
		classad.NewUndefinedValue(),            // and another
	}
	closeEnough := func(a, b classad.Value) bool {
		x, xerr := a.NumberValue()
		y, yerr := b.NumberValue()
		if xerr != nil || yerr != nil {
			return a.String() == b.String()
		}
		return math.Abs(x-y) <= 1e-9*math.Max(1, math.Abs(x))
	}
	fmt.Println(db.CheckAggregateEqual(stdDev{}, samples, closeEnough))

}
Output:
<nil>

func CheckSealedCopy added in v0.42.0

func CheckSealedCopy(srcSealed, dstSealed []string, cols map[string]string) error

CheckSealedCopy refuses a copy that would move a sealed value out from under its seal.

This is specific to copying rows and does not arise for a roll-up, which writes a SUMMARY. A copy moves stored values verbatim, so when the reader is entitled to sealed values -- it reads them decrypted -- and the target does not seal the column they land in, the statement writes in the clear what the source went to the trouble of encrypting at rest. Nothing downstream would report it: the copy succeeds, the rows read back correctly, and the only difference is on disk.

So it is refused, naming the columns, rather than silently sealing them in the target: changing a table's encryption from a DML statement would be a worse surprise than refusing one. An unprivileged copy needs no check -- a sealed value reaches that reader undefined, so there is nothing to leak -- but it does silently drop those columns, which is what redaction does everywhere.

srcSealed and dstSealed are the two tables' encrypted attribute names; cols maps each copied source attribute to the target attribute it lands in, and is nil for SELECT *, where every sealed source attribute keeps its own name.

func CompactInternPaths added in v0.30.0

func CompactInternPaths() (wirePath, astPath int64)

CompactInternPaths reports, process-wide, how compaction re-interned records: by transcoding the wire bytes, or by decoding into an ast and encoding it back. All-ast means the transcode is being attempted and refused on every record.

func ConstraintRefs added in v0.28.0

func ConstraintRefs(expr string) (refs []string, dynamic bool)

ConstraintRefs reports every attribute name an expression references, scoped or not, and whether the expression makes a reference this analysis cannot resolve statically (see dynamic below).

A constraint that does not parse returns no references and dynamic=false: it is rejected downstream where the parse error can be reported properly, and refusing it here would report the wrong reason.

func DeriveKey added in v0.41.0

func DeriveKey(ad *classad.ClassAd, cols []string) (string, error)

DeriveKey builds the storage key of an ad from the declared key columns.

Each column's value is rendered by GroupValueText, the same function that decides which rows form one GROUP. That is deliberate and it is the whole reason the derived key is trustworthy for the statement that motivates it: in `INSERT INTO summaries SELECT ... GROUP BY day, Owner`, the rows arriving are groups, and deriving their keys with the grouping relation means one group maps to one key -- always, and without this code having to re-derive what "the same" means. It also inherits the canonicalisation that relation already makes: 1 and 1.0 are one group and so one key, and the string "4" is kept apart from the number 4 because the rendering quotes it.

An undefined or error component is REFUSED rather than rendered. A missing key attribute is a mis-specified statement -- the SELECT did not produce the column the table says identifies a row -- and the alternative is worse than an error: every row missing that attribute would render the same `undefined` into its key, so they would all derive ONE key and silently overwrite each other down to a single surviving row. That is the exact failure this declaration exists to prevent, so it is refused at the one place that can still name which column was missing.

func DeriveKeys added in v0.41.0

func DeriveKeys(ads []*classad.ClassAd, cols []string) ([]string, error)

DeriveKeys builds the keys for a whole batch and refuses a batch in which two rows derive the same key.

The duplicate check is the point, not a nicety. Writing a batch whose keys collide is not an error anywhere below this: Txn.NewClassAd REPLACES the ad at a key, so N rows sharing a key would be written, counted as N, and leave ONE row behind -- a silent loss reported as a success. It happens whenever the declared key is coarser than the rows being inserted, which for `INSERT ... SELECT` means a GROUP BY that groups by more columns than the key names: a plausible statement, not a perverse one.

So the batch is refused whole, naming the colliding key, and nothing is written. The caller re-runs with a key that matches the grain of its rows.

func FallbackReasons added in v0.30.0

func FallbackReasons() (removal, bound, noBase, ineligible int64)

FallbackReasons reports, process-wide, why patch writes had to store a whole record rather than a delta: an attribute removal (which a delta cannot express), the chain reaching its bound, no whole record to chain to yet, or delta records not being in use. Each fallback costs a read of the stored ad, so this says which of those reads are worth attacking.

func GroupValue added in v0.38.0

func GroupValue(v classad.Value) classad.Value

GroupValue is GroupValueText's typed twin: the canonical VALUE of the group v belongs to. It is what AggRow.GroupValues carries.

THE INVARIANT, which is the whole point: GroupValueText(GroupValue(v)) == GroupValueText(v). The value names the same group the text does, so the two cannot drift -- a caller that keys on one and labels with the other is keying and labelling the same thing.

It is canonical rather than "the value that formed the group" because a group may hold values that are not identical. GroupValueText puts the integer 1 and the real 1.0 in one group, on purpose: they ARE one number. So "the first value seen" is not a property of the group at all -- it is a property of the scan order, and under a parallel fan-out it is not even stable between runs of the same query. Mapping every member to one value makes the answer a function of the group.

Only the integral real needs moving: its text is the integer's text, so the integer is the representative. Every other kind already renders uniquely for its group, so it is its own.

A LIST IS REBUILT from its elements mapped through this same function, because a list's group is decided element-wise (see groupListText) and so {1} and {1.0} are one group with one representative, {1}. Rebuilding also drops the unevaluated source expressions a list parsed from a literal carries, which are a property of how the value was written and not of the group -- {1,1+1} and {1,2} are one group and now one value.

func GroupValueText added in v0.37.0

func GroupValueText(v classad.Value) string

GroupValueText renders a value as its GROUP: the text that both names the group and decides which values share one.

GROUPING IS AN EQUIVALENCE RELATION, AND `==` IS NOT ONE. Two values belong in one group exactly when they are the same value, which neither ClassAd operator quite says:

a          b              a == b       a =?= b     one group?
undefined  undefined      undefined    true        yes
error      error          error        true        yes
4          "4"            error        false       no
true       "true"         error        false       no
"alice"    "Alice"        true         false       no
1          1.0            true         false       YES
true       1              true         false       no

`==` disqualifies itself twice over: under it an undefined value is not equal to ITSELF, so there is no group to put it in, and a cross-type pair answers `error` rather than "different". A relation that is neither reflexive nor total cannot partition rows. That is the same corner SQL turns, and it turns it the same way -- GROUP BY is defined on "is not distinct from", not on `=`, which is why NULL groups with NULL.

`=?=` IS total and reflexive, and for strings, bools and undefined it is exactly right. It is wrong in one place: it separates 1 from 1.0. Those are one quantity written two ways, and HTCondor ads genuinely carry both spellings of the same attribute, so splitting them gives two rows for one number -- a surprise with nothing behind it. The storage layer keys by (bits, type) and separates them; this is the layer that decides they are one group.

So the relation is: a number equals a number when they are numerically equal, a string equals a string when it is byte-identical, every other kind equals only its own kind and value, and undefined and error each form one group. Within numbers that is `==`; within strings it is `=?=`; across kinds it is neither, because neither answers.

WHY ONE FUNCTION AND NOT TWO. The key and the label are the same channel: AggRow.Group is what a caller reads AND what it keys on. A key finer than its label yields two output rows with one name -- which is what a tagged key over ValueText's labels produced, and TestArchiveGroupedStringColumn says it plainly: "group "undefined" appears twice". The rendering is made FAITHFUL instead, so distinct groups always read differently: a string arrives quoted, which is what keeps it from reading as the number or the bool it spells.

That faithfulness is also what lets 1 and 1.0 be one group WITHOUT hiding anything. They render alike because they ARE alike here; 4 and "4" render differently because they are not.

AND WHY THIS MUST NOT SHARE A RENDERER WITH joinPlan.key, which looks like the same job. The two have OPPOSITE requirements. A group key has to be exact: it partitions rows, so merging two values that are not the same value is a wrong row. A join's hash key has to be COARSE: it only narrows candidates, and the condition re-checks every surviving pair, so putting two unequal rows in one bucket costs a comparison -- while separating two rows the condition WOULD match loses them. A join's condition is usually `==`, so its key folds case on purpose, which is exactly what this must not do.

A NaN groups with other NaNs, though `NaN == NaN` is false, and an infinity with its own sign. Both are the same choice undefined and error get: a relation has no answer, and one group for all of them says more than one group per row. A bucketed series drops them before they reach here anyway.

A LIST OR A NESTED AD is reached by no operator at all: `{1,2} =?= {1,2}` is error, so even reflexivity fails there. They group by canonical text, which collects identical ones together -- the most that can be said about values no relation reaches, and more useful than one group per row. That text is a ClassAd literal like every other group's: a list reads "{ 1,2 }" and a nested ad "[a = 1]", so there is no kind whose group text has to be read differently from the rest, and ParseGroupValue inverts all of them.

func IsMatchAll added in v0.21.3

func IsMatchAll(constraint string) bool

IsMatchAll reports whether a constraint imposes no filter (an empty string or a literal "true"), so an aggregate over it covers every record. Shared with the mutable-table COUNT(*) fast path in dbrpc.

func IsMatchNone added in v0.27.1

func IsMatchNone(constraint string) bool

IsMatchNone reports whether a constraint can never match: it references no attribute, so its value is the same for every record, and that value is not TRUE.

The dual of IsMatchAll, and missing until `select min(ProcId) from history where false` was measured at 3.5s -- a full scan to establish that nothing matches. FALSE, UNDEFINED and ERROR all match nothing, since a record matches only when the constraint is boolean TRUE, so this folds `where false`, `where 1 == 2`, `where undefined` and `where error` alike.

The no-reference check is the same load-bearing one IsMatchAll documents, in the other direction: a record-dependent expression must never fold even when it evaluates non-TRUE against an empty ad. `JobStatus == 2` is UNDEFINED with JobStatus absent and matches nothing HERE, but matches plenty of records, so anything referencing an attribute is left to the scan.

func IsSystemKey added in v0.9.0

func IsSystemKey(key string) bool

IsSystemKey and SystemKey re-export the collections helpers so callers holding only a *db.DB (e.g. dbrpc building marker keys) can classify or construct a reserved system key without importing collections directly. A system key begins with a NUL byte and names a record hidden from client reads but retrievable by explicit LookupClassAd.

func LastDeltaDecodeFailure added in v0.30.5

func LastDeltaDecodeFailure() (stage, msg string)

LastDeltaDecodeFailure returns the most recently sampled decode error behind a refused delta chain merge, as (stage, message) where stage is "base" or "patch". The per-reason counts say how often the bytes would not decode; this says what the decoder objected to.

func LastNoBaseDetail added in v0.30.7

func LastNoBaseDetail() (versions int, chainBroken bool, sealedSkipped int)

LastNoBaseDetail returns the shape of the most recent delta chain that had no whole record: versions found, whether the walk truncated at a dead link, and how many sealed segments could not be probed for want of a key index.

func MatchSignature

func MatchSignature(ad *classad.ClassAd, significantAttrs []string) uint64

MatchSignature is HTCondor's autocluster key: a 64-bit checksum over the given significant attributes' expression text in ad. Two ads with textually identical significant attributes (same Requirements, same RequestCpus literal, ...) hash equal, so a matchmaker can compute one candidate list per distinct signature and reuse it for every identical request.

func ParseGroupValue added in v0.38.0

func ParseGroupValue(text string) classad.Value

ParseGroupValue reads a group's TEXT back as its value: GroupValueText's inverse, for a producer that has only the rendered form. The dbrpc client uses it against a server too old to send AggRow.GroupValues, so a caller gets values from every server and needs one path rather than two.

It inverts EVERY group exactly, with no kind excepted: GroupValueText renders a group as ClassAd literal syntax and nothing else, so a string arrives quoted and escaped, undefined and error as their keywords, a number as itself, a list as "{ 1,2 }" and a nested ad as "[a = 1]". Parsing that text is the exact reverse of writing it -- the reason the rendering had to be faithful, and why the two must change together.

The invariant it rests on, which TestParseGroupValueInvertsEveryGroup pins: for any value v, GroupValueText(ParseGroupValue(GroupValueText(v))) == GroupValueText(v). Round-tripping a group through its text lands in the same group, which is what lets a caller treat a label from an old server and a value from a new one as the same thing.

A text that does not parse comes back as the string it is, not an error value. This is the fallback path, and text that reached here came out of some server's renderer; reporting the group as "error" would be inventing a value the group never had, while the text always at least names the group it came from.

func PrivateConstraintRef added in v0.28.0

func PrivateConstraintRef(expr string) (attr string, dynamic bool)

PrivateConstraintRef reports the first private attribute an expression references, or "" if it references none. dynamic reports that the expression names attributes at runtime, which the caller must treat as unauthorized on an unprivileged connection: a reference it cannot see is one it cannot check.

What this CANNOT see, and what closes it: an ad may store a public attribute whose value is an expression over a private one (`Leak = ClaimId`). A constraint on `Leak` references nothing private and so passes here, while evaluation resolves ClaimId. Closing that needs the private value to be absent at EVALUATION time for an unprivileged reader, not a check on the constraint text -- the redacted decode walk already does this for rendering.

func RegisterAggregate added in v0.37.0

func RegisterAggregate(name string, agg Aggregate) error

RegisterAggregate registers agg under name, for use as AggSpec{Func: AggUser, UserFunc: name}.

It fails rather than overwriting, so that what an aggregate means cannot depend on package initialization order. A caller that MEANS to install over whatever is there says so, with ReplaceAggregate.

THIS REGISTRY OFFERS REPLACEMENT AND WITHDRAWAL AND THE SCALAR FUNCTION REGISTRY DOES NOT, and the difference is where the decision is recorded. There, compiling an expression captures the decision that a name was a UDF, along with its arity, and compiled programs are cached and shared -- so withdrawing a scalar registration would leave cached programs calling something that is gone. Nothing of the sort happens here. An aggregate name is resolved against this registry once per query, when the reduction is set up (resolveAggregates, and the columnar passes beside it), and nothing caches that resolution past the query that made it. Replacing or removing a registration is therefore well defined, and ReplaceAggregate and UnregisterAggregate do it.

ARITY DOES NOT CONSTRAIN A REPLACEMENT, because an aggregate registration does not carry one. A scalar registration declares min and max argument counts and every call site is checked against them when the expression compiles; an AggSpec names exactly one argument attribute whatever the aggregate is, so a replacement has no signature it could disagree with. What it has to keep is Accumulator's laws, which CheckAggregate tests and no registration can check.

func RegisteredAggregates added in v0.37.0

func RegisteredAggregates() []string

RegisteredAggregates returns the registered names, as registered, for diagnostics. The order is unspecified.

func ReplaceAggregate added in v0.38.2

func ReplaceAggregate(name string, agg Aggregate) error

ReplaceAggregate installs agg under name whether or not something is already registered there. It is the overwrite RegisterAggregate refuses, for the caller holding aggregates somewhere durable -- a table of stored modules an administrative command writes to -- which has to be able to CORRECT one. Validation is RegisterAggregate's: a nil aggregate and a name no query could refer to are both ErrBadAggregate, so nothing unusable reaches the registry by this door either.

IT REGISTERS AN ABSENT NAME rather than requiring a present one, because its caller is applying a stored definition and not editing a table: the command that writes a module and installs it runs the same line whether or not this process has seen that name before, and demanding presence would make every such caller write a lookup-then-branch that is also a race. A caller that wants the name to be free uses RegisterAggregate, which says so by failing.

THE NAME IS STORED AS GIVEN, so a replacement under a different casing than the registration it replaces leaves ONE entry -- lookups fold case -- carrying the NEW spelling, which is what RegisteredAggregates then reports. The spelling is diagnostics, and the useful thing to report is how the aggregate that is installed was asked for, not how an earlier one was.

A QUERY IN FLIGHT IS UNAFFECTED. A query resolves each name it uses to an Aggregate value once, before it reduces anything, and runs against that value; a replacement that lands mid-scan changes this map and reaches nothing already holding the old value. So a running query finishes under the implementation it started with, and one group's rows are never split across two of them -- which matters more than it sounds, because a half-and-half group is a wrong answer that looks exactly like a right one.

WHAT IT DOES CHANGE IS WHAT A LATER QUERY MEANS, mid-process and with no synchronization point anywhere: two queries seconds apart, naming the same aggregate, can compute different things, and nothing in either answer says so. That is the hazard, and it is inherent in being able to correct a definition at all -- a caller that cares which definition produced a number has to record that itself, because this registry holds only the current one.

func SealWalkStats added in v0.30.0

func SealWalkStats() (examined, deltas int64)

SealWalkStats reports what the collapse walk examined and found. See collections.SealWalkStats.

func SealedProbesSkipped added in v0.30.7

func SealedProbesSkipped() int64

SealedProbesSkipped reports how many sealed-segment key probes were skipped because the segment's key index is not built yet.

func SpliceStats added in v0.30.0

func SpliceStats() (spliced, spliceRefused, objectPath int64)

SpliceStats reports, process-wide, how delta merges were served: by splicing attribute bytes, by decoding after the splice refused, and by decoding because the caller wanted an object rather than bytes. All three matter -- a merge that never attempts a splice and one that attempts and refuses look identical in a one-number report.

func SystemKey added in v0.9.0

func SystemKey(name string) string

SystemKey builds a reserved system key from name (prefixing the NUL sentinel).

func UnreadableBaseReasonNames added in v0.30.3

func UnreadableBaseReasonNames() []string

UnreadableBaseReasonNames lists every reason name UnreadableBaseReasons can report, including ones currently at zero, so a consumer can publish a stable set of counters.

func UnreadableBaseReasons added in v0.30.3

func UnreadableBaseReasons() map[string]int64

UnreadableBaseReasons breaks UnreadableBaseRefusals down by WHY the base could not be read, keyed by reason name ("not-visible", "segment-gone", "reassemble", "delta-chain", "decode"). The repair differs per reason -- a snapshot miss, a reaped segment, a lost columnar payload, an unresolvable delta chain and a decode failure have nothing in common -- so the total alone cannot direct an investigation. The values sum to UnreadableBaseRefusals.

func UnreadableBaseRefusals added in v0.30.1

func UnreadableBaseRefusals() int64

UnreadableBaseRefusals reports, process-wide, how many patch writes were REFUSED because the key was present in the store but its current record could not be read. Nonzero means a key the store holds failed to resolve; the write was reported to the caller as a conflict rather than stored against an empty ad, which is what used to turn such a miss into an identity-less row.

func UnregisterAggregate added in v0.38.2

func UnregisterAggregate(name string) bool

UnregisterAggregate removes the registration of name, folding case as lookup does, and reports whether there was one to remove. It is for the caller taking a stored aggregate out of service, and it reports rather than returning nothing because "was it there" is the one thing that can surprise a caller reconciling a stored set against this process.

It does not validate the name: a name no query could refer to is a name nothing can be registered under, so it is simply absent and the answer is false.

A query in flight is unaffected, exactly as under ReplaceAggregate and for the same reason -- it holds the Aggregate it resolved, not this map. A query naming the aggregate AFTERWARDS fails with ErrNoSuchAggregate, the same answer a name that was never registered gets, since an unknown aggregate fails a query rather than being answered undefined.

func ValidTableName

func ValidTableName(name string) bool

ValidTableName reports whether name is usable as a table (and a directory): it must start with a letter or underscore and contain only letters, digits, underscores, and hyphens.

func ValueText added in v0.16.8

func ValueText(v classad.Value) string

ValueText renders a value as a plain display string, for an aggregate's RESULT. It is not a group rendering and must not be used as one: it drops the quoting and the decimal point that tell two groups apart. See GroupValueText.

Types

type Accumulator added in v0.37.0

type Accumulator interface {
	// Add folds one row's argument value into the state. The value is whatever the row's
	// argument attribute evaluated to, including undefined for a row that does not carry it;
	// skipping those is the aggregate's own decision, as it is for the built-ins.
	Add(v classad.Value)

	// Merge folds another accumulator's state into this one. The engine only ever passes an
	// accumulator made by the SAME registration, so an implementation may assert its own type;
	// a mismatch is a bug in the engine rather than an input to handle. Nothing reads other
	// afterwards, so an implementation may consume it rather than copying out of it.
	Merge(other Accumulator)

	// Result renders the group's answer. It may be called more than once and must return the
	// same value each time without changing the state.
	Result() classad.Value
}

Accumulator is one group's state for one user-defined aggregate.

THE STATE IS THE IMPLEMENTATION'S OWN, which is a deliberate departure from how the built-ins work. A built-in SUM/AVG/MIN/MAX here RETAINS EVERY ARGUMENT VALUE in the group and hands the whole slice to the reference implementation at the end, so that integer-exact sums, int/real promotion, boolean coercion and error propagation match the reference exactly rather than being a re-implementation of it. That is parity bought with memory: built-in aggregate state is O(group size), not O(1). An aggregate with its own bounded state is therefore BETTER BEHAVED than the built-in it imitates, which is the argument for giving this interface a state of its own instead of modelling it on the built-in accumulator and handing an implementation the value slice.

THE LAWS. Merge must be ASSOCIATIVE AND COMMUTATIVE, not merely associative. Nothing promises the order partial states are produced or combined in -- zone and segment identity has already produced one silent wrong-count bug in this engine -- so an aggregate whose merge is order-sensitive would be correct in every test and wrong in production, which is the worst failure mode available. Result must be repeatable and must not mutate the accumulator. Those are contracts an interface cannot enforce, so CheckAggregate exists to test one against them.

MERGE IS CALLED, twice over, and not as decoration. The columnar pass fans out across segment windows and folds one accumulator per worker, so merge is the reduction of that fan-out; and two storage groups that render to the same label are merged into one row. Which worker draws which segment is a race, which is why associativity alone is not enough and commutativity is in the contract: an order-sensitive merge is correct in every test its author writes and wrong in production, plausibly and only sometimes.

type AggFunc added in v0.16.8

type AggFunc uint8

AggFunc is a SQL aggregate function.

const (
	AggCount AggFunc = iota // COUNT(*) or COUNT(col)
	AggSum
	AggAvg
	AggMin
	AggMax
	// AggCountDistinct is COUNT(DISTINCT col): the number of distinct defined values of
	// the argument in the group. It is EXACT, and therefore keeps one entry per distinct
	// value per group while the scan runs -- fine for the attributes people group and
	// count by (Owner, JobStatus, a host name), but not something to point at a
	// unique-per-row attribute over a large history. There is deliberately no silent
	// switch to a sketch: a query written COUNT(DISTINCT ...) gets a true count, and the
	// approximate one is asked for by name (AggApproxCountDistinct).
	AggCountDistinct
	// AggUser is a user-defined aggregate, named by AggSpec.UserFunc and registered with
	// RegisterAggregate. It sits past AggMax on purpose: that is what routes a request
	// carrying one onto the RPC opcodes with a field for the NAME, so a server too old to
	// know them refuses rather than answering a different aggregate. A selector alone says
	// only "past MAX" -- which aggregate was meant is in UserFunc, and a frame that cannot
	// carry it cannot ask the question.
	AggUser

	// AggList collects the group's argument values into a LIST, bounded by a cap the
	// request carries.
	//
	// THE ORDER INSIDE THE CELL IS THE ORDER THE ROWS ARRIVED IN, which is the join's own
	// order and not a property of the group. A GROUP BY defines no order, the join planner
	// exchanges its sides on cost, and a parallel walk interleaves them -- so the same
	// query can return the same group's values in a different order on the next run, and a
	// caller that needs one sorts, or lists an expression that carries its own sort key.
	// That also decides WHICH values survive the cap: the first ones to arrive, which is to
	// say an arbitrary subset rather than the smallest or the earliest.
	//
	// It is the aggregate for a question that is not a reduction of the values but the
	// values themselves. An estimate over the (requested, used) PAIRS of a job's run
	// attempts cannot be rebuilt from a SUM and a COUNT of each half, so a caller asking it
	// had no choice but to take the rows -- which is the thing a join aggregate exists to
	// avoid sending.
	//
	// It keeps EVERY row's value, undefined and error included. Two lists over one group
	// are then positionally aligned -- element i of each came from the same row -- which is
	// what lets a caller ask for LIST(a) and LIST(b) and read the results as pairs.
	// Skipping undefined would misalign them, and the misalignment would be close to
	// invisible: the only symptom is two lists of different lengths, and a group where both
	// attributes go missing on the same rows does not even have that.
	//
	// A caller that would rather not depend on that can write the pair itself, where the
	// argument is an expression -- a join aggregate's is -- since LIST({a, b}) is a list of
	// pairs in one cell. It has to, if the two aggregates carry different FILTERs: a filter
	// is what decides which rows reach THIS aggregate, so two differently filtered lists are
	// two different selections of rows and nothing aligns them.
	//
	// It sits past AggUser, so a request carrying one reaches only a frame with a field for
	// its BOUND. AggregateValues refuses it outright for the same reason: an unbounded
	// list-valued aggregate turns a reduction back into the row stream it was reducing, one
	// cell at a time.
	AggList

	// AggApproxCountDistinct is COUNT(DISTINCT col) answered from a HyperLogLog built during
	// the scan: an ESTIMATE of the number of distinct defined values of the argument in the
	// group, in a fixed amount of memory per group.
	//
	// IT IS ASKED FOR BY NAME, and that is the whole reason it is a second function rather
	// than a size threshold inside the first one. A query written COUNT(DISTINCT ...) is a
	// query for a count, and answering it with a number that is close would be wrong rather
	// than fast -- wrong in a way nothing in the result says, since an estimate and a count
	// are the same digits. So AggCountDistinct is always exact, nothing routes to the sketch
	// on its own judgement, and a caller that would rather have the estimate says so here.
	//
	// WHAT IT COSTS AND WHAT IT BUYS. The exact function holds one entry per distinct value
	// per group for the duration of the scan, so its state grows with its own answer; this
	// holds approxDistinctRegisters one-byte registers per group whatever the cardinality --
	// 16 KiB at the fixed precision -- for a relative standard error of ApproxDistinctRelErr,
	// about 0.81%. That trade is worth making over a unique-per-row attribute on a large
	// history, where the exact count is a memory bound nobody can state in advance, and is
	// worth nothing at all over Owner or JobStatus, where the exact set is smaller than the
	// sketch and right.
	//
	// THE ERROR IS A STANDARD DEVIATION, not an interval: roughly a third of groups land
	// further out, and a single group can be several multiples away. A caller putting the
	// number in front of someone owes them the word "approximately"; see
	// ApproxDistinctRelErr.
	//
	// It composes with everything the exact function does -- a WHERE constraint, a GROUP BY
	// including a bucketed time series, a per-aggregate FILTER -- because the sketch is built
	// from the rows that reach the aggregate rather than read from a per-segment summary that
	// knows nothing about any of the three. The persisted per-segment sketches cover a whole
	// segment and an indexed attribute, so they could serve none of the three; the sketch
	// this builds is the one that composes.
	//
	// It sits past AggMax, which routes a request carrying it onto the extended opcodes, so a
	// server too old to know it refuses the query instead of returning "undefined" for the
	// column.
	AggApproxCountDistinct
)

type AggRequest added in v0.32.1

type AggRequest struct {
	// Constraint filters rows before grouping. Empty, or any expression that references no
	// attribute and evaluates TRUE, matches every row (IsMatchAll); one that references no
	// attribute and does not evaluate TRUE matches none (IsMatchNone).
	Constraint string
	// GroupCols is the GROUP BY, in output order. Empty means a scalar aggregate over the whole
	// match, which still yields exactly one row. A column may carry a bucket width, flooring a
	// numeric attribute into fixed-width buckets.
	GroupCols []GroupCol
	// Aggs are the outputs, in output order. A spec may carry its own FILTER, restricting that
	// one aggregate to the rows of its group matching it.
	Aggs []AggSpec
}

AggRequest is one grouped aggregate, stated independently of whatever answers it.

Every field a strategy may have to decline over lives here rather than in the strategy, which is the point of naming the type at all: a bucket width rides on its group column and a FILTER rides on its aggregate, so "can this be served" is a question about the request.

What is deliberately NOT here is execution: the caller's scan-stats block and its cancellation callback are not part of the question asked, they are how this particular run of it is being watched, so a ladder captures them when it is built rather than carrying them in the request.

type AggRow added in v0.16.8

type AggRow struct {
	Group  []string
	Values []string

	// GroupValues is Group as VALUES, one per group column, or nil when the producer
	// did not supply them. It exists so a caller rendering a group for some other
	// medium -- a JSON type, a chart label, a number it wants to do arithmetic on --
	// does not have to parse the text back. Group remains the group's identity and
	// its label; this is the same group said in types.
	//
	// It is a CANONICAL representative, not the first value that landed in the group:
	// GroupValue maps every member of a group to one value, so a group reads the same
	// whichever row formed it and whichever shard answered. That matters because 1 and
	// 1.0 are deliberately one group (see GroupValueText), so "the value that made the
	// group" is not well defined, and a parallel fan-out would otherwise report
	// whichever arrived first.
	//
	// Every kind arrives here as the value it is, lists and nested ads included. Those two
	// are grouped by their canonical text rather than by a relation (GroupValueText reaches
	// no relation for them), but that text is a ClassAd literal, so the value survives the
	// trip over dbrpc as well and a caller never finds a list standing in as a string.
	GroupValues []classad.Value

	// TypedValues is Values as VALUES, one per aggregate, or nil when the producer did not
	// supply them -- which is every producer that read them off the wire, because dbrpc's
	// aggregate cells are rendered text (opAggregateTyped types the group cells only).
	//
	// GroupValues exists so a caller need not parse a GROUP back; this is the other half,
	// and the halves are not symmetric. A group's text is a faithful ClassAd literal that
	// ParseGroupValue inverts exactly, so GroupValues is a convenience. An aggregate's text
	// is a RENDERING with no inverse -- ValueText drops a string's quotes -- so for an
	// in-process consumer that has to write the answer back into a ClassAd, this is not a
	// convenience but the only correct source. See aggAcc.resultValue.
	TypedValues []classad.Value

	// Elided is, per aggregate, how many of this group's values a BOUNDED aggregate left
	// out of its answer. It is nil when nothing was left out, which is every row a
	// producer without a bounded aggregate returns and most rows of one that has them.
	//
	// It exists because a cut list and a complete list are otherwise the same bytes. A
	// caller that reads a truncated cell as the whole of the group has been answered
	// wrongly and has no way to find out, which is the one failure a bounded list-valued
	// aggregate can have -- so the cut is reported rather than implied, and the caller can
	// decide between raising the bound, narrowing the query and showing the answer as the
	// sample it is.
	//
	// It is per CELL and not per row because two list-valued aggregates in one request can
	// carry different FILTERs and so fill at different rates: one of them being cut says
	// nothing about the other.
	Elided []int
}

AggRow is one group's result: the group-by column values followed by the aggregate values, all rendered as strings (aligned with the request's group columns and aggregate specs).

func AggregateValues added in v0.16.8

func AggregateValues(seq iter.Seq[[]classad.Value], attrs []string, groupCols []GroupCol, aggs []AggSpec, groupCol, aggCol []int, stop func() bool) (rows []AggRow, err error)

AggregateValues is the shared GROUP BY core: it buckets a sequence of projected value rows (each aligned to the attribute list AggProjection returned) by the (possibly time-bucketed) group tuple and reduces each group with the COUNT/SUM/ AVG/MIN/MAX accumulators, returning one AggRow per group in first-seen order. groupCol[i]/aggCol[i] index into each value row (aggCol[i] < 0 means COUNT(*)). With no group columns it returns a single row aggregating the whole sequence, still yielding one row over an empty sequence (SQL semantics: COUNT is 0, others undefined). If stop is non-nil it is polled per row and, when it returns true, the scan halts early with the groups accumulated so far -- used to abandon a scan whose client has gone away.

func ColumnarAggregate added in v0.25.2

func ColumnarAggregate(stats func(attr string) (NumStats, bool), groupCols []GroupCol, aggs []AggSpec) ([]AggRow, bool)

ColumnarAggregate answers aggs over constraint from the columnar accelerator when it can: exactly one aggregate, no grouping, no per-aggregate FILTER, and a numeric argument the current schema carries. ok=false means nothing was computed and the caller must scan.

COUNT(attr), MIN, MAX, SUM and AVG are served. The result TYPE follows the reference exactly (see numAggValue): SUM accumulates integers in int64 and only becomes a real once a real value appears, AVG is always a real, and MIN/MAX keep their element's type. A formatting difference would be as wrong as a numeric one and far easier to ship unnoticed, so the test compares text.

Three refusals. A FILTER is refused rather than approximated: the columnar pass knows nothing about it, so answering would report an unfiltered aggregate as if it were filtered. A column that turned up a BOOLEAN declines: the reference coerces booleans to 1/0 and then has a further quirk for a lone boolean element, which is not worth reproducing for data this pathological -- the scan gives the exact answer. And a MIN/MAX over a column holding a NaN declines, because the reference's answer there depends on the order the records are visited in (see minMaxDeclinesNaN).

func ColumnarUserAggregate added in v0.37.0

func ColumnarUserAggregate(values NumValueScan, groupCols []GroupCol, aggs []AggSpec) ([]AggRow, bool, error)

ColumnarUserAggregate answers one ungrouped user-defined aggregate from the columnar value pass, when the registered aggregate's accumulator takes column values. ok=false means nothing was computed and the caller must scan.

The refusals mirror ColumnarAggregate's and add one of its own. Grouping is refused because the pass yields values with no group key attached; a FILTER is refused rather than approximated, since the pass knows nothing about it and answering would report an unfiltered aggregate as a filtered one; and an accumulator that is not a ColumnAccumulator is refused because it has not made the two promises the pass requires.

func GroupedFromColumns added in v0.28.0

func GroupedFromColumns(src GroupStatsSource, constraint string, groupCols []GroupCol, aggs []AggSpec) ([]AggRow, bool)

GroupedFromColumns answers a GROUP BY from src's columns, or ok=false to scan.

Served: up to collections.MaxGroupCols group columns -- each numeric or STRING, each raw or floored into fixed-width buckets -- and aggregates drawn from COUNT(*), COUNT(attr), MIN, MAX, SUM and AVG over NUMERIC attributes the schema carries. A string group value is read through the block's string dictionary; an aggregate over a string column is still not this shape.

The bucketed form is what a time series asks ("per day", not "per second"), and it is the form this used to decline outright, sending every `GROUP BY time_bucket(t,'1d')` to a record scan while its ungrouped sibling was answered from the columns. Measured against a production history archive, the two differed by more than two orders of magnitude.

MORE THAN ONE column is the "per owner per day" shape -- one raw categorical column beside one bucketed time column -- which a dashboard asks constantly and which this declined outright for the same reason the bucketed form once did. The index tier answers the COUNT(*) form of it over a categorically indexed column with no WHERE clause, and nothing else: add an aggregate, a constraint, or a group column the index does not carry, and it fell to the record scan.

A per-aggregate FILTER is served: `COUNT(*) FILTER (WHERE JobStatus == 2)` beside a plain COUNT(*) answers two differently-conditioned questions in one pass, where it used to decline the whole query and send a shape the columns nearly answered back to the record scan. The filter is lowered to the same column narrowing the WHERE clause is lowered to, so it is served for exactly the expressions the WHERE clause is served for -- a conjunction of scalar comparisons against literals over numeric schema fields -- and a filter outside that declines the query.

COUNT(DISTINCT) declines: it needs the values, not their aggregate. More than MaxGroupCols columns declines: the storage key has that many component slots and a truncated key would merge groups the scan keeps apart. An UNGROUPED filtered aggregate declines too, because this whole path is the grouped one -- `SELECT COUNT(*) FILTER (WHERE ...)` with no GROUP BY still scans, and the ungrouped columnar rung above refuses a filter for the same reason the index rung does.

func GroupedUserFromColumns added in v0.37.0

func GroupedUserFromColumns(src GroupValuesSource, constraint string, groupCols []GroupCol,
	aggs []AggSpec) ([]AggRow, bool, error)

GroupedUserFromColumns answers a GROUP BY whose value aggregates are all user-defined, from src's columns. ok=false means nothing was computed and the caller must scan.

WHAT IT SERVES, and the one asymmetry with the built-in grouped rung. Every value aggregate must be an AggUser whose accumulator takes column values; COUNT(*) rides along, filtered or not, because a group's record count and its filtered counts are kept beside the accumulators rather than derived from them. A built-in SUM or MIN in the same request is refused: a group holds either the fixed statistics or the caller's accumulators, never both, so serving the mixture would mean two passes, and two passes over one request is a different plan rather than this one.

COLUMNS ARE NOT DE-DUPLICATED HERE, which is the opposite of what the built-in rung does. There, MIN(x) beside MAX(x) reads x once and shares one statistics accumulator, because the statistics are the same statistics. Two user-defined aggregates over one column are two different computations and need two accumulators, so each gets its own column entry and the column is read once per aggregate. Sharing them would hand both the same accumulator and silently answer one aggregate twice.

type AggSpec added in v0.16.8

type AggSpec struct {
	Func   AggFunc
	Arg    string
	Filter string

	// UserFunc names the registered aggregate when Func is AggUser, and is empty otherwise.
	// It is a separate field rather than an overload of Arg because a user-defined aggregate
	// still takes an argument attribute: p95(RequestMemory) is UserFunc "p95", Arg
	// "RequestMemory".
	UserFunc string
}

AggSpec is one aggregate in a query: a function over an argument attribute. Arg "*" (only meaningful for plain COUNT) counts every row in the group; otherwise Arg is an attribute name evaluated per ad.

Filter, when non-empty, is a ClassAd expression restricting this aggregate -- and only this one -- to the rows of its group where the expression is true (SQL's `COUNT(*) FILTER (WHERE ...)`). It is what lets one pass over the data answer several differently-conditioned questions at once:

SELECT Owner, COUNT(*), COUNT(*) FILTER (WHERE JobStatus == 2) FROM jobs GROUP BY Owner

Without it that is one scan per condition. The filter narrows an aggregate, never the group: a group whose rows all fail every filter still appears, with COUNT 0 and the other functions undefined, exactly as SQL has it.

type AggStrategy added in v0.32.1

type AggStrategy interface {
	// Name identifies the rung. It is descriptive -- it names the rung in the runner's error
	// and lets a test assert which rung served a shape -- and is deliberately not reported to
	// a caller, which would be a behaviour change rather than a description of one. The tier a
	// rung reports into the caller's ScanStats is the caller-visible statement, and several
	// rungs share one tier.
	Name() string
	// Answer serves req completely (ok=true) or declines it (ok=false, rows ignored).
	Answer(req AggRequest) (rows []AggRow, ok bool, err error)
}

AggStrategy is one rung of an aggregate ladder: it answers an AggRequest, or declines so the next rung may try.

Three rules hold for every implementation.

Decline freely, never answer partially. ok=false means "not me", for any reason and at any point -- a strategy that gets halfway and finds it cannot finish throws its partial result away and declines. The runner enforces the second half of that by ignoring the rows of a declining strategy outright, so a half-built result cannot reach a caller even by accident.

The last rung never declines. The record scan is the reference and always has an answer, so a ladder whose last rung declines is a programming error rather than a query the system cannot serve, and the runner reports it as one.

Every rung's answer equals the reference's for the same request, as a row SET: the index rung returns groups sorted by value and every other rung in first-seen order, and GROUP BY without ORDER BY defines neither. A rung that would render a value, partition a group or drop a row differently from the scan must decline instead. That is why a boolean in an aggregated column, a string argument, a per-aggregate FILTER, a predicate not proven free of CurrentTime and a chained table all fall through to the scan rather than being approximated.

err is separate from the decline: it reports a request that no rung can serve because it is malformed (COUNT(DISTINCT *), a filter that does not parse) or a read that failed, and it stops the ladder rather than advancing it. Today only the rungs that run the reduce engine -- the archive's match-none shortcut and both record scans -- can produce one; the rest answer or decline.

type Aggregate added in v0.37.0

type Aggregate interface {
	// NewAccumulator returns a fresh accumulator holding the aggregate's identity -- the answer
	// for a group that contributed no rows at all.
	NewAccumulator() Accumulator
}

Aggregate is a user-defined aggregate: a factory for the per-group accumulators that do the work. It carries no state itself, because one registration serves every group of every concurrent query.

func LookupAggregate added in v0.37.0

func LookupAggregate(name string) (Aggregate, bool)

LookupAggregate finds a registration by name, case-insensitively.

type ArchiveConfig added in v0.7.0

type ArchiveConfig struct {
	// SegmentSize is the sealed-segment file size in bytes (default 8 MiB).
	SegmentSize int
	// RowGroupBytes is the uncompressed record-bytes budget for one columnar row group. 0 takes the
	// default (see collections.Options.RowGroupBytes).
	//
	// Like the rest of this config it is read at CREATE time only -- a reopened archive uses the
	// persisted value. Use SetRowGroupBytes to change it on a live archive: unlike the index set, this
	// needs no reconciliation, because each block records its own layout, so a new budget governs
	// segments sealed from then on and older ones keep theirs.
	//
	// It trades point-lookup cost against how far compression can see across records: a larger group
	// stores the same data in fewer bytes and makes reading one record decompress more of it. The
	// default is measured on real ads, but the right point depends on the read mix and on how much of
	// the working set fits the block cache, which only a production-sized archive can answer.
	RowGroupBytes int
	// HotAttrs / CategoricalAttrs / ValueAttrs tune the per-segment hot header and
	// indexes; ZoneAttrs names numeric attributes to keep per-segment min/max on for
	// whole-segment query pruning (value-indexed attributes are included automatically).
	HotAttrs                     []string
	CategoricalAttrs, ValueAttrs []string
	// Unordered drops this archive's newest-first read order (collections.ArchiveOptions.
	// Unordered). An archive that is a history file must leave it alone: condor_history's
	// order is this table's whole contract. An archive whose rows are only aggregated or
	// joined has no such contract, and saying so lets the join planner choose which side to
	// hash on cost rather than on which one the query named first.
	//
	// Persisted with the rest of the config, because a reader that reordered its work on
	// Monday and preserved an order on Tuesday would be the worst of both.
	Unordered bool `json:",omitempty"`
	ZoneAttrs []string
	// AutoAttrs names the subset of the above created by the auto-tuner rather than by a
	// human. Provenance has to be persisted with the index set, or an auto index returns
	// from a restart indistinguishable from one someone asked for -- exempt from trimming
	// and from any future auto-drop, permanently, on the strength of nothing.
	AutoAttrs []string
	// CompositeAttrs configures composite (multi-column) indexes over attribute TUPLES (see
	// collections.ArchiveOptions.CompositeAttrs). Each tuple's attributes must also be indexed
	// in their own right, or the tuple is refused at open.
	//
	// It is persisted with the rest of the index set, and for the same reason: this file is
	// authoritative on reopen and rebuilds the whole set from it, so a composite missing from
	// it is a composite that does not come back -- along with the sidecar rebuild that creating
	// it paid for.
	CompositeAttrs [][]string `json:",omitempty"`
	// CategoricalComponentAttrs / ValueComponentAttrs name attributes that exist ONLY as
	// composite components: they carry no index of their own, so no constraint on one
	// alone is index-answered, and which list a name appears in fixes how the composite
	// encodes it. They are what lets a composite over (ClusterId, ProcId) be configured
	// without also paying for a standalone ProcId index nothing queries.
	//
	// Persisted with the rest of the index set, and `omitempty` for the same reason the
	// other index fields are: a configuration naming none must write the bytes it always
	// wrote.
	CategoricalComponentAttrs []string `json:",omitempty"`
	ValueComponentAttrs       []string `json:",omitempty"`
	// AutoComposites names the subset of CompositeAttrs the auto-tuner created, as AutoAttrs
	// does for the single-attribute indexes. Without it a composite the tuner created returns
	// from a restart indistinguishable from a configured one: no longer trimmable by the
	// memory budget, no longer droppable when nothing demands its tuple, permanently.
	AutoComposites [][]string `json:",omitempty"`
	// AutoHotAttrs names the hot attributes the demand-driven refresh (RefreshHotSet)
	// installed on top of HotAttrs, and is the hot-set counterpart of AutoAttrs: on reopen
	// the live set is HotAttrs plus these.
	//
	// It is a separate field rather than an update of HotAttrs because this file is
	// authoritative on reopen, so folding a sampled set back into HotAttrs would make the
	// tuner's guess indistinguishable from what an operator asked for -- and would quietly
	// discard the operator's list the first time a sample disagreed with it. HotAttrs stays
	// whatever a human configured; this records what the samples added.
	//
	// Without persisting it at all, a refreshed set would reset to the creation-time
	// HotAttrs on every restart, and the hot headers of the segments written since would be
	// the only trace that it ever ran.
	AutoHotAttrs []string
	// IndexBackfillBytes bounds how far back an index configuration change is carried at the
	// moment it is made: only segments within the newest N bytes have their existing index
	// rebuilt by AddIndex and Reindex; older ones keep the one they have until the
	// maintenance pass's budgeted backfill (MaintainOptions.ArchiveIndexBackfillBytes) or an
	// explicit ReindexAll reaches them (see collections.Options.IndexBackfillBytes).
	//
	// Zero takes defaultIndexBackfillBytes. A NEGATIVE value means no bound: the change is
	// carried across the whole archive in one go, which for history means decompressing every
	// record to add one index -- hours at scale, in a call that holds up whatever made it.
	// That is a reasonable thing to ask for deliberately and a bad thing to get by leaving a
	// field unset, which is why it is not what zero means.
	IndexBackfillBytes int64
	// SchemaRequireAttrs names attributes that enter the derived columnar schema whatever their
	// presence rate (see collections.Options.SchemaRequireAttrs). It is the CONFIGURED tier:
	// read at create, persisted here, and authoritative on reopen like the rest of this config,
	// so an application whose own configuration changed re-asserts it with SetSchemaRequireAttrs.
	//
	// An archive is where this matters most. Its records are written once, already in their
	// final state, so an attribute a record acquires only late in its life -- the resource
	// figures a job has only after it has run -- is simultaneously the thing history queries read
	// and the thing the presence floor excludes, because the records that never reached that
	// state are in the same table.
	SchemaRequireAttrs []string
	// SchemaRequireRuntime names the subset added at runtime rather than configured, kept apart
	// for the reason AutoAttrs is: a name that comes back from a restart should say where it came
	// from. SetSchemaRequireAttrs replaces the configured tier and leaves this one alone, so an
	// operator's experiment is not erased by the application re-asserting its config.
	SchemaRequireRuntime []string
	// GroupSchemaCount is how many secondary columnar schemas to derive for attributes the base
	// schema does not carry (see db.Config). 0 takes the default; a NEGATIVE value builds none.
	// An archive's maintenance cadence is long, so the stability gate's several derivations span
	// days rather than minutes -- the intended shape for a table whose structure changes on that
	// scale, and the reason an archive builds none for its first few days.
	GroupSchemaCount    int
	GroupStabilityRuns  int
	GroupMergeJaccard   float64
	GroupMaxPartialFrac float64
	// Retention bounds what rotation keeps (max segments / bytes / age). Zero keeps all.
	Retention collections.Retention
	// EncryptedAttrs is the explicit set of attributes sealed at rest (private attributes are always
	// sealed), as for a mutable table (see Config.EncryptedAttrs). Empty takes the catalog's default
	// set. A non-empty set is authoritative, and it is what SetEncryptedAttrs persists, so a runtime
	// change survives a restart exactly as a mutable table's does.
	EncryptedAttrs []string `json:",omitempty"`
}

ArchiveConfig configures an archive table. Dir is set by the catalog.

type ArchiveTable added in v0.7.0

type ArchiveTable struct {
	// contains filtered or unexported fields
}

An ArchiveTable is an append-only, rotated history store (the "condor history file" use case), exposed as a catalog table type alongside the mutable tables. Ads are appended once and never updated or deleted individually; old data is dropped in bulk by rotation. Queries are newest-first with an optional limit -- condor_history's "last K" -- with whole-segment pruning via zone maps. See collections.Archive.

func (*ArchiveTable) AcquireRead added in v0.33.0

func (t *ArchiveTable) AcquireRead() bool

AcquireRead registers the caller as an in-flight reader of the archive table; see DB.AcquireRead. Balance a true answer with ReleaseRead.

func (*ArchiveTable) AddHotAttrs added in v0.32.0

func (t *ArchiveTable) AddHotAttrs(names ...string) []string

AddHotAttrs pins the named attributes into the archive's hot set for future writes and persists them as human-configured, returning the resulting hot attributes.

func (*ArchiveTable) AddIndex added in v0.18.0

func (t *ArchiveTable) AddIndex(categorical, value []string) bool

AddIndex adds per-segment indexes on the named categorical and/or value attributes and rebuilds them over existing segments. Returns false if the index set was unchanged. A change is persisted (see saveIndexConfig).

func (*ArchiveTable) AddSchemaRequireAttrs added in v0.32.0

func (t *ArchiveTable) AddSchemaRequireAttrs(names ...string) (bool, error)

AddSchemaRequireAttrs / DropSchemaRequireAttrs change the set by name and persist it. Add goes to the runtime tier (it survives a restart); Drop removes from both, and dropping a configured attribute lasts only until the application re-asserts its configuration.

func (*ArchiveTable) Aggregate added in v0.16.8

func (t *ArchiveTable) Aggregate(constraint string, groupBy []string, aggs []AggSpec) ([]AggRow, error)

Aggregate runs a server-side GROUP BY over the archive's matches: it applies the constraint (using the archive's zone-map pruning, so segments no matching record can fall in are never scanned), groups by the raw group columns, and reduces each group with COUNT/SUM/AVG/MIN/MAX. It shares the exact grouping/reduce engine (AggregateValues) the mutable-table aggregate uses, so an archive aggregate behaves identically to the same aggregate over a live table -- only the (small) grouped result is produced, not every matched ad. With no group columns it returns a single row over the whole match.

func (*ArchiveTable) AggregateCols added in v0.23.1

func (t *ArchiveTable) AggregateCols(constraint string, groupCols []GroupCol, aggs []AggSpec) ([]AggRow, error)

AggregateCols is Aggregate where a group column may carry a bucket width, so a numeric attribute can be grouped into fixed-width buckets (the "per day" dimension) server-side.

func (*ArchiveTable) AggregateColsStats added in v0.29.6

func (t *ArchiveTable) AggregateColsStats(constraint string, groupCols []GroupCol, aggs []AggSpec, stats *collections.ScanStats) ([]AggRow, error)

AggregateColsStats is AggregateCols that also fills stats (may be nil) with the scan's work breakdown for EXPLAIN ANALYZE. Only the fall-through projected scan -- the slow path taken when the aggregated attribute is not a numeric field the columnar accelerator covers -- fills the per-record counts; the columnar/index fast paths do not reassemble records and leave them zero. So a nonzero, large RecordsReassembled here is the signature of the slow aggregate this trailer exists to surface.

Every return below also TAGS stats with the tier that served it (stats.Tier), because the strategies here differ by more than two orders of magnitude and zero counts were never a statement that a fast path ran -- only the absence of a scan, which a provably-empty constraint and a columnar answer produce alike. Tag at every return, including the ones that read nothing: an untagged return reports TierUnreported, which reads as "did not say" and cannot be told from a peer too old to report a tier at all.

func (*ArchiveTable) AggregateColsStatsRedacted added in v0.35.0

func (t *ArchiveTable) AggregateColsStatsRedacted(constraint string, groupCols []GroupCol, aggs []AggSpec, stats *collections.ScanStats) ([]AggRow, error)

AggregateColsStatsRedacted is AggregateColsStats for a caller NOT entitled to sealed values, as DB.AggregateColsStatsRedacted is for a mutable table: every rung answers as if the archive had no key, so a sealed value is undefined to the WHERE clause, the group key, the aggregate and its FILTER alike. A sealed attribute whose older records hold it in the clear reads as undefined in those too.

func (*ArchiveTable) Append added in v0.7.0

func (t *ArchiveTable) Append(ad *classad.ClassAd) error

Append adds one ad to the archive (append-only; there is no update or per-key delete). The ad is durable when Append returns.

func (*ArchiveTable) AppendDeferred added in v0.32.1

func (t *ArchiveTable) AppendDeferred(ad *classad.ClassAd) error

AppendDeferred adds one ad without waiting for it to reach disk: it is readable at once and durable at the next Flush. A caller persisting its own progress through the source calls Flush first. See collections.Archive.AppendDeferred.

func (*ArchiveTable) AppendOld added in v0.7.0

func (t *ArchiveTable) AppendOld(text string) error

AppendOld appends an ad parsed from old-ClassAd text (the qmgmt/history line format).

func (*ArchiveTable) AttrNDV added in v0.38.1

func (t *ArchiveTable) AttrNDV(attr string) (NDVEstimate, bool)

AttrNDV reports how many distinct values attr takes in this archive. See DB.AttrNDV.

func (*ArchiveTable) AttrTypeProfiles added in v0.34.2

func (t *ArchiveTable) AttrTypeProfiles(sampleMax int) ([]AttrTypeProfile, int)

AttrTypeProfiles profiles what this archive's attributes hold (see DB.AttrTypeProfiles).

func (*ArchiveTable) AttrTypeProfilesWith added in v0.36.0

func (t *ArchiveTable) AttrTypeProfilesWith(o AttrTypeOptions) ([]AttrTypeProfile, int)

AttrTypeProfilesWith is AttrTypeProfiles with the measurement selected by o (see DB.AttrTypeProfilesWith).

func (*ArchiveTable) AutoIndexNames added in v0.25.0

func (t *ArchiveTable) AutoIndexNames() []string

AutoIndexNames returns the names of the archive's auto-created indexes.

func (*ArchiveTable) AutoTune added in v0.25.0

func (t *ArchiveTable) AutoTune(opts AutoTuneOptions) AutoTuneResult

AutoTune adds indexes the observed query demand justifies, persisting the result so the backfill it just paid for is not thrown away and re-paid on the next restart.

Adds only: opts.DropUnused is forced off. On a mutable table an auto-drop is reversible at roughly the cost of the index; on an archive, re-adding means reading history back through the decompressor, so the cheap direction and the expensive direction are the wrong way round for acting on an absence of evidence.

func (*ArchiveTable) BackfillIndexes added in v0.33.0

func (t *ArchiveTable) BackfillIndexes(maxBytes int64) IndexBackfillResult

BackfillIndexes rebuilds, newest first and up to maxBytes of segment data (<= 0: no bound), the sidecars of sealed segments lacking postings for an indexed attribute.

func (*ArchiveTable) BuildAndEnableSchemaScan added in v0.25.2

func (t *ArchiveTable) BuildAndEnableSchemaScan(sampleMax, hotTopN int) bool

BuildAndEnableSchemaScan builds or extends the archive's columnar accelerator (see collections.Archive.BuildAndEnableSchemaScan).

func (*ArchiveTable) CategoricalGroupCounts added in v0.23.1

func (t *ArchiveTable) CategoricalGroupCounts(attr string) (map[string]int64, bool)

CategoricalGroupCounts returns the exact per-value record counts for a categorically indexed attribute, read from the per-segment indexes without scanning any record. ok is false when the indexes cannot fully account for every record (see the collections-level doc), in which case the caller must scan. Aggregate uses this for GROUP BY COUNT(*); it is exported so a planner can ask whether the cheap path is available.

func (*ArchiveTable) CategoricalGroupCountsBucketed added in v0.23.1

func (t *ArchiveTable) CategoricalGroupCountsBucketed(attr, bucketAttr string, width int64) (map[int64]map[string]int64, bool)

CategoricalGroupCountsBucketed is CategoricalGroupCounts split by a numeric bucket (floor(bucketAttr/width)*width) -- the "per group per day" shape. bucketAttr must be zone-mapped; ok is false when the counts cannot be established from the indexes.

Not yet reachable from SQL: the archive aggregate crosses dbrpc as a []string group list, which has no room for a bucket width, so a bucketed GROUP BY over an archive still falls back to client-side reduction. Wiring it through is a protocol change.

func (*ArchiveTable) CategoricalGroupCountsWhere added in v0.23.1

func (t *ArchiveTable) CategoricalGroupCountsWhere(attr, constraint string) (map[string]int64, bool)

CategoricalGroupCountsWhere is CategoricalGroupCounts restricted to records matching constraint. ok is false unless the constraint is a pure conjunction of numeric comparisons on zone-mapped attributes and the indexes account for every record.

func (*ArchiveTable) Close added in v0.7.0

func (t *ArchiveTable) Close() error

Close flushes and closes the archive.

func (*ArchiveTable) CodecStats added in v0.19.0

func (t *ArchiveTable) CodecStats(sampleMax int) CodecStats

CodecStats reports the archive's compression (codec, dict size, last retrain, sampled ratio).

func (*ArchiveTable) CompositeAttrs added in v0.38.1

func (t *ArchiveTable) CompositeAttrs() [][]string

CompositeAttrs returns the archive's composite index tuples, each in its configured order. As with a single-attribute index, these are built only in a sealed segment's sidecar, so HasSealedSidecar is what says whether any of them can answer a probe yet.

func (*ArchiveTable) Count added in v0.7.0

func (t *ArchiveTable) Count() int

Count is the number of records currently retained (reduced by rotation).

func (*ArchiveTable) CountConstraint added in v0.27.0

func (t *ArchiveTable) CountConstraint(constraint string) (int, bool)

CountConstraint counts the rows matching constraint via the columnar accelerator, or reports ok=false so the caller scans. See collections.Archive.CountConstraint.

func (*ArchiveTable) DropIndex added in v0.18.0

func (t *ArchiveTable) DropIndex(names ...string) bool

DropIndex removes the named per-segment indexes. Returns false if none matched. A change is persisted (see saveIndexConfig).

func (*ArchiveTable) DropSchemaRequireAttrs added in v0.32.0

func (t *ArchiveTable) DropSchemaRequireAttrs(names ...string) (bool, error)

func (*ArchiveTable) EncryptedAttrNames added in v0.29.0

func (t *ArchiveTable) EncryptedAttrNames() []string

EncryptedAttrNames reports the attributes this archive seals, matching DB.EncryptedAttrNames.

func (*ArchiveTable) EncryptionEnabled added in v0.29.0

func (t *ArchiveTable) EncryptionEnabled() bool

EncryptionEnabled reports ENCRYPTION AT REST, as DB.EncryptionEnabled does: the archive's master key is wrapped under pool keys. Without them the archive still seals (see SealsValues), but its master sits beside the data in the clear, which protects nothing and is reported here as false.

func (*ArchiveTable) Explain added in v0.25.2

func (t *ArchiveTable) Explain(constraint string) (QueryExplain, error)

MergePass merges cold segment runs when the archive has reached its trigger watermark, down to its target. Returns the number of merges performed.

An append-only table never compacts, so its segment count only grows, and every sealed segment costs a memory mapping at open. This is what bounds that; without it running, the count climbs until the daemon cannot start. Explain reports how a query against the archive would be executed (see collections.Archive.ExplainQuery), so `.explain` works on a history table rather than reporting it as a nonexistent table.

func (*ArchiveTable) Flush added in v0.32.1

func (t *ArchiveTable) Flush() error

Flush returns once every ad appended before it is durable, or reports why not. See collections.Archive.Flush.

func (*ArchiveTable) GCFloor added in v0.21.4

func (t *ArchiveTable) GCFloor() float64

GCFloor returns the current runtime GC watermark (0 when unset).

func (*ArchiveTable) GroupCountAll added in v0.28.0

func (t *ArchiveTable) GroupCountAll(groupAttr string) ([]collections.GroupCount, bool)

GroupCountAll is GroupCountConstraint over every row: the whole column's histogram. AggregateCols uses it for a grouped COUNT(*) whose constraint is match-all, which the predicate analysis behind GroupCountConstraint cannot serve because there is no predicate to analyze. See collections.Archive.GroupCountAll.

func (*ArchiveTable) GroupCountConstraint added in v0.28.0

func (t *ArchiveTable) GroupCountConstraint(constraint, groupAttr string) ([]collections.GroupCount, bool)

GroupCountConstraint answers a per-value COUNT(*) of groupAttr over the rows matching constraint via the columnar accelerator, or reports ok=false so the caller scans. AggregateCols uses it for the single-numeric-column GROUP BY shape; it is exported so a caller (and a test) can tell which tier answered. See collections.Archive.GroupCountConstraint.

func (*ArchiveTable) GroupSchemaAgreement added in v0.28.0

func (t *ArchiveTable) GroupSchemaAgreement(sampleMax, k int) GroupSchemaAgreement

GroupSchemaAgreement reports how well per-segment derivations agree with the archive-wide one.

func (*ArchiveTable) GroupSchemaChanges added in v0.29.13

func (t *ArchiveTable) GroupSchemaChanges() []GroupSchemaChange

GroupSchemaChanges returns the archive's committed-group change log (see DB.GroupSchemaChanges).

func (*ArchiveTable) GroupSchemaDrift added in v0.28.0

func (t *ArchiveTable) GroupSchemaDrift() GroupSchemaDrift

GroupSchemaDrift reports how the archive's derived groups have moved.

func (*ArchiveTable) GroupSchemaLastAgreement added in v0.29.13

func (t *ArchiveTable) GroupSchemaLastAgreement() (GroupSchemaLastAgreement, bool)

GroupSchemaLastAgreement returns the archive's last persisted per-segment agreement, if any.

func (*ArchiveTable) GroupSchemas added in v0.28.0

func (t *ArchiveTable) GroupSchemas(sampleMax, k int) GroupSchemaInfo

GroupSchemas derives and reports candidate group schemas for the archive (see DB.GroupSchemas).

func (*ArchiveTable) GroupStatsAll added in v0.28.0

func (t *ArchiveTable) GroupStatsAll(groupAttr string, aggAttrs []string) ([]collections.GroupStats, bool)

GroupStatsAll is GroupStatsConstraint over every row, for a constraint this layer has established is match-all (the predicate analysis behind GroupStatsConstraint has no predicate to work with there).

func (*ArchiveTable) GroupStatsAllAggs added in v0.33.2

func (t *ArchiveTable) GroupStatsAllAggs(groups []collections.GroupSpec,
	aggCols []collections.GroupAggCol, countFilters []string) ([]collections.GroupStats, bool)

GroupStatsAllAggs is GroupStatsConstraintAggs over every row, for a constraint this layer has established is match-all. See collections.Collection.GroupStatsAllAggs.

func (*ArchiveTable) GroupStatsAllBucketed added in v0.32.0

func (t *ArchiveTable) GroupStatsAllBucketed(groupAttr string, bucket int64, aggAttrs []string) ([]collections.GroupStats, bool)

GroupStatsAllBucketed is GroupStatsAll with the group column floored into buckets of bucket (0 = the raw value). See collections.Collection.GroupStatsQueryBucketed.

func (*ArchiveTable) GroupStatsAllCols added in v0.33.0

func (t *ArchiveTable) GroupStatsAllCols(groups []collections.GroupSpec, aggAttrs []string) ([]collections.GroupStats, bool)

GroupStatsAllCols is GroupStatsAllBucketed over SEVERAL group columns, for a constraint this layer has established is match-all. See collections.Collection.GroupStatsQueryCols.

func (*ArchiveTable) GroupStatsConstraint added in v0.28.0

func (t *ArchiveTable) GroupStatsConstraint(constraint, groupAttr string, aggAttrs []string) ([]collections.GroupStats, bool)

GroupStatsConstraint answers a per-group record count plus the aggregate inputs for each aggAttr over the rows matching constraint, via the columnar accelerator, or reports ok=false so the caller scans. AggregateCols uses it for the single-numeric-column GROUP BY shape; it is exported so a caller (and a test) can tell which tier answered. See collections.Archive.GroupStatsConstraint.

func (*ArchiveTable) GroupStatsConstraintAggs added in v0.33.2

func (t *ArchiveTable) GroupStatsConstraintAggs(constraint string, groups []collections.GroupSpec,
	aggCols []collections.GroupAggCol, countFilters []string) ([]collections.GroupStats, bool)

GroupStatsConstraintAggs is GroupStatsConstraintCols where an aggregate column may carry a per-aggregate FILTER. See collections.Collection.GroupStatsConstraintAggs.

func (*ArchiveTable) GroupStatsConstraintBucketed added in v0.32.0

func (t *ArchiveTable) GroupStatsConstraintBucketed(constraint, groupAttr string, bucket int64, aggAttrs []string) ([]collections.GroupStats, bool)

GroupStatsConstraintBucketed is GroupStatsConstraint with the group column floored into buckets of bucket (0 = the raw value). See collections.Collection.GroupStatsQueryBucketed.

func (*ArchiveTable) GroupStatsConstraintCols added in v0.33.0

func (t *ArchiveTable) GroupStatsConstraintCols(constraint string, groups []collections.GroupSpec, aggAttrs []string) ([]collections.GroupStats, bool)

GroupStatsConstraintCols is GroupStatsConstraintBucketed over SEVERAL group columns: the group is the tuple of their (individually bucketed) values, which is the "per owner per day" shape. See collections.Collection.GroupStatsQueryCols.

func (*ArchiveTable) GroupValuesAllAggs added in v0.37.0

func (t *ArchiveTable) GroupValuesAllAggs(groups []collections.GroupSpec,
	aggCols []collections.GroupAggCol, countFilters []string,
	newSink func() collections.GroupSink, mergeSink collections.MergeGroupSink) ([]collections.GroupValues, bool)

GroupValuesAllAggs is GroupValuesConstraintAggs over every row.

func (*ArchiveTable) GroupValuesConstraintAggs added in v0.37.0

func (t *ArchiveTable) GroupValuesConstraintAggs(constraint string, groups []collections.GroupSpec,
	aggCols []collections.GroupAggCol, countFilters []string,
	newSink func() collections.GroupSink, mergeSink collections.MergeGroupSink) ([]collections.GroupValues, bool)

GroupValuesConstraintAggs is GroupStatsConstraintAggs reduced into caller-supplied accumulators.

func (*ArchiveTable) HasSealedSidecar added in v0.38.0

func (t *ArchiveTable) HasSealedSidecar() bool

HasSealedSidecar reports whether any segment carries a mapped sidecar index, which is what says an index is built and not merely configured. See collections.Collection.HasSealedSidecar for why the join planner asks this rather than SidecarSizes().Segments > 0.

func (*ArchiveTable) HotAttrs added in v0.29.0

func (t *ArchiveTable) HotAttrs() []string

HotAttrs reports the archive's hot-header attributes, matching DB.HotAttrs for a mutable table.

func (*ArchiveTable) IndexBackfillPending added in v0.33.0

func (t *ArchiveTable) IndexBackfillPending() (segments int, bytes int64)

IndexBackfillPending reports the sealed segments, and their stored bytes, still lacking postings for some indexed attribute: the backfill's remaining work.

func (*ArchiveTable) IndexSizes added in v0.19.0

func (t *ArchiveTable) IndexSizes() IndexSizes

IndexSizes reports the per-attribute index byte footprint.

func (*ArchiveTable) IndexedAttrs added in v0.19.0

func (t *ArchiveTable) IndexedAttrs() (categorical, value []string)

IndexedAttrs returns the archive's categorical and value index attributes.

func (*ArchiveTable) MaintProgress added in v0.34.0

func (a *ArchiveTable) MaintProgress() collections.MaintProgress

MaintProgress reports the maintenance pass running on this archive, or the zero value when none is. It takes no lock, so it answers while the pass it describes holds the maintenance lock. See collections.Collection.MaintProgress.

func (*ArchiveTable) Maintain added in v0.38.3

func (t *ArchiveTable) Maintain(opts MaintainOptions)

Maintain runs one maintenance pass over this archive, as DB.Maintain does for a mutable table. Both kinds are driven from the same MaintainOptions; the STEPS differ, because what an append log needs maintaining is not what a mutable store needs, and that difference is the point rather than drift.

It lives here, beside DB.Maintain, rather than in whatever is driving the schedule. Until it did, an archive was maintained only by callers that happened to implement the pass themselves, so an embedder that opened a catalog and maintained its tables got nothing on the archive side -- no merge, no reindex, no tuning -- and the archive's segment count only grew. An append-only table never compacts, so that is not self-correcting: every sealed segment costs a memory mapping at open plus a second for its sidecar, exhausting vm.max_map_count stops the daemon STARTING, and the symptom does not appear until a restart.

The work is deliberately narrow -- drop whole segments past the configured retention, merge cold runs, rebuild indexes left stale by an interrupted pass, carry the index set to older segments within a per-pass budget, apply the demand-driven tuning, and, if asked, retrain the dictionary on a much longer clock.

REWRITING is not among them. It re-encodes every segment in place on every call, which at history scale is hours of I/O for a gain that is usually small, so it stays an explicit operator action. Retraining is in the pass but off unless configured, because it is cheap in the steady state and expensive exactly once -- see DefaultArchiveRetrainInterval.

Reports through MaintainOptions.OnAutoTune and OnRetrain rather than logging: nothing in this package logs, so the caller decides what is worth saying and names the table it happened to.

func (*ArchiveTable) MaintenanceStopping added in v0.34.0

func (a *ArchiveTable) MaintenanceStopping() bool

MaintenanceStopping reports whether this archive has been asked to stop maintenance. See collections.Collection.MaintenanceStopping.

func (*ArchiveTable) MergePass added in v0.24.1

func (t *ArchiveTable) MergePass(opts MergeOptions) int

func (*ArchiveTable) NumStats added in v0.25.2

func (t *ArchiveTable) NumStats(constraint, attr string) (NumStats, bool)

NumStats is DB.NumStats for an archive (history) table.

func (*ArchiveTable) NumValues added in v0.37.0

func (t *ArchiveTable) NumValues(constraint, attr string, newSink func() func(NumValue), merge func()) bool

NumValues is DB.NumValues for an archive (history) table.

func (*ArchiveTable) NumValuesRedacted added in v0.37.0

func (t *ArchiveTable) NumValuesRedacted(constraint, attr string, newSink func() func(NumValue), merge func()) bool

NumValuesRedacted is DB.NumValuesRedacted for an archive (history) table.

func (*ArchiveTable) OpStats added in v0.19.0

func (t *ArchiveTable) OpStats() OpStats

OpStats reports cumulative operational timings. An archive has no DB-level snapshot lock, so SnapshotLock is zero.

func (*ArchiveTable) Query added in v0.7.0

func (t *ArchiveTable) Query(constraint string) (iter.Seq[*classad.ClassAd], error)

Query returns the archived ads matching constraint, newest first. QueryLimit caps the result at the newest limit matches (<= 0 = all) -- the scan stops after the newest satisfying segments, so "last K" is cheap.

func (*ArchiveTable) QueryColumns added in v0.37.0

func (t *ArchiveTable) QueryColumns(constraint string, attrs []string, redact bool) (iter.Seq[*collections.Batch], error)

QueryColumns yields the values of attrs for the records matching constraint, in batches of typed columns, newest first (see DB.QueryColumns).

func (*ArchiveTable) QueryColumnsStats added in v0.37.0

func (t *ArchiveTable) QueryColumnsStats(constraint string, attrs []string, redact bool,
	stats *collections.ColumnScanStats) (iter.Seq[*collections.Batch], error)

QueryColumnsStats is QueryColumns that also fills stats (may be nil). See DB.QueryColumnsStats.

func (*ArchiveTable) QueryLimit added in v0.7.0

func (t *ArchiveTable) QueryLimit(constraint string, limit int) (iter.Seq[*classad.ClassAd], error)

func (*ArchiveTable) QueryLimitRedacted added in v0.35.0

func (t *ArchiveTable) QueryLimitRedacted(constraint string, limit int) (iter.Seq[*classad.ClassAd], error)

QueryLimitRedacted is QueryLimit for a caller NOT entitled to sealed values, as DB.QueryRedacted is for a mutable table: the constraint is matched and the ads decoded with no key, so a sealed attribute is undefined in both. An attribute the archive seals reads as undefined in its older records too, where it may be stored in the clear (see SetEncryptedAttrs).

func (*ArchiveTable) QueryProject added in v0.20.0

func (t *ArchiveTable) QueryProject(constraint string, attrs []string) (iter.Seq[[]classad.Value], error)

QueryProject scans the matching ads and yields each projected to just attrs' values, read wire-native where possible -- so an aggregate reads only the attributes it needs instead of fully decoding every record. Errors only on a malformed constraint.

func (*ArchiveTable) QueryProjectRedacted added in v0.36.0

func (t *ArchiveTable) QueryProjectRedacted(constraint string, attrs []string) (iter.Seq[[]classad.Value], error)

QueryProjectRedacted is QueryProject for a caller NOT entitled to sealed values: the constraint is matched and attrs are read with no key, so a sealed value is undefined to both. It is the archive counterpart of DB.QueryProjectRedacted, and a read surface that serves both table kinds needs the pair -- without it, an unprivileged peer reading history has no call to make.

func (*ArchiveTable) QueryProjectStats added in v0.29.6

func (t *ArchiveTable) QueryProjectStats(constraint string, attrs []string, stats *collections.ScanStats) (iter.Seq[[]classad.Value], error)

QueryProjectStats is QueryProject that also fills stats (may be nil) with the scan's work breakdown, so an EXPLAIN ANALYZE of an aggregate that falls to this projected scan can show the large RecordsReassembled that marks the slow per-record path.

func (*ArchiveTable) QueryProjectStatsRedacted added in v0.35.0

func (t *ArchiveTable) QueryProjectStatsRedacted(constraint string, attrs []string, stats *collections.ScanStats) (iter.Seq[[]classad.Value], error)

QueryProjectStatsRedacted is QueryProjectStats with QueryLimitRedacted's entitlement.

func (*ArchiveTable) QueryRawProjected added in v0.20.1

func (t *ArchiveTable) QueryRawProjected(constraint string, projection []string, redact bool) (iter.Seq[collections.RawAd], error)

QueryRawProjected yields each matching ad as a raw projected subset (only the projection attributes, rendered from the stored representation), newest first — the archive-side of the server-side projection op. redact strips private attributes. It mirrors db.DB.QueryRawProjected so the same wire op serves archives and mutable tables uniformly. chaseRefs is false, matching HTCondor's projection protocol (exactly the requested attributes).

func (*ArchiveTable) QueryRawProjectedRefs added in v0.24.0

func (t *ArchiveTable) QueryRawProjectedRefs(constraint string, projection []string, redact bool) (iter.Seq[collections.RawAd], error)

QueryRawProjectedRefs is QueryRawProjected with the projected expressions' attribute references resolved too, so each yielded ad evaluates self-contained. See db.DB.QueryRawProjectedRefs for why that is a separate call rather than the default.

func (*ArchiveTable) QueryRawProjectedRefsStats added in v0.29.1

func (t *ArchiveTable) QueryRawProjectedRefsStats(constraint string, projection []string, redact bool, stats *collections.ScanStats) (iter.Seq[collections.RawAd], error)

QueryRawProjectedRefsStats is QueryRawProjectedRefs that also fills stats (may be nil) with the per-scan work breakdown for EXPLAIN ANALYZE.

func (*ArchiveTable) QueryRawProjectedStats added in v0.29.0

func (t *ArchiveTable) QueryRawProjectedStats(constraint string, projection []string, redact bool, stats *collections.ScanStats) (iter.Seq[collections.RawAd], error)

QueryRawProjectedStats is QueryRawProjected that also fills stats (may be nil) with the per-scan work breakdown for EXPLAIN ANALYZE (segments scanned/pruned, records decided from columns vs reassembled, rows matched).

func (*ArchiveTable) QueryRawWire added in v0.35.0

func (t *ArchiveTable) QueryRawWire(constraint string, projection []string, redact bool) (iter.Seq[[]byte], error)

QueryRawWire yields each matching record as a self-contained WIRE-FORM ROW, newest first (see DB.QueryRawWire and collections.Archive.QueryRawWire). projection restricts the entries (empty = whole record); redact strips private attributes at the source.

func (*ArchiveTable) QueryRawWireRefs added in v0.35.0

func (t *ArchiveTable) QueryRawWireRefs(constraint string, projection []string, redact bool) (iter.Seq[[]byte], error)

QueryRawWireRefs is QueryRawWire whose projection also carries the attributes the projected expressions reference, transitively, so each yielded row EVALUATES self-contained. Which of the two a caller wants is decided as it is for a mutable table (see DB.QueryRawWireRefs): the plain form reproduces HTCondor's projection protocol, this one serves a consumer that will evaluate what it receives, for which the references are not optional.

func (*ArchiveTable) QueryRawWireRefsStats added in v0.35.0

func (t *ArchiveTable) QueryRawWireRefsStats(constraint string, projection []string, redact bool, stats *collections.ScanStats) (iter.Seq[[]byte], error)

QueryRawWireRefsStats is QueryRawWireRefs that also fills stats (may be nil).

func (*ArchiveTable) QueryRawWireStats added in v0.35.0

func (t *ArchiveTable) QueryRawWireStats(constraint string, projection []string, redact bool, stats *collections.ScanStats) (iter.Seq[[]byte], error)

QueryRawWireStats is QueryRawWire that also fills stats (may be nil) with the per-scan work breakdown. An empty/true constraint takes the no-WHERE scan path and leaves stats zero.

func (*ArchiveTable) RefreshHotSet added in v0.32.0

func (t *ArchiveTable) RefreshHotSet(sampleMax, topN int) int

RefreshHotSet recomputes the archive's hot set from accumulated query demand, installs it for future writes, and persists it. Returns how many attributes the sample chose.

The refreshed set governs records appended from now on and changes nothing already sealed. Older records pick it up as the merge pass folds their segments together, which re-encodes them under the hot set in effect at merge time; see collections.Archive.RefreshHotSet for why none of that needs a rewrite, and what a merge never reaches.

The configured HotAttrs are re-pinned afterwards: RefreshHotSet replaces the set outright, and an attribute an operator asked to keep hot should not be dropped because a 4096-record sample happened not to read it.

func (*ArchiveTable) Reindex added in v0.18.0

func (t *ArchiveTable) Reindex()

Reindex rebuilds the per-segment indexes of segments sealed since the last build, and of segments within the backfill horizon whose index predates the current configuration.

func (*ArchiveTable) ReindexAll added in v0.33.0

func (t *ArchiveTable) ReindexAll() IndexBackfillResult

ReindexAll is Reindex carried past the backfill horizon to every sealed segment, so each holds postings for every indexed attribute. It decompresses every segment that lacks one -- on a large archive, a long time -- and is meant for an operator who asked for it; the maintenance pass gets there incrementally through BackfillIndexes.

func (*ArchiveTable) ReleaseRead added in v0.33.0

func (t *ArchiveTable) ReleaseRead()

ReleaseRead balances an AcquireRead that returned true.

func (*ArchiveTable) ReschemaScan added in v0.25.3

func (t *ArchiveTable) ReschemaScan(sampleMax, hotTopN int) bool

ReschemaScan re-derives the archive's schema and adopts it (see DB.ReschemaScan).

func (*ArchiveTable) Retention added in v0.19.0

func (t *ArchiveTable) Retention() collections.Retention

Retention returns the archive's current retention bounds.

func (*ArchiveTable) RetrainDict added in v0.18.0

func (t *ArchiveTable) RetrainDict(sampleMax int) (int, error)

RetrainDict trains a fresh ZSTD dictionary from up to sampleMax records and recompresses every segment in place under it, returning the new dictionary's size in bytes.

func (*ArchiveTable) Rewrite added in v0.18.0

func (t *ArchiveTable) Rewrite() int

Rewrite recompresses and re-encodes every segment in place under the current codec and hot set, returning the number of records rewritten.

func (*ArchiveTable) Rotate added in v0.7.0

func (t *ArchiveTable) Rotate(now float64) (int, error)

Rotate drops sealed segments that fall outside the retention policy, given the current time (unix seconds, for age-based retention). Returns how many segments were dropped.

func (*ArchiveTable) RowGroupBytes added in v0.28.1

func (t *ArchiveTable) RowGroupBytes() int

RowGroupBytes reports the budget currently in effect (0 meaning the default).

func (*ArchiveTable) SaveDemand added in v0.25.0

func (t *ArchiveTable) SaveDemand()

SaveDemand checkpoints the archive's recorded query demand so index decisions survive a restart, ageing it by the time since the last checkpoint. See collections.SaveDemand.

func (*ArchiveTable) SchemaFit added in v0.25.3

func (t *ArchiveTable) SchemaFit(sampleMax int) ([]SchemaFieldFit, int)

SchemaFit measures the archive's derived schema against a fresh sample (see DB.SchemaFit).

func (*ArchiveTable) SchemaRequireAttrs added in v0.32.0

func (t *ArchiveTable) SchemaRequireAttrs() []string

SchemaRequireAttrs returns the archive's effective required-attribute set, and SchemaRequireSkipped the members of it the last derivation could not make a field of.

func (*ArchiveTable) SchemaRequireSkipped added in v0.32.0

func (t *ArchiveTable) SchemaRequireSkipped() []SchemaRequireSkip

func (*ArchiveTable) SchemaScanInfo added in v0.25.2

func (t *ArchiveTable) SchemaScanInfo() SchemaScanInfo

SchemaScanInfo reports the archive's columnar accelerator state.

func (*ArchiveTable) SealedIndexes added in v0.35.0

func (t *ArchiveTable) SealedIndexes() []string

SealedIndexes names the configured indexes this open left out because the sealing policy names their attribute. Normally empty; see openArchiveTable.

func (*ArchiveTable) SealsValues added in v0.35.0

func (t *ArchiveTable) SealsValues() bool

SealsValues reports whether this archive seals any attribute values at all -- private attributes always, and the explicit set -- as DB.SealsValues does. A table that seals nothing has nothing to withhold from a reader not entitled to sealed values.

func (*ArchiveTable) SetEncryptedAttrs added in v0.35.0

func (t *ArchiveTable) SetEncryptedAttrs(attrs []string) error

SetEncryptedAttrs replaces the explicit set of attributes sealed at rest and persists it (archiveconfig.json), as DB.SetEncryptedAttrs does for a mutable table. It errors if the archive has no key or a named attribute is indexed.

Appends from now on seal the new set. Records already appended keep the form they were written in until maintenance next rewrites their segment (a merge, a retrain, interning), which seals them; a history is not rewritten just for this. Until then a read not entitled to sealed values withholds the attribute by name, so what such a reader sees does not depend on when a record was written.

func (*ArchiveTable) SetGCFloor added in v0.21.4

func (t *ArchiveTable) SetGCFloor(floor float64)

SetGCFloor installs a runtime GC watermark (in the archive's MinAgeAttr units) so the next Rotate may reclaim already-consumed records early: a change-feed source passes the feed's GC floor (min ack over live subscribers) here to drain records every subscriber has read. It only shortens retention -- it never keeps data past the configured Retention ceilings, and never drops anything younger than Retention.MinAge. Unlike SetRetention this is NOT persisted -- the caller re-asserts it from the current live floor each pass, so a stale value can never GC data across a restart. Passing floor <= 0 clears it.

func (*ArchiveTable) SetRetention added in v0.19.0

func (t *ArchiveTable) SetRetention(r collections.Retention) error

SetRetention updates the retention bounds and persists them (archiveconfig.json), so they take effect on the next Rotate and survive a restart.

func (*ArchiveTable) SetRowGroupBytes added in v0.28.1

func (t *ArchiveTable) SetRowGroupBytes(n int) error

SetRowGroupBytes changes the columnar row-group budget and persists it (archiveconfig.json), so a value tried on a live archive survives a restart. 0 restores the default.

Unlike the index set this needs no reconciliation pass: every block records the layout it was written with, so the new budget governs segments sealed from now on while everything already on disk keeps reading as before. That is what makes it worth exposing -- the right value depends on the read mix and on how much of the working set fits the block cache, which only a production-sized archive can answer, and finding it must not mean rebuilding the archive.

func (*ArchiveTable) SetSchemaRequireAttrs added in v0.32.0

func (t *ArchiveTable) SetSchemaRequireAttrs(names []string) (bool, error)

SetSchemaRequireAttrs replaces the CONFIGURED required-attribute set and persists it (archiveconfig.json), which is how an application re-asserts its own configuration onto an archive that was created with a different one -- archiveconfig.json is authoritative on reopen, so the ArchiveConfig field alone takes effect only at create.

It reports whether the effective set changed, and re-derives nothing itself. The next maintenance pass adopts the schema the new set produces and rewrites the sealed segments onto it a ColumnarSegmentBudget per pass, so at history scale the change converges over several passes rather than reading the whole archive back at once.

func (*ArchiveTable) SidecarBytes added in v0.40.2

func (t *ArchiveTable) SidecarBytes() (segments int, mappedBytes int64)

SidecarBytes reports the sidecar count and mapped bytes without the structural breakdown. See collections.Collection.SidecarBytes.

func (*ArchiveTable) SidecarSizes added in v0.19.0

func (t *ArchiveTable) SidecarSizes() SidecarSizes

SidecarSizes reports the sealed-segment sidecar index bytes (mmap-backed, evictable).

func (*ArchiveTable) StaleIndexSegments added in v0.21.2

func (t *ArchiveTable) StaleIndexSegments() (stale, sealed int)

StaleIndexSegments reports how many sealed segments within the backfill horizon still carry an index built under an older configuration, and how many are sealed in total. Normally zero; segments beyond the horizon are counted by StaleIndexSegmentsByPolicy and, the ones missing an indexed attribute, by IndexBackfillPending.

func (*ArchiveTable) StaleIndexSegmentsByPolicy added in v0.25.0

func (t *ArchiveTable) StaleIndexSegmentsByPolicy() int

StaleIndexSegmentsByPolicy reports segments deliberately left on an older index configuration because they fall outside the backfill horizon -- expected, and not a sign of anything failing. Reported apart from StaleIndexSegments so an alert on the latter stays meaningful once a horizon is configured.

func (*ArchiveTable) Stats added in v0.19.0

func (t *ArchiveTable) Stats() Stats

Stats reports storage accounting (records, segments, arena/used/dead bytes).

func (*ArchiveTable) StopMaintenance added in v0.30.4

func (a *ArchiveTable) StopMaintenance()

StopMaintenance asks any maintenance pass running on this archive to stop at its next safe boundary. See collections.Collection.StopMaintenance.

func (*ArchiveTable) SuggestIndexes added in v0.38.3

func (t *ArchiveTable) SuggestIndexes(sampleMax int) []IndexSuggestion

SuggestIndexes reports the attributes observed query demand would justify indexing, ordered by demand, from the same tracker AutoTune acts on. Advisory: nothing is added by reporting it. See DB.SuggestIndexes.

There is no ArchiveTable.SuggestDrops. An archive's cheap direction and expensive direction are the wrong way round for an absence of evidence, for the reason AutoTune gives below: dropping an index is cheap and re-adding one reads history back through the decompressor. Naming an index for a human to drop puts that trade in front of someone who did not ask for it, which is the half of the asymmetry the add suggestions do not carry.

func (*ArchiveTable) TopK added in v0.29.1

func (t *ArchiveTable) TopK(constraint string, attrs []string, orderAttr string, desc bool, k int) ([][]classad.Value, error)

TopK returns the k rows matching constraint ordered by orderAttr -- the largest k when desc, the smallest k otherwise -- each projected to attrs, in sorted order. It is the server-side form of ORDER BY <orderAttr> {DESC|ASC} LIMIT k: only k rows are ever held or returned, not the whole match set, so a "newest few of a million" query does not ship a million rows to be sorted client-side.

orderAttr must resolve to a number (ClusterId, QDate, CompletionDate, ...); a row whose order value is not numeric has no place in the ordering and is skipped. If orderAttr is not among attrs it is fetched for the comparison and trimmed from the returned rows.

func (*ArchiveTable) TopKRedacted added in v0.35.0

func (t *ArchiveTable) TopKRedacted(constraint string, attrs []string, orderAttr string, desc bool, k int) ([][]classad.Value, error)

TopKRedacted is TopK for a caller NOT entitled to sealed values, as DB.TopKRedacted is for a mutable table: the constraint is matched, the rows ordered and the projection read with no key.

func (*ArchiveTable) TopKRowsPerGroup added in v0.42.0

func (t *ArchiveTable) TopKRowsPerGroup(constraint string, spec TopKRowsSpec) ([]TopKRowsGroup, error)

TopKRowsPerGroup is DB.TopKRowsPerGroup over a history table.

func (*ArchiveTable) TopKRowsPerGroupRedacted added in v0.42.0

func (t *ArchiveTable) TopKRowsPerGroupRedacted(constraint string, spec TopKRowsSpec) ([]TopKRowsGroup, error)

TopKRowsPerGroupRedacted is DB.TopKRowsPerGroupRedacted over a history table.

func (*ArchiveTable) TopKRowsPerGroupRedactedStats added in v0.42.0

func (t *ArchiveTable) TopKRowsPerGroupRedactedStats(constraint string, spec TopKRowsSpec, stats *collections.ScanStats) ([]TopKRowsGroup, error)

TopKRowsPerGroupRedactedStats is ArchiveTable.TopKRowsPerGroupRedacted filling stats.

func (*ArchiveTable) TopKRowsPerGroupStats added in v0.42.0

func (t *ArchiveTable) TopKRowsPerGroupStats(constraint string, spec TopKRowsSpec, stats *collections.ScanStats) ([]TopKRowsGroup, error)

TopKRowsPerGroupStats is ArchiveTable.TopKRowsPerGroup filling stats (may be nil).

func (*ArchiveTable) TopKRowsPerGroupWire added in v0.42.0

func (t *ArchiveTable) TopKRowsPerGroupWire(constraint string, spec TopKRowsSpec) ([]TopKRowsWireGroup, error)

TopKRowsPerGroupWire is DB.TopKRowsPerGroupWire over a history table.

func (*ArchiveTable) TopKRowsPerGroupWireRedacted added in v0.42.0

func (t *ArchiveTable) TopKRowsPerGroupWireRedacted(constraint string, spec TopKRowsSpec) ([]TopKRowsWireGroup, error)

TopKRowsPerGroupWireRedacted is DB.TopKRowsPerGroupWireRedacted over a history table.

func (*ArchiveTable) TopKRowsPerGroupWireRedactedStats added in v0.42.0

func (t *ArchiveTable) TopKRowsPerGroupWireRedactedStats(constraint string, spec TopKRowsSpec, stats *collections.ScanStats) ([]TopKRowsWireGroup, error)

TopKRowsPerGroupWireRedactedStats is ArchiveTable.TopKRowsPerGroupWireRedacted filling stats.

func (*ArchiveTable) TopKRowsPerGroupWireStats added in v0.42.0

func (t *ArchiveTable) TopKRowsPerGroupWireStats(constraint string, spec TopKRowsSpec, stats *collections.ScanStats) ([]TopKRowsWireGroup, error)

TopKRowsPerGroupWireStats is ArchiveTable.TopKRowsPerGroupWire filling stats (may be nil).

func (*ArchiveTable) TopKStats added in v0.29.4

func (t *ArchiveTable) TopKStats(constraint string, attrs []string, orderAttr string, desc bool, k int) ([][]classad.Value, collections.ScanStats, error)

TopKStats is TopK that also returns the cutoff scan's ScanStats, for EXPLAIN ANALYZE.

func (*ArchiveTable) TopKStatsRedacted added in v0.35.0

func (t *ArchiveTable) TopKStatsRedacted(constraint string, attrs []string, orderAttr string, desc bool, k int) ([][]classad.Value, collections.ScanStats, error)

TopKStatsRedacted is TopKStats with TopKRedacted's entitlement.

func (*ArchiveTable) Truncate added in v0.20.1

func (t *ArchiveTable) Truncate()

Truncate drops every record, resetting the archive to empty in place (see collections.Archive.Truncate). It is the destructive reset behind a from-scratch history re-sync: empty the table, then re-ingest from the source. The persisted config (indexes, zone maps, retention) is retained, so the archive keeps its shape.

func (*ArchiveTable) Unordered added in v0.38.0

func (t *ArchiveTable) Unordered() (u bool)

Unordered reports that this archive gives up newest-first reads (ArchiveConfig.Unordered), so a reader may produce its rows in whatever order is cheapest.

func (*ArchiveTable) Watch added in v0.20.2

func (t *ArchiveTable) Watch(ctx context.Context, cursor []byte) (iter.Seq[WatchEvent], error)

Watch streams the archive's change events (append-only: upserts and the catch-up/live markers, no deletes), converting collections.WatchEvent to the db WatchEvent used by the mutable-table watch so the two are wire-identical. An empty cursor replays the retained history, then goes live. See collections.Archive.Watch.

func (*ArchiveTable) WatchCursor added in v0.29.12

func (t *ArchiveTable) WatchCursor() ([]byte, error)

WatchCursor returns an opaque cursor at the current head of the archive's change log. Watch(cursor) then streams only what is appended after it, which is how a client tails an archive without first replaying everything rotation still retains.

func (*ArchiveTable) WatchRedacted added in v0.35.0

func (t *ArchiveTable) WatchRedacted(ctx context.Context, cursor []byte) (iter.Seq[WatchEvent], error)

WatchRedacted is Watch for a watcher NOT entitled to sealed values: each event's ad is decoded with no key, so a sealed attribute arrives undefined. See DB.WatchRedacted.

func (*ArchiveTable) WatchWatermark added in v0.40.0

func (t *ArchiveTable) WatchWatermark() uint64

WatchWatermark is the archive's closed watch clock, as DB.WatchWatermark is a table's: the version a catch-up has caught up to, for the replayed events that carry no stamp of their own. 0 when the archive has no watch.

func (*ArchiveTable) ZoneAttrs added in v0.21.2

func (t *ArchiveTable) ZoneAttrs() []string

ZoneAttrs returns the attributes carrying per-segment [min,max] zone maps, on which a range query prunes whole segments rather than only postings.

type AttrTypeOptions added in v0.36.0

type AttrTypeOptions = collections.AttrTypeOptions

AttrTypeOptions selects what an AttrTypeProfilesWith call measures (see collections.AttrTypeOptions).

type AttrTypeProfile added in v0.34.2

type AttrTypeProfile = collections.AttrTypeProfile

MergeOptions tunes a merge pass over an archive; see collections.MergeOptions. The zero value is usable and fills in defaults. AttrTypeProfile is one attribute's measured type profile (see collections.AttrTypeProfile).

type AttrValueCounts added in v0.36.0

type AttrValueCounts = collections.AttrValueCounts

AttrValueCounts is one measurement within a profile (see collections.AttrValueCounts).

type AutoTuneChange added in v0.38.2

type AutoTuneChange = collections.AutoTuneChange

type AutoTuneOptions added in v0.25.0

type AutoTuneOptions = collections.AutoTuneOptions

AutoTuneOptions, AutoTuneResult and AutoTuneChange are re-exported so a caller can tune an archive's indexes, and read back what a pass changed, without importing collections directly. AutoTuneChange is what a result's Changes hold, so a caller handed one by MaintainOptions.OnAutoTune needs it to say anything about them.

type AutoTuneResult added in v0.25.0

type AutoTuneResult = collections.AutoTuneResult

type Catalog

type Catalog struct {
	// contains filtered or unexported fields
}

A Catalog is a set of named tables, each an independent ClassAd store (its own keyspace, indexes, hot set, and persisted index config). Tables give the database more than one collection to work with -- e.g. separate machine and job ads -- without joins. A persistent catalog keeps each table in its own subdirectory (<dir>/tables/<name>), so tables are isolated on disk and the set is recovered by enumerating those subdirectories on open.

func OpenCatalog

func OpenCatalog(dir string) (*Catalog, error)

OpenCatalog opens the catalog rooted at dir with no encryption. See OpenCatalogConfig to enable encryption at rest.

func OpenCatalogConfig added in v0.7.0

func OpenCatalogConfig(cfg CatalogConfig) (*Catalog, error)

func (*Catalog) ArchiveTable added in v0.7.0

func (cat *Catalog) ArchiveTable(name string) (*ArchiveTable, bool)

ArchiveTable returns the archive (history) table named name.

func (*Catalog) ArchiveTables added in v0.7.0

func (cat *Catalog) ArchiveTables() []string

ArchiveTables returns the archive table names, sorted.

func (*Catalog) AttrNDV added in v0.38.1

func (cat *Catalog) AttrNDV(table, attr string) (NDVEstimate, bool)

AttrNDV reports one table's distinct-value count for attr, over either kind of table, so a join planner can ask about both sides the same way. ok is false when there is no such table or nothing indexes the attribute in it -- deliberately the same answer, because both mean the planner has no statistic and neither is a cardinality.

func (*Catalog) Close

func (cat *Catalog) Close() error

Close closes every table.

func (*Catalog) ConvertTableToMemory added in v0.10.2

func (cat *Catalog) ConvertTableToMemory(name string) error

ConvertTableToMemory drops the on-disk backing of an existing persistent table, keeping its current contents live in RAM only. It copies the table through a consistent snapshot into a fresh in-memory table (preserving ads, index configuration, hot set, codec, and encryption policy), swaps that in, then closes and deletes the on-disk original.

It is a no-op when the catalog itself is in-memory or the table is already RAM-only. Like Rewrite, it takes a consistent snapshot but does not globally quiesce writers, so a write that races the swap can be lost -- run it during low write activity. DAEMON-level: it changes a table's durability, so callers gate it above ordinary WRITE.

func (*Catalog) CreateArchiveTable added in v0.7.0

func (cat *Catalog) CreateArchiveTable(name string, cfg ArchiveConfig) (*ArchiveTable, error)

CreateArchiveTable creates (or returns the existing) append-only archive table named name, persisted under <dir>/archives/<name>. cfg configures indexes / zone maps / retention on first creation and is ignored for an existing archive.

func (*Catalog) CreateExporter added in v0.14.0

func (cat *Catalog) CreateExporter(def ExporterDef) error

CreateExporter registers a new exporter definition. The name must be a valid identifier and must not already be registered. Exporters occupy their own namespace, so an exporter may share a name with a table (an exporter named "jobs" that mirrors the "jobs" table is expected). On a persistent catalog the definition is written to disk before it is returned; on an in-memory catalog it lives only in memory.

func (*Catalog) CreateTable

func (cat *Catalog) CreateTable(name string) (*DB, error)

CreateTable creates (or returns the existing) table named name. Its data persists under <dir>/tables/<name> for a persistent catalog.

func (*Catalog) CreateTableInMemory added in v0.10.2

func (cat *Catalog) CreateTableInMemory(name string) (*DB, error)

CreateTableInMemory creates (or returns the existing) table named name as RAM-only, even in a persistent catalog -- shorthand for CreateTableOpts with InMemory set. If the table already exists its backing is unchanged (options are not re-applied); use ConvertTableToMemory to drop an existing persistent table's on-disk backing.

func (*Catalog) CreateTableOpts added in v0.10.1

func (cat *Catalog) CreateTableOpts(name string, opts TableOptions) (*DB, error)

CreateTableOpts is CreateTable with options; see TableOptions. If the table already exists it is returned as-is (options are not re-applied).

func (*Catalog) CreateView added in v0.12.0

func (cat *Catalog) CreateView(name string, spec ViewSpec) error

CreateView creates a materialized view named name over spec.BaseTable, materializing it synchronously (so a cardinality-limit overflow or an aggregation error fails the create and leaves nothing behind) and then maintaining it live from the base table's change stream. The definition is persisted for a persistent catalog.

func (*Catalog) DropArchiveTable added in v0.7.0

func (cat *Catalog) DropArchiveTable(name string) error

DropArchiveTable closes and removes the archive named name, deleting its on-disk data.

func (*Catalog) DropExporter added in v0.14.0

func (cat *Catalog) DropExporter(name string) error

DropExporter removes an exporter's definition and its resume state. It is not an error to drop an exporter that does not exist (idempotent teardown).

func (*Catalog) DropTable

func (cat *Catalog) DropTable(name string) error

DropTable closes and removes the table named name, deleting its on-disk data.

A query already streaming out of the table keeps reading it: the drop marks the table closed (nothing new may start reading it, and it is out of the catalog before anything else happens) and the actual unmap is handed to the last reader on its way out. Without that, unmapping under a slow consumer faulted the whole process -- a Go runtime fault, so not something a per-connection recover could contain. DropTable itself never waits: the reader is an iterator whose end it cannot see, and the catalog lock is held here.

The directory goes immediately regardless. Unlinking a mapped file does not disturb the mapping, and a deferred teardown writes nothing (DB.dropClose discards rather than flushes), so a table recreated under the same name afterwards cannot have its fresh files deleted or overwritten by the old one's tail.

func (*Catalog) DropView added in v0.12.0

func (cat *Catalog) DropView(name string) error

DropView removes a view: it stops the updater, drops the in-memory data, and deletes the persisted definition.

func (*Catalog) EnsureTable

func (cat *Catalog) EnsureTable(name string) (*DB, error)

EnsureTable creates the table if it does not exist, returning it.

func (*Catalog) ExplainJoin added in v0.38.0

func (cat *Catalog) ExplainJoin(spec JoinSpec, want JoinLimits) (JoinExplain, error)

ExplainJoin reports how the catalog would run this join, reading no records.

want lowers the limits the way Join does; the limits actually applied come back in the report, because a plan that will stop at a limit the caller did not set is a plan the caller needs to know about.

Example

ExampleCatalog_ExplainJoin reports the plan without running the join.

It is the way to find out which of the three paths a spec will take -- and, for a band, whether the step is windowed at all. A band engages only on a step with no equality, so a spec carrying both answers the same rows by hashing, and Banded says so.

package main

import (
	"fmt"

	"github.com/PelicanPlatform/classad/classad"
	"github.com/PelicanPlatform/classad/db"
)

// joinFixture builds an in-memory catalog holding the tables these examples join.
//
// A catalog with no Dir keeps every table in RAM, which is what makes a join runnable in an
// example: nothing is written, nothing has to be cleaned up, and the plan the store picks is
// the same one it would pick over a directory.
func joinFixture() *db.Catalog {
	cat, err := db.OpenCatalog("")
	if err != nil {
		panic(err)
	}
	rows := map[string][]string{

		"jobs": {
			`[ ClusterId = 1; Owner = "alice"; RequestCpus = 2 ]`,
			`[ ClusterId = 2; Owner = "bob";   RequestCpus = 8 ]`,
			`[ ClusterId = 3; Owner = "alice"; RequestCpus = 4 ]`,
			`[ ClusterId = 4; Owner = "carol"; RequestCpus = 1 ]`,
		},
		"owners": {
			`[ Owner = "alice"; AcctGroup = "physics" ]`,
			`[ Owner = "bob";   AcctGroup = "chem" ]`,
		},
		"groups": {
			`[ AcctGroup = "physics"; Quota = 400 ]`,
			`[ AcctGroup = "chem";    Quota = 100 ]`,
		},

		"runs": {
			`[ ClusterId = 1; Start = 100; End = 140 ]`,
			`[ ClusterId = 2; Start = 200; End = 220 ]`,
		},
		"events": {
			`[ Ts = 210; Kind = "evict" ]`,
			`[ Ts = 105; Kind = "start" ]`,
			`[ Ts = 500; Kind = "idle" ]`,
			`[ Ts = 139; Kind = "done" ]`,
			`[ Ts = 205; Kind = "start" ]`,
		},
	}
	for _, name := range []string{"jobs", "owners", "groups", "runs", "events"} {
		t, terr := cat.CreateTable(name)
		if terr != nil {
			panic(terr)
		}
		tx := t.Begin()
		for i, src := range rows[name] {
			ad, perr := classad.Parse(src)
			if perr != nil {
				panic(perr)
			}
			tx.NewClassAd(fmt.Sprintf("%s%d", name, i), ad)
		}
		if cerr := tx.Commit(); cerr != nil {
			panic(cerr)
		}
	}
	return cat
}

func main() {
	cat := joinFixture()
	defer cat.Close()

	spec := db.JoinSpec{
		Kind:  db.JoinInner,
		Left:  db.JoinSide{Table: "runs", Alias: "r"},
		Right: db.JoinSide{Table: "events", Alias: "e"},
		Cond:  []string{"e.Ts >= r.Start", "e.Ts <= r.End"},
		Band:  []db.JoinBand{{Attr: "Ts", Lo: "r.Start", Hi: "r.End"}},
	}
	ex, err := cat.ExplainJoin(spec, db.JoinLimits{})
	if err != nil {
		panic(err)
	}
	for i := range ex.Steps {
		st := &ex.Steps[i]
		fmt.Printf("step %s banded=%v bands=%d\n", st.Alias, st.Banded, len(st.Band))
	}

}
Output:
step e banded=true bands=1

func (*Catalog) Exporter added in v0.14.0

func (cat *Catalog) Exporter(name string) (ExporterDef, bool)

Exporter returns a single exporter definition.

func (*Catalog) Exporters added in v0.14.0

func (cat *Catalog) Exporters() []ExporterDef

Exporters returns all registered exporter definitions, sorted by name.

func (*Catalog) Join added in v0.35.0

func (cat *Catalog) Join(spec JoinSpec, want JoinLimits, stats *JoinStats) (iter.Seq[*classad.ClassAd], error)

Join pairs two or more tables and streams the joined rows.

Setup problems -- an unknown table, an expression that does not parse -- are returned before any row. A limit met part way through is reported in stats.Err, because by then the sequence has already yielded rows and has no other way to say it stopped early.

Example

ExampleCatalog_Join pairs two tables on an equality.

This is the two-table spelling: Left, Right, and the condition between them. Cond is the join condition in full and is what decides a pair; Equi names the equality conjuncts of it that may also drive the hash. Equi is an optimisation and Cond is the meaning, so a pair that the hash brought together is still checked against Cond.

package main

import (
	"fmt"
	"sort"

	"github.com/PelicanPlatform/classad/classad"
	"github.com/PelicanPlatform/classad/db"
)

// joinFixture builds an in-memory catalog holding the tables these examples join.
//
// A catalog with no Dir keeps every table in RAM, which is what makes a join runnable in an
// example: nothing is written, nothing has to be cleaned up, and the plan the store picks is
// the same one it would pick over a directory.
func joinFixture() *db.Catalog {
	cat, err := db.OpenCatalog("")
	if err != nil {
		panic(err)
	}
	rows := map[string][]string{

		"jobs": {
			`[ ClusterId = 1; Owner = "alice"; RequestCpus = 2 ]`,
			`[ ClusterId = 2; Owner = "bob";   RequestCpus = 8 ]`,
			`[ ClusterId = 3; Owner = "alice"; RequestCpus = 4 ]`,
			`[ ClusterId = 4; Owner = "carol"; RequestCpus = 1 ]`,
		},
		"owners": {
			`[ Owner = "alice"; AcctGroup = "physics" ]`,
			`[ Owner = "bob";   AcctGroup = "chem" ]`,
		},
		"groups": {
			`[ AcctGroup = "physics"; Quota = 400 ]`,
			`[ AcctGroup = "chem";    Quota = 100 ]`,
		},

		"runs": {
			`[ ClusterId = 1; Start = 100; End = 140 ]`,
			`[ ClusterId = 2; Start = 200; End = 220 ]`,
		},
		"events": {
			`[ Ts = 210; Kind = "evict" ]`,
			`[ Ts = 105; Kind = "start" ]`,
			`[ Ts = 500; Kind = "idle" ]`,
			`[ Ts = 139; Kind = "done" ]`,
			`[ Ts = 205; Kind = "start" ]`,
		},
	}
	for _, name := range []string{"jobs", "owners", "groups", "runs", "events"} {
		t, terr := cat.CreateTable(name)
		if terr != nil {
			panic(terr)
		}
		tx := t.Begin()
		for i, src := range rows[name] {
			ad, perr := classad.Parse(src)
			if perr != nil {
				panic(perr)
			}
			tx.NewClassAd(fmt.Sprintf("%s%d", name, i), ad)
		}
		if cerr := tx.Commit(); cerr != nil {
			panic(cerr)
		}
	}
	return cat
}

// show renders one joined row by evaluating each expression against it.
//
// A joined row is itself a ClassAd with one member per side, named by that side's Alias, so
// a caller reads it with ordinary ClassAd expressions -- `j.Owner`, `o.AcctGroup` -- and not
// with an accessor of the join's own.
func show(row *classad.ClassAd, exprs ...string) string {
	out := ""
	for i, src := range exprs {
		if i > 0 {
			out += " "
		}
		ex, err := classad.ParseExpr(src)
		if err != nil {
			panic(err)
		}
		out += src + "=" + ex.Eval(row).String()
	}
	return out
}

// drain collects every joined row, rendered, and panics on the error a limit reports.
//
// Sorted because row order is the scan's and the planner's, not the caller's: a join may walk
// its sides in parallel, and which worker finishes first is a race. Any caller that needs an
// order sorts, or asks for one with an ORDER BY of its own.
func drain(cat *db.Catalog, spec db.JoinSpec, exprs ...string) []string {
	var stats db.JoinStats
	seq, err := cat.Join(spec, db.JoinLimits{}, &stats)
	if err != nil {
		panic(err)
	}
	var out []string
	for row := range seq {
		out = append(out, show(row, exprs...))
	}

	if stats.Err != nil {
		panic(stats.Err)
	}
	sort.Strings(out)
	return out
}

func main() {
	cat := joinFixture()
	defer cat.Close()

	spec := db.JoinSpec{
		Kind:  db.JoinInner,
		Left:  db.JoinSide{Table: "jobs", Alias: "j"},
		Right: db.JoinSide{Table: "owners", Alias: "o"},
		Cond:  []string{"j.Owner == o.Owner"},
		// Left reads the row built so far, Right reads this step's own table -- so Right
		// is a bare attribute, not "o.Owner". See JoinEqui.
		Equi: []db.JoinEqui{{Left: "j.Owner", Right: "Owner"}},
	}
	for _, line := range drain(cat, spec, "j.ClusterId", "o.AcctGroup") {
		fmt.Println(line)
	}

}
Output:
j.ClusterId=1 o.AcctGroup="physics"
j.ClusterId=2 o.AcctGroup="chem"
j.ClusterId=3 o.AcctGroup="physics"
Example (Band)

ExampleCatalog_Join_band joins on a range instead of an equality.

"Which events fall inside each job's run window" has no equality in it at all. Written as Cond alone it is a nested loop over every pair, bounded only by the pair budget. Band says the same thing as a window over one attribute of the folded side: the side is sorted on Ts once, and each run binary-searches the events its window covers.

Cond still carries the predicate. The band narrows the candidates; it does not replace the condition, and dropping Cond here would mean something subtly different -- a band is over numbers, and declines what ClassAd's own comparison would answer differently.

package main

import (
	"fmt"
	"sort"

	"github.com/PelicanPlatform/classad/classad"
	"github.com/PelicanPlatform/classad/db"
)

// joinFixture builds an in-memory catalog holding the tables these examples join.
//
// A catalog with no Dir keeps every table in RAM, which is what makes a join runnable in an
// example: nothing is written, nothing has to be cleaned up, and the plan the store picks is
// the same one it would pick over a directory.
func joinFixture() *db.Catalog {
	cat, err := db.OpenCatalog("")
	if err != nil {
		panic(err)
	}
	rows := map[string][]string{

		"jobs": {
			`[ ClusterId = 1; Owner = "alice"; RequestCpus = 2 ]`,
			`[ ClusterId = 2; Owner = "bob";   RequestCpus = 8 ]`,
			`[ ClusterId = 3; Owner = "alice"; RequestCpus = 4 ]`,
			`[ ClusterId = 4; Owner = "carol"; RequestCpus = 1 ]`,
		},
		"owners": {
			`[ Owner = "alice"; AcctGroup = "physics" ]`,
			`[ Owner = "bob";   AcctGroup = "chem" ]`,
		},
		"groups": {
			`[ AcctGroup = "physics"; Quota = 400 ]`,
			`[ AcctGroup = "chem";    Quota = 100 ]`,
		},

		"runs": {
			`[ ClusterId = 1; Start = 100; End = 140 ]`,
			`[ ClusterId = 2; Start = 200; End = 220 ]`,
		},
		"events": {
			`[ Ts = 210; Kind = "evict" ]`,
			`[ Ts = 105; Kind = "start" ]`,
			`[ Ts = 500; Kind = "idle" ]`,
			`[ Ts = 139; Kind = "done" ]`,
			`[ Ts = 205; Kind = "start" ]`,
		},
	}
	for _, name := range []string{"jobs", "owners", "groups", "runs", "events"} {
		t, terr := cat.CreateTable(name)
		if terr != nil {
			panic(terr)
		}
		tx := t.Begin()
		for i, src := range rows[name] {
			ad, perr := classad.Parse(src)
			if perr != nil {
				panic(perr)
			}
			tx.NewClassAd(fmt.Sprintf("%s%d", name, i), ad)
		}
		if cerr := tx.Commit(); cerr != nil {
			panic(cerr)
		}
	}
	return cat
}

// show renders one joined row by evaluating each expression against it.
//
// A joined row is itself a ClassAd with one member per side, named by that side's Alias, so
// a caller reads it with ordinary ClassAd expressions -- `j.Owner`, `o.AcctGroup` -- and not
// with an accessor of the join's own.
func show(row *classad.ClassAd, exprs ...string) string {
	out := ""
	for i, src := range exprs {
		if i > 0 {
			out += " "
		}
		ex, err := classad.ParseExpr(src)
		if err != nil {
			panic(err)
		}
		out += src + "=" + ex.Eval(row).String()
	}
	return out
}

func main() {
	cat := joinFixture()
	defer cat.Close()

	spec := db.JoinSpec{
		Kind:  db.JoinInner,
		Left:  db.JoinSide{Table: "runs", Alias: "r"},
		Right: db.JoinSide{Table: "events", Alias: "e"},
		Cond:  []string{"e.Ts >= r.Start", "e.Ts <= r.End"},
		// Attr names the folded side's attribute; Lo and Hi are expressions over the
		// sides already in hand. An empty end is unbounded, which is how a one-sided
		// inequality is written.
		Band: []db.JoinBand{{Attr: "Ts", Lo: "r.Start", Hi: "r.End"}},
	}

	var stats db.JoinStats
	seq, err := cat.Join(spec, db.JoinLimits{}, &stats)
	if err != nil {
		panic(err)
	}
	var out []string
	for row := range seq {
		out = append(out, show(row, "r.ClusterId", "e.Ts", "e.Kind"))
	}
	if stats.Err != nil {
		panic(stats.Err)
	}
	sort.Strings(out)
	for _, line := range out {
		fmt.Println(line)
	}

	// The pairs examined are the ones inside a window, not the product of the two sides.
	// Five events against two runs is ten pairs scanned; the band examines four.
	fmt.Printf("pairs examined: %d\n", stats.Pairs)

}
Output:
r.ClusterId=1 e.Ts=105 e.Kind="start"
r.ClusterId=1 e.Ts=139 e.Kind="done"
r.ClusterId=2 e.Ts=205 e.Kind="start"
r.ClusterId=2 e.Ts=210 e.Kind="evict"
pairs examined: 4
Example (Chain)

ExampleCatalog_Join_chain folds in a third table.

Steps is the chain spelling: Left is the FROM table and each step adds one more. A chain is LEFT-DEEP, so the row grows one member at a time and each step's condition reads the row as it is after that step.

The second step is where the scope asymmetry shows. Its Equi.Left is "o.AcctGroup" -- it has to be qualified, because by then the row carries two sides and a bare AcctGroup would not say which. Equi.Right stays bare: it reads the table being folded in, which is unambiguous by construction.

package main

import (
	"fmt"
	"sort"

	"github.com/PelicanPlatform/classad/classad"
	"github.com/PelicanPlatform/classad/db"
)

// joinFixture builds an in-memory catalog holding the tables these examples join.
//
// A catalog with no Dir keeps every table in RAM, which is what makes a join runnable in an
// example: nothing is written, nothing has to be cleaned up, and the plan the store picks is
// the same one it would pick over a directory.
func joinFixture() *db.Catalog {
	cat, err := db.OpenCatalog("")
	if err != nil {
		panic(err)
	}
	rows := map[string][]string{

		"jobs": {
			`[ ClusterId = 1; Owner = "alice"; RequestCpus = 2 ]`,
			`[ ClusterId = 2; Owner = "bob";   RequestCpus = 8 ]`,
			`[ ClusterId = 3; Owner = "alice"; RequestCpus = 4 ]`,
			`[ ClusterId = 4; Owner = "carol"; RequestCpus = 1 ]`,
		},
		"owners": {
			`[ Owner = "alice"; AcctGroup = "physics" ]`,
			`[ Owner = "bob";   AcctGroup = "chem" ]`,
		},
		"groups": {
			`[ AcctGroup = "physics"; Quota = 400 ]`,
			`[ AcctGroup = "chem";    Quota = 100 ]`,
		},

		"runs": {
			`[ ClusterId = 1; Start = 100; End = 140 ]`,
			`[ ClusterId = 2; Start = 200; End = 220 ]`,
		},
		"events": {
			`[ Ts = 210; Kind = "evict" ]`,
			`[ Ts = 105; Kind = "start" ]`,
			`[ Ts = 500; Kind = "idle" ]`,
			`[ Ts = 139; Kind = "done" ]`,
			`[ Ts = 205; Kind = "start" ]`,
		},
	}
	for _, name := range []string{"jobs", "owners", "groups", "runs", "events"} {
		t, terr := cat.CreateTable(name)
		if terr != nil {
			panic(terr)
		}
		tx := t.Begin()
		for i, src := range rows[name] {
			ad, perr := classad.Parse(src)
			if perr != nil {
				panic(perr)
			}
			tx.NewClassAd(fmt.Sprintf("%s%d", name, i), ad)
		}
		if cerr := tx.Commit(); cerr != nil {
			panic(cerr)
		}
	}
	return cat
}

// show renders one joined row by evaluating each expression against it.
//
// A joined row is itself a ClassAd with one member per side, named by that side's Alias, so
// a caller reads it with ordinary ClassAd expressions -- `j.Owner`, `o.AcctGroup` -- and not
// with an accessor of the join's own.
func show(row *classad.ClassAd, exprs ...string) string {
	out := ""
	for i, src := range exprs {
		if i > 0 {
			out += " "
		}
		ex, err := classad.ParseExpr(src)
		if err != nil {
			panic(err)
		}
		out += src + "=" + ex.Eval(row).String()
	}
	return out
}

// drain collects every joined row, rendered, and panics on the error a limit reports.
//
// Sorted because row order is the scan's and the planner's, not the caller's: a join may walk
// its sides in parallel, and which worker finishes first is a race. Any caller that needs an
// order sorts, or asks for one with an ORDER BY of its own.
func drain(cat *db.Catalog, spec db.JoinSpec, exprs ...string) []string {
	var stats db.JoinStats
	seq, err := cat.Join(spec, db.JoinLimits{}, &stats)
	if err != nil {
		panic(err)
	}
	var out []string
	for row := range seq {
		out = append(out, show(row, exprs...))
	}

	if stats.Err != nil {
		panic(stats.Err)
	}
	sort.Strings(out)
	return out
}

func main() {
	cat := joinFixture()
	defer cat.Close()

	spec := db.JoinSpec{
		Left: db.JoinSide{Table: "jobs", Alias: "j"},
		Steps: []db.JoinStep{
			{
				Kind: db.JoinInner,
				Side: db.JoinSide{Table: "owners", Alias: "o"},
				Cond: []string{"j.Owner == o.Owner"},
				Equi: []db.JoinEqui{{Left: "j.Owner", Right: "Owner"}},
			},
			{
				Kind: db.JoinInner,
				Side: db.JoinSide{Table: "groups", Alias: "g"},
				Cond: []string{"o.AcctGroup == g.AcctGroup"},
				Equi: []db.JoinEqui{{Left: "o.AcctGroup", Right: "AcctGroup"}},
			},
		},
		// Post are the WHERE conjuncts: applied to the joined row, after the pair is formed.
		Post: []string{"g.Quota > 200"},
	}
	for _, line := range drain(cat, spec, "j.ClusterId", "g.AcctGroup", "g.Quota") {
		fmt.Println(line)
	}

}
Output:
j.ClusterId=1 g.AcctGroup="physics" g.Quota=400
j.ClusterId=3 g.AcctGroup="physics" g.Quota=400
Example (LeftOuter)

ExampleCatalog_Join_leftOuter keeps the left row when the right side matches nothing.

Cluster 4's owner has no row in owners. Under JoinInner it is absent from the answer; under JoinLeft it is present once, with every attribute of the right side undefined.

package main

import (
	"fmt"
	"sort"

	"github.com/PelicanPlatform/classad/classad"
	"github.com/PelicanPlatform/classad/db"
)

// joinFixture builds an in-memory catalog holding the tables these examples join.
//
// A catalog with no Dir keeps every table in RAM, which is what makes a join runnable in an
// example: nothing is written, nothing has to be cleaned up, and the plan the store picks is
// the same one it would pick over a directory.
func joinFixture() *db.Catalog {
	cat, err := db.OpenCatalog("")
	if err != nil {
		panic(err)
	}
	rows := map[string][]string{

		"jobs": {
			`[ ClusterId = 1; Owner = "alice"; RequestCpus = 2 ]`,
			`[ ClusterId = 2; Owner = "bob";   RequestCpus = 8 ]`,
			`[ ClusterId = 3; Owner = "alice"; RequestCpus = 4 ]`,
			`[ ClusterId = 4; Owner = "carol"; RequestCpus = 1 ]`,
		},
		"owners": {
			`[ Owner = "alice"; AcctGroup = "physics" ]`,
			`[ Owner = "bob";   AcctGroup = "chem" ]`,
		},
		"groups": {
			`[ AcctGroup = "physics"; Quota = 400 ]`,
			`[ AcctGroup = "chem";    Quota = 100 ]`,
		},

		"runs": {
			`[ ClusterId = 1; Start = 100; End = 140 ]`,
			`[ ClusterId = 2; Start = 200; End = 220 ]`,
		},
		"events": {
			`[ Ts = 210; Kind = "evict" ]`,
			`[ Ts = 105; Kind = "start" ]`,
			`[ Ts = 500; Kind = "idle" ]`,
			`[ Ts = 139; Kind = "done" ]`,
			`[ Ts = 205; Kind = "start" ]`,
		},
	}
	for _, name := range []string{"jobs", "owners", "groups", "runs", "events"} {
		t, terr := cat.CreateTable(name)
		if terr != nil {
			panic(terr)
		}
		tx := t.Begin()
		for i, src := range rows[name] {
			ad, perr := classad.Parse(src)
			if perr != nil {
				panic(perr)
			}
			tx.NewClassAd(fmt.Sprintf("%s%d", name, i), ad)
		}
		if cerr := tx.Commit(); cerr != nil {
			panic(cerr)
		}
	}
	return cat
}

// show renders one joined row by evaluating each expression against it.
//
// A joined row is itself a ClassAd with one member per side, named by that side's Alias, so
// a caller reads it with ordinary ClassAd expressions -- `j.Owner`, `o.AcctGroup` -- and not
// with an accessor of the join's own.
func show(row *classad.ClassAd, exprs ...string) string {
	out := ""
	for i, src := range exprs {
		if i > 0 {
			out += " "
		}
		ex, err := classad.ParseExpr(src)
		if err != nil {
			panic(err)
		}
		out += src + "=" + ex.Eval(row).String()
	}
	return out
}

// drain collects every joined row, rendered, and panics on the error a limit reports.
//
// Sorted because row order is the scan's and the planner's, not the caller's: a join may walk
// its sides in parallel, and which worker finishes first is a race. Any caller that needs an
// order sorts, or asks for one with an ORDER BY of its own.
func drain(cat *db.Catalog, spec db.JoinSpec, exprs ...string) []string {
	var stats db.JoinStats
	seq, err := cat.Join(spec, db.JoinLimits{}, &stats)
	if err != nil {
		panic(err)
	}
	var out []string
	for row := range seq {
		out = append(out, show(row, exprs...))
	}

	if stats.Err != nil {
		panic(stats.Err)
	}
	sort.Strings(out)
	return out
}

func main() {
	cat := joinFixture()
	defer cat.Close()

	spec := db.JoinSpec{
		Kind:  db.JoinLeft,
		Left:  db.JoinSide{Table: "jobs", Alias: "j"},
		Right: db.JoinSide{Table: "owners", Alias: "o"},
		Cond:  []string{"j.Owner == o.Owner"},
		Equi:  []db.JoinEqui{{Left: "j.Owner", Right: "Owner"}},
	}
	for _, line := range drain(cat, spec, "j.ClusterId", "o.AcctGroup") {
		fmt.Println(line)
	}

}
Output:
j.ClusterId=1 o.AcctGroup="physics"
j.ClusterId=2 o.AcctGroup="chem"
j.ClusterId=3 o.AcctGroup="physics"
j.ClusterId=4 o.AcctGroup=undefined

func (*Catalog) JoinAggregate added in v0.36.0

func (cat *Catalog) JoinAggregate(spec JoinAggSpec, want JoinLimits, stats *JoinStats, stop func() bool) ([]AggRow, error)

JoinAggregate runs a join and reduces its rows to one row per group, returning only the groups.

Setup problems -- an unknown table, a path that does not parse -- are returned before any row is read. A join that met a limit part way through is returned as an ERROR with no rows, which is the one place this deliberately differs from the row-streaming join: a truncated stream still handed the caller real rows and only had to say it stopped, whereas a truncated reduction is not a partial answer but a wrong one -- a COUNT that stopped early looks exactly like a smaller COUNT.

stats (may be nil) is filled with the join's work breakdown, including Groups, so a caller can see the reduction ratio it bought.

stop (may be nil) is polled per joined row by the reduce engine; returning true abandons the scan. It is checked again afterwards, so a cancelled aggregate is reported as cancelled rather than returned as the groups that happened to accumulate first.

THE TWO CAPS ON THE ANSWER FAIL DIFFERENTLY, which is not an inconsistency. Too many groups is REFUSED, because the groups this would return instead are an arbitrary subset -- the reduce engine's first-seen order is scan order, so a truncated answer is "a hundred thousand of your jobs, we will not say which" and reads exactly like a complete one. A list past its cap is CUT, because there the loss is nameable: the group is still the right group, its COUNT is still the count of every row, and AggRow.Elided says per cell how many values the list does not show. Degrading is worth more than refusing wherever the caller can be told precisely what it did not get, and is worth less than nothing where it cannot.

func (*Catalog) JoinWire added in v0.37.0

func (cat *Catalog) JoinWire(spec JoinSpec, want JoinLimits, redact bool, stats *JoinStats) (iter.Seq[[]byte], error)

JoinWire is Join with its rows in wire form: each row a self-contained inline-names ad, ready to be written to a frame or decoded by whoever wants an object.

redact withholds the private attributes of each side, which is what the text path's render does and what a row leaving the store for an unprivileged reader needs. The join itself still runs over the FULL rows -- redacting before the condition would make which rows match depend on who is asking.

The bytes handed to yield alias a buffer this reuses. A caller that keeps a row past the next iteration must copy it -- the same contract the store's own raw scans have.

Join is unchanged and remains the entry point for a caller that wants ads. This exists beside it rather than replacing it because a joined row's consumer decides what it wants: one that sorts or filters wants an object, and one that relays wants bytes.

func (*Catalog) JoinsBands added in v0.39.0

func (cat *Catalog) JoinsBands() bool

JoinsBands reports that this catalog's join understands a JoinBand.

A capability marker for the same reason JoinsChains is one, and a sharper case of it. A band is a CONJUNCT of the match, not a hint: a catalog that ignored one would pair rows the window excludes and answer with more rows than were asked for. A build whose join predates bands satisfies the same join interface and would do exactly that, so a relay deciding whether to advertise the band opcodes has to ask rather than infer.

func (*Catalog) JoinsChains added in v0.37.0

func (cat *Catalog) JoinsChains() bool

JoinsChains reports that this catalog's join accepts a chain of more than two tables.

It is a capability marker, for a relay deciding whether to advertise the chain opcode. A catalog from a build whose join predates chains satisfies the same join interface and would accept a chain request and answer a different query, so "it has a Join method" is not the question a relay needs answered.

func (*Catalog) LoadExporterState added in v0.14.0

func (cat *Catalog) LoadExporterState(name string) ([]byte, bool, error)

LoadExporterState returns an exporter's last checkpointed resume-state blob. The bool is false when the exporter has never checkpointed (a fresh exporter with no state yet), which callers treat as "start from the beginning", not as an error.

func (*Catalog) Restore added in v0.7.0

func (cat *Catalog) Restore(r io.Reader) error

Restore replaces each table named in the catalog snapshot with its snapshotted contents (creating the table if absent), each under that table's DB-wide lock. Tables present in the catalog but ABSENT from the snapshot are left untouched. Encrypted sections are opened with the catalog's pool keys (the tables share them).

func (*Catalog) SaveExporterState added in v0.14.0

func (cat *Catalog) SaveExporterState(name string, state []byte) error

SaveExporterState durably records an exporter's opaque resume-state blob, replacing any prior state. The exporter calls this after its data has been accepted downstream, so that on restart it resumes just past the last delivered change (at-least-once). The exporter must already exist.

func (*Catalog) Snapshot added in v0.7.0

func (cat *Catalog) Snapshot(w io.Writer) error

Snapshot writes a backup of every table in the catalog to w. Each table is captured under its own DB-wide lock (consistent per table); the set of tables is the snapshot's membership at the moment each name is read.

func (*Catalog) Table

func (cat *Catalog) Table(name string) (*DB, bool)

Table returns the table named name.

func (*Catalog) Tables

func (cat *Catalog) Tables() []string

Tables returns the mutable table names, sorted.

func (*Catalog) View added in v0.12.0

func (cat *Catalog) View(name string) (*View, bool)

View returns the named view.

func (*Catalog) ViewBacking added in v0.12.0

func (cat *Catalog) ViewBacking(name string) (*DB, bool)

ViewBacking returns the in-memory backing table of a view, so reads (SELECT/query) resolve a view name to its materialized rows.

func (*Catalog) ViewSealed added in v0.16.0

func (cat *Catalog) ViewSealed(name, constraint string) (seq iter.Seq[*classad.ClassAd], ok bool, err error)

ViewSealed streams a continuous aggregate's sealed (archived) rows matching the constraint, so a view read can union sealed history with the live backing. ok is false for a plain table or a gauge view (nothing sealed). The returned sequence is empty (not an error) for a continuous aggregate with no archive (in-memory catalog).

func (*Catalog) ViewSealedTable added in v0.34.3

func (cat *Catalog) ViewSealedTable(name string) (*ArchiveTable, bool)

ViewSealedTable returns the archive holding a continuous aggregate's sealed history, and whether name is a view that has one. It is ViewSealed's counterpart for a reader that must run an operation over the sealed half -- a projection, a top-K, an aggregate -- rather than consume its rows.

func (*Catalog) Views added in v0.12.0

func (cat *Catalog) Views() []string

Views returns the view names, sorted.

type CatalogConfig added in v0.7.0

type CatalogConfig struct {
	// OnOpenStep, when set, is called once per table and archive opened by OpenCatalogConfig, with how long
	// that one took. Catalog open is a loop over directories, so without it the whole thing is a single
	// number -- which is how a 15s startup stayed unattributed: every other phase reported 0s and this one
	// reported all of it, with nothing inside.
	//
	// kind is "table" or "archive". MAY BE CALLED CONCURRENTLY, from whichever goroutine opened that table --
	// opens run in parallel -- so a handler must be safe for concurrent use and must not block. It is still
	// called once per open, which is what keeps per-table attribution intact under parallelism: only the SUM
	// stops matching the wall clock.
	OnOpenStep func(kind, name string, d time.Duration)
	Dir        string
	// DeltaMax sets the delta-record chain bound (see Config.DeltaMax) for every table this
	// catalog opens, and DeltaMaxFor overrides it per table name. Zero takes DefaultDeltaMax --
	// delta records are ON by default -- and DeltaMaxOff turns them off.
	//
	// Both belong on the CATALOG rather than on CreateTableOpts: OpenCatalogConfig opens every
	// existing table directory at startup, so by the time a caller reaches CreateTable the table
	// is already open and per-call options are not re-applied. A per-table option would
	// therefore take effect the first time a table was created and be silently ignored on every
	// restart after -- a knob that works once.
	//
	// Note that a store which has written even one delta record replays them for the rest of its
	// life (the on-disk marker says so), so this is not reversible for data already written;
	// DeltaMaxOff stops NEW deltas and leaves the old ones readable.
	DeltaMax    int
	DeltaMaxFor map[string]int

	// DisableReadVerification turns off the per-record checksum check on reads for
	// EVERY table in this catalog. See Config.DisableReadVerification; this is the
	// catalog-wide form, because an operator reaching for it is reacting to a
	// whole-database read cost, not to one table.
	DisableReadVerification bool
	// PoolKeys enables encryption at rest for every table (each table's master key is
	// wrapped under these keys). EncryptedAttrs is the default explicit encrypted-attr
	// set for each table (private attributes are always encrypted); its values must not
	// refer to other attributes (see Config.EncryptedAttrs). See db/encrypt.go.
	PoolKeys       []KEK
	EncryptedAttrs []string

	// OnSealMigration, when set, is called for each table whose open rewrote segments to seal private
	// attributes that predate sealing, with how many segments it rewrote. It carries the TABLE NAME
	// because a catalog opens many, and "some table rewrote 400 segments" does not tell an operator
	// which one to expect the cost from.
	//
	// The per-table Config hook exists too, but a catalog builds those configs itself, so without this
	// the hook was unreachable for every table a daemon actually opens -- the only place the one-off
	// startup cost needs explaining.
	//
	// Called from the goroutine that opened that table, so it may run concurrently (see OnOpenStep).
	OnSealMigration func(table string, segments int)

	// SealMigrationWorkers bounds the concurrent segment rewrites within one table's migration (see
	// collections.MigrateSealedAttrs). 0 takes a per-table default. Tables are already opened in
	// parallel, so this multiplies with that.
	SealMigrationWorkers int
	// QueryParallelism applies Config.QueryParallelism to every table the catalog opens.
	QueryParallelism int

	// WatchRetain sets Config.WatchRetain -- how far back in time a table keeps its delete
	// markers, so a watcher that reconnects after a restart can be told which keys went away
	// instead of being sent back for a full replay -- for every table this catalog opens.
	// WatchRetainFor overrides it per table name, as DeltaMaxFor does. 0 (the default) keeps
	// nothing extra; the window is paid for in segments that cannot be reclaimed yet.
	//
	// Both belong on the CATALOG for the reason DeltaMax does: catalog open reopens every
	// existing table directory at startup, so a per-CreateTable option would take effect the
	// first time a table was created and be silently ignored on every restart after -- which
	// is exactly when a resuming watcher needs the window.
	//
	// This reaches MUTABLE tables only. An archive is append-only, has no deletes to retain,
	// and already carries its watch epoch across a clean shutdown unconditionally (see
	// collections/watchepoch.go); a name in WatchRetainFor that belongs to an archive has no
	// effect.
	WatchRetain    time.Duration
	WatchRetainFor map[string]time.Duration
}

CatalogConfig configures a catalog, including encryption at rest applied to every table. Dir empty is in-memory.

type CodecStats

type CodecStats = collections.CodecStats

Diagnostic and management types, re-exported from collections for callers that only import db.

type ColumnAccumulator added in v0.37.0

type ColumnAccumulator interface {
	Accumulator

	// AddNum folds one numeric column value.
	AddNum(v NumValue)
}

ColumnAccumulator is an Accumulator that can be fed straight from a numeric column instead of a boxed value per row.

IMPLEMENTING IT IS A DECLARATION OF TWO THINGS, because the columnar pass differs from the record scan in two ways that an aggregate can be sensitive to, and no signature can check either:

  1. ORDER. The pass walks segments, then blocks, skipping blocks a zone map rules out. That is not the order a record scan visits the same records in, and not insertion order. An aggregate whose answer depends on the order it is fed will give a DIFFERENT answer depending on which rung served the query, which is the worst kind of difference: both answers look right. The reference's own min()/max() cannot make this promise over a column holding a NaN, which is why the built-in path declines that case rather than answering it.

  2. WHAT IS OFFERED. The pass yields only the NUMERIC values of the column. A record whose argument is absent, undefined, a string, or an expression is not offered at all -- where the record scan calls Add for every row in the group, undefined included. So an aggregate that counts what it was given, or that treats undefined as a contribution, must not implement this interface: it would count different things on the two rungs. An aggregate that already skips non-numbers -- which is what the built-in SUM, AVG, MIN and MAX all do -- loses nothing.

Result must work from AddNum alone, since a query served by the columnar pass never calls Add.

type Config

type Config struct {
	Dir string
	// Ordered configures maintained, filtered, sorted indexes -- e.g. the negotiator's
	// resource-request lists (partition by Owner, sort by JobPrio then QDate), iterated
	// in order via Ordered. Optional.
	Ordered []OrderSpec
	// HotAttrs / CategoricalAttrs / ValueAttrs / MatchClosureRoots tune storage and
	// query/match push-down (see collections.Options). Optional.
	HotAttrs                     []string
	CategoricalAttrs, ValueAttrs []string
	MatchClosureRoots            []string
	// CompositeAttrs names attribute TUPLES to index together, so a conjunction pinning
	// every component of one is answered by a single lookup rather than by intersecting
	// each component's postings. Each component must also be indexed in its own right.
	// (`omitempty` for the same reason the archive's is: an unconfigured field must not
	// change the bytes an existing deployment's config file is rewritten with.)
	CompositeAttrs [][]string `json:",omitempty"`
	// CategoricalComponentAttrs / ValueComponentAttrs name attributes that exist ONLY as
	// composite components: they carry no index of their own, so no constraint on one
	// alone is index-answered, and which list a name appears in fixes how the composite
	// encodes it. They are what lets a composite over (ClusterId, ProcId) be configured
	// without also paying for a standalone ProcId index nothing queries.
	//
	// Persisted with the rest of the index set, and `omitempty` for the same reason the
	// other index fields are: a configuration naming none must write the bytes it always
	// wrote.
	CategoricalComponentAttrs []string `json:",omitempty"`
	ValueComponentAttrs       []string `json:",omitempty"`

	// DisableReadVerification turns off the per-record checksum check on reads for
	// this table. Verification is ON by default (see
	// collections.Options.DisableReadVerification): without it a bit flip inside a
	// stored ad is served as a plausible value rather than reported.
	//
	// This exists so an operator has a way out without downgrading. Turning it off
	// trades a few percent of read CPU for the chance of serving a wrong answer, and
	// should follow a measurement rather than a hunch.
	DisableReadVerification bool

	// DeltaMax bounds the delta-record chain length for this table; see DefaultDeltaMax and
	// DeltaMaxOff for what 0 and a negative value mean.
	//
	// It turns on delta records for this table and bounds the chain length: a
	// SetAttribute stores only the attributes it changed, until a key has accumulated
	// DeltaMax of them and the next write stores the whole ad again. See
	// collections.Options.DeltaMax for the trade.
	DeltaMax int

	// SchemaRequireAttrs names attributes that enter the derived columnar schema whatever their
	// presence rate, overriding the rule that an attribute becomes a fixed-slot field only when
	// it appears in at least 90% of sampled ads. See collections.Options.SchemaRequireAttrs for
	// what the override does and does not relax.
	//
	// Read at open and treated as the CONFIGURED tier: anything added later with
	// AddSchemaRequireAttrs is kept alongside it and persisted, so a runtime addition survives a
	// restart while a change to this field still takes effect on the next open.
	SchemaRequireAttrs []string

	// GroupSchemaCount is how many SECONDARY columnar schemas to derive: sets of attributes the
	// base schema does not carry which are present or absent together, stored columnar for the
	// ads that hold them without costing a slot in the ads that do not. 0 takes the default (4);
	// a NEGATIVE value builds none.
	//
	// A group's blocks are built only once its members have kept showing up together across
	// GroupStabilityRuns maintenance passes, so a freshly opened table builds none for a while --
	// SchemaScanInfo.GroupSchemas reports how many actually exist. See collections.Options.
	GroupSchemaCount    int
	GroupStabilityRuns  int
	GroupMergeJaccard   float64
	GroupMaxPartialFrac float64

	// ColumnarSegmentBudget is how many sealed segments one maintenance pass may rewrite into
	// COLUMNAR-NATIVE form -- every value the schema carries stored once, in the segment's own
	// columnar payload, and removed from the records. 0 takes the default (64 per pass, so an
	// existing table converges over several passes rather than stalling on one); a NEGATIVE value
	// disables the rewrite, leaving every sealed segment whole-record with a columnar copy beside
	// it in the sidecar. See collections.Options.ColumnarSegmentBudget for the measurement behind
	// the default and for what the two forms cost.
	//
	// The two forms answer the same questions and differ in what the columns can read: a sidecar
	// column holds literals only, while a columnarized one also holds the value an expression-valued
	// cell evaluates to. A caller that needs one of them specifically has to say so.
	ColumnarSegmentBudget int

	// SealMigrationWorkers bounds the concurrent segment rewrites when an existing store is migrated
	// to sealed private attributes at open (see collections.MigrateSealedAttrs). 0 takes a default
	// derived from GOMAXPROCS.
	SealMigrationWorkers int

	// OnSealMigration, if set, is called with the number of segments rewritten when the open migrated a
	// store to sealed private attributes. It exists so a daemon can log a one-off startup cost rather
	// than leaving an operator to wonder why the first open after an upgrade took longer.
	OnSealMigration func(segments int)

	// SegmentSize overrides the arena segment size in bytes (see collections.Options).
	// 0 uses the default (8 MiB). A smaller value seals segments sooner -- useful for
	// tests and for tuning the sealed-segment accelerators (columnar scan, sealed indexes).
	SegmentSize int

	// MutatingBlockCacheBytes and ArchiveBlockCacheBytes set the PROCESS-GLOBAL shared
	// decompressed-columnar-block cache budgets (see collections.Options): one budget shared by all
	// mutating tables and a separate one shared by all archive tables. They are global ceilings, not
	// per-DB or per-table, so a process opening many tables (as htcondordb does) no longer multiplies
	// a fixed per-table cache by the table count. 0 keeps the collections default (512 MiB each). The
	// same Config feeds both the mutating and archive opens, so setting them once configures the
	// whole process; the last non-zero value wins and resizes the live cache.
	MutatingBlockCacheBytes int64
	ArchiveBlockCacheBytes  int64

	// PoolKeys enables encryption at rest: the DB master key is wrapped under each of
	// these pool/signing keys (any one opens the DB; a rotated-in key is added on the
	// next open). Empty ⇒ encryption disabled. See db/encrypt.go.
	PoolKeys []KEK
	// EncryptedAttrs names the attributes whose values are sealed at rest (case-
	// insensitive). Only meaningful with PoolKeys set. An encrypted attribute may not
	// also be indexed, and its value must not refer to other attributes: reads do not
	// follow a reference inside a sealed value. See collections.Options.EncryptedAttrs.
	EncryptedAttrs []string

	// QueryParallelism is collections.Options.QueryParallelism: 0 auto, 1 serial, N>=2 a
	// per-query worker cap. Reachable here because the fan-out is not always a win --
	// when per-record work is small the scheduling can cost more than the scan -- and
	// without this the db layer gave no way to find that out, let alone configure it.
	QueryParallelism int
	// AggMinBytes is collections.Options.AggMinBytes: the live segment bytes an aggregate pass
	// must cover before it fans out across segments. 0 takes the measured default.
	AggMinBytes int
	// WatchRetain is collections.Options.WatchRetain: how far back in time a persistent
	// store keeps its delete markers, so a watcher that reconnects after a restart can
	// still be told which keys went away rather than being sent back for a full replay.
	//
	// 0 (the default) retains nothing extra, which is what a store with no reconnecting
	// watchers wants: the window is paid for in segments that cannot be reclaimed yet. Set
	// it where resuming matters more than the space, sized to how long a watcher may be
	// away -- a restart, a redeploy, a partition.
	WatchRetain time.Duration
}

type ConflictError

type ConflictError struct{ Keys []string }

ConflictError reports the keys whose writes lost an optimistic write-write race at commit. The other writes in the transaction committed; the caller re-reads and retries the conflicted keys.

func (*ConflictError) Error

func (e *ConflictError) Error() string

type Constraint

type Constraint struct {
	// contains filtered or unexported fields
}

Constraint is a compiled ClassAd boolean expression, for evaluating the same filter against many ads without re-parsing.

func ParseConstraint

func ParseConstraint(expr string) (*Constraint, error)

ParseConstraint compiles a ClassAd boolean expression.

func (*Constraint) Matches

func (c *Constraint) Matches(ad *classad.ClassAd) bool

Matches reports whether ad satisfies the constraint.

type CopyLimits added in v0.42.0

type CopyLimits struct {
	// MaxRows caps the rows copied in one statement.
	MaxRows int
	// MaxBytes caps the estimated heap of the rows held. It is the bound that actually
	// protects the process: a row's cost moves by more than twenty times with the width of
	// the ad, so a row count alone lets a wide table hold gigabytes while reporting itself
	// well inside its limit.
	MaxBytes int
}

CopyLimits bound what one row copy may hold. Like JoinLimits they are the store's own: a statement must not be able to ask a daemon to hold an unbounded amount.

They bound a HOLD, not a result size, and the hold is real: collections.Txn buffers every write by key until Commit, so a copy of N rows has N decoded ads in memory at the moment it commits. Committing in batches instead would trade that for a table that is visibly half-copied while it runs, which is the property the single transaction is there to avoid, so the bound is a refusal rather than a chunking strategy.

func DefaultCopyLimits added in v0.42.0

func DefaultCopyLimits() CopyLimits

DefaultCopyLimits are the limits a copy runs under when a caller names none. They are the join's hash-side budgets, because the thing being bounded is the same thing: decoded ads held in a map keyed by a string, for the duration of one operation.

type CopySpec added in v0.42.0

type CopySpec struct {
	Attrs []string
	Names []string
}

CopySpec names what a row copy reads and what it writes each value under.

Attrs empty means the WHOLE AD -- `SELECT *` -- which copies every attribute under its own name. Otherwise Attrs are the source attributes and Names the target attributes they are written under, aligned.

func (CopySpec) Validate added in v0.42.0

func (s CopySpec) Validate() error

Validate checks the spec on its own: one target name per source attribute, each a usable attribute name, none repeated and none private.

A private target name is refused for the reason InsertSpec.Validate refuses one, and here the concern is sharper: a projection RENAMES, so `SELECT <private> AS Public` is a laundering operation written in one line. Refusing a private name does not stop the reverse (a public source attribute written to a private name is harmless), and it does not by itself stop private data reaching a public column -- what stops that is the read entitlement, which hands an unprivileged caller undefined for a sealed value. This is the check that can be made from the names alone.

func (CopySpec) Whole added in v0.42.0

func (s CopySpec) Whole() bool

Whole reports whether this copies entire ads (SELECT *) rather than a projection.

type DB

type DB struct {
	// contains filtered or unexported fields
}

DB is an embedded ClassAd log. Safe for concurrent use.

func Open

func Open(dir string) (*DB, error)

Open opens a ClassAd log with default configuration. A non-empty dir makes it persistent (memory-mapped arenas under dir, recovered on reopen); an empty dir is in-memory. See OpenConfig for indexing / ordered-index configuration.

func OpenConfig

func OpenConfig(cfg Config) (*DB, error)

OpenConfig opens a ClassAd log with the given configuration. Every DB is stamped with a stable DB id (persisted alongside a persistent store, so it survives reopen) and a fresh instance id for this open. Until high-availability DB servers exist they are effectively the same identity, but a follower/replica shares the DB id while carrying its own instance id.

func (*DB) AcquireRead added in v0.33.0

func (db *DB) AcquireRead() bool

AcquireRead registers the caller as an in-flight reader of the table and reports whether it may read. False means the table has been closed or dropped. Every true answer MUST be balanced by ReleaseRead.

Holding it is what makes DropTable safe for a reader that is between segment pins -- see collections.Collection.AcquireRead. The scan methods below take it for the lifetime of the sequence they return, so an ordinary caller never needs it; it is exported for a caller that reads a table through several calls and needs the table to stay mapped across them.

func (*DB) AddHotAttrs

func (db *DB) AddHotAttrs(names ...string) []string

AddHotAttrs pins the named attributes into the hot set and returns the resulting hot attributes. The hot set is persisted.

func (*DB) AddIndex

func (db *DB) AddIndex(categorical, value []string) bool

AddIndex adds categorical and/or value indexes at runtime, returning whether the configuration changed. The spec is updated immediately; call Reindex to build the new index over existing ads. A change is persisted so it survives a restart.

func (*DB) AddSchemaRequireAttrs added in v0.32.0

func (db *DB) AddSchemaRequireAttrs(names ...string) (bool, error)

func (*DB) AggregateColsStats added in v0.32.1

func (db *DB) AggregateColsStats(constraint string, groupCols []GroupCol, aggs []AggSpec, stats *collections.ScanStats, stop func() bool) ([]AggRow, error)

AggregateColsStats runs a server-side GROUP BY over the table's matches: it applies the constraint, groups by the group-column tuple (a column may carry a bucket width, so a numeric attribute floors into fixed-width buckets -- the "per day" dimension), and reduces each group with COUNT/SUM/AVG/MIN/MAX. With no group columns it returns a single row over the whole match. It is the mutable-table counterpart of ArchiveTable.AggregateColsStats and shares its grouping and reduce engine (AggregateValues), so the same aggregate over a live table and over a history table answer identically rather than through two implementations that could drift.

stats (may be nil) is filled with the scan's work breakdown for EXPLAIN ANALYZE. Only the fall-through projected scan -- the slow path taken when the aggregate is not one the columnar accelerator covers -- fills it; the Len/count/columnar fast paths do not reassemble records and leave stats zero (RecordsVisited == 0), which the caller reads as "served without a per-record scan". A nonzero, large RecordsReassembled here is exactly the signature of the slow aggregate.

stop (may be nil) is polled per scanned row; returning true abandons the scan with whatever groups have accumulated, so a server can drop a scan whose client has gone away. It is a plain callback rather than a context because every other query entry point here takes none, and the one caller that cares already holds the connection's context.

func (*DB) AggregateColsStatsRedacted added in v0.35.0

func (db *DB) AggregateColsStatsRedacted(constraint string, groupCols []GroupCol, aggs []AggSpec, stats *collections.ScanStats, stop func() bool) ([]AggRow, error)

AggregateColsStatsRedacted is AggregateColsStats for a caller NOT entitled to sealed values: every rung answers as if the table had no key, so a sealed value is undefined to the WHERE clause, the group key, the aggregate and its FILTER alike. A count that a sealed value decided is as much an answer about it as the value itself.

func (*DB) AttrNDV added in v0.38.1

func (db *DB) AttrNDV(attr string) (NDVEstimate, bool)

AttrNDV reports how many distinct values attr takes in this table, from the per-segment index summaries alone. ok is false when nothing indexes attr, which a caller must treat as "unknown" rather than as a small number.

func (*DB) AttrTypeProfiles added in v0.34.2

func (db *DB) AttrTypeProfiles(sampleMax int) ([]AttrTypeProfile, int)

AttrTypeProfiles profiles what this table's attributes hold -- the classes of value each one carried across a sample, with exact bounds for the numeric ones. It is what a consumer that must publish one declared type per attribute reads to decide those types; see collections.AttrTypeProfiles for what it does and does not answer.

func (*DB) AttrTypeProfilesWith added in v0.36.0

func (db *DB) AttrTypeProfilesWith(o AttrTypeOptions) ([]AttrTypeProfile, int)

AttrTypeProfilesWith is AttrTypeProfiles with the measurement selected by o -- in particular with the EVALUATED classes alongside the stored ones, which is what a consumer that evaluates before it types has to read instead. See collections.AttrTypeProfilesWith.

func (*DB) BackupKey added in v0.7.0

func (db *DB) BackupKey() []byte

BackupKey returns a copy of the backup key -- the key that unwraps a snapshot's encryption, so an operator can escrow it and decrypt/restore backups without the pool keys. It is DISTINCT from the live-data key (it cannot decrypt the running store) and from the master. Returns nil when encryption is disabled.

func (*DB) Begin

func (db *DB) Begin() *Txn

Begin starts a new independent transaction.

func (*DB) BeginRedacted added in v0.28.0

func (db *DB) BeginRedacted() *Txn

BeginRedacted is Begin for a caller not entitled to sealed values: reads through the transaction decode with no key. SetAttribute and DeleteAttribute change only the attribute they name, sealed or not, and keep every other sealed value; where one has to be re-keyed into the rewritten record, the store does that itself and the transaction never sees it. NewClassAd and DestroyClassAd replace or remove the whole ad, as they do for any caller. See collections.Collection.BeginRedacted.

func (*DB) Chained added in v0.21.3

func (db *DB) Chained() bool

Chained reports whether the table has structural (parent-only) ads that Query hides, so a caller knows when Len equals the match-all row count (the COUNT(*) fast path).

func (*DB) Close

func (db *DB) Close() error

Close releases the log's resources.

func (*DB) CodecStats

func (db *DB) CodecStats(sampleMax int) CodecStats

CodecStats reports the storage codec's state and effectiveness (name, dictionary size, last-retrain time, and a sampled compression ratio).

func (*DB) Compact

func (db *DB) Compact() int

Compact reclaims dead space in shards whose dead-byte ratio warrants it, returning the number of shards compacted.

func (*DB) CompositeAttrs added in v0.38.1

func (db *DB) CompositeAttrs() [][]string

CompositeAttrs returns the attribute tuples indexed compositely, each in its configured order. A composite answers a conjunction pinning every one of its components to a single value, and nothing else.

func (*DB) CopyRowsInto added in v0.42.0

func (db *DB) CopyRowsInto(rows iter.Seq[*classad.ClassAd], spec CopySpec, limits CopyLimits,
	stop func() bool) (int, error)

CopyRowsInto copies rows into the table under keys derived from its declared primary key, as ONE transaction, and reports how many were written.

The rows STREAM: they are projected, keyed and buffered one at a time, and nothing holds the whole source. What the transaction itself holds is bounded by limits and a copy that would exceed them is refused with nothing written -- see CopyLimits for why the answer is a refusal rather than a commit every so often.

Like the roll-up this converges on a re-run instead of accumulating: each row's key is a function of the row, so copying the same source twice writes the same keys and the second run replaces the first. It is not a SYNC -- a row deleted from the source is not deleted from the target, because this statement only ever writes.

stop (may be nil) is polled per row so a caller that has gone away can end a long copy. A stopped copy writes NOTHING: the transaction is abandoned rather than committed with whatever it had reached, because a caller who stopped listening never learned how far it got, and a partial copy that nobody was told about is the one outcome here that cannot be reasoned about afterwards.

func (*DB) CountConstraint added in v0.23.0

func (db *DB) CountConstraint(constraint string) (int, bool)

CountConstraint counts the rows matching constraint via the columnar schema scan when the constraint is columnar-eligible (Native, numeric comparisons on one int schema field) and schema-scan is enabled; ok=false ⇒ the caller should use the normal count path. See collections.Collection.CountConstraint.

func (*DB) CountConstraintRedacted added in v0.35.0

func (db *DB) CountConstraintRedacted(constraint string) (int, bool)

CountConstraintRedacted is CountConstraint for a caller NOT entitled to sealed values. See collections.Collection.CountQueryRedacted.

func (*DB) Delete added in v0.8.0

func (db *DB) Delete(key string) (bool, error)

Delete removes the ad at key in its own optimistic transaction, retrying on a write-write conflict. It reports whether an ad was present to remove. Equivalent to Begin + DestroyClassAd + Commit with retry.

func (*DB) DeleteWhere added in v0.8.0

func (db *DB) DeleteWhere(constraint string) (int, error)

func (*DB) DeleteWhereRedacted added in v0.35.0

func (db *DB) DeleteWhereRedacted(constraint string) (int, error)

DeleteWhereRedacted is DeleteWhere for a caller NOT entitled to sealed values: the constraint is matched -- at the scan and again in the deleting transaction -- against ads decoded with no key, so a sealed value is undefined to it. How many rows a constraint deletes is an answer about every value it read, and a write session that may not see a value must not be able to delete by it either.

func (*DB) DeltaStats added in v0.30.0

func (db *DB) DeltaStats() (deltas, fulls int64)

DeltaStats reports how many of this table's writes were stored as delta records versus in full. Both zero means delta records are off (or nothing has been written). It is the only way to tell a working delta store from one that has quietly fallen back to storing whole ads -- which reads back perfectly correctly and saves nothing.

func (*DB) DropIndex

func (db *DB) DropIndex(names ...string) bool

DropIndex removes the named attributes from the configured indexes, returning whether the configuration changed. A change is persisted.

func (*DB) DropSchemaRequireAttrs added in v0.32.0

func (db *DB) DropSchemaRequireAttrs(names ...string) (bool, error)

func (*DB) EnableSchemaScan added in v0.23.0

func (db *DB) EnableSchemaScan(sampleMax, hotTopN int) bool

EnableSchemaScan builds the per-segment adschema columnar accelerator over the table's sealed segments and enables it, choosing the uncompressed hot numeric tier as the top-hotTopN int/real fields by query demand (see collections.Collection.BuildAndEnableSchemaScan). sampleMax bounds the ad sample. Opt-in; a table that never calls this is unaffected.

func (*DB) EncryptedAttrNames added in v0.7.0

func (db *DB) EncryptedAttrNames() []string

EncryptedAttrNames returns the explicit encrypted-attribute set (not the always-on private attributes). EncryptionEnabled reports whether encryption at rest is active.

func (*DB) EncryptionEnabled added in v0.7.0

func (db *DB) EncryptionEnabled() bool

EncryptionEnabled reports ENCRYPTION AT REST: the master key is wrapped under pool keys, so the data is protected from someone holding the disk. It is not "are values sealed" -- private attributes are always sealed, and without pool keys the master sits beside the data in the clear, which protects nothing and is reported here as false.

func (*DB) Explain

func (db *DB) Explain(constraint string) (QueryExplain, error)

Explain reports how the store would execute a constraint query -- which conjuncts are index-usable and the resulting access path.

func (*DB) ExplainMatch

func (db *DB) ExplainMatch(job *classad.ClassAd, targetConstraint string) MatchExplain

ExplainMatch reports how matchmaking job against this (resource) collection would execute: the job's Requirements rewritten over the slot with the job's attributes baked to constants, and which of the resulting probes prune via a configured index. targetConstraint, if non-empty, is the MATCH resource-side filter (WHERE TARGET / NOPREEMPT) melded into the explanation.

func (*DB) ForEach

func (db *DB) ForEach(fn func(ad *classad.ClassAd) bool)

ForEach calls fn for every committed ad, in no particular order, until fn returns false. It reads a consistent snapshot (concurrent writers do not block it).

func (*DB) ForEachSystemAd added in v0.9.0

func (db *DB) ForEachSystemAd(fn func(key string, ad *classad.ClassAd) bool)

ForEachSystemAd calls fn for every internal system-keyed ad and its key -- the reaper enumeration path. System records are durable bookkeeping (e.g. idempotency markers) stored in the same table as data but hidden from every client scan/query/DeleteWhere; this is the only way to enumerate them. Reads a consistent snapshot. See SystemKey.

func (*DB) GroupSchemaAgreement added in v0.28.0

func (db *DB) GroupSchemaAgreement(sampleMax, k int) GroupSchemaAgreement

GroupSchemaAgreement reports how well per-segment derivations agree with the table-wide one.

func (*DB) GroupSchemaChanges added in v0.29.13

func (db *DB) GroupSchemaChanges() []GroupSchemaChange

GroupSchemaChanges returns the committed-group change log: the moments the accelerator adopted a different set of secondary schemas, with each change's diff and reason. Reads persisted state only (no sampling), so it is available at READ.

func (*DB) GroupSchemaDrift added in v0.28.0

func (db *DB) GroupSchemaDrift() GroupSchemaDrift

GroupSchemaDrift reports how the derived groups have moved across retained derivations.

func (*DB) GroupSchemaLastAgreement added in v0.29.13

func (db *DB) GroupSchemaLastAgreement() (GroupSchemaLastAgreement, bool)

GroupSchemaLastAgreement returns the last persisted per-segment agreement result, if any.

func (*DB) GroupSchemas added in v0.28.0

func (db *DB) GroupSchemas(sampleMax, k int) GroupSchemaInfo

GroupSchemas derives and reports candidate group schemas: sets of attributes the base schema does not carry which are present or absent together, and could therefore be stored columnar for the ads that have them without costing a slot in the ads that do not. Report-only.

func (*DB) GroupStatsAll added in v0.28.0

func (db *DB) GroupStatsAll(groupAttr string, aggAttrs []string) ([]collections.GroupStats, bool)

GroupStatsAll is GroupStatsConstraint over every row, for a constraint the caller has established is match-all. Declines on a chained table for the reason above.

func (*DB) GroupStatsAllAggs added in v0.33.2

func (db *DB) GroupStatsAllAggs(groups []collections.GroupSpec, aggCols []collections.GroupAggCol,
	countFilters []string) ([]collections.GroupStats, bool)

GroupStatsAllAggs is GroupStatsConstraintAggs over every row. Declines on a chained table for the reason above. See collections.Collection.GroupStatsAllAggs.

func (*DB) GroupStatsAllBucketed added in v0.32.0

func (db *DB) GroupStatsAllBucketed(groupAttr string, bucket int64, aggAttrs []string) ([]collections.GroupStats, bool)

GroupStatsAllBucketed is GroupStatsAll with the group column floored into buckets of bucket (0 = the raw value). Declines on a chained table for the reason above. See collections.Collection.GroupStatsQueryBucketed.

func (*DB) GroupStatsAllCols added in v0.33.0

func (db *DB) GroupStatsAllCols(groups []collections.GroupSpec, aggAttrs []string) ([]collections.GroupStats, bool)

GroupStatsAllCols is GroupStatsAllBucketed over SEVERAL group columns. Declines on a chained table for the reason above. See collections.Collection.GroupStatsQueryCols.

func (*DB) GroupStatsConstraint added in v0.28.0

func (db *DB) GroupStatsConstraint(constraint, groupAttr string, aggAttrs []string) ([]collections.GroupStats, bool)

GroupStatsConstraint answers a per-group record count plus the aggregate inputs for each aggAttr over the rows matching constraint, via the columnar accelerator, or ok=false so the caller scans. This is the mutable-table half of what ArchiveTable already exposes; dbrpc's aggregate routes a grouped COUNT(*)/MIN/MAX/SUM/AVG through GroupedFromColumns for both.

A CHAINED table declines. The columnar paths read a record's own columns and know nothing about parentKeyFor/mergeParent, so on a chained table a group or aggregate attribute inherited from the parent would be missing from the column and the answer would differ from the scan's. (The same is true of the existing CountConstraint fast path, which does not guard it; chaining has no production consumer today -- only classad's own tests set IsStructural -- so that is a latent gap rather than a live bug.)

func (*DB) GroupStatsConstraintAggs added in v0.33.2

func (db *DB) GroupStatsConstraintAggs(constraint string, groups []collections.GroupSpec,
	aggCols []collections.GroupAggCol, countFilters []string) ([]collections.GroupStats, bool)

GroupStatsConstraintAggs is GroupStatsConstraintCols where an aggregate column may carry a per-aggregate FILTER. Declines on a chained table for the reason above. See collections.Collection.GroupStatsConstraintAggs.

func (*DB) GroupStatsConstraintBucketed added in v0.32.0

func (db *DB) GroupStatsConstraintBucketed(constraint, groupAttr string, bucket int64, aggAttrs []string) ([]collections.GroupStats, bool)

GroupStatsConstraintBucketed is GroupStatsConstraint with the group column floored into buckets of bucket (0 = the raw value). Declines on a chained table for the reason above. See collections.Collection.GroupStatsQueryBucketed.

func (*DB) GroupStatsConstraintCols added in v0.33.0

func (db *DB) GroupStatsConstraintCols(constraint string, groups []collections.GroupSpec, aggAttrs []string) ([]collections.GroupStats, bool)

GroupStatsConstraintCols is GroupStatsConstraintBucketed over SEVERAL group columns: the group is the tuple of their (individually bucketed) values. Declines on a chained table for the reason above. See collections.Collection.GroupStatsQueryCols.

func (*DB) GroupValuesAllAggs added in v0.37.0

func (db *DB) GroupValuesAllAggs(groups []collections.GroupSpec, aggCols []collections.GroupAggCol,
	countFilters []string, newSink func() collections.GroupSink, mergeSink collections.MergeGroupSink) ([]collections.GroupValues, bool)

GroupValuesAllAggs is GroupValuesConstraintAggs over every row.

func (*DB) GroupValuesConstraintAggs added in v0.37.0

func (db *DB) GroupValuesConstraintAggs(constraint string, groups []collections.GroupSpec,
	aggCols []collections.GroupAggCol, countFilters []string,
	newSink func() collections.GroupSink, mergeSink collections.MergeGroupSink) ([]collections.GroupValues, bool)

GroupValuesConstraintAggs is GroupStatsConstraintAggs reduced into caller-supplied accumulators. Declines on a chained table for the reason above.

func (*DB) HotAttrs

func (db *DB) HotAttrs() []string

HotAttrs returns the current hot attributes (front-loaded in each ad's hot header for cheap access).

func (*DB) ID

func (db *DB) ID() string

ID is the stable database identity (same across reopens of a persistent store).

func (*DB) InMemory added in v0.10.2

func (db *DB) InMemory() bool

InMemory reports whether the table's data lives only in RAM (no on-disk backing), i.e. it was opened with an empty Dir. Such a table is not recovered across restarts.

func (*DB) IndexSizes

func (db *DB) IndexSizes() IndexSizes

IndexSizes reports each configured index's resident bytes (with human/auto provenance) against the live data bytes -- the memory cost of indexing.

func (*DB) IndexedAttrs

func (db *DB) IndexedAttrs() (categorical, value []string)

IndexedAttrs returns the currently-indexed attribute names, split into categorical (string equality/membership) and value (numeric + range) indexes.

func (*DB) InsertDerived added in v0.41.0

func (db *DB) InsertDerived(ads []*classad.ClassAd) (int, error)

InsertDerived writes ads into the table under keys derived from its declared primary key, as ONE transaction.

One transaction because a roll-up that half-applied would be worse than one that did not run: the table would hold a mixture of this run's rows and the last run's, with no marker saying where the boundary is, and the natural fix (run it again) would not repair it.

Re-running a roll-up over the same source IS the expected use, and it converges rather than accumulating: a derived key is a function of the row's key columns, so a second run writes the same keys and Txn.NewClassAd replaces each. That is the whole reason the key is declared on the table and derived from the data instead of generated.

func (*DB) InsertSelectInto added in v0.41.0

func (db *DB) InsertSelectInto(rows []AggRow, spec InsertSpec) (int, error)

InsertSelectInto is the write half end to end: validate the spec against the target's declared key, build the ads and write them.

The key coverage is checked BEFORE any ad is built so the failure names the mismatch between the declaration and the statement -- "the key is (Day, Owner) and you inserted (Day, Jobs)" -- rather than surfacing later as the same complaint about every row in turn, which says the same thing len(rows) times and buries which column was meant.

func (*DB) InstanceID

func (db *DB) InstanceID() string

InstanceID is this open's identity (fresh each Open). Equal in spirit to ID until HA replicas exist, when replicas of one DB share ID but differ by InstanceID.

func (*DB) Keys

func (db *DB) Keys() []string

Keys returns every committed key at a consistent snapshot, in no particular order. Useful for administrative enumeration and for a replica that must clear its keyspace before a full re-sync (see the leader-follower replicator).

func (*DB) KeysWhere added in v0.20.4

func (db *DB) KeysWhere(constraint string) (iter.Seq[string], error)

KeysWhere returns an iterator over the storage keys of every row whose ad matches constraint. Unlike a query, it yields the DB keys themselves, so a caller can address matched rows for UPDATE/DELETE without depending on any self-reported key attribute in the ad. The constraint is parsed eagerly (a parse error returns before any scan); the scan is lazy and stops early if the caller's yield returns false. It matches against decoded ads (a full scan, like DeleteWhere).

func (*DB) KeysWhereRedacted added in v0.35.0

func (db *DB) KeysWhereRedacted(constraint string) (iter.Seq[string], error)

KeysWhereRedacted is KeysWhere for a caller NOT entitled to sealed values: each ad is matched as decoded with no key, so which keys come back says nothing about a sealed value.

func (*DB) Len

func (db *DB) Len() int

Len returns the number of committed ads a reader is shown: reserved system-keyed records are not among them, as they are not in Query either (including structural parent-only ads of a chained collection, which Query hides -- see Collection.Len / Chained).

func (*DB) LookupClassAd

func (db *DB) LookupClassAd(key string) (*classad.ClassAd, bool)

LookupClassAd returns the committed ad for key (the hash table, outside any transaction), or (nil, false).

func (*DB) LookupClassAdRedacted added in v0.28.0

func (db *DB) LookupClassAdRedacted(key string) (*classad.ClassAd, bool)

LookupClassAdRedacted is LookupClassAd for a caller not entitled to sealed values; see QueryRedacted.

func (*DB) MaintProgress added in v0.34.0

func (d *DB) MaintProgress() collections.MaintProgress

MaintProgress reports the maintenance pass running on this table, or the zero value when none is. It takes no lock, so it answers while the pass it describes holds the maintenance lock. See collections.Collection.MaintProgress.

func (*DB) Maintain

func (db *DB) Maintain(opts MaintainOptions)

Maintain runs one self-tuning pass: it auto-tunes indexes (adds demand-driven ones, trims auto indexes over the memory budget, never touches human-created ones), refreshes the hot-attribute set, and optionally retrains the compression dictionary. Index/hot changes are persisted. Synchronous; a server drives it on a schedule (see dbrpc.Server.StartMaintenance).

func (*DB) MaintenanceStopping added in v0.34.0

func (d *DB) MaintenanceStopping() bool

MaintenanceStopping reports whether this table has been asked to stop maintenance. See collections.Collection.MaintenanceStopping.

func (*DB) Match

func (db *DB) Match(job *classad.ClassAd) iter.Seq[*classad.ClassAd]

Match returns the ads that symmetrically match job (bilateral Requirements), pushed down to the store. For the negotiator's pick-best pattern, prefer MatchSorted.

func (*DB) MatchSorted

func (db *DB) MatchSorted(job *classad.ClassAd, limit int) []*classad.ClassAd

MatchSorted returns job's matches ranked by the job's Rank, best first, at most limit (<=0 = all) -- the negotiator resource-request path, with the store's deferred materialization so only the returned top-N are built.

func (*DB) MatchSortedRanked

func (db *DB) MatchSortedRanked(job *classad.ClassAd, limit int) []RankedMatch

MatchSortedRanked is MatchSorted that also returns each match's Rank. The job's Requirements prunes candidate slots via any covering index (the matchmaking pushdown) rather than bilaterally evaluating every slot.

func (*DB) MatchSortedRankedFiltered

func (db *DB) MatchSortedRankedFiltered(job *classad.ClassAd, targetConstraint string, limit int) ([]RankedMatch, error)

MatchSortedRankedFiltered is MatchSortedRanked restricted to slots that also satisfy targetConstraint (a MATCH's WHERE TARGET / NOPREEMPT filter over the resource ad). The constraint's index probes narrow the candidate scan (pushdown) and it is re-checked on each matched slot. An empty constraint is exactly MatchSortedRanked.

func (*DB) MatchSortedRankedFilteredRedacted added in v0.35.0

func (db *DB) MatchSortedRankedFilteredRedacted(job *classad.ClassAd, targetConstraint string, limit int) ([]RankedMatch, error)

MatchSortedRankedFilteredRedacted is MatchSortedRankedFiltered for a caller NOT entitled to sealed values: the slots, and the resource-side constraint on them, are evaluated with no key. See collections.Collection.MatchSortedRankedFilteredRedacted.

func (*DB) MatchSortedRedacted added in v0.35.0

func (db *DB) MatchSortedRedacted(job *classad.ClassAd, limit int) []*classad.ClassAd

MatchSortedRedacted is MatchSorted for a caller NOT entitled to sealed values: every slot is matched, ranked and returned as decoded with no key. See collections.Collection.MatchSortedRedacted.

func (*DB) NumStats added in v0.25.2

func (db *DB) NumStats(constraint, attr string) (NumStats, bool)

NumStats computes the aggregate inputs for attr over the records matching constraint, via the columnar scan. ok=false means the columnar path cannot serve it and the caller should scan.

func (*DB) NumValues added in v0.37.0

func (db *DB) NumValues(constraint, attr string, newSink func() func(NumValue), merge func()) bool

NumValues feeds the numeric values of attr, over the records matching constraint, to sinks via the columnar scan. false means the columnar path cannot serve it and the caller should scan.

The pass may fan out across segments: newSink is called once per worker and merge once at the end, both on the calling goroutine, and each sink is fed by one goroutine only. See collections.NumValuesQuery.

func (*DB) NumValuesRedacted added in v0.37.0

func (db *DB) NumValuesRedacted(constraint, attr string, newSink func() func(NumValue), merge func()) bool

NumValuesRedacted is DB.NumValues for a reader not entitled to sealed values.

func (*DB) OpStats added in v0.11.0

func (db *DB) OpStats() OpStats

OpStats returns the store's operational timing counters (see the OpStats type).

func (*DB) Ordered

func (db *DB) Ordered(index int, partition string, resume OrderCursor) iter.Seq[OrderedAd]

Ordered iterates one partition of the index-th configured ordered index in sort order (Config.Ordered), yielding each member ad with a resume cursor and its cluster signature (for run-length folding into resource-request lists). partition selects the run (e.g. an Owner); it is ignored for an index with no Partition. A zero resume starts at the beginning. The snapshot is O(1) and stable under concurrent churn.

func (*DB) OrderedRedacted added in v0.28.0

func (db *DB) OrderedRedacted(index int, partition string, resume OrderCursor) iter.Seq[OrderedAd]

OrderedRedacted is Ordered for a caller NOT entitled to sealed values; see QueryRedacted.

func (*DB) Persistent added in v0.33.2

func (db *DB) Persistent() bool

Persistent reports whether the table is backed by a directory (collections.Persistent). Both storage forms serve every query API; this is for a caller that needs to know which one it has -- a test asserting its own fixture, or a tool reporting the table's mode.

func (*DB) PrimaryKey added in v0.41.0

func (db *DB) PrimaryKey() []string

PrimaryKey returns the attributes whose values derive this table's storage keys, or nil if none is declared. The slice is a copy: the caller may keep or modify it.

func (*DB) Put added in v0.8.0

func (db *DB) Put(key string, ad *classad.ClassAd) error

Put inserts or replaces the ad at key in its own optimistic transaction, retrying on a write-write conflict (another committer touched key since this attempt's snapshot) until it lands or maxWriteAttempts is exhausted. A blind overwrite that loses the optimistic race would otherwise be silently dropped; Put is the retrying convenience for the common single-ad upsert so callers do not each reimplement the loop. Equivalent to Begin + NewClassAd + Commit.

func (*DB) Query

func (db *DB) Query(constraint string) (iter.Seq[*classad.ClassAd], error)

Query returns the committed ads matching the constraint expression (an "old ClassAd" boolean expression over each ad, e.g. `JobStatus == 2 && Owner == "alice"`). The store pushes the filter down -- indexed constraints visit only candidates -- so this is far cheaper than ForEach + client-side filtering. Errors only on a malformed constraint.

func (*DB) QueryAsOf added in v0.12.0

func (db *DB) QueryAsOf(constraint string, t time.Time) (iter.Seq[*classad.ClassAd], error)

QueryAsOf runs a point-in-time ("AS OF") query: it returns the ads matching the constraint as they were at time t. It errors on a malformed constraint, when time travel is not enabled on this table, or when t is older than the retained window.

func (*DB) QueryAsOfRedacted added in v0.35.0

func (db *DB) QueryAsOfRedacted(constraint string, t time.Time) (iter.Seq[*classad.ClassAd], error)

QueryAsOfRedacted is QueryAsOf for a caller NOT entitled to sealed values: the constraint is matched, and each ad decoded, with no key. See collections.Collection.QueryAsOfRedacted.

func (*DB) QueryColumns added in v0.37.0

func (db *DB) QueryColumns(constraint string, attrs []string, redact bool) (iter.Seq[*collections.Batch], error)

QueryColumns yields the values of attrs for the records matching constraint, in batches of typed columns (see collections.ScanColumns). An empty or "true" constraint reads every record. redact withholds values this reader is not entitled to, as the row reads do.

attrs is what the caller wants back, in the order it wants them: every batch carries exactly these columns, in this order, including for a record that has none of them -- which reports undefined rather than a short row, so a consumer can bind its output once.

func (*DB) QueryColumnsStats added in v0.37.0

func (db *DB) QueryColumnsStats(constraint string, attrs []string, redact bool,
	stats *collections.ColumnScanStats) (iter.Seq[*collections.Batch], error)

QueryColumnsStats is QueryColumns that also fills stats (may be nil) with the per-scan work breakdown -- including how many cells came from the block's columns and how many had to be read from the records themselves.

That split is worth a caller's attention rather than a footnote. A projection naming an attribute the table's schema does not carry is served correctly and reads every one of its rows the slow way, and nothing else in the result says so.

func (*DB) QueryProject

func (db *DB) QueryProject(constraint string, attrs []string) (iter.Seq[[]classad.Value], error)

QueryProject returns, for each ad matching the constraint, just the named attributes' values (aligned with attrs), read wire-native where possible so an aggregate or projection does not pay the full-ad decode Query costs. The yielded slice is reused across iterations; copy any value to retain it past the next step. Errors only on a malformed constraint.

func (*DB) QueryProjectRedacted added in v0.35.0

func (db *DB) QueryProjectRedacted(constraint string, attrs []string) (iter.Seq[[]classad.Value], error)

QueryProjectRedacted is QueryProject for a caller NOT entitled to sealed values: the constraint is matched and attrs are read with no key, so a sealed value is undefined to both. See collections.Collection.QueryProjectRedacted.

func (*DB) QueryProjectStats added in v0.32.1

func (db *DB) QueryProjectStats(constraint string, attrs []string, stats *collections.ScanStats) (iter.Seq[[]classad.Value], error)

QueryProjectStats is QueryProject that also fills stats (may be nil) with the scan's work breakdown -- the mutable-table counterpart of ArchiveTable.QueryProjectStats. It exists so an EXPLAIN ANALYZE of an aggregate that falls through to this projected scan can show where the time went, specifically a large RecordsReassembled, the slow per-record path. Stats accumulate as the returned sequence is consumed.

func (*DB) QueryProjectStatsRedacted added in v0.35.0

func (db *DB) QueryProjectStatsRedacted(constraint string, attrs []string, stats *collections.ScanStats) (iter.Seq[[]classad.Value], error)

QueryProjectStatsRedacted is QueryProjectStats with QueryProjectRedacted's entitlement.

func (*DB) QueryRaw added in v0.8.0

func (db *DB) QueryRaw(constraint string) (iter.Seq[collections.RawAd], error)

QueryRaw yields each matching ad as a collections.RawAd -- the wire-form attribute strings decoded straight from the stored representation with no AST, for a persistent (inline) store as well as an in-memory one -- so a whole-ad result set can be relayed without materializing and re-encoding each ad. Errors only on a malformed constraint.

func (*DB) QueryRawProjected added in v0.16.3

func (db *DB) QueryRawProjected(constraint string, projection []string, redact bool) (iter.Seq[collections.RawAd], error)

QueryRawProjected is QueryRaw restricted to the projected attribute names, applied inside the collection's decode walk: a non-projected attribute is skipped before any name resolution or value rendering, and a hot-header- covered projection is served from the hot header alone (see collections.ScanRawProjected). redact additionally strips private attributes. An empty projection means no attribute filter.

func (*DB) QueryRawProjectedFromSeq added in v0.29.5

func (db *DB) QueryRawProjectedFromSeq(constraint string, projection []string, after SeqCursor, limit int) (iter.Seq[collections.RawAd], *SeqPage, error)

QueryRawProjectedFromSeq is QueryRawProjected a page at a time: it yields at most limit matching ads after the cursor and reports where to resume.

The order is the store's commit sequence, which is what makes the cursor stable — see collections.QueryRawFromSeq. A caller pages by handing back the previous page's Next until More is false. The zero cursor starts at the beginning. After consuming a page's rows, check its Err: a cursor whose snapshot has been reclaimed yields nothing and Err wraps ErrSeqCursorExpired.

This is for a mutable table. An archive is already ordered newest-first with its limit pushed down, so a "last K" read there needs no cursor at all.

It is an entitled read: private attributes are kept and sealed values opened. Serve anyone else with QueryRawProjectedFromSeqRedacted.

func (*DB) QueryRawProjectedFromSeqRedacted added in v0.34.0

func (db *DB) QueryRawProjectedFromSeqRedacted(constraint string, projection []string, after SeqCursor, limit int) (iter.Seq[collections.RawAd], *SeqPage, error)

QueryRawProjectedFromSeqRedacted is QueryRawProjectedFromSeq for a caller not entitled to secrets, redacted as QueryRawProjected with redact set: private attributes are dropped and sealed values render as undefined, read with no key.

func (*DB) QueryRawProjectedRefs added in v0.24.0

func (db *DB) QueryRawProjectedRefs(constraint string, projection []string, redact bool) (iter.Seq[collections.RawAd], error)

QueryRawProjectedRefs is QueryRawProjected that also carries the attributes the projected expressions reference, transitively, so each yielded ad EVALUATES self-contained (collections chaseRefs).

The distinction matters whenever a projected attribute holds an expression rather than a literal, which is the norm for HTCondor data -- Requirements, Rank and friends are expressions over sibling attributes. Projecting to exactly the requested names drops those siblings, so the expression evaluates to undefined at the far end: asking for Requirements alone answers undefined where asking for the whole ad answers true.

QueryRawProjected remains the right call for a relay that must reproduce HTCondor's query protocol, which specifies exactly the requested attributes and nothing more. Use this one when the recipient is going to EVALUATE what it receives.

func (*DB) QueryRawProjectedRefsStats added in v0.29.1

func (db *DB) QueryRawProjectedRefsStats(constraint string, projection []string, redact bool, stats *collections.ScanStats) (iter.Seq[collections.RawAd], error)

QueryRawProjectedRefsStats is QueryRawProjectedRefs that also fills stats (may be nil) with the per-scan work breakdown, for EXPLAIN ANALYZE. An empty/true constraint takes the no-WHERE scan path and leaves stats zero.

func (*DB) QueryRawProjectedStats added in v0.32.0

func (db *DB) QueryRawProjectedStats(constraint string, projection []string, redact bool, stats *collections.ScanStats) (iter.Seq[collections.RawAd], error)

QueryRawProjectedStats is QueryRawProjected that also fills stats (may be nil) with the per-scan work breakdown -- the mutable-table counterpart of ArchiveTable.QueryRawProjectedStats. An empty/true constraint takes the no-WHERE scan path and leaves stats zero.

func (*DB) QueryRawRedacted added in v0.16.3

func (db *DB) QueryRawRedacted(constraint string) (iter.Seq[collections.RawAd], error)

QueryRawRedacted is QueryRaw with private (secret) attributes stripped inside the collection's decode walk -- an unprivileged consumer's whole-ad query pays no per-attribute re-classification and never renders a private value (see collections.ScanRawRedacted).

func (*DB) QueryRawWire added in v0.16.4

func (db *DB) QueryRawWire(constraint string, projection []string, redact bool) (iter.Seq[[]byte], error)

QueryRawWire yields each matching ad as a self-contained WIRE-FORM ROW (an inline-names subset ad assembled by slice copies -- see collections.ScanRawWire): the relay form for shipping ads to a remote consumer with the old-ClassAd render deferred to that consumer's client edge. projection restricts the entries (empty = whole ad); redact strips private attributes at the source. At-rest-encrypted values are opened during assembly (the consumer holds no data key).

Both storage forms serve wire rows: a persistent table's records already carry inline names, and an in-memory table's interned keys are rewritten to names as the row is assembled. ErrRawWireUnsupported is still returned, before any row, by a table that cannot produce self-contained rows at all -- a relay scan that yields nothing is indistinguishable from a query that matched nothing, so such a table must refuse rather than come back empty.

func (*DB) QueryRawWireRefs added in v0.33.2

func (db *DB) QueryRawWireRefs(constraint string, projection []string, redact bool) (iter.Seq[[]byte], error)

QueryRawWireRefs is QueryRawWire whose projection also carries the attributes the projected expressions reference, transitively, so each yielded ROW evaluates self-contained -- the wire-row twin of QueryRawProjectedRefs, and it returns the same attribute set that call does for the same projection.

Which of the two a caller wants is decided the same way: QueryRawWire reproduces HTCondor's projection protocol (exactly the requested attributes), and this one serves a consumer that will EVALUATE what it receives, for which the references are not optional. The closure is computed per record, inside the scan, because whether a projected attribute holds an expression is a property of the stored ad rather than of the query.

func (*DB) QueryRawWireRefsStats added in v0.33.2

func (db *DB) QueryRawWireRefsStats(constraint string, projection []string, redact bool, stats *collections.ScanStats) (iter.Seq[[]byte], error)

QueryRawWireRefsStats is QueryRawWireRefs that also fills stats (may be nil) with the per-scan work breakdown. An empty/true constraint takes the no-WHERE scan path and leaves stats zero.

func (*DB) QueryRawWireStats added in v0.32.0

func (db *DB) QueryRawWireStats(constraint string, projection []string, redact bool, stats *collections.ScanStats) (iter.Seq[[]byte], error)

QueryRawWireStats is QueryRawWire that also fills stats (may be nil) with the per-scan work breakdown. An empty/true constraint takes the no-WHERE scan path and leaves stats zero.

func (*DB) QueryRedacted added in v0.28.0

func (db *DB) QueryRedacted(constraint string) (iter.Seq[*classad.ClassAd], error)

QueryRedacted is Query for a caller NOT entitled to sealed values: the ads are decoded with no key, so a sealed attribute arrives undefined rather than opened. See collections.Collection.QueryRedacted.

Query decodes with the table's key and leaves it to the serializer to drop private attributes, which means an unprivileged reader's secret is decrypted in this process and then filtered on the way out.

func (*DB) RecordDemand

func (db *DB) RecordDemand(constraint string)

RecordDemand notes the attributes constraint filters on (for index suggestions) without scanning. Use it for filters applied outside the normal Query path -- e.g. a MATCH's resource-side WHERE TARGET constraint -- so those attributes still surface in SuggestIndexes. A malformed constraint is ignored.

func (*DB) RefreshHotSet

func (db *DB) RefreshHotSet(sampleMax, topN int) int

RefreshHotSet recomputes the hot set as the topN most frequent attributes from a sample of up to sampleMax live ads, returning how many were chosen. The resulting hot set is persisted.

func (*DB) Reindex

func (db *DB) Reindex()

Reindex rebuilds all configured indexes from the live ads.

func (*DB) ReleaseRead added in v0.33.0

func (db *DB) ReleaseRead()

ReleaseRead balances an AcquireRead that returned true. The last reader out of a dropped table is the one that unmaps it, so call it with no lock held.

func (*DB) ReschemaScan added in v0.25.2

func (db *DB) ReschemaScan(sampleMax, hotTopN int) bool

ReschemaScan derives a new schema from a fresh sample and adopts it unconditionally (routine maintenance adopts a re-derivation only when the change is material). Sidecar blocks are rebuilt before it returns; columnarized segments are rewritten one ColumnarSegmentBudget now and the rest by maintenance (see collections.Collection.ReschemaScan). Returns false if there was nothing to sample.

func (*DB) Restore added in v0.7.0

func (db *DB) Restore(r io.Reader) error

Restore replaces the entire DB with the contents of the snapshot in r. It holds the DB-wide lock exclusively: all writers are blocked and the truncate+reload is atomic. An encrypted snapshot is opened with this DB's pool keys against the snapshot's embedded master envelope, so a snapshot taken by any DB sharing a pool key can be restored.

func (*DB) RestoreWith added in v0.7.0

func (db *DB) RestoreWith(r io.Reader, keys SnapshotKeys) error

RestoreWith is Restore using any level of the key hierarchy (see SnapshotKeys).

func (*DB) RestoreWithBackupKey added in v0.7.0

func (db *DB) RestoreWithBackupKey(r io.Reader, backupKey []byte) error

RestoreWithBackupKey is Restore using an explicitly-provided backup key (from BackupKey / the backup.key command) to unwrap the snapshot key, instead of the DB's pool keys. This recovers an encrypted backup independently of the pool keys -- the escrow path for disaster recovery when the original pool keys are unavailable.

func (*DB) RetrainDict

func (db *DB) RetrainDict(sampleMax int) (int, error)

RetrainDict trains a fresh ZSTD dictionary from a sample of up to sampleMax live ads, switches new writes to it, and recompresses existing records under it. It returns the new dictionary's size in bytes. This is what turns on (or refreshes) compression for a collection that started with the identity codec. See collections.Collection.RetrainDict.

func (*DB) Rewrite

func (db *DB) Rewrite() int

Rewrite re-encodes every live ad with the current hot set (so a changed hot set applies to existing ads) and force-compacts, returning the number of ads rewritten. A maintenance operation -- see collections.Collection.Rewrite.

func (*DB) SchemaFit added in v0.25.2

func (db *DB) SchemaFit(sampleMax int) ([]SchemaFieldFit, int)

SchemaFit measures the current derived schema against a fresh sample, reporting each field's escape rate -- how often its value is not in the fixed slot, and how much of that is the attribute simply being absent. This is how you tell whether the schema still matches the data (see collections.Collection.SchemaFit). Returns nil when the accelerator is not enabled.

func (*DB) SchemaRequireAttrs added in v0.32.0

func (db *DB) SchemaRequireAttrs() []string

SchemaRequireAttrs returns the effective required-attribute set -- the attributes that enter the derived schema whatever their presence rate (see Config.SchemaRequireAttrs).

func (*DB) SchemaRequireSkipped added in v0.32.0

func (db *DB) SchemaRequireSkipped() []SchemaRequireSkip

SchemaRequireSkipped returns the required attributes the last schema derivation could not make a field of, with the reason (absent from the sample, or no dominant literal value).

func (*DB) SchemaScanInfo added in v0.23.1

func (db *DB) SchemaScanInfo() SchemaScanInfo

SchemaScanInfo reports the columnar (adschema) accelerator's state: whether it is enabled (so a numeric COUNT(*) WHERE routes to the columnar fast path), its hot columns, and segment coverage.

func (*DB) SealsValues added in v0.35.0

func (db *DB) SealsValues() bool

SealsValues reports whether this table seals any attribute values at all -- every persistent table does, since private attributes are always sealed, whether or not EncryptionEnabled; an in-memory one never does. A table that seals nothing has nothing to withhold from a reader not entitled to sealed values. See collections.Collection.SealsValues.

func (*DB) SetEncryptedAttrs added in v0.7.0

func (db *DB) SetEncryptedAttrs(attrs []string) error

SetEncryptedAttrs replaces the explicit set of attributes encrypted at rest (the human-toggled set; private attributes are always encrypted). It errors if encryption is disabled or a named attribute is indexed. The policy is persisted so it survives a restart and, in HA, lets a follower converge on reload. New writes seal the new set; existing records are sealed by the migration the next maintenance pass or open runs (see collections.MigrateSealedAttrs) -- compaction copies stored values as they are. Until then unprivileged reads withhold every attribute the set names, whatever form it is stored in. The named attributes' values must not refer to other attributes (see Config.EncryptedAttrs).

func (*DB) SetPrimaryKey added in v0.41.0

func (db *DB) SetPrimaryKey(attrs []string) error

SetPrimaryKey declares the attributes whose values derive this table's storage keys, and persists the declaration. An empty attrs clears it.

It does NOT rewrite the existing rows. Rows already stored keep the keys they were written under, so declaring a key on a populated table does not reorganise it or make the old rows addressable by the new rule -- the declaration governs writes that ask for a derived key from here on. That is the honest behaviour for a store whose key is its physical address, and the alternative (rekeying every row) is a table rewrite that should be asked for explicitly rather than implied by a DDL statement.

func (*DB) SetSchemaRequireAttrs added in v0.32.0

func (db *DB) SetSchemaRequireAttrs(names []string) (bool, error)

SetSchemaRequireAttrs replaces the configured required-attribute set, keeping anything added at runtime. AddSchemaRequireAttrs / DropSchemaRequireAttrs change it by name. Each reports whether the effective set changed, and persists the runtime tier so it survives a restart.

None of them re-derives anything itself; a configuration reload must not start a rewrite of the table. The next maintenance pass adopts the schema the changed set produces and converges the sealed segments onto it, a ColumnarSegmentBudget of them per pass; ReschemaScan adopts at once. The error return is always nil: index-config persistence here is best-effort, as it is for AddIndex and the hot set. It exists so both table types present the same surface -- an archive writes archiveconfig.json and can fail -- and so a caller that starts checking it does not have to change shape.

func (*DB) SetTimeTravel added in v0.12.0

func (db *DB) SetTimeTravel(maxDistance, checkpoint time.Duration)

SetTimeTravel enables (with a positive maxDistance), retunes, or disables (maxDistance <= 0) point-in-time queries for this table, and persists the setting so it survives a restart. A zero checkpoint interval uses the collections default (1 minute). Enabling is not retroactive -- only changes from now on are travelable.

func (*DB) SidecarBytes added in v0.40.2

func (db *DB) SidecarBytes() (segments int, mappedBytes int64)

SidecarBytes reports the sidecar count and mapped bytes without the MPH/bloom breakdown, for a caller that prints only the on-disk total. See collections.Collection.SidecarBytes for the cost difference, which is large on an archive with thousands of sealed segments.

func (*DB) SidecarSizes added in v0.29.0

func (db *DB) SidecarSizes() SidecarSizes

SidecarSizes reports the sealed-segment sidecar index bytes (mmap-backed, evictable), matching ArchiveTable.SidecarSizes. A mutable table has these sidecars too -- only the accessor was missing, so its .stats could not report the on-disk footprint an archive's could.

func (*DB) Snapshot added in v0.7.0

func (db *DB) Snapshot(w io.Writer) error

Snapshot writes a consistent backup of every ad to w. It holds the DB-wide lock shared, so it is consistent against writers without blocking other readers or snapshots.

func (*DB) SnapshotWithKey added in v0.7.0

func (db *DB) SnapshotWithKey(w io.Writer) ([]byte, error)

SnapshotWithKey is Snapshot that also returns the per-backup (snapshot) key it minted -- the finest-grained escrow key, which decrypts THIS backup only (unlike the backup key, which decrypts all of them). Returns nil when encryption is disabled.

func (*DB) StaleIndexSegments added in v0.29.0

func (db *DB) StaleIndexSegments() (stale, sealed int)

StaleIndexSegments reports how many sealed segments still carry an index built under an older configuration, and how many are sealed in total, matching ArchiveTable.StaleIndexSegments.

func (*DB) StartMaintenance

func (db *DB) StartMaintenance(interval time.Duration, opts MaintainOptions) (stop func())

StartMaintenance starts a background goroutine that runs Maintain with the given options every interval, returning a stop function. Prefer the catalog-wide dbrpc.Server.StartMaintenance, which also covers tables created later.

func (*DB) Stats

func (db *DB) Stats() Stats

Stats returns a snapshot of the store's storage (ad count, segment/arena/dead bytes) for observability.

func (*DB) StopMaintenance added in v0.30.4

func (d *DB) StopMaintenance()

StopMaintenance asks any maintenance pass running on this table to stop at its next safe boundary, so a caller closing the server does not wait out a whole in-flight pass. See collections.Collection.StopMaintenance.

func (*DB) StoredLen added in v0.38.0

func (db *DB) StoredLen() int

StoredLen returns the number of committed ads the table holds, system-keyed records included. For accounting what is stored; Len is for how many rows a reader sees.

func (*DB) SuggestDrops

func (db *DB) SuggestDrops(sampleMax int) []DropSuggestion

SuggestDrops recommends indexes to drop (unused or low-cardinality) from observed demand and a sample of up to sampleMax live ads.

func (*DB) SuggestIndexes

func (db *DB) SuggestIndexes(sampleMax int) []collections.IndexSuggestion

SuggestIndexes samples the store and returns attributes that queries filter on but are not yet indexed (advisory; a server may log or auto-apply them).

func (*DB) TimeTravel added in v0.12.0

func (db *DB) TimeTravel() (maxDistance, checkpoint time.Duration, enabled bool)

TimeTravel reports the table's current point-in-time settings and whether enabled.

func (*DB) TopK added in v0.29.1

func (db *DB) TopK(constraint string, attrs []string, orderAttr string, desc bool, k int) ([][]classad.Value, error)

TopK is ArchiveTable.TopK for a mutable table.

func (*DB) TopKRedacted added in v0.35.0

func (db *DB) TopKRedacted(constraint string, attrs []string, orderAttr string, desc bool, k int) ([][]classad.Value, error)

TopKRedacted is TopK for a caller NOT entitled to sealed values: the constraint is matched, the rows ordered and the projection read with no key, so a sealed value is undefined throughout -- it neither comes back nor decides which rows do or in what order.

func (*DB) TopKRowsPerGroup added in v0.42.0

func (db *DB) TopKRowsPerGroup(constraint string, spec TopKRowsSpec) ([]TopKRowsGroup, error)

TopKRowsPerGroup answers spec over the records matching constraint.

The groups come back in no defined order -- a GROUP BY defines none -- but each group's rows are ordered by the order attribute, best first. A group with fewer than K ranked rows returns all of them.

func (*DB) TopKRowsPerGroupRedacted added in v0.42.0

func (db *DB) TopKRowsPerGroupRedacted(constraint string, spec TopKRowsSpec) ([]TopKRowsGroup, error)

TopKRowsPerGroupRedacted is TopKRowsPerGroup with private attributes stripped at the source, for serving a reader not entitled to them.

This rung's exposure is NARROWER than the wire rung's: it returns one value per attribute the caller NAMED and nothing else, so a server can also refuse a private name up front, as the aggregate handler does. The pair exists anyway, because the two rungs answer the same question and a reviewer comparing them should not find one that can redact and one that cannot -- and because the caller that reaches for the value rung next will not be looking for a gap this one left.

func (*DB) TopKRowsPerGroupRedactedStats added in v0.42.0

func (db *DB) TopKRowsPerGroupRedactedStats(constraint string, spec TopKRowsSpec, stats *collections.ScanStats) ([]TopKRowsGroup, error)

TopKRowsPerGroupRedactedStats is TopKRowsPerGroupRedacted filling stats (may be nil).

func (*DB) TopKRowsPerGroupStats added in v0.42.0

func (db *DB) TopKRowsPerGroupStats(constraint string, spec TopKRowsSpec, stats *collections.ScanStats) ([]TopKRowsGroup, error)

TopKRowsPerGroupStats is TopKRowsPerGroup filling stats (may be nil) with the scan's work.

func (*DB) TopKRowsPerGroupWire added in v0.42.0

func (db *DB) TopKRowsPerGroupWire(constraint string, spec TopKRowsSpec) ([]TopKRowsWireGroup, error)

TopKRowsPerGroupWire is TopKRowsPerGroup with the rows left in wire form. See the note above for what that changes: nothing about which rows are chosen or their order, and two things about what comes back -- the rows are bytes, and they carry the group and order attributes whether or not the caller's Attrs named them.

func (*DB) TopKRowsPerGroupWireRedacted added in v0.42.0

func (db *DB) TopKRowsPerGroupWireRedacted(constraint string, spec TopKRowsSpec) ([]TopKRowsWireGroup, error)

TopKRowsPerGroupWireRedacted is TopKRowsPerGroupWire with private attributes stripped at the source, for serving a reader not entitled to them.

IT IS NOT OPTIONAL FOR AN UNPRIVILEGED CALLER. The rows this rung returns are the stored bytes, so an unredacted row carries every private attribute the projection happened to pull in -- including ones the caller never named, since the scan chases references. The aggregate path has the same pair for the same reason.

func (*DB) TopKRowsPerGroupWireRedactedStats added in v0.42.0

func (db *DB) TopKRowsPerGroupWireRedactedStats(constraint string, spec TopKRowsSpec, stats *collections.ScanStats) ([]TopKRowsWireGroup, error)

TopKRowsPerGroupWireRedactedStats is TopKRowsPerGroupWireRedacted filling stats (may be nil).

func (*DB) TopKRowsPerGroupWireStats added in v0.42.0

func (db *DB) TopKRowsPerGroupWireStats(constraint string, spec TopKRowsSpec, stats *collections.ScanStats) ([]TopKRowsWireGroup, error)

TopKRowsPerGroupWireStats is TopKRowsPerGroupWire filling stats (may be nil) with the scan's work.

func (*DB) TopKStats added in v0.29.4

func (db *DB) TopKStats(constraint string, attrs []string, orderAttr string, desc bool, k int) ([][]classad.Value, collections.ScanStats, error)

TopKStats is DB.TopK that also returns the cutoff scan's ScanStats, for EXPLAIN ANALYZE.

func (*DB) TopKStatsRedacted added in v0.35.0

func (db *DB) TopKStatsRedacted(constraint string, attrs []string, orderAttr string, desc bool, k int) ([][]classad.Value, collections.ScanStats, error)

TopKStatsRedacted is TopKStats with TopKRedacted's entitlement.

func (*DB) TrackedKeys added in v0.30.0

func (db *DB) TrackedKeys() int

TrackedKeys reports how many keys the delta tracker holds in memory (0 when delta records are off). It is the feature's one new in-memory structure, so it is the number to watch.

func (*DB) Truncate added in v0.7.0

func (db *DB) Truncate()

Truncate removes every ad from the DB, atomically against all writers (it takes the DB-wide lock exclusively). A DAEMON-level operation -- the first half of a restore, or a deliberate wipe.

func (*DB) UpdateOld added in v0.8.0

func (db *DB) UpdateOld(key, text string) error

UpdateOld ingests an ad from old-ClassAd wire text under key, skipping the AST build -- the wire-native ingest path (as collections.UpdateOld). It writes through the same shard storage, change log, and watch feed as a committed Put, but bypasses the optimistic-concurrency layer (last-writer-wins), which suits high-rate single-key upserts where per-key write-write races do not occur (a collector re-advertising its own ad).

An encrypted store no longer takes the AST path wholesale. The wire-native encoder cannot seal, so the ingest defers to the sealing path for an ad that HAS something to seal and streams the rest (see collections.encodeOld) -- for job and history ads, nearly all of them.

func (*DB) UpdateOldBatch added in v0.8.0

func (db *DB) UpdateOldBatch(items []OldAdText) error

UpdateOldBatch ingests many ads (key + old-ClassAd text) in one shard-commit batch -- the wire-native bulk ingest, so a burst of upserts costs one commit instead of one per ad. Bypasses the optimistic-concurrency layer (last-writer-wins) like UpdateOld. An encrypted store keeps the batch: sealing is decided per ad inside the ingest (see UpdateOld), so one commit still covers the whole batch.

func (*DB) VerifyingReads added in v0.31.4

func (db *DB) VerifyingReads() bool

VerifyingReads reports whether this table checks each record's stored checksum before using its bytes -- true unless Config.DisableReadVerification was set, and always false for an in-memory table. See collections.VerifyReadSkips for how many records that check has refused.

func (*DB) Watch

func (db *DB) Watch(ctx context.Context, cursor []byte) (iter.Seq[WatchEvent], error)

func (*DB) WatchCursor added in v0.7.0

func (db *DB) WatchCursor() ([]byte, error)

Watch streams changes committed after the given cursor (nil = from now). Cancel via ctx. Events arrive in commit order per key; see collections.Watch for the delivery and coalescing guarantees. WatchCursor returns an opaque cursor at the current head of the change log, so Watch(ctx, cursor) then streams only subsequent changes (no replay of current contents). Cheap; requires the store was opened with watch history.

func (*DB) WatchRedacted added in v0.28.0

func (db *DB) WatchRedacted(ctx context.Context, cursor []byte) (iter.Seq[WatchEvent], error)

WatchRedacted is Watch for a watcher NOT entitled to sealed values: each event's ad is decoded with no key, so a sealed attribute arrives undefined. A watch has the longest exposure of any read path -- it streams ads continuously. See collections.Collection.WatchRedacted.

func (*DB) WatchWatermark added in v0.38.3

func (db *DB) WatchWatermark() uint64

WatchWatermark is the DB's closed watch clock: every change at or below it has been handed to every attached watcher, and nothing committed later will be stamped at or below it. It is what a catch-up has caught up TO, and the one value that is both comparable across shards and non-decreasing. 0 when the store has no watch.

type DropSuggestion

type DropSuggestion = collections.DropSuggestion

Diagnostic and management types, re-exported from collections for callers that only import db.

type ExporterDef added in v0.14.0

type ExporterDef struct {
	Name string `json:"name"`
	// Kind selects the exporter implementation (e.g. "kafka"). The catalog does not
	// enumerate valid kinds; an unknown kind is simply ignored by any exporter that does
	// not recognize it.
	Kind string `json:"kind"`
	// Config is the exporter's own configuration, opaque to the catalog. It is stored and
	// returned verbatim.
	Config json.RawMessage `json:"config,omitempty"`
}

ExporterDef defines an external sink that mirrors this database's change stream elsewhere (for example, a Kafka change-data exporter that federates several instances through a broker). The catalog is a passive registry: it stores the definition and an opaque resume-state blob and never runs, interprets, or validates Config -- that is the job of the out-of-process exporter (which reads these over dbrpc). Keeping the DB free of any sink-specific dependency is deliberate.

type GroupAgreementItem added in v0.29.13

type GroupAgreementItem = collections.GroupAgreementItem

type GroupCol added in v0.16.8

type GroupCol struct {
	Attr        string
	BucketWidth int64
}

GroupCol is one GROUP BY column for a (possibly bucketed) aggregate: the attribute Attr, optionally floored into fixed-width buckets. BucketWidth == 0 groups by the raw attribute value; BucketWidth > 0 (seconds) groups by the epoch-aligned bucket floor(number(Attr)/BucketWidth)*BucketWidth, and a row whose Attr is not a finite number drops out of the result. This is the shared shape a bucketed aggregate and a bucketed materialized view can both group by.

type GroupSchemaAgreement added in v0.28.0

type GroupSchemaAgreement = collections.GroupSchemaAgreement

type GroupSchemaChange added in v0.29.13

type GroupSchemaChange = collections.GroupSchemaChange

type GroupSchemaDelta added in v0.29.13

type GroupSchemaDelta = collections.GroupSchemaDelta

type GroupSchemaDrift added in v0.28.0

type GroupSchemaDrift = collections.GroupSchemaDrift

type GroupSchemaEntry added in v0.28.0

type GroupSchemaEntry = collections.GroupSchemaEntry

type GroupSchemaInfo added in v0.28.0

type GroupSchemaInfo = collections.GroupSchemaInfo

GroupSchemaInfo, GroupSchemaEntry, GroupSchemaDrift and GroupSchemaAgreement are re-exported so a caller can read the group-schema reports without importing collections.

type GroupSchemaLastAgreement added in v0.29.13

type GroupSchemaLastAgreement = collections.GroupSchemaLastAgreement

type GroupStatsSource added in v0.28.0

type GroupStatsSource interface {
	// GroupStatsConstraintAggs answers the grouped stats for rows matching constraint, grouping by
	// the tuple of the given columns, each optionally floored into fixed-width buckets. An
	// aggregate column may carry a per-aggregate FILTER, and countFilters asks for a filtered
	// record count per group.
	GroupStatsConstraintAggs(constraint string, groups []collections.GroupSpec,
		aggCols []collections.GroupAggCol, countFilters []string) ([]collections.GroupStats, bool)
	// GroupStatsAllAggs answers them over every row, for a constraint the caller has established
	// is match-all (the predicate analysis has no predicate to work with there).
	GroupStatsAllAggs(groups []collections.GroupSpec, aggCols []collections.GroupAggCol,
		countFilters []string) ([]collections.GroupStats, bool)
}

GroupStatsSource is a table that can answer a grouped columnar aggregate: an archive or a mutable table. Both back onto the same collections primitive, so both get the same fast path and the same row-shaping rather than a copy per table type that could drift.

type GroupValuesSource added in v0.37.0

type GroupValuesSource interface {
	// GroupValuesConstraintAggs answers the groups for rows matching constraint, giving each
	// group a sink from newSink the first time it is seen.
	GroupValuesConstraintAggs(constraint string, groups []collections.GroupSpec,
		aggCols []collections.GroupAggCol, countFilters []string,
		newSink func() collections.GroupSink,
		mergeSink collections.MergeGroupSink) ([]collections.GroupValues, bool)
	// GroupValuesAllAggs answers them over every row, for a constraint the caller has established
	// is match-all.
	GroupValuesAllAggs(groups []collections.GroupSpec, aggCols []collections.GroupAggCol,
		countFilters []string, newSink func() collections.GroupSink,
		mergeSink collections.MergeGroupSink) ([]collections.GroupValues, bool)
}

GroupValuesSource is the storage-side grouped pass reduced into caller-supplied accumulators. It mirrors GroupStatsSource, which is the same pass reduced into NumStats.

type IndexBackfillResult added in v0.33.0

type IndexBackfillResult = collections.IndexBackfillResult

IndexBackfillResult reports one index backfill pass (see collections.IndexBackfillResult).

type IndexSize

type IndexSize = collections.IndexSize

Diagnostic and management types, re-exported from collections for callers that only import db.

type IndexSizes

type IndexSizes = collections.IndexSizes

Diagnostic and management types, re-exported from collections for callers that only import db.

type IndexSuggestion

type IndexSuggestion = collections.IndexSuggestion

Diagnostic and management types, re-exported from collections for callers that only import db.

type InsertSpec added in v0.41.0

type InsertSpec struct {
	GroupCols []GroupCol
	Aggs      []AggSpec
	Names     []string
}

InsertSpec names the columns of the rows being inserted.

Names has one entry per group column followed by one per aggregate -- the same order AggRow lays out Group and Values -- and each is the ATTRIBUTE the value is written under in the target table.

They are carried explicitly rather than derived from the source query because a group column's name is only sometimes its attribute (a time bucket has none) and an aggregate has none at all: `COUNT(*)` is a legal column header and not a legal ClassAd attribute name. A caller that let the header stand in for the attribute would be writing ads that no query can read back.

func (InsertSpec) Validate added in v0.41.0

func (s InsertSpec) Validate() error

Validate checks the spec on its own: one name per column, each a usable attribute, none repeated and none private.

A private name is refused for the same reason a private primary key column is (see validatePrimaryKey): the roll-up's output is an ordinary table that ordinary readers query, and an aggregate over private data written into a non-private attribute would launder it. Refusing the NAME is the narrow, checkable half of that -- it does not stop someone aggregating a private attribute into a public one, which is a question for whoever authorises the statement, but it does stop the result being stored somewhere that redaction no longer applies.

type JoinAggSpec added in v0.36.0

type JoinAggSpec struct {
	// Join is the join whose rows are reduced. Its Limit, if set, truncates the reduction's
	// INPUT, so a limited join aggregates a prefix of itself -- which is a different answer,
	// not a smaller one. Callers that want a bounded answer bound the groups, not the rows.
	Join JoinSpec
	// GroupCols is the GROUP BY, in output order. Each Attr is a ClassAd expression over the
	// composite row -- in practice a dotted path like `j.Owner`, but any expression the
	// evaluator accepts works, so `j.Owner` and `split(j.Cmd, "/")[0]` are the same kind of
	// thing here. Empty means a scalar aggregate over the whole join, which still yields
	// exactly one row.
	GroupCols []GroupCol
	// Aggs are the outputs, in output order. Arg is likewise an expression over the composite
	// row, except for "*" which counts rows. A spec's Filter, when set, is also an expression
	// over the composite row, and narrows that one aggregate rather than the group.
	Aggs []AggSpec
	// MaxGroups caps the groups the answer may carry; 0 takes the default. A request past
	// the cap is REFUSED rather than truncated, so see JoinAggregate for why.
	MaxGroups int
	// MaxListValues caps each list-valued aggregate's list within each group; 0 takes the
	// default. A list past the cap is CUT rather than refused, and the group reports how
	// many values it lost in AggRow.Elided.
	MaxListValues int
}

JoinAggSpec is a grouped aggregate over a join's rows.

It is an AggRequest whose rows come from a join instead of a table, which is why it has no Constraint of its own: a join already carries three places to filter (each side's own Constraint, the ON condition, and the Post clause over the composite row) and a fourth spelling of the same thing would only raise the question of where it ran.

type JoinBand added in v0.39.0

type JoinBand struct {
	// Attr names the attribute of THIS step's table the window is over.
	Attr string `json:"attr"`
	// Lo and Hi bound the window, as expressions over the sides folded in BEFORE this
	// step. An empty string leaves that end unbounded, which is how a one-sided
	// inequality is written: `h.Ts >= j.Start` is Lo alone.
	Lo string `json:"lo,omitempty"`
	Hi string `json:"hi,omitempty"`
	// LoOpen and HiOpen make that end exclusive -- `>` rather than `>=`.
	LoOpen bool `json:"loOpen,omitempty"`
	HiOpen bool `json:"hiOpen,omitempty"`
}

JoinBand is an ordered window over one attribute of a step's table.

It is a conjunct of the step's match, like JoinEqui: a row of the step's table joins only where its Attr falls inside the window, and the step's Cond still has to hold on top.

It is defined over NUMBERS. Attr joins when it evaluates to an integer, a real or a boolean -- the three the rest of the join already orders, with false below true -- and does not join when it is undefined, a string, a list, an error or NaN. A bound that does not evaluate to one of those three admits nothing, which is the rule an undefined hash key already follows: a key that cannot be compared matches nothing rather than everything.

That makes a band narrower than the ClassAd comparison it reads like, and deliberately so. ClassAd orders strings against strings -- `"abc" >= "abb"` is true -- and errors on a string against a number. A band answers neither: the first it declines, and the second it declines rather than erroring. A caller wanting ClassAd's own answer writes the comparison in Cond, where it is evaluated with ClassAd's semantics against every candidate; what is lost is the narrowing, not the question.

type JoinEqui added in v0.35.0

type JoinEqui struct {
	Left  string `json:"left"`
	Right string `json:"right"`
}

JoinEqui is one `<left expr> == <right expr>` conjunct of a step's condition, split into the half that reads what is already in the row and the half that reads the table being folded in. They drive the hash.

THE TWO HALVES ARE IN DIFFERENT SCOPES, which is the one asymmetry a reader has to hold. Right reads this step's own table, so it is a bare path -- `ClusterId`, not `c.ClusterId`. Left reads the accumulated row, which has one member per side folded in so far, so past the first step it must be QUALIFIED -- `a.ClusterId` -- because at the third step it might read the first table or the second. At the first step the row holds one side and a bare path is unambiguous, which is why the two-table spelling has always written Left bare and keeps working unchanged.

They narrow candidates only. Every surviving pair is still checked against the full condition, so the hash never has to reproduce ClassAd's equality semantics exactly -- it only has to avoid putting two rows in different buckets when they would compare equal.

type JoinExplain added in v0.38.0

type JoinExplain struct {
	// Planner names the rules that chose the order: "build-side" for a two-table join,
	// "chain" for three or more. They ask different questions -- which of two to hash,
	// against which of several to read FROM -- so which one spoke is part of the answer.
	Planner string `json:"planner"`
	// Sides are the tables in the order the PLAN reads them, the streamed side first.
	Sides []JoinSideExplain `json:"sides"`
	// AsWritten is the aliases in the order the CALLER wrote them. It differs from Sides
	// exactly when Reordered is true.
	AsWritten []string `json:"asWritten"`
	// Reordered reports that the plan reads the tables in a different order than the query
	// wrote them. It changes which order the rows come out in, so a reader who notices
	// that needs to be able to find out it was deliberate.
	Reordered bool `json:"reordered"`
	// OrderWhy is the planner's reason, set whether or not it moved anything.
	OrderWhy string `json:"orderWhy"`
	// OrderCost and OrderAsWrittenCost are what the chosen plan and the plan as written
	// were priced at, in records touched. Both are 0 when the planner declined before
	// pricing anything -- an order it may not change costs nothing to not compare.
	OrderCost          int `json:"orderCost,omitempty"`
	OrderAsWrittenCost int `json:"orderAsWrittenCost,omitempty"`
	// OuterDriven reports that the plan streams the FROM side in key batches and resolves
	// each batch against the inner side's index, holding neither side whole (db/joinouter.go).
	// OuterDrivenBatchKeys is how many FROM rows one batch covers.
	//
	// It changes what EstHashRows means: on this plan the hold is one batch's matches, so
	// there is no side-sized number to report and HashWhy says so instead.
	OuterDriven          bool `json:"outerDriven,omitempty"`
	OuterDrivenBatchKeys int  `json:"outerDrivenBatchKeys,omitempty"`
	// Steps are the folds after the streamed side, in plan order.
	Steps []JoinStepExplain `json:"steps"`
	// Limits are what the join would actually run under: the store's own, lowered to the
	// caller's where the caller asked for less.
	Limits JoinLimits `json:"limits"`

	// EstHashRows is an UPPER BOUND on the rows this plan would hold: every side but the
	// streamed one, each at what its own read is estimated to visit. It is the quantity
	// MaxHashRows bounds, and since the planner chooses which side to hash in order to
	// stay under that bound, it is the number that says whether the plan runs at all.
	//
	// A BOUND and not a prediction, and the gap can be enormous: measured on a selective
	// side joined to an indexed one, this is 8192 where 32 rows are held. An index probe
	// narrows a hashed side to the key values the streamed side turns out to hold, and how
	// many that is cannot be known without reading the streamed side, which this does not
	// do. A step reported as probe-eligible (ProbeAttr) may therefore hold a small fraction
	// of its share of this.
	//
	// Being a bound rather than an estimate is what makes it worth publishing, because it
	// is sound in the direction that matters. At or under MaxHashRows it is a GUARANTEE:
	// the plan cannot be refused for holding too much, since what is held is at most this.
	// Over it, the plan is refused unless a probe narrows one of the sides, and HashWhy
	// names the sides that might. An estimate 256x out would be worse than no number at
	// all; a bound 256x out still answers "will this run".
	//
	// 0 means a hashed side could not be sized at all, and HashWhy names it. A partial sum
	// is not reported: it would read as a bound on the plan and is not one.
	EstHashRows int `json:"estHashRows,omitempty"`
	// HashWhy states EstHashRows against the limit in words: within it, over it, or not
	// known. A plan over the limit is one the join will REFUSE, unless a probe narrows one
	// of the sides enough -- the single case where this report cannot be definite, and it
	// says so by naming the sides that might.
	HashWhy string `json:"hashWhy,omitempty"`
}

JoinExplain is how a join would be executed.

type JoinKind added in v0.35.0

type JoinKind uint8

JoinKind is which rows survive a join with no match.

const (
	// JoinInner keeps only paired rows.
	JoinInner JoinKind = iota
	// JoinLeft keeps every left row, null-extending the ones that paired with nothing.
	JoinLeft
	// JoinCross pairs every row with every row; its condition, if any, decides.
	JoinCross
	// JoinSemi keeps a row once when this side has at least one match, and carries nothing
	// from this side into it. It is SQL's `EXISTS`: the question is whether a match exists,
	// not what matched, so the walk stops at the first one and the row's fan-out over this
	// side is one however many rows matched.
	JoinSemi
	// JoinAnti keeps a row when this side has no match at all, and carries nothing from this
	// side into it. It is SQL's `NOT EXISTS`.
	//
	// It is the kind a `LEFT JOIN ... WHERE <right key> =?= undefined` was emulating, and it
	// differs from that emulation in what it costs rather than in what it answers: the
	// emulation forms every matching pair and then drops it, so a row that matched a hundred
	// times paid a hundred pairs to establish that it matched once.
	JoinAnti
)

type JoinLimits added in v0.35.0

type JoinLimits struct {
	// MaxHashRows caps the rows held for the hashed side. Only one side is held, but it is
	// held entirely.
	//
	// It is NOT a memory bound, which is what MaxHashBytes is for. A held row is a decoded
	// ad, and what that costs is set by how many ATTRIBUTES it carries: measured at roughly
	// 180 + 130*attrs bytes, so the same row count means 400 MB for a narrow row and 7.7 GB
	// for a 120-attribute one. A count cannot express a budget across that spread.
	MaxHashRows int
	// MaxHashBytes caps the ESTIMATED heap of the rows held, which is the bound that
	// actually protects the process. The estimate is rows * (hashRowBytesBase +
	// hashRowBytesPerAttr*width), with width sampled from the side rather than assumed.
	//
	// It is an estimate, so it is applied where estimates belong -- choosing a plan -- and
	// buildStage additionally counts what it really holds.
	MaxHashBytes int
	// MaxPairs caps candidate pairs EXAMINED. Neither the per-side cap nor hashing bounds
	// this: a key with one distinct value puts every row in one bucket, and that bucket is
	// the cross product. So the budget is spent where pairs are actually examined, which is
	// the only place that sees both sides.
	MaxPairs int
	// MaxRows caps joined rows produced.
	MaxRows int
}

JoinLimits bound what one join may spend. They are the store's own, not the caller's: a request must not be able to ask a daemon to hold or produce an unbounded amount, so these are applied here and a caller can only ask for less (see clamp).

func DefaultJoinAggLimits added in v0.36.0

func DefaultJoinAggLimits() JoinLimits

DefaultJoinAggLimits are the limits a join aggregate runs under when a caller names none.

They differ from DefaultJoinLimits in exactly one place, and for one reason: MaxRows bounds the joined rows PRODUCED, which in a streaming join is the bound on what crosses the wire. Here the rows are reduced where they are made and none of them crosses anything, so that bound is being applied to a quantity it was not written about -- and applying it would refuse the queries this is for, since a history join big enough to be worth reducing in the store is routinely past a million rows.

The budgets that bound real work are unchanged. MaxHashRows still bounds what is held, and MaxPairs -- which is what actually prices a join, since one bucket can be a cross product -- still bounds what is examined. MaxRows is raised to meet MaxPairs rather than removed, so the reduction's input is still bounded by something.

func DefaultJoinLimits added in v0.35.0

func DefaultJoinLimits() JoinLimits

DefaultJoinLimits are the limits a join runs under when a caller names none.

type JoinProbeStat added in v0.37.0

type JoinProbeStat struct {
	// Alias names the side this step folds in.
	Alias string
	// Attr is the attribute a membership constraint was built on, empty when none was.
	Attr string
	// Keys is how many distinct key values the scan was narrowed to, 0 when it was not.
	Keys int
	// EstRecords is the size the decision priced this side's scan at: the records its read
	// was estimated to visit, which is the table's whole size unless the side carries a
	// constraint of its own. 0 when the gates declined before a size was needed.
	//
	// It is reported because the decision cannot be checked without it. "Narrowing did not
	// pay" and "narrowing was priced against the wrong table size" read identically in a
	// plan that states only the outcome.
	EstRecords int
	// Why says why the scan was not narrowed, empty when it was.
	Why string
	// Tuple names the attribute tuple the scan was narrowed on as a WHOLE
	// ("ClusterId+ProcId"), empty when the narrowing was one membership constraint per
	// attribute, which admits their cross product rather than the tuples actually held.
	Tuple string
	// TupleComposite says whether a composite index answers that tuple in one lookup. False
	// with a non-empty Tuple means the narrowing is the same but each disjunct costs one
	// lookup per component, whose postings are then intersected.
	TupleComposite bool
	// TupleWhy says why the scan was not narrowed on the tuple as a whole, empty when it
	// was or when the step narrows on one attribute.
	//
	// Reported separately from Why because the two decisions are separate: a step can
	// decline the tuple form on cost and still narrow per attribute, in which case Why is
	// empty and this is not. Without it that step is indistinguishable from one that was
	// never a candidate.
	TupleWhy string
}

JoinProbeStat is one step's index-probe decision.

type JoinSide added in v0.35.0

type JoinSide struct {
	Table string
	// Alias is the member name this side takes in the joined row. It is what the caller's
	// expressions reference, so it must match what they were written against.
	Alias string
	// Constraint is the predicate pushed into this side's scan; "" matches everything.
	Constraint string
	// Project names the attributes this side contributes to the joined row. Nil carries the
	// whole ad.
	//
	// This is the output projection, and it is the one that matters: the rows do not cross
	// a wire until they are joined, so what bounds the result is how much of each paired ad
	// is carried, not how much was read.
	Project []string
}

JoinSide is one side of a join: which table, under what alias, and what to push into its own scan.

type JoinSideExplain added in v0.38.0

type JoinSideExplain struct {
	Alias      string `json:"alias"`
	Table      string `json:"table"`
	Constraint string `json:"constraint,omitempty"`
	// Streamed marks the one side the join reads as a stream. Every other side is hashed,
	// which is what MaxHashRows bounds.
	Streamed bool `json:"streamed"`
	// Read is the access path for this side's own constraint: the same report `.explain`
	// gives for a query against that table. It is what says whether the side is read
	// through an index or scanned, and EstVisited in it is the size the join priced it at.
	Read *QueryExplain `json:"read,omitempty"`
	// ReadWhy says why Read is absent -- a table this catalog does not have, or a
	// constraint that does not plan. A side whose read cannot be described is also one the
	// order decision could not price.
	ReadWhy string `json:"readWhy,omitempty"`
}

JoinSideExplain is one table of the join and how it would be read.

type JoinSpec added in v0.35.0

type JoinSpec struct {
	Kind        JoinKind
	Left, Right JoinSide

	// Steps is the chain spelling: one step per table after Left. When it is set, Right,
	// Kind, Cond and Equi must be empty -- two spellings in one spec have no single
	// meaning, and picking one would answer a query nobody wrote.
	Steps []JoinStep

	// Cond are the ON conjuncts that reference both sides -- the join condition proper --
	// as ClassAd expression text over the joined row. A pair matches when every one is true.
	Cond []string
	// Post are the WHERE conjuncts over the joined row, applied after the pair is formed
	// and, for a LEFT join, after an unmatched left row has been null-extended.
	Post []string
	// Equi are the equality conjuncts of Cond whose halves each read only one side.
	Equi []JoinEqui
	// Band are the ordered-range conjuncts of Cond, each a window over one attribute of
	// Right. See JoinBand.
	Band []JoinBand

	// Limit stops after this many joined rows (<= 0 = all).
	Limit int
}

JoinSpec is a planned join of two or more tables.

There are two spellings and a spec carries exactly one. Left/Right/Kind/Cond/Equi is the two-table spelling, unchanged and still what the two-table opcodes send. Left plus Steps is the chain spelling, where Left is the FROM table and each step folds in one more. The two are the same plan: the two-table spelling IS the chain with a single step, and compiles to it, so there is one pair loop rather than one per spelling.

type JoinStats added in v0.35.0

type JoinStats struct {
	HashRows int // rows held for the hashed side
	// HashBytes is the estimated heap of the rows held, from each side's sampled ad width.
	// It is what MaxHashBytes bounds, and it is an ESTIMATE -- the row count beside it is
	// exact, and the two disagree about how much trouble a join is in whenever the ads are
	// wider or narrower than the sample suggested.
	HashBytes int
	// OuterDrivenBatches counts the key batches an outer-driven join resolved against the
	// inner side's index (db/joinouter.go). It is 0 for every other plan, which is what
	// distinguishes a join that ran batched from one where that path declined -- a
	// distinction no other counter here makes, since a declined path and a cheap one
	// produce the same rows.
	//
	// On this path HashRows is a HIGH-WATER MARK over the batches rather than a total.
	OuterDrivenBatches int
	ScanRows           int // rows streamed from the other side
	// BandWindows counts the windows a band join binary-searched, one for each accumulated
	// row that reached a banded step, and BandCandidates the rows those windows admitted.
	// BandWindows is 0 for every other plan, which is what tells a join that ran banded
	// from one where the step was scanned whole -- the rows are the same either way.
	//
	// BandCandidates against BandWindows times the sorted side's rows is the saving: the
	// second is the pairs the same step would have examined with no band at all.
	BandWindows    int
	BandCandidates int
	Pairs          int // candidate pairs examined
	Rows           int // joined rows produced
	Groups         int // groups a reduced join returned (0 when the rows were not reduced)
	Err            error

	// ProbeAttr and ProbeKeys describe an index-based equi join: the attribute the hashed
	// side's scan was narrowed on, and how many distinct key values it was narrowed to.
	// ProbeKeys is 0 when the scan was not narrowed, and ProbeWhy then says why -- because
	// a plan that silently did not use an index reads exactly like one that had none.
	//
	// They describe the FIRST step, which for a two-table join is the only one. A longer
	// chain decides per step, and ProbeSteps carries all of them.
	ProbeAttr string
	ProbeKeys int
	ProbeWhy  string

	// ProbeSteps is the probe decision for each step of the chain, in order.
	ProbeSteps []JoinProbeStat

	// BuildSideSwapped reports that the planner exchanged the sides, so the side written
	// second was streamed and the side written first was hashed. BuildSideWhy says why, on
	// either answer, and BuildSideCost / BuildSideSwapCost are what the two orientations
	// were priced at in records touched (0 when neither was priced).
	//
	// Reported because a swap changes which order the rows come out in, and a reader who
	// notices that needs to be able to find out that it was deliberate.
	BuildSideSwapped  bool
	BuildSideWhy      string
	BuildSideCost     int
	BuildSideSwapCost int

	// ChainFromAlias names the table a chain was re-planned to read FROM, empty when the
	// chain was left as written. ChainOrderWhy says why on either answer, and the two costs
	// are what the chosen plan and the plan as written were priced at in records touched.
	//
	// The FROM side of a chain is where every step's index probe gets its keys, so this is
	// the decision that says whether the chain reads its tables through an index at all --
	// and, like a swap, it changes the order the rows come out in.
	ChainFromAlias          string
	ChainOrderWhy           string
	ChainOrderCost          int
	ChainOrderAsWrittenCost int
}

JoinStats is what the join did, and whether it finished.

Err carries a failure met WHILE streaming -- a limit reached, most likely -- which the result sequence cannot report, since it yields rows. A caller must check it after draining or it will read a truncated answer as a complete one.

type JoinStep added in v0.37.0

type JoinStep struct {
	Kind JoinKind
	// Side is the table this step folds in, under the alias the row will carry it as.
	Side JoinSide
	// Cond are this step's ON conjuncts, over the row AS IT IS AFTER this step. A conjunct
	// naming a side a LATER step folds in is refused rather than deferred: SQL does not
	// allow a forward reference, and deferring one would silently answer a different query.
	Cond []string
	// Equi are the equality conjuncts of Cond that drive this step's hash. Left is written
	// over the accumulated row and Right over this step's own table; see JoinEqui.
	Equi []JoinEqui
	// Band are the ordered-range conjuncts of Cond, each a window over one attribute of
	// this step's table. A step with no Equi and one Band is searched rather than scanned;
	// see JoinBand and db/joinband.go.
	Band []JoinBand
}

JoinStep folds one more table into the joined row.

A chain of tables is LEFT-DEEP: the row starts as the first side and each step adds one member to it. That is the only shape SQL's syntax can express without parentheses, and it is why an n-table join is this join run n-1 times rather than a second algorithm.

Kind is this step's own, and it decides null-extension for THIS side alone. In a left-deep chain a side is null-extendable exactly when its own step is a LEFT join: every earlier table is preserved by every later step, so being preserved once is being preserved throughout. That is why the rule is one comparison and not a walk back along the chain.

type JoinStepExplain added in v0.38.0

type JoinStepExplain struct {
	Alias string `json:"alias"`
	// Kind is this step's join: "inner", "left", "cross" -- or "right", which is a LEFT
	// join the planner chose to run from the other end, hashing the preserved side and
	// emitting its unmatched rows after the stream. "right" therefore always comes with
	// Reordered, and the two together are the whole statement: the caller's LEFT join,
	// read in the opposite order.
	Kind string `json:"kind"`
	// ProbeAttr is the attribute the step's table would be narrowed on, empty when it
	// would not be. ProbeWhy says why not, and is empty exactly when ProbeAttr is set.
	//
	// ProbeAttr being set is ELIGIBILITY, not a promise: see the file comment. The running
	// join still declines a step whose FROM side turns out to hold more distinct keys than
	// one constraint may carry.
	ProbeAttr string `json:"probeAttr,omitempty"`
	ProbeWhy  string `json:"probeWhy,omitempty"`
	// EstRecords is the size this table's read is estimated to visit: the whole table
	// unless the side carries a constraint of its own. It is both what narrowing this step
	// would avoid and, since every step holds its own table, this step's share of the
	// plan's EstHashRows.
	//
	// 0 only when the side could not be sized at all. It is filled in even where the probe
	// decision declined before it needed a size, because "how much does this step hold" is
	// asked of every step and not only of the ones an index could narrow.
	EstRecords int `json:"estRecords,omitempty"`
	// Cond and Equi are the step's conditions AS PLANNED. A reordered chain re-attaches
	// its conditions to different steps, so these are not always what the caller wrote --
	// which is the point of reporting them.
	Cond []string   `json:"cond,omitempty"`
	Equi []JoinEqui `json:"equi,omitempty"`
	// Band are the step's windows, and Banded says the step will be SEARCHED over the
	// first of them rather than scanned whole. The two differ: a step that also hashes
	// keeps its bands and is narrowed by the bucket instead, so Band is set there and
	// Banded is not.
	//
	// Banded is a promise, not an eligibility, and is the same test the walk makes. A
	// reader comparing it against JoinStats.BandWindows is comparing the plan against what
	// ran.
	Band   []JoinBand `json:"band,omitempty"`
	Banded bool       `json:"banded,omitempty"`
}

JoinStepExplain is one table folded into the accumulated row.

type KEK added in v0.7.0

type KEK = crypt.KEK

KEK is a pool / signing key that can wrap the DB master key. Re-exported so db callers need not import crypt.

type MaintainOptions

type MaintainOptions struct {
	// SampleMax caps the ads sampled for index tuning, hot-set frequency, and dictionary
	// training. Default 4096.
	SampleMax int
	// HotTopN refreshes the hot set to the topN most common attributes; 0 disables it.
	HotTopN int
	// Retrain retrains the ZSTD dictionary (recompacting + reindexing). Expensive on a
	// large store, so a server may run it on a longer cadence than the rest.
	Retrain bool
	// CompactInterval is the cadence of a lightweight, standalone compaction pass
	// (dbrpc.Server.StartMaintenance) that reclaims dead space independently of the
	// expensive retrain-dominated Maintain pass. Compaction is cheap and self-limiting
	// (it unlinks fully-dead segments for free and only recompacts shards past the
	// dead-byte threshold), so it runs far more often than Maintain -- essential for a
	// high-churn table (e.g. a collector re-advertising every few minutes) where dead
	// space would otherwise grow unbounded between the throttled retrain passes. 0 uses
	// a default cadence; a negative value disables the standalone compaction pass.
	CompactInterval time.Duration
	// MinIndexDemand is the minimum observed query count for the auto-tuner to add an
	// index; 0 leaves index auto-tune off (no demand-driven adds).
	MinIndexDemand int64
	// IndexBudgetHighFrac / IndexBudgetLowFrac / IndexBudgetSlackBytes bound auto-created
	// index memory as a fraction of the live data bytes (see collections.AutoTuneOptions);
	// 0 high frac disables the budget (auto indexes grow unbounded).
	IndexBudgetHighFrac   float64
	IndexBudgetLowFrac    float64
	IndexBudgetSlackBytes int64
	// ArchiveMerge tunes the merge pass run over each archive table. The zero value uses
	// the policy defaults, which target a segment count well clear of the process's mapping
	// budget.
	//
	// ArchiveMergeDisabled turns off the MERGE, leaving the rest of the archive pass running:
	// the stale-index reindex, the index backfill, the hot-set refresh, index auto-tune, the
	// schema scan and the demand checkpoint. It once meant the whole archive side, because
	// when it was written the archive side was the merge and the reindex that follows one;
	// the steps added since are not merges and do not re-encode anything, and the only one
	// that changes what future writes look like is governed by ArchiveHotRefreshDisabled.
	ArchiveMerge         MergeOptions
	ArchiveMergeDisabled bool
	// ArchiveRotateDisabled turns off RETENTION on archive tables: the step that drops whole
	// sealed segments falling outside ArchiveConfig.Retention (max segments / max bytes /
	// max age), plus the early drain of records a change feed has consumed (SetGCFloor).
	//
	// On by default, unlike the other archive steps that are off until configured, because
	// it is the only step in the pass that bounds the table's SIZE, and because it cannot
	// act on an archive that has not asked for it: retention is per-archive configuration
	// and the zero Retention keeps everything, so an archive with none configured drops
	// nothing no matter how often the pass runs. An embedder that configured retention and
	// never wired up a rotation loop of its own was getting unbounded growth from a policy
	// it had already written down.
	//
	// The switch exists because the step is the one irreversible thing the pass does. An
	// operator who wants the bounds enforced on a schedule of their own -- a quiet hour, a
	// snapshot taken first -- turns it off here and calls ArchiveTable.Rotate themselves.
	ArchiveRotateDisabled bool
	// ArchiveIndexBackfillBytes is the per-pass budget, in stored segment bytes, for carrying
	// the current index set to sealed archive segments that lack postings for an indexed
	// attribute -- the ones beyond ArchiveConfig.IndexBackfillBytes when the index was added.
	// Each pass rebuilds the newest such segments first until it has re-read this much, so a
	// large history is covered over several passes without one long stall. 0 takes
	// DefaultArchiveIndexBackfillBytes; a negative value turns the backfill off.
	ArchiveIndexBackfillBytes int64
	// ArchiveHotRefreshDisabled turns off the demand-driven hot-set refresh on ARCHIVE
	// tables, which otherwise runs on the same HotTopN the mutable tables use.
	//
	// It shares HotTopN rather than taking a threshold of its own because, alone among the
	// archive's self-tuning passes, it reads nothing back: the hot header is per record and
	// self-describing, so a refreshed set governs records written from now on and leaves
	// every sealed segment untouched and correct. The pass costs one bounded sample plus a
	// pointer swap -- nothing like the whole-history backfill behind ArchiveIndexMinDemand
	// or ArchiveSchemaScanHotTopN -- so there is nothing for an operator to weigh before
	// letting it run, and an off-by-default knob would leave it dark for the same reason
	// those two were.
	//
	// The escape hatch exists because the refresh does change what future segments look
	// like, and an archive whose hot set was chosen deliberately should be able to keep it.
	ArchiveHotRefreshDisabled bool
	// ArchiveIndexMinDemand is the minimum observed query demand for the auto-tuner to add
	// an index to an ARCHIVE table; 0 leaves archive index auto-tune off.
	//
	// It is separate from MinIndexDemand, and off by default, because the two costs are not
	// comparable. Indexing a mutable table is bounded by its live size; indexing an archive
	// is paid for by decompressing history, so the threshold wants to be set against
	// ArchiveConfig.IndexBackfillBytes -- how far back a new index is actually carried --
	// rather than inherited from the mutable side.
	//
	// Auto-DROP is deliberately not offered for archives at any setting. Dropping an index
	// is nearly free and re-adding it costs a backfill, so a wrong drop is expensive and
	// asymmetric in the direction that punishes acting on a weak signal.
	ArchiveIndexMinDemand int64
	// ArchiveSchemaScanHotTopN, when > 0, builds the per-segment columnar accelerator on
	// ARCHIVE tables too, keeping the topN most query-read numeric fields uncompressed.
	//
	// Separate from SchemaScanHotTopN, and off by default, for the same reason
	// ArchiveIndexMinDemand is: the first build reads every sealed record of the whole
	// history once, so turning it on is a deliberate decision about an existing deployment
	// rather than something an upgrade should start doing. Once built it is cheap -- an
	// archive's segments are immutable, so a block is never invalidated, and later passes
	// only cover newly-sealed segments.
	//
	// The payoff is confined to what the accelerator serves: single-int-field COUNT
	// comparisons (25-65x on a 50k-record archive). Other aggregates -- MAX, SUM, GROUP BY
	// -- take their existing paths and are unaffected.
	ArchiveSchemaScanHotTopN int
	// SchemaScanHotTopN, when > 0, builds/refreshes the per-segment adschema columnar
	// accelerator (used by CountConstraint's fast path) keeping the topN most query-read
	// numeric fields uncompressed. The first pass chooses a stable schema and hot set from
	// the current sample and demand; later passes only extend coverage to newly-sealed
	// segments, so a table opts into the columnar count path once maintenance runs and stays
	// covered as it grows. 0 leaves it off.
	SchemaScanHotTopN int

	// ArchiveRetrainInterval, when > 0, retrains an ARCHIVE's compression dictionary that
	// often. 0 leaves it off. DefaultArchiveRetrainInterval is the recommended cadence.
	//
	// Off by default, for the reason ArchiveIndexMinDemand and ArchiveSchemaScanHotTopN are:
	// the FIRST retrain of an inline archive reads and rewrites its whole history once, so
	// turning it on is a decision about an existing deployment rather than something an
	// upgrade should start doing. Steady-state cost is far lower -- see
	// DefaultArchiveRetrainInterval for what each half of that is.
	//
	// An interval rather than a per-pass bool (unlike Retrain, above) because the cadence
	// that suits it is much longer than the one that suits the rest of the pass, and because
	// the caller would otherwise have to keep the clock itself.
	//
	// The clock is per-process. It starts when the archive is opened, not at the zero time,
	// so a daemon that restarts often defers a retrain rather than running one on every
	// start; nothing persists the last-retrain time (CodecStats.LastRetrain is per-process
	// too), so a restart resets it -- in the conservative direction.
	ArchiveRetrainInterval time.Duration

	// OnRetrain, when set, is called after ArchiveTable.Maintain attempts a dictionary
	// retrain -- only when one was attempted, and whether or not it succeeded. Training
	// legitimately declines on small or homogeneous data, which is a Result with Err set and
	// the existing dictionary left in place, not a failure of the pass.
	OnRetrain func(RetrainResult) `json:"-"`

	// OnRotate, when set, is called after ArchiveTable.Maintain rotates the archive to its
	// retention bounds -- only when the rotation dropped something or failed, so the usual
	// pass over an archive that is within its bounds (or has none configured) says nothing.
	//
	// Called synchronously, on the maintenance goroutine, holding nothing the callback can
	// reach. It must not call back into the table.
	OnRotate func(RotateResult) `json:"-"`

	// OnAutoTune, when set, is called with what the pass's auto-tuner changed -- once per
	// pass, only when it changed something. Nil (the default) is the previous behaviour.
	//
	// It exists because the pass applied its decisions and then discarded them: Maintain
	// read res.Changed to decide whether to persist and dropped the Changes. An index
	// appearing or vanishing is the most consequential thing a background pass does -- a
	// composite moves the index generation, so every sealed segment is rebuilt to carry it
	// -- and nothing above this could say that it had happened, let alone which attribute
	// or why. A caller that wants to log it had no way to be told.
	//
	// Called synchronously, on the maintenance goroutine, holding nothing the callback can
	// reach. It must not call back into the table.
	OnAutoTune func(AutoTuneResult) `json:"-"`
}

MaintainOptions configures one maintenance pass (DB.Maintain).

func (MaintainOptions) ArchiveIndexBackfillBudget added in v0.33.0

func (o MaintainOptions) ArchiveIndexBackfillBudget() (int64, bool)

ArchiveIndexBackfillBudget resolves ArchiveIndexBackfillBytes: the byte budget for one pass, and false when the backfill is turned off.

func (MaintainOptions) ArchiveRetrainEvery added in v0.38.3

func (o MaintainOptions) ArchiveRetrainEvery() (time.Duration, bool)

ArchiveRetrainEvery resolves ArchiveRetrainInterval: how often to retrain, and false when retraining is off.

type MatchExplain

type MatchExplain = collections.MatchExplain

Diagnostic and management types, re-exported from collections for callers that only import db.

type MergeOptions added in v0.24.1

type MergeOptions = collections.MergeOptions

type NDVAccuracy added in v0.38.1

type NDVAccuracy = collections.NDVAccuracy

Diagnostic and management types, re-exported from collections for callers that only import db.

type NDVErr added in v0.38.1

type NDVErr = collections.NDVErr

Diagnostic and management types, re-exported from collections for callers that only import db.

type NDVEstimate added in v0.38.1

type NDVEstimate = collections.NDVEstimate

Diagnostic and management types, re-exported from collections for callers that only import db.

type NumStats added in v0.25.2

type NumStats = collections.NumStats

NumStats is one columnar pass's numeric aggregate inputs (see collections.NumStats).

type NumValue added in v0.37.0

type NumValue = collections.NumValue

NumValue is one value read from a numeric column (see collections.NumValue).

type NumValueKind added in v0.37.0

type NumValueKind = collections.NumValueKind

NumValueKind says how a column value was stored (see collections.NumValueKind).

type NumValueScan added in v0.37.0

type NumValueScan func(attr string, newSink func() func(NumValue), merge func()) bool

NumValueScan feeds the numeric values of attr, over the records a request matched, to one sink per worker. It reports false when the columnar path cannot serve the request and the caller must scan.

newSink is called once per worker and merge once afterwards, both on the CALLING goroutine, and each sink is fed by exactly one goroutine -- so an accumulator needs no synchronization of its own. A pass that declines calls merge not at all.

type OldAdText added in v0.8.0

type OldAdText struct {
	Key  string
	Text string
}

OldAdText is one keyed ad in old-ClassAd wire text, for UpdateOldBatch.

type OpStat added in v0.11.0

type OpStat = collections.OpStat

OpStat is one operation's cumulative call count and total wall-nanoseconds.

type OpStats added in v0.11.0

type OpStats struct {
	collections.OpStats
	SnapshotLock OpStat `json:"snapshotLock"`
	// SnapshotLockWait is the time committers spent BLOCKED on the shared snapshot lock (stalled
	// behind an exclusive Truncate/Restore) -- the waiters' side of SnapshotLock's holder time.
	SnapshotLockWait OpStat `json:"snapshotLockWait"`
}

OpStats is a snapshot of the store's operational timing counters -- where callers spent time blocked in, or holding, each stall point (see collections.OpStats) -- plus the DB-wide snapshot lock's exclusive-hold time (Truncate/Restore). Every value is a monotonic cumulative total; a scraper derives rate and mean latency from deltas.

type OrderCursor

type OrderCursor = collections.OrderCursor

OrderSpec, SortKey, OrderedAd, OrderCursor configure and drive maintained ordered indexes (the schedd priority-queue / resource-request-list pattern). Re-exported from collections so callers of db need not import it.

type OrderSpec

type OrderSpec = collections.OrderSpec

OrderSpec, SortKey, OrderedAd, OrderCursor configure and drive maintained ordered indexes (the schedd priority-queue / resource-request-list pattern). Re-exported from collections so callers of db need not import it.

type OrderedAd

type OrderedAd = collections.OrderedAd

OrderSpec, SortKey, OrderedAd, OrderCursor configure and drive maintained ordered indexes (the schedd priority-queue / resource-request-list pattern). Re-exported from collections so callers of db need not import it.

type ProbeExplain added in v0.21.2

type ProbeExplain = collections.ProbeExplain

Diagnostic and management types, re-exported from collections for callers that only import db.

type QueryExplain

type QueryExplain = collections.QueryExplain

Diagnostic and management types, re-exported from collections for callers that only import db.

type RankedMatch

type RankedMatch = collections.RankedMatch

RankedMatch is a matched ad with the job's Rank of it.

type Retention added in v0.19.0

type Retention = collections.Retention

Diagnostic and management types, re-exported from collections for callers that only import db.

type RetrainResult added in v0.38.3

type RetrainResult struct {
	// DictBytes is the size of the dictionary trained, 0 when training declined.
	DictBytes int
	// Took is how long the attempt ran, declined or not.
	Took time.Duration
	// Err is why training declined, nil on success. The archive keeps the dictionary it
	// already had and tries again after the interval.
	Err error
}

RetrainResult is what one dictionary retrain did (MaintainOptions.OnRetrain).

type RotateResult added in v0.38.3

type RotateResult struct {
	// Dropped is how many whole sealed segments fell outside retention and were released.
	Dropped int
	// Took is how long the rotation ran.
	Took time.Duration
	// Err is a failure to release a dropped segment's file (munmap/unlink), nil on success.
	// The segments are already out of service when it happens, so it leaks a file rather
	// than corrupting one, and it does not stop the rest of the maintenance pass.
	Err error
}

RotateResult is what one retention rotation did (MaintainOptions.OnRotate).

type SchemaFieldFit added in v0.25.2

type SchemaFieldFit = collections.SchemaFieldFit

Diagnostic and management types, re-exported from collections for callers that only import db.

type SchemaRequireSkip added in v0.32.0

type SchemaRequireSkip = collections.SchemaRequireSkip

SchemaRequireSkip is one required attribute the last schema derivation refused, with why.

type SchemaScanField added in v0.25.2

type SchemaScanField = collections.SchemaScanField

Diagnostic and management types, re-exported from collections for callers that only import db.

type SchemaScanGroup added in v0.29.13

type SchemaScanGroup = collections.SchemaScanGroup

Diagnostic and management types, re-exported from collections for callers that only import db.

type SchemaScanInfo added in v0.23.1

type SchemaScanInfo = collections.SchemaScanInfo

Diagnostic and management types, re-exported from collections for callers that only import db.

type SeqCursor added in v0.29.5

type SeqCursor = collections.SeqCursor

SeqCursor and SeqPage re-export the collections types so a caller of the DB API does not have to import the storage layer to paginate.

type SeqPage added in v0.29.5

type SeqPage = collections.SeqPage

SeqCursor and SeqPage re-export the collections types so a caller of the DB API does not have to import the storage layer to paginate.

type SidecarSizes added in v0.19.0

type SidecarSizes = collections.SidecarSizes

Diagnostic and management types, re-exported from collections for callers that only import db.

type SnapshotKeys added in v0.7.0

type SnapshotKeys struct {
	SnapshotKey []byte
	BackupKey   []byte
	MasterKey   []byte
	PoolKeys    []KEK
}

SnapshotKeys carries any ONE level of the key hierarchy sufficient to decrypt a snapshot. A snapshot is decryptable with, in decreasing specificity: the per-backup SnapshotKey (seals the body directly), the BackupKey (unwraps the snapshot key), the MasterKey (derives the backup key), or PoolKeys (open the embedded master envelope). Restore uses the most specific field provided; all are optional (plain Restore uses the DB's own pool keys). This lets an operator escrow whichever level fits their recovery model and decrypt a backup even without the original pool keys or a running daemon.

type SortKey

type SortKey = collections.SortKey

OrderSpec, SortKey, OrderedAd, OrderCursor configure and drive maintained ordered indexes (the schedd priority-queue / resource-request-list pattern). Re-exported from collections so callers of db need not import it.

type Stats

type Stats = collections.Stats

Diagnostic and management types, re-exported from collections for callers that only import db.

type TableOptions added in v0.10.1

type TableOptions struct {
	// InMemory makes the table's data live only in RAM even in a persistent
	// catalog: no <dir>/tables/<name> directory is created, so the table is not
	// recovered across restarts (it reappears empty). Useful for high-churn,
	// reconstructible data (e.g. frequently-replaced ads) that is not worth the
	// disk I/O of persistence. In an already-in-memory catalog it is a no-op.
	InMemory bool
}

TableOptions tunes how a table is created. The zero value creates a normal table (persistent when the catalog has a directory).

type TopKOrderTerm added in v0.42.0

type TopKOrderTerm struct {
	Attr string
	Desc bool
}

TopKOrderTerm is one term of the ordering, most significant first.

func OrderOne added in v0.42.0

func OrderOne(attr string, desc bool) []TopKOrderTerm

OrderOne is the spec for the common single-term ordering.

type TopKRowsGroup added in v0.42.0

type TopKRowsGroup struct {
	Values []classad.Value
	Rows   [][]classad.Value
}

TopKRowsGroup is one group's answer: the group columns' values and its best rows, ordered.

type TopKRowsSource added in v0.42.0

type TopKRowsSource interface {
	QueryProjectStats(constraint string, attrs []string, stats *collections.ScanStats) (iter.Seq[[]classad.Value], error)
	QueryProjectStatsRedacted(constraint string, attrs []string, stats *collections.ScanStats) (iter.Seq[[]classad.Value], error)
	QueryRawWireRefsStats(constraint string, projection []string, redact bool, stats *collections.ScanStats) (iter.Seq[[]byte], error)
}

TopKRowsSource is a table the top-k rungs can read: the mutable DB and an ArchiveTable both expose exactly these two scans with these signatures, which is why one implementation serves both rather than the archive growing a parallel copy of it.

History is where "the top N per group" mostly points -- the five longest jobs each owner ran last month is a question about completed jobs, not queued ones -- so an archive that could not answer it was the bigger gap of the two.

type TopKRowsSpec added in v0.42.0

type TopKRowsSpec struct {
	// GroupCols are the PARTITION BY columns. Empty means one partition holding every row,
	// which is the plain "top k overall" and is served here too.
	GroupCols []GroupCol
	// OrderBy is the ordering, most significant term first. It must name at least one term.
	OrderBy []TopKOrderTerm
	K       int
	// Attrs are the attributes each returned row carries, as QueryProject's are. The group
	// columns and the ordering attributes are NOT added for you: a caller that wants them in
	// the rows asks for them, exactly as it would of a projection.
	Attrs []string
}

TopKRowsSpec is one "k rows per group" request.

Every ordering attribute must be numeric in the rows that are to be ranked. A row where ANY of them is absent, non-numeric or a NaN is NOT ranked and does not appear -- the same rule the numeric aggregates follow, and the only one under which "the newest five" means anything. Such a row still belongs to its group for any other purpose; this call simply has nothing to order it by.

WHY THE ORDERING IS A LIST. "The top 5" is only a well-defined SET of rows when the ordering is total over them. Where it ties across the K-th place, which rows come back is a choice, and this makes a different choice than a client sorting the same rows does -- measured, not supposed: see TestTopKTieBreakingAgainstAClientRanking. A caller that needs the answer to be determined writes an ordering that cannot tie, which for a job table is QDate then ClusterId, both already in the ad. A single term remains the common spelling and is what a caller writes when it does not care which of several equal rows it gets.

type TopKRowsWireGroup added in v0.42.0

type TopKRowsWireGroup struct {
	Values []classad.Value
	Rows   [][]byte
}

TopKRowsWireGroup is one group's answer with its rows left in wire form.

Values names the group exactly as TopKRowsGroup.Values does. Rows are self-contained wire ads, best first, each owned by the caller.

type Txn

type Txn struct {
	// contains filtered or unexported fields
}

Txn is an independent optimistic transaction. Operations are buffered and applied at Commit under snapshot-isolation OCC. A *Txn is not safe for concurrent use by multiple goroutines; independent transactions are.

func (*Txn) Abort

func (t *Txn) Abort()

Abort discards the transaction's buffered operations. Nothing is written.

func (*Txn) Commit

func (t *Txn) Commit() error

Commit applies the buffered operations. It returns a *ConflictError if any key was modified by another committer since this transaction's snapshot (the non-conflicted operations still committed), or nil on full success.

func (*Txn) CommitNondurable

func (t *Txn) CommitNondurable() error

CommitNondurable is Commit that defers the disk durability sync (classad_log.h CommitNondurableTransaction): the writes are visible immediately but their flush is batched to a later durable commit. On an in-memory DB it is identical to Commit.

func (*Txn) DeleteAttribute

func (t *Txn) DeleteAttribute(key, name string)

DeleteAttribute removes one attribute of key (classad_log.h LogDeleteAttribute). A no-op if key or the attribute is absent.

func (*Txn) DestroyClassAd

func (t *Txn) DestroyClassAd(key string)

DestroyClassAd removes key (classad_log.h LogDestroyClassAd).

func (*Txn) Has added in v0.29.11

func (t *Txn) Has(key string) bool

Has reports whether key exists as the transaction sees it (its own buffered writes over the snapshot, exactly as LookupClassAd does) without decoding the stored record. Use it wherever only presence matters: LookupClassAd's decode of a wide ad dominates the cost of an ingest that asks the question once per operation.

func (*Txn) KeysWhere added in v0.24.0

func (t *Txn) KeysWhere(constraint string) (iter.Seq[string], error)

KeysWhere returns the storage keys of the rows matching the constraint as the transaction sees them (DB.KeysWhere with the transaction's writes overlaid). It is what lets an UPDATE or DELETE inside a transaction address a row the transaction itself created. Errors only on a malformed constraint.

func (*Txn) LookupAttr

func (t *Txn) LookupAttr(key, name string) (string, bool)

LookupAttr returns the unparsed expression of one attribute as the transaction sees it (classad_log.h LookupInTransaction), or ("", false).

func (*Txn) LookupClassAd

func (t *Txn) LookupClassAd(key string) (*classad.ClassAd, bool)

LookupClassAd returns key's ad as the transaction sees it: its own buffered writes (read-your-writes) merged over the snapshot (classad_log.h Lookup + the LookupInTransaction overlay in one call).

func (*Txn) NewClassAd

func (t *Txn) NewClassAd(key string, ad *classad.ClassAd)

NewClassAd stores ad under key (classad_log.h LogNewClassAd). An existing ad at key is replaced.

func (*Txn) NewClassAdOld added in v0.25.1

func (t *Txn) NewClassAdOld(key, text string) bool

NewClassAdOld stores an ad supplied as old-ClassAd text under key, encoding it straight to the stored wire form instead of parsing it into a ClassAd for the commit to re-encode. The transaction is unaffected: the write is buffered, conflict-checked and committed exactly as NewClassAd's is.

It reports whether the wire-native path was taken. False means the caller must parse the text and use NewClassAd -- an encrypted store seals its values and the streaming encoder does not seal, and a few ad shapes (a repeated attribute name, an escape the fast lexer would read differently) defer to the reference parser by design.

func (*Txn) Query added in v0.24.0

func (t *Txn) Query(constraint string) (iter.Seq[*classad.ClassAd], error)

Query returns the ads matching the constraint as the transaction sees them: the committed rows with the transaction's own buffered writes overlaid, so a query inside a transaction observes work the transaction has not committed yet. DB.Query cannot -- it reads the committed store -- which is why a caller that has staged writes and then wants to read them back must go through the transaction.

Isolation is snapshot plus read-your-writes: the committed half is read at the transaction's own snapshot sequence, the same one Get reads at, so a scan and a point lookup in one transaction agree and a concurrent commit is invisible to both. The cost is a full scan -- reading at a past sequence and overlaying by key both need the per-record walk the indexed query path skips. See collections.Txn.Query. Errors only on a malformed constraint.

func (*Txn) SetAttribute

func (t *Txn) SetAttribute(key, name, expr string) error

SetAttribute sets one attribute of key to the expression parsed from expr (classad_log.h LogSetAttribute) -- a read-modify-write within the transaction, so it composes with the transaction's own earlier writes to key. The ad is created if absent.

expr is parsed with OLD-ClassAd semantics, because that is what LogSetAttribute does: it calls ParseClassAdRvalExpr, which is a ClassAdParser with SetOldClassAd(true) (compat_classad_util.cpp). The difference is string literals -- old syntax gives them no escape interpretation at all, so a job_queue.log value like regexp("host\.example\.org") keeps its backslashes and means what its author wrote. Parsing it as new syntax silently deleted them (there, \. is the undefined escape that decays to .), turning that regex into one matching any character.

type UnappliedError added in v0.30.2

type UnappliedError struct {
	Keys []string
	// Reasons is parallel to Keys: why each one could not be composed ("delta-no-base",
	// "delta-chain-broken", "too-deep: ..."). Carried so a caller can log what happened rather
	// than only that something did.
	Reasons []string
	// Errs is parallel to Keys: the error behind each one, or nil where the reason alone says it.
	// Unwrap exposes the non-nil ones to errors.Is and errors.As.
	Errs []error
}

UnappliedError reports writes that could not be composed at all, as distinct from a conflict. Re-applying the identical write cannot succeed -- the key is present but the record behind it will not come back -- so a caller should record these and make progress rather than retry. Treating them as conflicts made a tailer rewind and re-apply forever, escalating to full log replays that changed nothing. See collections.CommitResult.Unapplied.

A write refused because a value nests too deeply to store is one of these: errors.Is(err, ErrTooDeep) reports it, and its reason names the attribute.

func (*UnappliedError) Error added in v0.30.2

func (e *UnappliedError) Error() string

func (*UnappliedError) Unwrap added in v0.34.3

func (e *UnappliedError) Unwrap() []error

Unwrap returns the errors behind the unapplied keys, so errors.Is(err, ErrTooDeep) finds a refused write among them.

type View added in v0.12.0

type View struct {
	// contains filtered or unexported fields
}

View is a live materialized view: its spec, the per-key contribution store, per-group accumulators, and an in-memory backing DB holding one rendered ad per group (so the view is queried like a table). All mutable state is guarded by mu.

func (*View) AggregateColsStats added in v0.34.3

func (v *View) AggregateColsStats(constraint string, groupCols []GroupCol, aggs []AggSpec, stats *collections.ScanStats, stop func() bool) ([]AggRow, error)

AggregateColsStats reduces an aggregate over the WHOLE view: the live backing and, for a continuous aggregate that has sealed, its archive too.

Unlike the top-K above, this does NOT reduce each half and combine the results, because not every aggregate can be combined from finished ones. SUM, MIN, MAX and COUNT can; AVG cannot (an average of averages is not the average), and COUNT(DISTINCT) cannot at all, since a count has thrown away the values it would have to compare. Special-casing the mergeable ones would leave the other two either wrong or refused.

So the union is reduced ONCE, in a single pass over both halves' projected rows, through the same reducer the scanning path uses. That gives up the columnar fast paths, and the reason it is still the right trade is that only a continuous aggregate seals: its materialized rows are one per bucket per series and bounded by the view's cardinality cap, so the scan is small by construction. A gauge view never reaches here -- it has no archive, and falls through to the backing's own ladder with every fast path intact.

func (*View) Backing added in v0.12.0

func (v *View) Backing() *DB

Backing returns the in-memory table holding the materialized rows (read-only in intent).

func (*View) Cursor added in v0.12.0

func (v *View) Cursor() []byte

Cursor returns the last durable resume cursor (for persistence).

func (*View) LateDrops added in v0.15.0

func (v *View) LateDrops() int64

LateDrops reports how many base rows were dropped because their time bucket was already sealed (out-of-window late data).

func (*View) Seal added in v0.16.0

func (v *View) Seal(now int64)

Seal appends and evicts every bucket whose window has closed as of now (unix seconds) into the archive, advancing the watermark. It is what the periodic tick invokes; exported so an operator (or a test) can force sealing at a chosen instant.

func (*View) SealedArchive added in v0.34.3

func (v *View) SealedArchive() (*ArchiveTable, bool)

SealedArchive returns the continuous aggregate's sealed history as an archive table, and whether it has one (a gauge view and an in-memory catalog do not).

SealedQuery streams its rows, which is all a whole-ad read needs. A projected read, a top-K and an aggregate each need to run their OWN operation over the sealed half rather than re-derive it from rows, which is what this exposes: the sealed half is an ordinary archive table and answers the same reads as one.

The caller must treat the two halves as one table. A view that seals holds its rows in two places -- the live backing and this -- and a read that consults only one of them is silently short, not merely stale.

func (*View) SealedQuery added in v0.16.0

func (v *View) SealedQuery(constraint string) (iter.Seq[*classad.ClassAd], error)

SealedQuery streams this continuous aggregate's sealed (archived) rows matching the constraint. It returns an empty sequence for a gauge view or an in-memory continuous aggregate (no archive). Reads union this with the live backing so a view read returns the full series, not just the unsealed buckets.

func (*View) SeriesCount added in v0.12.0

func (v *View) SeriesCount() int

SeriesCount returns the current number of groups (Prometheus series).

func (*View) Spec added in v0.12.0

func (v *View) Spec() ViewSpec

Spec returns the view's definition.

func (*View) State added in v0.12.0

func (v *View) State() (ViewState, error)

State reports the view's lifecycle state and any failure error.

func (*View) TopK added in v0.34.3

func (v *View) TopK(constraint string, attrs []string, orderAttr string, desc bool, k int) ([][]classad.Value, error)

TopK answers a top-K over the WHOLE view: the live backing and, for a continuous aggregate that has sealed, its archive too.

It is not a top-K of the backing with the archive's rows appended. The k best of a union are within the union of each half's k best, so each half runs its own top-K -- keeping the columnar cutoff scan that makes a top-K cheaper than a sort -- and the two results are merged through the same heap the single-table path uses. Reusing that heap is the point: a merge with its own comparator would order the union by one rule and a single half by another, and the difference would show only once a view started sealing.

func (*View) TopKStats added in v0.34.3

func (v *View) TopKStats(constraint string, attrs []string, orderAttr string, desc bool, k int) ([][]classad.Value, collections.ScanStats, error)

TopKStats is TopK that also returns the scan breakdown, summed over both halves.

type ViewAggFunc added in v0.12.0

type ViewAggFunc string

ViewAggFunc is a delta-maintainable aggregate.

const (
	ViewCount ViewAggFunc = "count"
	ViewSum   ViewAggFunc = "sum"
	ViewAvg   ViewAggFunc = "avg"
)

type ViewGroupCol added in v0.12.0

type ViewGroupCol struct {
	Attr        string `json:"attr"`
	Alias       string `json:"alias"`
	BucketWidth int64  `json:"bucketWidth,omitempty"` // seconds; 0 = raw value
}

ViewGroupCol is one GROUP BY column: the base-table attribute and the alias it is stored under in the materialized rows (e.g. attr "Owner", alias "label_owner"). When BucketWidth > 0, the column is time-bucketed: the (numeric, unix-seconds) attribute is floored to epoch-aligned buckets of that width, turning the view into a time series -- each interval is a distinct, permanent group rather than an overwritten gauge. This is the same shape as dbrpc.GroupCol.

type ViewMetric added in v0.12.0

type ViewMetric struct {
	Func  ViewAggFunc `json:"func"`
	Arg   string      `json:"arg"`
	Alias string      `json:"alias"`
}

ViewMetric is one aggregate output: the function, its argument attribute ("*" for COUNT(*)), and the alias it is stored under (e.g. "metric_jobs").

type ViewSpec added in v0.12.0

type ViewSpec struct {
	BaseTable string         `json:"baseTable"`
	Groups    []ViewGroupCol `json:"groups"`
	Metrics   []ViewMetric   `json:"metrics"`
	// Cardinality is the hard maximum number of groups (output series). Exceeding it fails
	// the build (CreateView error) or, at runtime, moves the view to the failed state.
	Cardinality int `json:"cardinality"`
	// SelectText is the original SELECT for display (.views); not used for execution.
	SelectText string `json:"selectText"`

	// Grace is seconds to wait after a time bucket's window closes before sealing it
	// (so late-arriving base rows still land). Only meaningful for a continuous aggregate
	// (a spec with a time-bucketed group column). 0 seals as soon as the window closes.
	Grace int64 `json:"grace,omitempty"`
	// Retention is seconds of sealed history to keep in the continuous aggregate's archive;
	// 0 keeps all.
	Retention int64 `json:"retention,omitempty"`
}

ViewSpec is a materialized view's persisted definition.

func (ViewSpec) IsContinuous added in v0.15.0

func (s ViewSpec) IsContinuous() bool

IsContinuous reports whether the view is a continuous aggregate (has a time bucket).

func (ViewSpec) Validate added in v0.12.0

func (s ViewSpec) Validate() error

Validate checks a spec is well-formed and delta-maintainable.

type ViewState added in v0.12.0

type ViewState uint8

ViewState is a view's lifecycle state.

const (
	// ViewBuilding: the initial catch-up is in progress.
	ViewBuilding ViewState = iota
	// ViewActive: built and maintaining live.
	ViewActive
	// ViewStale: definition loaded but the base table is absent; will bind on the next
	// activation once the base table exists.
	ViewStale
	// ViewFailed: a runtime error (e.g. cardinality exceeded); the updater has stopped.
	ViewFailed
)

func (ViewState) String added in v0.12.0

func (s ViewState) String() string

type WatchEvent

type WatchEvent struct {
	Kind   WatchKind
	Key    string
	Ad     *classad.ClassAd // nil for a delete
	Cursor []byte
	// Stamp names this change in a clock shared by every shard, so two readers of the same
	// DB give the same change the same number. A server that owes its clients one
	// comparable version -- a Kubernetes resourceVersion, say -- takes it from here instead
	// of keeping a counter of its own, which would restart with the process and differ
	// between replicas.
	//
	// It is not a position: resuming is Cursor's job. Live events carry it; catch-up events
	// do not, and WatchSynced carries the watermark instead. See collections.WatchEvent for
	// what it does and does not order.
	Stamp uint64
	// Reason says why a WatchReset was issued; WatchResetNone on every other kind. A
	// consumer that treats every Reset alike cannot tell records it LOST (retention) from
	// records it is merely being re-sent (a restart, or its first subscribe).
	Reason WatchResetReason
}

WatchEvent is one change to the log. Cursor resumes a watch just after this event (survives reconnect / restart, so no change is missed).

type WatchKind

type WatchKind uint8

WatchKind classifies a watch event.

const (
	// WatchUpsert: key was added or updated; Ad holds its new value.
	WatchUpsert WatchKind = iota
	// WatchDelete: key was removed; Ad is nil.
	WatchDelete
	// WatchReset: discard prior state and rebuild from the upserts that follow (the
	// initial full replay, or a re-sync after the cursor fell out of retention). Key
	// and Ad are empty.
	WatchReset
	// WatchSynced: end of the initial catch-up/replay; the watcher is now live. Cursor is
	// a durable resume point (and, after a Reset, the point at which a shadow rebuild goes
	// live). Key and Ad are empty.
	WatchSynced
	// WatchResync: the live stream fell behind; reconnect with the last persisted cursor
	// (which re-enters catch-up). Key and Ad are empty; no state is implied.
	WatchResync
)

type WatchResetReason added in v0.40.0

type WatchResetReason = collections.WatchResetReason

WatchResetReason says why a WatchReset was issued. It mirrors collections.WatchResetReason 1:1; see there for what each cause means and what an operator should do about it.

type Watcher

type Watcher struct {
	// contains filtered or unexported fields
}

Watcher is a watch whose readiness is signalled on a file descriptor -- for the C (DaemonCore) side, which registers the fd in its poll loop and, on wakeup, drains events with Next. Go writes a single wakeup byte when the queue goes non-empty (coalesced: the reader drains fully per wakeup). The fd is a pipe or eventfd the caller created and owns; Watcher never closes it.

func NewWatcher

func NewWatcher(db *DB, notifyFD int, cursor []byte) (*Watcher, error)

NewWatcher starts watching db, signalling notifyFD when events are queued. cursor nil starts from now.

func (*Watcher) Next

func (w *Watcher) Next() (WatchEvent, bool)

Next dequeues one event without blocking. ok is false when the queue is empty; the caller (having been woken via the fd) drains until ok is false.

func (*Watcher) Stop

func (w *Watcher) Stop()

Stop ends the watch. The notify fd is not closed (the caller owns it).

Directories

Path Synopsis
Package main builds the embedded ClassAd log as a C archive: it exports C symbols (cadb_*) mirroring HTCondor's classad_log.h, so a C++ interface (built on libcondor_utils) can sit on top.
Package main builds the embedded ClassAd log as a C archive: it exports C symbols (cadb_*) mirroring HTCondor's classad_log.h, so a C++ interface (built on libcondor_utils) can sit on top.
Package replicate is the transport-neutral core for replicating a db table/archive's change stream into another store: an in-memory Change (a db.WatchEvent enriched with replication metadata) and an idempotent Sink that applies Changes into a *db.DB or *db.ArchiveTable.
Package replicate is the transport-neutral core for replicating a db table/archive's change stream into another store: an in-memory Change (a db.WatchEvent enriched with replication metadata) and an idempotent Sink that applies Changes into a *db.DB or *db.ArchiveTable.
Package scriptagg registers an aggregate written in ClassAd Script, so a GROUP BY can name it exactly as it names SUM.
Package scriptagg registers an aggregate written in ClassAd Script, so a GROUP BY can name it exactly as it names SUM.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL