scriptagg

package
v0.41.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 16, 2026 License: Apache-2.0 Imports: 6 Imported by: 0

Documentation

Overview

Package scriptagg registers an aggregate written in ClassAd Script, so a GROUP BY can name it exactly as it names SUM.

It is the aggregate counterpart of collections/scriptudf, which registers a script function as a scalar user-defined function, and the two are deliberately the same shape: a module exports functions by name, registration picks the ones it needs, and what the engine ends up holding is the same kind of thing a Go or wasm implementation would have produced.

AN AGGREGATE IS FOUR PURE FUNCTIONS THREADING A STATE VALUE, and that is the whole of the language support it needs. There is no accumulator object in the script, no mutation and no handle:

def init():            the identity -- the answer for a group that contributed no rows
def add(state, v):     the state with one row's value folded in
def merge(a, b):       two partial states combined
def result(state):     the group's answer

The state is an ordinary ClassAd value and the Go side holds exactly one of them per group. A scalar state -- a running sum, a maximum, a count -- is a number; anything needing two numbers at once is a LIST, which a module function builds natively because it has no ad for a list literal to be lazy against (see vm's OpMakeList). That is why AVG is expressible here at all: its state is a sum and a count, and a scalar cannot carry both.

THE LAWS ARE THE SCRIPT AUTHOR'S TO KEEP. db.Accumulator states them -- merge associative AND commutative, result repeatable and non-mutating -- and nothing here can check them, because the four functions are the author's code. db.CheckAggregate tests one against them over a sample, and an aggregate written here should be run through it in the author's own tests before it is registered. The failure it prevents is the bad one: the columnar pass fans out across segment windows and which worker draws which segment is a race, so an order-sensitive merge is right in every test its author writes and wrong in production, plausibly and only sometimes.

WHY THIS LIVES UNDER db AND NOT BESIDE scriptudf. An accumulator cannot be reached through a structurally-matching interface the way the scalar registry can: db.Accumulator's Merge takes a db.Accumulator, and Go satisfies interfaces by type identity, so a copy declared elsewhere would be a different type db would not accept. The import therefore has to be real, and collections -- where vm lives -- is the module db depends on, so the edge can only run this way round. wasmudf reaches the same conclusion for the same reason.

Example

Example registers an aggregate written in ClassAd Script and runs a GROUP BY with it, exactly as a query naming SUM would run.

The state is a list because the answer needs two numbers at once -- a sum and a count -- and a mean cannot be carried in one. The four functions are pure and the state is threaded through them; nothing is mutated and nothing is held between calls.

package main

import (
	"fmt"
	"sort"

	"github.com/PelicanPlatform/classad/classad"
	"github.com/PelicanPlatform/classad/collections/vm"
	"github.com/PelicanPlatform/classad/db"
	"github.com/PelicanPlatform/classad/db/scriptagg"
)

func main() {
	mod, err := vm.ParseScriptModule(`
def init():
    return {0, 0}

def add(s, v):
    return isReal(v) || isInteger(v) ? {s[0] + v, s[1] + 1} : s

def merge(a, b):
    return {a[0] + b[0], a[1] + b[1]}

def result(s):
    return s[1] > 0 ? real(s[0]) / s[1] : undefined
`)
	if err != nil {
		panic(err)
	}
	agg, err := scriptagg.Register("exampleMean", mod, scriptagg.Spec{})
	if err != nil {
		panic(err)
	}

	// The laws the reduction relies on are the script's to keep, so they are tested rather than
	// assumed. The sample holds what a row can really contain, not only what the aggregate was
	// written for.
	fmt.Println("laws:", db.CheckAggregate(agg, []classad.Value{
		classad.NewIntValue(1024),
		classad.NewIntValue(2048),
		classad.NewRealValue(4096.5),
		classad.NewStringValue("lots"),
		classad.NewUndefinedValue(),
	}))

	d, err := db.OpenConfig(db.Config{})
	if err != nil {
		panic(err)
	}
	defer d.Close()

	tx := d.Begin()
	for i, src := range []string{
		`[ Pool = "cm1"; Memory = 1024 ]`,
		`[ Pool = "cm1"; Memory = 2048 ]`,
		`[ Pool = "cm1"; Memory = 4096 ]`,
		`[ Pool = "cm2"; Memory = 8192 ]`,
		`[ Pool = "cm2"; Memory = "lots" ]`, // not a number: `add` skips it
	} {
		ad, perr := classad.Parse(src)
		if perr != nil {
			panic(perr)
		}
		tx.NewClassAd(fmt.Sprintf("k%d", i), ad)
	}
	if cerr := tx.Commit(); cerr != nil {
		panic(cerr)
	}

	rows, err := d.AggregateColsStats("true",
		[]db.GroupCol{{Attr: "Pool"}},
		[]db.AggSpec{
			{Func: db.AggCount, Arg: "*"},
			{Func: db.AggUser, UserFunc: "exampleMean", Arg: "Memory"},
		}, nil, nil)
	if err != nil {
		panic(err)
	}
	sort.Slice(rows, func(i, j int) bool { return rows[i].Group[0] < rows[j].Group[0] })
	for _, r := range rows {
		fmt.Printf("%s count=%s mean=%s\n", r.Group[0], r.Values[0], r.Values[1])
	}

}
Output:
laws: <nil>
"cm1" count=3 mean=2389.3333333333335
"cm2" count=2 mean=8192

Index

Examples

Constants

This section is empty.

Variables

View Source
var ErrBadSpec = errors.New("scriptagg: the module does not define a usable aggregate")

ErrBadSpec is returned when the module does not offer the four functions this needs, in the shapes it needs them. It wraps whatever the compiler said, so a loader can print one error and a caller can still tell a malformed aggregate from a missing name.

Functions

This section is empty.

Types

type Aggregate

type Aggregate struct {
	// contains filtered or unexported fields
}

Aggregate is a compiled script aggregate, ready for db.RegisterAggregate.

It is READ-ONLY once built, apart from its counters, and safe to use from any number of goroutines: a vm.ScriptFunc allocates its own register file per call, and the state an accumulator threads is a value rather than a reference into anything shared.

func New

func New(mod *vm.ScriptModule, spec Spec) (*Aggregate, error)

New compiles the module's four functions and returns the aggregate they describe.

EVERYTHING CHECKABLE IS CHECKED HERE, which is the same order scriptudf uses and matters for the same reason: a registration that can fail at query time fails for every caller of that query, and says so in a place nobody is watching. So each function is compiled -- which applies the purity gate, so a function reaching the ad is refused here -- each arity is checked against the role it fills, and `init` is CALLED, so an aggregate whose identity is itself a failure is a build error rather than a column of ERROR later.

The arities are not negotiable because the engine supplies the arguments: init takes none, add takes the state and one row's value, merge takes two states, result takes one.

func Register

func Register(name string, mod *vm.ScriptModule, spec Spec) (*Aggregate, error)

Register compiles the aggregate and registers it under name, so a request naming db.AggSpec{Func: db.AggUser, UserFunc: name} reaches it.

Compilation comes first and registration second, as it does for a scalar script function: a name is only ever registered against four functions that compiled, passed the purity gate and produced an identity. The registry's own failures come back unwrapped -- db.ErrAggregateExists for a name already taken, db.ErrBadAggregate for one no query could refer to.

func (*Aggregate) Columnar

func (a *Aggregate) Columnar() bool

Columnar reports whether the aggregate declared itself fit for the columnar value pass.

func (*Aggregate) NewAccumulator

func (a *Aggregate) NewAccumulator() db.Accumulator

NewAccumulator returns a fresh accumulator holding the identity.

func (*Aggregate) Stats

func (a *Aggregate) Stats() Stats

Stats snapshots the counters. The fields are read independently, so a snapshot taken while a query is running can be internally inconsistent by a call or two.

type Spec

type Spec struct {
	Init, Add, Merge, Result string

	// Columnar opts the aggregate into the columnar value pass, which is how SUM is served: one
	// pass over the column's blocks, whole blocks ruled out by the zone map, no record decoded.
	// The alternative is the record scan, which is correct and slower, never wrong.
	//
	// IT IS A PROMISE, NOT A SPEED SETTING, and the promise is one this package cannot check for
	// the author. The pass yields values in BLOCK ORDER, which is not insertion order and not the
	// same order twice, so `add` must give the same answer whatever order it sees the values in;
	// and it yields only the column's NUMERIC values, so a row whose attribute is absent or a
	// string is never offered -- an `add` that counts those would count a different set of rows on
	// the two rungs and answer the same query two ways depending on what has been sealed.
	//
	// db.CheckAggregate tests both of those for an aggregate that declares it. Set this, run
	// CheckAggregate, and keep the test.
	Columnar bool
}

Spec names the module's four functions and states what the author promises about them.

An empty name means the default, which is the function's role spelled in lower case -- init, add, merge, result. A module written for this package needs no Spec fields at all; the names exist for a module that exports several aggregates and has to tell them apart.

type Stats

type Stats struct {
	Accumulators uint64
	Merges       uint64
	Results      uint64
}

Stats is a snapshot of one aggregate's counters.

Accumulators is the one to read first: it is how many per-group, per-worker states a query made, so a value above the group count is the columnar fan-out reaching the aggregate. A test asserting that a fan-out happened has nothing else to assert on, because a pass that quietly stopped fanning out gives the same answers.

THERE IS NO PER-ROW COUNTER. Add is the hot loop -- once per row per group -- and an atomic increment on it would be a contended write on the one path that has to stay cheap. Merges and Results are per partial state and per group, which is thousands of times rarer.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL