Documentation
¶
Overview ¶
Package transform compiles a yaml definition of column transforms into a plan that runs over Arrow record batches. A definition is parsed once, type-checked against a resource schema once, and then applied to every batch as a fixed list of whole-column operations. Rows are visited only inside a kernel's inner loop, never by the plan itself.
This file is the interpreter. The compiler emits ops over nodes, and Apply walks them against a frame of Arrow columns. Nothing outside this file turns a literal into a scalar, a column into a datum, or calls a kernel.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ErrInvalid = errors.New("transform: invalid definition")
ErrInvalid is the sentinel every compile-time failure wraps.
Functions ¶
func Analyze ¶
Analyze compiles like Compile and also reports the type of every sub-expression that compiled, in compile order, and the schema the steps produce, whether or not the whole definition did. A builder shows those while the author is mid-edit.
func Grammar ¶
func Grammar() []byte
Grammar is the JSON schema every definition must satisfy, for clients that want to check a document's shape before sending it.
func GrammarVersion ¶
func GrammarVersion() int
GrammarVersion is the version a definition must declare.
Types ¶
type Analysis ¶
type Analysis struct {
Types []ExpressionType
Layout rowmodel.Schema
}
Analysis is what Analyze learns besides the plan: the type of every sub-expression that compiled, and the schema as the steps leave it. Layout is filled in whether or not the definition compiled: the entries of a step that compiled still apply, and the ones that failed are skipped, so a builder can offer the right columns to the steps after a broken one.
type ArgSpec ¶
type ArgSpec struct {
Name string
DisplayName string
Types []rowmodel.LogicalType // accepted logical types; empty accepts any
Literal bool // must be a literal
Column bool // must be a column-shaped value
Optional bool
Variadic bool
}
ArgSpec constrains one positional argument of a catalog function. Types lists the logical types it accepts; an empty list accepts any. Literal and Column say whether the argument must be a constant or a column. Variadic on the last argument lets a call repeat it any number of times.
type Definition ¶
type Definition struct {
Version int `yaml:"version"`
Resources map[string]Resource `yaml:"resources"`
}
Definition is a parsed transform document: a step list per resource.
func Parse ¶
func Parse(data []byte) (*Definition, error)
Parse checks a yaml document against the grammar and decodes it. Parse knows nothing about schemas; column and type checks happen in Compile.
type Errors ¶
type Errors struct {
Issues []Issue
}
Errors collects every problem a compile pass finds, so an author sees all of them at once rather than one per attempt.
type Expr ¶
Expr is a column reference, a literal, or a function call, and exactly one of Col, Lit, or Fn is set. Literals are never nil, so Lit != nil is the test for one.
type ExpressionType ¶
type ExpressionType struct {
Path string
Logical rowmodel.LogicalType
}
ExpressionType is the logical type of one compiled sub-expression, at the same kind of path an Issue uses, so a client reads types and issues off one map. resources["r"].steps[2].compute["x"] is a whole expression, resources["r"].steps[2].compute["x"].concat[1] its second argument, and resources["r"].steps[2].where.and[0].gt[1] a value inside a condition. A literal that was widened to a column's type reports the widened type.
type FunctionSpec ¶
type FunctionSpec struct {
Name string
DisplayName string
Description string
OperatorSymbol string
ConditionOperator bool
ConditionJoin string
VariadicAddLabel string
Args []ArgSpec
Returns rowmodel.LogicalType
SameType bool
ReturnsInput bool
}
FunctionSpec is a catalog function's signature: its arguments, the logical type it returns, and presentation metadata for a builder to show. An OperatorSymbol marks a binary function that can use compact infix UI. A SameType function needs every argument to share one type, and at least one column among them; a literal is widened to the column's type when it fits. ReturnsInput means the result takes that shared type.
func Functions ¶
func Functions() []FunctionSpec
Functions returns every catalog function's signature, in catalog order.
func FunctionsFor ¶
func FunctionsFor(t rowmodel.LogicalType) []FunctionSpec
FunctionsFor returns the functions that accept a column of the given type in some argument, in catalog order. It is the same acceptance the compiler applies, so a builder offering these never proposes a call Compile rejects.
type Plan ¶
type Plan struct {
// contains filtered or unexported fields
}
Plan is a compiled transform for one resource: the schema it produces and the ordered column operations that produce it. A Plan never changes after Compile returns and may be applied from many goroutines at once.
func Compile ¶
func Compile(def *Definition, in rowmodel.Schema) (*Plan, error)
Compile resolves the definition's steps for in.Resource against in and returns the plan that applies them. Every column reference and function call is checked here, so Apply never meets a name or type it has not seen. A resource the definition does not mention compiles to an identity plan.
func (*Plan) Apply ¶
func (p *Plan) Apply(ctx context.Context, mem memory.Allocator, rec arrow.RecordBatch) (arrow.RecordBatch, error)
Apply runs the plan over rec and returns a new record the caller owns. The leading columns of rec must match the schema the plan was compiled against. Any columns after them pass through unchanged and land after the plan's output, so a caller may append its own fields to a batch without the plan knowing. Columns the plan does not touch are shared with rec, not copied.
type Resource ¶
type Resource struct {
Steps []Step `yaml:"steps"`
}
Resource holds the steps applied to one resource, in order.
type Step ¶
type Step struct {
Rename map[string]string `yaml:"rename,omitempty"`
Drop []string `yaml:"drop,omitempty"`
Compute map[string]Expr `yaml:"compute,omitempty"`
Where *Expr `yaml:"where,omitempty"`
}
Step is one of rename, drop, or compute; the grammar allows exactly one per step. Where belongs to compute alone and limits which rows it rewrites. No step removes rows. Entries of one compute are independent of each other, so their order carries no meaning; columns a compute creates are appended in name order.
Source Files
Directories