Documentation
¶
Index ¶
- Constants
- func AllIndices(n int) []int
- func Avg[T Numeric](col []T) float64
- func CoercePredicateValue(val interface{}, ft FieldType) (interface{}, error)
- func Count[T any](col []T) int
- func DistinctString(col []string) []string
- func FilterIndices[T Numeric](col []T, op func(T) bool) []int
- func FilterIndicesBool(col []bool, op func(bool) bool) []int
- func FilterIndicesString(col []string, op func(string) bool) []int
- func Gather[T any](col []T, indices []int) []T
- func GroupCount(groupCol []string) map[string]int
- func GroupCountIndices(groupCol []string, indices []int) map[string]int
- func GroupSum[T Numeric](groupCol []string, valCol []T) map[string]T
- func GroupSumIndices[T Numeric](groupCol []string, valCol []T, indices []int) map[string]T
- func Max[T Numeric](col []T) T
- func Min[T Numeric](col []T) T
- func PutTokeniser(t *Tokeniser)
- func SkipValue(tokens []Token, i int) int
- func SortIndicesBy[T Numeric](col []T, indices []int, desc bool)
- func SortIndicesByString(col []string, indices []int, desc bool)
- func Sum[T Numeric](col []T) T
- type Atom
- type AtomRegistry
- func (r *AtomRegistry) FullName(a Atom) (string, bool)
- func (r *AtomRegistry) Lookup(name string) (Atom, bool)
- func (r *AtomRegistry) MustRegister(name string) Atom
- func (r *AtomRegistry) NeedsFullVerify(a Atom) bool
- func (r *AtomRegistry) Register(name string) (Atom, error)
- func (r *AtomRegistry) VerifyMatch(a Atom, keyBytes []byte) bool
- type ColumnStore
- type FieldExtractor
- type FieldPredicate
- type FieldSpec
- type FieldType
- type FilterFieldEntry
- type FilterResult
- type Numeric
- type PredicateOp
- type PredicateSet
- type Token
- type TokenType
- type Tokeniser
Constants ¶
const MaxNestingDepth = 10000
MaxNestingDepth bounds how deeply the tokeniser will recurse through nested objects and arrays. Beyond this, Tokenise returns an error rather than recursing further (D-003): unbounded recursion on attacker-shaped nesting produces a `fatal error: stack overflow`, which is not a panic and cannot be caught by recover(), so it would kill the process. The limit matches the standard library's json decoder ceiling (10000), keeping jsonic self- protecting independent of who fills the document column.
Variables ¶
This section is empty.
Functions ¶
func CoercePredicateValue ¶
CoercePredicateValue converts an interface{} value (typically from OQL parsing) into the appropriate Go type for the given FieldType. This is used when building FieldPredicates from parsed OQL WHERE expressions.
func Count ¶
Count returns the number of elements in a slice. Trivial, but provided for API consistency with other aggregate functions.
func DistinctString ¶
DistinctString returns the unique values in a string column, preserving first-occurrence order.
func FilterIndices ¶
FilterIndices returns the indices of elements that satisfy the predicate. The returned slice is allocated once at ~25% of the column length (a reasonable estimate for selective filters).
func FilterIndicesBool ¶
FilterIndicesBool returns the indices of bool elements that satisfy the predicate.
func FilterIndicesString ¶
FilterIndicesString returns the indices of string elements that satisfy the predicate.
func GroupCount ¶
GroupCount counts elements per group key.
func GroupCountIndices ¶
GroupCountIndices counts elements per group key using an index subset.
func GroupSumIndices ¶
GroupSumIndices groups by string column using an index subset.
func PutTokeniser ¶
func PutTokeniser(t *Tokeniser)
PutTokeniser returns a tokeniser to the pool for reuse. The input reference is cleared to avoid retaining large byte slices.
func SkipValue ¶
SkipValue advances past a single JSON value (including nested objects and arrays) starting at token index i. Returns the index of the next token after the skipped value.
func SortIndicesBy ¶
SortIndicesBy sorts a set of row indices by the values in a numeric column. When desc is true, sorts in descending order.
func SortIndicesByString ¶
SortIndicesByString sorts row indices by string column values.
Types ¶
type Atom ¶
type Atom uint64
Atom is a uint64-packed representation of a field name for O(1) key matching during JSON token walking. Names up to 8 bytes are packed directly (one byte per position); longer names use FNV-1a hashing.
func MakeAtom ¶
MakeAtom converts a string to an Atom. For names <= 8 bytes, the packing is bijective (no collisions possible). For names > 8 bytes, FNV-1a is used and the caller should use AtomRegistry to detect hash collisions.
func MakeAtomBytes ¶
MakeAtomBytes converts a byte slice to an Atom without allocation.
func (Atom) MatchBytes ¶
MatchBytes reports whether a byte slice matches this Atom. For short names (<= 8 bytes), this is a single integer comparison. For hashed names, it compares the hash value.
type AtomRegistry ¶
type AtomRegistry struct {
// contains filtered or unexported fields
}
AtomRegistry manages a set of field names and their Atom representations. It detects hash collisions for names longer than 8 bytes (where FNV-1a is used). For names <= 8 bytes, packing is bijective and collision-free.
The registry is safe for concurrent reads after initial registration. Registration itself is mutex-protected.
func NewAtomRegistry ¶
func NewAtomRegistry() *AtomRegistry
NewAtomRegistry creates a new empty registry.
func (*AtomRegistry) FullName ¶
func (r *AtomRegistry) FullName(a Atom) (string, bool)
FullName returns the registered name for an atom. Returns ("", false) if the atom was not registered.
func (*AtomRegistry) Lookup ¶
func (r *AtomRegistry) Lookup(name string) (Atom, bool)
Lookup returns the Atom for a previously registered name. Returns (0, false) if the name was not registered.
func (*AtomRegistry) MustRegister ¶
func (r *AtomRegistry) MustRegister(name string) Atom
MustRegister is like Register but panics on collision.
func (*AtomRegistry) NeedsFullVerify ¶
func (r *AtomRegistry) NeedsFullVerify(a Atom) bool
NeedsFullVerify reports whether an atom match requires full string comparison (true for FNV-1a hashed names > 8 bytes).
func (*AtomRegistry) Register ¶
func (r *AtomRegistry) Register(name string) (Atom, error)
Register adds a field name to the registry and returns its Atom. If the name collides with an existing (different) name, Register returns an error. This should be called at query planning time, not in the hot path.
func (*AtomRegistry) VerifyMatch ¶
func (r *AtomRegistry) VerifyMatch(a Atom, keyBytes []byte) bool
VerifyMatch checks whether a byte slice from the JSON input truly matches an atom. For short names (<= 8 bytes), atom comparison is sufficient. For hashed names, it compares the full string.
type ColumnStore ¶
type ColumnStore struct {
Strings map[Atom][]string
Ints map[Atom][]int64
Floats map[Atom][]float64
Bools map[Atom][]bool
Rows int
}
ColumnStore holds typed columnar data indexed by Atom. Each column is a contiguous slice of a single type, enabling tight loops that the CPU can pipeline. Row N across all columns corresponds to the same entity.
func ExtractRows ¶
func ExtractRows(blobs [][]byte, fields []FieldSpec, registry *AtomRegistry, copyStrings bool) *ColumnStore
ExtractRows tokenises and extracts fields from multiple JSON blobs into a single ColumnStore. This is the primary entry point for batch processing.
func NewColumnStore ¶
func NewColumnStore(capacity int) *ColumnStore
NewColumnStore creates a new column store. The capacity hint is used to pre-allocate column slices (not required for correctness).
func (*ColumnStore) IncrementRows ¶
func (cs *ColumnStore) IncrementRows()
IncrementRows increments the row counter. Called after extracting fields from one JSON object.
func (*ColumnStore) String ¶
func (cs *ColumnStore) String() string
func (*ColumnStore) ToMaps ¶
func (cs *ColumnStore) ToMaps(nameMap map[Atom]string) []map[string]interface{}
ToMaps converts the columnar data back to the traditional []map[string]interface{} representation. This is the compatibility bridge for executor code that hasn't been migrated to columnar input.
The nameMap maps Atom -> field name string for the output maps.
type FieldExtractor ¶
type FieldExtractor struct {
Fields []FieldSpec
Registry *AtomRegistry // optional, for collision verification
CopyStrings bool // if true, copy strings from input (safe for pooled buffers)
// contains filtered or unexported fields
}
FieldExtractor walks a tokenised JSON object and populates a ColumnStore with only the specified fields. Unrecognised fields are skipped without allocation.
The copyStrings parameter controls whether string values are copied from the input buffer. When true (production use), strings are heap- allocated and safe to use after the input buffer is recycled. When false (benchmark use), zero-copy unsafeString is used.
func NewFieldExtractor ¶
func NewFieldExtractor(fields []FieldSpec, registry *AtomRegistry, copyStrings bool) *FieldExtractor
NewFieldExtractor creates a new extractor for the given fields. If registry is non-nil, hashed atom matches (names > 8 bytes) are verified against the full string to detect collisions.
func (*FieldExtractor) Extract ¶
func (fe *FieldExtractor) Extract(tok *Tokeniser, cs *ColumnStore)
Extract walks the tokens from a single JSON object and appends extracted field values to the appropriate columns in the store. After calling Extract for each row, call cs.IncrementRows().
The method expects the token stream to start with TokObjStart. Nested objects and arrays are skipped correctly but not extracted.
type FieldPredicate ¶
type FieldPredicate struct {
Name string // JSON key name
Atom Atom // pre-computed atom for fast key matching
Type FieldType // expected value type of the field
Op PredicateOp // comparison operator
Val interface{} // comparison target: string, float64, int64, bool, or []interface{} for IN
}
FieldPredicate describes a filter condition on a single JSON field. The Value is stored as the native Go type that will be compared against the tokenised JSON value.
func MakeFieldPredicate ¶
func MakeFieldPredicate(name string, typ FieldType, op PredicateOp, val interface{}) FieldPredicate
MakeFieldPredicate creates a predicate, computing the Atom from the name.
func (*FieldPredicate) EvalTokenValue ¶
EvalTokenValue evaluates a predicate against a raw token from the JSON input. Returns (matched, fieldWasSeen). If the token type doesn't match the predicate's expected type, returns (false, true) — the field was seen but the type mismatch means no match.
type FieldSpec ¶
type FieldSpec struct {
Name string // JSON key name
Atom Atom // pre-computed atom for fast matching
Type FieldType // expected value type
}
FieldSpec describes a field to extract from a JSON object.
func MakeFieldSpec ¶
MakeFieldSpec creates a FieldSpec, computing the Atom from the name.
type FieldType ¶
type FieldType uint8
FieldType identifies the expected type of a field for extraction.
type FilterFieldEntry ¶
FilterFieldEntry pairs an Atom with its output field name.
func MakeFilterFieldEntries ¶
func MakeFilterFieldEntries(names []string) []FilterFieldEntry
MakeFilterFieldEntries builds a slice of FilterFieldEntry from names.
type FilterResult ¶
type FilterResult struct {
Passed bool // true if all predicates matched
Data map[string]interface{} // extracted fields (only populated if Passed)
}
FilterResult holds the outcome of a filtered extraction pass.
func FilterExtractFromTokens ¶
func FilterExtractFromTokens( tok *Tokeniser, outputFields []FilterFieldEntry, preds *PredicateSet, ) FilterResult
FilterExtractFromTokens walks a tokenised JSON object, extracts the requested fields, and evaluates the predicate set in a single pass.
The walk extracts both predicate fields and output fields as they are encountered (JSON key order is arbitrary). At the end of the object, all predicates are evaluated. If any predicate field was not found in the object, the predicate is treated as not matching (closed-world).
This avoids allocating the output map entirely for rows that fail the predicate — the main performance win of B4.
Parameters:
- tok: tokeniser with a completed Tokenise() call
- outputFields: the SELECT fields to extract (FilterFieldEntry-style)
- preds: predicate set to evaluate (may be nil for no filtering)
Returns a FilterResult. If preds is nil, Passed is always true.
type PredicateOp ¶
type PredicateOp uint8
PredicateOp identifies a comparison operator for field predicates.
const ( OpEq PredicateOp = iota // = OpNeq // != OpLt // < OpLte // <= OpGt // > OpGte // >= OpIn // IN (value list) OpLike // LIKE (string pattern) )
func (PredicateOp) String ¶
func (op PredicateOp) String() string
String returns the operator symbol.
type PredicateSet ¶
type PredicateSet struct {
Predicates []FieldPredicate
// contains filtered or unexported fields
}
PredicateSet is an AND-combined set of field predicates. All must match for a row to pass. This represents the subset of WHERE clauses that can be evaluated during tokenisation.
func NewPredicateSet ¶
func NewPredicateSet(preds []FieldPredicate) *PredicateSet
NewPredicateSet creates a predicate set from the given predicates.
func (*PredicateSet) LookupAtom ¶
func (ps *PredicateSet) LookupAtom(a Atom) int
LookupAtom returns the predicate index for a given atom, or -1 if the atom doesn't correspond to any predicate field.
type Token ¶
type Token struct {
Type TokenType
Start uint32 // byte offset of token content start
End uint32 // byte offset past last byte of token content
}
Token represents a single JSON token as a (type, start, end) triple referencing byte offsets in the original input. No copies are made.
type Tokeniser ¶
type Tokeniser struct {
// contains filtered or unexported fields
}
Tokeniser is a zero-allocation JSON tokeniser that produces a flat array of tokens referencing the original input byte slice.
func GetTokeniser ¶
func GetTokeniser() *Tokeniser
GetTokeniser returns a tokeniser from the pool, or allocates a new one.
func (*Tokeniser) TokenBytes ¶
TokenBytes returns the raw bytes of a token from the input.
func (*Tokeniser) TokenCount ¶
TokenCount returns the number of tokens produced.
func (*Tokeniser) TokenString ¶
TokenString extracts the string content of a TokString token from the input. This is a zero-copy operation using unsafeString; the returned string is only valid while the input byte slice is alive.
Source Files