Documentation
¶
Index ¶
- Constants
- Variables
- func ContainsKnn(f Filter) bool
- func ContainsSourceFilter(f Filter) bool
- func ContainsText(f Filter) bool
- func FilterTreeAny(f Filter, pred func(Filter) bool) bool
- func GuaranteesPresence(f Filter, fieldName string) bool
- func IsTextScoreSort(s Sort) bool
- func MayTighten(f Filter) bool
- func PrefixSuccessor(p []byte) []byte
- type All
- type And
- type Bound
- type Bounds
- type Comp
- type CompOp
- type Exists
- type Filter
- type In
- type Key
- type Knn
- type KnnOpt
- type Modifier
- type ModifierChain
- type ModifyFunc
- type Nor
- type Not
- type Operator
- type Or
- type RawFilter
- type RawSort
- type Regexp
- type Size
- type Sort
- type SortField
- type Sorts
- type Text
- type TextClause
- type TextOp
- type TextScoreSort
- type Type
- type TypeFilter
Constants ¶
const ( // KnnMaxK bounds $k. Policy, not semantics: raising it later is // non-breaking, lowering it is not — start conservative. KnnMaxK = 10_000 // KnnMaxEf bounds $ef, the ANN candidate/beam depth. KnnMaxEf = 65_536 )
const ( TypeNull = Type(anyenc.TypeNull) TypeNumber = Type(anyenc.TypeNumber) TypeString = Type(anyenc.TypeString) TypeFalse = Type(anyenc.TypeFalse) TypeTrue = Type(anyenc.TypeTrue) TypeArray = Type(anyenc.TypeArray) TypeObject = Type(anyenc.TypeObject) TypeVectorF32 = Type(anyenc.TypeVectorF32) TypeObjectID = Type(anyenc.TypeObjectID) )
Variables ¶
var ErrVectorNotOrderable = errors.New("any-store: a vector is not orderable")
ErrVectorNotOrderable is returned when an ordering operator ($gt/$gte/$lt/$lte) is given a vector operand. Vectors are not points on the scalar order (Rule V): the anyenc tag order sorts every vector above every scalar, which made such a comparison true for every document — emptying the collection on Delete. Equality ($eq/$ne) against a vector remains legal: it is byte equality.
Functions ¶
func ContainsKnn ¶
ContainsKnn reports whether the tree contains a Knn ANYWHERE. It MUST walk the whole tree, pointer nodes included: a Knn under Not would evaluate !false == match-all (see the Knn type comment), so every consumer that rejects bad placements needs the full walk.
func ContainsSourceFilter ¶
ContainsSourceFilter reports whether the tree contains a SOURCE filter — one matched by an index (Text, Knn) rather than by Ok. RECURSIVE, not a shallow type check: the only legal shapes are Key{path, Knn} and And{Text, …}, so a shallow test would return false for every real query and protect nobody. External consumers that post-filter by calling Filter.Ok directly (the subscription pattern) MUST reject these: Text.Ok matches everything, Knn.Ok matches nothing.
func ContainsText ¶
ContainsText reports whether the tree contains a Text ANYWHERE, pointer nodes included — same full-walk contract as ContainsKnn. Consumers use it both to reject bad placements ($text under $or/$nor/$not) and as the post-condition that the fts residual extraction stripped every Text node.
func FilterTreeAny ¶
FilterTreeAny walks the standard filter tree and reports whether pred is true for any node. It descends And/Or/Nor/Not/Key — in BOTH their value and pointer forms: every composite here has value-receiver methods, so a pointer-built &Not{…}/&Key{…}/&Or{…} satisfies Filter too, and a walker that switches only on value types silently skips such nodes. That is not a stylistic gap — ContainsKnn is a mass-delete guard (see Knn), and a missed node is a guard bypass.
func GuaranteesPresence ¶
GuaranteesPresence reports whether every document matching f must carry a present, non-null value at fieldName. It is true only when some top-level conjunct constrains fieldName with a predicate that rejects BOTH a missing field (a nil value) and an explicit null — exactly the two cases a sparse index omits (see index.writeValues: it drops a key when any indexed field is nil or TypeNull).
The planner uses this to decide whether a SPARSE index can represent the whole matching set: a sparse index that omits some matching document would silently drop rows if seeked. Probing the field's own sub-filter with Ok keeps this in lockstep with match semantics, so it is exact rather than a rule-of-thumb — e.g. {$exists:true} yields false here because an explicit-null document matches $exists:true yet is absent from the sparse index. An OR, a $exists:false, a $ne, an equality-to-null, an $in containing null, or no predicate on the field at all all yield false, keeping that sparse index out of consideration (the planner then falls back to a complete index or scan).
func IsTextScoreSort ¶
IsTextScoreSort reports whether s requests relevance ordering (directly or as the sole element of a Sorts list).
func MayTighten ¶
MayTighten reports whether f contains any conjunction that could make TightIndexBounds differ from IndexBounds for SOME field — an And node with more than one child, reachable outside Or/Not/Nor (which the tight channel delegates wide). Callers use it to skip the tight walk for the common single-predicate filters where the channels are provably identical.
func PrefixSuccessor ¶
PrefixSuccessor returns the smallest byte string greater than every string prefixed by p: trailing 0xFF bytes are stripped and the last remaining byte incremented, into a fresh slice. Used as an EXCLUSIVE End, the result admits exactly the p-prefixed keys — unlike the append-0xFF inclusive idiom, which excludes any key continuing with a 0xFF byte after the prefix (longer sorts greater). Returns nil (+inf) when p is all-0xFF or empty.
Types ¶
type And ¶
type And []Filter
func (And) IndexBounds ¶
IndexBounds returns a SOUND OVER-APPROXIMATION of the index bounds for this conjunction: the bounds of the first conjunct that constrains fieldName. The result must be a SUPERSET of the matching set so the index seek (and Iter) never miss a doc.
Intersecting conjunct bounds (the original I-04 fix) is UNSOUND for ARRAY/multi-key fields: array filter semantics match each conjunct against the whole array independently, so a doc matches {$gte:2,$lte:3} when one element is >=2 and a DIFFERENT element is <=3 — it need not have any element in [2,3]. Narrowing the seek to the intersection drops such docs from both Count and Iter (and a FilterIter cannot re-add what the seek skipped). The over-approximation here is correct for scalar AND array fields because every matching doc has an element satisfying the first conjunct, hence in its bounds; a FilterIter re-checks the full conjunction.
The CountOnly fast path skips that FilterIter, so it would over-count when these over-approx bounds are a strict superset of the matches. It is gated separately: indexCoversFilter rejects a covered field carrying more than one predicate, so the fast path is only taken when bounds exactly equal the matches (a single In/Eq/range per field). See docs/known-issues.md (I-04).
Consumers that can prove fan-out entries are absent (pk namespace, indexes with a scalar-proven multikey flag) — or that only ESTIMATE — get the intersected variant from TightIndexBounds (query/tight_bounds.go); this method's contract stays over-approximating regardless.
type Bound ¶
type Bounds ¶
type Bounds []Bound
func TightIndexBounds ¶
TightIndexBounds computes the INTERSECTED ("tight") bounds for fieldName: where a conjunction carries several bound-contributing conjuncts on the same field ({"f":{"$gt":lo,"$lt":hi}}, {$in:[...],$ne:v}, nested $and forms), the contributions are interval-intersected instead of keeping only the first one the way And.IndexBounds does.
SOUNDNESS — read before adding consumers. Tight bounds are NOT a sound seek range in general: array filter semantics match each conjunct against the whole array independently, so {tags:{$gte:2,$lte:3}} matches {tags:[5,1]} (5>=2 via one element, 1<=3 via another) with NO index entry in [2,3] — a seek narrowed to the intersection silently drops the doc from Iter AND Count (docs/known-issues.md I-04, revert ba92bc7). Tight bounds may drive:
- cost/selectivity ESTIMATION: always (estimates cannot drop docs);
- actual SEEKS: only when the scanned namespace provably holds no fan-out (multi-key) entries — the primary-key namespace (array pks are rejected on write, see ErrArrayPrimaryKey) or a secondary index whose persisted multikey flag proves it scalar-only.
empty=true reports that the intersection is PROVABLY EMPTY as a value set ({"f":{"$gt":5,"$lt":3}}). It does NOT mean the filter is unsatisfiable: under array semantics {f:[6,1]} MATCHES that filter (6>5, 1<3), so callers may treat empty as "zero results" only under the same fan-out-free proof as seeks; everywhere else they must fall back to the wide bounds. When empty=true the returned bounds are nil — never feed that to code that reads len==0 as "unconstrained".
Node handling: Key routes the field; And/*And intersect their contributing children (children contributing nothing are unconstrained, not narrowing); every other node — including Or — delegates to its own IndexBounds, so Or keeps its all-or-nothing union and never produces empty. Bounds bytes alias filter-owned pre-clipped (cap==len) memory and are never mutated in place, per docs/query-filter-contract.md.
func (Bounds) Contains ¶
Contains reports whether val lies within any range in bs. val is a raw anyenc-encoded tuple component (e.g. the field-value prefix of an index key, with docId suffix stripped).
func (Bounds) Intersect ¶
Intersect returns the lex-intersection of this bound set with other. A bound set is a union of value ranges; the intersection is the set of values present in BOTH unions. Used by TightIndexBounds to combine the bounds contributed by multiple conjuncts on the same field — NEVER by IndexBounds itself, whose over-approximation contract (see And.IndexBounds) is what keeps array/multi-key seeks sound.
An empty result (returned as a zero-length Bounds) means the two sets share no values — distinct from "no narrowing", which preserves the input. Callers must not feed a zero-length result into code that reads len==0 as "unconstrained". The pairwise intersections are sorted and merged before returning. Result endpoints alias the input bounds' Start/End bytes and are never mutated, preserving the cap==len sharing contract (docs/query-filter-contract.md).
func (Bounds) SortAndMerge ¶
SortAndMerge sorts bounds by Start key and merges overlapping/adjacent entries in a single O(N log N) pass. Use after batch-appending multiple bounds.
type Comp ¶
func (*Comp) IndexBounds ¶
IndexBounds stays a sound over-approximation under Rule V: it may select keys that Ok now rejects (an ordering op against a vector selects a tag range and then matches nothing), never the reverse. Every plan that uses these bounds still runs the residual filter — the covering CountOnly shortcuts require a PointLookup with FIXED bounds, and an ordering op is neither — so a stricter Ok cannot make a plan return rows it should not.
func (*Comp) Ok ¶
Rule V: a vector is not a point on the scalar order.
anyenc resolves a cross-type comparison on the type tag, and TypeVectorF32 (10) sorts above every scalar tag (1..9). So an ordering op with a vector on either side used to be true for EVERY document — {"v":{"$gt":1}} on a vector field, and symmetrically {"anyField":{"$lt":{"$vector":[..]}}} on any field at all, even an absent one. On Query.Delete that emptied the collection with err=nil, no vector index required. $eq/$ne are unaffected: byte equality is well-defined for a vector.
The parser rejects the operand-side spelling outright (ErrVectorNotOrderable); these guards are what protect hand-built filters, which never see the parser.
type Filter ¶
type Filter interface {
// Ok evaluates whether the given value satisfies the filter condition.
// docBuf is optional and can be nil, it's used to reuse memory between
// calls: with a warm per-query DocBuffer, Ok is alloc-free
// (TestFilterOkAllocFree).
Ok(v *anyenc.Value, docBuf *syncpool.DocBuffer) bool
// IndexBounds appends bounds to the bs for the given field if it is applicable
IndexBounds(fieldName string, bs Bounds) (bounds Bounds)
fmt.Stringer
}
Filter is built once (ParseCondition or a constructor) and is immutable afterwards: it may be shared by any number of queries running concurrently. Ok and IndexBounds never mutate filter state and need no synchronization; bound bytes returned by IndexBounds that alias filter memory are clipped to cap == len so downstream appends reallocate instead of writing into the shared filter. Pinned by TestFilterConcurrentReuse and TestIndexBounds_FilterOwnedBytesAreCapped; see docs/query-filter-contract.md.
func MustParseCondition ¶
func ParseCondition ¶
type In ¶
type In struct {
// Values is the membership set, keyed by anyenc-marshaled value. A
// built filter is immutable: once handed to a query it may be used by
// many queries concurrently, so Values must not be mutated (IndexBounds
// and Ok read it and the pre-sorted view without synchronization).
// Build In via NewInValue.
Values map[string]struct{}
// contains filtered or unexported fields
}
func NewInValue ¶
type Key ¶
type Knn ¶
type Knn struct {
Query []float32 // non-empty, finite; len checked against the index dim at detection
K int // required, 1..KnnMaxK
Ef int // 0 = auto; candidate/beam depth (numCandidates); >= K when set
Index string // optional: vector index name (disambiguates 2 indexes on one field)
}
Knn is the {"<field>":{"$knn":{"$query":[…],"$k":N}}} approximate-nearest-neighbour clause. It is NOT a predicate: it selects the K documents whose vector at this field the field's vector index ranks closest to Query. "Is this document one of the k nearest?" is a property of the CANDIDATE SET, not of the document, so no per-document truth value exists.
Ok therefore returns FALSE, unconditionally, and is unreachable in a correct plan: the executor detects the clause on EVERY verb, drives the query from the ANN source, strips the clause from the residual, and hard-errors on any placement where it could not be stripped. The false FAILS CLOSED — a Knn that somehow reaches a document-by-document evaluator matches nothing rather than everything. Contrast query.Text, whose Ok returns TRUE: fail-open on Query.Delete costs the collection. Never do that here.
FAIL-CLOSED IS LOAD-BEARING IN TWO PLACES, both of which must be kept in sync:
- ContainsKnn must walk Not/Nor: Not{Knn}.Ok == !false == MATCH-ALL. A $knn that reaches a Not is a mass delete through the front door, reachable from Query.Delete.
- GuaranteesPresence must special-case Knn: it calls the inner filter's Ok DIRECTLY and reads !Ok(nil) && !Ok(null) as "guarantees presence" — so a fail-closed Ok yields TRUE, the AGGRESSIVE answer.
func NewKnn ¶
NewKnn is the programmatic constructor for the $knn clause: wrap it in a query.Key naming the vector field. It exists because programmatic consumers build their ANN filter as a Go value and never touch JSON; every validation the parser applies is re-checked at query build time (detection), which is the only validation a hand-built filter ever sees.
func (Knn) IndexBounds ¶
IndexBounds returns bs verbatim: a Knn clause contributes no range-index bounds. Verbatim matters — Or.IndexBounds detects "this branch contributed nothing" by comparing lengths, and a shorter return silently discards accumulated sibling bounds.
func (Knn) Validate ¶
Validate checks the $knn argument rules — the ONE copy shared by the parser (parseKnn, after its shape/presence checks) and the executor's detection walk (which re-validates because programmatic NewKnn filters never see the parser). Message texts are the normative parse-error strings.
type Modifier ¶
type Modifier interface {
Modify(a *anyenc.Arena, v *anyenc.Value) (result *anyenc.Value, modified bool, err error)
}
func MustParseModifier ¶
func ParseModifier ¶
type ModifierChain ¶
type ModifierChain []Modifier
type ModifyFunc ¶
type RawFilter ¶
RawFilter is an optional fast path implemented by filters that can evaluate against the raw encoded document (as stored, root already decoded — see anyenc.Parser.DecodedDoc) without building the full Value tree. Unindexed scans use it to REJECT documents cheaply: the field walk skips unrelated values length-only, so a low-selectivity scan avoids parsing ~everything it discards (the full parse dominated the scan profile). Accepted documents are re-parsed by the caller as before, so downstream semantics are unchanged.
handled=false means this filter (or a required sub-filter) cannot decide from the raw bytes — the caller must fall back to Ok on the parsed document. Implementations must guarantee that when handled=true, ok equals what Ok would return on the parsed document (TestFilterOkRawParity).
type RawSort ¶
type RawSort interface {
AppendKeyRaw(k anyenc.Tuple, doc []byte, buf *syncpool.DocBuffer) (anyenc.Tuple, bool)
}
RawSort is an optional fast path implemented by sorts that can build their key from the raw encoded document (root already decoded — see anyenc.Parser.DecodedDoc) without a full parse. The unindexed sort path fetches and parses every candidate document just to read the sort fields; seeking the fields' raw fragments instead skips the tree build and the scanning of everything past them (see RawFilter for the same idea on the filter side).
handled=false means the key could not be built from the raw bytes (an array container on a path, malformed bytes) — k is returned unchanged (any partial append truncated) and the caller must fall back to AppendKey on the parsed document. When handled=true, the appended bytes are identical to what AppendKey would produce (TestSortAppendKeyRawParity).
type Sort ¶
type Sort interface {
Fields() []SortField
AppendKey(k anyenc.Tuple, v *anyenc.Value) anyenc.Tuple
}
func MustParseSort ¶
type Text ¶
type Text struct {
// Search is the raw $search string, preserved verbatim for String() and as
// the fallback source when Clauses is nil (a directly-constructed Text).
Search string
// DefaultAnd is set by {"$defaultOperator":"and"}: bare (should) terms then
// become required (AND) rather than the default OR.
DefaultAnd bool
// Clauses are the parsed clauses of the whole $text predicate: the should
// clauses from $search (phrases, prefix*, plain terms) plus the required and
// excluded clauses from the $require / $exclude sub-fields. Populated by
// ParseCondition; a Text built directly with only Search set has nil Clauses
// and the executor re-parses Search on demand (yielding should clauses only).
Clauses []TextClause
}
Text is the {"$text":{"$search":"..."}} full-text predicate. Matching and ranking are performed by the full-text index (the query is driven by an FTS scan), so as a residual predicate Ok always passes: any document the FTS scan yields has already matched. It contributes no index bounds.
CAUTION: this makes Text meaningful only inside an FTS-driven query. Used standalone — calling Ok directly to post-filter documents, as subscription-style consumers of this package do — a $text condition silently matches EVERY document; Ok never evaluates the search text. Placement rules ($text only at the top level or inside $and, at most one per query) are also enforced by the executor, not here. See docs/fts/DESIGN.md.
func (Text) ParsedClauses ¶
func (t Text) ParsedClauses() []TextClause
ParsedClauses returns the parsed clauses, parsing Search on demand for a Text that was constructed directly (Clauses nil) rather than via ParseCondition.
type TextClause ¶
type TextClause struct {
// Raw is the unanalyzed clause text: a single word, or the inner text of a
// "quoted phrase".
Raw string
// Phrase marks a "..." clause: Raw analyzes to an ordered term list matched
// by positional adjacency.
Phrase bool
// Prefix marks a trailing-* clause (foo*): Raw's single analyzed term is
// expanded against the vocabulary. Mutually exclusive with Phrase.
Prefix bool
// Op is the clause's boolean role.
Op TextOp
}
TextClause is one parsed unit of a $text $search string: a single term, a quoted phrase, or a prefix term, together with its boolean role. The raw text is NOT analyzed here — the query package stays free of the fts analyzer; the FTS executor analyzes each clause with the index's own pipeline.
func ParseTextSearch ¶
func ParseTextSearch(s string) []TextClause
ParseTextSearch splits a $text $search string into clauses. It is a purely syntactic scan — no analysis/tokenization happens here (that is the FTS executor's job, using the index's analyzer). Every clause it returns is a should (OR) clause; required/excluded clauses come from the typed $require / $exclude sub-fields instead (see parseText), not from inline string signs.
- "double quoted" → a phrase clause (positional adjacency).
- word* → a prefix clause (vocabulary expansion).
- word → a plain term clause.
Deliberately there is NO inline +/- boolean syntax. Parsing boolean intent out of a raw human search string is a footgun for an embedded library whose common use is "forward the end-user's search box straight into $search": a user typing "-9" (meaning −9°C) or "+1" (a vote) would silently get an exclusion/require. The same characters are harmless here — they are punctuation the analyzer drops — so "-9" is just the term "9". Boolean require/exclude lives in the structured $require / $exclude arrays, which are unambiguous and run through the same analyzer. An app that wants single-box power-user syntax can parse it itself and populate those fields. A trailing * is still honoured on bare words (prefix expansion has negligible false-positive rate in natural text); it is left literal inside a phrase, where positional expansion would be combinatorial.
type TextOp ¶
type TextOp uint8
TextOp classifies a $text clause's boolean role within the query.
const ( // TextShould is a default clause: OR'd with the other should-clauses, unless // the query sets DefaultAnd, in which case it is required. TextShould TextOp = iota // TextMust is a +term: always required (AND), regardless of DefaultAnd. TextMust // TextMustNot is a -term: documents containing it are excluded. TextMustNot )
type TextScoreSort ¶
type TextScoreSort struct {
Field string
}
TextScoreSort marks a request to order full-text results by descending relevance ({"<field>":{"$meta":"textScore"}}). It carries no document field — the full-text scan already yields documents in score order — so AppendKey and Fields are inert; the query layer recognises the type and keeps scan order.
func (TextScoreSort) AppendKeyRaw ¶
func (TextScoreSort) AppendKeyRaw(k anyenc.Tuple, _ []byte, _ *syncpool.DocBuffer) (anyenc.Tuple, bool)
AppendKeyRaw is inert like AppendKey: relevance order comes from the full-text scan itself, no document field is read.
func (TextScoreSort) Fields ¶
func (TextScoreSort) Fields() []SortField
func (TextScoreSort) String ¶
func (TextScoreSort) String() string
type TypeFilter ¶
func (TypeFilter) IndexBounds ¶
func (e TypeFilter) IndexBounds(fieldName string, bs Bounds) (bounds Bounds)
func (TypeFilter) String ¶
func (e TypeFilter) String() string
Source Files