Documentation
¶
Overview ¶
Package search is the chassis-owned lexical search store: a durable, tenant-scoped place to index text records and find them again by the words they contain, exposed to txcl as txco://search/*.
**Separation of concerns.** This package does lexical retrieval and nothing else. It stores no embeddings and never talks to a model. Semantic retrieval is chassis/vector's job; fusing the two lanes belongs to the stack, above both primitives.
**The collection is the ranking unit.** Term statistics are local to one (tenant, collection), so one collection's documents never move another's ranking. A query addresses exactly one collection.
**One filter grammar.** Filter is the vector store's Filter, by alias, so a stack narrows both retrieval lanes with the same object and the two cannot drift.
Index ¶
- Constants
- func Register(name string, c Constructor)
- func ValidateCollectionName(name string) error
- func ValidateItems(items []Item) error
- func ValidateQuery(q string, limit int) (int, error)
- func ValidateSelector(sel Selector) error
- func ValidateUpdate(filter Filter, ch Change) error
- type AnalyzerMismatchError
- type Change
- type CodedError
- type Collection
- type CollectionNotFoundError
- type Condition
- type Config
- type Constructor
- type FieldSet
- type Filter
- type Hit
- type InvalidArgError
- type Item
- type Manifest
- type Mutation
- type MutationOp
- type RecoveringError
- type Selector
- type Sequenced
- type Snapshotter
- type Store
- type TooLargeError
- type UnavailableError
Constants ¶
const ( CodeDisabled = "txco_search_disabled" CodeNoTenant = "txco_search_no_tenant" // CodeStore is the code of any backend error that carries none. CodeStore = "txco_search_store" )
Codes the op layer raises itself, for conditions no backend sees.
const ( MaxTextBytes = 64 << 10 MaxShortFieldBytes = 1 << 10 // title, heading, name MaxEntities = 64 MaxEntityBytes = 256 MaxMetadataBytes = 16 << 10 MaxMetadataKeys = 64 MaxIDBytes = 512 MaxItemsPerUpsert = 500 MaxIDsPerDelete = 500 MaxQueryBytes = 8 << 10 MaxNameBytes = 256 // a collection name DefaultLimit = 10 MaxLimit = 100 )
Limits of the stack-facing API. They are public limits, not engine limits: every backend enforces the same ones by calling the Validate functions, so a request that works on a laptop works hosted.
const CountPending = -1
CountPending is returned in place of a count by a backend that accepted a Delete or an Update but has not applied it yet. The bundled backend applies synchronously and never returns it.
const ManifestFile = "manifest.json"
ManifestFile is the manifest's name inside a snapshot directory.
const StoreNone = "none"
StoreNone is the --search-store value that turns lexical search off. Open returns a nil Store for it, and the ops answer txco_search_disabled.
Variables ¶
This section is empty.
Functions ¶
func Register ¶
func Register(name string, c Constructor)
Register adds a backend constructor. Called from a backend package's init().
func ValidateCollectionName ¶
ValidateCollectionName rejects a name a backend could not address.
func ValidateItems ¶
ValidateItems checks one upsert: its size, and each item's id and fields. A repeated id inside one upsert is refused, because "which one wins" would differ between backends.
func ValidateQuery ¶
ValidateQuery checks a query string and clamps its limit: zero or less takes DefaultLimit, and anything above MaxLimit is MaxLimit.
func ValidateSelector ¶
ValidateSelector checks that a Delete names records exactly one way.
func ValidateUpdate ¶
ValidateUpdate checks that an Update is conditional and changes something.
Types ¶
type AnalyzerMismatchError ¶
type AnalyzerMismatchError struct {
Collection string
Field string // "analyzer_version" | "scoring_model"
Existing string
Requested string
}
AnalyzerMismatchError is returned when a collection is pinned to an analyzer or scoring model other than the one asked for, or than this build provides. The fix is a re-index of the collection, never an edit of the pin.
func (*AnalyzerMismatchError) Code ¶
func (e *AnalyzerMismatchError) Code() string
func (*AnalyzerMismatchError) Error ¶
func (e *AnalyzerMismatchError) Error() string
type Change ¶
type Change struct {
Merge map[string]any `json:"merge,omitempty"`
Fields *FieldSet `json:"fields,omitempty"`
}
Change is what an Update does to each record its filter admits. Merge is a shallow metadata merge, RFC 7396 style: a key replaces the record's key, and a nil value removes it. Fields replaces short searched fields. Text is never changed by an update: new text is a new upsert.
type CodedError ¶
CodedError carries a stable txco_search_* code surfaced on the op envelope's `search.error.code` so rule authors dispatch uniformly with `WHEN @search.error EXEC ...`.
type Collection ¶
type Collection struct {
Name string `json:"name"`
AnalyzerVersion string `json:"analyzer_version"`
ScoringModel string `json:"scoring_model"`
// Records is filled by DescribeCollection and ListCollections.
Records uint64 `json:"records"`
}
Collection is one lexical corpus: the ranking unit, the physical index unit, and the unit a query addresses. AnalyzerVersion and ScoringModel are pinned when the collection is created. Changing either means re-indexing it.
type CollectionNotFoundError ¶
CollectionNotFoundError is returned by Upsert/Query/Delete/Update for an unknown (tenant, collection).
func (*CollectionNotFoundError) Code ¶
func (e *CollectionNotFoundError) Code() string
func (*CollectionNotFoundError) Error ¶
func (e *CollectionNotFoundError) Error() string
type Config ¶
type Config struct {
// Path is the bundled backend's root directory (--search-path).
Path string
// MaxOpenIndexes bounds how many collection indexes the bundled backend
// holds open at once (--search-max-open-indexes). Zero takes its default.
MaxOpenIndexes int
}
Config carries backend-selecting options resolved from chassis config. A backend reads only the fields it needs. Adding a field here doesn't affect existing backends (same posture as vector.Config).
type Constructor ¶
Constructor builds a Store from resolved config.
type FieldSet ¶
type FieldSet struct {
Name *string `json:"name,omitempty"`
Title *string `json:"title,omitempty"`
Heading *string `json:"heading,omitempty"`
}
FieldSet replaces the short searched fields it names. A nil field is left alone; a pointer to "" clears it.
type Filter ¶
Filter narrows a query, a delete or an update by metadata. It is the vector store's grammar (eq, in, not_in, gte, lte, gt, lt; conjunction), including its two rules: the special field "id" addresses the record id, and an absent metadata field passes not_in.
type Hit ¶
type Hit struct {
ID string `json:"id"`
Rank int `json:"rank"`
Text string `json:"text"`
Metadata map[string]any `json:"metadata,omitempty"`
}
Hit is one query result. Rank is 1 for the best hit. No score is exposed: the contract is "best first", and a raw engine score is not portable across analyzer changes, backends or shards. Text is returned because a lexical hit may be absent from the vector lane's results, and the stack needs its body.
type InvalidArgError ¶
type InvalidArgError struct{ Reason string }
InvalidArgError is a malformed request: a missing id, an unknown filter op, an update with no filter.
func (*InvalidArgError) Code ¶
func (e *InvalidArgError) Code() string
func (*InvalidArgError) Error ¶
func (e *InvalidArgError) Error() string
type Item ¶
type Item struct {
ID string `json:"id"`
Text string `json:"text,omitempty"`
Title string `json:"title,omitempty"`
Heading string `json:"heading,omitempty"`
Name string `json:"name,omitempty"`
Entities []string `json:"entities,omitempty"`
Metadata map[string]any `json:"metadata,omitempty"`
}
Item is one searchable record. ID is unique within a collection; an upsert of an existing ID replaces the record. Text, Title, Heading, Name and Entities are searched, and a backend may weight them differently. Metadata is filtered on, never searched.
type Manifest ¶
type Manifest struct {
Tenant string `json:"tenant"`
Collection string `json:"collection"`
AppliedSeq uint64 `json:"applied_seq"`
Records uint64 `json:"records"`
AnalyzerVersion string `json:"analyzer_version"`
ScoringModel string `json:"scoring_model"`
Engine string `json:"engine"`
CreatedAt string `json:"created_at"`
}
Manifest describes one snapshot of one collection.
type Mutation ¶
type Mutation struct {
Op MutationOp `json:"op"`
Collection Collection `json:"collection,omitempty"` // ensure
Items []Item `json:"items,omitempty"` // upsert
Select Selector `json:"select,omitempty"` // delete
Filter Filter `json:"filter,omitempty"` // update
Change Change `json:"change,omitempty"` // update
}
Mutation is one write to one collection, in a form that does not depend on how it arrived. Every write a Store accepts reduces to one, so a backend has a single apply path, and a backend that ships writes elsewhere ships exactly what the far side applies.
type MutationOp ¶
type MutationOp string
MutationOp names a write.
const ( OpEnsure MutationOp = "ensure" OpUpsert MutationOp = "upsert" OpDelete MutationOp = "delete" OpUpdate MutationOp = "update" OpDrop MutationOp = "drop" )
type RecoveringError ¶
type RecoveringError struct{ Reason string }
RecoveringError is returned by a backend whose engine is up but has not yet caught up, so an answer now would be silently incomplete.
func (*RecoveringError) Code ¶
func (e *RecoveringError) Code() string
func (*RecoveringError) Error ¶
func (e *RecoveringError) Error() string
type Selector ¶
type Selector struct {
IDs []string `json:"ids,omitempty"`
Filter Filter `json:"filter,omitempty"`
}
Selector names the records a Delete removes: IDs, or a Filter with at least one condition. Exactly one of the two is set.
type Sequenced ¶
type Sequenced interface {
// ApplyAt applies m to the collection unless the collection has already
// applied a mutation at seq or later, in which case it does nothing and
// reports applied == false. The position is stored in the same write as
// the mutation's last change, so after a crash a collection is never
// ahead of, or behind, the position it reports. A replay of the same log
// from any earlier point therefore converges on the same records.
//
// OpEnsure creates the collection, OpDrop removes it, and the other ops
// need it to exist (CollectionNotFoundError otherwise). count is what the
// matching Store method would have returned.
ApplyAt(ctx context.Context, tenant, collection string, seq uint64, m Mutation) (count int, applied bool, err error)
// AppliedSeq reports the position of the last mutation the collection
// applied. found is false when the collection is not on this node's disk.
AppliedSeq(ctx context.Context, tenant, collection string) (seq uint64, found bool, err error)
}
Sequenced applies mutations that carry a position in some ordered log.
type Snapshotter ¶
type Snapshotter interface {
// StagingDir is a scratch directory on the same filesystem as the store,
// so that Install can move a snapshot into place instead of copying it.
StagingDir() (string, error)
// Snapshot writes a complete, self-contained copy of the collection into
// dir, which must not exist. The collection stays open and writable
// meanwhile. The copy is opened and checked before Snapshot returns, and
// the manifest records what it actually holds, not what was asked for.
Snapshot(ctx context.Context, tenant, collection, dir string) (Manifest, error)
// Install makes the snapshot at dir (as written by Snapshot) this store's
// copy of the collection, replacing any it has. dir is consumed.
Install(ctx context.Context, tenant, collection, dir string) (Manifest, error)
}
Snapshotter copies a collection out of a live store, and back in.
type Store ¶
type Store interface {
// EnsureCollection creates the collection if absent and is a no-op if it
// exists. c.AnalyzerVersion and c.ScoringModel may be left empty to take
// the backend's; a value that differs from an existing collection's pin,
// or that the backend cannot provide, returns AnalyzerMismatchError.
EnsureCollection(ctx context.Context, tenant string, c Collection) error
// DescribeCollection returns the pinned collection and its record count.
// found is false (nil error) when it doesn't exist.
DescribeCollection(ctx context.Context, tenant, name string) (c Collection, found bool, err error)
// Upsert inserts or replaces items by ID, all or nothing. Returns the
// number of items written.
Upsert(ctx context.Context, tenant, collection string, items []Item) (int, error)
// Query returns up to limit hits for q among the records filter admits,
// best first. q is plain language: a quoted string is a phrase, an
// identifier is matched whole, and the rest is OR-ed. No engine syntax is
// recognised. An empty result is not an error.
Query(ctx context.Context, tenant, collection, q string, limit int, filter Filter) ([]Hit, error)
// Delete removes the records sel names. Returns the number removed, or
// CountPending.
Delete(ctx context.Context, tenant, collection string, sel Selector) (int, error)
// Update applies ch to every record filter admits, leaving text alone.
// filter must carry at least one condition: an unconditional rewrite is
// refused with InvalidArgError. The filter is evaluated when the update is
// applied. Returns the number of records matched, or CountPending.
Update(ctx context.Context, tenant, collection string, filter Filter, ch Change) (int, error)
// ListCollections returns the tenant's collections, sorted by name. Not a
// hot path.
ListCollections(ctx context.Context, tenant string) ([]Collection, error)
// DropCollection removes a collection and all its records, returning the
// number of records removed. A missing collection is not an error
// (returns 0).
DropCollection(ctx context.Context, tenant, name string) (int, error)
// per-node. The bundled backend is per-node (returns false).
Shared() bool
// Close flushes and releases the backend. A search index must be closed.
Close() error
}
Store is the backend-agnostic lexical search interface. Implementations must be safe for concurrent use. tenant scopes every call (collections are namespaced by (tenant, name)); the op layer pins it from TenantScope, never from the envelope.
type TooLargeError ¶
TooLargeError is a request past one of the limits in limits.go.
func (*TooLargeError) Code ¶
func (e *TooLargeError) Code() string
func (*TooLargeError) Error ¶
func (e *TooLargeError) Error() string
type UnavailableError ¶
UnavailableError is returned by a backend that cannot reach its engine right now. A stack should treat it as "no lexical results", not as a failure of the request.
func (*UnavailableError) Code ¶
func (e *UnavailableError) Code() string
func (*UnavailableError) Error ¶
func (e *UnavailableError) Error() string
Directories
¶
| Path | Synopsis |
|---|---|
|
Package blevestore is the bundled lexical search backend: one Bleve index per (tenant, collection) on local disk.
|
Package blevestore is the bundled lexical search backend: one Bleve index per (tenant, collection) on local disk. |
|
searchbench
command
Command searchbench indexes a file of records into one blevestore collection and runs a file of queries against it.
|
Command searchbench indexes a file of records into one blevestore collection and runs a file of queries against it. |
|
Package searchtest holds the backend-agnostic conformance suite for search.Store implementations.
|
Package searchtest holds the backend-agnostic conformance suite for search.Store implementations. |