microfts2

package module
v0.4.4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 5, 2026 License: MIT Imports: 22 Imported by: 0

README

microfts2

A dynamic trigram index backed by bbolt, written in Go. Usable as a CLI tool or as a library.

microfts2 indexes files into raw byte trigrams (24-bit, 16M possible) organized by chunks, then maintains an inverted index for fast substring search. The index is maintained incrementally — every add/remove updates it immediately.

Install

go install microfts/cmd/microfts@latest

Or clone and build:

git clone https://github.com/zot/microfts2.git
cd microfts2
go build ./cmd/microfts

Quick Start

# Create a database
microfts init -db myindex

# Register a chunking strategy (any command that outputs range\tcontent lines)
microfts strategy add -db myindex -name lines -cmd "microfts chunk-lines"

# Add files
microfts add -db myindex -strategy lines src/*.go

# Search
microfts search -db myindex "func Open"
# output: src/db.go:198-260

# Check for stale files
microfts stale -db myindex

# Refresh stale files and search in one step
microfts search -r -db myindex "func Open"

How It Works

Every byte is its own value — no character set mapping. Three consecutive bytes form a 24-bit trigram. UTF-8 multibyte characters produce cross-boundary byte trigrams; character-internal trigrams are skipped.

Search computes the query's trigrams, optionally filters them via a caller-supplied TrigramFilter, intersects posting lists from the inverted index, and returns matching file/chunk locations.

Single-tree design: All records live in one bbolt bucket, distinguished by a prefix byte. Trigram frequency counts (sparse C records), file metadata, settings, and the inverted trigram-to-chunk mapping share the one tree, maintained incrementally on every add/remove.

Dynamic trigram filtering: Query trigram selection is handled at search time via TrigramFilter functions. Stock filters include FilterAll (use all trigrams), FilterByRatio (skip high-frequency trigrams), and FilterBestN (keep N most selective). Callers can supply custom filters.

Staleness detection: Each indexed file records its modification time and SHA-256 hash. Checking mod time first avoids hashing unchanged files. The -r flag refreshes stale files before any command.

CLI Reference

All commands require -db <path>. Optional: -db-name for a custom bucket name (default fts).

Command Description
init Create a new database. -case-insensitive, -aliases
add Add files. -strategy <name>
search Search for text. -regex, -score coverage|density, -verify
delete Remove files from the database
reindex Re-chunk files with a different strategy. -strategy <name>
score Score named files against a query. -score coverage|density
stale List stale and missing files
strategy add|remove|list Manage chunking strategies
chunk-lines Built-in chunker: one chunk per line
chunk-lines-overlap Built-in chunker: overlapping line windows. -lines, -overlap
chunk-words-overlap Built-in chunker: overlapping word windows. -words, -overlap, -pattern

Global flag: -r refreshes all stale files before running the command. Usable standalone (microfts -r -db path) or combined (microfts search -r -db path query).

Library API

import "microfts2"

// Lifecycle
db, err := microfts.Create(path, microfts.Options{})
db, err := microfts.Open(path, microfts.Options{})
db.Close()

// Content
fileid, err := db.AddFile(filepath, strategyName)
db.RemoveFile(filepath)
fileid, err := db.Reindex(filepath, strategyName)

// Search
results, err := db.Search("query", microfts.WithTrigramFilter(microfts.FilterAll))
results, err := db.SearchRegex("pattern")

// Scoring
chunks, err := db.ScoreFile("query", filepath, microfts.CoverageScore)

// Staleness
status, err := db.CheckFile(filepath)    // FileStatus{Path, Status, FileID, Strategy}
statuses, err := db.StaleFiles()         // all indexed files
refreshed, err := db.RefreshStale("")    // reindex stale files

// Strategies
db.AddStrategy(name, command)
db.AddStrategyFunc(name, fn)
db.RemoveStrategy(name)

Chunking Strategies

A chunking strategy is either an external command or a Go function that takes a file and produces chunks. Each chunk has an opaque range label and text content to index.

External chunker example using awk:

microfts strategy add -db myindex -name awk-lines \
  -cmd "awk 'BEGIN{pos=0} {pos+=length(\$0)+1; print pos}'"

License

MIT

Documentation

Index

Constants

View Source 
const BitsetSize = 2097152 // 2^21 bytes = 2^24 bits

Variables

View Source 
var ErrAlreadyIndexed = errors.New("file already indexed")

ErrAlreadyIndexed is returned when AddFile is called for a path that already has F records in the database. Use Reindex or AppendChunks instead. R215

View Source 
var ErrAppendBoundary = errors.New("AppendChunks: non-AppendAware chunker produced zero chunks for non-empty content; appended bytes may straddle a chunk boundary — use Reindex")

ErrAppendBoundary is returned by AppendChunks when the registered chunker is not an AppendAwareChunker and produces zero chunks from non-empty content — the appended bytes likely straddle a chunk boundary the chunker can't see without prior context, so silently appending would lose them. Callers should detect this with errors.Is and fall through to Reindex (or another full refresh) instead. R623, R624

View Source 
var ErrNoChunks = errors.New("chunker produced no chunks")

ErrNoChunks is returned when a chunker produces zero chunks for a file.

View Source 
var LangC = BracketLang{
	LineComments:  []string{"//"},
	BlockComments: [][2]string{{"/*", "*/"}},
	Brackets: []BracketGroup{
		{Open: []string{"{"}, Close: []string{"}"}},
		{Open: []string{"("}, Close: []string{")"}},
		{Open: []string{"["}, Close: []string{"]"}},
		{Open: []string{`"`}, Close: []string{`"`}, Escape: `\`, AllowedInner: []string{}},
		{Open: []string{"'"}, Close: []string{"'"}, Escape: `\`, AllowedInner: []string{}},
	},
}

LangC is the bracket language config for C/C++.

View Source 
var LangGo = BracketLang{
	LineComments:  []string{"//"},
	BlockComments: [][2]string{{"/*", "*/"}},
	Brackets: []BracketGroup{
		{Open: []string{"{"}, Close: []string{"}"}},
		{Open: []string{"("}, Close: []string{")"}},
		{Open: []string{"["}, Close: []string{"]"}},
		{Open: []string{`"`}, Close: []string{`"`}, Escape: `\`, AllowedInner: []string{}},
		{Open: []string{"`"}, Close: []string{"`"}, AllowedInner: []string{}},
		{Open: []string{"'"}, Close: []string{"'"}, Escape: `\`, AllowedInner: []string{}},
	},
}

LangGo is the bracket language config for Go.

View Source 
var LangJS = BracketLang{
	LineComments:  []string{"//"},
	BlockComments: [][2]string{{"/*", "*/"}},
	Brackets: []BracketGroup{
		{Open: []string{"{"}, Close: []string{"}"}},
		{Open: []string{"("}, Close: []string{")"}},
		{Open: []string{"["}, Close: []string{"]"}},
		{Open: []string{`"`}, Close: []string{`"`}, Escape: `\`, AllowedInner: []string{}},
		{Open: []string{"'"}, Close: []string{"'"}, Escape: `\`, AllowedInner: []string{}},
		{Open: []string{"`"}, Close: []string{"`"}, Escape: `\`, AllowedInner: []string{"${"}},
		{Open: []string{"${"}, Close: []string{"}"}, AllowedParent: []string{"`"}},
	},
}

LangJS is the bracket language config for JavaScript. Backquote template literals permit ${...} interpolation back to code mode.

View Source 
var LangJava = LangC

LangJava is the bracket language config for Java.

View Source 
var LangLisp = BracketLang{
	LineComments: []string{";"},
	Brackets: []BracketGroup{
		{Open: []string{"("}, Close: []string{")"}},
		{Open: []string{"["}, Close: []string{"]"}},
		{Open: []string{`"`}, Close: []string{`"`}, Escape: `\`, AllowedInner: []string{}},
	},
}

LangLisp is the bracket language config for Lisp/Scheme/Clojure.

View Source 
var LangNginx = BracketLang{
	LineComments: []string{"#"},
	Brackets: []BracketGroup{
		{Open: []string{"{"}, Close: []string{"}"}},
		{Open: []string{`"`}, Close: []string{`"`}, Escape: `\`, AllowedInner: []string{}},
		{Open: []string{"'"}, Close: []string{"'"}, AllowedInner: []string{}},
	},
}

LangNginx is the bracket language config for nginx.

View Source 
var LangPascal = BracketLang{
	BlockComments: [][2]string{{"{", "}"}, {"(*", "*)"}},
	Brackets: []BracketGroup{
		{
			Open:       []string{"begin", "record", "class"},
			Separators: []string{},
			Close:      []string{"end"},
		},
		{
			Open:       []string{"if"},
			Separators: []string{"then", "else"},
			Close:      []string{"end"},
		},
		{Open: []string{"("}, Close: []string{")"}},
		{Open: []string{"["}, Close: []string{"]"}},
		{Open: []string{"'"}, Close: []string{"'"}, AllowedInner: []string{}},
	},
}

LangPascal is the bracket language config for Pascal.

View Source 
var LangShell = BracketLang{
	LineComments: []string{"#"},
	Brackets: []BracketGroup{
		{
			Open:       []string{"if"},
			Separators: []string{"then", "elif", "else"},
			Close:      []string{"fi"},
		},
		{
			Open:       []string{"while", "for"},
			Separators: []string{"do"},
			Close:      []string{"done"},
		},
		{
			Open:  []string{"case"},
			Close: []string{"esac"},
		},
		{Open: []string{"{"}, Close: []string{"}"}},
		{Open: []string{"("}, Close: []string{")"}},
		{Open: []string{`"`}, Close: []string{`"`}, Escape: `\`, AllowedInner: []string{}},
		{Open: []string{"'"}, Close: []string{"'"}, AllowedInner: []string{}},
	},
}

LangShell is the bracket language config for Bourne shell / bash.

Functions

func DecodeByteRangeLocator added in v0.4.0 

func DecodeByteRangeLocator(loc []byte) (int, int, bool)

DecodeByteRangeLocator parses a locator written by EncodeByteRangeLocator. Returns (start, end, ok). ok=false on malformed input. // CRC: crc-Chunker.md | R594

func DecodeFilename 

func DecodeFilename(keys [][]byte) string

DecodeFilename reconstructs a filename from chained N record keys. Keys must be in order (part 0, 1, ..., 255).

func DecodeTrigram 

func DecodeTrigram(v uint32) string

DecodeTrigram converts a 24-bit trigram value back to a 3-byte string. Bytes that are 0 (whitespace-encoded) are shown as spaces.

func EncodeByteRangeLocator added in v0.4.0 

func EncodeByteRangeLocator(start, end int) []byte

EncodeByteRangeLocator encodes a byte range [start, end) as varint pairs. Used by built-in line-oriented chunkers to populate Chunk.Locator so random-access retrieval and append-merge can avoid line-offset table builds. CRC: crc-Chunker.md | R594

func FinalKey 

func FinalKey(filename string) []byte

FinalKey returns the final N record key for direct fileid lookup.

func LineChunkFunc 

func LineChunkFunc(_ string, content []byte, yield func(Chunk) bool) error

LineChunkFunc is a built-in ChunkFunc that yields one chunk per line. Range is "N-N" (1-based line number). LineChunkFunc yields one chunk per newline-terminated line. Range is "N-N"; Locator carries the chunk's byte range so AppendAware merge and random-access retrieval can skip line-offset rebuilds. // CRC: crc-Chunker.md | R594

func MarkdownChunkFunc 

func MarkdownChunkFunc(_ string, content []byte, yield func(Chunk) bool) error

MarkdownChunkFunc splits markdown content into paragraph-based chunks. Heading lines start new chunks; a heading and its following paragraph (up to the next blank line or heading) form one chunk. Blank lines are boundaries only and are not included in any chunk's content. Fenced code blocks (``` or ~~~) suppress blank-line splitting — all lines from opening fence through matching close belong to the current chunk. // R465, R466, R467, R468 Headline merging: after a heading chunk, consecutive tag-only chunks (lines starting with @) and one following content chunk are absorbed into a single merged chunk with internal blank lines included. // R563, R564, R565, R566, R567, R568, R569, R570

func PairGet 

func PairGet(pairs []Pair, key string) ([]byte, bool)

PairGet returns the value for the first Pair matching key, or nil if not found.

func ScoreOverlap 

func ScoreOverlap(queryTrigrams []uint32, chunkCounts map[uint32]int, _ int) float64

ScoreOverlap: count of matching query trigrams, no normalization (OR semantics).

func TrigramValue 

func TrigramValue(a, b, c byte) uint32

TrigramValue computes the 24-bit trigram from three byte values.

func UnmarshalTValue 

func UnmarshalTValue(data []byte) ([]uint64, error)

UnmarshalTValue decodes a TRecord value. Trigram must be set separately (from key).

func UnmarshalWValue 

func UnmarshalWValue(data []byte) ([]uint64, error)

UnmarshalWValue decodes a WRecord value. Same format as TRecord.

func ValidateAliases 

func ValidateAliases(aliases map[byte]byte) error

ValidateAliases returns an error if any alias source or target byte is non-ASCII (≥ 0x80). Aliasing UTF-8 continuation or leading bytes would corrupt multibyte characters and break character-internal trigram skipping.

func WithIndexedChunkCallback added in v0.4.0 

func WithIndexedChunkCallback(fn IndexedChunkCallback) interface {
	IndexOption
	AppendOption
}

WithIndexedChunkCallback supplies a chunkid-aware callback. Fires once per genuinely newly-inserted chunkid (not per-emitted-chunk): chunks whose content matches an existing C record (refcount bumped) do NOT fire. The callback receives the chunk plus its freshly-written C record. The CRecord is txn-attached, so callbacks may call FileRecord and other DB lookups during the fire.

Coexists with WithChunkCallback — the text-only callback fires for every yielded chunk (vector-embedding path), the indexed callback only for genuinely new ones.

Fires from the LMDB indexing paths: AddFile, AddFileWithContent, Reindex (and its variants), RefreshStale, AppendChunks. Does NOT fire for the in-memory overlay paths (AddTmpFile / UpdateTmpFile / AppendTmpFile); tmp:// documents don't have persistent CRecords to deliver.

func WithRemovedChunkCallback added in v0.4.0 

func WithRemovedChunkCallback(fn RemovedChunkCallback) interface {
	IndexOption
	AppendOption
	RemoveOption
}

WithRemovedChunkCallback supplies a callback that fires once per chunk being orphaned (refcount dropped to zero) by the current operation. Fires before the C record is deleted, with txn context attached. The CRecord carries hash, trigrams, tokens, and attrs — enough for ark and similar consumers to clean up chunkid-keyed external state (embeddings, tag stores).

Works as IndexOption (Reindex, RefreshStale — for chunks the file used to reference but no longer does), AppendOption (AppendChunks — for the replaced-last-chunk case under AppendAwareChunker), and RemoveOption (RemoveFile — for chunks the removed file used to reference).

Types

type AppendAwareChunker added in v0.4.0 

type AppendAwareChunker interface {
	AppendChunks(path string, lastLocator []byte, newBytes []byte, yield func(Chunk) bool) (replacedLast bool, err error)
}

AppendAwareChunker is the optional interface a chunker implements to handle append-merge boundary cases. Given the locator of the file's last existing chunk (nil if none) and the new bytes being appended, the chunker yields the chunks that replace the trailing region. replacedLast=true tells the dispatcher to drop the F record's last entry and decrement that chunkid's fileid count in its C record (cascading orphan cleanup as in RemoveFile). Chunkers that don't implement this interface get the default behavior in DB.AppendChunks: chunk newBytes alone and append. // CRC: crc-Chunker.md | R601, R602, R603, R604, R610

type AppendOption 

type AppendOption interface {
	// contains filtered or unexported methods
}

AppendOption configures AppendChunks behavior. R158

func WithAppendChunkCallback deprecated 

func WithAppendChunkCallback(fn ChunkCallback) AppendOption

WithAppendChunkCallback supplies a chunk callback for append methods. CRC: crc-DB.md | R471

Deprecated: use WithIndexedChunkCallback for chunkid-aware callbacks, or WithChunkCallback if only chunk text is needed (now accepted by both indexing and append methods via dualOption — see WithChunkCallback).

func WithBaseLine 

func WithBaseLine(n int) AppendOption

WithBaseLine sets the 1-based line number offset for line-based chunker ranges. When non-zero, "start-end" ranges are adjusted by adding this offset. R162

func WithContentHash 

func WithContentHash(hash string) AppendOption

WithContentHash sets the full-file SHA-256 hash (caller pre-computed). R159

func WithFileLength 

func WithFileLength(n int64) AppendOption

WithFileLength sets the full file size after append. R161

func WithModTime 

func WithModTime(t int64) AppendOption

WithModTime sets the file modification time (Unix nanoseconds). R160

type Bitset 

type Bitset [BitsetSize]byte

Bitset is a fixed-size bitset for 16,777,216 trigrams (2^24).

func (*Bitset) Bytes 

func (b *Bitset) Bytes() []byte

Bytes returns the bitset as a byte slice for storage.

func (*Bitset) Count 

func (b *Bitset) Count() int

Count returns the number of set bits.

func (*Bitset) ForEach 

func (b *Bitset) ForEach(fn func(uint32))

ForEach calls fn for each set bit in ascending order.

func (*Bitset) FromBytes 

func (b *Bitset) FromBytes(data []byte)

FromBytes loads the bitset from stored bytes.

func (*Bitset) Set 

func (b *Bitset) Set(trigram uint32)

Set sets the bit for the given trigram.

func (*Bitset) Test 

func (b *Bitset) Test(trigram uint32) bool

Test returns whether the bit for the given trigram is set.

type BracketGroup 

type BracketGroup struct {
	Open          []string
	Separators    []string
	Close         []string
	Escape        string
	AllowedInner  []string
	AllowedParent []string
}

BracketGroup defines one set of matching brackets, code or string-like. R309 Separators are mid-group markers (e.g. "else" between "if"/"end").

AllowedInner controls scanning inside the group (R618): 

nil           → code mode: full scanning, all bracket groups recognized
non-nil slice → scan-restricted mode: only Close, Escape, and listed openers
                are recognized; everything else is literal text
                (use []string{} for pure raw mode with no escape hatches)

AllowedParent restricts where this bracket may be recognized (R619): 

nil           → recognized in any context (default for top-level brackets)
non-nil slice → only recognized when scanning is currently inside one of
                the listed openers (e.g. "${" inside "`")

Escape (R617) is consumed inside a scan-restricted group along with the byte that follows it; empty means no escaping (raw strings).

nil and []string{} are semantically distinct for AllowedInner and AllowedParent (R622).

type BracketLang 

type BracketLang struct {
	LineComments  []string       // e.g. "//", "#", "--"
	BlockComments [][2]string    // e.g. {{"/*", "*/"}, {"<!--", "-->"}}
	Brackets      []BracketGroup // open/separator/close sets — includes strings and word brackets
}

BracketLang defines the lexical rules for one language. R307 Strings are expressed as scan-restricted bracket groups in Brackets; no separate StringDelims field exists.

func LangByName 

func LangByName(name string) (BracketLang, bool)

LangByName returns a BracketLang config by name, or false if not found.

type CRecord 

type CRecord struct {
	ChunkID    uint64
	Hash       [32]byte
	ContentLen int
	Trigrams   []TrigramEntry
	Tokens     []TokenEntry
	Attrs      []Pair
	FileIDs    []FileIDCount // refcounted: per-fileid occurrence count // CRC: crc-DB.md | R595
	// contains filtered or unexported fields
}

CRecord is the per-chunk record. Self-describing: everything needed for search, scoring, filtering, and removal. Carries unexported db/bkt — the chunk is tied to the bucket that read it.

func UnmarshalCValue 

func UnmarshalCValue(data []byte) (CRecord, error)

UnmarshalCValue decodes a CRecord value. ChunkID must be set separately (from key). R498: hash + contentLen + trigrams + tokens + attrs + fileids

func (*CRecord) Bucket added in v0.4.0 

func (c *CRecord) Bucket() *bbolt.Bucket

Bucket returns the bbolt bucket this record was read from. Implements TxnHolder. R664

func (*CRecord) DB 

func (c *CRecord) DB() *DB

DB returns the database this record belongs to.

func (*CRecord) DecFileID added in v0.4.0 

func (c *CRecord) DecFileID(fileid uint64) bool

DecFileID removes one occurrence; returns true when the entry was removed entirely. // CRC: crc-DB.md | R597

func (*CRecord) FileIDCountFor added in v0.4.0 

func (c *CRecord) FileIDCountFor(fileid uint64) uint64

FileIDCountFor returns the occurrence count for the given fileid, or 0 if absent.

func (*CRecord) FileRecord 

func (c *CRecord) FileRecord(fileid uint64) (FRecord, error)

FileRecord navigates to an F record within the same transaction.

func (*CRecord) IncFileID added in v0.4.0 

func (c *CRecord) IncFileID(fileid uint64)

IncFileID adds an occurrence for fileid. // CRC: crc-DB.md | R596

func (*CRecord) MarshalValue 

func (c *CRecord) MarshalValue() []byte

MarshalValue encodes the CRecord value (everything except the key prefix and chunkid). hash + contentLen + trigrams + tokens + attrs + [fileid, count] pairs // R497, R595

func (*CRecord) RemoveFileID added in v0.4.0 

func (c *CRecord) RemoveFileID(fileid uint64) bool

RemoveFileID drops all occurrences of fileid in one shot. // CRC: crc-DB.md | R599

func (*CRecord) Tx added in v0.4.0 

func (c *CRecord) Tx() *bbolt.Tx

Tx returns the bbolt transaction this record is part of. R663, R664

type Chunk 

type Chunk struct {
	Range   []byte
	Locator []byte
	Content []byte
	Attrs   []Pair
}

Chunk is a single chunk yielded by a Chunker. Range is the human-readable label used for CLI output (e.g. "1-10" for lines). Locator is the chunker's per-occurrence opaque bytes used for fast random-access retrieval and append-merge resume; nil if the chunker doesn't need it. Content is the chunk text. Range, Locator, and Content are reusable buffers — the caller must copy before the next yield. Attrs is optional per-chunk metadata (per-content, lives on C record). // CRC: crc-Chunker.md | R587, R588, R589, R590, R591

type ChunkCache 

type ChunkCache struct {
	// contains filtered or unexported fields
}

ChunkCache is a per-query cache for file content and chunked data. Avoids redundant file reads and re-chunking when processing search results.

func (*ChunkCache) ChunkText 

func (cc *ChunkCache) ChunkText(fpath, rangeLabel string) ([]byte, bool)

ChunkText returns a single chunk's content by range label. R536 Convenience wrapper — resolves chunkID via rangeIds, delegates to ChunkTextWithId.

func (*ChunkCache) ChunkTextWithId added in v0.3.0 

func (cc *ChunkCache) ChunkTextWithId(fpath string, chunkID uint64) ([]byte, bool)

ChunkTextWithId returns a single chunk's content by ChunkID. R535 Fast path for callers that already have a chunkID (e.g. SearchResult).

func (*ChunkCache) GetChunks 

func (cc *ChunkCache) GetChunks(fpath, targetRange string, before, after int) ([]ChunkResult, error)

GetChunks retrieves the target chunk and up to before/after positional neighbors. R544 Same contract as DB.GetChunks but cached.

type ChunkCallback 

type ChunkCallback func(chunkText string)

ChunkCallback receives clean chunk text during indexing. Called once per chunk, in chunk order. The string is a copy, safe to retain. CRC: crc-DB.md | R469

type ChunkFilter 

type ChunkFilter func(chunk CRecord) bool

ChunkFilter receives a CRecord during candidate evaluation. Return true to keep the chunk, false to reject it. The CRecord carries transaction context — use Txn() and DB() for lookups.

type ChunkFunc 

type ChunkFunc func(path string, content []byte, yield func(Chunk) bool) error

ChunkFunc is a generator that yields chunks for a file. Convenience type — wrap with FuncChunker to get a full Chunker.

func RunChunkerFunc 

func RunChunkerFunc(cmd string) ChunkFunc

RunChunkerFunc returns a ChunkFunc that executes an external command. The command receives the filepath as an argument and outputs one chunk per line on stdout as "range\tcontent".

type ChunkResult 

type ChunkResult struct {
	Path    string `json:"path"`
	Range   string `json:"range"`
	Content string `json:"content"`
	Index   int    `json:"index"` // 0-based position in the file's chunk list
	Attrs   []Pair `json:"attrs,omitempty"`
}

ChunkResult holds a single chunk with its content and position. R201

type Chunker 

type Chunker interface {
	Chunks(path string, content []byte, yield func(Chunk) bool) error
}

Chunker is the content-based chunking interface for text formats. // CRC: crc-Chunker.md | R502

func BracketChunker 

func BracketChunker(lang BracketLang) Chunker

BracketChunker returns a Chunker for the given language config. R320

func IndentChunker 

func IndentChunker(lang BracketLang, tabWidth int) Chunker

IndentChunker returns a Chunker for indentation-scoped languages. R333 tabWidth controls how tabs count for column calculation (0 = one column per tab).

type ChunkerMetadata added in v0.4.0 

type ChunkerMetadata interface {
	IsWritable() bool      // true for editable text, false for binary / read-only formats
	CommentSyntax() string // line-comment delimiter, "" when n/a
}

ChunkerMetadata is the optional interface a chunker implements to expose editability and the line-comment delimiter of its underlying language. Callers (e.g. ark's curation workshop) type-assert against it; chunkers that don't implement it are treated as IsWritable=true, CommentSyntax="". CRC: crc-Chunker.md | R639

type DB 

type DB struct {
	// contains filtered or unexported fields
}

func Create 

func Create(path string, opts Options) (*DB, error)

Seq: seq-init.md | R660, R662

func Open 

func Open(path string, opts Options) (*DB, error)

Open opens an existing microfts2 database. Returns an error if the file does not exist or the fts bucket is absent. R660, R662

func (*DB) AddChunker 

func (db *DB) AddChunker(name string, c any) error

CRC: crc-DB.md | R293, R518, R519, R533 c may additionally implement RandomAccessChunker for fast-path retrieval; detection happens at dispatch time via type assertion.

func (*DB) AddFile 

func (db *DB) AddFile(fpath, strategy string, opts ...IndexOption) (uint64, error)

Seq: seq-add.md | R477

func (*DB) AddFileWithContent 

func (db *DB) AddFileWithContent(fpath, strategy string, opts ...IndexOption) (uint64, []byte, error)

CRC: crc-DB.md | R120, R478

func (*DB) AddStrategy 

func (db *DB) AddStrategy(name, cmd string) error

func (*DB) AddStrategyFunc 

func (db *DB) AddStrategyFunc(name string, fn ChunkFunc) error

CRC: crc-DB.md | R294

func (*DB) AddTmpFile 

func (db *DB) AddTmpFile(path, strategy string, content []byte, opts ...IndexOption) (uint64, error)

CRC: crc-DB.md | Seq: seq-tmp-add.md | R358, R359, R360 AddTmpFile indexes a tmp:// document in the in-memory overlay. CRC: crc-DB.md | R480

func (*DB) AppendChunks 

func (db *DB) AppendChunks(fileid uint64, content []byte, strategy string, opts ...AppendOption) error

AppendChunks adds chunks to an existing file without full reindex. content is only the appended bytes, not the full file. CRC: crc-DB.md | Seq: seq-append.md R150, R151, R152, R153, R154, R155, R156, R157, R163, R164, R165, R166, R167, R168, R604, R605, R606, R608

func (*DB) AppendTmpFile 

func (db *DB) AppendTmpFile(path, strategy string, content []byte, opts ...AppendOption) (uint64, error)

CRC: crc-DB.md | R428-R442, R483 AppendTmpFile appends content to an existing tmp:// document, creating it if it doesn't exist. New chunks are indexed from the appended content without touching existing chunks.

func (*DB) BM25Func 

func (db *DB) BM25Func(queryTrigrams []uint32) (ScoreFunc, error)

CRC: crc-DB.md | R274, R277, R278 BM25Func reads T records for per-trigram document frequencies and I record counters for corpus statistics, then returns a BM25 ScoreFunc closure.

func (*DB) CheckFile 

func (db *DB) CheckFile(fpath string) (FileStatus, error)

CheckFile checks whether an indexed file is fresh, stale, or missing on disk.

func (*DB) ChunkContentLens added in v0.3.0 

func (db *DB) ChunkContentLens(fileid uint64) ([]int, error)

CRC: crc-DB.md | R500, R501 ChunkContentLens returns the byte length of each chunk's content for a file, in chunk-list order. Reads F record for chunk list, then each C record's ContentLen.

func (*DB) Close 

func (db *DB) Close() error

func (*DB) Copy 

func (db *DB) Copy() *DB

CRC: crc-DB.md | R459, R460, R461, R462 Copy returns a shallow copy of the DB sharing the bbolt handle, overlay, and chunker registry. Caches are nil — the copy lazy-loads from committed bbolt state. Intended for short-lived write transactions in a separate goroutine.

func (*DB) DB added in v0.4.0 

func (db *DB) DB() *bbolt.DB

DB returns the underlying bbolt database. R661

func (*DB) FileIDPaths 

func (db *DB) FileIDPaths() (map[uint64]string, error)

CRC: crc-DB.md | R448, R449, R450, R454

func (*DB) FileInfoByID 

func (db *DB) FileInfoByID(fileid uint64) (FRecord, error)

FileInfoByID resolves a fileid to its FRecord.

func (*DB) GetChunks 

func (db *DB) GetChunks(fpath, targetRange string, before, after int) ([]ChunkResult, error)

Seq: seq-chunks.md | Seq: seq-chunker-dispatch.md | R197, R198, R199, R200, R201, R202, R203, R529, R530 GetChunks retrieves the target chunk (identified by range label) and up to before/after positional neighbors. Returns chunks in positional order. Fast path: if the resolved chunker implements RandomAccessChunker, extract each chunk in the window directly via GetChunk instead of re-running the streaming pass from the top of the file.

func (*DB) HasTmp 

func (db *DB) HasTmp() bool

CRC: crc-DB.md | R377 HasTmp reports whether any tmp:// documents exist in the overlay.

func (*DB) InvalidateCaches 

func (db *DB) InvalidateCaches()

CRC: crc-DB.md | R463, R464 InvalidateCaches nils the path and FRecord caches, forcing lazy reload on next access. Does not reset overlayOnce.

func (*DB) NewChunkCache 

func (db *DB) NewChunkCache() *ChunkCache

NewChunkCache creates a per-query chunk cache.

func (*DB) NewSearchCache 

func (db *DB) NewSearchCache() func()

CRC: crc-DB.md | R456

func (*DB) QueryTrigramCounts 

func (db *DB) QueryTrigramCounts(query string) ([]TrigramCount, error)

QueryTrigramCounts extracts trigrams from a query string and returns their corpus document frequencies. For diagnostic/inspection use.

func (*DB) ReadCRecord added in v0.4.0 

func (db *DB) ReadCRecord(tx *bbolt.Tx, chunkID uint64) (CRecord, error)

CRC: crc-DB.md | R571, R664 ReadCRecord fetches and unmarshals a C record by chunkID, attaching db/bucket to the result so Bucket(), Tx(), DB(), and FileRecord(fileid) work on it. Must be called inside a bbolt View or Update txn — the bucket is part of the CRecord.

func (*DB) RecordCounts 

func (db *DB) RecordCounts() (map[byte]RecordStats, error)

CRC: crc-DB.md | R443, R444, R445

func (*DB) RefreshStale 

func (db *DB) RefreshStale(strategy string, opts ...IndexOption) ([]FileStatus, error)

RefreshStale reindexes all stale files. If strategy is empty, each file's existing strategy is used. Returns the list of stale/missing files. CRC: crc-DB.md | R479

func (*DB) Reindex 

func (db *DB) Reindex(fpath, strategy string, opts ...IndexOption) (uint64, error)

func (*DB) ReindexWithCallback added in v0.3.0 

func (db *DB) ReindexWithCallback(fpath, strategy string, fn ReindexCallback, opts ...IndexOption) (uint64, error)

CRC: crc-DB.md | R556, R557, R558, R559, R560, R561, R562

func (*DB) ReindexWithContent 

func (db *DB) ReindexWithContent(fpath, strategy string, opts ...IndexOption) (uint64, []byte, error)

CRC: crc-DB.md | R121

func (*DB) RemoveFile 

func (db *DB) RemoveFile(fpath string, opts ...RemoveOption) error

func (*DB) RemoveFileWithCallback added in v0.3.0 

func (db *DB) RemoveFileWithCallback(fpath string, fn RemoveCallback) error

CRC: crc-DB.md | R547, R548, R549, R550, R551, R552, R553, R554

func (*DB) RemoveStrategy 

func (db *DB) RemoveStrategy(name string) error

func (*DB) RemoveTmpFile 

func (db *DB) RemoveTmpFile(path string) error

CRC: crc-DB.md | Seq: seq-tmp-add.md | R364, R365 RemoveTmpFile removes a tmp:// document from the overlay.

func (*DB) ScoreFile 

func (db *DB) ScoreFile(query, fpath string, fn ScoreFunc, opts ...SearchOption) ([]ScoredChunk, error)

Seq: seq-score.md | R178, R179, R180 ScoreFile returns per-chunk scores for a single file using the given scoring function.

func (*DB) Search 

func (db *DB) Search(query string, opts ...SearchOption) (*SearchResults, error)

Seq: seq-search.md | R178, R179, R180, R181, R182

func (*DB) SearchFuzzy 

func (db *DB) SearchFuzzy(query string, k int, opts ...SearchOption) (*SearchResults, error)

CRC: crc-DB.md | Seq: seq-fuzzy-trigram.md | R418, R419, R420, R421, R422, R423, R425, R427 SearchFuzzy performs fast typo-tolerant search using two phases: Phase 1: trigram OR-union tally from T record posting lists (select top-k) Phase 2: C record re-score with ScoreCoverage for the top-k winners

func (*DB) SearchMulti 

func (db *DB) SearchMulti(query string, strategies map[string]ScoreFunc, k int, opts ...SearchOption) ([]MultiSearchResult, error)

CRC: crc-DB.md | Seq: seq-search-multi.md | R283, R284, R285, R287, R288, R289, R290

func (*DB) SearchRegex 

func (db *DB) SearchRegex(pattern string, opts ...SearchOption) (*SearchResults, error)

Seq: seq-search.md SearchRegex searches using a regex pattern against the full trigram index.

func (*DB) SetFileLength added in v0.4.0 

func (db *DB) SetFileLength(fileid uint64, length int64) error

SetFileLength rewrites only the FileLength field of a file's F record. Intended for callers that need to advance the append high-water mark without re-chunking — e.g. "checkpoint" operations that skip processing of historical content. All other fields, including Chunks, Tokens, ContentHash, and ModTime, are preserved. Returns an error if the file is unknown.

func (*DB) Settings 

func (db *DB) Settings() Settings

Settings returns the current database settings.

func (*DB) StaleFiles 

func (db *DB) StaleFiles() ([]FileStatus, error)

StaleFiles returns the status of every indexed file.

func (*DB) TmpContent 

func (db *DB) TmpContent(path string) (*bytes.Reader, error)

CRC: crc-DB.md | R378 TmpContent returns a reader over the raw stored content of a tmp:// document.

func (*DB) TmpFileIDs 

func (db *DB) TmpFileIDs() map[uint64]struct{}

CRC: crc-DB.md | R369 TmpFileIDs returns the set of all current tmp:// fileids.

func (*DB) TmpFileInfos added in v0.3.0 

func (db *DB) TmpFileInfos() []TmpFileInfo

TmpFileInfos returns status info for all tmp:// documents.

func (*DB) UpdateTmpFile 

func (db *DB) UpdateTmpFile(path, strategy string, content []byte, opts ...IndexOption) error

CRC: crc-DB.md | Seq: seq-tmp-add.md | R361, R362, R363, R481 UpdateTmpFile replaces the content of an existing tmp:// document.

func (*DB) Version 

func (db *DB) Version() (string, error)

CRC: crc-DB.md

type FRecord 

type FRecord struct {
	FileID      uint64
	ModTime     int64
	ContentHash [32]byte
	FileLength  int64
	Strategy    string
	Names       []string
	Chunks      []FileChunkEntry
	Tokens      []TokenEntry
}

FRecord is the per-file record. Metadata, ordered chunks, file-level token bag.

func UnmarshalFHeader 

func UnmarshalFHeader(data []byte) (FRecord, error)

R451, R452: UnmarshalFHeader decodes only the header fields of an F record value: ModTime, ContentHash, FileLength, Strategy, and Names. Skips Chunks and Tokens.

func UnmarshalFValue 

func UnmarshalFValue(data []byte) (FRecord, error)

UnmarshalFValue decodes an FRecord value. FileID must be set separately (from key).

func (*FRecord) MarshalValue 

func (f *FRecord) MarshalValue() []byte

MarshalValue encodes the FRecord value (everything except the key prefix and fileid). Chunk entries are [chunkid, location, locator] — both location and locator are length-prefixed bytes (either may be empty). // R592

type FileChunkEntry 

type FileChunkEntry struct {
	ChunkID  uint64
	Location string
	Locator  []byte
}

FileChunkEntry is a per-occurrence entry in the F record's chunk list. Location is the chunker's Range (human display); Locator is the chunker's Locator (per-occurrence opaque bytes for fast retrieval and append-merge resume). Both are length-prefixed in the wire format and either may be empty. CRC: crc-DB.md | R592

type FileChunker added in v0.3.0 

type FileChunker interface {
	FileChunks(path string, old [32]byte, yield func(Chunk) bool) ([32]byte, error)
}

FileChunker is the file-based chunking interface for binary formats. // CRC: crc-Chunker.md | R505, R506, R507, R508, R509, R510, R511 FileChunks reads from path, computes the SHA-256 hash, and yields chunks. If old is non-zero and matches the file hash, chunking may be skipped (yield never called). Returns the content hash. Zero hash signals no content. The method is named FileChunks (not Chunks) so a single type can also implement Chunker.

type FileIDCount added in v0.4.0 

type FileIDCount struct {
	FileID uint64
	Count  uint64
}

FileIDCount records how many times a chunk occurs in a file. // CRC: crc-DB.md | R595

type FileStatus 

type FileStatus struct {
	Path     string
	Status   string // "fresh", "stale", "missing"
	FileID   uint64
	Strategy string
}

FileStatus is returned by CheckFile and StaleFiles.

type FuncChunker 

type FuncChunker struct {
	Fn ChunkFunc
}

FuncChunker wraps a bare ChunkFunc into a Chunker. // CRC: crc-Chunker.md

func (FuncChunker) Chunks 

func (fc FuncChunker) Chunks(path string, content []byte, yield func(Chunk) bool) error

type HRecord 

type HRecord struct {
	Hash    [32]byte
	ChunkID uint64
}

HRecord maps content hash to chunkid.

type IndexOption 

type IndexOption interface {
	// contains filtered or unexported methods
}

IndexOption configures indexing methods (AddFile, AddFileWithContent, RefreshStale, AddTmpFile, UpdateTmpFile). CRC: crc-DB.md | R472

func WithChunkCallback 

func WithChunkCallback(fn ChunkCallback) IndexOption

WithChunkCallback supplies a chunk callback for indexing methods. CRC: crc-DB.md | R470

type IndexStatus 

type IndexStatus struct {
	Built bool
}

IndexStatus reports the state of the index.

type IndexedChunk added in v0.4.0 

type IndexedChunk struct {
	Chunk   Chunk
	CRecord CRecord
}

IndexedChunk pairs a yielded chunk with the freshly-written C record. Delivered to WithIndexedChunkCallback only when the chunk's content was genuinely new in this indexing call (the C record was created, not bumped). On the persistent path the CRecord carries txn context, so callbacks may call FileRecord and other DB lookups during the fire. On the overlay (tmp://) path there is no bbolt transaction: CRecord.Tx() and CRecord.DB() return nil, and only the chunkid / hash / attrs / fileIDs are meaningful. Overlay chunkids count down from MaxUint64, so callers can distinguish them from persistent chunkids by range.

type IndexedChunkCallback added in v0.4.0 

type IndexedChunkCallback func(IndexedChunk)

IndexedChunkCallback receives IndexedChunk values for newly-inserted chunkids. Fires after the C record has been written. CRecord.FileIDs contains a single entry: the file currently being indexed with Count=1.

type KeyPair 

type KeyPair struct {
	Key   []byte
	Value []byte // nil for non-final parts; caller sets fileid on final part
}

KeyPair is an N record key/value pair for filename key chains.

func EncodeFilename 

func EncodeFilename(filename string) []KeyPair

EncodeFilename returns N record key/value pairs for a filename. Short filenames (≤509 bytes) produce a single final key. Longer filenames are split across chained keys.

type LineChunker added in v0.3.0 

type LineChunker struct{}

LineChunker exposes LineChunkFunc as a Chunker + FileChunker + RandomAccessChunker + AppendAwareChunker. // CRC: crc-Chunker.md | R531, R633, R634, R635, R636

func (LineChunker) AppendChunks added in v0.4.0 

func (LineChunker) AppendChunks(path string, lastLocator []byte, newBytes []byte, yield func(Chunk) bool) (bool, error)

func (LineChunker) Chunks added in v0.3.0 

func (LineChunker) Chunks(path string, content []byte, yield func(Chunk) bool) error

func (LineChunker) CommentSyntax added in v0.4.0 

func (LineChunker) CommentSyntax() string

func (LineChunker) FileChunks added in v0.4.0 

func (LineChunker) FileChunks(path string, old [32]byte, yield func(Chunk) bool) ([32]byte, error)

func (LineChunker) GetChunk added in v0.3.0 

func (LineChunker) GetChunk(path string, data []byte, customData *any, chunk *Chunk) error

func (LineChunker) IsWritable added in v0.4.0 

func (LineChunker) IsWritable() bool

CRC: crc-Chunker.md | R640

type MarkdownChunker added in v0.3.0 

type MarkdownChunker struct{}

MarkdownChunker exposes MarkdownChunkFunc as a Chunker + FileChunker + RandomAccessChunker + AppendAwareChunker. // CRC: crc-Chunker.md | R531, R601-R604, R610, R633, R636

func (MarkdownChunker) AppendChunks added in v0.4.0 

func (MarkdownChunker) AppendChunks(path string, lastLocator []byte, newBytes []byte, yield func(Chunk) bool) (bool, error)

AppendChunks delegates to appendByRechunkResume so markdown's paragraph/fence/heading-merge boundaries are recognised across the append boundary via re-chunking from the previous last chunk's start through EOF.

func (MarkdownChunker) Chunks added in v0.3.0 

func (MarkdownChunker) Chunks(path string, content []byte, yield func(Chunk) bool) error

func (MarkdownChunker) CommentSyntax added in v0.4.0 

func (MarkdownChunker) CommentSyntax() string

func (MarkdownChunker) FileChunks added in v0.4.0 

func (MarkdownChunker) FileChunks(path string, old [32]byte, yield func(Chunk) bool) ([32]byte, error)

func (MarkdownChunker) GetChunk added in v0.3.0 

func (MarkdownChunker) GetChunk(path string, data []byte, customData *any, chunk *Chunk) error

func (MarkdownChunker) IsWritable added in v0.4.0 

func (MarkdownChunker) IsWritable() bool

CRC: crc-Chunker.md | R640

type MultiSearchResult 

type MultiSearchResult struct {
	Strategy string
	Results  []SearchResult
}

CRC: crc-DB.md | R286 MultiSearchResult holds one strategy's results from SearchMulti.

type Options 

type Options struct {
	CaseInsensitive bool
	Aliases         map[byte]byte // maps input bytes to replacement bytes before trigram extraction
	DBName          string        // subdatabase name, default "fts"
	// Timeout bounds how long Open/Create waits for the bbolt file lock
	// before returning bbolt.ErrTimeout. Zero blocks until the lock is
	// available (bbolt default). The index is single-process; a bounded
	// timeout lets a second opener fail fast instead of hanging while
	// another holds the DB. R672
	Timeout time.Duration
}

Options configures database creation and opening. R666

type Pair 

type Pair struct {
	Key   []byte
	Value []byte
}

Pair is an opaque key-value pair for per-chunk metadata. Allows duplicate keys. Mirrors the DB wire format.

func CopyPairs 

func CopyPairs(src []Pair) []Pair

CopyPairs deep-copies a slice of Pair.

type RandomAccessChunker added in v0.3.0 

type RandomAccessChunker interface {
	GetChunk(path string, data []byte, customData *any, chunk *Chunk) error
}

RandomAccessChunker retrieves a single chunk by pre-filled range. // CRC: crc-Chunker.md | R524, R525, R526, R527, R528 Caller pre-fills chunk.Range (from F record Location) and chunk.Attrs (from C record). Chunker fills chunk.Content and may replace or augment chunk.Attrs. customData is per-file scratch — nil on first call, chunker populates lazily and reuses.

type RecordStats 

type RecordStats struct {
	Count      int64
	KeyBytes   int64
	ValueBytes int64
}

CRC: crc-DB.md | R445

type ReindexCallback added in v0.3.0 

type ReindexCallback func(tx *bbolt.Tx, orphanedChunkIDs, newChunkIDs []uint64) error

ReindexCallback receives the bbolt write transaction, the orphaned chunk IDs from removal, and the new chunk IDs from re-indexing. CRC: crc-DB.md | R555, R665

type RemoveCallback added in v0.3.0 

type RemoveCallback func(tx *bbolt.Tx, orphanedChunkIDs []uint64) error

RemoveCallback receives the bbolt write transaction and the list of chunk IDs that were orphaned (deleted from the index) during file removal. CRC: crc-DB.md | R546, R665

type RemoveOption added in v0.4.0 

type RemoveOption interface {
	// contains filtered or unexported methods
}

RemoveOption configures RemoveFile behavior.

type RemovedChunkCallback added in v0.4.0 

type RemovedChunkCallback func(CRecord)

RemovedChunkCallback receives the CRecord of a chunk being orphaned (refcount dropped to zero) during the current operation. Fires before the C record is deleted, with txn context attached so callbacks can perform reads. ark uses this to unindex chunkid-keyed external data (embeddings, tag stores) without racing the deletion.

type ScoreFunc 

type ScoreFunc func(queryTrigrams []uint32, chunkCounts map[uint32]int, chunkTokenCount int) float64

ScoreFunc computes a relevance score for a chunk. queryTrigrams: active query trigrams. chunkCounts: trigram -> occurrence count in the chunk. chunkTokenCount: number of tokens (words) in the chunk. 

var ScoreCoverage ScoreFunc = scoreCoverage

ScoreCoverage is the coverage scoring function: fraction of active query trigrams present in chunk. 

var ScoreDensityFunc ScoreFunc = scoreDensity

ScoreDensityFunc is the density scoring function for direct use with ScoreFile.

func ScoreBM25 

func ScoreBM25(idf map[uint32]float64, avgTokenCount float64) ScoreFunc

CRC: crc-DB.md | R272, R273 ScoreBM25 returns a ScoreFunc closure implementing BM25 ranking. idf maps trigram codes to their inverse document frequency. avgTokenCount is the average chunk token count across the corpus.

type ScoredChunk 

type ScoredChunk struct {
	Range string
	Score float64
}

ScoredChunk is a per-chunk trigram match score from ScoreFile.

type SearchOption 

type SearchOption func(*searchConfig)

SearchOption configures search behavior.

func WithAfter 

func WithAfter(t time.Time) SearchOption

WithAfter keeps chunks with timestamp >= t. Checks "timestamp" attr first (parsed as Unix nanos); falls back to file mod time from F record. CRC: crc-DB.md | R258

func WithBefore 

func WithBefore(t time.Time) SearchOption

WithBefore keeps chunks with timestamp < t. Same fallback as WithAfter. CRC: crc-DB.md | R259

func WithChunkCache 

func WithChunkCache(cc *ChunkCache) SearchOption

WithChunkCache threads an external ChunkCache through post-filters (verify, regex, except-regex). When present, post-filters use the cache instead of re-reading files. R486

func WithChunkFilter 

func WithChunkFilter(fn ChunkFilter) SearchOption

WithChunkFilter adds a chunk filter. Multiple calls accumulate (AND semantics).

func WithCoverage 

func WithCoverage() SearchOption

WithCoverage uses coverage scoring (default): matching / total active query trigrams.

func WithDensity 

func WithDensity() SearchOption

WithDensity uses token-density scoring for long queries.

func WithExcept 

func WithExcept(ids map[uint64]struct{}) SearchOption

WithExcept excludes chunks from the given file IDs. Multiple WithExcept calls union: IDs from every call are excluded.

func WithExceptRegex 

func WithExceptRegex(patterns ...string) SearchOption

WithExceptRegex adds subtract post-filters: any match rejects the chunk. Multiple calls accumulate patterns. R184, R185

func WithLoose 

func WithLoose() SearchOption

CRC: crc-DB.md | Seq: seq-fuzzy-search.md | R336 WithLoose enables OR semantics at the term level: a chunk matches if it contains any query term's trigrams. Default scoring: terms matched / total terms.

func WithNoTmp 

func WithNoTmp() SearchOption

CRC: crc-DB.md | R376 WithNoTmp excludes tmp:// overlay documents from search results.

func WithOnly 

func WithOnly(ids map[uint64]struct{}) SearchOption

WithOnly restricts search to chunks from the given file IDs. Multiple WithOnly calls intersect: only IDs present in every call survive.

func WithOverlap 

func WithOverlap() SearchOption

CRC: crc-DB.md | R271 WithOverlap uses overlap scoring: matching trigram count, no normalization.

func WithProximityRerank 

func WithProximityRerank(topN int) SearchOption

CRC: crc-DB.md | R279 WithProximityRerank reranks the top-N results by query term proximity in chunk text.

func WithRegexFilter 

func WithRegexFilter(patterns ...string) SearchOption

WithRegexFilter adds AND post-filters: every pattern must match chunk content. Multiple calls accumulate patterns. R183, R185

func WithScoring 

func WithScoring(fn ScoreFunc) SearchOption

WithScoring uses a custom scoring function.

func WithTrigramFilter 

func WithTrigramFilter(fn TrigramFilter) SearchOption

WithTrigramFilter supplies a caller-defined trigram selection function.

func WithVerify 

func WithVerify() SearchOption

WithVerify enables post-filter verification: after trigram intersection, read chunk text from disk and verify each query term appears as a case-insensitive substring. Eliminates trigram false positives. R124, R125

type SearchResult 

type SearchResult struct {
	Path  string
	Range string
	Score float64
	// contains filtered or unexported fields
}

SearchResult is a single match from Search. R99, R490, R491

type SearchResults 

type SearchResults struct {
	Results []SearchResult
	Status  IndexStatus
}

SearchResults wraps search matches with index health status.

type Settings 

type Settings struct {
	CaseInsensitive    bool
	Aliases            map[byte]byte     // byte→byte alias mapping
	ChunkingStrategies map[string]string // name→cmd (empty cmd = func strategy)
}

Settings holds the in-memory representation of I records.

type TRecord 

type TRecord struct {
	Trigram  uint32
	ChunkIDs []uint64
}

TRecord is the trigram inverted index entry.

func (*TRecord) MarshalValue 

func (t *TRecord) MarshalValue() []byte

MarshalValue encodes the TRecord value (packed chunkid list).

type TmpChunkInfo added in v0.3.0 

type TmpChunkInfo struct {
	Location string
	Size     int
}

TmpChunkInfo holds info about one chunk of a tmp:// document.

type TmpFileInfo added in v0.3.0 

type TmpFileInfo struct {
	Path       string
	FileID     uint64
	Strategy   string
	ContentLen int
	ChunkCount int
	ChunkSizes []int
	Chunks     []TmpChunkInfo
}

TmpFileInfo holds status info for a tmp:// document.

type TokenEntry 

type TokenEntry struct {
	Token string
	Count int
}

TokenEntry pairs a token string with its occurrence count.

type TrigramCount 

type TrigramCount struct {
	Trigram uint32
	Count   int
}

TrigramCount pairs a trigram code with its corpus document frequency.

func FilterAll 

func FilterAll(trigrams []TrigramCount, _ int) []TrigramCount

FilterAll uses every query trigram. No filtering.

type TrigramEntry 

type TrigramEntry struct {
	Trigram uint32
	Count   int
}

TrigramEntry pairs a trigram code with its per-chunk occurrence count.

type TrigramFilter 

type TrigramFilter func(trigrams []TrigramCount, totalChunks int) []TrigramCount

TrigramFilter decides which trigrams to use for a given query. It receives the query's trigrams with their corpus-wide document frequencies, and the total number of indexed chunks. It returns the subset to search with.

func FilterBestN 

func FilterBestN(n int) TrigramFilter

FilterBestN returns a TrigramFilter that keeps the N trigrams with the lowest document frequency.

func FilterByRatio 

func FilterByRatio(maxRatio float64) TrigramFilter

CRC: crc-DB.md | Seq: seq-search.md | R141, R674 FilterByRatio returns a TrigramFilter that skips trigrams appearing in more than maxRatio of total chunks. E.g., 0.50 skips trigrams in >50% of chunks. Below two chunks every present trigram is in 100% of chunks, so the ratio cannot discriminate; filtering there would drop every trigram and make each query unanswerable, so the filter passes them all through instead.

type Trigrams 

type Trigrams struct {
	// contains filtered or unexported fields
}

Trigrams extracts raw byte trigrams from text. Every byte is its own value — no character set mapping. Whitespace bytes are boundaries; runs collapse. Case insensitivity via bytes.ToLower(). Byte aliases applied before extraction.

func NewTrigrams 

func NewTrigrams(caseInsensitive bool, aliases map[byte]byte) *Trigrams

NewTrigrams creates a trigram extractor.

func (*Trigrams) EncodeTrigram 

func (t *Trigrams) EncodeTrigram(s string) (uint32, bool)

EncodeTrigram converts a 3-byte string to a 24-bit trigram using the same encoding as ExtractTrigrams: case folding, aliases, whitespace→0. Returns 0, false if the trigram cannot appear in the index (e.g. all whitespace, or consecutive whitespace which encode() collapses away).

func (*Trigrams) ExtractTrigrams 

func (t *Trigrams) ExtractTrigrams(data []byte) []uint32

ExtractTrigrams extracts all trigrams from data. Character-internal trigrams (windows entirely within a multibyte UTF-8 char) are skipped.

func (*Trigrams) TrigramCounts 

func (t *Trigrams) TrigramCounts(data []byte) map[uint32]int

TrigramCounts extracts trigrams with occurrence counts. Character-internal trigrams (windows entirely within a multibyte UTF-8 char) are skipped.

type TxnHolder 

type TxnHolder interface {
	Bucket() *bbolt.Bucket
}

TxnHolder carries a bbolt transaction's fts bucket. R667 CRecord implements it; bucketWrap wraps raw buckets from View/Update blocks.

type WRecord 

type WRecord struct {
	TokenHash uint32
	ChunkIDs  []uint64
}

WRecord is the token inverted index entry.

func (*WRecord) MarshalValue 

func (w *WRecord) MarshalValue() []byte

MarshalValue encodes the WRecord value (packed chunkid list, same as TRecord).

Source Files

View all Source files

Directories

Path Synopsis
cmd
microfts command

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.