roaringrange

package module
v0.42.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 16, 2026 License: MIT Imports: 20 Imported by: 0

README

roaringrange

CI Coverage Status Go Report Card Go Reference Maintainability Rating Reliability Rating Security Rating Snyk

Static, range-fetchable full-text search built on roaring bitmaps — query a multi-million-document trigram index in the browser, over HTTP Range requests, with no backend.

The index is one static file on object storage (S3/CDN). The browser fetches a tiny sparse view once (~tens of KB), then each query pulls a few small byte ranges — independent of corpus size. The multi-GB index is never downloaded whole. Postings are portable RoaringBitmaps, so writers and readers interoperate byte-for-byte with zero re-encoding: build an index directly with the Rust build module, or transcode an existing roaringsearch index with the Go writer — both emit the same files the Rust/WASM (or Go) reader reads.

Live demos

  • OpenAlex — openalex.evefreeman.com: 484M scholarly works, citation-ranked, faceted (year/type/open-access/language/topic), with trigram, whole-word (term, with default-on BM25 relevance), semantic (IVFPQ), and hybrid search. Trigram defaults to a regional /search Lambda for speed; the client-side range-read modes (split-set, monolith, term, semantic — no backend) are one toggle away. (Reproducible — see examples/openalex/.)

How it fits together

build (Go):   corpus ─roaringsearch─▶ FTSR ─roaringrange.Transcode─▶ RRS (.rrs) ─▶ S3/CDN
              (optional) facets ──────────────roaringrange.WriteFacets─▶ RRSF (.rrf) ─▶ S3/CDN
build (Rust): corpus ──build::write_index / write_facets / write_records──▶ .rrs + .rrf + records
browser (Rust/WASM): .rrs/.rrf/records on CDN ─HTTP Range─▶ Catalog (= Index + FacetIndex + RecordStore)

Doc IDs are assigned at build time in descending static rank (citations, holdings, …), so ascending doc ID = rank. Each posting is one roaring bitmap read in rank-ordered buckets: a query ANDs the small top bucket (the 65,536 top-ranked docs) to get the ranked top-K and only pages deeper buckets when paginating past it. See docs/format.svg and docs/search.svg.

This is the core design trade-off: ranking is baked in up front as a static rank in doc-ID order, in exchange for near-constant per-query fetch cost. (Term and hybrid modes add optional, default-on BM25 lexical relevance via the RRSB .rrb sidecar.)

How it compares

Inspired by lunr.js and Pagefind — both deliver full-text search to static sites with no backend. lunr loads the whole index into memory; Pagefind pioneered fetching only the index shards a query needs. roaringrange pushes that "fetch only what you query" idea to millions of records by HTTP-Range-reading a single roaring-bitmap index file, trading a general query-time relevance pipeline for a baked-in static rank (with optional default-on BM25 on the term/hybrid modes).

lunr.js Pagefind roaringrange
backend none none none
index transport whole index in memory many shard files, per query one file, HTTP Range
typical scale hundreds–few thousand docs up to ~100k+ pages millions–100M+ records
per-query bytes 0 after full load (load can be MBs+) ~tens–hundreds KB ~KB–few MB (≈ constant)
matching stemmed terms; wildcard, fuzzy, boosts stemmed words; partial trigram substring; fuzzy (tolerate-N)
ranking TF-IDF / BM25 relevance BM25-like relevance static rank (query-independent) in doc-ID order, plus default-on BM25 on term/hybrid (RRSB sidecar)
facets / filters fielded search (no facet counts) filters + facet counts facets + live counts (sidecar)
build input JS objects / prebuilt JSON crawls built HTML pages any records (Go via roaringsearch, or the Rust builder)
sweet spot embedding a search library in a JS app (you own the index) static sites & docs, small to large very large catalogs & datasets

In short: lunr.js when you want to embed a search library and control indexing in code; Pagefind for static-site search from small to large; roaringrange when you have a lot of records and want a single range-fetched file with static-rank ordering and facets.

Repository layout

path what
*.go (root) core Go module (github.com/freeeve/roaringrange — go get github.com/freeeve/roaringrange): Transcode (FTSR→RRS), Open/Index reference reader, WriteFacets, NgramKeys
*.md (root) the frozen on-disk format specs — see the On-disk formats table below
rust/ Rust crate roaringrange: reader (Catalog over Index/FacetIndex/RecordStore) + native build writers; both exposed to WASM (wasm-pack). Optional vector feature adds the RRVI similarity-search reader + IVFPQ trainer
rust/examples/ runnable examples, each with a //! header stating its purpose and exact cargo run … --example … [--features …] command
python/ PyO3 bindings (pip install roaringrange): Builder (text index) + VectorBuilder (similarity index) over the core build writers
conformance/ cross-library test: roaringsearch build ⇄ roaringrange(go) read must agree
examples/openalex/ the OpenAlex demo: Go loader, parallel Rust builder/, download.sh, static web UI
docs/ architecture diagrams (SVG)

On-disk formats

roaringrange is an à-la-carte family of composable index files over one shared rank-ordered doc-ID space. Each has a 4-char magic; the magic and the file extension are intentionally not always identical (records and the re-rank sidecar use app-facing names). Every format's reader and builder are listed below.

Magic Ext Spec Reader Builder What
RRSI .rrs FORMAT.md Index / RrsIndex Rust write_index, Go Transcode trigram text index
RRSF .rrf FACETS.md FacetIndex / RrfFacets Rust write_facets, Go WriteFacets facet sidecar
RRSR .idx+.bin(+.dict) RECORDS.md RecordStore / RrsRecords Rust write_records, Go WriteRecords record store (paired files; optional zstd dict)
RRIL .rril LOOKUP.md Lookup / RrsLookup Rust write_lookup identifier exact-match index
RRSC .rrsc SORTCOLS.md SortCols / RrsSortCols Rust write_sortcols static sort/rank columns
RRTI .rrt TERMS.md TermIndex / RrtIndex Rust write_term_index, Python term index (blocked, range-fetched dict)
RRSB .rrb BM25.md ImpactIndex / RrbIndex Rust write_impacts BM25 impact sidecar for the term index (lexical relevance)
RRVI .rrvi VECTORS.md VectorIndex / RrviIndex Rust build_ivfpq, Python IVFPQ vector index
RRVR .rrvi.rerank VECTORS.md RerankStore Rust write_rerank bf16 re-rank sidecar
RRM2 .rrm2 VECTORS.md Model2vec / Model2vecEmbedder python/scripts/model2vec_export.py in-browser query embedder
RRHC .rrhc HOTCACHE.md Hotcache Rust write_hotcache boot bundle (manifest + inlined members)
RRSS .rrss SPLITSET.md SplitSet / RrssIndex Rust SplitSetBuilder, Go, Python split-set manifest (splits are .rrs/.rrt)
Which builder/reader exists in which language

Readers ship as wasm/JS (the browser is the only reader runtime); builders run server-side in Rust, Go, or Python. Rust is the reference — Go and Python expose deliberate subsets.

Capability (format) Rust Go Python JS (read)
Trigram index RRSI build + read build build read
Facets RRSF build + read build build read
Records RRSR build + read build build read
Lookup RRIL build + read — — read
Sort columns RRSC build + read — — read
Terms RRTI build + read — build read
BM25 impacts RRSB build + read — — read
Vectors RRVI build + read — build read
Model2vec RRM2 read — export¹ read
Hotcache RRHC build + read — — —²
Split set RRSS (trigram) build + read build³ build read
Split set RRSS (term) build + read — build read

¹ via python/scripts/model2vec_export.py.  ² no JS reader yet (server-side only).  ³ trigram bodies only (Go builds the split set + per-split facet sidecars, but not RRTI term bodies).

The core Go module has no dependency on roaringsearch — it parses the FTSR byte format directly. The n-gram key derivation is reproduced independently in Go (here), Go (roaringsearch), and Rust (the reader); the go/conformance/ module is the guard that keeps all three byte-compatible.

Command-line inspector

roaringrange is a CLI that inspects any of these files: it auto-detects the format from the 4-byte magic and dumps or converts the contents to JSON.

# macOS / Linux (Homebrew):
brew install freeeve/tap/roaringrange

# any platform with a Go toolchain:
go install github.com/freeeve/roaringrange/cmd/roaringrange@latest

# or download a prebuilt binary (macOS/Linux/Windows, amd64/arm64) from
# https://github.com/freeeve/roaringrange/releases

roaringrange info    index.rrt                 # header summary for any format
roaringrange dump    facets.rrf --json         # full structural dump as JSON
roaringrange dump    terms.rrt  --postings --limit 20
roaringrange records store.idx store.bin --dict store.dict --id 0
roaringrange get     terms.rrt  --term roaring # look up one key's posting
roaringrange get     terms.rrt  --prefix roar  # prefix-matched terms + union posting

info and dump work on every format (RRSI, RRSR, RRTI, RRVI, RRVR, RRSF, RRSB, RRSS, RRHC, RRSC, RRIL); --postings includes bitmap/vector contents, --limit/--offset page large tables.

Quick start

Build an index — two paths to the same files. Assign doc IDs in descending static rank first, so the head holds the top-K.

Rust (direct): split each posting into head/tail, then write the index + an optional facet sidecar + record store:

use roaringrange::build::{write_index, write_facets, write_records, split_posting};
let entries = postings.iter()
    .map(|(k, bm)| { let (h, t) = split_posting(bm); (*k, h, t) })
    .collect();
write_index(rrs_w, 3, 0, entries)?;            // → .rrs
write_facets(rrf_w, facet_fields)?;            // → .rrf  (optional)
write_records(bin_w, idx_w, &records)?;        // → record store (optional)

For a corpus whose index exceeds RAM, build it in doc-ID-range chunks and fold them into one standard .rrs with build::chunk::{write_partial, merge_partials_to_rrs}.

Go (transcode a roaringsearch index):

rr.Transcode(ftsrReader, rrsWriter)             // FTSR → .rrs
rr.WriteFacets(rrfWriter, []rr.FacetField{...}) // → .rrf  (optional)
rr.WriteRecords(binW, idxW, records)            // → record store (optional)

Read it (Rust/WASM): build the reader, then open a Catalog and search:

cd rust && wasm-pack build --target web --features wasm

import init, { RrsCatalog } from "./roaringrange.js";
await init();
const cat = await RrsCatalog.openAll("index.rrs", "index.rrf", "records.idx", "records.bin");
const page = await cat.search("query", 0, 25, 0, '[["type","article"]]');
// page.ids = ranked doc IDs · page.records · page.facetCounts

Catalog/Index/FacetIndex/RecordStore (and RrsIndex/RrsRecords) stay available for advanced use. Host the files on anything that supports HTTP Range (S3 + CloudFront works well) and point the reader at the URLs.

Measured (full corpora, range-fetched)

library catalog 9.6M OpenAlex 47.8M (with abstracts)
text index (.rrs) 1.4 GB 11.5 GB
boot — index sparse ~52 KB ~0.5 MB
typical query tens–hundreds of KB ~1–6 MB head+tail (less head-only)
compute 2–14 µs —

Boot and per-query cost stay ~constant as the corpus grows; size lives in the postings (≈0.4 bytes per trigram-document incidence — roaring is near-optimal), so the lever for a smaller index is indexing less text per doc, not the encoding.

The Rust builder reaches the 47.8M-work OpenAlex corpus that backed the earlier demo — an 11.5 GB index of 30.3M unique trigrams, built in ~57 min at ~52 GB peak RAM with no swap — and sublinearly (≈half the naive linear projection), as the trigram vocabulary saturates and roaring absorbs the added postings. The current 484M-work demo index is 113 GB over 114.6M trigrams, built by the chunked/phased builder. With facets and records attached, a full boot is a few MB: the index sparse, the facet metadata + top-category heads (the facet tails stay range-fetched, not loaded up front), and the record-store header.

Costs — server default vs. client-side range reads

The unit of cost is bytes moved per query — the demo's link is bandwidth-bound (measured ~1–3.5 MB/s down, ~150–200 ms RTT), so per-query bytes, not CPU, set wall time. Storage is nearly free: the full 484M index family — trigram monolith + term + vector + records + facet sidecars + the geometric trigram (19-tier) and term (12-tier) split sets + the BM25 .rrb sidecar (plus the legacy flat split set still parked on S3) — is 550 GB ≈ **$13/month**, and there is no idle cost: nothing runs when nobody searches. The demo defaults to a regional /search Lambda for trigram (~1–3 KB, ~0.66 s, faceting included); the client-side range-read modes — searchable with no backend at all — are one toggle away, and that's where bytes climb:

mode (484M, warm) per query one-time resident boot
trigram — server (default) ~1–3 KB none
trigram — client geo split ~240–300 KB ~1.5 KB manifest
trigram — client monolith ~0.87–1.07 MB ~1.7 MB
term — client geo split ~40–270 KB ~1 KB manifest
term — client monolith ~15–20 KB ~76 MB (resident dict)
semantic (8 IVFPQ probes) ~2–9.6 MB ~64 MB (vector + embedder)
records page (25 cards) ~25–50 KB —
client facet filter (membership) ~tens of KB —
server facet filter ~2 KB (+ exact counts) none

Client bytes drop via pruning (read only the tiers that can match) and geometric tiering caps a worst-case descent at ~log-many visits. Faceting is cheap client-side too: the demo post-filters the ranked candidates with a membership read of the selected category — only the 64K-doc buckets the candidates occupy (container-granularity seeks on the .rrf, the same offset-table seek the trigram tail scan uses), not the whole category bitmap — so a category that runs to tens of MB whole costs ~tens of KB here, and counts come from the resident facet heads with no fetch. The server path's facet edge is exact totals + counts, not bytes. Behind a CDN with a real free tier (CloudFront: 1 TB + 10M req/mo), the monthly bill — baselined on the server default (~2 KB/query) and a representative client query (geo split, ~0.3 MB / ~30 range-GETs):

queries/mo server default (this) client range reads always-on box managed search
10k–100k ~$13 (inside free tier) ~$13 ~$100–150 flat $700+
1M ~$20 ~$30–50 ~$100–150 $700+
10M ~$110–150 ~$300–400 ~$150+ $1,500+

Honest conclusions:

  • Below a few hundred thousand queries a month, nothing is cheaper — or lower-ops. Server or client, you're inside the CDN free tier with no box to babysit, and the artifacts are immutable (the demo still works untouched years later). In the in-browser semantic mode the query text never leaves the browser at all.
  • For interactive latency on a modest connection, the server path wins on the heavy modes — KB instead of the client trigram-monolith's ~1 MB or semantic's MBs, and no 64–76 MB resident boots; faceting is cheap either way (membership reads), so the server's facet edge is exact counts, not bytes. The client-side range-read modes (examples/search-lambda is the server side) stay to demonstrate the no-backend story and its tradeoffs, not as the speed default.
  • The sweet spot is large corpus × modest traffic, and it widens as the corpus shrinks. Per-query bytes scale with the corpus, so a smaller index makes the client-side path cheaper and the CDN free tier go further; the demo runs the full 484M corpus.

The two paths compose — the same artifacts serve both — so the demo's production shape is now server-side by default with the client-side range-read modes as the no-backend option. The dictionary records every posting's byte size, so a client can estimate a query's cost before fetching anything and auto-route the expensive ones to the Lambda.

Tried and shelved

Experiments the data argued against — recorded so we don't relitigate them.

Inverting common-trigram postings

Idea: a very common trigram has a huge posting, but its complement (the docs that lack it) is small — so store the complement plus a one-bit "inverted" flag and ANDNOT it during intersection, cutting the bytes a query fetches.

Why it didn't pan out: roaring stores each 65,536-doc block as either an array (≤4096 docs, 2 B each) or a flat 8 KB bitmap for any cardinality in (4096, 61440]. A common trigram's complement usually lands in that same band, so it is also an 8 KB bitmap — inversion only shrinks a block denser than ~94% (complement becomes a small array, or empty). Measured on the earlier 47.8M index (rust/examples/density, e.g. cargo run --release --example density -- "machine learning" "posthuman became"): the hottest trigram, the, is only ~52% dense — OpenAlex lacks an abstract for roughly half its works, so half the corpus is title-only text — and just 0.0–0.6% of posting bytes sit in >94%-dense blocks. Net saving across those queries: ~0.1%. Not worth a format-version bump + reindex.

It would pay on a corpus where common trigrams are near-universal (full text with an abstract on every doc); here it's the partial abstract coverage that flattens the density. The density example re-runs the analysis on any index/query.

Rarest-trigram candidates + verify

Idea: skip the common trigrams' multi-MB postings entirely — seed candidates by intersecting only the rarest trigrams, then verify each against its record's stored text (title + abstract + authors + venue), keeping the true matches.

Why it didn't pan out: client-side, egress is floored by the result-set size. The candidate set can't shrink below the number of results, and verifying that many records costs ~result_count × record_size. Measured on the earlier 47.8M index (rust/examples/candidates): machine learning (171k results) still has ~195k candidates after seeding the 4 rarest trigrams → ~190 MB of record verification, worse than the 53 MB full intersection. It helps only sparse results (posthuman became, 317 → ~12 MB vs 30 MB), which the lazy tail already gates behind an explicit load. Index::search_candidates and the candidates example stay for re-measuring.

Returning just result IDs needs the intersection to run server-side (a thin Lambda@Edge over in-region postings) — the one lever that beats the result-set floor.

Vector / similarity search (optional)

Beyond trigram text search, the crate has a range-fetchable similarity index in the same ethos: an RRVI (IVFPQ) file whose coarse centroids and PQ codebooks boot once, after which each query range-fetches only the nprobe nearest clusters' codes and scans them with asymmetric distance computation — top-k nearest vectors in ~constant bytes, independent of corpus size. vector_id == doc_id, so a hit reuses the same record store and can hybridize with the trigram result set.

Build it from Rust (build_ivfpq, behind the vector feature) or Python (VectorBuilder), or train at scale with FAISS and export the same layout (build_ivfpq_from_parts / roaringrange.write_rrvi_from_faiss, verified against the reader at recall@10 ≈ 0.9995). The reader VectorIndex is pure Rust with a browser binding (RrviIndex, wasm-pack build --features "wasm vector"). See VECTORS.md. Live on the demo: the in-browser model2vec query embedder (RrviIndex + Model2vecEmbedder) and term/trigram hybrid (reciprocal-rank fusion); an optional EmbeddingGemma Lambda embedder is parked.

Development

Enable the formatting pre-commit hook (runs gofmt + cargo fmt --check on staged changes, matching CI):

git config core.hooksPath .githooks

CI runs go test ./... in go/, the go/conformance/ module, go vet on the example, cargo test + fmt + clippy for the reader (a second pass with --features vector), and builds + tests the Python extension on CPython 3.12–3.14.

License

MIT — see LICENSE.

Documentation

Overview

Package roaringrange turns a roaringsearch (FTSR) index into a range-fetchable static index (RRS, see FORMAT.md) that a browser can query over HTTP Range requests with no backend.

The layout puts the whole n-gram dictionary in one contiguous, key-sorted block so a reader loads a sparse view in a single ranged GET, then fetches each posting by absolute byte offset. Postings are byte-identical portable RoaringBitmaps, so transcoding copies them verbatim and the Rust/WASM reader deserializes them with the `roaring` crate. Optional faceting is layered on via a companion sidecar (see FACETS.md).

Index

Constants

View Source 
const (

	// DefaultK1 / DefaultB are the BM25 parameters (match the Rust DEFAULT_K1/_B).
	DefaultK1 float32 = 1.2
	DefaultB  float32 = 0.75
)
View Source 
const (
	// MagicFacet is the RRSF facet sidecar magic.
	MagicFacet = "RRSF"
	// VersionFacet is the RRSF format version number.
	VersionFacet = 1
)
View Source 
const (
	// MagicRecord is the RRSR record-store index magic.
	MagicRecord = "RRSR"
	// VersionRecord is the RRSR format version this package's writer emits
	// (untagged raw records).
	VersionRecord = 1
)
View Source 
const (
	// MagicSplitSet is the RRSS split-set manifest magic.
	MagicSplitSet = "RRSS"
	// VersionSplitSet is the RRSS format version number.
	VersionSplitSet = 1

	// PolicyTiered assigns docs to splits by rank (top-cited first).
	PolicyTiered = 0
	// PolicyStableKey assigns docs by ingest order; rank is a query-time RRSC column.
	PolicyStableKey = 1

	// BodyKindTrigram marks split data files as trigram RRS indexes (header byte 9; the
	// default, so older manifests read back as trigram). Go only builds trigram bodies.
	BodyKindTrigram = 0
	// BodyKindTerm marks split data files as term RRTI (FST) indexes (header byte 9).
	BodyKindTerm = 1

	// SplitSetFlagBloom marks per-split term Bloom-filter summaries present (header flag).
	SplitSetFlagBloom = 1 << 0
	// SplitSetFlagFacet marks per-split facet-presence summaries present (header flag).
	SplitSetFlagFacet = 1 << 1
	// SplitSetFlagTime marks per-split time min/max summaries present (header flag).
	SplitSetFlagTime = 1 << 2
	// SplitSetFlagTombstones marks per-split tombstone postings present (header flag).
	SplitSetFlagTombstones = 1 << 3
	// SplitSetFlagCaseSensitive marks a case-sensitive split set: n-gram and facet keys were
	// not lowercased, so a query derives keys without folding. Unset (the default) keeps every
	// manifest byte-identical. Mirrors the Rust splitset::FLAG_CASE_SENSITIVE.
	SplitSetFlagCaseSensitive = 1 << 4

	// SplitFlagHasTombstone marks a split that carries a tombstone posting (per-split flag).
	SplitFlagHasTombstone = 1 << 0
	// SplitFlagAbsoluteIDs marks a split that stores absolute global doc IDs (per-split flag).
	SplitFlagAbsoluteIDs = 1 << 1

	// SortColFlagDescending marks a descending rank sort-column (higher value = better rank).
	SortColFlagDescending = 1 << 0
)
View Source 
const (
	// Magic is the RRS index magic.
	Magic = "RRSI"
	// Version is the RRS format version number.
	Version = 3

	// VersionV4 is the RRS format version emitted only for a case-sensitive index: identical
	// to v3 plus a trailing flags u16 at offset 16. Default (case-folding) builds stay v3.
	VersionV4 = 4

	// DefaultStride is the default sparse-index stride.
	DefaultStride = 512
)

Variables

View Source 
var (
	// ErrSrcMagic is returned when the source is not a roaringsearch (FTSR) index.
	ErrSrcMagic = errors.New("source is not a roaringsearch (FTSR) index")
	// ErrMagic is returned when an index does not start with the RRS magic.
	ErrMagic = errors.New("bad RRS magic")
	// ErrVersion is returned when an index's format version is unsupported. The RRS
	// reader is v3-only, matching the Rust reference reader; a v2 file has a different
	// header and dictionary layout and would otherwise misparse silently.
	ErrVersion = errors.New("unsupported RRS format version")
	// ErrTruncated is returned when an index ends before its declared structure.
	ErrTruncated = errors.New("truncated index")
	// ErrCompressedRecord is returned when a version-2 record store holds a
	// zstd-compressed frame (tag 1) but the store was opened without a dictionary
	// decoder — open it with OpenRecordStoreWithDict to inflate compressed records.
	ErrCompressedRecord = errors.New("compressed record (zstd frame) requires opening the store with OpenRecordStoreWithDict")
)

Functions

func FacetKey 

func FacetKey(field, category string) uint64

FacetKey derives the category key as FNV-1a 64-bit over lower(field), a 0x1f separator, and lower(category). See FACETS.md. Equivalent to FacetKeyWith with case folding on (the default).

func FacetKeyWith added in v0.25.0 

func FacetKeyWith(field, category string, caseFold bool) uint64

FacetKeyWith is FacetKey with an explicit case-fold flag. When caseFold is false the field/category are hashed verbatim (a case-sensitive index keeps "Smith" and "smith" distinct). Build and pruning sides must pass the same mode. Mirrors the Rust facet_key.

func NgramKeys 

func NgramKeys(query string, gramSize int) []uint64

NgramKeys derives the deduplicated n-gram keys for a query. The query is split on whitespace and each word is keyed independently (normalize: keep Unicode letters/digits, lowercase; then key each gramSize-rune window), unioning keys in first-seen order. Per-word keying avoids cross-word boundary trigrams — a query like "legends travis" must not require "dst" from legend·s·t·ravis. Mirrors roaringsearch's per-word query matching. See FORMAT.md.

The key derivation here must stay byte-compatible with roaringsearch's builder (and the Rust reader's port); the cross-library test in ./conformance is the guard that enforces it.

func NgramKeysWith added in v0.25.0 

func NgramKeysWith(query string, gramSize int, caseFold bool) []uint64

NgramKeysWith is NgramKeys with an explicit case-fold flag. When caseFold is false the kept letters/digits are not lowercased, so a case-sensitive trigram index keys on the original case. The builder and reader must agree; the choice is recorded in the RRSI header (a v4 flags field) so the reader derives the same caseFold. See FORMAT.md.

func QuantizeImpact added in v0.24.0 

func QuantizeImpact(tf uint32, dl, avgdl, k1, b float32) byte

QuantizeImpact quantizes one (term, doc) BM25 impact to a byte (1–255, never 0 — presence in the posting implies a nonzero contribution): the tf component with the document-length norm folded in, scaled by k1+1. The arithmetic is float32 throughout and rounds half-away-from-zero, matching the Rust quantize_impact byte-for-byte (f32→f64 widening is exact, and math.Round uses the same rounding rule as f32::round). The saturating float→int + clamp mirrors Rust's `as i64`.

func TermTokenize added in v0.24.0 

func TermTokenize(text string) []string

TermTokenize is the base RRTI tokenizer (no stemming / stop words): maximal runs of Rust-alphanumeric runes, lowercased with the full per-rune mapping. Mirrors the Rust terms::tokenize. Equivalent to TermTokenizeWith with case folding on (the default).

func TermTokenizeWith added in v0.25.0 

func TermTokenizeWith(text string, caseFold bool) []string

TermTokenizeWith is the base tokenizer with an explicit case-fold flag. When caseFold is false the token runes are kept verbatim (a case-sensitive index); the boundary rule (maximal Rust-alphanumeric runs) is unchanged. Mirrors the Rust terms::tokenize_with.

func Transcode 

func Transcode(src io.Reader, dst io.Writer) error

Transcode reads a roaringsearch FTSR index from src and writes the v3 RRS range-fetchable layout to dst using the default stride. Each term is written as one portable RoaringBitmap posting. Entries are sorted by key ascending. See FORMAT.md.

func TranscodeStride 

func TranscodeStride(src io.Reader, dst io.Writer, stride int) error

TranscodeStride is Transcode with an explicit sparse-index stride. A stride of zero or less is replaced by DefaultStride.

func WriteFacets 

func WriteFacets(dst io.Writer, fields []FacetField) error

WriteFacets writes the RRSF facet sidecar for the given fields to dst. Each category posting is split into head (docs [0,65536)) and tail (docs [65536, ∞)) portable RoaringBitmaps, mirroring the text index. Categories are grouped by field and sorted by key within a field. See FACETS.md.

func WriteFacetsWith added in v0.25.0 

func WriteFacetsWith(dst io.Writer, fields []FacetField, caseNormalization bool) error

WriteFacetsWith is WriteFacets with an explicit caseNormalization flag. true (the default) lowercases field/category for the facet-key hash and writes a byte-identical v1 sidecar; false keys on the raw bytes (a case-sensitive index) and sets the reserved-field flag so a split-set's facet pruning recomputes keys identically. Category display names are stored verbatim either way. Mirrors the Rust write_facets_with.

func WriteHotcache added in v0.24.0 

func WriteHotcache(dst io.Writer, members []MemberSpec, inlineThreshold uint32) error

WriteHotcache writes the RRHC bundle for members to dst — byte-for-byte with the Rust write_hotcache. A member whose boot fits inlineThreshold is copied into the inlined-boot blob (returned free with the single GET); larger boots are referenced by (BootOff, BootLen) in the member's own data file.

func WriteImpacts added in v0.24.0 

func WriteImpacts(dst io.Writer, dict []DictEntry, acc *ImpactsAccumulator, k1, b float32) error

WriteImpacts writes the RRSB sidecar to dst for a finished .rrt whose dictionary is dict over the stats in acc — byte-for-byte with the Rust write_impacts. Every dict term must have accumulated stats (else error — a tokenizer-config mismatch would otherwise mis-address every later term). k1/b are the BM25 parameters used.

PERFORMANCE: this emits many small (~20-byte) writes to dst; pass a buffered writer (e.g. bufio.Writer) when dst is a file or socket and flush it after. The library does not buffer internally.

func WriteIndex added in v0.24.0 

func WriteIndex(dst io.Writer, gramSize uint16, stride int, entries []IndexEntry) error

WriteIndex writes a v3 RRS (RRSI) trigram index over key->posting entries to dst — the byte-for-byte Go mirror of the Rust build::write_index. Entries are sorted by key here (so the caller need not pre-sort), and a stride of zero or less becomes DefaultStride. Each posting must be one portable RoaringBitmap (see roaring.Bitmap.ToBytes). See FORMAT.md.

func WriteIndexWith added in v0.25.0 

func WriteIndexWith(dst io.Writer, gramSize uint16, stride int, entries []IndexEntry, caseNormalization bool) error

WriteIndexWith is WriteIndex with an explicit caseNormalization flag. true (the default) emits a v3 index byte-identical to before; false emits a case-sensitive v4 index (the caller must have keyed entries with NgramKeysWith(.., false)). Mirrors the Rust build::write_index_with.

func WriteLookup added in v0.24.0 

func WriteLookup(dst io.Writer, entries []LookupEntry) error

WriteLookup writes the RRIL index over entries to dst — byte-for-byte with the Rust write_lookup. Identifiers are normalized and double-hashed, empties dropped, and the records sorted by (hash, doc) with a stable sort (matching Rust's stable sort_by on equal-hash, equal-doc ties).

PERFORMANCE: this emits many small (~16-byte) writes to dst; pass a buffered writer (e.g. bufio.Writer) when dst is a file or socket and flush it after. The library does not buffer internally.

func WritePerm added in v0.24.0 

func WritePerm(dst io.Writer, primaryOfSecondary []uint32) error

WritePerm writes a one-column u32 RRSC store named "primary" mapping a secondary doc-ID space back to the primary one (primaryOfSecondary[secondary] = primary) — byte-for-byte with the Rust write_perm.

func WriteRRVI added in v0.24.0 

func WriteRRVI(dst io.Writer, m *ivfpq.Model) error

WriteRRVI serializes a trained IVFPQ model to dst in the RRVI byte layout the Rust VectorIndex (and the wasm reader) read over HTTP Range — the byte-for-byte mirror of the Rust Ivfpq::write. Layout (all little-endian): a 48-byte header, an optional OPQ rotation, the coarse centroids, the PQ codebooks, an nlist×12 cluster directory (absolute list offset + count), then each cluster's [ids][codes]. See VECTORS.md.

func WriteRecords added in v0.24.0 

func WriteRecords(bin, idx io.Writer, records [][]byte) error

WriteRecords writes a record store from an in-memory slice of records, in doc-ID order — a convenience over RecordWriter for callers that already hold every record. See RECORDS.md.

func WriteRecordsZstd added in v0.24.0 

func WriteRecordsZstd(bin, idx io.Writer, records [][]byte, dict []byte) error

WriteRecordsZstd writes a version-2 (framed) RRSR record store that compresses each record against the shared dict, in doc-ID order — the klauspost counterpart of WriteRecords. Each record is framed [tag][payload]: the smaller of a zstd frame (tag 1) and the raw bytes (tag 0), so a record never grows; a zero-length record stays zero-length (no tag), matching the version-1 convention. The same dict must be shipped to the reader as the *.dict sidecar and passed to OpenRecordStoreWithDict. The frame bytes differ from the Rust libzstd builder but read back identically through either reader.

func WriteRerank added in v0.24.0 

func WriteRerank(dst io.Writer, dim int, vectors [][]float32, l2Normalize bool) error

WriteRerank writes the RRVR bf16 re-rank sidecar (read by the Rust RerankStore): a 20-byte header then a dense bf16 array of vectors keyed by doc ID (slice index == doc ID). Every vector must have length dim. Set l2Normalize for an inner-product index so the stored vectors match the unit-sphere space the index was built in. Byte-for-byte with the Rust write_rerank.

func WriteSortcols added in v0.24.0 

func WriteSortcols(dst io.Writer, cols []SortColumn) error

WriteSortcols writes the RRSC store for cols to dst — byte-for-byte with the Rust write_sortcols. Every column must hold the same number of values (one per doc).

func WriteSplitSet added in v0.24.0 

func WriteSplitSet(w io.Writer, splits []SplitSpec, config SplitSetConfig) error

WriteSplitSet writes the RRSS manifest for splits to w, in the order given (base splits first, then delta splits — BaseCount marks the boundary). The output is byte-for-byte identical to the Rust writer for the same inputs. See SPLITSET.md.

PERFORMANCE: this emits many small (~56-byte) writes to w; pass a buffered writer (e.g. bufio.Writer) when w is a file or socket and flush it after. The library does not buffer internally.

func WriteSplitsetBundle added in v0.33.0 

func WriteSplitsetBundle(dst io.Writer, built *BuiltSplitSet, maxSplits int, inlineThreshold uint32) error

WriteSplitsetBundle writes an .rrhc boot bundle over built to dst — byte-for-byte with the Rust write_splitset_bundle: one inlined member per split (in seal/rank order, so the top tiers come first), each carrying its split's boot region. Both split bodies are handled by their leading magic: a trigram RRS boot (header + sparse index) or a term RRTI boot (header + router FST).

maxSplits caps how many splits are inlined (0 = all): a corpus with a large top tier inlines only the splits a top-K query is likely to open, keeping the first GET small. inlineThreshold is forwarded to WriteHotcache: a split whose boot exceeds it is referenced rather than inlined, so its boot lookup misses and it simply cold-opens — the bundle never forces a fat first GET. Errors only on a malformed split header or an I/O failure.

func WriteTermIndex added in v0.24.0 

func WriteTermIndex(dst io.Writer, postings map[string]*roaring.Bitmap, headBoundary uint32, lang TermLanguage, stopwords bool, blockCap int) error

WriteTermIndex writes an RRTI v2 term index over postings (term → bitmap of the shared rank-order doc IDs) to dst — byte-for-byte the Rust write_term_index_from_postings. headBoundary is the head/tail doc-ID split (a multiple of 65536); language/stopwords are recorded in the header so the reader tokenizes queries identically; blockCap of 0 takes the default.

func WriteTermIndexFull added in v0.26.0 

func WriteTermIndexFull(dst io.Writer, postings map[string]*roaring.Bitmap, headBoundary uint32, lang TermLanguage, stem, stopwords, caseNormalization bool, blockCap int) error

WriteTermIndexFull writes an RRTI v2 term index with independent stem and stopwords filters over a shared language (mirrors the Rust write_term_index_from_postings). stem controls the stemmed header flag; the language byte is recorded when either filter is on, and both filters require a language (a filter on with none set is an error). blockCap of 0 takes the default. It is a thin error-only wrapper over WriteTermIndexFullDict for source compatibility.

func WriteTermIndexWith added in v0.25.0 

func WriteTermIndexWith(dst io.Writer, postings map[string]*roaring.Bitmap, headBoundary uint32, lang TermLanguage, stopwords, caseNormalization bool, blockCap int) error

WriteTermIndexWith is WriteTermIndex with an explicit caseNormalization flag. true (the default) lowercases nothing differently and writes a byte-identical header; false records the case-sensitive flag (termFlagCaseSensitive) so the reader skips query-side folding. Back-compat: a non-None language implies stemming — use WriteTermIndexFull to control stemming independently (e.g. stop words without stemming).

Types

type BuiltSplitSet added in v0.24.0 

type BuiltSplitSet struct {
	Manifest []byte
	Splits   []NamedSplit
	Facets   []NamedSplit
}

BuiltSplitSet is the output of SplitSetBuilder.Finish: the manifest, each split's (filename, RRS bytes), and — for a faceted build — each split's (filename, RRSF bytes) facet sidecar. The caller writes them out; nothing is written here.

type ColumnMeta added in v0.29.0 

type ColumnMeta struct {
	Name string
	Type string
	Rows uint32
}

ColumnMeta is the public view of one column's shape for `info`/`dump`.

type ColumnValues added in v0.24.0 

type ColumnValues interface {
	Len() int
	// contains filtered or unexported methods
}

ColumnValues is one sort column's dense values in doc-ID order. The concrete type (U16Column / U32Column / I32Column / F32Column) selects the on-disk value type.

type DictEntry added in v0.24.0 

type DictEntry struct {
	Term    string
	HeadOff uint64
}

DictEntry is one (term, posting head_off) pair from the paired .rrt dictionary, in dictionary order (ascending head_off).

func WriteTermIndexFullDict added in v0.28.0 

func WriteTermIndexFullDict(dst io.Writer, postings map[string]*roaring.Bitmap, headBoundary uint32, lang TermLanguage, stem, stopwords, caseNormalization bool, blockCap int) ([]DictEntry, error)

WriteTermIndexFullDict is WriteTermIndexFull that additionally returns the dictionary it wrote: the (term, posting head_off) pairs in byte-lexicographic dictionary order, carrying the *real* head offsets front-coded into the .rrt. Feed the result straight to WriteImpacts to build a BM25 .rrb sidecar that addresses this exact .rrt--the offsets are the ones the reader recovers on lookup, not fabricated. The bytes written to dst are identical to WriteTermIndexFull; only the extra return value differs.

type F32Column added in v0.24.0 

type F32Column []float32

U16Column / U32Column / I32Column / F32Column are the four on-disk value types.

func (F32Column) Len added in v0.24.0 

func (c F32Column) Len() int

type FacetCatMeta added in v0.29.0 

type FacetCatMeta struct {
	Key         uint64
	Name        string
	Cardinality uint32
	// contains filtered or unexported fields
}

FacetCatMeta is one category's directory entry (without its posting bytes).

type FacetCategory 

type FacetCategory struct {
	Name   string
	Bitmap *roaring.Bitmap
}

FacetCategory is one category value within a field and the doc-ID posting of the documents carrying it.

type FacetField 

type FacetField struct {
	Name       string
	Categories []FacetCategory
}

FacetField is a named facet field with its categories. Mirrors one field of a roaringsearch BitmapFilter.

type FacetFieldMeta added in v0.29.0 

type FacetFieldMeta struct {
	Name     string
	CatStart uint32
	CatCount uint32
}

FacetFieldMeta is one facet field's directory entry.

type FacetIndex added in v0.29.0 

type FacetIndex struct {
	CaseSensitive bool
	// contains filtered or unexported fields
}

FacetIndex is a reference reader over an RRSF sidecar accessed by byte range. The directories are resident after Open; postings are fetched per Posting call.

func OpenFacets added in v0.29.0 

func OpenFacets(r io.ReaderAt) (*FacetIndex, error)

OpenFacets reads and validates the RRSF header and both directories. Postings are read lazily.

func (*FacetIndex) Categories added in v0.29.0 

func (f *FacetIndex) Categories() []FacetCatMeta

Categories returns every category's directory entry, in stored order (grouped by field, sorted by key within a field).

func (*FacetIndex) Category added in v0.29.0 

func (f *FacetIndex) Category(key uint64) (FacetCatMeta, bool)

Category returns the directory entry for a category key. Category keys are not globally sorted (only within a field), so this scans linearly.

func (*FacetIndex) Fields added in v0.29.0 

func (f *FacetIndex) Fields() []FacetFieldMeta

Fields returns every field's directory entry.

func (*FacetIndex) Posting added in v0.29.0 

func (f *FacetIndex) Posting(key uint64) (*roaring.Bitmap, bool, error)

Posting returns the merged doc posting for a category key, or ok=false if the key is absent.

func (*FacetIndex) ReadAll added in v0.29.0 

func (f *FacetIndex) ReadAll() ([]FacetField, error)

ReadAll reconstructs the []FacetField that WriteFacets consumes (postings included), so WriteFacetsWith(ReadAll(x), !x.CaseSensitive) reproduces x.

type Field added in v0.29.0 

type Field struct {
	Key string
	Val any
}

Field is one decoded header value, kept in an ordered slice so text output is deterministic (a map would randomize key order).

type FileInfo added in v0.29.0 

type FileInfo struct {
	Magic   string
	Name    string
	Ext     string
	Version uint16
	Fields  []Field
}

FileInfo is the format-agnostic header view produced by a Describe function: the magic, human name, extension, version, and the format-specific decoded header fields (counts, sizes, and decoded flag bits). It is what `roaringrange info` renders, in text or JSON, for any recognized file.

func OpenHeader added in v0.29.0 

func OpenHeader(r io.ReaderAt) (*FileInfo, error)

OpenHeader detects the format and returns its FileInfo, reading only the fixed header. It is the entry point for `roaringrange info` and works on every registered format without opening postings, vectors, or other bodies.

type Format added in v0.29.0 

type Format struct {
	Magic    string
	Name     string
	Ext      string
	Describe func(io.ReaderAt) (*FileInfo, error)
}

Format describes one recognized on-disk format: its 4-byte magic, a human name, the conventional extension, and a Describe that reads only the fixed header (not the body) and returns a FileInfo. Formats self-register via register in init.

func DetectFormat added in v0.29.0 

func DetectFormat(r io.ReaderAt) (Format, error)

DetectFormat reads the leading 4-byte magic and returns the matching Format. Returns ErrMagic when the magic is not a recognized roaringrange format (this includes the sidecar .dict, which has no magic of its own).

func Formats added in v0.29.0 

func Formats() []Format

Formats returns the registered formats sorted by magic, for help text and tests.

type Hotcache added in v0.29.0 

type Hotcache struct {
	// contains filtered or unexported fields
}

Hotcache is a reference reader over an RRHC bundle. The whole manifest is resident after Open.

func OpenHotcache added in v0.29.0 

func OpenHotcache(r io.ReaderAt) (*Hotcache, error)

OpenHotcache reads and validates the RRHC header and rehydrates every MemberSpec, inlining the boot bytes for members that were stored inline.

func (*Hotcache) Boot added in v0.29.0 

func (h *Hotcache) Boot(i int) []byte

Boot returns the i-th member's inlined boot bytes, or nil if the boot is referenced (not stored in the bundle) or i is out of range.

func (*Hotcache) Members added in v0.29.0 

func (h *Hotcache) Members() []MemberSpec

Members returns every member spec. Inlined members carry their BootBytes; referenced members have BootBytes == nil (their boot lives in DataFile at [BootOff, BootOff+BootLen)).

type I32Column added in v0.24.0 

type I32Column []int32

U16Column / U32Column / I32Column / F32Column are the four on-disk value types.

func (I32Column) Len added in v0.24.0 

func (c I32Column) Len() int

type ImpactHeader added in v0.29.0 

type ImpactHeader struct {
	Scale, K1, B, AvgDL float32
	TermCount           uint32
	DocCount            uint64
}

ImpactHeader is the RRSB scoring configuration.

type ImpactIndex added in v0.29.0 

type ImpactIndex struct {
	// contains filtered or unexported fields
}

ImpactIndex is a reference reader over an RRSB sidecar accessed by byte range. Only the header is resident; entries and impacts are fetched per lookup.

func OpenImpacts added in v0.29.0 

func OpenImpacts(r io.ReaderAt) (*ImpactIndex, error)

OpenImpacts reads and validates the RRSB header.

func (*ImpactIndex) Entry added in v0.29.0 

func (b *ImpactIndex) Entry(headOff uint64) (rel uint64, card uint32, ok bool, err error)

Entry finds the entry for a posting head_off by binary search over the ascending entries table. ok is false when no term maps to that head_off.

func (*ImpactIndex) EntryAt added in v0.29.0 

func (b *ImpactIndex) EntryAt(i int) (headOff, rel uint64, card uint32, err error)

EntryAt reads the i-th entry: its posting head_off, the byte offset of its impacts relative to the impacts region, and the number of docs (impact bytes).

func (*ImpactIndex) Header added in v0.29.0 

func (b *ImpactIndex) Header() ImpactHeader

Header returns the scoring configuration.

func (*ImpactIndex) Impacts added in v0.29.0 

func (b *ImpactIndex) Impacts(headOff uint64) ([]byte, bool, error)

Impacts returns the per-doc quantized impact bytes for a posting head_off, in the posting's doc order. ok is false when no term maps to that head_off.

func (*ImpactIndex) Len added in v0.29.0 

func (b *ImpactIndex) Len() int

Len reports the number of term entries.

type ImpactsAccumulator added in v0.24.0 

type ImpactsAccumulator struct {
	// contains filtered or unexported fields
}

ImpactsAccumulator gathers per-term (doc, tf) plus per-doc lengths over a corpus added in ascending doc-ID order (the shared rank order), for WriteImpacts. Mirrors the Rust bm25::build::ImpactsAccumulator: the per-term lists are ascending by doc by construction, matching posting iteration order.

func NewImpactsAccumulator added in v0.24.0 

func NewImpactsAccumulator(tok *TermTokenizer) *ImpactsAccumulator

NewImpactsAccumulator builds an accumulator over tok, which MUST match the .rrt build's tokenizer (same language / stop-word config) or the vocabularies diverge.

func (*ImpactsAccumulator) AddDoc added in v0.24.0 

func (a *ImpactsAccumulator) AddDoc(text string) uint32

AddDoc tokenizes text as the next sequential doc ID and returns that ID.

func (*ImpactsAccumulator) DocCount added in v0.24.0 

func (a *ImpactsAccumulator) DocCount() uint32

DocCount is the number of documents accumulated so far.

type Index 

type Index struct {

	// GramSize is the n-gram window the index was built with.
	GramSize int
	// CaseFold reports whether queries should lowercase their n-grams before keying
	// (false for a v4 case-sensitive index). Callers deriving keys should use
	// NgramKeysWith(query, GramSize, CaseFold) so they key exactly as the index was built.
	CaseFold bool
	// contains filtered or unexported fields
}

Index is a reference reader over a v3 RRS index accessed by byte range. It reads only the 16-byte header and the sparse index up front (mirroring the browser reader's boot ranged GETs); the dictionary and postings are fetched lazily, one ranged ReadAt per block, never in their entirety. See FORMAT.md.

func Open 

func Open(r io.ReaderAt) (*Index, error)

Open reads and parses the RRS header and sparse index via ranged reads. It does not read the dictionary or postings; those are fetched per lookup.

func (*Index) NgramCount 

func (s *Index) NgramCount() int

NgramCount returns the number of n-grams in the dictionary.

func (*Index) Posting added in v0.24.0 

func (s *Index) Posting(key uint64) (data []byte, ok bool, err error)

Posting returns the full posting bytes for key via one ranged dictionary read and one ranged posting read, or ok=false if key is absent.

func (*Index) Postings added in v0.27.0 

func (s *Index) Postings(keys []uint64) (map[uint64][]byte, error)

Postings returns the posting bytes for each present key, keyed by key. The dict blocks are read once per distinct sparse block (deduping keys that share a block), then one ranged read per present posting -- so an n-key query costs (distinct dict blocks + present postings) reads, not up to 2n. Absent keys are omitted from the map.

type IndexEntry added in v0.24.0 

type IndexEntry struct {
	Key     uint64
	Posting []byte
}

IndexEntry is one v3 RRS dictionary entry for WriteIndex: a trigram key and its single serialized portable-RoaringBitmap posting. The public mirror of the internal indexEntry.

type LookupEntry added in v0.24.0 

type LookupEntry struct {
	ID  string
	Doc uint32
}

LookupEntry is one (identifier, doc-ID) pair fed to WriteLookup.

type LookupIndex added in v0.29.0 

type LookupIndex struct {
	// contains filtered or unexported fields
}

LookupIndex is a reference reader over an RRIL index accessed by byte range. It reads only the 16-byte header up front; table rows are fetched per probe.

func OpenLookup added in v0.29.0 

func OpenLookup(r io.ReaderAt) (*LookupIndex, error)

OpenLookup reads and validates the RRIL header. Rows are read lazily.

func (*LookupIndex) Count added in v0.29.0 

func (l *LookupIndex) Count() int

Count reports the number of table rows.

func (*LookupIndex) Entries added in v0.29.0 

func (l *LookupIndex) Entries() ([]LookupRecord, error)

Entries reads and returns every table row in stored (hash, doc) order. The whole table is read in one bounded range read.

func (*LookupIndex) Lookup added in v0.29.0 

func (l *LookupIndex) Lookup(id string) ([]uint32, error)

Lookup returns the doc ID(s) carrying the given identifier. The identifier is normalized and hashed exactly as WriteLookup did (reusing normalizeID/idFNV64a), the table is binary-searched by the primary hash, and equal-hash rows are kept only when the verify hash also matches. Returns nil (not an error) for a miss.

type LookupRecord added in v0.29.0 

type LookupRecord struct {
	Hash   uint64
	Verify uint32
	Doc    uint32
}

LookupRecord is one raw table row: the primary and verify hashes and the doc ID. The source identifier is not stored (it is hashed away), so a dump exposes the hashes rather than the original string.

type MemberSpec added in v0.24.0 

type MemberSpec struct {
	Tag       MemberTag
	DataFile  string
	BootOff   uint64
	BootLen   uint32
	BootBytes []byte
}

MemberSpec describes one hotcache member: its format tag, the data-file name its per-query reads go to, the boot byte-range within that file, and the boot bytes themselves (so the writer can inline or measure them). BootLen must equal len(BootBytes).

type MemberTag added in v0.24.0 

type MemberTag uint16

MemberTag is a hotcache member's format type (the on-disk u16 tag) — values match the Rust hotcache::MemberTag::to_u16. 

const (
	MemberRrs      MemberTag = 1
	MemberRrti     MemberTag = 2
	MemberRrsf     MemberTag = 3
	MemberRrvi     MemberTag = 4
	MemberRrsrIdx  MemberTag = 5
	MemberRrsrBin  MemberTag = 6
	MemberRrsrDict MemberTag = 7
	MemberRril     MemberTag = 8
	MemberRrm2     MemberTag = 9
	MemberRrss     MemberTag = 10
)

type NamedSplit added in v0.24.0 

type NamedSplit struct {
	Name  string
	Bytes []byte
}

NamedSplit is one emitted split: its filename and RRS bytes.

type NgramKeyer added in v0.27.0 

type NgramKeyer struct {
	// contains filtered or unexported fields
}

NgramKeyer derives n-gram keys while reusing its scratch buffers across calls, for the per-document ingest hot path where NgramKeysWith's fresh map + rune slice per call cost hundreds of millions of allocations over a full 484M-doc build. Not safe for concurrent use; hold one per builder.

func (*NgramKeyer) Keys added in v0.27.0 

func (k *NgramKeyer) Keys(query string, gramSize int, caseFold bool) []uint64

Keys returns the deduplicated n-gram keys for query, byte-identical to NgramKeysWith but reusing internal buffers. The returned slice aliases the keyer's buffer and is valid only until the next Keys call -- copy it if it must outlive that.

type RecordStore added in v0.24.0 

type RecordStore struct {
	// contains filtered or unexported fields
}

RecordStore is a reference reader over an RRSR record store accessed by byte range: an offset index (idx) over a record blob (bin). It mirrors the browser reader's access pattern — one ranged read of the 16-byte offset pair in the index, one ranged read of the record slice in the blob. See RECORDS.md.

func OpenRecordStore added in v0.24.0 

func OpenRecordStore(idx, bin io.ReaderAt) (*RecordStore, error)

OpenRecordStore boots the store: reads the 16-byte index header and validates magic and version. idx addresses the offset index, bin the record blob. Accepts version 1 (untagged raw records) and version 2 ([tag][payload]-framed records, matching the Rust reader); a version-2 zstd-compressed frame (tag 1) errors at Get — open with OpenRecordStoreWithDict (records_zstd.go) to attach a dictionary-backed zstd decoder.

func OpenRecordStoreWithDict added in v0.24.0 

func OpenRecordStoreWithDict(idx, bin io.ReaderAt, dict []byte) (*RecordStore, error)

OpenRecordStoreWithDict boots an RRSR store like OpenRecordStore but attaches a dictionary-backed zstd decoder, so version-2 zstd frames (tag 1) inflate at Get — the mirror of the Rust RecordStore::open_with_dict. A version-1 (raw) store ignores the dictionary. The dict must be the *.dict sidecar the store was built against (Rust libzstd or the Go WriteRecordsZstd builder — either reads here).

func (*RecordStore) Get added in v0.24.0 

func (s *RecordStore) Get(id uint32) (data []byte, ok bool, err error)

Get returns the raw record bytes for doc id via one ranged index read (the 16-byte offset pair) and one ranged blob read, or ok=false if id is out of range. A zero-length record (a doc with no stored fields) returns ok=true with empty bytes.

func (*RecordStore) GetMany added in v0.27.0 

func (s *RecordStore) GetMany(ids []uint32) (map[uint32][]byte, error)

GetMany returns the decoded record bytes for each in-range id, keyed by id. Doc IDs are rank-ordered by construction, so a top-k page's ids are frequently near-contiguous: GetMany reads the offset-table entries in coalesced waves and then the record blobs in coalesced waves (bridging gaps <= recordCoalesceGap), so ~20 near-contiguous records cost a handful of ranged reads rather than ~2 per id. Out-of-range ids are omitted; a zero-length record maps to empty bytes. Results are identical to calling Get on each id.

func (*RecordStore) Len added in v0.24.0 

func (s *RecordStore) Len() uint32

Len returns the number of records (doc IDs 0..Len).

type RecordWriter added in v0.24.0 

type RecordWriter struct {
	// contains filtered or unexported fields
}

RecordWriter streams the RRSR record store: record bytes are pushed one at a time in doc-ID order, so a builder that produces records incrementally never has to hold them all in memory. The concatenated record bytes go to bin and a range-fetchable offset index to idx. Records are opaque to the library — the caller chooses the encoding (JSON, msgpack, …); the store only frames them for O(1) Range lookup by doc ID. Mirrors the Rust RecordWriter. See RECORDS.md.

count is written into the header up front, so the caller must know the record total in advance and call Write exactly that many times (the offset table is sized for count+1 entries).

func NewRecordWriter added in v0.24.0 

func NewRecordWriter(bin, idx io.Writer, count uint32) (*RecordWriter, error)

NewRecordWriter opens a streaming record store for count records, writing the 16-byte RRSR header and the leading off[0]=0 to idx. Push each record with Write in ascending doc-ID order.

PERFORMANCE: Write issues one bin write plus one 8-byte idx write per record, so pass BUFFERED writers (e.g. bufio.Writer) for bin and idx when they are files or sockets — otherwise a full-corpus build pays ~2 syscalls per record. The library does not buffer internally (so it never double-buffers a caller that already does); flush your buffers after the final Write / Finish.

func (*RecordWriter) Finish added in v0.27.0 

func (w *RecordWriter) Finish() error

Finish verifies the writer produced exactly the record count declared in the header. The header commits to count up front and the offset table is sized for count+1 entries, so an under- or over-count leaves the store internally inconsistent — a reader Get of a high id would read past the written offsets. Call it after the last Write; it writes nothing (the output stays byte-identical for a correct writer) and only reports the mismatch. Streaming callers that cannot know the total in advance should size the writer to the count they will actually emit.

func (*RecordWriter) Write added in v0.24.0 

func (w *RecordWriter) Write(rec []byte) error

Write appends one record's bytes to the blob and its cumulative end offset to the index. A zero-length record (a doc with no stored fields) stays addressable.

func (*RecordWriter) Written added in v0.24.0 

func (w *RecordWriter) Written() uint32

Written returns the number of records written so far.

type RerankStore added in v0.29.0 

type RerankStore struct {
	Dim int
	N   uint64
	// contains filtered or unexported fields
}

RerankStore is a reference reader over an RRVR bf16 sidecar accessed by byte range: dense bf16 vectors keyed by doc ID (index == doc ID).

func OpenRerank added in v0.29.0 

func OpenRerank(r io.ReaderAt) (*RerankStore, error)

OpenRerank reads and validates the RRVR header.

func (*RerankStore) Vector added in v0.29.0 

func (s *RerankStore) Vector(doc uint32) ([]float32, bool, error)

Vector decodes the bf16 vector stored for a doc ID back to f32. ok is false when doc is out of range.

type SortColSpec added in v0.24.0 

type SortColSpec struct {
	Name       string // the RRSC data-file name holding the rank column
	Column     uint16 // column index within that RRSC
	Descending bool   // whether a higher value ranks better
}

SortColSpec is the stable-key rank source recorded in the manifest header.

type SortColumn added in v0.24.0 

type SortColumn struct {
	Name   string
	Values ColumnValues
}

SortColumn is one named sort column: a display name plus its dense values.

type SortcolStore added in v0.29.0 

type SortcolStore struct {
	// contains filtered or unexported fields
}

SortcolStore is a reference reader over an RRSC store accessed by byte range. It reads the header, column directory, and name blob up front; each column's dense values are fetched per Column call.

func OpenSortcols added in v0.29.0 

func OpenSortcols(r io.ReaderAt) (*SortcolStore, error)

OpenSortcols reads and validates the RRSC header, column directory, and name blob. Column values are read lazily.

func (*SortcolStore) Column added in v0.29.0 

func (s *SortcolStore) Column(i int) (SortColumn, error)

Column reads and decodes the i-th column's dense values into a typed ColumnValues (the same type WriteSortcols consumes).

func (*SortcolStore) Columns added in v0.29.0 

func (s *SortcolStore) Columns() []ColumnMeta

Columns returns the shape of every column without reading its values.

func (*SortcolStore) ReadAll added in v0.29.0 

func (s *SortcolStore) ReadAll() ([]SortColumn, error)

ReadAll reads every column into the SortColumn slice WriteSortcols consumes, so WriteSortcols(ReadAll(x)) reproduces x byte-for-byte.

func (*SortcolStore) Rows added in v0.29.0 

func (s *SortcolStore) Rows() int

Rows reports the per-column value count (one per doc).

type SplitBuildConfig added in v0.24.0 

type SplitBuildConfig struct {
	Policy          int          // PolicyTiered | PolicyStableKey
	ByteCap         uint64       // per-split seal target (the FIRST tier's cap when CapMax > 0)
	CapMax          uint64       // geometric tiering: tier i's cap = min(ByteCap << i, CapMax); 0 = flat
	GramSize        uint16       // n-gram window (e.g. 3)
	HeadBoundary    uint32       // split head/tail boundary; 0 -> 65536
	Stride          uint32       // sparse-index stride; 0 -> DefaultStride
	NamePrefix      string       // split filenames: ‹prefix›-s00000.rrs, ...
	SortCol         *SortColSpec // stable-key rank source, or nil
	BloomBitsPerKey uint32       // per-split term Bloom bits/key (0 disables)
	// CaseSensitive builds a case-sensitive split set: n-gram and facet keys are NOT
	// lowercased (v4 RRS splits, case-sensitive RRSF keys, the manifest's case-sensitive
	// flag). The zero value (false) is the default case-folding behavior, byte-identical to
	// before. (This is the inverse of the Rust SplitBuildConfig.case_normalization field; Go
	// uses the zero-value-safe inverse so existing configs are unaffected.)
	CaseSensitive bool
}

SplitBuildConfig is the build-time configuration for a SplitSetBuilder.

type SplitSet added in v0.29.0 

type SplitSet struct {
	Config SplitSetConfig
	Splits []SplitSpec
}

SplitSet is a reference reader over an RRSS manifest. The whole manifest is resident after Open.

func OpenSplitSet added in v0.29.0 

func OpenSplitSet(r io.ReaderAt) (*SplitSet, error)

OpenSplitSet reads and validates the RRSS manifest, rehydrating the config and every split entry.

type SplitSetBuilder added in v0.24.0 

type SplitSetBuilder struct {
	// contains filtered or unexported fields
}

SplitSetBuilder accumulates documents and seals them into byte-capped RRS splits.

func NewSplitSetBuilder added in v0.24.0 

func NewSplitSetBuilder(cfg SplitBuildConfig) *SplitSetBuilder

NewSplitSetBuilder creates a builder. A HeadBoundary/Stride of 0 take the RRS defaults.

func (*SplitSetBuilder) AddFaceted added in v0.24.0 

func (b *SplitSetBuilder) AddFaceted(text string, facets map[string][]string) (uint32, error)

AddFaceted tokenizes text and appends it as one document, recording its facet memberships (each field mapped to the categories this document belongs to). Each sealed split then gets its own RRSF facet sidecar plus a facet-presence summary, so a facet-filtered query can skip a split lacking a selected category. Returns the global id.

func (*SplitSetBuilder) AddKeys added in v0.24.0 

func (b *SplitSetBuilder) AddKeys(keys []uint64) (uint32, error)

AddKeys appends one document by its (deduplicated) n-gram keys, returning its global doc id. A keyword-less document still consumes an id (dense id space).

func (*SplitSetBuilder) AddText added in v0.24.0 

func (b *SplitSetBuilder) AddText(text string) (uint32, error)

AddText tokenizes text into n-gram keys and appends it as one document, returning its global doc id.

func (*SplitSetBuilder) DocCount added in v0.24.0 

func (b *SplitSetBuilder) DocCount() uint32

DocCount is the number of documents added so far.

func (*SplitSetBuilder) Finish added in v0.24.0 

func (b *SplitSetBuilder) Finish() (*BuiltSplitSet, error)

Finish seals the final open split and serializes the manifest, returning the manifest bytes and every split's (filename, RRS bytes). Errors if a single document's postings alone exceed the byte cap (a degenerate corpus).

func (*SplitSetBuilder) SetFacetDigest added in v0.39.0 

func (b *SplitSetBuilder) SetFacetDigest(topPerField int)

SetFacetDigest emits a facet digest into each sealed split's manifest summary (TLV tag 3): the top topPerField categories per field by full-corpus count, with names, counts, and posting ranges -- so a reader prices panel facet counts from the resident manifest with no sidecar meta boot. 0 (the default) emits no digest and keeps every existing manifest byte-identical. Mirrors the Rust with_facet_digest.

func (*SplitSetBuilder) SetFacetDigestV2 added in v0.42.0 

func (b *SplitSetBuilder) SetFacetDigestV2(topPerField int)

SetFacetDigestV2 is like SetFacetDigest but emits the v2 digest (tag 6), which additionally carries each top-k category's tail container directory so a reader can skip the per-category tail-header read for a large tail. Opt-in; v1 stays the default. Mirrors the Rust with_facet_digest_v2.

type SplitSetConfig added in v0.24.0 

type SplitSetConfig struct {
	Policy    int          // PolicyTiered | PolicyStableKey
	BodyKind  uint8        // BodyKindTrigram (RRS) | BodyKindTerm (RRTI); 0 keeps older manifests byte-identical
	TierCount uint16       // number of rank tiers (tiered); 0 for stable-key
	BaseCount uint32       // splits [0, BaseCount) are base, the rest delta
	ByteCap   uint64       // the per-split byte cap (informational)
	GramSize  uint16       // n-gram window the splits were built with (for Bloom pruning); 0 for a term-bodied set
	SortCol   *SortColSpec // stable-key rank source, or nil
	Flags     uint16       // header summary-presence flags
}

SplitSetConfig is the manifest-level configuration.

type SplitSpec added in v0.24.0 

type SplitSpec struct {
	DataFile string // the split's RRS data-file name
	Tier     uint16 // rank tier (tiered policy; 0 for stable-key / delta)
	DocCount uint32 // docs in the split
	DocIDLo  uint32 // min global doc id present (inclusive); the local-id base
	DocIDHi  uint32 // max global doc id present (inclusive)
	Epoch    uint64 // flush/build epoch (supersession ordering)
	ByteSize uint64 // the split RRS file size in bytes
	Flags    uint16 // per-split flags (SplitFlagHasTombstone | SplitFlagAbsoluteIDs)
	Summary  []byte // opaque summary TLV bytes (Bloom / facet / time / tombstone)
}

SplitSpec is one split recorded in the manifest. The split's RRS bytes live in its own DataFile and are not passed here.

type TermHeader added in v0.29.0 

type TermHeader struct {
	TermCount     uint32
	HeadBoundary  uint32
	BlockCap      uint32
	Language      TermLanguage
	Stemmed       bool
	Stopwords     bool
	CaseSensitive bool
}

TermHeader is the RRTI index configuration decoded from the header.

type TermIndex added in v0.29.0 

type TermIndex struct {
	// contains filtered or unexported fields
}

TermIndex is a reference reader over an RRTI index accessed by byte range. Only the router FST is resident; dict blocks and postings are range-read per lookup.

func OpenTermIndex added in v0.29.0 

func OpenTermIndex(r io.ReaderAt) (*TermIndex, error)

OpenTermIndex reads and validates the RRTI header and resident router FST. Dict blocks and postings are read lazily.

func (*TermIndex) Complete added in v0.30.0 

func (t *TermIndex) Complete(prefix string, maxTerms int) ([]string, error)

Complete autocompletes prefix: up to maxTerms dictionary terms that start with it, in lexicographic order. The prefix is case-folded the way the index's tokenizer folds (verbatim for a case-sensitive index). Range-reads only the dict blocks spanning the prefix. Mirrors the Rust reader's complete.

func (*TermIndex) Dict added in v0.29.0 

func (t *TermIndex) Dict() []DictEntry

Dict collects the whole dictionary as (term, head_off) entries in dictionary order (the shape WriteImpacts consumes and WriteTermIndexFullDict returns).

func (*TermIndex) Header added in v0.29.0 

func (t *TermIndex) Header() TermHeader

Header returns the index configuration.

func (*TermIndex) LookupTerm added in v0.29.0 

func (t *TermIndex) LookupTerm(token string) (*roaring.Bitmap, bool, error)

LookupTerm returns the doc posting for an exact dictionary token (already in the stored, tokenized form). ok is false when the token is absent.

func (*TermIndex) Posting added in v0.29.0 

func (t *TermIndex) Posting(query string) (*roaring.Bitmap, bool, error)

Posting tokenizes query with the index's tokenizer and returns the union of the resulting tokens' postings (a single-word query resolves to one token). ok is false when the query yields no token or none is present.

func (*TermIndex) PrefixPostings added in v0.30.0 

func (t *TermIndex) PrefixPostings(prefix string, limit int) ([]TermPosting, bool, error)

PrefixPostings returns up to limit dictionary terms starting with prefix, in lexicographic order, each with its full posting. truncated is true when more than limit terms matched. The prefix is case-folded the way the index's tokenizer folds; only the dict blocks spanning the prefix and the matched terms' postings are range-read.

func (*TermIndex) SearchPrefix added in v0.30.0 

func (t *TermIndex) SearchPrefix(prefix string, limit int) (docs []uint32, truncated bool, err error)

SearchPrefix returns up to limit doc IDs matching any dictionary term that starts with prefix (the union of every matching term's posting), ascending — which is descending rank under the rank-remapped doc-ID space. At most maxPrefixTerms terms are unioned; truncated is true when the prefix matched more, making the result a bounded approximation of the full union. Head bitmaps are read first and the tails only when the heads underfill limit, mirroring the Rust reader's search_prefix_capped / union_locs.

func (*TermIndex) Terms added in v0.29.0 

func (t *TermIndex) Terms() iter.Seq2[string, uint64]

Terms iterates every dictionary term with its posting head offset, in byte-lexicographic dictionary order, decoding one block at a time.

type TermLanguage added in v0.24.0 

type TermLanguage uint8

TermLanguage selects the Snowball stemmer recorded in the RRTI header (the on-disk language byte; values match the Rust Language::to_u8). 

const (
	// TermLanguageNone builds an unstemmed index with no stop-word list.
	TermLanguageNone TermLanguage = 0
	// The Snowball languages, byte values matching the Rust Language::to_u8. All are wired
	// for stemming on the Go build side (see stemAlgorithm) and every language has a
	// stop-word list (stopwordFile), so a stemmed and/or stop-words index can be built in
	// any of them.
	TermLanguageEnglish    TermLanguage = 1
	TermLanguageSpanish    TermLanguage = 2
	TermLanguageArabic     TermLanguage = 3
	TermLanguageDanish     TermLanguage = 4
	TermLanguageDutch      TermLanguage = 5
	TermLanguageFinnish    TermLanguage = 6
	TermLanguageFrench     TermLanguage = 7
	TermLanguageGerman     TermLanguage = 8
	TermLanguageGreek      TermLanguage = 9
	TermLanguageHungarian  TermLanguage = 10
	TermLanguageItalian    TermLanguage = 11
	TermLanguageNorwegian  TermLanguage = 12
	TermLanguagePortuguese TermLanguage = 13
	TermLanguageRomanian   TermLanguage = 14
	TermLanguageRussian    TermLanguage = 15
	TermLanguageSwedish    TermLanguage = 16
	TermLanguageTamil      TermLanguage = 17
	TermLanguageTurkish    TermLanguage = 18
)

type TermPosting added in v0.30.0 

type TermPosting struct {
	Term    string
	Posting *roaring.Bitmap
}

TermPosting pairs a prefix-matched dictionary term with its full posting.

type TermSplitBuildConfig added in v0.24.0 

type TermSplitBuildConfig struct {
	Policy       int          // PolicyTiered | PolicyStableKey
	ByteCap      uint64       // per-split seal target (the FIRST tier's cap when CapMax > 0)
	CapMax       uint64       // geometric tiering: tier i's cap = min(ByteCap << i, CapMax); 0 = flat
	HeadBoundary uint32       // head/tail doc-ID split; 0 -> 65536
	NamePrefix   string       // split filenames: ‹prefix›-s00000.rrt, …
	SortCol      *SortColSpec // stable-key rank source, or nil
	Language     TermLanguage // index language, shared by Stem and Stopwords; required when either is set
	Stem         bool         // apply Snowball stemming in Language (independent of Stopwords)
	Stopwords    bool         // remove the language's stop words (and from queries); requires Language
	// CaseSensitive builds a case-sensitive term split set: terms and facet keys are NOT
	// lowercased (each split's RRTI carries the case-sensitive flag, the RRSF keys are
	// case-sensitive, and the manifest sets its case-sensitive flag). The zero value (false)
	// is the default case-folding behavior, byte-identical to before. Inverse of the Rust
	// TermSplitBuildConfig.case_normalization (Go uses the zero-value-safe inverse).
	CaseSensitive bool
}

TermSplitBuildConfig configures a TermSplitSetBuilder.

type TermSplitSetBuilder added in v0.24.0 

type TermSplitSetBuilder struct {
	// contains filtered or unexported fields
}

TermSplitSetBuilder accumulates documents and seals them into byte-capped RRTI splits with per-split facet sidecars.

func NewTermSplitSetBuilder added in v0.24.0 

func NewTermSplitSetBuilder(cfg TermSplitBuildConfig) *TermSplitSetBuilder

NewTermSplitSetBuilder creates a builder. A HeadBoundary of 0 takes the RRS default.

func (*TermSplitSetBuilder) AddFaceted added in v0.24.0 

func (b *TermSplitSetBuilder) AddFaceted(text string, facets map[string][]string) (uint32, error)

AddFaceted is AddText plus the document's facet memberships, recorded into the open split's RRSF sidecar and facet-presence summary.

func (*TermSplitSetBuilder) AddText added in v0.24.0 

func (b *TermSplitSetBuilder) AddText(text string) (uint32, error)

AddText tokenizes text and appends it as one document, returning its global doc id. A token-less document still consumes an id (dense id space).

func (*TermSplitSetBuilder) DocCount added in v0.24.0 

func (b *TermSplitSetBuilder) DocCount() uint32

DocCount is the number of documents added so far.

func (*TermSplitSetBuilder) Finish added in v0.24.0 

func (b *TermSplitSetBuilder) Finish() (*BuiltSplitSet, error)

Finish seals the final open split and serializes the manifest (BodyKindTerm, gramSize 0), returning the manifest and every split's bytes. Errors if any single document's postings alone exceed the byte cap.

func (*TermSplitSetBuilder) SetFacetDigest added in v0.39.0 

func (b *TermSplitSetBuilder) SetFacetDigest(topPerField int)

SetFacetDigest emits a facet digest into each sealed split's manifest summary (TLV tag 3) -- see SplitSetBuilder.SetFacetDigest; identical semantics for term splits.

func (*TermSplitSetBuilder) SetFacetDigestV2 added in v0.42.0 

func (b *TermSplitSetBuilder) SetFacetDigestV2(topPerField int)

SetFacetDigestV2 emits the v2 facet digest (tag 6, with tail container directory) -- see SplitSetBuilder.SetFacetDigestV2; identical semantics for term splits. Opt-in; v1 default.

type TermTokenizer added in v0.24.0 

type TermTokenizer struct {
	// contains filtered or unexported fields
}

TermTokenizer is the configured token-filter chain (base tokenize → optional stop-word removal → optional Snowball stemming), fixed at build time and recorded in the header so queries tokenize identically. Mirrors the Rust terms::Tokenizer.

func NewTermTokenizer added in v0.24.0 

func NewTermTokenizer(lang TermLanguage, stopwords bool) *TermTokenizer

NewTermTokenizer builds the chain for the given language / stop-word setting, with case folding on (the default). Mirrors the Rust Tokenizer::new(.., true).

func NewTermTokenizerFull added in v0.26.0 

func NewTermTokenizerFull(lang TermLanguage, stem, stopwords, caseFold bool) *TermTokenizer

NewTermTokenizerFull builds the chain with independent stem and stopwords filters over a shared language (mirrors the Rust Tokenizer::with). The stemmer is created only when stem is set, so an index can strip a language's stop words without stemming. Every go-stemmers language is wired (see stemAlgorithm); a stem request for TermLanguageNone or an unsupported byte leaves the stemmer nil.

func NewTermTokenizerWith added in v0.25.0 

func NewTermTokenizerWith(lang TermLanguage, stopwords, caseFold bool) *TermTokenizer

NewTermTokenizerWith builds the chain with an explicit case-fold flag. Back-compat: a non-None language implies stemming (the historical coupling). Prefer NewTermTokenizerFull to strip a language's stop words without stemming.

func (*TermTokenizer) Tokenize added in v0.24.0 

func (t *TermTokenizer) Tokenize(text string) []string

Tokenize applies the full chain to text.

type TrigramMonolithBuilder added in v0.24.0 

type TrigramMonolithBuilder struct {
	// contains filtered or unexported fields
}

TrigramMonolithBuilder accumulates trigram postings over a whole corpus into a single v3 RRS index. Docs are added in ascending doc-ID order — each Add returns the doc's id, and an empty doc (no trigrams) still consumes an id so the doc-ID space stays dense and aligned with the records / facet / lookup sidecars. Write seals the accumulated postings to one RRSI index, byte-identical to a split set's single split over the same docs. All postings are held in memory; for a 100+ GB corpus use the chunked Rust builder instead.

func NewTrigramMonolithBuilder added in v0.24.0 

func NewTrigramMonolithBuilder(gramSize uint16, stride int) *TrigramMonolithBuilder

NewTrigramMonolithBuilder opens a monolith builder for the given trigram size and sparse stride, with case folding on (the default). A gramSize of zero defaults to 3; a stride of zero or less to DefaultStride.

func NewTrigramMonolithBuilderWith added in v0.25.0 

func NewTrigramMonolithBuilderWith(gramSize uint16, stride int, caseSensitive bool) *TrigramMonolithBuilder

NewTrigramMonolithBuilderWith opens a monolith builder; when caseSensitive is true the trigram keys are not lowercased and Write emits a case-sensitive v4 index. caseSensitive false reproduces the default (case-folding) behavior byte-for-byte.

func (*TrigramMonolithBuilder) AddKeys added in v0.24.0 

func (b *TrigramMonolithBuilder) AddKeys(keys []uint64) uint32

AddKeys indexes the given trigram keys under the next doc ID and returns that ID. An empty keys slice still advances the doc-ID space (the doc is indexed as having no trigrams).

func (*TrigramMonolithBuilder) AddText added in v0.24.0 

func (b *TrigramMonolithBuilder) AddText(text string) uint32

AddText tokenizes text into gramSize-gram trigram keys and indexes them under the next doc ID, returning that ID. Mirrors SplitSetBuilder.AddText.

func (*TrigramMonolithBuilder) DocCount added in v0.24.0 

func (b *TrigramMonolithBuilder) DocCount() uint32

DocCount returns the number of docs added so far (the next doc ID).

func (*TrigramMonolithBuilder) Write added in v0.24.0 

func (b *TrigramMonolithBuilder) Write(dst io.Writer) error

Write seals the accumulated postings into one v3 RRS index on dst — each trigram's bitmap serialized as one portable posting, laid out key-sorted via WriteIndex.

type U16Column added in v0.24.0 

type U16Column []uint16

U16Column / U32Column / I32Column / F32Column are the four on-disk value types.

func (U16Column) Len added in v0.24.0 

func (c U16Column) Len() int

type U32Column added in v0.24.0 

type U32Column []uint32

U16Column / U32Column / I32Column / F32Column are the four on-disk value types.

func (U32Column) Len added in v0.24.0 

func (c U32Column) Len() int

type VectorHeader added in v0.29.0 

type VectorHeader struct {
	Dim, Nlist, M int
	Nbits         uint8
	Metric        ivfpq.Metric
	N             uint64
	HasOPQ        bool
}

VectorHeader is the RRVI index shape.

type VectorIndex added in v0.29.0 

type VectorIndex struct {
	// contains filtered or unexported fields
}

VectorIndex is a reference reader over an RRVI index accessed by byte range. The centroids, codebooks, optional OPQ, and directory are resident after Open; each cluster's ids/codes are fetched per Cluster call.

func OpenRRVI added in v0.29.0 

func OpenRRVI(r io.ReaderAt) (*VectorIndex, error)

OpenRRVI reads and validates the RRVI header and resident regions (OPQ, centroids, codebooks, cluster directory). Cluster ids/codes are read lazily.

func (*VectorIndex) Cluster added in v0.29.0 

func (v *VectorIndex) Cluster(c int) (ids []uint32, codes []byte, err error)

Cluster reads the c-th cluster's doc IDs and packed PQ codes (M bytes per vector).

func (*VectorIndex) Header added in v0.29.0 

func (v *VectorIndex) Header() VectorHeader

Header returns the index shape.

func (*VectorIndex) ToModel added in v0.29.0 

func (v *VectorIndex) ToModel() (*ivfpq.Model, error)

ToModel reconstructs the in-memory ivfpq.Model (reading every cluster), so WriteRRVI(ToModel(x)) reproduces x byte-for-byte.

Source Files

View all Source files

Directories

Path Synopsis
cmd
roaringrange command
Command roaringrange inspects roaringrange index files: it auto-detects the format from the 4-byte magic and prints a header summary (info), dumps the full structural contents as JSON (dump), decodes a record store (records), or looks up a single key (get).
 Command roaringrange inspects roaringrange index files: it auto-detects the format from the 4-byte magic and prints a header summary (info), dumps the full structural contents as JSON (dump), decodes a record store (records), or looks up a single key (get).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.