raglit

package module
v0.9.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 10, 2026 License: MIT Imports: 65 Imported by: 0

README

raglit

A local document RAG index you can stand up in one command. Point it at a folder, ask questions, or hand it to an agent as an MCP tool. One portable SQLite file is the whole index — pure-Go, single static binary, no services to run.

Quickstart

Three ways to install. Each is one binary: CLI, daemon and MCP server; pure Go, no C compiler.

how gets updates
release tarball raglit_{os}_{arch}.tar.gz (+ SHA256SUMS) from GitHub releases everything, web UI included doctor --json checks the latest release
go install github.com/iodesystems/raglit/cmd/raglit@latest CLI, daemon, MCP; the web UI is a placeholder page (it needs node to build) doctor --json checks the Go proxy
make install in a checkout everything; rebuilds itself when the checkout changes doctor --json checks the checkout's upstream

raglit doctor says what this install can do and what would add more: models per role (embedding, vision, text, summary, audio) and external extractors (poppler, pandoc, tesseract, …). --json prints it for tools; raglit mcp sends the same summary as an MCP log notification (logger raglit/setup).

# build
go install github.com/iodesystems/raglit/cmd/raglit@latest

# 1. per-project setup — pick an OpenAI-compatible endpoint + models (a wizard)
cd my-project
raglit init                # writes ./.raglit/ here

# 2. ingest a folder (code, markdown, text; PDFs get OCR'd)
raglit ingest ./src --wait

# 3. ask (works from any subdirectory — raglit finds ./.raglit)
raglit search "how does the auth token refresh work?"

That's it. init asks for a base URL + API key and lists the endpoint's models so you can pick a vision model and an embedding model, plus a project name (defaults to the directory name); everything else uses sensible defaults (you never pass model flags again). The project name namespaces this repo's indexes on the shared daemon (below), so two projects both using the default index never collide. When the endpoint reports capabilities (a corrallm-class server), each pick list is filtered to the models that fit the role — image-capable models for OCR, embedding models for --embed — instead of the whole catalog; a plain OpenAI server shows all models.

raglit init is project-local: it writes ./.raglit/ in the current directory, so each repo or sub-project owns its own index and config. Any command run inside the tree discovers the nearest .raglit/ by walking up (like git), so you can run raglit search from a deep subdirectory. With no .raglit/ found, commands fall back to $RAGLIT_HOME, else ~/local/raglit. Override the location anywhere with --home DIR.

On success init prints the MCP server setup (a claude mcp add-json line and a .mcp.json block) plus the ingest/search commands for reference.

No endpoint handy? Every offline piece works without one: raglit demo runs a self-contained tour, and text/code ingest needs no model at all — the deterministic fragmenter runs fully offline (only OCR and figure description need a vision model).

What it does

  • Ingest folders, files, or URLs (file://, http(s)://) — lazily (queued) and followed with --wait. Each item runs a staged pipeline: a scanned page goes img→paged-text (OCR cascade: cheap tesseract→gibberish-gate→vision VLM); a born-digital PDF page uses its text layer, no OCR. Then one per-document fragmenter choice: text, code, and cheap-OCR'd pages are split by a deterministic overlapping-window fragmenter (no model — windows snapped to line/paragraph edges, each fragment carrying its source offsets); a document any page of which the VLM transcribed is LLM-segmented as a whole (coherent ~500-word units — functions bound with their docs — an open fragment carried across page boundaries). The choice is stored per document (frag_mode = text-overlap | llm-seg, shown in list_documents); window / stride / floor are config-tunable and capped by the embed model's probed input limit. Every stage and its engine is recorded per job.

  • Mail archives (.eml, .mbox) are read as PAGES, one per message — an archive is not one document. A 24 MB .eml holding a decade of nested forwards read as "page 1" is unciteable; read as pages, a quotation lands on which message. Headers are kept (for mail the routing IS the evidence), enclosed message/rfc822 messages become pages of their own, transfer encodings and non-UTF-8 charsets are decoded, and multipart/alternative indexes the plain text rather than the same words twice. Attachments are named with their media type and size; set extract_email_attachments (project-wide or per index) and they are also extracted byte-for-byte into <archive>.raglit-attachments/ with a MANIFEST.md recording which message, sender, date and sha256 each came from — so the survey inside the archive becomes an ordinary file the next sync indexes, with its chain intact. Off by default: one archive can carry 69 files.

  • Figures explained into the index — while the VLM transcribes a scanned page it also describes each figure/diagram/chart inline ([FIGURE: …]), so a diagram becomes searchable text with no extra infrastructure. Each figure is also recorded as a media object anchored to the fragment holding it and embedded on its own — from the image (via nomic-embed-vision, which shares nomic-embed-text's space so a text query still matches) or, with no image embedder, from its description. search_figures ranks figures directly, and get_document returns a document's figures alongside its text. (Escalating a born-digital page that carries a figure to the VLM is opt-in: ocr.describe_figures.)

  • Indexing work is deduped: the daemon caches each processed document in a shared pool keyed by (recipe, file-hash) — where recipe is the models + config that shape the output — so the same file, in ANY index or on a retry, is reused (fragments + vectors + page images copied in, mode pooled) instead of re-running the LLM. Re-indexing under different models is a new recipe, so it reprocesses. (Embedded/single-index mode dedups per index by content hash.) The pool is bounded but lax by default: it grows freely up to a byte budget (--pool-max-bytes, default 4 GiB — counting cached payloads and page images) and trims by evicting the oldest-accessed entries — so merges and retries keep reusing pooled work rather than re-indexing it. --pool-max (entry cap) and --pool-ttl (evict unused; off by default) are optional; POST /api/pool/gc runs it on demand and GET /api/pool reports size.

  • Search — BM25 (--mode bm25), vectors (--mode vec), or hybrid RRF (--mode hybrid; the default on an index with embeddings, else BM25). Each hit carries similarity (raw cosine, when the vector side ranked it) and matched_terms / query_terms (term coverage, stopwords out), so a caller can cut weak hits at a fixed threshold. Optionally scoped to an index and/or a path subtree (--path /repo/src/api/ — a prefix match, so hierarchical corpora don't need a separate index per directory). Results are precise citations: document → page → fragment. search_figures does the same over described figures.

  • MCP over HTTP — a client needs a URL and a key and runs nothing:

    raglit project create acme --root /srv/acme   # prints the MCP URL and the key, once
    

    The client connects to <daemon>/mcp/acme with Authorization: Bearer <key>. The key reaches that project's indexes and nothing else. On top of the tools below it gets upload (send a document; list_documents reports each content_hash, so send only what changed) and manage_project (create, drop, watch, unwatch, watches). Reading or watching a directory on the daemon's host needs a --root grant. raglit project key acme mints another key; raglit key revoke ends one. The indexes with no project are /mcp/root, for the admin key. One daemon is one trust domain: its models and its shared pool serve every project on it, so a vault with other rules uses another daemon.

  • MCP — expose the index(es) to any MCP client (Claude Desktop, agentkit):

    raglit mcp
    

    Tools: search, search_figures, list_documents, get_document, ingest, index_status, list_indexes, ocr. search / search_figures take an optional path prefix. raglit init prints a ready-to-paste MCP config (Claude Code + generic .mcp.json) pinned to this project's .raglit/.

    raglit mcp --manage adds upload and manage_project, the tools below for a client with a project. They are off by default because they change indexes.

    An agent that needs a whole document's text: search to find a hit (or list_documents with a name filter to find it by filename), then get_document with that path (or a unique filename substring) to read the full indexed text — per-page plus a joined blob (overlapping fragments reassembled exactly once via their offsets), plus the document's figures, with optional page range and a max_chars cap that bounds the whole response (pages included). ocr is the other read path: it extracts text from a file/URL you supply directly (not from the index).

Commands

raglit init                          configure endpoint + models (wizard)
raglit ingest TARGET... [--wait]     queue folders / files / URLs (lazy; --wait follows the daemon)
raglit search "query" [--mode M] [--path P]  M = bm25 | vec | hybrid; P = path-prefix scope
raglit status                        documents/fragments, queue progress, rate, ETAs
raglit identify [DOC...] [--list]    what a document IS: caption, summary, kind (never renames)
raglit prune [--index N] [--yes]     documents whose local file is gone (already out of search); --yes removes
raglit mcp                           stdio MCP server (old name: serve)
raglit daemon                        HTTP API + workers + review UI at /
raglit review                        the daemon, framed as the status/job/OCR review UI
raglit demo                          offline, self-contained tour

--home DIR overrides the index home (default: nearest ./.raglit walking up, else $RAGLIT_HOME, else ~/local/raglit); --index NAME selects a named index within it. With no --index, commands use the config's default_index (set in the wizard), falling back to default. On the shared daemon, --index NAME is this project's own index; --index <namespace>:<index> addresses a reachable namespace (your project or a shared one).

Configured sources (raglit sync)

Instead of passing paths every time, declare source roots + rules in the project's .raglit/config.json and run raglit sync — it resolves them to files and enqueues each (the content-hash dedup skips unchanged ones, so re-syncing is cheap). Rules layer project → index → root (ignore is unioned and always wins; include is overridable per root), each root's .gitignore is honored, and a built-in default drops dot-dirs / node_modules / vendor. Multi-index is native.

{
  // ...endpoint + models...
  "ignore":    ["**/*.min.js"],     // project-scoped default excludes (this config only)
  "gitignore": true,                 // honor each root's .gitignore (default)
  "indexes": {
    "code": {
      "roots":   [".", "../shared-lib"],
      "include": ["*.go", "*.ts", "*.py", "*.md"],   // a file must match one
      "ignore":  ["*_test.go", "gen/**"]              // merged with project + built-in
    },
    "docs": { "roots": [ { "path": "./docs", "include": ["*.md", "*.pdf"] } ] }
  }
}
raglit sync                       # ingest every configured index's roots
raglit sync --index code --dry-run  # preview one index's matched files

Globs: no / matches the basename (*.go); with /, the path (gen/**, **/x). sync routes to the daemon when daemon_url/--daemon is set, else the local home.

Watch ("watch": true): register a project and the daemon keeps its roots fresh — on an interval it re-plans the sources (same rules) and re-ingests changed files, dropping documents whose source file was deleted. raglit sync auto- registers when watch:true; manage it with raglit watch:

raglit watch          # register this project (idempotent)
raglit watch list     # what the daemon is watching (project, file count)
raglit watch stop     # unregister this project

It's a poll (default every 5s, --watch-interval), so an unchanged file that slips through is a no-op thanks to the content-hash dedup; registrations persist under the daemon root and reload on restart.

Daemon mode

By default every client — mcp and the CLI (ingest/search/status/ sync) — talks to a single shared per-user daemon, auto-starting it if none is running. There is no in-process mode: N sessions each opening the index would be N processes on one SQLite file, running their own workers and calling the LLM independently → write contention + duplicated indexing work, and every one of them skipping auth. One daemon (single writer + worker pool + LLM caller, scoped storage, shared dedup pool) is the only path. raglit demo and the tests start a throwaway daemon of their own.

# nothing to start — the first client brings the daemon up (at 127.0.0.1:7420,
# storage under $RAGLIT_ROOT / ~/.raglit), and every session connects to it:
raglit ingest ./my-project
raglit search "rollback procedure"
raglit mcp                         # MCP over the shared daemon

# run it explicitly (foreground) if you prefer, or point at a remote one:
raglit daemon --addr 127.0.0.1:7420    # workers + HTTP API + review UI + OpenAPI + GraphQL
raglit search --daemon http://host:7420 "…"   # or RAGLIT_DAEMON / config daemon_url
raglit daemon --stop                    # signal the running daemon to shut down

Because that one daemon serves every project, each client namespaces its indexes by the config's project name: the daemon index is <project>__<local>, and a project's "search all" is scoped to <project>__* — so two repos both using default don't share storage, and neither sees the other's documents. The project name is required to start a daemon-routed client (mcp or CLI); --project overrides it. A client with no project still reads (by directory scope or by name) but does not write. The <project>__ prefix is internal — search/status/list show plain local names.

Shared docs (shared): common material — a home ~/doc, a team handbook — is indexed once under its own project (say shared), and other projects opt into reading it by listing that namespace in their config. A project's "search all" then spans its own indexes plus each shared namespace, and shared hits keep their shared__ tag so you can see where they came from. No duplication per project. Address one specific index in a reachable namespace with --index <namespace>:<index> (e.g. --index shared:handbook) — for reads and for writes (so a project can contribute to a shared corpus); a bare --index name is always this project's own, and an unreachable namespace is refused (writes error, reads return nothing). Branches stay project-only.

// ~/doc, indexed once:  raglit --home ~/.raglit-shared ingest ~/doc   (project "shared")
{ "project": "alpha", "shared": ["shared"], "daemon_url": "http://127.0.0.1:7420" }
// `alpha` now searches alpha__* + shared__*; a project without "shared" stays isolated.

On startup the daemon records <root>/daemon.json ({pid, addr, root, ...}) and removes it on clean shutdown. Clients read it to discover the daemon's real address — so one on a non-default port is found instead of a duplicate being spawned on 7420 — after verifying the pid is alive and it answers /api/health (a stale file is ignored). raglit daemon --stop reads it to signal that pid.

raglit daemon is a multi-protocol server (huma + gwag/gat): the same operations are REST + in-process GraphQL (/graphql) + gRPC off one port, with OpenAPI at /openapi.json. Storage is scoped per index under --root (default ~/.raglit, so each index lives at ~/.raglit/indexes/<name>/); --home DIR selects a single-index layout instead. mcp becomes a thin client to the daemon when daemon_url/--daemon is set, so many mcp instances share one daemon.

Branches (copy-on-write, worktree-style): POST /api/branches {name,parent} forks a branch whose reads overlay the parent at document grain (writes/deletes touch the branch only); GET /api/branches lists them with age + last-access; DELETE /api/branches?name= drops one.

Review UI

The daemon also serves a self-contained web UI at / — status, job control, and OCR review. raglit review is the same server with a friendlier banner:

raglit review --addr 127.0.0.1:7420        # then open http://127.0.0.1:7420/
  • Status — documents, fragments, and live job counts (done/running/pending/ failed) + throughput, auto-refreshing.
  • Job control — the full ingest queue as a table; retry an errored or done job (requeues it) and cancel a pending one. Each job shows a mode badge (text-overlap = deterministic windows, llm-seg = a VLM-transcribed doc segmented by the model, pooled/unchanged = reused/skipped) and expands to its pipeline stages — the series of tasks it ran: fetch → extract → [ocr] → segment → [embed] → commit, each tagged with the engine that handled it (text-layer, pandoc, tesseract, vision, llm, text-overlap…), so a failure shows exactly which stage broke.
  • OCR review — pick a document to see its pages: the saved page image beside the indexed text, an engine badge per page (text = born-digital/plain, vision = the VLM OCR'd it), and a Re-OCR (cascade) button that reruns the cheap→gate→VLM cascade on that page's image to show the raw transcription and which engine handled it.

Ingest records per-page provenance (engine + page image) under <home>/pages/; documents indexed before this feature show no OCR pages until re-ingested. Control-plane routes live under /api/*.

Auth: never open. Every route except /api/health needs a bearer token, on every bind, loopback included. With no OIDC issuer configured the daemon runs in admin-token mode: it writes <root>/admin.token (0600) and accepts that token. With auth.issuer set it also accepts OIDC tokens. A client sends $RAGLIT_TOKEN, else the output of $RAGLIT_TOKEN_COMMAND (or config auth.token_command), else, for a daemon on a loopback address, the admin token file that daemon's /api/health names; so on a single-user machine nothing needs configuring. A browser signs in with raglit service login-admin.

Keys and session branches. The admin mints an API token for a service (raglit key mint --principal aw4 --out FILE) and hands it an index (raglit index code owner aw4). A service or a person mints a session key for one session (raglit key mint --session ID) and gives it to the agent as $RAGLIT_TOKEN. A session key reads its project and the global indexes, as far as its issuer may, and writes only its session's branches: code@ID, a copy-on-write branch of code that exists from its first write, reads as code until then, and is deleted 7 days after its last write (session_branch_ttl_hours). It expires 7 days after its last use. A session key mints a child key for one of its branches (--branch code@ID); revoking a key revokes the keys under it. A session's notes go in with raglit ingest --index sessions@ID --name N --text-file - --identity N.json, the identity recorded as a supplied caption with no model call.

For agents (agentkit)

raglit's search output is exactly the shape agentkit's ragnotify.ParseHits consumes, so one raglit mcp drives both a model's explicit searches and agentkit's proactive "live-watch" pings (the finder scopes which indexes it watches via Opts.ExtraArgs {"index": ...} — all by default). raglit's agent.DocFinder also plugs straight into a Session's FinderPreparer.

How it's built

  • SQLite FTS5 gives BM25 + the document:page:fragment index in one pure-Go dependency (modernc.org/sqlite, no CGo). Vectors are stored as BLOBs, cosine brute-forced (fine for a local corpus).
  • Two fragmenters, chosen by whether a model is already in the loop. Text, code, and cheap-OCR'd pages take a deterministic overlapping-window splitter with exact source offsets — no model, and overlap gives every hit its surrounding context (get_document reassembles the document once despite the overlap). A page a VLM transcribed is instead LLM-segmented (via agentkit) with a schema-validated fix-loop and a safe fallback, an open fragment carried across boundaries. Figures the VLM sees are described inline and embedded (image or description) for search_figures.
  • Multi-index — one home holds several named indexes; mcp searches all (RRF-merged, tagged), a scoped subset, or a path subtree.

Home layout

./.raglit/                 (per-project; or $RAGLIT_HOME / ~/local/raglit / --home)
  config.json              endpoint + model settings (from `raglit init`)
  index.sqlite             the default index (index-<name>.sqlite for others)
  uploads/                 uploaded documents (the index is their only home)
  pages/                   page images for OCR

Everything raglit holds for an index is under one directory — copy it, back it up, or delete it wholesale. A local-disk document is read from its file and not copied: when the file is gone, the document is out of search and raglit prune removes it.

Roadmap

  • ◻ Vector reranking; opt-in summaries for oversized fragments.

Documentation

Overview

Package raglit is a local, composable document RAG index.

The whole index is ONE portable SQLite file. SQLite's FTS5 extension gives BM25 lexical ranking built in, so "BM25" and "the document:page:fragment index" collapse into a single dependency — modernc.org/sqlite, which is pure-Go (no CGo) and thus builds to a single static binary. That is the point of raglit: a tool small enough to drop into any workflow (index a folder, grep it semantically) that scales up to a real service by swapping the agent.DocFinder impl (see finder.go) for a remote one — no rewrite.

Grain: documents → fragments(page, ord, text). A "fragment" is one indexable unit (a paragraph, a chunk, an OCR'd page region); page + ord locate it back in the source so a hit is a precise citation, not just "somewhere in the PDF".

Vectors are deliberately absent in v1. FTS5 lexical BM25 is the floor; a vector sidecar (sqlite-vec, or a custom NSW file) is added only if lexical recall proves insufficient — measured, not assumed.

Embedder surface

A program may embed this package as a library (dund does). The surface it may rely on is:

  • OpenHome and OpenIndex (with Home); StoreSet; Models (OnUsage, Code), NewWorker, NewChannels, NewEmbedder (with VectorClient), NewIdentifier, Models.Identifier, NewSegmenter (Segmenter.SegmentText, SegResult.Degraded).
  • Code index: CodeSegmenter and CodeSpan (Models.Code, nil by default); Fragment.StartLine, EndLine and Symbol; Hit.StartLine, EndLine, Symbol and ContentID; Store.FragmentExists(contentID).
  • Store ingest: Enqueue, Ingest (Document, Fragment), SetEmbedder, SetIdentifier, SetTags (caller-owned tags, also on a path not yet ingested).
  • Store search: Search, VecSearch, HybridSearch and their Path and Filtered variants with SearchFilter.
  • Store vectors: EmbedModel, SyncEmbedModel(model, dim), MissingVectors, BackfillVectors, DropVectors; index metadata: Meta, SetMeta.
  • Store documents: Documents (DocSummary, with ContentHash and AddedAt), DocumentHash, RemoveDocument, Prune(dry, deleteOriginals) to collect originals, Close.
  • Branch: SetParent, BranchDiff. Jobs: Jobs, Job, ClaimNext; Hit, Job and JobInfo; Runner (Start, Drain, CutOff, Done).
  • Worktree branch: OpenBranch, BranchMeta (Root, ParentRoot), WorktreeOf, GCBranches, ErrParentOnly.

api_test.go uses exactly this surface, so a break fails this repository's own tests. It follows semantic versioning from v0.5.0; v0.6.0 added the vector, metadata, segmentation, listing and tag calls; v0.7.0 added the code-index calls above and Models.OnUsage; v0.8.0 adds the worktree branch calls. Everything else in the package is internal to raglit and may change in any release.

Index

Constants

View Source
const (
	OpRelationPut = "relation.put"
	// OpAuthorityPut — a person ruled which reading of a source governs.
	OpAuthorityPut = "authority.put"
	OpSlicePut     = "slice.put"
	OpSliceDelete  = "slice.delete"
	OpPageCorrect  = "page.correct"
	OpDocWithdraw  = "doc.withdraw"
	OpDocRestore   = "doc.restore"
)

Audit op names. Values, not derived from Go identifiers: they are written to a file that outlives any refactor, and a rename must not orphan history.

View Source
const (
	CaptionProfileLegal = "legal"
	CaptionProfileDocs  = "docs"
)
View Source
const (
	// IdentityByMachine / IdentityByPerson are gen_source's two values.
	IdentityByMachine = "machine"
	IdentityByPerson  = "person"
	// IdentitySupplied is a caption the caller handed in with the document
	// (SupplyIdentity): the fold summary a session already wrote. Not a
	// ruling — a re-ingest that supplies another replaces it — and not owed
	// to the model, so no identity or tags job is queued for it.
	IdentitySupplied = "supplied"
)
View Source
const (
	IdentityAskFull   = "identity"
	IdentityAskTags   = "tags"
	IdentityAskFields = "fields"
)

The three asks a queued job can be. One queue, because they are the same shape of work — a bounded model call per document, hundreds of them, drained at the endpoint's real concurrency.

View Source
const (
	// SQLOwnWords constrains fragments to the document's own indexed words —
	// the transcription and the descriptions that ARE its content — excluding
	// the fragments that are about it. Unqualified column, for a query with one
	// table.
	SQLOwnWords = `origin NOT IN ('identity', 'fields')`
	// SQLOwnWordsF is the same, qualified for a query that joins `fragments f`.
	SQLOwnWordsF = `f.origin NOT IN ('identity', 'fields')`
)

A note for anything that filters fragments by origin.

There are two kinds of generated text in this column and they are not interchangeable. 'identity' is a caption ABOUT a document — excluded from the document's own words, because a re-run must read the document and not the last caption of it. 'described' IS a document's content: a photograph has no other text, and dropping it means the photograph has none.

So the predicate for "the document's own indexed words" is NOT `origin = ”`. Six queries were written that way before descriptions existed, and each one silently changed meaning the moment they did: the document list reported photographs as empty, get_document returned nothing for them, and the cross-index pool would have re-imported them with their only content stripped.

It is not `origin <> 'identity'` either, which is how it was written next. 'fields' — a document type's schema, filled out (docfields.go) — is the same kind of thing a caption is: a record ABOUT the document. Named the same way and added the same way, it would have walked straight back into the same trap from the other side, with every extraction reappearing as the document's own text in get_document and in its fragment count.

So the predicate is this constant, in every query that means it. Adding a generated origin means deciding which list it joins, ONCE, here.

View Source
const (
	KeyAPI     = "api"
	KeySession = "session"
	KeyProject = "project"
)
View Source
const (
	// ReadingMachine — a model's output, unreviewed. Every ingest starts here.
	ReadingMachine = "machine"
	// ReadingAdapted — a machine reading a person has changed: a corrected page,
	// a re-attributed speaker turn. Better than machine and not attested, because
	// nobody has said the whole of it is right.
	ReadingAdapted = "adapted"
	// ReadingAttested — a person has ruled on it. What may be quoted.
	ReadingAttested = "attested"
)

Reading levels, in the order authority accrues.

View Source
const (
	MethodASR       = "oidio-asr"
	MethodVision    = "vision-ocr"
	MethodTextLayer = "text-layer"
	MethodRegion    = "region"
	MethodPandoc    = "pandoc"
	MethodEmail     = "email-mime"
	MethodCompiled  = "compiled"
	MethodVerbatim  = "verbatim"
)

Reading methods — how the text was produced.

View Source
const (
	VerdictRead      = "read"      // or empty
	VerdictEscalate  = "escalate"  // the frame is wrong and I cannot fix it from here
	VerdictIllegible = "illegible" // right frame, the pixels are not there
	VerdictUnknown   = "unknown"   // the model returned a word this package does not define

	ActionRetransform = "retransform" // box was right, rendering was wrong
	ActionRepick      = "repick"      // box was wrong
	ActionKeep        = "keep"        // the child was mistaken; its reading stands
	ActionAbandon     = "abandon"     // nothing here can be read
)

What a region may claim about its own frame, and what a parent may answer.

View Source
const (
	// FlagLowResolution — this region was seen at materially less detail than a
	// letter page gets. Pure arithmetic, available before the model is called.
	FlagLowResolution = "low-resolution"
	// FlagRepetition — the model looped. On a page this fails the document; on a
	// region it is a reason to descend.
	FlagRepetition = "repetition"
	// There was a FlagClipped here, meant to catch text cut at a region
	// boundary. It measured ink touching the border and it did not work:
	// measured on the survey, a blank margin scored 0.196, a text block 0.161
	// and the drawing interior 0.055 — ANTI-correlated, because what it actually
	// found was the sheet's own border frame. It fired on every region of the
	// first real run, and a flag that is always on trains people to ignore
	// flags.
	//
	// Removed rather than tuned. The underlying problem — a cut like "REPLAT OF
	// BLO" — is now PREVENTED by padding descents (see descentPad) instead of
	// reported after the fact.
	// FlagCycled — descent stopped because it stopped paying: a transform
	// rendered to bytes already seen, or cleared no flag.
	FlagCycled = "cycled"
	// FlagExhausted — the model proposed nothing further. The one POSITIVE
	// signal, and the only one meaning "this is as good as it gets".
	FlagExhausted = "exhausted"
	// FlagBlurred — the crop's strokes are smeared, by the variance of its
	// laplacian against the measured spread of real pages. Computed on the
	// FULL-RESOLUTION crop before any call, because the model is shown a
	// downsample and cannot tell blur from smallness.
	FlagBlurred = "blurred"
	// FlagFaded — the crop's luminance is crushed into a narrow band, the way a
	// photocopy of a photocopy is. The damage with the most to gain from repair:
	// a faded crop read NOTHING unfiltered and most of its facts with contrast.
	FlagFaded = "faded"
	// FlagTransformSuspect — this region's reading has almost nothing in common
	// with what its PARENT said is here. Not a claim that the text is wrong: a
	// claim that the TRANSFORM is, because a crop or a rotation applied the wrong
	// way round does not garble the content, it returns a reading about something
	// else. Observed as clockwise-for-counter-clockwise and as a bad root.
	//
	// Recorded rather than corrected. Acting on it means re-rendering the PARENT
	// — the child cannot fix a rotation it did not choose — and that loop is not
	// built.
	FlagTransformSuspect = "transform-suspect"
	// FlagEscalated — this region's frame was referred back to its parent, and
	// the parent answered. Recorded whatever the answer was, because "a human
	// looked at why this looks wrong" is worth knowing even when the verdict was
	// that nothing was wrong.
	FlagEscalated = "escalated"
	// FlagAbandoned — the parent judged that nothing here can be read at any
	// treatment. The region keeps whatever its parent said was here: a region
	// nobody could read is not a region nobody described.
	FlagAbandoned = "abandoned"
	// FlagUnknownVerdict — the region answered with a verdict this package does
	// not define. Kept visible so invented vocabulary does not read as silence.
	FlagUnknownVerdict = "unknown-verdict"
	// FlagUnread — this region could not be read at all: the call failed rather
	// than returning something poor. Its children may still have been read, so
	// the tree below it is real and only this node's own account is missing.
	FlagUnread = "unread"
	// FlagBudget — descent stopped because the CALL budget ran out, not because
	// the region was finished. Recorded so a thin read is never mistaken for a
	// complete one.
	//
	// Not to be confused with FlagDepthReached. Measured 2026-08-04 over the
	// corpus's oversize pages: of 70 budget flags, 58 were tiles that stopped
	// because MaxDepth said to — normal termination reported as exhaustion. A
	// flag that is always on trains people to ignore flags, which is the same
	// reason FlagClipped was removed.
	FlagBudget = "budget"
	// FlagDepthReached — this region stopped descending because MaxDepth said to.
	// Normal termination, not a shortfall: at MaxDepth 1 every tile ends this way
	// by construction, and calling that a spent budget makes the budget flag
	// meaningless on exactly the sheets it matters for.
	FlagDepthReached = "depth-reached"
)

Region flags. Not a score: a confidence number here would be an invented statistic. These are conditions that either hold or do not.

View Source
const (
	// MinRunChars is the shortest run of identical text counted as a match, in
	// folded characters. 40 characters is six or seven words.
	//
	// This is the phrase-noise cutoff. Swept 1/20/40/80/200 on three pairs chosen
	// to span the difficulty:
	//
	//	                                        1      20     40     80    200
	//	Graham declaration vs its own email
	//	  attachment (byte-identical files)     .930   .927   .927   .927  .927
	//	two DIFFERENT Havern deeds, 1993 QCD
	//	  vs 2008 deed (same county forms)      .307   .301   .259   .033  .000
	//	1946 survey vs 1992 reconveyance
	//	  (unrelated)                           .010   .010   .000   .000  .000
	//
	// The shape is what decides it: a true duplicate's coverage does not move at
	// all across the whole sweep, so raising the floor costs nothing there, while
	// unrelated documents go to exactly zero by 40. The sensitive case is the
	// middle one — two different instruments on the same county's forms — and 40
	// leaves it at 0.26, less than half of DuplicateThreshold.
	//
	// Not 80, which would take that middle case to 0.03: it would also discard a
	// genuine paragraph-length quotation, which is a finding this corpus turns on.
	// Report-level noise is controlled by MinReportChars instead, which can be
	// raised without making short real matches invisible.
	MinRunChars = 40

	// MaxPostingsPerHash caps how many times a shingle may occur in the compared
	// document before it is ignored as an anchor.
	//
	// This is not a tuning knob, it is what keeps the alignment from going
	// quadratic on a degenerate document. A shingle occurring thousands of times is
	// repetitive filler — a run of dot leaders, a table of blank cells, a failed OCR
	// emitting one phrase over and over — and it cannot anchor a unique alignment,
	// only multiply seeds.
	//
	// Measured over every document on this corpus: the worst
	// (201503110042-2008-06-predecessor-survey-lotB.pdf, whose transcription is
	// largely one repeated string) would generate 67,829,506 seeds against itself
	// uncapped, and 510 capped. Five orders of magnitude, from one document. Same
	// reasoning as repeat masking in a sequence aligner.
	MaxPostingsPerHash = 24

	// DuplicateThreshold is the block coverage at which a document is called a copy
	// rather than an overlap.
	//
	// The measured separation on this corpus is wide. Two DIFFERENT recorded
	// instruments on the same county's forms reach 0.26; the worst genuinely
	// byte-identical pair reaches 0.92 (the Graham declaration against its own email
	// attachment — under 1.00 only because generic-text masking removes the same
	// passages from both sides). So anything from about 0.3 to 0.9 separates those.
	//
	// 0.55 is set by the one case that falls INSIDE that gap, and it is the case the
	// feature exists for: the operative 2008 record of survey is reproduced on page 4
	// of the 2008 lot certification, and aligns at 0.589 rather than ~0.95 because
	// the certification's OCR is substantially garbled ("LAURENCE MOONION" for
	// Clarence Brannock, "HAVERN TACOMA SHORE" for Havern Lake Shore). A threshold
	// of 0.60 called that pair a mere `overlap`, missing a real sub-document
	// containment by 0.011.
	//
	// Setting a threshold from a single example is normally how a measurement gets
	// fitted to noise. It is defensible here because the example is the DEGRADED-COPY
	// case rather than an outlier — degraded copies are the population this tool
	// serves — and because 0.55 is still more than twice the level two different
	// instruments reach. The error direction is deliberate: a false `duplicate`
	// costs one human comparison, a missed one lets a second copy of an instrument
	// enter the record unnoticed.
	DuplicateThreshold = 0.55
)
View Source
const (
	// FacetText is the words: what the document says, or what was spoken.
	FacetText = "text"
	// FacetSpeaker is who said them. Only diarized readings have it, and it is
	// the facet that fails independently of the words.
	FacetSpeaker = "speaker"
	// FacetLayout is reading order and structure — which column follows which,
	// where a table's cells belong. A page can be read perfectly and assembled
	// wrongly.
	FacetLayout = "layout"
	// FacetSubject is what a model says an image DEPICTS. Not a transcription of
	// anything: nobody wrote "a red Chevrolet Malibu", the model inferred it.
	FacetSubject = "subject"
)

Facets — the parts of a reading that can be trusted separately.

View Source
const CutOffWait = 10 * time.Second

CutOffWait is how long cancelled jobs get to write 'pending' back after a Drain's grace ends. A job that does not return in time keeps its 'running' row and is marked 'error' at the next start, as before.

View Source
const DefaultIdentitySlots = 4

DefaultIdentitySlots is how wide this worker's caption pipeline is.

It used to be a CAPACITY CLAIM — "two, because that is what the endpoint serves concurrently" — and that was a number about a server written down in raglit, which is exactly the kind that goes stale when the model layout changes. It is no longer one: the identity model has its own admission channel (modelchan.go), learned from the server's own backpressure, and that is what bounds requests now.

So this is just how many captions may be ASSEMBLED and in flight at once. Four rather than two, because the extra callers cost a goroutine each and simply wait in Acquire when the model is narrow — while a model that turns out to serve several would otherwise be held to two by a constant nobody revisited.

View Source
const DefaultIngestRunners = 8

DefaultIngestRunners is how many documents may be in flight at once.

Not a capacity claim — the model channels are that. It is how many documents can be PARTWAY through, and it wants to be comfortably more than the number of models, so that a document waiting on the vision channel never occupies the slot a document needing only embedding could use. Runners are goroutines; the scarce thing is downstream.

View Source
const DefaultMaxImageTokens = 16384

DefaultMaxImageTokens matches --image-max-tokens as configured on this fleet. It belongs to the SERVER, not the document, so it is a policy input rather than a constant folded into the arithmetic.

View Source
const DefaultSTTTimeout = 4 * time.Hour

DefaultSTTTimeout bounds one transcription when Timeout is unset. Four hours covers a full day's hearing at diarization's roughly-realtime cost with room for a loaded box; it is a runaway guard, not a service-level expectation.

View Source
const DefaultSessionBranchTTL = 7 * 24 * time.Hour

DefaultSessionBranchTTL is how long a session branch outlives its last write (USER 2026-10-30: kept 7 days after the session's last write, so a later session can still search it, then removed).

View Source
const DescribedUnmeasured = -1

DescribedUnmeasured marks a reading whose described fraction is UNKNOWN, as distinct from known to be zero.

The measurement is taken from layout markup that only exists during ingest, so a document indexed before that measurement existed cannot be scored without re-reading it — and 0 would claim a model made none of it up, about exactly the documents most likely to be screenshots. Negative so it can never be mistaken for a percentage, and queryable, so the set that needs re-reading can be listed rather than guessed at.

View Source
const FoldWidth = 32

FoldWidth is the shingle width, in folded bytes.

Measured on the ardley-v-brannock corpus — 233 indexed documents, 2.49 MB of fragment text, 441 true pages, 1.59 MB folded — by re-sketching the whole index at w = 16/24/32/48/64 and running the all-pairs audit at each:

w    reported pairs   duplicate-or-containment   hardest real pair
16        292                  56                      0.40
24        316                  58                      0.49
32        399                  57                      0.59
48        281                  65                      0.55
64        364                  60                      0.44

Every width finds every known duplicate and every known containment, and the known false positive is suppressed at every width. So w does NOT decide the boilerplate problem — generic-text masking does (see Mask in similar.go). What w decides is how much of a DEGRADED copy is recovered, and the last column is that: the 2008 record of survey against the copy reproduced inside the 2008 lot certification, whose OCR is substantially garbled. Coverage peaks at w=32 and falls away in both directions.

It falls off above 32 for the obvious reason — one wrong character destroys w shingles, so a wide window is fragile to a bad read. It falls off BELOW 32 for a less obvious one: a short shingle is far more likely to occur across many unrelated documents, so it clears the generic-DF cutoff and real text gets masked away. At w=16 even the byte-identical Graham pair drops to 0.85 coverage, against 1.00 at w=48. Both failure modes are worst exactly on the pages that are hardest to read, which is where the tool has to work.

View Source
const FragOriginDescribed = "described"

FragOriginDescribed marks a fragment whose text is a model's account of an IMAGE rather than a transcription of writing.

It sits beside origin='identity' (identity.go), which marks a generated caption, and means the same thing in the same column: findable by it, not quotable from it. A corpus that may be cited needs the difference between "the fax says" and "the model thinks the car is a Chevrolet" to survive into the search result, and until this existed it did not.

View Source
const FragOriginFrontmatter = "frontmatter"

FragOriginFrontmatter marks the one fragment that holds a Markdown file's YAML frontmatter block, delimiters included.

Unlike 'identity' and 'fields' this fragment is NOT generated: its text is the file's own bytes [0,N), with those offsets. So it stays inside SQLOwnWords — get_document reassembles the stored text exactly, and a quote taken from the block still verifies against it. The mark exists so a renderer can say this passage is document metadata and not part of the body (Hit.Caveat), and so the block is not windowed as body text.

View Source
const GenericDocFreq = 12

GenericDocFreq is the number of DISTINCT documents a sampled shingle must occur in before its text is masked out of every alignment.

Set from the measured frequency distribution of this corpus rather than from a round number. Of the distinct sampled shingles on this corpus, 17,801 occur in exactly one document and 4,739 in two; the count is in single digits by 9 documents, and then a thin tail runs out to 56. That tail IS the corpus's chrome — the lawyer's confidentiality notice reaches 56 documents, an email header template 32 — and it is only about 60 sampled shingles, which is why suppressing it costs so little.

The cutoff must sit ABOVE the number of documents a genuinely reproduced instrument appears in, and that is what fixes the value. Measured here: the operative 2008 record of survey's text is detectably reproduced in 13 documents, 4 of them substantially (coverage above 0.25). A cutoff at or below 4 would begin masking the very instrument this exists to find, and that failure is silent.

Measured across 8/12/18/25 on the all-pairs audit — 774 duplicate-or-containment pairs with no masking at all, then 51/57/57/60. The cutoff barely matters once masking exists; what the six extra pairs at 8 are is a judgment call rather than a correctness one. All six are VENDOR TEMPLATES — three Chicago Title order notifications for the same file number, two "Scanned from a Xerox Multifunction Printer" notices — so calling them duplicates is defensible and so is suppressing them. 12 keeps them (above the 4 substantive reproductions, below the 13 incidental ones) and --generic-df exists because that judgment belongs to whoever is triaging, not to this constant.

View Source
const GlobalPattern = ":global"

GlobalPattern in a Read list matches every index with no project namespace ("code", not "proj__code").

View Source
const IngestRetry5xxAttempts = 9

IngestRetry5xxAttempts rides out an upstream episode of roughly two minutes. With exponential backoff capped at 30s the schedule is about 1+2+4+8+16+30+30+30s; RetryBudget still stops it after that.

Ingest is a BATCH, not a chat turn, and agentkit's default 5xx cap of five attempts is tuned for the latter. A 33-page document is ~33 requests, so one upstream blip outlasting ~15s of backoff kills the whole document — measured on a real corpus, nothing over 20 pages ever completed.

The wall clock is still bounded by RetryBudget, which is the guard that should bind here. Raising attempts trades a longer wait on a sick upstream for finishing the document, which for a transcription backlog is the right trade: the alternative is not "fail fast", it is "re-read 30 pages".

View Source
const KeyIdleTTL = 7 * 24 * time.Hour

KeyIdleTTL is how long a session key lives past its last use: the same 7 days a session branch outlives its last write.

View Source
const KeyPrincipalPrefix = "key#"

KeyPrincipalPrefix marks a session key's own principal.

View Source
const MinClassifyChars = 250

classify names the relation from block coverage.

Coverage and not the shingle containments, because containment is what shared boilerplate inflates: two unrelated county deeds reached ContainProbe 0.34 on statutory phrasing alone, which would have been called a containment. Their block coverage was 0.06. MinClassifyChars is the least matched text a duplicate or containment claim may rest on, in folded characters.

A coverage ratio is unusable on a short document, and this corpus is full of short documents. 0.83 coverage of a 400-character email is a signature block; 0.83 of a two-page deed is the deed. 250 folded characters is roughly a substantial paragraph — below it the pair is reported as `overlap`, which says "these share text" without asserting one is a copy of the other.

View Source
const MinReportChars = MinClassifyChars

MinReportChars is the least matched text worth reporting at all, in folded characters. Same value as MinClassifyChars, deliberately: a pair too small to say anything about is not worth a line of output either.

Gating on matched LENGTH rather than on coverage, which was the first attempt and was wrong in both directions. A 0.05 coverage floor let through 1,440 pairs on this corpus whose median matched text was 75 folded characters — two shingles, a shared address line — while a floor high enough to kill those (0.20, leaving 150 pairs) would also discard a genuine paragraph-length quotation inside a long brief, which is 0.6% of a 40,000-character title commitment and is exactly the finding this corpus turns on. Length is the property that says whether there is anything there; coverage says what it means once there is.

Measured: 250 folded characters takes the audit from 1,497 reported pairs to 257, of which 57 are duplicate-or-containment. It discards nothing strong — every duplicate and containment survives it, because they all rest on thousands of matched characters, not hundreds.

250 FOLDED characters is about 305 raw ones on this corpus's text (folding removes roughly 18% of it as spaces and punctuation), so the floor is a ~50-word passage. A shorter quotation than that is genuinely below it and needs --min-chars to surface; that is the cost, and it is the one this floor trades for not reporting 1,240 pairs whose median shared text is 75 characters.

View Source
const MinSharedSamples = 3

MinSharedSamples is how many sampled shingles a document (or one of its pages) must share with the probe to be rescored.

At mod 32 an average page (3,600 folded chars) contributes ~90 samples, so 3 samples is a few percent of one page — far below anything that would be reported, which is the point: phase 1 should over-generate and let the exact rescore decide.

Measured with the operative record of survey as probe (100 sampled shingles): minShared 1 -> 19 candidate documents, 2 -> 16, 3 -> 16, 5 -> 11. So 3 costs nothing against 1 and 5 starts discarding candidates that would have been rescored. The whole probe (index load plus candidate selection) is 120 ms.

View Source
const NoisyFloor = 0.60

NoisyFloor separates "this page is the same page, read imperfectly" from "this page is different".

Set from the corpus rather than by taste: the 2008 lot certification's copy of the record of survey — the worst OCR in ardley-v-brannock, reading "LAURENCE MOONION" for Clarence Brannock — still holds most of its shingles against the clean copy. A floor low enough to call that page divergent would call every scanned page divergent, and the distinction this whole file exists to draw would be lost.

View Source
const PageJoin = "\n\n"

PageJoin separates pages in FoldedText.Raw. It folds away entirely, so it can never create or destroy a shingle — the fold is what makes the choice of separator irrelevant.

View Source
const ProjectHomeName = ".raglit"

ProjectHomeName is the per-project home directory `raglit init` creates in the working directory. Commands discover it by walking up from the cwd, so raglit is project-local like git: one .raglit/ per project, found from any subdir.

View Source
const ProjectPrincipalPrefix = "project:"

ProjectPrincipalPrefix marks a project key's principal, "project:<ns>". Every key of a project shares it, so the indexes one key creates stay reachable after that key is revoked and another minted.

View Source
const RootProject = "root"

RootProject is the name /mcp/root gives the indexes with no project namespace. It is reserved: no project can be called it.

View Source
const SampleMod = 32

SampleMod is the sampling divisor for the stored sketch: a shingle is kept when hash % SampleMod == 0.

This is NOT a MinHash signature, and the difference is the load-bearing part. A fixed-K MinHash signature estimates Jaccard well and CONTAINMENT not at all: the fraction of agreeing signature positions between a 2-page deed and the 40-page commitment containing it estimates |A∩B|/|A∪B| ≈ 0.05, and there is no way to recover |A∩B|/|A| = 1.0 from it, because the signature has thrown away the sizes. A first pass at this used K=128 MinHash and reported the deed-inside- commitment case as a 0.05 similarity with no containment signal — the exact failure the whole feature exists to avoid.

A mod-p sample (FracMinHash) is CONSISTENT: whether a shingle is sampled depends only on the shingle, never on the set it came from, so sample(A) ∩ sample(B) = sample(A ∩ B) and |A∩B| is estimable directly. Sizes are kept separately (shingle_pages.total), so both measures fall out.

32 keeps 1 shingle in 32. Measured on this corpus: 441 true pages, 1,587,596 folded characters, 1,290,196 distinct shingles, of which 39,778 are sampled — 1 in 32.4, so the sampling is doing what it says. That is ~90 samples on an average page (3,600 folded chars). The smallest region worth reporting is a paragraph, ~250 folded characters, which contributes ~8 samples at full containment — comfortably above the 3-sample floor candidate generation uses.

The whole index is 39,778 rows, against 1.29 M rows if the shingles were stored whole: a 32x saving on a table that has to be scanned per probe, and candidate generation loses nothing by it, because every candidate pair is then rescored EXACTLY over all shingles (see Compare in similar.go).

Variables

View Source
var (
	// ErrNoIdentifier means this store was never given a model to caption with.
	ErrNoIdentifier = errors.New("raglit: no identity model configured")
	// ErrIdentityKept means the document already has an identity that this run
	// must not replace — a person's, or a machine's without --force.
	ErrIdentityKept = errors.New("raglit: identity already recorded")
)

Errors IdentifyDocument returns for the two states that are not failures.

View Source
var DefaultLaneSlots = map[Lane]int{LaneHeavy: 1, LaneLight: 3}

DefaultLaneSlots is how many jobs each lane runs at once.

Heavy is 1 and should stay 1 unless the vision endpoint genuinely admits more than one request: a second concurrent page does not start earlier, it blocks inside the server, and its retries and timeouts are then invisible here.

Light is 3 rather than "as many as there are indexes". The ceiling that matters is the embedding endpoint, which is one server however many callers there are; three keeps it busy across a batch without turning one raglit into something that needs its own rate limit.

View Source
var ErrBadKey = errors.New("raglit: unknown, revoked or expired key")

ErrBadKey is a token that is unknown, revoked, expired or malformed.

View Source
var ErrForbidden = errors.New("raglit: only the index owner may change it")

ErrForbidden is a write refused on an index the caller can read.

View Source
var ErrIgnored = errors.New("raglit: ignored")

ErrIgnored is Enqueue refusing a path an ignore pattern matches. Callers that enqueue many paths skip it and count it: one ignored file must not fail the batch it came in.

View Source
var ErrNoDocType = errors.New("raglit: this document is not one of the index's document types")

ErrNoDocType is a document that has not resolved as any registered type. Not a failure: most documents in most corpora are not forms.

View Source
var ErrNoIndex = errors.New("raglit: no such index")

ErrNoIndex is what an unreadable index looks like from outside.

View Source
var ErrNoPageImages = errors.New("raglit: no embedded page images (born-digital PDF? pagify handles image/scanned PDFs only)")

ErrNoPageImages is returned when a PDF has no embedded page images — i.e. a born-digital, vector-text PDF. raglit's pagify is pure-Go (pdfcpu) and pulls images that are ALREADY in the PDF (scanned / image PDFs); it does not rasterize vector pages. Rasterization would need a native renderer, out of scope by design.

View Source
var ErrParentOnly = errors.New("raglit: the document is in the parent index only; a branch cannot change a parent's document")

ErrParentOnly is the error (errors.Is) a branch returns for a write to a document that only its parent holds. A branch cannot change a parent's document; ingest the document into the branch first.

Functions

func AppendAudit

func AppendAudit(path string, ev AuditEvent) error

AppendAudit writes one event. The only writer of the trail.

Sync before returning: an event that is in the page cache and not on disk is an event a power cut turns into a decision nobody can account for.

func ApplyRegionFilter

func ApplyRegionFilter(img image.Image, f RegionFilter) image.Image

ApplyRegionFilter runs one repair over a crop. An unknown filter returns the image unchanged — the caller validates; this does not silently invent one.

func AttachCheapOCR added in v0.5.0

func AttachCheapOCR(ocr *OCR, oc OCRConfig, render RenderPolicy) error

AttachCheapOCR enables the cascade's cheap first-pass engine on an OCR, with render as the resolution policy. An error is a misconfigured engine; the OCR is then left VLM-only, and the caller should report it rather than fail — a bad OCR knob must not break ingestion.

func AuditPath

func AuditPath(projectDir string) string

AuditPath is a project's trail — beside the documents, never under .raglit/.

func AuthorityFromTrail

func AuthorityFromTrail(anyDoc string) func(string) (string, bool)

AuthorityFromTrail builds the resolver Store.SetAuthorityResolver wants, from the judgement trail of the project a document belongs to.

anyDoc is any document in the corpus; the trail is found by walking up from it, the same way a correction is. Returns nil when there is no trail, which leaves the level order in charge — the right answer for a corpus nobody has ruled on, and better than an empty resolver that would claim every source is unruled.

Read ONCE and held: a search must not open and replay a trail per hit, and the rulings are a small append-only file that changes when a person acts, not while a query runs. A daemon that wants a change picked up re-installs it.

func ChildParent

func ChildParent(p string) string

ChildParent returns the bundle a child path derives from, or "".

func ContentIDs added in v0.7.0

func ContentIDs(path string, frags []Fragment) []string

ContentIDs is the content id Store.Ingest gives each of frags, in order, for a document at path: the id a Hit carries and FragmentExists takes. A caller that keeps its own fragments (the code index) uses it to decide what it has already embedded without asking the index for every one.

Exact for Store.Ingest, which replaces every fragment of the document and numbers them in the order given. A fragment Ingest skips (blank, or bytes that are not text) still gets an id here, one that no stored fragment has.

func DPICapForArea

func DPICapForArea(areaSqIn float64, maxTokens int) int

DPICapForArea is the highest resolution at which a sheet of areaSqIn can be shown WITHOUT the server downscaling it — the bound that falls as the canvas grows, and the reason a 27x36.7in sheet cannot be read whole at any setting.

func DamageOf

func DamageOf(img image.Image) []string

DamageOf reports what is measurably wrong with a crop's PIXELS, as flags.

Taken on the full-resolution crop, which is the point: the model is shown a downsampled version, and at that scale it cannot tell a blurred page from a small picture of a sharp one. Measured — asked to diagnose a crop blurred at sigma 1.6, the vision model answered "skew" and "noise" with 0.9 confidence and prescribed a deskew, which is the one repair that makes a blurred region WORSE. This function is what it could not see.

func DamageOfPNG

func DamageOfPNG(data []byte) []string

DamageOfPNG measures encoded image bytes — the form a region's crop is recorded and reproduced in, so a pass that re-renders a recorded read can measure exactly what was read without decoding it itself.

func DefaultRoot

func DefaultRoot() string

DefaultRoot is the DAEMON's storage root — $RAGLIT_ROOT, else ~/.raglit — under which each index is scoped at indexes/<name>/ (its own Home: index.sqlite + originals/ + pages/). This is the multi-index server layout (OpenScopedRegistry), distinct from a project Home (a single embedded index). The daemon's own config (endpoint + models) lives at <root>/config.json.

func DefaultTrust

func DefaultTrust(method string) map[string]Confidence

DefaultTrust is what a reading produced by this method can be trusted on, before anybody has ruled.

A facet that is ABSENT is not untrusted, it is not claimed: a page transcription asserts nothing about who spoke, and reporting 0 for it would read as "we know it is wrong".

func DescribableRead

func DescribableRead(raw string) bool

DescribableRead reports whether a page's raw text came from a reader that SAYS what it described — the only kind that can be measured.

The distinction between "described nothing" and "cannot tell" lives here, and it is not academic. A layout-aware read wraps every block in a labelled region and a figure-caption read leaves `[FIGURE: ...]` inline; a page from either can be scored, and 0 means the model described nothing. A page with NEITHER came from a reader that was never asked to mark descriptions — an older prompt, a different model — and its text is a transcription and a description run together with no seam. Measured on the fence-dispute index: 654 of 1422 cached vision pages are like this.

Scoring those 0 would be the worst available answer, because it looks measured. See DescribedUnmeasured.

func DescribedFraction

func DescribedFraction(raw string) float64

DescribedFraction is how much of a page's INDEXABLE text came from the model describing an image, between 0 and 1. A page with no such regions is 0; a photograph is ~1.

BOTH description mechanisms count. A layout-aware read wraps what it describes in an Image region with the text in an <img alt>; the figure-caption path writes `[FIGURE: ...]` inline instead (figures.go), and the corpus holds both — measured on the fence-dispute index, 768 cached pages carry layout markup and 201 carry figure markers, with no overlap. Counting only the first scored a page whose only description is `[FIGURE: Logo for Harrow Brand & Teal…]` at zero, which is not "we could not tell" but "a model made none of this up".

func DynamicRange

func DynamicRange(img image.Image) int

DynamicRange is the 1st-to-99th percentile spread of luminance. Percentiles rather than min/max because one black speck and one white pixel would report a full range on a page that is entirely mid-grey.

func EnqueueBuiltinIgnore

func EnqueueBuiltinIgnore() []string

EnqueueBuiltinIgnore is the ignore list every index applies, for display.

func EscalatePrompt

func EscalatePrompt(question string) string

EscalatePrompt asks a parent to decide GEOMETRY, and forbids a transcription.

Separate from Prompt because it takes an argument — the case being put — and because it replaces rather than composes. `keep` means the child's reading stands, never "here is a better one": anything the parent transcribed here would be the low-resolution invention this package exists to prevent.

func EstimateTokens

func EstimateTokens(s string) int

EstimateTokens is the token count raglit assumes for a piece of text.

Deliberately an OVER-estimate: it must never say a string is smaller than the tokenizer will. Shared so fragment sizing and batch sizing agree, because disagreeing about the unit is how a limit gets enforced in one place and ignored in another.

func ExtForContentType

func ExtForContentType(mime string) string

ExtForContentType maps a content type to a file extension for materializing fetched/base64 bytes to a temp file (external tools detect format by extension). Empty when unknown — the caller should sniff or default.

func ExtractEmailAttachments

func ExtractEmailAttachments(srcPath, archivePath, destDir string) (int, string, error)

ExtractEmailAttachments writes every attachment in the archive at srcPath into a sidecar beside archivePath, and returns how many files it wrote.

srcPath and archivePath differ on the ingest path: the worker materialises fetched bytes to a temp file, but the sidecar belongs next to the document's real location in the corpus. Passing both keeps the bytes and the identity from being confused for one another.

The archive is parsed a second time rather than threaded through EmailText. EmailText must stay pure — the `ocr` MCP tool calls it — and a second walk of 24 MB costs milliseconds, which is not worth a sink parameter that one of the two callers would always pass nil for.

func FigurePromptVersion

func FigurePromptVersion() int

FigurePromptVersion is the current figure-description prompt version, folded into recipe hashes (pool + frag_recipe) so a prompt change reprocesses the affected documents.

func FlattenForIndex

func FlattenForIndex(s string) string

FlattenForIndex is what the index should hold for a page: layout markup removed, markdown tables reduced to plain lines.

The two halves were written weeks apart for the same reason and neither was wired to anything. Composed here so there is ONE answer to "what goes in the index", and one place to change it.

func FlattenMarkdownForIndex

func FlattenMarkdownForIndex(md string) string

FlattenMarkdownForIndex turns a stored transcription into the plain lines the index should hold.

The artifact keeps its markdown: a monument table read as a table is worth having, and a reader opening <doc>.raglit-transcription.md should see one. The INDEX wants neither the pipes nor the rules — FTS5 tokenises them away as separators, but they cost fragment budget, they carry into embeddings as noise, and a cell boundary inside a phrase is a boundary the searcher did not type.

Cells become space-separated on one line, so a row stays one unit of text and a phrase spanning two columns still matches. Alignment rows (|---|:--:|) carry no content and are dropped entirely.

func GCBranches added in v0.8.0

func GCBranches(branches []Home, alive func(root string) bool) (removed int, err error)

GCBranches removes each branch home whose branch.json names a root for which alive returns false, and returns how many it removed. A home with no branch.json, or a branch with no root (a session branch), is never removed. The caller lists its branch homes and must not have them open.

func HEICToPNG

func HEICToPNG(ctx context.Context, path string) ([]byte, error)

HEICToPNG decodes a HEIC/HEIF photo to PNG bytes for the OCR path. PNG, not JPEG: these are photographed surveys, plats, permits and correspondence headed straight for OCR, and a second lossy generation ahead of OCR costs accuracy on exactly the small print that matters — the first (HEIC's own) generation is unavoidable, a second is not.

-auto-orient matters and is not a no-op to skip: a phone's sensor is landscape, and a portrait shot only reads right-side-up because of a rotation signal carried alongside the pixels — an OCR pass over the raw sensor orientation reads sideways text and transcribes it to nothing. HEIF carries that signal two ways: a container-level irot/imir transform (the MIAF-compliant mechanism nearly all camera-generated HEIC actually uses) or, occasionally, a bare Exif Orientation tag as compatibility metadata. -auto-orient applies either; ImageMagick's HEIC decoder historically only honored the container form (github.com/ImageMagick/ImageMagick issue #1232, fixed by commit ba470aad in the version this machine has), which is the form real camera output actually carries, so this is not a synthetic concern.

func HashHex

func HashHex(b []byte) string

HashHex is the hex sha256 of b — exported so callers can build pool recipe keys.

func HaveHEIC

func HaveHEIC() bool

HaveHEIC reports whether a HEIC/HEIF → PNG converter is on PATH.

func HaveLegacyDoc

func HaveLegacyDoc() bool

HaveLegacyDoc reports whether a binary .doc converter is on PATH.

func HavePandoc

func HavePandoc() bool

func HavePoppler

func HavePoppler() bool

HavePoppler / HavePandoc report whether the external extractors are available, so callers (and `raglit doctor`) can degrade gracefully.

func HaveXLS

func HaveXLS() bool

HaveXLS reports whether xls2csv (the legacy-.xls converter, part of the same catdoc package as antiword's fallback) is on PATH.

func HintBlock

func HintBlock(hint string) string

HintBlock renders the hint for a prompt, or "" when there is none. One wording, so the four prompts that carry it cannot present it differently — and labelled as the corpus owner's words, so a model weighs it as context about the collection rather than as an instruction from the document.

func HitPage

func HitPage(startPage int, spansJSON, text, query string) int

HitPage resolves the page a match sits on, given the fragment's recorded page spans and the query that found it.

startPage is returned unchanged when the fragment covers one page, when the spans are missing, or when no query term can be located — all three mean "nothing better is knowable here".

func IdentityKinds

func IdentityKinds() []string

IdentityKinds is every vocabulary, for a help string or a UI filter.

func IdentityRoleKinds

func IdentityRoleKinds() []string

IdentityRoleKinds is the vocabulary, for a help string or a UI filter.

func IdentityTextHash

func IdentityTextHash(text string) string

IdentityTextHash fingerprints the text a caption is written from. Content only — the same words in the same order hash the same however they were fragmented, so a re-fragmentation that changes no words does not read as a changed document.

func ImageTooSmallToBeAPage

func ImageTooSmallToBeAPage(data []byte) (bool, string)

ImageTooSmallToBeAPage reports whether an image is below the floor, with its dimensions for the message. Undecodable images pass — a format this build cannot read is not evidence of anything, and the OCR path will say so better.

func IndexTrees added in v0.9.0

func IndexTrees(ctx context.Context, stores map[string]*Store, root string, o TreeOptions) (map[string]TreeResult, error)

IndexTrees is IndexTree over several indexes: o.Route picks each file's index, and a document under root in an index its file does not route to is Removed from it. A file routed to a name not in stores is in none.

func Inited

func Inited(h Home) bool

Inited reports whether a home has a usable config.

func IsChildPath

func IsChildPath(p string) bool

IsChildPath reports whether a document path denotes a derived slice rather than a file. Anything that checks the filesystem must ask this first: a child has no file, and reading its absence as "the source vanished" would delete documents that are working exactly as designed.

func IsDescribedPage

func IsDescribedPage(raw string) bool

IsDescribedPage reports whether a page is essentially a model's description of an image rather than a transcription of text.

The BINARY question, and it stays binary because what it decides is binary: whether a fragment may be quoted as the record. A page mostly transcribed with a figure in it must not be marked, because the transcription is real.

func IsGeneratedSidecar

func IsGeneratedSidecar(path string) bool

IsGeneratedSidecar reports whether a path is raglit's own output rather than a source document. Generated output is never a source: indexing a transcription produces a transcription of the transcription, and the backlog grows every time the indexer runs.

func IsMailArchive

func IsMailArchive(path string) bool

IsMailArchive reports whether a path is one this reader handles.

func IsMinified

func IsMinified(head []byte) bool

IsMinified reports whether these bytes have no line structure a human put there — a minified bundle, a generated blob, a one-line data dump.

Such a file is TEXT, so IsOpaque passes it and the reader indexes it happily: measured here, a 289 KB minified xterm.js would fragment into hundreds of chunks of unreadable JavaScript and embed every one, answering no question anyone will ever ask. It is git-tracked, so a caller filtering by .gitignore cannot help — the defence has to be about the bytes.

func IsMixedPage

func IsMixedPage(raw string) bool

IsMixedPage reports a page that both transcribes and describes.

The case the binary test misses, and it is not rare — it is what a SCREENSHOT is. chandra reading a page of SMS messages transcribes the messages and narrates the phone around them: the status bar, the app icons, the microphone and camera buttons.

measured on the fence-dispute SMS exhibit 2026-08-15
  15 pages, all read by the VLM, 15% of the document described
  per page 0% to 28% — 13 of 15 pages carry some, 2 carry none

Every one of those pages failed IsDescribedPage and was recorded as pure transcription, so the reading claimed `text 90%` and claimed nothing at all about subject. That overstates one thing and hides another: a reader searching the text gets the model's prose back mixed in with the record — arrow, battery, blurry, button, camera, compose, icon, logo, microphone, navigation — with no sign of which is which.

func IsOpaque

func IsOpaque(head []byte) bool

IsOpaque reports whether these bytes have no representation raglit can index: not text, not an image, not audio or video, not a document format.

It exists because KindUnknown means two different things and the reader treats them the same. "A text file whose extension I do not recognise" and "a compiled binary" both fall through to being read as plain text, and reading bytes as text never fails — so a 27 MB ELF executable indexed cleanly, filled an index with mojibake, and reported `done`. Measured: dun handed its own build output to an ingest and most of a pool's growth from 49 MB to 237 MB was compiled binaries being segmented and embedded as though they were documents.

http.DetectContentType is the stdlib's WHATWG sniffer: it names the common text, image, audio, video and archive types and says application/octet-stream for everything else. Measured on the file that prompted this — the ELF sniffs as application/octet-stream, every .go/.md/.sum in the same tree as text/plain. `file(1)` answers correctly too and was considered; it costs a process per file and an external binary, against the pure-Go single-binary property this project holds elsewhere (see embed.go).

The NUL check is the second signal for the same reason git uses it: a byte that cannot appear in text is the cheapest proof there is, and it catches binaries whose first bytes happen to sniff as something benign.

audio/ and video/ are deliberately ABSENT from the pass list below, though the sniffer names them and an earlier version of this function let them through. This list is not "types the sniffer recognises", it is "types that may fall through to the TEXT READER" — which is the only thing past this guard.

Media has a route of its own now (KindAudio → oidio → index the transcript), and Worker.route takes it before anything reaches here: by extension, by served content type, and failing both by sniffing the bytes. So a recording should never arrive at this function at all, and one that does is a file whose identity nothing could establish. Handing THAT to the text reader is the failure this guard exists to stop, so the prefixes stay out.

Restoring them would not re-enable audio ingest — it would only re-open the fall-through. Measured before the route existed: an .mp4 of a court hearing was stopped by IsMinified alone, on the incidental grounds that its bytes hold a 5000-character run with no newline (208,179 in that file). The .opus beside it tops out at 3,508 across 7 MB and would have gone straight through. A line-length heuristic agreeing with the right answer is not the same as a check that knows it.

func IsRegionsSidecar

func IsRegionsSidecar(path string) bool

IsRegionsSidecar reports whether a path is raglit's own region output.

func IsTranscription

func IsTranscription(path string) bool

IsTranscription reports whether a path is raglit's own transcription output.

func IsTransient added in v0.9.0

func IsTransient(err error) bool

IsTransient reports whether err is a failure outside the document: the context cancelled or timed out, a dial or connection error, an HTTP 5xx, 429, 401, 403, 404 or 408 from a model or fetch endpoint, or a spent model budget (402, which refills). A document error (extract or parse failure, unsupported or unreadable content) is not transient.

func JudgementsPath deprecated

func JudgementsPath(projectDir string) string

JudgementsPath is retained so callers compile unchanged. The trail is the store; there is no database. JudgementsPath is retained as an alias for AuditPath.

Deprecated: there is no judgements database. The name outlived the file it named, and callers asking "where are the rulings" want the trail.

func LaplacianVariance

func LaplacianVariance(img image.Image) float64

LaplacianVariance is the variance of the 4-neighbour Laplacian: the standard cheap focus measure. Sharp edges produce large second derivatives, blur flattens them, so a low number means the strokes have been smeared.

func LegacyAttachmentDir

func LegacyAttachmentDir(archivePath string) string

LegacyAttachmentDir is where an archive's attachments USED to be extracted to: a sidecar directory in the corpus, beside the archive. Kept only so the migration can find them; nothing writes here any more. See Home.AttachmentDir.

func LegacyDocText

func LegacyDocText(ctx context.Context, path string) (string, error)

LegacyDocText converts a pre-2007 binary Word .doc to plain text.

Why not LibreOffice. `libreoffice --headless --convert-to docx` is the obvious answer and it is the wrong one here: it is a ~1GB dependency to read a format that is almost always a letter, it costs seconds per file against milliseconds, and it serialises on a single user-profile lock — so raglit's concurrent workers would either collide or need a throwaway -env:UserInstallation per job. antiword is ~200KB and does nothing else.

The trade it makes: antiword drops embedded images, so a .doc whose content is a scanned figure extracts as empty. That is the right trade for correspondence and agreements; if a .doc ever matters for its figures, convert it to PDF and let the PDF path rasterize and describe them.

func LegacyRelationsPath

func LegacyRelationsPath(projectDir string) string

LegacyRelationsPath is the append-only file rulings used to live in, kept only so an existing corpus can be migrated into the database once.

func MaterializeSlice

func MaterializeSlice(s *Store, sl Slice) (int, error)

MaterializeSlice writes the derived child document for a declaration.

The child's text is the parent's pages, carried over unchanged, and its page numbers stay the PARENT's: a child of pages 3-6 has pages 3, 4, 5, 6. Never renumbered to 1-4 — a quotation cited from the child has to be checkable against the exhibit as filed, and renumbering makes "p. 1" mean two different sheets depending on which document you are holding.

One fragment per page rather than the windowing fragmenter. The child exists to be cited and scoped, and the parent is already indexed at fragment grain for deep search; keeping page and fragment the same unit here means an offset in a child result needs no translation to become a page in the bundle.

Idempotent: Ingest replaces a document's fragments, so re-running after the parent is re-read refreshes the child. That is the whole reason the bundle stays the single source of bytes — one thing to fix, and the children follow.

func NativeDPI

func NativeDPI(ctx context.Context, pdfPath string, page int) int

NativeDPI reports the resolution of the largest image on a PDF page, or 0 when there is none (born-digital) or poppler is unavailable.

Largest rather than first: a scanned sheet often carries a small logo or signature stamp alongside the page raster, and the smaller image's ppi says nothing about what the page holds.

func NormalizeAnyKind

func NormalizeAnyKind(s string) (string, bool)

NormalizeAnyKind maps a kind onto the first profile that knows it. A bare legal term ("deed", "court filing") is accepted for what was stored before kinds had families.

func NormalizeIndexName

func NormalizeIndexName(name string) string

NormalizeIndexName is the exported form of normalizeIndexName, for clients that build daemon index names (e.g. project namespacing).

func NormalizeKind

func NormalizeKind(s string) (string, bool)

NormalizeKind maps a kind onto the vocabularies — any profile's, since a person may file a document under any family — returning ok=false when no profile knows it. A bare legal term ("deed") is what was stored before kinds had families, and still maps.

func NormalizeRole

func NormalizeRole(s string) (string, bool)

NormalizeRole maps a model's role tag onto identityRoleKinds, returning ok=false when it is not a term this vocabulary knows.

func NormalizeTypeName

func NormalizeTypeName(s string) string

NormalizeTypeName is how a type name is compared: lowercased and whitespace-collapsed, so "Work Order" and "work order" are one type rather than two that each hold half the corpus.

func OfficeText

func OfficeText(ctx context.Context, path string) (string, error)

OfficeText extracts text from any KindOffice document, choosing the converter by extension — except an OLE2 header overrides the extension, because a renamed .doc is still a .doc underneath and pandoc still cannot read it. Callers route KindOffice here rather than to PandocText directly, so adding a format that pandoc cannot read is a change in one place.

func PageAt

func PageAt(startPage int, spans []PageSpan, off int) int

PageAt resolves a byte offset inside a fragment's text to the page it falls on. This is the reason the column exists: a search hit is an offset, and without the boundaries the only answer available was the fragment's start page, which is wrong for everything after the first stitch.

func PandocText

func PandocText(ctx context.Context, path string) (string, error)

PandocText converts an office/markup document (docx, odt, epub, html, pptx, …) to plain text via pandoc, which auto-detects the input format from the file extension. The path must have the right extension.

func ParseRange

func ParseRange(s string) (from, to int, err error)

ParseRange reads "3-6" or "3" as an inclusive page range.

Inclusive because that is how a person reads a document: "pages 3-6" is four pages, and a half-open range here would silently drop the last page of every slice anybody declared by eye.

func PlanSources

func PlanSources(cfg Config, baseDir string) (map[string][]string, error)

PlanSources returns, per index name, the absolute file paths its configured roots + rules select. baseDir is the project directory (relative roots resolve against it). It shells out to `git ls-files` for a root's .gitignore semantics when the root is a git work tree and cfg.Gitignore isn't false.

func PrincipalFrom

func PrincipalFrom(ctx context.Context) string

PrincipalFrom returns the authenticated account, "" when there is none.

func ProjectDir

func ProjectDir() (string, bool)

ProjectDir is the directory that OWNS this project — the one holding .raglit/, found by walking up from the cwd like git.

Distinct from the index Home on purpose. The Home is derived storage: it is gitignored in every project that has one, it is rebuilt by a reindex, and on a daemon it does not even sit near the documents. Anything DURABLE about the corpus — a human's ruling that two documents are versions of one instrument, which nothing can recompute — belongs here instead, beside the documents it is about, where git records it and a folder sync carries it to the other machines.

func ProjectDirForDoc

func ProjectDirForDoc(docPath string) string

projectFlagForDoc walks up from a document for the nearest project config and lets it decide, falling back to the store's setting when there is none. See writebackForDoc for why the document's path, and not the daemon's config, is the right place to ask. ProjectDirForDoc finds the project a document belongs to, by walking up from the document itself until it meets a .raglit/ directory.

The daemon needs this and cannot use ProjectDir(): it runs from its own home and its working directory has nothing to do with the corpus. What it always has is the document's absolute path, and the corpus layout answers the rest — the same walk that already decides per-project writeback settings.

Returns "" when the document sits under no project, which is a real state (an ad-hoc file ingested by path) and not an error.

func Prompt

func Prompt(k PromptKind, mods ...Mod) string

Prompt assembles the instruction for one read.

Modifiers on a kind that does not accept them are DROPPED, not silently appended: see AcceptsMods for why each would contradict its prompt.

func RandomHex added in v0.4.1

func RandomHex(n int) string

RandomHex is n random bytes, hex encoded.

func Recipe

func Recipe(w, mod int) string

Recipe identifies the comparison parameters. Mirrors documents.frag_recipe: a parameter change should mark exactly what it affects, and a finding computed under one recipe is not comparable to one computed under another.

func RecordReadingsInto

func RecordReadingsInto(js Judgements, s *Store)

RecordReadingsInto wires a judgement store to an index so that every change to a page's ACTIVE reading lands as a row.

This is the hook the interface exposes, used for its intended purpose: a correction invalidates things it does not own — the export beside the document and the fragments in the index — and neither notices on its own. Registering here means a person correcting a page leaves a visible, queryable version history rather than a silently changed file.

func RegionSHA

func RegionSHA(data []byte) string

imageSHA identifies a rendering by its bytes.

This is also the cycle detector, at no extra cost. A transform that renders to bytes already seen IS the cycle: rotating 90 degrees four times returns the original SHA, and re-asking for the same bbox at the same dpi and rotation returns the original SHA. RegionSHA digests rendered image bytes the way a region's record does. Exported so a caller outside this package can construct or check a region record against an image it rendered itself.

func RegionsPath

func RegionsPath(docPath string) string

RegionsPath is where a document's region reads live: beside it, like the transcription.

func RegionsSuffix

func RegionsSuffix() string

RegionsSuffix is what a region sidecar's name ends in, for a caller walking a tree and mapping sidecars back to the documents they describe.

func RenderTranscription

func RenderTranscription(docPath string, pages []TranscribedPage) string

func RenderTranscriptionCorrected

func RenderTranscriptionCorrected(docPath string, pages []TranscribedPage, corrections map[int]PageCorrection) string

RenderTranscription formats pages as markdown.

Deterministic: same pages in, same bytes out, so a diff between two transcriptions is a real change in what was read rather than formatting noise. RenderTranscriptionCorrected renders with a person's page corrections applied.

The corrections are NOT stored in the file this produces. That file is regenerated on every read, and corrections kept in it were destroyed twice by ordinary re-reads before they were moved into the judgement store. Rendering applies them; the store keeps them.

func RerenderRegion

func RerenderRegion(page image.Image, reg *Region) ([]byte, error)

RerenderRegion reproduces the crop a region was read from: same bbox, same rotation, same page rasterization.

page must be the page rasterized at reg.DPI — the caller owns that, because rasterizing a PDF needs a native renderer this package deliberately does not link (see pagify.go). The result is byte-identical to what the descent produced; VerifyRegionRender is how you find out, rather than assuming.

This is the image to put in front of a human. It is the passage at the resolution it was read at, which is the whole point, and where a context downscale intervened it is SHARPER than what the model got — more detail on the same pixels, never different pixels. RerenderRegionAsSeen is for the other question.

func RerenderRegionAsSeen

func RerenderRegionAsSeen(page image.Image, reg *Region) ([]byte, error)

RerenderRegionAsSeen is the crop with the context downscales replayed: what the model was actually given, rather than what was cropped for it.

A diagnostic, not the attestation image. It answers "could this have been read at all", which is a different question from "does the document say this", and it is deliberately NOT what reg.SHA256 covers: the digest is taken before the call, because it is also the descent's cycle detector. So this is reproducible — same crop, same factor, same count — but verifiable only through the crop it is derived from.

func ResolveExtractedAttachments

func ResolveExtractedAttachments(dirs ...string) func(string) string

ResolveExtractedAttachments maps sha256 → extracted file path, by hashing what is in the given directories.

By CONTENT, not by name. The extractor dedups within an archive — the same PDF on five messages in a thread is one file cited five times — so the filename it chose encodes the FIRST message that carried it and reconstructing it per message would point four of those five at a file that does not exist.

Hashing the directory on each call is affordable because these directories hold tens of files, and it cannot drift from whatever is actually on disk, which reconstructing a name from the extractor's format string could. Several directories because an archive may not have been migrated yet: current extractions land in raglit's storage (Home.AttachmentDir) and older ones are still in the corpus sidecar (LegacyAttachmentDir). Reading both means the thread view works either side of the migration.

func ResolveFragParams

func ResolveFragParams(window, stride, floor, embedLimit int) (int, int, int)

ResolveFragParams applies defaults for non-positive values, caps the window by the embed model's input limit when known (embedLimit > 0), and enforces the windower's invariants: stride ≤ ¾·window (so a boundary-snapped window can't leave a coverage gap before the next window's start) and floor ≤ window.

func STTReading

func STTReading(mediaPath, sha256, producer string, r *sttResponse) (*attest.Reading, error)

STTReading converts oidio's response into the reading attest reviews.

One unit per speaker turn, located by TIME — which is the locator attest already defines for audio and the reason Locator has a Time arm at all. The speaker becomes the unit's Label rather than part of its Text, because they are separately correctable: a reviewer can rule that the words are right and the speaker is wrong, and attest's own comment says conflating those two is what made oidio's earlier workbench painful. Sealed before it is returned, so the units carry content ids rather than the "t0, t1, …" handles used while building. Seal treats those as the producer's own handles and rewrites them, which is exactly what they are.

func Sampled

func Sampled(h uint64, mod int) bool

Sampled reports whether a shingle hash is in the stored sketch.

func SaveConfig

func SaveConfig(h Home, cfg Config) error

SaveConfig writes cfg to <home>/config.json (0600 — it holds a token), creating the home layout if needed.

func ScopeMatches

func ScopeMatches(scopes []string, dir string) bool

ScopeMatches reports whether dir lies in one of scopes. A plain directory matches itself and everything under it; a pattern with * or ? is a glob in the ignore-pattern syntax (** crosses directories), matched against dir.

func SessionBranchParent added in v0.4.0

func SessionBranchParent(name string) (string, bool)

SessionBranchParent returns the index a session branch name overlays ("proj__code@s1" → "proj__code"), and false for a name that is not one.

func Shingles

func Shingles(folded string, w int) []uint64

Shingles hashes every width-w window of folded text, at stride 1.

Result index i is the shingle STARTING at folded offset i, and callers depend on that: the alignment in similar.go recovers matched spans by chaining positions, so a position must mean an offset and nothing else. Fewer than w bytes yields nothing — a page too short to shingle contributes no evidence, which is correct, but it is also why a page's folded length is stored alongside its shingle count.

func SourceFile

func SourceFile(path string) (string, bool)

SourceFile is the file on disk a document was read from, and false when the document has none to check: a URL, or a relative path.

A page range (slices.go, "<file>#p3-8") is its parent's file. A file inside another file (an archive member) is not detected here: its path does not exist even while the archive does, so it would read as missing. Members are written out to real files (emailattach.go), so none exist today.

func StripLayoutMarkup

func StripLayoutMarkup(s string) string

StripLayoutMarkup removes a VLM's layout markup, keeping the text and the figure descriptions.

Block tags become line breaks and inline tags become nothing — the distinction is the whole point, and getting it backwards is worse than not stripping at all, because it silently corrupts phrases rather than obviously leaving tags.

func SuspectDocs

func SuspectDocs(root string) (map[string][]int, error)

SuspectDocs returns the documents under root whose transcription has at least one page that does not look like a page, with the page numbers.

Reads the SIDECARS rather than the index: the sidecar is what a human and kgraph both actually consume, so a document whose sidecar is wrong is wrong for every purpose that matters even if the index rows happen to be fine.

func SuspectPages

func SuspectPages(pages []TranscribedPage) map[int]string

SuspectPages returns the page numbers a human should look at, with reasons.

func Suspicion

func Suspicion(text string) string

Suspicion returns a human-readable reason this page looks wrong, or "".

The reason is written for the person who has to act on it: what was seen, and what it usually means.

func TagLine

func TagLine(tags []TagCount) string

TagLine renders tags as "tag(count), tag(count)" — the form the identity prompt takes its vocabulary in, and the form the digest prints.

func TilesNeeded

func TilesNeeded(areaSqIn float64, dpi, maxTokens int) int

TilesNeeded is how many equal pieces a sheet must be cut into for every pixel to reach the model at dpi. Ceil, because a fractional tile still costs a call and a sheet 1.1x over the cap still loses a tenth of itself.

func TokensPerSqInAt

func TokensPerSqInAt(dpi int) float64

TokensPerSqInAt is the encoder's cost of one square inch rendered at dpi.

func TokensToChars

func TokensToChars(tokens int) int

TokensToChars converts a token budget to the character budget that cannot exceed it, whatever the text.

The exact inverse of EstimateTokens, allowance included. Without subtracting it the budget overshot its own cap by the allowance — a budget that fails its own check is worse than no budget, because it looks like a guarantee.

func TranscriptText

func TranscriptText(name string, r *sttResponse) string

TranscriptText renders the transcript raglit INDEXES.

Written for a reader and for retrieval, not as a data format — the reading sidecar is the data format. Speaker and timestamp lead each turn so a search hit can be taken back to the recording, which is the whole reason a transcript of a hearing is worth indexing at all.

func TranscriptionPath

func TranscriptionPath(docPath string) string

TranscriptionPath is where a document's transcription lives: beside it, with a suffix that says what wrote it, so a re-transcription is detectable and a human can tell it from a hand-written note.

func UploadURL

func UploadURL(path string) string

UploadURL is the queue/document identity of an upload saved at path.

func ValidRegionFilter

func ValidRegionFilter(f RegionFilter) bool

ValidRegionFilter reports whether a name is one this package can apply. A proposal naming anything else is refused rather than silently ignored, so a model inventing "denoise" is visible instead of quietly doing nothing.

func ValidateFieldSchema

func ValidateFieldSchema(raw json.RawMessage) error

ValidateFieldSchema refuses a schema that is not a JSON Schema OBJECT with properties. The extraction call turns it into a tool's `parameters`, and an endpoint handed a malformed one fails in a way that reads as a model problem.

func VerifyRegionRender

func VerifyRegionRender(reg *Region, data []byte) error

VerifyRegionRender reports whether a re-render is the image the region's text was read from.

Byte equality, via the digest the read itself recorded. A weaker check — same bbox, same rotation — would pass an image cropped from a page rasterized by a different tool at the same nominal dpi, which is precisely the substitution a human attesting a quotation is being asked to rule out.

func WithPrincipal

func WithPrincipal(ctx context.Context, p string) context.Context

WithPrincipal records the authenticated account on a request context.

func WorktreeOf added in v0.8.0

func WorktreeOf(dir string) (mainRoot string, ok bool)

WorktreeOf returns the work tree of the main checkout when dir is inside a linked git worktree. ok is false for a main checkout, a bare repository's worktree and a directory outside git. It asks git (rev-parse --git-common-dir) and, when git cannot run, reads the .git file's gitdir pointer. It only answers the question; the caller decides which index is the parent.

mainRoot is spelled as git records it, which is usually with symlinks resolved, and may differ from the spelling of dir and of the paths an index holds. Before passing it to OpenBranch, spell it as the parent index's document paths are spelled (see OpenBranch).

func WriteRegionDoc

func WriteRegionDoc(docPath string, pages ...RegionPage) (string, error)

WriteRegionDoc merges page reads into the sidecar beside a document.

Merges rather than replaces: `raglit regions` reads ONE page, and a run over page 7 must not delete the record of page 3 — the crop that page 3's quotation was read from would go with it, and the quotation would still be in the filing.

func WriteTranscription

func WriteTranscription(docPath string, pages []TranscribedPage) (string, error)

WriteTranscription materialises the transcription beside the document.

Best-effort by design: a corpus can be read-only, or on a mount that is gone, and failing an otherwise-good ingest because a convenience file could not be written would be the wrong trade. The caller logs and continues.

func WriteTranscriptionCorrected

func WriteTranscriptionCorrected(docPath string, pages []TranscribedPage, corrections map[int]PageCorrection) (string, error)

WriteTranscriptionCorrected writes the export with a person's corrections applied. See RenderTranscriptionCorrected for why they are applied here rather than stored here.

Types

type ACL

type ACL struct {
	Owner      string     `json:"owner"`
	Visibility Visibility `json:"visibility"`
}

ACL is an index's owner and visibility, stored at <index home>/acl.json.

type Assembler

type Assembler struct {

	// MinChars: absorb sub-floor siblings up to this size (0 disables the floor).
	// MaxChars: never absorb past this.
	MinChars, MaxChars int
	// contains filtered or unexported fields
}

func NewAssembler

func NewAssembler(sink func(page, ord int, text string, spans []PageSpan) error) *Assembler

NewAssembler builds an Assembler; sink finalizes a closed fragment (e.g. insert row + hand to the embed pipeline).

func (*Assembler) Close

func (a *Assembler) Close() error

Close finalizes the last open fragment (end of document).

func (*Assembler) Feed

func (a *Assembler) Feed(page int, r SegResult) error

Feed processes one unit's segmentation. page is the unit's page number (0 for text windows, or a running window index).

func (*Assembler) OpenText

func (a *Assembler) OpenText() string

OpenText is the current open (deferred) fragment's text, or "" — passed to the next unit's segmentation as continuation context.

type AttachmentMove

type AttachmentMove struct {
	Archive string `json:"archive"`
	OldPath string `json:"old_path"`
	NewPath string `json:"new_path"`
	// Err is set when this one could not be moved. The migration continues: a
	// half-moved corpus with a report is recoverable, a migration that stops at
	// the first bad file leaves nobody knowing which side of it they are on.
	Err string `json:"error,omitempty"`
}

AttachmentMove is one document this migration would move, or moved.

type AttestationRow

type AttestationRow struct {
	Doc       string
	Seq       int
	Unit      string
	Kind      string
	Blanket   bool
	Text      string
	Label     string
	Note      string
	Statement string
	Payload   string
	RuledBy   string
	Auth      string
	RuledAt   string
}

AttestationRow is one entry of one asset's log, as stored.

type AuditEvent

type AuditEvent struct {
	Op string `json:"op"`
	// At is when the event was appended, not when a person decided — the decision
	// date lives on the payload. Both matter and they are not the same: a ruling
	// made in July and imported in September is one of each.
	At string `json:"at"`
	By string `json:"by,omitempty"`

	Relation   *Mark           `json:"relation,omitempty"`
	Slice      *Slice          `json:"slice,omitempty"`
	SliceID    string          `json:"slice_id,omitempty"`
	Correction *PageCorrection `json:"correction,omitempty"`
	Withdrawal *Withdrawal     `json:"withdrawal,omitempty"`
	// RestorePath names the document a doc.restore returns to the corpus.
	RestorePath string `json:"restore_path,omitempty"`
	// Authority names the reading a person ruled governs for a source. See
	// ReadingAuthority — it is a ruling, so it lives in the trail rather than in
	// the index, which is derived and thrown away by a reindex.
	Authority *ReadingAuthority `json:"authority,omitempty"`
}

AuditEvent is one recorded mutation.

func ReadAudit

func ReadAudit(path string) ([]AuditEvent, error)

ReadAudit parses the whole trail, in order. A missing file is an empty trail, which is the normal state of a corpus nobody has ruled on.

A malformed line is an ERROR and not a skip. The trail is the source of the database, so silently ignoring a line it cannot parse would rebuild a database that quietly disagrees with the record — the failure this whole arrangement is meant to make impossible.

type AuthConfig

type AuthConfig struct {
	// Issuer is the OIDC issuer URL (discovery at <issuer>/.well-known/
	// openid-configuration). Empty → the daemon does not authenticate.
	Issuer string `json:"issuer,omitempty"`
	// Audience is the token's required aud: the client id (or project id) the
	// IdP issues raglit tokens for.
	Audience string `json:"audience,omitempty"`
	// Admin is the principal ("<iss>#<sub>") that owns every index without an
	// acl.json — the indexes from before auth. Required when Issuer is set.
	Admin string `json:"admin,omitempty"`
	// ClientID and RedirectURL turn on BROWSER login (authorization code +
	// PKCE, session cookie) for the review UI. ClientID is a public (no secret)
	// web/user-agent application at the IdP; RedirectURL is
	// <daemon public URL>/auth/callback, registered there exactly. Both empty →
	// the browser gets 401 like any client without a token.
	ClientID    string `json:"client_id,omitempty"`
	RedirectURL string `json:"redirect_url,omitempty"`
	// TokenCommand prints a bearer token on stdout. The client runs it when
	// $RAGLIT_TOKEN is unset, and again after a 401. A headless client (a
	// Zitadel machine user) puts its client-credentials exchange here.
	TokenCommand string `json:"token_command,omitempty"`
}

AuthConfig is the daemon's OIDC setup (issuer/audience/admin) and the client's token source (token_command). One block because one config.json serves both sides on a single machine; a remote client sets only token_command.

type Block

type Block struct {
	ProbeFromPage int `json:"probe_from_page"`
	ProbeToPage   int `json:"probe_to_page"`
	MatchFromPage int `json:"match_from_page"`
	MatchToPage   int `json:"match_to_page"`
	// Runs is how many separate identical spans the block was chained from. 1
	// means the region is identical throughout; a high count on a short block
	// means the two copies differ constantly, which is itself the finding.
	Runs int `json:"runs"`
	// MatchedChars is folded characters proved identical inside the block;
	// SpanChars is the block's extent on the probe side. Agreement is their ratio.
	MatchedChars int `json:"matched_chars"`
	SpanChars    int `json:"span_chars"`
	// Gaps are the disagreements inside the block, largest first, capped.
	Gaps []Gap `json:"gaps,omitempty"`
	// contains filtered or unexported fields
}

Block is a chained group of runs — one reportable aligned region, with the page ranges that make it actionable.

func (Block) Agreement

func (b Block) Agreement() float64

Agreement is the fraction of the block's extent proved identical. Below 1.0 the two copies of this passage are not the same text, and Gaps says where.

type BranchChange

type BranchChange struct {
	Path       string `json:"path"`
	Change     string `json:"change"` // "added", "changed", "same" (a copy with the parent's bytes) or "deleted"
	Hash       string `json:"hash,omitempty"`
	ParentHash string `json:"parent_hash,omitempty"`
	Title      string `json:"title,omitempty"`
}

BranchChange is one document a branch differs from its parent on.

type BranchInfo

type BranchInfo struct {
	Name           string `json:"name"`
	Parent         string `json:"parent"`
	CreatedAt      int64  `json:"created_at"`
	LastAccessedAt int64  `json:"last_accessed_at"`
	Documents      int    `json:"documents"`
}

BranchInfo describes a branch for listing: lineage, age, last access, and its LOCAL document count (the branch's own diffs over the parent).

type BranchMeta

type BranchMeta struct {
	Parent         string `json:"parent"`
	Root           string `json:"root,omitempty"`
	ParentRoot     string `json:"parent_root,omitempty"`
	CreatedAt      int64  `json:"created_at"`
	LastAccessedAt int64  `json:"last_accessed_at"`
}

BranchMeta records a branch's lineage + access times. Stored as branch.json in the branch's scoped Home; absent for non-branch indexes.

Parent is an index name for a Registry branch and the parent's home directory for one opened with OpenBranch. Root and ParentRoot are the worktree's and the main checkout's directories (OpenBranch); empty for a branch that does no path translation.

Fields:

  • Parent: the parent index (name or home directory, see above).
  • Root: the worktree directory this branch indexes; empty without path translation.
  • ParentRoot: the main checkout directory the parent indexes; empty without path translation.
  • CreatedAt: when the branch was forked, in Unix nanoseconds.
  • LastAccessedAt: when the branch was last read, in Unix nanoseconds. Updated at most once a minute, so it can lag a real query by up to that long.

func ReadBranch added in v0.8.1

func ReadBranch(home Home) (BranchMeta, bool, error)

ReadBranch reads home's branch.json without opening the index. ok is false, with a nil error, when the home has no branch.json (a plain index). An unreadable or malformed branch.json is an error, so a caller does not take a damaged branch for a plain index.

type Changes added in v0.9.0

type Changes struct {
	Root     string
	Paths    []string
	Degraded bool
}

Changes is one burst. Paths are absolute, sorted, and may name directories and paths that no longer exist; IndexTree's Paths takes them as they are. Empty Paths means the whole tree (Rescan, an event overflow). Degraded means the watcher hit the OS watch limit and stopped adding watches: from then on changes in new directories are not seen, and the caller rescans or polls on its own trigger.

type ChannelStat

type ChannelStat struct {
	Model    string `json:"model"`
	Width    int    `json:"width"`
	InFlight int    `json:"in_flight"`
	Peak     int    `json:"peak"`
	Calls    int    `json:"calls"`
	N429     int    `json:"n_429"`
	// Chilled is true while backpressure is still holding the width down.
	Chilled bool `json:"chilled,omitempty"`
}

ChannelStat is one model's admission channel, for the status report.

type Channels

type Channels struct {

	// Max is the widest any channel may grow. 0 → maxChannelWidth.
	//
	// The one knob worth configuring, because the ceiling is the only part of
	// this that depends on WHAT KIND of endpoint is behind it rather than on
	// evidence. A local card serving one model at a time is never going past a
	// handful; a hosted provider bought for throughput will take far more, and
	// there the ceiling — not the provider — would be the limit. Everything else
	// is learned.
	Max int
	// contains filtered or unexported fields
}

Channels holds one admission channel per model name.

func NewChannels

func NewChannels() *Channels

NewChannels builds an empty registry. A model is created on first use, so nothing needs to know the model layout in advance — which is the property that makes a changing layout a non-event.

func (*Channels) Acquire

func (c *Channels) Acquire(ctx context.Context, model string) (release func(ok bool), err error)

Acquire takes a slot for one call to model, blocking until one is free or ctx ends. The returned release must be called exactly once.

release takes the call's outcome: ok=false for a call that failed for any reason other than backpressure. Backpressure is reported separately by Note429, because agentkit RETRIES a 429 internally — the call that eventually succeeds still met backpressure, and a controller that only looked at final outcomes would never see it.

func (*Channels) Gate

func (c *Channels) Gate(inner Chatter, model string) Chatter

Gate wraps a Chatter so every completion takes a slot in model's channel.

Chatter and VectorClient are the only two seams raglit calls a model through — OCR, segmentation and identity all take a Chatter; embedding takes a VectorClient — so gating these two covers every model call in the process without touching any of the four call sites.

func (*Channels) GateVector

func (c *Channels) GateVector(inner VectorClient, model string) VectorClient

GateVector wraps an embedding client the same way. Embed is synchronous, so the slot spans the call exactly.

func (*Channels) Note429

func (c *Channels) Note429(model string)

Note429 records backpressure for a model: halve the width and refuse to grow for a cooldown.

Multiplicative decrease against additive increase, which is the pairing that converges when several clients share one resource — and raglit IS one of several. Halving on the first 429 rather than after a few is deliberate: the server has already queued or rejected the call by the time it says this, so the cost of over-asking has been paid.

func (*Channels) Observe

func (c *Channels) Observe(model string, next func(llm.RetryEvent)) func(llm.RetryEvent)

Observe returns an llm.Client.OnRetry hook for model that feeds backpressure into the controller and then chains to next (which may be nil).

Chained rather than replacing, because the RetryTally is already on that hook and its record — what a job had to survive — is what the health report reads.

func (*Channels) Stats

func (c *Channels) Stats() []ChannelStat

Stats reports every model this process has called, widest first. Reported because the width is now a LEARNED number rather than a configured one, and a learned number nobody can see is indistinguishable from a bug.

type Chatter

type Chatter interface {
	ChatStream(ctx context.Context, messages []llm.Message, tools []llm.ToolDef,
		opts *llm.ChatOpts) (<-chan llm.StreamChunk, error)
}

Chatter is the sliver of *llm.Client the OCR and segmentation paths need: one streaming completion. An interface so the model can be stubbed in tests.

STREAMING, not Chat: see the file doc. A non-streaming client cannot be protected from a runaway generation, because by the time it returns, the tokens are already paid for.

type CodeSegmenter added in v0.7.0

type CodeSegmenter interface {
	// ForPath names the language of a path, from its extension. ok is false
	// for a file the segmenter cannot parse.
	ForPath(path string) (lang string, ok bool)
	// Symbols returns the declarations of content, parsed as lang.
	Symbols(lang string, content []byte) ([]CodeSpan, error)
	// Sections returns one span per heading section of a Markdown document.
	Sections(content []byte) ([]CodeSpan, error)
	// Runtime names the parser and its version. It is in the pool recipe,
	// because two parsers can cut the same file differently.
	Runtime() string
}

CodeSegmenter cuts source text into CodeSpans. The root package defines the interface and does not implement it: raglitd sets Models.Code to an adapter over poly-lsp-mcp's segment package, so the parser and its grammars are not in the engine's import closure.

type CodeSpan added in v0.7.0

type CodeSpan struct {
	Sym, Class string
	StartLine  int
	EndLine    int
	// CommentStartLine is the first line of the doc comment above the
	// declaration, 0 when there is none.
	CommentStartLine int
	// BodyStartLine is the line where a callable's body begins, 0 when it has
	// none.
	BodyStartLine int
}

CodeSpan is one cut of a source file, as a CodeSegmenter reports it: a declaration of a code file or a heading section of a Markdown file. Lines are 1-based and EndLine is exclusive. Sym is the dotted symbol path ("Server.Start") or, for a section, the heading path. Spans come in source order and are flat: nesting shows only in the ranges, a nested span lying inside its parent's.

The same shape as poly-lsp-mcp's segment.Span, so an adapter is a field copy.

type Confidence

type Confidence int

Confidence is how far a facet of a reading can be relied on, 0-100.

100 means "this is the thing itself, or a person has ruled on it". Everything below is a machine's output, and the number is a POLICY default for that machinery — not a measurement of this particular reading, which nothing can produce. Where a measurement exists it is cited on the constant.

const (
	// TrustExact — the bytes are the text. A plain file, a text layer lifted
	// from a PDF, a mail body decoded from its transfer encoding: no model is
	// involved and there is nothing to be wrong about beyond the file itself.
	TrustExact Confidence = 100
	// TrustRuled — a person went through it. The ceiling, and the only way a
	// machine-produced facet reaches it.
	TrustRuled Confidence = 100
	// TrustPandoc — a converter's reading of a structured document. Not exact:
	// a .docx has a layout that prose does not, and pandoc chooses.
	TrustPandoc Confidence = 95
	// TrustASRWords — a recogniser's words. High: modern ASR on clear audio is
	// mostly right, and it is wrong in ways a reader can see (a mangled name, a
	// dropped clause) rather than in ways that read as fluent.
	TrustASRWords Confidence = 85
	// TrustVisionText — a VLM's transcription of a page. Measured on this setup,
	// not guessed: chandra scored 89.6% on olmOCR-bench long_tiny_text against a
	// published 92.3, and 14/17 on the fence-dispute ground-truth checks.
	TrustVisionText Confidence = 90
	// TrustSubject — a model's account of what an image shows. The lowest, and
	// the one that most needs saying out loud, because it reads as fact: a plate
	// number, a make and model, a colour, none of it written by anybody.
	TrustSubject Confidence = 80
	// TrustDiarization — who spoke. Lower than the words on purpose. The
	// diarizer's own output says so, labelling turns "blended, attribution
	// uncertain", and a verified transcript exists precisely because this is the
	// facet people have to correct.
	TrustDiarization Confidence = 65
	// TrustLayout — reading order over a page a model assembled.
	TrustLayout Confidence = 85
)

Default confidences, by what produced the facet.

type Config

type Config struct {
	BaseURL     string `json:"base_url"`
	APIKey      string `json:"api_key"`
	VisionModel string `json:"vision_model"`
	EmbedModel  string `json:"embed_model"`
	// AudioModel names oidio's transcriber, which reads a recording so raglit can
	// index the transcript and review the reading. Empty → audio ingest is
	// unavailable and an audio job says so; a corpus of documents needs no
	// transcriber.
	//
	// Through corrallm — which is how oidio is actually deployed — the ids are
	// "oidio-stt-diarize" (speaker-labelled segments) and "oidio-stt" (plain
	// text), served from the SAME endpoint as the vision and embed models.
	// Standalone, oidio calls the same two "stt-diarize" and "stt".
	AudioModel string `json:"audio_model,omitempty"`
	// AudioBaseURL overrides where the transcriber lives. Empty → BaseURL, which
	// is right whenever oidio is behind the same broker as everything else.
	//
	// This started as a REQUIRED field of its own, on the reasoning that oidio is
	// a separate process on :8077 and assuming one gateway fronts both would make
	// a working setup unreachable. The :8077 in oidio's own example config is its
	// standalone-dev address; the deployment puts it under corrallm, whose
	// OpenAI proxy already re-exports it alongside the vision and embed models on
	// one base URL. So the separate endpoint is the exception, not the rule, and
	// the field earns its keep as an override rather than a second thing to set.
	AudioBaseURL string `json:"audio_base_url,omitempty"`
	// EmbedLimitChars caps a fragment's size to what the embed model accepts as one
	// input — probed once and stored (DiscoverEmbedLimit), so the deterministic
	// fragmenter's ceiling is bounded by the model, not by taste. 0 = not probed
	// (the fragmenter falls back to FragWindow uncapped).
	EmbedLimitChars int `json:"embed_limit_chars,omitempty"`
	// EmbedLimitTokens is the same limit in the unit the embedder counts in, and
	// it is the one that decides. EmbedLimitChars is a conversion of it at two
	// characters per token; a scanned court brief is 1.57, so a fragment can sit
	// inside the character ceiling and still be refused. 0 = not established.
	EmbedLimitTokens int `json:"embed_limit_tokens,omitempty"`
	// SegmentInputLimitTokens caps ONE segmentation REQUEST — prompt, carried-over
	// fragment and page text — to what the chat endpoint accepts, asked once and
	// stored (the server's own n_ctx where it states one, else llm.DiscoverContext).
	// The mirror of EmbedLimitChars on the other side of the call: that one bounds
	// what comes back, this one bounds what is sent.
	//
	// TOKENS, because that is the unit every endpoint states its limit in.
	// EmbedLimitChars stays in characters because it sizes a fragment, and the
	// conversion between the two is not a constant: measured with the model's own
	// tokenizer, this corpus runs 4.66 chars/token on prose and 1.16 on a survey
	// legal description.
	//
	// 0 = not established (no cap). Set it to override on a proxy that accepts a
	// prompt its backend will later refuse — which cannot be measured from
	// outside.
	SegmentInputLimitTokens int `json:"segment_input_limit_tokens,omitempty"`
	// FragWindow / FragStride / FragFloor tune the deterministic overlapping-window
	// text fragmenter (chars). 0 → defaults (1000 / 750 / 300; see fragment.go). window > stride
	// gives the overlap; floor folds a short tail. They feed frag_recipe, so a
	// change marks the affected documents for reprocessing.
	FragWindow int `json:"frag_window,omitempty"`
	FragStride int `json:"frag_stride,omitempty"`
	FragFloor  int `json:"frag_floor,omitempty"`
	// TranscriptionModel and DescribeModel are the per-kind ingest models (F11):
	// when ingest descends a page into zones, it reads each zone with the model
	// for its KIND — text zones go to the transcription model, diagram/figure
	// zones to the multimodal describer. A transcription model (chandra-ocr-2)
	// reads drawings poorly and a describer is wasted on paragraphs, so one
	// VisionModel covering both roles costs whichever side it was chosen for.
	//
	// Both empty → routing is a no-op: every zone is read by VisionModel exactly
	// as today, which is the live default (init does not ask for these). Set
	// only one and the other kind keeps reading VisionModel; set both and a page
	// can be split across two models. A zone whose kind is neither text nor
	// diagram/figure falls back to VisionModel either way.
	TranscriptionModel string `json:"transcription_model,omitempty"`
	DescribeModel      string `json:"describe_model,omitempty"`
	// IdentityModel names the chat model that says what a document IS — a
	// caption, a summary and a kind, asked once per document on the assembled
	// transcript (identity.go). Empty → the segment model, else the vision
	// model; set this to caption with a different text model than either.
	IdentityModel string `json:"identity_model,omitempty"`
	// IdentityProvider is IdentityModel's provider. Empty → the legacy
	// base_url/api_key, as before providers. Unused when IdentityModel is empty:
	// identity then runs on the segment model and its provider.
	IdentityProvider string `json:"identity_provider,omitempty"`
	// IdentitySlots is how many captioning requests are in the model at once.
	// 0 → DefaultIdentitySlots (2), which is what this endpoint serves
	// concurrently. Raising it past what the server actually runs does not make
	// anything faster: the extra requests wait INSIDE the server, where raglit
	// cannot see them, cannot resume them, and cannot tell them apart from an
	// ingest job's OCR call waiting for the same slot.
	IdentitySlots int `json:"identity_slots,omitempty"`
	// SessionBranchTTLHours is how long a session branch ("idx@session")
	// outlives its last write before the daemon deletes it. 0 → 168 (7 days:
	// USER 2026-10-30, "7 day to save thrash").
	SessionBranchTTLHours int `json:"session_branch_ttl_hours,omitempty"`

	// ModelChannelMax caps how wide any one model's admission channel may grow
	// (modelchan.go). 0 → the built-in ceiling.
	//
	// Every other number in that controller is LEARNED — a model starts at one
	// slot and widens only while calls succeed, halving whenever the server
	// pushes back. This is the exception, because the right ceiling depends on
	// what kind of endpoint is behind the name and no amount of evidence
	// distinguishes "has not said no yet" from "will take a hundred": a card
	// serving one model locally is never going past a handful, while a hosted
	// provider bought for throughput will, and there the ceiling would be the
	// only thing limiting it.
	ModelChannelMax int `json:"model_channel_max,omitempty"`
	// NoIdentity turns document identity off. Documents then carry only the
	// filename they arrived with — which for a scanner-named corpus is a list
	// nobody can navigate, so this is for a corpus whose names are already good
	// (a code tree) or an endpoint you do not want the extra call on.
	NoIdentity bool `json:"no_identity,omitempty"`
	// CodeSegmenter cuts source files at their declarations and Markdown at
	// its headings (KindCode), one fragment per declaration or section, each
	// with its lines and symbol path. raglitd's segmenter is poly-lsp-mcp's
	// tree-sitter segment package (cmd/raglit/codesegment.go). False → code
	// and Markdown are read into windows, as before. Turning it on changes the
	// KindCode pool key, so a changed file is cut again on its next ingest. An
	// unchanged file is not: the unchanged-bytes check skips it before the pool
	// key is looked at, so files indexed before the switch keep their windows
	// until `raglit ingest --fresh` re-reads them.
	// The daemon flag --code-segmenter turns it on as well.
	CodeSegmenter bool `json:"code_segmenter,omitempty"`
	// DefaultIndex is the index used when a command gives no --index. Empty →
	// "default". Set it in the wizard to make one named index your working default.
	DefaultIndex string `json:"default_index,omitempty"`
	// Project names this project. On the SHARED daemon it namespaces every index
	// this client touches (daemon index = "<project>__<local>"), so two projects
	// both using index "default" don't collide, and a project's "search all" stays
	// within its own indexes. Required to start a daemon-routed client (serve/CLI);
	// --embedded/--db (single-session, in-process) don't need it. Set in the wizard.
	Project string `json:"project,omitempty"`
	// Shared lists OTHER project namespaces this project also SEARCHES (reads only).
	// A "search all" spans "<project>__*" plus each "<shared>__*", so common docs
	// (e.g. ~/doc indexed once under a project named "shared") are visible from
	// every project that opts in — without duplicating them per project. Writes
	// (ingest/branch) still target this project only. Empty → fully isolated.
	Shared []string `json:"shared,omitempty"`
	// Watch, when true, tells the daemon to keep this project's configured source
	// roots fresh: after you register it (a `raglit sync` with watch:true, or
	// `raglit watch`), the daemon re-scans the roots on an interval and re-ingests
	// changed files / drops deleted ones. Needs `indexes` (the roots to watch).
	Watch bool `json:"watch,omitempty"`

	// WritebackTranscriptionMd materialises <doc>.raglit-transcription.md beside
	// every ingested document, project-wide. A per-index setting of the same name
	// overrides this for that index.
	WritebackTranscriptionMd bool `json:"writeback_transcription_md,omitempty"`
	// ExtractEmailAttachments writes a mail archive's attachments into
	// <archive>.raglit-attachments/ beside it, project-wide, so the documents
	// inside an .eml/.mbox become ordinary files the next sync indexes. A
	// per-index setting of the same name overrides this for that index.
	ExtractEmailAttachments bool `json:"extract_email_attachments,omitempty"`
	// SegmentModel splits transcribed TEXT into fragments. Empty → the vision
	// model, which is what this always used to be.
	//
	// They are different jobs and the best model for one is not the best for the
	// other. The segmenter is asked for a structured tool call over text; an OCR
	// specialist is chosen for reading pixels and need not do the first well.
	// Measured on a 2-page 1947 deed, same cached OCR, four arms:
	//
	//	chandra + markup     degraded — no valid JSON
	//	Qwen    + markup     done, dropped 1184 of 1234 chars
	//	chandra + flattened  degraded — no valid JSON
	//	Qwen    + flattened  done, clean
	//
	// Corpus-wide the same split: chandra degraded 29 of 116 documents (25%),
	// Qwen 4 of 254 (1.6%).
	SegmentModel string `json:"segment_model,omitempty"`
	// DaemonURL, when set, makes this a CLIENT config: commands route to the
	// raglit daemon at this URL (http(s)://host:port) instead of opening a local
	// index. The daemon owns storage (scoped per index, under its own home), so
	// the local .raglit/ then holds config only. Precedence for the effective
	// daemon: --daemon flag > $RAGLIT_DAEMON > this. Empty → local (embedded) mode.
	DaemonURL string `json:"daemon_url,omitempty"`
	// OCR configures the cheap tier — either as the cascade's first pass or as
	// the vision model's spelling reference (OCRConfig.Mode). Zero value →
	// VLM-only (every page transcribed by the vision model).
	OCR OCRConfig `json:"ocr,omitempty"`
	// ImageEmbed optionally configures an image embedder for FIGURES: when its
	// Model is set, a figure is embedded from its IMAGE instead of its description.
	// nomic-embed-vision-v1.5 shares nomic-embed-text's space, so image figures stay
	// searchable by the same text query — requires EmbedModel to be nomic-embed-text.
	ImageEmbed ImageEmbedConfig `json:"image_embed,omitempty"`

	// Ignore is this config's default exclude globs (project-scoped — it does not
	// affect other projects' configs). Merged with a built-in default (dot-dirs,
	// node_modules, vendor) and the per-index / per-root ignores; ignore wins.
	Ignore []string `json:"ignore,omitempty"`
	// Gitignore, when nil or true, also honors each root's .gitignore chain.
	Gitignore *bool `json:"gitignore,omitempty"`
	// Indexes declares named indexes and the source roots + rules that feed them,
	// for `raglit sync`. Multi-index: one project can define several.
	Indexes map[string]IndexConfig `json:"indexes,omitempty"`
	// Auth turns on OIDC for the daemon and tells a client how to get a token.
	// Nil → no auth, every index readable by anyone who can reach the daemon.
	Auth *AuthConfig `json:"auth,omitempty"`

	// Providers are named model endpoints, and each role picks one
	// (USER 2026-10-22: provider, then model — set from the UI). A role with
	// no provider uses BaseURL/APIKey above, so a config written before
	// providers existed behaves exactly as it did.
	Providers       map[string]Provider `json:"providers,omitempty"`
	VisionProvider  string              `json:"vision_provider,omitempty"`
	EmbedProvider   string              `json:"embed_provider,omitempty"`
	SegmentProvider string              `json:"segment_provider,omitempty"`
	AudioProvider   string              `json:"audio_provider,omitempty"`
}

Config is raglit's model-connection setup, written by `raglit init` into <home>/config.json. It's OpenAI-standard: a base URL + token, plus the two model ids raglit needs — a vision model (image in → text, for OCR) and an embedding model (text in → vector, for --embed / vector search). Kept out of the index so the same corpus can be re-pointed at a different endpoint.

func LoadConfig

func LoadConfig(h Home) (cfg Config, exists bool, err error)

LoadConfig reads the home's config. exists is false (with nil error) when the home has not been initialized yet — the caller decides whether that's fatal.

func (*Config) Endpoint

func (c *Config) Endpoint(provider string) (baseURL, apiKey string, ok bool)

Endpoint is the base URL and key of a named provider. "" or an unknown name → the legacy BaseURL/APIKey, with ok=false for an unknown name.

func (Config) PerKindModelsConfigured

func (c Config) PerKindModelsConfigured() bool

PerKindModelsConfigured reports whether either per-kind ingest model is set. While both are empty the F11 routing is a no-op: every zone is read by the one VisionModel, unchanged from today, and callers do not need to branch at all on zone kind.

func (*Config) SegmentProviderName

func (c *Config) SegmentProviderName() string

SegmentProviderName is the segment role's provider. A segment role with no model of its own runs on the vision model, and so on the vision provider.

func (Config) StrategyFor

func (c Config) StrategyFor(index string) StrategyConfig

StrategyFor resolves the read policy for one index: the index's own choice, else the project default, else the zero value (today's behavior).

An index naming a strategy that does not exist gets the zero value rather than an error, and that is deliberate — a typo'd strategy name must not stop an ingest, and the alternative (silently using the default strategy) would hide the typo behind plausible output. Callers that want to complain can check StrategyNamed.

func (Config) StrategyNamed

func (c Config) StrategyNamed(name string) (StrategyConfig, bool)

StrategyNamed looks a strategy up by name. ok=false for an empty or unknown name, in which case the zero StrategyConfig is returned.

type ContextProber

type ContextProber interface {
	// ContextWindow is the limit as the SERVER states it (llama.cpp /props),
	// ok=false when it does not state one.
	ContextWindow(ctx context.Context) (int, bool)
	// DiscoverContext measures it by overflowing, which costs O(log N) requests
	// and returns a lower bound. The fallback, not the first choice.
	DiscoverContext(ctx context.Context) (int, error)
}

ContextProber is the part of *llm.Client this needs. An interface so a test can answer without an endpoint, and so raglit does not take a dependency on the client type to ask two questions.

type Coverage

type Coverage struct {
	Parent    string  `json:"parent"`
	Pages     int     `json:"pages"`
	Covered   int     `json:"covered"`
	Uncovered []int   `json:"uncovered,omitempty"`
	Overlaps  [][]int `json:"overlaps,omitempty"` // [page, count] for pages in >1 slice
}

Coverage reports which of a bundle's pages belong to no slice, and where slices overlap.

This is what says a bundle is FULLY linearized rather than merely started. Without it "we sliced that scan" is unfalsifiable — pages 23-27 belonging to nothing looks identical to a finished job.

Overlap is reported, not refused: a declaration and the exhibit bound into it genuinely occupy the same pages, and a tool that forbade that would force the person to lie about one of them.

type DPIDecision

type DPIDecision struct {
	DPI       int
	Reason    string  // "need" | "cap" | "native" | "base"
	CapDPI    int     // what the token budget allowed
	NativeDPI int     // what the source holds (0 = unknown)
	NeedDPI   int     // what the glyph measurement asked for
	WantDPI   int     // the resolution asked for, BEFORE the cap
	Tiles     int     // canvases needed to deliver WantDPI
	Density   float64 // tokens/sq in delivered at DPI
}

DPIDecision is the chosen resolution and WHICH bound chose it. The reason is carried because the three failures look identical in a rendered page and are fixed differently: "need" means the text is fine, "cap" means tile it, and "native" means no rendering setting will ever help and the scan is the limit.

func ChooseDPI

func ChooseDPI(needDPI int, areaSqIn float64, nativeDPI, maxTokens int, p RenderPolicy) DPIDecision

ChooseDPI applies all three bounds.

nativeDPI of 0 means "unknown", and an unknown source resolution must NOT silently become a bound: a born-digital page has no native raster at all, and treating that as zero would refuse to render it.

func (DPIDecision) String

func (d DPIDecision) String() string

String renders a decision for a log line or a trace record.

type DiffShape

type DiffShape struct {
	Pages int `json:"pages"`
	// Clean is pages matching at 1.0, Noisy is pages disagreeing a little
	// (>= NoisyFloor), Divergent is pages disagreeing a lot.
	Clean     int `json:"clean"`
	Noisy     int `json:"noisy"`
	Divergent int `json:"divergent"`
	Missing   int `json:"missing"`
}

DiffShape describes the distribution of disagreement across pages.

func Shape

func Shape(pages []PagePair) DiffShape

Shape counts how the pages fall out. The counts are the finding: "12 clean, 1 divergent" is a refiling, "13 noisy" is a second scan, and both can average to the same number.

type DocContent

type DocContent struct {
	Path      string        `json:"path"`
	Title     string        `json:"title"`
	Pages     []DocPageText `json:"pages"`
	Text      string        `json:"text"`
	Truncated bool          `json:"truncated"`
	Figures   []FigureRef   `json:"figures,omitempty"`
}

DocContent is a document's indexed text: per-page plus a single joined blob, plus any figures explained into it.

type DocCost

type DocCost struct {
	Path         string
	Hash         string
	Passages     int
	Embeddings   int
	PageReads    int
	Bytes        int64   // passage text + embeddings + saved original and page images
	ModelSeconds float64 // import jobs that did model work (not pooled, not unchanged)
}

DocCost is what one document of one layer cost to hold.

type DocDiff

type DocDiff struct {
	// SameBytes short-circuits everything below it.
	SameBytes bool `json:"same_bytes"`
	// Match is the whole-document alignment: relation, coverage, blocks, gaps,
	// and the numeric tokens that differ. Reused rather than recomputed so a
	// diff and a `similar` report of the same pair can never disagree.
	Match DocMatch `json:"match"`
	// Pages is one row per page of A, in page order.
	Pages []PagePair `json:"pages"`
	// OnlyInB lists pages of B that no page of A matched — the other half of
	// "what is missing", which a table keyed on A cannot show.
	OnlyInB []int `json:"only_in_b,omitempty"`
	// Shape summarises where the disagreement lives, which is the actual input
	// to the copy-versus-version decision.
	Shape DiffShape `json:"shape"`
}

DocDiff is the full comparison of two transcripts.

type DocFields

type DocFields struct {
	Type   string          `json:"type"`
	Fields json.RawMessage `json:"fields"`
	// Source is 'machine' or 'person'. A person's extraction is never
	// regenerated over — the same rule a person's caption follows.
	Source   string `json:"source,omitempty"`
	Model    string `json:"model,omitempty"`
	At       int64  `json:"at,omitempty"`
	TextHash string `json:"text_hash,omitempty"`
	// TypeHash fingerprints the type DEFINITION these fields were read under
	// (DocType.Hash), so an extraction that answers a schema's older questions
	// can be told from one that answers its current ones. Empty means unknown,
	// which is treated as stale.
	TypeHash string `json:"type_hash,omitempty"`
}

DocFields is what a document's type schema says, filled out.

func (DocFields) ByPerson

func (f DocFields) ByPerson() bool

ByPerson reports whether a person wrote these fields.

func (DocFields) Empty

func (f DocFields) Empty() bool

Empty reports whether nothing has been extracted for this document.

type DocIdentity

type DocIdentity struct {
	Name    string `json:"name"`
	Summary string `json:"summary"`
	Kind    string `json:"kind"`
	// ContentTags are 3–5 short noun phrases describing what the document is
	// ABOUT (subject matter, entities, procedures). Open vocabulary but
	// constrained in shape; see validateIdentity.
	ContentTags []string `json:"content_tags,omitempty"`
	// RoleTags are 1–3 terms from identityRoleKinds describing what job the
	// document does in the corpus (documentation, reference, overview…).
	// Closed vocabulary so they stay groupable and filterable.
	RoleTags []string `json:"role_tags,omitempty"`
	// DocType is which of the INDEX's registered document types this resolved
	// as, empty for none. Its vocabulary is per-index and authored (doctype.go),
	// unlike Kind, which is raglit's and closed. A document that resolves as one
	// is asked to fill out that type's schema; most documents resolve as none,
	// and that is the normal case rather than a failure.
	DocType string `json:"doc_type,omitempty"`
	// Source is 'machine' (a model read the transcript) or 'person' (someone
	// corrected it). A person's identity is never overwritten by a re-run.
	Source string `json:"source,omitempty"`
	Model  string `json:"model,omitempty"`
	At     int64  `json:"at,omitempty"` // unix nanos
	// TextHash fingerprints the text this caption was written from, so a
	// transcript that changes underneath it can be detected. See
	// documents.gen_text_hash.
	TextHash string `json:"text_hash,omitempty"`
}

DocIdentity is what a document is, in one record: the caption, the summary, the kind, and who said so.

func (DocIdentity) ByPerson

func (d DocIdentity) ByPerson() bool

ByPerson reports whether a person, not a model, is the author of this caption.

func (DocIdentity) Empty

func (d DocIdentity) Empty() bool

Empty reports whether nothing has been established about this document.

type DocKind

type DocKind int

DocKind is how a source document should be extracted to text. raglit routes each ingested file to the cheapest extractor that fits its kind — the "cascade all the way down" idea: a PDF page uses its text layer if it has one, OCR if not; office/markup goes through pandoc; images go through OCR; text is read.

const (
	KindText   DocKind = iota // .txt/.md/… or text/* → read + segment
	KindPDF                   // .pdf → per-page hybrid (text layer or OCR)
	KindImage                 // .png/.jpg/… → OCR
	KindOffice                // .docx/.odt/.epub/.html/… → pandoc → text
	// KindEmail is an .eml archive: one PAGE per nested message, headers kept.
	// Not KindText — read as plain text an email archive is mostly base64, and
	// read as one page a quotation from the fifth message can only be located
	// somewhere in the whole file.
	KindEmail
	// KindSpreadsheet is .xlsx/.xls: one PAGE per sheet, same reasoning as
	// KindEmail — a fact on the "Q3 Budget" tab must be locatable to THAT tab,
	// not buried in a single page mixing every sheet in the workbook. Not
	// KindOffice: pandoc has no xlsx/xls reader at all (its own
	// --list-input-formats omits both — csv is the only spreadsheet format it
	// takes), and even if it did, OfficeText's one-flat-string-per-document
	// shape is the wrong output for a multi-sheet source.
	KindSpreadsheet
	// KindAudio is a recording — audio or a video container carrying speech.
	// Transcribed by oidio (see stt.go) and indexed as the TRANSCRIPT, with the
	// machine reading kept beside it for attest to review. raglit does not read
	// audio itself and is not meant to: it consumes a reading and verifies it.
	//
	// Video is here rather than in a kind of its own because what raglit wants
	// from a hearing recording is the speech, and oidio takes the container
	// whole. A kind that split them would have two routes to the same producer.
	KindAudio
	KindUnknown
	// KindCode is a text file the embedder's CodeSegmenter (Models.Code) takes:
	// source code by extension (CodeSegmenter.ForPath), and Markdown. It is cut
	// at declarations or heading sections, not into windows (codesegment.go).
	// Never returned by ClassifyDoc: only a worker with a segmenter routes to
	// it, so without one these files stay KindText or KindUnknown. After
	// KindUnknown so that no existing constant changes value.
	KindCode
)

func ClassifyDoc

func ClassifyDoc(name, contentType string) DocKind

ClassifyDoc routes a source by extension first, then content-type. Extension wins because it is the most reliable signal for a local file; content-type covers extensionless HTTP fetches.

func ClassifyPath

func ClassifyPath(path, contentType string) DocKind

ClassifyPath is ClassifyDoc for a file that exists on disk: the extension decides, and only when it decides nothing does the content get a say.

The order matters and is not the obvious one. Sniffing FIRST would be more "correct" in the abstract and worse in practice — a .docx and a .odt are both zips, a .md and a .csv are both text, and the extension separates them where the bytes do not. The bytes are the fallback, not the authority.

func SniffBytes

func SniffBytes(head []byte) DocKind

SniffBytes is SniffKind for content already in hand — the ingest path holds the bytes and never the file, so a path-only sniff would not reach it.

func SniffKind

func SniffKind(path string) DocKind

SniffKind reads a file's leading bytes and says what it is, ignoring its name.

ClassifyDoc routes on the extension because for a local file the extension is the most reliable signal available — but it is only a signal, and a file that has NO extension gets KindUnknown, which the worker reads as plain text.

For a PDF that is not a near-miss, it is garbage: the index fills with `%PDF-1.7`, `obj` and `endobj` while the document's actual text stays unsearchable, and nothing reports an error because reading bytes as text always "works". Two attachments in ardley-v-brannock arrived exactly this way, their names truncated at 102 characters by the tool that unpacked them, losing the `.pdf` — the same class of defect the email extractor's own fix names ("an unnamed attachment kept no extension, so nothing indexed it").

Returns KindUnknown when nothing matches, so the caller keeps its existing behaviour and this can only ever add routing, never redirect a file that the extension already classified.

func (DocKind) String

func (k DocKind) String() string

String names a kind stably, for the pool key.

These strings are part of a CACHE KEY, so they are written out by hand rather than derived from the constant's position: renumbering the iota, or inserting a kind in the middle, must not silently re-key every cached document. Adding a new name here is free; changing an existing one invalidates that kind's cache, which is the correct behaviour if the routing itself changed and a mistake otherwise.

type DocMatch

type DocMatch struct {
	Path  string `json:"path"`
	Title string `json:"title,omitempty"`

	Relation Relation `json:"relation"`

	// Jaccard is |A∩B|/|A∪B| over distinct shingles: "are these the same
	// document". Exact, not estimated.
	Jaccard float64 `json:"jaccard"`
	// ContainProbe is |A∩B|/|A| — how much of the PROBE occurs in this document.
	// ContainMatch is the reverse. Asymmetric on purpose; a single number here
	// cannot express "wholly inside".
	ContainProbe float64 `json:"contain_probe"`
	ContainMatch float64 `json:"contain_match"`

	// BlockCoverProbe / BlockCoverMatch are the fraction of each side covered by
	// runs of at least MinRunChars. These are the numbers Relation is decided on,
	// because they are the ones shared boilerplate cannot inflate.
	BlockCoverProbe float64 `json:"block_cover_probe"`
	BlockCoverMatch float64 `json:"block_cover_match"`

	SharedShingles int `json:"shared_shingles"`
	ProbeShingles  int `json:"probe_shingles"`
	MatchShingles  int `json:"match_shingles"`

	// GenericChars is how much of the PROBE was excluded as corpus-generic (an
	// email footer, a caption repeated across unrelated filings). Reported rather
	// than applied silently: masking is the one step that can hide a real
	// duplicate, so when a document is mostly generic the person reading the
	// result has to be able to see that is why the score is low.
	GenericChars int `json:"generic_chars,omitempty"`

	// MatchedChars is the total folded characters proved identical, counting
	// overlaps once. Coverage alone cannot be acted on for a short document: 0.83
	// of a 400-character email is a signature block, and 0.83 of a deed is a deed.
	MatchedChars int `json:"matched_chars"`

	Blocks []Block `json:"blocks,omitempty"`

	// NumericOnlyInProbe / NumericOnlyInMatch are numeric tokens (auditor file
	// numbers, distances, bearings, dollar amounts, dates) that appear inside the
	// aligned region on ONE side only.
	//
	// This is the headline finding for a legal corpus and it is cheap. Two copies
	// of one deed that align at 0.97 and disagree about "25.00" versus "30.00" are
	// not a housekeeping problem; in this matter there is an open question about
	// which of two 25-foot strips a plat depicts, and one exhibit's OCR was found
	// fabricated outright. A digits-level diff over the aligned span puts that in
	// front of a person instead of averaging it into a score.
	NumericOnlyInProbe []string `json:"numeric_only_in_probe,omitempty"`
	NumericOnlyInMatch []string `json:"numeric_only_in_match,omitempty"`
}

DocMatch is one document that shares text with the probe.

func Compare

func Compare(probe, match FoldedText, w int, probeMask, matchMask Mask) DocMatch

Compare is the exact pairwise comparison: no sampling, no estimate.

Candidate generation samples (see shingle.go), because probing an index must not read every document. Scoring does not, because these numbers end up in a human decision about which copy of an instrument a filing cites, and an estimate that is usually right is the wrong tool for that. At this corpus size exactness is nearly free: the whole 233-document index folds to 1.59 MB, so a pair costs two shingle passes over a few tens of kilobytes. The full all-pairs audit of the corpus — 233 probes, each rescoring its candidates exactly — runs in 27 seconds.

One asymmetry to know about, because it looks like a bug and is not: MatchedChars, BlockCoverProbe and the reported blocks are all measured on the PROBE side, so Compare(a,b) and Compare(b,a) do not report the same matched length. A passage of the smaller document that recurs at nine places in the larger one is 383 characters measured on the small side and 3,612 on the large one; both are true statements about different questions. Jaccard is symmetric, the containments are deliberately not, and the blocks answer "where in the PROBE", which is what a triage caller asked. probeMask and matchMask may be nil, which is how the pure two-document tests exercise the scoring without a corpus to derive frequencies from.

type DocPageText

type DocPageText struct {
	Page int    `json:"page"`
	Text string `json:"text"`
}

DocPageText is one page's reassembled text.

type DocRef

type DocRef struct {
	Path  string `json:"path"`
	Title string `json:"title"`
}

DocRef identifies one indexed document (its stable path key + title).

type DocSummary

type DocSummary struct {
	Path      string         `json:"path"`
	Title     string         `json:"title"`
	Fragments int            `json:"fragments"`
	Pages     int            `json:"pages"`     // OCR-tracked pages (page ≥ 1)
	Vision    int            `json:"vision"`    // pages that used the VLM
	FragMode  string         `json:"frag_mode"` // how it was fragmented: text-overlap | llm-seg | code | sections
	Engines   map[string]int `json:"engines"`   // engine → page count
	// AddedAt is when the document was last really ingested (Unix nanoseconds).
	// A re-ingest of an unchanged file is skipped before commit (worker.go), so
	// it does not move AddedAt.
	AddedAt int64 `json:"added_at"`
	// SrcSize / SrcMtime are the source file's bytes and modified time (unix
	// seconds); 0 when unknown (srcstat.go).
	SrcSize  int64 `json:"src_size,omitempty"`
	SrcMtime int64 `json:"src_mtime,omitempty"`
	// ContentHash is documents.content_hash: the sha256 hex of the raw source
	// bytes, not of the extracted text, and the same value as
	// Store.DocumentHash; "" for a document never hashed.
	ContentHash string `json:"content_hash,omitempty"`
	// GenName / GenKind / GenSource are what the document IS (identity.go), for a
	// list a person can navigate. Both names travel: the caption is what a reader
	// needs, and the filename is what everything else joins on — and where they
	// disagree, THAT is the finding.
	GenName   string `json:"gen_name,omitempty"`
	GenKind   string `json:"gen_kind,omitempty"`
	GenSource string `json:"gen_source,omitempty"` // machine | person
	// GenSummary is the generated account of what the document covers, and
	// GenModel the model that wrote the caption (empty for a person's). Carried
	// so a list can show what a document is without a request per row.
	GenSummary string `json:"gen_summary,omitempty"`
	GenModel   string `json:"gen_model,omitempty"`
	// GenContentTags / GenRoleTags are the document's tags (identity.go):
	// what it is about, and what job it does in the corpus.
	GenContentTags []string `json:"gen_content_tags,omitempty"`
	GenRoleTags    []string `json:"gen_role_tags,omitempty"`
}

DocSummary is a document with counts for the review UI's document list.

type DocType

type DocType struct {
	Name string `json:"name"`
	// Description is one line saying how to RECOGNISE one — it is what the
	// identity call is shown when choosing among the registered types, so it
	// describes the document, not the fields.
	Description string `json:"description,omitempty"`
	// Prompt is the extraction instruction: how to read these fields out of this
	// kind of document, including what its conventions mean. Travels with the
	// schema because a schema alone produces a confidently filled-in form.
	Prompt string `json:"prompt,omitempty"`
	// Schema is a JSON Schema object describing the fields — the `parameters` of
	// the extraction tool call.
	Schema json.RawMessage `json:"schema,omitempty"`
	// Gold is the documents the schema was proposed from, kept so a revision can
	// be judged against the same examples rather than whatever is at hand.
	Gold      []string `json:"gold,omitempty"`
	Model     string   `json:"model,omitempty"`
	CreatedAt int64    `json:"created_at,omitempty"`
	UpdatedAt int64    `json:"updated_at,omitempty"`
}

DocType is one registered document type of an index.

func (DocType) FieldNames

func (t DocType) FieldNames() []string

FieldNames lists the schema's top-level properties, in schema order where the JSON preserves it and alphabetically otherwise. For a summary line.

func (DocType) Hash

func (t DocType) Hash() string

Hash fingerprints the parts of a type definition that shape an EXTRACTION: the schema, the reading instructions, and the description (which the extraction prompt carries as "this document is a X — <description>").

The name is deliberately NOT in it. A rename is a rename; it does not change what was asked of the document, and re-extracting a corpus because somebody fixed a capital letter is a bill for nothing. The gold paths are out for the same reason — they say where the schema came from, not what it asks.

The schema is hashed through a decode/re-encode round trip, so a reformatted-but-identical schema hashes the same: Go marshals map keys in sorted order, which makes that canonical. A schema that will not decode is hashed as its raw bytes, which is the conservative answer — it will simply read as changed more often than it is.

type Document

type Document struct {
	Path      string
	Title     string
	Fragments []Fragment
}

Document is a source doc plus its fragments. Path is the unique key.

type EmailAttachmentRef

type EmailAttachmentRef struct {
	Name string `json:"name"`
	Mime string `json:"mime,omitempty"`
	Size int64  `json:"size,omitempty"`
	Sum  string `json:"sum,omitempty"`
	// Path is the extracted file beside the archive, when extraction ran and the
	// bytes are on disk. Empty means the attachment was seen and not extracted —
	// which is a different statement from "there is no attachment", and the UI
	// must not render them the same way.
	Path string `json:"path,omitempty"`
}

EmailAttachmentRef is an attachment as a reader can act on it.

type EmailHeader

type EmailHeader struct {
	Name  string `json:"name"`
	Value string `json:"value"`
}

EmailHeader is one header line.

type EmailMessage

type EmailMessage struct {
	// Page is the page number this message occupies in the indexed document, so
	// a citation from search lands on the same message a reader is looking at.
	Page int `json:"page"`
	// Depth is enclosure: 0 is a top-level message in the archive, 1 is a message
	// forwarded inside one, and so on. This is the thread structure.
	Depth   int    `json:"depth"`
	From    string `json:"from,omitempty"`
	To      string `json:"to,omitempty"`
	Cc      string `json:"cc,omitempty"`
	Date    string `json:"date,omitempty"`
	Subject string `json:"subject,omitempty"`
	// Headers is EVERY header, name and value, sorted. The page text keeps four
	// and counts the rest; this is the rest.
	Headers []EmailHeader `json:"headers,omitempty"`
	Body    string        `json:"body,omitempty"`
	// Attachments are what this message carried, whether or not extraction ran.
	// "A 4 MB PDF called survey.pdf was sent on this date" is itself evidence.
	Attachments []EmailAttachmentRef `json:"attachments,omitempty"`
	// Notes is structure the reader could not parse, carried rather than dropped.
	Notes []string `json:"notes,omitempty"`
}

EmailMessage is one message in an archive, structured for display.

func EmailMessages

func EmailMessages(path string, resolve func(sum string) string) ([]EmailMessage, error)

EmailMessages reads an archive into structured messages. Pure: it reads the file and writes nothing, for the reason EmailText gives.

resolve, when non-nil, maps an attachment's sha256 to the path it was extracted to. Passing nil returns attachments with no Path, which is honest for an archive whose attachments were never extracted.

type Embedder

type Embedder struct {
	Client      VectorClient
	Model       string
	DocPrefix   string
	QueryPrefix string
	// BatchLimitChars caps ONE embed request's total input. 0 → a conservative
	// default. The server's limit applies to the whole request, so this bounds
	// the batch, not the individual fragment.
	BatchLimitChars int
}

Embedder turns text into normalized vectors. nomic-embed-text is ASYMMETRIC: documents and queries must carry different task prefixes or retrieval quality drops, so DocPrefix / QueryPrefix default to the nomic convention. Override them (to "") for a model that doesn't use prefixes.

func NewEmbedder

func NewEmbedder(c VectorClient, model string) *Embedder

NewEmbedder builds an Embedder with the nomic prefixes.

func (*Embedder) DiscoverEmbedLimit

func (e *Embedder) DiscoverEmbedLimit(ctx context.Context) (int, error)

DiscoverEmbedLimit probes the largest single input (in characters) the embed endpoint accepts without error — the same shape as llm.DiscoverContext. It grows exponentially to find a length that FAILS, then binary-searches the boundary. Only works on endpoints that REJECT an over-long input; a tolerant endpoint that truncates returns maxEmbedProbeChars (treated as "effectively unbounded"). Store the result in Config.EmbedLimitChars to cap the fragment ceiling by the model, not by taste.

Only a rejection that says the input is too long moves the boundary. Any other failure stops the probe with that error. Counting every error as "too long" made a busy endpoint teach a limit far below the real one: corrallm answers 429 when a model's slots are full, the client waits out its retry budget, and the probe then read the give-up as a refusal of that length.

func (*Embedder) EmbedDocs

func (e *Embedder) EmbedDocs(ctx context.Context, texts []string) ([][]float32, error)

EmbedDocs embeds document fragments (DocPrefix), normalized.

Splits into as many requests as the endpoint's batch budget needs. Chunking lives HERE rather than at the call sites so every caller is covered by construction — there are three, and the one that overflowed was the only one anybody would have thought to fix.

func (*Embedder) EmbedQuery

func (e *Embedder) EmbedQuery(ctx context.Context, text string) ([]float32, error)

EmbedQuery embeds a search query (QueryPrefix), normalized.

type EnqueueItem added in v0.8.0

type EnqueueItem struct {
	URL, Title string
	Fresh      bool
}

EnqueueItem is one entry of an EnqueueMany batch.

type EnqueueResult added in v0.8.0

type EnqueueResult struct {
	ID  int64
	Err error
}

EnqueueResult is the outcome for one EnqueueItem: the job id (new, or the existing job that already covers it), or the per-item refusal.

type ErrIdentityTooShort

type ErrIdentityTooShort struct{ Chars int }

ErrIdentityTooShort reports a document with too little text to identify.

func (*ErrIdentityTooShort) Error

func (e *ErrIdentityTooShort) Error() string

type ErrSegmentShort

type ErrSegmentShort struct {
	Page    int
	In, Out int
}

ErrSegmentShort reports that the model-segmenter did not account for a page.

func (*ErrSegmentShort) Detail

func (e *ErrSegmentShort) Detail() string

Detail is the short form for a stage log.

func (*ErrSegmentShort) Error

func (e *ErrSegmentShort) Error() string

type Fetched

type Fetched struct {
	Data        []byte
	Title       string // basename, for the document title
	IsPDF       bool   // route to OCR vs plain-text fragmenting
	ContentType string // HTTP Content-Type, for format routing (empty for local files)
}

Fetched is the raw content behind an ingest URL, plus enough to route it.

func Fetch

func Fetch(ctx context.Context, rawURL string) (Fetched, error)

Fetch resolves an ingest URL to bytes. Supported schemes:

  • file://<path> a local file
  • http(s)://... a remote GET
  • (no scheme) treated as a local filesystem path

PDF is detected by a .pdf extension or an application/pdf content type.

type FieldsCoverage

type FieldsCoverage struct {
	Type      string `json:"type"`
	Resolved  int    `json:"resolved"`
	Extracted int    `json:"extracted"`
	// Stale is extractions that answer an older schema or an older transcript.
	// Counted separately from Extracted because "88 of 88 extracted" over a
	// schema edited yesterday is a coverage report that lies.
	Stale int `json:"stale,omitempty"`
}

FieldsCoverage counts, per registered type, how many documents resolved as it and how many have been extracted. What `raglit fields --list` reports.

type FieldsStale

type FieldsStale string

FieldsStale says why an extraction is not current, or that it is.

const (
	FieldsCurrent     FieldsStale = ""             // answers the type's current questions, from the document's current text
	FieldsNone        FieldsStale = "none"         // never extracted
	FieldsSchemaMoved FieldsStale = "schema"       // the type definition changed under it
	FieldsTextMoved   FieldsStale = "text"         // the document's transcript changed under it
	FieldsTypeGone    FieldsStale = "type removed" // the type it was read under is no longer registered
	FieldsTypeChanged FieldsStale = "type differs" // the document now resolves as a DIFFERENT type
)

func (FieldsStale) Reason

func (f FieldsStale) Reason() string

Reason is a phrase for a report.

func (FieldsStale) Stale

func (f FieldsStale) Stale() bool

Stale reports whether this is anything other than current.

type FigureHit

type FigureHit struct {
	MediaID     int64   `json:"media_id"`
	Path        string  `json:"path"`
	Title       string  `json:"title"`
	Page        int     `json:"page"`
	Description string  `json:"description"`
	ImagePath   string  `json:"image_path"`
	FragmentID  int64   `json:"fragment_id"`
	Score       float64 `json:"score"`
}

FigureHit is one figure matched by SearchFigures: the figure plus where it lives, ranked by similarity to the query.

type FigureRef

type FigureRef struct {
	Page        int    `json:"page"`
	Kind        string `json:"kind"`
	Description string `json:"description"`
	ImagePath   string `json:"image_path,omitempty"`
	Bbox        string `json:"bbox,omitempty"`
}

FigureRef is one figure attached to a document (or a search hit): its description and image, located by page.

type Finder

type Finder struct {
	Store *Store
	// Limit caps fragments pulled from the index per query before collapsing to
	// documents. Default 20.
	Limit int
}

Finder adapts a raglit Store to agent.DocFinder, so a local index drops straight into agentkit's proactive-retrieval seam (agent.FinderPreparer) and its MCP tool bridge — the SAME interface a remote ragtag client satisfies. That is the scale path: start with a local .sqlite Finder, swap to a remote impl later without touching the agent wiring.

A DocFinder hit is per-DOCUMENT (the notify layer dedups by DocID and injects a pointer), but Search returns per-FRAGMENT rows. Finder collapses fragments to the best-scoring one per document, so a doc pings once with its strongest passage as the pointer line.

func NewFinder

func NewFinder(s *Store) *Finder

NewFinder wraps a Store as an agent.DocFinder.

func (*Finder) Find

func (f *Finder) Find(ctx context.Context, texts []string) ([]agent.DocHit, error)

Find implements agent.DocFinder: it searches the joined conversation text and returns one DocHit per matched document (best fragment wins). DocID is the document path — stable, and what a fetch tool would use to pull the full doc.

type FoldRecord

type FoldRecord struct {
	Hash      string `json:"hash"`
	Layer     string `json:"layer"`
	Source    string `json:"source"`    // the layer document's path
	BasePath  string `json:"base_path"` // the base document's path (an upload URL)
	Title     string `json:"title,omitempty"`
	By        string `json:"by"`
	FoldedAt  string `json:"folded_at"`
	Settled   bool   `json:"settled"` // provenance note and source notes copied
	SettledAt string `json:"settled_at,omitempty"`
}

FoldRecord is one content folded into the base, kept in <base home>/folds.json.

type FoldReport

type FoldReport struct {
	ProjectID     string       `json:"project_id"`
	Base          string       `json:"base"`
	DryRun        bool         `json:"dry_run"`
	LayersScanned []string     `json:"layers_scanned"` // public layers read
	Folded        []FoldRecord `json:"folded"`
	AlreadyInBase int          `json:"already_in_base"` // layer documents whose content the base holds
	Settled       int          `json:"settled"`         // earlier folds whose notes were copied this run
	Pending       int          `json:"pending"`         // earlier folds still importing
	Skipped       []string     `json:"skipped,omitempty"`
}

FoldReport is what one run did (or, dry, would do).

type FoldedPage

type FoldedPage struct {
	Page  int
	Start int
	End   int
}

FoldedPage locates one page inside a FoldedText as a half-open [Start,End) span of folded offsets. A page whose text folds to nothing (a blank scan, or a page that is pure figure) is still listed, with Start == End: page numbering has to stay complete or every alignment after it reports the wrong page.

type FoldedText

type FoldedText struct {
	// Raw is the text the fold was taken of — pages joined by PageJoin. Offs
	// indexes into this.
	Raw string
	// Body is the fold: lowercase letters and digits only, no separators.
	Body string
	// Offs[i] is the byte offset in Raw of the i'th byte of Body.
	Offs []int
	// Pages are page boundaries in FOLDED (Body) offsets, in page order.
	Pages []FoldedPage
}

FoldedText is text reduced to its comparable form, with the index needed to map a finding back to something a person can read.

func FoldPages

func FoldPages(pages []PageText) FoldedText

FoldPages reduces paged text to its comparable form.

Kept byte-for-byte compatible with kgraph's foldTextIdx: lowercase, keep unicode letters and digits, drop everything else, and record the source offset of every byte written. Digits are kept STRICTLY — no date or number canonicalisation of any kind. In this corpus the highest-value tokens on a page are exactly the ones a normaliser would eat: an auditor file number (AF#201503110045), a bearing (N 89°14'32" W), a distance (25.00 feet). Two copies of one deed that disagree about a distance are a serious finding, and any fold that rounds, reformats or strips digits reports them as agreeing.

The separators between those digits DO fold away — "AF#201503110045", "AF 201503110045" and "AF# 201503110045" all become "af201503110045" — which is the same reason kgraph's normalize() drops punctuation from an identifier.

func (FoldedText) PageAt

func (f FoldedText) PageAt(off int) int

PageAt reports which page a folded offset lies on, and 0 when it lies past the end. Binary search would be faster; page counts here are in the tens.

type FragConfig

type FragConfig struct {
	Window, Stride, Floor int
	EmbedLimit            int
	FigurePrompt          int
	// SegmentInputLimit caps ONE segmentation request — prompt, carried-over
	// fragment and page text together — at what the chat endpoint accepts, IN
	// TOKENS, which is the unit the endpoint counts in. Distinct from EmbedLimit,
	// which bounds the fragments that come BACK, in characters, because that
	// limit belongs to the embedder. Measuring only the second is how a page too
	// big to send reached the endpoint and failed the document. 0 → unknown, no
	// cap.
	SegmentInputLimit int
	// TokenCounter counts tokens as the serving model does, when the endpoint can
	// (llm.Client.CountTokens). nil → sizes are estimated from a ratio learned
	// per document instead. The difference is not cosmetic: measured with the
	// model's own tokenizer, this corpus runs 4.66 characters per token on prose
	// and 1.16 on a survey legal description.
	TokenCounter TokenCounter
	// EmbedLimitTokens and EmbedTokenCounter are the same pair for the EMBEDDER,
	// and they are a separate model with a separate tokenizer — nomic-embed-text
	// answers in BERT ids where the chat model answers in its own.
	//
	// This is the limit that actually refused documents. EmbedLimit above is a
	// character number converted from the embedder's 8192-token context at an
	// assumed two characters per token, i.e. 16128 — and a scanned court brief
	// reaches 10240 tokens inside that, so the fragment was allowed, sent, and
	// refused with a 500 that failed the whole document at the embed stage.
	// Characters cannot express this limit; only tokens can.
	EmbedLimitTokens  int
	EmbedTokenCounter TokenCounter
}

FragConfig tunes the deterministic text fragmenter and identifies the per- document fragmentation recipe. Window/Stride/Floor are 0 → defaults; EmbedLimit caps the window by the embed model's input limit (0 → uncapped); FigurePrompt is the figure-description prompt version, folded into frag_recipe so a prompt change marks affected documents stale.

type Fragment

type Fragment struct {
	Page int    // 1-based page number (0 for pageless sources like plain text)
	Ord  int    // fragment order within the page
	Text string // the searchable text
	// StartLine and EndLine are the half-open [StartLine, EndLine) 1-based line
	// span of Text in the source, 0/0 for unknown. Stored as given.
	StartLine, EndLine int
	// Symbol is the dotted symbol path the fragment is ("Server.Start"), empty
	// for none. It is part of the fragment's content id.
	Symbol string
}

Fragment is one indexable unit at the document:page:fragment grain.

type GCPolicy

type GCPolicy struct {
	MaxBytes     int64         // trim total payload bytes below this (LRU)
	MaxEntries   int           // trim entry count below this (LRU)
	MaxAgeUnused time.Duration // also evict anything unused at least this long
}

GCPolicy bounds the pool. All limits are optional (zero = disabled). The default posture is deliberately LAX — a byte budget only — so merges and retries keep reusing pooled work instead of paying to re-index it; the pool is trimmed to MaxBytes by evicting the OLDEST-accessed (LRU) entries, which are the ones least likely to be needed again.

type Gap

type Gap struct {
	ProbePage int    `json:"probe_page"`
	MatchPage int    `json:"match_page"`
	ProbeText string `json:"probe_text"`
	MatchText string `json:"match_text"`
}

Gap is one disagreement inside an aligned block: the unmatched text between two runs, from both sides, as a person would read it.

type GibberishConfig

type GibberishConfig struct {
	// MinMeanConfidence: pages whose mean recognizer confidence is
	// below this fall back. PP-OCR reports >0.9 mean on clean print;
	// 0.55 only catches genuinely struggling pages.
	MinMeanConfidence float64 `json:"min_mean_confidence"`
	// MinWordlikeFraction: fraction of whitespace tokens that must
	// look like words (letters with vowels, or numbers). Below this,
	// the text is mostly symbol-soup → fall back. Only applied once
	// the page has at least MinTokensForLexical tokens (short pages
	// have too few tokens for a stable ratio).
	MinWordlikeFraction float64 `json:"min_wordlike_fraction"`
	// MinTokensForLexical gates the lexical test: pages with fewer
	// tokens than this are judged on confidence alone.
	MinTokensForLexical int `json:"min_tokens_for_lexical"`
	// MaxJunkRuneFraction: fraction of replacement (U+FFFD) /
	// control runes above which the page is junk regardless of
	// confidence. Catches mojibake the recognizer was "confident"
	// about.
	MaxJunkRuneFraction float64 `json:"max_junk_rune_fraction"`
}

GibberishConfig tunes the detector that decides whether a cheap PaddleOCR page is trustworthy or needs a vision-model re-OCR. Defaults are PRECISION-BIASED: thresholds sit well below clean- print norms so only clearly-bad pages (handwriting, degraded scans, garbled figures) trip the fallback. The goal is to keep the expensive vision calls rare; the cost of a false-negative is a slightly noisy page, the cost of a false-positive is a wasted Qwen call.

Zero value → all defaults via withDefaults(). Overrides arrive through the OCR config (config.OCR.Gibberish) and are carried on OCR.Gate; see the cascade in ocr.go.

func (GibberishConfig) IsGibberish

func (g GibberishConfig) IsGibberish(po PageOCR) (bool, string)

IsGibberish reports whether a PaddleOCR page should be re-OCR'd by the vision model, with a short human-readable reason for tracing.

A zero-box page is NOT gibberish — it's an empty/blank page (or a pure figure with no detectable text), which the cascade emits as empty rather than paying the vision model for every blank. A figure whose labels paddle garbles will instead trip on low confidence or the lexical test, which is the case the user cares about ("graphs need Qwen").

type Hit

type Hit struct {
	ID    int64 // fragment id (stable key for fusing rankings)
	Path  string
	Title string
	Page  int
	Ord   int
	Text  string
	Score float64
	// StartLine and EndLine are the fragment's half-open 1-based line span in
	// its source, 0/0 when unknown (llm-seg, a paged document, an older index).
	StartLine, EndLine int
	// Symbol is the fragment's dotted symbol path, empty for none.
	Symbol string
	// ContentID is the fragment's stable id (contentid.go): the same across a
	// re-ingest of unchanged text, unlike ID. See Store.FragmentExists.
	ContentID string
	// Origin is empty for the document's own words, and names what a machine
	// made otherwise: "identity" for a generated caption/summary (identity.go),
	// "described" for a model's account of an IMAGE (indextext.go) — the 700
	// words chandra writes about a photograph, naming a car's make and its
	// licence plate, none of which anybody wrote down.
	//
	// These rank in the same list on purpose: a summary is how a document whose
	// body never says "purchase and sale agreement" becomes findable by that
	// query, and a description is the only way a photograph is findable at all.
	// But they are a machine's words, so every renderer says so and nothing
	// quotes from them.
	Origin string
	// Trust is how far the text of this hit can be relied on, and about WHAT.
	// Nil when nothing is recorded about how the document was read, which is not
	// a claim of reliability — see HitTrust.
	Trust *HitTrust
	// Similarity is the cosine of the query and fragment vectors, set on a hit
	// a vector search ranked. Nil when the vector side did not rank it (bm25,
	// or a hybrid hit outside the vector pool). Score in hybrid is a fused
	// rank, so a fixed cut-off reads this instead.
	Similarity *float64
	// Matched is the distinct query terms the fragment contains, in query
	// order, and QueryTerms how many terms the query had after stopwords. A
	// long query matched on one word is a weak hit whatever its bm25.
	Matched    []string
	QueryTerms int
	// contains filtered or unexported fields
}

Hit is one BM25-ranked fragment. Score is normalized so HIGHER is better (the opposite of SQLite's raw bm25(), which returns more-negative for better matches) — matching agentkit's DocHit.Score convention.

func CollapseToAuthoritative

func CollapseToAuthoritative(hits []Hit, readingOf func(doc string) (Reading, bool),
	readingsOf func(source string) []Reading, governs func(source string) (string, bool)) []Hit

CollapseToAuthoritative drops hits that are readings of a source some OTHER hit reads better, keeping rank order.

Search ranks fragments, and two readings of one recording are two sets of fragments saying nearly the same thing — so a 44-minute hearing came back twice, once as a verified transcript with people's names and once as machine output with speaker-1 and speaker-6. Both are real; only one should be the answer, and it must not be decided by which happened to score higher.

Documents with no reading recorded are untouched. That is most of a corpus, and a hit is never removed for want of a row nothing wrote.

func (Hit) Caveat

func (h Hit) Caveat() string

Caveat is the caution for THIS hit: the fragment first, then the document.

Two levels, because a reader reads a FRAGMENT and the document-level number can be honestly small while the excerpt in front of them is entirely a model's. Measured on this corpus: a 2002 record of survey scores 3% described — correctly, it is a transcribed sheet with one figure on it — and the top excerpt a survey query returns from it is `[FIGURE: Section map showing grid lines and section number 6]`, which is nobody's words. A document-level caveat alone stays silent there.

func (Hit) IsDescription

func (h Hit) IsDescription() bool

IsDescription reports whether this hit is a DESCRIPTION of a document — its caption and summary — rather than text from the document itself. True for a machine's and for a person's alike: neither is the instrument, and neither can be quoted as if it were.

type HitTrust

type HitTrust struct {
	// Method is what produced the text: text-layer, vision-ocr, oidio-asr…
	Method string `json:"method,omitempty"`
	// Level is machine, adapted or attested — whether a person has been through it.
	Level string `json:"level,omitempty"`
	// DescribedPct is how much of the DOCUMENT is a model's description rather
	// than a transcription, or DescribedUnmeasured when it cannot be known.
	DescribedPct int `json:"described_pct,omitempty"`
	// Facets is the per-facet confidence: text, speaker, layout, subject. A facet
	// that is absent is NOT claimed rather than distrusted.
	Facets map[string]Confidence `json:"facets,omitempty"`
	// Summary renders Facets weakest-first for a reader ("subject 80%", "text 90%").
	Summary []string `json:"summary,omitempty"`
	// RuledBy is the person who ruled, when one has.
	RuledBy string `json:"ruled_by,omitempty"`
}

HitTrust is the caveat a search result carries: what produced this text, and what a person has ruled on.

A ranked list presents every row in the same shape, and the shape reads as "the document says this". For most rows that is right. For others the sentence was written by a model looking at a picture, and the two are indistinguishable once they are excerpts in a panel.

Origin already marks the clear-cut case, but only when EVERY page a fragment touches is ≥90% description — so a survey sheet measured at 88%, whose entire indexed text is a model's account of a map down to which annotation arrow is which colour, comes back with an empty Origin and reads as the record. That is the case this exists for, and it is one bind away from deriving backing in caselit.

Reported, never enforced. Nothing here decides that a hit may not be used — that is a person's ruling, and presenting a confidence as a verdict would be the same mistake as presenting rank as authority.

func (*HitTrust) Caveat

func (t *HitTrust) Caveat() string

Caveat is the one line a renderer shows beside an excerpt, or "" when the hit carries no caution worth spending a line on.

Deliberately silent for an ordinary transcription: a caveat on every row is a caveat on none, and the rows that need one stop standing out.

type Home

type Home string

Home is raglit's on-disk layout: everything for one index under one directory, so the whole thing is a single portable unit you can copy, back up, or delete wholesale.

<home>/
  index.sqlite   the FTS5 index (documents:pages:fragments)
  originals/     ingested source files, kept so the index is self-contained
  pages/         rasterized/extracted page images for OCR (filled by Slice C)

Default location: $RAGLIT_HOME, else ~/local/raglit. Override per-invocation with the CLI --home flag.

func DefaultHome

func DefaultHome() Home

DefaultHome resolves the default index home: $RAGLIT_HOME if set, else ~/local/raglit. Falls back to ./raglit-home if the user home is undiscoverable.

func DiscoverHome

func DiscoverHome() Home

DiscoverHome resolves the home for a command given no explicit --home: the nearest ancestor .raglit/ (walking up from the cwd), else DefaultHome() ($RAGLIT_HOME, else ~/local/raglit). `raglit init` instead writes ./.raglit unconditionally, so a fresh project always gets its own local home.

func ScopedIndexHome

func ScopedIndexHome(root, name string) Home

ScopedIndexHome is the daemon's Home for one named index under a scoped root: <root>/indexes/<name>. Exported so a CLI can READ the index the daemon owns rather than the project-local one.

The two are not the same database and routinely disagree. A daemon-routed project ingests through the daemon, so <root>/indexes/<ns>__<name> is the corpus; the project's own .raglit/index.sqlite is whatever was last written locally, which may be years of documents behind. A command that opens the wrong one answers confidently about a corpus nobody is using.

func (Home) AttachmentDir

func (h Home) AttachmentDir(archivePath string) string

AttachmentDir is where one archive's attachments live, keyed by the archive's path the same way OriginalPath and PageDir are keyed. Deterministic, so nothing has to record it.

func (Home) AttachmentsDir

func (h Home) AttachmentsDir() string

AttachmentsDir holds files extracted OUT of ingested documents — today, mail attachments.

In raglit's storage, not beside the archive. Extraction used to write <archive>.raglit-attachments/ into the corpus: 204 files across 44 directories in one legal evidence tree, for a feature raglit owns end to end. An extracted attachment is a real document — it is ingested, captioned and searchable — but where its bytes LIVE is raglit's business, exactly as it already is for uploads/ and pages/.

func (Home) ConfigPath

func (h Home) ConfigPath() string

ConfigPath is the model-connection config (base URL, token, model ids).

func (Home) Ensure

func (h Home) Ensure() error

Ensure creates the home layout (home + pages/ + attachments/) if absent.

func (Home) IndexPath

func (h Home) IndexPath() string

IndexPath is the home's primary (default) SQLite index file.

func (Home) OriginalPath

func (h Home) OriginalPath(docPath string) string

OriginalPath is where the original for a given document path is stored — a deterministic function of the doc path (hash-prefixed to avoid basename collisions across directories), so no DB column is needed to find it back.

func (Home) OriginalsDir

func (h Home) OriginalsDir() string

OriginalsDir is where raglit used to copy every ingested source file. Nothing writes it now and nothing ever read it back (USER 2026-10-24: a local file is checked on disk, and missing means prunable; an upload already keeps its bytes in uploads/). Prune reports what is left and removes it with --delete-orphan-originals.

func (Home) PageDir

func (h Home) PageDir(docPath string) string

PageDir is the per-document subdirectory of pages/ that holds its page images. Deterministic from the doc path, like OriginalPath.

func (Home) PagesDir

func (h Home) PagesDir() string

PagesDir holds page images derived from originals (OCR input).

func (Home) TranscriptDir

func (h Home) TranscriptDir(mediaPath string) string

TranscriptDir is where one recording's transcript and reading live, keyed by the recording's path the same way OriginalPath and PageDir are keyed. Deterministic, so nothing has to record it.

func (Home) TranscriptsDir

func (h Home) TranscriptsDir() string

TranscriptsDir holds transcripts of ingested recordings.

In raglit's storage, beside originals/ and pages/, for the reason the attachments migration states: where a generated artifact's bytes live is raglit's business, not the corpus's. A transcript is more clearly raglit's own output than an extracted attachment ever was — nobody put it in the evidence tree and nothing outside raglit refers to it by path.

func (Home) UploadsDir

func (h Home) UploadsDir() string

UploadsDir is where uploaded documents live in a home.

type Identifier

type Identifier struct {
	Client Chatter
	// Model is recorded on the document, so a caption can be told from one a
	// different model wrote (and from a person's).
	Model      string
	MaxRetries int // JSON fix-loop attempts after the first try (default 2)
	// contains filtered or unexported fields
}

Identifier asks a model what a document is. One call per document, on the assembled transcript rather than per page — cheap next to the OCR that produced the transcript.

func NewIdentifier

func NewIdentifier(c Chatter, model string) *Identifier

NewIdentifier builds an Identifier over a chat client (an *llm.Client).

func (*Identifier) ExtractFields

func (id *Identifier) ExtractFields(ctx context.Context, text string, t DocType, hint string) (DocFields, error)

ExtractFields asks the model to fill out one type's schema.

The type's schema becomes the tool's parameters, so the SAME fix loop that holds identity to its shape holds an extraction to the corpus owner's — a schema authored months ago is enforced by the machinery that was already here, rather than by a second, worse copy of it.

func (*Identifier) Identify

func (id *Identifier) Identify(ctx context.Context, ask IdentityAsk) (DocIdentity, error)

Identify reads a document's assembled text and returns what it is.

The FILENAME IS DELIBERATELY NOT PASSED. This exists because filenames lie, and a model given "Lead-Based Paint (1).pdf" alongside a purchase and sale agreement will hedge toward the name it was handed — which reproduces exactly the failure the caption is supposed to catch. The document says what it is.

Returns an error rather than a fallback. A document with no caption is a document a person can still find by its filename; a document with a WRONG caption is one whose list entry now lies with a machine's confidence. There is no degraded answer worth storing here. tagContext is the index's established tag vocabulary — see Store.TagContext. It is a PARAMETER rather than a field because one *Identifier is shared by every index in a registry and by every slot of the captioning queue: a field would be a data race, and would carry one index's vocabulary into the next.

func (*Identifier) IdentifyTags

func (id *Identifier) IdentifyTags(ctx context.Context, ask IdentityAsk) (content, roles []string, err error)

IdentifyTags asks for TAGS ALONE, leaving a caption that already exists alone with it. The backfill for a corpus captioned before tags existed: the full identity would rewrite hundreds of names that are already right (and a person's, which must never be regenerated), for the sake of two columns.

Returns the content and role tags. Source/Model/At are the CALLER's to merge, because what is being written is part of an identity somebody else already authored.

type IdentityAsk

type IdentityAsk struct {
	// Text is the document's own words, as indexed.
	Text string
	// TagContext is the index's established tag vocabulary (Store.TagContext),
	// so new tags align with terms the corpus already uses.
	TagContext string
	// IndexHint is what the corpus owner says about reading this collection
	// (Store.IndexHint).
	IndexHint string
	// DocTypes are the index's registered document types, which the answer may
	// choose one of. Empty → the doc_type field is not asked for at all, which
	// is right for an index that has registered none.
	DocTypes []DocType
	// Path is the document's path, which picks the caption profile and, for a
	// profile that shows it, is quoted in the prompt.
	Path string
	// Profile is the caption rules (profileFor); nil means legal.
	Profile *captionProfile
}

IdentityAsk is everything one identity call needs to know that is not the prompt: the document, and the three things about the INDEX that shape the answer.

A struct rather than four parameters, and a parameter rather than fields on the Identifier, because one *Identifier is shared by every index in a registry and by every slot of the captioning queue — state there is a data race and carries one index's context into the next.

type IdentityJob

type IdentityJob struct {
	ID    int64  `json:"id"`
	Path  string `json:"path"`
	State string `json:"state"` // pending|running|done|skipped|error
	Force bool   `json:"force,omitempty"`
	// Mode is WHICH ask this job is: IdentityAskFull, IdentityAskTags (leave the
	// caption alone — the backfill for a corpus captioned before tags existed)
	// or IdentityAskFields (fill out the schema of the type it resolved as).
	Mode       string `json:"mode,omitempty"`
	Error      string `json:"error,omitempty"`
	EnqueuedAt int64  `json:"enqueued_at,omitempty"`
	StartedAt  int64  `json:"started_at,omitempty"`
	FinishedAt int64  `json:"finished_at,omitempty"`
}

IdentityJob is one queued captioning task.

type IdentityQueueStatus

type IdentityQueueStatus struct {
	Pending int `json:"pending"`
	Running int `json:"running"`
	Done    int `json:"done"`
	// Skipped is documents there was nothing to caption for — a scanned page with
	// forty characters on it, an empty attachment. Its own state rather than a
	// failure: nothing went wrong, nothing will go better next time, and counting
	// it as failed puts a permanent red number on a corpus that is fine. Kept out
	// of the re-queue for the same reason.
	Skipped int `json:"skipped"`
	Failed  int `json:"failed"`
}

IdentityQueueStatus is the queue in one line: how much is left, how much is running, how much finished, and how much failed.

func (IdentityQueueStatus) Empty

func (q IdentityQueueStatus) Empty() bool

Empty reports whether there is nothing outstanding.

type IdentityStatus

type IdentityStatus struct {
	Path string `json:"path"`
	DocIdentity
}

IdentityStatus is one document's identity, for a listing.

type IdentityWorker

type IdentityWorker struct {
	Store *Store
	// Slots is how many model calls are in flight. 0 → DefaultIdentitySlots.
	Slots int
	// IdlePoll is how long Run waits when the queue is empty. Default 2s.
	IdlePoll time.Duration
	// OnDone, when set, is called for each finished job — for a CLI that wants
	// to print progress. Called from the committer, one at a time.
	OnDone func(job IdentityJob, id DocIdentity, err error)
	// OnReclaim, when set, reports rows requeued from a dead process at the top
	// of a Drain. Worth saying out loud: those documents are about to be redone,
	// so a count that does not match what was asked for has an explanation.
	OnReclaim func(n int)
}

IdentityWorker drains the captioning queue.

Shaped as a pipeline rather than as N self-contained workers, because the model is the scarce resource and everything else is sqlite: a loader claims rows and reassembles document text, `Slots` callers do NOTHING but the model call, and a committer writes the captions and closes the rows. A slot is held for the request and not a millisecond longer.

func (*IdentityWorker) Drain

func (w *IdentityWorker) Drain(ctx context.Context) (int, error)

Drain works the queue until it is empty, returning how many jobs finished (successfully or not). This is the whole sweep for a CLI; Run is the daemon's version, which waits for more work instead of returning.

func (*IdentityWorker) Run

func (w *IdentityWorker) Run(ctx context.Context)

Run drains forever, sleeping IdlePoll between empty polls, until ctx is done.

type ImageEmbedConfig

type ImageEmbedConfig struct {
	Model  string `json:"model,omitempty"`   // e.g. nomic-embed-vision-v1.5; empty → disabled
	URL    string `json:"url,omitempty"`     // full endpoint; empty → Nomic Atlas image API
	APIKey string `json:"api_key,omitempty"` // empty → reuse the main api_key
}

ImageEmbedConfig configures the optional figure IMAGE embedder (see Config.ImageEmbed). Model empty → disabled (figures embed their description).

type ImageEmbedder

type ImageEmbedder interface {
	EmbedImage(ctx context.Context, mime string, data []byte) ([]float32, error)
	// Aligned reports whether the image vectors share the text query embedder's
	// space, so a text query can cosine against them (true for nomic-vision paired
	// with nomic-text). False → image vectors are stored but not text-searchable.
	Aligned() bool
}

ImageEmbedder embeds a figure IMAGE into a vector (a CLIP-style tower). raglit ships nomic-vision (imageembed.go); with none configured, figures fall back to embedding their description with the text embedder.

type IndexConfig

type IndexConfig struct {
	Roots   []Root   `json:"roots,omitempty"`
	Include []string `json:"include,omitempty"` // a file must match one to be indexed
	Ignore  []string `json:"ignore,omitempty"`  // merged with project + built-in ignore
	// WritebackTranscriptionMd NO LONGER DOES ANYTHING, and is kept only so an
	// existing config still parses.
	//
	// It made ingest write <doc>.raglit-transcription.md beside every document it
	// read — 407 files into one legal evidence tree, each duplicating text raglit
	// already held in `pages`, in the fragments, and on /api/doc-detail. raglit
	// then needed IsGeneratedSidecar, a builtinIgnore entry and a refusal inside
	// the writer itself to avoid tripping over its own output.
	//
	// `raglit transcribe` still writes one. The difference is that a person asked.
	WritebackTranscriptionMd bool `json:"writeback_transcription_md,omitempty"`
	// ExtractEmailAttachments writes a mail archive's attachments into
	// <archive>.raglit-attachments/ beside it, with a MANIFEST.md recording which
	// message each came from. Off by default for the same reason: an archive can
	// carry 69 files and putting them in somebody's corpus uninvited is not an
	// indexer's call. Unlike a transcription the extracted files ARE indexable —
	// they are originals that travelled inside an envelope, not derived output —
	// so the next `sync` picks them up as ordinary files.
	ExtractEmailAttachments bool `json:"extract_email_attachments,omitempty"`
	// OCRStrategy names an entry in OCRConfig.Strategies. Empty → the project's
	// OCR.Strategy, else today's behavior.
	//
	// Per index because that is the grain at which document KIND is already
	// declared here: `records/` is recorded surveys and `correspondence/` is
	// letters, and they want different amounts of work per page. Attaching the
	// policy to the corpus rather than to the command is the whole point — a
	// strategy retyped as a flag is one that gets forgotten.
	OCRStrategy string `json:"ocr_strategy,omitempty"`
}

IndexConfig is one index's source definition: the roots to scan and the include/ignore globs that apply to them (overridable per root).

type IndexDigest

type IndexDigest struct {
	// Name is the index this describes. Set by a caller reporting across
	// several; a store does not know what it is called.
	Name string `json:"index,omitempty"`
	// Path is the subtree this digest was computed over, empty for the whole
	// index. Reported because a digest of the whole index shown for a
	// path-scoped search claims coverage the subtree does not have.
	Path      string     `json:"path,omitempty"`
	Documents int        `json:"documents"`
	Untagged  int        `json:"untagged,omitempty"`
	Kinds     []TagCount `json:"kinds,omitempty"`
	Content   []TagCount `json:"content,omitempty"`
	Roles     []TagCount `json:"roles,omitempty"`
	// Types is the index's SCHEMAED documents — which registered document types
	// its documents resolved as, and how many of each have been extracted. Part
	// of the digest because "this corpus holds 88 work orders whose fields are
	// readable" is the single most useful thing that can be said about an index
	// that has any, and nothing else says it.
	Types []FieldsCoverage `json:"types,omitempty"`
	// About is the generated paragraph, when one has been written and still
	// matches the corpus it was written from. See IndexAbout.
	About string `json:"about,omitempty"`
	// AboutStale marks an About written from a materially different corpus —
	// present so a caller can show it AND say it is behind, rather than
	// choosing between a stale summary and none.
	AboutStale bool `json:"about_stale,omitempty"`
}

IndexDigest is what an index contains, counted from what is already stored.

func (IndexDigest) Empty

func (d IndexDigest) Empty() bool

Empty reports whether there is nothing to say about this index.

type IndexMeta added in v0.5.0

type IndexMeta struct {
	Owner      string
	Visibility Visibility
	Bytes      int64 // the index home on disk, every file in it
	Status     Status
}

IndexMeta is what a signed-in caller may know about an index it cannot read: who owns it, whether it is public, how much disk it takes and how its queue stands. No document, passage or path from inside it.

The web UI lists every index with this (Carl 2026-11-04: "metadata only unless admin"), so an index a project key created is visible to the people running the daemon, as an index with an owner, rather than missing.

type Job

type Job struct {
	ID         int64
	URL        string
	Title      string
	State      JobState
	Error      string
	Fragments  int
	EnqueuedAt int64
	StartedAt  int64
	FinishedAt int64
	// Fresh re-reads the document even when nothing changed: it skips both the
	// unchanged-bytes fast path and the cross-index pool. The escape hatch for a
	// cached result that is wrong for a reason the cache key cannot see.
	Fresh bool
}

Job is one queued ingestion of a URL.

type JobInfo

type JobInfo struct {
	ID        int64  `json:"id"`
	URL       string `json:"url"`
	Title     string `json:"title"`
	State     string `json:"state"`
	Error     string `json:"error"`
	Fragments int    `json:"fragments"`
	Mode      string `json:"mode"` // 'llm' | 'offline' | ” — segmentation mode
	// Lane is which scheduler ran it: 'heavy' (vision/OCR, one GPU slot) or
	// 'light' (everything else, several at once). See lane.go.
	Lane       string `json:"lane"`
	EnqueuedAt int64  `json:"enqueued_at"`
	StartedAt  int64  `json:"started_at"`
	FinishedAt int64  `json:"finished_at"`
}

JobInfo is a full ingest-job row for the review UI's job table (all states).

type JobStage

type JobStage struct {
	// ID is the row's own identity, and the only stable one a stage has: a
	// RETRIED job appends a whole second run under the same job_id with seq
	// restarting at 1, so (job_id, seq) names up to four different rows. Job 927
	// on the fence-dispute index is four runs.
	ID     int64  `json:"id"`
	Seq    int    `json:"seq"`
	Name   string `json:"name"`
	Engine string `json:"engine"`
	State  string `json:"state"`
	Detail string `json:"detail"`
	At     int64  `json:"at"`
}

JobStage is one recorded pipeline step.

type JobState

type JobState string

JobState is the lifecycle of an ingest job.

const (
	JobPending JobState = "pending"
	JobRunning JobState = "running"
	JobDone    JobState = "done"
	JobError   JobState = "error"
)

type JudgementStore

type JudgementStore struct {
	// contains filtered or unexported fields
}

The judgement store: what a PERSON decided about this corpus.

Nothing here can be recomputed. raglit can measure that two documents share 97% of their text; it cannot decide whether that means one instrument scanned twice or one instrument amended. A person decides, and the decision has to be written down or it is done again next week.

raglit-audit.jsonl IS the store. Not a log of it, not a backup of it — the thing itself, read at open and answered from memory.

It was briefly a SQLite database projected from the trail, with a schema, a generated query layer, a rebuild path and a table that duplicated the trail verbatim. On the corpus that motivated all of it: twelve events, eleven rows, a 68 KB database projecting a 12 KB file. Parsing the file is faster than opening the database, the constraints were already enforced in Go before any write, and the indexes covered eleven rows. It bought nothing and cost a concept everybody had to learn.

If the trail ever grows past what is comfortable to parse, cache it. Until a measurement says so, answering at runtime is both simpler and correct by construction — there is no projection to drift, catch up, or rebuild.

func OpenJudgements

func OpenJudgements(auditPath string) (*JudgementStore, error)

OpenJudgements reads a project's trail. A missing trail is a corpus nobody has ruled on, which is the normal starting state.

The projection is gone. This answers from the trail, in memory, and there is no second database to drift, catch up or rebuild — verdicts that DO want a queryable projection now land in the index itself, see attestations.go.

func (*JudgementStore) Authorities

func (s *JudgementStore) Authorities() []ReadingAuthority

Authorities returns every authority ruling.

func (*JudgementStore) Authority

func (s *JudgementStore) Authority(source string) (ReadingAuthority, bool)

Authority returns the ruling for a source, if a person has made one.

func (*JudgementStore) Close

func (s *JudgementStore) Close() error

Close exists so callers need not care that there is nothing to close.

func (*JudgementStore) CoverageOf

func (s *JudgementStore) CoverageOf(parent string, pages int) (Coverage, error)

CoverageOf reports which of a bundle's pages belong to no slice.

func (*JudgementStore) DeleteSlice

func (s *JudgementStore) DeleteSlice(id string) error

DeleteSlice removes a declaration. The trail keeps the creation AND the deletion, so replaying lands in the same place and the history stays readable.

func (*JudgementStore) History

func (s *JudgementStore) History(kind, subject string) ([]AuditEvent, error)

History returns every event recorded on one subject, oldest first.

func (*JudgementStore) OnTranscriptionChange

func (s *JudgementStore) OnTranscriptionChange(fn func(TranscriptionChange))

OnTranscriptionChange registers a listener for a page's active reading changing. Listeners run in registration order, synchronously, after the event is durable in the trail.

func (*JudgementStore) PageCorrections

func (s *JudgementStore) PageCorrections(doc string) (map[int]PageCorrection, error)

PageCorrections returns the ACTIVE correction for each page of a document.

func (*JudgementStore) PageReadings

func (s *JudgementStore) PageReadings(doc string, page int) []PageCorrection

PageReadings returns every reading ever recorded for one page, oldest first. The last is the active one; the rest are superseded and kept.

func (*JudgementStore) PutAuthority

func (s *JudgementStore) PutAuthority(a ReadingAuthority) error

PutAuthority records which reading of a source governs.

Keyed on the SOURCE, not on a pair: a third reading arriving later is ordered against the others by this one ruling rather than needing a new one for every combination. Re-ruling replaces — a person changing their mind is one fact, and the trail keeps both.

func (*JudgementStore) PutPageCorrection

func (s *JudgementStore) PutPageCorrection(c PageCorrection) error

PutPageCorrection records what a person read off a page.

It carries what it SUPERSEDES. A correction attests that a better reading exists; it does not erase the old one, which stays in the trail as the reading the index held when the document was cited.

func (*JudgementStore) PutRelation

func (s *JudgementStore) PutRelation(m Mark) error

PutRelation records a ruling on a pair, normalizing the sides so the pair is one fact however it was given.

func (*JudgementStore) PutSlice

func (s *JudgementStore) PutSlice(sl Slice) error

PutSlice records that a page range of a bundle is a document.

func (*JudgementStore) Relation

func (s *JudgementStore) Relation(a, b string) (Mark, bool, error)

Relation returns the ruling on a pair, in either order.

func (*JudgementStore) Relations

func (s *JudgementStore) Relations() ([]Mark, error)

Relations returns every ruling, in a stable order.

func (*JudgementStore) RelationsFor

func (s *JudgementStore) RelationsFor(doc string) ([]Mark, error)

RelationsFor returns every ruling involving one document.

func (*JudgementStore) Restore

func (s *JudgementStore) Restore(path, by string) error

Restore returns a withdrawn document to the corpus. The withdrawal stays in the trail: a decision reversed is still a decision that was made.

func (*JudgementStore) Slice

func (s *JudgementStore) Slice(id string) (Slice, bool, error)

Slice returns one declaration by id.

func (*JudgementStore) SliceParents

func (s *JudgementStore) SliceParents() ([]string, error)

SliceParents lists the bundles that have any slice.

func (*JudgementStore) Slices

func (s *JudgementStore) Slices() ([]Slice, error)

Slices returns every declaration, parent then page order.

func (*JudgementStore) SlicesOf

func (s *JudgementStore) SlicesOf(parent string) ([]Slice, error)

SlicesOf returns one bundle's declarations, in page order.

func (*JudgementStore) Withdraw

func (s *JudgementStore) Withdraw(w Withdrawal) error

Withdraw records that a document is out of the corpus, and why.

Appends first, then applies — the order audit.go explains: a process that dies between leaves the trail ahead of the state, which a re-read fixes, where the reverse would leave a decision in memory that nothing recorded.

func (*JudgementStore) Withdrawals

func (s *JudgementStore) Withdrawals() []Withdrawal

Withdrawals lists every document ruled out, path order.

func (*JudgementStore) Withdrawn

func (s *JudgementStore) Withdrawn(path string) (Withdrawal, bool)

Withdrawn reports the ruling that took a document out of the corpus, if any.

type Judgements

type Judgements interface {
	// Relations
	PutRelation(m Mark) error
	Relation(a, b string) (Mark, bool, error)
	Relations() ([]Mark, error)
	RelationsFor(doc string) ([]Mark, error)

	// Slices
	PutSlice(sl Slice) error
	DeleteSlice(id string) error
	Slice(id string) (Slice, bool, error)
	Slices() ([]Slice, error)
	SlicesOf(parent string) ([]Slice, error)
	SliceParents() ([]string, error)
	CoverageOf(parent string, pages int) (Coverage, error)

	// Page readings
	PutPageCorrection(c PageCorrection) error
	PageCorrections(doc string) (map[int]PageCorrection, error)
	PageReadings(doc string, page int) []PageCorrection

	// Provenance
	History(kind, subject string) ([]AuditEvent, error)

	// Notification
	OnTranscriptionChange(fn func(TranscriptionChange))

	Close() error
}

The judgement surface, and why it is an interface.

Today's implementation answers every question by walking maps built from raglit-audit.jsonl at open. That is right for a trail of a few thousand events and wrong at some size nobody has measured yet.

So callers depend on this and never on the implementation. When a measurement says the walk is too slow, a cached implementation — a SQLite projection, an mmap'd index, whatever the measurement argues for — satisfies the same interface and is swapped in at construction. Nothing above changes, and the trail stays the record either way.

The write methods are here deliberately. A cache that could be read through but written around would drift on the first write that bypassed it.

type Key added in v0.4.0

type Key struct {
	ID        string   `json:"id"`
	Hash      string   `json:"hash,omitempty"`
	Kind      string   `json:"kind"`
	Principal string   `json:"principal"`
	Issuer    string   `json:"issuer"`
	Parent    string   `json:"parent,omitempty"`
	Read      []string `json:"read,omitempty"`
	Write     []string `json:"write,omitempty"`
	Roots     []string `json:"roots,omitempty"`
	Label     string   `json:"label,omitempty"`
	CreatedAt int64    `json:"created_at"`
	LastUsed  int64    `json:"last_used,omitempty"`
	Revoked   bool     `json:"revoked,omitempty"`
}

Key is one stored credential. Hash is never returned to a caller.

func (Key) AllowsRead added in v0.4.0

func (k Key) AllowsRead(name string) bool

AllowsRead reports whether k's Read patterns admit index name (no patterns: every index its issuer may read).

func (Key) AllowsWrite added in v0.4.0

func (k Key) AllowsWrite(name string) bool

AllowsWrite reports whether k's Write patterns admit index name (no patterns: whatever the ACL lets its principal write).

type KeyStore added in v0.4.0

type KeyStore struct {
	// contains filtered or unexported fields
}

KeyStore holds the keys. Safe for concurrent use.

func OpenKeyStore added in v0.4.0

func OpenKeyStore(p string) (*KeyStore, error)

OpenKeyStore loads (or starts) the store at path.

func (*KeyStore) Authenticate added in v0.4.0

func (ks *KeyStore) Authenticate(token string, now time.Time) (Key, error)

Authenticate returns the key a token belongs to, refusing one that is revoked (itself or any ancestor) or, for a session key, idle past KeyIdleTTL. Its last use is recorded.

func (*KeyStore) ByPrincipal added in v0.4.0

func (ks *KeyStore) ByPrincipal(p string) (Key, bool)

ByPrincipal returns the live session key acting as p ("key#<id>").

func (*KeyStore) Flush added in v0.4.0

func (ks *KeyStore) Flush() error

Flush writes any pending last-use times.

func (*KeyStore) HasProject added in v0.4.1

func (ks *KeyStore) HasProject(ns string) bool

HasProject reports whether a project has a live key.

func (*KeyStore) IsAncestor added in v0.4.0

func (ks *KeyStore) IsAncestor(anc, p string) bool

IsAncestor reports whether principal anc is a session key above key principal p (its parent, its parent's parent, ...).

func (*KeyStore) List added in v0.4.0

func (ks *KeyStore) List(by, admin string) []Key

List returns the keys by may see: every key for the admin, else the keys it issued and the keys under them. Hashes are left out.

func (*KeyStore) Mint added in v0.4.0

func (ks *KeyStore) Mint(k Key) (string, Key, error)

Mint stores k (ID, Hash and CreatedAt are filled in) and returns its token, the only time the token exists in the clear.

func (*KeyStore) Revoke added in v0.4.0

func (ks *KeyStore) Revoke(by, admin, id string) error

Revoke revokes key id and, through Authenticate's ancestor check, every key under it. by must be the admin, the issuer of the key or of a key above it, or an ancestor key.

func (*KeyStore) RevokeProject added in v0.4.1

func (ks *KeyStore) RevokeProject(ns string) error

RevokeProject revokes every key of a project, and through the ancestor check every key under them.

type Lane

type Lane string

Scheduling lanes — what a job COSTS, decided before it runs.

The queue used to be one serial round-robin: one goroutine, one job at a time, one index per turn (cmd/raglit/serve.go). So a forty-minute OCR of a twenty-two page scan held every other index's turn, and a markdown file that needs three seconds of embedding waited behind it. That is the complaint this answers, and it is not a throughput problem — the two kinds of work do not even contend for the same resource.

LaneHeavy is vision: a page at a time through a VLM, minutes per document, and bounded by ONE GPU admission slot. Running two at once does not go faster; it queues inside the endpoint where nothing here can see or resume them. That is the same reason the identity queue drains one index at a time, and the reason OCR moved to a model on the second card.

LaneLight is everything else: pandoc, a mail parser, a spreadsheet reader, the deterministic fragmenter, and embedding — which is a batched call measured in seconds. Several at once is fine and is the point.

The split is by RESOURCE, not by speed. A large text file is not fast, and it still belongs in light, because what makes heavy serial is the GPU slot rather than the wall clock.

const (
	LaneHeavy Lane = "heavy"
	LaneLight Lane = "light"
)

func LaneFor

func LaneFor(url string) Lane

LaneFor guesses a lane from a URL alone, which is all an enqueue has.

A guess, and deliberately so: the real decision needs the BYTES (Worker.route sniffs content when the name says nothing), and a queue that fetched every document to decide where to queue it would do the expensive half of the work twice. So the extension decides, and the worker corrects the row once it knows — see Worker.ProcessJob.

Two ways it is wrong, both survivable. A PDF that turns out to carry a text layer is read without a model, so it occupied the heavy lane for seconds instead of minutes: wasteful, not incorrect. An extensionless file that sniffs to a scan runs its OCR in the light lane, which is the case worth naming — one light slot is busy for minutes. It is one job, the row is corrected, and a retry goes to the right lane.

func LaneForKind

func LaneForKind(k DocKind) Lane

LaneForKind maps a routed document to the lane that can afford it.

KindAudio is heavy, and is the longest single job raglit has. A 44-minute hearing transcribes in about twenty minutes; the light lane's three slots exist for work measured in seconds, and three recordings would hold all of them while every text and email ingest queued behind. That is precisely the case LaneFor's comment names as the one worth naming, arrived at from the other direction.

Heavy's single slot also serialises them, which is what the far end wants: oidio is one process holding one recogniser, so a second concurrent transcription does not start earlier — it blocks inside the server, exactly as the vision endpoint does.

type LaneStatus

type LaneStatus struct {
	Pending int `json:"pending"`
	Running int `json:"running"`
	// Slots is how many jobs this lane runs at once — the number that says
	// whether a pending count is a queue or a moment.
	Slots int `json:"slots"`
}

LaneStatus is one lane's share of the queue.

type LayerSavings

type LayerSavings struct {
	Name       string `json:"name"`
	Documents  int    `json:"documents"`
	Duplicates int    `json:"duplicates"` // documents another layer (or the base) also holds
}

LayerSavings is one layer's share of the duplication.

type LayoutBox

type LayoutBox struct {
	// Normalised to 0-1000 per axis, as the model emits them.
	//
	// One tag per field, deliberately. `X0, Y0, X1, Y1 int ` + "`" + `json:"x0"` + "`" + “ applies the
	// SAME tag to all four, encoding/json suppresses the conflicting names, and
	// every box goes over the wire with NO COORDINATES — the overlay draws
	// nothing. The Go tests passed throughout, because they exercised Pct() on
	// the struct and never the wire format.
	X0    int    `json:"x0"`
	Y0    int    `json:"y0"`
	X1    int    `json:"x1"`
	Y1    int    `json:"y1"`
	Label string `json:"label"`
	Text  string `json:"text"`
}

LayoutBox is one block the model reported: where it is, what kind it thought it was, and what it read there.

func ParseLayoutBoxes

func ParseLayoutBoxes(raw string) []LayoutBox

ParseLayoutBoxes extracts the blocks from a page's raw transcription.

Returns nil when the text carries no boxes — a tesseract page, a text-layer page, or a model that does not emit them. A caller renders the text alone in that case; "no layout" is an ordinary answer, not an error.

func (LayoutBox) Pct

func (b LayoutBox) Pct() (left, top, width, height float64)

Pct returns the box as CSS percentages (left, top, width, height), which is the whole reason the normalised space is convenient: no image dimensions, no aspect correction, and it stays correct at any rendered size.

type LayoutOpts

type LayoutOpts struct {
	// DPI the image was rendered at. Everything below is derived from it.
	DPI int // 0 → 200
	// GapIn is how far apart ink may be, IN INCHES, and still belong to one
	// cluster. The single knob that decides whether the lines of a paragraph
	// merge into one block (they should) or a drawing's separate labels merge
	// into one (they must not).
	//
	// 0.25in sits above text line pitch (~0.2in for 10pt) and below the spacing
	// between labels on a survey (~0.5in and up), which is the whole window this
	// has to land in.
	GapIn float64 // 0 → 0.25
	// MinIn is the smallest side of a cluster worth reading; below it is a speck.
	MinIn float64 // 0 → 0.08
	// CellPx overrides the derived working resolution. Segmentation does not need
	// full resolution, and a 40 megapixel sheet does need to not be scanned per
	// pixel — derived as one cell per 0.04in, so the grid means the same thing at
	// any render resolution.
	CellPx int // 0 → DPI/25
	// GapCells and MinCells override the derived cell counts. Prefer GapIn/MinIn.
	GapCells int
	MinCells int
	// MaxClusters caps the result. Above it the smallest are merged into their
	// nearest neighbour rather than dropped — losing a label is worse than
	// reading it alongside its neighbour.
	MaxClusters int // 0 → 24
	// Disabled falls back to the geometric grid — for a corpus where segmentation
	// misreads the layout, and for testing the grid itself, which is otherwise
	// unreachable once clusters are preferred.
	Disabled bool
}

LayoutOpts tunes cluster finding.

The distances are PHYSICAL — inches — and the cell grid is derived from DPI, because a pixel means nothing on its own. The first version fixed GapCells at 3 cells of 8px, which is 0.12 inch at 200 DPI: narrower than the line spacing it was supposed to bridge. So a certificate block fragmented into many tight crops, a number split across two of them was read by neither, and wiring the segmenter into the descent scored 12/17 where the geometric grid scored 16/17.

type Mark

type Mark struct {
	A    string   `json:"a"`
	B    string   `json:"b"`
	Kind MarkKind `json:"kind"`
	// Supersedes is the path that GOVERNS, for a version pair whose order is
	// known. Empty means the pair is two versions with no ruling on precedence —
	// a real and common state (two undated drafts), not a missing field.
	Supersedes string `json:"supersedes,omitempty"`
	Note       string `json:"note,omitempty"`
	By         string `json:"by,omitempty"`
	At         string `json:"at,omitempty"`

	// The evidence as it stood when the ruling was made, so a later reader can
	// see what was in front of the person. Not used for anything: a ruling is not
	// re-derived from these, it is quoted.
	Relation Relation `json:"relation,omitempty"`
	Coverage float64  `json:"coverage,omitempty"`
}

Mark is one ruling on a pair of documents.

A and B are index-relative document paths, ORDER-NORMALIZED (A < B) so the pair is one fact rather than two and is found from either side. Supersedes is not normalized — it names a side, and which side is the whole content of an ordering.

func ReadLegacyRelations

func ReadLegacyRelations(path string) ([]Mark, error)

ReadLegacyRelations parses the retired relations.jsonl. Later lines win, as the append-only format intended. Missing file is not an error.

func (Mark) Normalize

func (m Mark) Normalize() Mark

Normalize puts a mark's two sides in canonical order. Supersedes is untouched.

func (Mark) Other

func (m Mark) Other(doc string) (string, bool)

Other returns the side of the pair that is not doc, and whether doc is in it.

type MarkKind

type MarkKind string

MarkKind is a person's ruling on an overlapping pair.

const (
	// MarkCopy — the same instrument, with no substantive difference. Whichever
	// copy is more legible is the one to cite; the other is redundant.
	MarkCopy MarkKind = "copy"
	// MarkVersion — the same instrument, differing in substance. Both are
	// evidence, and Supersedes records which one governs when that is known.
	MarkVersion MarkKind = "version"
	// MarkUnrelated — the overlap is real but means nothing: shared county forms,
	// a quoted paragraph, a common exhibit. Recorded rather than left blank so the
	// pair stops coming back as an open question.
	MarkUnrelated MarkKind = "unrelated"
)

type Mask

type Mask []iv

Mask marks regions of a document that are CORPUS-GENERIC — text that recurs across unrelated documents and therefore says nothing about whether two documents are copies of each other.

This is the single most important correction measured on real data, and the failure it fixes was not the one expected. Legal boilerplate (statutory deed forms, notary blocks) turned out to be survivable: it produces scattered runs that a 40-character floor and a coverage measure already discount. What broke the tool was EMAIL CHROME. Two entirely different one-paragraph emails from the same lawyer shared their header template and a 450-character confidentiality notice — 593 of 800 folded characters — and were reported as `duplicate` at 0.75/0.69 coverage.

Measured on the all-pairs audit: 774 duplicate-or-containment findings with no masking, 57 with it, and the same 5 known-true pairs present in both. A triage tool that cries duplicate 774 times on a 233-document corpus is a tool nobody runs twice. The specific pair above went from `duplicate` 0.75/0.69 to `overlap` 0.09/0.09 with 981 of its characters masked.

Neither Jaccard, containment, nor contiguity can separate that case: the shared text is long, contiguous, and genuinely identical. Only its DOCUMENT FREQUENCY distinguishes it. So generic regions are removed from the alignment.

Intervals are in FOLDED offsets, half-open, sorted and non-overlapping.

func BuildMask

func BuildMask(f FoldedText, generic map[uint64]bool, w, mod int) Mask

BuildMask locates a document's corpus-generic regions in folded offsets.

Only a 1-in-mod sample of shingles is known to be generic, so a generic REGION is reconstructed by widening each generic sample's window. The widening is what makes this work rather than merely shred runs: sampled shingles inside a generic region are spaced about `mod` apart, so windows of width w abut only when mod <= w and leave gaps otherwise. Padding by mod on each side closes those gaps for any mod, and the cost is up to mod characters of real text at each region boundary — under the MinRunChars floor, so it cannot turn a real match into a miss on its own.

Without the padding, at mod == w the masks met exactly and a one-character rounding left every other footer unmasked, which read as the cutoff being wrong.

type Mod

type Mod func(*strings.Builder)

Mod is a modifier: something true about the PIXELS this reader was handed, or about what the caller wants. Modifiers never change the question.

func WithDamage

func WithDamage(damage []string) Mod

WithDamage reports what was MEASURED about the pixels, and what may be asked about it.

Measured 2026-08-03: the variance of the laplacian identifies a blurred crop that the model itself diagnoses as "skew" with 0.9 confidence — and the deskew it then prescribes makes a blurred region worse. So the number is what notices, and the model is what decides, because only it can see whether the region is a faded fax or a drawing that is mostly white space.

func WithGrid

func WithGrid(grid string) Mod

WithGrid tells a computed tile that it is one, and what that obliges it to leave alone.

The 45% rule is asymmetric on purpose. Cells overlap by descentPadIn, so an item near a seam is partly visible from both sides; without a rule both guess. The two failure modes are not equal — a duplicated monument call is recoverable by anyone reading the transcript, a dropped one is invisible. At 45% nothing can be dropped by geometry: if this cell holds under 45% of an item, the neighbour necessarily holds over 55% and takes it.

func WithHint

func WithHint(hint string) Mod

WithHint states what the caller is looking for.

Appended rather than woven in: the instruction competes with the image for the model's attention, and a hint that rewrote the whole prompt would also change the parts that make the answer parseable.

type ModelRole added in v0.5.0

type ModelRole struct {
	BaseURL string
	APIKey  string
	Model   string
}

ModelRole is one model a worker calls: where it is served, the key for it, and the model id. Model empty → the role is off.

type Models added in v0.5.0

type Models struct {
	// Vision reads page images (OCR). Empty Model → scans and images are read by
	// the local engine alone (tesseract or OCR.CheapEngine), or fail as
	// unreadable when there is none.
	Vision ModelRole
	// Segment splits transcribed text into fragments. Empty Model → the vision
	// model; empty BaseURL → the vision endpoint.
	Segment ModelRole
	// Embed makes vectors. NewWorker probes it for its input limit; the
	// embedder itself is set on the Store or Registry, built by Models.Embedder.
	Embed ModelRole
	// Audio is the speech-to-text model. Empty Model → audio jobs fail with a
	// clear message.
	Audio ModelRole
	// Identity captions each ingested document (identity.go). Empty Model →
	// identity is off. Empty BaseURL → the segment endpoint.
	Identity ModelRole
	// OCR selects and tunes the cheap first-pass engine, and its default
	// strategy supplies the resolution policy.
	OCR OCRConfig
	// Fragmenter parameters. Zero → the defaults (ResolveFragParams).
	FragWindow, FragStride, FragFloor int
	// Configured input limits. Zero → probed from the endpoint and cached on
	// the index.
	EmbedLimitChars, EmbedLimitTokens, SegmentInputLimitTokens int
	// Channels is the per-model admission registry (modelchan.go). Required:
	// NewWorker and Identifier panic on nil. Model capacity is a property of the
	// SERVER, so build one registry for the process with NewChannels, set its
	// Max (the per-model channel ceiling, config model_channel_max), and pass it
	// to every index's worker. A registry per worker would split one server's
	// capacity between controllers that each learn it alone.
	Channels *Channels
	// OnUsage, when set, is called once per completed model response with the
	// role ("vision", "segment", "identity", "embed"), that role's model id and
	// the tokens the endpoint reported, so a host can total or cap spend per
	// role. It runs on the calling goroutine of the request, possibly from many
	// at once: it must be safe for concurrent use and must not block. An
	// OnUsage already on a client is chained, not replaced. nil → no change.
	//
	// Coverage: the four roles above only. STT, image embed and the OCR engine
	// use plain HTTP clients and do not report usage. "embed" reports only
	// through an Embedder from Models.Embedder (or a client wrapped with
	// UsageVector): set that one on the Store or Registry. The embed limit
	// probes NewWorker makes do not report.
	//
	// "embed" is an ESTIMATE, not a reported count: agentkit's Client.Embed
	// discards the response's usage, so UsageVector reports ceil(bytes/4) of
	// input. Code often runs below four bytes per token, so the estimate can
	// be low.
	OnUsage func(role, model string, u llm.Usage)
	// Code cuts source files at their declarations and Markdown at its
	// headings (KindCode, codesegment.go). nil → both are read into windows,
	// as before. Its Runtime() is in the KindCode pool key (poolRecipe).
	Code CodeSegmenter
}

Models is everything NewWorker needs to know about the models and the ingest settings. Plain data: it does not read a Home, a config file or flags. raglit fills it from config.json and its flags; an embedder of the library fills it from its own config.

func (Models) Embedder added in v0.7.0

func (m Models) Embedder() *Embedder

Embedder builds the embedder for ingest and search: the batch retry policy, the embed model's admission channel with its 429 hook, and the OnUsage report as role "embed". nil when no Embed model is set. m.Channels must be set. Set the result on the Store (SetEmbedder) or the Registry.

func (Models) Identifier added in v0.5.0

func (m Models) Identifier() *Identifier

Identifier builds the document-identity model client, or returns nil when identity is off (no Identity model). m.Channels must be set.

Same batch retry policy as the rest of ingest: this call happens at the end of a document that may have cost thirty OCR requests, and losing the caption to a transient 502 wastes the one moment the whole transcript is in hand.

func (Models) UsageVector added in v0.7.0

func (m Models) UsageVector(inner VectorClient) VectorClient

UsageVector wraps an embedding client so each successful call reports to m.OnUsage as role "embed". Returns inner unchanged when OnUsage is nil.

The token count is an estimate (see Models.OnUsage): ceil(bytes/4) over the inputs sent. The model id is the one the call names, else m.Embed.Model.

type NomicVisionEmbedder

type NomicVisionEmbedder struct {
	URL    string // full endpoint; "" → defaultNomicVisionURL
	APIKey string
	Model  string // "" → defaultNomicVisionModel
	HTTP   *http.Client
}

NomicVisionEmbedder embeds figure images via nomic-embed-vision over HTTP. Its vectors are L2-normalized (like the text embedder's), so cosine is a dot product.

func NewNomicVisionEmbedder

func NewNomicVisionEmbedder(url, apiKey, model string) *NomicVisionEmbedder

NewNomicVisionEmbedder builds the embedder. url/model empty → the nomic defaults.

func (*NomicVisionEmbedder) Aligned

func (e *NomicVisionEmbedder) Aligned() bool

Aligned reports that nomic-vision shares nomic-text's space — true by construction (this embedder is the nomic pair). The caller is responsible for keeping the text embed model on nomic-embed-text; a mismatch silently degrades figure ranking rather than erroring.

func (*NomicVisionEmbedder) EmbedImage

func (e *NomicVisionEmbedder) EmbedImage(ctx context.Context, mime string, data []byte) ([]float32, error)

EmbedImage posts the image as multipart form-data (model + images file) and returns the L2-normalized embedding.

type Note

type Note struct {
	ID int64 `json:"id"`
	// Page is 0 for a note about the whole document. Stored as NULL in that
	// case — the column is nullable so "about the document" and "about page 0"
	// cannot be confused, and page 0 is a real value here (text windows carry
	// it).
	Page      int    `json:"page,omitempty"`
	Body      string `json:"body"`
	Author    string `json:"author,omitempty"`
	CreatedAt string `json:"created_at,omitempty"`
}

Note is one person's comment on a document, or on one of its pages.

type OCR

type OCR struct {
	Client Chatter
	// Model is the vision model's id, recorded on every page this OCR reads
	// (page_readings.model). Not used to call anything — the client already knows
	// which model it talks to — but a transcription whose author is unrecorded
	// cannot be told from one a different model produced, and a corpus outlives
	// several models.
	Model  string
	Prompt string // transcription instruction; "" → Prompt(PromptPlain)
	// Collection is what the corpus owner says about reading THIS collection
	// (Store.IndexHint) — appended to every reading prompt.
	//
	// It reaches the transcription rather than only the later asks because
	// "how to decode" is a property of the pixels: "RO" on a garage's paperwork
	// is a repair order, the second column of a carbon copy is the customer's,
	// and a model reading one page cannot infer either. It is part of the
	// READING RECIPE for the same reason — a changed hint changes what a page
	// says, and pooled work read under the old one must not be replayed under
	// the new.
	Collection string
	// Trace, when non-nil, receives one line per decision this OCR takes.
	//
	// It exists because nothing else reports what a call DID. `raglit doctor`
	// says what is configured; the page cache, the cheap tier, the assist, the
	// repetition guard and the tool list are all invisible at the call site, and
	// a reader of the output cannot tell a cheap-engine page from a VLM page, an
	// assisted prompt from a bare one, or a cut stream from a complete one.
	//
	// Written to be read by a person mid-investigation, not parsed.
	Trace io.Writer

	// Render is the automatic per-page resolution policy. Zero → the measured
	// package defaults, which is what every page got when these were constants.
	// Carried on OCR rather than passed down because OCR is already threaded to
	// every extract path, and a policy that has to be plumbed separately is one
	// that will be plumbed to some paths and not others.
	Render RenderPolicy

	// TraceJSONL, when non-nil, receives ONE JSON OBJECT PER LINE describing each
	// interaction and the transforms applied to it.
	//
	// Separate from Trace rather than a format flag on it, because the two answer
	// to different readers. Trace is prose for a person mid-investigation; this is
	// for diffing two runs, joining a failure to the exact bytes that produced it,
	// and counting what happened across a corpus — none of which survives being
	// parsed back out of a sentence.
	//
	// Every event carries img_sha (first 12 hex of the image bytes), and that is
	// what makes a line joinable: the same crop at the same scale has the same sha
	// across models and runs, so "these two models disagreed on THIS image" is a
	// join rather than a reconstruction.
	TraceJSONL io.Writer

	// TraceCtx is merged into every event this OCR emits. The region walk sets it
	// per call — doc, page, region, tokens/in² — so a line in the log says WHICH
	// crop it describes rather than only which image sha.
	//
	// A field rather than a parameter because the walk already swaps o.Prompt per
	// call and restores it after; carrying the context the same way keeps the two
	// in step, and there is no path where one is set and the other is not.
	TraceCtx map[string]any
	Cheap    PageEngine      // optional cheap first pass; nil → VLM-only
	Gate     GibberishConfig // when the cheap pass escalates to the VLM (zero → defaults)
	// DescribeFigures is the FIGURE gate (§3a): a born-digital PDF page carrying an
	// embedded image is rasterized to the VLM even when its text layer is clean, so
	// its diagrams get described. Orthogonal to the gibberish gate (which judges
	// TEXT quality); OFF by default because it flips such pages to the (costly)
	// vision path and thus to llm-seg.
	DescribeFigures bool

	// MaxTokens caps one page transcription; 0 → defaultOCRMaxTokens. See
	// chat.go for why an unbounded transcription is not a safe default.
	MaxTokens int

	// Assist changes what the cheap engine is FOR: not a substitute for the
	// vision model but a spelling reference handed to it.
	//
	// The cascade's bargain is "if the cheap read looks clean, keep it and skip
	// the model". That is a cost decision, and on a corpus of filed documents it
	// buys the wrong thing: measured here, tesseract reads a surveyor's
	// certificate as 20123164 with 86% confidence — clean-looking, wrong, and
	// the gibberish gate passes it.
	//
	// The two readers fail differently, which is what makes the assist work. The
	// vision model normalises proper names (DACE → DACF) and tesseract does
	// not; tesseract mangles digits and the vision model, at sufficient
	// resolution, does not. So the model reads the page, with the cheap engine's
	// WORDS for spelling and its NUMBERS REMOVED — measured on the disputed
	// record of survey, that combination got all four checked facts right,
	// including one auditor's file number that no other configuration read
	// correctly. Handing over the numbers as well loses two of them: the model
	// copies them, whatever the instructions say.
	Assist bool
}

OCR transcribes page images to text. It runs a CASCADE: an optional cheap first-pass engine (tesseract / paddleocr), gated by a gibberish detector, and falls back to a vision-capable chat model (e.g. gemma-4-12b on bonsai, via corrallm) only when the cheap pass is missing, errors, or looks like garbage. With no Cheap engine it is VLM-only — the original behavior.

func NewOCR

func NewOCR(c Chatter) *OCR

NewOCR wraps a Chatter (an *llm.Client) as an OCR transcriber. The cheap tier is off by default; set OCR.Cheap (see BuildPageEngine) to enable the cascade.

func TesseractOnlyOCR added in v0.5.0

func TesseractOnlyOCR(oc OCRConfig) *OCR

TesseractOnlyOCR is an OCR with no vision model: the configured cheap engine, else tesseract when it is on PATH. nil when neither exists. Every page is taken from the engine as read (OCR.PageAsSeen), and figures are not described, since describing one needs the vision model.

func (*OCR) AskWithHint

func (o *OCR) AskWithHint(hint string) func(context.Context, PageImage, RegionAbout) (RegionReading, error)

AskWithHint is AskWithOCR with a caller's statement of what is being looked for appended to whichever prompt applies.

Appended rather than woven in: the instruction competes with the image for the model's attention, and a hint that rewrites the whole prompt would also change the parts that make the answer parseable.

func (*OCR) AskWithOCR

func (o *OCR) AskWithOCR() func(context.Context, PageImage, RegionAbout) (RegionReading, error)

AskWithOCR builds an Ask that drives a VLM through the existing OCR client, so the region walk inherits the page cache, the repetition guard and the retry policy already in place.

func (*OCR) Page

func (o *OCR) Page(ctx context.Context, img PageImage) (string, error)

Page transcribes one page image and returns the trimmed text.

func (*OCR) PageAsSeen

func (o *OCR) PageAsSeen(ctx context.Context, img PageImage) (text, engine string, shrinks int, err error)

PageAsSeen is PageWithEngine plus the one thing a caller needs to reproduce what the model actually looked at: how many times the image was downscaled to fit the context before it was read.

Separate rather than a wider PageWithEngine because only the region walk cares — it records the number per region so the crop can be re-rendered exactly, and every other caller wants the text and the engine and nothing else.

func (*OCR) PageWithEngine

func (o *OCR) PageWithEngine(ctx context.Context, img PageImage) (text, engine string, err error)

PageWithEngine transcribes one page and reports which engine produced it: the cheap engine's Name() when its result passed the gibberish gate, else "vision". The cascade never drops a page — a cheap-engine error or a gibberish verdict escalates to the VLM rather than returning the bad text.

type OCRConfig

type OCRConfig struct {
	CheapEngine   string          `json:"cheap_engine,omitempty"`   // "none" | "tesseract" | "paddleocr"
	PaddleURL     string          `json:"paddle_url,omitempty"`     // sidecar base URL when cheap_engine=paddleocr
	TesseractBin  string          `json:"tesseract_bin,omitempty"`  // tesseract binary; "" → "tesseract"
	TesseractLang string          `json:"tesseract_lang,omitempty"` // -l language; "" → "eng"
	Gibberish     GibberishConfig `json:"gibberish,omitempty"`      // gate overrides; zero → precision-biased defaults
	// DescribeFigures escalates born-digital PDF pages that carry an embedded image
	// to the VLM so their figures are described inline (§3a figure gate). OFF by
	// default: it forces those pages onto the vision/llm-seg path (costly).
	DescribeFigures bool `json:"describe_figures,omitempty"`
	// Mode decides what the cheap engine is for.
	//
	//   "cascade" (default) — it reads first, and the vision model is paid for
	//     only when the gibberish gate rejects the result. A cost decision.
	//   "assist" — the vision model reads every page, with the cheap engine's
	//     WORDS as a spelling reference and its numbers removed. See OCR.Assist:
	//     the two readers fail differently, and this is how each one's strength
	//     is used without importing the other's errors.
	//
	// A corpus where a misread digit is expensive wants "assist"; a corpus where
	// throughput matters more than a certificate number wants "cascade".
	Mode string `json:"mode,omitempty"`

	// Strategies are named bundles of read policy, selectable per index via
	// IndexConfig.OCRStrategy. They exist because one corpus is not one kind of
	// page: a records/ folder of recorded surveys wants descent, tiling and a
	// hint about monument calls, while correspondence/ wants none of it and
	// should not pay for the check. Before this the knobs lived only on
	// `raglit regions` flags, so a policy could not be attached to a corpus at
	// all — it had to be retyped per invocation and was forgotten by the next.
	Strategies map[string]StrategyConfig `json:"strategies,omitempty"`

	// Strategy names the default for indexes that do not choose one. Empty means
	// the zero StrategyConfig, which is today's behavior exactly.
	Strategy string `json:"strategy,omitempty"`
}

OCRConfig selects and tunes the cheap first-pass OCR engine. The cascade tries this engine before the vision model and escalates only when the page looks like gibberish (see ocr.go, gibberish.go). CheapEngine="none" (the default) disables the cheap tier — the cascade is then VLM-only.

type OCRLine

type OCRLine struct {
	Text       string  `json:"text"`
	Confidence float64 `json:"confidence"`
}

OCRLine is one recognized text box from a cheap page-OCR engine, with the recognizer's confidence (0..1). The gibberish gate reads the per-line scores in aggregate (MeanConfidence); downstream only the joined Text matters.

type OffsetFrag

type OffsetFrag struct {
	Start, End int
	Text       string
}

OffsetFrag is one windowed fragment plus its half-open [Start,End) span into the source text it was cut from.

func OverlapFragments

func OverlapFragments(text string, window, stride, floor int) []OffsetFrag

OverlapFragments cuts text into overlapping windows. window/stride/floor must already be resolved (ResolveFragParams). A document at or under one window is a single fragment with no overlap; a tail shorter than floor is folded into the final window rather than emitted as a near-duplicate ("barely-over document").

type PaddleEngine

type PaddleEngine struct {
	BaseURL string
	HTTP    *http.Client
}

PaddleEngine is a PageEngine backed by an out-of-process PaddleOCR sidecar (the one ragtag used; optionally installed via docker). It POSTs the page image to <BaseURL>/ocr and decodes the bespoke {text, lines, mean_confidence, box_count} contract. Higher print accuracy than tesseract, at the cost of a running sidecar.

func NewPaddleEngine

func NewPaddleEngine(baseURL string) *PaddleEngine

NewPaddleEngine builds a PaddleEngine for the sidecar base URL. The timeout is generous: CPU PP-OCR on a dense page can take a couple seconds.

func (*PaddleEngine) Name

func (e *PaddleEngine) Name() string

func (*PaddleEngine) OCRPage

func (e *PaddleEngine) OCRPage(ctx context.Context, img PageImage) (PageOCR, error)

OCRPage sends one page image to the sidecar and returns its PageOCR. The Content-Type is taken from the page's mime (the sidecar decodes png as readily as jpeg), defaulting to image/jpeg. Transport / decode / non-200 errors propagate so the cascade can fall back to the vision model.

type PageCorrection

type PageCorrection struct {
	Doc  string `json:"doc"`
	Page int    `json:"page"`
	Text string `json:"text"`
	Note string `json:"note,omitempty"`
	By   string `json:"by,omitempty"`
	At   string `json:"at,omitempty"`
	// Supersedes is the reading this one replaced — the machine's text, or an
	// earlier correction.
	//
	// A correction is an ATTESTATION that a better reading exists, not an erasure
	// of the old one. What the machine said is evidence: it is what the index held
	// when a document was cited, it is what a stale quotation will match, and
	// "the OCR read 2015031104" is itself a fact about how reliable this sheet is.
	// Keeping it means the trail carries the whole lineage of a page and says
	// which reading is ACTIVE, rather than only the winner.
	Supersedes string `json:"supersedes,omitempty"`
}

PageCorrection is what a person read off a page that the machine got wrong.

Text replaces the machine read for that page when the transcription is rendered. Note records HOW it was established — the magnification, the source image — because a corrected identifier that cannot be re-checked is only a different unverified value.

type PageEngine

type PageEngine interface {
	OCRPage(ctx context.Context, img PageImage) (PageOCR, error)
	Name() string // short id for the per-page engine tag: "tesseract" | "paddleocr"
}

PageEngine is a CHEAP first-pass OCR over one rendered page image — the tier the cascade tries before paying for the vision model (see OCR.Page). It is the swappable "other OCR system" slot: a local CLI (tesseract) or a bespoke sidecar (paddleocr) are interchangeable here.

func BuildPageEngine

func BuildPageEngine(cfg OCRConfig) (PageEngine, error)

BuildPageEngine constructs the configured cheap engine, or nil for "none" (the cascade is then VLM-only). An unknown name or a paddleocr backend with no URL is a configuration error the caller should surface.

type PageImage

type PageImage struct {
	Page int    // 1-based page number
	Path string // where it was written (empty when outDir == "")
	Mime string // "image/png", "image/jpeg", …
	Data []byte // raw image bytes
}

PageImage is one page's image, ready for OCR.

func Pagify

func Pagify(pdfPath, outDir string) ([]PageImage, error)

Pagify extracts embedded page images from an image/scanned PDF, in page order. If outDir is non-empty each image is also written there as p<NN>.<ext> (with -<objnr> appended when a page carries multiple images). Returns ErrNoPageImages for a PDF with nothing to extract.

func PagifyBytes

func PagifyBytes(pdf []byte) ([]PageImage, error)

PagifyBytes extracts embedded page images from an in-memory image/scanned PDF — the MCP tool's rasterization path, so a caller can pass PDF bytes without a temp file. Same semantics as Pagify with an empty outDir (no files written).

type PageLayout

type PageLayout struct {
	Doc string `json:"doc"`
	// Page is 1-based, as everything user-facing in raglit is.
	Page int `json:"page"`
	// Engine and Model say WHO read it — a tesseract page has no boxes and that
	// is worth showing rather than looking like a failure.
	Engine string `json:"engine"`
	Model  string `json:"model"`
	// HasImage is false when the page image was never saved or has been cleaned
	// up; the text panes still work.
	HasImage bool `json:"has_image"`
	// ImgW/ImgH are the page image's pixel size, read from its header only.
	//
	// The boxes are normalised and so need no dimensions to PLACE — but a
	// reconstruction of the page from text alone needs the page's SHAPE, and a
	// client rendering it beside the image (not over it) has nothing to infer
	// that from. Zero when unknown; a caller then falls back to the block list.
	ImgW  int         `json:"img_w,omitempty"`
	ImgH  int         `json:"img_h,omitempty"`
	Boxes []LayoutBox `json:"boxes"`
	// Raw is the model's own output, markup and all. Indexed is what search
	// actually matches against. Showing both side by side is the point: they
	// have differed by 40% of their bytes and nothing surfaced it.
	Raw     string `json:"raw"`
	Indexed string `json:"indexed"`
}

PageLayout is everything needed to render one page three ways: the picture, what the model reported about it, and what the index actually holds.

type PageOCR

type PageOCR struct {
	Text           string
	Lines          []OCRLine
	MeanConfidence float64
	BoxCount       int
	// MedianGlyphPx is the median height of recognised text, in pixels of the
	// image as given — min(width, height) per word, because text that runs
	// sideways up a sheet has its height on the other axis. 0 when the engine
	// does not report geometry.
	//
	// It is how a page says it needs a better look: the pages every reader read
	// correctly measure 18-25 px here, and the page they all misread measures 10.
	// See renderDPIFor.
	MedianGlyphPx int
}

PageOCR is a cheap engine's result for one page image. Text is the recognized lines joined in reading order; MeanConfidence and BoxCount are the cheap signals the gibberish gate fuses to decide whether the page needs a vision-model re-OCR. BoxCount == 0 means no text at all (blank page or a pure figure) — treated as an empty page, not gibberish.

type PagePair

type PagePair struct {
	APage int `json:"a_page"`
	// BPage is 0 when nothing in B matches this page at all — a page present in
	// one document and absent from the other, which is the strongest per-page
	// finding available and is invisible in any average.
	BPage int `json:"b_page,omitempty"`
	// Rate is |A∩B| / |A| over the page's distinct shingles: how much of THIS
	// page of A occurs on that page of B. Asymmetric on purpose — a short page
	// wholly inside a long one is 1.0 here and small in reverse.
	Rate float64 `json:"rate"`
	// Identical is exact equality of the two pages' folded text. A rate of 1.0
	// is not the same claim: shingles can be a subset while the text differs in
	// what falls between them.
	Identical bool `json:"identical"`
	AChars    int  `json:"a_chars"`
	BChars    int  `json:"b_chars"`
}

PagePair is one page of A and the page of B it best corresponds to.

func PageRates

func PageRates(a, b FoldedText, w int) ([]PagePair, []int)

PageRates computes, for every page of a, the page of b it best corresponds to.

Exact, not sampled. The stored sketches hold a 1-in-N sample, which is right for finding candidates across a corpus and wrong here: these numbers are read by a person deciding which of two instruments a filing cites, and a rate that is usually right is the wrong instrument for that decision.

type PageReading

type PageReading struct {
	Doc    string `json:"doc"`
	Page   int    `json:"page"`
	Seq    int    `json:"seq"`
	Text   string `json:"text"`
	Source string `json:"source"` // machine | corrected
	// Engine and Model say WHICH reader produced this, where Source says only
	// whether it was a machine at all. A corpus read by a text layer, then
	// tesseract, then one vision model, then another cannot answer "how far do I
	// trust this page" while all four are equally "machine".
	Engine string `json:"engine,omitempty"` // vision | tesseract | paddleocr | text-layer
	Model  string `json:"model,omitempty"`  // the model id behind the engine, when it has one
	Note   string `json:"note,omitempty"`
	By     string `json:"by,omitempty"`
	At     string `json:"at,omitempty"`
	Active bool   `json:"active"`
}

PageReading is one version of a page's text.

type PageReview

type PageReview struct {
	Page      int    `json:"page"`
	Engine    string `json:"engine"`
	Vision    bool   `json:"vision"`
	HasImage  bool   `json:"has_image"`
	Fragments int    `json:"fragments"`
	Text      string `json:"text"`
	// TextChars and DescribedChars are how much indexable text this page produced
	// and how much of it was the model DESCRIBING an image rather than reading
	// one. Measured at ingest — the markup that says so does not survive into
	// Text, so this cannot be recomputed from what is returned here.
	TextChars      int `json:"text_chars,omitempty"`
	DescribedChars int `json:"described_chars,omitempty"`
}

PageReview is one page's OCR review: the engine that produced it, whether it needed the VLM, whether a page image is on disk, and the text indexed for it.

func (PageReview) DescribedPct

func (p PageReview) DescribedPct() int

DescribedPct is how much of this page a model made up, 0-100.

type PageSketch

type PageSketch struct {
	Page int
	// Chars is the page's folded length. Kept because Total alone cannot tell a
	// blank page (no shingles, nothing to say) from a page shorter than one
	// shingle (no shingles, but it does hold text).
	Chars int
	// Total is the count of DISTINCT shingles on the page — the denominator of
	// containment. Sampling makes the numerator estimable and would make this
	// unrecoverable, so it is stored.
	Total int
	// Hashes are the sampled shingles, distinct and sorted.
	Hashes []uint64
}

PageSketch is one page's stored evidence: the sampled shingle hashes, plus the sizes that sampling throws away.

func SketchPages

func SketchPages(f FoldedText, w, mod int) []PageSketch

SketchPages builds the stored sketch for a folded document, one entry per page. Pages are emitted even when they hold no shingles, so a re-sketch replaces a document's rows completely and page numbering stays contiguous.

type PageSpan

type PageSpan struct {
	Off  int `json:"off"`  // byte offset into the fragment's text
	Page int `json:"page"` // the page whose content begins there
}

PageSpan marks where a page's content begins inside a fragment's text.

A fragment spans pages whenever the assembler absorbs a continuation, or a sub-floor sibling, from the following page. Keeping only the start page made `fragments.page` right for the beginning of a fragment and wrong for the rest, so a hit inside one could not be resolved to the page it actually sits on.

func DecodePageSpans

func DecodePageSpans(s string) []PageSpan

DecodePageSpans parses what encodePageSpans wrote; nil on empty or malformed, so a caller degrades to the start page rather than failing.

type PageText

type PageText struct {
	Page   int
	Text   string
	Engine string
	// Err records why THIS page has no text, when it failed on its own.
	//
	// A page that fails is not a blank page, and the difference has to survive:
	// downstream, an empty Text with no Err is a legitimately blank sheet, and an
	// empty Text with an Err is a hole that somebody should come back to.
	Err string
}

PageText is one page's extracted text plus the engine that produced it, for the `ocr` MCP tool's paged output. Engine is "text" for a PDF text layer, pandoc, or a plain-text read; "tesseract"/"paddleocr"/"vision" for an OCR'd (scanned/image) page.

func EmailText

func EmailText(path string) ([]PageText, error)

EmailText renders a mail archive as paged text, one page per message.

Pure: it reads and returns, and writes nothing. That is load-bearing — the `ocr` MCP tool calls this to read a file the user named, and a read tool that materialises 69 files as a side effect is a tool nobody can trust. Attachment extraction is ExtractEmailAttachments, called separately by ingest.

func ExtractPaged

func ExtractPaged(ctx context.Context, path string, ocr *OCR) ([]PageText, error)

ExtractPaged extracts a document to paged text — the `ocr` MCP tool's core. It routes by kind: a PDF runs the text-layer/OCR hybrid, office/markup goes through pandoc (one page), an image runs the OCR cascade, and text is read directly. ocr may be nil when no page needs OCR (a born-digital PDF, office, or text); a scanned page with a nil ocr is a clear error.

func ParseTranscription

func ParseTranscription(md string) []PageText

ParseTranscription reads a raglit transcription sidecar back into pages.

The inverse of RenderTranscription, and the reason it exists is the upload case: a client sends a file whose text this index has never read, but whose sidecar is already beside it from a previous ingest elsewhere. Re-OCRing it to compare would need a vision model, which puts the whole check behind a network call and out of CI. The sidecar's `## Page N` headings are the documented contract, so parsing them is safe — and a heading whose number does not parse is skipped rather than guessed at, because a wrong page number in an alignment is worse than a missing one.

func SpreadsheetPages

func SpreadsheetPages(ctx context.Context, path string) ([]PageText, error)

SpreadsheetPages extracts a workbook to one PageText per sheet. .xlsx is read natively (no external tool exists for it — see xlsxExts); .xls goes through xls2csv.

type PendingItem

type PendingItem struct {
	ID         int64   `json:"id"`
	URL        string  `json:"url"`
	State      string  `json:"state"`       // pending | running
	ETASeconds float64 `json:"eta_seconds"` // estimate to completion; 0 = unknown
}

PendingItem is one queued (not-yet-done) job plus an ETA estimate.

type Pool

type Pool struct {
	// contains filtered or unexported fields
}

Pool is the shared processed-document cache (pool.sqlite + pool-pages/) under a daemon root.

func OpenPool

func OpenPool(root string) (*Pool, error)

OpenPool opens (creating if needed) the pool under root: <root>/pool.sqlite and <root>/pool-pages/.

func (*Pool) Close

func (p *Pool) Close() error

Close releases the pool database.

func (*Pool) FileDir

func (p *Pool) FileDir(fileHash string) string

FileDir is where a file's pooled page images live.

func (*Pool) GC

func (p *Pool) GC(pol GCPolicy) (int, error)

GC evicts pooled documents per policy, oldest-accessed first, and removes any pool-pages/<file> dir no surviving entry references. Returns the number of entries evicted.

func (*Pool) Get

func (p *Pool) Get(recipeHash, fileHash string) (PooledDoc, bool, error)

Get returns the cached processed document for (recipe, file), if present.

func (*Pool) PurgeFiles added in v0.4.1

func (p *Pool) PurgeFiles(fileHashes []string) (int, error)

PurgeFiles removes every recipe's cached work for these files, page images included. Returns the number of entries removed.

func (*Pool) Put

func (p *Pool) Put(recipeHash, fileHash string, doc PooledDoc) error

Put caches a processed document. Page images (doc.Pages[].Image = absolute source paths) are copied into pool-pages/<file>/ and rewritten to basenames.

func (*Pool) Stats

func (p *Pool) Stats() (PoolStats, error)

Stats reports the pool's size. Bytes counts both the cached payloads (fragments + vectors) and the pool-pages images on disk.

type PoolStats

type PoolStats struct {
	Entries int   `json:"entries"` // cached (recipe, file) documents
	Files   int   `json:"files"`   // distinct source files (pool-pages dirs)
	Bytes   int64 `json:"bytes"`   // total cached-payload bytes (the GC budget basis)
}

PoolStats is a snapshot of the pool for reporting.

type PooledDoc

type PooledDoc struct {
	Title      string           `json:"title"`
	FragMode   string           `json:"frag_mode,omitempty"`
	FragRecipe string           `json:"frag_recipe,omitempty"`
	Fragments  []PooledFragment `json:"fragments"`
	Pages      []PooledPage     `json:"pages"`
	// Identity is the document's caption/summary/kind (identity.go), carried for
	// the same reason PageSpans is: reuse replays a document without OCR or the
	// model, so anything a model established has to survive here or it is lost on
	// every reuse — and the identity call is a model call, which is the cost the
	// pool exists to avoid paying twice.
	//
	// The identity FRAGMENT is not in Fragments — it is rebuilt from this on
	// import, so a pooled document cannot end up with two of them, and an older
	// payload that predates this field simply has no caption.
	Identity *DocIdentity `json:"identity,omitempty"`
}

PooledDoc is a fully-processed document: the reusable output of one ingest. Media rows are NOT serialized — they are recomputed on reuse from the pooled fragments' [FIGURE: ...] markers + restored page images (extractMedia), which reproduces them exactly.

type PooledFragment

type PooledFragment struct {
	Page     int `json:"page"`
	Ord      int `json:"ord"`
	StartOff int `json:"start_off,omitempty"`
	EndOff   int `json:"end_off,omitempty"`
	// PageSpans is where each page begins inside Text for a fragment that spans
	// pages. Carried through the pool because this is the CHEAP reuse path: a
	// pooled document is replayed without OCR or segmentation, so anything the
	// fragmenter worked out has to survive here or it is lost on every reuse —
	// and a pooled doc is shared across projects, so losing it here loses it
	// everywhere. Omitted when the fragment lies on one page; absent in payloads
	// pooled before the field existed, which read back as nil.
	PageSpans []PageSpan `json:"page_spans,omitempty"`
	Text      string     `json:"text"`
	Vec       []float32  `json:"vec,omitempty"`
	// Origin marks text nobody wrote — "described", a model's account of an
	// image. Carried for the reason PageSpans is, and more urgently: reuse
	// replays a document without OCR, and the evidence for the mark (layout
	// markup) is gone by then, so a mark dropped here can never be recomputed.
	//
	// Without it a photograph pooled from one index arrived in the next as an
	// ORDINARY fragment: a model's "a red Chevrolet Malibu, licence plate
	// CEP0912" indexed with nothing to say a model made it up, quotable as the
	// record. Empty for transcription, which is nearly everything.
	Origin string `json:"origin,omitempty"`
	// Line span and symbol path (Fragment), carried for the reason PageSpans
	// is. Absent in payloads pooled before the fields existed: 0/0 and none.
	StartLine int    `json:"start_line,omitempty"`
	EndLine   int    `json:"end_line,omitempty"`
	Symbol    string `json:"symbol,omitempty"`
}

PooledFragment is one cached fragment: its text, source span, and (optional) vector.

type PooledPage

type PooledPage struct {
	Page   int    `json:"page"`
	Engine string `json:"engine"`
	Image  string `json:"image,omitempty"`
	// How much of this page a model DESCRIBED rather than transcribed. Same
	// reason as PooledFragment.Origin: measured from layout markup that reuse
	// never sees, so it travels or it is lost. Absent in payloads pooled before
	// the field existed, which read back as 0 — see recordIngestReading on why
	// that is treated as UNMEASURED and not as "nothing described".
	TextChars      int `json:"text_chars,omitempty"`
	DescribedChars int `json:"described_chars,omitempty"`
}

PooledPage is one cached page's provenance. Image is an absolute source path on export, rewritten to a pool-pages basename when stored.

type Problem

type Problem struct {
	Kind ProblemKind `json:"kind"`
	// Subject is the document path, or the job's target when no document exists.
	Subject string `json:"subject"`
	JobID   int64  `json:"job_id,omitempty"`
	// Stage is where it went wrong, when a stage recorded it.
	Stage string `json:"stage,omitempty"`
	// Detail is the recorded evidence, verbatim — the endpoint's own words where
	// there are any. Summarising it here is how "increase the physical batch
	// size" became "status 500" and cost a week.
	Detail string `json:"detail,omitempty"`
	// Fix is the command that addresses it, ready to run.
	Fix string `json:"fix,omitempty"`
}

Problem is one thing worth a person's attention.

type ProblemKind

type ProblemKind string

ProblemKind names a class of trouble, stable enough to filter on.

const (
	// ProblemNoFragments — indexed and unsearchable. The worst kind: it looks
	// like a document from every angle except the one that matters.
	ProblemNoFragments ProblemKind = "no-fragments"
	// ProblemNoPages — a paged document whose pages were never recorded, so page
	// images, the transcript and every page-level citation have nothing behind
	// them.
	ProblemNoPages ProblemKind = "no-pages"
	// ProblemJobFailed — an ingest job that errored. Carries the stage it died in,
	// because "failed" without a stage sends the reader to the wrong half.
	ProblemJobFailed ProblemKind = "job-failed"
	// ProblemDegraded — the model would not segment a page and it was stored
	// whole. Not an error: the text is there and searchable, but it is one
	// undivided block where retrieval expects fragments.
	ProblemDegraded ProblemKind = "segment-degraded"
	// ProblemRetries — a job that only completed after the endpoint made it fight
	// for it. Not a failure, and the earliest warning that one is coming.
	ProblemRetries ProblemKind = "llm-retries"
	// ProblemGenerated — a document that is raglit's OWN output: a transcription
	// or a region record written beside a real document and then indexed as if it
	// were one. Never a judgement call, always a mistake, and self-feeding —
	// indexing a transcription writes a transcription of it.
	ProblemGenerated ProblemKind = "generated-indexed"
	// ProblemWithdrawn — ruled out of the corpus on purpose. Reported so a
	// document that is absent BY DECISION is never mistaken for one that is
	// absent by accident, which is the confusion withdrawal exists to end.
	ProblemWithdrawn ProblemKind = "withdrawn"
	// ProblemMissingFile — an indexed document whose file is no longer at the
	// path the row records.
	//
	// The quietest failure in this list, because search keeps working: the
	// fragments live in SQLite and answer queries exactly as before. What breaks
	// is everything that goes back to the FILE — download the original, re-read
	// it, show a page image, write a reading so it can be reviewed — and each of
	// those fails on its own, far from anything that would name the cause.
	//
	// Found on the corpus this was written against, and only because a review
	// sweep tried to open 355 documents and 43 were not there. Nothing else had
	// noticed. Most were a deliberate archival move (documents/ → legacy/) and
	// several were content-preserving renames whose new path was never ingested,
	// so four court filings existed on disk under better names and were absent
	// from the corpus entirely.
	//
	// Usually RECOVERABLE, which is why the fix is not simply "forget": the bytes
	// are generally still in the tree. 39 of those 43 were found on disk at
	// another path. A row is a pointer, and a moved file breaks the pointer, not
	// the document.
	ProblemMissingFile ProblemKind = "missing-file"
	// ProblemUnreadPage — a page the OCR could not read, in a document that was
	// otherwise indexed. Created deliberately by the ingest salvage: one failed
	// page used to discard the whole document, and now it is recorded as a hole
	// so the rest can be kept. That trade is only sound if the hole is FINDABLE —
	// a partial document nobody knows is partial is the more dangerous object,
	// because search returns it and a reader assumes the absence is the record's.
	ProblemUnreadPage ProblemKind = "page-unread"
	// ProblemPartialPage — a page the model read only as far as a structural
	// loop: every row with content is present, the blank remainder and anything
	// BELOW it was never seen (see enginePartialVision).
	//
	// Separate from page-unread because the two need different judgement. An
	// unread page is a hole a reader can see. A salvaged page reads like a whole
	// page — it has a heading, rows, a shape — and the part that is missing left
	// no gap in the text to notice. On the two Record of Ownership sheets what
	// went missing was the "Adjoining Property" heading under the blank rows,
	// which is precisely the kind of absence a reader would take for the
	// record's own silence.
	//
	// Salvage is still the right trade: the alternative was dropping the page
	// entire, and it dropped a 1972 chain-of-title entry with it. The trade is
	// only sound while the partial is FINDABLE, which is what this row is for.
	ProblemPartialPage ProblemKind = "page-partial"
	// ProblemEmptySource — the file is there and it is zero bytes.
	//
	// Reported rather than failed, because having no content is a fact about the
	// DOCUMENT and being unable to import it is a fact about the importer; saying
	// the second when the first is true sends the reader to the wrong half. A
	// 0-byte .docx failed as `pandoc .docx: exit status 63`, which names the tool
	// and reads as a broken install.
	//
	// It must not be silent either. The text path did not fail at all on an empty
	// file — it indexed a document with no fragments, which is ProblemNoFragments:
	// a row that looks like a document from every angle except the one that
	// matters. Usually a broken copy or a truncated export, and the document it
	// should have carried is absent from the record.
	ProblemEmptySource ProblemKind = "empty-source"
)

type ProjectMeta

type ProjectMeta struct {
	ID        string `json:"id"`
	Label     string `json:"label,omitempty"`
	CreatedAt int64  `json:"created_at"`
}

ProjectMeta marks an index as a project's base: <home>/project.json.

type ProjectSavings

type ProjectSavings struct {
	ProjectID string         `json:"project_id"`
	Base      string         `json:"base"`
	Layers    []LayerSavings `json:"layers"`
	// Counted over the layers the caller can read. A document held by N
	// readable indexes counts N-1 times: the copies beyond the first.
	DuplicateDocuments  int     `json:"duplicate_documents"`
	DuplicatePassages   int     `json:"duplicate_passages"`
	DuplicateEmbeddings int     `json:"duplicate_embeddings"`
	DuplicatePageReads  int     `json:"duplicate_page_reads"`
	DuplicateBytes      int64   `json:"duplicate_bytes"`
	ModelSeconds        float64 `json:"model_seconds"` // model time spent importing the copies
	// Indexes one layer search reads now (itself, the base, its readable
	// siblings) and with a maintained base (itself and the base).
	FanoutNow      int `json:"fanout_now"`
	FanoutWithBase int `json:"fanout_with_base"`
	// Recommend is true when a maintained base would pay: two or more layers,
	// copies between them, and a base that holds nothing of its own.
	Recommend bool   `json:"recommend"`
	Reason    string `json:"reason"`
	Note      string `json:"note"`
}

ProjectSavings is the report for one project id.

type PromptKind

type PromptKind string

PromptKind is the QUESTION being asked. Four, not one with options: a crop is asked to transcribe, a root to account for a sheet, a parent to decide geometry, and a plain page to transcribe with no JSON at all.

const (
	// PromptPlain is the whole-page read on the ingest path: transcription only,
	// no JSON, no proposals. It is the pass that scores 14/14 on stamped
	// recording numbers and it is deliberately the simplest thing here.
	PromptPlain PromptKind = "plain"
	// PromptRoot accounts for a whole sheet and says where to look closely. Its
	// description is the PRIMARY product — an area it fails to name is an area
	// no later pass will crop, so the account decides coverage.
	PromptRoot PromptKind = "root"
	// PromptCrop reads one region. Transcription is the primary product.
	PromptCrop PromptKind = "crop"
	// PromptEscalate asks a PARENT what to do about a child that cannot read
	// itself. It REPLACES the question rather than modifying it, and forbids
	// transcription outright: the parent is looking at the coarse view that the
	// close-up exists because it cannot resolve.
	PromptEscalate PromptKind = "escalate"
)

func (PromptKind) AcceptsMods

func (k PromptKind) AcceptsMods() bool

AcceptsMods reports whether modifiers mean anything for a kind.

Plain takes none because it is not looking at a crop — a grid rule or a damage report about "this image" has no referent on a whole page. Escalate takes none because it forbids transcription, and every modifier here is about how to transcribe; appending one contradicts the question.

type Proposal

type Proposal struct {
	Kind MarkKind `json:"kind"`
	// Why is the evidence in one line, for a person deciding. Written to be read
	// aloud in a review, not parsed.
	Why string `json:"why"`
	// Confident is whether this can be acted on without reading the documents.
	// Only exact-byte and near-exact alignment reach it.
	Confident bool `json:"confident"`
}

Proposal is what raglit thinks a pair is, and why. It is never written to relations.jsonl: a proposal is an argument put in front of a person, and the file holds rulings.

func Propose

func Propose(m DocMatch) Proposal

Propose reads a computed match and says what the pair probably is.

The rule, and why it is this rule.

Coverage decides whether the two are the same instrument at all; it cannot decide copy versus version, because a re-recorded deed and a scan of a deed both align at 0.97. What decides that is the SHAPE of the disagreement, and there are two kinds of disagreement in this corpus that look alike in a score and mean opposite things:

OCR NOISE      diffuse, alphabetic, one-sided. The 2008 lot certification
               reads "LAURENCE MOONION" for Clarence Brannock. Nothing was
               refiled; a scan was read badly. This is a COPY.
A SUBSTITUTION localized, numeric, and PAIRED — a number present only on one
               side sits against a different number present only on the other.
               An auditor file number, a distance, a date. Something was
               changed and filed. This is a VERSION.

So the test for a version is paired numeric disagreement: tokens exclusive to each side, on BOTH sides. A number dropped from one side only is an omission, which OCR produces constantly, and does not qualify.

This is a heuristic and it is allowed to be, because nothing acts on it — it proposes, a person rules, and the ruling is what gets stored. Its job is to put the right pairs and the right evidence in front of that person, not to be right on its own. Where it cannot tell, it says so rather than guessing.

type Provider

type Provider struct {
	// Kind is the preset it was made from: "custom", "openrouter", "litellm".
	// Informational; the base URL is what is called.
	Kind    string `json:"kind,omitempty"`
	BaseURL string `json:"base_url"`
	// APIKey is sent as a bearer token. Empty → no Authorization header.
	APIKey string `json:"api_key,omitempty"`
	// APIKeyEnv and APIKeyFile say where the key is instead of holding it: an
	// environment variable of the process that makes the call, or a file whose
	// contents, trimmed, are the key. A tool that already holds the user's key
	// (dun) points at it rather than copying it into this file. APIKey wins
	// when set. Read at each call, so a rotated key needs no restart.
	//
	// A daemon under systemd or launchd does not inherit a shell's variables,
	// so api_key_env reaches it only through its unit; api_key_file reaches it
	// always.
	APIKeyEnv  string `json:"api_key_env,omitempty"`
	APIKeyFile string `json:"api_key_file,omitempty"`
}

Provider is one OpenAI-compatible endpoint.

func (Provider) Key

func (p Provider) Key() string

Key is the bearer token: APIKey, else $APIKeyEnv, else APIKeyFile's contents ("~/" is the home directory). "" when none of them gives one.

type PruneDir

type PruneDir struct {
	Dir       string `json:"dir"`
	Documents int    `json:"documents"`
}

PruneDir is one directory's share of a prune.

type PruneResult

type PruneResult struct {
	// Documents whose local file was not found.
	Documents int        `json:"documents"`
	Dirs      []PruneDir `json:"dirs"`
	// Removed is false on a dry run.
	Removed bool `json:"removed"`
	// Page and figure images under pages/ that no document names. Removed with
	// the documents.
	OrphanPages     int   `json:"orphan_pages"`
	OrphanPageBytes int64 `json:"orphan_page_bytes"`
	// Copies under originals/ (Home.OriginalsDir), except an upload's.
	DanglingOriginals     int   `json:"dangling_originals"`
	DanglingOriginalBytes int64 `json:"dangling_original_bytes"`
	// OriginalsRemoved is true when deleteOriginals removed them.
	OriginalsRemoved bool `json:"originals_removed"`
}

PruneResult is what Prune removes, or would remove.

type PurgeResult

type PurgeResult struct {
	Pattern   string   `json:"pattern"`
	Documents int      `json:"documents"`
	Sample    []string `json:"sample"`
	// Pending jobs on matching paths, dropped with the documents.
	IngestJobs  int `json:"ingest_jobs"`
	CaptionJobs int `json:"caption_jobs"`
}

PurgeResult is what a purge by pattern removes, or would remove.

type Reading

type Reading struct {
	SourceSHA256 string `json:"source_sha256,omitempty"`
	SourcePath   string `json:"source_path,omitempty"`
	DocPath      string `json:"doc_path"`
	Method       string `json:"method,omitempty"`
	Level        string `json:"level"`
	ProducedBy   string `json:"produced_by,omitempty"`
	RuledBy      string `json:"ruled_by,omitempty"`
	At           int64  `json:"at,omitempty"`
	// Text is the reading itself — what gets indexed when this reading governs.
	Text string `json:"text,omitempty"`
	// Data is the structured form the text was rendered from, as an attest.Reading:
	// units, each with a locator into the asset.
	//
	// ONE shape for every kind, which is attest's whole design — a Unit is "one
	// machine claim about one piece of the asset", and the locator differs by
	// kind rather than the model differing. A diarized recording's units are
	// turns located by TIME; a scanned sheet's are page or region reads located
	// by AREA; a mail archive's are messages located by SPAN. So an audio
	// transcript and a page transcription are attestable in the same way, by the
	// same log, and a correction to either is an attestation rather than an edit
	// to what the machine said.
	//
	// Empty is normal — a plain text extraction has no structure worth keeping
	// beyond its text.
	Data string `json:"data,omitempty"`
	// Ruled is what a person has been through, per facet, as JSON. See trust.go.
	Ruled string `json:"ruled,omitempty"`
	// Describes marks a reading that says what its source DEPICTS rather than
	// what it says — a model's account of a photograph.
	Describes bool `json:"describes,omitempty"`
	// DescribedPct is how much of this reading the model DESCRIBED rather than
	// transcribed, 0-100, or DescribedUnmeasured. See IsMixedPage: a screenshot is
	// both, and a flag cannot say so.
	DescribedPct int `json:"described_pct,omitempty"`
}

Reading is one account of one source.

func AuthoritativeReading

func AuthoritativeReading(rs []Reading, ruled func(source string) (string, bool)) (Reading, bool)

AuthoritativeReading picks the reading of a source that governs.

A person's ruling first, always. Levels only order what nobody has ruled on: attested over adapted over machine, and the NEWEST wins a tie, because a second reading at the same level is a re-read and a re-read supersedes.

The ruling is looked up through `ruled`, which the caller supplies from the judgement trail — this package must not open it, because the trail lives beside the documents and the index does not know where that is.

func (Reading) LowestTrust

func (r Reading) LowestTrust() Confidence

LowestTrust is the weakest thing this reading claims, which is what a reader deciding whether to quote it needs. 0 when it claims nothing.

func (Reading) Trust

func (r Reading) Trust() map[string]Confidence

Trust returns this reading's confidence per facet: the method's defaults, with anything a person has ruled on layered over them.

func (Reading) TrustSummary

func (r Reading) TrustSummary() []string

TrustSummary renders the facets in a stable order, weakest first, for a reader and for a log line.

func (Reading) WithRuling

func (r Reading) WithRuling(facet string, c Confidence) Reading

WithRuling returns the reading with a facet marked as ruled on by a person.

Per facet, because that is how ruling actually happens: somebody sits down and checks the speaker attribution of a hearing, or corrects the wording of a page, and almost never both at once. A transcript whose attribution was ruled on and whose words were not is the common case, not an edge one.

type ReadingAuthority

type ReadingAuthority struct {
	// Source is the sha256 of the asset both readings read. The ruling is about
	// a SOURCE, not a pair, so a third reading arriving later does not need a
	// second ruling to be ordered against the first two.
	Source string `json:"source"`
	// Doc is the document holding the reading that governs.
	Doc string `json:"doc"`
	// Why is the grounds. A ruling without them is a preference nobody can check
	// later, which is the thing every other ruling here refuses to be.
	Why string `json:"why,omitempty"`
	By  string `json:"by,omitempty"`
	At  string `json:"at,omitempty"`
}

ReadingAuthority is a person's ruling that one reading of a source governs.

The corpus holds several accounts of one thing — a machine transcript and a verified one, a whole-page read and a region descent — and which of them may be quoted is not something the corpus can compute. Levels give a default order (attested over adapted over machine); this overrides it, because a person may know that the older attested reading is the one to cite, or that a machine reading of a re-scan beats a verified reading of a bad scan.

type Rect

type Rect struct {
	X float64 `json:"x"`
	Y float64 `json:"y"`
	W float64 `json:"w"`
	H float64 `json:"h"`
}

Rect is a region of a page in normalized coordinates — 0..1 of the page's width and height. Normalized rather than pixels so a region survives being re-rendered at a different dpi, which is exactly what a transform does.

Lowercase in JSON, matching RegionProposal: the model answers in x/y/w/h and the record has no business spelling the same four numbers differently.

func LayoutClusters

func LayoutClusters(img image.Image, o LayoutOpts) []Rect

LayoutClusters returns the content clusters of an image, as fractions of it.

Empty when there is nothing to find, which callers must treat as "fall back to the geometric grid" rather than as "the page is blank".

func LayoutClustersPNG

func LayoutClustersPNG(data []byte, o LayoutOpts) []Rect

LayoutClustersPNG segments encoded bytes — the form a region's crop has.

type Reference

type Reference struct {
	From    string `json:"from"`
	Page    int    `json:"page,omitempty"`
	To      string `json:"to"`
	Excerpt string `json:"excerpt"`
}

Reference is one fragment citing another document.

type Region

type Region struct {
	// ID addresses this region within its document: "p1" for a page's root,
	// "p1.0" and "p1.0.2" for what descent found under it.
	//
	// Path-shaped rather than a digest because a diff of two reads has to stay
	// readable, and because the path IS the ancestry — a hit can be reported as
	// sheet → drawing interior → lot C corner without a lookup. It addresses a
	// position in a RECORDED tree, not a piece of paper: a second read of the
	// same sheet proposes different regions and renumbers. SHA256 is what says
	// whether an id still points at the pixels the text came from.
	ID       string `json:"id,omitempty"`
	Page     int    `json:"page"`
	BBox     Rect   `json:"bbox"` // page coordinates
	Rotation int    `json:"rotation"`
	Kind     string `json:"kind,omitempty"` // overview|text-block|table|drawing|legend
	// Verdict is this region's own claim about its frame — empty when it simply
	// read. Kept on the record because "this could not be read as framed" is a
	// fact about the READ, and a later reader deserves to know a region said so.
	Verdict string `json:"verdict,omitempty"`
	// Because is one line explaining a verdict or the parent's answer to it.
	Because string `json:"because,omitempty"`
	// Grid names this region's cell when it came from tileRegion rather than from
	// something the model named — "row 2 of 4, column 3 of 4". Recorded because a
	// cell's reading has to be read knowing it was one: a cell deliberately does
	// NOT transcribe an item its neighbour holds more of.
	Grid string `json:"grid,omitempty"`
	// Filter is the repair applied to the crop BEFORE it was read, and it is part
	// of the render: the digest covers the filtered bytes, so re-rendering
	// without it produces a different image and VerifyRegionRender says so.
	Filter RegionFilter `json:"filter,omitempty"`
	Text   string       `json:"text,omitempty"`
	Flags  []string     `json:"flags,omitempty"`
	Depth  int          `json:"depth"`
	// DPI is what the PAGE was rasterized at before this region was cropped out
	// of it. Recorded per region rather than per document because a re-render
	// that gets the crop right and the resolution wrong reproduces the geometry
	// and not the detail, and detail is the entire subject here.
	DPI int `json:"dpi,omitempty"`
	// SHA256 digests the PNG this region was cropped to before it was read.
	//
	// The descent already computes it to detect a cycle; keeping it rather than
	// discarding it is what makes attestation possible. A re-render that hashes
	// to this is the image that produced Text. One that does not is a different
	// artifact, and saying so is more useful than quietly showing it anyway.
	//
	// BEFORE the read, not after: it has to be, because it is also the cycle
	// detector and the cycle has to be caught before the call is paid for. Where
	// a context downscale intervened, Downscales says so and
	// RerenderRegionAsSeen replays it.
	SHA256 string `json:"sha256,omitempty"`
	// Downscales is how many times the rendered crop had to be shrunk to fit the
	// model's context before it could be read.
	//
	// The one part of what the model saw that the crop geometry does not capture:
	// an overflowing page is re-rendered smaller MID-CALL (see maxContextShrinks),
	// so a re-render that skips the shrink is sharper than the image the text
	// came from. Recorded rather than prevented, because refusing to shrink would
	// fail the page instead of reading it.
	Downscales int `json:"downscales,omitempty"`
	// TokensPerSqIn is what this region was actually seen at. Recorded because
	// it is the one quality number available BEFORE any model call, and on its
	// own it condemns a transcription taken at four.
	TokensPerSqIn float64 `json:"tokens_per_sq_in"`
	// Clusters are this region's measured label clusters, as fractions of it.
	Clusters []Rect `json:"clusters,omitempty"`
	// Computed marks a box derived here (grid tile or cluster), not proposed by
	// a model. Such a box does not escalate.
	Computed bool      `json:"computed,omitempty"`
	Children []*Region `json:"children,omitempty"`
}

Region is one node of the read: a piece of a page, at a rotation and scale, with what the model said about it.

Everything needed to RE-RENDER the region is on the node, because a human asked to attest that a document says what a fact quotes must be shown the image the words came from. When the text was read off a rotated, zoomed crop, handing the human the whole page hands them a different artifact — one where the passage is at the resolution that produced the failure in the first place.

func (*Region) AddFlag

func (r *Region) AddFlag(f string)

AddFlag records a condition on this region, once. Exported because a pass that MEASURES a recorded read — rather than producing one — belongs outside this package; see `raglit regions --backfill-damage`.

func (*Region) FindByID

func (r *Region) FindByID(id string) *Region

FindByID resolves a region id within this tree, or nil.

func (*Region) Flatten

func (r *Region) Flatten() []*Region

Flatten walks the tree depth-first, parents before children.

func (*Region) Leaves

func (r *Region) Leaves() []*Region

Leaves are the regions with no children — what gets embedded and searched. Interior nodes stay searchable through their own overview text, which is why a missed region costs detail rather than coverage.

func (*Region) String

func (r *Region) String() string

String renders the tree for a human, one line per region.

type RegionAbout

type RegionAbout struct {
	Depth int
	// Damage is what DamageOf measured on these pixels — see damageSuffix. The
	// model is shown a downsample and cannot tell a blurred crop from a small one.
	Damage []string
	// Grid is set when this region is one CELL of a computed grid rather than a
	// block somebody named: "row 2 of 4, column 3 of 4". A cell has neighbours it
	// cannot see and overlaps them, which changes what it should transcribe — see
	// gridSuffix.
	Grid string
	// Escalation is set only on TURN 3: a parent being asked what to do about a
	// child that reported its frame was broken. Present means "answer with an
	// action, not a transcription" — see escalationSuffix.
	Escalation string

	// Doc, Page, RegionID and TokensPerSqIn identify WHICH crop of WHICH page of
	// WHICH document this call is reading.
	//
	// They exist for the trace, and they are what makes a descent observable at
	// all. Without them every call in a walk looks identical in the record —
	// same model, same document, forty images — so a tree that spent its budget
	// in the margins is indistinguishable from one that read the title block,
	// and neither can be joined back to the region it came from. The region id
	// is the join key to the recorded tree; tokens/in² is the number that says
	// whether the crop could have been read at all.
	Doc           string
	Page          int
	RegionID      string
	TokensPerSqIn float64

	// FitsWhole means this region needs no subdivision: every pixel reaches the
	// model in ONE canvas at this resolution. The root is then asked to
	// TRANSCRIBE rather than to account for the sheet and propose where to look.
	//
	// Salience is only worth paying for when something will act on it. Measured
	// on the bench's 94 sq in record of survey: the root prompt returned 4 of 7
	// checks where a plain read of the same pixels returned 7 of 7 — it described
	// the sheet and proposed regions nobody descended into, because there was
	// nothing to descend into. The three it lost were recording numbers.
	FitsWhole bool
}

RegionAbout is what a region is told about itself before it is read.

type RegionDoc

type RegionDoc struct {
	Doc     string       `json:"doc"` // base name, for a human opening the file
	Version int          `json:"version"`
	Pages   []RegionPage `json:"pages"`
}

RegionDoc is a document's region reads — one tree per page that was read this way. Not every page is: the descent is for a sheet whose text is physically too small to survive one look, and a letter page does not need it.

func ReadRegionDoc

func ReadRegionDoc(docPath string) (*RegionDoc, bool, error)

ReadRegionDoc loads the sidecar beside a document. A missing sidecar is not an error — most documents are not read this way — and reports false.

func (*RegionDoc) Find

func (d *RegionDoc) Find(id string) (RegionPage, *Region)

Find resolves a region id anywhere in the document. The id carries its page ("p3.1"), but a caller holding an id from a citation should not have to parse it to use it.

func (*RegionDoc) PageRead

func (d *RegionDoc) PageRead(n int) (RegionPage, bool)

PageRead returns the read for a page number, if this document has one.

type RegionFilter

type RegionFilter string

RegionFilter is a repair applied to a region's crop before it is read. A closed set on purpose: every one of these was measured on a page, and the measurements do not transfer to a filter nobody ran.

const (
	// FilterNone is the crop as rendered.
	FilterNone RegionFilter = ""
	// FilterContrast is CLAHE at the setting that was measured to help — clip 2.0
	// over an 8x8 tile grid. The STRONGER setting (clip 4.0, 16x16) recovered
	// nothing on the same page, so the parameters are not a detail.
	FilterContrast RegionFilter = "contrast"
	// FilterSharpen is a mild unsharp mask, sigma 1.0 amount 0.4. Also a measured
	// setting: at sigma 2.0 amount 0.8 the same page went back to its unfiltered
	// score.
	FilterSharpen RegionFilter = "sharpen"
)

type RegionPage

type RegionPage struct {
	Page int `json:"page"`
	// WidthIn, HeightIn are the sheet's physical size — the input that decides
	// everything here, because the encoder's budget is per image and square
	// inches are what resolution is spent on.
	WidthIn  float64 `json:"width_in,omitempty"`
	HeightIn float64 `json:"height_in,omitempty"`
	// DPI, PxW, PxH describe the page rasterization every region was cropped out
	// of. Recorded together on purpose: dpi alone says what was ASKED for, and
	// the pixel dimensions say what was produced. A renderer that disagrees about
	// the second while agreeing about the first is caught here rather than
	// discovered as an unexplained digest mismatch.
	DPI  int     `json:"dpi"`
	PxW  int     `json:"px_w,omitempty"`
	PxH  int     `json:"px_h,omitempty"`
	Root *Region `json:"root"`
	// Text is this page's transcription, assembled from the tree. Byte for byte
	// what belongs under `## Page N` in the transcription sidecar.
	Text string `json:"text"`
	// Spans map byte ranges of Text to the region that produced them.
	//
	// Out here rather than inline in the markdown, and that is the load-bearing
	// choice: `## Page N` is a contract two separate consumers already parse, and
	// a marker interleaved with the text would land INSIDE the quotations they
	// match against. The transcription stays exactly what it was; the attribution
	// lives in a file nothing else reads.
	Spans []RegionSpan `json:"spans,omitempty"`
}

RegionPage is one page's read: the tree, the transcription assembled from it, and which region each part of that transcription came from.

func (RegionPage) RegionAt

func (p RegionPage) RegionAt(off int) *Region

RegionAt reports the region a byte offset in the page's transcription came from, or nil if the offset falls between spans.

The offset is into RegionPage.Text. A consumer that located a quotation in the markdown sidecar has an offset into the FILE; subtract the start of the page's section — the text under `## Page N` is this text verbatim — or use RegionsForQuote, which does not need offsets at all.

func (RegionPage) RegionsForQuote

func (p RegionPage) RegionsForQuote(quote string) []*Region

RegionsForQuote reports the regions whose text contains a quotation, deepest first.

Deepest first because a deeper region was read at a narrower field of view, which is the higher-resolution reading of the same words and therefore the one to show. A quotation matching only an interior node is a quotation nothing looked at closely — worth knowing, and the flags on that node will usually say why.

Matched on folded text — letters and digits, lowercased — because the words went through a vision model and hyphenation, spacing and punctuation are exactly what it renders inconsistently. A consumer that located the quote in the transcription folded it the same way to find it there.

type RegionProposal

type RegionProposal struct {
	X        float64 `json:"x"`
	Y        float64 `json:"y"`
	W        float64 `json:"w"`
	H        float64 `json:"h"`
	Rotation int     `json:"rotation"`
	Reason   string  `json:"reason"`
	Kind     string  `json:"kind"`
	// Filter is a repair the model is asking for on the SAME pixels. Named by the
	// model rather than applied by measurement alone: the flags say what is wrong,
	// and the model is the one looking at the region when it decides what to do
	// about it. An unknown name is refused in route rather than ignored here.
	Filter RegionFilter `json:"filter"`
	// Margin is INCHES of extra paper the region wants on every side — a region
	// REFINING its own frame rather than escalating.
	//
	// Escalate when the transform is fundamentally broken; refine when the fix
	// needs no knowledge of the larger document. A word truncated at the region's
	// own edge is the second kind: the region can see the cut, "half an inch more"
	// is a complete statement of the remedy, and asking a parent that cannot see
	// the region's six-point text to adjudicate it buys nothing.
	//
	// Inches, not a fraction, for the reason descentPadIn is a length: half an
	// inch has to mean the same thing at every depth, and a fraction of a small
	// region vanishes exactly where cuts are worst.
	Margin float64 `json:"margin"`
}

RegionProposal is one sub-region the model thinks is worth a closer look.

type RegionReader

type RegionReader struct {
	// Ask returns the model's reading of one rendered region. Separated from the
	// walk so the descent rules — budgets, routing, cycle detection — are
	// testable without a model, which is most of what can go wrong here.
	// About carries what this region knows about ITSELF and cannot see: what its
	// pixels measured as, and whether it is one cell of a grid. Bundled rather
	// than passed as more parameters, because everything here exists for the same
	// reason — the model is looking at a crop and some of what governs its answer
	// is not in the crop.
	Ask func(ctx context.Context, img PageImage, about RegionAbout) (RegionReading, error)

	// PageWIn, PageHIn are the sheet's physical size. The whole reason this
	// exists is that a bigger sheet buys no extra tokens, so the physical size
	// is the input that decides everything.
	PageWIn, PageHIn float64
	DPI              int

	// MaxDepth bounds descent. 0 means READ THE SHEET AND STOP, which is the
	// default: the root reading is written to account for the whole sheet, so
	// descending is for when a specific area needs the characters, not something
	// every page should pay for. A survey descends into a title block because
	// somebody asked about the certificate number, not because it was scanned.
	MaxDepth      int
	MaxChildren   int // per node, 0 → 8
	MaxTransforms int // per region, 0 → 2
	MaxCalls      int // whole document, 0 → 40
	// MaxEscalations bounds turn 3 for the whole document (0 → 4).
	//
	// Separate from MaxCalls because an escalation is the most expensive call in
	// the walk — it is asked at the PARENT's resolution, which on an E-size sheet
	// is four tokens per square inch, the worst per-token value anywhere in the
	// descent. It should be rare; if it is not, the region proposals are bad and
	// tiling is the cheaper answer.
	MaxEscalations int
	MinRegionIn    float64

	// Hint is what the caller is looking for, threaded into every prompt.
	//
	// The model proposing regions is looking at a view where the thing you want
	// may be physically unresolvable — 6pt bearing labels on a sheet seen at 4
	// tokens per square inch — so it cannot propose what it cannot see. A hint
	// supplies the salience the view cannot: "read every bearing, distance and
	// monument call on the drawing" costs nothing and beats any amount of extra
	// budget, because budget alone only re-asks the same blind view.
	Hint string

	// Segment cuts along measured ink clusters instead of the geometric grid.
	// OFF by default: measured worse on this corpus — see withTiles.
	Segment bool

	// Layout tunes cluster finding when Segment is set. Zero value derives its
	// distances from DPI.
	Layout LayoutOpts

	// Tile turns on geometric subdivision of large low-resolution DRAWINGS
	// instead of asking where to look. See tileRegion.
	Tile bool

	// Doc is the document these regions belong to, carried only so the trace can
	// say which one. A walk reads one document; the record may hold many.
	Doc string

	// Trace, when non-nil, receives one line per TURN and per decision taken
	// about it: what was read, at what resolution, what it proposed, what was
	// routed to a descent and what to a transform, what escalated and what the
	// parent said. A tree printed at the end says where the walk ARRIVED; this
	// says what it did on the way, which is the part that explains a bad tree.
	Trace io.Writer
	// contains filtered or unexported fields
}

RegionReader walks a page as a tree of regions.

The model is asked, at every node, for two things: what this whole region IS, and which sub-regions merit a closer look. Descent recurses into the second while keeping the first, which is what makes a bad region set cost DETAIL rather than coverage.

func (*RegionReader) Read

func (rr *RegionReader) Read(ctx context.Context, page image.Image, pageNo int) (*Region, error)

Read walks the page and returns the root region.

func (*RegionReader) ReadInto

func (rr *RegionReader) ReadInto(ctx context.Context, page image.Image, pageNo int, docPath, regionID string) (*Region, error)

ReadInto re-reads ONE recorded region and grafts the result back into the tree.

The alternative on offer was raising --depth, which re-runs the whole page and re-derives the same split — so a tree that spent its budget on the margins could never be sent back into the drawing. Everything needed was already recorded: the bbox, the rotation and the dpi identify the crop exactly, which is the property `raglit region` uses to put the same pixels back on screen.

The subtree is REPLACED rather than merged. A re-read is a new reading, and keeping the old children beside the new ones would leave two accounts of the same ground with nothing to say which was current.

type RegionReading

type RegionReading struct {
	// Transcription is the characters on the page, verbatim, markdown for
	// STRUCTURE only. It feeds the full-text index, where someone types an exact
	// recording number and expects this document back.
	//
	// Split from Description on 2026-08-06 because one field could not be both.
	// Its doc comment used to admit as much — "a transcription for a text block,
	// a summary for a drawing" — and every tile of a survey came back
	// kind:drawing, so every tile returned a summary. Six tiles at 43-46
	// tokens/in², above the readable baseline, and not one transcribed a
	// character.
	Transcription string `json:"transcription_markdown"`
	// Description is what this IS, for the index that answers a searcher who
	// does not know the words on the page.
	Description string `json:"description"`
	Kind        string `json:"kind"`
	// Regions are proposals in THIS region's coordinates, 0..1.
	Regions []RegionProposal `json:"regions"`
	// Repeated is set when the generation looped. On a page that fails the
	// document; on a region it is a reason to descend.
	Repeated bool `json:"-"`
	// Downscales is how many times the reader shrank the image before the model
	// could read it. Reported back so the region can record what was actually
	// looked at rather than what was handed over.
	Downscales int `json:"-"`
	// Verdict is a region's claim about its OWN frame, when it has one. Empty
	// means "read" — nothing to report.
	//
	// The line it draws: escalate when the transform is fundamentally broken,
	// refine when the fix needs no knowledge of the larger document. A refinement
	// is expressed as a proposal and handled here; an escalation is the one thing
	// a region cannot do for itself, because the box and the rotation came from
	// its parent.
	Verdict string `json:"verdict"`
	// Action is a PARENT's answer when it is asked what to do about a child that
	// escalated. See escalationPrompt.
	Action string `json:"action"`
	// Because is one line for a human reading the record, on either.
	Because string `json:"because"`
}

RegionReading is what the model returns for one region.

func ParseRegionReading

func ParseRegionReading(s string) RegionReading

ParseRegionReading pulls the JSON object out of a model reply.

Tolerant of the wrapping models add — prose before, a fenced block, a trailing apology. A reply that carries no object at all is still usable as a description: losing the transcription because the JSON was malformed would throw away the part that matters.

type RegionSpan

type RegionSpan struct {
	Region string `json:"region"`
	Start  int    `json:"start"`
	End    int    `json:"end"`
}

RegionSpan attributes a byte range of a page's transcription to a region.

func RegionTranscript

func RegionTranscript(root *Region) (string, []RegionSpan)

RegionTranscript assembles a page's transcription from its region tree, and says which region every part of it came from.

Parents before children in reading order — Flatten's order, which is also the order the tree prints in. EVERY node's text is emitted, not only the leaves: an interior node describes the whole of what is under it, and that is the design's coverage guarantee — a bad region set is supposed to cost detail rather than coverage. Dropping the overviews here would put the loss straight back.

It does duplicate: an overview and the leaves that refine it describe the same block twice. That is the honest artifact, and it is also why the spans matter. An overview read at four tokens per square inch is precisely where invented text lives, and a reader who can see WHICH region a sentence came from can see that it came from a low-resolution one — and ask for that crop.

Whether leaves should replace fragments outright is still open; see docs/design/hierarchical-regions.md. This assembles a transcription and decides nothing about fragmentation.

type Registration

type Registration struct {
	ProjectID string `json:"project_id"`
	// Namespace prefixes every name below; it is the client's project setting
	// for the daemon MCP.
	Namespace string `json:"namespace"`
	Base      string `json:"base"`
	// Private and Public are the caller's two layers — where the caller writes.
	Private    string `json:"private"`
	Public     string `json:"public"`
	NewBase    bool   `json:"new_base"`
	NewPrivate bool   `json:"new_private"`
	NewPublic  bool   `json:"new_public"`
}

Registration answers RegisterProject.

type Registry

type Registry struct {
	// contains filtered or unexported fields
}

Registry manages a SET of named indexes. Two storage layouts:

  • single-home (OpenRegistry): all indexes live in one Home as sibling sqlite files sharing its originals/ + pages/ — "default" is index.sqlite, any other name is index-<name>.sqlite. This is the embedded/project layout.
  • scoped (OpenScopedRegistry): each index is its OWN Home under <root>/indexes/<name>/ (own index.sqlite + originals/ + pages/), fully isolated. This is the DAEMON's multi-index layout — and the substrate for branch storage (a branch is a scoped store layered over a parent).

func OpenRegistry

func OpenRegistry(home Home) (*Registry, error)

OpenRegistry prepares a single-home registry (all indexes as sqlite siblings in one home).

func OpenScopedRegistry

func OpenScopedRegistry(root string) (*Registry, error)

OpenScopedRegistry prepares a registry whose indexes are each their own Home under <root>/indexes/<name>/ — the daemon's scoped, per-index storage.

func (*Registry) ACLEnabled

func (r *Registry) ACLEnabled() bool

ACLEnabled reports whether access control is on.

func (*Registry) ACLOf

func (r *Registry) ACLOf(name string) ACL

ACLOf returns an index's ACL, defaulting to admin-owned and private.

func (*Registry) Admin

func (r *Registry) Admin() string

Admin is the principal that owns indexes without an acl.json; "" when access control is off.

func (*Registry) CanRead

func (r *Registry) CanRead(ctx context.Context, name string) bool

CanRead applies read(i) to the caller on ctx. Always true with access control off, and always true for the admin.

The admin used to read only what it owned or what was public, like anyone else. Indexes a project key creates (over /mcp/{project}, as para does) are owned by "project:<ns>" and private, so no browser identity — the admin's included — could see them: they were missing from the web UI with no error, and nothing could recover them but TransferOwnership. The admin runs the daemon and can read its files anyway; the ACL hiding them from it protected nothing (Carl 2026-11-04: "admin gets full access").

Implicit search scopes do not take this override (OwnReadable): the admin's own sessions search what they own, not every project's corpus.

func (*Registry) Close

func (r *Registry) Close() error

Close releases every open index.

func (*Registry) DeleteAs

func (r *Registry) DeleteAs(ctx context.Context, name string, branch bool) error

DeleteAs removes an index or branch the caller owns.

func (*Registry) DeleteBranch

func (r *Registry) DeleteBranch(name string) error

func (*Registry) DeleteIndex

func (r *Registry) DeleteIndex(name string) error

DeleteBranch closes and removes a branch (its scoped storage), leaving the parent untouched. Errors if `name` is not a branch. DeleteIndex removes an index entirely — its store, its directory, its originals and its pages.

THE POOL IS NOT TOUCHED, and that is the whole point of the split. The pool cost LLM calls (extract/OCR/segment/embed) and is shared by every index; an index is the cheap part, rebuildable FROM the pool without a single model call. Deleting the expensive half to reclaim the cheap one would turn a cleanup into a re-embedding bill, so index lifetime is the caller's business and pool lifetime belongs to Pool.GC.

Refuses an index that a branch is layered over: a branch reads branch-over-parent, so removing the parent leaves it reading half a conversation with no way to say which half is missing.

func (*Registry) EnableACL

func (r *Registry) EnableACL(admin string) error

EnableACL turns on per-index access control. admin owns every index that has no acl.json. Scoped registries only: single-home indexes share one directory, so there is nowhere to put a per-index ACL.

func (*Registry) ExclusiveHashes added in v0.4.1

func (r *Registry) ExclusiveHashes(names []string) ([]string, error)

ExclusiveHashes is the source-file hashes held by the indexes in names and by no other index on this daemon: what can leave the shared pool without costing another index its reuse, and without one project learning about another's files.

func (*Registry) Existing

func (r *Registry) Existing(name string) (*Store, error)

Existing opens an index only if its storage is already there, and never creates one.

Get creates on demand, which is right for ingest — POST /ingest to a new index name should make it — and WRONG for anything that walks a list of names. The worker loops do exactly that: they take reg.Names(), then Get each one in turn. Delete a branch while that pass is in flight and the loop reaches a name whose directory has just gone, Get recreates the whole index, and the thing that was deleted is back — empty, listed everywhere, and impossible to remove while the daemon runs.

Observed, not theorised: a branch deleted through the UI returned ok, and its index.sqlite was recreated within the same second by the round-robin.

Single-home mode has no such race — DeleteIndex and DeleteBranch both refuse without scoped storage — so there it is Get.

func (*Registry) FoldProject

func (r *Registry) FoldProject(ctx context.Context, id string, dry bool) (FoldReport, error)

FoldProject folds the public layers of project id into its base, as the caller, who must own the base. dry reports without writing anything.

func (*Registry) ForkBranch

func (r *Registry) ForkBranch(name, parent string) error

ForkBranch creates a branch index `name` whose reads overlay `parent` (copy-on-write). Errors if the name is taken, the parent is unknown, or this is not a scoped registry.

func (*Registry) ForkBranchAs

func (r *Registry) ForkBranchAs(ctx context.Context, name, parent string) error

ForkBranchAs forks a branch the caller owns, private. The caller must be able to read the parent: a fork is a read of everything in it.

func (*Registry) Get

func (r *Registry) Get(name string) (*Store, error)

Get opens (once, then caches) the named index, creating it if absent.

func (*Registry) HasIdentifier added in v0.4.0

func (r *Registry) HasIdentifier() bool

HasIdentifier reports whether an identity model is configured: captions, tags and field extraction need one.

func (*Registry) IsAdmin added in v0.5.0

func (r *Registry) IsAdmin(ctx context.Context) bool

IsAdmin reports whether the caller on ctx is the daemon's admin. False with access control off: there is no admin then.

func (*Registry) Keys added in v0.4.0

func (r *Registry) Keys() *KeyStore

Keys is the registry's key store, nil when none is set.

func (*Registry) LayersOf

func (r *Registry) LayersOf(base string) []string

LayersOf lists the branches whose parent is base.

func (*Registry) ListBranches

func (r *Registry) ListBranches() ([]BranchInfo, error)

ListBranches enumerates the scoped registry's branches with lineage + age + last-access + local doc count, sorted by name.

func (*Registry) Meta added in v0.5.0

func (r *Registry) Meta(name string) (IndexMeta, bool)

Meta reads an index's metadata without the read check. False when the index does not exist; nothing is created.

func (*Registry) Names

func (r *Registry) Names() []string

Names lists the indexes present on disk (always including "default"), plus any opened this session, sorted.

func (*Registry) OwnReadable added in v0.5.0

func (r *Registry) OwnReadable(ctx context.Context, names []string) []string

OwnReadable is Readable without the admin's override: what an implicit search scope (no index named, "all", @cwd) reads. The admin token is what clients on the daemon's own machine use, and an override there would make every local session search every project's corpus.

func (*Registry) ParentOf

func (r *Registry) ParentOf(name string) string

ParentOf is a branch's parent, "" for an index that is not a branch.

func (*Registry) ProjectBase

func (r *Registry) ProjectBase(id string) string

ProjectBase returns the base index registered for a project id, "" if none.

func (*Registry) ProjectIDOf

func (r *Registry) ProjectIDOf(base string) string

ProjectIDOf is the project id a base index was registered under, "" when it is not a project base.

func (*Registry) ProjectSavings

func (r *Registry) ProjectSavings(ctx context.Context, id string) (ProjectSavings, error)

ProjectSavings reports, for a project id, what a maintained base would save.

func (*Registry) Read

func (r *Registry) Read(ctx context.Context, name string) (*Store, error)

Read opens an index the caller may read. An index the caller may not read, or that does not exist, is ErrNoIndex — and is never created, with access control on or off: a read of a wrong name used to leave an empty index behind, listed everywhere, which made the next choice of name harder.

func (*Registry) Readable

func (r *Registry) Readable(ctx context.Context, names []string) []string

Readable filters names to the ones the caller may read, keeping order.

func (*Registry) ReapSessionBranches added in v0.4.0

func (r *Registry) ReapSessionBranches(ttl time.Duration, now time.Time) ([]string, error)

ReapSessionBranches deletes every session branch ("idx@session") whose last write is older than ttl, and returns their names. The last write is the newest modification time of the branch's storage files: every write lands in the index file or its write-ahead log.

func (*Registry) RegisterProject

func (r *Registry) RegisterProject(ctx context.Context, id, label string) (Registration, error)

RegisterProject is register-or-layer, atomic per registry: the base for id (created public and empty when absent) and the caller's private and public layers over it (each created when absent). Repeating a registration returns the same names.

func (*Registry) Scoped added in v0.4.0

func (r *Registry) Scoped() bool

Scoped reports whether the registry keeps one directory per index (the daemon's layout) rather than a single home.

func (*Registry) ScopedFor

func (r *Registry) ScopedFor(ctx context.Context, dir string) []string

ScopedFor is the indexes a search from dir reads when it names none: those whose scopes match dir, and the ambient global ones. Only indexes the caller may read as an owner (OwnReadable): the admin's override is for indexes it names.

func (*Registry) SetEmbedder

func (r *Registry) SetEmbedder(e *Embedder)

SetEmbedder makes every index (already open or opened later) embed on ingest.

func (*Registry) SetIdentifier

func (r *Registry) SetIdentifier(id *Identifier)

SetIdentifier makes every index (already open or opened later) caption its ingested documents (identity.go). nil disables it.

func (*Registry) SetImageEmbedder

func (r *Registry) SetImageEmbedder(ie ImageEmbedder)

SetImageEmbedder makes every index (already open or opened later) embed FIGURE IMAGES on ingest (else figures embed their description). nil disables it.

func (*Registry) SetKeyStore added in v0.4.0

func (r *Registry) SetKeyStore(ks *KeyStore)

SetKeyStore gives the registry the keys whose principals its access checks resolve (keys.go).

func (*Registry) SetVisibility

func (r *Registry) SetVisibility(ctx context.Context, name string, v Visibility) error

SetVisibility changes who may read an index. Owner only.

func (*Registry) Siblings

func (r *Registry) Siblings(ctx context.Context, name string) []string

Siblings lists the branches that share name's parent and that the caller may read — what a search on name fans out to besides name itself. Empty for an index that is not a branch. With access control off every sibling is readable.

func (*Registry) TransferOwnership

func (r *Registry) TransferOwnership(ctx context.Context, name, to string) error

TransferOwnership hands an index to another principal — how a maintainer service takes over a project's base. The current owner or the admin may do it; visibility is unchanged. Readers of a branch are unaffected; writers are now the new owner only.

func (*Registry) Write

func (r *Registry) Write(ctx context.Context, name string) (*Store, error)

Write opens an index the caller owns. A new name is created and owned by the caller, private. An index the caller can read but does not own is ErrForbidden; one it cannot read is ErrNoIndex.

type Relation

type Relation string

Relation names how a pair of documents overlap. Deliberately four values and not a score: the caller is a triage flow that has to DO something different in each case, and "0.63" does not say which.

const (
	// RelIdentical — the two files are byte-for-byte the same. Not a shingle
	// finding at all: it comes from documents.content_hash, which ingest already
	// records for its own re-index skip.
	//
	// Kept separate from RelDuplicate because it is a strictly stronger claim and
	// because the shingle scores UNDERSTATE it. Three byte-identical pairs exist on
	// this corpus and none scored 1.00 — the Graham declaration against its own
	// email attachment came back 0.93/0.92, because generic-text masking removes
	// the same passages from both sides and coverage is measured over what is left.
	// A triage flow told "0.93 similar" about two identical files will ask a human
	// to compare them; told "identical", it can act.
	RelIdentical Relation = "identical"
	// RelDuplicate — each document is substantially the whole of the other.
	RelDuplicate Relation = "duplicate"
	// RelProbeInside — the probe is contained in the match. The upload is an
	// instrument we already hold as part of a larger filing.
	RelProbeInside Relation = "probe-inside-match"
	// RelMatchInside — the match is contained in the probe. The upload is a larger
	// filing that swallows something we already hold standalone; the held copy is
	// probably the better evidence.
	RelMatchInside Relation = "match-inside-probe"
	// RelOverlap — they share passages without either containing the other.
	// Quotation, a shared exhibit, or a partial reproduction.
	RelOverlap Relation = "overlap"
)

type RenderPolicy

type RenderPolicy struct {
	BaseDPI          int `json:"base_dpi,omitempty"`            // 0 → 200
	MaxDPI           int `json:"max_dpi,omitempty"`             // 0 → 600
	TargetGlyphPx    int `json:"target_glyph_px,omitempty"`     // 0 → 20
	SmallTextGlyphPx int `json:"small_text_glyph_px,omitempty"` // 0 → 14
}

RenderPolicy is the automatic per-page resolution rule, which was four package constants and therefore the one thing here that genuinely required a rebuild to change.

The rule: measure the page's own glyph height with the cheap engine, and if it is below SmallTextGlyphPx re-render at whatever DPI brings it to TargetGlyphPx, capped at MaxDPI. Zero fields keep the measured defaults — 200/600/20/14 — which were chosen on this corpus and should not be moved casually. Raising TargetGlyphPx costs tokens on every small-text page.

type RetrySummary

type RetrySummary struct {
	N429, N5xx, NTransport int
	Recovered, GaveUp      int
	// Waited is the sum of the delays the client scheduled — the wall-clock this
	// job spent holding rather than working. The number that says whether an hour
	// was spent waiting or generating, which one call's Elapsed cannot.
	Waited time.Duration
	// Worst is the longest single call's time in the retry loop.
	Worst time.Duration
	// LastStatus / LastBody are the most recent server answer and its own words.
	// The body is the actionable half — "increase the physical batch size" —
	// and is empty far more often than not.
	LastStatus int
	LastBody   string
	// MaxAhead is the deepest queue a fair-share proxy reported us standing in.
	MaxAhead int
}

RetrySummary is one job's retry history, flattened.

func (RetrySummary) Detail

func (s RetrySummary) Detail() string

Detail renders the summary for a stage row.

func (RetrySummary) Empty

func (s RetrySummary) Empty() bool

Empty reports whether nothing worth recording happened, which is the normal case and must stay silent — a stage row on every healthy job is noise, and noise is why the real ones were never noticed.

type RetryTally

type RetryTally struct {
	// contains filtered or unexported fields
}

RetryTally accumulates llm.RetryEvents for one job.

Safe for concurrent use: the worker is single-threaded per index today, but a tally that silently under-counts the moment that changes is the same class of error it exists to catch.

func (*RetryTally) Observe

func (t *RetryTally) Observe(e llm.RetryEvent)

Observe is the llm.Client.OnRetry hook.

func (*RetryTally) Take

func (t *RetryTally) Take() RetrySummary

Take returns the accumulated summary and clears it, so the next job starts from zero. One call per job, at the end of it.

type Root

type Root struct {
	Path    string   `json:"path"`
	Include []string `json:"include,omitempty"`
	Ignore  []string `json:"ignore,omitempty"`
}

Root is a source directory, optionally with its own include/ignore overriding the index's. In JSON it is EITHER a bare path string OR {path, include, ignore}.

func (*Root) UnmarshalJSON

func (r *Root) UnmarshalJSON(b []byte) error

UnmarshalJSON accepts a bare string ("./src") or an object ({"path":"./gen","include":["*.go"]}).

type Run

type Run struct {
	AStart, AEnd int
	BStart, BEnd int
}

Run is one maximal span of identical text: A[AStart:AEnd) equals B[BStart:BEnd) byte for byte in folded form, and the two spans are the same length.

func (Run) Len

func (r Run) Len() int

Len is the run's length in folded characters.

type Runner added in v0.5.0

type Runner struct {
	// Stores is the set of indexes whose queues are worked. Required.
	Stores StoreSet
	// NewWorker builds the worker for one index. It is called once per index per
	// runner, the first time that runner takes a job from the index. Required.
	// A NewWorker that returns an error leaves the job pending for a later pass.
	NewWorker func(*Store) (*Worker, error)
	// Runners is how many ingest jobs may run at once, across every index.
	// < 1 → DefaultIngestRunners.
	Runners int
	// IdentitySlots is each index's caption pipeline width (IdentityWorker.Slots).
	// 0 → DefaultIdentitySlots.
	IdentitySlots int
	// Logf receives the per-job and per-index log lines. nil → log.Printf.
	Logf func(format string, args ...any)
	// Notef receives the drain's notices (a cut-off, a job that did not stop).
	// nil → a "raglit: " line on stderr.
	Notef func(format string, args ...any)
	// contains filtered or unexported fields
}

Runner is the ingest scheduler: one pool of runners draining every index's queue, the identity (captioning) queue beside it, and the drain that winds both down before a stop or a re-exec.

Fill the fields, call Start once, and call Drain (or CutOff) to stop.

The identity queue drains only on a store whose identifier is set (Store.SetIdentifier); on any other it is skipped without a log line, because no identity model is a normal configuration. NewWorker sets it, but Runner calls NewWorker only when an ingest runner takes a job from that index. So set each store's identifier before Start (raglitd does, through Registry.SetIdentifier), or captions queued on an index with no ingest since Start wait until that index's next ingest.

func (*Runner) CutOff added in v0.5.0

func (r *Runner) CutOff()

CutOff cancels the running jobs now; they requeue themselves. It also ends the claiming. Safe before Start and more than once.

func (*Runner) Done added in v0.5.0

func (r *Runner) Done() <-chan struct{}

Done is closed when the ingest runners and the caption loop have both returned, including every caption drain the loop started. After it closes the Runner touches no store, so a caller may close them. A caption call in flight when the claim context ends is cancelled through that context, so the wait is short. nil before Start.

func (*Runner) Drain added in v0.5.0

func (r *Runner) Drain(grace time.Duration) bool

Drain winds the ingest queue down before the process exits or re-execs.

Before it, a stop cut off whatever was running. The process exited under the jobs, the next start found their rows still 'running' with a dead pid, and reclaimOrphanedJobs marked them 'error' — correct for a crash, wrong for a restart that someone asked for. Every restart of a busy daemon left errors to retry by hand. The re-exec paths waited for an idle queue instead, while the queue went on claiming, so a busy queue could hold a re-exec back without end.

A drain stops claiming first, then gives the running jobs a grace period to finish, then cancels them. A job cancelled this way goes back to 'pending' (Worker.ProcessJob), so the next process runs it. Captions and re-embedding are not drained: an orphaned caption is requeued at start (ReclaimIdentityJobs), and re-embedding resumes from the fragments that still have no vector.

Drain reports whether every runner returned. Safe to call more than once and from more than one goroutine. Before Start it returns true at once.

func (*Runner) Start added in v0.5.0

func (r *Runner) Start(parent context.Context) (claim, work context.Context)

Start begins working the queues and returns at once.

It returns two contexts, both under parent. claim ends when the Runner stops claiming (Drain, CutOff, or parent ending); work ends when running jobs are cancelled (a Drain past its grace, CutOff, or parent ending). They are two contexts so a stop can let running jobs finish. A caller may run its own background loops under claim and its own request-scoped work under work, so they stop with the queue.

Start must be called once.

type STT

type STT struct {
	// BaseURL is the endpoint's root — normally the same broker as the vision and
	// embed models, since corrallm re-exports oidio's backends beside them.
	BaseURL string
	APIKey  string
	// Model is the oidio model id — "stt-diarize" for speaker-labelled segments,
	// "stt" for plain text. Recorded as the reading's Producer, because a corpus
	// outlives several models and a transcript whose author is unrecorded cannot
	// be told from one a different model produced.
	Model string
	// Chan admits one transcription at a time per model, at a width learned from
	// the server's own backpressure. nil → ungated, which is what a test wants
	// and what a single-caller CLI can afford.
	//
	// Wired because the first real run proved it necessary and NOT wiring it was
	// the bug modelchan.go's comment describes. Four hearings queued together,
	// the daemon started all four, and corrallm answered two of them
	// `429 {"capacity":2,"in_flight":2,"reason":"queue-timeout","retry_after":9}`
	// — two 40-minute recordings failed after uploading, having done the work of
	// getting there. A transcription is the most expensive request raglit makes
	// and the least affordable one to throw away on a retryable refusal.
	Chan *Channels
	// HTTP overrides the client (tests). nil → a client with Timeout.
	HTTP *http.Client
	// GatewayBackoff overrides how long to wait after a 502/503/504. 0 →
	// sttGatewayBackoff. Set short in tests, which must not sit out a real
	// restart window to prove they retry.
	GatewayBackoff time.Duration
	// Timeout bounds one transcription. Diarization costs roughly a minute per
	// minute of audio, so this is measured in hours, not the seconds an HTTP
	// default assumes: the 20-minute hearing that prompted this would trip a
	// 30-second timeout every time and look like a broken endpoint.
	Timeout time.Duration
}

STT turns a recording into a reading. nil on a Worker → an audio job fails with a clear message, the same way a PDF job does without a vision model.

func (*STT) Transcribe

func (s *STT) Transcribe(ctx context.Context, name string, data []byte) (*sttResponse, error)

Transcribe uploads the recording and returns oidio's reading.

name is the original filename; it rides along in the multipart part because oidio's decoder uses the extension as a hint and an upload called "file" with no extension is a container it has to guess at.

Gated and retried, because the upload is the expensive part. corrallm refuses a saturated backend with a 429 that carries how long to wait, and throwing a 40-minute recording away on one is discarding minutes of transfer for a condition that says, in the response body, when it will have passed.

type SearchFilter added in v0.5.0

type SearchFilter struct {
	// PathPrefix keeps documents whose path starts with it (see pathPredicate).
	PathPrefix string
	// Tags keeps documents that carry EVERY one of these tags, each as a whole
	// tag in the caller's tags (Store.SetTags), gen_content_tags or
	// gen_role_tags — "paint" does not match "lead paint". A tag is compared the way writeIdentity stores it:
	// lowercased, its whitespace collapsed (normalizeContentTag). An entry that
	// normalizes to nothing is ignored.
	Tags []string
	// Fields keeps documents whose extracted fields (doc_fields.fields) have each
	// key equal to its value. Only a JSON STRING value matches: a number or a
	// boolean in the record does not equal its spelling here. A document with
	// no extraction matches no field constraint.
	Fields map[string]string
}

SearchFilter scopes a search to the documents that qualify, BEFORE anything is ranked. The constraints go into each layer's WHERE, next to the path scope, so a limit counts only qualifying fragments: a filtered query with limit 1 returns the best QUALIFYING fragment, not the best fragment of the corpus that the filter then throws away. Filtering after ranking would return fewer answers than were asked for, or none, and that reads as a thinner corpus.

Every constraint must hold (AND). The zero value constrains nothing.

type SegResult

type SegResult struct {
	ContinuesPrevious bool      `json:"continues_previous"`
	Fragments         []Segment `json:"fragments"`

	// Degraded names why this unit was NOT segmented by the model, when the fix
	// loop ran out and run() fell back to the whole unit as one fragment. Empty
	// on a real segmentation.
	//
	// It exists because that fallback returns a nil error and a plausible result:
	// the document completes, the page count is right, and the only evidence that
	// a page came back as an undivided block is a reason that used to be dropped
	// on the floor. Not part of the model's answer — `json:"-"` so a model cannot
	// claim it, and so unmarshalling a reply cannot clear it.
	Degraded string `json:"-"`
}

SegResult is the model's structured segmentation of one unit.

type Segment

type Segment struct {
	Text string `json:"text"`
}

Segment is one fragment the model carved out of a unit.

func SplitOversized

func SplitOversized(ctx context.Context, b *TokenBudget, frags []Segment) []Segment

SplitOversized bounds fragments the model returned, so a fragment can never be larger than the embedding model will take as one input.

The deterministic windower is already capped by ResolveFragParams. The LLM path was not: it asks for "roughly 400-800 words" and takes whatever comes back, so a dense page could yield a fragment no embedder would accept. The failure was silent in the worst way — the fragment reached the API, the API returned a 500, and the whole document failed with a message about batch sizes.

Splitting rather than TRUNCATING is the point. A truncated fragment is indexed, searchable, and quietly missing its tail, which is the same class of failure as a transcription that reads complete. A split fragment keeps every character; only the boundary is arbitrary.

Bounded in TOKENS by the embedder's own budget. It used to convert the token limit to characters at two characters per token and check the result with an estimate built on the same ratio — which is not a bound: a survey legal description is 1.16 characters per token, so a piece could pass both checks and still be half again over the real limit.

type Segmenter

type Segmenter struct {
	Client     Chatter
	MaxRetries int // JSON fix-loop attempts after the first try (default 2)
	// Collection is what the corpus owner says about reading THIS collection
	// (Store.IndexHint), appended to the segmentation prompt. Where a fragment
	// boundary falls depends on knowing what the document IS: a lab panel's
	// results are one unit and a work order's line items are another, and
	// nothing on the page says so.
	Collection string

	// FragBudget is the ceiling a fragment must respect — the EMBEDDER's limit,
	// in the tokens it counts in. nil → unchecked.
	//
	// Enforced by asking the model again rather than by cutting, for as long as
	// the fix loop allows: a model re-splitting its own fragment cuts at a
	// heading or a paragraph, and a splitter cuts at whatever falls near the
	// budget.
	//
	// It replaces a MaxFragChars that nothing ever assigned, so this check has
	// been returning "nothing is oversized" for every document ever indexed and
	// the re-prompt it guards has never run once.
	FragBudget *TokenBudget

	// MaxTokens caps one unit's segmentation; 0 → maxTokensFor(the unit), i.e.
	// scaled to the input, since the answer is the input re-emitted. See chat.go.
	MaxTokens int
	// contains filtered or unexported fields
}

Segmenter runs schema-validated LLM segmentation with a fix-loop.

func NewSegmenter

func NewSegmenter(c Chatter) *Segmenter

NewSegmenter builds a Segmenter over a chat client (an *llm.Client).

func (*Segmenter) SegmentText

func (sg *Segmenter) SegmentText(ctx context.Context, text, openText string) (SegResult, error)

SegmentText segments a window of text/code.

type SimilarOpts

type SimilarOpts struct {
	Width int // shingle width; 0 = FoldWidth
	Mod   int // sample divisor; 0 = SampleMod
	// MinChars is the matched-length floor for reporting, in folded characters.
	// 0 means MinReportChars.
	MinChars int
	// MinCoverage is an ADDITIONAL block-coverage floor, in either direction. 0
	// means no coverage floor — length is the primary gate (see MinReportChars).
	MinCoverage float64
	// Limit caps reported matches; 0 = unlimited.
	Limit int
	// Self keeps the probe's own path in the results. Off by default (a document
	// is trivially identical to itself); on for verifying a rebuild.
	Self bool
	// GenericDF overrides GenericDocFreq. Exposed because the right cutoff is a
	// property of the CORPUS — a corpus of 3,000 emails needs a higher one than a
	// corpus of 200 recorded instruments — and because a person who suspects a
	// duplicate was masked away needs to be able to raise it and look.
	GenericDF int
	// NoMask disables generic-text masking entirely. For diagnosing "why was this
	// not reported": if it appears with masking off and not on, its text is corpus
	// chrome and the finding was correct to suppress.
	NoMask bool
}

SimilarOpts are the knobs `raglit similar` exposes. Zero values mean defaults, so a caller can pass an empty struct.

type SimilarReport

type SimilarReport struct {
	// Probe is the path compared. Indexed is whether it was read from the index
	// (rather than extracted from a file on disk), because that decides whether
	// the probe's own text has been through this index's OCR or a different run of
	// it — two reads of one scan differ, and a lower score against an otherwise
	// identical document is explained by that.
	Probe   string `json:"probe"`
	Indexed bool   `json:"indexed"`
	// Source says where the probe's TEXT came from when it was not the index:
	// "extracted" (read locally, no OCR) or "transcription:<file>" (a sidecar
	// written by an earlier ingest). Recorded because a lower score against an
	// otherwise identical document is usually explained by this — two OCR reads of
	// one scan differ, and comparing an extraction against an OCR'd copy is not
	// comparing like with like.
	Source string `json:"source,omitempty"`
	Pages  int    `json:"pages"`
	Chars  int    `json:"chars"`
	// Shingled is the probe's distinct shingle count. Zero means the probe holds
	// less text than one shingle — a blank scan, or a page that is pure figure —
	// and NO match can be reported. Distinguishing that from "nothing similar
	// found" matters: the first is "we cannot tell", the second is "we checked".
	Shingled int `json:"shingled"`
	// Exact names indexed documents whose SOURCE BYTES are the same as the probe's,
	// path -> title. Independent of every score here and stronger than all of them;
	// present even when the probe's text could not be read at all.
	Exact map[string]string `json:"exact,omitempty"`
	// GenericChars is how much of the probe was masked as corpus-generic. A probe
	// that is almost entirely generic (a bare email footer, a cover sheet) will
	// report no matches, and this is what says why.
	GenericChars int        `json:"generic_chars,omitempty"`
	Matches      []DocMatch `json:"matches"`
	// Recipe is the fold/shingle parameters the report was produced under, so a
	// stored finding can be told from one computed under different settings.
	Recipe string `json:"recipe"`
}

SimilarReport is what `raglit similar` returns. Shaped to be consumed programmatically first — the upload triage flow is the caller that matters, and it needs to branch on Relation and read page ranges, not parse a table.

type SketchStatus

type SketchStatus struct {
	Documents   int      `json:"documents"`
	Sketched    int      `json:"sketched"`
	Pages       int      `json:"pages"`
	IndexRows   int      `json:"index_rows"`
	Recipe      string   `json:"recipe"`
	Unsketched  []string `json:"unsketched,omitempty"`
	StaleRecipe []string `json:"stale_recipe,omitempty"`
}

SketchStatus is what the index knows about its own sketch coverage. Reported rather than assumed, because an unsketched document is UNCHECKED and reporting it as clean is the one wrong answer here.

type Slice

type Slice struct {
	// ID is the child's stable identity. Facts cite it, so it must survive
	// re-reads and must NOT be derived from the page range: a bundle re-paginated
	// by a better read would otherwise rename every child that cites it.
	ID     string `json:"id"`
	Parent string `json:"parent"`
	// From and To are inclusive PARENT page numbers.
	From int `json:"from"`
	To   int `json:"to"`
	// Title is what this instrument is, in a person's words. The one field that
	// makes a slice worth more than the page range it wraps.
	Title string `json:"title,omitempty"`
	Note  string `json:"note,omitempty"`
	By    string `json:"by,omitempty"`
	At    string `json:"at,omitempty"`
}

Slice declares that a page range of Parent is a document in its own right.

func (Slice) ChildPath

func (s Slice) ChildPath() string

ChildPath is the derived document's path in the index: the parent's path with the range appended after '#'.

A '#' suffix rather than a new file, because there IS no new file and the path should say so. It also keeps provenance in front of anyone reading a search result — a hit in "scan.pdf#p3-6" is visibly a hit inside scan.pdf, which a separate filename would hide.

func (Slice) Pages

func (s Slice) Pages() []int

Pages returns the parent page numbers this slice covers.

type SliceProposal

type SliceProposal struct {
	From  int    `json:"from"`
	To    int    `json:"to"`
	Form  string `json:"form,omitempty"`
	Title string `json:"title"`
	Why   string `json:"why"`
	// Shared names the other forms that start on this proposal's FIRST page.
	//
	// Two one-page forms printed on one sheet cannot both be a page range, and
	// the answer is not to pick one: a slice mints a document from a range, and
	// the range would be a lie about half of it. Those are `seen-in` claims —
	// observed inside, no identity minted, no page arithmetic — so the proposal
	// says so rather than quietly proposing a slice that is wrong.
	Shared []string `json:"shared_page_with,omitempty"`
}

SliceProposal is a candidate instrument inside a bundle.

func ProposeSlices

func ProposeSlices(pages []PageText) []SliceProposal

ProposeSlices reads a bundle's pages and proposes the instruments in it.

func (SliceProposal) Sliceable

func (p SliceProposal) Sliceable() bool

Sliceable reports whether this proposal should become a slice at all.

type StageLog

type StageLog struct {
	// contains filtered or unexported fields
}

StageLog records stages for one job. A nil *StageLog is a no-op, so the ingest pipeline can be driven with no job (CLI) or with one (the worker/daemon).

func (*StageLog) Done

func (sl *StageLog) Done(name, engine, detail string)

Done / Skip / Fail are shorthands for the common states.

func (*StageLog) Fail

func (sl *StageLog) Fail(name, engine string, err error)

func (*StageLog) Record

func (sl *StageLog) Record(name, engine, state, detail string)

Record appends one stage. Errors are swallowed — stage recording is observability, and must never fail an ingest.

func (*StageLog) Skip

func (sl *StageLog) Skip(name, detail string)

type StaleFields

type StaleFields struct {
	Path   string      `json:"path"`
	Type   string      `json:"type"`
	Reason FieldsStale `json:"reason"`
}

StaleFields is one extraction that is owed a re-run, and why.

type Status

type Status struct {
	Documents  int           `json:"documents"`    // docs indexed
	Fragments  int           `json:"fragments"`    // fragments indexed
	Done       int           `json:"done"`         // completed jobs
	Running    int           `json:"running"`      // in-flight jobs
	Pending    int           `json:"pending"`      // queued jobs
	Failed     int           `json:"failed"`       // errored jobs
	RatePerMin float64       `json:"rate_per_min"` // recent completion rate (jobs/min); 0 = unknown
	Items      []PendingItem `json:"items"`        // running + pending, in processing order, with ETAs
	// Identity is the CAPTIONING queue (identityqueue.go), which is separate work
	// on the same endpoint: an ingest job is minutes of OCR over many pages, a
	// caption is one bounded call. Reported here because "what is this index
	// still owed" is one question, and a 400-document sweep outstanding is the
	// kind of thing that should not need a second command to notice.
	Identity IdentityQueueStatus `json:"identity"`
	// Lanes is the queue BY SCHEDULING LANE — heavy (vision/OCR, one GPU slot)
	// and light (everything else, several at once). Reported because "12 pending"
	// answers nothing on its own: twelve scans is an afternoon and twelve
	// markdown files is under a minute, and which one it is used to be
	// unknowable without reading every URL.
	Lanes map[string]LaneStatus `json:"lanes"`
	// Prunable is how many documents' source files are gone (prune.go): out of
	// search already, removed by `raglit prune`. Up to a minute stale.
	Prunable int `json:"prunable,omitempty"`
}

Status is a snapshot of the index + ingest queue.

func NewStatus

func NewStatus() Status

NewStatus is a zero Status with a non-nil Items, so an idle queue marshals as `"items":[]` rather than `null` — null-vs-empty-array trips naive consumers. Every producer of a Status (IndexStatus, the daemon/MCP aggregators) starts here.

type Store

type Store struct {
	// contains filtered or unexported fields
}

Store is a handle to one raglit index file. When opened via OpenHome it also knows a Home layout and copies ingested originals into it.

func Open

func Open(path string) (*Store, error)

func OpenBranch added in v0.8.0

func OpenBranch(parent Home, branch Home, root, parentRoot string) (*Store, error)

OpenBranch opens the branch at home branch over the index at home parent, creating the branch home and its branch.json when they do not exist. It is idempotent: a second call opens the same branch. It returns an error when an existing branch.json names a different parent, root or parent root, or when branch holds an index that is not a branch.

root and parentRoot are absolute directories: the worktree's and the main checkout's. With both set, the branch translates paths at the overlay boundary (see the top of worktree.go), so its index stores only the files that differ from the parent's, plus tombstones. With both empty, the branch does no translation. One set and the other empty is an error.

Translation is a string prefix match: OpenBranch does not resolve symlinks. root must be spelled as the paths the branch's documents are ingested under, and parentRoot as the paths the parent index stores. A different spelling (a symlinked directory, or git's resolved path from WorktreeOf against an index of unresolved paths) translates nothing and reports nothing: every worktree file counts as new, and the parent's documents show at their main-checkout paths. root may be inside parentRoot.

The parent index must exist. OpenBranch opens its own handle on it and does not wire the parent's own parent. The returned store owns that handle: its Close closes the parent too. Two branches of one parent each hold a handle.

func OpenBranchOn added in v0.8.1

func OpenBranchOn(parent *Store, branch Home, root, parentRoot string) (*Store, error)

OpenBranchOn is OpenBranch over parent, an index the caller has open already, instead of a handle of its own. parent must be the default index of its home (OpenHome); branch.json records that home, as OpenBranch does. The returned store does not own parent: its Close leaves parent open, and the caller must keep parent open until the branch is closed.

func OpenHome

func OpenHome(home Home) (*Store, error)

OpenHome opens the home's primary ("default") index. Use Open for a raw path or an in-memory test index; OpenIndex for a named index within the home.

func OpenIndex

func OpenIndex(home Home, name string) (*Store, error)

OpenIndex opens a NAMED index within a home (sharing its originals/pages), so one home can hold several indexes. "default" (or "") is the home's primary index.sqlite; any other name is index-<name>.sqlite. Created if absent. Ingesting a doc whose Path is a real file copies it into <home>/originals/.

func OpenIndexRO

func OpenIndexRO(home Home, name string) (*Store, error)

OpenIndexRO opens an index for READING only.

The daemon is the single writer for its indexes — that is what keeps one worker pool from fighting a CLI over the same file. A reporting command still needs to see that data, and SQLite in WAL mode allows any number of concurrent readers alongside the writer, so reading is safe where writing is not.

It does not create anything. An index that is not there is an error rather than a new empty database, because a command that silently invents one reports "nothing found" about a corpus it never opened.

func (*Store) ActiveReadings

func (s *Store) ActiveReadings(ctx context.Context, doc string) (map[int]PageReading, error)

ActiveReadings returns the reading IN FORCE for each page of a document, by page number. Pages nobody has ruled on are absent — a document with no corrections returns an empty map, which is the common case and costs one indexed lookup.

func (*Store) AddNote

func (s *Store) AddNote(path string, n Note) (Note, error)

AddNote records a note. page <= 0 files it against the whole document.

An empty body is refused rather than stored. A blank note is not a state anybody meant to reach — it is a stray click — and one sitting in a thread costs every later reader the moment it takes to work out that it says nothing.

func (*Store) AddPageReading

func (s *Store) AddPageReading(ctx context.Context, r PageReading) error

AddPageReading appends a reading and makes it the active one.

Appends rather than replaces. The previous reading stays, marked inactive, so "what did this page say before somebody checked it" remains answerable — which is the whole reason a correction is an attestation rather than an edit.

func (*Store) AllowPatterns

func (s *Store) AllowPatterns() []string

AllowPatterns is this index's allow list: a path matching one is exempt from the global ignores (EnqueueBuiltinIgnore), so an index that IS an iOS project can take `**/Pods/**`. The index's own ignore list still applies — both are this index's choices, and refusing is the safe way to settle them.

func (*Store) Ambient

func (s *Store) Ambient() bool

Ambient reports whether a global index is read by every search.

func (*Store) AttachmentDirFor

func (s *Store) AttachmentDirFor(archivePath string) string

Open opens (creating if needed) a raglit index at path. Use ":memory:" for a throwaway index (tests). foreign_keys is ON so a document delete cascades to its fragments; WAL keeps concurrent readers unblocked during ingest. AttachmentDirFor is where this index stores files extracted out of the archive at archivePath. Empty when the store has no home (":memory:", a bare Open), which is the one case extraction cannot run.

func (*Store) Attestations

func (s *Store) Attestations(doc string) ([]AttestationRow, error)

Attestations returns one document's rows in log order.

func (*Store) AttestationsForUnit

func (s *Store) AttestationsForUnit(doc, unit string) ([]AttestationRow, error)

AttestationsForUnit returns every ruling on one unit, in log order.

Every ruling, not the one in force: a retracted confirmation and the retraction are both facts about what happened, and which one governs is attest.Resolve's answer rather than SQL's.

func (*Store) AttestedDocs

func (s *Store) AttestedDocs() (map[string]int, error)

AttestedDocs reports how many entries each document has, for a review index that can say what has been looked at without opening every log.

Deliberately NOT a coverage percentage. How much of a document is reviewed is a question about the CURRENT reading, and this table holds verdicts against units that may no longer exist — see the schema note on orphans. A count of rulings presented as coverage would report a document as done after a re-read invalidated every one of them.

func (*Store) BackfillLanes

func (s *Store) BackfillLanes() (int, error)

BackfillLanes assigns a lane to pending rows that predate the column, and reports how many it set.

An index that has been running since before lanes existed has an EMPTY lane on every queued row, and no lane claims those — so without this, upgrading the daemon silently strands whatever was already in the queue. Pending only: a running row is owned, and a terminal one is never claimed again.

func (*Store) BackfillSourceStats

func (s *Store) BackfillSourceStats() (int, error)

BackfillSourceStats records the stat of every document whose file is unknown so far, and returns how many it filled. Files no longer on disk stay unknown.

func (*Store) BackfillVectors

func (s *Store) BackfillVectors(ctx context.Context, batch int) (int, error)

BackfillVectors embeds up to batch fragments that have no vector, then figure descriptions with no text-space vector, and returns how many it gave a vector. A fragment whose text already has a vector in this model's space (another fragment's, here or in a branch's parent) gets a copy of it rather than an embed call (R12a, reuse.go). 0 means there is nothing left (or no embedder, or the index is paused).

func (*Store) BranchDiff

func (s *Store) BranchDiff() ([]BranchChange, error)

BranchDiff lists the documents this branch holds or deleted, against what its parent shows now (the parent's own overlay, if it is a branch too). Compared by content hash; a document with no hash on either side counts as "changed". A tombstone for a path the parent does not show is left out: it hides nothing. A non-branch store has no diff.

func (*Store) CanSearchByMeaning

func (s *Store) CanSearchByMeaning() bool

CanSearchByMeaning reports whether vector and hybrid search can run: an embedder is set. Without one they fail on every call, so a UI hides them.

func (*Store) CancelJob

func (s *Store) CancelJob(id int64) error

CancelJob removes a pending job from the queue. Only pending jobs can be canceled — a running job is mid-flight and done/error jobs are terminal.

func (*Store) CaptionProfile added in v0.4.1

func (s *Store) CaptionProfile() string

CaptionProfile is the owner's choice for this index, "" when they made none.

func (*Store) ChatInputLimitTokens

func (s *Store) ChatInputLimitTokens(ctx context.Context, p ContextProber, model string, configured int) int

ChatInputLimitTokens returns the largest single request this index's chat model accepts, in tokens, asking once and remembering the answer.

Asked before probed. A server that states its own n_ctx is one request for the real number; the probe is O(log N) requests for a lower bound, and on an endpoint that accepts anything it returns its own ceiling rather than a fact.

`configured` wins over both: an operator who knows the number should not wait, and a proxy that accepts a prompt its backend will later refuse cannot be measured from outside at all.

Zero means "unknown, do not cap" — the same reading the fragment ceiling gives it. An invented limit would cut pages that were fine, and the failure it would be guarding against announces itself in the endpoint's own words.

func (*Store) ClaimNext

func (s *Store) ClaimNext() (*Job, error)

ClaimNext claims the oldest pending job in this index, whatever it is.

The exported claim the daemon's runners use. There was briefly a lane parameter here: the scheduler sorted jobs into `heavy` and `light` and gave each a slot count, to stop a slow scan holding up a text file. The problem was real and the cure was in the wrong place — what serialised them was not their KIND but one shared "the GPU admits one" budget covering three models on three cards. Admission is per model now (modelchan.go), so a transcription and an embedding take different channels and neither waits on the other. The lane column is still written and still reported; nothing schedules on it.

func (*Store) ClaimNextInLane

func (s *Store) ClaimNextInLane(lane Lane) (*Job, error)

ClaimNextInLane claims the oldest pending job in one scheduling lane, or (nil, nil) when that lane has nothing here.

The lane belongs in the CLAIM rather than in a filter afterwards. A dispatcher that claimed the oldest job and then put it back because it was the wrong kind would move the row's state twice per pass and race the other lane for it; a dispatcher that claimed and then held it would be the serial queue again under a new name.

func (*Store) ClearFailed added in v0.9.0

func (s *Store) ClearFailed(paths ...string) error

ClearFailed forgets the recorded ingest failures of paths (document paths, bare or file://; either spelling clears both), or of every path when none is given, so the next IndexTree queues those files again on the same bytes.

func (*Store) ClearSketches

func (s *Store) ClearSketches() error

ClearSketches drops every stored sketch.

Needed for a recipe change, and it has to be a truncate rather than a per-document rebuild: a document deleted from the index since the last build keeps its shingle rows (the ON DELETE CASCADE only fires when the document row itself goes, and a re-ingest replaces rather than deletes), so those rows go on generating candidates that resolve to a path TruePages cannot read. That showed up as `similar` reporting matches against documents that were not there.

func (*Store) Close

func (s *Store) Close() error

Close releases the underlying database.

func (*Store) DedupeQueue

func (s *Store) DedupeQueue() (int, error)

DedupeQueue collapses duplicate pending jobs for the same url, keeping the oldest, and returns how many rows it removed.

For queues that grew before Enqueue deduped. A running job is left alone: a worker owns it, and deleting a row out from under one is how a job ends up stuck in `running` forever with nothing to finish it.

func (*Store) DeleteDocType

func (s *Store) DeleteDocType(name string) error

DeleteDocType removes a type. The extractions made under it are LEFT ALONE: they are what documents said, and deleting a type is a statement about the index's vocabulary, not a retraction of everything read through it.

func (*Store) DeleteDocument

func (s *Store) DeleteDocument(path string) error

DeleteDocument removes a document from THIS store and, when it's a branch, marks the path tombstoned so the parent's version stops showing through the overlay.

page_readings keys on the path with no foreign key, so the cascade from documents does not reach it; it is deleted here, in the same transaction. The whole removal is one transaction that takes the write lock first.

func (*Store) DeleteNote

func (s *Store) DeleteNote(id int64) error

DeleteNote removes one note by id.

Hard delete, not a tombstone. A note is somebody's comment, not a ruling anything downstream replays, so there is nothing that has to be able to see that it once existed — which is the only reason attestations accumulate instead.

func (*Store) DocCosts

func (s *Store) DocCosts() ([]DocCost, error)

DocCosts lists this store's OWN documents (not a parent's) with their cost. Documents with no content hash are left out: there is nothing to compare.

func (*Store) DocJobs added in v0.2.0

func (s *Store) DocJobs(targets ...string) ([]JobInfo, error)

DocJobs returns the finished ingest jobs (done or error) for one document, oldest first: when it was read into the index, and from where. targets are the forms a job's URL can take for it (the absolute path, the display path); each matches bare and as a file:// URL.

A query of its own rather than a filter over Jobs(): that answers "the last N", and a document read a thousand jobs ago would have no history at all.

func (*Store) DocReadingHistory

func (s *Store) DocReadingHistory(doc string) ([]PageReading, error)

DocReadingHistory returns every recorded reading of every page of one document, in page then arrival order.

Every version, not the active one. The whole reason readings accumulate as rows is that a correction must not erase what it replaced: the superseded text is what the index held when the document was cited, it is what a stale quotation elsewhere still matches, and "the OCR read 2015031104 for 201503110045" is itself evidence of how far a sheet can be trusted. A history that shows only the current answer throws away the thing the table exists for.

func (*Store) DocReview

func (s *Store) DocReview(path string) (title string, pages []PageReview, err error)

DocReview returns a document's title and per-page OCR review (pages ≥ 1, in order). Page text is the concatenation of the fragments indexed for the page — i.e. what OCR/segmentation actually produced. Returns (‑, nil, nil) when the path is unknown.

func (*Store) DocText

func (s *Store) DocText(exactPath string, from, to, maxChars int) (DocContent, error)

DocText overlays: the branch's copy if present, else (unless tombstoned) the parent's.

func (*Store) DocTypeByName

func (s *Store) DocTypeByName(name string) (DocType, error)

DocTypeByName returns one registered type. An unregistered name is an error, not a zero value: extracting against a type nobody defined would produce a record whose shape is whatever the model felt like.

func (*Store) DocTypeRefs

func (s *Store) DocTypeRefs() ([]DocType, error)

DocTypeRefs is the type list as the identity call is shown it: the names it may choose from, and one line each on how to recognise one.

func (*Store) DocTypes

func (s *Store) DocTypes() ([]DocType, error)

DocTypes lists the index's registered types, by name.

func (*Store) DocumentAddedAt added in v0.2.0

func (s *Store) DocumentAddedAt(path string) int64

DocumentAddedAt is when the document was last written into the index, in Unix nanoseconds; 0 when it is not in the index. Re-ingest updates it.

func (*Store) DocumentFields

func (s *Store) DocumentFields(path string) (DocFields, error)

DocumentFields returns what has been extracted for a document. A document with no extraction returns the zero value and no error — that is a state, not a failure. An unknown path IS an error.

func (*Store) DocumentHash

func (s *Store) DocumentHash(path string) (string, error)

DocumentHash returns a document's stored source-content hash, or "" if the doc is unknown or has none. The hash is the sha256 hex of the raw source bytes, not of the extracted text. The worker uses it to skip re-ingesting unchanged content (dedup work).

A branch with a root (OpenBranch) reads through to its parent for a path it has no row for and did not tombstone, so a worktree file whose bytes equal the main checkout's is skipped rather than copied. A branch with no root does not.

func (*Store) DocumentIdentity

func (s *Store) DocumentIdentity(path string) (DocIdentity, error)

DocumentIdentity returns what is recorded about a document. A document with no caption yet returns the zero value and no error — that is a state, not a failure. An unknown path IS an error.

func (*Store) DocumentPaths

func (s *Store) DocumentPaths() ([]string, error)

DocumentPaths lists every document path in the index, branch overlay applied: the branch's own documents, then the parent's that the branch does not shadow.

func (*Store) Documents

func (s *Store) Documents() ([]DocSummary, error)

Documents overlays: branch docs, then parent docs not shadowed.

func (*Store) DocumentsMissingFields

func (s *Store) DocumentsMissingFields() ([]string, error)

DocumentsMissingFields lists documents that resolved as a registered type but have no extraction yet — the work list for `raglit fields`.

func (*Store) DocumentsMissingIdentity

func (s *Store) DocumentsMissingIdentity() ([]string, error)

DocumentsMissingIdentity lists the paths of indexed documents with no caption yet, oldest first — the work list for `raglit identify`. A corpus indexed before identity existed is entirely this list.

func (*Store) DocumentsMissingTags

func (s *Store) DocumentsMissingTags() ([]string, error)

DocumentsMissingTags lists documents that HAVE a caption but no tags — a corpus captioned before tags existed. Its own selector rather than a flag on the one above, because the two are different work: one needs a caption written, the other needs a caption LEFT ALONE and two columns filled in.

func (*Store) DropVectors added in v0.6.0

func (s *Store) DropVectors() (int, error)

DropVectors deletes every vector the text embedder made, so a forced re-embed (BackfillVectors) rebuilds them all. Fragments, their FTS rows and the model and dim records are untouched. Returns the number of vectors deleted.

Nothing made before the drop comes back: an ingest or backfill in flight writes no vectors (vecGen), and a branch stops reusing its parent's vectors until SyncEmbedModel next drops them for another model or length. The branch's re-embed sends every fragment to the embedder. The parent itself is untouched: drop its vectors separately to re-embed it too.

func (*Store) EmbedLimitChars

func (s *Store) EmbedLimitChars(ctx context.Context, e *Embedder, configured int) int

EmbedLimitChars returns the largest single input this index's embed model accepts, probing once and remembering the answer.

Discovered rather than configured, because it is a fact about the endpoint. It was previously left at zero here — meaning "unknown", which ResolveFragParams reads as "no cap" — so fragments were sized by taste (9000 characters) with nothing checking them against what the model would take. The first symptom was a document failing with a 500 about batch sizes.

`configured` wins when set: an operator who knows the number should not have to wait for a probe, and an endpoint that silently truncates cannot be probed at all.

func (*Store) EmbedLimitTokens

func (s *Store) EmbedLimitTokens(ctx context.Context, p ContextProber, model string, configured int) int

EmbedLimitTokens returns what the embed model accepts as one input, in tokens.

The limit that actually refused documents. EmbedLimitChars answers the same question in characters, converted from this number at two characters per token — 16128 for an 8192-token embedder. A scanned court brief reaches 10240 tokens inside 16128 characters, so the fragment passed every check raglit had and was refused by the endpoint:

input (10240 tokens) is too large to process.
increase the physical batch size (current batch size: 8192)

which failed the whole document at the embed stage, after its OCR and its segmentation had already been paid for.

Note this is a DIFFERENT model from the chat one, with a different tokenizer and a much smaller window — 8192 against 180224 on this endpoint. Asking the chat model's limit and applying it here would be off by twenty-two times.

func (*Store) EmbedModel

func (s *Store) EmbedModel() string

EmbedModel is the model this index's vectors are recorded as coming from, "" when no record exists yet.

func (*Store) Enqueue

func (s *Store) Enqueue(url, title string) (int64, error)

Enqueue adds a pending ingest job for url and returns its id. It does not fetch or index anything — a worker does that later (lazy).

A url already waiting is NOT queued again; the existing job's id is returned. Without this the queue inflates every time anything requeues — a watch fires, a batch is re-run, a sweep is retried — and the work is done two and three times over. Measured on the ardley corpus before the check: 162 queued rows for 115 distinct documents, one court memorandum queued FIVE times, roughly 29% of a nine-hour backlog that was pure repetition.

Deduped against pending and running only, never against done. Re-indexing a document that has changed is legitimate and must stay possible; what is never useful is the same file sitting in the queue twice at once.

func (*Store) EnqueueFields

func (s *Store) EnqueueFields(path string, force bool) (bool, error)

EnqueueFields queues one document for the SCHEMA ask: fill out the fields of the document type it resolved as.

func (*Store) EnqueueFieldsFor

func (s *Store) EnqueueFieldsFor(paths []string, force bool) (int, error)

EnqueueFieldsFor queues every given path for the schema ask.

func (*Store) EnqueueFresh

func (s *Store) EnqueueFresh(url, title string, fresh bool) (int64, error)

EnqueueFresh is Enqueue with an explicit re-read flag. A fresh job never dedupes against a pending one: the caller is asking for work the queued job would specifically NOT do.

func (*Store) EnqueueIdentity

func (s *Store) EnqueueIdentity(path string, force bool) (bool, error)

EnqueueIdentity queues one document for captioning. Idempotent per path: an existing pending/running row is left alone (re-queueing a document already in flight buys a duplicate model call and nothing else), and a TERMINAL row — done, skipped or error — is reset to pending. Returns false when an in-flight job already covers it.

Reviving a skipped row is deliberate and is the reason this takes a path at all: the bulk enqueue leaves skips alone (nothing to read is not work), but a document named explicitly, or a --force sweep, is somebody saying that has changed — which it does the moment a document is re-OCR'd.

func (*Store) EnqueueIdentityFor

func (s *Store) EnqueueIdentityFor(paths []string, force bool) (int, error)

EnqueueIdentityFor queues every given path, reporting how many rows it added or revived. Paths already in flight are skipped, not duplicated.

func (*Store) EnqueueMany added in v0.8.0

func (s *Store) EnqueueMany(items []EnqueueItem) ([]EnqueueResult, error)

EnqueueMany enqueues every item in ONE write transaction and returns one result per item, in order. Each item behaves as EnqueueFresh does: empty, generated-sidecar and ignored urls are refused per item (Err set, batch continues), and an item already covered by a pending/running job returns that job's id — including a job inserted earlier in the same batch.

One transaction because one Enqueue per file costs a commit (fsync) each: ~826 files took 5-12 s. BEGIN IMMEDIATE takes the write lock up front, so the batch waits under busy_timeout like any writer and cannot hit the SQLITE_BUSY_SNAPSHOT upgrade failure EnqueueFresh documents. The returned error is only for a transaction or database failure; then nothing is committed.

func (*Store) EnqueueMissingFields

func (s *Store) EnqueueMissingFields(force bool) (int, error)

EnqueueMissingFields queues every document that resolved as a registered type and has no extraction yet — or, with force, every document that resolved as one. A person's extraction is still refused downstream.

func (*Store) EnqueueMissingIdentities

func (s *Store) EnqueueMissingIdentities(force bool) (int, error)

EnqueueMissingIdentities queues every document with no caption yet — or, with force, every document. Returns how many were queued.

func (*Store) EnqueueMissingTags

func (s *Store) EnqueueMissingTags(force bool) (int, error)

EnqueueMissingTags queues every CAPTIONED document that has no tags — or, with force, every captioned document. The backfill for a corpus captioned before tags existed.

func (*Store) EnqueueTags

func (s *Store) EnqueueTags(path string, force bool) (bool, error)

EnqueueTags queues one document for the TAGS-ONLY ask. Same queue, same rows, same resumability — a backfill of hundreds is the same shape of work as a captioning sweep and has no business being a second mechanism.

func (*Store) EnqueueTagsFor

func (s *Store) EnqueueTagsFor(paths []string, force bool) (int, error)

EnqueueTagsFor queues every given path for the tags-only ask.

func (*Store) ExportDoc

func (s *Store) ExportDoc(path string) (PooledDoc, error)

ExportDoc reads a freshly-ingested document back out of an index as a PooledDoc (fragments + vectors + page provenance, page Image = absolute source path), for caching in the pool.

func (*Store) ExtractFields

func (s *Store) ExtractFields(ctx context.Context, path string, force bool) (DocFields, error)

ExtractFields fills out a document's type schema and stores the result.

Returns ErrIdentityKept when an extraction already exists (unless forced), and when a person wrote it — a person's extraction is never regenerated, forced or not.

func (*Store) ExtractableMissing

func (s *Store) ExtractableMissing() ([]string, error)

ExtractableMissing is every document that OWES an extraction: one that resolved as a registered type and has none, and one whose extraction answers a schema or a transcript that has since moved. A person's is never owed again.

func (*Store) FieldsCoverage

func (s *Store) FieldsCoverage() ([]FieldsCoverage, error)

FieldsCoverage reports extraction coverage per type, most-resolved first.

func (*Store) FieldsStaleness

func (s *Store) FieldsStaleness() ([]StaleFields, error)

FieldsStaleness lists every MACHINE extraction that is not current, with the reason. A person's is never stale: it is theirs, and a schema edit does not make what they wrote wrong.

Documents that have never been extracted are NOT here — they are DocumentsMissingFields, which is a different report and a different sentence to say to somebody.

func (*Store) FindPathByHash

func (s *Store) FindPathByHash(hash, path string) (string, error)

FindPathByHash returns the path of a DIFFERENT document row that stores the same content hash, or ("", nil) when there is no such row.

It is the lookup reconciliation uses when a job's source path no longer resolves: the bytes may simply have MOVED, and a document already indexed under the new path carries the hash that identifies them. Excluding the caller's own path is what keeps a plain re-ingest (same path, same content) from finding itself and "moving" onto itself.

func (*Store) ForgetJob

func (s *Store) ForgetJob(id int64) error

ForgetJob deletes a terminal job row and its stages.

Not a cancel and not a retry: this is "that attempt is not coming back and I do not want to be told about it again". The health report already drops rows whose file is gone, because that is decidable; this is for the ones only a person can call — a remote URL that is genuinely dead, a failure nobody is going to chase.

Refused on a live job. A pending job has a cancel and a running one is mid-flight; deleting either loses work that is still in front of somebody.

It removes evidence, and that is the point of restricting it to a person's explicit act. What the row records is an ATTEMPT, not a fact about the corpus — nothing cites it, and a log nobody can prune is one nobody reads.

func (*Store) FragmentCount

func (s *Store) FragmentCount() (int, error)

FragmentCount is the number of fragments, for the cost of a re-embed.

func (*Store) FragmentExists added in v0.7.0

func (s *Store) FragmentExists(contentID string) (bool, error)

FragmentExists reports whether a fragment with this content id is in the index. On a branch it overlays the parent the way search does: the parent's fragment counts unless its document is shadowed (rewritten or tombstoned in the branch).

func (*Store) HybridSearch

func (s *Store) HybridSearch(ctx context.Context, query string, limit int) ([]Hit, error)

HybridSearch is HybridSearchPath with no path constraint.

func (*Store) HybridSearchFiltered added in v0.5.0

func (s *Store) HybridSearchFiltered(ctx context.Context, query string, f SearchFilter, limit int) ([]Hit, error)

HybridSearchFiltered is HybridSearchPath constrained by a SearchFilter. Both sides filter before they rank, so the fusion only ever sees qualifying fragments; RRF itself is unchanged.

func (*Store) HybridSearchPath

func (s *Store) HybridSearchPath(ctx context.Context, query, pathPrefix string, limit int) ([]Hit, error)

HybridSearchPath fuses BM25 and vector rankings with Reciprocal Rank Fusion (RRF) — the standard, score-scale-agnostic combiner: a fragment's fused score is the sum over each ranked list of 1/(rrfK + rank). It over-fetches from each side, so a fragment strong on either signal surfaces. Both sides honor the optional path subtree scope. Requires an embedder. Returns up to limit fragments, best fused first.

func (*Store) IdenticalGroups

func (s *Store) IdenticalGroups() ([][]string, error)

IdenticalGroups returns every set of indexed documents whose SOURCE BYTES are the same, as groups of paths, each group sorted and the groups sorted by their first path.

This is the one relation in the corpus that needs no shingles, no thresholds and no judgement. Two files with one sha256 are the same document — not similar to it, not probably it — so the answer does not depend on any of the tuning the rest of this file is careful about, and it is available on an index that has never been sketched.

Grouped rather than paired because a document held three times is ONE finding. Emitting three pairs invites a reader to rule on them separately and reach two different answers about one set of bytes.

func (*Store) IdentifyDocument

func (s *Store) IdentifyDocument(ctx context.Context, path string, force bool) (DocIdentity, error)

IdentifyDocument captions one already-indexed document: read its indexed text, ask the model what it is, store it. This is the re-runnable half — a corpus of 406 documents is not going to be re-OCR'd to get captions, and a caption is worth having on documents indexed long before this existed.

force replaces a MACHINE caption. It does not replace a PERSON's: someone who corrected a caption did so because the model's was wrong, and re-running the same model on the same text produces the same wrong answer. Changing a person's caption is done by recording another one.

func (*Store) Identities

func (s *Store) Identities() ([]IdentityStatus, error)

Identities lists every document's identity, newest first, including the ones with none — a corpus's captioning coverage is exactly as interesting as the captions.

func (*Store) IdentityJobs

func (s *Store) IdentityJobs(state string, limit int) ([]IdentityJob, error)

IdentityJobs lists the queue, newest first, optionally only one state.

func (*Store) IdentityQueue

func (s *Store) IdentityQueue() (IdentityQueueStatus, error)

IdentityQueue reports the queue's counts.

func (*Store) IdentityText

func (s *Store) IdentityText(ctx context.Context, path string) (string, error)

IdentityText is the text a caption is written from: the reading IN FORCE for each page, not the machine's first attempt at it.

The distinction is the whole of why this function exists. Fragments hold what OCR produced, and they are deliberately NOT rewritten when a person corrects a page — re-fragmenting would move every offset and invalidate every citation already taken. So a correction lives in `page_readings` as the active reading, and anything that wants what the document SAYS has to go there.

Captioning is exactly such a consumer, and the case that proves it is the one this was found on: a disputed record of survey whose corrected page fixed the surveyor's name and certificate number. Those are the facts a caption states. Written from fragments, the caption would have asserted, in a machine's voice and at the top of a document list, the very reading a person had already ruled wrong.

Pages with no ruling fall back to the indexed text, so a corpus nobody has corrected — the overwhelming majority — is unaffected and pays one indexed lookup.

func (*Store) IgnorePatterns

func (s *Store) IgnorePatterns() []string

IgnorePatterns is this index's own ignore list: globs a path must not match to be enqueued. Matched against the absolute path, so a directory anywhere is `**/name/**`; a glob with no "/" matches the file name.

func (*Store) IgnoredBy

func (s *Store) IgnoredBy(path string) string

IgnoredBy returns the pattern that refuses path at Enqueue, "" when none does. The index's own ignore list first; then the global list, unless the index's allow list exempts the path.

func (*Store) ImportVerifiedTranscripts

func (s *Store) ImportVerifiedTranscripts(dryRun bool, ruledBy string) ([]TranscriptMatch, error)

ImportVerifiedTranscripts finds verified transcripts ON DISK and records them as ATTESTED readings of the recordings they transcribe — so search collapses to one, and the one it keeps is the transcript a person ruled on.

From DISK, not from the index, and that ordering is the point. A transcript beside a recording should not be a document at all: it is a reading, and once the corpus stops indexing these sidecars there is nothing in the index to walk. An importer that could only see what was already indexed would stop working the moment its job was done.

Candidates come from the READINGS, not from the documents: a recording with a machine reading is the only thing a transcript can be adopted onto, and the digest to join on lives there. Transcripts are looked for in the directories those recordings came from, which is where every producer puts them.

dryRun reports the matches and records nothing.

func (*Store) IndexAbout

func (s *Store) IndexAbout() (about string, stale bool, err error)

IndexAbout is the stored paragraph and whether it is behind the corpus. Empty when none has been written; writing one is WriteIndexAbout.

func (*Store) IndexDigest

func (s *Store) IndexDigest() (IndexDigest, error)

IndexDigest counts the whole index, keeping the ten most common content tags and every kind and role — the closed vocabularies are short enough to show entire, and a truncated one reads as an absence.

func (*Store) IndexDigestFor

func (s *Store) IndexDigestFor(pathPrefix string, topN int) (IndexDigest, error)

IndexDigestFor counts what the index holds under a path prefix — empty for the whole index. topN caps each list; 0 means "all of them".

func (*Store) IndexHint

func (s *Store) IndexHint() string

IndexHint is what a person tells the models about THIS corpus, in their own words: how to decode it, which of its ambiguities resolve which way, what matters in it.

It exists because the model is answering a general question about a specific corpus, and the corpus is the half it cannot see. "RO" on a garage's paperwork is a repair order, not "received"; a survey's marginal figures are bearings, not measurements; a scanned carbon copy's second column is the customer's, not a duplicate. None of that is inferable from one page, all of it is obvious to whoever owns the corpus, and every prompt that reads a document is worse for not having it.

So it travels the whole way down: the page transcription, the segmentation, the identity, and the field extraction. It is part of the READING recipe for that reason — a changed hint changes what a page says, and pooled work read under the old one must not be replayed under the new.

func (*Store) IndexStatus

func (s *Store) IndexStatus() (Status, error)

IndexStatus reports queue progress: counts, a recent processing rate, and a per-item ETA (queue position × recent average job duration). ETA/rate are 0 until at least one job has completed (no basis to estimate).

func (*Store) Ingest

func (s *Store) Ingest(ctx context.Context, doc Document) error

Ingest upserts a document and (re)indexes its fragments in one transaction. Re-ingesting the same Path is idempotent: the doc's old fragments are dropped and replaced, so re-running an index over a changed file converges rather than duplicating. Empty-text fragments are skipped.

func (*Store) IngestPDF

func (s *Store) IngestPDF(ctx context.Context, ocr *OCR, pdfPath string) (int, error)

IngestPDF pagifies an image/scanned PDF and indexes it via LLM segmentation: the vision model (ocr's client) reads each page image and carves coherent fragments, with open fragments stitched across page boundaries and vectors embedded concurrently. Page images are written under the home's pages/ dir when the store has a home, else a temp dir. Returns the number of fragments indexed.

func (*Store) IngestPooled

func (s *Store) IngestPooled(ctx context.Context, docPath, title string, doc PooledDoc, poolFileDir string) (int, error)

IngestPooled writes a cached PooledDoc into this index — the CHEAP reuse path (no LLM/OCR/embed): fragments + their cached vectors, and page provenance with each image copied from poolFileDir into this index's pages/. Atomic (commitDoc).

func (*Store) Job

func (s *Store) Job(id int64) (JobInfo, error)

Job returns one job by id, whatever its state and however old it is.

Jobs() answers "the last N", which is the wrong shape for a LINK. The health report cites a job by id — an ingest that had to fight the endpoint, a failure from a batch weeks ago — and a reader following that citation lands on a window of the newest rows that does not contain it. Measured on the fence-dispute index: the cited job was 927, the newest was 1467, and the page rendered two hundred rows none of which was the one the link was about.

func (*Store) JobStages

func (s *Store) JobStages(jobID int64) ([]JobStage, error)

JobStages returns a job's stages in the order they were RECORDED — see the ORDER BY in ListJobStages for why that is not the order by seq.

func (*Store) Jobs

func (s *Store) Jobs(state string, limit int) ([]JobInfo, error)

Jobs lists ingest jobs, newest first, via a metaquery Builder: state filters to one lifecycle state (pending|running|done|error); "" or "all" returns every state. limit ≤ 0 → 100.

func (*Store) LearnLimitFromError

func (s *Store) LearnLimitFromError(model, errText string) (int, bool)

LearnLimitFromError records a limit the endpoint stated in a rejection.

Better than the table and better than the probe, because it is not an inference: the server named its own number. Called on an embed failure so the next ingest is sized correctly even if the model was unknown or was swapped under a stored value.

Stored in tokens as well as characters. The token number is the one the fragment budget enforces, and it is the one that was wrong: probed as the model's context window (8192) on a server whose physical batch was 4096, so a CSV fragment of 5250 tokens passed every check and was refused (2026-10-22). A stated number only ever LOWERS the token limit; the server's word for one input never makes a smaller stored limit larger.

func (*Store) LearnedEmbedLimitTokens

func (s *Store) LearnedEmbedLimitTokens(model string) int

LearnedEmbedLimitTokens is the stored per-input token limit for model, 0 when none is stored.

func (*Store) LegacyAttachmentDirFor

func (s *Store) LegacyAttachmentDirFor(archivePath string) string

LegacyAttachmentDirFor is the corpus sidecar an archive's attachments used to be written to, for readers that must work before the migration has run.

func (*Store) MarkForReingest

func (s *Store) MarkForReingest(path string) error

MarkForReingest clears a document's content hash so the next ingest redoes it.

The hash is the dedup lever, so clearing it is the least destructive way to force a re-read: the document row, its fragments and its OCR page cache all survive, and the pages already transcribed are not read again.

func (*Store) MatchDocuments

func (s *Store) MatchDocuments(ref string) ([]DocRef, error)

MatchDocuments overlays: branch matches, then parent matches not shadowed.

func (*Store) MergeTags

func (s *Store) MergeTags(ctx context.Context, m TagMerge) (TagMergeResult, error)

MergeTags collapses From into To across every document in this index, re-indexing each changed document's identity fragment so the searchable text and the columns cannot disagree.

Content tags only. Roles are a closed vocabulary and cannot drift; a role outside it is a validation failure to fix at the source, not a tag to merge.

The caption is untouched, and so is its authorship: a person's identity keeps gen_source='person'. Merging a tag is not overruling anybody's caption.

func (*Store) Meta

func (s *Store) Meta(key string) (string, bool)

Meta reads a per-index setting.

func (*Store) MigrateExtractedAttachments

func (s *Store) MigrateExtractedAttachments(dryRun bool) ([]AttachmentMove, error)

MigrateExtractedAttachments moves this index's corpus-side attachment directories into raglit's storage, rewriting the document rows to match.

dryRun reports what it would do and touches nothing — the only responsible default for something that renames files in an evidence tree.

func (*Store) MissingCount

func (s *Store) MissingCount() int

MissingCount is len(MissingDocuments), cached for missingTTL; -1 when the count failed. Prune resets it.

func (*Store) MissingDocuments

func (s *Store) MissingDocuments() ([]string, error)

MissingDocuments is this layer's documents whose local source file is gone, among those read from a file (fromFileSQL).

func (*Store) MissingVectors

func (s *Store) MissingVectors() (int, error)

MissingVectors counts fragments with no vector. Only the document's own words: identity and fields fragments are about a document and are never embedded (SQLOwnWordsF).

func (*Store) NewStageLog

func (s *Store) NewStageLog(jobID int64) *StageLog

NewStageLog returns a recorder bound to a job. Returns nil for jobID ≤ 0 so callers without a job (the CLI IngestPDF path) record nothing.

func (*Store) Notes

func (s *Store) Notes(path string) ([]Note, error)

Notes returns every note on a document, oldest first.

Oldest first because a note thread is read as it was written: the later note usually answers the earlier one, and reversing that makes a correction arrive before the thing it corrects.

func (*Store) OversizedDocs

func (s *Store) OversizedDocs(limit int) (map[string]int, error)

OversizedDocs lists documents holding a fragment larger than limit, with the worst offender's size. These are the documents whose fragments were cut to a standard the embed model does not accept.

func (*Store) PageImagePath

func (s *Store) PageImagePath(path string, page int) (string, error)

PageImagePath returns the absolute path of a document page's saved image, or "" if there is none. The daemon validates the path is under the home's pages/ dir before serving it. A stored path from a home that has since been copied or moved is mapped back under this home's pages/ (rerootPageImage).

func (*Store) PageLayoutFor

func (s *Store) PageLayoutFor(docPath string, page int) (*PageLayout, error)

PageLayoutFor assembles one page's layout view.

Raw text comes from the OCR cache, which is keyed by the image's sha256 and therefore survives re-ingests, model changes and the flattener — it is the only place the model's original output is guaranteed to still exist.

func (*Store) PageReadings

func (s *Store) PageReadings(ctx context.Context, doc string, page int) ([]PageReading, error)

PageReadings returns every recorded version of one page, oldest first.

func (*Store) PagesRoot

func (s *Store) PagesRoot() string

PagesRoot is the home's pages directory (absolute), for the daemon to bound page-image serving to the home. Empty for a no-home store.

func (*Store) PagesWithLayout

func (s *Store) PagesWithLayout(docPath string) map[int]bool

PagesWithLayout reports which of a document's pages have layout blocks, so a page LIST can decide whether to offer a Layout tab without fetching each page's blocks. One query plus a memoised hash per page.

func (*Store) Path

func (s *Store) Path() string

Path is the index file path (or ":memory:").

func (*Store) Paused

func (s *Store) Paused() bool

Paused reports whether this index runs no work. Jobs still queue while it is paused, and run when it is resumed.

func (*Store) Problems

func (s *Store) Problems(ctx context.Context) ([]Problem, error)

Problems reports everything wrong with this index, worst first.

Ordered by consequence, not by table: a document that cannot be found matters more than one that was retried into place. A caller rendering only the first screen still sees the things that lose data.

func (*Store) ProposeDocType

func (s *Store) ProposeDocType(ctx context.Context, name string, goldPaths []string) (DocType, error)

ProposeDocType reads gold documents and proposes the type: how to recognise one, how to read it, and its fields.

The authoring step, and the reason document types are practical at all. Writing a JSON Schema by hand for a form you are looking at is exactly the work a model does well, and doing it from SEVERAL examples is what stops the schema being a transcription of one document's quirks.

The proposal is returned, not stored — a schema nobody read before it started filling in records is a schema nobody will trust afterwards.

func (*Store) Prune

func (s *Store) Prune(dry, deleteOriginals bool) (PruneResult, error)

Prune removes every document whose local source file is gone, with its fragments, vectors, pages, fields, caption rows and page images, and pending ingest jobs on those files, and the page images no document names. dry reports the same and removes nothing. deleteOriginals removes the dangling copies under originals/, dry or not: they are reported always and removed only when asked.

Dirs groups by the top-most directory that is gone, so a deleted worktree is one line however deep its tree was; a file deleted from a directory that still exists is counted under that directory.

func (*Store) Purge

func (s *Store) Purge(pattern string, dry bool) (PurgeResult, error)

Purge removes every document whose path matches pattern, with its fragments, vectors, pages, fields and page images, and drops pending ingest and caption jobs on matching paths. dry reports the same counts and removes nothing. Files on the user's disk are untouched; an upload's file goes with its document (RemoveDocument). Matched like an ignore pattern: against the absolute path, or the file name for a glob with no "/".

func (*Store) PurgeDocPageCache

func (s *Store) PurgeDocPageCache(ctx context.Context, path string) (int, error)

PurgeDocPageCache drops the cached OCR for every page of one document and returns how many entries it removed.

It works by rendering the document exactly as ingest does and computing the same keys, which is sound because the render is deterministic at a fixed dpi (measured, not assumed). A page whose render has changed — a different poppler, a re-exported PDF — simply will not match, and the miss is harmless: the page was going to be re-read anyway.

func (*Store) PutAttestations

func (s *Store) PutAttestations(doc string, entries []attest.Entry) error

PutAttestations replaces this document's projected rows with the log as it now stands.

Delete-then-insert rather than an incremental append, because the log is authoritative and append-only: re-projecting the whole of it is idempotent and cannot drift, while tracking a high-water mark can silently miss an entry a second machine appended between syncs. The logs are small — one asset's rulings — so the cost is not worth the class of bug.

func (*Store) ReadingFor

func (s *Store) ReadingFor(docPath string) (Reading, bool, error)

ReadingFor returns what this document is a reading of, if anything is recorded.

func (*Store) ReadingsOfSource

func (s *Store) ReadingsOfSource(sha string) ([]Reading, error)

ReadingsOfSource returns every reading of one source, oldest first.

Empty for an empty digest, deliberately: readings that could not name their source are not all readings of each other, and grouping them would assert a relationship nobody established.

func (*Store) ReclaimIdentityJobs

func (s *Store) ReclaimIdentityJobs() (int, error)

ReclaimIdentityJobs returns rows left 'running' by a process that is gone to pending, and reports how many.

Requeued rather than failed, which is the opposite of what ingest does with its orphans — and the difference is the work. An ingest job may have been killed BY its document (an OOM on a huge scan), so retrying it on every start is a crash loop. A caption is one bounded request on text already in the index: if it died, the process died, and redoing it costs one call.

func (*Store) RecordFields

func (s *Store) RecordFields(ctx context.Context, path string, f DocFields, by string) (DocFields, error)

RecordFields is a PERSON's extraction: what they say the fields are. Never regenerated over.

func (*Store) RecordIdentity

func (s *Store) RecordIdentity(ctx context.Context, path string, d DocIdentity, by string) (DocIdentity, error)

RecordIdentity is a PERSON saying what a document is. It supersedes whatever the machine said and is not regenerated afterwards.

Fields left empty keep what is already recorded, so correcting only the name does not silently blank the summary. by is recorded in Model's place — the column names who produced the claim, and for a person that is the person.

func (*Store) RecordReading

func (s *Store) RecordReading(r Reading) error

RecordReading records what a document is a reading of. Idempotent per document — a re-ingest replaces the row, because it is a new reading of the same thing.

An empty SourceSHA256 is accepted and stored. A producer that cannot say what it read is a fact worth keeping: oidio's diarized.json names no media and carries no digest, so every transcript derived from one arrives here unable to point at its recording. Refusing the row would hide that; storing it makes the gap countable and fixable.

func (*Store) RecordSourceStat

func (s *Store) RecordSourceStat(path string) error

RecordSourceStat stores the size and modified time of the file behind the document at path, now. A path with no local file is left at 0 (unknown). It does not vouch for the stored bytes, so IndexTree hashes the file next pass.

func (*Store) ReferencesTo

func (s *Store) ReferencesTo(ctx context.Context, path string) ([]Reference, error)

ReferencesTo finds fragments that CITE a path, so a withdrawal does not leave live references pointing at a document nobody can retrieve.

Matched on the path and on its file name, because a citation in this corpus is written either way — "documents/drafts/x.md" in a packet, "x.md" in a note. Fragments of the withdrawn document itself are excluded: a document citing its own name is not a dangling reference.

func (*Store) RemoveDocument

func (s *Store) RemoveDocument(path string, stored bool) error

RemoveDocument deletes a document as DeleteDocument does, and the page and figure images it made under this index's pages/. stored also deletes the document's own bytes when they are in this index's storage (an upload, an extracted attachment). A file outside the index's storage is never touched, whatever stored says. A copy under originals/ is left for prune's --delete-orphan-originals.

func (*Store) RequeueJob

func (s *Store) RequeueJob(id int64) error

RequeueJob returns a claimed-but-unstarted job to the queue.

Distinct from RetryJob, which is a person deciding to run something again: this is the scheduler putting back a row it claimed and then could not hand to a runner, because the daemon is shutting down. Without it the row says `running` forever with no process behind it — the state reclaimOrphanedJobs exists to repair, which is worth not creating on a clean exit.

Restricted to rows that never started, so it cannot rewind a job mid-ingest.

func (*Store) Restore

func (s *Store) Restore(path string) error

Restore returns a withdrawn document to the corpus. It does not re-ingest — the file has to be indexed again, which is the caller's next step.

func (*Store) RetryJob

func (s *Store) RetryJob(id int64) error

RetryJob requeues an errored or done job: state → pending, error cleared, timestamps reset, so the worker picks it up again. Errors if the job isn't in a retryable state.

func (*Store) SameBytesAs

func (s *Store) SameBytesAs(docPath string) (map[string]string, error)

SameBytesAs lists other indexed documents whose source bytes hash the same, path -> title.

Reusing documents.content_hash rather than adding anything: ingest already records it to skip re-indexing unchanged files, so exact duplication across PATHS was sitting in the index unqueried. On this corpus it names three pairs, one of which is a misfiled document — `2019-04-09-sale-disclosure-packet-COMPLETE-Form34-35-LotCert-Form17.pdf` is byte-identical to `2019-mls-listing-broker-remarks.pdf` and holds an MLS listing sheet, so the four instruments its filename claims are not in it.

A document with an empty content_hash is skipped rather than matched: ” is "not recorded", and grouping on it would report every synthetic document as a duplicate of every other.

func (*Store) SameBytesHash

func (s *Store) SameBytesHash(hash string) (map[string]string, error)

SameBytesHash lists indexed documents whose recorded source hash equals hash.

This is the upload case, and it is the cheapest true answer available: hash the file the client just sent and ask whether the index already holds those exact bytes. It runs before any extraction, so it needs no OCR, no model, and answers for a scanned PDF that raglit could not otherwise read at all — which is the one input the shingle path has to refuse.

func (*Store) SaveUpload

func (s *Store) SaveUpload(name string, data []byte) (string, error)

SaveUpload writes data as the upload called name and returns its absolute path. name is relative and slash-separated ("notes/2026/a.md"); anything that would land outside the uploads directory is refused.

func (*Store) Scopes

func (s *Store) Scopes() (scopes []string, set bool)

Scopes is this index's directory patterns. set is false when they were never written, which reads as an opt-in global index and is what the daemon's one-time migration and an ingest's scope claim look for.

func (*Store) Search

func (s *Store) Search(query string, limit int) ([]Hit, error)

Search is SearchPath with no path constraint.

func (*Store) SearchFigures

func (s *Store) SearchFigures(ctx context.Context, query string, limit int) ([]FigureHit, error)

SearchFigures is SearchFiguresPath with no path constraint.

func (*Store) SearchFiguresOwnLayer

func (s *Store) SearchFiguresOwnLayer(ctx context.Context, query, pathPrefix string, limit int) ([]FigureHit, error)

SearchFiguresOwnLayer is SearchFiguresPath over this layer's own documents only — for sibling fan-out, where the caller's layer already covers the parent.

func (*Store) SearchFiguresPath

func (s *Store) SearchFiguresPath(ctx context.Context, query, pathPrefix string, limit int) ([]FigureHit, error)

SearchFiguresPath ranks figures by cosine similarity of the query to each figure's embedding, best first, optionally constrained to a path subtree (pathPrefix). Only figures whose vector is comparable to a TEXT query are considered — the description embedding, and an aligned image embedding (nomic-vision, whose space nomic-text shares). Non-aligned image embeddings are skipped (they await an image-query path). Requires a text embedder.

func (*Store) SearchFiltered added in v0.5.0

func (s *Store) SearchFiltered(query string, f SearchFilter, limit int) ([]Hit, error)

SearchFiltered is SearchPath constrained by a SearchFilter. Each layer applies the filter in its own WHERE, before its limit. A branch's own copy of a document shadows the parent's whether or not the copy qualifies, so a parent row whose tags or fields the branch has since changed cannot answer for it.

func (*Store) SearchOwnLayer

func (s *Store) SearchOwnLayer(ctx context.Context, query, mode, pathPrefix string, limit int) ([]Hit, error)

SearchOwnLayer searches this layer's OWN documents only — not its parent. For sibling fan-out: the caller's layer already overlays the shared parent, so searching a sibling's overlay again would return the parent twice. mode is "" / "bm25", "vec" or "hybrid".

func (*Store) SearchPath

func (s *Store) SearchPath(query, pathPrefix string, limit int) ([]Hit, error)

SearchPath overlays branch-over-parent: branch hits, then parent hits whose doc is not shadowed, re-ranked by score and truncated to limit — optionally constrained to a path subtree (pathPrefix). A non-branch store (parent == nil) is just its local FTS.

func (*Store) ServesPageImage

func (s *Store) ServesPageImage(img string) bool

ServesPageImage reports whether img is a page image this store may serve: under its own pages directory, or — for a branch — under an ancestor's, for the parent documents that show through it.

func (*Store) SetAllowPatterns

func (s *Store) SetAllowPatterns(patterns []string) ([]string, error)

SetAllowPatterns replaces this index's allow list, cleaned the same way.

func (*Store) SetAmbient

func (s *Store) SetAmbient(ambient bool) error

SetAmbient makes a global index ambient (true) or opt-in (false).

func (*Store) SetAuthorityResolver

func (s *Store) SetAuthorityResolver(fn func(source string) (string, bool))

SetAuthorityResolver supplies the person's ruling about which reading of a source governs (readings.go).

INJECTED, because the rulings live in the trail beside the documents and this package must not go looking for it: an index does not know where its corpus is, and one that guessed would answer differently depending on where it was opened from. Unset → the level order decides, which is the right default for a corpus nobody has ruled on.

func (*Store) SetCaptionProfile added in v0.4.1

func (s *Store) SetCaptionProfile(profile string, now int64) error

SetCaptionProfile records the owner's choice; "" returns to the corpus's shape. It applies to captions written from now on. Existing captions change only by `raglit identify --force`.

func (*Store) SetDocType

func (s *Store) SetDocType(t DocType) error

SetDocType registers or replaces a type.

func (*Store) SetDocumentFields

func (s *Store) SetDocumentFields(ctx context.Context, path string, f DocFields) error

SetDocumentFields records an extraction and re-indexes its fragment, in one transaction — so the row and the searchable text can never disagree.

A person's extraction is what a re-run must not touch; that rule lives in the CALLER, because this is also how a correction is written.

On a branch, a document only the parent holds is an error wrapping ErrParentOnly.

func (*Store) SetDocumentHash

func (s *Store) SetDocumentHash(path, hash string) error

SetDocumentHash records a document's source-content hash after a successful ingest, so an unchanged re-ingest can be skipped.

func (*Store) SetDocumentIdentity

func (s *Store) SetDocumentIdentity(ctx context.Context, path string, d DocIdentity) error

SetDocumentIdentity records what a document is and re-indexes its identity fragment, in one transaction — so the columns and the searchable text can never disagree.

A person's identity (Source 'person') is what a re-run must not touch; that rule lives in the CALLER, because this is also how a correction is written.

func (*Store) SetEmbedder

func (s *Store) SetEmbedder(e *Embedder)

SetEmbedder enables vector search: fragments are embedded on Ingest and VecSearch/HybridSearch become available. nil disables it.

A reembed of an index with ingests in flight is, in this order: SetEmbedder with the new embedder, then SyncEmbedModel (or DropVectors), then BackfillVectors until it returns 0. An in-flight ingest that embedded with the old embedder then leaves no vectors behind (vecMu); its fragments are missing a vector, and the backfill embeds them with the new one. The other order (drop, then swap) leaves a window in which old vectors survive.

func (*Store) SetExtractEmailAttachments

func (s *Store) SetExtractEmailAttachments(v bool)

SetExtractEmailAttachments turns mail-archive attachment extraction on.

func (*Store) SetIdentifier

func (s *Store) SetIdentifier(id *Identifier)

SetIdentifier enables document identity: each ingested document is captioned, summarised and typed by the model (identity.go), and the summary is indexed. nil disables it — an ingest then leaves whatever identity the document already had, because a caption is not invalidated by re-reading the same file.

func (*Store) SetIgnorePatterns

func (s *Store) SetIgnorePatterns(patterns []string) ([]string, error)

SetIgnorePatterns replaces this index's ignore list. Blank entries and repeats are dropped. It does not touch documents already indexed; a purge does that.

func (*Store) SetImageEmbedder

func (s *Store) SetImageEmbedder(ie ImageEmbedder)

SetImageEmbedder enables IMAGE embeddings for figures (a CLIP-style tower): each figure is embedded from its image rather than its description. nil (the default) → figures embed their description with the text embedder instead.

func (*Store) SetIndexHint

func (s *Store) SetIndexHint(hint string, now int64) error

SetIndexHint records it. Empty clears it.

func (*Store) SetJobLane

func (s *Store) SetJobLane(id int64, lane Lane) error

SetJobLane records which lane a job actually belongs in. Used by the worker once routing has read the bytes and disagreed with the guess from the URL.

func (*Store) SetJobURL

func (s *Store) SetJobURL(id int64, url string) error

SetJobURL re-points a job at a new URL. Reconciliation uses it when a job's stored source path no longer resolves but the same content is indexed at another path: the job's identity (its id) is unchanged, only the pointer it follows moves, so a subsequent retry ingests the document under the job's existing identity.

func (*Store) SetMeta

func (s *Store) SetMeta(key, value string, now int64) error

SetMeta records a per-index setting.

func (*Store) SetParent

func (s *Store) SetParent(p *Store)

SetParent makes this store a branch over p (branch-over-parent overlay reads).

func (*Store) SetPaused

func (s *Store) SetPaused(paused bool) error

SetPaused pauses or resumes this index. Stored in the index, so a restart keeps it. A job already running finishes; nothing new is claimed.

func (*Store) SetScopes

func (s *Store) SetScopes(scopes []string) ([]string, error)

SetScopes replaces this index's directory patterns. Blank entries and repeats are dropped; a plain directory is cleaned. An empty list makes the index global.

func (*Store) SetTags added in v0.6.0

func (s *Store) SetTags(path string, tags []string) error

SetTags replaces the caller's tags on the document at path. An empty slice clears them. Each tag is normalized the way the model's are (normalizeContentTag) and repeats are dropped, but the model-tag rules do not apply: no length or word limit, no count, no role vocabulary. A caller's tag is its own vocabulary, "kind:strategy" included.

On a path with no document yet the tags are held, and the next ingest of that path to commit applies them, so "enqueue, then tag" works without waiting for the job. A later SetTags before that commit replaces the held ones.

On a branch, a document that only the parent holds is an error: a branch cannot edit the parent's document until copy-up exists (the same rule as SetDocumentFields). The error wraps ErrParentOnly.

func (*Store) SiblingReadings

func (s *Store) SiblingReadings(docPath string) ([]Reading, error)

SiblingReadings returns the OTHER readings of whatever this document reads — the answer to "show me the raw one" from a verified transcript, and the reverse.

func (*Store) SimilarIndexed

func (s *Store) SimilarIndexed(docPath string, opts SimilarOpts) (SimilarReport, error)

SimilarIndexed compares an already-indexed document against the rest of the index, and additionally reports any file byte-identical to it.

func (*Store) SimilarTo

func (s *Store) SimilarTo(probePath string, probe FoldedText, opts SimilarOpts) (SimilarReport, error)

SimilarTo compares folded probe text against the index and returns the report.

Takes already-folded text rather than a path so the caller decides where the probe came from — an indexed document, a file on disk, or a raglit transcription sidecar. That mattered: making this open the file itself meant a probe could only be something raglit could extract, and the upload flow needs to compare text it already has in hand without writing it anywhere first.

func (*Store) SketchAll

func (s *Store) SketchAll(w, mod int) (done int, errs []error)

SketchAll builds sketches for every document that has none under the current recipe. Returns how many documents were sketched. A document whose text cannot be read is skipped with its error collected rather than aborting the run: one unreadable document must not leave the rest of the corpus unchecked, which is the state in which `similar` silently reports "nothing found".

func (*Store) SketchDoc

func (s *Store) SketchDoc(docPath string, w, mod int) (int, error)

SketchDoc builds (or rebuilds) one document's page sketches from the index's own text.

Reads the document through TruePages, so a page number in a later alignment is the page the text is actually on, not the page its fragment opened on. Replaces the document's rows wholesale: an incremental update would have to know which shingles a re-OCR'd page lost, and a re-read changes a page completely often enough that the bookkeeping would cost more than the rewrite.

func (*Store) SketchStatusFor

func (s *Store) SketchStatusFor(w, mod int) (SketchStatus, error)

SketchStatusFor reports coverage under a recipe.

func (*Store) SketchedPaths

func (s *Store) SketchedPaths(w, mod int) ([]string, error)

SketchedPaths lists documents carrying a sketch under this recipe, in a stable order. The all-pairs audit walks these and nothing else — comparing a document with no sketch would silently fall back to zero candidates and report it clean.

func (*Store) StoragePath

func (s *Store) StoragePath(path string) (string, bool)

StoragePath is where a storage document's bytes are on THIS machine: the file under this store's home (or its parent's, for a branch) that StorageRel names. Empty and false for a corpus document, for a path that would leave the home, and for a file that is not there.

func (*Store) StorageRel

func (s *Store) StorageRel(path string) (string, bool)

StorageRel reports whether a document's bytes live in raglit's own storage for this index (extracted attachments, uploads, transcripts) rather than in the corpus, and if so its path relative to that storage.

Such documents are real documents, but their directory is raglit's business: letting ~/.raglit/indexes/<name>/attachments/... take part in the corpus's common directory widened one legal corpus's root to the user's home, which put /home/<user>/... in front of every path and made the review list walk the whole home directory.

Two matches. Under this store's home is the ordinary case. The second is a home that has been COPIED or moved: the rows still carry the path of the home they were written in, and the only stable part of it is the ".../indexes/<this index>/" layout. A branch also asks its parent, whose documents show through it with the parent's home in their paths.

func (*Store) SupersededPages

func (s *Store) SupersededPages(ctx context.Context) ([]PageReading, error)

SupersededPages lists documents holding a page whose reading was replaced — the corpus's record of where a machine read was found wanting.

func (*Store) SupplyIdentity added in v0.4.0

func (s *Store) SupplyIdentity(path string, d DocIdentity, by string) error

SupplyIdentity records the caption a document will carry when its next ingest commits: name required, summary cut to identityMaxSummaryChars (no minimum: a supplied summary is what the caller had), recorded as IdentitySupplied by `by`. It is consumed by that commit.

func (*Store) SyncEmbedModel

func (s *Store) SyncEmbedModel(model string, dim int) (dropped bool, err error)

SyncEmbedModel records model and dim as the source of this index's vectors and reports whether existing vectors were dropped because they came from another model or have another length. An index with no model record adopts model as-is: it is an index from before the record existed, whose vectors are from the model configured when they were made, and re-embedding every index on upgrade would cost the whole corpus for nothing.

dim 0 means the caller does not know it: the dimension is then neither compared nor recorded. With no dim record, the length of a stored vector stands in for one, so an index from before the record still drops vectors that the configured model can no longer match.

func (*Store) TagContext

func (s *Store) TagContext() string

TagContext is this index's established tag vocabulary, as one line for the identity prompt — the mechanism that keeps "lead paint" from drifting into "LBP", "paint inspection" and "lead-based paint hazards" across a corpus. Empty for a fresh index, and empty rather than an error when the read fails: a caption without the vocabulary is worse than one with it, not a failure.

func (*Store) TagDocument

func (s *Store) TagDocument(ctx context.Context, path string, force bool) (DocIdentity, error)

TagDocument fills in a document's tags WITHOUT touching its caption.

The caption's authorship is preserved exactly — a person's stays a person's, a machine's keeps the model and timestamp that wrote it — because tags are not a re-reading of the document, they are two columns that were never asked for. Returns ErrIdentityKept when the document already has tags, unless forced.

func (*Store) TagNeighbours

func (s *Store) TagNeighbours() (map[string][]string, error)

TagNeighbours reports, for each content tag, the other tags that share a significant word with it — the drift a person is being asked to rule on.

A LEXICAL proposal and nothing more. It cannot see that "escrow closing" and "settlement" are the same thing, and it will pair "lead paint" with "lead paint disclosure", which are not. That is why nothing here writes.

func (*Store) Tags added in v0.6.0

func (s *Store) Tags(path string) ([]string, error)

Tags returns the caller's tags on the document at path, in the order SetTags stored them. A document with none returns nil. An unknown path is an error; tags still held for it are not reported.

A branch with a root (OpenBranch) reads a parent-only document's tags through to the parent. A branch with no root does not.

func (*Store) TranscriptDirFor

func (s *Store) TranscriptDirFor(mediaPath string) string

TranscriptDirFor is where this index stores the transcript and machine reading of the recording at mediaPath. Empty when the store has no home, the same case AttachmentDirFor cannot serve — an in-memory index has nowhere to put a sidecar, and the caller records that as a skipped stage rather than a failure.

func (*Store) TruePages

func (s *Store) TruePages(exactPath string) ([]PageText, error)

TruePages reassembles a document at TRUE page grain, honouring page_spans.

DocText's Pages array is not this, and the difference is the reason this exists. The assembler absorbs a page's continuation (or a sub-floor sibling) from the NEXT page into an open fragment and keeps only the START page in `fragments.page`, so DocText attributes every byte of a multi-page fragment to the page the fragment opened on. Measured on this index: 67 of 460 fragments carry page boundaries, and honouring them recovers 441 true pages where grouping by `fragments.page` finds only 277. Dace's declaration is the clearest case — 14 fragments opening on pages 1, 4, 14, 17, 18, 23, 24, 28 and 33, so DocText reports it as a 9-page document and every page number after the first stitch is wrong. That is exactly wrong for anything that wants to SAY where a finding is.

page_spans was added to fix that ("a search hit inside one could not be resolved to the page it is actually on", per the schema) and, until this, was written by ingest, carried through the pool, and read by nothing but its own tests. Similarity is its first consumer: "pages 12-14 of X are pages 1-3 of Y" is the whole deliverable, and a page number that is silently the fragment's start page makes that report a lie.

Pages come back in page order, contiguous from the lowest page present, with empty pages included — a consumer resolving an alignment to a page depends on the numbering being complete, the same contract the transcription markdown makes. Raw SQL, not sqlc, and not by preference. The generated ListFragmentsForDoc does not select page_spans, and regenerating is currently destructive: `sqlc generate` with the installed toolchain (sqlc v1.30.0 against the metaquery plugin built on plugin-sdk-go v1.23.0) rewrites the SQL text of EVERY query in internal/db — each one comes back rotated two characters, e.g.

const cancelJob = `-- name: CancelJob :execrows
);

DELETE FROM ingest_jobs WHERE id=? AND state='pendin`

That is 60+ silently broken queries for the sake of one added column, so this follows the precedent the schema file already sets for FTS5 and the vector scan: what sqlc cannot express here stays raw. Fix the plugin, then fold this back in.

func (*Store) VecSearch

func (s *Store) VecSearch(ctx context.Context, query string, limit int) ([]Hit, error)

VecSearch is VecSearchPath with no path constraint.

func (*Store) VecSearchFiltered added in v0.5.0

func (s *Store) VecSearchFiltered(ctx context.Context, query string, f SearchFilter, limit int) ([]Hit, error)

VecSearchFiltered is VecSearchPath constrained by a SearchFilter, applied in each layer's scan before the ranking and the limit.

func (*Store) VecSearchPath

func (s *Store) VecSearchPath(ctx context.Context, query, pathPrefix string, limit int) ([]Hit, error)

VecSearchPath embeds the query and ranks fragments by cosine similarity, best first, optionally constrained to a path subtree (pathPrefix). Brute-force: it scans every stored vector in scope (fine for a local corpus; see embed.go). Requires SetEmbedder. Score is cosine in [-1,1] (higher = better). Fragments without a vector (indexed before embeddings were enabled) are invisible.

func (*Store) Withdraw

func (s *Store) Withdraw(w Withdrawal) error

Withdraw removes a document from this index and records why.

Idempotent: withdrawing an already-withdrawn path refreshes the grounds and removes any rows a re-ingest slipped in before the decision was made.

func (*Store) Withdrawals

func (s *Store) Withdrawals() ([]Withdrawal, error)

Withdrawals lists every ruling, path order.

func (*Store) WithdrawnReason

func (s *Store) WithdrawnReason(path string) (string, bool)

WithdrawnReason reports why a path is out of the corpus, if it is.

func (*Store) WriteIndexAbout

func (s *Store) WriteIndexAbout(ctx context.Context) (string, error)

WriteIndexAbout asks the model what this index is and stores the answer.

Uses the identity model, because it is the one already configured to read this corpus and the task is the same task one level up.

type StoreSet added in v0.5.0

type StoreSet interface {
	Names() []string
	// Existing returns the open store for name, or an error when there is none.
	// It must not create an index.
	Existing(name string) (*Store, error)
}

StoreSet is the indexes a Runner works: their names, and the open store for a name. It is read again on every pass, so a set that gains an index while the Runner runs (a Registry does, when an index is created) has that index's queue worked from the next pass on. *Registry satisfies it.

type StrategyConfig

type StrategyConfig struct {
	// Descend is RegionReader.MaxDepth. 0 reads the sheet and stops, which is
	// what every page does today.
	Descend int `json:"descend,omitempty"`
	// Tile subdivides a large low-resolution DRAWING geometrically instead of
	// asking the model where to look — arithmetic instead of a call, and on an
	// E-size sheet the asking is the part that does not work.
	Tile bool `json:"tile,omitempty"`
	// Hint is threaded into every prompt. Cheapest lever here by a distance: the
	// model proposing regions is looking at a view where the thing you want may
	// be physically unresolvable, so it cannot propose what it cannot see. A
	// sentence naming the target beats any amount of extra budget.
	Hint string `json:"hint,omitempty"`

	// Budgets. Zero → RegionReader's own defaults (8 / 2 / 40 / 4).
	MaxCalls       int     `json:"max_calls,omitempty"`
	MaxChildren    int     `json:"max_children,omitempty"`
	MaxTransforms  int     `json:"max_transforms,omitempty"`
	MaxEscalations int     `json:"max_escalations,omitempty"`
	MinRegionIn    float64 `json:"min_region_in,omitempty"`

	// Render overrides the automatic per-page resolution policy.
	Render RenderPolicy `json:"render,omitempty"`

	// AutoDescend descends WITHOUT being asked, but only for a page the pixels
	// say needs it: the same low-resolution test the descent already applies
	// internally (below half the letter-page token density). Measured over a 998
	// page corpus that is 14 pages in 11 documents — 1.4% — so it is affordable
	// in a way "descend everything" is not, and it targets exactly the sheets a
	// single-shot read fails on. Off by default; Descend still bounds it.
	AutoDescend bool `json:"auto_descend,omitempty"`
}

StrategyConfig is how much work a page is worth.

Every field is zero-valued to today's behavior, so adding this section changes nothing until something is set. That matters more than brevity here: this governs model spend, and a config format whose defaults are not the current behavior turns an upgrade into a bill.

type TagCount

type TagCount struct {
	Tag   string `json:"tag"`
	Count int    `json:"count"`
}

TagCount is one tag or kind and how many documents carry it.

type TagMerge

type TagMerge struct {
	From []string `json:"from"`
	To   string   `json:"to"`
}

TagMerge is one collapse: every tag in From becomes To.

func ParseTagMerge

func ParseTagMerge(spec string) (TagMerge, error)

ParseTagMerge reads the "a,b=>c" form: the tags to collapse, then the tag to collapse them into. Whitespace around any term is ignored.

func (TagMerge) String

func (m TagMerge) String() string

String renders a merge back in the form ParseTagMerge reads.

type TagMergeResult

type TagMergeResult struct {
	To        string   `json:"to"`
	From      []string `json:"from"`
	Documents int      `json:"documents"` // documents whose tags changed
	Collapsed []string `json:"collapsed"` // the From tags actually present
	Missing   []string `json:"missing"`   // From tags no document carried
}

TagMergeResult is what a merge did.

type TesseractEngine

type TesseractEngine struct {
	Bin  string // tesseract binary; "" → "tesseract"
	Lang string // -l language; "" → "eng"
}

TesseractEngine runs the tesseract CLI over each page: no cgo, no daemon, one optional system dependency. Fast and ~free relative to a vision LLM, but blind to handwriting and stylized figures — which is exactly what the gibberish gate exists to catch and escalate.

func NewTesseractEngine

func NewTesseractEngine(bin, lang string) *TesseractEngine

NewTesseractEngine builds a TesseractEngine with defaults filled.

func (*TesseractEngine) Name

func (e *TesseractEngine) Name() string

func (*TesseractEngine) OCRPage

func (e *TesseractEngine) OCRPage(ctx context.Context, img PageImage) (PageOCR, error)

OCRPage pipes the page image to `tesseract stdin stdout -l <lang> tsv` and parses the TSV (per-word confidence + line grouping). A run error propagates so the cascade falls back to the vision model rather than dropping the page.

type TokenBudget

type TokenBudget struct {

	// Overhead is text the CONSUMER prepends that a count of the fragment cannot
	// see. The embedder sends DocPrefix + fragment ("search_document: " for
	// nomic), so a fragment sized to the limit exactly is over it by the prefix.
	Overhead string
	// Reserve covers what the tokenizer adds around the input and does not report
	// — the BOS/EOS pair, counted because add_special is off when measuring a
	// piece of a larger prompt.
	//
	// Both were learned the same way: a fragment sized to 8192 was refused at
	// "input (8194 tokens) is too large to process". Two tokens. A budget that
	// misses by two is not a budget, and it fails the document exactly as
	// completely as one that misses by two thousand.
	Reserve int
	// contains filtered or unexported fields
}

TokenBudget sizes text against a token limit for one document.

Exact where it can be: with a tokenizer, a prefix is measured rather than predicted, and the learned ratio only chooses where to start looking so the search converges in a call or two instead of a binary search.

func NewTokenBudget

func NewTokenBudget(ctx context.Context, counter TokenCounter, limitTokens int) *TokenBudget

NewTokenBudget builds a budget over an optional counter. A nil counter, or a limit of 0, yields a budget that never cuts.

func (*TokenBudget) Fit

func (b *TokenBudget) Fit(ctx context.Context, overhead, rest string, minTake int) int

Fit returns how many characters of `rest` can be sent alongside `overhead` characters of prompt without exceeding the limit — len(rest) when it all fits.

With a tokenizer this MEASURES rather than predicts: the ratio picks a starting guess and each check corrects it, so a page of prose and a page of coordinates both converge in a call or two. The alternative — one ratio for the whole corpus — is what put nine documents past the limit and cut the rest to a fifth of what the endpoint would have taken.

minTake guarantees progress: when the overhead alone leaves no room, cutting to fit is not available and a zero-length take would never terminate.

func (*TokenBudget) FitOne

func (b *TokenBudget) FitOne(ctx context.Context, rest string, minTake int) int

FitOne is Fit against the budget's own Overhead — the form the fragment splitters use, where the overhead is a property of the consumer rather than of the call.

func (*TokenBudget) FitSuffix

func (b *TokenBudget) FitSuffix(ctx context.Context, s string, room int) int

FitSuffix returns how many characters of the END of `s` fit in `room` tokens.

The mirror of Fit, and it exists because the two cuts protect different things. A take keeps the FRONT of the remaining page, because that is what comes next. A carried-over fragment keeps its TAIL, because continuation is decided by how it ends.

func (*TokenBudget) Fits

func (b *TokenBudget) Fits(ctx context.Context, text string) bool

Fits reports whether text is inside the budget once the consumer's overhead is counted. The check every splitter shares, so they agree about what fits.

func (*TokenBudget) Tokens

func (b *TokenBudget) Tokens(ctx context.Context, text string) int

Tokens counts text, exactly when possible and by the learned ratio otherwise.

func (*TokenBudget) Unlimited

func (b *TokenBudget) Unlimited() bool

Unlimited reports whether this budget will never cut.

type TokenCounter

type TokenCounter interface {
	CountTokens(ctx context.Context, text string) (int, bool)
}

TokenCounter counts tokens as the serving model does. Implemented by *llm.Client; ok=false means this endpoint cannot count, which is a fact to work around rather than an error.

type ToolEnv

type ToolEnv struct {
	// Who describes the process this was measured in — "shell" or "daemon (pid
	// 1234)". Set by whoever renders it; a probe cannot know what it will be
	// called.
	Who string `json:"who"`
	// PATH as this process has it. The single most useful line when two probes
	// disagree, and the reason a tool "is not installed" when it is.
	PATH string `json:"path_env"`
	// Tools, in a stable order so two environments can be diffed line by line.
	Tools []ToolStatus `json:"tools"`
	// GOOS, because an install hint for apt is noise on a mac.
	GOOS string `json:"goos"`
}

ToolEnv is the whole external-tool picture for one process.

func ProbeTools

func ProbeTools() ToolEnv

ProbeTools measures this process's view of the external extractors.

Deliberately does NOT run each tool's --version: this is called on an HTTP path that an operator may poll, LookPath is a stat and a version probe is a fork, and "which binary would we run" is the question that was actually being got wrong.

func (ToolEnv) Disagreements

func (e ToolEnv) Disagreements(other ToolEnv) []string

Disagreements lists the tools two environments see differently — missing in one, or resolving to different binaries.

Both halves matter. "Present here, absent there" is the failure that started this. "Present in both, different paths" is quieter and worse: two pandocs of different vintages read the same .docx into different text, and nothing in an index records which one ran.

func (ToolEnv) PathDiff

func (e ToolEnv) PathDiff(other ToolEnv) (onlyMine, onlyOther []string)

PathDiff reports PATH entries one environment has and the other does not, which is nearly always the CAUSE when Disagreements is non-empty.

type ToolStatus

type ToolStatus struct {
	// Name is what the router looks for ("pandoc"), not the package that ships it.
	Name string `json:"name"`
	// Found is whether it resolved on PATH.
	Found bool `json:"found"`
	// Path is where it resolved, empty when not found. Reported because two
	// processes can both find a tool and find DIFFERENT ONES.
	Path string `json:"path,omitempty"`
	// Purpose is what stops working without it, in the operator's terms.
	Purpose string `json:"purpose"`
	// Optional marks a tool whose absence degrades rather than breaks.
	Optional bool `json:"optional"`
	// Install is the hint printed when it is missing.
	Install string `json:"install,omitempty"`
}

ToolStatus is one external tool as this process sees it.

type TranscribedFigure

type TranscribedFigure struct {
	Kind        string
	Description string
}

TranscribedFigure is a described diagram or image on a page. A scanned survey is mostly figure, and a transcription that drops it says the page is blank.

type TranscribedPage

type TranscribedPage struct {
	Page    int
	Text    string
	Figures []TranscribedFigure
	// Corrected is a person's replacement for this page, when one exists. Set by
	// RenderTranscriptionCorrected; never read off disk.
	Corrected PageCorrection
}

TranscribedPage is one page of a transcription.

type TranscriptMatch

type TranscriptMatch struct {
	Transcript string  `json:"transcript"`
	Recording  string  `json:"recording,omitempty"`
	Source     string  `json:"source,omitempty"`
	Score      float64 `json:"score"`
	// Trust is what the adopted reading would be worth, per facet.
	Trust []string `json:"trust,omitempty"`
	// Why is set when nothing was adopted, and says which of the two failure
	// shapes it was: nothing close enough, or nothing to compare against.
	Why string `json:"why,omitempty"`
}

TranscriptMatch is one verified transcript and the recording it belongs to.

type TranscriptionChange

type TranscriptionChange struct {
	Doc  string
	Page int
	// Active is the reading now in force.
	Active PageCorrection
	// Superseded is the text it replaced — a machine read, or an earlier
	// correction. Empty when this is the first recorded reading for the page.
	Superseded string
}

TranscriptionChange reports that a page's ACTIVE reading changed.

The event exists because a correction invalidates things it does not own. The .raglit-transcription.md export beside the document still shows the old text; the index still holds it in fragments, so a search still returns the reading a person just replaced. Neither notices on its own.

Superseded carries what was replaced rather than only what won. A correction is an attestation that a better reading exists, not an erasure: the old text is what the index held when a document was cited, and it is what a stale quotation elsewhere will still match.

type TreeOptions added in v0.9.0

type TreeOptions struct {
	Rules TreeRules
	// Paths: check only these (absolute, or relative to the root). A directory
	// covers what is under it; a path that no longer exists counts its
	// documents as Removed. Empty = the whole tree.
	Paths []string
	// Route picks the index (a key of IndexTrees' map) for a file. Optional
	// with one store.
	Route func(path string) string
	// Keep spares a document under the root that the rules do not select:
	// one whose path (absolute, no scheme) Keep reports true for is not
	// Removed. For roots that overlap in one index, each with its own rules:
	// Keep is "another root selects it".
	Keep func(path string) bool
	// Verify hashes every file. By default a file whose size and modified
	// time (ns) equal the stat recorded when its bytes were last read is
	// Current without being read (srcstat.go). The trade-off: a change that
	// keeps both the size and the mtime (a tool that restores the mtime, a
	// clock set back) is missed until the stat changes or a Verify pass. A
	// file modified within 2 s of its stat is always hashed.
	Verify bool
}

TreeOptions are IndexTree's options.

type TreeResult added in v0.9.0

type TreeResult struct {
	Queued, Current, Removed, Failed, Skipped int
	Errors                                    []error
}

TreeResult counts what IndexTree did per file. Errors are per-file failures (a read, an enqueue); the run goes on past them.

func IndexTree added in v0.9.0

func IndexTree(ctx context.Context, st *Store, root string, o TreeOptions) (TreeResult, error)

IndexTree brings st in step with the files under root. See TreeOptions.

func (*TreeResult) Add added in v0.9.0

func (r *TreeResult) Add(o TreeResult)

Add sums o into r.

type TreeRules added in v0.9.0

type TreeRules struct {
	// Include globs, relative to the root; empty = every file. A glob with no
	// "/" matches the file name, as in PlanSources.
	Include []string
	// Ignore globs, relative to the root, on top of the built-in ignores.
	Ignore []string
	// Gitignore: nil or true = honor .gitignore (git ls-files) in a git work
	// tree; false = walk the directory.
	Gitignore *bool
	// Skip drops an absolute path (a file, or for WatchTree a directory).
	Skip func(path string) bool
	// MaxBytes: a larger file counts as Skipped. 0 = 2 MiB; negative = no
	// limit. A document whose file grew past the limit is kept, not Removed.
	MaxBytes int64
}

TreeRules choose the files of a tree. IndexTree and WatchTree share them.

func (TreeRules) Admits added in v0.9.0

func (r TreeRules) Admits(root, path string) bool

Admits reports whether the path rules (built-in ignores, Ignore, Include, Skip) select the file at path under root. Not .gitignore, and not the file's size or bytes. False outside root.

type TreeWatcher added in v0.9.0

type TreeWatcher struct {
	// contains filtered or unexported fields
}

TreeWatcher watches one root. Make one with WatchTree.

func WatchTree added in v0.9.0

func WatchTree(ctx context.Context, root string, r TreeRules, o WatchOptions, fn func(Changes)) (*TreeWatcher, error)

WatchTree watches the directories under root that r does not exclude (never .git, node_modules or a gitignored directory), adding directories as they appear, and calls fn with one Changes per burst. fn runs on the watcher's goroutine, one call at a time. The watcher stops on Stop, on ctx's end, or after o.Idle.

func (*TreeWatcher) Rescan added in v0.9.0

func (w *TreeWatcher) Rescan()

Rescan reports everything selected, as one burst (Changes with no Paths), as soon as MinInterval allows.

func (*TreeWatcher) Root added in v0.9.0

func (w *TreeWatcher) Root() string

Root is the watched directory, absolute and clean.

func (*TreeWatcher) Stop added in v0.9.0

func (w *TreeWatcher) Stop()

Stop ends the watcher. It only signals and never waits, so it is safe from inside fn and from a goroutine fn waits on: the watcher ends once a callback in progress returns, and calls fn no more. Wait waits for that.

func (*TreeWatcher) Touch added in v0.9.0

func (w *TreeWatcher) Touch()

Touch resets the idle timer.

func (*TreeWatcher) Wait added in v0.9.0

func (w *TreeWatcher) Wait()

Wait blocks until the watcher has ended (Stop, ctx's end, or Idle) and its callback in progress, if any, has returned. Wait must not be called from inside fn, or from a goroutine fn waits on: it would wait for itself.

type VectorClient

type VectorClient interface {
	Embed(ctx context.Context, model string, input []string) ([][]float32, error)
}

VectorClient is the sliver of *llm.Client the embedder needs. An interface so tests supply deterministic vectors without a network.

type Visibility

type Visibility string

Visibility says who may read an index.

const (
	Private Visibility = "private"
	Public  Visibility = "public"
)

type WatchHub added in v0.9.0

type WatchHub struct {
	// contains filtered or unexported fields
}

WatchHub shares watchers and index runs across callers (R12e): one TreeWatcher per root however many subscribe, and one index run per root at a time.

func NewWatchHub added in v0.9.0

func NewWatchHub(o WatchOptions) *WatchHub

NewWatchHub makes a hub whose watchers use o.

func (*WatchHub) Index added in v0.9.0

func (h *WatchHub) Index(ctx context.Context, key string, do func(context.Context) (TreeResult, error)) (TreeResult, error)

Index runs do for key (a root, or any string naming one tree), one run at a time. A call while no run is going starts one and gets its result. A call during a run joins the one follow-up run that starts when it ends: every call that arrives during the same run shares that follow-up and its result, and the follow-up runs the do of the last of them. So do must cover everything asked so far (drain a set of pending paths, say), not only its own caller's request. A caller whose ctx ends stops waiting; the run goes on, under a context that keeps ctx's values and is cancelled only by Stop.

func (*WatchHub) Stop added in v0.9.0

func (h *WatchHub) Stop()

Stop stops every watcher and cancels the index runs in flight. Subscriptions end; a later Subscribe or Index fails. Stop only signals and never waits, so it is safe from inside a callback or an Index do; Wait waits.

func (*WatchHub) Subscribe added in v0.9.0

func (h *WatchHub) Subscribe(root string, r TreeRules, fn func(Changes)) (unsubscribe func(), err error)

Subscribe calls fn with every burst under root. The first subscriber of a root starts its watcher; the watcher skips only what the built-in rules and that subscriber's Gitignore setting skip, and each subscriber's Ignore, Include and Skip then filter the paths it is given (a burst left with no paths is not delivered, unless it is a whole-tree or Degraded one). The last unsubscribe stops the watcher. unsubscribe only signals and never waits, so it is safe from inside fn and from a goroutine fn waits on: once it returns, no new call of fn starts, but a call in progress may still be running (Stop and Wait wait for it). A watcher that idles out (WatchOptions Idle) drops its subscribers; the next Subscribe starts a new one.

func (*WatchHub) Wait added in v0.9.0

func (h *WatchHub) Wait()

Wait blocks until every watcher the hub started has ended, with its callback in progress, and every index run has returned. Call it after Stop; before Stop it waits for watchers to idle out, and must not race a Subscribe. Wait must not be called from inside a callback or an Index do, or from a goroutine one waits on: it would wait for itself.

type WatchOptions added in v0.9.0

type WatchOptions struct {
	// Quiet: a burst ends after this long with no event. Default 500 ms.
	Quiet time.Duration
	// MaxDelay: a burst ends this long after its first event, even if events
	// keep coming. Default 5 s.
	MaxDelay time.Duration
	// MinInterval: at most one burst per this long. Default 2 s.
	MinInterval time.Duration
	// Idle: the watcher stops itself after this long with no Touch and no
	// event. 0 = never.
	Idle time.Duration
}

WatchOptions tune a watcher's debounce. A zero field takes its default; a negative MinInterval means none.

type Withdrawal

type Withdrawal struct {
	Path   string `json:"path"`
	Reason string `json:"reason"`
	By     string `json:"by,omitempty"`
	// At is when the decision was made, which is not necessarily when it was
	// appended — the same distinction AuditEvent.At draws.
	At string `json:"at,omitempty"`
}

Withdrawal is a document ruled OUT of the corpus, and why.

Removing a document from an index is a delete; withdrawing it is a decision, and the difference is the reason. A delete leaves nothing behind, so the next sweep re-ingests the file and the next reader wonders why a document that plainly exists is not in the index. A withdrawal says what was decided, so it survives re-ingest and answers the question.

The case it was written for: a corpus of legal evidence that had absorbed the drafts folder — outreach letters, settlement proposals, argument written by the party themselves. A search for "the Brannocks acted in good faith" hit a draft letter and returned it looking like a source. A draft is argument, not record; only a letter actually SENT and declared is evidence of anything.

type Worker

type Worker struct {
	Store *Store
	// OCR ingests PDF jobs. nil → a PDF job fails with a clear message (a text
	// corpus needs no vision model).
	OCR *OCR
	// STT transcribes audio jobs, by asking oidio. nil → an audio job fails with
	// a clear message, the same way a PDF job does with no vision model: a corpus
	// of documents needs no transcriber and should not be told to configure one.
	STT *STT
	// Segmenter LLM-segments a document ONLY when a page escalated to the VLM (the
	// llm-seg path, decided per-document in ingestUnits). Text/code never escalate,
	// so they take the deterministic overlapping-window fragmenter regardless of
	// whether a model is configured. nil → PDF/image llm-seg unavailable.
	Segmenter *Segmenter
	// Code cuts source files at declarations and Markdown at headings
	// (KindCode, codesegment.go). nil → those files are text, read into
	// windows as before.
	Code CodeSegmenter
	// Frag tunes the deterministic text fragmenter (window/stride/floor) and caps
	// the fragment ceiling by the embed model's probed input limit. Zero → defaults.
	Frag FragConfig
	// Fetcher overrides URL resolution (tests). nil → Fetch (file://, http(s)://).
	Fetcher func(ctx context.Context, url string) (Fetched, error)
	// IdlePoll is how long Run waits when the queue is empty. Default 500ms.
	IdlePoll time.Duration
	// Pool + RecipeHash enable cross-index dedup of INDEXING work: a fresh ingest
	// is cached in the pool keyed by (RecipeHash, source-file hash); a matching
	// key — from ANY index, or a retry — is reused (fragments + vectors + images
	// copied in) instead of re-running the LLM. RecipeHash captures the models +
	// config that shape the output, so alt models are a new key. nil Pool →
	// per-index dedup only (content_hash).
	Pool       *Pool
	RecipeHash string
	// Retries collects what the LLM client had to survive for the job in flight,
	// recorded as a stage row when it is not empty. nil → retries stay invisible,
	// which is what they were.
	Retries *RetryTally
}

Worker drains the ingest queue (queue.go): claim a pending job → fetch its URL → turn it into a Document (plain-text fragmenting, or PDF → OCR) → Ingest → mark done/error. Run it in the background of `serve`, or step it once from the CLI. A per-URL failure is recorded on the job, not fatal — one bad URL never stops the worker.

func NewWorker added in v0.5.0

func NewWorker(store *Store, m Models, pool *Pool) *Worker

NewWorker wires a Worker with OCR (PDF) + LLM segmentation (text/code) when a vision model is configured; text-window sizing is resolved lazily on the first text job (probing + caching the model's context). No model → offline blank-line text + PDF-fails-gracefully.

It also sets the store's identifier. pool, when non-nil, enables the cross-index pool (daemon only). m.Channels must be set (Models.Channels).

func (*Worker) Drain

func (w *Worker) Drain(ctx context.Context) (int, error)

Drain processes jobs until the queue is empty, returning how many it handled.

func (*Worker) ProcessJob

func (w *Worker) ProcessJob(ctx context.Context, job *Job) error

ProcessJob runs a job that has ALREADY been claimed.

Split from ProcessOne so a scheduler can separate the two halves. Claiming is a transaction; running is minutes of OCR. The lane dispatcher claims in a round-robin over every index — which is what makes it fair — and hands the job to whichever slot is free, so one long job occupies a slot instead of the whole queue. ProcessOne stays the claim-and-run pair the CLI drain and the tests use.

func (*Worker) ProcessOne

func (w *Worker) ProcessOne(ctx context.Context) (processed bool, err error)

ProcessOne claims and processes a single job. processed is false when the queue is empty. A fetch/ingest failure is recorded on the job and returns (true, nil); a returned error is infrastructure (db) failure only.

func (*Worker) Run

func (w *Worker) Run(ctx context.Context)

Run drains the queue forever, sleeping IdlePoll between empty polls, until the context is canceled. This is the background loop `serve` starts.

Directories

Path Synopsis
Package attest is the machine-read-plus-human-verdict framework that oidio, raglit and kgraph each built separately and none of them built completely.
Package attest is the machine-read-plus-human-verdict framework that oidio, raglit and kgraph each built separately and none of them built completely.
Package client is raglit's Go client: how another program asks raglit about raglit's data.
Package client is raglit's Go client: how another program asks raglit about raglit's data.
cmd
raglit command
Command raglit is a document index and search service for LLM sessions: BM25 and vector search over SQLite, at document:page:fragment grain.
Command raglit is a document index and search service for LLM sessions: BM25 and vector search over SQLite, at document:page:fragment grain.
internal
db
update
Package update is how a release build of raglit gets its next build: a manifest of downloads (URL, sha256 and size per os/arch) signed with minisign, an installer that verifies a download before anything runs it, and the daily check that stages a new build for the daemon to move to.
Package update is how a release build of raglit gets its next build: a manifest of downloads (URL, sha256 and size per os/arch) signed with minisign, an installer that verifies a download before anything runs it, and the daily check that stages a new build for the daemon to move to.
tools
mkmanifest command
mkmanifest writes a release's raglit-manifest.json (make release-manifest): one raglit entry per archive in -dist (raglit_<os>_<arch>.tar.gz, sha256 and size computed here, URL -url/<archive>), and key_next from -key-next when that file exists.
mkmanifest writes a release's raglit-manifest.json (make release-manifest): one raglit entry per archive in -dist (raglit_<os>_<arch>.tar.gz, sha256 and size computed here, URL -url/<archive>), and key_next from -key-next when that file exists.
notices command
Command notices checks the licenses of everything a raglit release ships and writes the third-party notices that go in the release tarball.
Command notices checks the licenses of everything a raglit release ships and writes the third-party notices that go in the release tarball.
Package web carries the built review SPA.
Package web carries the built review SPA.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL