Documentation
¶
Overview ¶
Package raglit is a local, composable document RAG index.
The whole index is ONE portable SQLite file. SQLite's FTS5 extension gives BM25 lexical ranking built in, so "BM25" and "the document:page:fragment index" collapse into a single dependency — modernc.org/sqlite, which is pure-Go (no CGo) and thus builds to a single static binary. That is the point of raglit: a tool small enough to drop into any workflow (index a folder, grep it semantically) that scales up to a real service by swapping the agent.DocFinder impl (see finder.go) for a remote one — no rewrite.
Grain: documents → fragments(page, ord, text). A "fragment" is one indexable unit (a paragraph, a chunk, an OCR'd page region); page + ord locate it back in the source so a hit is a precise citation, not just "somewhere in the PDF".
Vectors are deliberately absent in v1. FTS5 lexical BM25 is the floor; a vector sidecar (sqlite-vec, or a custom NSW file) is added only if lexical recall proves insufficient — measured, not assumed.
Embedder surface ¶
A program may embed this package as a library (dund does). The surface it may rely on is:
- OpenHome and OpenIndex (with Home); StoreSet; Models (OnUsage, Code), NewWorker, NewChannels, NewEmbedder (with VectorClient), NewIdentifier, Models.Identifier, NewSegmenter (Segmenter.SegmentText, SegResult.Degraded).
- Code index: CodeSegmenter and CodeSpan (Models.Code, nil by default); Fragment.StartLine, EndLine and Symbol; Hit.StartLine, EndLine, Symbol and ContentID; Store.FragmentExists(contentID).
- Store ingest: Enqueue, Ingest (Document, Fragment), SetEmbedder, SetIdentifier, SetTags (caller-owned tags, also on a path not yet ingested).
- Store search: Search, VecSearch, HybridSearch and their Path and Filtered variants with SearchFilter.
- Store vectors: EmbedModel, SyncEmbedModel(model, dim), MissingVectors, BackfillVectors, DropVectors; index metadata: Meta, SetMeta.
- Store documents: Documents (DocSummary, with ContentHash and AddedAt), DocumentHash, RemoveDocument, Prune(dry, deleteOriginals) to collect originals, Close.
- Branch: SetParent, BranchDiff. Jobs: Jobs, Job, ClaimNext; Hit, Job and JobInfo; Runner (Start, Drain, CutOff, Done).
- Worktree branch: OpenBranch, BranchMeta (Root, ParentRoot), WorktreeOf, GCBranches, ErrParentOnly.
api_test.go uses exactly this surface, so a break fails this repository's own tests. It follows semantic versioning from v0.5.0; v0.6.0 added the vector, metadata, segmentation, listing and tag calls; v0.7.0 added the code-index calls above and Models.OnUsage; v0.8.0 adds the worktree branch calls. Everything else in the package is internal to raglit and may change in any release.
Index ¶
- Constants
- Variables
- func AppendAudit(path string, ev AuditEvent) error
- func ApplyRegionFilter(img image.Image, f RegionFilter) image.Image
- func AttachCheapOCR(ocr *OCR, oc OCRConfig, render RenderPolicy) error
- func AuditPath(projectDir string) string
- func AuthorityFromTrail(anyDoc string) func(string) (string, bool)
- func ChildParent(p string) string
- func ContentIDs(path string, frags []Fragment) []string
- func DPICapForArea(areaSqIn float64, maxTokens int) int
- func DamageOf(img image.Image) []string
- func DamageOfPNG(data []byte) []string
- func DefaultRoot() string
- func DefaultTrust(method string) map[string]Confidence
- func DescribableRead(raw string) bool
- func DescribedFraction(raw string) float64
- func DynamicRange(img image.Image) int
- func EnqueueBuiltinIgnore() []string
- func EscalatePrompt(question string) string
- func EstimateTokens(s string) int
- func ExtForContentType(mime string) string
- func ExtractEmailAttachments(srcPath, archivePath, destDir string) (int, string, error)
- func FigurePromptVersion() int
- func FlattenForIndex(s string) string
- func FlattenMarkdownForIndex(md string) string
- func GCBranches(branches []Home, alive func(root string) bool) (removed int, err error)
- func HEICToPNG(ctx context.Context, path string) ([]byte, error)
- func HashHex(b []byte) string
- func HaveHEIC() bool
- func HaveLegacyDoc() bool
- func HavePandoc() bool
- func HavePoppler() bool
- func HaveXLS() bool
- func HintBlock(hint string) string
- func HitPage(startPage int, spansJSON, text, query string) int
- func IdentityKinds() []string
- func IdentityRoleKinds() []string
- func IdentityTextHash(text string) string
- func ImageTooSmallToBeAPage(data []byte) (bool, string)
- func IndexTrees(ctx context.Context, stores map[string]*Store, root string, o TreeOptions) (map[string]TreeResult, error)
- func Inited(h Home) bool
- func IsChildPath(p string) bool
- func IsDescribedPage(raw string) bool
- func IsGeneratedSidecar(path string) bool
- func IsMailArchive(path string) bool
- func IsMinified(head []byte) bool
- func IsMixedPage(raw string) bool
- func IsOpaque(head []byte) bool
- func IsRegionsSidecar(path string) bool
- func IsTranscription(path string) bool
- func IsTransient(err error) bool
- func JudgementsPath(projectDir string) stringdeprecated
- func LaplacianVariance(img image.Image) float64
- func LegacyAttachmentDir(archivePath string) string
- func LegacyDocText(ctx context.Context, path string) (string, error)
- func LegacyRelationsPath(projectDir string) string
- func MaterializeSlice(s *Store, sl Slice) (int, error)
- func NativeDPI(ctx context.Context, pdfPath string, page int) int
- func NormalizeAnyKind(s string) (string, bool)
- func NormalizeIndexName(name string) string
- func NormalizeKind(s string) (string, bool)
- func NormalizeRole(s string) (string, bool)
- func NormalizeTypeName(s string) string
- func OfficeText(ctx context.Context, path string) (string, error)
- func PageAt(startPage int, spans []PageSpan, off int) int
- func PandocText(ctx context.Context, path string) (string, error)
- func ParseRange(s string) (from, to int, err error)
- func PlanSources(cfg Config, baseDir string) (map[string][]string, error)
- func PrincipalFrom(ctx context.Context) string
- func ProjectDir() (string, bool)
- func ProjectDirForDoc(docPath string) string
- func Prompt(k PromptKind, mods ...Mod) string
- func RandomHex(n int) string
- func Recipe(w, mod int) string
- func RecordReadingsInto(js Judgements, s *Store)
- func RegionSHA(data []byte) string
- func RegionsPath(docPath string) string
- func RegionsSuffix() string
- func RenderTranscription(docPath string, pages []TranscribedPage) string
- func RenderTranscriptionCorrected(docPath string, pages []TranscribedPage, corrections map[int]PageCorrection) string
- func RerenderRegion(page image.Image, reg *Region) ([]byte, error)
- func RerenderRegionAsSeen(page image.Image, reg *Region) ([]byte, error)
- func ResolveExtractedAttachments(dirs ...string) func(string) string
- func ResolveFragParams(window, stride, floor, embedLimit int) (int, int, int)
- func STTReading(mediaPath, sha256, producer string, r *sttResponse) (*attest.Reading, error)
- func Sampled(h uint64, mod int) bool
- func SaveConfig(h Home, cfg Config) error
- func ScopeMatches(scopes []string, dir string) bool
- func SessionBranchParent(name string) (string, bool)
- func Shingles(folded string, w int) []uint64
- func SourceFile(path string) (string, bool)
- func StripLayoutMarkup(s string) string
- func SuspectDocs(root string) (map[string][]int, error)
- func SuspectPages(pages []TranscribedPage) map[int]string
- func Suspicion(text string) string
- func TagLine(tags []TagCount) string
- func TilesNeeded(areaSqIn float64, dpi, maxTokens int) int
- func TokensPerSqInAt(dpi int) float64
- func TokensToChars(tokens int) int
- func TranscriptText(name string, r *sttResponse) string
- func TranscriptionPath(docPath string) string
- func UploadURL(path string) string
- func ValidRegionFilter(f RegionFilter) bool
- func ValidateFieldSchema(raw json.RawMessage) error
- func VerifyRegionRender(reg *Region, data []byte) error
- func WithPrincipal(ctx context.Context, p string) context.Context
- func WorktreeOf(dir string) (mainRoot string, ok bool)
- func WriteRegionDoc(docPath string, pages ...RegionPage) (string, error)
- func WriteTranscription(docPath string, pages []TranscribedPage) (string, error)
- func WriteTranscriptionCorrected(docPath string, pages []TranscribedPage, corrections map[int]PageCorrection) (string, error)
- type ACL
- type Assembler
- type AttachmentMove
- type AttestationRow
- type AuditEvent
- type AuthConfig
- type Block
- type BranchChange
- type BranchInfo
- type BranchMeta
- type Changes
- type ChannelStat
- type Channels
- func (c *Channels) Acquire(ctx context.Context, model string) (release func(ok bool), err error)
- func (c *Channels) Gate(inner Chatter, model string) Chatter
- func (c *Channels) GateVector(inner VectorClient, model string) VectorClient
- func (c *Channels) Note429(model string)
- func (c *Channels) Observe(model string, next func(llm.RetryEvent)) func(llm.RetryEvent)
- func (c *Channels) Stats() []ChannelStat
- type Chatter
- type CodeSegmenter
- type CodeSpan
- type Confidence
- type Config
- type ContextProber
- type Coverage
- type DPIDecision
- type DiffShape
- type DocContent
- type DocCost
- type DocDiff
- type DocFields
- type DocIdentity
- type DocKind
- type DocMatch
- type DocPageText
- type DocRef
- type DocSummary
- type DocType
- type Document
- type EmailAttachmentRef
- type EmailHeader
- type EmailMessage
- type Embedder
- type EnqueueItem
- type EnqueueResult
- type ErrIdentityTooShort
- type ErrSegmentShort
- type Fetched
- type FieldsCoverage
- type FieldsStale
- type FigureHit
- type FigureRef
- type Finder
- type FoldRecord
- type FoldReport
- type FoldedPage
- type FoldedText
- type FragConfig
- type Fragment
- type GCPolicy
- type Gap
- type GibberishConfig
- type Hit
- type HitTrust
- type Home
- func (h Home) AttachmentDir(archivePath string) string
- func (h Home) AttachmentsDir() string
- func (h Home) ConfigPath() string
- func (h Home) Ensure() error
- func (h Home) IndexPath() string
- func (h Home) OriginalPath(docPath string) string
- func (h Home) OriginalsDir() string
- func (h Home) PageDir(docPath string) string
- func (h Home) PagesDir() string
- func (h Home) TranscriptDir(mediaPath string) string
- func (h Home) TranscriptsDir() string
- func (h Home) UploadsDir() string
- type Identifier
- func (id *Identifier) ExtractFields(ctx context.Context, text string, t DocType, hint string) (DocFields, error)
- func (id *Identifier) Identify(ctx context.Context, ask IdentityAsk) (DocIdentity, error)
- func (id *Identifier) IdentifyTags(ctx context.Context, ask IdentityAsk) (content, roles []string, err error)
- type IdentityAsk
- type IdentityJob
- type IdentityQueueStatus
- type IdentityStatus
- type IdentityWorker
- type ImageEmbedConfig
- type ImageEmbedder
- type IndexConfig
- type IndexDigest
- type IndexMeta
- type Job
- type JobInfo
- type JobStage
- type JobState
- type JudgementStore
- func (s *JudgementStore) Authorities() []ReadingAuthority
- func (s *JudgementStore) Authority(source string) (ReadingAuthority, bool)
- func (s *JudgementStore) Close() error
- func (s *JudgementStore) CoverageOf(parent string, pages int) (Coverage, error)
- func (s *JudgementStore) DeleteSlice(id string) error
- func (s *JudgementStore) History(kind, subject string) ([]AuditEvent, error)
- func (s *JudgementStore) OnTranscriptionChange(fn func(TranscriptionChange))
- func (s *JudgementStore) PageCorrections(doc string) (map[int]PageCorrection, error)
- func (s *JudgementStore) PageReadings(doc string, page int) []PageCorrection
- func (s *JudgementStore) PutAuthority(a ReadingAuthority) error
- func (s *JudgementStore) PutPageCorrection(c PageCorrection) error
- func (s *JudgementStore) PutRelation(m Mark) error
- func (s *JudgementStore) PutSlice(sl Slice) error
- func (s *JudgementStore) Relation(a, b string) (Mark, bool, error)
- func (s *JudgementStore) Relations() ([]Mark, error)
- func (s *JudgementStore) RelationsFor(doc string) ([]Mark, error)
- func (s *JudgementStore) Restore(path, by string) error
- func (s *JudgementStore) Slice(id string) (Slice, bool, error)
- func (s *JudgementStore) SliceParents() ([]string, error)
- func (s *JudgementStore) Slices() ([]Slice, error)
- func (s *JudgementStore) SlicesOf(parent string) ([]Slice, error)
- func (s *JudgementStore) Withdraw(w Withdrawal) error
- func (s *JudgementStore) Withdrawals() []Withdrawal
- func (s *JudgementStore) Withdrawn(path string) (Withdrawal, bool)
- type Judgements
- type Key
- type KeyStore
- func (ks *KeyStore) Authenticate(token string, now time.Time) (Key, error)
- func (ks *KeyStore) ByPrincipal(p string) (Key, bool)
- func (ks *KeyStore) Flush() error
- func (ks *KeyStore) HasProject(ns string) bool
- func (ks *KeyStore) IsAncestor(anc, p string) bool
- func (ks *KeyStore) List(by, admin string) []Key
- func (ks *KeyStore) Mint(k Key) (string, Key, error)
- func (ks *KeyStore) Revoke(by, admin, id string) error
- func (ks *KeyStore) RevokeProject(ns string) error
- type Lane
- type LaneStatus
- type LayerSavings
- type LayoutBox
- type LayoutOpts
- type Mark
- type MarkKind
- type Mask
- type Mod
- type ModelRole
- type Models
- type NomicVisionEmbedder
- type Note
- type OCR
- func (o *OCR) AskWithHint(hint string) func(context.Context, PageImage, RegionAbout) (RegionReading, error)
- func (o *OCR) AskWithOCR() func(context.Context, PageImage, RegionAbout) (RegionReading, error)
- func (o *OCR) Page(ctx context.Context, img PageImage) (string, error)
- func (o *OCR) PageAsSeen(ctx context.Context, img PageImage) (text, engine string, shrinks int, err error)
- func (o *OCR) PageWithEngine(ctx context.Context, img PageImage) (text, engine string, err error)
- type OCRConfig
- type OCRLine
- type OffsetFrag
- type PaddleEngine
- type PageCorrection
- type PageEngine
- type PageImage
- type PageLayout
- type PageOCR
- type PagePair
- type PageReading
- type PageReview
- type PageSketch
- type PageSpan
- type PageText
- type PendingItem
- type Pool
- func (p *Pool) Close() error
- func (p *Pool) FileDir(fileHash string) string
- func (p *Pool) GC(pol GCPolicy) (int, error)
- func (p *Pool) Get(recipeHash, fileHash string) (PooledDoc, bool, error)
- func (p *Pool) PurgeFiles(fileHashes []string) (int, error)
- func (p *Pool) Put(recipeHash, fileHash string, doc PooledDoc) error
- func (p *Pool) Stats() (PoolStats, error)
- type PoolStats
- type PooledDoc
- type PooledFragment
- type PooledPage
- type Problem
- type ProblemKind
- type ProjectMeta
- type ProjectSavings
- type PromptKind
- type Proposal
- type Provider
- type PruneDir
- type PruneResult
- type PurgeResult
- type Reading
- type ReadingAuthority
- type Rect
- type Reference
- type Region
- type RegionAbout
- type RegionDoc
- type RegionFilter
- type RegionPage
- type RegionProposal
- type RegionReader
- type RegionReading
- type RegionSpan
- type Registration
- type Registry
- func (r *Registry) ACLEnabled() bool
- func (r *Registry) ACLOf(name string) ACL
- func (r *Registry) Admin() string
- func (r *Registry) CanRead(ctx context.Context, name string) bool
- func (r *Registry) Close() error
- func (r *Registry) DeleteAs(ctx context.Context, name string, branch bool) error
- func (r *Registry) DeleteBranch(name string) error
- func (r *Registry) DeleteIndex(name string) error
- func (r *Registry) EnableACL(admin string) error
- func (r *Registry) ExclusiveHashes(names []string) ([]string, error)
- func (r *Registry) Existing(name string) (*Store, error)
- func (r *Registry) FoldProject(ctx context.Context, id string, dry bool) (FoldReport, error)
- func (r *Registry) ForkBranch(name, parent string) error
- func (r *Registry) ForkBranchAs(ctx context.Context, name, parent string) error
- func (r *Registry) Get(name string) (*Store, error)
- func (r *Registry) HasIdentifier() bool
- func (r *Registry) IsAdmin(ctx context.Context) bool
- func (r *Registry) Keys() *KeyStore
- func (r *Registry) LayersOf(base string) []string
- func (r *Registry) ListBranches() ([]BranchInfo, error)
- func (r *Registry) Meta(name string) (IndexMeta, bool)
- func (r *Registry) Names() []string
- func (r *Registry) OwnReadable(ctx context.Context, names []string) []string
- func (r *Registry) ParentOf(name string) string
- func (r *Registry) ProjectBase(id string) string
- func (r *Registry) ProjectIDOf(base string) string
- func (r *Registry) ProjectSavings(ctx context.Context, id string) (ProjectSavings, error)
- func (r *Registry) Read(ctx context.Context, name string) (*Store, error)
- func (r *Registry) Readable(ctx context.Context, names []string) []string
- func (r *Registry) ReapSessionBranches(ttl time.Duration, now time.Time) ([]string, error)
- func (r *Registry) RegisterProject(ctx context.Context, id, label string) (Registration, error)
- func (r *Registry) Scoped() bool
- func (r *Registry) ScopedFor(ctx context.Context, dir string) []string
- func (r *Registry) SetEmbedder(e *Embedder)
- func (r *Registry) SetIdentifier(id *Identifier)
- func (r *Registry) SetImageEmbedder(ie ImageEmbedder)
- func (r *Registry) SetKeyStore(ks *KeyStore)
- func (r *Registry) SetVisibility(ctx context.Context, name string, v Visibility) error
- func (r *Registry) Siblings(ctx context.Context, name string) []string
- func (r *Registry) TransferOwnership(ctx context.Context, name, to string) error
- func (r *Registry) Write(ctx context.Context, name string) (*Store, error)
- type Relation
- type RenderPolicy
- type RetrySummary
- type RetryTally
- type Root
- type Run
- type Runner
- type STT
- type SearchFilter
- type SegResult
- type Segment
- type Segmenter
- type SimilarOpts
- type SimilarReport
- type SketchStatus
- type Slice
- type SliceProposal
- type StageLog
- type StaleFields
- type Status
- type Store
- func Open(path string) (*Store, error)
- func OpenBranch(parent Home, branch Home, root, parentRoot string) (*Store, error)
- func OpenBranchOn(parent *Store, branch Home, root, parentRoot string) (*Store, error)
- func OpenHome(home Home) (*Store, error)
- func OpenIndex(home Home, name string) (*Store, error)
- func OpenIndexRO(home Home, name string) (*Store, error)
- func (s *Store) ActiveReadings(ctx context.Context, doc string) (map[int]PageReading, error)
- func (s *Store) AddNote(path string, n Note) (Note, error)
- func (s *Store) AddPageReading(ctx context.Context, r PageReading) error
- func (s *Store) AllowPatterns() []string
- func (s *Store) Ambient() bool
- func (s *Store) AttachmentDirFor(archivePath string) string
- func (s *Store) Attestations(doc string) ([]AttestationRow, error)
- func (s *Store) AttestationsForUnit(doc, unit string) ([]AttestationRow, error)
- func (s *Store) AttestedDocs() (map[string]int, error)
- func (s *Store) BackfillLanes() (int, error)
- func (s *Store) BackfillSourceStats() (int, error)
- func (s *Store) BackfillVectors(ctx context.Context, batch int) (int, error)
- func (s *Store) BranchDiff() ([]BranchChange, error)
- func (s *Store) CanSearchByMeaning() bool
- func (s *Store) CancelJob(id int64) error
- func (s *Store) CaptionProfile() string
- func (s *Store) ChatInputLimitTokens(ctx context.Context, p ContextProber, model string, configured int) int
- func (s *Store) ClaimNext() (*Job, error)
- func (s *Store) ClaimNextInLane(lane Lane) (*Job, error)
- func (s *Store) ClearFailed(paths ...string) error
- func (s *Store) ClearSketches() error
- func (s *Store) Close() error
- func (s *Store) DedupeQueue() (int, error)
- func (s *Store) DeleteDocType(name string) error
- func (s *Store) DeleteDocument(path string) error
- func (s *Store) DeleteNote(id int64) error
- func (s *Store) DocCosts() ([]DocCost, error)
- func (s *Store) DocJobs(targets ...string) ([]JobInfo, error)
- func (s *Store) DocReadingHistory(doc string) ([]PageReading, error)
- func (s *Store) DocReview(path string) (title string, pages []PageReview, err error)
- func (s *Store) DocText(exactPath string, from, to, maxChars int) (DocContent, error)
- func (s *Store) DocTypeByName(name string) (DocType, error)
- func (s *Store) DocTypeRefs() ([]DocType, error)
- func (s *Store) DocTypes() ([]DocType, error)
- func (s *Store) DocumentAddedAt(path string) int64
- func (s *Store) DocumentFields(path string) (DocFields, error)
- func (s *Store) DocumentHash(path string) (string, error)
- func (s *Store) DocumentIdentity(path string) (DocIdentity, error)
- func (s *Store) DocumentPaths() ([]string, error)
- func (s *Store) Documents() ([]DocSummary, error)
- func (s *Store) DocumentsMissingFields() ([]string, error)
- func (s *Store) DocumentsMissingIdentity() ([]string, error)
- func (s *Store) DocumentsMissingTags() ([]string, error)
- func (s *Store) DropVectors() (int, error)
- func (s *Store) EmbedLimitChars(ctx context.Context, e *Embedder, configured int) int
- func (s *Store) EmbedLimitTokens(ctx context.Context, p ContextProber, model string, configured int) int
- func (s *Store) EmbedModel() string
- func (s *Store) Enqueue(url, title string) (int64, error)
- func (s *Store) EnqueueFields(path string, force bool) (bool, error)
- func (s *Store) EnqueueFieldsFor(paths []string, force bool) (int, error)
- func (s *Store) EnqueueFresh(url, title string, fresh bool) (int64, error)
- func (s *Store) EnqueueIdentity(path string, force bool) (bool, error)
- func (s *Store) EnqueueIdentityFor(paths []string, force bool) (int, error)
- func (s *Store) EnqueueMany(items []EnqueueItem) ([]EnqueueResult, error)
- func (s *Store) EnqueueMissingFields(force bool) (int, error)
- func (s *Store) EnqueueMissingIdentities(force bool) (int, error)
- func (s *Store) EnqueueMissingTags(force bool) (int, error)
- func (s *Store) EnqueueTags(path string, force bool) (bool, error)
- func (s *Store) EnqueueTagsFor(paths []string, force bool) (int, error)
- func (s *Store) ExportDoc(path string) (PooledDoc, error)
- func (s *Store) ExtractFields(ctx context.Context, path string, force bool) (DocFields, error)
- func (s *Store) ExtractableMissing() ([]string, error)
- func (s *Store) FieldsCoverage() ([]FieldsCoverage, error)
- func (s *Store) FieldsStaleness() ([]StaleFields, error)
- func (s *Store) FindPathByHash(hash, path string) (string, error)
- func (s *Store) ForgetJob(id int64) error
- func (s *Store) FragmentCount() (int, error)
- func (s *Store) FragmentExists(contentID string) (bool, error)
- func (s *Store) HybridSearch(ctx context.Context, query string, limit int) ([]Hit, error)
- func (s *Store) HybridSearchFiltered(ctx context.Context, query string, f SearchFilter, limit int) ([]Hit, error)
- func (s *Store) HybridSearchPath(ctx context.Context, query, pathPrefix string, limit int) ([]Hit, error)
- func (s *Store) IdenticalGroups() ([][]string, error)
- func (s *Store) IdentifyDocument(ctx context.Context, path string, force bool) (DocIdentity, error)
- func (s *Store) Identities() ([]IdentityStatus, error)
- func (s *Store) IdentityJobs(state string, limit int) ([]IdentityJob, error)
- func (s *Store) IdentityQueue() (IdentityQueueStatus, error)
- func (s *Store) IdentityText(ctx context.Context, path string) (string, error)
- func (s *Store) IgnorePatterns() []string
- func (s *Store) IgnoredBy(path string) string
- func (s *Store) ImportVerifiedTranscripts(dryRun bool, ruledBy string) ([]TranscriptMatch, error)
- func (s *Store) IndexAbout() (about string, stale bool, err error)
- func (s *Store) IndexDigest() (IndexDigest, error)
- func (s *Store) IndexDigestFor(pathPrefix string, topN int) (IndexDigest, error)
- func (s *Store) IndexHint() string
- func (s *Store) IndexStatus() (Status, error)
- func (s *Store) Ingest(ctx context.Context, doc Document) error
- func (s *Store) IngestPDF(ctx context.Context, ocr *OCR, pdfPath string) (int, error)
- func (s *Store) IngestPooled(ctx context.Context, docPath, title string, doc PooledDoc, poolFileDir string) (int, error)
- func (s *Store) Job(id int64) (JobInfo, error)
- func (s *Store) JobStages(jobID int64) ([]JobStage, error)
- func (s *Store) Jobs(state string, limit int) ([]JobInfo, error)
- func (s *Store) LearnLimitFromError(model, errText string) (int, bool)
- func (s *Store) LearnedEmbedLimitTokens(model string) int
- func (s *Store) LegacyAttachmentDirFor(archivePath string) string
- func (s *Store) MarkForReingest(path string) error
- func (s *Store) MatchDocuments(ref string) ([]DocRef, error)
- func (s *Store) MergeTags(ctx context.Context, m TagMerge) (TagMergeResult, error)
- func (s *Store) Meta(key string) (string, bool)
- func (s *Store) MigrateExtractedAttachments(dryRun bool) ([]AttachmentMove, error)
- func (s *Store) MissingCount() int
- func (s *Store) MissingDocuments() ([]string, error)
- func (s *Store) MissingVectors() (int, error)
- func (s *Store) NewStageLog(jobID int64) *StageLog
- func (s *Store) Notes(path string) ([]Note, error)
- func (s *Store) OversizedDocs(limit int) (map[string]int, error)
- func (s *Store) PageImagePath(path string, page int) (string, error)
- func (s *Store) PageLayoutFor(docPath string, page int) (*PageLayout, error)
- func (s *Store) PageReadings(ctx context.Context, doc string, page int) ([]PageReading, error)
- func (s *Store) PagesRoot() string
- func (s *Store) PagesWithLayout(docPath string) map[int]bool
- func (s *Store) Path() string
- func (s *Store) Paused() bool
- func (s *Store) Problems(ctx context.Context) ([]Problem, error)
- func (s *Store) ProposeDocType(ctx context.Context, name string, goldPaths []string) (DocType, error)
- func (s *Store) Prune(dry, deleteOriginals bool) (PruneResult, error)
- func (s *Store) Purge(pattern string, dry bool) (PurgeResult, error)
- func (s *Store) PurgeDocPageCache(ctx context.Context, path string) (int, error)
- func (s *Store) PutAttestations(doc string, entries []attest.Entry) error
- func (s *Store) ReadingFor(docPath string) (Reading, bool, error)
- func (s *Store) ReadingsOfSource(sha string) ([]Reading, error)
- func (s *Store) ReclaimIdentityJobs() (int, error)
- func (s *Store) RecordFields(ctx context.Context, path string, f DocFields, by string) (DocFields, error)
- func (s *Store) RecordIdentity(ctx context.Context, path string, d DocIdentity, by string) (DocIdentity, error)
- func (s *Store) RecordReading(r Reading) error
- func (s *Store) RecordSourceStat(path string) error
- func (s *Store) ReferencesTo(ctx context.Context, path string) ([]Reference, error)
- func (s *Store) RemoveDocument(path string, stored bool) error
- func (s *Store) RequeueJob(id int64) error
- func (s *Store) Restore(path string) error
- func (s *Store) RetryJob(id int64) error
- func (s *Store) SameBytesAs(docPath string) (map[string]string, error)
- func (s *Store) SameBytesHash(hash string) (map[string]string, error)
- func (s *Store) SaveUpload(name string, data []byte) (string, error)
- func (s *Store) Scopes() (scopes []string, set bool)
- func (s *Store) Search(query string, limit int) ([]Hit, error)
- func (s *Store) SearchFigures(ctx context.Context, query string, limit int) ([]FigureHit, error)
- func (s *Store) SearchFiguresOwnLayer(ctx context.Context, query, pathPrefix string, limit int) ([]FigureHit, error)
- func (s *Store) SearchFiguresPath(ctx context.Context, query, pathPrefix string, limit int) ([]FigureHit, error)
- func (s *Store) SearchFiltered(query string, f SearchFilter, limit int) ([]Hit, error)
- func (s *Store) SearchOwnLayer(ctx context.Context, query, mode, pathPrefix string, limit int) ([]Hit, error)
- func (s *Store) SearchPath(query, pathPrefix string, limit int) ([]Hit, error)
- func (s *Store) ServesPageImage(img string) bool
- func (s *Store) SetAllowPatterns(patterns []string) ([]string, error)
- func (s *Store) SetAmbient(ambient bool) error
- func (s *Store) SetAuthorityResolver(fn func(source string) (string, bool))
- func (s *Store) SetCaptionProfile(profile string, now int64) error
- func (s *Store) SetDocType(t DocType) error
- func (s *Store) SetDocumentFields(ctx context.Context, path string, f DocFields) error
- func (s *Store) SetDocumentHash(path, hash string) error
- func (s *Store) SetDocumentIdentity(ctx context.Context, path string, d DocIdentity) error
- func (s *Store) SetEmbedder(e *Embedder)
- func (s *Store) SetExtractEmailAttachments(v bool)
- func (s *Store) SetIdentifier(id *Identifier)
- func (s *Store) SetIgnorePatterns(patterns []string) ([]string, error)
- func (s *Store) SetImageEmbedder(ie ImageEmbedder)
- func (s *Store) SetIndexHint(hint string, now int64) error
- func (s *Store) SetJobLane(id int64, lane Lane) error
- func (s *Store) SetJobURL(id int64, url string) error
- func (s *Store) SetMeta(key, value string, now int64) error
- func (s *Store) SetParent(p *Store)
- func (s *Store) SetPaused(paused bool) error
- func (s *Store) SetScopes(scopes []string) ([]string, error)
- func (s *Store) SetTags(path string, tags []string) error
- func (s *Store) SiblingReadings(docPath string) ([]Reading, error)
- func (s *Store) SimilarIndexed(docPath string, opts SimilarOpts) (SimilarReport, error)
- func (s *Store) SimilarTo(probePath string, probe FoldedText, opts SimilarOpts) (SimilarReport, error)
- func (s *Store) SketchAll(w, mod int) (done int, errs []error)
- func (s *Store) SketchDoc(docPath string, w, mod int) (int, error)
- func (s *Store) SketchStatusFor(w, mod int) (SketchStatus, error)
- func (s *Store) SketchedPaths(w, mod int) ([]string, error)
- func (s *Store) StoragePath(path string) (string, bool)
- func (s *Store) StorageRel(path string) (string, bool)
- func (s *Store) SupersededPages(ctx context.Context) ([]PageReading, error)
- func (s *Store) SupplyIdentity(path string, d DocIdentity, by string) error
- func (s *Store) SyncEmbedModel(model string, dim int) (dropped bool, err error)
- func (s *Store) TagContext() string
- func (s *Store) TagDocument(ctx context.Context, path string, force bool) (DocIdentity, error)
- func (s *Store) TagNeighbours() (map[string][]string, error)
- func (s *Store) Tags(path string) ([]string, error)
- func (s *Store) TranscriptDirFor(mediaPath string) string
- func (s *Store) TruePages(exactPath string) ([]PageText, error)
- func (s *Store) VecSearch(ctx context.Context, query string, limit int) ([]Hit, error)
- func (s *Store) VecSearchFiltered(ctx context.Context, query string, f SearchFilter, limit int) ([]Hit, error)
- func (s *Store) VecSearchPath(ctx context.Context, query, pathPrefix string, limit int) ([]Hit, error)
- func (s *Store) Withdraw(w Withdrawal) error
- func (s *Store) Withdrawals() ([]Withdrawal, error)
- func (s *Store) WithdrawnReason(path string) (string, bool)
- func (s *Store) WriteIndexAbout(ctx context.Context) (string, error)
- type StoreSet
- type StrategyConfig
- type TagCount
- type TagMerge
- type TagMergeResult
- type TesseractEngine
- type TokenBudget
- func (b *TokenBudget) Fit(ctx context.Context, overhead, rest string, minTake int) int
- func (b *TokenBudget) FitOne(ctx context.Context, rest string, minTake int) int
- func (b *TokenBudget) FitSuffix(ctx context.Context, s string, room int) int
- func (b *TokenBudget) Fits(ctx context.Context, text string) bool
- func (b *TokenBudget) Tokens(ctx context.Context, text string) int
- func (b *TokenBudget) Unlimited() bool
- type TokenCounter
- type ToolEnv
- type ToolStatus
- type TranscribedFigure
- type TranscribedPage
- type TranscriptMatch
- type TranscriptionChange
- type TreeOptions
- type TreeResult
- type TreeRules
- type TreeWatcher
- type VectorClient
- type Visibility
- type WatchHub
- type WatchOptions
- type Withdrawal
- type Worker
Constants ¶
const ( OpRelationPut = "relation.put" // OpAuthorityPut — a person ruled which reading of a source governs. OpAuthorityPut = "authority.put" OpSlicePut = "slice.put" OpSliceDelete = "slice.delete" OpPageCorrect = "page.correct" OpDocWithdraw = "doc.withdraw" OpDocRestore = "doc.restore" )
Audit op names. Values, not derived from Go identifiers: they are written to a file that outlives any refactor, and a rename must not orphan history.
const ( CaptionProfileLegal = "legal" CaptionProfileDocs = "docs" )
const ( // IdentityByMachine / IdentityByPerson are gen_source's two values. IdentityByMachine = "machine" IdentityByPerson = "person" // IdentitySupplied is a caption the caller handed in with the document // (SupplyIdentity): the fold summary a session already wrote. Not a // ruling — a re-ingest that supplies another replaces it — and not owed // to the model, so no identity or tags job is queued for it. IdentitySupplied = "supplied" )
const ( IdentityAskFull = "identity" IdentityAskTags = "tags" IdentityAskFields = "fields" )
The three asks a queued job can be. One queue, because they are the same shape of work — a bounded model call per document, hundreds of them, drained at the endpoint's real concurrency.
const ( // SQLOwnWords constrains fragments to the document's own indexed words — // the transcription and the descriptions that ARE its content — excluding // the fragments that are about it. Unqualified column, for a query with one // table. SQLOwnWords = `origin NOT IN ('identity', 'fields')` // SQLOwnWordsF is the same, qualified for a query that joins `fragments f`. SQLOwnWordsF = `f.origin NOT IN ('identity', 'fields')` )
A note for anything that filters fragments by origin.
There are two kinds of generated text in this column and they are not interchangeable. 'identity' is a caption ABOUT a document — excluded from the document's own words, because a re-run must read the document and not the last caption of it. 'described' IS a document's content: a photograph has no other text, and dropping it means the photograph has none.
So the predicate for "the document's own indexed words" is NOT `origin = ”`. Six queries were written that way before descriptions existed, and each one silently changed meaning the moment they did: the document list reported photographs as empty, get_document returned nothing for them, and the cross-index pool would have re-imported them with their only content stripped.
It is not `origin <> 'identity'` either, which is how it was written next. 'fields' — a document type's schema, filled out (docfields.go) — is the same kind of thing a caption is: a record ABOUT the document. Named the same way and added the same way, it would have walked straight back into the same trap from the other side, with every extraction reappearing as the document's own text in get_document and in its fragment count.
So the predicate is this constant, in every query that means it. Adding a generated origin means deciding which list it joins, ONCE, here.
const ( KeyAPI = "api" KeySession = "session" KeyProject = "project" )
const ( // ReadingMachine — a model's output, unreviewed. Every ingest starts here. ReadingMachine = "machine" // ReadingAdapted — a machine reading a person has changed: a corrected page, // a re-attributed speaker turn. Better than machine and not attested, because // nobody has said the whole of it is right. ReadingAdapted = "adapted" // ReadingAttested — a person has ruled on it. What may be quoted. ReadingAttested = "attested" )
Reading levels, in the order authority accrues.
const ( MethodASR = "oidio-asr" MethodVision = "vision-ocr" MethodTextLayer = "text-layer" MethodRegion = "region" MethodPandoc = "pandoc" MethodEmail = "email-mime" MethodCompiled = "compiled" MethodVerbatim = "verbatim" )
Reading methods — how the text was produced.
const ( VerdictRead = "read" // or empty VerdictEscalate = "escalate" // the frame is wrong and I cannot fix it from here VerdictIllegible = "illegible" // right frame, the pixels are not there VerdictUnknown = "unknown" // the model returned a word this package does not define ActionRetransform = "retransform" // box was right, rendering was wrong ActionRepick = "repick" // box was wrong ActionKeep = "keep" // the child was mistaken; its reading stands ActionAbandon = "abandon" // nothing here can be read )
What a region may claim about its own frame, and what a parent may answer.
const ( // FlagLowResolution — this region was seen at materially less detail than a // letter page gets. Pure arithmetic, available before the model is called. FlagLowResolution = "low-resolution" // FlagRepetition — the model looped. On a page this fails the document; on a // region it is a reason to descend. FlagRepetition = "repetition" // There was a FlagClipped here, meant to catch text cut at a region // boundary. It measured ink touching the border and it did not work: // measured on the survey, a blank margin scored 0.196, a text block 0.161 // and the drawing interior 0.055 — ANTI-correlated, because what it actually // found was the sheet's own border frame. It fired on every region of the // first real run, and a flag that is always on trains people to ignore // flags. // // Removed rather than tuned. The underlying problem — a cut like "REPLAT OF // BLO" — is now PREVENTED by padding descents (see descentPad) instead of // reported after the fact. // FlagCycled — descent stopped because it stopped paying: a transform // rendered to bytes already seen, or cleared no flag. FlagCycled = "cycled" // FlagExhausted — the model proposed nothing further. The one POSITIVE // signal, and the only one meaning "this is as good as it gets". FlagExhausted = "exhausted" // FlagBlurred — the crop's strokes are smeared, by the variance of its // laplacian against the measured spread of real pages. Computed on the // FULL-RESOLUTION crop before any call, because the model is shown a // downsample and cannot tell blur from smallness. FlagBlurred = "blurred" // FlagFaded — the crop's luminance is crushed into a narrow band, the way a // photocopy of a photocopy is. The damage with the most to gain from repair: // a faded crop read NOTHING unfiltered and most of its facts with contrast. FlagFaded = "faded" // FlagTransformSuspect — this region's reading has almost nothing in common // with what its PARENT said is here. Not a claim that the text is wrong: a // claim that the TRANSFORM is, because a crop or a rotation applied the wrong // way round does not garble the content, it returns a reading about something // else. Observed as clockwise-for-counter-clockwise and as a bad root. // // Recorded rather than corrected. Acting on it means re-rendering the PARENT // — the child cannot fix a rotation it did not choose — and that loop is not // built. FlagTransformSuspect = "transform-suspect" // FlagEscalated — this region's frame was referred back to its parent, and // the parent answered. Recorded whatever the answer was, because "a human // looked at why this looks wrong" is worth knowing even when the verdict was // that nothing was wrong. FlagEscalated = "escalated" // FlagAbandoned — the parent judged that nothing here can be read at any // treatment. The region keeps whatever its parent said was here: a region // nobody could read is not a region nobody described. FlagAbandoned = "abandoned" // FlagUnknownVerdict — the region answered with a verdict this package does // not define. Kept visible so invented vocabulary does not read as silence. FlagUnknownVerdict = "unknown-verdict" // FlagUnread — this region could not be read at all: the call failed rather // than returning something poor. Its children may still have been read, so // the tree below it is real and only this node's own account is missing. FlagUnread = "unread" // FlagBudget — descent stopped because the CALL budget ran out, not because // the region was finished. Recorded so a thin read is never mistaken for a // complete one. // // Not to be confused with FlagDepthReached. Measured 2026-08-04 over the // corpus's oversize pages: of 70 budget flags, 58 were tiles that stopped // because MaxDepth said to — normal termination reported as exhaustion. A // flag that is always on trains people to ignore flags, which is the same // reason FlagClipped was removed. FlagBudget = "budget" // FlagDepthReached — this region stopped descending because MaxDepth said to. // Normal termination, not a shortfall: at MaxDepth 1 every tile ends this way // by construction, and calling that a spent budget makes the budget flag // meaningless on exactly the sheets it matters for. FlagDepthReached = "depth-reached" )
Region flags. Not a score: a confidence number here would be an invented statistic. These are conditions that either hold or do not.
const ( // MinRunChars is the shortest run of identical text counted as a match, in // folded characters. 40 characters is six or seven words. // // This is the phrase-noise cutoff. Swept 1/20/40/80/200 on three pairs chosen // to span the difficulty: // // 1 20 40 80 200 // Graham declaration vs its own email // attachment (byte-identical files) .930 .927 .927 .927 .927 // two DIFFERENT Havern deeds, 1993 QCD // vs 2008 deed (same county forms) .307 .301 .259 .033 .000 // 1946 survey vs 1992 reconveyance // (unrelated) .010 .010 .000 .000 .000 // // The shape is what decides it: a true duplicate's coverage does not move at // all across the whole sweep, so raising the floor costs nothing there, while // unrelated documents go to exactly zero by 40. The sensitive case is the // middle one — two different instruments on the same county's forms — and 40 // leaves it at 0.26, less than half of DuplicateThreshold. // // Not 80, which would take that middle case to 0.03: it would also discard a // genuine paragraph-length quotation, which is a finding this corpus turns on. // Report-level noise is controlled by MinReportChars instead, which can be // raised without making short real matches invisible. MinRunChars = 40 // MaxPostingsPerHash caps how many times a shingle may occur in the compared // document before it is ignored as an anchor. // // This is not a tuning knob, it is what keeps the alignment from going // quadratic on a degenerate document. A shingle occurring thousands of times is // repetitive filler — a run of dot leaders, a table of blank cells, a failed OCR // emitting one phrase over and over — and it cannot anchor a unique alignment, // only multiply seeds. // // Measured over every document on this corpus: the worst // (201503110042-2008-06-predecessor-survey-lotB.pdf, whose transcription is // largely one repeated string) would generate 67,829,506 seeds against itself // uncapped, and 510 capped. Five orders of magnitude, from one document. Same // reasoning as repeat masking in a sequence aligner. MaxPostingsPerHash = 24 // DuplicateThreshold is the block coverage at which a document is called a copy // rather than an overlap. // // The measured separation on this corpus is wide. Two DIFFERENT recorded // instruments on the same county's forms reach 0.26; the worst genuinely // byte-identical pair reaches 0.92 (the Graham declaration against its own email // attachment — under 1.00 only because generic-text masking removes the same // passages from both sides). So anything from about 0.3 to 0.9 separates those. // // 0.55 is set by the one case that falls INSIDE that gap, and it is the case the // feature exists for: the operative 2008 record of survey is reproduced on page 4 // of the 2008 lot certification, and aligns at 0.589 rather than ~0.95 because // the certification's OCR is substantially garbled ("LAURENCE MOONION" for // Clarence Brannock, "HAVERN TACOMA SHORE" for Havern Lake Shore). A threshold // of 0.60 called that pair a mere `overlap`, missing a real sub-document // containment by 0.011. // // Setting a threshold from a single example is normally how a measurement gets // fitted to noise. It is defensible here because the example is the DEGRADED-COPY // case rather than an outlier — degraded copies are the population this tool // serves — and because 0.55 is still more than twice the level two different // instruments reach. The error direction is deliberate: a false `duplicate` // costs one human comparison, a missed one lets a second copy of an instrument // enter the record unnoticed. DuplicateThreshold = 0.55 )
const ( // FacetText is the words: what the document says, or what was spoken. FacetText = "text" // FacetSpeaker is who said them. Only diarized readings have it, and it is // the facet that fails independently of the words. FacetSpeaker = "speaker" // FacetLayout is reading order and structure — which column follows which, // where a table's cells belong. A page can be read perfectly and assembled // wrongly. FacetLayout = "layout" // FacetSubject is what a model says an image DEPICTS. Not a transcription of // anything: nobody wrote "a red Chevrolet Malibu", the model inferred it. FacetSubject = "subject" )
Facets — the parts of a reading that can be trusted separately.
const CutOffWait = 10 * time.Second
CutOffWait is how long cancelled jobs get to write 'pending' back after a Drain's grace ends. A job that does not return in time keeps its 'running' row and is marked 'error' at the next start, as before.
const DefaultIdentitySlots = 4
DefaultIdentitySlots is how wide this worker's caption pipeline is.
It used to be a CAPACITY CLAIM — "two, because that is what the endpoint serves concurrently" — and that was a number about a server written down in raglit, which is exactly the kind that goes stale when the model layout changes. It is no longer one: the identity model has its own admission channel (modelchan.go), learned from the server's own backpressure, and that is what bounds requests now.
So this is just how many captions may be ASSEMBLED and in flight at once. Four rather than two, because the extra callers cost a goroutine each and simply wait in Acquire when the model is narrow — while a model that turns out to serve several would otherwise be held to two by a constant nobody revisited.
const DefaultIngestRunners = 8
DefaultIngestRunners is how many documents may be in flight at once.
Not a capacity claim — the model channels are that. It is how many documents can be PARTWAY through, and it wants to be comfortably more than the number of models, so that a document waiting on the vision channel never occupies the slot a document needing only embedding could use. Runners are goroutines; the scarce thing is downstream.
const DefaultMaxImageTokens = 16384
DefaultMaxImageTokens matches --image-max-tokens as configured on this fleet. It belongs to the SERVER, not the document, so it is a policy input rather than a constant folded into the arithmetic.
const DefaultSTTTimeout = 4 * time.Hour
DefaultSTTTimeout bounds one transcription when Timeout is unset. Four hours covers a full day's hearing at diarization's roughly-realtime cost with room for a loaded box; it is a runaway guard, not a service-level expectation.
const DefaultSessionBranchTTL = 7 * 24 * time.Hour
DefaultSessionBranchTTL is how long a session branch outlives its last write (USER 2026-10-30: kept 7 days after the session's last write, so a later session can still search it, then removed).
const DescribedUnmeasured = -1
DescribedUnmeasured marks a reading whose described fraction is UNKNOWN, as distinct from known to be zero.
The measurement is taken from layout markup that only exists during ingest, so a document indexed before that measurement existed cannot be scored without re-reading it — and 0 would claim a model made none of it up, about exactly the documents most likely to be screenshots. Negative so it can never be mistaken for a percentage, and queryable, so the set that needs re-reading can be listed rather than guessed at.
const FoldWidth = 32
FoldWidth is the shingle width, in folded bytes.
Measured on the ardley-v-brannock corpus — 233 indexed documents, 2.49 MB of fragment text, 441 true pages, 1.59 MB folded — by re-sketching the whole index at w = 16/24/32/48/64 and running the all-pairs audit at each:
w reported pairs duplicate-or-containment hardest real pair 16 292 56 0.40 24 316 58 0.49 32 399 57 0.59 48 281 65 0.55 64 364 60 0.44
Every width finds every known duplicate and every known containment, and the known false positive is suppressed at every width. So w does NOT decide the boilerplate problem — generic-text masking does (see Mask in similar.go). What w decides is how much of a DEGRADED copy is recovered, and the last column is that: the 2008 record of survey against the copy reproduced inside the 2008 lot certification, whose OCR is substantially garbled. Coverage peaks at w=32 and falls away in both directions.
It falls off above 32 for the obvious reason — one wrong character destroys w shingles, so a wide window is fragile to a bad read. It falls off BELOW 32 for a less obvious one: a short shingle is far more likely to occur across many unrelated documents, so it clears the generic-DF cutoff and real text gets masked away. At w=16 even the byte-identical Graham pair drops to 0.85 coverage, against 1.00 at w=48. Both failure modes are worst exactly on the pages that are hardest to read, which is where the tool has to work.
const FragOriginDescribed = "described"
FragOriginDescribed marks a fragment whose text is a model's account of an IMAGE rather than a transcription of writing.
It sits beside origin='identity' (identity.go), which marks a generated caption, and means the same thing in the same column: findable by it, not quotable from it. A corpus that may be cited needs the difference between "the fax says" and "the model thinks the car is a Chevrolet" to survive into the search result, and until this existed it did not.
const FragOriginFrontmatter = "frontmatter"
FragOriginFrontmatter marks the one fragment that holds a Markdown file's YAML frontmatter block, delimiters included.
Unlike 'identity' and 'fields' this fragment is NOT generated: its text is the file's own bytes [0,N), with those offsets. So it stays inside SQLOwnWords — get_document reassembles the stored text exactly, and a quote taken from the block still verifies against it. The mark exists so a renderer can say this passage is document metadata and not part of the body (Hit.Caveat), and so the block is not windowed as body text.
const GenericDocFreq = 12
GenericDocFreq is the number of DISTINCT documents a sampled shingle must occur in before its text is masked out of every alignment.
Set from the measured frequency distribution of this corpus rather than from a round number. Of the distinct sampled shingles on this corpus, 17,801 occur in exactly one document and 4,739 in two; the count is in single digits by 9 documents, and then a thin tail runs out to 56. That tail IS the corpus's chrome — the lawyer's confidentiality notice reaches 56 documents, an email header template 32 — and it is only about 60 sampled shingles, which is why suppressing it costs so little.
The cutoff must sit ABOVE the number of documents a genuinely reproduced instrument appears in, and that is what fixes the value. Measured here: the operative 2008 record of survey's text is detectably reproduced in 13 documents, 4 of them substantially (coverage above 0.25). A cutoff at or below 4 would begin masking the very instrument this exists to find, and that failure is silent.
Measured across 8/12/18/25 on the all-pairs audit — 774 duplicate-or-containment pairs with no masking at all, then 51/57/57/60. The cutoff barely matters once masking exists; what the six extra pairs at 8 are is a judgment call rather than a correctness one. All six are VENDOR TEMPLATES — three Chicago Title order notifications for the same file number, two "Scanned from a Xerox Multifunction Printer" notices — so calling them duplicates is defensible and so is suppressing them. 12 keeps them (above the 4 substantive reproductions, below the 13 incidental ones) and --generic-df exists because that judgment belongs to whoever is triaging, not to this constant.
const GlobalPattern = ":global"
GlobalPattern in a Read list matches every index with no project namespace ("code", not "proj__code").
const IngestRetry5xxAttempts = 9
IngestRetry5xxAttempts rides out an upstream episode of roughly two minutes. With exponential backoff capped at 30s the schedule is about 1+2+4+8+16+30+30+30s; RetryBudget still stops it after that.
Ingest is a BATCH, not a chat turn, and agentkit's default 5xx cap of five attempts is tuned for the latter. A 33-page document is ~33 requests, so one upstream blip outlasting ~15s of backoff kills the whole document — measured on a real corpus, nothing over 20 pages ever completed.
The wall clock is still bounded by RetryBudget, which is the guard that should bind here. Raising attempts trades a longer wait on a sick upstream for finishing the document, which for a transcription backlog is the right trade: the alternative is not "fail fast", it is "re-read 30 pages".
const KeyIdleTTL = 7 * 24 * time.Hour
KeyIdleTTL is how long a session key lives past its last use: the same 7 days a session branch outlives its last write.
const KeyPrincipalPrefix = "key#"
KeyPrincipalPrefix marks a session key's own principal.
const MinClassifyChars = 250
classify names the relation from block coverage.
Coverage and not the shingle containments, because containment is what shared boilerplate inflates: two unrelated county deeds reached ContainProbe 0.34 on statutory phrasing alone, which would have been called a containment. Their block coverage was 0.06. MinClassifyChars is the least matched text a duplicate or containment claim may rest on, in folded characters.
A coverage ratio is unusable on a short document, and this corpus is full of short documents. 0.83 coverage of a 400-character email is a signature block; 0.83 of a two-page deed is the deed. 250 folded characters is roughly a substantial paragraph — below it the pair is reported as `overlap`, which says "these share text" without asserting one is a copy of the other.
const MinReportChars = MinClassifyChars
MinReportChars is the least matched text worth reporting at all, in folded characters. Same value as MinClassifyChars, deliberately: a pair too small to say anything about is not worth a line of output either.
Gating on matched LENGTH rather than on coverage, which was the first attempt and was wrong in both directions. A 0.05 coverage floor let through 1,440 pairs on this corpus whose median matched text was 75 folded characters — two shingles, a shared address line — while a floor high enough to kill those (0.20, leaving 150 pairs) would also discard a genuine paragraph-length quotation inside a long brief, which is 0.6% of a 40,000-character title commitment and is exactly the finding this corpus turns on. Length is the property that says whether there is anything there; coverage says what it means once there is.
Measured: 250 folded characters takes the audit from 1,497 reported pairs to 257, of which 57 are duplicate-or-containment. It discards nothing strong — every duplicate and containment survives it, because they all rest on thousands of matched characters, not hundreds.
250 FOLDED characters is about 305 raw ones on this corpus's text (folding removes roughly 18% of it as spaces and punctuation), so the floor is a ~50-word passage. A shorter quotation than that is genuinely below it and needs --min-chars to surface; that is the cost, and it is the one this floor trades for not reporting 1,240 pairs whose median shared text is 75 characters.
MinSharedSamples is how many sampled shingles a document (or one of its pages) must share with the probe to be rescored.
At mod 32 an average page (3,600 folded chars) contributes ~90 samples, so 3 samples is a few percent of one page — far below anything that would be reported, which is the point: phase 1 should over-generate and let the exact rescore decide.
Measured with the operative record of survey as probe (100 sampled shingles): minShared 1 -> 19 candidate documents, 2 -> 16, 3 -> 16, 5 -> 11. So 3 costs nothing against 1 and 5 starts discarding candidates that would have been rescored. The whole probe (index load plus candidate selection) is 120 ms.
const NoisyFloor = 0.60
NoisyFloor separates "this page is the same page, read imperfectly" from "this page is different".
Set from the corpus rather than by taste: the 2008 lot certification's copy of the record of survey — the worst OCR in ardley-v-brannock, reading "LAURENCE MOONION" for Clarence Brannock — still holds most of its shingles against the clean copy. A floor low enough to call that page divergent would call every scanned page divergent, and the distinction this whole file exists to draw would be lost.
const PageJoin = "\n\n"
PageJoin separates pages in FoldedText.Raw. It folds away entirely, so it can never create or destroy a shingle — the fold is what makes the choice of separator irrelevant.
const ProjectHomeName = ".raglit"
ProjectHomeName is the per-project home directory `raglit init` creates in the working directory. Commands discover it by walking up from the cwd, so raglit is project-local like git: one .raglit/ per project, found from any subdir.
const ProjectPrincipalPrefix = "project:"
ProjectPrincipalPrefix marks a project key's principal, "project:<ns>". Every key of a project shares it, so the indexes one key creates stay reachable after that key is revoked and another minted.
const RootProject = "root"
RootProject is the name /mcp/root gives the indexes with no project namespace. It is reserved: no project can be called it.
const SampleMod = 32
SampleMod is the sampling divisor for the stored sketch: a shingle is kept when hash % SampleMod == 0.
This is NOT a MinHash signature, and the difference is the load-bearing part. A fixed-K MinHash signature estimates Jaccard well and CONTAINMENT not at all: the fraction of agreeing signature positions between a 2-page deed and the 40-page commitment containing it estimates |A∩B|/|A∪B| ≈ 0.05, and there is no way to recover |A∩B|/|A| = 1.0 from it, because the signature has thrown away the sizes. A first pass at this used K=128 MinHash and reported the deed-inside- commitment case as a 0.05 similarity with no containment signal — the exact failure the whole feature exists to avoid.
A mod-p sample (FracMinHash) is CONSISTENT: whether a shingle is sampled depends only on the shingle, never on the set it came from, so sample(A) ∩ sample(B) = sample(A ∩ B) and |A∩B| is estimable directly. Sizes are kept separately (shingle_pages.total), so both measures fall out.
32 keeps 1 shingle in 32. Measured on this corpus: 441 true pages, 1,587,596 folded characters, 1,290,196 distinct shingles, of which 39,778 are sampled — 1 in 32.4, so the sampling is doing what it says. That is ~90 samples on an average page (3,600 folded chars). The smallest region worth reporting is a paragraph, ~250 folded characters, which contributes ~8 samples at full containment — comfortably above the 3-sample floor candidate generation uses.
The whole index is 39,778 rows, against 1.29 M rows if the shingles were stored whole: a 32x saving on a table that has to be scanned per probe, and candidate generation loses nothing by it, because every candidate pair is then rescored EXACTLY over all shingles (see Compare in similar.go).
Variables ¶
var ( // ErrNoIdentifier means this store was never given a model to caption with. ErrNoIdentifier = errors.New("raglit: no identity model configured") // ErrIdentityKept means the document already has an identity that this run // must not replace — a person's, or a machine's without --force. ErrIdentityKept = errors.New("raglit: identity already recorded") )
Errors IdentifyDocument returns for the two states that are not failures.
var DefaultLaneSlots = map[Lane]int{LaneHeavy: 1, LaneLight: 3}
DefaultLaneSlots is how many jobs each lane runs at once.
Heavy is 1 and should stay 1 unless the vision endpoint genuinely admits more than one request: a second concurrent page does not start earlier, it blocks inside the server, and its retries and timeouts are then invisible here.
Light is 3 rather than "as many as there are indexes". The ceiling that matters is the embedding endpoint, which is one server however many callers there are; three keeps it busy across a batch without turning one raglit into something that needs its own rate limit.
var ErrBadKey = errors.New("raglit: unknown, revoked or expired key")
ErrBadKey is a token that is unknown, revoked, expired or malformed.
var ErrForbidden = errors.New("raglit: only the index owner may change it")
ErrForbidden is a write refused on an index the caller can read.
var ErrIgnored = errors.New("raglit: ignored")
ErrIgnored is Enqueue refusing a path an ignore pattern matches. Callers that enqueue many paths skip it and count it: one ignored file must not fail the batch it came in.
var ErrNoDocType = errors.New("raglit: this document is not one of the index's document types")
ErrNoDocType is a document that has not resolved as any registered type. Not a failure: most documents in most corpora are not forms.
var ErrNoIndex = errors.New("raglit: no such index")
ErrNoIndex is what an unreadable index looks like from outside.
var ErrNoPageImages = errors.New("raglit: no embedded page images (born-digital PDF? pagify handles image/scanned PDFs only)")
ErrNoPageImages is returned when a PDF has no embedded page images — i.e. a born-digital, vector-text PDF. raglit's pagify is pure-Go (pdfcpu) and pulls images that are ALREADY in the PDF (scanned / image PDFs); it does not rasterize vector pages. Rasterization would need a native renderer, out of scope by design.
var ErrParentOnly = errors.New("raglit: the document is in the parent index only; a branch cannot change a parent's document")
ErrParentOnly is the error (errors.Is) a branch returns for a write to a document that only its parent holds. A branch cannot change a parent's document; ingest the document into the branch first.
Functions ¶
func AppendAudit ¶
func AppendAudit(path string, ev AuditEvent) error
AppendAudit writes one event. The only writer of the trail.
Sync before returning: an event that is in the page cache and not on disk is an event a power cut turns into a decision nobody can account for.
func ApplyRegionFilter ¶
func ApplyRegionFilter(img image.Image, f RegionFilter) image.Image
ApplyRegionFilter runs one repair over a crop. An unknown filter returns the image unchanged — the caller validates; this does not silently invent one.
func AttachCheapOCR ¶ added in v0.5.0
func AttachCheapOCR(ocr *OCR, oc OCRConfig, render RenderPolicy) error
AttachCheapOCR enables the cascade's cheap first-pass engine on an OCR, with render as the resolution policy. An error is a misconfigured engine; the OCR is then left VLM-only, and the caller should report it rather than fail — a bad OCR knob must not break ingestion.
func AuthorityFromTrail ¶
AuthorityFromTrail builds the resolver Store.SetAuthorityResolver wants, from the judgement trail of the project a document belongs to.
anyDoc is any document in the corpus; the trail is found by walking up from it, the same way a correction is. Returns nil when there is no trail, which leaves the level order in charge — the right answer for a corpus nobody has ruled on, and better than an empty resolver that would claim every source is unruled.
Read ONCE and held: a search must not open and replay a trail per hit, and the rulings are a small append-only file that changes when a person acts, not while a query runs. A daemon that wants a change picked up re-installs it.
func ChildParent ¶
ChildParent returns the bundle a child path derives from, or "".
func ContentIDs ¶ added in v0.7.0
ContentIDs is the content id Store.Ingest gives each of frags, in order, for a document at path: the id a Hit carries and FragmentExists takes. A caller that keeps its own fragments (the code index) uses it to decide what it has already embedded without asking the index for every one.
Exact for Store.Ingest, which replaces every fragment of the document and numbers them in the order given. A fragment Ingest skips (blank, or bytes that are not text) still gets an id here, one that no stored fragment has.
func DPICapForArea ¶
DPICapForArea is the highest resolution at which a sheet of areaSqIn can be shown WITHOUT the server downscaling it — the bound that falls as the canvas grows, and the reason a 27x36.7in sheet cannot be read whole at any setting.
func DamageOf ¶
DamageOf reports what is measurably wrong with a crop's PIXELS, as flags.
Taken on the full-resolution crop, which is the point: the model is shown a downsampled version, and at that scale it cannot tell a blurred page from a small picture of a sharp one. Measured — asked to diagnose a crop blurred at sigma 1.6, the vision model answered "skew" and "noise" with 0.9 confidence and prescribed a deskew, which is the one repair that makes a blurred region WORSE. This function is what it could not see.
func DamageOfPNG ¶
DamageOfPNG measures encoded image bytes — the form a region's crop is recorded and reproduced in, so a pass that re-renders a recorded read can measure exactly what was read without decoding it itself.
func DefaultRoot ¶
func DefaultRoot() string
DefaultRoot is the DAEMON's storage root — $RAGLIT_ROOT, else ~/.raglit — under which each index is scoped at indexes/<name>/ (its own Home: index.sqlite + originals/ + pages/). This is the multi-index server layout (OpenScopedRegistry), distinct from a project Home (a single embedded index). The daemon's own config (endpoint + models) lives at <root>/config.json.
func DefaultTrust ¶
func DefaultTrust(method string) map[string]Confidence
DefaultTrust is what a reading produced by this method can be trusted on, before anybody has ruled.
A facet that is ABSENT is not untrusted, it is not claimed: a page transcription asserts nothing about who spoke, and reporting 0 for it would read as "we know it is wrong".
func DescribableRead ¶
DescribableRead reports whether a page's raw text came from a reader that SAYS what it described — the only kind that can be measured.
The distinction between "described nothing" and "cannot tell" lives here, and it is not academic. A layout-aware read wraps every block in a labelled region and a figure-caption read leaves `[FIGURE: ...]` inline; a page from either can be scored, and 0 means the model described nothing. A page with NEITHER came from a reader that was never asked to mark descriptions — an older prompt, a different model — and its text is a transcription and a description run together with no seam. Measured on the fence-dispute index: 654 of 1422 cached vision pages are like this.
Scoring those 0 would be the worst available answer, because it looks measured. See DescribedUnmeasured.
func DescribedFraction ¶
DescribedFraction is how much of a page's INDEXABLE text came from the model describing an image, between 0 and 1. A page with no such regions is 0; a photograph is ~1.
BOTH description mechanisms count. A layout-aware read wraps what it describes in an Image region with the text in an <img alt>; the figure-caption path writes `[FIGURE: ...]` inline instead (figures.go), and the corpus holds both — measured on the fence-dispute index, 768 cached pages carry layout markup and 201 carry figure markers, with no overlap. Counting only the first scored a page whose only description is `[FIGURE: Logo for Harrow Brand & Teal…]` at zero, which is not "we could not tell" but "a model made none of this up".
func DynamicRange ¶
DynamicRange is the 1st-to-99th percentile spread of luminance. Percentiles rather than min/max because one black speck and one white pixel would report a full range on a page that is entirely mid-grey.
func EnqueueBuiltinIgnore ¶
func EnqueueBuiltinIgnore() []string
EnqueueBuiltinIgnore is the ignore list every index applies, for display.
func EscalatePrompt ¶
EscalatePrompt asks a parent to decide GEOMETRY, and forbids a transcription.
Separate from Prompt because it takes an argument — the case being put — and because it replaces rather than composes. `keep` means the child's reading stands, never "here is a better one": anything the parent transcribed here would be the low-resolution invention this package exists to prevent.
func EstimateTokens ¶
EstimateTokens is the token count raglit assumes for a piece of text.
Deliberately an OVER-estimate: it must never say a string is smaller than the tokenizer will. Shared so fragment sizing and batch sizing agree, because disagreeing about the unit is how a limit gets enforced in one place and ignored in another.
func ExtForContentType ¶
ExtForContentType maps a content type to a file extension for materializing fetched/base64 bytes to a temp file (external tools detect format by extension). Empty when unknown — the caller should sniff or default.
func ExtractEmailAttachments ¶
ExtractEmailAttachments writes every attachment in the archive at srcPath into a sidecar beside archivePath, and returns how many files it wrote.
srcPath and archivePath differ on the ingest path: the worker materialises fetched bytes to a temp file, but the sidecar belongs next to the document's real location in the corpus. Passing both keeps the bytes and the identity from being confused for one another.
The archive is parsed a second time rather than threaded through EmailText. EmailText must stay pure — the `ocr` MCP tool calls it — and a second walk of 24 MB costs milliseconds, which is not worth a sink parameter that one of the two callers would always pass nil for.
func FigurePromptVersion ¶
func FigurePromptVersion() int
FigurePromptVersion is the current figure-description prompt version, folded into recipe hashes (pool + frag_recipe) so a prompt change reprocesses the affected documents.
func FlattenForIndex ¶
FlattenForIndex is what the index should hold for a page: layout markup removed, markdown tables reduced to plain lines.
The two halves were written weeks apart for the same reason and neither was wired to anything. Composed here so there is ONE answer to "what goes in the index", and one place to change it.
func FlattenMarkdownForIndex ¶
FlattenMarkdownForIndex turns a stored transcription into the plain lines the index should hold.
The artifact keeps its markdown: a monument table read as a table is worth having, and a reader opening <doc>.raglit-transcription.md should see one. The INDEX wants neither the pipes nor the rules — FTS5 tokenises them away as separators, but they cost fragment budget, they carry into embeddings as noise, and a cell boundary inside a phrase is a boundary the searcher did not type.
Cells become space-separated on one line, so a row stays one unit of text and a phrase spanning two columns still matches. Alignment rows (|---|:--:|) carry no content and are dropped entirely.
func GCBranches ¶ added in v0.8.0
GCBranches removes each branch home whose branch.json names a root for which alive returns false, and returns how many it removed. A home with no branch.json, or a branch with no root (a session branch), is never removed. The caller lists its branch homes and must not have them open.
func HEICToPNG ¶
HEICToPNG decodes a HEIC/HEIF photo to PNG bytes for the OCR path. PNG, not JPEG: these are photographed surveys, plats, permits and correspondence headed straight for OCR, and a second lossy generation ahead of OCR costs accuracy on exactly the small print that matters — the first (HEIC's own) generation is unavoidable, a second is not.
-auto-orient matters and is not a no-op to skip: a phone's sensor is landscape, and a portrait shot only reads right-side-up because of a rotation signal carried alongside the pixels — an OCR pass over the raw sensor orientation reads sideways text and transcribes it to nothing. HEIF carries that signal two ways: a container-level irot/imir transform (the MIAF-compliant mechanism nearly all camera-generated HEIC actually uses) or, occasionally, a bare Exif Orientation tag as compatibility metadata. -auto-orient applies either; ImageMagick's HEIC decoder historically only honored the container form (github.com/ImageMagick/ImageMagick issue #1232, fixed by commit ba470aad in the version this machine has), which is the form real camera output actually carries, so this is not a synthetic concern.
func HaveHEIC ¶
func HaveHEIC() bool
HaveHEIC reports whether a HEIC/HEIF → PNG converter is on PATH.
func HaveLegacyDoc ¶
func HaveLegacyDoc() bool
HaveLegacyDoc reports whether a binary .doc converter is on PATH.
func HavePandoc ¶
func HavePandoc() bool
func HavePoppler ¶
func HavePoppler() bool
HavePoppler / HavePandoc report whether the external extractors are available, so callers (and `raglit doctor`) can degrade gracefully.
func HaveXLS ¶
func HaveXLS() bool
HaveXLS reports whether xls2csv (the legacy-.xls converter, part of the same catdoc package as antiword's fallback) is on PATH.
func HintBlock ¶
HintBlock renders the hint for a prompt, or "" when there is none. One wording, so the four prompts that carry it cannot present it differently — and labelled as the corpus owner's words, so a model weighs it as context about the collection rather than as an instruction from the document.
func HitPage ¶
HitPage resolves the page a match sits on, given the fragment's recorded page spans and the query that found it.
startPage is returned unchanged when the fragment covers one page, when the spans are missing, or when no query term can be located — all three mean "nothing better is knowable here".
func IdentityKinds ¶
func IdentityKinds() []string
IdentityKinds is every vocabulary, for a help string or a UI filter.
func IdentityRoleKinds ¶
func IdentityRoleKinds() []string
IdentityRoleKinds is the vocabulary, for a help string or a UI filter.
func IdentityTextHash ¶
IdentityTextHash fingerprints the text a caption is written from. Content only — the same words in the same order hash the same however they were fragmented, so a re-fragmentation that changes no words does not read as a changed document.
func ImageTooSmallToBeAPage ¶
ImageTooSmallToBeAPage reports whether an image is below the floor, with its dimensions for the message. Undecodable images pass — a format this build cannot read is not evidence of anything, and the OCR path will say so better.
func IndexTrees ¶ added in v0.9.0
func IndexTrees(ctx context.Context, stores map[string]*Store, root string, o TreeOptions) (map[string]TreeResult, error)
IndexTrees is IndexTree over several indexes: o.Route picks each file's index, and a document under root in an index its file does not route to is Removed from it. A file routed to a name not in stores is in none.
func IsChildPath ¶
IsChildPath reports whether a document path denotes a derived slice rather than a file. Anything that checks the filesystem must ask this first: a child has no file, and reading its absence as "the source vanished" would delete documents that are working exactly as designed.
func IsDescribedPage ¶
IsDescribedPage reports whether a page is essentially a model's description of an image rather than a transcription of text.
The BINARY question, and it stays binary because what it decides is binary: whether a fragment may be quoted as the record. A page mostly transcribed with a figure in it must not be marked, because the transcription is real.
func IsGeneratedSidecar ¶
IsGeneratedSidecar reports whether a path is raglit's own output rather than a source document. Generated output is never a source: indexing a transcription produces a transcription of the transcription, and the backlog grows every time the indexer runs.
func IsMailArchive ¶
IsMailArchive reports whether a path is one this reader handles.
func IsMinified ¶
IsMinified reports whether these bytes have no line structure a human put there — a minified bundle, a generated blob, a one-line data dump.
Such a file is TEXT, so IsOpaque passes it and the reader indexes it happily: measured here, a 289 KB minified xterm.js would fragment into hundreds of chunks of unreadable JavaScript and embed every one, answering no question anyone will ever ask. It is git-tracked, so a caller filtering by .gitignore cannot help — the defence has to be about the bytes.
func IsMixedPage ¶
IsMixedPage reports a page that both transcribes and describes.
The case the binary test misses, and it is not rare — it is what a SCREENSHOT is. chandra reading a page of SMS messages transcribes the messages and narrates the phone around them: the status bar, the app icons, the microphone and camera buttons.
measured on the fence-dispute SMS exhibit 2026-08-15 15 pages, all read by the VLM, 15% of the document described per page 0% to 28% — 13 of 15 pages carry some, 2 carry none
Every one of those pages failed IsDescribedPage and was recorded as pure transcription, so the reading claimed `text 90%` and claimed nothing at all about subject. That overstates one thing and hides another: a reader searching the text gets the model's prose back mixed in with the record — arrow, battery, blurry, button, camera, compose, icon, logo, microphone, navigation — with no sign of which is which.
func IsOpaque ¶
IsOpaque reports whether these bytes have no representation raglit can index: not text, not an image, not audio or video, not a document format.
It exists because KindUnknown means two different things and the reader treats them the same. "A text file whose extension I do not recognise" and "a compiled binary" both fall through to being read as plain text, and reading bytes as text never fails — so a 27 MB ELF executable indexed cleanly, filled an index with mojibake, and reported `done`. Measured: dun handed its own build output to an ingest and most of a pool's growth from 49 MB to 237 MB was compiled binaries being segmented and embedded as though they were documents.
http.DetectContentType is the stdlib's WHATWG sniffer: it names the common text, image, audio, video and archive types and says application/octet-stream for everything else. Measured on the file that prompted this — the ELF sniffs as application/octet-stream, every .go/.md/.sum in the same tree as text/plain. `file(1)` answers correctly too and was considered; it costs a process per file and an external binary, against the pure-Go single-binary property this project holds elsewhere (see embed.go).
The NUL check is the second signal for the same reason git uses it: a byte that cannot appear in text is the cheapest proof there is, and it catches binaries whose first bytes happen to sniff as something benign.
audio/ and video/ are deliberately ABSENT from the pass list below, though the sniffer names them and an earlier version of this function let them through. This list is not "types the sniffer recognises", it is "types that may fall through to the TEXT READER" — which is the only thing past this guard.
Media has a route of its own now (KindAudio → oidio → index the transcript), and Worker.route takes it before anything reaches here: by extension, by served content type, and failing both by sniffing the bytes. So a recording should never arrive at this function at all, and one that does is a file whose identity nothing could establish. Handing THAT to the text reader is the failure this guard exists to stop, so the prefixes stay out.
Restoring them would not re-enable audio ingest — it would only re-open the fall-through. Measured before the route existed: an .mp4 of a court hearing was stopped by IsMinified alone, on the incidental grounds that its bytes hold a 5000-character run with no newline (208,179 in that file). The .opus beside it tops out at 3,508 across 7 MB and would have gone straight through. A line-length heuristic agreeing with the right answer is not the same as a check that knows it.
func IsRegionsSidecar ¶
IsRegionsSidecar reports whether a path is raglit's own region output.
func IsTranscription ¶
IsTranscription reports whether a path is raglit's own transcription output.
func IsTransient ¶ added in v0.9.0
IsTransient reports whether err is a failure outside the document: the context cancelled or timed out, a dial or connection error, an HTTP 5xx, 429, 401, 403, 404 or 408 from a model or fetch endpoint, or a spent model budget (402, which refills). A document error (extract or parse failure, unsupported or unreadable content) is not transient.
func JudgementsPath
deprecated
JudgementsPath is retained so callers compile unchanged. The trail is the store; there is no database. JudgementsPath is retained as an alias for AuditPath.
Deprecated: there is no judgements database. The name outlived the file it named, and callers asking "where are the rulings" want the trail.
func LaplacianVariance ¶
LaplacianVariance is the variance of the 4-neighbour Laplacian: the standard cheap focus measure. Sharp edges produce large second derivatives, blur flattens them, so a low number means the strokes have been smeared.
func LegacyAttachmentDir ¶
LegacyAttachmentDir is where an archive's attachments USED to be extracted to: a sidecar directory in the corpus, beside the archive. Kept only so the migration can find them; nothing writes here any more. See Home.AttachmentDir.
func LegacyDocText ¶
LegacyDocText converts a pre-2007 binary Word .doc to plain text.
Why not LibreOffice. `libreoffice --headless --convert-to docx` is the obvious answer and it is the wrong one here: it is a ~1GB dependency to read a format that is almost always a letter, it costs seconds per file against milliseconds, and it serialises on a single user-profile lock — so raglit's concurrent workers would either collide or need a throwaway -env:UserInstallation per job. antiword is ~200KB and does nothing else.
The trade it makes: antiword drops embedded images, so a .doc whose content is a scanned figure extracts as empty. That is the right trade for correspondence and agreements; if a .doc ever matters for its figures, convert it to PDF and let the PDF path rasterize and describe them.
func LegacyRelationsPath ¶
LegacyRelationsPath is the append-only file rulings used to live in, kept only so an existing corpus can be migrated into the database once.
func MaterializeSlice ¶
MaterializeSlice writes the derived child document for a declaration.
The child's text is the parent's pages, carried over unchanged, and its page numbers stay the PARENT's: a child of pages 3-6 has pages 3, 4, 5, 6. Never renumbered to 1-4 — a quotation cited from the child has to be checkable against the exhibit as filed, and renumbering makes "p. 1" mean two different sheets depending on which document you are holding.
One fragment per page rather than the windowing fragmenter. The child exists to be cited and scoped, and the parent is already indexed at fragment grain for deep search; keeping page and fragment the same unit here means an offset in a child result needs no translation to become a page in the bundle.
Idempotent: Ingest replaces a document's fragments, so re-running after the parent is re-read refreshes the child. That is the whole reason the bundle stays the single source of bytes — one thing to fix, and the children follow.
func NativeDPI ¶
NativeDPI reports the resolution of the largest image on a PDF page, or 0 when there is none (born-digital) or poppler is unavailable.
Largest rather than first: a scanned sheet often carries a small logo or signature stamp alongside the page raster, and the smaller image's ppi says nothing about what the page holds.
func NormalizeAnyKind ¶
NormalizeAnyKind maps a kind onto the first profile that knows it. A bare legal term ("deed", "court filing") is accepted for what was stored before kinds had families.
func NormalizeIndexName ¶
NormalizeIndexName is the exported form of normalizeIndexName, for clients that build daemon index names (e.g. project namespacing).
func NormalizeKind ¶
NormalizeKind maps a kind onto the vocabularies — any profile's, since a person may file a document under any family — returning ok=false when no profile knows it. A bare legal term ("deed") is what was stored before kinds had families, and still maps.
func NormalizeRole ¶
NormalizeRole maps a model's role tag onto identityRoleKinds, returning ok=false when it is not a term this vocabulary knows.
func NormalizeTypeName ¶
NormalizeTypeName is how a type name is compared: lowercased and whitespace-collapsed, so "Work Order" and "work order" are one type rather than two that each hold half the corpus.
func OfficeText ¶
OfficeText extracts text from any KindOffice document, choosing the converter by extension — except an OLE2 header overrides the extension, because a renamed .doc is still a .doc underneath and pandoc still cannot read it. Callers route KindOffice here rather than to PandocText directly, so adding a format that pandoc cannot read is a change in one place.
func PageAt ¶
PageAt resolves a byte offset inside a fragment's text to the page it falls on. This is the reason the column exists: a search hit is an offset, and without the boundaries the only answer available was the fragment's start page, which is wrong for everything after the first stitch.
func PandocText ¶
PandocText converts an office/markup document (docx, odt, epub, html, pptx, …) to plain text via pandoc, which auto-detects the input format from the file extension. The path must have the right extension.
func ParseRange ¶
ParseRange reads "3-6" or "3" as an inclusive page range.
Inclusive because that is how a person reads a document: "pages 3-6" is four pages, and a half-open range here would silently drop the last page of every slice anybody declared by eye.
func PlanSources ¶
PlanSources returns, per index name, the absolute file paths its configured roots + rules select. baseDir is the project directory (relative roots resolve against it). It shells out to `git ls-files` for a root's .gitignore semantics when the root is a git work tree and cfg.Gitignore isn't false.
func PrincipalFrom ¶
PrincipalFrom returns the authenticated account, "" when there is none.
func ProjectDir ¶
ProjectDir is the directory that OWNS this project — the one holding .raglit/, found by walking up from the cwd like git.
Distinct from the index Home on purpose. The Home is derived storage: it is gitignored in every project that has one, it is rebuilt by a reindex, and on a daemon it does not even sit near the documents. Anything DURABLE about the corpus — a human's ruling that two documents are versions of one instrument, which nothing can recompute — belongs here instead, beside the documents it is about, where git records it and a folder sync carries it to the other machines.
func ProjectDirForDoc ¶
projectFlagForDoc walks up from a document for the nearest project config and lets it decide, falling back to the store's setting when there is none. See writebackForDoc for why the document's path, and not the daemon's config, is the right place to ask. ProjectDirForDoc finds the project a document belongs to, by walking up from the document itself until it meets a .raglit/ directory.
The daemon needs this and cannot use ProjectDir(): it runs from its own home and its working directory has nothing to do with the corpus. What it always has is the document's absolute path, and the corpus layout answers the rest — the same walk that already decides per-project writeback settings.
Returns "" when the document sits under no project, which is a real state (an ad-hoc file ingested by path) and not an error.
func Prompt ¶
func Prompt(k PromptKind, mods ...Mod) string
Prompt assembles the instruction for one read.
Modifiers on a kind that does not accept them are DROPPED, not silently appended: see AcceptsMods for why each would contradict its prompt.
func Recipe ¶
Recipe identifies the comparison parameters. Mirrors documents.frag_recipe: a parameter change should mark exactly what it affects, and a finding computed under one recipe is not comparable to one computed under another.
func RecordReadingsInto ¶
func RecordReadingsInto(js Judgements, s *Store)
RecordReadingsInto wires a judgement store to an index so that every change to a page's ACTIVE reading lands as a row.
This is the hook the interface exposes, used for its intended purpose: a correction invalidates things it does not own — the export beside the document and the fragments in the index — and neither notices on its own. Registering here means a person correcting a page leaves a visible, queryable version history rather than a silently changed file.
func RegionSHA ¶
imageSHA identifies a rendering by its bytes.
This is also the cycle detector, at no extra cost. A transform that renders to bytes already seen IS the cycle: rotating 90 degrees four times returns the original SHA, and re-asking for the same bbox at the same dpi and rotation returns the original SHA. RegionSHA digests rendered image bytes the way a region's record does. Exported so a caller outside this package can construct or check a region record against an image it rendered itself.
func RegionsPath ¶
RegionsPath is where a document's region reads live: beside it, like the transcription.
func RegionsSuffix ¶
func RegionsSuffix() string
RegionsSuffix is what a region sidecar's name ends in, for a caller walking a tree and mapping sidecars back to the documents they describe.
func RenderTranscription ¶
func RenderTranscription(docPath string, pages []TranscribedPage) string
func RenderTranscriptionCorrected ¶
func RenderTranscriptionCorrected(docPath string, pages []TranscribedPage, corrections map[int]PageCorrection) string
RenderTranscription formats pages as markdown.
Deterministic: same pages in, same bytes out, so a diff between two transcriptions is a real change in what was read rather than formatting noise. RenderTranscriptionCorrected renders with a person's page corrections applied.
The corrections are NOT stored in the file this produces. That file is regenerated on every read, and corrections kept in it were destroyed twice by ordinary re-reads before they were moved into the judgement store. Rendering applies them; the store keeps them.
func RerenderRegion ¶
RerenderRegion reproduces the crop a region was read from: same bbox, same rotation, same page rasterization.
page must be the page rasterized at reg.DPI — the caller owns that, because rasterizing a PDF needs a native renderer this package deliberately does not link (see pagify.go). The result is byte-identical to what the descent produced; VerifyRegionRender is how you find out, rather than assuming.
This is the image to put in front of a human. It is the passage at the resolution it was read at, which is the whole point, and where a context downscale intervened it is SHARPER than what the model got — more detail on the same pixels, never different pixels. RerenderRegionAsSeen is for the other question.
func RerenderRegionAsSeen ¶
RerenderRegionAsSeen is the crop with the context downscales replayed: what the model was actually given, rather than what was cropped for it.
A diagnostic, not the attestation image. It answers "could this have been read at all", which is a different question from "does the document say this", and it is deliberately NOT what reg.SHA256 covers: the digest is taken before the call, because it is also the descent's cycle detector. So this is reproducible — same crop, same factor, same count — but verifiable only through the crop it is derived from.
func ResolveExtractedAttachments ¶
ResolveExtractedAttachments maps sha256 → extracted file path, by hashing what is in the given directories.
By CONTENT, not by name. The extractor dedups within an archive — the same PDF on five messages in a thread is one file cited five times — so the filename it chose encodes the FIRST message that carried it and reconstructing it per message would point four of those five at a file that does not exist.
Hashing the directory on each call is affordable because these directories hold tens of files, and it cannot drift from whatever is actually on disk, which reconstructing a name from the extractor's format string could. Several directories because an archive may not have been migrated yet: current extractions land in raglit's storage (Home.AttachmentDir) and older ones are still in the corpus sidecar (LegacyAttachmentDir). Reading both means the thread view works either side of the migration.
func ResolveFragParams ¶
ResolveFragParams applies defaults for non-positive values, caps the window by the embed model's input limit when known (embedLimit > 0), and enforces the windower's invariants: stride ≤ ¾·window (so a boundary-snapped window can't leave a coverage gap before the next window's start) and floor ≤ window.
func STTReading ¶
STTReading converts oidio's response into the reading attest reviews.
One unit per speaker turn, located by TIME — which is the locator attest already defines for audio and the reason Locator has a Time arm at all. The speaker becomes the unit's Label rather than part of its Text, because they are separately correctable: a reviewer can rule that the words are right and the speaker is wrong, and attest's own comment says conflating those two is what made oidio's earlier workbench painful. Sealed before it is returned, so the units carry content ids rather than the "t0, t1, …" handles used while building. Seal treats those as the producer's own handles and rewrites them, which is exactly what they are.
func SaveConfig ¶
SaveConfig writes cfg to <home>/config.json (0600 — it holds a token), creating the home layout if needed.
func ScopeMatches ¶
ScopeMatches reports whether dir lies in one of scopes. A plain directory matches itself and everything under it; a pattern with * or ? is a glob in the ignore-pattern syntax (** crosses directories), matched against dir.
func SessionBranchParent ¶ added in v0.4.0
SessionBranchParent returns the index a session branch name overlays ("proj__code@s1" → "proj__code"), and false for a name that is not one.
func Shingles ¶
Shingles hashes every width-w window of folded text, at stride 1.
Result index i is the shingle STARTING at folded offset i, and callers depend on that: the alignment in similar.go recovers matched spans by chaining positions, so a position must mean an offset and nothing else. Fewer than w bytes yields nothing — a page too short to shingle contributes no evidence, which is correct, but it is also why a page's folded length is stored alongside its shingle count.
func SourceFile ¶
SourceFile is the file on disk a document was read from, and false when the document has none to check: a URL, or a relative path.
A page range (slices.go, "<file>#p3-8") is its parent's file. A file inside another file (an archive member) is not detected here: its path does not exist even while the archive does, so it would read as missing. Members are written out to real files (emailattach.go), so none exist today.
func StripLayoutMarkup ¶
StripLayoutMarkup removes a VLM's layout markup, keeping the text and the figure descriptions.
Block tags become line breaks and inline tags become nothing — the distinction is the whole point, and getting it backwards is worse than not stripping at all, because it silently corrupts phrases rather than obviously leaving tags.
func SuspectDocs ¶
SuspectDocs returns the documents under root whose transcription has at least one page that does not look like a page, with the page numbers.
Reads the SIDECARS rather than the index: the sidecar is what a human and kgraph both actually consume, so a document whose sidecar is wrong is wrong for every purpose that matters even if the index rows happen to be fine.
func SuspectPages ¶
func SuspectPages(pages []TranscribedPage) map[int]string
SuspectPages returns the page numbers a human should look at, with reasons.
func Suspicion ¶
Suspicion returns a human-readable reason this page looks wrong, or "".
The reason is written for the person who has to act on it: what was seen, and what it usually means.
func TagLine ¶
TagLine renders tags as "tag(count), tag(count)" — the form the identity prompt takes its vocabulary in, and the form the digest prints.
func TilesNeeded ¶
TilesNeeded is how many equal pieces a sheet must be cut into for every pixel to reach the model at dpi. Ceil, because a fractional tile still costs a call and a sheet 1.1x over the cap still loses a tenth of itself.
func TokensPerSqInAt ¶
TokensPerSqInAt is the encoder's cost of one square inch rendered at dpi.
func TokensToChars ¶
TokensToChars converts a token budget to the character budget that cannot exceed it, whatever the text.
The exact inverse of EstimateTokens, allowance included. Without subtracting it the budget overshot its own cap by the allowance — a budget that fails its own check is worse than no budget, because it looks like a guarantee.
func TranscriptText ¶
TranscriptText renders the transcript raglit INDEXES.
Written for a reader and for retrieval, not as a data format — the reading sidecar is the data format. Speaker and timestamp lead each turn so a search hit can be taken back to the recording, which is the whole reason a transcript of a hearing is worth indexing at all.
func TranscriptionPath ¶
TranscriptionPath is where a document's transcription lives: beside it, with a suffix that says what wrote it, so a re-transcription is detectable and a human can tell it from a hand-written note.
func ValidRegionFilter ¶
func ValidRegionFilter(f RegionFilter) bool
ValidRegionFilter reports whether a name is one this package can apply. A proposal naming anything else is refused rather than silently ignored, so a model inventing "denoise" is visible instead of quietly doing nothing.
func ValidateFieldSchema ¶
func ValidateFieldSchema(raw json.RawMessage) error
ValidateFieldSchema refuses a schema that is not a JSON Schema OBJECT with properties. The extraction call turns it into a tool's `parameters`, and an endpoint handed a malformed one fails in a way that reads as a model problem.
func VerifyRegionRender ¶
VerifyRegionRender reports whether a re-render is the image the region's text was read from.
Byte equality, via the digest the read itself recorded. A weaker check — same bbox, same rotation — would pass an image cropped from a page rasterized by a different tool at the same nominal dpi, which is precisely the substitution a human attesting a quotation is being asked to rule out.
func WithPrincipal ¶
WithPrincipal records the authenticated account on a request context.
func WorktreeOf ¶ added in v0.8.0
WorktreeOf returns the work tree of the main checkout when dir is inside a linked git worktree. ok is false for a main checkout, a bare repository's worktree and a directory outside git. It asks git (rev-parse --git-common-dir) and, when git cannot run, reads the .git file's gitdir pointer. It only answers the question; the caller decides which index is the parent.
mainRoot is spelled as git records it, which is usually with symlinks resolved, and may differ from the spelling of dir and of the paths an index holds. Before passing it to OpenBranch, spell it as the parent index's document paths are spelled (see OpenBranch).
func WriteRegionDoc ¶
func WriteRegionDoc(docPath string, pages ...RegionPage) (string, error)
WriteRegionDoc merges page reads into the sidecar beside a document.
Merges rather than replaces: `raglit regions` reads ONE page, and a run over page 7 must not delete the record of page 3 — the crop that page 3's quotation was read from would go with it, and the quotation would still be in the filing.
func WriteTranscription ¶
func WriteTranscription(docPath string, pages []TranscribedPage) (string, error)
WriteTranscription materialises the transcription beside the document.
Best-effort by design: a corpus can be read-only, or on a mount that is gone, and failing an otherwise-good ingest because a convenience file could not be written would be the wrong trade. The caller logs and continues.
func WriteTranscriptionCorrected ¶
func WriteTranscriptionCorrected(docPath string, pages []TranscribedPage, corrections map[int]PageCorrection) (string, error)
WriteTranscriptionCorrected writes the export with a person's corrections applied. See RenderTranscriptionCorrected for why they are applied here rather than stored here.
Types ¶
type ACL ¶
type ACL struct {
Owner string `json:"owner"`
Visibility Visibility `json:"visibility"`
}
ACL is an index's owner and visibility, stored at <index home>/acl.json.
type Assembler ¶
type Assembler struct {
// MinChars: absorb sub-floor siblings up to this size (0 disables the floor).
// MaxChars: never absorb past this.
MinChars, MaxChars int
// contains filtered or unexported fields
}
func NewAssembler ¶
NewAssembler builds an Assembler; sink finalizes a closed fragment (e.g. insert row + hand to the embed pipeline).
type AttachmentMove ¶
type AttachmentMove struct {
Archive string `json:"archive"`
OldPath string `json:"old_path"`
NewPath string `json:"new_path"`
// Err is set when this one could not be moved. The migration continues: a
// half-moved corpus with a report is recoverable, a migration that stops at
// the first bad file leaves nobody knowing which side of it they are on.
Err string `json:"error,omitempty"`
}
AttachmentMove is one document this migration would move, or moved.
type AttestationRow ¶
type AttestationRow struct {
Doc string
Seq int
Unit string
Kind string
Blanket bool
Text string
Label string
Note string
Statement string
Payload string
RuledBy string
Auth string
RuledAt string
}
AttestationRow is one entry of one asset's log, as stored.
type AuditEvent ¶
type AuditEvent struct {
Op string `json:"op"`
// At is when the event was appended, not when a person decided — the decision
// date lives on the payload. Both matter and they are not the same: a ruling
// made in July and imported in September is one of each.
At string `json:"at"`
By string `json:"by,omitempty"`
Relation *Mark `json:"relation,omitempty"`
Slice *Slice `json:"slice,omitempty"`
SliceID string `json:"slice_id,omitempty"`
Correction *PageCorrection `json:"correction,omitempty"`
Withdrawal *Withdrawal `json:"withdrawal,omitempty"`
// RestorePath names the document a doc.restore returns to the corpus.
RestorePath string `json:"restore_path,omitempty"`
// Authority names the reading a person ruled governs for a source. See
// ReadingAuthority — it is a ruling, so it lives in the trail rather than in
// the index, which is derived and thrown away by a reindex.
Authority *ReadingAuthority `json:"authority,omitempty"`
}
AuditEvent is one recorded mutation.
func ReadAudit ¶
func ReadAudit(path string) ([]AuditEvent, error)
ReadAudit parses the whole trail, in order. A missing file is an empty trail, which is the normal state of a corpus nobody has ruled on.
A malformed line is an ERROR and not a skip. The trail is the source of the database, so silently ignoring a line it cannot parse would rebuild a database that quietly disagrees with the record — the failure this whole arrangement is meant to make impossible.
type AuthConfig ¶
type AuthConfig struct {
// Issuer is the OIDC issuer URL (discovery at <issuer>/.well-known/
// openid-configuration). Empty → the daemon does not authenticate.
Issuer string `json:"issuer,omitempty"`
// Audience is the token's required aud: the client id (or project id) the
// IdP issues raglit tokens for.
Audience string `json:"audience,omitempty"`
// Admin is the principal ("<iss>#<sub>") that owns every index without an
// acl.json — the indexes from before auth. Required when Issuer is set.
Admin string `json:"admin,omitempty"`
// ClientID and RedirectURL turn on BROWSER login (authorization code +
// PKCE, session cookie) for the review UI. ClientID is a public (no secret)
// web/user-agent application at the IdP; RedirectURL is
// <daemon public URL>/auth/callback, registered there exactly. Both empty →
// the browser gets 401 like any client without a token.
ClientID string `json:"client_id,omitempty"`
RedirectURL string `json:"redirect_url,omitempty"`
// TokenCommand prints a bearer token on stdout. The client runs it when
// $RAGLIT_TOKEN is unset, and again after a 401. A headless client (a
// Zitadel machine user) puts its client-credentials exchange here.
TokenCommand string `json:"token_command,omitempty"`
}
AuthConfig is the daemon's OIDC setup (issuer/audience/admin) and the client's token source (token_command). One block because one config.json serves both sides on a single machine; a remote client sets only token_command.
type Block ¶
type Block struct {
ProbeFromPage int `json:"probe_from_page"`
ProbeToPage int `json:"probe_to_page"`
MatchFromPage int `json:"match_from_page"`
MatchToPage int `json:"match_to_page"`
// Runs is how many separate identical spans the block was chained from. 1
// means the region is identical throughout; a high count on a short block
// means the two copies differ constantly, which is itself the finding.
Runs int `json:"runs"`
// MatchedChars is folded characters proved identical inside the block;
// SpanChars is the block's extent on the probe side. Agreement is their ratio.
MatchedChars int `json:"matched_chars"`
SpanChars int `json:"span_chars"`
// Gaps are the disagreements inside the block, largest first, capped.
Gaps []Gap `json:"gaps,omitempty"`
// contains filtered or unexported fields
}
Block is a chained group of runs — one reportable aligned region, with the page ranges that make it actionable.
type BranchChange ¶
type BranchChange struct {
Path string `json:"path"`
Change string `json:"change"` // "added", "changed", "same" (a copy with the parent's bytes) or "deleted"
Hash string `json:"hash,omitempty"`
ParentHash string `json:"parent_hash,omitempty"`
Title string `json:"title,omitempty"`
}
BranchChange is one document a branch differs from its parent on.
type BranchInfo ¶
type BranchInfo struct {
Name string `json:"name"`
Parent string `json:"parent"`
CreatedAt int64 `json:"created_at"`
LastAccessedAt int64 `json:"last_accessed_at"`
Documents int `json:"documents"`
}
BranchInfo describes a branch for listing: lineage, age, last access, and its LOCAL document count (the branch's own diffs over the parent).
type BranchMeta ¶
type BranchMeta struct {
Parent string `json:"parent"`
Root string `json:"root,omitempty"`
ParentRoot string `json:"parent_root,omitempty"`
CreatedAt int64 `json:"created_at"`
LastAccessedAt int64 `json:"last_accessed_at"`
}
BranchMeta records a branch's lineage + access times. Stored as branch.json in the branch's scoped Home; absent for non-branch indexes.
Parent is an index name for a Registry branch and the parent's home directory for one opened with OpenBranch. Root and ParentRoot are the worktree's and the main checkout's directories (OpenBranch); empty for a branch that does no path translation.
Fields:
- Parent: the parent index (name or home directory, see above).
- Root: the worktree directory this branch indexes; empty without path translation.
- ParentRoot: the main checkout directory the parent indexes; empty without path translation.
- CreatedAt: when the branch was forked, in Unix nanoseconds.
- LastAccessedAt: when the branch was last read, in Unix nanoseconds. Updated at most once a minute, so it can lag a real query by up to that long.
func ReadBranch ¶ added in v0.8.1
func ReadBranch(home Home) (BranchMeta, bool, error)
ReadBranch reads home's branch.json without opening the index. ok is false, with a nil error, when the home has no branch.json (a plain index). An unreadable or malformed branch.json is an error, so a caller does not take a damaged branch for a plain index.
type Changes ¶ added in v0.9.0
Changes is one burst. Paths are absolute, sorted, and may name directories and paths that no longer exist; IndexTree's Paths takes them as they are. Empty Paths means the whole tree (Rescan, an event overflow). Degraded means the watcher hit the OS watch limit and stopped adding watches: from then on changes in new directories are not seen, and the caller rescans or polls on its own trigger.
type ChannelStat ¶
type ChannelStat struct {
Model string `json:"model"`
Width int `json:"width"`
InFlight int `json:"in_flight"`
Peak int `json:"peak"`
Calls int `json:"calls"`
N429 int `json:"n_429"`
// Chilled is true while backpressure is still holding the width down.
Chilled bool `json:"chilled,omitempty"`
}
ChannelStat is one model's admission channel, for the status report.
type Channels ¶
type Channels struct {
// Max is the widest any channel may grow. 0 → maxChannelWidth.
//
// The one knob worth configuring, because the ceiling is the only part of
// this that depends on WHAT KIND of endpoint is behind it rather than on
// evidence. A local card serving one model at a time is never going past a
// handful; a hosted provider bought for throughput will take far more, and
// there the ceiling — not the provider — would be the limit. Everything else
// is learned.
Max int
// contains filtered or unexported fields
}
Channels holds one admission channel per model name.
func NewChannels ¶
func NewChannels() *Channels
NewChannels builds an empty registry. A model is created on first use, so nothing needs to know the model layout in advance — which is the property that makes a changing layout a non-event.
func (*Channels) Acquire ¶
Acquire takes a slot for one call to model, blocking until one is free or ctx ends. The returned release must be called exactly once.
release takes the call's outcome: ok=false for a call that failed for any reason other than backpressure. Backpressure is reported separately by Note429, because agentkit RETRIES a 429 internally — the call that eventually succeeds still met backpressure, and a controller that only looked at final outcomes would never see it.
func (*Channels) Gate ¶
Gate wraps a Chatter so every completion takes a slot in model's channel.
Chatter and VectorClient are the only two seams raglit calls a model through — OCR, segmentation and identity all take a Chatter; embedding takes a VectorClient — so gating these two covers every model call in the process without touching any of the four call sites.
func (*Channels) GateVector ¶
func (c *Channels) GateVector(inner VectorClient, model string) VectorClient
GateVector wraps an embedding client the same way. Embed is synchronous, so the slot spans the call exactly.
func (*Channels) Note429 ¶
Note429 records backpressure for a model: halve the width and refuse to grow for a cooldown.
Multiplicative decrease against additive increase, which is the pairing that converges when several clients share one resource — and raglit IS one of several. Halving on the first 429 rather than after a few is deliberate: the server has already queued or rejected the call by the time it says this, so the cost of over-asking has been paid.
func (*Channels) Observe ¶
func (c *Channels) Observe(model string, next func(llm.RetryEvent)) func(llm.RetryEvent)
Observe returns an llm.Client.OnRetry hook for model that feeds backpressure into the controller and then chains to next (which may be nil).
Chained rather than replacing, because the RetryTally is already on that hook and its record — what a job had to survive — is what the health report reads.
func (*Channels) Stats ¶
func (c *Channels) Stats() []ChannelStat
Stats reports every model this process has called, widest first. Reported because the width is now a LEARNED number rather than a configured one, and a learned number nobody can see is indistinguishable from a bug.
type Chatter ¶
type Chatter interface {
ChatStream(ctx context.Context, messages []llm.Message, tools []llm.ToolDef,
opts *llm.ChatOpts) (<-chan llm.StreamChunk, error)
}
Chatter is the sliver of *llm.Client the OCR and segmentation paths need: one streaming completion. An interface so the model can be stubbed in tests.
STREAMING, not Chat: see the file doc. A non-streaming client cannot be protected from a runaway generation, because by the time it returns, the tokens are already paid for.
type CodeSegmenter ¶ added in v0.7.0
type CodeSegmenter interface {
// ForPath names the language of a path, from its extension. ok is false
// for a file the segmenter cannot parse.
ForPath(path string) (lang string, ok bool)
// Symbols returns the declarations of content, parsed as lang.
Symbols(lang string, content []byte) ([]CodeSpan, error)
// Sections returns one span per heading section of a Markdown document.
Sections(content []byte) ([]CodeSpan, error)
// Runtime names the parser and its version. It is in the pool recipe,
// because two parsers can cut the same file differently.
Runtime() string
}
CodeSegmenter cuts source text into CodeSpans. The root package defines the interface and does not implement it: raglitd sets Models.Code to an adapter over poly-lsp-mcp's segment package, so the parser and its grammars are not in the engine's import closure.
type CodeSpan ¶ added in v0.7.0
type CodeSpan struct {
Sym, Class string
StartLine int
EndLine int
// CommentStartLine is the first line of the doc comment above the
// declaration, 0 when there is none.
CommentStartLine int
// BodyStartLine is the line where a callable's body begins, 0 when it has
// none.
BodyStartLine int
}
CodeSpan is one cut of a source file, as a CodeSegmenter reports it: a declaration of a code file or a heading section of a Markdown file. Lines are 1-based and EndLine is exclusive. Sym is the dotted symbol path ("Server.Start") or, for a section, the heading path. Spans come in source order and are flat: nesting shows only in the ranges, a nested span lying inside its parent's.
The same shape as poly-lsp-mcp's segment.Span, so an adapter is a field copy.
type Confidence ¶
type Confidence int
Confidence is how far a facet of a reading can be relied on, 0-100.
100 means "this is the thing itself, or a person has ruled on it". Everything below is a machine's output, and the number is a POLICY default for that machinery — not a measurement of this particular reading, which nothing can produce. Where a measurement exists it is cited on the constant.
const ( // TrustExact — the bytes are the text. A plain file, a text layer lifted // from a PDF, a mail body decoded from its transfer encoding: no model is // involved and there is nothing to be wrong about beyond the file itself. TrustExact Confidence = 100 // TrustRuled — a person went through it. The ceiling, and the only way a // machine-produced facet reaches it. TrustRuled Confidence = 100 // TrustPandoc — a converter's reading of a structured document. Not exact: // a .docx has a layout that prose does not, and pandoc chooses. TrustPandoc Confidence = 95 // TrustASRWords — a recogniser's words. High: modern ASR on clear audio is // mostly right, and it is wrong in ways a reader can see (a mangled name, a // dropped clause) rather than in ways that read as fluent. TrustASRWords Confidence = 85 // TrustVisionText — a VLM's transcription of a page. Measured on this setup, // not guessed: chandra scored 89.6% on olmOCR-bench long_tiny_text against a // published 92.3, and 14/17 on the fence-dispute ground-truth checks. TrustVisionText Confidence = 90 // TrustSubject — a model's account of what an image shows. The lowest, and // the one that most needs saying out loud, because it reads as fact: a plate // number, a make and model, a colour, none of it written by anybody. TrustSubject Confidence = 80 // TrustDiarization — who spoke. Lower than the words on purpose. The // diarizer's own output says so, labelling turns "blended, attribution // uncertain", and a verified transcript exists precisely because this is the // facet people have to correct. TrustDiarization Confidence = 65 // TrustLayout — reading order over a page a model assembled. TrustLayout Confidence = 85 )
Default confidences, by what produced the facet.
type Config ¶
type Config struct {
BaseURL string `json:"base_url"`
APIKey string `json:"api_key"`
VisionModel string `json:"vision_model"`
EmbedModel string `json:"embed_model"`
// AudioModel names oidio's transcriber, which reads a recording so raglit can
// index the transcript and review the reading. Empty → audio ingest is
// unavailable and an audio job says so; a corpus of documents needs no
// transcriber.
//
// Through corrallm — which is how oidio is actually deployed — the ids are
// "oidio-stt-diarize" (speaker-labelled segments) and "oidio-stt" (plain
// text), served from the SAME endpoint as the vision and embed models.
// Standalone, oidio calls the same two "stt-diarize" and "stt".
AudioModel string `json:"audio_model,omitempty"`
// AudioBaseURL overrides where the transcriber lives. Empty → BaseURL, which
// is right whenever oidio is behind the same broker as everything else.
//
// This started as a REQUIRED field of its own, on the reasoning that oidio is
// a separate process on :8077 and assuming one gateway fronts both would make
// a working setup unreachable. The :8077 in oidio's own example config is its
// standalone-dev address; the deployment puts it under corrallm, whose
// OpenAI proxy already re-exports it alongside the vision and embed models on
// one base URL. So the separate endpoint is the exception, not the rule, and
// the field earns its keep as an override rather than a second thing to set.
AudioBaseURL string `json:"audio_base_url,omitempty"`
// EmbedLimitChars caps a fragment's size to what the embed model accepts as one
// input — probed once and stored (DiscoverEmbedLimit), so the deterministic
// fragmenter's ceiling is bounded by the model, not by taste. 0 = not probed
// (the fragmenter falls back to FragWindow uncapped).
EmbedLimitChars int `json:"embed_limit_chars,omitempty"`
// EmbedLimitTokens is the same limit in the unit the embedder counts in, and
// it is the one that decides. EmbedLimitChars is a conversion of it at two
// characters per token; a scanned court brief is 1.57, so a fragment can sit
// inside the character ceiling and still be refused. 0 = not established.
EmbedLimitTokens int `json:"embed_limit_tokens,omitempty"`
// SegmentInputLimitTokens caps ONE segmentation REQUEST — prompt, carried-over
// fragment and page text — to what the chat endpoint accepts, asked once and
// stored (the server's own n_ctx where it states one, else llm.DiscoverContext).
// The mirror of EmbedLimitChars on the other side of the call: that one bounds
// what comes back, this one bounds what is sent.
//
// TOKENS, because that is the unit every endpoint states its limit in.
// EmbedLimitChars stays in characters because it sizes a fragment, and the
// conversion between the two is not a constant: measured with the model's own
// tokenizer, this corpus runs 4.66 chars/token on prose and 1.16 on a survey
// legal description.
//
// 0 = not established (no cap). Set it to override on a proxy that accepts a
// prompt its backend will later refuse — which cannot be measured from
// outside.
SegmentInputLimitTokens int `json:"segment_input_limit_tokens,omitempty"`
// FragWindow / FragStride / FragFloor tune the deterministic overlapping-window
// text fragmenter (chars). 0 → defaults (1000 / 750 / 300; see fragment.go). window > stride
// gives the overlap; floor folds a short tail. They feed frag_recipe, so a
// change marks the affected documents for reprocessing.
FragWindow int `json:"frag_window,omitempty"`
FragStride int `json:"frag_stride,omitempty"`
FragFloor int `json:"frag_floor,omitempty"`
// TranscriptionModel and DescribeModel are the per-kind ingest models (F11):
// when ingest descends a page into zones, it reads each zone with the model
// for its KIND — text zones go to the transcription model, diagram/figure
// zones to the multimodal describer. A transcription model (chandra-ocr-2)
// reads drawings poorly and a describer is wasted on paragraphs, so one
// VisionModel covering both roles costs whichever side it was chosen for.
//
// Both empty → routing is a no-op: every zone is read by VisionModel exactly
// as today, which is the live default (init does not ask for these). Set
// only one and the other kind keeps reading VisionModel; set both and a page
// can be split across two models. A zone whose kind is neither text nor
// diagram/figure falls back to VisionModel either way.
TranscriptionModel string `json:"transcription_model,omitempty"`
DescribeModel string `json:"describe_model,omitempty"`
// IdentityModel names the chat model that says what a document IS — a
// caption, a summary and a kind, asked once per document on the assembled
// transcript (identity.go). Empty → the segment model, else the vision
// model; set this to caption with a different text model than either.
IdentityModel string `json:"identity_model,omitempty"`
// IdentityProvider is IdentityModel's provider. Empty → the legacy
// base_url/api_key, as before providers. Unused when IdentityModel is empty:
// identity then runs on the segment model and its provider.
IdentityProvider string `json:"identity_provider,omitempty"`
// IdentitySlots is how many captioning requests are in the model at once.
// 0 → DefaultIdentitySlots (2), which is what this endpoint serves
// concurrently. Raising it past what the server actually runs does not make
// anything faster: the extra requests wait INSIDE the server, where raglit
// cannot see them, cannot resume them, and cannot tell them apart from an
// ingest job's OCR call waiting for the same slot.
IdentitySlots int `json:"identity_slots,omitempty"`
// SessionBranchTTLHours is how long a session branch ("idx@session")
// outlives its last write before the daemon deletes it. 0 → 168 (7 days:
// USER 2026-10-30, "7 day to save thrash").
SessionBranchTTLHours int `json:"session_branch_ttl_hours,omitempty"`
// ModelChannelMax caps how wide any one model's admission channel may grow
// (modelchan.go). 0 → the built-in ceiling.
//
// Every other number in that controller is LEARNED — a model starts at one
// slot and widens only while calls succeed, halving whenever the server
// pushes back. This is the exception, because the right ceiling depends on
// what kind of endpoint is behind the name and no amount of evidence
// distinguishes "has not said no yet" from "will take a hundred": a card
// serving one model locally is never going past a handful, while a hosted
// provider bought for throughput will, and there the ceiling would be the
// only thing limiting it.
ModelChannelMax int `json:"model_channel_max,omitempty"`
// NoIdentity turns document identity off. Documents then carry only the
// filename they arrived with — which for a scanner-named corpus is a list
// nobody can navigate, so this is for a corpus whose names are already good
// (a code tree) or an endpoint you do not want the extra call on.
NoIdentity bool `json:"no_identity,omitempty"`
// CodeSegmenter cuts source files at their declarations and Markdown at
// its headings (KindCode), one fragment per declaration or section, each
// with its lines and symbol path. raglitd's segmenter is poly-lsp-mcp's
// tree-sitter segment package (cmd/raglit/codesegment.go). False → code
// and Markdown are read into windows, as before. Turning it on changes the
// KindCode pool key, so a changed file is cut again on its next ingest. An
// unchanged file is not: the unchanged-bytes check skips it before the pool
// key is looked at, so files indexed before the switch keep their windows
// until `raglit ingest --fresh` re-reads them.
// The daemon flag --code-segmenter turns it on as well.
CodeSegmenter bool `json:"code_segmenter,omitempty"`
// DefaultIndex is the index used when a command gives no --index. Empty →
// "default". Set it in the wizard to make one named index your working default.
DefaultIndex string `json:"default_index,omitempty"`
// Project names this project. On the SHARED daemon it namespaces every index
// this client touches (daemon index = "<project>__<local>"), so two projects
// both using index "default" don't collide, and a project's "search all" stays
// within its own indexes. Required to start a daemon-routed client (serve/CLI);
// --embedded/--db (single-session, in-process) don't need it. Set in the wizard.
Project string `json:"project,omitempty"`
// A "search all" spans "<project>__*" plus each "<shared>__*", so common docs
// (e.g. ~/doc indexed once under a project named "shared") are visible from
// every project that opts in — without duplicating them per project. Writes
// (ingest/branch) still target this project only. Empty → fully isolated.
Shared []string `json:"shared,omitempty"`
// Watch, when true, tells the daemon to keep this project's configured source
// roots fresh: after you register it (a `raglit sync` with watch:true, or
// `raglit watch`), the daemon re-scans the roots on an interval and re-ingests
// changed files / drops deleted ones. Needs `indexes` (the roots to watch).
Watch bool `json:"watch,omitempty"`
// WritebackTranscriptionMd materialises <doc>.raglit-transcription.md beside
// every ingested document, project-wide. A per-index setting of the same name
// overrides this for that index.
WritebackTranscriptionMd bool `json:"writeback_transcription_md,omitempty"`
// ExtractEmailAttachments writes a mail archive's attachments into
// <archive>.raglit-attachments/ beside it, project-wide, so the documents
// inside an .eml/.mbox become ordinary files the next sync indexes. A
// per-index setting of the same name overrides this for that index.
ExtractEmailAttachments bool `json:"extract_email_attachments,omitempty"`
// SegmentModel splits transcribed TEXT into fragments. Empty → the vision
// model, which is what this always used to be.
//
// They are different jobs and the best model for one is not the best for the
// other. The segmenter is asked for a structured tool call over text; an OCR
// specialist is chosen for reading pixels and need not do the first well.
// Measured on a 2-page 1947 deed, same cached OCR, four arms:
//
// chandra + markup degraded — no valid JSON
// Qwen + markup done, dropped 1184 of 1234 chars
// chandra + flattened degraded — no valid JSON
// Qwen + flattened done, clean
//
// Corpus-wide the same split: chandra degraded 29 of 116 documents (25%),
// Qwen 4 of 254 (1.6%).
SegmentModel string `json:"segment_model,omitempty"`
// DaemonURL, when set, makes this a CLIENT config: commands route to the
// raglit daemon at this URL (http(s)://host:port) instead of opening a local
// index. The daemon owns storage (scoped per index, under its own home), so
// the local .raglit/ then holds config only. Precedence for the effective
// daemon: --daemon flag > $RAGLIT_DAEMON > this. Empty → local (embedded) mode.
DaemonURL string `json:"daemon_url,omitempty"`
// OCR configures the cheap tier — either as the cascade's first pass or as
// the vision model's spelling reference (OCRConfig.Mode). Zero value →
// VLM-only (every page transcribed by the vision model).
OCR OCRConfig `json:"ocr,omitempty"`
// ImageEmbed optionally configures an image embedder for FIGURES: when its
// Model is set, a figure is embedded from its IMAGE instead of its description.
// nomic-embed-vision-v1.5 shares nomic-embed-text's space, so image figures stay
// searchable by the same text query — requires EmbedModel to be nomic-embed-text.
ImageEmbed ImageEmbedConfig `json:"image_embed,omitempty"`
// Ignore is this config's default exclude globs (project-scoped — it does not
// affect other projects' configs). Merged with a built-in default (dot-dirs,
// node_modules, vendor) and the per-index / per-root ignores; ignore wins.
Ignore []string `json:"ignore,omitempty"`
// Gitignore, when nil or true, also honors each root's .gitignore chain.
Gitignore *bool `json:"gitignore,omitempty"`
// Indexes declares named indexes and the source roots + rules that feed them,
// for `raglit sync`. Multi-index: one project can define several.
Indexes map[string]IndexConfig `json:"indexes,omitempty"`
// Auth turns on OIDC for the daemon and tells a client how to get a token.
// Nil → no auth, every index readable by anyone who can reach the daemon.
Auth *AuthConfig `json:"auth,omitempty"`
// Providers are named model endpoints, and each role picks one
// (USER 2026-10-22: provider, then model — set from the UI). A role with
// no provider uses BaseURL/APIKey above, so a config written before
// providers existed behaves exactly as it did.
Providers map[string]Provider `json:"providers,omitempty"`
VisionProvider string `json:"vision_provider,omitempty"`
EmbedProvider string `json:"embed_provider,omitempty"`
SegmentProvider string `json:"segment_provider,omitempty"`
AudioProvider string `json:"audio_provider,omitempty"`
}
Config is raglit's model-connection setup, written by `raglit init` into <home>/config.json. It's OpenAI-standard: a base URL + token, plus the two model ids raglit needs — a vision model (image in → text, for OCR) and an embedding model (text in → vector, for --embed / vector search). Kept out of the index so the same corpus can be re-pointed at a different endpoint.
func LoadConfig ¶
LoadConfig reads the home's config. exists is false (with nil error) when the home has not been initialized yet — the caller decides whether that's fatal.
func (*Config) Endpoint ¶
Endpoint is the base URL and key of a named provider. "" or an unknown name → the legacy BaseURL/APIKey, with ok=false for an unknown name.
func (Config) PerKindModelsConfigured ¶
PerKindModelsConfigured reports whether either per-kind ingest model is set. While both are empty the F11 routing is a no-op: every zone is read by the one VisionModel, unchanged from today, and callers do not need to branch at all on zone kind.
func (*Config) SegmentProviderName ¶
SegmentProviderName is the segment role's provider. A segment role with no model of its own runs on the vision model, and so on the vision provider.
func (Config) StrategyFor ¶
func (c Config) StrategyFor(index string) StrategyConfig
StrategyFor resolves the read policy for one index: the index's own choice, else the project default, else the zero value (today's behavior).
An index naming a strategy that does not exist gets the zero value rather than an error, and that is deliberate — a typo'd strategy name must not stop an ingest, and the alternative (silently using the default strategy) would hide the typo behind plausible output. Callers that want to complain can check StrategyNamed.
func (Config) StrategyNamed ¶
func (c Config) StrategyNamed(name string) (StrategyConfig, bool)
StrategyNamed looks a strategy up by name. ok=false for an empty or unknown name, in which case the zero StrategyConfig is returned.
type ContextProber ¶
type ContextProber interface {
// ContextWindow is the limit as the SERVER states it (llama.cpp /props),
// ok=false when it does not state one.
ContextWindow(ctx context.Context) (int, bool)
// DiscoverContext measures it by overflowing, which costs O(log N) requests
// and returns a lower bound. The fallback, not the first choice.
DiscoverContext(ctx context.Context) (int, error)
}
ContextProber is the part of *llm.Client this needs. An interface so a test can answer without an endpoint, and so raglit does not take a dependency on the client type to ask two questions.
type Coverage ¶
type Coverage struct {
Parent string `json:"parent"`
Pages int `json:"pages"`
Covered int `json:"covered"`
Uncovered []int `json:"uncovered,omitempty"`
Overlaps [][]int `json:"overlaps,omitempty"` // [page, count] for pages in >1 slice
}
Coverage reports which of a bundle's pages belong to no slice, and where slices overlap.
This is what says a bundle is FULLY linearized rather than merely started. Without it "we sliced that scan" is unfalsifiable — pages 23-27 belonging to nothing looks identical to a finished job.
Overlap is reported, not refused: a declaration and the exhibit bound into it genuinely occupy the same pages, and a tool that forbade that would force the person to lie about one of them.
type DPIDecision ¶
type DPIDecision struct {
DPI int
Reason string // "need" | "cap" | "native" | "base"
CapDPI int // what the token budget allowed
NativeDPI int // what the source holds (0 = unknown)
NeedDPI int // what the glyph measurement asked for
WantDPI int // the resolution asked for, BEFORE the cap
Tiles int // canvases needed to deliver WantDPI
Density float64 // tokens/sq in delivered at DPI
}
DPIDecision is the chosen resolution and WHICH bound chose it. The reason is carried because the three failures look identical in a rendered page and are fixed differently: "need" means the text is fine, "cap" means tile it, and "native" means no rendering setting will ever help and the scan is the limit.
func ChooseDPI ¶
func ChooseDPI(needDPI int, areaSqIn float64, nativeDPI, maxTokens int, p RenderPolicy) DPIDecision
ChooseDPI applies all three bounds.
nativeDPI of 0 means "unknown", and an unknown source resolution must NOT silently become a bound: a born-digital page has no native raster at all, and treating that as zero would refuse to render it.
func (DPIDecision) String ¶
func (d DPIDecision) String() string
String renders a decision for a log line or a trace record.
type DiffShape ¶
type DiffShape struct {
Pages int `json:"pages"`
// Clean is pages matching at 1.0, Noisy is pages disagreeing a little
// (>= NoisyFloor), Divergent is pages disagreeing a lot.
Clean int `json:"clean"`
Noisy int `json:"noisy"`
Divergent int `json:"divergent"`
Missing int `json:"missing"`
}
DiffShape describes the distribution of disagreement across pages.
type DocContent ¶
type DocContent struct {
Path string `json:"path"`
Title string `json:"title"`
Pages []DocPageText `json:"pages"`
Text string `json:"text"`
Truncated bool `json:"truncated"`
Figures []FigureRef `json:"figures,omitempty"`
}
DocContent is a document's indexed text: per-page plus a single joined blob, plus any figures explained into it.
type DocCost ¶
type DocCost struct {
Path string
Hash string
Passages int
Embeddings int
PageReads int
Bytes int64 // passage text + embeddings + saved original and page images
ModelSeconds float64 // import jobs that did model work (not pooled, not unchanged)
}
DocCost is what one document of one layer cost to hold.
type DocDiff ¶
type DocDiff struct {
// SameBytes short-circuits everything below it.
SameBytes bool `json:"same_bytes"`
// Match is the whole-document alignment: relation, coverage, blocks, gaps,
// and the numeric tokens that differ. Reused rather than recomputed so a
// diff and a `similar` report of the same pair can never disagree.
Match DocMatch `json:"match"`
// Pages is one row per page of A, in page order.
Pages []PagePair `json:"pages"`
// OnlyInB lists pages of B that no page of A matched — the other half of
// "what is missing", which a table keyed on A cannot show.
OnlyInB []int `json:"only_in_b,omitempty"`
// Shape summarises where the disagreement lives, which is the actual input
// to the copy-versus-version decision.
Shape DiffShape `json:"shape"`
}
DocDiff is the full comparison of two transcripts.
type DocFields ¶
type DocFields struct {
Type string `json:"type"`
Fields json.RawMessage `json:"fields"`
// Source is 'machine' or 'person'. A person's extraction is never
// regenerated over — the same rule a person's caption follows.
Source string `json:"source,omitempty"`
Model string `json:"model,omitempty"`
At int64 `json:"at,omitempty"`
TextHash string `json:"text_hash,omitempty"`
// TypeHash fingerprints the type DEFINITION these fields were read under
// (DocType.Hash), so an extraction that answers a schema's older questions
// can be told from one that answers its current ones. Empty means unknown,
// which is treated as stale.
TypeHash string `json:"type_hash,omitempty"`
}
DocFields is what a document's type schema says, filled out.
type DocIdentity ¶
type DocIdentity struct {
Name string `json:"name"`
Summary string `json:"summary"`
Kind string `json:"kind"`
// ContentTags are 3–5 short noun phrases describing what the document is
// ABOUT (subject matter, entities, procedures). Open vocabulary but
// constrained in shape; see validateIdentity.
ContentTags []string `json:"content_tags,omitempty"`
// RoleTags are 1–3 terms from identityRoleKinds describing what job the
// document does in the corpus (documentation, reference, overview…).
// Closed vocabulary so they stay groupable and filterable.
RoleTags []string `json:"role_tags,omitempty"`
// DocType is which of the INDEX's registered document types this resolved
// as, empty for none. Its vocabulary is per-index and authored (doctype.go),
// unlike Kind, which is raglit's and closed. A document that resolves as one
// is asked to fill out that type's schema; most documents resolve as none,
// and that is the normal case rather than a failure.
DocType string `json:"doc_type,omitempty"`
// Source is 'machine' (a model read the transcript) or 'person' (someone
// corrected it). A person's identity is never overwritten by a re-run.
Source string `json:"source,omitempty"`
Model string `json:"model,omitempty"`
At int64 `json:"at,omitempty"` // unix nanos
// TextHash fingerprints the text this caption was written from, so a
// transcript that changes underneath it can be detected. See
// documents.gen_text_hash.
TextHash string `json:"text_hash,omitempty"`
}
DocIdentity is what a document is, in one record: the caption, the summary, the kind, and who said so.
func (DocIdentity) ByPerson ¶
func (d DocIdentity) ByPerson() bool
ByPerson reports whether a person, not a model, is the author of this caption.
func (DocIdentity) Empty ¶
func (d DocIdentity) Empty() bool
Empty reports whether nothing has been established about this document.
type DocKind ¶
type DocKind int
DocKind is how a source document should be extracted to text. raglit routes each ingested file to the cheapest extractor that fits its kind — the "cascade all the way down" idea: a PDF page uses its text layer if it has one, OCR if not; office/markup goes through pandoc; images go through OCR; text is read.
const ( KindText DocKind = iota // .txt/.md/… or text/* → read + segment KindPDF // .pdf → per-page hybrid (text layer or OCR) KindImage // .png/.jpg/… → OCR KindOffice // .docx/.odt/.epub/.html/… → pandoc → text // KindEmail is an .eml archive: one PAGE per nested message, headers kept. // Not KindText — read as plain text an email archive is mostly base64, and // read as one page a quotation from the fifth message can only be located // somewhere in the whole file. KindEmail // KindSpreadsheet is .xlsx/.xls: one PAGE per sheet, same reasoning as // KindEmail — a fact on the "Q3 Budget" tab must be locatable to THAT tab, // not buried in a single page mixing every sheet in the workbook. Not // KindOffice: pandoc has no xlsx/xls reader at all (its own // --list-input-formats omits both — csv is the only spreadsheet format it // takes), and even if it did, OfficeText's one-flat-string-per-document // shape is the wrong output for a multi-sheet source. KindSpreadsheet // KindAudio is a recording — audio or a video container carrying speech. // Transcribed by oidio (see stt.go) and indexed as the TRANSCRIPT, with the // machine reading kept beside it for attest to review. raglit does not read // audio itself and is not meant to: it consumes a reading and verifies it. // // Video is here rather than in a kind of its own because what raglit wants // from a hearing recording is the speech, and oidio takes the container // whole. A kind that split them would have two routes to the same producer. KindAudio KindUnknown // KindCode is a text file the embedder's CodeSegmenter (Models.Code) takes: // source code by extension (CodeSegmenter.ForPath), and Markdown. It is cut // at declarations or heading sections, not into windows (codesegment.go). // Never returned by ClassifyDoc: only a worker with a segmenter routes to // it, so without one these files stay KindText or KindUnknown. After // KindUnknown so that no existing constant changes value. KindCode )
func ClassifyDoc ¶
ClassifyDoc routes a source by extension first, then content-type. Extension wins because it is the most reliable signal for a local file; content-type covers extensionless HTTP fetches.
func ClassifyPath ¶
ClassifyPath is ClassifyDoc for a file that exists on disk: the extension decides, and only when it decides nothing does the content get a say.
The order matters and is not the obvious one. Sniffing FIRST would be more "correct" in the abstract and worse in practice — a .docx and a .odt are both zips, a .md and a .csv are both text, and the extension separates them where the bytes do not. The bytes are the fallback, not the authority.
func SniffBytes ¶
SniffBytes is SniffKind for content already in hand — the ingest path holds the bytes and never the file, so a path-only sniff would not reach it.
func SniffKind ¶
SniffKind reads a file's leading bytes and says what it is, ignoring its name.
ClassifyDoc routes on the extension because for a local file the extension is the most reliable signal available — but it is only a signal, and a file that has NO extension gets KindUnknown, which the worker reads as plain text.
For a PDF that is not a near-miss, it is garbage: the index fills with `%PDF-1.7`, `obj` and `endobj` while the document's actual text stays unsearchable, and nothing reports an error because reading bytes as text always "works". Two attachments in ardley-v-brannock arrived exactly this way, their names truncated at 102 characters by the tool that unpacked them, losing the `.pdf` — the same class of defect the email extractor's own fix names ("an unnamed attachment kept no extension, so nothing indexed it").
Returns KindUnknown when nothing matches, so the caller keeps its existing behaviour and this can only ever add routing, never redirect a file that the extension already classified.
func (DocKind) String ¶
String names a kind stably, for the pool key.
These strings are part of a CACHE KEY, so they are written out by hand rather than derived from the constant's position: renumbering the iota, or inserting a kind in the middle, must not silently re-key every cached document. Adding a new name here is free; changing an existing one invalidates that kind's cache, which is the correct behaviour if the routing itself changed and a mistake otherwise.
type DocMatch ¶
type DocMatch struct {
Path string `json:"path"`
Title string `json:"title,omitempty"`
Relation Relation `json:"relation"`
// Jaccard is |A∩B|/|A∪B| over distinct shingles: "are these the same
// document". Exact, not estimated.
Jaccard float64 `json:"jaccard"`
// ContainProbe is |A∩B|/|A| — how much of the PROBE occurs in this document.
// ContainMatch is the reverse. Asymmetric on purpose; a single number here
// cannot express "wholly inside".
ContainProbe float64 `json:"contain_probe"`
ContainMatch float64 `json:"contain_match"`
// BlockCoverProbe / BlockCoverMatch are the fraction of each side covered by
// runs of at least MinRunChars. These are the numbers Relation is decided on,
// because they are the ones shared boilerplate cannot inflate.
BlockCoverProbe float64 `json:"block_cover_probe"`
BlockCoverMatch float64 `json:"block_cover_match"`
ProbeShingles int `json:"probe_shingles"`
MatchShingles int `json:"match_shingles"`
// GenericChars is how much of the PROBE was excluded as corpus-generic (an
// email footer, a caption repeated across unrelated filings). Reported rather
// than applied silently: masking is the one step that can hide a real
// duplicate, so when a document is mostly generic the person reading the
// result has to be able to see that is why the score is low.
GenericChars int `json:"generic_chars,omitempty"`
// MatchedChars is the total folded characters proved identical, counting
// overlaps once. Coverage alone cannot be acted on for a short document: 0.83
// of a 400-character email is a signature block, and 0.83 of a deed is a deed.
MatchedChars int `json:"matched_chars"`
Blocks []Block `json:"blocks,omitempty"`
// NumericOnlyInProbe / NumericOnlyInMatch are numeric tokens (auditor file
// numbers, distances, bearings, dollar amounts, dates) that appear inside the
// aligned region on ONE side only.
//
// This is the headline finding for a legal corpus and it is cheap. Two copies
// of one deed that align at 0.97 and disagree about "25.00" versus "30.00" are
// not a housekeeping problem; in this matter there is an open question about
// which of two 25-foot strips a plat depicts, and one exhibit's OCR was found
// fabricated outright. A digits-level diff over the aligned span puts that in
// front of a person instead of averaging it into a score.
NumericOnlyInProbe []string `json:"numeric_only_in_probe,omitempty"`
NumericOnlyInMatch []string `json:"numeric_only_in_match,omitempty"`
}
DocMatch is one document that shares text with the probe.
func Compare ¶
func Compare(probe, match FoldedText, w int, probeMask, matchMask Mask) DocMatch
Compare is the exact pairwise comparison: no sampling, no estimate.
Candidate generation samples (see shingle.go), because probing an index must not read every document. Scoring does not, because these numbers end up in a human decision about which copy of an instrument a filing cites, and an estimate that is usually right is the wrong tool for that. At this corpus size exactness is nearly free: the whole 233-document index folds to 1.59 MB, so a pair costs two shingle passes over a few tens of kilobytes. The full all-pairs audit of the corpus — 233 probes, each rescoring its candidates exactly — runs in 27 seconds.
One asymmetry to know about, because it looks like a bug and is not: MatchedChars, BlockCoverProbe and the reported blocks are all measured on the PROBE side, so Compare(a,b) and Compare(b,a) do not report the same matched length. A passage of the smaller document that recurs at nine places in the larger one is 383 characters measured on the small side and 3,612 on the large one; both are true statements about different questions. Jaccard is symmetric, the containments are deliberately not, and the blocks answer "where in the PROBE", which is what a triage caller asked. probeMask and matchMask may be nil, which is how the pure two-document tests exercise the scoring without a corpus to derive frequencies from.
type DocPageText ¶
DocPageText is one page's reassembled text.
type DocSummary ¶
type DocSummary struct {
Path string `json:"path"`
Title string `json:"title"`
Fragments int `json:"fragments"`
Pages int `json:"pages"` // OCR-tracked pages (page ≥ 1)
Vision int `json:"vision"` // pages that used the VLM
FragMode string `json:"frag_mode"` // how it was fragmented: text-overlap | llm-seg | code | sections
Engines map[string]int `json:"engines"` // engine → page count
// AddedAt is when the document was last really ingested (Unix nanoseconds).
// A re-ingest of an unchanged file is skipped before commit (worker.go), so
// it does not move AddedAt.
AddedAt int64 `json:"added_at"`
// SrcSize / SrcMtime are the source file's bytes and modified time (unix
// seconds); 0 when unknown (srcstat.go).
SrcSize int64 `json:"src_size,omitempty"`
SrcMtime int64 `json:"src_mtime,omitempty"`
// ContentHash is documents.content_hash: the sha256 hex of the raw source
// bytes, not of the extracted text, and the same value as
// Store.DocumentHash; "" for a document never hashed.
ContentHash string `json:"content_hash,omitempty"`
// GenName / GenKind / GenSource are what the document IS (identity.go), for a
// list a person can navigate. Both names travel: the caption is what a reader
// needs, and the filename is what everything else joins on — and where they
// disagree, THAT is the finding.
GenName string `json:"gen_name,omitempty"`
GenKind string `json:"gen_kind,omitempty"`
GenSource string `json:"gen_source,omitempty"` // machine | person
// GenSummary is the generated account of what the document covers, and
// GenModel the model that wrote the caption (empty for a person's). Carried
// so a list can show what a document is without a request per row.
GenSummary string `json:"gen_summary,omitempty"`
GenModel string `json:"gen_model,omitempty"`
// GenContentTags / GenRoleTags are the document's tags (identity.go):
// what it is about, and what job it does in the corpus.
GenContentTags []string `json:"gen_content_tags,omitempty"`
GenRoleTags []string `json:"gen_role_tags,omitempty"`
}
DocSummary is a document with counts for the review UI's document list.
type DocType ¶
type DocType struct {
Name string `json:"name"`
// Description is one line saying how to RECOGNISE one — it is what the
// identity call is shown when choosing among the registered types, so it
// describes the document, not the fields.
Description string `json:"description,omitempty"`
// Prompt is the extraction instruction: how to read these fields out of this
// kind of document, including what its conventions mean. Travels with the
// schema because a schema alone produces a confidently filled-in form.
Prompt string `json:"prompt,omitempty"`
// Schema is a JSON Schema object describing the fields — the `parameters` of
// the extraction tool call.
Schema json.RawMessage `json:"schema,omitempty"`
// Gold is the documents the schema was proposed from, kept so a revision can
// be judged against the same examples rather than whatever is at hand.
Gold []string `json:"gold,omitempty"`
Model string `json:"model,omitempty"`
CreatedAt int64 `json:"created_at,omitempty"`
UpdatedAt int64 `json:"updated_at,omitempty"`
}
DocType is one registered document type of an index.
func (DocType) FieldNames ¶
FieldNames lists the schema's top-level properties, in schema order where the JSON preserves it and alphabetically otherwise. For a summary line.
func (DocType) Hash ¶
Hash fingerprints the parts of a type definition that shape an EXTRACTION: the schema, the reading instructions, and the description (which the extraction prompt carries as "this document is a X — <description>").
The name is deliberately NOT in it. A rename is a rename; it does not change what was asked of the document, and re-extracting a corpus because somebody fixed a capital letter is a bill for nothing. The gold paths are out for the same reason — they say where the schema came from, not what it asks.
The schema is hashed through a decode/re-encode round trip, so a reformatted-but-identical schema hashes the same: Go marshals map keys in sorted order, which makes that canonical. A schema that will not decode is hashed as its raw bytes, which is the conservative answer — it will simply read as changed more often than it is.
type EmailAttachmentRef ¶
type EmailAttachmentRef struct {
Name string `json:"name"`
Mime string `json:"mime,omitempty"`
Size int64 `json:"size,omitempty"`
Sum string `json:"sum,omitempty"`
// Path is the extracted file beside the archive, when extraction ran and the
// bytes are on disk. Empty means the attachment was seen and not extracted —
// which is a different statement from "there is no attachment", and the UI
// must not render them the same way.
Path string `json:"path,omitempty"`
}
EmailAttachmentRef is an attachment as a reader can act on it.
type EmailHeader ¶
EmailHeader is one header line.
type EmailMessage ¶
type EmailMessage struct {
// Page is the page number this message occupies in the indexed document, so
// a citation from search lands on the same message a reader is looking at.
Page int `json:"page"`
// Depth is enclosure: 0 is a top-level message in the archive, 1 is a message
// forwarded inside one, and so on. This is the thread structure.
Depth int `json:"depth"`
From string `json:"from,omitempty"`
To string `json:"to,omitempty"`
Cc string `json:"cc,omitempty"`
Date string `json:"date,omitempty"`
Subject string `json:"subject,omitempty"`
// Headers is EVERY header, name and value, sorted. The page text keeps four
// and counts the rest; this is the rest.
Headers []EmailHeader `json:"headers,omitempty"`
Body string `json:"body,omitempty"`
// Attachments are what this message carried, whether or not extraction ran.
// "A 4 MB PDF called survey.pdf was sent on this date" is itself evidence.
Attachments []EmailAttachmentRef `json:"attachments,omitempty"`
// Notes is structure the reader could not parse, carried rather than dropped.
Notes []string `json:"notes,omitempty"`
}
EmailMessage is one message in an archive, structured for display.
func EmailMessages ¶
func EmailMessages(path string, resolve func(sum string) string) ([]EmailMessage, error)
EmailMessages reads an archive into structured messages. Pure: it reads the file and writes nothing, for the reason EmailText gives.
resolve, when non-nil, maps an attachment's sha256 to the path it was extracted to. Passing nil returns attachments with no Path, which is honest for an archive whose attachments were never extracted.
type Embedder ¶
type Embedder struct {
Client VectorClient
Model string
DocPrefix string
QueryPrefix string
// BatchLimitChars caps ONE embed request's total input. 0 → a conservative
// default. The server's limit applies to the whole request, so this bounds
// the batch, not the individual fragment.
BatchLimitChars int
}
Embedder turns text into normalized vectors. nomic-embed-text is ASYMMETRIC: documents and queries must carry different task prefixes or retrieval quality drops, so DocPrefix / QueryPrefix default to the nomic convention. Override them (to "") for a model that doesn't use prefixes.
func NewEmbedder ¶
func NewEmbedder(c VectorClient, model string) *Embedder
NewEmbedder builds an Embedder with the nomic prefixes.
func (*Embedder) DiscoverEmbedLimit ¶
DiscoverEmbedLimit probes the largest single input (in characters) the embed endpoint accepts without error — the same shape as llm.DiscoverContext. It grows exponentially to find a length that FAILS, then binary-searches the boundary. Only works on endpoints that REJECT an over-long input; a tolerant endpoint that truncates returns maxEmbedProbeChars (treated as "effectively unbounded"). Store the result in Config.EmbedLimitChars to cap the fragment ceiling by the model, not by taste.
Only a rejection that says the input is too long moves the boundary. Any other failure stops the probe with that error. Counting every error as "too long" made a busy endpoint teach a limit far below the real one: corrallm answers 429 when a model's slots are full, the client waits out its retry budget, and the probe then read the give-up as a refusal of that length.
func (*Embedder) EmbedDocs ¶
EmbedDocs embeds document fragments (DocPrefix), normalized.
Splits into as many requests as the endpoint's batch budget needs. Chunking lives HERE rather than at the call sites so every caller is covered by construction — there are three, and the one that overflowed was the only one anybody would have thought to fix.
type EnqueueItem ¶ added in v0.8.0
EnqueueItem is one entry of an EnqueueMany batch.
type EnqueueResult ¶ added in v0.8.0
EnqueueResult is the outcome for one EnqueueItem: the job id (new, or the existing job that already covers it), or the per-item refusal.
type ErrIdentityTooShort ¶
type ErrIdentityTooShort struct{ Chars int }
ErrIdentityTooShort reports a document with too little text to identify.
func (*ErrIdentityTooShort) Error ¶
func (e *ErrIdentityTooShort) Error() string
type ErrSegmentShort ¶
ErrSegmentShort reports that the model-segmenter did not account for a page.
func (*ErrSegmentShort) Detail ¶
func (e *ErrSegmentShort) Detail() string
Detail is the short form for a stage log.
func (*ErrSegmentShort) Error ¶
func (e *ErrSegmentShort) Error() string
type Fetched ¶
type Fetched struct {
Data []byte
Title string // basename, for the document title
IsPDF bool // route to OCR vs plain-text fragmenting
ContentType string // HTTP Content-Type, for format routing (empty for local files)
}
Fetched is the raw content behind an ingest URL, plus enough to route it.
type FieldsCoverage ¶
type FieldsCoverage struct {
Type string `json:"type"`
Resolved int `json:"resolved"`
Extracted int `json:"extracted"`
// Stale is extractions that answer an older schema or an older transcript.
// Counted separately from Extracted because "88 of 88 extracted" over a
// schema edited yesterday is a coverage report that lies.
Stale int `json:"stale,omitempty"`
}
FieldsCoverage counts, per registered type, how many documents resolved as it and how many have been extracted. What `raglit fields --list` reports.
type FieldsStale ¶
type FieldsStale string
FieldsStale says why an extraction is not current, or that it is.
const ( FieldsCurrent FieldsStale = "" // answers the type's current questions, from the document's current text FieldsNone FieldsStale = "none" // never extracted FieldsSchemaMoved FieldsStale = "schema" // the type definition changed under it FieldsTextMoved FieldsStale = "text" // the document's transcript changed under it FieldsTypeGone FieldsStale = "type removed" // the type it was read under is no longer registered FieldsTypeChanged FieldsStale = "type differs" // the document now resolves as a DIFFERENT type )
func (FieldsStale) Stale ¶
func (f FieldsStale) Stale() bool
Stale reports whether this is anything other than current.
type FigureHit ¶
type FigureHit struct {
MediaID int64 `json:"media_id"`
Path string `json:"path"`
Title string `json:"title"`
Page int `json:"page"`
Description string `json:"description"`
ImagePath string `json:"image_path"`
FragmentID int64 `json:"fragment_id"`
Score float64 `json:"score"`
}
FigureHit is one figure matched by SearchFigures: the figure plus where it lives, ranked by similarity to the query.
type FigureRef ¶
type FigureRef struct {
Page int `json:"page"`
Kind string `json:"kind"`
Description string `json:"description"`
ImagePath string `json:"image_path,omitempty"`
Bbox string `json:"bbox,omitempty"`
}
FigureRef is one figure attached to a document (or a search hit): its description and image, located by page.
type Finder ¶
type Finder struct {
Store *Store
// Limit caps fragments pulled from the index per query before collapsing to
// documents. Default 20.
Limit int
}
Finder adapts a raglit Store to agent.DocFinder, so a local index drops straight into agentkit's proactive-retrieval seam (agent.FinderPreparer) and its MCP tool bridge — the SAME interface a remote ragtag client satisfies. That is the scale path: start with a local .sqlite Finder, swap to a remote impl later without touching the agent wiring.
A DocFinder hit is per-DOCUMENT (the notify layer dedups by DocID and injects a pointer), but Search returns per-FRAGMENT rows. Finder collapses fragments to the best-scoring one per document, so a doc pings once with its strongest passage as the pointer line.
type FoldRecord ¶
type FoldRecord struct {
Hash string `json:"hash"`
Layer string `json:"layer"`
Source string `json:"source"` // the layer document's path
BasePath string `json:"base_path"` // the base document's path (an upload URL)
Title string `json:"title,omitempty"`
By string `json:"by"`
FoldedAt string `json:"folded_at"`
Settled bool `json:"settled"` // provenance note and source notes copied
SettledAt string `json:"settled_at,omitempty"`
}
FoldRecord is one content folded into the base, kept in <base home>/folds.json.
type FoldReport ¶
type FoldReport struct {
ProjectID string `json:"project_id"`
Base string `json:"base"`
DryRun bool `json:"dry_run"`
LayersScanned []string `json:"layers_scanned"` // public layers read
Folded []FoldRecord `json:"folded"`
AlreadyInBase int `json:"already_in_base"` // layer documents whose content the base holds
Settled int `json:"settled"` // earlier folds whose notes were copied this run
Pending int `json:"pending"` // earlier folds still importing
Skipped []string `json:"skipped,omitempty"`
}
FoldReport is what one run did (or, dry, would do).
type FoldedPage ¶
FoldedPage locates one page inside a FoldedText as a half-open [Start,End) span of folded offsets. A page whose text folds to nothing (a blank scan, or a page that is pure figure) is still listed, with Start == End: page numbering has to stay complete or every alignment after it reports the wrong page.
type FoldedText ¶
type FoldedText struct {
// Raw is the text the fold was taken of — pages joined by PageJoin. Offs
// indexes into this.
Raw string
// Body is the fold: lowercase letters and digits only, no separators.
Body string
// Offs[i] is the byte offset in Raw of the i'th byte of Body.
Offs []int
// Pages are page boundaries in FOLDED (Body) offsets, in page order.
Pages []FoldedPage
}
FoldedText is text reduced to its comparable form, with the index needed to map a finding back to something a person can read.
func FoldPages ¶
func FoldPages(pages []PageText) FoldedText
FoldPages reduces paged text to its comparable form.
Kept byte-for-byte compatible with kgraph's foldTextIdx: lowercase, keep unicode letters and digits, drop everything else, and record the source offset of every byte written. Digits are kept STRICTLY — no date or number canonicalisation of any kind. In this corpus the highest-value tokens on a page are exactly the ones a normaliser would eat: an auditor file number (AF#201503110045), a bearing (N 89°14'32" W), a distance (25.00 feet). Two copies of one deed that disagree about a distance are a serious finding, and any fold that rounds, reformats or strips digits reports them as agreeing.
The separators between those digits DO fold away — "AF#201503110045", "AF 201503110045" and "AF# 201503110045" all become "af201503110045" — which is the same reason kgraph's normalize() drops punctuation from an identifier.
func (FoldedText) PageAt ¶
func (f FoldedText) PageAt(off int) int
PageAt reports which page a folded offset lies on, and 0 when it lies past the end. Binary search would be faster; page counts here are in the tens.
type FragConfig ¶
type FragConfig struct {
Window, Stride, Floor int
EmbedLimit int
FigurePrompt int
// SegmentInputLimit caps ONE segmentation request — prompt, carried-over
// fragment and page text together — at what the chat endpoint accepts, IN
// TOKENS, which is the unit the endpoint counts in. Distinct from EmbedLimit,
// which bounds the fragments that come BACK, in characters, because that
// limit belongs to the embedder. Measuring only the second is how a page too
// big to send reached the endpoint and failed the document. 0 → unknown, no
// cap.
SegmentInputLimit int
// TokenCounter counts tokens as the serving model does, when the endpoint can
// (llm.Client.CountTokens). nil → sizes are estimated from a ratio learned
// per document instead. The difference is not cosmetic: measured with the
// model's own tokenizer, this corpus runs 4.66 characters per token on prose
// and 1.16 on a survey legal description.
TokenCounter TokenCounter
// EmbedLimitTokens and EmbedTokenCounter are the same pair for the EMBEDDER,
// and they are a separate model with a separate tokenizer — nomic-embed-text
// answers in BERT ids where the chat model answers in its own.
//
// This is the limit that actually refused documents. EmbedLimit above is a
// character number converted from the embedder's 8192-token context at an
// assumed two characters per token, i.e. 16128 — and a scanned court brief
// reaches 10240 tokens inside that, so the fragment was allowed, sent, and
// refused with a 500 that failed the whole document at the embed stage.
// Characters cannot express this limit; only tokens can.
EmbedLimitTokens int
EmbedTokenCounter TokenCounter
}
FragConfig tunes the deterministic text fragmenter and identifies the per- document fragmentation recipe. Window/Stride/Floor are 0 → defaults; EmbedLimit caps the window by the embed model's input limit (0 → uncapped); FigurePrompt is the figure-description prompt version, folded into frag_recipe so a prompt change marks affected documents stale.
type Fragment ¶
type Fragment struct {
Page int // 1-based page number (0 for pageless sources like plain text)
Ord int // fragment order within the page
Text string // the searchable text
// StartLine and EndLine are the half-open [StartLine, EndLine) 1-based line
// span of Text in the source, 0/0 for unknown. Stored as given.
StartLine, EndLine int
// Symbol is the dotted symbol path the fragment is ("Server.Start"), empty
// for none. It is part of the fragment's content id.
Symbol string
}
Fragment is one indexable unit at the document:page:fragment grain.
type GCPolicy ¶
type GCPolicy struct {
MaxBytes int64 // trim total payload bytes below this (LRU)
MaxEntries int // trim entry count below this (LRU)
MaxAgeUnused time.Duration // also evict anything unused at least this long
}
GCPolicy bounds the pool. All limits are optional (zero = disabled). The default posture is deliberately LAX — a byte budget only — so merges and retries keep reusing pooled work instead of paying to re-index it; the pool is trimmed to MaxBytes by evicting the OLDEST-accessed (LRU) entries, which are the ones least likely to be needed again.
type Gap ¶
type Gap struct {
ProbePage int `json:"probe_page"`
MatchPage int `json:"match_page"`
ProbeText string `json:"probe_text"`
MatchText string `json:"match_text"`
}
Gap is one disagreement inside an aligned block: the unmatched text between two runs, from both sides, as a person would read it.
type GibberishConfig ¶
type GibberishConfig struct {
// MinMeanConfidence: pages whose mean recognizer confidence is
// below this fall back. PP-OCR reports >0.9 mean on clean print;
// 0.55 only catches genuinely struggling pages.
MinMeanConfidence float64 `json:"min_mean_confidence"`
// MinWordlikeFraction: fraction of whitespace tokens that must
// look like words (letters with vowels, or numbers). Below this,
// the text is mostly symbol-soup → fall back. Only applied once
// the page has at least MinTokensForLexical tokens (short pages
// have too few tokens for a stable ratio).
MinWordlikeFraction float64 `json:"min_wordlike_fraction"`
// MinTokensForLexical gates the lexical test: pages with fewer
// tokens than this are judged on confidence alone.
MinTokensForLexical int `json:"min_tokens_for_lexical"`
// MaxJunkRuneFraction: fraction of replacement (U+FFFD) /
// control runes above which the page is junk regardless of
// confidence. Catches mojibake the recognizer was "confident"
// about.
MaxJunkRuneFraction float64 `json:"max_junk_rune_fraction"`
}
GibberishConfig tunes the detector that decides whether a cheap PaddleOCR page is trustworthy or needs a vision-model re-OCR. Defaults are PRECISION-BIASED: thresholds sit well below clean- print norms so only clearly-bad pages (handwriting, degraded scans, garbled figures) trip the fallback. The goal is to keep the expensive vision calls rare; the cost of a false-negative is a slightly noisy page, the cost of a false-positive is a wasted Qwen call.
Zero value → all defaults via withDefaults(). Overrides arrive through the OCR config (config.OCR.Gibberish) and are carried on OCR.Gate; see the cascade in ocr.go.
func (GibberishConfig) IsGibberish ¶
func (g GibberishConfig) IsGibberish(po PageOCR) (bool, string)
IsGibberish reports whether a PaddleOCR page should be re-OCR'd by the vision model, with a short human-readable reason for tracing.
A zero-box page is NOT gibberish — it's an empty/blank page (or a pure figure with no detectable text), which the cascade emits as empty rather than paying the vision model for every blank. A figure whose labels paddle garbles will instead trip on low confidence or the lexical test, which is the case the user cares about ("graphs need Qwen").
type Hit ¶
type Hit struct {
ID int64 // fragment id (stable key for fusing rankings)
Path string
Title string
Page int
Ord int
Text string
Score float64
// StartLine and EndLine are the fragment's half-open 1-based line span in
// its source, 0/0 when unknown (llm-seg, a paged document, an older index).
StartLine, EndLine int
// Symbol is the fragment's dotted symbol path, empty for none.
Symbol string
// ContentID is the fragment's stable id (contentid.go): the same across a
// re-ingest of unchanged text, unlike ID. See Store.FragmentExists.
ContentID string
// Origin is empty for the document's own words, and names what a machine
// made otherwise: "identity" for a generated caption/summary (identity.go),
// "described" for a model's account of an IMAGE (indextext.go) — the 700
// words chandra writes about a photograph, naming a car's make and its
// licence plate, none of which anybody wrote down.
//
// These rank in the same list on purpose: a summary is how a document whose
// body never says "purchase and sale agreement" becomes findable by that
// query, and a description is the only way a photograph is findable at all.
// But they are a machine's words, so every renderer says so and nothing
// quotes from them.
Origin string
// Trust is how far the text of this hit can be relied on, and about WHAT.
// Nil when nothing is recorded about how the document was read, which is not
// a claim of reliability — see HitTrust.
Trust *HitTrust
// Similarity is the cosine of the query and fragment vectors, set on a hit
// a vector search ranked. Nil when the vector side did not rank it (bm25,
// or a hybrid hit outside the vector pool). Score in hybrid is a fused
// rank, so a fixed cut-off reads this instead.
Similarity *float64
// Matched is the distinct query terms the fragment contains, in query
// order, and QueryTerms how many terms the query had after stopwords. A
// long query matched on one word is a weak hit whatever its bm25.
Matched []string
QueryTerms int
// contains filtered or unexported fields
}
Hit is one BM25-ranked fragment. Score is normalized so HIGHER is better (the opposite of SQLite's raw bm25(), which returns more-negative for better matches) — matching agentkit's DocHit.Score convention.
func CollapseToAuthoritative ¶
func CollapseToAuthoritative(hits []Hit, readingOf func(doc string) (Reading, bool), readingsOf func(source string) []Reading, governs func(source string) (string, bool)) []Hit
CollapseToAuthoritative drops hits that are readings of a source some OTHER hit reads better, keeping rank order.
Search ranks fragments, and two readings of one recording are two sets of fragments saying nearly the same thing — so a 44-minute hearing came back twice, once as a verified transcript with people's names and once as machine output with speaker-1 and speaker-6. Both are real; only one should be the answer, and it must not be decided by which happened to score higher.
Documents with no reading recorded are untouched. That is most of a corpus, and a hit is never removed for want of a row nothing wrote.
func (Hit) Caveat ¶
Caveat is the caution for THIS hit: the fragment first, then the document.
Two levels, because a reader reads a FRAGMENT and the document-level number can be honestly small while the excerpt in front of them is entirely a model's. Measured on this corpus: a 2002 record of survey scores 3% described — correctly, it is a transcribed sheet with one figure on it — and the top excerpt a survey query returns from it is `[FIGURE: Section map showing grid lines and section number 6]`, which is nobody's words. A document-level caveat alone stays silent there.
func (Hit) IsDescription ¶
IsDescription reports whether this hit is a DESCRIPTION of a document — its caption and summary — rather than text from the document itself. True for a machine's and for a person's alike: neither is the instrument, and neither can be quoted as if it were.
type HitTrust ¶
type HitTrust struct {
// Method is what produced the text: text-layer, vision-ocr, oidio-asr…
Method string `json:"method,omitempty"`
// Level is machine, adapted or attested — whether a person has been through it.
Level string `json:"level,omitempty"`
// DescribedPct is how much of the DOCUMENT is a model's description rather
// than a transcription, or DescribedUnmeasured when it cannot be known.
DescribedPct int `json:"described_pct,omitempty"`
// Facets is the per-facet confidence: text, speaker, layout, subject. A facet
// that is absent is NOT claimed rather than distrusted.
Facets map[string]Confidence `json:"facets,omitempty"`
// Summary renders Facets weakest-first for a reader ("subject 80%", "text 90%").
Summary []string `json:"summary,omitempty"`
// RuledBy is the person who ruled, when one has.
RuledBy string `json:"ruled_by,omitempty"`
}
HitTrust is the caveat a search result carries: what produced this text, and what a person has ruled on.
A ranked list presents every row in the same shape, and the shape reads as "the document says this". For most rows that is right. For others the sentence was written by a model looking at a picture, and the two are indistinguishable once they are excerpts in a panel.
Origin already marks the clear-cut case, but only when EVERY page a fragment touches is ≥90% description — so a survey sheet measured at 88%, whose entire indexed text is a model's account of a map down to which annotation arrow is which colour, comes back with an empty Origin and reads as the record. That is the case this exists for, and it is one bind away from deriving backing in caselit.
Reported, never enforced. Nothing here decides that a hit may not be used — that is a person's ruling, and presenting a confidence as a verdict would be the same mistake as presenting rank as authority.
type Home ¶
type Home string
Home is raglit's on-disk layout: everything for one index under one directory, so the whole thing is a single portable unit you can copy, back up, or delete wholesale.
<home>/ index.sqlite the FTS5 index (documents:pages:fragments) originals/ ingested source files, kept so the index is self-contained pages/ rasterized/extracted page images for OCR (filled by Slice C)
Default location: $RAGLIT_HOME, else ~/local/raglit. Override per-invocation with the CLI --home flag.
func DefaultHome ¶
func DefaultHome() Home
DefaultHome resolves the default index home: $RAGLIT_HOME if set, else ~/local/raglit. Falls back to ./raglit-home if the user home is undiscoverable.
func DiscoverHome ¶
func DiscoverHome() Home
DiscoverHome resolves the home for a command given no explicit --home: the nearest ancestor .raglit/ (walking up from the cwd), else DefaultHome() ($RAGLIT_HOME, else ~/local/raglit). `raglit init` instead writes ./.raglit unconditionally, so a fresh project always gets its own local home.
func ScopedIndexHome ¶
ScopedIndexHome is the daemon's Home for one named index under a scoped root: <root>/indexes/<name>. Exported so a CLI can READ the index the daemon owns rather than the project-local one.
The two are not the same database and routinely disagree. A daemon-routed project ingests through the daemon, so <root>/indexes/<ns>__<name> is the corpus; the project's own .raglit/index.sqlite is whatever was last written locally, which may be years of documents behind. A command that opens the wrong one answers confidently about a corpus nobody is using.
func (Home) AttachmentDir ¶
AttachmentDir is where one archive's attachments live, keyed by the archive's path the same way OriginalPath and PageDir are keyed. Deterministic, so nothing has to record it.
func (Home) AttachmentsDir ¶
AttachmentsDir holds files extracted OUT of ingested documents — today, mail attachments.
In raglit's storage, not beside the archive. Extraction used to write <archive>.raglit-attachments/ into the corpus: 204 files across 44 directories in one legal evidence tree, for a feature raglit owns end to end. An extracted attachment is a real document — it is ingested, captioned and searchable — but where its bytes LIVE is raglit's business, exactly as it already is for uploads/ and pages/.
func (Home) ConfigPath ¶
ConfigPath is the model-connection config (base URL, token, model ids).
func (Home) OriginalPath ¶
OriginalPath is where the original for a given document path is stored — a deterministic function of the doc path (hash-prefixed to avoid basename collisions across directories), so no DB column is needed to find it back.
func (Home) OriginalsDir ¶
OriginalsDir is where raglit used to copy every ingested source file. Nothing writes it now and nothing ever read it back (USER 2026-10-24: a local file is checked on disk, and missing means prunable; an upload already keeps its bytes in uploads/). Prune reports what is left and removes it with --delete-orphan-originals.
func (Home) PageDir ¶
PageDir is the per-document subdirectory of pages/ that holds its page images. Deterministic from the doc path, like OriginalPath.
func (Home) TranscriptDir ¶
TranscriptDir is where one recording's transcript and reading live, keyed by the recording's path the same way OriginalPath and PageDir are keyed. Deterministic, so nothing has to record it.
func (Home) TranscriptsDir ¶
TranscriptsDir holds transcripts of ingested recordings.
In raglit's storage, beside originals/ and pages/, for the reason the attachments migration states: where a generated artifact's bytes live is raglit's business, not the corpus's. A transcript is more clearly raglit's own output than an extracted attachment ever was — nobody put it in the evidence tree and nothing outside raglit refers to it by path.
func (Home) UploadsDir ¶
UploadsDir is where uploaded documents live in a home.
type Identifier ¶
type Identifier struct {
Client Chatter
// Model is recorded on the document, so a caption can be told from one a
// different model wrote (and from a person's).
Model string
MaxRetries int // JSON fix-loop attempts after the first try (default 2)
// contains filtered or unexported fields
}
Identifier asks a model what a document is. One call per document, on the assembled transcript rather than per page — cheap next to the OCR that produced the transcript.
func NewIdentifier ¶
func NewIdentifier(c Chatter, model string) *Identifier
NewIdentifier builds an Identifier over a chat client (an *llm.Client).
func (*Identifier) ExtractFields ¶
func (id *Identifier) ExtractFields(ctx context.Context, text string, t DocType, hint string) (DocFields, error)
ExtractFields asks the model to fill out one type's schema.
The type's schema becomes the tool's parameters, so the SAME fix loop that holds identity to its shape holds an extraction to the corpus owner's — a schema authored months ago is enforced by the machinery that was already here, rather than by a second, worse copy of it.
func (*Identifier) Identify ¶
func (id *Identifier) Identify(ctx context.Context, ask IdentityAsk) (DocIdentity, error)
Identify reads a document's assembled text and returns what it is.
The FILENAME IS DELIBERATELY NOT PASSED. This exists because filenames lie, and a model given "Lead-Based Paint (1).pdf" alongside a purchase and sale agreement will hedge toward the name it was handed — which reproduces exactly the failure the caption is supposed to catch. The document says what it is.
Returns an error rather than a fallback. A document with no caption is a document a person can still find by its filename; a document with a WRONG caption is one whose list entry now lies with a machine's confidence. There is no degraded answer worth storing here. tagContext is the index's established tag vocabulary — see Store.TagContext. It is a PARAMETER rather than a field because one *Identifier is shared by every index in a registry and by every slot of the captioning queue: a field would be a data race, and would carry one index's vocabulary into the next.
func (*Identifier) IdentifyTags ¶
func (id *Identifier) IdentifyTags(ctx context.Context, ask IdentityAsk) (content, roles []string, err error)
IdentifyTags asks for TAGS ALONE, leaving a caption that already exists alone with it. The backfill for a corpus captioned before tags existed: the full identity would rewrite hundreds of names that are already right (and a person's, which must never be regenerated), for the sake of two columns.
Returns the content and role tags. Source/Model/At are the CALLER's to merge, because what is being written is part of an identity somebody else already authored.
type IdentityAsk ¶
type IdentityAsk struct {
// Text is the document's own words, as indexed.
Text string
// TagContext is the index's established tag vocabulary (Store.TagContext),
// so new tags align with terms the corpus already uses.
TagContext string
// IndexHint is what the corpus owner says about reading this collection
// (Store.IndexHint).
IndexHint string
// DocTypes are the index's registered document types, which the answer may
// choose one of. Empty → the doc_type field is not asked for at all, which
// is right for an index that has registered none.
DocTypes []DocType
// Path is the document's path, which picks the caption profile and, for a
// profile that shows it, is quoted in the prompt.
Path string
// Profile is the caption rules (profileFor); nil means legal.
Profile *captionProfile
}
IdentityAsk is everything one identity call needs to know that is not the prompt: the document, and the three things about the INDEX that shape the answer.
A struct rather than four parameters, and a parameter rather than fields on the Identifier, because one *Identifier is shared by every index in a registry and by every slot of the captioning queue — state there is a data race and carries one index's context into the next.
type IdentityJob ¶
type IdentityJob struct {
ID int64 `json:"id"`
Path string `json:"path"`
State string `json:"state"` // pending|running|done|skipped|error
Force bool `json:"force,omitempty"`
// Mode is WHICH ask this job is: IdentityAskFull, IdentityAskTags (leave the
// caption alone — the backfill for a corpus captioned before tags existed)
// or IdentityAskFields (fill out the schema of the type it resolved as).
Mode string `json:"mode,omitempty"`
Error string `json:"error,omitempty"`
EnqueuedAt int64 `json:"enqueued_at,omitempty"`
StartedAt int64 `json:"started_at,omitempty"`
FinishedAt int64 `json:"finished_at,omitempty"`
}
IdentityJob is one queued captioning task.
type IdentityQueueStatus ¶
type IdentityQueueStatus struct {
Pending int `json:"pending"`
Running int `json:"running"`
Done int `json:"done"`
// Skipped is documents there was nothing to caption for — a scanned page with
// forty characters on it, an empty attachment. Its own state rather than a
// failure: nothing went wrong, nothing will go better next time, and counting
// it as failed puts a permanent red number on a corpus that is fine. Kept out
// of the re-queue for the same reason.
Skipped int `json:"skipped"`
Failed int `json:"failed"`
}
IdentityQueueStatus is the queue in one line: how much is left, how much is running, how much finished, and how much failed.
func (IdentityQueueStatus) Empty ¶
func (q IdentityQueueStatus) Empty() bool
Empty reports whether there is nothing outstanding.
type IdentityStatus ¶
type IdentityStatus struct {
Path string `json:"path"`
DocIdentity
}
IdentityStatus is one document's identity, for a listing.
type IdentityWorker ¶
type IdentityWorker struct {
Store *Store
// Slots is how many model calls are in flight. 0 → DefaultIdentitySlots.
Slots int
// IdlePoll is how long Run waits when the queue is empty. Default 2s.
IdlePoll time.Duration
// OnDone, when set, is called for each finished job — for a CLI that wants
// to print progress. Called from the committer, one at a time.
OnDone func(job IdentityJob, id DocIdentity, err error)
// OnReclaim, when set, reports rows requeued from a dead process at the top
// of a Drain. Worth saying out loud: those documents are about to be redone,
// so a count that does not match what was asked for has an explanation.
OnReclaim func(n int)
}
IdentityWorker drains the captioning queue.
Shaped as a pipeline rather than as N self-contained workers, because the model is the scarce resource and everything else is sqlite: a loader claims rows and reassembles document text, `Slots` callers do NOTHING but the model call, and a committer writes the captions and closes the rows. A slot is held for the request and not a millisecond longer.
func (*IdentityWorker) Drain ¶
func (w *IdentityWorker) Drain(ctx context.Context) (int, error)
Drain works the queue until it is empty, returning how many jobs finished (successfully or not). This is the whole sweep for a CLI; Run is the daemon's version, which waits for more work instead of returning.
func (*IdentityWorker) Run ¶
func (w *IdentityWorker) Run(ctx context.Context)
Run drains forever, sleeping IdlePoll between empty polls, until ctx is done.
type ImageEmbedConfig ¶
type ImageEmbedConfig struct {
Model string `json:"model,omitempty"` // e.g. nomic-embed-vision-v1.5; empty → disabled
URL string `json:"url,omitempty"` // full endpoint; empty → Nomic Atlas image API
APIKey string `json:"api_key,omitempty"` // empty → reuse the main api_key
}
ImageEmbedConfig configures the optional figure IMAGE embedder (see Config.ImageEmbed). Model empty → disabled (figures embed their description).
type ImageEmbedder ¶
type ImageEmbedder interface {
EmbedImage(ctx context.Context, mime string, data []byte) ([]float32, error)
// Aligned reports whether the image vectors share the text query embedder's
// space, so a text query can cosine against them (true for nomic-vision paired
// with nomic-text). False → image vectors are stored but not text-searchable.
Aligned() bool
}
ImageEmbedder embeds a figure IMAGE into a vector (a CLIP-style tower). raglit ships nomic-vision (imageembed.go); with none configured, figures fall back to embedding their description with the text embedder.
type IndexConfig ¶
type IndexConfig struct {
Roots []Root `json:"roots,omitempty"`
Include []string `json:"include,omitempty"` // a file must match one to be indexed
Ignore []string `json:"ignore,omitempty"` // merged with project + built-in ignore
// WritebackTranscriptionMd NO LONGER DOES ANYTHING, and is kept only so an
// existing config still parses.
//
// It made ingest write <doc>.raglit-transcription.md beside every document it
// read — 407 files into one legal evidence tree, each duplicating text raglit
// already held in `pages`, in the fragments, and on /api/doc-detail. raglit
// then needed IsGeneratedSidecar, a builtinIgnore entry and a refusal inside
// the writer itself to avoid tripping over its own output.
//
// `raglit transcribe` still writes one. The difference is that a person asked.
WritebackTranscriptionMd bool `json:"writeback_transcription_md,omitempty"`
// ExtractEmailAttachments writes a mail archive's attachments into
// <archive>.raglit-attachments/ beside it, with a MANIFEST.md recording which
// message each came from. Off by default for the same reason: an archive can
// carry 69 files and putting them in somebody's corpus uninvited is not an
// indexer's call. Unlike a transcription the extracted files ARE indexable —
// they are originals that travelled inside an envelope, not derived output —
// so the next `sync` picks them up as ordinary files.
ExtractEmailAttachments bool `json:"extract_email_attachments,omitempty"`
// OCRStrategy names an entry in OCRConfig.Strategies. Empty → the project's
// OCR.Strategy, else today's behavior.
//
// Per index because that is the grain at which document KIND is already
// declared here: `records/` is recorded surveys and `correspondence/` is
// letters, and they want different amounts of work per page. Attaching the
// policy to the corpus rather than to the command is the whole point — a
// strategy retyped as a flag is one that gets forgotten.
OCRStrategy string `json:"ocr_strategy,omitempty"`
}
IndexConfig is one index's source definition: the roots to scan and the include/ignore globs that apply to them (overridable per root).
type IndexDigest ¶
type IndexDigest struct {
// Name is the index this describes. Set by a caller reporting across
// several; a store does not know what it is called.
Name string `json:"index,omitempty"`
// Path is the subtree this digest was computed over, empty for the whole
// index. Reported because a digest of the whole index shown for a
// path-scoped search claims coverage the subtree does not have.
Path string `json:"path,omitempty"`
Documents int `json:"documents"`
Untagged int `json:"untagged,omitempty"`
Kinds []TagCount `json:"kinds,omitempty"`
Content []TagCount `json:"content,omitempty"`
Roles []TagCount `json:"roles,omitempty"`
// Types is the index's SCHEMAED documents — which registered document types
// its documents resolved as, and how many of each have been extracted. Part
// of the digest because "this corpus holds 88 work orders whose fields are
// readable" is the single most useful thing that can be said about an index
// that has any, and nothing else says it.
Types []FieldsCoverage `json:"types,omitempty"`
// About is the generated paragraph, when one has been written and still
// matches the corpus it was written from. See IndexAbout.
About string `json:"about,omitempty"`
// AboutStale marks an About written from a materially different corpus —
// present so a caller can show it AND say it is behind, rather than
// choosing between a stale summary and none.
AboutStale bool `json:"about_stale,omitempty"`
}
IndexDigest is what an index contains, counted from what is already stored.
func (IndexDigest) Empty ¶
func (d IndexDigest) Empty() bool
Empty reports whether there is nothing to say about this index.
type IndexMeta ¶ added in v0.5.0
type IndexMeta struct {
Owner string
Visibility Visibility
Bytes int64 // the index home on disk, every file in it
Status Status
}
IndexMeta is what a signed-in caller may know about an index it cannot read: who owns it, whether it is public, how much disk it takes and how its queue stands. No document, passage or path from inside it.
The web UI lists every index with this (Carl 2026-11-04: "metadata only unless admin"), so an index a project key created is visible to the people running the daemon, as an index with an owner, rather than missing.
type Job ¶
type Job struct {
ID int64
URL string
Title string
State JobState
Error string
Fragments int
EnqueuedAt int64
StartedAt int64
FinishedAt int64
// Fresh re-reads the document even when nothing changed: it skips both the
// unchanged-bytes fast path and the cross-index pool. The escape hatch for a
// cached result that is wrong for a reason the cache key cannot see.
Fresh bool
}
Job is one queued ingestion of a URL.
type JobInfo ¶
type JobInfo struct {
ID int64 `json:"id"`
URL string `json:"url"`
Title string `json:"title"`
State string `json:"state"`
Error string `json:"error"`
Fragments int `json:"fragments"`
Mode string `json:"mode"` // 'llm' | 'offline' | ” — segmentation mode
// Lane is which scheduler ran it: 'heavy' (vision/OCR, one GPU slot) or
// 'light' (everything else, several at once). See lane.go.
Lane string `json:"lane"`
EnqueuedAt int64 `json:"enqueued_at"`
StartedAt int64 `json:"started_at"`
FinishedAt int64 `json:"finished_at"`
}
JobInfo is a full ingest-job row for the review UI's job table (all states).
type JobStage ¶
type JobStage struct {
// ID is the row's own identity, and the only stable one a stage has: a
// RETRIED job appends a whole second run under the same job_id with seq
// restarting at 1, so (job_id, seq) names up to four different rows. Job 927
// on the fence-dispute index is four runs.
ID int64 `json:"id"`
Seq int `json:"seq"`
Name string `json:"name"`
Engine string `json:"engine"`
State string `json:"state"`
Detail string `json:"detail"`
At int64 `json:"at"`
}
JobStage is one recorded pipeline step.
type JudgementStore ¶
type JudgementStore struct {
// contains filtered or unexported fields
}
The judgement store: what a PERSON decided about this corpus.
Nothing here can be recomputed. raglit can measure that two documents share 97% of their text; it cannot decide whether that means one instrument scanned twice or one instrument amended. A person decides, and the decision has to be written down or it is done again next week.
raglit-audit.jsonl IS the store. Not a log of it, not a backup of it — the thing itself, read at open and answered from memory.
It was briefly a SQLite database projected from the trail, with a schema, a generated query layer, a rebuild path and a table that duplicated the trail verbatim. On the corpus that motivated all of it: twelve events, eleven rows, a 68 KB database projecting a 12 KB file. Parsing the file is faster than opening the database, the constraints were already enforced in Go before any write, and the indexes covered eleven rows. It bought nothing and cost a concept everybody had to learn.
If the trail ever grows past what is comfortable to parse, cache it. Until a measurement says so, answering at runtime is both simpler and correct by construction — there is no projection to drift, catch up, or rebuild.
func OpenJudgements ¶
func OpenJudgements(auditPath string) (*JudgementStore, error)
OpenJudgements reads a project's trail. A missing trail is a corpus nobody has ruled on, which is the normal starting state.
The projection is gone. This answers from the trail, in memory, and there is no second database to drift, catch up or rebuild — verdicts that DO want a queryable projection now land in the index itself, see attestations.go.
func (*JudgementStore) Authorities ¶
func (s *JudgementStore) Authorities() []ReadingAuthority
Authorities returns every authority ruling.
func (*JudgementStore) Authority ¶
func (s *JudgementStore) Authority(source string) (ReadingAuthority, bool)
Authority returns the ruling for a source, if a person has made one.
func (*JudgementStore) Close ¶
func (s *JudgementStore) Close() error
Close exists so callers need not care that there is nothing to close.
func (*JudgementStore) CoverageOf ¶
func (s *JudgementStore) CoverageOf(parent string, pages int) (Coverage, error)
CoverageOf reports which of a bundle's pages belong to no slice.
func (*JudgementStore) DeleteSlice ¶
func (s *JudgementStore) DeleteSlice(id string) error
DeleteSlice removes a declaration. The trail keeps the creation AND the deletion, so replaying lands in the same place and the history stays readable.
func (*JudgementStore) History ¶
func (s *JudgementStore) History(kind, subject string) ([]AuditEvent, error)
History returns every event recorded on one subject, oldest first.
func (*JudgementStore) OnTranscriptionChange ¶
func (s *JudgementStore) OnTranscriptionChange(fn func(TranscriptionChange))
OnTranscriptionChange registers a listener for a page's active reading changing. Listeners run in registration order, synchronously, after the event is durable in the trail.
func (*JudgementStore) PageCorrections ¶
func (s *JudgementStore) PageCorrections(doc string) (map[int]PageCorrection, error)
PageCorrections returns the ACTIVE correction for each page of a document.
func (*JudgementStore) PageReadings ¶
func (s *JudgementStore) PageReadings(doc string, page int) []PageCorrection
PageReadings returns every reading ever recorded for one page, oldest first. The last is the active one; the rest are superseded and kept.
func (*JudgementStore) PutAuthority ¶
func (s *JudgementStore) PutAuthority(a ReadingAuthority) error
PutAuthority records which reading of a source governs.
Keyed on the SOURCE, not on a pair: a third reading arriving later is ordered against the others by this one ruling rather than needing a new one for every combination. Re-ruling replaces — a person changing their mind is one fact, and the trail keeps both.
func (*JudgementStore) PutPageCorrection ¶
func (s *JudgementStore) PutPageCorrection(c PageCorrection) error
PutPageCorrection records what a person read off a page.
It carries what it SUPERSEDES. A correction attests that a better reading exists; it does not erase the old one, which stays in the trail as the reading the index held when the document was cited.
func (*JudgementStore) PutRelation ¶
func (s *JudgementStore) PutRelation(m Mark) error
PutRelation records a ruling on a pair, normalizing the sides so the pair is one fact however it was given.
func (*JudgementStore) PutSlice ¶
func (s *JudgementStore) PutSlice(sl Slice) error
PutSlice records that a page range of a bundle is a document.
func (*JudgementStore) Relation ¶
func (s *JudgementStore) Relation(a, b string) (Mark, bool, error)
Relation returns the ruling on a pair, in either order.
func (*JudgementStore) Relations ¶
func (s *JudgementStore) Relations() ([]Mark, error)
Relations returns every ruling, in a stable order.
func (*JudgementStore) RelationsFor ¶
func (s *JudgementStore) RelationsFor(doc string) ([]Mark, error)
RelationsFor returns every ruling involving one document.
func (*JudgementStore) Restore ¶
func (s *JudgementStore) Restore(path, by string) error
Restore returns a withdrawn document to the corpus. The withdrawal stays in the trail: a decision reversed is still a decision that was made.
func (*JudgementStore) Slice ¶
func (s *JudgementStore) Slice(id string) (Slice, bool, error)
Slice returns one declaration by id.
func (*JudgementStore) SliceParents ¶
func (s *JudgementStore) SliceParents() ([]string, error)
SliceParents lists the bundles that have any slice.
func (*JudgementStore) Slices ¶
func (s *JudgementStore) Slices() ([]Slice, error)
Slices returns every declaration, parent then page order.
func (*JudgementStore) SlicesOf ¶
func (s *JudgementStore) SlicesOf(parent string) ([]Slice, error)
SlicesOf returns one bundle's declarations, in page order.
func (*JudgementStore) Withdraw ¶
func (s *JudgementStore) Withdraw(w Withdrawal) error
Withdraw records that a document is out of the corpus, and why.
Appends first, then applies — the order audit.go explains: a process that dies between leaves the trail ahead of the state, which a re-read fixes, where the reverse would leave a decision in memory that nothing recorded.
func (*JudgementStore) Withdrawals ¶
func (s *JudgementStore) Withdrawals() []Withdrawal
Withdrawals lists every document ruled out, path order.
func (*JudgementStore) Withdrawn ¶
func (s *JudgementStore) Withdrawn(path string) (Withdrawal, bool)
Withdrawn reports the ruling that took a document out of the corpus, if any.
type Judgements ¶
type Judgements interface {
// Relations
PutRelation(m Mark) error
Relation(a, b string) (Mark, bool, error)
Relations() ([]Mark, error)
RelationsFor(doc string) ([]Mark, error)
// Slices
PutSlice(sl Slice) error
DeleteSlice(id string) error
Slice(id string) (Slice, bool, error)
Slices() ([]Slice, error)
SlicesOf(parent string) ([]Slice, error)
SliceParents() ([]string, error)
CoverageOf(parent string, pages int) (Coverage, error)
// Page readings
PutPageCorrection(c PageCorrection) error
PageCorrections(doc string) (map[int]PageCorrection, error)
PageReadings(doc string, page int) []PageCorrection
// Provenance
History(kind, subject string) ([]AuditEvent, error)
// Notification
OnTranscriptionChange(fn func(TranscriptionChange))
Close() error
}
The judgement surface, and why it is an interface.
Today's implementation answers every question by walking maps built from raglit-audit.jsonl at open. That is right for a trail of a few thousand events and wrong at some size nobody has measured yet.
So callers depend on this and never on the implementation. When a measurement says the walk is too slow, a cached implementation — a SQLite projection, an mmap'd index, whatever the measurement argues for — satisfies the same interface and is swapped in at construction. Nothing above changes, and the trail stays the record either way.
The write methods are here deliberately. A cache that could be read through but written around would drift on the first write that bypassed it.
type Key ¶ added in v0.4.0
type Key struct {
ID string `json:"id"`
Hash string `json:"hash,omitempty"`
Kind string `json:"kind"`
Principal string `json:"principal"`
Issuer string `json:"issuer"`
Parent string `json:"parent,omitempty"`
Read []string `json:"read,omitempty"`
Write []string `json:"write,omitempty"`
Roots []string `json:"roots,omitempty"`
Label string `json:"label,omitempty"`
CreatedAt int64 `json:"created_at"`
LastUsed int64 `json:"last_used,omitempty"`
Revoked bool `json:"revoked,omitempty"`
}
Key is one stored credential. Hash is never returned to a caller.
func (Key) AllowsRead ¶ added in v0.4.0
AllowsRead reports whether k's Read patterns admit index name (no patterns: every index its issuer may read).
func (Key) AllowsWrite ¶ added in v0.4.0
AllowsWrite reports whether k's Write patterns admit index name (no patterns: whatever the ACL lets its principal write).
type KeyStore ¶ added in v0.4.0
type KeyStore struct {
// contains filtered or unexported fields
}
KeyStore holds the keys. Safe for concurrent use.
func OpenKeyStore ¶ added in v0.4.0
OpenKeyStore loads (or starts) the store at path.
func (*KeyStore) Authenticate ¶ added in v0.4.0
Authenticate returns the key a token belongs to, refusing one that is revoked (itself or any ancestor) or, for a session key, idle past KeyIdleTTL. Its last use is recorded.
func (*KeyStore) ByPrincipal ¶ added in v0.4.0
ByPrincipal returns the live session key acting as p ("key#<id>").
func (*KeyStore) HasProject ¶ added in v0.4.1
HasProject reports whether a project has a live key.
func (*KeyStore) IsAncestor ¶ added in v0.4.0
IsAncestor reports whether principal anc is a session key above key principal p (its parent, its parent's parent, ...).
func (*KeyStore) List ¶ added in v0.4.0
List returns the keys by may see: every key for the admin, else the keys it issued and the keys under them. Hashes are left out.
func (*KeyStore) Mint ¶ added in v0.4.0
Mint stores k (ID, Hash and CreatedAt are filled in) and returns its token, the only time the token exists in the clear.
func (*KeyStore) Revoke ¶ added in v0.4.0
Revoke revokes key id and, through Authenticate's ancestor check, every key under it. by must be the admin, the issuer of the key or of a key above it, or an ancestor key.
func (*KeyStore) RevokeProject ¶ added in v0.4.1
RevokeProject revokes every key of a project, and through the ancestor check every key under them.
type Lane ¶
type Lane string
Scheduling lanes — what a job COSTS, decided before it runs.
The queue used to be one serial round-robin: one goroutine, one job at a time, one index per turn (cmd/raglit/serve.go). So a forty-minute OCR of a twenty-two page scan held every other index's turn, and a markdown file that needs three seconds of embedding waited behind it. That is the complaint this answers, and it is not a throughput problem — the two kinds of work do not even contend for the same resource.
LaneHeavy is vision: a page at a time through a VLM, minutes per document, and bounded by ONE GPU admission slot. Running two at once does not go faster; it queues inside the endpoint where nothing here can see or resume them. That is the same reason the identity queue drains one index at a time, and the reason OCR moved to a model on the second card.
LaneLight is everything else: pandoc, a mail parser, a spreadsheet reader, the deterministic fragmenter, and embedding — which is a batched call measured in seconds. Several at once is fine and is the point.
The split is by RESOURCE, not by speed. A large text file is not fast, and it still belongs in light, because what makes heavy serial is the GPU slot rather than the wall clock.
func LaneFor ¶
LaneFor guesses a lane from a URL alone, which is all an enqueue has.
A guess, and deliberately so: the real decision needs the BYTES (Worker.route sniffs content when the name says nothing), and a queue that fetched every document to decide where to queue it would do the expensive half of the work twice. So the extension decides, and the worker corrects the row once it knows — see Worker.ProcessJob.
Two ways it is wrong, both survivable. A PDF that turns out to carry a text layer is read without a model, so it occupied the heavy lane for seconds instead of minutes: wasteful, not incorrect. An extensionless file that sniffs to a scan runs its OCR in the light lane, which is the case worth naming — one light slot is busy for minutes. It is one job, the row is corrected, and a retry goes to the right lane.
func LaneForKind ¶
LaneForKind maps a routed document to the lane that can afford it.
KindAudio is heavy, and is the longest single job raglit has. A 44-minute hearing transcribes in about twenty minutes; the light lane's three slots exist for work measured in seconds, and three recordings would hold all of them while every text and email ingest queued behind. That is precisely the case LaneFor's comment names as the one worth naming, arrived at from the other direction.
Heavy's single slot also serialises them, which is what the far end wants: oidio is one process holding one recogniser, so a second concurrent transcription does not start earlier — it blocks inside the server, exactly as the vision endpoint does.
type LaneStatus ¶
type LaneStatus struct {
Pending int `json:"pending"`
Running int `json:"running"`
// Slots is how many jobs this lane runs at once — the number that says
// whether a pending count is a queue or a moment.
Slots int `json:"slots"`
}
LaneStatus is one lane's share of the queue.
type LayerSavings ¶
type LayerSavings struct {
Name string `json:"name"`
Documents int `json:"documents"`
Duplicates int `json:"duplicates"` // documents another layer (or the base) also holds
}
LayerSavings is one layer's share of the duplication.
type LayoutBox ¶
type LayoutBox struct {
// Normalised to 0-1000 per axis, as the model emits them.
//
// One tag per field, deliberately. `X0, Y0, X1, Y1 int ` + "`" + `json:"x0"` + "`" + “ applies the
// SAME tag to all four, encoding/json suppresses the conflicting names, and
// every box goes over the wire with NO COORDINATES — the overlay draws
// nothing. The Go tests passed throughout, because they exercised Pct() on
// the struct and never the wire format.
X0 int `json:"x0"`
Y0 int `json:"y0"`
X1 int `json:"x1"`
Y1 int `json:"y1"`
Label string `json:"label"`
Text string `json:"text"`
}
LayoutBox is one block the model reported: where it is, what kind it thought it was, and what it read there.
func ParseLayoutBoxes ¶
ParseLayoutBoxes extracts the blocks from a page's raw transcription.
Returns nil when the text carries no boxes — a tesseract page, a text-layer page, or a model that does not emit them. A caller renders the text alone in that case; "no layout" is an ordinary answer, not an error.
type LayoutOpts ¶
type LayoutOpts struct {
// DPI the image was rendered at. Everything below is derived from it.
DPI int // 0 → 200
// GapIn is how far apart ink may be, IN INCHES, and still belong to one
// cluster. The single knob that decides whether the lines of a paragraph
// merge into one block (they should) or a drawing's separate labels merge
// into one (they must not).
//
// 0.25in sits above text line pitch (~0.2in for 10pt) and below the spacing
// between labels on a survey (~0.5in and up), which is the whole window this
// has to land in.
GapIn float64 // 0 → 0.25
// MinIn is the smallest side of a cluster worth reading; below it is a speck.
MinIn float64 // 0 → 0.08
// CellPx overrides the derived working resolution. Segmentation does not need
// full resolution, and a 40 megapixel sheet does need to not be scanned per
// pixel — derived as one cell per 0.04in, so the grid means the same thing at
// any render resolution.
CellPx int // 0 → DPI/25
// GapCells and MinCells override the derived cell counts. Prefer GapIn/MinIn.
GapCells int
MinCells int
// MaxClusters caps the result. Above it the smallest are merged into their
// nearest neighbour rather than dropped — losing a label is worse than
// reading it alongside its neighbour.
MaxClusters int // 0 → 24
// Disabled falls back to the geometric grid — for a corpus where segmentation
// misreads the layout, and for testing the grid itself, which is otherwise
// unreachable once clusters are preferred.
Disabled bool
}
LayoutOpts tunes cluster finding.
The distances are PHYSICAL — inches — and the cell grid is derived from DPI, because a pixel means nothing on its own. The first version fixed GapCells at 3 cells of 8px, which is 0.12 inch at 200 DPI: narrower than the line spacing it was supposed to bridge. So a certificate block fragmented into many tight crops, a number split across two of them was read by neither, and wiring the segmenter into the descent scored 12/17 where the geometric grid scored 16/17.
type Mark ¶
type Mark struct {
A string `json:"a"`
B string `json:"b"`
Kind MarkKind `json:"kind"`
// Supersedes is the path that GOVERNS, for a version pair whose order is
// known. Empty means the pair is two versions with no ruling on precedence —
// a real and common state (two undated drafts), not a missing field.
Supersedes string `json:"supersedes,omitempty"`
Note string `json:"note,omitempty"`
By string `json:"by,omitempty"`
At string `json:"at,omitempty"`
// The evidence as it stood when the ruling was made, so a later reader can
// see what was in front of the person. Not used for anything: a ruling is not
// re-derived from these, it is quoted.
Relation Relation `json:"relation,omitempty"`
Coverage float64 `json:"coverage,omitempty"`
}
Mark is one ruling on a pair of documents.
A and B are index-relative document paths, ORDER-NORMALIZED (A < B) so the pair is one fact rather than two and is found from either side. Supersedes is not normalized — it names a side, and which side is the whole content of an ordering.
func ReadLegacyRelations ¶
ReadLegacyRelations parses the retired relations.jsonl. Later lines win, as the append-only format intended. Missing file is not an error.
type MarkKind ¶
type MarkKind string
MarkKind is a person's ruling on an overlapping pair.
const ( // MarkCopy — the same instrument, with no substantive difference. Whichever // copy is more legible is the one to cite; the other is redundant. MarkCopy MarkKind = "copy" // MarkVersion — the same instrument, differing in substance. Both are // evidence, and Supersedes records which one governs when that is known. MarkVersion MarkKind = "version" // a quoted paragraph, a common exhibit. Recorded rather than left blank so the // pair stops coming back as an open question. MarkUnrelated MarkKind = "unrelated" )
type Mask ¶
type Mask []iv
Mask marks regions of a document that are CORPUS-GENERIC — text that recurs across unrelated documents and therefore says nothing about whether two documents are copies of each other.
This is the single most important correction measured on real data, and the failure it fixes was not the one expected. Legal boilerplate (statutory deed forms, notary blocks) turned out to be survivable: it produces scattered runs that a 40-character floor and a coverage measure already discount. What broke the tool was EMAIL CHROME. Two entirely different one-paragraph emails from the same lawyer shared their header template and a 450-character confidentiality notice — 593 of 800 folded characters — and were reported as `duplicate` at 0.75/0.69 coverage.
Measured on the all-pairs audit: 774 duplicate-or-containment findings with no masking, 57 with it, and the same 5 known-true pairs present in both. A triage tool that cries duplicate 774 times on a 233-document corpus is a tool nobody runs twice. The specific pair above went from `duplicate` 0.75/0.69 to `overlap` 0.09/0.09 with 981 of its characters masked.
Neither Jaccard, containment, nor contiguity can separate that case: the shared text is long, contiguous, and genuinely identical. Only its DOCUMENT FREQUENCY distinguishes it. So generic regions are removed from the alignment.
Intervals are in FOLDED offsets, half-open, sorted and non-overlapping.
func BuildMask ¶
func BuildMask(f FoldedText, generic map[uint64]bool, w, mod int) Mask
BuildMask locates a document's corpus-generic regions in folded offsets.
Only a 1-in-mod sample of shingles is known to be generic, so a generic REGION is reconstructed by widening each generic sample's window. The widening is what makes this work rather than merely shred runs: sampled shingles inside a generic region are spaced about `mod` apart, so windows of width w abut only when mod <= w and leave gaps otherwise. Padding by mod on each side closes those gaps for any mod, and the cost is up to mod characters of real text at each region boundary — under the MinRunChars floor, so it cannot turn a real match into a miss on its own.
Without the padding, at mod == w the masks met exactly and a one-character rounding left every other footer unmasked, which read as the cutoff being wrong.
type Mod ¶
Mod is a modifier: something true about the PIXELS this reader was handed, or about what the caller wants. Modifiers never change the question.
func WithDamage ¶
WithDamage reports what was MEASURED about the pixels, and what may be asked about it.
Measured 2026-08-03: the variance of the laplacian identifies a blurred crop that the model itself diagnoses as "skew" with 0.9 confidence — and the deskew it then prescribes makes a blurred region worse. So the number is what notices, and the model is what decides, because only it can see whether the region is a faded fax or a drawing that is mostly white space.
func WithGrid ¶
WithGrid tells a computed tile that it is one, and what that obliges it to leave alone.
The 45% rule is asymmetric on purpose. Cells overlap by descentPadIn, so an item near a seam is partly visible from both sides; without a rule both guess. The two failure modes are not equal — a duplicated monument call is recoverable by anyone reading the transcript, a dropped one is invisible. At 45% nothing can be dropped by geometry: if this cell holds under 45% of an item, the neighbour necessarily holds over 55% and takes it.
type ModelRole ¶ added in v0.5.0
ModelRole is one model a worker calls: where it is served, the key for it, and the model id. Model empty → the role is off.
type Models ¶ added in v0.5.0
type Models struct {
// Vision reads page images (OCR). Empty Model → scans and images are read by
// the local engine alone (tesseract or OCR.CheapEngine), or fail as
// unreadable when there is none.
Vision ModelRole
// Segment splits transcribed text into fragments. Empty Model → the vision
// model; empty BaseURL → the vision endpoint.
Segment ModelRole
// Embed makes vectors. NewWorker probes it for its input limit; the
// embedder itself is set on the Store or Registry, built by Models.Embedder.
Embed ModelRole
// Audio is the speech-to-text model. Empty Model → audio jobs fail with a
// clear message.
Audio ModelRole
// Identity captions each ingested document (identity.go). Empty Model →
// identity is off. Empty BaseURL → the segment endpoint.
Identity ModelRole
// OCR selects and tunes the cheap first-pass engine, and its default
// strategy supplies the resolution policy.
OCR OCRConfig
// Fragmenter parameters. Zero → the defaults (ResolveFragParams).
FragWindow, FragStride, FragFloor int
// Configured input limits. Zero → probed from the endpoint and cached on
// the index.
EmbedLimitChars, EmbedLimitTokens, SegmentInputLimitTokens int
// Channels is the per-model admission registry (modelchan.go). Required:
// NewWorker and Identifier panic on nil. Model capacity is a property of the
// SERVER, so build one registry for the process with NewChannels, set its
// Max (the per-model channel ceiling, config model_channel_max), and pass it
// to every index's worker. A registry per worker would split one server's
// capacity between controllers that each learn it alone.
Channels *Channels
// OnUsage, when set, is called once per completed model response with the
// role ("vision", "segment", "identity", "embed"), that role's model id and
// the tokens the endpoint reported, so a host can total or cap spend per
// role. It runs on the calling goroutine of the request, possibly from many
// at once: it must be safe for concurrent use and must not block. An
// OnUsage already on a client is chained, not replaced. nil → no change.
//
// Coverage: the four roles above only. STT, image embed and the OCR engine
// use plain HTTP clients and do not report usage. "embed" reports only
// through an Embedder from Models.Embedder (or a client wrapped with
// UsageVector): set that one on the Store or Registry. The embed limit
// probes NewWorker makes do not report.
//
// "embed" is an ESTIMATE, not a reported count: agentkit's Client.Embed
// discards the response's usage, so UsageVector reports ceil(bytes/4) of
// input. Code often runs below four bytes per token, so the estimate can
// be low.
OnUsage func(role, model string, u llm.Usage)
// Code cuts source files at their declarations and Markdown at its
// headings (KindCode, codesegment.go). nil → both are read into windows,
// as before. Its Runtime() is in the KindCode pool key (poolRecipe).
Code CodeSegmenter
}
Models is everything NewWorker needs to know about the models and the ingest settings. Plain data: it does not read a Home, a config file or flags. raglit fills it from config.json and its flags; an embedder of the library fills it from its own config.
func (Models) Embedder ¶ added in v0.7.0
Embedder builds the embedder for ingest and search: the batch retry policy, the embed model's admission channel with its 429 hook, and the OnUsage report as role "embed". nil when no Embed model is set. m.Channels must be set. Set the result on the Store (SetEmbedder) or the Registry.
func (Models) Identifier ¶ added in v0.5.0
func (m Models) Identifier() *Identifier
Identifier builds the document-identity model client, or returns nil when identity is off (no Identity model). m.Channels must be set.
Same batch retry policy as the rest of ingest: this call happens at the end of a document that may have cost thirty OCR requests, and losing the caption to a transient 502 wastes the one moment the whole transcript is in hand.
func (Models) UsageVector ¶ added in v0.7.0
func (m Models) UsageVector(inner VectorClient) VectorClient
UsageVector wraps an embedding client so each successful call reports to m.OnUsage as role "embed". Returns inner unchanged when OnUsage is nil.
The token count is an estimate (see Models.OnUsage): ceil(bytes/4) over the inputs sent. The model id is the one the call names, else m.Embed.Model.
type NomicVisionEmbedder ¶
type NomicVisionEmbedder struct {
URL string // full endpoint; "" → defaultNomicVisionURL
APIKey string
Model string // "" → defaultNomicVisionModel
HTTP *http.Client
}
NomicVisionEmbedder embeds figure images via nomic-embed-vision over HTTP. Its vectors are L2-normalized (like the text embedder's), so cosine is a dot product.
func NewNomicVisionEmbedder ¶
func NewNomicVisionEmbedder(url, apiKey, model string) *NomicVisionEmbedder
NewNomicVisionEmbedder builds the embedder. url/model empty → the nomic defaults.
func (*NomicVisionEmbedder) Aligned ¶
func (e *NomicVisionEmbedder) Aligned() bool
Aligned reports that nomic-vision shares nomic-text's space — true by construction (this embedder is the nomic pair). The caller is responsible for keeping the text embed model on nomic-embed-text; a mismatch silently degrades figure ranking rather than erroring.
func (*NomicVisionEmbedder) EmbedImage ¶
func (e *NomicVisionEmbedder) EmbedImage(ctx context.Context, mime string, data []byte) ([]float32, error)
EmbedImage posts the image as multipart form-data (model + images file) and returns the L2-normalized embedding.
type Note ¶
type Note struct {
ID int64 `json:"id"`
// Page is 0 for a note about the whole document. Stored as NULL in that
// case — the column is nullable so "about the document" and "about page 0"
// cannot be confused, and page 0 is a real value here (text windows carry
// it).
Page int `json:"page,omitempty"`
Body string `json:"body"`
Author string `json:"author,omitempty"`
CreatedAt string `json:"created_at,omitempty"`
}
Note is one person's comment on a document, or on one of its pages.
type OCR ¶
type OCR struct {
Client Chatter
// Model is the vision model's id, recorded on every page this OCR reads
// (page_readings.model). Not used to call anything — the client already knows
// which model it talks to — but a transcription whose author is unrecorded
// cannot be told from one a different model produced, and a corpus outlives
// several models.
Model string
Prompt string // transcription instruction; "" → Prompt(PromptPlain)
// Collection is what the corpus owner says about reading THIS collection
// (Store.IndexHint) — appended to every reading prompt.
//
// It reaches the transcription rather than only the later asks because
// "how to decode" is a property of the pixels: "RO" on a garage's paperwork
// is a repair order, the second column of a carbon copy is the customer's,
// and a model reading one page cannot infer either. It is part of the
// READING RECIPE for the same reason — a changed hint changes what a page
// says, and pooled work read under the old one must not be replayed under
// the new.
Collection string
// Trace, when non-nil, receives one line per decision this OCR takes.
//
// It exists because nothing else reports what a call DID. `raglit doctor`
// says what is configured; the page cache, the cheap tier, the assist, the
// repetition guard and the tool list are all invisible at the call site, and
// a reader of the output cannot tell a cheap-engine page from a VLM page, an
// assisted prompt from a bare one, or a cut stream from a complete one.
//
// Written to be read by a person mid-investigation, not parsed.
Trace io.Writer
// Render is the automatic per-page resolution policy. Zero → the measured
// package defaults, which is what every page got when these were constants.
// Carried on OCR rather than passed down because OCR is already threaded to
// every extract path, and a policy that has to be plumbed separately is one
// that will be plumbed to some paths and not others.
Render RenderPolicy
// TraceJSONL, when non-nil, receives ONE JSON OBJECT PER LINE describing each
// interaction and the transforms applied to it.
//
// Separate from Trace rather than a format flag on it, because the two answer
// to different readers. Trace is prose for a person mid-investigation; this is
// for diffing two runs, joining a failure to the exact bytes that produced it,
// and counting what happened across a corpus — none of which survives being
// parsed back out of a sentence.
//
// Every event carries img_sha (first 12 hex of the image bytes), and that is
// what makes a line joinable: the same crop at the same scale has the same sha
// across models and runs, so "these two models disagreed on THIS image" is a
// join rather than a reconstruction.
TraceJSONL io.Writer
// TraceCtx is merged into every event this OCR emits. The region walk sets it
// per call — doc, page, region, tokens/in² — so a line in the log says WHICH
// crop it describes rather than only which image sha.
//
// A field rather than a parameter because the walk already swaps o.Prompt per
// call and restores it after; carrying the context the same way keeps the two
// in step, and there is no path where one is set and the other is not.
TraceCtx map[string]any
Cheap PageEngine // optional cheap first pass; nil → VLM-only
Gate GibberishConfig // when the cheap pass escalates to the VLM (zero → defaults)
// DescribeFigures is the FIGURE gate (§3a): a born-digital PDF page carrying an
// embedded image is rasterized to the VLM even when its text layer is clean, so
// its diagrams get described. Orthogonal to the gibberish gate (which judges
// TEXT quality); OFF by default because it flips such pages to the (costly)
// vision path and thus to llm-seg.
DescribeFigures bool
// MaxTokens caps one page transcription; 0 → defaultOCRMaxTokens. See
// chat.go for why an unbounded transcription is not a safe default.
MaxTokens int
// Assist changes what the cheap engine is FOR: not a substitute for the
// vision model but a spelling reference handed to it.
//
// The cascade's bargain is "if the cheap read looks clean, keep it and skip
// the model". That is a cost decision, and on a corpus of filed documents it
// buys the wrong thing: measured here, tesseract reads a surveyor's
// certificate as 20123164 with 86% confidence — clean-looking, wrong, and
// the gibberish gate passes it.
//
// The two readers fail differently, which is what makes the assist work. The
// vision model normalises proper names (DACE → DACF) and tesseract does
// not; tesseract mangles digits and the vision model, at sufficient
// resolution, does not. So the model reads the page, with the cheap engine's
// WORDS for spelling and its NUMBERS REMOVED — measured on the disputed
// record of survey, that combination got all four checked facts right,
// including one auditor's file number that no other configuration read
// correctly. Handing over the numbers as well loses two of them: the model
// copies them, whatever the instructions say.
Assist bool
}
OCR transcribes page images to text. It runs a CASCADE: an optional cheap first-pass engine (tesseract / paddleocr), gated by a gibberish detector, and falls back to a vision-capable chat model (e.g. gemma-4-12b on bonsai, via corrallm) only when the cheap pass is missing, errors, or looks like garbage. With no Cheap engine it is VLM-only — the original behavior.
func NewOCR ¶
NewOCR wraps a Chatter (an *llm.Client) as an OCR transcriber. The cheap tier is off by default; set OCR.Cheap (see BuildPageEngine) to enable the cascade.
func TesseractOnlyOCR ¶ added in v0.5.0
TesseractOnlyOCR is an OCR with no vision model: the configured cheap engine, else tesseract when it is on PATH. nil when neither exists. Every page is taken from the engine as read (OCR.PageAsSeen), and figures are not described, since describing one needs the vision model.
func (*OCR) AskWithHint ¶
func (o *OCR) AskWithHint(hint string) func(context.Context, PageImage, RegionAbout) (RegionReading, error)
AskWithHint is AskWithOCR with a caller's statement of what is being looked for appended to whichever prompt applies.
Appended rather than woven in: the instruction competes with the image for the model's attention, and a hint that rewrites the whole prompt would also change the parts that make the answer parseable.
func (*OCR) AskWithOCR ¶
func (o *OCR) AskWithOCR() func(context.Context, PageImage, RegionAbout) (RegionReading, error)
AskWithOCR builds an Ask that drives a VLM through the existing OCR client, so the region walk inherits the page cache, the repetition guard and the retry policy already in place.
func (*OCR) PageAsSeen ¶
func (o *OCR) PageAsSeen(ctx context.Context, img PageImage) (text, engine string, shrinks int, err error)
PageAsSeen is PageWithEngine plus the one thing a caller needs to reproduce what the model actually looked at: how many times the image was downscaled to fit the context before it was read.
Separate rather than a wider PageWithEngine because only the region walk cares — it records the number per region so the crop can be re-rendered exactly, and every other caller wants the text and the engine and nothing else.
func (*OCR) PageWithEngine ¶
PageWithEngine transcribes one page and reports which engine produced it: the cheap engine's Name() when its result passed the gibberish gate, else "vision". The cascade never drops a page — a cheap-engine error or a gibberish verdict escalates to the VLM rather than returning the bad text.
type OCRConfig ¶
type OCRConfig struct {
CheapEngine string `json:"cheap_engine,omitempty"` // "none" | "tesseract" | "paddleocr"
PaddleURL string `json:"paddle_url,omitempty"` // sidecar base URL when cheap_engine=paddleocr
TesseractBin string `json:"tesseract_bin,omitempty"` // tesseract binary; "" → "tesseract"
TesseractLang string `json:"tesseract_lang,omitempty"` // -l language; "" → "eng"
Gibberish GibberishConfig `json:"gibberish,omitempty"` // gate overrides; zero → precision-biased defaults
// DescribeFigures escalates born-digital PDF pages that carry an embedded image
// to the VLM so their figures are described inline (§3a figure gate). OFF by
// default: it forces those pages onto the vision/llm-seg path (costly).
DescribeFigures bool `json:"describe_figures,omitempty"`
// Mode decides what the cheap engine is for.
//
// "cascade" (default) — it reads first, and the vision model is paid for
// only when the gibberish gate rejects the result. A cost decision.
// "assist" — the vision model reads every page, with the cheap engine's
// WORDS as a spelling reference and its numbers removed. See OCR.Assist:
// the two readers fail differently, and this is how each one's strength
// is used without importing the other's errors.
//
// A corpus where a misread digit is expensive wants "assist"; a corpus where
// throughput matters more than a certificate number wants "cascade".
Mode string `json:"mode,omitempty"`
// Strategies are named bundles of read policy, selectable per index via
// IndexConfig.OCRStrategy. They exist because one corpus is not one kind of
// page: a records/ folder of recorded surveys wants descent, tiling and a
// hint about monument calls, while correspondence/ wants none of it and
// should not pay for the check. Before this the knobs lived only on
// `raglit regions` flags, so a policy could not be attached to a corpus at
// all — it had to be retyped per invocation and was forgotten by the next.
Strategies map[string]StrategyConfig `json:"strategies,omitempty"`
// Strategy names the default for indexes that do not choose one. Empty means
// the zero StrategyConfig, which is today's behavior exactly.
Strategy string `json:"strategy,omitempty"`
}
OCRConfig selects and tunes the cheap first-pass OCR engine. The cascade tries this engine before the vision model and escalates only when the page looks like gibberish (see ocr.go, gibberish.go). CheapEngine="none" (the default) disables the cheap tier — the cascade is then VLM-only.
type OCRLine ¶
OCRLine is one recognized text box from a cheap page-OCR engine, with the recognizer's confidence (0..1). The gibberish gate reads the per-line scores in aggregate (MeanConfidence); downstream only the joined Text matters.
type OffsetFrag ¶
OffsetFrag is one windowed fragment plus its half-open [Start,End) span into the source text it was cut from.
func OverlapFragments ¶
func OverlapFragments(text string, window, stride, floor int) []OffsetFrag
OverlapFragments cuts text into overlapping windows. window/stride/floor must already be resolved (ResolveFragParams). A document at or under one window is a single fragment with no overlap; a tail shorter than floor is folded into the final window rather than emitted as a near-duplicate ("barely-over document").
type PaddleEngine ¶
PaddleEngine is a PageEngine backed by an out-of-process PaddleOCR sidecar (the one ragtag used; optionally installed via docker). It POSTs the page image to <BaseURL>/ocr and decodes the bespoke {text, lines, mean_confidence, box_count} contract. Higher print accuracy than tesseract, at the cost of a running sidecar.
func NewPaddleEngine ¶
func NewPaddleEngine(baseURL string) *PaddleEngine
NewPaddleEngine builds a PaddleEngine for the sidecar base URL. The timeout is generous: CPU PP-OCR on a dense page can take a couple seconds.
func (*PaddleEngine) Name ¶
func (e *PaddleEngine) Name() string
func (*PaddleEngine) OCRPage ¶
OCRPage sends one page image to the sidecar and returns its PageOCR. The Content-Type is taken from the page's mime (the sidecar decodes png as readily as jpeg), defaulting to image/jpeg. Transport / decode / non-200 errors propagate so the cascade can fall back to the vision model.
type PageCorrection ¶
type PageCorrection struct {
Doc string `json:"doc"`
Page int `json:"page"`
Text string `json:"text"`
Note string `json:"note,omitempty"`
By string `json:"by,omitempty"`
At string `json:"at,omitempty"`
// Supersedes is the reading this one replaced — the machine's text, or an
// earlier correction.
//
// A correction is an ATTESTATION that a better reading exists, not an erasure
// of the old one. What the machine said is evidence: it is what the index held
// when a document was cited, it is what a stale quotation will match, and
// "the OCR read 2015031104" is itself a fact about how reliable this sheet is.
// Keeping it means the trail carries the whole lineage of a page and says
// which reading is ACTIVE, rather than only the winner.
Supersedes string `json:"supersedes,omitempty"`
}
PageCorrection is what a person read off a page that the machine got wrong.
Text replaces the machine read for that page when the transcription is rendered. Note records HOW it was established — the magnification, the source image — because a corrected identifier that cannot be re-checked is only a different unverified value.
type PageEngine ¶
type PageEngine interface {
OCRPage(ctx context.Context, img PageImage) (PageOCR, error)
Name() string // short id for the per-page engine tag: "tesseract" | "paddleocr"
}
PageEngine is a CHEAP first-pass OCR over one rendered page image — the tier the cascade tries before paying for the vision model (see OCR.Page). It is the swappable "other OCR system" slot: a local CLI (tesseract) or a bespoke sidecar (paddleocr) are interchangeable here.
func BuildPageEngine ¶
func BuildPageEngine(cfg OCRConfig) (PageEngine, error)
BuildPageEngine constructs the configured cheap engine, or nil for "none" (the cascade is then VLM-only). An unknown name or a paddleocr backend with no URL is a configuration error the caller should surface.
type PageImage ¶
type PageImage struct {
Page int // 1-based page number
Path string // where it was written (empty when outDir == "")
Mime string // "image/png", "image/jpeg", …
Data []byte // raw image bytes
}
PageImage is one page's image, ready for OCR.
func Pagify ¶
Pagify extracts embedded page images from an image/scanned PDF, in page order. If outDir is non-empty each image is also written there as p<NN>.<ext> (with -<objnr> appended when a page carries multiple images). Returns ErrNoPageImages for a PDF with nothing to extract.
func PagifyBytes ¶
PagifyBytes extracts embedded page images from an in-memory image/scanned PDF — the MCP tool's rasterization path, so a caller can pass PDF bytes without a temp file. Same semantics as Pagify with an empty outDir (no files written).
type PageLayout ¶
type PageLayout struct {
Doc string `json:"doc"`
// Page is 1-based, as everything user-facing in raglit is.
Page int `json:"page"`
// Engine and Model say WHO read it — a tesseract page has no boxes and that
// is worth showing rather than looking like a failure.
Engine string `json:"engine"`
Model string `json:"model"`
// HasImage is false when the page image was never saved or has been cleaned
// up; the text panes still work.
HasImage bool `json:"has_image"`
// ImgW/ImgH are the page image's pixel size, read from its header only.
//
// The boxes are normalised and so need no dimensions to PLACE — but a
// reconstruction of the page from text alone needs the page's SHAPE, and a
// client rendering it beside the image (not over it) has nothing to infer
// that from. Zero when unknown; a caller then falls back to the block list.
ImgW int `json:"img_w,omitempty"`
ImgH int `json:"img_h,omitempty"`
Boxes []LayoutBox `json:"boxes"`
// Raw is the model's own output, markup and all. Indexed is what search
// actually matches against. Showing both side by side is the point: they
// have differed by 40% of their bytes and nothing surfaced it.
Raw string `json:"raw"`
Indexed string `json:"indexed"`
}
PageLayout is everything needed to render one page three ways: the picture, what the model reported about it, and what the index actually holds.
type PageOCR ¶
type PageOCR struct {
Text string
Lines []OCRLine
MeanConfidence float64
BoxCount int
// MedianGlyphPx is the median height of recognised text, in pixels of the
// image as given — min(width, height) per word, because text that runs
// sideways up a sheet has its height on the other axis. 0 when the engine
// does not report geometry.
//
// It is how a page says it needs a better look: the pages every reader read
// correctly measure 18-25 px here, and the page they all misread measures 10.
// See renderDPIFor.
MedianGlyphPx int
}
PageOCR is a cheap engine's result for one page image. Text is the recognized lines joined in reading order; MeanConfidence and BoxCount are the cheap signals the gibberish gate fuses to decide whether the page needs a vision-model re-OCR. BoxCount == 0 means no text at all (blank page or a pure figure) — treated as an empty page, not gibberish.
type PagePair ¶
type PagePair struct {
APage int `json:"a_page"`
// BPage is 0 when nothing in B matches this page at all — a page present in
// one document and absent from the other, which is the strongest per-page
// finding available and is invisible in any average.
BPage int `json:"b_page,omitempty"`
// Rate is |A∩B| / |A| over the page's distinct shingles: how much of THIS
// page of A occurs on that page of B. Asymmetric on purpose — a short page
// wholly inside a long one is 1.0 here and small in reverse.
Rate float64 `json:"rate"`
// Identical is exact equality of the two pages' folded text. A rate of 1.0
// is not the same claim: shingles can be a subset while the text differs in
// what falls between them.
Identical bool `json:"identical"`
AChars int `json:"a_chars"`
BChars int `json:"b_chars"`
}
PagePair is one page of A and the page of B it best corresponds to.
func PageRates ¶
func PageRates(a, b FoldedText, w int) ([]PagePair, []int)
PageRates computes, for every page of a, the page of b it best corresponds to.
Exact, not sampled. The stored sketches hold a 1-in-N sample, which is right for finding candidates across a corpus and wrong here: these numbers are read by a person deciding which of two instruments a filing cites, and a rate that is usually right is the wrong instrument for that decision.
type PageReading ¶
type PageReading struct {
Doc string `json:"doc"`
Page int `json:"page"`
Seq int `json:"seq"`
Text string `json:"text"`
Source string `json:"source"` // machine | corrected
// Engine and Model say WHICH reader produced this, where Source says only
// whether it was a machine at all. A corpus read by a text layer, then
// tesseract, then one vision model, then another cannot answer "how far do I
// trust this page" while all four are equally "machine".
Engine string `json:"engine,omitempty"` // vision | tesseract | paddleocr | text-layer
Model string `json:"model,omitempty"` // the model id behind the engine, when it has one
Note string `json:"note,omitempty"`
By string `json:"by,omitempty"`
At string `json:"at,omitempty"`
Active bool `json:"active"`
}
PageReading is one version of a page's text.
type PageReview ¶
type PageReview struct {
Page int `json:"page"`
Engine string `json:"engine"`
Vision bool `json:"vision"`
HasImage bool `json:"has_image"`
Fragments int `json:"fragments"`
Text string `json:"text"`
// TextChars and DescribedChars are how much indexable text this page produced
// and how much of it was the model DESCRIBING an image rather than reading
// one. Measured at ingest — the markup that says so does not survive into
// Text, so this cannot be recomputed from what is returned here.
TextChars int `json:"text_chars,omitempty"`
DescribedChars int `json:"described_chars,omitempty"`
}
PageReview is one page's OCR review: the engine that produced it, whether it needed the VLM, whether a page image is on disk, and the text indexed for it.
func (PageReview) DescribedPct ¶
func (p PageReview) DescribedPct() int
DescribedPct is how much of this page a model made up, 0-100.
type PageSketch ¶
type PageSketch struct {
Page int
// Chars is the page's folded length. Kept because Total alone cannot tell a
// blank page (no shingles, nothing to say) from a page shorter than one
// shingle (no shingles, but it does hold text).
Chars int
// Total is the count of DISTINCT shingles on the page — the denominator of
// containment. Sampling makes the numerator estimable and would make this
// unrecoverable, so it is stored.
Total int
// Hashes are the sampled shingles, distinct and sorted.
Hashes []uint64
}
PageSketch is one page's stored evidence: the sampled shingle hashes, plus the sizes that sampling throws away.
func SketchPages ¶
func SketchPages(f FoldedText, w, mod int) []PageSketch
SketchPages builds the stored sketch for a folded document, one entry per page. Pages are emitted even when they hold no shingles, so a re-sketch replaces a document's rows completely and page numbering stays contiguous.
type PageSpan ¶
type PageSpan struct {
Off int `json:"off"` // byte offset into the fragment's text
Page int `json:"page"` // the page whose content begins there
}
PageSpan marks where a page's content begins inside a fragment's text.
A fragment spans pages whenever the assembler absorbs a continuation, or a sub-floor sibling, from the following page. Keeping only the start page made `fragments.page` right for the beginning of a fragment and wrong for the rest, so a hit inside one could not be resolved to the page it actually sits on.
func DecodePageSpans ¶
DecodePageSpans parses what encodePageSpans wrote; nil on empty or malformed, so a caller degrades to the start page rather than failing.
type PageText ¶
type PageText struct {
Page int
Text string
Engine string
// Err records why THIS page has no text, when it failed on its own.
//
// A page that fails is not a blank page, and the difference has to survive:
// downstream, an empty Text with no Err is a legitimately blank sheet, and an
// empty Text with an Err is a hole that somebody should come back to.
Err string
}
PageText is one page's extracted text plus the engine that produced it, for the `ocr` MCP tool's paged output. Engine is "text" for a PDF text layer, pandoc, or a plain-text read; "tesseract"/"paddleocr"/"vision" for an OCR'd (scanned/image) page.
func EmailText ¶
EmailText renders a mail archive as paged text, one page per message.
Pure: it reads and returns, and writes nothing. That is load-bearing — the `ocr` MCP tool calls this to read a file the user named, and a read tool that materialises 69 files as a side effect is a tool nobody can trust. Attachment extraction is ExtractEmailAttachments, called separately by ingest.
func ExtractPaged ¶
ExtractPaged extracts a document to paged text — the `ocr` MCP tool's core. It routes by kind: a PDF runs the text-layer/OCR hybrid, office/markup goes through pandoc (one page), an image runs the OCR cascade, and text is read directly. ocr may be nil when no page needs OCR (a born-digital PDF, office, or text); a scanned page with a nil ocr is a clear error.
func ParseTranscription ¶
ParseTranscription reads a raglit transcription sidecar back into pages.
The inverse of RenderTranscription, and the reason it exists is the upload case: a client sends a file whose text this index has never read, but whose sidecar is already beside it from a previous ingest elsewhere. Re-OCRing it to compare would need a vision model, which puts the whole check behind a network call and out of CI. The sidecar's `## Page N` headings are the documented contract, so parsing them is safe — and a heading whose number does not parse is skipped rather than guessed at, because a wrong page number in an alignment is worse than a missing one.
type PendingItem ¶
type PendingItem struct {
ID int64 `json:"id"`
URL string `json:"url"`
State string `json:"state"` // pending | running
ETASeconds float64 `json:"eta_seconds"` // estimate to completion; 0 = unknown
}
PendingItem is one queued (not-yet-done) job plus an ETA estimate.
type Pool ¶
type Pool struct {
// contains filtered or unexported fields
}
Pool is the shared processed-document cache (pool.sqlite + pool-pages/) under a daemon root.
func OpenPool ¶
OpenPool opens (creating if needed) the pool under root: <root>/pool.sqlite and <root>/pool-pages/.
func (*Pool) GC ¶
GC evicts pooled documents per policy, oldest-accessed first, and removes any pool-pages/<file> dir no surviving entry references. Returns the number of entries evicted.
func (*Pool) PurgeFiles ¶ added in v0.4.1
PurgeFiles removes every recipe's cached work for these files, page images included. Returns the number of entries removed.
type PoolStats ¶
type PoolStats struct {
Entries int `json:"entries"` // cached (recipe, file) documents
Files int `json:"files"` // distinct source files (pool-pages dirs)
Bytes int64 `json:"bytes"` // total cached-payload bytes (the GC budget basis)
}
PoolStats is a snapshot of the pool for reporting.
type PooledDoc ¶
type PooledDoc struct {
Title string `json:"title"`
FragMode string `json:"frag_mode,omitempty"`
FragRecipe string `json:"frag_recipe,omitempty"`
Fragments []PooledFragment `json:"fragments"`
Pages []PooledPage `json:"pages"`
// Identity is the document's caption/summary/kind (identity.go), carried for
// the same reason PageSpans is: reuse replays a document without OCR or the
// model, so anything a model established has to survive here or it is lost on
// every reuse — and the identity call is a model call, which is the cost the
// pool exists to avoid paying twice.
//
// The identity FRAGMENT is not in Fragments — it is rebuilt from this on
// import, so a pooled document cannot end up with two of them, and an older
// payload that predates this field simply has no caption.
Identity *DocIdentity `json:"identity,omitempty"`
}
PooledDoc is a fully-processed document: the reusable output of one ingest. Media rows are NOT serialized — they are recomputed on reuse from the pooled fragments' [FIGURE: ...] markers + restored page images (extractMedia), which reproduces them exactly.
type PooledFragment ¶
type PooledFragment struct {
Page int `json:"page"`
Ord int `json:"ord"`
StartOff int `json:"start_off,omitempty"`
EndOff int `json:"end_off,omitempty"`
// PageSpans is where each page begins inside Text for a fragment that spans
// pages. Carried through the pool because this is the CHEAP reuse path: a
// pooled document is replayed without OCR or segmentation, so anything the
// fragmenter worked out has to survive here or it is lost on every reuse —
// and a pooled doc is shared across projects, so losing it here loses it
// everywhere. Omitted when the fragment lies on one page; absent in payloads
// pooled before the field existed, which read back as nil.
PageSpans []PageSpan `json:"page_spans,omitempty"`
Text string `json:"text"`
Vec []float32 `json:"vec,omitempty"`
// Origin marks text nobody wrote — "described", a model's account of an
// image. Carried for the reason PageSpans is, and more urgently: reuse
// replays a document without OCR, and the evidence for the mark (layout
// markup) is gone by then, so a mark dropped here can never be recomputed.
//
// Without it a photograph pooled from one index arrived in the next as an
// ORDINARY fragment: a model's "a red Chevrolet Malibu, licence plate
// CEP0912" indexed with nothing to say a model made it up, quotable as the
// record. Empty for transcription, which is nearly everything.
Origin string `json:"origin,omitempty"`
// Line span and symbol path (Fragment), carried for the reason PageSpans
// is. Absent in payloads pooled before the fields existed: 0/0 and none.
StartLine int `json:"start_line,omitempty"`
EndLine int `json:"end_line,omitempty"`
Symbol string `json:"symbol,omitempty"`
}
PooledFragment is one cached fragment: its text, source span, and (optional) vector.
type PooledPage ¶
type PooledPage struct {
Page int `json:"page"`
Engine string `json:"engine"`
Image string `json:"image,omitempty"`
// How much of this page a model DESCRIBED rather than transcribed. Same
// reason as PooledFragment.Origin: measured from layout markup that reuse
// never sees, so it travels or it is lost. Absent in payloads pooled before
// the field existed, which read back as 0 — see recordIngestReading on why
// that is treated as UNMEASURED and not as "nothing described".
TextChars int `json:"text_chars,omitempty"`
DescribedChars int `json:"described_chars,omitempty"`
}
PooledPage is one cached page's provenance. Image is an absolute source path on export, rewritten to a pool-pages basename when stored.
type Problem ¶
type Problem struct {
Kind ProblemKind `json:"kind"`
// Subject is the document path, or the job's target when no document exists.
Subject string `json:"subject"`
JobID int64 `json:"job_id,omitempty"`
// Stage is where it went wrong, when a stage recorded it.
Stage string `json:"stage,omitempty"`
// Detail is the recorded evidence, verbatim — the endpoint's own words where
// there are any. Summarising it here is how "increase the physical batch
// size" became "status 500" and cost a week.
Detail string `json:"detail,omitempty"`
// Fix is the command that addresses it, ready to run.
Fix string `json:"fix,omitempty"`
}
Problem is one thing worth a person's attention.
type ProblemKind ¶
type ProblemKind string
ProblemKind names a class of trouble, stable enough to filter on.
const ( // ProblemNoFragments — indexed and unsearchable. The worst kind: it looks // like a document from every angle except the one that matters. ProblemNoFragments ProblemKind = "no-fragments" // ProblemNoPages — a paged document whose pages were never recorded, so page // images, the transcript and every page-level citation have nothing behind // them. ProblemNoPages ProblemKind = "no-pages" // ProblemJobFailed — an ingest job that errored. Carries the stage it died in, // because "failed" without a stage sends the reader to the wrong half. ProblemJobFailed ProblemKind = "job-failed" // ProblemDegraded — the model would not segment a page and it was stored // whole. Not an error: the text is there and searchable, but it is one // undivided block where retrieval expects fragments. ProblemDegraded ProblemKind = "segment-degraded" // ProblemRetries — a job that only completed after the endpoint made it fight // for it. Not a failure, and the earliest warning that one is coming. ProblemRetries ProblemKind = "llm-retries" // ProblemGenerated — a document that is raglit's OWN output: a transcription // or a region record written beside a real document and then indexed as if it // were one. Never a judgement call, always a mistake, and self-feeding — // indexing a transcription writes a transcription of it. ProblemGenerated ProblemKind = "generated-indexed" // ProblemWithdrawn — ruled out of the corpus on purpose. Reported so a // document that is absent BY DECISION is never mistaken for one that is // absent by accident, which is the confusion withdrawal exists to end. ProblemWithdrawn ProblemKind = "withdrawn" // ProblemMissingFile — an indexed document whose file is no longer at the // path the row records. // // The quietest failure in this list, because search keeps working: the // fragments live in SQLite and answer queries exactly as before. What breaks // is everything that goes back to the FILE — download the original, re-read // it, show a page image, write a reading so it can be reviewed — and each of // those fails on its own, far from anything that would name the cause. // // Found on the corpus this was written against, and only because a review // sweep tried to open 355 documents and 43 were not there. Nothing else had // noticed. Most were a deliberate archival move (documents/ → legacy/) and // several were content-preserving renames whose new path was never ingested, // so four court filings existed on disk under better names and were absent // from the corpus entirely. // // Usually RECOVERABLE, which is why the fix is not simply "forget": the bytes // are generally still in the tree. 39 of those 43 were found on disk at // another path. A row is a pointer, and a moved file breaks the pointer, not // the document. ProblemMissingFile ProblemKind = "missing-file" // ProblemUnreadPage — a page the OCR could not read, in a document that was // otherwise indexed. Created deliberately by the ingest salvage: one failed // page used to discard the whole document, and now it is recorded as a hole // so the rest can be kept. That trade is only sound if the hole is FINDABLE — // a partial document nobody knows is partial is the more dangerous object, // because search returns it and a reader assumes the absence is the record's. ProblemUnreadPage ProblemKind = "page-unread" // ProblemPartialPage — a page the model read only as far as a structural // loop: every row with content is present, the blank remainder and anything // BELOW it was never seen (see enginePartialVision). // // Separate from page-unread because the two need different judgement. An // unread page is a hole a reader can see. A salvaged page reads like a whole // page — it has a heading, rows, a shape — and the part that is missing left // no gap in the text to notice. On the two Record of Ownership sheets what // went missing was the "Adjoining Property" heading under the blank rows, // which is precisely the kind of absence a reader would take for the // record's own silence. // // Salvage is still the right trade: the alternative was dropping the page // entire, and it dropped a 1972 chain-of-title entry with it. The trade is // only sound while the partial is FINDABLE, which is what this row is for. ProblemPartialPage ProblemKind = "page-partial" // ProblemEmptySource — the file is there and it is zero bytes. // // Reported rather than failed, because having no content is a fact about the // DOCUMENT and being unable to import it is a fact about the importer; saying // the second when the first is true sends the reader to the wrong half. A // 0-byte .docx failed as `pandoc .docx: exit status 63`, which names the tool // and reads as a broken install. // // It must not be silent either. The text path did not fail at all on an empty // file — it indexed a document with no fragments, which is ProblemNoFragments: // a row that looks like a document from every angle except the one that // matters. Usually a broken copy or a truncated export, and the document it // should have carried is absent from the record. ProblemEmptySource ProblemKind = "empty-source" )
type ProjectMeta ¶
type ProjectMeta struct {
ID string `json:"id"`
Label string `json:"label,omitempty"`
CreatedAt int64 `json:"created_at"`
}
ProjectMeta marks an index as a project's base: <home>/project.json.
type ProjectSavings ¶
type ProjectSavings struct {
ProjectID string `json:"project_id"`
Base string `json:"base"`
Layers []LayerSavings `json:"layers"`
// Counted over the layers the caller can read. A document held by N
// readable indexes counts N-1 times: the copies beyond the first.
DuplicateDocuments int `json:"duplicate_documents"`
DuplicatePassages int `json:"duplicate_passages"`
DuplicateEmbeddings int `json:"duplicate_embeddings"`
DuplicatePageReads int `json:"duplicate_page_reads"`
DuplicateBytes int64 `json:"duplicate_bytes"`
ModelSeconds float64 `json:"model_seconds"` // model time spent importing the copies
// Indexes one layer search reads now (itself, the base, its readable
// siblings) and with a maintained base (itself and the base).
FanoutNow int `json:"fanout_now"`
FanoutWithBase int `json:"fanout_with_base"`
// Recommend is true when a maintained base would pay: two or more layers,
// copies between them, and a base that holds nothing of its own.
Recommend bool `json:"recommend"`
Reason string `json:"reason"`
Note string `json:"note"`
}
ProjectSavings is the report for one project id.
type PromptKind ¶
type PromptKind string
PromptKind is the QUESTION being asked. Four, not one with options: a crop is asked to transcribe, a root to account for a sheet, a parent to decide geometry, and a plain page to transcribe with no JSON at all.
const ( // PromptPlain is the whole-page read on the ingest path: transcription only, // no JSON, no proposals. It is the pass that scores 14/14 on stamped // recording numbers and it is deliberately the simplest thing here. PromptPlain PromptKind = "plain" // PromptRoot accounts for a whole sheet and says where to look closely. Its // description is the PRIMARY product — an area it fails to name is an area // no later pass will crop, so the account decides coverage. PromptRoot PromptKind = "root" // PromptCrop reads one region. Transcription is the primary product. PromptCrop PromptKind = "crop" // PromptEscalate asks a PARENT what to do about a child that cannot read // itself. It REPLACES the question rather than modifying it, and forbids // transcription outright: the parent is looking at the coarse view that the // close-up exists because it cannot resolve. PromptEscalate PromptKind = "escalate" )
func (PromptKind) AcceptsMods ¶
func (k PromptKind) AcceptsMods() bool
AcceptsMods reports whether modifiers mean anything for a kind.
Plain takes none because it is not looking at a crop — a grid rule or a damage report about "this image" has no referent on a whole page. Escalate takes none because it forbids transcription, and every modifier here is about how to transcribe; appending one contradicts the question.
type Proposal ¶
type Proposal struct {
Kind MarkKind `json:"kind"`
// Why is the evidence in one line, for a person deciding. Written to be read
// aloud in a review, not parsed.
Why string `json:"why"`
// Confident is whether this can be acted on without reading the documents.
// Only exact-byte and near-exact alignment reach it.
Confident bool `json:"confident"`
}
Proposal is what raglit thinks a pair is, and why. It is never written to relations.jsonl: a proposal is an argument put in front of a person, and the file holds rulings.
func Propose ¶
Propose reads a computed match and says what the pair probably is.
The rule, and why it is this rule.
Coverage decides whether the two are the same instrument at all; it cannot decide copy versus version, because a re-recorded deed and a scan of a deed both align at 0.97. What decides that is the SHAPE of the disagreement, and there are two kinds of disagreement in this corpus that look alike in a score and mean opposite things:
OCR NOISE diffuse, alphabetic, one-sided. The 2008 lot certification
reads "LAURENCE MOONION" for Clarence Brannock. Nothing was
refiled; a scan was read badly. This is a COPY.
A SUBSTITUTION localized, numeric, and PAIRED — a number present only on one
side sits against a different number present only on the other.
An auditor file number, a distance, a date. Something was
changed and filed. This is a VERSION.
So the test for a version is paired numeric disagreement: tokens exclusive to each side, on BOTH sides. A number dropped from one side only is an omission, which OCR produces constantly, and does not qualify.
This is a heuristic and it is allowed to be, because nothing acts on it — it proposes, a person rules, and the ruling is what gets stored. Its job is to put the right pairs and the right evidence in front of that person, not to be right on its own. Where it cannot tell, it says so rather than guessing.
type Provider ¶
type Provider struct {
// Kind is the preset it was made from: "custom", "openrouter", "litellm".
// Informational; the base URL is what is called.
Kind string `json:"kind,omitempty"`
BaseURL string `json:"base_url"`
// APIKey is sent as a bearer token. Empty → no Authorization header.
APIKey string `json:"api_key,omitempty"`
// APIKeyEnv and APIKeyFile say where the key is instead of holding it: an
// environment variable of the process that makes the call, or a file whose
// contents, trimmed, are the key. A tool that already holds the user's key
// (dun) points at it rather than copying it into this file. APIKey wins
// when set. Read at each call, so a rotated key needs no restart.
//
// A daemon under systemd or launchd does not inherit a shell's variables,
// so api_key_env reaches it only through its unit; api_key_file reaches it
// always.
APIKeyEnv string `json:"api_key_env,omitempty"`
APIKeyFile string `json:"api_key_file,omitempty"`
}
Provider is one OpenAI-compatible endpoint.
type PruneResult ¶
type PruneResult struct {
// Documents whose local file was not found.
Documents int `json:"documents"`
Dirs []PruneDir `json:"dirs"`
// Removed is false on a dry run.
Removed bool `json:"removed"`
// Page and figure images under pages/ that no document names. Removed with
// the documents.
OrphanPages int `json:"orphan_pages"`
OrphanPageBytes int64 `json:"orphan_page_bytes"`
// Copies under originals/ (Home.OriginalsDir), except an upload's.
DanglingOriginals int `json:"dangling_originals"`
DanglingOriginalBytes int64 `json:"dangling_original_bytes"`
// OriginalsRemoved is true when deleteOriginals removed them.
OriginalsRemoved bool `json:"originals_removed"`
}
PruneResult is what Prune removes, or would remove.
type PurgeResult ¶
type PurgeResult struct {
Pattern string `json:"pattern"`
Documents int `json:"documents"`
Sample []string `json:"sample"`
// Pending jobs on matching paths, dropped with the documents.
IngestJobs int `json:"ingest_jobs"`
CaptionJobs int `json:"caption_jobs"`
}
PurgeResult is what a purge by pattern removes, or would remove.
type Reading ¶
type Reading struct {
SourceSHA256 string `json:"source_sha256,omitempty"`
SourcePath string `json:"source_path,omitempty"`
DocPath string `json:"doc_path"`
Method string `json:"method,omitempty"`
Level string `json:"level"`
ProducedBy string `json:"produced_by,omitempty"`
RuledBy string `json:"ruled_by,omitempty"`
At int64 `json:"at,omitempty"`
// Text is the reading itself — what gets indexed when this reading governs.
Text string `json:"text,omitempty"`
// Data is the structured form the text was rendered from, as an attest.Reading:
// units, each with a locator into the asset.
//
// ONE shape for every kind, which is attest's whole design — a Unit is "one
// machine claim about one piece of the asset", and the locator differs by
// kind rather than the model differing. A diarized recording's units are
// turns located by TIME; a scanned sheet's are page or region reads located
// by AREA; a mail archive's are messages located by SPAN. So an audio
// transcript and a page transcription are attestable in the same way, by the
// same log, and a correction to either is an attestation rather than an edit
// to what the machine said.
//
// Empty is normal — a plain text extraction has no structure worth keeping
// beyond its text.
Data string `json:"data,omitempty"`
// Ruled is what a person has been through, per facet, as JSON. See trust.go.
Ruled string `json:"ruled,omitempty"`
// Describes marks a reading that says what its source DEPICTS rather than
// what it says — a model's account of a photograph.
Describes bool `json:"describes,omitempty"`
// DescribedPct is how much of this reading the model DESCRIBED rather than
// transcribed, 0-100, or DescribedUnmeasured. See IsMixedPage: a screenshot is
// both, and a flag cannot say so.
DescribedPct int `json:"described_pct,omitempty"`
}
Reading is one account of one source.
func AuthoritativeReading ¶
AuthoritativeReading picks the reading of a source that governs.
A person's ruling first, always. Levels only order what nobody has ruled on: attested over adapted over machine, and the NEWEST wins a tie, because a second reading at the same level is a re-read and a re-read supersedes.
The ruling is looked up through `ruled`, which the caller supplies from the judgement trail — this package must not open it, because the trail lives beside the documents and the index does not know where that is.
func (Reading) LowestTrust ¶
func (r Reading) LowestTrust() Confidence
LowestTrust is the weakest thing this reading claims, which is what a reader deciding whether to quote it needs. 0 when it claims nothing.
func (Reading) Trust ¶
func (r Reading) Trust() map[string]Confidence
Trust returns this reading's confidence per facet: the method's defaults, with anything a person has ruled on layered over them.
func (Reading) TrustSummary ¶
TrustSummary renders the facets in a stable order, weakest first, for a reader and for a log line.
func (Reading) WithRuling ¶
func (r Reading) WithRuling(facet string, c Confidence) Reading
WithRuling returns the reading with a facet marked as ruled on by a person.
Per facet, because that is how ruling actually happens: somebody sits down and checks the speaker attribution of a hearing, or corrects the wording of a page, and almost never both at once. A transcript whose attribution was ruled on and whose words were not is the common case, not an edge one.
type ReadingAuthority ¶
type ReadingAuthority struct {
// Source is the sha256 of the asset both readings read. The ruling is about
// a SOURCE, not a pair, so a third reading arriving later does not need a
// second ruling to be ordered against the first two.
Source string `json:"source"`
// Doc is the document holding the reading that governs.
Doc string `json:"doc"`
// Why is the grounds. A ruling without them is a preference nobody can check
// later, which is the thing every other ruling here refuses to be.
Why string `json:"why,omitempty"`
By string `json:"by,omitempty"`
At string `json:"at,omitempty"`
}
ReadingAuthority is a person's ruling that one reading of a source governs.
The corpus holds several accounts of one thing — a machine transcript and a verified one, a whole-page read and a region descent — and which of them may be quoted is not something the corpus can compute. Levels give a default order (attested over adapted over machine); this overrides it, because a person may know that the older attested reading is the one to cite, or that a machine reading of a re-scan beats a verified reading of a bad scan.
type Rect ¶
type Rect struct {
X float64 `json:"x"`
Y float64 `json:"y"`
W float64 `json:"w"`
H float64 `json:"h"`
}
Rect is a region of a page in normalized coordinates — 0..1 of the page's width and height. Normalized rather than pixels so a region survives being re-rendered at a different dpi, which is exactly what a transform does.
Lowercase in JSON, matching RegionProposal: the model answers in x/y/w/h and the record has no business spelling the same four numbers differently.
func LayoutClusters ¶
func LayoutClusters(img image.Image, o LayoutOpts) []Rect
LayoutClusters returns the content clusters of an image, as fractions of it.
Empty when there is nothing to find, which callers must treat as "fall back to the geometric grid" rather than as "the page is blank".
func LayoutClustersPNG ¶
func LayoutClustersPNG(data []byte, o LayoutOpts) []Rect
LayoutClustersPNG segments encoded bytes — the form a region's crop has.
type Reference ¶
type Reference struct {
From string `json:"from"`
Page int `json:"page,omitempty"`
To string `json:"to"`
Excerpt string `json:"excerpt"`
}
Reference is one fragment citing another document.
type Region ¶
type Region struct {
// ID addresses this region within its document: "p1" for a page's root,
// "p1.0" and "p1.0.2" for what descent found under it.
//
// Path-shaped rather than a digest because a diff of two reads has to stay
// readable, and because the path IS the ancestry — a hit can be reported as
// sheet → drawing interior → lot C corner without a lookup. It addresses a
// position in a RECORDED tree, not a piece of paper: a second read of the
// same sheet proposes different regions and renumbers. SHA256 is what says
// whether an id still points at the pixels the text came from.
ID string `json:"id,omitempty"`
Page int `json:"page"`
BBox Rect `json:"bbox"` // page coordinates
Rotation int `json:"rotation"`
Kind string `json:"kind,omitempty"` // overview|text-block|table|drawing|legend
// Verdict is this region's own claim about its frame — empty when it simply
// read. Kept on the record because "this could not be read as framed" is a
// fact about the READ, and a later reader deserves to know a region said so.
Verdict string `json:"verdict,omitempty"`
// Because is one line explaining a verdict or the parent's answer to it.
Because string `json:"because,omitempty"`
// Grid names this region's cell when it came from tileRegion rather than from
// something the model named — "row 2 of 4, column 3 of 4". Recorded because a
// cell's reading has to be read knowing it was one: a cell deliberately does
// NOT transcribe an item its neighbour holds more of.
Grid string `json:"grid,omitempty"`
// Filter is the repair applied to the crop BEFORE it was read, and it is part
// of the render: the digest covers the filtered bytes, so re-rendering
// without it produces a different image and VerifyRegionRender says so.
Filter RegionFilter `json:"filter,omitempty"`
Text string `json:"text,omitempty"`
Flags []string `json:"flags,omitempty"`
Depth int `json:"depth"`
// DPI is what the PAGE was rasterized at before this region was cropped out
// of it. Recorded per region rather than per document because a re-render
// that gets the crop right and the resolution wrong reproduces the geometry
// and not the detail, and detail is the entire subject here.
DPI int `json:"dpi,omitempty"`
// SHA256 digests the PNG this region was cropped to before it was read.
//
// The descent already computes it to detect a cycle; keeping it rather than
// discarding it is what makes attestation possible. A re-render that hashes
// to this is the image that produced Text. One that does not is a different
// artifact, and saying so is more useful than quietly showing it anyway.
//
// BEFORE the read, not after: it has to be, because it is also the cycle
// detector and the cycle has to be caught before the call is paid for. Where
// a context downscale intervened, Downscales says so and
// RerenderRegionAsSeen replays it.
SHA256 string `json:"sha256,omitempty"`
// Downscales is how many times the rendered crop had to be shrunk to fit the
// model's context before it could be read.
//
// The one part of what the model saw that the crop geometry does not capture:
// an overflowing page is re-rendered smaller MID-CALL (see maxContextShrinks),
// so a re-render that skips the shrink is sharper than the image the text
// came from. Recorded rather than prevented, because refusing to shrink would
// fail the page instead of reading it.
Downscales int `json:"downscales,omitempty"`
// TokensPerSqIn is what this region was actually seen at. Recorded because
// it is the one quality number available BEFORE any model call, and on its
// own it condemns a transcription taken at four.
TokensPerSqIn float64 `json:"tokens_per_sq_in"`
// Clusters are this region's measured label clusters, as fractions of it.
Clusters []Rect `json:"clusters,omitempty"`
// Computed marks a box derived here (grid tile or cluster), not proposed by
// a model. Such a box does not escalate.
Computed bool `json:"computed,omitempty"`
Children []*Region `json:"children,omitempty"`
}
Region is one node of the read: a piece of a page, at a rotation and scale, with what the model said about it.
Everything needed to RE-RENDER the region is on the node, because a human asked to attest that a document says what a fact quotes must be shown the image the words came from. When the text was read off a rotated, zoomed crop, handing the human the whole page hands them a different artifact — one where the passage is at the resolution that produced the failure in the first place.
func (*Region) AddFlag ¶
AddFlag records a condition on this region, once. Exported because a pass that MEASURES a recorded read — rather than producing one — belongs outside this package; see `raglit regions --backfill-damage`.
type RegionAbout ¶
type RegionAbout struct {
Depth int
// Damage is what DamageOf measured on these pixels — see damageSuffix. The
// model is shown a downsample and cannot tell a blurred crop from a small one.
Damage []string
// Grid is set when this region is one CELL of a computed grid rather than a
// block somebody named: "row 2 of 4, column 3 of 4". A cell has neighbours it
// cannot see and overlaps them, which changes what it should transcribe — see
// gridSuffix.
Grid string
// Escalation is set only on TURN 3: a parent being asked what to do about a
// child that reported its frame was broken. Present means "answer with an
// action, not a transcription" — see escalationSuffix.
Escalation string
// Doc, Page, RegionID and TokensPerSqIn identify WHICH crop of WHICH page of
// WHICH document this call is reading.
//
// They exist for the trace, and they are what makes a descent observable at
// all. Without them every call in a walk looks identical in the record —
// same model, same document, forty images — so a tree that spent its budget
// in the margins is indistinguishable from one that read the title block,
// and neither can be joined back to the region it came from. The region id
// is the join key to the recorded tree; tokens/in² is the number that says
// whether the crop could have been read at all.
Doc string
Page int
RegionID string
TokensPerSqIn float64
// FitsWhole means this region needs no subdivision: every pixel reaches the
// model in ONE canvas at this resolution. The root is then asked to
// TRANSCRIBE rather than to account for the sheet and propose where to look.
//
// Salience is only worth paying for when something will act on it. Measured
// on the bench's 94 sq in record of survey: the root prompt returned 4 of 7
// checks where a plain read of the same pixels returned 7 of 7 — it described
// the sheet and proposed regions nobody descended into, because there was
// nothing to descend into. The three it lost were recording numbers.
FitsWhole bool
}
RegionAbout is what a region is told about itself before it is read.
type RegionDoc ¶
type RegionDoc struct {
Doc string `json:"doc"` // base name, for a human opening the file
Version int `json:"version"`
Pages []RegionPage `json:"pages"`
}
RegionDoc is a document's region reads — one tree per page that was read this way. Not every page is: the descent is for a sheet whose text is physically too small to survive one look, and a letter page does not need it.
func ReadRegionDoc ¶
ReadRegionDoc loads the sidecar beside a document. A missing sidecar is not an error — most documents are not read this way — and reports false.
type RegionFilter ¶
type RegionFilter string
RegionFilter is a repair applied to a region's crop before it is read. A closed set on purpose: every one of these was measured on a page, and the measurements do not transfer to a filter nobody ran.
const ( // FilterNone is the crop as rendered. FilterNone RegionFilter = "" // FilterContrast is CLAHE at the setting that was measured to help — clip 2.0 // over an 8x8 tile grid. The STRONGER setting (clip 4.0, 16x16) recovered // nothing on the same page, so the parameters are not a detail. FilterContrast RegionFilter = "contrast" // FilterSharpen is a mild unsharp mask, sigma 1.0 amount 0.4. Also a measured // setting: at sigma 2.0 amount 0.8 the same page went back to its unfiltered // score. FilterSharpen RegionFilter = "sharpen" )
type RegionPage ¶
type RegionPage struct {
Page int `json:"page"`
// WidthIn, HeightIn are the sheet's physical size — the input that decides
// everything here, because the encoder's budget is per image and square
// inches are what resolution is spent on.
WidthIn float64 `json:"width_in,omitempty"`
HeightIn float64 `json:"height_in,omitempty"`
// DPI, PxW, PxH describe the page rasterization every region was cropped out
// of. Recorded together on purpose: dpi alone says what was ASKED for, and
// the pixel dimensions say what was produced. A renderer that disagrees about
// the second while agreeing about the first is caught here rather than
// discovered as an unexplained digest mismatch.
DPI int `json:"dpi"`
PxW int `json:"px_w,omitempty"`
PxH int `json:"px_h,omitempty"`
Root *Region `json:"root"`
// Text is this page's transcription, assembled from the tree. Byte for byte
// what belongs under `## Page N` in the transcription sidecar.
Text string `json:"text"`
// Spans map byte ranges of Text to the region that produced them.
//
// Out here rather than inline in the markdown, and that is the load-bearing
// choice: `## Page N` is a contract two separate consumers already parse, and
// a marker interleaved with the text would land INSIDE the quotations they
// match against. The transcription stays exactly what it was; the attribution
// lives in a file nothing else reads.
Spans []RegionSpan `json:"spans,omitempty"`
}
RegionPage is one page's read: the tree, the transcription assembled from it, and which region each part of that transcription came from.
func (RegionPage) RegionAt ¶
func (p RegionPage) RegionAt(off int) *Region
RegionAt reports the region a byte offset in the page's transcription came from, or nil if the offset falls between spans.
The offset is into RegionPage.Text. A consumer that located a quotation in the markdown sidecar has an offset into the FILE; subtract the start of the page's section — the text under `## Page N` is this text verbatim — or use RegionsForQuote, which does not need offsets at all.
func (RegionPage) RegionsForQuote ¶
func (p RegionPage) RegionsForQuote(quote string) []*Region
RegionsForQuote reports the regions whose text contains a quotation, deepest first.
Deepest first because a deeper region was read at a narrower field of view, which is the higher-resolution reading of the same words and therefore the one to show. A quotation matching only an interior node is a quotation nothing looked at closely — worth knowing, and the flags on that node will usually say why.
Matched on folded text — letters and digits, lowercased — because the words went through a vision model and hyphenation, spacing and punctuation are exactly what it renders inconsistently. A consumer that located the quote in the transcription folded it the same way to find it there.
type RegionProposal ¶
type RegionProposal struct {
X float64 `json:"x"`
Y float64 `json:"y"`
W float64 `json:"w"`
H float64 `json:"h"`
Rotation int `json:"rotation"`
Reason string `json:"reason"`
Kind string `json:"kind"`
// Filter is a repair the model is asking for on the SAME pixels. Named by the
// model rather than applied by measurement alone: the flags say what is wrong,
// and the model is the one looking at the region when it decides what to do
// about it. An unknown name is refused in route rather than ignored here.
Filter RegionFilter `json:"filter"`
// Margin is INCHES of extra paper the region wants on every side — a region
// REFINING its own frame rather than escalating.
//
// Escalate when the transform is fundamentally broken; refine when the fix
// needs no knowledge of the larger document. A word truncated at the region's
// own edge is the second kind: the region can see the cut, "half an inch more"
// is a complete statement of the remedy, and asking a parent that cannot see
// the region's six-point text to adjudicate it buys nothing.
//
// Inches, not a fraction, for the reason descentPadIn is a length: half an
// inch has to mean the same thing at every depth, and a fraction of a small
// region vanishes exactly where cuts are worst.
Margin float64 `json:"margin"`
}
RegionProposal is one sub-region the model thinks is worth a closer look.
type RegionReader ¶
type RegionReader struct {
// Ask returns the model's reading of one rendered region. Separated from the
// walk so the descent rules — budgets, routing, cycle detection — are
// testable without a model, which is most of what can go wrong here.
// About carries what this region knows about ITSELF and cannot see: what its
// pixels measured as, and whether it is one cell of a grid. Bundled rather
// than passed as more parameters, because everything here exists for the same
// reason — the model is looking at a crop and some of what governs its answer
// is not in the crop.
Ask func(ctx context.Context, img PageImage, about RegionAbout) (RegionReading, error)
// PageWIn, PageHIn are the sheet's physical size. The whole reason this
// exists is that a bigger sheet buys no extra tokens, so the physical size
// is the input that decides everything.
PageWIn, PageHIn float64
DPI int
// MaxDepth bounds descent. 0 means READ THE SHEET AND STOP, which is the
// default: the root reading is written to account for the whole sheet, so
// descending is for when a specific area needs the characters, not something
// every page should pay for. A survey descends into a title block because
// somebody asked about the certificate number, not because it was scanned.
MaxDepth int
MaxChildren int // per node, 0 → 8
MaxTransforms int // per region, 0 → 2
MaxCalls int // whole document, 0 → 40
// MaxEscalations bounds turn 3 for the whole document (0 → 4).
//
// Separate from MaxCalls because an escalation is the most expensive call in
// the walk — it is asked at the PARENT's resolution, which on an E-size sheet
// is four tokens per square inch, the worst per-token value anywhere in the
// descent. It should be rare; if it is not, the region proposals are bad and
// tiling is the cheaper answer.
MaxEscalations int
MinRegionIn float64
// Hint is what the caller is looking for, threaded into every prompt.
//
// The model proposing regions is looking at a view where the thing you want
// may be physically unresolvable — 6pt bearing labels on a sheet seen at 4
// tokens per square inch — so it cannot propose what it cannot see. A hint
// supplies the salience the view cannot: "read every bearing, distance and
// monument call on the drawing" costs nothing and beats any amount of extra
// budget, because budget alone only re-asks the same blind view.
Hint string
// Segment cuts along measured ink clusters instead of the geometric grid.
// OFF by default: measured worse on this corpus — see withTiles.
Segment bool
// Layout tunes cluster finding when Segment is set. Zero value derives its
// distances from DPI.
Layout LayoutOpts
// Tile turns on geometric subdivision of large low-resolution DRAWINGS
// instead of asking where to look. See tileRegion.
Tile bool
// Doc is the document these regions belong to, carried only so the trace can
// say which one. A walk reads one document; the record may hold many.
Doc string
// Trace, when non-nil, receives one line per TURN and per decision taken
// about it: what was read, at what resolution, what it proposed, what was
// routed to a descent and what to a transform, what escalated and what the
// parent said. A tree printed at the end says where the walk ARRIVED; this
// says what it did on the way, which is the part that explains a bad tree.
Trace io.Writer
// contains filtered or unexported fields
}
RegionReader walks a page as a tree of regions.
The model is asked, at every node, for two things: what this whole region IS, and which sub-regions merit a closer look. Descent recurses into the second while keeping the first, which is what makes a bad region set cost DETAIL rather than coverage.
func (*RegionReader) ReadInto ¶
func (rr *RegionReader) ReadInto(ctx context.Context, page image.Image, pageNo int, docPath, regionID string) (*Region, error)
ReadInto re-reads ONE recorded region and grafts the result back into the tree.
The alternative on offer was raising --depth, which re-runs the whole page and re-derives the same split — so a tree that spent its budget on the margins could never be sent back into the drawing. Everything needed was already recorded: the bbox, the rotation and the dpi identify the crop exactly, which is the property `raglit region` uses to put the same pixels back on screen.
The subtree is REPLACED rather than merged. A re-read is a new reading, and keeping the old children beside the new ones would leave two accounts of the same ground with nothing to say which was current.
type RegionReading ¶
type RegionReading struct {
// Transcription is the characters on the page, verbatim, markdown for
// STRUCTURE only. It feeds the full-text index, where someone types an exact
// recording number and expects this document back.
//
// Split from Description on 2026-08-06 because one field could not be both.
// Its doc comment used to admit as much — "a transcription for a text block,
// a summary for a drawing" — and every tile of a survey came back
// kind:drawing, so every tile returned a summary. Six tiles at 43-46
// tokens/in², above the readable baseline, and not one transcribed a
// character.
Transcription string `json:"transcription_markdown"`
// Description is what this IS, for the index that answers a searcher who
// does not know the words on the page.
Description string `json:"description"`
Kind string `json:"kind"`
// Regions are proposals in THIS region's coordinates, 0..1.
Regions []RegionProposal `json:"regions"`
// Repeated is set when the generation looped. On a page that fails the
// document; on a region it is a reason to descend.
Repeated bool `json:"-"`
// Downscales is how many times the reader shrank the image before the model
// could read it. Reported back so the region can record what was actually
// looked at rather than what was handed over.
Downscales int `json:"-"`
// Verdict is a region's claim about its OWN frame, when it has one. Empty
// means "read" — nothing to report.
//
// The line it draws: escalate when the transform is fundamentally broken,
// refine when the fix needs no knowledge of the larger document. A refinement
// is expressed as a proposal and handled here; an escalation is the one thing
// a region cannot do for itself, because the box and the rotation came from
// its parent.
Verdict string `json:"verdict"`
// Action is a PARENT's answer when it is asked what to do about a child that
// escalated. See escalationPrompt.
Action string `json:"action"`
// Because is one line for a human reading the record, on either.
Because string `json:"because"`
}
RegionReading is what the model returns for one region.
func ParseRegionReading ¶
func ParseRegionReading(s string) RegionReading
ParseRegionReading pulls the JSON object out of a model reply.
Tolerant of the wrapping models add — prose before, a fenced block, a trailing apology. A reply that carries no object at all is still usable as a description: losing the transcription because the JSON was malformed would throw away the part that matters.
type RegionSpan ¶
type RegionSpan struct {
Region string `json:"region"`
Start int `json:"start"`
End int `json:"end"`
}
RegionSpan attributes a byte range of a page's transcription to a region.
func RegionTranscript ¶
func RegionTranscript(root *Region) (string, []RegionSpan)
RegionTranscript assembles a page's transcription from its region tree, and says which region every part of it came from.
Parents before children in reading order — Flatten's order, which is also the order the tree prints in. EVERY node's text is emitted, not only the leaves: an interior node describes the whole of what is under it, and that is the design's coverage guarantee — a bad region set is supposed to cost detail rather than coverage. Dropping the overviews here would put the loss straight back.
It does duplicate: an overview and the leaves that refine it describe the same block twice. That is the honest artifact, and it is also why the spans matter. An overview read at four tokens per square inch is precisely where invented text lives, and a reader who can see WHICH region a sentence came from can see that it came from a low-resolution one — and ask for that crop.
Whether leaves should replace fragments outright is still open; see docs/design/hierarchical-regions.md. This assembles a transcription and decides nothing about fragmentation.
type Registration ¶
type Registration struct {
ProjectID string `json:"project_id"`
// Namespace prefixes every name below; it is the client's project setting
// for the daemon MCP.
Namespace string `json:"namespace"`
Base string `json:"base"`
// Private and Public are the caller's two layers — where the caller writes.
Private string `json:"private"`
Public string `json:"public"`
NewBase bool `json:"new_base"`
NewPrivate bool `json:"new_private"`
NewPublic bool `json:"new_public"`
}
Registration answers RegisterProject.
type Registry ¶
type Registry struct {
// contains filtered or unexported fields
}
Registry manages a SET of named indexes. Two storage layouts:
- single-home (OpenRegistry): all indexes live in one Home as sibling sqlite files sharing its originals/ + pages/ — "default" is index.sqlite, any other name is index-<name>.sqlite. This is the embedded/project layout.
- scoped (OpenScopedRegistry): each index is its OWN Home under <root>/indexes/<name>/ (own index.sqlite + originals/ + pages/), fully isolated. This is the DAEMON's multi-index layout — and the substrate for branch storage (a branch is a scoped store layered over a parent).
func OpenRegistry ¶
OpenRegistry prepares a single-home registry (all indexes as sqlite siblings in one home).
func OpenScopedRegistry ¶
OpenScopedRegistry prepares a registry whose indexes are each their own Home under <root>/indexes/<name>/ — the daemon's scoped, per-index storage.
func (*Registry) ACLEnabled ¶
ACLEnabled reports whether access control is on.
func (*Registry) Admin ¶
Admin is the principal that owns indexes without an acl.json; "" when access control is off.
func (*Registry) CanRead ¶
CanRead applies read(i) to the caller on ctx. Always true with access control off, and always true for the admin.
The admin used to read only what it owned or what was public, like anyone else. Indexes a project key creates (over /mcp/{project}, as para does) are owned by "project:<ns>" and private, so no browser identity — the admin's included — could see them: they were missing from the web UI with no error, and nothing could recover them but TransferOwnership. The admin runs the daemon and can read its files anyway; the ACL hiding them from it protected nothing (Carl 2026-11-04: "admin gets full access").
Implicit search scopes do not take this override (OwnReadable): the admin's own sessions search what they own, not every project's corpus.
func (*Registry) DeleteBranch ¶
func (*Registry) DeleteIndex ¶
DeleteBranch closes and removes a branch (its scoped storage), leaving the parent untouched. Errors if `name` is not a branch. DeleteIndex removes an index entirely — its store, its directory, its originals and its pages.
THE POOL IS NOT TOUCHED, and that is the whole point of the split. The pool cost LLM calls (extract/OCR/segment/embed) and is shared by every index; an index is the cheap part, rebuildable FROM the pool without a single model call. Deleting the expensive half to reclaim the cheap one would turn a cleanup into a re-embedding bill, so index lifetime is the caller's business and pool lifetime belongs to Pool.GC.
Refuses an index that a branch is layered over: a branch reads branch-over-parent, so removing the parent leaves it reading half a conversation with no way to say which half is missing.
func (*Registry) EnableACL ¶
EnableACL turns on per-index access control. admin owns every index that has no acl.json. Scoped registries only: single-home indexes share one directory, so there is nowhere to put a per-index ACL.
func (*Registry) ExclusiveHashes ¶ added in v0.4.1
ExclusiveHashes is the source-file hashes held by the indexes in names and by no other index on this daemon: what can leave the shared pool without costing another index its reuse, and without one project learning about another's files.
func (*Registry) Existing ¶
Existing opens an index only if its storage is already there, and never creates one.
Get creates on demand, which is right for ingest — POST /ingest to a new index name should make it — and WRONG for anything that walks a list of names. The worker loops do exactly that: they take reg.Names(), then Get each one in turn. Delete a branch while that pass is in flight and the loop reaches a name whose directory has just gone, Get recreates the whole index, and the thing that was deleted is back — empty, listed everywhere, and impossible to remove while the daemon runs.
Observed, not theorised: a branch deleted through the UI returned ok, and its index.sqlite was recreated within the same second by the round-robin.
Single-home mode has no such race — DeleteIndex and DeleteBranch both refuse without scoped storage — so there it is Get.
func (*Registry) FoldProject ¶
FoldProject folds the public layers of project id into its base, as the caller, who must own the base. dry reports without writing anything.
func (*Registry) ForkBranch ¶
ForkBranch creates a branch index `name` whose reads overlay `parent` (copy-on-write). Errors if the name is taken, the parent is unknown, or this is not a scoped registry.
func (*Registry) ForkBranchAs ¶
ForkBranchAs forks a branch the caller owns, private. The caller must be able to read the parent: a fork is a read of everything in it.
func (*Registry) HasIdentifier ¶ added in v0.4.0
HasIdentifier reports whether an identity model is configured: captions, tags and field extraction need one.
func (*Registry) IsAdmin ¶ added in v0.5.0
IsAdmin reports whether the caller on ctx is the daemon's admin. False with access control off: there is no admin then.
func (*Registry) ListBranches ¶
func (r *Registry) ListBranches() ([]BranchInfo, error)
ListBranches enumerates the scoped registry's branches with lineage + age + last-access + local doc count, sorted by name.
func (*Registry) Meta ¶ added in v0.5.0
Meta reads an index's metadata without the read check. False when the index does not exist; nothing is created.
func (*Registry) Names ¶
Names lists the indexes present on disk (always including "default"), plus any opened this session, sorted.
func (*Registry) OwnReadable ¶ added in v0.5.0
OwnReadable is Readable without the admin's override: what an implicit search scope (no index named, "all", @cwd) reads. The admin token is what clients on the daemon's own machine use, and an override there would make every local session search every project's corpus.
func (*Registry) ProjectBase ¶
ProjectBase returns the base index registered for a project id, "" if none.
func (*Registry) ProjectIDOf ¶
ProjectIDOf is the project id a base index was registered under, "" when it is not a project base.
func (*Registry) ProjectSavings ¶
ProjectSavings reports, for a project id, what a maintained base would save.
func (*Registry) Read ¶
Read opens an index the caller may read. An index the caller may not read, or that does not exist, is ErrNoIndex — and is never created, with access control on or off: a read of a wrong name used to leave an empty index behind, listed everywhere, which made the next choice of name harder.
func (*Registry) ReapSessionBranches ¶ added in v0.4.0
ReapSessionBranches deletes every session branch ("idx@session") whose last write is older than ttl, and returns their names. The last write is the newest modification time of the branch's storage files: every write lands in the index file or its write-ahead log.
func (*Registry) RegisterProject ¶
RegisterProject is register-or-layer, atomic per registry: the base for id (created public and empty when absent) and the caller's private and public layers over it (each created when absent). Repeating a registration returns the same names.
func (*Registry) Scoped ¶ added in v0.4.0
Scoped reports whether the registry keeps one directory per index (the daemon's layout) rather than a single home.
func (*Registry) ScopedFor ¶
ScopedFor is the indexes a search from dir reads when it names none: those whose scopes match dir, and the ambient global ones. Only indexes the caller may read as an owner (OwnReadable): the admin's override is for indexes it names.
func (*Registry) SetEmbedder ¶
SetEmbedder makes every index (already open or opened later) embed on ingest.
func (*Registry) SetIdentifier ¶
func (r *Registry) SetIdentifier(id *Identifier)
SetIdentifier makes every index (already open or opened later) caption its ingested documents (identity.go). nil disables it.
func (*Registry) SetImageEmbedder ¶
func (r *Registry) SetImageEmbedder(ie ImageEmbedder)
SetImageEmbedder makes every index (already open or opened later) embed FIGURE IMAGES on ingest (else figures embed their description). nil disables it.
func (*Registry) SetKeyStore ¶ added in v0.4.0
SetKeyStore gives the registry the keys whose principals its access checks resolve (keys.go).
func (*Registry) SetVisibility ¶
SetVisibility changes who may read an index. Owner only.
func (*Registry) Siblings ¶
Siblings lists the branches that share name's parent and that the caller may read — what a search on name fans out to besides name itself. Empty for an index that is not a branch. With access control off every sibling is readable.
func (*Registry) TransferOwnership ¶
TransferOwnership hands an index to another principal — how a maintainer service takes over a project's base. The current owner or the admin may do it; visibility is unchanged. Readers of a branch are unaffected; writers are now the new owner only.
type Relation ¶
type Relation string
Relation names how a pair of documents overlap. Deliberately four values and not a score: the caller is a triage flow that has to DO something different in each case, and "0.63" does not say which.
const ( // RelIdentical — the two files are byte-for-byte the same. Not a shingle // finding at all: it comes from documents.content_hash, which ingest already // records for its own re-index skip. // // Kept separate from RelDuplicate because it is a strictly stronger claim and // because the shingle scores UNDERSTATE it. Three byte-identical pairs exist on // this corpus and none scored 1.00 — the Graham declaration against its own // email attachment came back 0.93/0.92, because generic-text masking removes // the same passages from both sides and coverage is measured over what is left. // A triage flow told "0.93 similar" about two identical files will ask a human // to compare them; told "identical", it can act. RelIdentical Relation = "identical" // RelDuplicate — each document is substantially the whole of the other. RelDuplicate Relation = "duplicate" // RelProbeInside — the probe is contained in the match. The upload is an // instrument we already hold as part of a larger filing. RelProbeInside Relation = "probe-inside-match" // RelMatchInside — the match is contained in the probe. The upload is a larger // filing that swallows something we already hold standalone; the held copy is // probably the better evidence. RelMatchInside Relation = "match-inside-probe" // RelOverlap — they share passages without either containing the other. // Quotation, a shared exhibit, or a partial reproduction. RelOverlap Relation = "overlap" )
type RenderPolicy ¶
type RenderPolicy struct {
BaseDPI int `json:"base_dpi,omitempty"` // 0 → 200
MaxDPI int `json:"max_dpi,omitempty"` // 0 → 600
TargetGlyphPx int `json:"target_glyph_px,omitempty"` // 0 → 20
SmallTextGlyphPx int `json:"small_text_glyph_px,omitempty"` // 0 → 14
}
RenderPolicy is the automatic per-page resolution rule, which was four package constants and therefore the one thing here that genuinely required a rebuild to change.
The rule: measure the page's own glyph height with the cheap engine, and if it is below SmallTextGlyphPx re-render at whatever DPI brings it to TargetGlyphPx, capped at MaxDPI. Zero fields keep the measured defaults — 200/600/20/14 — which were chosen on this corpus and should not be moved casually. Raising TargetGlyphPx costs tokens on every small-text page.
type RetrySummary ¶
type RetrySummary struct {
N429, N5xx, NTransport int
Recovered, GaveUp int
// Waited is the sum of the delays the client scheduled — the wall-clock this
// job spent holding rather than working. The number that says whether an hour
// was spent waiting or generating, which one call's Elapsed cannot.
Waited time.Duration
// Worst is the longest single call's time in the retry loop.
Worst time.Duration
// LastStatus / LastBody are the most recent server answer and its own words.
// The body is the actionable half — "increase the physical batch size" —
// and is empty far more often than not.
LastStatus int
LastBody string
// MaxAhead is the deepest queue a fair-share proxy reported us standing in.
MaxAhead int
}
RetrySummary is one job's retry history, flattened.
func (RetrySummary) Detail ¶
func (s RetrySummary) Detail() string
Detail renders the summary for a stage row.
func (RetrySummary) Empty ¶
func (s RetrySummary) Empty() bool
Empty reports whether nothing worth recording happened, which is the normal case and must stay silent — a stage row on every healthy job is noise, and noise is why the real ones were never noticed.
type RetryTally ¶
type RetryTally struct {
// contains filtered or unexported fields
}
RetryTally accumulates llm.RetryEvents for one job.
Safe for concurrent use: the worker is single-threaded per index today, but a tally that silently under-counts the moment that changes is the same class of error it exists to catch.
func (*RetryTally) Observe ¶
func (t *RetryTally) Observe(e llm.RetryEvent)
Observe is the llm.Client.OnRetry hook.
func (*RetryTally) Take ¶
func (t *RetryTally) Take() RetrySummary
Take returns the accumulated summary and clears it, so the next job starts from zero. One call per job, at the end of it.
type Root ¶
type Root struct {
Path string `json:"path"`
Include []string `json:"include,omitempty"`
Ignore []string `json:"ignore,omitempty"`
}
Root is a source directory, optionally with its own include/ignore overriding the index's. In JSON it is EITHER a bare path string OR {path, include, ignore}.
func (*Root) UnmarshalJSON ¶
UnmarshalJSON accepts a bare string ("./src") or an object ({"path":"./gen","include":["*.go"]}).
type Run ¶
Run is one maximal span of identical text: A[AStart:AEnd) equals B[BStart:BEnd) byte for byte in folded form, and the two spans are the same length.
type Runner ¶ added in v0.5.0
type Runner struct {
// Stores is the set of indexes whose queues are worked. Required.
Stores StoreSet
// NewWorker builds the worker for one index. It is called once per index per
// runner, the first time that runner takes a job from the index. Required.
// A NewWorker that returns an error leaves the job pending for a later pass.
NewWorker func(*Store) (*Worker, error)
// Runners is how many ingest jobs may run at once, across every index.
// < 1 → DefaultIngestRunners.
Runners int
// IdentitySlots is each index's caption pipeline width (IdentityWorker.Slots).
// 0 → DefaultIdentitySlots.
IdentitySlots int
// Logf receives the per-job and per-index log lines. nil → log.Printf.
Logf func(format string, args ...any)
// Notef receives the drain's notices (a cut-off, a job that did not stop).
// nil → a "raglit: " line on stderr.
Notef func(format string, args ...any)
// contains filtered or unexported fields
}
Runner is the ingest scheduler: one pool of runners draining every index's queue, the identity (captioning) queue beside it, and the drain that winds both down before a stop or a re-exec.
Fill the fields, call Start once, and call Drain (or CutOff) to stop.
The identity queue drains only on a store whose identifier is set (Store.SetIdentifier); on any other it is skipped without a log line, because no identity model is a normal configuration. NewWorker sets it, but Runner calls NewWorker only when an ingest runner takes a job from that index. So set each store's identifier before Start (raglitd does, through Registry.SetIdentifier), or captions queued on an index with no ingest since Start wait until that index's next ingest.
func (*Runner) CutOff ¶ added in v0.5.0
func (r *Runner) CutOff()
CutOff cancels the running jobs now; they requeue themselves. It also ends the claiming. Safe before Start and more than once.
func (*Runner) Done ¶ added in v0.5.0
func (r *Runner) Done() <-chan struct{}
Done is closed when the ingest runners and the caption loop have both returned, including every caption drain the loop started. After it closes the Runner touches no store, so a caller may close them. A caption call in flight when the claim context ends is cancelled through that context, so the wait is short. nil before Start.
func (*Runner) Drain ¶ added in v0.5.0
Drain winds the ingest queue down before the process exits or re-execs.
Before it, a stop cut off whatever was running. The process exited under the jobs, the next start found their rows still 'running' with a dead pid, and reclaimOrphanedJobs marked them 'error' — correct for a crash, wrong for a restart that someone asked for. Every restart of a busy daemon left errors to retry by hand. The re-exec paths waited for an idle queue instead, while the queue went on claiming, so a busy queue could hold a re-exec back without end.
A drain stops claiming first, then gives the running jobs a grace period to finish, then cancels them. A job cancelled this way goes back to 'pending' (Worker.ProcessJob), so the next process runs it. Captions and re-embedding are not drained: an orphaned caption is requeued at start (ReclaimIdentityJobs), and re-embedding resumes from the fragments that still have no vector.
Drain reports whether every runner returned. Safe to call more than once and from more than one goroutine. Before Start it returns true at once.
func (*Runner) Start ¶ added in v0.5.0
Start begins working the queues and returns at once.
It returns two contexts, both under parent. claim ends when the Runner stops claiming (Drain, CutOff, or parent ending); work ends when running jobs are cancelled (a Drain past its grace, CutOff, or parent ending). They are two contexts so a stop can let running jobs finish. A caller may run its own background loops under claim and its own request-scoped work under work, so they stop with the queue.
Start must be called once.
type STT ¶
type STT struct {
// BaseURL is the endpoint's root — normally the same broker as the vision and
// embed models, since corrallm re-exports oidio's backends beside them.
BaseURL string
APIKey string
// Model is the oidio model id — "stt-diarize" for speaker-labelled segments,
// "stt" for plain text. Recorded as the reading's Producer, because a corpus
// outlives several models and a transcript whose author is unrecorded cannot
// be told from one a different model produced.
Model string
// Chan admits one transcription at a time per model, at a width learned from
// the server's own backpressure. nil → ungated, which is what a test wants
// and what a single-caller CLI can afford.
//
// Wired because the first real run proved it necessary and NOT wiring it was
// the bug modelchan.go's comment describes. Four hearings queued together,
// the daemon started all four, and corrallm answered two of them
// `429 {"capacity":2,"in_flight":2,"reason":"queue-timeout","retry_after":9}`
// — two 40-minute recordings failed after uploading, having done the work of
// getting there. A transcription is the most expensive request raglit makes
// and the least affordable one to throw away on a retryable refusal.
Chan *Channels
// HTTP overrides the client (tests). nil → a client with Timeout.
HTTP *http.Client
// GatewayBackoff overrides how long to wait after a 502/503/504. 0 →
// sttGatewayBackoff. Set short in tests, which must not sit out a real
// restart window to prove they retry.
GatewayBackoff time.Duration
// Timeout bounds one transcription. Diarization costs roughly a minute per
// minute of audio, so this is measured in hours, not the seconds an HTTP
// default assumes: the 20-minute hearing that prompted this would trip a
// 30-second timeout every time and look like a broken endpoint.
Timeout time.Duration
}
STT turns a recording into a reading. nil on a Worker → an audio job fails with a clear message, the same way a PDF job does without a vision model.
func (*STT) Transcribe ¶
Transcribe uploads the recording and returns oidio's reading.
name is the original filename; it rides along in the multipart part because oidio's decoder uses the extension as a hint and an upload called "file" with no extension is a container it has to guess at.
Gated and retried, because the upload is the expensive part. corrallm refuses a saturated backend with a 429 that carries how long to wait, and throwing a 40-minute recording away on one is discarding minutes of transfer for a condition that says, in the response body, when it will have passed.
type SearchFilter ¶ added in v0.5.0
type SearchFilter struct {
// PathPrefix keeps documents whose path starts with it (see pathPredicate).
PathPrefix string
// Tags keeps documents that carry EVERY one of these tags, each as a whole
// tag in the caller's tags (Store.SetTags), gen_content_tags or
// gen_role_tags — "paint" does not match "lead paint". A tag is compared the way writeIdentity stores it:
// lowercased, its whitespace collapsed (normalizeContentTag). An entry that
// normalizes to nothing is ignored.
Tags []string
// Fields keeps documents whose extracted fields (doc_fields.fields) have each
// key equal to its value. Only a JSON STRING value matches: a number or a
// boolean in the record does not equal its spelling here. A document with
// no extraction matches no field constraint.
Fields map[string]string
}
SearchFilter scopes a search to the documents that qualify, BEFORE anything is ranked. The constraints go into each layer's WHERE, next to the path scope, so a limit counts only qualifying fragments: a filtered query with limit 1 returns the best QUALIFYING fragment, not the best fragment of the corpus that the filter then throws away. Filtering after ranking would return fewer answers than were asked for, or none, and that reads as a thinner corpus.
Every constraint must hold (AND). The zero value constrains nothing.
type SegResult ¶
type SegResult struct {
ContinuesPrevious bool `json:"continues_previous"`
Fragments []Segment `json:"fragments"`
// Degraded names why this unit was NOT segmented by the model, when the fix
// loop ran out and run() fell back to the whole unit as one fragment. Empty
// on a real segmentation.
//
// It exists because that fallback returns a nil error and a plausible result:
// the document completes, the page count is right, and the only evidence that
// a page came back as an undivided block is a reason that used to be dropped
// on the floor. Not part of the model's answer — `json:"-"` so a model cannot
// claim it, and so unmarshalling a reply cannot clear it.
Degraded string `json:"-"`
}
SegResult is the model's structured segmentation of one unit.
type Segment ¶
type Segment struct {
Text string `json:"text"`
}
Segment is one fragment the model carved out of a unit.
func SplitOversized ¶
func SplitOversized(ctx context.Context, b *TokenBudget, frags []Segment) []Segment
SplitOversized bounds fragments the model returned, so a fragment can never be larger than the embedding model will take as one input.
The deterministic windower is already capped by ResolveFragParams. The LLM path was not: it asks for "roughly 400-800 words" and takes whatever comes back, so a dense page could yield a fragment no embedder would accept. The failure was silent in the worst way — the fragment reached the API, the API returned a 500, and the whole document failed with a message about batch sizes.
Splitting rather than TRUNCATING is the point. A truncated fragment is indexed, searchable, and quietly missing its tail, which is the same class of failure as a transcription that reads complete. A split fragment keeps every character; only the boundary is arbitrary.
Bounded in TOKENS by the embedder's own budget. It used to convert the token limit to characters at two characters per token and check the result with an estimate built on the same ratio — which is not a bound: a survey legal description is 1.16 characters per token, so a piece could pass both checks and still be half again over the real limit.
type Segmenter ¶
type Segmenter struct {
Client Chatter
MaxRetries int // JSON fix-loop attempts after the first try (default 2)
// Collection is what the corpus owner says about reading THIS collection
// (Store.IndexHint), appended to the segmentation prompt. Where a fragment
// boundary falls depends on knowing what the document IS: a lab panel's
// results are one unit and a work order's line items are another, and
// nothing on the page says so.
Collection string
// FragBudget is the ceiling a fragment must respect — the EMBEDDER's limit,
// in the tokens it counts in. nil → unchecked.
//
// Enforced by asking the model again rather than by cutting, for as long as
// the fix loop allows: a model re-splitting its own fragment cuts at a
// heading or a paragraph, and a splitter cuts at whatever falls near the
// budget.
//
// It replaces a MaxFragChars that nothing ever assigned, so this check has
// been returning "nothing is oversized" for every document ever indexed and
// the re-prompt it guards has never run once.
FragBudget *TokenBudget
// MaxTokens caps one unit's segmentation; 0 → maxTokensFor(the unit), i.e.
// scaled to the input, since the answer is the input re-emitted. See chat.go.
MaxTokens int
// contains filtered or unexported fields
}
Segmenter runs schema-validated LLM segmentation with a fix-loop.
func NewSegmenter ¶
NewSegmenter builds a Segmenter over a chat client (an *llm.Client).
type SimilarOpts ¶
type SimilarOpts struct {
Width int // shingle width; 0 = FoldWidth
Mod int // sample divisor; 0 = SampleMod
// MinChars is the matched-length floor for reporting, in folded characters.
// 0 means MinReportChars.
MinChars int
// MinCoverage is an ADDITIONAL block-coverage floor, in either direction. 0
// means no coverage floor — length is the primary gate (see MinReportChars).
MinCoverage float64
// Limit caps reported matches; 0 = unlimited.
Limit int
// Self keeps the probe's own path in the results. Off by default (a document
// is trivially identical to itself); on for verifying a rebuild.
Self bool
// GenericDF overrides GenericDocFreq. Exposed because the right cutoff is a
// property of the CORPUS — a corpus of 3,000 emails needs a higher one than a
// corpus of 200 recorded instruments — and because a person who suspects a
// duplicate was masked away needs to be able to raise it and look.
GenericDF int
// NoMask disables generic-text masking entirely. For diagnosing "why was this
// not reported": if it appears with masking off and not on, its text is corpus
// chrome and the finding was correct to suppress.
NoMask bool
}
SimilarOpts are the knobs `raglit similar` exposes. Zero values mean defaults, so a caller can pass an empty struct.
type SimilarReport ¶
type SimilarReport struct {
// Probe is the path compared. Indexed is whether it was read from the index
// (rather than extracted from a file on disk), because that decides whether
// the probe's own text has been through this index's OCR or a different run of
// it — two reads of one scan differ, and a lower score against an otherwise
// identical document is explained by that.
Probe string `json:"probe"`
Indexed bool `json:"indexed"`
// Source says where the probe's TEXT came from when it was not the index:
// "extracted" (read locally, no OCR) or "transcription:<file>" (a sidecar
// written by an earlier ingest). Recorded because a lower score against an
// otherwise identical document is usually explained by this — two OCR reads of
// one scan differ, and comparing an extraction against an OCR'd copy is not
// comparing like with like.
Source string `json:"source,omitempty"`
Pages int `json:"pages"`
Chars int `json:"chars"`
// Shingled is the probe's distinct shingle count. Zero means the probe holds
// less text than one shingle — a blank scan, or a page that is pure figure —
// and NO match can be reported. Distinguishing that from "nothing similar
// found" matters: the first is "we cannot tell", the second is "we checked".
Shingled int `json:"shingled"`
// Exact names indexed documents whose SOURCE BYTES are the same as the probe's,
// path -> title. Independent of every score here and stronger than all of them;
// present even when the probe's text could not be read at all.
Exact map[string]string `json:"exact,omitempty"`
// GenericChars is how much of the probe was masked as corpus-generic. A probe
// that is almost entirely generic (a bare email footer, a cover sheet) will
// report no matches, and this is what says why.
GenericChars int `json:"generic_chars,omitempty"`
Matches []DocMatch `json:"matches"`
// Recipe is the fold/shingle parameters the report was produced under, so a
// stored finding can be told from one computed under different settings.
Recipe string `json:"recipe"`
}
SimilarReport is what `raglit similar` returns. Shaped to be consumed programmatically first — the upload triage flow is the caller that matters, and it needs to branch on Relation and read page ranges, not parse a table.
type SketchStatus ¶
type SketchStatus struct {
Documents int `json:"documents"`
Sketched int `json:"sketched"`
Pages int `json:"pages"`
IndexRows int `json:"index_rows"`
Recipe string `json:"recipe"`
Unsketched []string `json:"unsketched,omitempty"`
StaleRecipe []string `json:"stale_recipe,omitempty"`
}
SketchStatus is what the index knows about its own sketch coverage. Reported rather than assumed, because an unsketched document is UNCHECKED and reporting it as clean is the one wrong answer here.
type Slice ¶
type Slice struct {
// ID is the child's stable identity. Facts cite it, so it must survive
// re-reads and must NOT be derived from the page range: a bundle re-paginated
// by a better read would otherwise rename every child that cites it.
ID string `json:"id"`
Parent string `json:"parent"`
// From and To are inclusive PARENT page numbers.
From int `json:"from"`
To int `json:"to"`
// Title is what this instrument is, in a person's words. The one field that
// makes a slice worth more than the page range it wraps.
Title string `json:"title,omitempty"`
Note string `json:"note,omitempty"`
By string `json:"by,omitempty"`
At string `json:"at,omitempty"`
}
Slice declares that a page range of Parent is a document in its own right.
func (Slice) ChildPath ¶
ChildPath is the derived document's path in the index: the parent's path with the range appended after '#'.
A '#' suffix rather than a new file, because there IS no new file and the path should say so. It also keeps provenance in front of anyone reading a search result — a hit in "scan.pdf#p3-6" is visibly a hit inside scan.pdf, which a separate filename would hide.
type SliceProposal ¶
type SliceProposal struct {
From int `json:"from"`
To int `json:"to"`
Form string `json:"form,omitempty"`
Title string `json:"title"`
Why string `json:"why"`
//
// Two one-page forms printed on one sheet cannot both be a page range, and
// the answer is not to pick one: a slice mints a document from a range, and
// the range would be a lie about half of it. Those are `seen-in` claims —
// observed inside, no identity minted, no page arithmetic — so the proposal
// says so rather than quietly proposing a slice that is wrong.
Shared []string `json:"shared_page_with,omitempty"`
}
SliceProposal is a candidate instrument inside a bundle.
func ProposeSlices ¶
func ProposeSlices(pages []PageText) []SliceProposal
ProposeSlices reads a bundle's pages and proposes the instruments in it.
func (SliceProposal) Sliceable ¶
func (p SliceProposal) Sliceable() bool
Sliceable reports whether this proposal should become a slice at all.
type StageLog ¶
type StageLog struct {
// contains filtered or unexported fields
}
StageLog records stages for one job. A nil *StageLog is a no-op, so the ingest pipeline can be driven with no job (CLI) or with one (the worker/daemon).
type StaleFields ¶
type StaleFields struct {
Path string `json:"path"`
Type string `json:"type"`
Reason FieldsStale `json:"reason"`
}
StaleFields is one extraction that is owed a re-run, and why.
type Status ¶
type Status struct {
Documents int `json:"documents"` // docs indexed
Fragments int `json:"fragments"` // fragments indexed
Done int `json:"done"` // completed jobs
Running int `json:"running"` // in-flight jobs
Pending int `json:"pending"` // queued jobs
Failed int `json:"failed"` // errored jobs
RatePerMin float64 `json:"rate_per_min"` // recent completion rate (jobs/min); 0 = unknown
Items []PendingItem `json:"items"` // running + pending, in processing order, with ETAs
// Identity is the CAPTIONING queue (identityqueue.go), which is separate work
// on the same endpoint: an ingest job is minutes of OCR over many pages, a
// caption is one bounded call. Reported here because "what is this index
// still owed" is one question, and a 400-document sweep outstanding is the
// kind of thing that should not need a second command to notice.
Identity IdentityQueueStatus `json:"identity"`
// Lanes is the queue BY SCHEDULING LANE — heavy (vision/OCR, one GPU slot)
// and light (everything else, several at once). Reported because "12 pending"
// answers nothing on its own: twelve scans is an afternoon and twelve
// markdown files is under a minute, and which one it is used to be
// unknowable without reading every URL.
Lanes map[string]LaneStatus `json:"lanes"`
// Prunable is how many documents' source files are gone (prune.go): out of
// search already, removed by `raglit prune`. Up to a minute stale.
Prunable int `json:"prunable,omitempty"`
}
Status is a snapshot of the index + ingest queue.
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store is a handle to one raglit index file. When opened via OpenHome it also knows a Home layout and copies ingested originals into it.
func OpenBranch ¶ added in v0.8.0
OpenBranch opens the branch at home branch over the index at home parent, creating the branch home and its branch.json when they do not exist. It is idempotent: a second call opens the same branch. It returns an error when an existing branch.json names a different parent, root or parent root, or when branch holds an index that is not a branch.
root and parentRoot are absolute directories: the worktree's and the main checkout's. With both set, the branch translates paths at the overlay boundary (see the top of worktree.go), so its index stores only the files that differ from the parent's, plus tombstones. With both empty, the branch does no translation. One set and the other empty is an error.
Translation is a string prefix match: OpenBranch does not resolve symlinks. root must be spelled as the paths the branch's documents are ingested under, and parentRoot as the paths the parent index stores. A different spelling (a symlinked directory, or git's resolved path from WorktreeOf against an index of unresolved paths) translates nothing and reports nothing: every worktree file counts as new, and the parent's documents show at their main-checkout paths. root may be inside parentRoot.
The parent index must exist. OpenBranch opens its own handle on it and does not wire the parent's own parent. The returned store owns that handle: its Close closes the parent too. Two branches of one parent each hold a handle.
func OpenBranchOn ¶ added in v0.8.1
OpenBranchOn is OpenBranch over parent, an index the caller has open already, instead of a handle of its own. parent must be the default index of its home (OpenHome); branch.json records that home, as OpenBranch does. The returned store does not own parent: its Close leaves parent open, and the caller must keep parent open until the branch is closed.
func OpenHome ¶
OpenHome opens the home's primary ("default") index. Use Open for a raw path or an in-memory test index; OpenIndex for a named index within the home.
func OpenIndex ¶
OpenIndex opens a NAMED index within a home (sharing its originals/pages), so one home can hold several indexes. "default" (or "") is the home's primary index.sqlite; any other name is index-<name>.sqlite. Created if absent. Ingesting a doc whose Path is a real file copies it into <home>/originals/.
func OpenIndexRO ¶
OpenIndexRO opens an index for READING only.
The daemon is the single writer for its indexes — that is what keeps one worker pool from fighting a CLI over the same file. A reporting command still needs to see that data, and SQLite in WAL mode allows any number of concurrent readers alongside the writer, so reading is safe where writing is not.
It does not create anything. An index that is not there is an error rather than a new empty database, because a command that silently invents one reports "nothing found" about a corpus it never opened.
func (*Store) ActiveReadings ¶
ActiveReadings returns the reading IN FORCE for each page of a document, by page number. Pages nobody has ruled on are absent — a document with no corrections returns an empty map, which is the common case and costs one indexed lookup.
func (*Store) AddNote ¶
AddNote records a note. page <= 0 files it against the whole document.
An empty body is refused rather than stored. A blank note is not a state anybody meant to reach — it is a stray click — and one sitting in a thread costs every later reader the moment it takes to work out that it says nothing.
func (*Store) AddPageReading ¶
func (s *Store) AddPageReading(ctx context.Context, r PageReading) error
AddPageReading appends a reading and makes it the active one.
Appends rather than replaces. The previous reading stays, marked inactive, so "what did this page say before somebody checked it" remains answerable — which is the whole reason a correction is an attestation rather than an edit.
func (*Store) AllowPatterns ¶
AllowPatterns is this index's allow list: a path matching one is exempt from the global ignores (EnqueueBuiltinIgnore), so an index that IS an iOS project can take `**/Pods/**`. The index's own ignore list still applies — both are this index's choices, and refusing is the safe way to settle them.
func (*Store) AttachmentDirFor ¶
Open opens (creating if needed) a raglit index at path. Use ":memory:" for a throwaway index (tests). foreign_keys is ON so a document delete cascades to its fragments; WAL keeps concurrent readers unblocked during ingest. AttachmentDirFor is where this index stores files extracted out of the archive at archivePath. Empty when the store has no home (":memory:", a bare Open), which is the one case extraction cannot run.
func (*Store) Attestations ¶
func (s *Store) Attestations(doc string) ([]AttestationRow, error)
Attestations returns one document's rows in log order.
func (*Store) AttestationsForUnit ¶
func (s *Store) AttestationsForUnit(doc, unit string) ([]AttestationRow, error)
AttestationsForUnit returns every ruling on one unit, in log order.
Every ruling, not the one in force: a retracted confirmation and the retraction are both facts about what happened, and which one governs is attest.Resolve's answer rather than SQL's.
func (*Store) AttestedDocs ¶
AttestedDocs reports how many entries each document has, for a review index that can say what has been looked at without opening every log.
Deliberately NOT a coverage percentage. How much of a document is reviewed is a question about the CURRENT reading, and this table holds verdicts against units that may no longer exist — see the schema note on orphans. A count of rulings presented as coverage would report a document as done after a re-read invalidated every one of them.
func (*Store) BackfillLanes ¶
BackfillLanes assigns a lane to pending rows that predate the column, and reports how many it set.
An index that has been running since before lanes existed has an EMPTY lane on every queued row, and no lane claims those — so without this, upgrading the daemon silently strands whatever was already in the queue. Pending only: a running row is owned, and a terminal one is never claimed again.
func (*Store) BackfillSourceStats ¶
BackfillSourceStats records the stat of every document whose file is unknown so far, and returns how many it filled. Files no longer on disk stay unknown.
func (*Store) BackfillVectors ¶
BackfillVectors embeds up to batch fragments that have no vector, then figure descriptions with no text-space vector, and returns how many it gave a vector. A fragment whose text already has a vector in this model's space (another fragment's, here or in a branch's parent) gets a copy of it rather than an embed call (R12a, reuse.go). 0 means there is nothing left (or no embedder, or the index is paused).
func (*Store) BranchDiff ¶
func (s *Store) BranchDiff() ([]BranchChange, error)
BranchDiff lists the documents this branch holds or deleted, against what its parent shows now (the parent's own overlay, if it is a branch too). Compared by content hash; a document with no hash on either side counts as "changed". A tombstone for a path the parent does not show is left out: it hides nothing. A non-branch store has no diff.
func (*Store) CanSearchByMeaning ¶
CanSearchByMeaning reports whether vector and hybrid search can run: an embedder is set. Without one they fail on every call, so a UI hides them.
func (*Store) CancelJob ¶
CancelJob removes a pending job from the queue. Only pending jobs can be canceled — a running job is mid-flight and done/error jobs are terminal.
func (*Store) CaptionProfile ¶ added in v0.4.1
CaptionProfile is the owner's choice for this index, "" when they made none.
func (*Store) ChatInputLimitTokens ¶
func (s *Store) ChatInputLimitTokens(ctx context.Context, p ContextProber, model string, configured int) int
ChatInputLimitTokens returns the largest single request this index's chat model accepts, in tokens, asking once and remembering the answer.
Asked before probed. A server that states its own n_ctx is one request for the real number; the probe is O(log N) requests for a lower bound, and on an endpoint that accepts anything it returns its own ceiling rather than a fact.
`configured` wins over both: an operator who knows the number should not wait, and a proxy that accepts a prompt its backend will later refuse cannot be measured from outside at all.
Zero means "unknown, do not cap" — the same reading the fragment ceiling gives it. An invented limit would cut pages that were fine, and the failure it would be guarding against announces itself in the endpoint's own words.
func (*Store) ClaimNext ¶
ClaimNext claims the oldest pending job in this index, whatever it is.
The exported claim the daemon's runners use. There was briefly a lane parameter here: the scheduler sorted jobs into `heavy` and `light` and gave each a slot count, to stop a slow scan holding up a text file. The problem was real and the cure was in the wrong place — what serialised them was not their KIND but one shared "the GPU admits one" budget covering three models on three cards. Admission is per model now (modelchan.go), so a transcription and an embedding take different channels and neither waits on the other. The lane column is still written and still reported; nothing schedules on it.
func (*Store) ClaimNextInLane ¶
ClaimNextInLane claims the oldest pending job in one scheduling lane, or (nil, nil) when that lane has nothing here.
The lane belongs in the CLAIM rather than in a filter afterwards. A dispatcher that claimed the oldest job and then put it back because it was the wrong kind would move the row's state twice per pass and race the other lane for it; a dispatcher that claimed and then held it would be the serial queue again under a new name.
func (*Store) ClearFailed ¶ added in v0.9.0
ClearFailed forgets the recorded ingest failures of paths (document paths, bare or file://; either spelling clears both), or of every path when none is given, so the next IndexTree queues those files again on the same bytes.
func (*Store) ClearSketches ¶
ClearSketches drops every stored sketch.
Needed for a recipe change, and it has to be a truncate rather than a per-document rebuild: a document deleted from the index since the last build keeps its shingle rows (the ON DELETE CASCADE only fires when the document row itself goes, and a re-ingest replaces rather than deletes), so those rows go on generating candidates that resolve to a path TruePages cannot read. That showed up as `similar` reporting matches against documents that were not there.
func (*Store) DedupeQueue ¶
DedupeQueue collapses duplicate pending jobs for the same url, keeping the oldest, and returns how many rows it removed.
For queues that grew before Enqueue deduped. A running job is left alone: a worker owns it, and deleting a row out from under one is how a job ends up stuck in `running` forever with nothing to finish it.
func (*Store) DeleteDocType ¶
DeleteDocType removes a type. The extractions made under it are LEFT ALONE: they are what documents said, and deleting a type is a statement about the index's vocabulary, not a retraction of everything read through it.
func (*Store) DeleteDocument ¶
DeleteDocument removes a document from THIS store and, when it's a branch, marks the path tombstoned so the parent's version stops showing through the overlay.
page_readings keys on the path with no foreign key, so the cascade from documents does not reach it; it is deleted here, in the same transaction. The whole removal is one transaction that takes the write lock first.
func (*Store) DeleteNote ¶
DeleteNote removes one note by id.
Hard delete, not a tombstone. A note is somebody's comment, not a ruling anything downstream replays, so there is nothing that has to be able to see that it once existed — which is the only reason attestations accumulate instead.
func (*Store) DocCosts ¶
DocCosts lists this store's OWN documents (not a parent's) with their cost. Documents with no content hash are left out: there is nothing to compare.
func (*Store) DocJobs ¶ added in v0.2.0
DocJobs returns the finished ingest jobs (done or error) for one document, oldest first: when it was read into the index, and from where. targets are the forms a job's URL can take for it (the absolute path, the display path); each matches bare and as a file:// URL.
A query of its own rather than a filter over Jobs(): that answers "the last N", and a document read a thousand jobs ago would have no history at all.
func (*Store) DocReadingHistory ¶
func (s *Store) DocReadingHistory(doc string) ([]PageReading, error)
DocReadingHistory returns every recorded reading of every page of one document, in page then arrival order.
Every version, not the active one. The whole reason readings accumulate as rows is that a correction must not erase what it replaced: the superseded text is what the index held when the document was cited, it is what a stale quotation elsewhere still matches, and "the OCR read 2015031104 for 201503110045" is itself evidence of how far a sheet can be trusted. A history that shows only the current answer throws away the thing the table exists for.
func (*Store) DocReview ¶
func (s *Store) DocReview(path string) (title string, pages []PageReview, err error)
DocReview returns a document's title and per-page OCR review (pages ≥ 1, in order). Page text is the concatenation of the fragments indexed for the page — i.e. what OCR/segmentation actually produced. Returns (‑, nil, nil) when the path is unknown.
func (*Store) DocText ¶
func (s *Store) DocText(exactPath string, from, to, maxChars int) (DocContent, error)
DocText overlays: the branch's copy if present, else (unless tombstoned) the parent's.
func (*Store) DocTypeByName ¶
DocTypeByName returns one registered type. An unregistered name is an error, not a zero value: extracting against a type nobody defined would produce a record whose shape is whatever the model felt like.
func (*Store) DocTypeRefs ¶
DocTypeRefs is the type list as the identity call is shown it: the names it may choose from, and one line each on how to recognise one.
func (*Store) DocumentAddedAt ¶ added in v0.2.0
DocumentAddedAt is when the document was last written into the index, in Unix nanoseconds; 0 when it is not in the index. Re-ingest updates it.
func (*Store) DocumentFields ¶
DocumentFields returns what has been extracted for a document. A document with no extraction returns the zero value and no error — that is a state, not a failure. An unknown path IS an error.
func (*Store) DocumentHash ¶
DocumentHash returns a document's stored source-content hash, or "" if the doc is unknown or has none. The hash is the sha256 hex of the raw source bytes, not of the extracted text. The worker uses it to skip re-ingesting unchanged content (dedup work).
A branch with a root (OpenBranch) reads through to its parent for a path it has no row for and did not tombstone, so a worktree file whose bytes equal the main checkout's is skipped rather than copied. A branch with no root does not.
func (*Store) DocumentIdentity ¶
func (s *Store) DocumentIdentity(path string) (DocIdentity, error)
DocumentIdentity returns what is recorded about a document. A document with no caption yet returns the zero value and no error — that is a state, not a failure. An unknown path IS an error.
func (*Store) DocumentPaths ¶
DocumentPaths lists every document path in the index, branch overlay applied: the branch's own documents, then the parent's that the branch does not shadow.
func (*Store) Documents ¶
func (s *Store) Documents() ([]DocSummary, error)
Documents overlays: branch docs, then parent docs not shadowed.
func (*Store) DocumentsMissingFields ¶
DocumentsMissingFields lists documents that resolved as a registered type but have no extraction yet — the work list for `raglit fields`.
func (*Store) DocumentsMissingIdentity ¶
DocumentsMissingIdentity lists the paths of indexed documents with no caption yet, oldest first — the work list for `raglit identify`. A corpus indexed before identity existed is entirely this list.
func (*Store) DocumentsMissingTags ¶
DocumentsMissingTags lists documents that HAVE a caption but no tags — a corpus captioned before tags existed. Its own selector rather than a flag on the one above, because the two are different work: one needs a caption written, the other needs a caption LEFT ALONE and two columns filled in.
func (*Store) DropVectors ¶ added in v0.6.0
DropVectors deletes every vector the text embedder made, so a forced re-embed (BackfillVectors) rebuilds them all. Fragments, their FTS rows and the model and dim records are untouched. Returns the number of vectors deleted.
Nothing made before the drop comes back: an ingest or backfill in flight writes no vectors (vecGen), and a branch stops reusing its parent's vectors until SyncEmbedModel next drops them for another model or length. The branch's re-embed sends every fragment to the embedder. The parent itself is untouched: drop its vectors separately to re-embed it too.
func (*Store) EmbedLimitChars ¶
EmbedLimitChars returns the largest single input this index's embed model accepts, probing once and remembering the answer.
Discovered rather than configured, because it is a fact about the endpoint. It was previously left at zero here — meaning "unknown", which ResolveFragParams reads as "no cap" — so fragments were sized by taste (9000 characters) with nothing checking them against what the model would take. The first symptom was a document failing with a 500 about batch sizes.
`configured` wins when set: an operator who knows the number should not have to wait for a probe, and an endpoint that silently truncates cannot be probed at all.
func (*Store) EmbedLimitTokens ¶
func (s *Store) EmbedLimitTokens(ctx context.Context, p ContextProber, model string, configured int) int
EmbedLimitTokens returns what the embed model accepts as one input, in tokens.
The limit that actually refused documents. EmbedLimitChars answers the same question in characters, converted from this number at two characters per token — 16128 for an 8192-token embedder. A scanned court brief reaches 10240 tokens inside 16128 characters, so the fragment passed every check raglit had and was refused by the endpoint:
input (10240 tokens) is too large to process. increase the physical batch size (current batch size: 8192)
which failed the whole document at the embed stage, after its OCR and its segmentation had already been paid for.
Note this is a DIFFERENT model from the chat one, with a different tokenizer and a much smaller window — 8192 against 180224 on this endpoint. Asking the chat model's limit and applying it here would be off by twenty-two times.
func (*Store) EmbedModel ¶
EmbedModel is the model this index's vectors are recorded as coming from, "" when no record exists yet.
func (*Store) Enqueue ¶
Enqueue adds a pending ingest job for url and returns its id. It does not fetch or index anything — a worker does that later (lazy).
A url already waiting is NOT queued again; the existing job's id is returned. Without this the queue inflates every time anything requeues — a watch fires, a batch is re-run, a sweep is retried — and the work is done two and three times over. Measured on the ardley corpus before the check: 162 queued rows for 115 distinct documents, one court memorandum queued FIVE times, roughly 29% of a nine-hour backlog that was pure repetition.
Deduped against pending and running only, never against done. Re-indexing a document that has changed is legitimate and must stay possible; what is never useful is the same file sitting in the queue twice at once.
func (*Store) EnqueueFields ¶
EnqueueFields queues one document for the SCHEMA ask: fill out the fields of the document type it resolved as.
func (*Store) EnqueueFieldsFor ¶
EnqueueFieldsFor queues every given path for the schema ask.
func (*Store) EnqueueFresh ¶
EnqueueFresh is Enqueue with an explicit re-read flag. A fresh job never dedupes against a pending one: the caller is asking for work the queued job would specifically NOT do.
func (*Store) EnqueueIdentity ¶
EnqueueIdentity queues one document for captioning. Idempotent per path: an existing pending/running row is left alone (re-queueing a document already in flight buys a duplicate model call and nothing else), and a TERMINAL row — done, skipped or error — is reset to pending. Returns false when an in-flight job already covers it.
Reviving a skipped row is deliberate and is the reason this takes a path at all: the bulk enqueue leaves skips alone (nothing to read is not work), but a document named explicitly, or a --force sweep, is somebody saying that has changed — which it does the moment a document is re-OCR'd.
func (*Store) EnqueueIdentityFor ¶
EnqueueIdentityFor queues every given path, reporting how many rows it added or revived. Paths already in flight are skipped, not duplicated.
func (*Store) EnqueueMany ¶ added in v0.8.0
func (s *Store) EnqueueMany(items []EnqueueItem) ([]EnqueueResult, error)
EnqueueMany enqueues every item in ONE write transaction and returns one result per item, in order. Each item behaves as EnqueueFresh does: empty, generated-sidecar and ignored urls are refused per item (Err set, batch continues), and an item already covered by a pending/running job returns that job's id — including a job inserted earlier in the same batch.
One transaction because one Enqueue per file costs a commit (fsync) each: ~826 files took 5-12 s. BEGIN IMMEDIATE takes the write lock up front, so the batch waits under busy_timeout like any writer and cannot hit the SQLITE_BUSY_SNAPSHOT upgrade failure EnqueueFresh documents. The returned error is only for a transaction or database failure; then nothing is committed.
func (*Store) EnqueueMissingFields ¶
EnqueueMissingFields queues every document that resolved as a registered type and has no extraction yet — or, with force, every document that resolved as one. A person's extraction is still refused downstream.
func (*Store) EnqueueMissingIdentities ¶
EnqueueMissingIdentities queues every document with no caption yet — or, with force, every document. Returns how many were queued.
func (*Store) EnqueueMissingTags ¶
EnqueueMissingTags queues every CAPTIONED document that has no tags — or, with force, every captioned document. The backfill for a corpus captioned before tags existed.
func (*Store) EnqueueTags ¶
EnqueueTags queues one document for the TAGS-ONLY ask. Same queue, same rows, same resumability — a backfill of hundreds is the same shape of work as a captioning sweep and has no business being a second mechanism.
func (*Store) EnqueueTagsFor ¶
EnqueueTagsFor queues every given path for the tags-only ask.
func (*Store) ExportDoc ¶
ExportDoc reads a freshly-ingested document back out of an index as a PooledDoc (fragments + vectors + page provenance, page Image = absolute source path), for caching in the pool.
func (*Store) ExtractFields ¶
ExtractFields fills out a document's type schema and stores the result.
Returns ErrIdentityKept when an extraction already exists (unless forced), and when a person wrote it — a person's extraction is never regenerated, forced or not.
func (*Store) ExtractableMissing ¶
ExtractableMissing is every document that OWES an extraction: one that resolved as a registered type and has none, and one whose extraction answers a schema or a transcript that has since moved. A person's is never owed again.
func (*Store) FieldsCoverage ¶
func (s *Store) FieldsCoverage() ([]FieldsCoverage, error)
FieldsCoverage reports extraction coverage per type, most-resolved first.
func (*Store) FieldsStaleness ¶
func (s *Store) FieldsStaleness() ([]StaleFields, error)
FieldsStaleness lists every MACHINE extraction that is not current, with the reason. A person's is never stale: it is theirs, and a schema edit does not make what they wrote wrong.
Documents that have never been extracted are NOT here — they are DocumentsMissingFields, which is a different report and a different sentence to say to somebody.
func (*Store) FindPathByHash ¶
FindPathByHash returns the path of a DIFFERENT document row that stores the same content hash, or ("", nil) when there is no such row.
It is the lookup reconciliation uses when a job's source path no longer resolves: the bytes may simply have MOVED, and a document already indexed under the new path carries the hash that identifies them. Excluding the caller's own path is what keeps a plain re-ingest (same path, same content) from finding itself and "moving" onto itself.
func (*Store) ForgetJob ¶
ForgetJob deletes a terminal job row and its stages.
Not a cancel and not a retry: this is "that attempt is not coming back and I do not want to be told about it again". The health report already drops rows whose file is gone, because that is decidable; this is for the ones only a person can call — a remote URL that is genuinely dead, a failure nobody is going to chase.
Refused on a live job. A pending job has a cancel and a running one is mid-flight; deleting either loses work that is still in front of somebody.
It removes evidence, and that is the point of restricting it to a person's explicit act. What the row records is an ATTEMPT, not a fact about the corpus — nothing cites it, and a log nobody can prune is one nobody reads.
func (*Store) FragmentCount ¶
FragmentCount is the number of fragments, for the cost of a re-embed.
func (*Store) FragmentExists ¶ added in v0.7.0
FragmentExists reports whether a fragment with this content id is in the index. On a branch it overlays the parent the way search does: the parent's fragment counts unless its document is shadowed (rewritten or tombstoned in the branch).
func (*Store) HybridSearch ¶
HybridSearch is HybridSearchPath with no path constraint.
func (*Store) HybridSearchFiltered ¶ added in v0.5.0
func (s *Store) HybridSearchFiltered(ctx context.Context, query string, f SearchFilter, limit int) ([]Hit, error)
HybridSearchFiltered is HybridSearchPath constrained by a SearchFilter. Both sides filter before they rank, so the fusion only ever sees qualifying fragments; RRF itself is unchanged.
func (*Store) HybridSearchPath ¶
func (s *Store) HybridSearchPath(ctx context.Context, query, pathPrefix string, limit int) ([]Hit, error)
HybridSearchPath fuses BM25 and vector rankings with Reciprocal Rank Fusion (RRF) — the standard, score-scale-agnostic combiner: a fragment's fused score is the sum over each ranked list of 1/(rrfK + rank). It over-fetches from each side, so a fragment strong on either signal surfaces. Both sides honor the optional path subtree scope. Requires an embedder. Returns up to limit fragments, best fused first.
func (*Store) IdenticalGroups ¶
IdenticalGroups returns every set of indexed documents whose SOURCE BYTES are the same, as groups of paths, each group sorted and the groups sorted by their first path.
This is the one relation in the corpus that needs no shingles, no thresholds and no judgement. Two files with one sha256 are the same document — not similar to it, not probably it — so the answer does not depend on any of the tuning the rest of this file is careful about, and it is available on an index that has never been sketched.
Grouped rather than paired because a document held three times is ONE finding. Emitting three pairs invites a reader to rule on them separately and reach two different answers about one set of bytes.
func (*Store) IdentifyDocument ¶
IdentifyDocument captions one already-indexed document: read its indexed text, ask the model what it is, store it. This is the re-runnable half — a corpus of 406 documents is not going to be re-OCR'd to get captions, and a caption is worth having on documents indexed long before this existed.
force replaces a MACHINE caption. It does not replace a PERSON's: someone who corrected a caption did so because the model's was wrong, and re-running the same model on the same text produces the same wrong answer. Changing a person's caption is done by recording another one.
func (*Store) Identities ¶
func (s *Store) Identities() ([]IdentityStatus, error)
Identities lists every document's identity, newest first, including the ones with none — a corpus's captioning coverage is exactly as interesting as the captions.
func (*Store) IdentityJobs ¶
func (s *Store) IdentityJobs(state string, limit int) ([]IdentityJob, error)
IdentityJobs lists the queue, newest first, optionally only one state.
func (*Store) IdentityQueue ¶
func (s *Store) IdentityQueue() (IdentityQueueStatus, error)
IdentityQueue reports the queue's counts.
func (*Store) IdentityText ¶
IdentityText is the text a caption is written from: the reading IN FORCE for each page, not the machine's first attempt at it.
The distinction is the whole of why this function exists. Fragments hold what OCR produced, and they are deliberately NOT rewritten when a person corrects a page — re-fragmenting would move every offset and invalidate every citation already taken. So a correction lives in `page_readings` as the active reading, and anything that wants what the document SAYS has to go there.
Captioning is exactly such a consumer, and the case that proves it is the one this was found on: a disputed record of survey whose corrected page fixed the surveyor's name and certificate number. Those are the facts a caption states. Written from fragments, the caption would have asserted, in a machine's voice and at the top of a document list, the very reading a person had already ruled wrong.
Pages with no ruling fall back to the indexed text, so a corpus nobody has corrected — the overwhelming majority — is unaffected and pays one indexed lookup.
func (*Store) IgnorePatterns ¶
IgnorePatterns is this index's own ignore list: globs a path must not match to be enqueued. Matched against the absolute path, so a directory anywhere is `**/name/**`; a glob with no "/" matches the file name.
func (*Store) IgnoredBy ¶
IgnoredBy returns the pattern that refuses path at Enqueue, "" when none does. The index's own ignore list first; then the global list, unless the index's allow list exempts the path.
func (*Store) ImportVerifiedTranscripts ¶
func (s *Store) ImportVerifiedTranscripts(dryRun bool, ruledBy string) ([]TranscriptMatch, error)
ImportVerifiedTranscripts finds verified transcripts ON DISK and records them as ATTESTED readings of the recordings they transcribe — so search collapses to one, and the one it keeps is the transcript a person ruled on.
From DISK, not from the index, and that ordering is the point. A transcript beside a recording should not be a document at all: it is a reading, and once the corpus stops indexing these sidecars there is nothing in the index to walk. An importer that could only see what was already indexed would stop working the moment its job was done.
Candidates come from the READINGS, not from the documents: a recording with a machine reading is the only thing a transcript can be adopted onto, and the digest to join on lives there. Transcripts are looked for in the directories those recordings came from, which is where every producer puts them.
dryRun reports the matches and records nothing.
func (*Store) IndexAbout ¶
IndexAbout is the stored paragraph and whether it is behind the corpus. Empty when none has been written; writing one is WriteIndexAbout.
func (*Store) IndexDigest ¶
func (s *Store) IndexDigest() (IndexDigest, error)
IndexDigest counts the whole index, keeping the ten most common content tags and every kind and role — the closed vocabularies are short enough to show entire, and a truncated one reads as an absence.
func (*Store) IndexDigestFor ¶
func (s *Store) IndexDigestFor(pathPrefix string, topN int) (IndexDigest, error)
IndexDigestFor counts what the index holds under a path prefix — empty for the whole index. topN caps each list; 0 means "all of them".
func (*Store) IndexHint ¶
IndexHint is what a person tells the models about THIS corpus, in their own words: how to decode it, which of its ambiguities resolve which way, what matters in it.
It exists because the model is answering a general question about a specific corpus, and the corpus is the half it cannot see. "RO" on a garage's paperwork is a repair order, not "received"; a survey's marginal figures are bearings, not measurements; a scanned carbon copy's second column is the customer's, not a duplicate. None of that is inferable from one page, all of it is obvious to whoever owns the corpus, and every prompt that reads a document is worse for not having it.
So it travels the whole way down: the page transcription, the segmentation, the identity, and the field extraction. It is part of the READING recipe for that reason — a changed hint changes what a page says, and pooled work read under the old one must not be replayed under the new.
func (*Store) IndexStatus ¶
IndexStatus reports queue progress: counts, a recent processing rate, and a per-item ETA (queue position × recent average job duration). ETA/rate are 0 until at least one job has completed (no basis to estimate).
func (*Store) Ingest ¶
Ingest upserts a document and (re)indexes its fragments in one transaction. Re-ingesting the same Path is idempotent: the doc's old fragments are dropped and replaced, so re-running an index over a changed file converges rather than duplicating. Empty-text fragments are skipped.
func (*Store) IngestPDF ¶
IngestPDF pagifies an image/scanned PDF and indexes it via LLM segmentation: the vision model (ocr's client) reads each page image and carves coherent fragments, with open fragments stitched across page boundaries and vectors embedded concurrently. Page images are written under the home's pages/ dir when the store has a home, else a temp dir. Returns the number of fragments indexed.
func (*Store) IngestPooled ¶
func (s *Store) IngestPooled(ctx context.Context, docPath, title string, doc PooledDoc, poolFileDir string) (int, error)
IngestPooled writes a cached PooledDoc into this index — the CHEAP reuse path (no LLM/OCR/embed): fragments + their cached vectors, and page provenance with each image copied from poolFileDir into this index's pages/. Atomic (commitDoc).
func (*Store) Job ¶
Job returns one job by id, whatever its state and however old it is.
Jobs() answers "the last N", which is the wrong shape for a LINK. The health report cites a job by id — an ingest that had to fight the endpoint, a failure from a batch weeks ago — and a reader following that citation lands on a window of the newest rows that does not contain it. Measured on the fence-dispute index: the cited job was 927, the newest was 1467, and the page rendered two hundred rows none of which was the one the link was about.
func (*Store) JobStages ¶
JobStages returns a job's stages in the order they were RECORDED — see the ORDER BY in ListJobStages for why that is not the order by seq.
func (*Store) Jobs ¶
Jobs lists ingest jobs, newest first, via a metaquery Builder: state filters to one lifecycle state (pending|running|done|error); "" or "all" returns every state. limit ≤ 0 → 100.
func (*Store) LearnLimitFromError ¶
LearnLimitFromError records a limit the endpoint stated in a rejection.
Better than the table and better than the probe, because it is not an inference: the server named its own number. Called on an embed failure so the next ingest is sized correctly even if the model was unknown or was swapped under a stored value.
Stored in tokens as well as characters. The token number is the one the fragment budget enforces, and it is the one that was wrong: probed as the model's context window (8192) on a server whose physical batch was 4096, so a CSV fragment of 5250 tokens passed every check and was refused (2026-10-22). A stated number only ever LOWERS the token limit; the server's word for one input never makes a smaller stored limit larger.
func (*Store) LearnedEmbedLimitTokens ¶
LearnedEmbedLimitTokens is the stored per-input token limit for model, 0 when none is stored.
func (*Store) LegacyAttachmentDirFor ¶
LegacyAttachmentDirFor is the corpus sidecar an archive's attachments used to be written to, for readers that must work before the migration has run.
func (*Store) MarkForReingest ¶
MarkForReingest clears a document's content hash so the next ingest redoes it.
The hash is the dedup lever, so clearing it is the least destructive way to force a re-read: the document row, its fragments and its OCR page cache all survive, and the pages already transcribed are not read again.
func (*Store) MatchDocuments ¶
MatchDocuments overlays: branch matches, then parent matches not shadowed.
func (*Store) MergeTags ¶
MergeTags collapses From into To across every document in this index, re-indexing each changed document's identity fragment so the searchable text and the columns cannot disagree.
Content tags only. Roles are a closed vocabulary and cannot drift; a role outside it is a validation failure to fix at the source, not a tag to merge.
The caption is untouched, and so is its authorship: a person's identity keeps gen_source='person'. Merging a tag is not overruling anybody's caption.
func (*Store) MigrateExtractedAttachments ¶
func (s *Store) MigrateExtractedAttachments(dryRun bool) ([]AttachmentMove, error)
MigrateExtractedAttachments moves this index's corpus-side attachment directories into raglit's storage, rewriting the document rows to match.
dryRun reports what it would do and touches nothing — the only responsible default for something that renames files in an evidence tree.
func (*Store) MissingCount ¶
MissingCount is len(MissingDocuments), cached for missingTTL; -1 when the count failed. Prune resets it.
func (*Store) MissingDocuments ¶
MissingDocuments is this layer's documents whose local source file is gone, among those read from a file (fromFileSQL).
func (*Store) MissingVectors ¶
MissingVectors counts fragments with no vector. Only the document's own words: identity and fields fragments are about a document and are never embedded (SQLOwnWordsF).
func (*Store) NewStageLog ¶
NewStageLog returns a recorder bound to a job. Returns nil for jobID ≤ 0 so callers without a job (the CLI IngestPDF path) record nothing.
func (*Store) Notes ¶
Notes returns every note on a document, oldest first.
Oldest first because a note thread is read as it was written: the later note usually answers the earlier one, and reversing that makes a correction arrive before the thing it corrects.
func (*Store) OversizedDocs ¶
OversizedDocs lists documents holding a fragment larger than limit, with the worst offender's size. These are the documents whose fragments were cut to a standard the embed model does not accept.
func (*Store) PageImagePath ¶
PageImagePath returns the absolute path of a document page's saved image, or "" if there is none. The daemon validates the path is under the home's pages/ dir before serving it. A stored path from a home that has since been copied or moved is mapped back under this home's pages/ (rerootPageImage).
func (*Store) PageLayoutFor ¶
func (s *Store) PageLayoutFor(docPath string, page int) (*PageLayout, error)
PageLayoutFor assembles one page's layout view.
Raw text comes from the OCR cache, which is keyed by the image's sha256 and therefore survives re-ingests, model changes and the flattener — it is the only place the model's original output is guaranteed to still exist.
func (*Store) PageReadings ¶
PageReadings returns every recorded version of one page, oldest first.
func (*Store) PagesRoot ¶
PagesRoot is the home's pages directory (absolute), for the daemon to bound page-image serving to the home. Empty for a no-home store.
func (*Store) PagesWithLayout ¶
PagesWithLayout reports which of a document's pages have layout blocks, so a page LIST can decide whether to offer a Layout tab without fetching each page's blocks. One query plus a memoised hash per page.
func (*Store) Paused ¶
Paused reports whether this index runs no work. Jobs still queue while it is paused, and run when it is resumed.
func (*Store) Problems ¶
Problems reports everything wrong with this index, worst first.
Ordered by consequence, not by table: a document that cannot be found matters more than one that was retried into place. A caller rendering only the first screen still sees the things that lose data.
func (*Store) ProposeDocType ¶
func (s *Store) ProposeDocType(ctx context.Context, name string, goldPaths []string) (DocType, error)
ProposeDocType reads gold documents and proposes the type: how to recognise one, how to read it, and its fields.
The authoring step, and the reason document types are practical at all. Writing a JSON Schema by hand for a form you are looking at is exactly the work a model does well, and doing it from SEVERAL examples is what stops the schema being a transcription of one document's quirks.
The proposal is returned, not stored — a schema nobody read before it started filling in records is a schema nobody will trust afterwards.
func (*Store) Prune ¶
func (s *Store) Prune(dry, deleteOriginals bool) (PruneResult, error)
Prune removes every document whose local source file is gone, with its fragments, vectors, pages, fields, caption rows and page images, and pending ingest jobs on those files, and the page images no document names. dry reports the same and removes nothing. deleteOriginals removes the dangling copies under originals/, dry or not: they are reported always and removed only when asked.
Dirs groups by the top-most directory that is gone, so a deleted worktree is one line however deep its tree was; a file deleted from a directory that still exists is counted under that directory.
func (*Store) Purge ¶
func (s *Store) Purge(pattern string, dry bool) (PurgeResult, error)
Purge removes every document whose path matches pattern, with its fragments, vectors, pages, fields and page images, and drops pending ingest and caption jobs on matching paths. dry reports the same counts and removes nothing. Files on the user's disk are untouched; an upload's file goes with its document (RemoveDocument). Matched like an ignore pattern: against the absolute path, or the file name for a glob with no "/".
func (*Store) PurgeDocPageCache ¶
PurgeDocPageCache drops the cached OCR for every page of one document and returns how many entries it removed.
It works by rendering the document exactly as ingest does and computing the same keys, which is sound because the render is deterministic at a fixed dpi (measured, not assumed). A page whose render has changed — a different poppler, a re-exported PDF — simply will not match, and the miss is harmless: the page was going to be re-read anyway.
func (*Store) PutAttestations ¶
PutAttestations replaces this document's projected rows with the log as it now stands.
Delete-then-insert rather than an incremental append, because the log is authoritative and append-only: re-projecting the whole of it is idempotent and cannot drift, while tracking a high-water mark can silently miss an entry a second machine appended between syncs. The logs are small — one asset's rulings — so the cost is not worth the class of bug.
func (*Store) ReadingFor ¶
ReadingFor returns what this document is a reading of, if anything is recorded.
func (*Store) ReadingsOfSource ¶
ReadingsOfSource returns every reading of one source, oldest first.
Empty for an empty digest, deliberately: readings that could not name their source are not all readings of each other, and grouping them would assert a relationship nobody established.
func (*Store) ReclaimIdentityJobs ¶
ReclaimIdentityJobs returns rows left 'running' by a process that is gone to pending, and reports how many.
Requeued rather than failed, which is the opposite of what ingest does with its orphans — and the difference is the work. An ingest job may have been killed BY its document (an OOM on a huge scan), so retrying it on every start is a crash loop. A caption is one bounded request on text already in the index: if it died, the process died, and redoing it costs one call.
func (*Store) RecordFields ¶
func (s *Store) RecordFields(ctx context.Context, path string, f DocFields, by string) (DocFields, error)
RecordFields is a PERSON's extraction: what they say the fields are. Never regenerated over.
func (*Store) RecordIdentity ¶
func (s *Store) RecordIdentity(ctx context.Context, path string, d DocIdentity, by string) (DocIdentity, error)
RecordIdentity is a PERSON saying what a document is. It supersedes whatever the machine said and is not regenerated afterwards.
Fields left empty keep what is already recorded, so correcting only the name does not silently blank the summary. by is recorded in Model's place — the column names who produced the claim, and for a person that is the person.
func (*Store) RecordReading ¶
RecordReading records what a document is a reading of. Idempotent per document — a re-ingest replaces the row, because it is a new reading of the same thing.
An empty SourceSHA256 is accepted and stored. A producer that cannot say what it read is a fact worth keeping: oidio's diarized.json names no media and carries no digest, so every transcript derived from one arrives here unable to point at its recording. Refusing the row would hide that; storing it makes the gap countable and fixable.
func (*Store) RecordSourceStat ¶
RecordSourceStat stores the size and modified time of the file behind the document at path, now. A path with no local file is left at 0 (unknown). It does not vouch for the stored bytes, so IndexTree hashes the file next pass.
func (*Store) ReferencesTo ¶
ReferencesTo finds fragments that CITE a path, so a withdrawal does not leave live references pointing at a document nobody can retrieve.
Matched on the path and on its file name, because a citation in this corpus is written either way — "documents/drafts/x.md" in a packet, "x.md" in a note. Fragments of the withdrawn document itself are excluded: a document citing its own name is not a dangling reference.
func (*Store) RemoveDocument ¶
RemoveDocument deletes a document as DeleteDocument does, and the page and figure images it made under this index's pages/. stored also deletes the document's own bytes when they are in this index's storage (an upload, an extracted attachment). A file outside the index's storage is never touched, whatever stored says. A copy under originals/ is left for prune's --delete-orphan-originals.
func (*Store) RequeueJob ¶
RequeueJob returns a claimed-but-unstarted job to the queue.
Distinct from RetryJob, which is a person deciding to run something again: this is the scheduler putting back a row it claimed and then could not hand to a runner, because the daemon is shutting down. Without it the row says `running` forever with no process behind it — the state reclaimOrphanedJobs exists to repair, which is worth not creating on a clean exit.
Restricted to rows that never started, so it cannot rewind a job mid-ingest.
func (*Store) Restore ¶
Restore returns a withdrawn document to the corpus. It does not re-ingest — the file has to be indexed again, which is the caller's next step.
func (*Store) RetryJob ¶
RetryJob requeues an errored or done job: state → pending, error cleared, timestamps reset, so the worker picks it up again. Errors if the job isn't in a retryable state.
func (*Store) SameBytesAs ¶
SameBytesAs lists other indexed documents whose source bytes hash the same, path -> title.
Reusing documents.content_hash rather than adding anything: ingest already records it to skip re-indexing unchanged files, so exact duplication across PATHS was sitting in the index unqueried. On this corpus it names three pairs, one of which is a misfiled document — `2019-04-09-sale-disclosure-packet-COMPLETE-Form34-35-LotCert-Form17.pdf` is byte-identical to `2019-mls-listing-broker-remarks.pdf` and holds an MLS listing sheet, so the four instruments its filename claims are not in it.
A document with an empty content_hash is skipped rather than matched: ” is "not recorded", and grouping on it would report every synthetic document as a duplicate of every other.
func (*Store) SameBytesHash ¶
SameBytesHash lists indexed documents whose recorded source hash equals hash.
This is the upload case, and it is the cheapest true answer available: hash the file the client just sent and ask whether the index already holds those exact bytes. It runs before any extraction, so it needs no OCR, no model, and answers for a scanned PDF that raglit could not otherwise read at all — which is the one input the shingle path has to refuse.
func (*Store) SaveUpload ¶
SaveUpload writes data as the upload called name and returns its absolute path. name is relative and slash-separated ("notes/2026/a.md"); anything that would land outside the uploads directory is refused.
func (*Store) Scopes ¶
Scopes is this index's directory patterns. set is false when they were never written, which reads as an opt-in global index and is what the daemon's one-time migration and an ingest's scope claim look for.
func (*Store) SearchFigures ¶
SearchFigures is SearchFiguresPath with no path constraint.
func (*Store) SearchFiguresOwnLayer ¶
func (s *Store) SearchFiguresOwnLayer(ctx context.Context, query, pathPrefix string, limit int) ([]FigureHit, error)
SearchFiguresOwnLayer is SearchFiguresPath over this layer's own documents only — for sibling fan-out, where the caller's layer already covers the parent.
func (*Store) SearchFiguresPath ¶
func (s *Store) SearchFiguresPath(ctx context.Context, query, pathPrefix string, limit int) ([]FigureHit, error)
SearchFiguresPath ranks figures by cosine similarity of the query to each figure's embedding, best first, optionally constrained to a path subtree (pathPrefix). Only figures whose vector is comparable to a TEXT query are considered — the description embedding, and an aligned image embedding (nomic-vision, whose space nomic-text shares). Non-aligned image embeddings are skipped (they await an image-query path). Requires a text embedder.
func (*Store) SearchFiltered ¶ added in v0.5.0
SearchFiltered is SearchPath constrained by a SearchFilter. Each layer applies the filter in its own WHERE, before its limit. A branch's own copy of a document shadows the parent's whether or not the copy qualifies, so a parent row whose tags or fields the branch has since changed cannot answer for it.
func (*Store) SearchOwnLayer ¶
func (s *Store) SearchOwnLayer(ctx context.Context, query, mode, pathPrefix string, limit int) ([]Hit, error)
SearchOwnLayer searches this layer's OWN documents only — not its parent. For sibling fan-out: the caller's layer already overlays the shared parent, so searching a sibling's overlay again would return the parent twice. mode is "" / "bm25", "vec" or "hybrid".
func (*Store) SearchPath ¶
SearchPath overlays branch-over-parent: branch hits, then parent hits whose doc is not shadowed, re-ranked by score and truncated to limit — optionally constrained to a path subtree (pathPrefix). A non-branch store (parent == nil) is just its local FTS.
func (*Store) ServesPageImage ¶
ServesPageImage reports whether img is a page image this store may serve: under its own pages directory, or — for a branch — under an ancestor's, for the parent documents that show through it.
func (*Store) SetAllowPatterns ¶
SetAllowPatterns replaces this index's allow list, cleaned the same way.
func (*Store) SetAmbient ¶
SetAmbient makes a global index ambient (true) or opt-in (false).
func (*Store) SetAuthorityResolver ¶
SetAuthorityResolver supplies the person's ruling about which reading of a source governs (readings.go).
INJECTED, because the rulings live in the trail beside the documents and this package must not go looking for it: an index does not know where its corpus is, and one that guessed would answer differently depending on where it was opened from. Unset → the level order decides, which is the right default for a corpus nobody has ruled on.
func (*Store) SetCaptionProfile ¶ added in v0.4.1
SetCaptionProfile records the owner's choice; "" returns to the corpus's shape. It applies to captions written from now on. Existing captions change only by `raglit identify --force`.
func (*Store) SetDocType ¶
SetDocType registers or replaces a type.
func (*Store) SetDocumentFields ¶
SetDocumentFields records an extraction and re-indexes its fragment, in one transaction — so the row and the searchable text can never disagree.
A person's extraction is what a re-run must not touch; that rule lives in the CALLER, because this is also how a correction is written.
On a branch, a document only the parent holds is an error wrapping ErrParentOnly.
func (*Store) SetDocumentHash ¶
SetDocumentHash records a document's source-content hash after a successful ingest, so an unchanged re-ingest can be skipped.
func (*Store) SetDocumentIdentity ¶
SetDocumentIdentity records what a document is and re-indexes its identity fragment, in one transaction — so the columns and the searchable text can never disagree.
A person's identity (Source 'person') is what a re-run must not touch; that rule lives in the CALLER, because this is also how a correction is written.
func (*Store) SetEmbedder ¶
SetEmbedder enables vector search: fragments are embedded on Ingest and VecSearch/HybridSearch become available. nil disables it.
A reembed of an index with ingests in flight is, in this order: SetEmbedder with the new embedder, then SyncEmbedModel (or DropVectors), then BackfillVectors until it returns 0. An in-flight ingest that embedded with the old embedder then leaves no vectors behind (vecMu); its fragments are missing a vector, and the backfill embeds them with the new one. The other order (drop, then swap) leaves a window in which old vectors survive.
func (*Store) SetExtractEmailAttachments ¶
SetExtractEmailAttachments turns mail-archive attachment extraction on.
func (*Store) SetIdentifier ¶
func (s *Store) SetIdentifier(id *Identifier)
SetIdentifier enables document identity: each ingested document is captioned, summarised and typed by the model (identity.go), and the summary is indexed. nil disables it — an ingest then leaves whatever identity the document already had, because a caption is not invalidated by re-reading the same file.
func (*Store) SetIgnorePatterns ¶
SetIgnorePatterns replaces this index's ignore list. Blank entries and repeats are dropped. It does not touch documents already indexed; a purge does that.
func (*Store) SetImageEmbedder ¶
func (s *Store) SetImageEmbedder(ie ImageEmbedder)
SetImageEmbedder enables IMAGE embeddings for figures (a CLIP-style tower): each figure is embedded from its image rather than its description. nil (the default) → figures embed their description with the text embedder instead.
func (*Store) SetIndexHint ¶
SetIndexHint records it. Empty clears it.
func (*Store) SetJobLane ¶
SetJobLane records which lane a job actually belongs in. Used by the worker once routing has read the bytes and disagreed with the guess from the URL.
func (*Store) SetJobURL ¶
SetJobURL re-points a job at a new URL. Reconciliation uses it when a job's stored source path no longer resolves but the same content is indexed at another path: the job's identity (its id) is unchanged, only the pointer it follows moves, so a subsequent retry ingests the document under the job's existing identity.
func (*Store) SetParent ¶
SetParent makes this store a branch over p (branch-over-parent overlay reads).
func (*Store) SetPaused ¶
SetPaused pauses or resumes this index. Stored in the index, so a restart keeps it. A job already running finishes; nothing new is claimed.
func (*Store) SetScopes ¶
SetScopes replaces this index's directory patterns. Blank entries and repeats are dropped; a plain directory is cleaned. An empty list makes the index global.
func (*Store) SetTags ¶ added in v0.6.0
SetTags replaces the caller's tags on the document at path. An empty slice clears them. Each tag is normalized the way the model's are (normalizeContentTag) and repeats are dropped, but the model-tag rules do not apply: no length or word limit, no count, no role vocabulary. A caller's tag is its own vocabulary, "kind:strategy" included.
On a path with no document yet the tags are held, and the next ingest of that path to commit applies them, so "enqueue, then tag" works without waiting for the job. A later SetTags before that commit replaces the held ones.
On a branch, a document that only the parent holds is an error: a branch cannot edit the parent's document until copy-up exists (the same rule as SetDocumentFields). The error wraps ErrParentOnly.
func (*Store) SiblingReadings ¶
SiblingReadings returns the OTHER readings of whatever this document reads — the answer to "show me the raw one" from a verified transcript, and the reverse.
func (*Store) SimilarIndexed ¶
func (s *Store) SimilarIndexed(docPath string, opts SimilarOpts) (SimilarReport, error)
SimilarIndexed compares an already-indexed document against the rest of the index, and additionally reports any file byte-identical to it.
func (*Store) SimilarTo ¶
func (s *Store) SimilarTo(probePath string, probe FoldedText, opts SimilarOpts) (SimilarReport, error)
SimilarTo compares folded probe text against the index and returns the report.
Takes already-folded text rather than a path so the caller decides where the probe came from — an indexed document, a file on disk, or a raglit transcription sidecar. That mattered: making this open the file itself meant a probe could only be something raglit could extract, and the upload flow needs to compare text it already has in hand without writing it anywhere first.
func (*Store) SketchAll ¶
SketchAll builds sketches for every document that has none under the current recipe. Returns how many documents were sketched. A document whose text cannot be read is skipped with its error collected rather than aborting the run: one unreadable document must not leave the rest of the corpus unchecked, which is the state in which `similar` silently reports "nothing found".
func (*Store) SketchDoc ¶
SketchDoc builds (or rebuilds) one document's page sketches from the index's own text.
Reads the document through TruePages, so a page number in a later alignment is the page the text is actually on, not the page its fragment opened on. Replaces the document's rows wholesale: an incremental update would have to know which shingles a re-OCR'd page lost, and a re-read changes a page completely often enough that the bookkeeping would cost more than the rewrite.
func (*Store) SketchStatusFor ¶
func (s *Store) SketchStatusFor(w, mod int) (SketchStatus, error)
SketchStatusFor reports coverage under a recipe.
func (*Store) SketchedPaths ¶
SketchedPaths lists documents carrying a sketch under this recipe, in a stable order. The all-pairs audit walks these and nothing else — comparing a document with no sketch would silently fall back to zero candidates and report it clean.
func (*Store) StoragePath ¶
StoragePath is where a storage document's bytes are on THIS machine: the file under this store's home (or its parent's, for a branch) that StorageRel names. Empty and false for a corpus document, for a path that would leave the home, and for a file that is not there.
func (*Store) StorageRel ¶
StorageRel reports whether a document's bytes live in raglit's own storage for this index (extracted attachments, uploads, transcripts) rather than in the corpus, and if so its path relative to that storage.
Such documents are real documents, but their directory is raglit's business: letting ~/.raglit/indexes/<name>/attachments/... take part in the corpus's common directory widened one legal corpus's root to the user's home, which put /home/<user>/... in front of every path and made the review list walk the whole home directory.
Two matches. Under this store's home is the ordinary case. The second is a home that has been COPIED or moved: the rows still carry the path of the home they were written in, and the only stable part of it is the ".../indexes/<this index>/" layout. A branch also asks its parent, whose documents show through it with the parent's home in their paths.
func (*Store) SupersededPages ¶
func (s *Store) SupersededPages(ctx context.Context) ([]PageReading, error)
SupersededPages lists documents holding a page whose reading was replaced — the corpus's record of where a machine read was found wanting.
func (*Store) SupplyIdentity ¶ added in v0.4.0
func (s *Store) SupplyIdentity(path string, d DocIdentity, by string) error
SupplyIdentity records the caption a document will carry when its next ingest commits: name required, summary cut to identityMaxSummaryChars (no minimum: a supplied summary is what the caller had), recorded as IdentitySupplied by `by`. It is consumed by that commit.
func (*Store) SyncEmbedModel ¶
SyncEmbedModel records model and dim as the source of this index's vectors and reports whether existing vectors were dropped because they came from another model or have another length. An index with no model record adopts model as-is: it is an index from before the record existed, whose vectors are from the model configured when they were made, and re-embedding every index on upgrade would cost the whole corpus for nothing.
dim 0 means the caller does not know it: the dimension is then neither compared nor recorded. With no dim record, the length of a stored vector stands in for one, so an index from before the record still drops vectors that the configured model can no longer match.
func (*Store) TagContext ¶
TagContext is this index's established tag vocabulary, as one line for the identity prompt — the mechanism that keeps "lead paint" from drifting into "LBP", "paint inspection" and "lead-based paint hazards" across a corpus. Empty for a fresh index, and empty rather than an error when the read fails: a caption without the vocabulary is worse than one with it, not a failure.
func (*Store) TagDocument ¶
TagDocument fills in a document's tags WITHOUT touching its caption.
The caption's authorship is preserved exactly — a person's stays a person's, a machine's keeps the model and timestamp that wrote it — because tags are not a re-reading of the document, they are two columns that were never asked for. Returns ErrIdentityKept when the document already has tags, unless forced.
func (*Store) TagNeighbours ¶
TagNeighbours reports, for each content tag, the other tags that share a significant word with it — the drift a person is being asked to rule on.
A LEXICAL proposal and nothing more. It cannot see that "escrow closing" and "settlement" are the same thing, and it will pair "lead paint" with "lead paint disclosure", which are not. That is why nothing here writes.
func (*Store) Tags ¶ added in v0.6.0
Tags returns the caller's tags on the document at path, in the order SetTags stored them. A document with none returns nil. An unknown path is an error; tags still held for it are not reported.
A branch with a root (OpenBranch) reads a parent-only document's tags through to the parent. A branch with no root does not.
func (*Store) TranscriptDirFor ¶
TranscriptDirFor is where this index stores the transcript and machine reading of the recording at mediaPath. Empty when the store has no home, the same case AttachmentDirFor cannot serve — an in-memory index has nowhere to put a sidecar, and the caller records that as a skipped stage rather than a failure.
func (*Store) TruePages ¶
TruePages reassembles a document at TRUE page grain, honouring page_spans.
DocText's Pages array is not this, and the difference is the reason this exists. The assembler absorbs a page's continuation (or a sub-floor sibling) from the NEXT page into an open fragment and keeps only the START page in `fragments.page`, so DocText attributes every byte of a multi-page fragment to the page the fragment opened on. Measured on this index: 67 of 460 fragments carry page boundaries, and honouring them recovers 441 true pages where grouping by `fragments.page` finds only 277. Dace's declaration is the clearest case — 14 fragments opening on pages 1, 4, 14, 17, 18, 23, 24, 28 and 33, so DocText reports it as a 9-page document and every page number after the first stitch is wrong. That is exactly wrong for anything that wants to SAY where a finding is.
page_spans was added to fix that ("a search hit inside one could not be resolved to the page it is actually on", per the schema) and, until this, was written by ingest, carried through the pool, and read by nothing but its own tests. Similarity is its first consumer: "pages 12-14 of X are pages 1-3 of Y" is the whole deliverable, and a page number that is silently the fragment's start page makes that report a lie.
Pages come back in page order, contiguous from the lowest page present, with empty pages included — a consumer resolving an alignment to a page depends on the numbering being complete, the same contract the transcription markdown makes. Raw SQL, not sqlc, and not by preference. The generated ListFragmentsForDoc does not select page_spans, and regenerating is currently destructive: `sqlc generate` with the installed toolchain (sqlc v1.30.0 against the metaquery plugin built on plugin-sdk-go v1.23.0) rewrites the SQL text of EVERY query in internal/db — each one comes back rotated two characters, e.g.
const cancelJob = `-- name: CancelJob :execrows ); DELETE FROM ingest_jobs WHERE id=? AND state='pendin`
That is 60+ silently broken queries for the sake of one added column, so this follows the precedent the schema file already sets for FTS5 and the vector scan: what sqlc cannot express here stays raw. Fix the plugin, then fold this back in.
func (*Store) VecSearchFiltered ¶ added in v0.5.0
func (s *Store) VecSearchFiltered(ctx context.Context, query string, f SearchFilter, limit int) ([]Hit, error)
VecSearchFiltered is VecSearchPath constrained by a SearchFilter, applied in each layer's scan before the ranking and the limit.
func (*Store) VecSearchPath ¶
func (s *Store) VecSearchPath(ctx context.Context, query, pathPrefix string, limit int) ([]Hit, error)
VecSearchPath embeds the query and ranks fragments by cosine similarity, best first, optionally constrained to a path subtree (pathPrefix). Brute-force: it scans every stored vector in scope (fine for a local corpus; see embed.go). Requires SetEmbedder. Score is cosine in [-1,1] (higher = better). Fragments without a vector (indexed before embeddings were enabled) are invisible.
func (*Store) Withdraw ¶
func (s *Store) Withdraw(w Withdrawal) error
Withdraw removes a document from this index and records why.
Idempotent: withdrawing an already-withdrawn path refreshes the grounds and removes any rows a re-ingest slipped in before the decision was made.
func (*Store) Withdrawals ¶
func (s *Store) Withdrawals() ([]Withdrawal, error)
Withdrawals lists every ruling, path order.
func (*Store) WithdrawnReason ¶
WithdrawnReason reports why a path is out of the corpus, if it is.
func (*Store) WriteIndexAbout ¶
WriteIndexAbout asks the model what this index is and stores the answer.
Uses the identity model, because it is the one already configured to read this corpus and the task is the same task one level up.
type StoreSet ¶ added in v0.5.0
type StoreSet interface {
Names() []string
// Existing returns the open store for name, or an error when there is none.
// It must not create an index.
Existing(name string) (*Store, error)
}
StoreSet is the indexes a Runner works: their names, and the open store for a name. It is read again on every pass, so a set that gains an index while the Runner runs (a Registry does, when an index is created) has that index's queue worked from the next pass on. *Registry satisfies it.
type StrategyConfig ¶
type StrategyConfig struct {
// Descend is RegionReader.MaxDepth. 0 reads the sheet and stops, which is
// what every page does today.
Descend int `json:"descend,omitempty"`
// Tile subdivides a large low-resolution DRAWING geometrically instead of
// asking the model where to look — arithmetic instead of a call, and on an
// E-size sheet the asking is the part that does not work.
Tile bool `json:"tile,omitempty"`
// Hint is threaded into every prompt. Cheapest lever here by a distance: the
// model proposing regions is looking at a view where the thing you want may
// be physically unresolvable, so it cannot propose what it cannot see. A
// sentence naming the target beats any amount of extra budget.
Hint string `json:"hint,omitempty"`
// Budgets. Zero → RegionReader's own defaults (8 / 2 / 40 / 4).
MaxCalls int `json:"max_calls,omitempty"`
MaxChildren int `json:"max_children,omitempty"`
MaxTransforms int `json:"max_transforms,omitempty"`
MaxEscalations int `json:"max_escalations,omitempty"`
MinRegionIn float64 `json:"min_region_in,omitempty"`
// Render overrides the automatic per-page resolution policy.
Render RenderPolicy `json:"render,omitempty"`
// AutoDescend descends WITHOUT being asked, but only for a page the pixels
// say needs it: the same low-resolution test the descent already applies
// internally (below half the letter-page token density). Measured over a 998
// page corpus that is 14 pages in 11 documents — 1.4% — so it is affordable
// in a way "descend everything" is not, and it targets exactly the sheets a
// single-shot read fails on. Off by default; Descend still bounds it.
AutoDescend bool `json:"auto_descend,omitempty"`
}
StrategyConfig is how much work a page is worth.
Every field is zero-valued to today's behavior, so adding this section changes nothing until something is set. That matters more than brevity here: this governs model spend, and a config format whose defaults are not the current behavior turns an upgrade into a bill.
type TagMerge ¶
TagMerge is one collapse: every tag in From becomes To.
func ParseTagMerge ¶
ParseTagMerge reads the "a,b=>c" form: the tags to collapse, then the tag to collapse them into. Whitespace around any term is ignored.
type TagMergeResult ¶
type TagMergeResult struct {
To string `json:"to"`
From []string `json:"from"`
Documents int `json:"documents"` // documents whose tags changed
Collapsed []string `json:"collapsed"` // the From tags actually present
Missing []string `json:"missing"` // From tags no document carried
}
TagMergeResult is what a merge did.
type TesseractEngine ¶
type TesseractEngine struct {
Bin string // tesseract binary; "" → "tesseract"
Lang string // -l language; "" → "eng"
}
TesseractEngine runs the tesseract CLI over each page: no cgo, no daemon, one optional system dependency. Fast and ~free relative to a vision LLM, but blind to handwriting and stylized figures — which is exactly what the gibberish gate exists to catch and escalate.
func NewTesseractEngine ¶
func NewTesseractEngine(bin, lang string) *TesseractEngine
NewTesseractEngine builds a TesseractEngine with defaults filled.
func (*TesseractEngine) Name ¶
func (e *TesseractEngine) Name() string
type TokenBudget ¶
type TokenBudget struct {
// Overhead is text the CONSUMER prepends that a count of the fragment cannot
// see. The embedder sends DocPrefix + fragment ("search_document: " for
// nomic), so a fragment sized to the limit exactly is over it by the prefix.
Overhead string
// Reserve covers what the tokenizer adds around the input and does not report
// — the BOS/EOS pair, counted because add_special is off when measuring a
// piece of a larger prompt.
//
// Both were learned the same way: a fragment sized to 8192 was refused at
// "input (8194 tokens) is too large to process". Two tokens. A budget that
// misses by two is not a budget, and it fails the document exactly as
// completely as one that misses by two thousand.
Reserve int
// contains filtered or unexported fields
}
TokenBudget sizes text against a token limit for one document.
Exact where it can be: with a tokenizer, a prefix is measured rather than predicted, and the learned ratio only chooses where to start looking so the search converges in a call or two instead of a binary search.
func NewTokenBudget ¶
func NewTokenBudget(ctx context.Context, counter TokenCounter, limitTokens int) *TokenBudget
NewTokenBudget builds a budget over an optional counter. A nil counter, or a limit of 0, yields a budget that never cuts.
func (*TokenBudget) Fit ¶
Fit returns how many characters of `rest` can be sent alongside `overhead` characters of prompt without exceeding the limit — len(rest) when it all fits.
With a tokenizer this MEASURES rather than predicts: the ratio picks a starting guess and each check corrects it, so a page of prose and a page of coordinates both converge in a call or two. The alternative — one ratio for the whole corpus — is what put nine documents past the limit and cut the rest to a fifth of what the endpoint would have taken.
minTake guarantees progress: when the overhead alone leaves no room, cutting to fit is not available and a zero-length take would never terminate.
func (*TokenBudget) FitOne ¶
FitOne is Fit against the budget's own Overhead — the form the fragment splitters use, where the overhead is a property of the consumer rather than of the call.
func (*TokenBudget) FitSuffix ¶
FitSuffix returns how many characters of the END of `s` fit in `room` tokens.
The mirror of Fit, and it exists because the two cuts protect different things. A take keeps the FRONT of the remaining page, because that is what comes next. A carried-over fragment keeps its TAIL, because continuation is decided by how it ends.
func (*TokenBudget) Fits ¶
func (b *TokenBudget) Fits(ctx context.Context, text string) bool
Fits reports whether text is inside the budget once the consumer's overhead is counted. The check every splitter shares, so they agree about what fits.
func (*TokenBudget) Tokens ¶
func (b *TokenBudget) Tokens(ctx context.Context, text string) int
Tokens counts text, exactly when possible and by the learned ratio otherwise.
func (*TokenBudget) Unlimited ¶
func (b *TokenBudget) Unlimited() bool
Unlimited reports whether this budget will never cut.
type TokenCounter ¶
TokenCounter counts tokens as the serving model does. Implemented by *llm.Client; ok=false means this endpoint cannot count, which is a fact to work around rather than an error.
type ToolEnv ¶
type ToolEnv struct {
// Who describes the process this was measured in — "shell" or "daemon (pid
// 1234)". Set by whoever renders it; a probe cannot know what it will be
// called.
Who string `json:"who"`
// PATH as this process has it. The single most useful line when two probes
// disagree, and the reason a tool "is not installed" when it is.
PATH string `json:"path_env"`
// Tools, in a stable order so two environments can be diffed line by line.
Tools []ToolStatus `json:"tools"`
// GOOS, because an install hint for apt is noise on a mac.
GOOS string `json:"goos"`
}
ToolEnv is the whole external-tool picture for one process.
func ProbeTools ¶
func ProbeTools() ToolEnv
ProbeTools measures this process's view of the external extractors.
Deliberately does NOT run each tool's --version: this is called on an HTTP path that an operator may poll, LookPath is a stat and a version probe is a fork, and "which binary would we run" is the question that was actually being got wrong.
func (ToolEnv) Disagreements ¶
Disagreements lists the tools two environments see differently — missing in one, or resolving to different binaries.
Both halves matter. "Present here, absent there" is the failure that started this. "Present in both, different paths" is quieter and worse: two pandocs of different vintages read the same .docx into different text, and nothing in an index records which one ran.
type ToolStatus ¶
type ToolStatus struct {
// Name is what the router looks for ("pandoc"), not the package that ships it.
Name string `json:"name"`
// Found is whether it resolved on PATH.
Found bool `json:"found"`
// Path is where it resolved, empty when not found. Reported because two
// processes can both find a tool and find DIFFERENT ONES.
Path string `json:"path,omitempty"`
// Purpose is what stops working without it, in the operator's terms.
Purpose string `json:"purpose"`
// Optional marks a tool whose absence degrades rather than breaks.
Optional bool `json:"optional"`
// Install is the hint printed when it is missing.
Install string `json:"install,omitempty"`
}
ToolStatus is one external tool as this process sees it.
type TranscribedFigure ¶
TranscribedFigure is a described diagram or image on a page. A scanned survey is mostly figure, and a transcription that drops it says the page is blank.
type TranscribedPage ¶
type TranscribedPage struct {
Page int
Text string
Figures []TranscribedFigure
// Corrected is a person's replacement for this page, when one exists. Set by
// RenderTranscriptionCorrected; never read off disk.
Corrected PageCorrection
}
TranscribedPage is one page of a transcription.
type TranscriptMatch ¶
type TranscriptMatch struct {
Transcript string `json:"transcript"`
Recording string `json:"recording,omitempty"`
Source string `json:"source,omitempty"`
Score float64 `json:"score"`
// Trust is what the adopted reading would be worth, per facet.
Trust []string `json:"trust,omitempty"`
// Why is set when nothing was adopted, and says which of the two failure
// shapes it was: nothing close enough, or nothing to compare against.
Why string `json:"why,omitempty"`
}
TranscriptMatch is one verified transcript and the recording it belongs to.
type TranscriptionChange ¶
type TranscriptionChange struct {
Doc string
Page int
// Active is the reading now in force.
Active PageCorrection
// Superseded is the text it replaced — a machine read, or an earlier
// correction. Empty when this is the first recorded reading for the page.
Superseded string
}
TranscriptionChange reports that a page's ACTIVE reading changed.
The event exists because a correction invalidates things it does not own. The .raglit-transcription.md export beside the document still shows the old text; the index still holds it in fragments, so a search still returns the reading a person just replaced. Neither notices on its own.
Superseded carries what was replaced rather than only what won. A correction is an attestation that a better reading exists, not an erasure: the old text is what the index held when a document was cited, and it is what a stale quotation elsewhere will still match.
type TreeOptions ¶ added in v0.9.0
type TreeOptions struct {
Rules TreeRules
// Paths: check only these (absolute, or relative to the root). A directory
// covers what is under it; a path that no longer exists counts its
// documents as Removed. Empty = the whole tree.
Paths []string
// Route picks the index (a key of IndexTrees' map) for a file. Optional
// with one store.
Route func(path string) string
// Keep spares a document under the root that the rules do not select:
// one whose path (absolute, no scheme) Keep reports true for is not
// Removed. For roots that overlap in one index, each with its own rules:
// Keep is "another root selects it".
Keep func(path string) bool
// Verify hashes every file. By default a file whose size and modified
// time (ns) equal the stat recorded when its bytes were last read is
// Current without being read (srcstat.go). The trade-off: a change that
// keeps both the size and the mtime (a tool that restores the mtime, a
// clock set back) is missed until the stat changes or a Verify pass. A
// file modified within 2 s of its stat is always hashed.
Verify bool
}
TreeOptions are IndexTree's options.
type TreeResult ¶ added in v0.9.0
TreeResult counts what IndexTree did per file. Errors are per-file failures (a read, an enqueue); the run goes on past them.
func IndexTree ¶ added in v0.9.0
func IndexTree(ctx context.Context, st *Store, root string, o TreeOptions) (TreeResult, error)
IndexTree brings st in step with the files under root. See TreeOptions.
type TreeRules ¶ added in v0.9.0
type TreeRules struct {
// Include globs, relative to the root; empty = every file. A glob with no
// "/" matches the file name, as in PlanSources.
Include []string
// Ignore globs, relative to the root, on top of the built-in ignores.
Ignore []string
// Gitignore: nil or true = honor .gitignore (git ls-files) in a git work
// tree; false = walk the directory.
Gitignore *bool
// Skip drops an absolute path (a file, or for WatchTree a directory).
Skip func(path string) bool
// MaxBytes: a larger file counts as Skipped. 0 = 2 MiB; negative = no
// limit. A document whose file grew past the limit is kept, not Removed.
MaxBytes int64
}
TreeRules choose the files of a tree. IndexTree and WatchTree share them.
type TreeWatcher ¶ added in v0.9.0
type TreeWatcher struct {
// contains filtered or unexported fields
}
TreeWatcher watches one root. Make one with WatchTree.
func WatchTree ¶ added in v0.9.0
func WatchTree(ctx context.Context, root string, r TreeRules, o WatchOptions, fn func(Changes)) (*TreeWatcher, error)
WatchTree watches the directories under root that r does not exclude (never .git, node_modules or a gitignored directory), adding directories as they appear, and calls fn with one Changes per burst. fn runs on the watcher's goroutine, one call at a time. The watcher stops on Stop, on ctx's end, or after o.Idle.
func (*TreeWatcher) Rescan ¶ added in v0.9.0
func (w *TreeWatcher) Rescan()
Rescan reports everything selected, as one burst (Changes with no Paths), as soon as MinInterval allows.
func (*TreeWatcher) Root ¶ added in v0.9.0
func (w *TreeWatcher) Root() string
Root is the watched directory, absolute and clean.
func (*TreeWatcher) Stop ¶ added in v0.9.0
func (w *TreeWatcher) Stop()
Stop ends the watcher. It only signals and never waits, so it is safe from inside fn and from a goroutine fn waits on: the watcher ends once a callback in progress returns, and calls fn no more. Wait waits for that.
func (*TreeWatcher) Touch ¶ added in v0.9.0
func (w *TreeWatcher) Touch()
Touch resets the idle timer.
func (*TreeWatcher) Wait ¶ added in v0.9.0
func (w *TreeWatcher) Wait()
Wait blocks until the watcher has ended (Stop, ctx's end, or Idle) and its callback in progress, if any, has returned. Wait must not be called from inside fn, or from a goroutine fn waits on: it would wait for itself.
type VectorClient ¶
type VectorClient interface {
Embed(ctx context.Context, model string, input []string) ([][]float32, error)
}
VectorClient is the sliver of *llm.Client the embedder needs. An interface so tests supply deterministic vectors without a network.
type Visibility ¶
type Visibility string
Visibility says who may read an index.
const ( Private Visibility = "private" Public Visibility = "public" )
type WatchHub ¶ added in v0.9.0
type WatchHub struct {
// contains filtered or unexported fields
}
WatchHub shares watchers and index runs across callers (R12e): one TreeWatcher per root however many subscribe, and one index run per root at a time.
func NewWatchHub ¶ added in v0.9.0
func NewWatchHub(o WatchOptions) *WatchHub
NewWatchHub makes a hub whose watchers use o.
func (*WatchHub) Index ¶ added in v0.9.0
func (h *WatchHub) Index(ctx context.Context, key string, do func(context.Context) (TreeResult, error)) (TreeResult, error)
Index runs do for key (a root, or any string naming one tree), one run at a time. A call while no run is going starts one and gets its result. A call during a run joins the one follow-up run that starts when it ends: every call that arrives during the same run shares that follow-up and its result, and the follow-up runs the do of the last of them. So do must cover everything asked so far (drain a set of pending paths, say), not only its own caller's request. A caller whose ctx ends stops waiting; the run goes on, under a context that keeps ctx's values and is cancelled only by Stop.
func (*WatchHub) Stop ¶ added in v0.9.0
func (h *WatchHub) Stop()
Stop stops every watcher and cancels the index runs in flight. Subscriptions end; a later Subscribe or Index fails. Stop only signals and never waits, so it is safe from inside a callback or an Index do; Wait waits.
func (*WatchHub) Subscribe ¶ added in v0.9.0
func (h *WatchHub) Subscribe(root string, r TreeRules, fn func(Changes)) (unsubscribe func(), err error)
Subscribe calls fn with every burst under root. The first subscriber of a root starts its watcher; the watcher skips only what the built-in rules and that subscriber's Gitignore setting skip, and each subscriber's Ignore, Include and Skip then filter the paths it is given (a burst left with no paths is not delivered, unless it is a whole-tree or Degraded one). The last unsubscribe stops the watcher. unsubscribe only signals and never waits, so it is safe from inside fn and from a goroutine fn waits on: once it returns, no new call of fn starts, but a call in progress may still be running (Stop and Wait wait for it). A watcher that idles out (WatchOptions Idle) drops its subscribers; the next Subscribe starts a new one.
func (*WatchHub) Wait ¶ added in v0.9.0
func (h *WatchHub) Wait()
Wait blocks until every watcher the hub started has ended, with its callback in progress, and every index run has returned. Call it after Stop; before Stop it waits for watchers to idle out, and must not race a Subscribe. Wait must not be called from inside a callback or an Index do, or from a goroutine one waits on: it would wait for itself.
type WatchOptions ¶ added in v0.9.0
type WatchOptions struct {
// Quiet: a burst ends after this long with no event. Default 500 ms.
Quiet time.Duration
// MaxDelay: a burst ends this long after its first event, even if events
// keep coming. Default 5 s.
MaxDelay time.Duration
// MinInterval: at most one burst per this long. Default 2 s.
MinInterval time.Duration
// Idle: the watcher stops itself after this long with no Touch and no
// event. 0 = never.
Idle time.Duration
}
WatchOptions tune a watcher's debounce. A zero field takes its default; a negative MinInterval means none.
type Withdrawal ¶
type Withdrawal struct {
Path string `json:"path"`
Reason string `json:"reason"`
By string `json:"by,omitempty"`
// At is when the decision was made, which is not necessarily when it was
// appended — the same distinction AuditEvent.At draws.
At string `json:"at,omitempty"`
}
Withdrawal is a document ruled OUT of the corpus, and why.
Removing a document from an index is a delete; withdrawing it is a decision, and the difference is the reason. A delete leaves nothing behind, so the next sweep re-ingests the file and the next reader wonders why a document that plainly exists is not in the index. A withdrawal says what was decided, so it survives re-ingest and answers the question.
The case it was written for: a corpus of legal evidence that had absorbed the drafts folder — outreach letters, settlement proposals, argument written by the party themselves. A search for "the Brannocks acted in good faith" hit a draft letter and returned it looking like a source. A draft is argument, not record; only a letter actually SENT and declared is evidence of anything.
type Worker ¶
type Worker struct {
Store *Store
// OCR ingests PDF jobs. nil → a PDF job fails with a clear message (a text
// corpus needs no vision model).
OCR *OCR
// STT transcribes audio jobs, by asking oidio. nil → an audio job fails with
// a clear message, the same way a PDF job does with no vision model: a corpus
// of documents needs no transcriber and should not be told to configure one.
STT *STT
// Segmenter LLM-segments a document ONLY when a page escalated to the VLM (the
// llm-seg path, decided per-document in ingestUnits). Text/code never escalate,
// so they take the deterministic overlapping-window fragmenter regardless of
// whether a model is configured. nil → PDF/image llm-seg unavailable.
Segmenter *Segmenter
// Code cuts source files at declarations and Markdown at headings
// (KindCode, codesegment.go). nil → those files are text, read into
// windows as before.
Code CodeSegmenter
// Frag tunes the deterministic text fragmenter (window/stride/floor) and caps
// the fragment ceiling by the embed model's probed input limit. Zero → defaults.
Frag FragConfig
// Fetcher overrides URL resolution (tests). nil → Fetch (file://, http(s)://).
Fetcher func(ctx context.Context, url string) (Fetched, error)
// IdlePoll is how long Run waits when the queue is empty. Default 500ms.
IdlePoll time.Duration
// Pool + RecipeHash enable cross-index dedup of INDEXING work: a fresh ingest
// is cached in the pool keyed by (RecipeHash, source-file hash); a matching
// key — from ANY index, or a retry — is reused (fragments + vectors + images
// copied in) instead of re-running the LLM. RecipeHash captures the models +
// config that shape the output, so alt models are a new key. nil Pool →
// per-index dedup only (content_hash).
Pool *Pool
RecipeHash string
// Retries collects what the LLM client had to survive for the job in flight,
// recorded as a stage row when it is not empty. nil → retries stay invisible,
// which is what they were.
Retries *RetryTally
}
Worker drains the ingest queue (queue.go): claim a pending job → fetch its URL → turn it into a Document (plain-text fragmenting, or PDF → OCR) → Ingest → mark done/error. Run it in the background of `serve`, or step it once from the CLI. A per-URL failure is recorded on the job, not fatal — one bad URL never stops the worker.
func NewWorker ¶ added in v0.5.0
NewWorker wires a Worker with OCR (PDF) + LLM segmentation (text/code) when a vision model is configured; text-window sizing is resolved lazily on the first text job (probing + caching the model's context). No model → offline blank-line text + PDF-fails-gracefully.
It also sets the store's identifier. pool, when non-nil, enables the cross-index pool (daemon only). m.Channels must be set (Models.Channels).
func (*Worker) Drain ¶
Drain processes jobs until the queue is empty, returning how many it handled.
func (*Worker) ProcessJob ¶
ProcessJob runs a job that has ALREADY been claimed.
Split from ProcessOne so a scheduler can separate the two halves. Claiming is a transaction; running is minutes of OCR. The lane dispatcher claims in a round-robin over every index — which is what makes it fair — and hands the job to whichever slot is free, so one long job occupies a slot instead of the whole queue. ProcessOne stays the claim-and-run pair the CLI drain and the tests use.
func (*Worker) ProcessOne ¶
ProcessOne claims and processes a single job. processed is false when the queue is empty. A fetch/ingest failure is recorded on the job and returns (true, nil); a returned error is infrastructure (db) failure only.
Source Files
¶
- acl.go
- attachmigrate.go
- attestations.go
- audit.go
- branch.go
- captionprofile.go
- chat.go
- chatlimit.go
- codesegment.go
- config.go
- contentid.go
- docdiff.go
- docfields.go
- docfiles.go
- docget.go
- docpaths.go
- doctype.go
- dpi.go
- email.go
- emailattach.go
- embed.go
- extract.go
- failedsources.go
- fetch.go
- figures.go
- filter.go
- finder.go
- fold.go
- fragment.go
- frontmatter.go
- gibberish.go
- hitmatch.go
- hitpage.go
- home.go
- identity.go
- identityqueue.go
- imageembed.go
- importreadings.go
- indexcontrol.go
- indexdigest.go
- indexmeta.go
- indexscope.go
- indextext.go
- judgements.go
- judgements_iface.go
- keys.go
- lane.go
- layout.go
- modelchan.go
- newworker.go
- notes.go
- ocr.go
- ocrengine.go
- pagelayout.go
- pagereadings.go
- pagify.go
- pipeline.go
- pool.go
- problems.go
- project.go
- prompts.go
- prune.go
- queue.go
- readings.go
- reembed.go
- regiondoc.go
- regionfilter.go
- regionread.go
- regions.go
- registry.go
- relations.go
- reread.go
- retrytally.go
- reuse.go
- review.go
- runner.go
- savings.go
- segment.go
- shingle.go
- similar.go
- similarstore.go
- sliceprop.go
- slices.go
- sources.go
- srcstat.go
- stages.go
- store.go
- stt.go
- suspect.go
- tagmerge.go
- tags.go
- tokenbudget.go
- tools.go
- transcribe.go
- transient.go
- tree.go
- treewatch.go
- trust.go
- upload.go
- watchhub.go
- window.go
- withdraw.go
- worker.go
- worktree.go
Directories
¶
| Path | Synopsis |
|---|---|
|
Package attest is the machine-read-plus-human-verdict framework that oidio, raglit and kgraph each built separately and none of them built completely.
|
Package attest is the machine-read-plus-human-verdict framework that oidio, raglit and kgraph each built separately and none of them built completely. |
|
Package client is raglit's Go client: how another program asks raglit about raglit's data.
|
Package client is raglit's Go client: how another program asks raglit about raglit's data. |
|
cmd
|
|
|
raglit
command
Command raglit is a document index and search service for LLM sessions: BM25 and vector search over SQLite, at document:page:fragment grain.
|
Command raglit is a document index and search service for LLM sessions: BM25 and vector search over SQLite, at document:page:fragment grain. |
|
internal
|
|
|
update
Package update is how a release build of raglit gets its next build: a manifest of downloads (URL, sha256 and size per os/arch) signed with minisign, an installer that verifies a download before anything runs it, and the daily check that stages a new build for the daemon to move to.
|
Package update is how a release build of raglit gets its next build: a manifest of downloads (URL, sha256 and size per os/arch) signed with minisign, an installer that verifies a download before anything runs it, and the daily check that stages a new build for the daemon to move to. |
|
tools
|
|
|
mkmanifest
command
mkmanifest writes a release's raglit-manifest.json (make release-manifest): one raglit entry per archive in -dist (raglit_<os>_<arch>.tar.gz, sha256 and size computed here, URL -url/<archive>), and key_next from -key-next when that file exists.
|
mkmanifest writes a release's raglit-manifest.json (make release-manifest): one raglit entry per archive in -dist (raglit_<os>_<arch>.tar.gz, sha256 and size computed here, URL -url/<archive>), and key_next from -key-next when that file exists. |
|
notices
command
Command notices checks the licenses of everything a raglit release ships and writes the third-party notices that go in the release tarball.
|
Command notices checks the licenses of everything a raglit release ships and writes the third-party notices that go in the release tarball. |
|
Package web carries the built review SPA.
|
Package web carries the built review SPA. |