Documentation
¶
Overview ¶
Package corpus loads a topic's documents (sources and wiki articles) and provides everything decisions need to reason about them without a model: body hashes, value hashes, BM25 retrieval, the alias dictionary and mention scan, section-aware excerpts, provenance state, the `.decisions/state.jsonl` store and the byte-preserving owned-key writer.
Index ¶
- Constants
- func BodyHash(body string) string
- func DefaultFields(d *Document) map[string]string
- func DefaultWeights() map[string]float64
- func DocumentState(d *Document, maxTokens int, terms []string) map[string]any
- func EstimateTokens(text string) int
- func Excerpt(body string, maxTokens int, terms []string) string
- func Head(body string, maxChars int) string
- func IsMergedListKey(key string) bool
- func ListPaths(paths []string, limit int) string
- func MatchGlob(pattern, p string) bool
- func Tokenize(text string) []string
- func Unclassified(doc *Document, row *StateRow, contract string, banks map[string]string) bool
- func ValidGlob(pattern string) bool
- func ValueHash(v any) string
- type BM25
- type Corpus
- type Dictionary
- type Document
- func (d *Document) Concepts() []string
- func (d *Document) Criterion() string
- func (d *Document) Depth() (float64, bool)
- func (d *Document) Entities() []string
- func (d *Document) Genre() string
- func (d *Document) Locked() bool
- func (d *Document) Provenance() map[string]any
- func (d *Document) Questions() []string
- func (d *Document) Relevance() string
- func (d *Document) SourceKind() string
- func (d *Document) SourceURL() string
- func (d *Document) Summary() string
- func (d *Document) Triage() string
- type Hit
- type Kind
- type LoadOptions
- type Mention
- type Skip
- type StateMeta
- type StateRow
- type StateStore
- func (s *StateStore) Compact() error
- func (s *StateStore) FindByBodyHash(h string) []StateRow
- func (s *StateStore) Get(path string) (*StateRow, bool)
- func (s *StateStore) Lookup(path, bodyHash string, exists func(path string) bool) (*StateRow, bool)
- func (s *StateStore) Path() string
- func (s *StateStore) Put(row StateRow) error
- func (s *StateStore) Rows() []StateRow
- type WriteResult
- type Writer
Constants ¶
const ( StatusWritten = "written" StatusUnchanged = "unchanged" StatusSkippedLocked = "skipped:locked" StatusSkippedChanged = "skipped:changed" // StatusSkippedUserKey is the run-summary label for WriteResult.SkippedUserKeys. StatusSkippedUserKey = "skipped:user-key" )
Write statuses reported in WriteResult.Status.
const (
// HeadChars is the number of body characters indexed by DefaultFields.
HeadChars = 2000
)
BM25 parameters.
const StateFile = resolve.DecisionsDir + "/state.jsonl"
StateFile is the topic-relative path of the document state store.
Variables ¶
This section is empty.
Functions ¶
func DefaultFields ¶
DefaultFields extracts the standard retrieval fields of a document: title, aliases, criterion, summary, entities, questions and head (the first HeadChars characters of the body with wikilinks flattened).
func DefaultWeights ¶
DefaultWeights are the field weights used with DefaultFields.
func DocumentState ¶
DocumentState is the single state builder every decision purpose uses for a document (spec §4.4): `{"title", "excerpt", "provenance"}`, with the excerpt bounded by maxTokens and biased toward sections matching terms (see Excerpt).
func EstimateTokens ¶
EstimateTokens estimates the token count of text as runes/4, rounded up. It is a deliberately cheap, provider-independent approximation used only to bound state size.
func Excerpt ¶
Excerpt returns a section-aware excerpt of body within about maxTokens tokens (EstimateTokens). A body that fits is returned unchanged. Otherwise the body is split into sections at ATX headings (outside fenced code) and kept in document order by priority: the head (the text before the first heading plus the first section), then every section containing one of terms (case-insensitive), then the remaining sections, while the budget allows. Each run of omitted sections is replaced by `[... N sections omitted ...]`; a head section larger than the whole budget is cut and ends with `[... truncated ...]`. The tail is never dropped without a marker.
func Head ¶
Head flattens wikilinks (`[[a|b]]` → `b`, `[[a]]` → `a`) and returns at most maxChars runes of the trimmed result.
func IsMergedListKey ¶
IsMergedListKey reports whether kb merges new items into a user-owned value of key instead of leaving it alone (`aliases` and the relation lists).
func ListPaths ¶
ListPaths renders document paths for a run summary: at most limit of them, comma-separated, then "(+N more)" when some were left out.
func MatchGlob ¶
MatchGlob reports whether the slash path p matches pattern. Matching is segment-aware: `*`, `?` and `[...]` match within one path segment (as in path.Match) and a `**` segment matches zero or more whole segments, so `raw/audit-*/**` matches every file below any `raw/audit-...` folder. Leading `./` and surrounding slashes are ignored on both sides; a malformed pattern matches nothing.
func Tokenize ¶
Tokenize lowercases text, splits it on every rune that is not a Unicode letter or digit, drops stopwords and keeps tokens of at least 2 runes.
func Unclassified ¶
Unclassified reports whether doc needs a (re)classification: it has no state row, or any bank in banks was recorded with another version, judged on another body, or judged under another contract hash (when contract is non-empty). Writes by other commands never refresh these per-bank records, so a body edit followed by `kb link` still leaves doc unclassified.
func ValueHash ¶
ValueHash is the sha256 hex digest of the canonical JSON of the normalized value: strings are trimmed, lists keep their order ([]any of strings hashes like []string), numbers encode as JSON numbers (int 2 and float 2.0 hash alike), maps encode with sorted keys and times as their frontmatter text (`2006-01-02` at midnight, RFC 3339 otherwise). A value written by kb and the same value parsed back from YAML therefore hash identically.
Types ¶
type BM25 ¶
type BM25 struct {
// contains filtered or unexported fields
}
BM25 is an in-memory BM25F index over documents: each named field has its own length normalization and a weight applied to its term frequency.
func NewBM25 ¶
func NewBM25(docs []*Document, fields func(*Document) map[string]string, weights map[string]float64) *BM25
NewBM25 indexes docs. fields extracts the text of every named field of a document; weights gives each field's weight (fields missing from weights weigh 1). Parameters: k1 = 1.2, b = 0.75.
type Corpus ¶
type Corpus struct {
// Root is the topic root directory.
Root string
// Skipped lists documents skipped because they could not be read or
// their frontmatter did not parse.
Skipped []Skip
// contains filtered or unexported fields
}
Corpus is the loaded set of topic documents.
func Load ¶
func Load(topicRoot string, opts LoadOptions) (*Corpus, error)
Load reads the topic at topicRoot. Sources are `raw/**/*.md` except `raw/_quarantine/**`, `raw/codebase/**` and documents whose `source_kind` starts with `codebase-`; articles are `wiki/concepts/**/*.md`. Hub files (`log.md`, `CLAUDE.md`, `AGENTS.md`, `wiki/index/**`, `outputs/**`, `.decisions/**`, `bases/**`) fall outside both trees and dot directories are never walked, so neither is ever loaded. A document whose frontmatter does not parse is skipped and recorded in Corpus.Skipped; it never fails the load.
func (*Corpus) ByBodyHash ¶
ByBodyHash returns the documents whose body hash is h, sorted by path.
func (*Corpus) ByPath ¶
ByPath returns the document at the topic-relative path p (with or without `.md`), or nil.
type Dictionary ¶
type Dictionary struct {
// contains filtered or unexported fields
}
Dictionary is the alias dictionary of a topic's articles: each article's title and `aliases`.
func NewDictionary ¶
func NewDictionary(articles []*Document) *Dictionary
NewDictionary builds the dictionary over articles. Terms shorter than 3 runes are skipped unless they are all-caps acronyms of at least 2 runes; such short acronyms match case-sensitively, every other term case-insensitively.
func (*Dictionary) Mentions ¶
func (d *Dictionary) Mentions(text string) []Mention
Mentions scans text for dictionary terms at word boundaries, skipping leading frontmatter, fenced and inline code, heading lines, URLs, and existing wikilinks and markdown links. At each position the longest term wins; overlapping shorter terms are not reported. Mentions are returned in text order, one per article per occurrence.
type Document ¶
type Document struct {
// Path is the topic-relative slash path (with `.md`).
Path string
// AbsPath is the absolute filesystem path.
AbsPath string
// Title is the frontmatter title, else the first `# ` heading, else the
// file stem.
Title string
Kind Kind
// Frontmatter holds the parsed frontmatter values.
Frontmatter map[string]any
// Raw is the full file text as read.
Raw string
// Body is the text after the frontmatter.
Body string
// BodyHash is BodyHash(Body).
BodyHash string
// Aliases are the Obsidian `aliases` values.
Aliases []string
ModTime time.Time
}
Document is one loaded topic document.
func ReadDocument ¶
ReadDocument loads one topic document from disk. relative is the topic-relative path.
func (*Document) Provenance ¶
Provenance returns the document's provenance state (spec §4.4): `path` (topic-relative), `source_kind`, `source_host` (host of `source_url`, lowercased, `www.` stripped), `ingest_batch` and `ingest_query`. Empty fields are omitted, never guessed.
func (*Document) SourceKind ¶
SourceKind returns the `source_kind` key.
type LoadOptions ¶
type LoadOptions struct {
// Exclude lists topic-relative globs (see MatchGlob) of documents to skip.
Exclude []string
}
LoadOptions configures Load.
type Mention ¶
type Mention struct {
Article *Document
// Term is the dictionary term (title or alias) that matched.
Term string
// Start and End are byte offsets of the matched text in the scanned text.
Start int
End int
// Sentence is the sentence around the match, trimmed and bounded to about
// 300 characters.
Sentence string
}
Mention is one occurrence of an article title or alias in a body.
type StateMeta ¶
type StateMeta struct {
// Contract is the selection contract hash ("" keeps the row's value).
Contract string
// Banks maps question bank ids to versions; merged into the row. Each
// listed bank is also recorded as judged on the document's body (the one
// the write started from) under the contract.
Banks map[string]string
// Facts are per-document outcomes merged into the row; an empty value
// deletes the fact.
Facts map[string]string
}
StateMeta is the classification context recorded in the state row.
type StateRow ¶
type StateRow struct {
Path string `json:"path"`
BodyHash string `json:"body_hash"`
Contract string `json:"contract"`
Banks map[string]string `json:"banks"`
Written map[string]string `json:"written"`
BankBody map[string]string `json:"bank_body,omitempty"`
BankContract map[string]string `json:"bank_contract,omitempty"`
WrittenBody map[string]string `json:"written_body,omitempty"`
Facts map[string]string `json:"facts,omitempty"`
Updated string `json:"updated"`
}
StateRow is one row of `<topic>/.decisions/state.jsonl` (spec §6): the bookkeeping kb keeps per document instead of writing it to frontmatter.
BodyHash and Contract describe the document at kb's last write (rename detection follows BodyHash). Freshness is tracked per judgment instead: a write by one command (link, review, ingest) must not make another command's answers look current. BankBody and BankContract record, per question bank, the body and contract its answers were computed on; WrittenBody records, per owned key, the body its value was derived from; Facts holds small per-document outcomes other runs aggregate (for example the primary concept). Rows written before these maps existed read as if every bank and key was judged on BodyHash under Contract.
func (*StateRow) JudgedBody ¶
JudgedBody returns the body hash the answers of bank were computed on (BodyHash for rows that predate per-bank tracking).
func (*StateRow) JudgedContract ¶
JudgedContract returns the contract hash bank was judged under (Contract for rows that predate per-bank tracking).
type StateStore ¶
type StateStore struct {
// contains filtered or unexported fields
}
StateStore is the append-only document state store. The last row per path wins. It is safe for concurrent use.
func OpenState ¶
func OpenState(topicRoot string) (*StateStore, error)
OpenState reads `<topicRoot>/.decisions/state.jsonl`. A missing file is an empty store. A torn last line (a crash mid-append) is ignored; any other malformed line is an error.
func (*StateStore) Compact ¶
func (s *StateStore) Compact() error
Compact atomically rewrites the state file with only the latest row per path, sorted by path.
func (*StateStore) FindByBodyHash ¶
func (s *StateStore) FindByBodyHash(h string) []StateRow
FindByBodyHash returns the latest rows whose body hash is h, sorted by path.
func (*StateStore) Get ¶
func (s *StateStore) Get(path string) (*StateRow, bool)
Get returns the latest row for the topic-relative path.
func (*StateStore) Lookup ¶
Lookup returns the row of path. When path has no row it performs rename detection: the first row with the same body hash whose own path no longer exists (exists reports false) follows the file and is returned with Path set to path. The store itself is not modified.
func (*StateStore) Path ¶
func (s *StateStore) Path() string
Path returns the absolute path of the state file.
func (*StateStore) Put ¶
func (s *StateStore) Put(row StateRow) error
Put appends row as one whole JSON line and makes it the latest row for its path.
func (*StateStore) Rows ¶
func (s *StateStore) Rows() []StateRow
Rows returns the latest row of every path, sorted by path.
type WriteResult ¶
type WriteResult struct {
// Written lists the keys whose value changed on disk, sorted.
Written []string
// SkippedUserKeys lists keys left alone because they belong to the user
// (present and not last written by kb), sorted.
SkippedUserKeys []string
// BodyWritten reports whether ApplyBody replaced the body.
BodyWritten bool
// Status is StatusWritten, StatusUnchanged, StatusSkippedLocked or
// StatusSkippedChanged.
Status string
}
WriteResult reports what Apply or ApplyBody did.
type Writer ¶
type Writer struct {
// contains filtered or unexported fields
}
Writer applies kb-owned frontmatter keys (spec §6) to topic documents, byte-preserving everything else, and records each write in the state store. It is safe for concurrent use; writes to one document are serialized.
func NewWriter ¶
func NewWriter(topicRoot string, store *StateStore, now func() time.Time) *Writer
NewWriter returns a writer for the topic at topicRoot. now defaults to time.Now.
func (*Writer) Apply ¶
Apply writes updates (owned key → value; a nil value deletes the key) to doc. Rules, in order:
- any non-owned key is refused with an error;
- a document with `locked: true` is left alone (skipped:locked);
- the file is re-read: if its body changed since doc was loaded nothing is written (skipped:changed); other non-owned edits are fine because only owned key blocks are replaced in the fresh content;
- a key is written only if it is absent or its current value hash equals the state row's written hash (kb wrote it and nobody edited it since); otherwise it belongs to the user and is reported in SkippedUserKeys. `aliases` is merged instead (case-insensitive dedupe, never removes), even when user-owned, and so is a user-owned relation list (`related`, `extends`, `prerequisite`, `example_of`, `contradicts`, `affects`, `supersedes`): new targets are appended, deduplicated by normalized link target, and existing items are never removed or reordered;
- unchanged values are not rewritten;
- the file is replaced atomically (temp file + rename, mode kept), and only then the state row is appended with the new body hash, contract, merged banks and merged written hashes. The banks of meta, and every key that now holds the requested value, are recorded as derived from the body the write started from; judgments recorded earlier keep their own body hash, so a write never makes another command's answers look current. A body replacement that only adds or changes wikilink markup (link insertion) carries every judgment made on the old body over to the new one. If the file changed between the read and the rename, the whole step is retried once, then reported as skipped:changed.
On success doc is refreshed in place with the written content.
func (*Writer) ApplyBody ¶
func (w *Writer) ApplyBody(doc *Document, newBody string, updates map[string]any, meta StateMeta) (WriteResult, error)
ApplyBody is Apply plus a body replacement (link insertion, recapture). The body is replaced only while the file's body hash still equals doc.BodyHash; the same key rules apply to updates.
Source Files