knowledge

package
v0.7.58-dev.3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 12, 2026 License: MIT Imports: 21 Imported by: 0

Documentation

Overview

Package knowledge is the knowledge pack: an LLM wiki — compiled markdown pages with an index — together with the documents it was compiled from, at most 10 MB, that a run searches through the tools server's knowledge tools over a read-only /knowledge mount. Everything that makes, stores or attaches a pack reads the format from here: the runner (run --knowledge), the Creator's apply step, the local store, the desktop.

A pack is one directory:

<name>/
  manifest.json   name, id, dates, creator, schema, pages, bytes, the
                  tree's sha256, the scan record
  wiki/
    index.md      every page, one line each, by section (generated)
    log.md        append-only: every ingest, with the documents it took
    <section>/<slug>.md   the pages: YAML frontmatter, [src: doc:<path>]
  sources/        the documents as given, and their extracted text

The tree's digest is the SHA-256 of a canonical tar of wiki/ and sources/ (TreeDigest); it is what the receipt records.

Index

Constants

View Source
const (
	ChangesFile   = "changes.json"
	ChangesSchema = "knowledge/changes/v1"
)

ChangesFile is the Creator's account of one ingest, beside the files it names; ChangesSchema is its format.

View Source
const (
	ManifestFile = "manifest.json"
	WikiDir      = "wiki"
	SourcesDir   = "sources"
	IndexFile    = "index.md"
	LogFile      = "log.md"
)

The pack's fixed entries.

View Source
const MaxBytes = 10 << 20

MaxBytes caps a pack Agentry keeps or builds — every file in it, the manifest included: one uploaded, downloaded, shared, shipped in a release or made by a build In Agentry. Counted in MiB, the unit of the release archive's own cap (rule V8). It is also a build's cap when the caller names none.

View Source
const MaxFiles = 2000

MaxFiles caps the files in a pack's tree: the registry refuses a tree with more, so no client keeps one either.

View Source
const MaxLocalBytes = 1 << 30

MaxLocalBytes caps a pack made and kept on this computer, documents included: a local build may take up to 1 GB of documents. Such a pack past MaxBytes stays on this computer: Agentry takes none past MaxBytes.

View Source
const Schema = "knowledge/v1"

Schema is the pack format this package reads and writes.

View Source
const SectionIndexAbove = 300

SectionIndexAbove is the page count past which the index lists sections and each section gets an index of its own (past a few hundred pages it lists sections whose own index.md lists their pages), so the file a query starts from stays small.

Variables

View Source
var (
	// ErrTooLarge is a pack past its cap: MaxLocalBytes on this computer,
	// MaxBytes for Agentry or a build that names none.
	ErrTooLarge = errors.New("knowledge pack is past its size cap")
	// ErrDigestMismatch is a tree that no longer matches its manifest.
	ErrDigestMismatch = errors.New("knowledge pack's files do not match the digest in its manifest")
)

Functions

func CopyPack

func CopyPack(src, dst string) error

CopyPack copies the pack at src to dst, which must not exist: the manifest and the regular files under wiki/ and sources/, nothing else — a link, a device or a pipe is refused, never followed — and never more than MaxLocalBytes in all, so a pack changed after it was checked cannot pull anything else into the copy. Every read goes through an os.Root at src. The copy is for Load to verify; CopyPack itself checks no digest.

func CountPages

func CountPages(n int) string

CountPages is a page count as a person reads it: "1 page", "3 pages".

func FormatCap

func FormatCap(n int64) string

FormatCap is a cap as a person reads it: "10 MB", "1 GB".

func GenerateIndex

func GenerateIndex(dir, name string) (map[string][]byte, error)

GenerateIndex returns the index files the pack at dir should hold, by slash path relative to the pack: wiki/index.md, and each section's wiki/<section>/index.md once the pack is past SectionIndexAbove pages.

func Pages

func Pages(dir string) (int, error)

Pages counts the wiki's pages under dir: every .md file under wiki/ but the index files and the log.

func TreeFiles

func TreeFiles(dir string) ([]string, error)

TreeFiles lists the regular files of the pack at dir, under wiki/ and sources/, as sorted slash paths relative to it: what TreeDigest hashes and the knowledge tools serve.

func ValidName

func ValidName(name string) error

ValidName checks a pack name: 1-64 lowercase letters, digits, '.', '-' or '_', starting with a letter or a digit — one directory component on every filesystem, case-insensitive ones included.

func WriteIndex

func WriteIndex(dir, name string) error

WriteIndex writes GenerateIndex's files into the pack at dir and removes any section index it no longer calls for.

Types

type ApplyOptions

type ApplyOptions struct {
	// Name names a new pack; ignored when the pack exists.
	Name string
	// Now stamps the manifest and the log entry.
	Now time.Time
	// Creator is recorded as what last extended the pack.
	Creator Creator
	// MaxBytes caps the pack Apply makes, documents included: MaxBytes
	// when zero (a build In Agentry's, and any caller that names none);
	// MaxLocalBytes for a build on this computer.
	MaxBytes int64
}

ApplyOptions are the facts Apply cannot read from the output.

type Changes

type Changes struct {
	Schema string `json:"schema"`
	// Write names files, relative to the pack, under wiki/ or sources/,
	// each carried at the same relative path beside changes.json.
	Write []string `json:"write,omitempty"`
	// Delete names files, relative to the pack, to remove.
	Delete []string `json:"delete,omitempty"`
	// Documents are the input documents this ingest took, for the log.
	Documents []string `json:"documents,omitempty"`
	// Summary is one line for the log.
	Summary string `json:"summary,omitempty"`
}

Changes is changes.json.

type Creator

type Creator struct {
	CLI   string `json:"cli,omitempty"`
	Model string `json:"model,omitempty"`
}

Creator names what made or last extended a pack.

type Digest

type Digest struct {
	SHA256 string
	Bytes  int64
	Files  int
}

Digest is a tree's digest and size.

func ExtractTree

func ExtractTree(r io.Reader, dst, want string) (d Digest, err error)

ExtractTree reads a canonical tree tar — what WriteTree writes, and what a sealed pack opens to — from r into dst, and checks it is the tree want names. dst must not exist, or be an empty directory; it is made 0700, since what lands in it is a pack's words in the clear.

Every entry must be a regular file with a clean slash path under wiki/ or sources/, in byte order, each once; the whole tree at most MaxLocalBytes and MaxFiles. The stream's SHA-256 must be want, and the files written must re-encode to the same digest, so the bytes were the canonical tar of their files and nothing else ("one pack, one digest"). Every write goes through an os.Root at dst. A refused tree leaves nothing: a directory ExtractTree made is removed, and an empty one it was given is emptied again.

func TreeDigest

func TreeDigest(dir string) (Digest, error)

TreeDigest hashes a pack's wiki/ and sources/ as a canonical tar, so that any implementation can reproduce it: for every regular file, in byte order of its slash path relative to the pack (e.g. "sources/a.txt"), a POSIX ustar header — name, mode "0000644", uid and gid "0000000", size and mtime in octal (mtime 0), typeflag '0', magic "ustar\0", version "00", empty linkname, uname, gname and prefix, devmajor and devminor "0000000", each numeric field NUL-terminated, the checksum "%06o\0 " — preceded by a PAX extended header only when a name needs one (longer than 100 bytes, or not ASCII), then the data padded to 512 bytes; after the last file, two zero blocks and no record padding. This is exactly what archive/tar writes for these headers. Anything that is not a directory or a regular file is refused: a pack holds files, never links.

func WriteTree

func WriteTree(dir string, w io.Writer) (Digest, error)

WriteTree writes dir's wiki/ and sources/ to w as the canonical tar TreeDigest hashes — the plaintext a sealed pack holds (the core's pkg/packseal seals it) — and returns its digest.

type Drop

type Drop struct {
	Path   string
	Reason string
	// Kept is true for a delete kept back, and for a page kept with its
	// dead links unlinked.
	Kept bool
}

Drop is a change Apply did not make as written: a written file that failed the lint (reverted to what the pack held, or not added), a delete kept back because a page that stays needs the file, or a written page kept with its links to nothing in the pack made plain text.

type Finding

type Finding struct {
	// Page is the slash path relative to the pack, e.g. wiki/topics/x.md.
	Page string
	// Line is the 1-based line, or 0 for the file as a whole.
	Line int
	Rule Rule
	// Target is, for a link or a citation, the slash path relative to the
	// pack that it names and that is missing; empty otherwise.
	Target string
	// Href is, for a link to nothing in the pack, the link as the page
	// writes it, so the apply step can make it plain text.
	Href    string
	Message string
}

Finding is one lint failure.

func Lint

func Lint(dir, name string) ([]Finding, error)

Lint checks the pack at dir, whose name the index is generated under. The error is for I/O alone; every problem with the pack is a Finding.

func (Finding) String

func (f Finding) String() string

type Manifest

type Manifest struct {
	Schema  string    `json:"schema"`
	Name    string    `json:"name"`
	ID      string    `json:"id,omitempty"`
	Created time.Time `json:"created,omitzero"`
	Updated time.Time `json:"updated,omitzero"`
	Creator Creator   `json:"creator,omitzero"`
	// Pages counts the wiki's pages, index and log aside.
	Pages int `json:"pages"`
	// Bytes is the tree's size: every file under wiki/ and sources/.
	Bytes int64 `json:"bytes"`
	// SHA256 is TreeDigest's "sha256:<hex>".
	SHA256 string `json:"sha256"`
	Scan   Scan   `json:"scan"`
}

Manifest is manifest.json, in the clear beside the tree it describes.

type Pack

type Pack struct {
	Dir      string
	Manifest Manifest
}

Pack is a pack on disk whose manifest and tree agree.

func Load

func Load(dir string) (*Pack, error)

Load reads and verifies the pack at dir: the manifest parses and names this schema and a valid name, the layout is the pack's (manifest.json, wiki/ with its index and log, sources/, nothing else), the whole pack is within MaxLocalBytes, and the tree's digest is the manifest's. Agentry's smaller cap is checked where a pack goes to Agentry (the store's upload).

func Unpack

func Unpack(r io.Reader, want, dst string, m Manifest) (p *Pack, err error)

Unpack makes the pack directory dst from a canonical tree tar whose digest is want: the tree extracted and checked (ExtractTree), a manifest written from m with the tree's own schema, counts and digest in place of whatever m said, and the pack loaded. It is how a pack kept somewhere as a tree — a registry's copy downloaded, a sandbox host's staged tree for a hosted build — becomes a directory the Creator and the runner read. A refused tree or pack leaves nothing in dst.

type Result

type Result struct {
	Manifest Manifest
	Written  []string
	Deleted  []string
	Dropped  []Drop
}

Result is what one Apply did.

func Apply

func Apply(packDir, outDir string, o ApplyOptions) (*Result, error)

Apply applies the output in outDir to the pack at packDir, creating the pack when packDir holds none. Every path the output names is the model's, so each is held to changePath's rules and then read and written only through an os.Root — at outDir for what the output carries, at the staging directory for what the next pack holds — so no name, link or separator in it reaches a file outside either.

type Rule

type Rule string

Rule names what a finding is about.

const (
	RuleFrontmatter Rule = "frontmatter"
	RuleSlug        Rule = "slug"
	RuleLink        Rule = "link"
	RuleCitation    Rule = "citation"
	RuleIndex       Rule = "index"
	RuleLayout      Rule = "layout"
)

The lint's rules.

type Scan

type Scan struct {
	// Scanned is true when the manifest carries a registry's record.
	Scanned bool
	Report  json.RawMessage
	// Verified is set by code that confirmed the record with the registry
	// that wrote it; never read from or written to a file.
	Verified bool
}

Scan is the pack's scan record: "not scanned" for every pack no registry has scanned — every pack made on a machine — or the record the registry wrote, kept as written. A record in a manifest is only a claim: anyone can write one into a file, so it reads as scanned only once code that asked the registry has set Verified, which this client's local paths never do.

func (Scan) Describe

func (s Scan) Describe() string

Describe is the record as a receipt and a person read it: a record no one here has confirmed says so rather than claiming a scan.

func (Scan) MarshalJSON

func (s Scan) MarshalJSON() ([]byte, error)

MarshalJSON writes "not scanned" or the registry's record.

func (*Scan) UnmarshalJSON

func (s *Scan) UnmarshalJSON(b []byte) error

UnmarshalJSON reads either form; any other word is refused, so a record can never claim a scan by spelling.

Directories

Path Synopsis
Package build makes or extends a knowledge pack with the Knowledge Creator: it stages the documents, decodes them before any agent sees them (pkg/runner/decode), runs the built-in Creator over their text with the pack as it stands attached (pkg/runner/localrun, pkg/runner/engine), and applies the pages it wrote — with the documents themselves and their decoded text, which only this step, never the model, puts in the pack's sources/ — through pkg/knowledge's apply step.
Package build makes or extends a knowledge pack with the Knowledge Creator: it stages the documents, decodes them before any agent sees them (pkg/runner/decode), runs the built-in Creator over their text with the pack as it stands attached (pkg/runner/localrun, pkg/runner/engine), and applies the pages it wrote — with the documents themselves and their decoded text, which only this step, never the model, puts in the pack's sources/ — through pkg/knowledge's apply step.
Package creator is the Knowledge Creator: the QuickAgent package that compiles documents into a knowledge pack, embedded in the CLI and run by the runner as a built-in package — part of the application, never a release a person installs.
Package creator is the Knowledge Creator: the QuickAgent package that compiles documents into a knowledge pack, embedded in the CLI and run by the runner as a built-in package — part of the application, never a release a person installs.
Package knowledgetest makes knowledge packs for other packages' tests, through the format's own apply step, the way every pack a person attaches is made.
Package knowledgetest makes knowledge packs for other packages' tests, through the format's own apply step, the way every pack a person attaches is made.
Package remote is the client of a registry's knowledge routes: wrap and unwrap a pack's data key, create a pack, upload and download its sealed tree, list and delete, for the account a session token or a personal access token with knowledge:read and knowledge:write speaks for.
Package remote is the client of a registry's knowledge routes: wrap and unwrap a pack's data key, create a pack, upload and download its sealed tree, list and delete, for the account a session token or a personal access token with knowledge:read and knowledge:write speaks for.
Package ship lays a knowledge pack into a QuickAgent's package so the release carries it: the pack's files under knowledge/<name>/ with the manifest.json the registry holds a shipped pack to, the entry in the package's knowledge.ships and the patterns of publish.include that put it in the archive, and the sidecar that records where the pack was laid down from.
Package ship lays a knowledge pack into a QuickAgent's package so the release carries it: the pack's files under knowledge/<name>/ with the manifest.json the registry holds a shipped pack to, the entry in the package's knowledge.ships and the patterns of publish.include that put it in the archive, and the sidecar that records where the pack was laid down from.
Package store is the local knowledge store: the knowledge packs a person keeps on this machine, sealed, under the runner home the CLI and the desktop share, so that both clients list, open, attach and export the same packs.
Package store is the local knowledge store: the knowledge packs a person keeps on this machine, sealed, under the runner home the CLI and the desktop share, so that both clients list, open, attach and export the same packs.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL