Documentation
¶
Overview ¶
Package artifact defines lore's portable corpus format: a single, versioned, self-contained byte stream holding one collection — its embedding-space and chunker pins, metadata, documents, chunks, and vectors — so a collection can be exported to a file and reconstructed elsewhere with its invariants intact.
The stream is framed: an 8-byte magic, a 4-byte big-endian format version, then a gob-encoded Bundle. Framing the version lets an importer reject a newer format with a clear error instead of a gob failure. The wire types are deliberately decoupled from the domain types (their own plain structs) so the format stays stable across domain refactors; the export/import use cases map between the two.
Index ¶
Constants ¶
const ( // Magic prefixes every artifact, distinguishing it from random bytes and from // an age-encrypted stream (which carries its own header). Magic = "LORECORP" // FormatVersion is the current frame version. Bump it when the Bundle wire // shape changes incompatibly; Read rejects anything newer. FormatVersion uint32 = 1 )
Variables ¶
var ( // ErrBadFormat means the stream is not a lore artifact (wrong or short magic). ErrBadFormat = errors.New("not a lore artifact") // ErrUnsupportedVersion means the artifact's format version is newer than this // binary understands. ErrUnsupportedVersion = errors.New("unsupported artifact format version") )
Functions ¶
Types ¶
type Bundle ¶
type Bundle struct {
Collection Collection
Documents []Document
}
Bundle is a whole exported collection.
type Chunk ¶
Chunk is one stored chunk with its embedding vector. The chunk and document IDs are not stored — they are deterministically re-derived from the (target collection, source URI, seq) on import, which keeps the artifact rename-safe.
type Chunker ¶
type Chunker struct {
Strategy string
Version int
Size int
Overlap int
Tokenizer string
ContextPrefix bool
}
Chunker is the serialized chunker pin (mirrors domain.ChunkerSpec).
type Collection ¶
type Collection struct {
Name string
Model string
Dimensions int
CreatedAt time.Time
Sources []string
Chunker Chunker
}
Collection carries the collection's identity, metadata, and the pins that must travel with it so the reconstructed collection enforces the same invariants. A zero Chunker means the source collection predated chunker pinning.
type Document ¶
type Document struct {
SourceURI string
Hash string
IngestedAt time.Time
Fingerprint string
Metadata map[string]string
Chunks []Chunk
}
Document is one source document and its chunks (with their vectors), enough to reconstruct the document, its chunks, and its index entries on import. Metadata is an additive field: gob ignores it when read by a binary that predates it, and it decodes to nil from an artifact written without it, so no format-version bump is required (the version gates only incompatible shape changes).
Source Files