Documentation
¶
Overview ¶
Package document defines the canonical serializable Document content value shared by extraction, retrieval, and model-facing components.
NewDocument requires text, media, or both. Metadata is JSON-safe and belongs to the document itself; query-specific relevance belongs to the vector-store match value. Clone provides an independent snapshot when ownership crosses a component boundary.
Formatter is the one contract for rendering a document as text, shared by etl pipelines and rag context assembly. TextFormatter renders the text alone and rejects media it cannot carry. Extraction, splitting, identifier assignment, batching, and loading policy live in the separate etl module.
Example ¶
package main
import (
"fmt"
"github.com/Tangerg/scope/core/document"
)
func main() {
doc, err := document.NewDocument("Scope are wild cats.", nil)
if err != nil {
panic(err)
}
doc.ID = "doc-1"
if setErr := doc.Metadata.Set("source", "field-guide"); setErr != nil {
panic(setErr)
}
source, _, err := doc.Metadata.Decode[string]("source")
if err != nil {
panic(err)
}
fmt.Println(doc.ID, source)
}
Output: doc-1 field-guide
Index ¶
- Constants
- Variables
- func ValidateStoredMetadata(facts metadata.Map) error
- type ByteRange
- type Chunk
- type Document
- func (d *Document) Clone() *Document
- func (Document) JSONSchemaModel() any
- func (d Document) MarshalJSON() ([]byte, error)
- func (d *Document) MarshalStoredMetadata() ([]byte, error)
- func (d *Document) UnmarshalJSON(data []byte) error
- func (d *Document) UnmarshalStoredMetadata(data []byte) error
- func (d *Document) Validate() error
- type Formatter
- type FormatterFunc
- type Provenance
- type SourceName
- type TextFormatter
Examples ¶
Constants ¶
const StoredProvenanceKey = "_scope_document_provenance"
StoredProvenanceKey is reserved in the physical JSON metadata cell. It is never part of the searchable business metadata returned by a store.
Variables ¶
var ErrInvalidDocument = errors.New("document: invalid document")
var ErrUnsupportedMedia = errors.New("document: formatter does not support media")
ErrUnsupportedMedia reports a document whose media a text formatter cannot represent. Formatters return it instead of dropping the media silently.
var ErrUnversionedSource = errors.New("document: source revision is unknown")
ErrUnversionedSource distinguishes local material from a reproducible source.
Functions ¶
func ValidateStoredMetadata ¶ added in v0.60.0
ValidateStoredMetadata rejects business fields that would collide with reserved data in a shared physical metadata cell.
Types ¶
type ByteRange ¶ added in v0.59.0
ByteRange is a half-open interval in the original source bytes. A nil range means that a transform could not prove an interval in the original.
func (ByteRange) JSONSchemaModel ¶ added in v0.60.0
func (ByteRange) MarshalJSON ¶ added in v0.60.0
func (*ByteRange) UnmarshalJSON ¶ added in v0.60.0
type Chunk ¶ added in v0.59.0
type Chunk struct {
Parent string `json:"parent,omitempty"`
Index int `json:"index"`
Total int `json:"total"`
}
Chunk identifies an emitted part within its parent, after empty parts have been removed. Nested splitting appends a step instead of overwriting history.
func (Chunk) JSONSchemaModel ¶ added in v0.60.0
func (Chunk) MarshalJSON ¶ added in v0.60.0
func (*Chunk) UnmarshalJSON ¶ added in v0.60.0
type Document ¶
type Document struct {
ID string `json:"id,omitempty"`
Text string `json:"text,omitempty"`
Media *media.Media `json:"media,omitzero"`
Metadata metadata.Map `json:"metadata,omitzero"`
Provenance *Provenance `json:"provenance,omitzero"`
}
Document is the canonical content carrier. Clone recursively snapshots media and metadata so indexing and retrieval boundaries never retain caller-owned mutable buffers.
func NewDocument ¶
NewDocument requires text or media; their agreement remains the caller's responsibility.
func (Document) JSONSchemaModel ¶ added in v0.60.0
func (Document) MarshalJSON ¶ added in v0.13.0
func (*Document) MarshalStoredMetadata ¶ added in v0.60.0
MarshalStoredMetadata encodes the document's business facts and optional provenance in one native JSON cell. Native filter projections must still be built from d.Metadata, never by indexing this physical representation.
func (*Document) UnmarshalJSON ¶ added in v0.13.0
func (*Document) UnmarshalStoredMetadata ¶ added in v0.60.0
UnmarshalStoredMetadata restores both fields without changing the document's identity or content. Invalid native data leaves the document untouched. The raw cell may separately be retained as the backend's conditional-write token.
type Formatter ¶ added in v0.41.0
type Formatter interface {
// Format renders one valid document without mutating or retaining it. It
// must be deterministic, because the text feeds chunk identities,
// embeddings, and model context. Media the policy cannot represent returns
// an error rather than being dropped.
Format(doc *Document) (string, error)
}
Formatter renders a document as text under one frozen policy; consumers needing different representations configure separate formatters.
type FormatterFunc ¶ added in v0.41.0
type Provenance ¶ added in v0.59.0
type Provenance struct {
Source string `json:"source,omitempty"`
Revision string `json:"revision,omitempty"`
Chunks []Chunk `json:"chunks,omitempty"`
Bytes *ByteRange `json:"bytes,omitzero"`
Page int `json:"page,omitzero"` // one-based; zero means unknown
}
Provenance describes where a document came from, independently of its store ID and searchable business metadata. Source and Revision identify one original; a relative chunk path can still be known for an anonymous in-memory input.
func NewProvenance ¶ added in v0.59.0
func NewProvenance(source string, data []byte) *Provenance
NewProvenance binds a named source to the digest of its original bytes. Reading the same name after an edit therefore creates a different revision.
func (*Provenance) Clone ¶ added in v0.59.0
func (p *Provenance) Clone() *Provenance
func (*Provenance) Excerpt ¶ added in v0.59.0
func (p *Provenance) Excerpt() *Provenance
Excerpt preserves only locations that apply to the entire submitted input. A remote segmenter or text transform cannot establish its original byte range.
func (*Provenance) RequireRevision ¶ added in v0.59.0
func (p *Provenance) RequireRevision() error
RequireRevision is the admission boundary for reproducible ingestion and source citations. Relative positions alone cannot identify an original.
func (*Provenance) Split ¶ added in v0.59.0
func (p *Provenance) Split(parent string, index, total int) *Provenance
Split adds a proven, relative position to the surviving source evidence.
func (*Provenance) Validate ¶ added in v0.59.0
func (p *Provenance) Validate() error
type SourceName ¶ added in v0.61.0
type SourceName string
SourceName can be admitted before reading the bytes that establish a revision. Its empty value denotes an anonymous source.
func (SourceName) Validate ¶ added in v0.61.0
func (s SourceName) Validate() error
type TextFormatter ¶ added in v0.41.0
type TextFormatter struct{}
TextFormatter renders the document text alone and rejects media with ErrUnsupportedMedia. Its zero value is ready to use.
Source Files