Documentation
¶
Overview ¶
Package document defines the canonical serializable Document content value shared by extraction, retrieval, and model-facing components.
NewDocument requires text, media, or both. Metadata is JSON-safe and belongs to the document itself; query-specific relevance belongs to the vector-store match value. Clone provides an independent snapshot when ownership crosses a component boundary.
Formatter is the one contract for rendering a document as text, shared by etl pipelines and rag context assembly. TextFormatter renders the text alone and rejects media it cannot carry. Extraction, splitting, identifier assignment, batching, and loading policy live in the separate etl module.
Example ¶
package main
import (
"fmt"
"github.com/Tangerg/scope/core/document"
)
func main() {
doc, err := document.NewDocument("Scope are wild cats.", nil)
if err != nil {
panic(err)
}
doc.ID = "doc-1"
if setErr := doc.Metadata.Set("source", "field-guide"); setErr != nil {
panic(setErr)
}
source, _, err := doc.Metadata.Decode[string]("source")
if err != nil {
panic(err)
}
fmt.Println(doc.ID, source)
}
Output: doc-1 field-guide
Index ¶
Examples ¶
Constants ¶
This section is empty.
Variables ¶
var ErrInvalidDocument = errors.New("document: invalid document")
var ErrUnsupportedMedia = errors.New("document: formatter does not support media")
ErrUnsupportedMedia reports a document whose media a text formatter cannot represent. Formatters return it instead of dropping the media silently.
Functions ¶
This section is empty.
Types ¶
type Document ¶
type Document struct {
ID string `json:"id,omitempty"`
Text string `json:"text,omitempty"`
Media *media.Media `json:"media,omitzero"`
Metadata metadata.Map `json:"metadata,omitzero"`
}
Document is the canonical content carrier. Clone recursively snapshots media and metadata so indexing and retrieval boundaries never retain caller-owned mutable buffers.
func NewDocument ¶
NewDocument requires text or media; their agreement remains the caller's responsibility.
func (Document) MarshalJSON ¶ added in v0.13.0
func (*Document) UnmarshalJSON ¶ added in v0.13.0
type Formatter ¶ added in v0.41.0
type Formatter interface {
// Format renders one valid document without mutating or retaining it. It
// must be deterministic, because the text feeds chunk identities,
// embeddings, and model context. Media the policy cannot represent returns
// an error rather than being dropped.
Format(doc *Document) (string, error)
}
Formatter renders a document as text under one frozen policy; consumers needing different representations configure separate formatters.
type FormatterFunc ¶ added in v0.41.0
type TextFormatter ¶ added in v0.41.0
type TextFormatter struct{}
TextFormatter renders the document text alone and rejects media with ErrUnsupportedMedia. Its zero value is ready to use.
Source Files