Documentation
¶
Overview ¶
Package etl provides explicit extract-transform-load building blocks for documents.
Format readers extract source data into core document values. Formatters, splitting, identifier assignment, and batching transform those values for a downstream index. A Splitter applies one TextSplitter policy, such as SeparatorSplitter, TokenSplitter, or markdown.TextSplitter, to documents and owns chunk metadata and lineage for every policy. Named readers bind the source and revision to the original bytes in document.Provenance. Splitting appends a chunk to its parent's path, leaving business metadata untouched. Byte positions are present only where a reader can prove an exact half-open interval in the original input.
github.com/Tangerg/scope/etl/text.FileWriter is a filesystem load target; semantic indexing composes ETL output with rag/index.Writer.
The base module owns the root package plus the text, JSON, and Markdown packages. HTML and PDF are separate leaf modules because their parser dependencies are materially larger.
Index ¶
- Constants
- Variables
- type IDAssigner
- type IDAssignerConfig
- type IDGenerator
- type MetadataHeaderFormatter
- type MetadataHeaderFormatterConfig
- type SHA256IDGenerator
- type SeparatorSplitter
- type SeparatorSplitterConfig
- type SourceBudget
- type Splitter
- type SplitterConfig
- type TextSplitter
- type TextSplitterFunc
- type TokenCountBatcher
- type TokenCountBatcherConfig
- type TokenSplitter
- type TokenSplitterConfig
- type UUIDGenerator
Examples ¶
Constants ¶
const (
// DefaultMaxSearchWork bounds token-window search per chunk.
DefaultMaxSearchWork = 8 << 20
)
const DefaultMaxSourceBytes int64 = 32 * 1024 * 1024
Variables ¶
var ( ErrInvalidSourceBudget = errors.New("etl: invalid source budget") ErrNilSource = errors.New("etl: source must not be nil") // ErrSourceConsumed means this stream reader has already attempted its one // read. Reopen the complete input to retry; the current offset is unknown. ErrSourceConsumed = errors.New("etl: source stream already consumed") // ErrSourceTooLarge reports that no partial payload is returned. ErrSourceTooLarge = errors.New("etl: source exceeds byte budget") )
var ( ErrInvalidTextEncoding = errors.New("etl: invalid text encoding") ErrUnsupportedDocumentMedia = errors.New("etl: text splitting does not support document media") )
var ErrChunkBudgetTooSmall = errors.New("etl: chunk token budget is too small")
ErrChunkBudgetTooSmall means no nonempty trimmed source prefix fits the token budget.
var ErrChunkLimitExceeded = errors.New("etl: chunk limit exceeded")
ErrChunkLimitExceeded reports that splitting would exceed the configured chunk count.
var ErrNilDocument = errors.New("etl: document must not be nil")
var ErrSearchBudgetExceeded = errors.New("etl: token window search budget exceeded")
ErrSearchBudgetExceeded means prefix search stopped before proving whether a nonempty chunk fits.
Functions ¶
This section is empty.
Types ¶
type IDAssigner ¶
type IDAssigner struct {
// contains filtered or unexported fields
}
IDAssigner assigns identifiers to independent copies of input documents. Caller-owned documents, metadata, and media remain untouched.
func NewIDAssigner ¶
func NewIDAssigner(config IDAssignerConfig) (*IDAssigner, error)
func (*IDAssigner) Assign ¶
func (i *IDAssigner) Assign(ctx context.Context, docs []*document.Document) ([]*document.Document, error)
Assign validates and clones every document before assigning IDs. It returns no partial output on failure and never mutates the input slice or documents. Generated identities must satisfy the canonical document contract.
type IDAssignerConfig ¶
type IDAssignerConfig struct {
Generator IDGenerator
// Overwrite replaces existing IDs instead of preserving them.
Overwrite bool
}
type IDGenerator ¶
type IDGenerator interface {
// Generate returns a non-blank UTF-8 identifier without surrounding whitespace
// for one document without mutating it. Content-addressed implementations must
// be deterministic; random implementations must still honor ctx and reject nil
// documents.
Generate(ctx context.Context, document *document.Document) (string, error)
}
IDGenerator produces an identifier for a document. Implementations may derive identity from content or generate an unconditional random identity.
type MetadataHeaderFormatter ¶ added in v0.41.0
type MetadataHeaderFormatter struct {
// contains filtered or unexported fields
}
MetadataHeaderFormatter renders a *document.Document as
key1: value1 key2: value2 <document text>
Example ¶
package main
import (
"fmt"
"github.com/Tangerg/scope/core/document"
"github.com/Tangerg/scope/etl"
)
func main() {
doc, err := document.NewDocument("Scope keeps document text provider-neutral.", nil)
if err != nil {
panic(err)
}
formatted, err := etl.NewMetadataHeaderFormatter(etl.MetadataHeaderFormatterConfig{}).Format(doc)
if err != nil {
panic(err)
}
fmt.Println(formatted)
}
Output: Scope keeps document text provider-neutral.
func NewMetadataHeaderFormatter ¶ added in v0.41.0
func NewMetadataHeaderFormatter(config MetadataHeaderFormatterConfig) MetadataHeaderFormatter
NewMetadataHeaderFormatter snapshots formatting policy into an immutable formatter.
func (MetadataHeaderFormatter) Format ¶ added in v0.41.0
func (m MetadataHeaderFormatter) Format(doc *document.Document) (string, error)
Format sorts metadata keys because map order would make rendered text, and therefore embeddings and token counts, nondeterministic. Without rendered metadata the output is the document text alone. Media cannot be rendered as text and returns document.ErrUnsupportedMedia.
type MetadataHeaderFormatterConfig ¶ added in v0.41.0
type MetadataHeaderFormatterConfig struct {
// ExcludedMetadata lists metadata keys omitted from rendered output.
ExcludedMetadata []string
}
type SHA256IDGenerator ¶
type SHA256IDGenerator struct {
// contains filtered or unexported fields
}
SHA256IDGenerator builds a content-addressable identifier from a document's text, media, metadata and known provenance. The existing ID is deliberately excluded so the digest remains stable before and after assignment.
func NewSHA256IDGenerator ¶
func NewSHA256IDGenerator(salt []byte) SHA256IDGenerator
NewSHA256IDGenerator snapshots salt so identical documents retain stable IDs without sharing caller-owned bytes.
type SeparatorSplitter ¶ added in v0.60.0
type SeparatorSplitter struct {
// contains filtered or unexported fields
}
SeparatorSplitter is the TextSplitter that breaks text on a fixed separator.
func NewSeparatorSplitter ¶ added in v0.60.0
func NewSeparatorSplitter(config SeparatorSplitterConfig) (*SeparatorSplitter, error)
type SeparatorSplitterConfig ¶ added in v0.60.0
type SeparatorSplitterConfig struct {
Separator string
}
SeparatorSplitterConfig configures fixed-separator chunking. The zero Separator uses a newline.
type SourceBudget ¶
type SourceBudget struct {
// contains filtered or unexported fields
}
SourceBudget is the shared memory-safety contract for whole-source readers. Its zero value uses DefaultMaxSourceBytes. A custom budget must be created with NewSourceBudget, so readers never have an implicit unlimited mode.
func NewSourceBudget ¶
func NewSourceBudget(maxBytes int64) (SourceBudget, error)
func (SourceBudget) MaxBytes ¶
func (s SourceBudget) MaxBytes() int64
func (SourceBudget) ReadAll ¶
ReadAll consumes at most MaxBytes plus one detection byte. It returns no partial payload when the source exceeds the budget and preserves source and context errors for errors.Is/errors.As. Cancellation stops subsequent reads; interruption of an in-flight Read remains the source's responsibility.
Example ¶
package main
import (
"context"
"fmt"
"strings"
"github.com/Tangerg/scope/etl"
)
func main() {
budget, err := etl.NewSourceBudget(16)
if err != nil {
panic(err)
}
data, err := budget.ReadAll(context.Background(), strings.NewReader("bounded input"))
if err != nil {
panic(err)
}
fmt.Println(string(data), budget.MaxBytes())
}
Output: bounded input 16
type Splitter ¶
type Splitter struct {
// contains filtered or unexported fields
}
Splitter applies a TextSplitter to documents, clones source metadata onto every chunk, and records chunk lineage. Documents containing media are rejected because a text policy cannot assign media to chunks. Chunks have no ID; an IDAssigner identifies them, including before a nested split.
func NewSplitter ¶
func NewSplitter(config SplitterConfig) (*Splitter, error)
type SplitterConfig ¶
type SplitterConfig struct {
// TextSplitter is required.
TextSplitter TextSplitter
}
type TextSplitter ¶
TextSplitter is the atomic policy a Splitter applies to each document's text: it owns where chunks break, while Splitter owns metadata and lineage.
type TextSplitterFunc ¶ added in v0.60.0
type TokenCountBatcher ¶
type TokenCountBatcher struct {
// contains filtered or unexported fields
}
TokenCountBatcher preserves document order and rejects any document that exceeds the batch token budget. Split oversized documents before batching.
func NewTokenCountBatcher ¶
func NewTokenCountBatcher(config TokenCountBatcherConfig) (*TokenCountBatcher, error)
type TokenCountBatcherConfig ¶
type TokenCountBatcherConfig struct {
Counter tokenizer.TextCounter
// MaxTokens is the required provider input limit. The batching layer has no
// provider-neutral default because model limits differ.
MaxTokens int
// Reserve is the fraction of MaxTokens held back from each batch. Zero
// means no reserve.
Reserve float64
// Formatter renders each document before counting. Nil counts the
// document text an index embeds; media contributes no text tokens.
Formatter document.Formatter
}
type TokenSplitter ¶
type TokenSplitter struct {
// contains filtered or unexported fields
}
TokenSplitter is the TextSplitter that breaks text into token-bounded chunks and prefers a sentence boundary once the configured minimum token count has been reached.
func NewTokenSplitter ¶
func NewTokenSplitter(config TokenSplitterConfig) (*TokenSplitter, error)
type TokenSplitterConfig ¶
type TokenSplitterConfig struct {
Tokenizer tokenizer.Tokenizer
MaxTokensPerChunk int
MinTokensPerChunk int
MaxChunks int
// MaxSearchWork bounds bytes rendered or encoded and tokens decoded per
// prefix search. Each operation costs at least one; zero uses DefaultMaxSearchWork.
MaxSearchWork int
PreserveNewlines bool
}
TokenSplitterConfig configures token-aware chunking. Zero sizing values use documented defaults; negative values are rejected.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
html
module
|
|
|
internal
|
|
|
origin
Package origin binds each reader's bounded input snapshot to its original revision.
|
Package origin binds each reader's bounded input snapshot to its original revision. |
|
tokenwindow
Package tokenwindow preserves source text when vocabulary tokens split UTF-8.
|
Package tokenwindow preserves source text when vocabulary tokens split UTF-8. |
|
Package json reads JSON values into documents.
|
Package json reads JSON values into documents. |
|
Package markdown extracts and transforms Markdown documents.
|
Package markdown extracts and transforms Markdown documents. |
|
pdf
module
|
|
|
Package text reads plain text into documents and writes documents to text files.
|
Package text reads plain text into documents and writes documents to text files. |