Documentation
¶
Overview ¶
Package etl provides explicit extract-transform-load building blocks for documents.
Format readers extract source data into core document values. Formatters, splitters, identifier assignment, and batching transform those values for a downstream index. github.com/Tangerg/scope/etl/text.FileWriter provides a concrete filesystem load target; vector-store loading remains owned by core/vectorstore capabilities. The root package owns format-independent document processing; format packages own source decoding and concrete output lifecycles.
The base module owns the root package plus text, JSON, and Markdown packages. HTML and PDF remain optional leaf modules because their parser dependencies are materially larger. The core document package remains a serializable data contract and does not depend on ETL.
Index ¶
- Constants
- Variables
- type Formatter
- type FormatterFunc
- type IDAssigner
- type IDAssignerConfig
- type IDGenerator
- type SHA256IDGenerator
- type SimpleFormatter
- type SimpleFormatterConfig
- type SourceBudget
- type Splitter
- type SplitterConfig
- type TextFormatter
- type TextSplitter
- type TextSplitterConfig
- type TokenCountBatcher
- type TokenCountBatcherConfig
- type TokenSplitter
- type TokenSplitterConfig
- type UUIDGenerator
Examples ¶
Constants ¶
const ( // MetadataKeyParentID holds the source document's ID. It is omitted when // the source document has no ID. MetadataKeyParentID = "parent_document_id" // MetadataKeyChunkIndex holds the zero-based position among emitted chunks. MetadataKeyChunkIndex = "chunk_index" // MetadataKeyChunkTotal holds the number of chunks emitted for the source. MetadataKeyChunkTotal = "chunk_total" )
Chunk-lineage metadata keys stamped by Splitter on every emitted chunk.
const (
// DefaultMaxSearchWork bounds token-window search per chunk.
DefaultMaxSearchWork = 8 << 20
)
const DefaultMaxSourceBytes int64 = 32 * 1024 * 1024
DefaultMaxSourceBytes is the bounded zero-value policy for whole-source readers.
Variables ¶
var ( // ErrInvalidSourceBudget identifies a non-positive or unrepresentable bound. ErrInvalidSourceBudget = errors.New("etl: invalid source budget") // ErrNilSource rejects an absent reader before attempting extraction. ErrNilSource = errors.New("etl: source must not be nil") // ErrSourceTooLarge reports that no partial payload is returned. ErrSourceTooLarge = errors.New("etl: source exceeds byte budget") )
var ( // ErrInvalidTextEncoding identifies document text that cannot be split as // valid UTF-8. ErrInvalidTextEncoding = errors.New("etl: invalid text encoding") // ErrUnsupportedDocumentMedia rejects media whose relationship to text // chunks cannot be represented by a text splitting policy. ErrUnsupportedDocumentMedia = errors.New("etl: text splitting does not support document media") )
var ErrChunkBudgetTooSmall = errors.New("etl: chunk token budget is too small")
ErrChunkBudgetTooSmall means no nonempty trimmed source prefix fits the token budget.
var ErrChunkLimitExceeded = errors.New("etl: chunk limit exceeded")
ErrChunkLimitExceeded prevents token splitting from producing an unbounded number of documents.
var ErrNilDocument = errors.New("etl: document must not be nil")
ErrNilDocument rejects absent content at transformation boundaries.
var ErrSearchBudgetExceeded = errors.New("etl: token window search budget exceeded")
ErrSearchBudgetExceeded means prefix search stopped before proving whether a nonempty chunk fits. It is distinct from ErrChunkBudgetTooSmall.
Functions ¶
This section is empty.
Types ¶
type Formatter ¶
type Formatter interface {
// Format renders one valid document under the receiver's frozen policy. It
// must not mutate or retain the document and must be deterministic so the
// same pipeline input produces stable downstream chunks and identities.
Format(document *document.Document) (string, error)
}
Formatter renders a document according to one frozen formatting policy. Consumers that need different representations should use independently configured formatters instead of passing consumer-specific modes per call.
type FormatterFunc ¶
FormatterFunc adapts a pure document projection to Formatter.
type IDAssigner ¶
type IDAssigner struct {
// contains filtered or unexported fields
}
IDAssigner assigns identifiers to independent copies of input documents. Caller-owned documents, metadata, and media remain untouched.
func NewIDAssigner ¶
func NewIDAssigner(config IDAssignerConfig) (*IDAssigner, error)
NewIDAssigner rejects a missing generator rather than choosing ambient identity policy.
type IDAssignerConfig ¶
type IDAssignerConfig struct {
// Generator is required.
Generator IDGenerator
// Overwrite replaces existing IDs instead of preserving them.
Overwrite bool
}
IDAssignerConfig binds one deterministic or random identity policy to documents that do not already own an ID.
type IDGenerator ¶
type IDGenerator interface {
// Generate returns a non-blank identifier for one document without mutating
// it. Content-addressed implementations must be deterministic; random
// implementations must still honor ctx and reject nil documents.
Generate(ctx context.Context, document *document.Document) (string, error)
}
IDGenerator produces an identifier for a document. Implementations may derive identity from content or generate an unconditional random identity.
type SHA256IDGenerator ¶
type SHA256IDGenerator struct {
// contains filtered or unexported fields
}
SHA256IDGenerator builds a content-addressable identifier from a document's text, media, and metadata. The existing ID is deliberately excluded so the digest remains stable before and after assignment.
func NewSHA256IDGenerator ¶
func NewSHA256IDGenerator(salt []byte) SHA256IDGenerator
NewSHA256IDGenerator snapshots salt so identical documents retain stable IDs without sharing caller-owned bytes.
type SimpleFormatter ¶
type SimpleFormatter struct {
// contains filtered or unexported fields
}
SimpleFormatter renders a *document.Document as
key1: value1 key2: value2 <document text>
Metadata keys can be excluded when constructing the formatter. Use independently configured formatters when different consumers need different representations.
Example:
f := etl.NewSimpleFormatter(etl.SimpleFormatterConfig{
ExcludedMetadata: []string{"row_id", "internal"},
})
Example ¶
package main
import (
"fmt"
"github.com/Tangerg/scope/core/document"
"github.com/Tangerg/scope/etl"
)
func main() {
doc, err := document.NewDocument("Scope keeps document text provider-neutral.", nil)
if err != nil {
panic(err)
}
formatted, err := etl.NewSimpleFormatter(etl.SimpleFormatterConfig{}).Format(doc)
if err != nil {
panic(err)
}
fmt.Println(formatted)
}
Output: Scope keeps document text provider-neutral.
func NewSimpleFormatter ¶
func NewSimpleFormatter(config SimpleFormatterConfig) SimpleFormatter
NewSimpleFormatter snapshots formatting policy into an immutable formatter.
func (SimpleFormatter) Format ¶
func (s SimpleFormatter) Format(doc *document.Document) (string, error)
Format renders doc by emitting filtered metadata as `key: value` lines (sorted by key — map iteration order would make the rendered text, and thus embedding inputs and token counts, non-deterministic) followed by a blank line and the document text. With no metadata (filtered empty), the output is just doc.Text — no leading newlines.
type SimpleFormatterConfig ¶
type SimpleFormatterConfig struct {
// ExcludedMetadata lists metadata keys omitted from rendered output.
ExcludedMetadata []string
}
SimpleFormatterConfig controls the stable textual projection used before splitting or indexing.
type SourceBudget ¶
type SourceBudget struct {
// contains filtered or unexported fields
}
SourceBudget is the shared memory-safety contract for whole-source readers. Its zero value uses DefaultMaxSourceBytes. A custom budget must be created with NewSourceBudget, so readers never have an implicit unlimited mode.
func NewSourceBudget ¶
func NewSourceBudget(maxBytes int64) (SourceBudget, error)
NewSourceBudget makes a custom whole-source memory bound explicit.
func (SourceBudget) MaxBytes ¶
func (s SourceBudget) MaxBytes() int64
func (SourceBudget) ReadAll ¶
ReadAll consumes at most MaxBytes plus one detection byte. It returns no partial payload when the source exceeds the budget and preserves source and context errors for errors.Is/errors.As. Cancellation stops subsequent reads; interruption of an in-flight Read remains the source's responsibility.
Example ¶
package main
import (
"context"
"fmt"
"strings"
"github.com/Tangerg/scope/etl"
)
func main() {
budget, err := etl.NewSourceBudget(16)
if err != nil {
panic(err)
}
data, err := budget.ReadAll(context.Background(), strings.NewReader("bounded input"))
if err != nil {
panic(err)
}
fmt.Println(string(data), budget.MaxBytes())
}
Output: bounded input 16
type Splitter ¶
type Splitter struct {
// contains filtered or unexported fields
}
Splitter applies a text splitting policy to documents, clones source metadata onto every chunk, and records chunk lineage. Documents containing media are rejected because a text policy cannot assign media to chunks.
func NewSplitter ¶
func NewSplitter(config SplitterConfig) (*Splitter, error)
NewSplitter rejects missing policy and snapshots the optional ID boundary.
type SplitterConfig ¶
type SplitterConfig struct {
// SplitFunc is required and owns the text splitting policy.
SplitFunc func(context.Context, string) ([]string, error)
// IDGenerator, when set, assigns an ID to each emitted chunk.
IDGenerator IDGenerator
}
SplitterConfig separates chunking policy from optional identity assignment.
type TextFormatter ¶
type TextFormatter struct{}
TextFormatter renders only document text. Its zero value is ready to use.
type TextSplitter ¶
type TextSplitter struct {
// contains filtered or unexported fields
}
TextSplitter splits text on a fixed separator and enriches document chunks with the same lineage behavior as Splitter.
func NewTextSplitter ¶
func NewTextSplitter(config TextSplitterConfig) (*TextSplitter, error)
NewTextSplitter treats the separator as a literal boundary and preserves the configured identity policy for emitted chunks.
type TextSplitterConfig ¶
type TextSplitterConfig struct {
Separator string
// IDGenerator, when set, assigns an ID to every emitted chunk.
IDGenerator IDGenerator
}
TextSplitterConfig configures fixed-separator chunking. The zero Separator uses a newline.
type TokenCountBatcher ¶
type TokenCountBatcher struct {
// contains filtered or unexported fields
}
TokenCountBatcher carves a document slice into batches that fit downstream embedding-service token limits. Document order is preserved across batches so callers can map embeddings back by position.
A single document whose token count exceeds the per-batch budget is rejected with an error — the caller is expected to split it first (see TokenSplitter).
func NewTokenCountBatcher ¶
func NewTokenCountBatcher(config TokenCountBatcherConfig) (*TokenCountBatcher, error)
NewTokenCountBatcher validates budgets before any document reaches a load boundary.
type TokenCountBatcherConfig ¶
type TokenCountBatcherConfig struct {
// Counter is required.
Counter tokenizer.TextCounter
// MaxTokens is the required provider input limit. The batching layer has no
// provider-neutral default because model limits differ.
MaxTokens int
// Reserve is the fraction of MaxTokens held back from each batch. Zero
// means no reserve.
Reserve float64
// Formatter renders each document before counting. Nil uses document
// text without metadata.
Formatter Formatter
}
TokenCountBatcherConfig binds one tokenizer to hard per-document and per-batch token budgets.
type TokenSplitter ¶
type TokenSplitter struct {
// contains filtered or unexported fields
}
TokenSplitter splits document text into token-bounded chunks and prefers a sentence boundary once the configured minimum token count has been reached.
func NewTokenSplitter ¶
func NewTokenSplitter(config TokenSplitterConfig) (*TokenSplitter, error)
NewTokenSplitter validates token and chunk bounds before retaining the tokenizer.
func (*TokenSplitter) Split ¶
func (t *TokenSplitter) Split(ctx context.Context, docs []*document.Document) ([]*document.Document, error)
Split emits token-bounded document chunks with cloned metadata and lineage.
type TokenSplitterConfig ¶
type TokenSplitterConfig struct {
Tokenizer tokenizer.Tokenizer
MaxTokensPerChunk int
MinTokensPerChunk int
MaxChunks int
// MaxSearchWork bounds bytes rendered or encoded and tokens decoded per
// prefix search. Each operation costs at least one; zero uses DefaultMaxSearchWork.
MaxSearchWork int
PreserveNewlines bool
IDGenerator IDGenerator
}
TokenSplitterConfig configures token-aware chunking. Zero sizing values use documented defaults; negative values are rejected.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
html
module
|
|
|
internal
|
|
|
tokenwindow
Package tokenwindow preserves source text when vocabulary tokens split UTF-8.
|
Package tokenwindow preserves source text when vocabulary tokens split UTF-8. |
|
Package json reads JSON values into documents.
|
Package json reads JSON values into documents. |
|
Package markdown extracts and transforms Markdown documents.
|
Package markdown extracts and transforms Markdown documents. |
|
pdf
module
|
|
|
Package text reads plain text into documents and writes documents to text files.
|
Package text reads plain text into documents and writes documents to text files. |