README
¶
document
doc := document.Document{
ID: "doc-1",
Format: document.FormatMarkdown,
Text: document.NormalizeText("\ufeff# Title\r\nBody"),
}
span := document.Span{StartByte: 0, EndByte: len(doc.Text)}
text, ok := span.Text(doc.Text)
document owns provider-neutral document value types: Document, Section,
Element, Span, Format, and Metadata.
Callers own parsing, source schemas, and semantic interpretation. This package only normalizes text and validates byte/rune offsets so downstream chunking, ingest, retrieval, provenance, and test fixtures can share one document shape.
Element records generic pre-chunk structure such as titles, narrative text,
list items, tables, code, page breaks, and unknown blocks. It stores the parser
strategy label and byte span, but it does not select or implement a parser.
Documentation
¶
Overview ¶
Package document defines provider-neutral document, section, and span primitives. It keeps parsing policy with callers and models normalized text, formats, metadata, and byte/rune offset conversion.
Package document defines provider-neutral document primitives.
Index ¶
- Variables
- func ByteOffsetForRune(text string, runeIndex int) (int, bool)
- func NormalizeText(text string) string
- func RuneOffsetForByte(text string, byteOffset int) (int, bool)
- type Document
- type Element
- type ElementKind
- type Format
- type Metadata
- type ParserStrategy
- type ReaderOption
- type Section
- type Span
Examples ¶
Constants ¶
This section is empty.
Variables ¶
var ( // ErrInputTooLarge indicates that reader input exceeded the configured byte limit. ErrInputTooLarge = errors.New("document input too large") // ErrInvalidUTF8 indicates that reader input was not valid UTF-8 text. ErrInvalidUTF8 = errors.New("document input is not valid UTF-8") )
Functions ¶
func ByteOffsetForRune ¶
ByteOffsetForRune returns the byte offset for a rune index.
func NormalizeText ¶
NormalizeText normalizes line endings and strips a UTF-8 BOM.
Types ¶
type Document ¶
type Document struct {
ID string
Title string
Format Format
Text string
Sections []Section
Metadata Metadata
}
Document is normalized text plus structural metadata.
func FromReader ¶ added in v0.11.0
FromReader reads bounded UTF-8 text and constructs a normalized document. It does not infer parsing behavior from the filename or format.
Example ¶
package main
import (
"fmt"
"strings"
"github.com/dotcommander/reliquary/document"
)
func main() {
doc, err := document.FromReader(
"doc-1",
strings.NewReader("\ufeffAlpha\r\nBeta"),
document.WithFilename("notes.txt"),
document.WithMetadata(map[string]string{"source": "example"}),
)
if err != nil {
panic(err)
}
fmt.Println(doc.ID, doc.Title, doc.Format)
fmt.Println(doc.Text)
}
Output: doc-1 notes.txt text Alpha Beta
type Element ¶
type Element struct {
ID string
Kind ElementKind
Strategy ParserStrategy
Span Span
Text string
Metadata Metadata
}
Element is a span-backed, provider-neutral unit detected before chunking.
func (Element) ElementText ¶
ElementText returns the element text, preferring the span when it is valid for source and falling back to Element.Text.
Example ¶
package main
import (
"fmt"
"github.com/dotcommander/reliquary/document"
)
func main() {
source := "# Title\n\nBody text"
element := document.Element{
ID: "el-1",
Kind: document.ElementKindTitle,
Strategy: document.ParserStrategyMarkdown,
Span: document.Span{StartByte: 2, EndByte: 7},
Text: "fallback title",
}
fmt.Println(element.Kind, element.ElementText(source))
}
Output: title Title
type ElementKind ¶
type ElementKind string
ElementKind identifies generic structure detected before chunking.
const ( ElementKindTitle ElementKind = "title" ElementKindNarrative ElementKind = "narrative_text" ElementKindListItem ElementKind = "list_item" ElementKindTable ElementKind = "table" ElementKindCode ElementKind = "code" ElementKindPageBreak ElementKind = "page_break" ElementKindUnknown ElementKind = "unknown" )
type Format ¶
type Format string
Format identifies a document or span format without owning parsing policy.
type ParserStrategy ¶
type ParserStrategy string
ParserStrategy labels the caller-owned parser strategy that produced an element. It is descriptive only and does not select parser behavior.
const ( ParserStrategyPlainText ParserStrategy = "plain_text" ParserStrategyMarkdown ParserStrategy = "markdown" ParserStrategyHTML ParserStrategy = "html" ParserStrategyPDF ParserStrategy = "pdf" ParserStrategyUnknown ParserStrategy = "unknown" )
type ReaderOption ¶ added in v0.11.0
type ReaderOption func(*readerConfig)
ReaderOption configures a document created by FromReader.
func WithFilename ¶ added in v0.11.0
func WithFilename(filename string) ReaderOption
WithFilename sets the document title. It does not select a parser or infer a format from the filename.
func WithFormat ¶ added in v0.11.0
func WithFormat(format Format) ReaderOption
WithFormat sets the document format.
func WithMaxBytes ¶ added in v0.11.0
func WithMaxBytes(maxBytes int64) ReaderOption
WithMaxBytes sets the maximum number of input bytes accepted by FromReader. Nonpositive limits are invalid.
func WithMetadata ¶ added in v0.11.0
func WithMetadata(metadata map[string]string) ReaderOption
WithMetadata snapshots metadata for the constructed document.
type Span ¶
Span is a byte-offset range into a UTF-8 string.
func (Span) Text ¶
Text returns the substring covered by the span.
Example ¶
package main
import (
"fmt"
"github.com/dotcommander/reliquary/document"
)
func main() {
text := document.NormalizeText("\ufeffAlpha\r\nBeta")
span := document.Span{StartByte: 0, EndByte: 5}
value, ok := span.Text(text)
fmt.Println(value, ok)
}
Output: Alpha true
Source Files