README
¶
document
doc := document.Document{
ID: "doc-1",
Format: document.FormatMarkdown,
Text: document.NormalizeText("\ufeff# Title\r\nBody"),
}
span := document.Span{StartByte: 0, EndByte: len(doc.Text)}
text, ok := span.Text(doc.Text)
document owns provider-neutral document value types: Document, Section,
Element, Span, Format, and Metadata.
Callers own parsing, source schemas, and semantic interpretation. This package only normalizes text and validates byte/rune offsets so downstream chunking, ingest, retrieval, provenance, and test fixtures can share one document shape.
Element records generic pre-chunk structure such as titles, narrative text,
list items, tables, code, page breaks, and unknown blocks. It stores the parser
strategy label and byte span, but it does not select or implement a parser.
Documentation
¶
Overview ¶
Package document defines provider-neutral document, section, and span primitives. It keeps parsing policy with callers and models normalized text, formats, metadata, and byte/rune offset conversion.
Package document defines provider-neutral document primitives.
Index ¶
Examples ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func ByteOffsetForRune ¶
ByteOffsetForRune returns the byte offset for a rune index.
func NormalizeText ¶
NormalizeText normalizes line endings and strips a UTF-8 BOM.
Types ¶
type Document ¶
type Document struct {
ID string
Title string
Format Format
Text string
Sections []Section
Metadata Metadata
}
Document is normalized text plus structural metadata.
type Element ¶
type Element struct {
ID string
Kind ElementKind
Strategy ParserStrategy
Span Span
Text string
Metadata Metadata
}
Element is a span-backed, provider-neutral unit detected before chunking.
func (Element) ElementText ¶
ElementText returns the element text, preferring the span when it is valid for source and falling back to Element.Text.
Example ¶
package main
import (
"fmt"
"github.com/dotcommander/reliquary/document"
)
func main() {
source := "# Title\n\nBody text"
element := document.Element{
ID: "el-1",
Kind: document.ElementKindTitle,
Strategy: document.ParserStrategyMarkdown,
Span: document.Span{StartByte: 2, EndByte: 7},
Text: "fallback title",
}
fmt.Println(element.Kind, element.ElementText(source))
}
Output: title Title
type ElementKind ¶
type ElementKind string
ElementKind identifies generic structure detected before chunking.
const ( ElementKindTitle ElementKind = "title" ElementKindNarrative ElementKind = "narrative_text" ElementKindListItem ElementKind = "list_item" ElementKindTable ElementKind = "table" ElementKindCode ElementKind = "code" ElementKindPageBreak ElementKind = "page_break" ElementKindUnknown ElementKind = "unknown" )
type Format ¶
type Format string
Format identifies a document or span format without owning parsing policy.
type ParserStrategy ¶
type ParserStrategy string
ParserStrategy labels the caller-owned parser strategy that produced an element. It is descriptive only and does not select parser behavior.
const ( ParserStrategyPlainText ParserStrategy = "plain_text" ParserStrategyMarkdown ParserStrategy = "markdown" ParserStrategyHTML ParserStrategy = "html" ParserStrategyPDF ParserStrategy = "pdf" ParserStrategyUnknown ParserStrategy = "unknown" )
type Span ¶
Span is a byte-offset range into a UTF-8 string.
func (Span) Text ¶
Text returns the substring covered by the span.
Example ¶
package main
import (
"fmt"
"github.com/dotcommander/reliquary/document"
)
func main() {
text := document.NormalizeText("\ufeffAlpha\r\nBeta")
span := document.Span{StartByte: 0, EndByte: 5}
value, ok := span.Text(text)
fmt.Println(value, ok)
}
Output: Alpha true
Source Files