Affected by 
Documentation
¶
Overview ¶
Package sourceloc maps parsed Markdown back to positions in the original file: it re-aims parser-reported source blocks after the ingestion pipeline rewrites the text, aligns parser output to the structure of office files, and derives the per-chunk locators that citations open.
Index ¶
- func Align(markdown string, units []Unit) []types.SourceBlock
- func AlignStructure(fileType string, data []byte, markdown string) ([]types.SourceBlock, error)
- func AppendLocator(list types.SourceLocators, loc types.SourceLocator) types.SourceLocators
- func CleanQuote(text string) string
- func MergeLocators(a, b types.SourceLocators) types.SourceLocators
- func RemapBlocks(blocks []types.SourceBlock, oldText, newText string) []types.SourceBlock
- func SortBlocks(blocks []types.SourceBlock)
- func SupportsStructure(fileType string) bool
- func TextBlocks(content string) []types.SourceBlock
- type Index
- type Remapper
- type Unit
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Align ¶
func Align(markdown string, units []Unit) []types.SourceBlock
Align maps complete source units, never extending a hit through unmatched content. Unknown gaps must stay unknown: assigning them to the preceding block can point a citation at an entirely different page.
func AlignStructure ¶
AlignStructure reads the structure of an office file and aligns the parser's markdown against it. It returns nil when the file type is not supported or nothing could be placed.
func AppendLocator ¶
func AppendLocator(list types.SourceLocators, loc types.SourceLocator) types.SourceLocators
AppendLocator appends loc to list, folding it into the last entry when both point at the same place.
func CleanQuote ¶
CleanQuote strips Markdown image and link destinations and HTML tags from text, collapses whitespace, and caps it at types.SourceLocatorQuoteMax runes.
func MergeLocators ¶
func MergeLocators(a, b types.SourceLocators) types.SourceLocators
MergeLocators unions two locator lists, dropping exact duplicates, for search results that the chat pipeline merges.
func RemapBlocks ¶
func RemapBlocks(blocks []types.SourceBlock, oldText, newText string) []types.SourceBlock
RemapBlocks re-aims blocks recorded against oldText at newText. Blocks that collapse to nothing are dropped.
func SortBlocks ¶
func SortBlocks(blocks []types.SourceBlock)
SortBlocks orders blocks by start offset.
func SupportsStructure ¶
SupportsStructure reports whether StructureUnits can read fileType.
func TextBlocks ¶
func TextBlocks(content string) []types.SourceBlock
TextBlocks splits a plain-text original (Markdown, TXT) that the parser passed through unchanged into paragraph blocks whose text locators point at the same rune ranges of the original file.
Types ¶
type Index ¶
type Index struct {
// contains filtered or unexported fields
}
Index answers "which source positions does this range of the final Markdown cover" for the chunker's output.
func NewIndex ¶
func NewIndex(markdown string, blocks []types.SourceBlock) *Index
NewIndex builds an index over blocks recorded against markdown.
func (*Index) Locators ¶
func (x *Index) Locators(start, end int) types.SourceLocators
Locators returns the locators of the blocks overlapping the rune range [start, end), each quoting the overlapping text. Adjacent locators that point at the same place (one page without a region, one slide, a run of rows) are folded together.
func (*Index) LocatorsAt ¶
func (x *Index) LocatorsAt(pos int) types.SourceLocators
LocatorsAt returns the locators of the blocks containing rune offset pos, without quotes. Used to place an image reference found in the markdown.
type Remapper ¶
type Remapper struct {
// contains filtered or unexported fields
}
Remapper translates rune offsets in one version of a text to the matching offsets in a rewritten version.
Ingestion rewrites parser output before chunking (line endings, HTML tables, image references). Every rewrite is local, so the two texts share most of their lines: the remapper pairs identical lines, maps offsets inside a paired line directly, and spreads offsets proportionally across the unpaired stretches between pairs. That keeps it correct for any future rewrite step without each step having to report its edits.
func NewRemapper ¶
NewRemapper prepares a best-effort offset mapping. Deprecated: source positions must use RemapBlocks, which verifies full text.
type Unit ¶
type Unit struct {
Text string
Locator types.SourceLocator
}
Unit is one structural piece of an original file (a paragraph, a slide shape, a sheet row) with the locator it should resolve to.
Source Files