Documentation
¶
Overview ¶
Package loaders provides document loaders for different file formats.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type CSVLoader ¶
type CSVLoader struct {
// contains filtered or unexported fields
}
CSVLoader loads CSV files as knowledge documents (one document per row).
func NewCSVLoader ¶
NewCSVLoader creates a loader for CSV files. If hasHeader is true, the first row is treated as column names and used to label fields.
type ChunkStrategy ¶
type ChunkStrategy int
ChunkStrategy defines how text is split into chunks.
const ( ChunkByCharacters ChunkStrategy = iota ChunkBySentences ChunkByParagraphs )
type Chunker ¶
type Chunker struct {
Strategy ChunkStrategy
ChunkSize int
Overlap int
}
Chunker splits text into chunks using configurable strategies.
func NewChunker ¶
func NewChunker(strategy ChunkStrategy, chunkSize, overlap int) *Chunker
NewChunker creates a chunker with the given strategy.
type JSONLoader ¶
type JSONLoader struct {
// contains filtered or unexported fields
}
JSONLoader loads JSON files as knowledge documents. Supports both JSON arrays (one document per element) and single objects (one document).
func NewJSONLoader ¶
func NewJSONLoader(paths []string) *JSONLoader
NewJSONLoader creates a loader for JSON files.
type PDFLoader ¶
type PDFLoader struct {
// contains filtered or unexported fields
}
PDFLoader loads PDF files as knowledge documents by extracting text content. It uses a lightweight pure-Go text extraction approach that handles common PDF text streams without requiring external dependencies like poppler.
func NewPDFLoader ¶
NewPDFLoader creates a loader for PDF files. chunkSize controls the maximum characters per document chunk (0 = no chunking). overlap controls how many characters overlap between consecutive chunks.
type TextLoader ¶
type TextLoader struct {
// contains filtered or unexported fields
}
TextLoader loads plain text files as knowledge documents.
func NewTextLoader ¶
func NewTextLoader(paths []string, chunkSize, overlap int) *TextLoader
NewTextLoader creates a loader for plain text files. chunkSize controls the maximum characters per document chunk (0 = no chunking). overlap controls how many characters overlap between consecutive chunks.
type WebLoader ¶
type WebLoader struct {
// contains filtered or unexported fields
}
WebLoader loads web pages as knowledge documents by scraping URL content and extracting readable text. It strips HTML tags and normalizes whitespace.
func NewWebLoader ¶
NewWebLoader creates a loader for web pages. chunkSize controls the maximum characters per document chunk (0 = no chunking). overlap controls how many characters overlap between consecutive chunks.
Source Files