Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Chunk ¶
type Chunk struct {
Index int `json:"index"`
Text string `json:"text"`
ByteStart int `json:"byte_start"`
ByteEnd int `json:"byte_end"`
}
Chunk represents a single piece of the original text along with its position in the source document.
func ChunkBytes ¶
ChunkBytes splits data into overlapping chunks of at most chunkSize bytes, preferring to break on paragraph, then sentence, then whitespace boundaries so words are not sliced in half. overlap bytes from the end of one chunk are repeated at the start of the next chunk to preserve context across chunk boundaries (useful for retrieval quality).
type EmbeddedChunk ¶
Source Files
Click to show internal directories.
Click to hide internal directories.