pipeline

package

v1.0.0 Latest Latest Go to latest Published: Mar 29, 2026 License: MIT Imports: 7 Imported by: 0

Details

This section is empty.

This section is empty.

This section is empty.

type ChunkedFile struct {
	Path   string
	Chunks []EmbeddedChunk
}

ChunkedFile represents a file that has been chunked and embedded.

type EmbeddedChunk struct {
	Content string
	Index   int
	Vector  []float32
}

EmbeddedChunk holds a chunk's text and its vector embedding.

type Result struct {
	Path       string
	ChunkCount int
	Embedded   bool
}

Result holds the output of the full pipeline for a single file.

func Run(source string, workers int, chunkSize int, chunkOverlap int, maxMem int, emb embedder.Embedder, db *storage.DB) ([]Result, error)

Run executes the crawl-chunk-embed-storage pipeline. If emb is nil, it skips the embedding step.