indexer

package
v0.11.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 29, 2026 License: MIT Imports: 17 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func IsTextFile added in v0.7.0 

func IsTextFile(ext string) bool

IsTextFile reports whether ext (e.g. ".go") is a supported text/code extension.

Types

type Indexer 

type Indexer struct {
	// contains filtered or unexported fields
}

Indexer scans a workspace directory for PDF and text/code files and runs the IndexAgent on batches of documents to produce semantic labels per page.

func New 

func New(dir string, docStore *docindex.Store, lr *agent.LoopRunner) *Indexer

New creates a new Indexer. Pass an empty model to use the runner's default.

func (*Indexer) IndexDocument 

func (idx *Indexer) IndexDocument(ctx context.Context, filePath string) error

IndexDocument extracts text from a PDF or text/code file, builds keyword corpora, then runs the IndexAgent in batches to produce labels. This is the single-file entry point — it delegates to the batch infrastructure.

func (*Indexer) Progress added in v0.9.0 

func (idx *Indexer) Progress() *ProgressTracker

Progress returns the progress tracker for external monitoring.

func (*Indexer) Run 

func (idx *Indexer) Run(ctx context.Context) error

Run scans dir recursively for PDF and text/code files and indexes each one. It groups small text files into batches (Solution 1), runs batches in parallel (Solution 2), and publishes progress events (Solution 5).

func (*Indexer) WithExcludes added in v0.7.0 

func (idx *Indexer) WithExcludes(patterns []string) *Indexer

WithExcludes sets additional patterns to skip during the directory walk. Patterns are matched against directory names and file basenames using filepath.Match.

func (*Indexer) WithMaxConcurrent added in v0.9.0 

func (idx *Indexer) WithMaxConcurrent(n int) *Indexer

WithMaxConcurrent sets the number of parallel indexing sessions.

func (*Indexer) WithMaxKeywordsBatch added in v0.9.0 

func (idx *Indexer) WithMaxKeywordsBatch(n int) *Indexer

WithMaxKeywordsBatch sets the maximum number of keywords per LLM batch.

func (*Indexer) WithModel 

func (idx *Indexer) WithModel(model string) *Indexer

WithModel sets the model override for sessions created by the indexer.

type PageCorpus 

type PageCorpus struct {
	PageNum  int
	Keywords []string
}

PageCorpus holds the deduplicated keyword corpus for a single PDF page.

func BuildCorpora 

func BuildCorpora(pages []PageText) []PageCorpus

BuildCorpora converts page texts to keyword corpora in parallel. Each corpus is deduplicated, lowercased, and stripped of stop words and non-alphabetic tokens.

type PageText 

type PageText struct {
	PageNum int
	Text    string
}

PageText holds the raw extracted text for a single PDF page.

func ExtractPages 

func ExtractPages(pdfPath string) ([]PageText, error)

ExtractPages reads all pages from the PDF at pdfPath in parallel and returns a slice of PageText sorted by page number.

func ExtractTextFile added in v0.7.0 

func ExtractTextFile(filePath string) ([]PageText, error)

ExtractTextFile reads a text or code file and returns it as a single PageText. HTML/XML/template files have tags stripped before keyword extraction.

type ProgressTracker added in v0.9.0 

type ProgressTracker struct {
	Total     atomic.Int32
	Completed atomic.Int32
	Failed    atomic.Int32
	Current   atomic.Value // stores string
}

ProgressTracker tracks indexing progress and publishes events via the bus.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.