worker

package
v0.14.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jan 2, 2026 License: MIT Imports: 12 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func CalculatePriority

func CalculatePriority(info os.FileInfo) int

CalculatePriority calculates job priority based on file modification time More recently modified files get higher priority

Types

type Job

type Job struct {
	Path     string
	Info     os.FileInfo
	Priority int // Higher priority = process first
}

Job represents a file processing job

type JobResult

type JobResult struct {
	Entry     types.IndexEntry
	Embedding []float32 // Optional embedding for the file's summary
	Error     error
}

JobResult represents the result of processing a job

type Pool

type Pool struct {
	// contains filtered or unexported fields
}

Pool manages parallel file processing with rate limiting

func NewPool

func NewPool(
	workers int,
	rateLimitPerMin int,
	embeddingRateLimitPerMin int,
	metadataExtractor *metadata.Extractor,
	semanticProvider semantic.Provider,
	providerName string,
	modelName string,
	embeddingProvider embeddings.Provider,
	embeddingCache *embeddings.Cache,
	cacheManager *cache.Manager,
	logger *slog.Logger,
	ctx context.Context,
) *Pool

NewPool creates a new worker pool

func (*Pool) GetStats

func (p *Pool) GetStats() PoolStats

GetStats returns current pool statistics

func (*Pool) Results

func (p *Pool) Results() <-chan JobResult

Results returns the result channel

func (*Pool) Start

func (p *Pool) Start()

Start starts the worker pool

func (*Pool) Stop

func (p *Pool) Stop()

Stop stops the worker pool gracefully

func (*Pool) Submit

func (p *Pool) Submit(job Job)

Submit submits a job to the pool

func (*Pool) SubmitBatch

func (p *Pool) SubmitBatch(jobs []Job)

SubmitBatch submits multiple jobs, sorted by priority

type PoolStats

type PoolStats struct {
	JobsProcessed      int
	JobsQueued         int
	CacheHits          int
	APICalls           int
	EmbeddingAPICalls  int
	EmbeddingCacheHits int
	Errors             int
}

PoolStats tracks worker pool statistics

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL