Documentation
¶
Overview ¶
Package scan owns the per-file processing pipeline: smart filtering, token counting, and optional signature compression. It turns a file's bytes into an enriched FileEntry for the collection and picker flows, so walkers and renderers stay free of tokenizer/compressor details.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ErrFileSkipped = errors.New("file skipped by smart filter")
ErrFileSkipped is returned by Process when the smart filter rejects the file. The specific reason is wrapped so callers can report or ignore it. It distinguishes a deliberate skip from a processing failure.
Functions ¶
This section is empty.
Types ¶
type Mode ¶
type Mode int
Mode selects how a file is processed.
const ( // ModeFull keeps the raw content with a single token count. Used by // dump/diff/watch when no signature mode is requested. ModeFull Mode = iota // ModePicker retains both the full and signature variants plus their // token counts, leaving the active view to the caller. The entry's Tokens // is left unset (-1); selection reconciles it with the chosen view. ModePicker // ModeSignatures renders the compressed signature summary into Content, // with token counts reflecting the compressed text. ModeSignatures )
type Options ¶
type Options struct {
TokenizeModel string
SmartFilter bool
SmartMaxTokens int
// Compress enables signature compression. It is off for ordinary
// full-content scans, which never touch the compressor, so the tree-sitter
// grammars and parser pools are not loaded unnecessarily.
Compress bool
Logger *logger.Logger
}
Options configures a Processor.
type Processor ¶
type Processor struct {
// contains filtered or unexported fields
}
Processor transforms a file's content into an enriched FileEntry.
The tokenizer, compressor, and smart evaluator are shared across Process calls. The tokenizer's codec is safe for concurrent Count calls (its regexp runner comes from a sync.Pool), and the compressor pools its tree-sitter parsers, so one Processor may be used by many workers.
func New ¶
New returns a Processor for the given options. Tokenizer construction can fail, hence the error return.
func (*Processor) Process ¶
Process enriches one file's content according to mode. It returns ErrFileSkipped (wrapping the smart-filter reason) when the file is rejected by name, content, or token guardrail. Token-counting failures log a warning and yield a zero count so a counting error never aborts a scan.
Source Files