scan

type CachedResult ¶ added in v1.3.0

type CachedResult struct {
	Content      []byte
	Tokens       int
	TokensFull   int
	SigContent   []byte
	TokensSig    int
	IsCompressed bool
	Language     string
}

CachedResult is the expensive half of Process: the enriched fields a cache hit replays verbatim so a hit is indistinguishable from a miss. Content aliases the caller's bytes (or the one-off compressed output); like SigContent it is shared read-only and must never be mutated.

type ContentCache ¶ added in v1.3.0

type ContentCache struct {
	// contains filtered or unexported fields
}

ContentCache maps sha256(content) + processing options to computed token counts and signature summaries, so watch re-renders and repeated scans skip tree-sitter compression and BPE counting for unchanged files. It is safe for concurrent use by walker workers.

Cached SigContent and Content slices are shared read-only: entries produced from a hit must never mutate them. The key includes the tokenizer model so one cache is never poisoned by counts from another encoding.

func NewContentCache() *ContentCache

func (c *ContentCache) Get(content []byte, mode Mode, model string) (CachedResult, bool)

func (c *ContentCache) GetHashed(hash [32]byte, mode Mode, model string) (CachedResult, bool)

func (c *ContentCache) Put(content []byte, mode Mode, model string, result CachedResult)

func (c *ContentCache) PutHashed(hash [32]byte, mode Mode, model string, result CachedResult)

type Mode ¶

type Mode int

Mode selects how a file is processed.

const (
	// ModeFull keeps the raw content with a single token count. Used by
	// dump/diff/watch when no signature mode is requested.
	ModeFull Mode = iota
	// ModePicker retains both the full and signature variants plus their
	// token counts, leaving the active view to the caller. The entry's Tokens
	// is left unset (-1); selection reconciles it with the chosen view.
	ModePicker
	// ModeSignatures renders the compressed signature summary into Content,
	// with token counts reflecting the compressed text.
	ModeSignatures
)

type Options ¶

type Options struct {
	TokenizeModel  string
	SmartFilter    bool
	SmartMaxTokens int
	// Evaluator, when non-nil, is the smart filter to share with the
	// caller's other passes (skeleton, walker pre-read). It carries the
	// path-decision memo across them; when nil and SmartFilter is set, New
	// builds a private evaluator.
	Evaluator *smart.Evaluator
	// Compress enables signature compression. It is off for ordinary
	// full-content scans, which never touch the compressor, so the tree-sitter
	// grammars and parser pools are not loaded unnecessarily.
	Compress bool
	Logger   *logger.Logger
	// Cache memoizes token counts and signature summaries by content hash.
	// When nil, New installs a fresh instance; callers that want hits across
	// processors (e.g. a long-lived App serving watch re-renders) inject one.
	Cache *ContentCache
}

Options configures a Processor.

type Processor ¶

type Processor struct {
	// contains filtered or unexported fields
}

Processor transforms a file's content into an enriched FileEntry.

The tokenizer, compressor, and smart evaluator are shared across Process calls. The tokenizer's codec is safe for concurrent Count calls (its regexp runner comes from a sync.Pool), and the compressor pools its tree-sitter parsers, so one Processor may be used by many workers.

func New(opts Options) (*Processor, error)

func (p *Processor) Process(path string, content []byte, mode Mode) (format.FileEntry, error)