Documentation
¶
Overview ¶
Package filefilter provides a concurrent pipeline for filtering files by size, content type, and glob patterns.
Index ¶
- Variables
- func ExpandExcludeNames(names []string) ([]string, error)
- func IsTextContent(data []byte) bool
- func ReadFileHeader(path string, n int64) ([]byte, error)
- type Analytics
- type FileFilter
- type Option
- func WithAnalytics(analytics Analytics) Option
- func WithConcurrency(n int) Option
- func WithExcludeGlobs(userPatterns []string) Option
- func WithFilters(filters ...FileFilter) Option
- func WithInvocationContext(invocationCtx workflow.InvocationContext) Option
- func WithLogger(logger *zerolog.Logger) Option
- type Pipeline
Constants ¶
This section is empty.
Variables ¶
var ErrPathNotAllowed = errors.New("paths are not allowed in exclude rules")
ErrPathNotAllowed is returned when an exclude rule contains a path separator.
Functions ¶
func ExpandExcludeNames ¶
ExpandExcludeNames validates and converts user-provided names into global glob patterns. It enforces the rule that inputs must be basenames (example: "node_modules"), not paths.
func IsTextContent ¶
IsTextContent determines if the data slice contains text content based on the null byte method. See: https://docs.google.com/document/d/1GYir_j0ITTxg_CqyAw8BeUZYCCUyNMAePbGw5nsTGYE/
Types ¶
type Analytics ¶
type Analytics interface {
RecordSizeFiltered(total int)
RecordFileFilterTimeMs(startTime time.Time)
}
Analytics defines the metrics recording interface used by the filter pipeline.
type FileFilter ¶
FileFilter defines the contract for any logic that decides if a file should be dropped.
func FileSizeFilter ¶
func FileSizeFilter(logger *zerolog.Logger) FileFilter
FileSizeFilter returns a filter that drops empty files and files larger than 1 MB.
func TextFileOnlyFilter ¶
func TextFileOnlyFilter(logger *zerolog.Logger) FileFilter
TextFileOnlyFilter returns a filter that drops binary files based on a null-byte heuristic.
type Option ¶
type Option func(*Pipeline)
Option defines the functional option type.
func WithAnalytics ¶
WithAnalytics sets the analytics recorder for filter metrics.
func WithConcurrency ¶
WithConcurrency allows overriding the default worker count.
func WithExcludeGlobs ¶
WithExcludeGlobs adds user-defined patterns to the pipeline's exclude list.
func WithFilters ¶
func WithFilters(filters ...FileFilter) Option
WithFilters allows passing multiple filters (FileSizeFilter, TextFileOnlyFilter).
func WithInvocationContext ¶
func WithInvocationContext(invocationCtx workflow.InvocationContext) Option
WithInvocationContext sets the invocation context the file filter is built from, so filtering follows the invocation's configuration. Without it, filtering falls back to GAF's defaults.
func WithLogger ¶
WithLogger sets the logger for the pipeline.
type Pipeline ¶
type Pipeline struct {
// contains filtered or unexported fields
}
Pipeline holds the configuration for the filtering process.
func NewPipeline ¶
NewPipeline creates a filter pipeline with reasonable defaults. Default concurrency is set to runtime.NumCPU().
Source Files