Documentation
¶
Overview ¶
Package parallelwalkdir walks directory trees with a bounded pool of directory workers and streams matching files on channels.
Concurrency ¶
Workers read dirWork only from a buffered dirCh. Subdirectories are enqueued on an unbounded schedule FIFO (internal/queue.Queue[*dirWork]). One feeder goroutine dequeues with DequeueWait, is the sole dirCh writer, and is the only caller of pending.Add before each send. The coordinator waits on idleMu/idleCond until the schedule queue is empty and processing and feeding atomics are zero, then pending.Wait (serialized with feeder Add via pendingMu), stops the feeder, closes dirCh, waits for workers, and closes results and errs.
Visit tracking uses FNV-1a 128-bit hashes of canonical directory path bytes (EvalSymlinks when needed) to avoid symlink loops. Hash collisions may silently skip a directory (accepted policy).
Filtering (pick one include mode) ¶
- WithBasenameInclude: match entry basenames with zero-alloc suffix checks derived from the compiled regexp pattern (cached by pattern string). Use for production discovery walks (see files.WalkImageDir). Unsupported patterns panic when the option is applied.
- WithRegexpInclude: match entry basenames with regexp.Regexp.Match (may allocate under load). Tests and patterns not supported by suffix derivation.
- WithValidationFunc: custom filter on reported path bytes and fs.FileInfo.
WithSizeNotZero drops zero-length files. WithRegexpExclude skips paths whose reported path matches. Include options are mutually exclusive with each other and with WithValidationFunc.
WithDirEntCatalog: GetDirEntMapFunc runs before readDirAllFn per directory; CheckIfFileModifiedFunc gates sendReported on PathHash keys from HashPathBytes(entry.Name). Requires WithBasenameInclude and WithSizeNotZero. Production discovery passes files.DiscoveryDirEntModifiedWithStats from WalkImageDir; unchanged files are dropped at walk time and never reach discovery workers. FNV-128 key collisions may mis-skip files (accepted).
Results ¶
ParallelWalk returns buffered results and errs channels. Drain both until closed. Walk errors are non-fatal; callers log or count them. Scheduled directory paths are carried as owned []byte inside dirWork. ReportedFile.Path is a subslice of the walk's WalkPathStore arena (symlink-aware reported path), plus mtime and size from stat.
Test seams ¶
Package vars evalSymlinksAt, oszaReadDir, readDirAllFn, fileInfoForDirEntryFn, walkLstatAt, walkLstatPathLen, walkStatAt, walkStatPathLen, and walkLstatJoin default to production implementations and may be overridden in tests.
Index ¶
- type CheckIfFileModifiedFunc
- type DirEntState
- type GetDirEntMapFunc
- type Option
- func WithBasenameInclude(r *regexp.Regexp) Option
- func WithContext(ctx context.Context) Option
- func WithDirChCapacity(n int) Option
- func WithDirEntCatalog(getMap GetDirEntMapFunc, check CheckIfFileModifiedFunc) Option
- func WithMaxReportedFiles(n int64) Option
- func WithMaxWorkers(n int) Option
- func WithRegexpExclude(r *regexp.Regexp) Option
- func WithRegexpInclude(r *regexp.Regexp) Option
- func WithSizeNotZero() Option
- func WithValidationFunc(vf WalkDirFunc) Option
- func WithWalkPathStore(store *WalkPathStore) Option
- type PathHash
- type ReportedFile
- type WalkDirFunc
- type WalkPathStore
- type Walker
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type CheckIfFileModifiedFunc ¶ added in v0.16.0
type CheckIfFileModifiedFunc func(name []byte, info fs.FileInfo, state DirEntState, inMap bool) (modified bool, err error)
CheckIfFileModifiedFunc decides whether a regular file must be reported on results. modified == true means send on results (needs discovery processing).
type DirEntState ¶ added in v0.16.0
type DirEntState struct {
InvalidPathValid bool
InvalidMtime int64
InvalidSize int64
FileIDValid bool
FileMtimeValid bool
FileMtime int64
FileSizeValid bool
FileSize int64
FileMD5Valid bool
}
DirEntState holds catalog fields needed to decide modified vs unchanged without a per-file DB call.
type GetDirEntMapFunc ¶ added in v0.16.0
type GetDirEntMapFunc func(ctx context.Context, reportedDir []byte) (map[PathHash]DirEntState, error)
GetDirEntMapFunc loads per-directory catalog state keyed by HashPathBytes(basename).
type Option ¶
type Option func(*Walker)
Option configures a Walker.
func WithBasenameInclude ¶ added in v0.16.0
WithBasenameInclude matches file basenames with suffix checks derived from r. Patterns must be an anchored alternation of literal suffix fragments (see basenameSuffixesFromPattern); derivation is verified against r at build time. Mutually exclusive with WithRegexpInclude and WithValidationFunc.
func WithContext ¶
func WithDirChCapacity ¶ added in v0.16.0
WithDirChCapacity sets the buffered dirCh capacity. Zero uses workers()*1024. Values > 0 set an exact buffer size (tests only; production does not set this).
func WithDirEntCatalog ¶ added in v0.16.0
func WithDirEntCatalog(getMap GetDirEntMapFunc, check CheckIfFileModifiedFunc) Option
WithDirEntCatalog enables per-directory catalog filtering before reporting files. Requires WithBasenameInclude and WithSizeNotZero. Mutually exclusive with WithRegexpInclude and WithValidationFunc.
func WithMaxReportedFiles ¶ added in v0.16.0
WithMaxReportedFiles stops the walk after n matching files are sent on results. Intended for benchmarks and sampling; production discovery does not set this.
func WithMaxWorkers ¶ added in v0.16.0
WithMaxWorkers sets the number of directory worker goroutines. Values <= 0 use runtime.GOMAXPROCS(0). Intended for tests; production discovery does not set this.
func WithRegexpExclude ¶
WithRegexpExclude skips directory entries whose reported path matches r. Applied to both files and directories before descent.
func WithRegexpInclude ¶
WithRegexpInclude limits files to those whose basename matches r via Match. Mutually exclusive with WithBasenameInclude and WithValidationFunc.
func WithSizeNotZero ¶
func WithSizeNotZero() Option
WithSizeNotZero excludes zero-length files. Mutually exclusive with WithValidationFunc.
func WithValidationFunc ¶
func WithValidationFunc(vf WalkDirFunc) Option
WithValidationFunc supplies custom file filtering on reported path bytes. Mutually exclusive with WithRegexpInclude, WithBasenameInclude, and WithSizeNotZero.
func WithWalkPathStore ¶ added in v0.16.0
func WithWalkPathStore(store *WalkPathStore) Option
WithContext sets cancellation for the walk. When cancelled, the feeder stops without feeding queued work that was not yet sent on dirCh; workers and schedule submit paths exit as soon as practical; results and errs close after workers finish. WithWalkPathStore sets the bump arena for reported paths. When unset, ParallelWalk allocates a default store for the Walker.
type PathHash ¶ added in v0.16.0
type PathHash struct {
Hi, Lo uint64
}
PathHash is a 128-bit FNV-1a digest of canonical path bytes, suitable as a comparable map key (for example map[PathHash]struct{}). Hi and Lo are the big-endian high and low 64-bit words respectively—the same layout as hash/fnv.New128a().Sum(nil).
func HashPathBytes ¶ added in v0.16.0
HashPathBytes returns the FNV-1a 128-bit hash of path. The slice is read only for the duration of the call; callers must not mutate path concurrently.
type ReportedFile ¶ added in v0.16.0
type ReportedFile struct {
Path []byte // Arena subslice from the walk's WalkPathStore; valid until store Reset or discard.
ModTimeUnix int64 // From the FileInfo used for filtering
SizeBytes int64
}
ReportedFile is one file sent on the ParallelWalk results channel.
type WalkDirFunc ¶
WalkDirFunc decides inclusion from the reported path and FileInfo. path is the reported path slice; valid only for the call; do not retain. Mutually exclusive with WithRegexpInclude, WithBasenameInclude, and WithSizeNotZero.
type WalkPathStore ¶ added in v0.16.0
type WalkPathStore struct {
// contains filtered or unexported fields
}
WalkPathStore bump-allocates reported path bytes for one or more walks on the same store. ReportedFile.Path values are subslices of the arena until Reset runs and nothing still references bytes from the prior bump generation.
func NewWalkPathStore ¶ added in v0.16.0
func NewWalkPathStore() *WalkPathStore
NewWalkPathStore returns an empty path arena.
func (*WalkPathStore) AppendPath ¶ added in v0.16.0
func (s *WalkPathStore) AppendPath(src []byte, n int) []byte
AppendPath copies src[:n] into the arena and returns a subslice valid until Reset or the store is discarded.
func (*WalkPathStore) Reset ¶ added in v0.16.0
func (s *WalkPathStore) Reset()
Reset clears the bump offset; retained capacity is reused on the next walk.
type Walker ¶
type Walker struct {
// contains filtered or unexported fields
}
Walker holds state for a parallel directory traversal: a fixed pool of worker goroutines reading from dirCh, an unbounded schedule queue and feeder that is the sole writer to dirCh, in-flight work accounting (pending), visited-path loop detection, and result/error channels.
func NewWalker ¶
NewWalker creates a Walker and applies options. Default worker count is runtime.GOMAXPROCS(0) when WithMaxWorkers is unset or <= 0.
Panics if include options conflict: WithValidationFunc with any other filter option, or both WithRegexpInclude and WithBasenameInclude.
func (*Walker) ParallelWalk ¶
func (w *Walker) ParallelWalk(rootPath string) (<-chan ReportedFile, <-chan error)
ParallelWalk traverses rootPath. The returned channels are read-only; both close when the walk finishes. The same Walker may be reused for another walk.
WithMaxReportedFiles wraps the walk context in a cancel func; reaching the limit cancels further scheduling.
Source Files