Documentation
¶
Overview ¶
Package fileproc provides concurrent file processing utilities.
Index ¶
- Constants
- func ForEachFile[T any](files []string, fn func(string) (T, error)) []T
- func ForEachFileN[T any](files []string, maxWorkers int, fn func(string) (T, error), ...) []T
- func ForEachFileWithErrors[T any](files []string, fn func(string) (T, error), onError ErrorFunc) []T
- func ForEachFileWithProgress[T any](files []string, fn func(string) (T, error), onProgress ProgressFunc) []T
- func ForEachFileWithResource[T any, R any](files []string, initResource func() (R, error), closeResource func(R), ...) []T
- func MapFiles[T any](files []string, fn func(*parser.Parser, string) (T, error)) []T
- func MapFilesN[T any](files []string, maxWorkers int, fn func(*parser.Parser, string) (T, error), ...) []T
- func MapFilesNWithSizeLimit[T any](files []string, maxWorkers int, maxSize int64, ...) []T
- func MapFilesWithErrors[T any](files []string, fn func(*parser.Parser, string) (T, error), onError ErrorFunc) []T
- func MapFilesWithProgress[T any](files []string, fn func(*parser.Parser, string) (T, error), ...) []T
- func MapFilesWithSizeLimit[T any](files []string, maxSize int64, fn func(*parser.Parser, string) (T, error), ...) []T
- type ErrorFunc
- type ProcessingError
- type ProcessingErrors
- func ForEachFileCollectErrors[T any](files []string, fn func(string) (T, error)) ([]T, *ProcessingErrors)
- func ForEachFileCollectErrorsWithProgress[T any](files []string, fn func(string) (T, error), onProgress ProgressFunc) ([]T, *ProcessingErrors)
- func ForEachFileWithContext[T any](ctx context.Context, files []string, fn func(string) (T, error)) ([]T, *ProcessingErrors)
- func ForEachFileWithContextAndProgress[T any](ctx context.Context, files []string, fn func(string) (T, error), ...) ([]T, *ProcessingErrors)
- func MapFilesCollectErrors[T any](files []string, fn func(*parser.Parser, string) (T, error)) ([]T, *ProcessingErrors)
- func MapFilesCollectErrorsWithProgress[T any](files []string, fn func(*parser.Parser, string) (T, error), ...) ([]T, *ProcessingErrors)
- func MapFilesWithContext[T any](ctx context.Context, files []string, ...) ([]T, *ProcessingErrors)
- func MapFilesWithContextAndProgress[T any](ctx context.Context, files []string, ...) ([]T, *ProcessingErrors)
- type ProgressFunc
Constants ¶
const DefaultWorkerMultiplier = 2
DefaultWorkerMultiplier is the multiplier applied to NumCPU for worker count. 2x is optimal for mixed I/O and CGO workloads.
Variables ¶
This section is empty.
Functions ¶
func ForEachFile ¶
ForEachFile processes files in parallel, calling fn for each file. No parser is provided; use this for non-AST operations (e.g., SATD scanning). Uses 2x NumCPU workers by default.
func ForEachFileN ¶
func ForEachFileN[T any](files []string, maxWorkers int, fn func(string) (T, error), onProgress ProgressFunc, onError ErrorFunc) []T
ForEachFileN processes files with configurable worker count and callbacks. If maxWorkers is <= 0, defaults to 2x NumCPU.
func ForEachFileWithErrors ¶
func ForEachFileWithErrors[T any](files []string, fn func(string) (T, error), onError ErrorFunc) []T
ForEachFileWithErrors processes files in parallel with error callback.
func ForEachFileWithProgress ¶
func ForEachFileWithProgress[T any](files []string, fn func(string) (T, error), onProgress ProgressFunc) []T
ForEachFileWithProgress processes files in parallel with optional progress callback.
func ForEachFileWithResource ¶
func ForEachFileWithResource[T any, R any]( files []string, initResource func() (R, error), closeResource func(R), fn func(R, string) (T, error), onProgress ProgressFunc, ) []T
ForEachFileWithResource processes files in parallel, calling fn for each file with a per-worker resource. The initResource function is called once per worker to create the resource (e.g., git repo handle). The closeResource function is called when the worker is done to release the resource. Uses 2x NumCPU workers by default.
func MapFiles ¶
MapFiles processes files in parallel, calling fn for each file with a dedicated parser. Results are collected and returned in arbitrary order. Errors from individual files are silently skipped; use MapFilesWithErrors for error handling. Uses 2x NumCPU workers by default (optimal for mixed I/O and CGO workloads).
func MapFilesN ¶
func MapFilesN[T any](files []string, maxWorkers int, fn func(*parser.Parser, string) (T, error), onProgress ProgressFunc, onError ErrorFunc) []T
MapFilesN processes files with configurable worker count and callbacks. If maxWorkers is <= 0, defaults to 2x NumCPU.
func MapFilesNWithSizeLimit ¶
func MapFilesNWithSizeLimit[T any](files []string, maxWorkers int, maxSize int64, fn func(*parser.Parser, string) (T, error), onProgress ProgressFunc, onError ErrorFunc) []T
MapFilesNWithSizeLimit processes files with configurable worker count and file size limit. Files exceeding maxSize bytes are skipped. If maxSize is 0, no limit is enforced.
func MapFilesWithErrors ¶
func MapFilesWithErrors[T any](files []string, fn func(*parser.Parser, string) (T, error), onError ErrorFunc) []T
MapFilesWithErrors processes files in parallel with error callback. The onError callback is invoked for each file that fails processing.
func MapFilesWithProgress ¶
func MapFilesWithProgress[T any](files []string, fn func(*parser.Parser, string) (T, error), onProgress ProgressFunc) []T
MapFilesWithProgress processes files in parallel with optional progress callback.
func MapFilesWithSizeLimit ¶
func MapFilesWithSizeLimit[T any](files []string, maxSize int64, fn func(*parser.Parser, string) (T, error), onProgress ProgressFunc, onError ErrorFunc) []T
MapFilesWithSizeLimit processes files in parallel, skipping files that exceed maxSize. If maxSize is 0, no limit is enforced. Files that exceed the limit are silently skipped unless onError is provided.
Types ¶
type ErrorFunc ¶
ErrorFunc is called when a file processing error occurs. Receives the file path and the error. If nil, errors are silently skipped.
type ProcessingError ¶
ProcessingError represents an error that occurred while processing a file.
func (ProcessingError) Error ¶
func (e ProcessingError) Error() string
type ProcessingErrors ¶
type ProcessingErrors struct {
Errors []ProcessingError
// contains filtered or unexported fields
}
ProcessingErrors collects multiple file processing errors.
func ForEachFileCollectErrors ¶
func ForEachFileCollectErrors[T any](files []string, fn func(string) (T, error)) ([]T, *ProcessingErrors)
ForEachFileCollectErrors processes files in parallel and collects all errors. Returns results and any errors that occurred during processing.
func ForEachFileCollectErrorsWithProgress ¶
func ForEachFileCollectErrorsWithProgress[T any](files []string, fn func(string) (T, error), onProgress ProgressFunc) ([]T, *ProcessingErrors)
ForEachFileCollectErrorsWithProgress processes files in parallel with progress callback and collects errors.
func ForEachFileWithContext ¶
func ForEachFileWithContext[T any](ctx context.Context, files []string, fn func(string) (T, error)) ([]T, *ProcessingErrors)
ForEachFileWithContext processes files in parallel with context cancellation support.
func ForEachFileWithContextAndProgress ¶
func ForEachFileWithContextAndProgress[T any](ctx context.Context, files []string, fn func(string) (T, error), onProgress ProgressFunc) ([]T, *ProcessingErrors)
ForEachFileWithContextAndProgress processes files with context and progress callback.
func MapFilesCollectErrors ¶
func MapFilesCollectErrors[T any](files []string, fn func(*parser.Parser, string) (T, error)) ([]T, *ProcessingErrors)
MapFilesCollectErrors processes files in parallel and collects all errors. Returns results and any errors that occurred during processing.
func MapFilesCollectErrorsWithProgress ¶
func MapFilesCollectErrorsWithProgress[T any](files []string, fn func(*parser.Parser, string) (T, error), onProgress ProgressFunc) ([]T, *ProcessingErrors)
MapFilesCollectErrorsWithProgress processes files in parallel with progress callback and collects errors.
func MapFilesWithContext ¶
func MapFilesWithContext[T any](ctx context.Context, files []string, fn func(*parser.Parser, string) (T, error)) ([]T, *ProcessingErrors)
MapFilesWithContext processes files in parallel with context cancellation support. Returns results collected before cancellation and any errors including context errors.
func MapFilesWithContextAndProgress ¶
func MapFilesWithContextAndProgress[T any](ctx context.Context, files []string, fn func(*parser.Parser, string) (T, error), onProgress ProgressFunc) ([]T, *ProcessingErrors)
MapFilesWithContextAndProgress processes files with context and progress callback.
func (*ProcessingErrors) Add ¶
func (e *ProcessingErrors) Add(path string, err error)
Add appends an error to the collection (thread-safe).
func (*ProcessingErrors) Error ¶
func (e *ProcessingErrors) Error() string
Error implements the error interface.
func (*ProcessingErrors) HasErrors ¶
func (e *ProcessingErrors) HasErrors() bool
HasErrors returns true if any errors were collected.
func (*ProcessingErrors) Unwrap ¶
func (e *ProcessingErrors) Unwrap() error
Unwrap returns nil (ProcessingErrors doesn't wrap a single error).
Source Files