Documentation
¶
Overview ¶
Package wfp turns files into WFP (Winnowing Fingerprint) fingerprints — the payload a SCANOSS scan uploads. All entry points fingerprint through a bounded worker pool; they differ in what they hand back. Folder and Files return the combined stream sorted by path together with the per-file detail behind it. Stream and StreamFolder write each file's block to an io.Writer as it completes and retain nothing, so memory stays bounded whatever the tree size — at the price of completion-order output.
Folder and StreamFolder take a directory and apply the filtering rules themselves, which is what a caller holding a tree wants. Files and Stream take a list and apply none: they fingerprint whatever they are given, because the list may have come from somewhere other than a walk — an explicit selection, an archive stream, a file named on a command line.
The asymmetry is deliberate but not obvious from the names, so a caller building the list from a directory itself should filter it first, with filter.Collect and one of its profiles.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Stream ¶
func Stream(files []string, workers int, root string, w io.Writer, onProgress func(done, total int)) ([]error, error)
Stream fingerprints the files and writes each block to w as it completes, retaining none of them: memory stays bounded by the in-flight window whatever the tree size.
Block order is completion order, by design: the server matches per block, so order does not affect scan results. Use Files where byte-reproducible output matters.
The []error lists the files that could not be fingerprinted — best-effort, like Result.Errors. The error reports a failed write to w, which is fatal: a stream missing a block is unusable, so nothing further is written, though the pool is still drained so every worker finishes and progress reaches its total.
func StreamFolder ¶
func StreamFolder(dir string, filters *filter.Options, workers int, w io.Writer, onProgress func(done, total int)) ([]error, error)
StreamFolder collects the files worth fingerprinting under dir and streams their fingerprints to w — Folder's collection with Stream's memory profile and contract: block order is completion order, and nothing is retained.
Types ¶
type FileFingerprint ¶
type FileFingerprint struct {
Path string // path of the fingerprinted file, relative to the scan root
Hash string // whole-file hash
Size int // file size in bytes
Fingerprint string // the WFP text itself, "file=..." and its minutiae
}
FileFingerprint is one file's fingerprint: what the scan uploads about it, and what every stage between hashing and upload passes around.
It lives here, beside the function that produces it, because it is the vocabulary the fingerprinting packages speak to each other in — the worker pool and the scan service both name it. Kept out of reach, it would leave those packages with signatures no caller could write down.
type Result ¶
type Result struct {
WFP []byte // the combined stream, ready to upload
Files []FileFingerprint // per-file detail, sorted by path
Errors []error // files that could not be fingerprinted
Skipped int // files the filters excluded; always 0 from Files, which selects nothing
}
Result is what Files and Folder produce: the stream to upload, the per-file detail behind it, and the files it could not fingerprint.
WFP and Files are two views of one run, not a choice: a scan uploads WFP, while a caller reporting on individual files needs Hash and Size, which the combined stream does not carry. Errors is a field rather than a second return value because fingerprinting is best-effort — a run that skipped three unreadable files still produced a usable WFP — and a caller that ignores it should have to say so.
func Files ¶
Files fingerprints the given files through a bounded worker pool. onProgress, if non-nil, is called as each file completes (done, total), counting failures too, so done reaches total.
It does no selection: which files are worth fingerprinting was decided by whoever built the list. Fingerprinting one file is Files with a one-file slice — a separate single-file function would be a second way to do the same thing, and the caller would still have to combine the output itself to upload it.
TODO(#77): filter here as well, once the rules can be applied to a list. Folder filters and Files does not, which their names do not suggest: a caller that hands over every file in a directory gets a WFP full of files a scan would have skipped.
func Folder ¶
func Folder(dir string, filters *filter.Options, workers int, onProgress func(done, total int)) (Result, error)
Folder collects the files worth fingerprinting under dir and fingerprints them. A nil filters uses the fingerprinting profile, which is what a caller holding a directory and no opinion about the rules wants.
It is the entry point to reach for with a directory in hand: Files fingerprints whatever it is given, so building the list yourself means deciding for yourself what a scan should skip.
Source Files