Documentation
¶
Overview ¶
Package distfiles resolves and prepares the directory pkgdev is given as --distdir, and holds the single implementation every caller shares.
It exists because the two callers had drifted: `overlay manifest` took a configurable, persistent directory while the autoupdate sweep hardcoded a temporary one. Keeping the resolution here — below both feature packages, rather than exported from one of them — is what stops that divergence from arising again.
The directory this package hands back may be the host's real DISTDIR: the same directory portage downloads into, shared with emerge and with every other package operation on the machine. It is therefore NOT necessarily ours to delete. A resolved Dir carries its own provenance in Created, and Cleanup removes the directory only when this process is the one that made it; a directory the caller named — or the host's own DISTDIR — survives the run untouched.
There are two entry points because the two commands promise their users different defaults, not because the resolution differs:
- Resolve is the autoupdate path. With nothing configured it lands on the DISTDIR the host itself names, and never on a temporary directory.
- ResolveOrTemp is `overlay manifest`. With nothing configured it makes a throwaway temporary directory, which is exactly what that command's --help promises ("no sudo is required").
Both share one expansion-and-creation helper, so the drift this package was created to end cannot come back through the parts that are genuinely common.
Only Resolve ends with the writability pre-flight (Probe). That is not an oversight in ResolveOrTemp: `overlay manifest` ships today without such a check, its default distdir is one it has just created, and turning a non-writable named --distdir into a hard failure there would change the behaviour of a command this package was only supposed to move.
The package depends only on the Go standard library.
Index ¶
- Constants
- Variables
- func Locate(ctx context.Context, explicit, configured string) (string, bool)
- func ManifestDistLines(body []byte) []byte
- func ParseManifestDistFilenames(manifestPath string) []string
- func PrepopulateFromCache(distdir, cacheDir string, names []string) int
- func Probe(dir string) error
- func Quarantine(distdir string, manifestNames, expected []string) ([]string, error)
- func ReadManifestDistFilenames(manifestPath string) ([]string, error)
- func ResolveCache(userDir, distdir string) string
- func TempRoot(ctx context.Context) string
- type Dir
- type FetchLock
- type FetchScope
- type FileError
- type ManifestMerge
Constants ¶
const DefaultCache = "/var/cache/distfiles"
DefaultCache is the documented system path this package falls back to, and it plays two roles:
- the default for the --distfiles-cache flag: the read-only portage cache consulted to skip re-downloading distfiles already on disk;
- the last resort of Resolve's precedence — used only when the host cannot be asked where its DISTDIR is.
internal/overlay re-exports it as DefaultDistfilesCache so the CLI keeps the name it has always used.
const DownloadSuffix = ".__download__"
DownloadSuffix is the name portage gives a download still in progress: it fetches into "<name>.__download__" and renames to "<name>" only once the file is complete (_download_suffix in portage's package/ebuild/fetch.py). A file carrying it is a partial by construction, whatever its size.
Variables ¶
var ErrDistdirNotWritable = errors.New("distdir is not writable")
ErrDistdirNotWritable reports that the resolved distdir could not be prepared for this run: it does not exist and could not be created, or it exists and the invoking user cannot write to it. Callers test it with errors.Is; the specifics — which directory, and what the operating system said — travel in the wrapped error Probe or Resolve builds around it.
The two conditions share one sentinel because they are one condition ("does not exist, cannot be created, or is not writable") and every caller acts on them identically: fail the manifest step naming the path, and do not hand the failure to the LLM fixer. Splitting them would make the gate in internal/autoupdate ask two questions to reach one answer, and a gate that misses a rung invokes an agent against a machine fault.
var ErrDistfileLocked = errors.New("distfile is locked by another writer")
ErrDistfileLocked reports that another writer holds a distfile this run needs and did not let go within the bounded wait. Callers test it with errors.Is; which distfile, which directory and which process holds it travel in the wrapped message, because those are what an operator acts on.
It is a distinct sentinel from ErrDistdirNotWritable on purpose. "Somebody else is downloading this right now" is a transient state that resolves by itself and whose right answer is to fail this package and let the next run have it; "this directory cannot be written to" is an environment fault that will still be there next time. The failure classifier must be able to tell them apart without reading either message.
var ErrUnsupportedHomeForm = errors.New("unsupported home directory form")
ErrUnsupportedHomeForm is returned (wrapped) for a path whose first element is "~" followed by a user name, such as "~alice/distfiles". Only "~" and "~/..." are expanded; reading "~alice" as "$HOME/alice" would silently point at a directory under the CURRENT user's home, so the form is refused instead.
Functions ¶
func Locate ¶
Locate reports the distdir to READ from, and whether there is one at all.
Resolve's two side effects are wrong for a read-only gate: expandAndCreate CREATES the directory and Probe WRITES into it. A gate that only opens archives on disk needs no write permission, so a portage-owned DISTDIR the user cannot write is usable to it; and a DISTDIR that does not exist is an ANSWER ("nothing here to read"), not a directory to create.
So Locate shares Resolve's precedence (explicit, configured, then the host's `portageq distdir`) and nothing else. It has no DefaultCache rung and no temporary fallback: each would turn "I could not look" into "I looked and found nothing". Do not fold it back into Resolve.
found is false when no rung named a candidate, when the candidate does not exist or is not a directory, or on any other Stat error: the caller's question is "can I read here", not "why not". ctx bounds the host query on the last rung, as it does for Resolve.
func ManifestDistLines ¶
ManifestDistLines keeps the DIST records of a Manifest and drops everything else, each surviving line byte-for-byte.
A staged validation tree holds one candidate ebuild and none of the package's other files, so the published Manifest's EBUILD, AUX and MISC records name files it does not have. Only the DIST records describe what the two trees genuinely share: the upstream archive. (A non-thin repository still refuses a candidate with no EBUILD record; staging imposes thin-manifests for that half.)
Lines, not a parser: Portage verifies these digests, so a kept line must be the bytes that were read. The trailing newline is re-established rather than preserved per line, the one normalisation, and no digest is read through it. Empty in, or no DIST record found, yields nil: "this describes nothing".
func ParseManifestDistFilenames ¶
ParseManifestDistFilenames extracts the filenames listed on `DIST <name> ...` lines of a Gentoo Manifest. Missing files, read errors, or malformed lines yield an empty slice — prepopulation treats this as "nothing to reuse".
func PrepopulateFromCache ¶
PrepopulateFromCache symlinks each cached distfile into distdir so pkgdev can validate it locally instead of re-downloading. Returns the count of successfully linked files. Files already present in distdir, missing from the cache, or causing symlink errors are silently skipped — pkgdev will download them as a fallback.
func Probe ¶
Probe verifies that dir can actually be written to, by creating a small file inside it and removing it again. It returns nil when the directory is usable and an error wrapping ErrDistdirNotWritable when it is not.
It runs before pkgdev, so an unwritable distdir is an environment failure reported at once, not a download failure handed to an LLM fixer that cannot repair a permission. The default is the host's DISTDIR, portage:portage 0775, so writability depends on the invoking user's GROUPS; where they lack portage the answer must be this error naming the directory, never a retreat to somewhere writable such as a tmpfs.
The probe file's name is built from the PID and a counter, never from input, so concurrent calls never collide. It is opened 0600 without O_EXCL, so a leftover from a killed predecessor with a reused PID does not read as unwritable, but with O_NOFOLLOW, so a planted symlink fails with ELOOP and its target is untouched. The file is removed by the same call, including when the write fails. An empty dir is refused: filepath.Join would probe the working directory.
func Quarantine ¶
Quarantine runs before pkgdev: every distfile the new version expects is looked at in the resolved distdir, and one that is present but cannot be verified is moved aside instead of being reused. Portage's default FETCHCOMMAND writes straight to the final name, so a killed fetch leaves a TRUNCATED file there; on a bump the current Manifest does not list it, and the next run would digest it and the overlay publish that checksum.
An absent name needs nothing. A present name listed in manifestNames (the DIST filenames of the CURRENT Manifest) is reused as is. A present unlisted name is renamed to a sibling quarantine name and reported: moved, not deleted, because the host's directory is not ours and a move is recoverable. It returns each file's NEW name (the original is its prefix). On an error the caller must not run pkgdev; what was moved still comes back. A failed inspection fails closed rather than meaning absent.
Names are untrusted and reduced with distfileName (Base, then a lexical refusal of "..", "." and "/") before joining. Lookup is os.Lstat and the move os.Rename, so symlinks, dangling ones included, are moved, never followed; directories are left alone. A concurrent rename's loser sees ENOENT.
func ReadManifestDistFilenames ¶
ReadManifestDistFilenames is ParseManifestDistFilenames' error-returning sibling: the same names for a file that can be read, and the read failure itself for one that cannot.
ParseManifestDistFilenames answers an unreadable Manifest with nil, which is indistinguishable from "readable, and declaring no DIST". That is wrong for a caller whose empty slice is AUTHORITATIVE ("this package publishes no archive"): it would report "I could not look" as a fact. It is one read, not a probe plus a parse, because a Manifest replaced between two reads would answer for two different files.
The error travels unwrapped: each call site words this failure itself, to the byte, because operators read those reports. Unwrapped also keeps errors.Is(err, fs.ErrNotExist) answerable for callers that tell an absent Manifest from an unreadable one.
func ResolveCache ¶
ResolveCache validates the configured cache directory once per run. Returns the absolute path when the directory exists and is a real directory distinct from distdir, or "" when prepopulation should be skipped (cache disabled, missing, unreadable, a "~name" path, a home that cannot be resolved, or pointing at the same path as distdir).
func TempRoot ¶
TempRoot returns the directory a THROWAWAY distdir should be created under — the host's own PORTAGE_TMPDIR — or "" when the question cannot be answered. "" is exactly what os.MkdirTemp reads as "use os.TempDir()", so
os.MkdirTemp(distfiles.TempRoot(ctx), "…")
is disk-backed where possible and today's behaviour elsewhere.
Resolve's answer is shared, persistent and not ours to delete; this one is a directory that is ours, will be deleted, and must not be RAM. The LLM manifest fixer needs it for a private distdir. os.TempDir() on the measured host is a 31 GB tmpfs, the same defect Resolve removes from the manifest step. PORTAGE_TMPDIR is set in make.conf and NOT exported to a user process, so os.Getenv would return "" and land back on the tmpfs: only portageq can answer.
Types ¶
type Dir ¶
type Dir struct {
// Path is the absolute path pkgdev is given as --distdir.
Path string
// Created reports whether this process created the directory for this
// run: only a directory we made is a directory we may remove.
//
// What counts as "we made it" differs between the two entry points, and
// the difference is deliberate — each matches the promise its command
// makes to its users. Do not unify them:
//
// - ResolveOrTemp sets Created only for the temporary directory it
// makes when no path was supplied. A directory the caller NAMED is
// never Created, whether it already existed or had to be mkdir'd: a
// named distdir is a persistent download cache the caller expects to
// find again next run, and it may well be the host's own DISTDIR.
// - Resolve sets Created for any path it had to create, because on that
// path the directory was chosen by this process rather than named by
// the user. A directory this run conjured is one it must take
// away again.
Created bool
}
Dir is a resolved distfiles directory together with the provenance that decides who may delete it.
func Resolve ¶
Resolve returns the directory the autoupdate path gives pkgdev as --distdir, by this precedence, highest first:
- explicit — the --distdir flag, with `overlay manifest`'s meaning;
- configured — autoupdate.distdir from the config file;
- the DISTDIR the host's package manager reports (`portageq distdir`);
- DefaultCache, and only when that query cannot be answered.
os.TempDir() is nowhere in that list on purpose: on the host that hit the defect it is a 31 GB tmpfs, and large fetches died there with wget's "Cannot write to '…' (Success)", the disguise ENOSPC wears on tmpfs. A directory that already existed has Created = false; only a path this call made is Created (stricter than ResolveOrTemp, see Dir.Created).
The chosen directory is returned only after Probe proved it writable, with NO fallback: retreating elsewhere is the defect being removed. On failure the Dir carries the chosen Path, for the message only, with Created = false. A done ctx on the host-query rung returns the cancellation wrapped with %w rather than falling through to DefaultCache.
func ResolveOrTemp ¶
ResolveOrTemp returns the directory `overlay manifest` gives pkgdev as --distdir, keeping the default that command documents in its own --help: an unset --distdir means a temporary directory discarded after the run, which is why that command needs no sudo (see cmd/bentoo/overlay_manifest.go).
When userDir is empty, a temporary directory is created and the returned Dir is Created, so Cleanup removes it when the run finishes. When userDir is set it is expanded (~ and relative paths) and created if missing, and the returned Dir is never Created: the caller asked for that specific path, so it is preserved across runs as a persistent download cache.
That last rule is why this is not Resolve. Resolve marks a directory it had to create as Created; here a named --distdir is a cache by contract and never ours to delete. Both rules are correct for their own caller (see Dir.Created).
On failure the returned Dir is the zero value, whose Cleanup does nothing.
func (Dir) Cleanup ¶
func (d Dir) Cleanup()
Cleanup removes the directory when this process created it, and is a no-op otherwise. It is safe on a zero Dir and safe to call more than once, so `defer dir.Cleanup()` can be written immediately after Resolve returns.
A removal failure is deliberately ignored: cleanup runs when the outcome of the run is already decided, and a leftover temporary directory is housekeeping noise the caller cannot act on.
type FetchLock ¶
type FetchLock struct {
// contains filtered or unexported fields
}
FetchLock is the exclusive claim one manifest step holds over the distfile names it is about to fetch, for the whole of its pkgdev invocation, so that concurrent writers never fetch onto the same path.
It also underpins FetchScope's ownership rule ("absent before the fetch, therefore ours now"), which only holds if nothing else writes those names during the download between RecordFetchScope's Lstat and CleanupFailedFetch's Remove; without the lock our failure branch could delete another worker's download. Quarantine has the same Lstat-to-Rename gap. Hence the order:
resolve -> LOCK -> Quarantine -> prepopulate -> record -> pkgdev -> on failure: cleanup -> RELEASE
Release is safe to defer right after a successful LockFetch and safe to call more than once.
It does NOT cover the host's package manager: portage serializes fetches under FEATURES=distlocks, but pkgdev does not take part, so a sweep and an emerge wanting the same distfile are not serialized by anything. That is a documented limitation; a private handshake with portage would be worse.
func LockFetch ¶
LockFetch takes an exclusive lock on every distfile name in names and returns them as one claim, or an error having released whatever it took: holding three of four locks leaves the fourth unprotected.
Each name is reduced with distfileName and dropped if it is not a filename, which keeps the lock file inside the shared distdir despite untrusted input. Duplicates collapse (locking a name twice would self-deadlock), and names are locked in sorted order so two workers sharing distfiles cannot wait on each other in a cycle. An empty distdir is refused, as in Probe: filepath.Join("", name) would lock in the working directory and separate nobody.
ctx ends the wait as lockWait does, whichever comes first, because a waiter holds a sweep worker slot. A ctx already done is refused before any lock file is created; one that ends during contention ends the wait at once and releases the locks taken so far. Both errors wrap ctx.Err(), not ErrDistfileLocked: being cancelled is not contention.
A caller that gets an error MUST NOT go on to fetch; it reports the package as failed.
func (*FetchLock) Release ¶
func (l *FetchLock) Release()
Release gives up every lock in the claim. It is safe on nil, safe on a claim that holds nothing, and safe to call twice, so `defer lock.Release()` can be written straight after LockFetch returns.
Each lock file is unlinked while its flock is still held and only after the file at the path has been confirmed to be the one this claim created, so a release can never take away a lock that has since become somebody else's.
Nothing is reported. A failed unlink is not something the caller can act on, and it is not something anyone has to act on either: the file left behind belongs to a process that is about to stop existing, and the kernel drops its flock when it does, which is exactly the state the next acquirer reaps. The mechanism that survives a SIGKILL survives a failed unlink for free.
type FetchScope ¶
type FetchScope struct {
// contains filtered or unexported fields
}
FetchScope is the record of which distfiles a package's manifest step may remove if its fetch fails, and the only thing that can authorise a removal.
Only "the incomplete artefact this run created for that fetch" may go. In the host's real DISTDIR that cannot be worked out after the fact: sweep workers run concurrently, so a file another worker is writing and one this run truncated look alike. It is MEASURED BEFORE the fetch instead: expected names absent when the step began are the only names a later file can be ours under. The set lives in an unexported field only RecordFetchScope fills, so no caller can point the removal at another name; the zero value removes nothing, the direction every mistake must fail in.
resolve distdir -> Quarantine -> prepopulate from cache -> RECORD -> pkgdev -> on failure: CleanupFailedFetch
Recording before Quarantine would class a doomed file's name as present and leave a truncated refetch behind. Recording before prepopulation would make cleanup throw away the verified cache links the reuse exists to keep. The per-distfile lock, held across pkgdev, stops a second worker fetching into a recorded name between the record and the failure.
func RecordFetchScope ¶
func RecordFetchScope(distdir string, expected []string) (FetchScope, error)
RecordFetchScope takes the measurement CleanupFailedFetch later spends: which of the distfile names the new version expects are NOT in distdir at the moment the manifest step begins. See FetchScope for where in the step it belongs: the value is a snapshot of one instant.
Each expected name is recorded only when absent. A present name is never recorded, whoever wrote it, which keeps one worker's failure off another's file. A name whose inspection failed for any other reason is not recorded either: a name we could not read is a name we cannot claim.
The lookup is os.Lstat, not os.Stat: os.Stat reports a dangling cache symlink as absent, which would let cleanup unlink a link it never created. Every name is reduced with distfileName first and dropped if it does not reduce to a filename, since the names are untrusted input; duplicates collapse. An empty distdir is refused: filepath.Join("", name) resolves against the working directory.
func (FetchScope) CleanupFailedFetch ¶
func (s FetchScope) CleanupFailedFetch() ([]string, error)
CleanupFailedFetch removes the artefacts this run created and returns their names, so the caller can report them: this package never logs. Call it ONLY on the manifest step's failure branch; after a success the same names are freshly verified distfiles whose reuse must be kept.
Only a recorded name with something under it now is removed, so the result is what actually went away, as bare filenames. A directory is never removed (os.Remove would rmdir an empty host subdirectory); a symlink is unlinked, never followed, as its target is the read-only cache. Each name is checked with distfileName again right before removal, because a name carrying a separator would pass the directory check and this is the operation with the worst blast radius.
A failed removal does not stop the others; failures are joined into one error returned ALONGSIDE the removed names. The zero value and an empty set are no-ops, so a defer can call it on any error; an empty set is also the normal result when every distfile was already on disk. Names with an empty distdir are refused: the removal would resolve against the working directory.
type FileError ¶
type FileError struct {
Name string // the distfile name; empty when the directory itself failed
Source string // the directory the file was read from
Err error
}
FileError is one file a distdir operation could not handle. The operation goes on with the next file, so a caller receives every failure at once and decides how to report it; this package does not log.
func CarryOver ¶
CarryOver moves the completed downloads of one private distdir into another and returns how many moved, plus one FileError per file that could not be.
Only top-level regular files move. A symlink stays behind because the usual one here points into the distfiles cache, and the directory it would land in can belong to an agent holding a Write tool: a link there would let that agent overwrite the cache entry. A directory, fifo or device is never opened (opening a fifo blocks), and a name ending in DownloadSuffix is a partial.
A name already present in to is left alone, whatever it is: nothing is replaced, so an entry planted there is never written through. A move that fails leaves the file where it was. from and to are expected to share a filesystem; across filesystems every rename fails and is reported.
func CopyFromSources ¶
CopyFromSources copies each named distfile that distdir does not yet hold from the first source directory holding it as a regular file, and returns how many were copied plus one FileError per copy that failed.
It copies, never links: the directory it fills can belong to an agent holding a Write tool, and a link there would let that agent overwrite the source. On Linux io.Copy between two files uses copy_file_range, which reflinks on filesystems that support it, so a copy costs no space there.
A source entry is stat'ed before it is opened, and anything that is not a regular file (or a symlink to one) is passed over unopened: opening a fifo blocks. A name that is not a single path element is skipped without a read or a write, since names come from upstream-influenced Manifests. A name already present in distdir is never replaced. Each copy is written under a temporary name and renamed into place only once complete; a failed copy leaves neither the temporary file nor the final name, and the name is not retried from a later source.
type ManifestMerge ¶
type ManifestMerge struct {
Kept, Written, Replaced, Dropped int
}
ManifestMerge counts what MergeManifestDist did: published records kept, staged records written, published records displaced by a staged record of the same filename, and published records dropped for having no usable filename.
func MergeManifestDist ¶
func MergeManifestDist(published, staged []byte) ([]byte, ManifestMerge)
MergeManifestDist returns the DIST records of staged plus those of published whose filename staged does not declare, ordered by filename, each line exactly as read. A filename published declares twice keeps its first record. Non-DIST records are written from neither side; nil means no record at all.
Source Files