Documentation
¶
Overview ¶
Package filecache is a node-local disk cache of downloaded artifacts, keyed by opaque identities (see Key). An artifact wanted by N concurrent callers is fetched once, publication into the cache is atomic and crash-safe, and entries are evicted under byte-budget pressure without breaking consumers.
On-disk layout, under a store's root (which the store owns exclusively):
entries/<sha256(key)>/
data # the cached file (or directory tree)
meta.json # canonical key + creation time; debugging only
tmp/ # in-flight fetches; same filesystem as entries/, so
# publication is one atomic rename
.rm-* # retired entries awaiting removal
An entry directory's mtime is its last-use clock. Nothing on a correctness path reads meta.json: entries are matched to keys by hashing the keys. Everything under entries/ is a complete, published entry; SweepDebris clears crash leftovers from the other two locations at startup.
Contracts ¶
Eviction never invalidates what a caller already received:
- GetFileTo serves a hit as a hard link, created in the same locked step that resolves the hit. Evicting the entry later removes only the cache's own link; the consumer's file stays valid.
- GetFileCopyTo serves a private copy. The consumer owns the inode and may mutate it, and a copy in progress reads a held-open handle, so a concurrent eviction cannot corrupt it.
- The store's min age (WithMinAge) vetoes eviction of entries younger than the window between publication and a consumer's use becoming visible. Size it for the slowest consumer.
Entries are published read-only (0444): a consumer writing through its hard link fails with EACCES instead of corrupting the shared bytes. Consumers that must mutate a file use GetFileCopyTo.
Keys are identities, not addresses. The store never invalidates: a key must name content that is immutable at its source (see URIKey); changed content must arrive under a new key.
Index ¶
- type EvictStats
- type FileFetcher
- type Key
- type Option
- type Store
- func (s *Store) EvictUnused(ctx context.Context, targetBytes int64, dryRun bool) (EvictStats, error)
- func (s *Store) GetFileCopyTo(ctx context.Context, key Key, dst string, fetch FileFetcher) error
- func (s *Store) GetFileTo(ctx context.Context, key Key, dst string, fetch FileFetcher) error
- func (s *Store) SweepDebris(ctx context.Context) (SweepStats, error)
- func (s *Store) TotalBytes(ctx context.Context) (int64, error)
- type SweepStats
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type EvictStats ¶
type EvictStats struct {
// Retired counts entries renamed out of the cache's namespace: from
// that rename on, lookups miss and refetch, whether or not the later
// physical removal succeeded (FreedBytes tracks that part). In a dry
// run, the entries a real pass would have retired.
Retired int
// FreedBytes counts allocated (on-disk) bytes actually returned to the
// filesystem: the footprints of retired entries whose data had no
// consumer hard links left and whose physical removal completed. An
// entry retired but not removed (a RemoveAll failure, reported in the
// returned error) is excluded; its bytes sit in a .rm-* dir until
// SweepDebris. In a dry run this is the would-free estimate.
FreedBytes int64
// PendingBytes counts allocated bytes of retired entries whose data is
// still hard-linked by a consumer: the cache's claim is gone, but the
// kernel frees the space only when the last consumer link is removed.
PendingBytes int64
// SkippedYoung counts entries vetoed by the store's min age.
SkippedYoung int
// SkippedBusy counts entries vetoed at retire time: a hit or a fresh
// fetch moved their last-use clock after the pass listed them.
SkippedBusy int
}
EvictStats reports what an EvictUnused pass did (or, dry-run, would do).
type FileFetcher ¶
FileFetcher downloads a single-file artifact to dstPath, creating the file itself so it can seek and truncate for sparse output. The store runs it at most once per key across concurrent callers, detached from their contexts and bounded by the fetch timeout. Its error reaches every waiting caller wrapped with %w, so errors.Is classification sees through the store.
type Key ¶
type Key struct {
// contains filtered or unexported fields
}
Key identifies a cache entry. It is an identity, not an address: it says nothing about where the bytes come from (the fetcher does), only what they are, so two sources serving the same identity share one entry.
Keys are built via constructors only. The zero Key is invalid.
func SHA256Key ¶
SHA256Key returns the key for a content-addressed artifact, identified by the lowercase hex sha256 of its bytes. Uppercase digits are normalized so equal digests always yield equal keys.
func URIKey ¶
URIKey returns the key for an artifact identified by an immutable source, e.g. URIKey(goldenSnapshotURI, fileName). The parts must identify content that never changes underneath them; the cache has no invalidation, so a republished URI would serve stale bytes forever. Callers pass at least one non-empty part.
type Option ¶
type Option func(*Store)
Option configures a Store.
func WithFetchTimeout ¶
WithFetchTimeout sets the per-fetch bound.
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store is one on-disk cache. It is safe for concurrent use and assumes it is the only writer under its root (one atelet per node).
func (*Store) EvictUnused ¶
func (s *Store) EvictUnused(ctx context.Context, targetBytes int64, dryRun bool) (EvictStats, error)
EvictUnused frees cache space until targetBytes of actually-freeable bytes are reclaimed or no eligible entries remain, least-recently-used first. Entries younger than the store's min age are skipped, and a hit or fetch racing the pass vetoes its entry at retire time. Retiring an entry never invalidates files already served from it: a consumer's hard link keeps its bytes (counted as pending until the link goes), so eviction only ever costs the next caller a re-download.
With dryRun, nothing is touched and the stats report what a real pass would have chosen. Passes are pressure-driven: the caller decides when and how much; a non-positive target is a no-op.
func (*Store) GetFileCopyTo ¶
GetFileCopyTo materializes the artifact identified by key at dst as a private, hole-preserving copy, fetching it with fetch if it is not cached. Unlike GetFileTo's hard link, the caller owns the resulting inode (mode 0600) and may mutate it in place, and dst may be on any filesystem. dst must be an absolute path that does not exist yet.
Concurrent calls for one key share a single fetch, as in GetFileTo. A copy in progress reads a held-open handle, so a concurrent eviction cannot corrupt it.
func (*Store) GetFileTo ¶
GetFileTo materializes the artifact identified by key at dst, fetching it with fetch if it is not cached. dst must be an absolute path that does not exist yet and lives on the cache's filesystem (the same mount, not just the same disk): on success it is a hard link to the read-only cache copy, so the caller keeps a valid file regardless of later eviction, and the entry is published mode 0444 so an in-place write fails loudly rather than corrupting the shared bytes.
Concurrent calls for one key share a single fetch. A caller whose ctx is canceled returns early with ctx.Err() while the fetch keeps running for the others; there is no negative caching, so after a failed fetch the next call starts fresh.
func (*Store) SweepDebris ¶
func (s *Store) SweepDebris(ctx context.Context) (SweepStats, error)
SweepDebris removes crash debris: everything under tmp/ (fetches a crash cut short) and every .rm-* dir at the store root (evictions that renamed but never removed). It must run once at startup, before the store serves requests, and not again: a later sweep would delete the working directories of in-flight fetches. No periodic schedule is needed either — debris only appears at crash or eviction time, and eviction passes retry leftover .rm-* removals themselves.
Removal failures are joined and reported after the sweep visits everything, so one bad path does not shadow the rest; published entries are never touched.
type SweepStats ¶
type SweepStats struct {
// TmpRemoved counts removed tmp/ children (unfinished fetches).
TmpRemoved int
// RetiredRemoved counts removed .rm-* dirs (interrupted evictions).
RetiredRemoved int
}
SweepStats reports what a SweepDebris pass removed.
Source Files