Documentation
¶
Overview ¶
Cache eviction for the machine-local source cache.
Why the cache needed a reclaimer at all ¶
Resolve writes one FULL clone per repo+ref (CacheKey), and the only removal anywhere in this package is the explicit Refresh. A project that bumps its pin every few days therefore accumulates one complete checkout per bump, forever. Measured on a developer machine: <UserCacheDir>/forge/sources held 16 clones of one repository totalling 9.2 GB, because each entry also carries whatever the frontend install wrote into it (web/node_modules was 0.54 GB of a 0.67 GB entry).
Nothing about that is a correctness bug — every entry is a valid, re-fetchable materialization of a reviewable pin — which is exactly why it went unnoticed until the disk filled.
Recording last use: why the metadata file's MTIME ¶
Eviction needs to know when an entry was last USED, which is not when it was fetched. Three candidates, and the choice matters:
- Rewriting a timestamp INTO .forge-source.json. Rejected. That file is the completion marker: its presence is what makes an entry a cache hit (see Resolve). Rewriting it on every hit puts the one file that must never be half-written on the hot path, and a crash mid-write would downgrade a complete 700 MB entry to a miss and force a full re-clone. The cost of being wrong is far higher than the information is worth.
- The entry DIRECTORY's mtime. Rejected. A directory's mtime tracks changes to its own direct children, not reads, so it records the fetch and not the use. It is also perturbed by anything that writes into the tree — an `npm install` in a subdir — which conflates "something wrote here" with "forge resolved this pin".
- The metadata file's MTIME, touched with os.Chtimes. CHOSEN.
Chtimes writes no content, so an interrupted touch cannot corrupt the completion marker; the worst case is a missed touch, which costs one re-fetch of a re-fetchable pin. The file already exists on every complete entry, so caches that predate this code need no backfill: their mtime is the fetch time, which is a correct lower bound for last use. And there is no second marker file to keep in sync with the first.
Why an entry in use cannot be evicted ¶
The recency guard is the first protection. Resolve touches the metadata on every cache HIT, so any process currently building from an entry touched it moments ago and cannot be in a set whose youngest member is MaxAge (default 14 days) old. The open-file scan covers the case the recency guard cannot — a process that resolved an entry and then held it (an open file, or its cwd) for longer than MaxAge, such as a dev server left running from a cached frontend source.
When the scan is unavailable (no lsof, Windows) or does not finish, every candidate is retained. It used to proceed on recency alone, and it also accepted a killed lsof's partial listing as complete; both decided a deletion on evidence that could not see the process using the entry. Retaining costs disk until the next pass; deleting a live checkout costs the process using it.
Nothing persists a resolved cache path into a project's .forge/ state; the resolved directory is rewritten into an in-memory FrontendConfig and never serialized (see internal/cli/frontend_source.go and BuildState, which records image/tag/digest and no source paths). So there is no stored reference an eviction could invalidate.
Package gitsource resolves a declared cross-repo source — "that repo at that ref" — to a directory on the local filesystem, so a component whose code lives in ANOTHER repository can be built from a checkout that contains only this one.
Background: every cross-repo mechanism forge had was filesystem-only. A frontend declared `path = "../reliant/web"` builds on a laptop that happens to have the sibling checkout and fails in CI, where actions/checkout clones a single repository. Worse than the failure is the success: when the sibling IS present, the version that ships is whatever happened to be checked out, so two machines produce different artifacts from identical commits. The Go half of the same dependency has been a pinned, proxy-resolved module all along; only the frontend and image halves were an unpinned path.
A GitSource closes that asymmetry. `repo` + `ref` + optional `subdir` is a reviewable, promotable pin that resolves identically on a laptop, in a container, and in CI — the same property go.mod gives the Go half.
The cache ¶
Fetches land in a machine-local cache keyed by repo AND ref (<UserCacheDir>/forge/sources/<repo-slug>-<digest>), so a second build of the same pin does no network work at all, and two pins of the same repo coexist rather than fighting over one directory. A materialized checkout records what produced it in .forge-source.json; the presence of that file is what makes a directory a cache HIT, so a fetch interrupted halfway leaves no directory that a later build would mistake for complete.
Immutability is the caller's to choose. A full commit sha, or a tag under a project that does not move tags, gives a cache that can never be stale. A branch ref is cached the same way — forge does not re-fetch a branch on every build, because a dependency that silently changes underneath a build is the exact non-reproducibility this package exists to remove. Delete the cache entry (or Refresh) to pick up a moved ref.
The local override — why it is a file and not a mode ¶
Pinning without an escape hatch would force every one-line frontend edit through a push, a tag and a re-fetch, which is how a feature gets resented rather than adopted. So a source may be overridden to a local directory, the same way a gitignored go.work bridges a project's Go side to a local forge checkout during development (`forge project new --link-forge`).
The override lives in `.forge/source-overrides.yaml` — machine-local, gitignored, and read only when present. Two properties are deliberate:
- It is EXPLICIT. forge never auto-adopts a sibling checkout it happens to find on disk. Silent adoption would reintroduce the unpinned build this package exists to eliminate, and it would do it invisibly, on the one machine (a maintainer's laptop) least likely to notice.
- It is DECLARATIVE and out of the build command. The override is state in a file under the developer's control, not a flag that has to be remembered on every invocation or an env var that leaks into a child process.
Because the file is never committed, an override cannot follow a change into CI: the pin is what builds there, always.
Code generated by forge. DO NOT EDIT. forge:hash=8c2a44772b0dded742ceb10a7e8a514c64665b00b51bfff0a6fe859db16a313b forge-owned: regenerated every run — do not edit (forge project disown to take ownership) Source: contract.go in this package.
To customize: edit contract.go (the interface IS the public surface) and re-run "forge generate". This file is regenerated unconditionally.
Index ¶
- Constants
- Variables
- func DefaultCacheRoot() (string, error)
- func LoadOverrides(projectDir string) (map[string]string, error)
- func WriteOverrides(projectDir string, sources map[string]string) error
- type EvictPolicy
- type Eviction
- type Fetcher
- type GitFetcher
- type Metadata
- type MockFetcher
- type MockSourceResolver
- type Option
- type Resolution
- type Resolver
- type Source
- type SourceResolver
Constants ¶
const ( OverridesDirName = ".forge" OverridesFileName = "source-overrides.yaml" )
OverridesDirName / OverridesFileName locate the machine-local source override file, `.forge/source-overrides.yaml`. It sits under `.forge/` with forge's other machine-local state, and it is never committed — which is the property that makes it safe. An override that could travel with a change would silently un-pin CI, defeating the entire point of a pinned source.
const DefaultEvictKeepPerSlug = 2
DefaultEvictKeepPerSlug is how many of the most recently used entries for one repository are retained REGARDLESS of age.
This is the floor that makes eviction safe for an idle project. A repository pinned to one ref and not built for a month has exactly one entry, and it is the entry the next build needs; evicting it on age alone would turn the first build after a holiday into a full re-clone. Two, not one, so that a developer flipping between a release pin and a main pin keeps both warm.
const DefaultEvictMaxAge = 14 * 24 * time.Hour
DefaultEvictMaxAge is how long an entry may go unresolved before it becomes a candidate. Two weeks is long enough that a pin a developer returns to between sprints survives, and short enough that abandoned bumps do not accumulate for a quarter.
const MetadataFile = ".forge-source.json"
MetadataFile is the completion marker inside a cache entry.
Variables ¶
var SlimDirs = []string{"node_modules"}
SlimDirs are the directories an entry's slimming removes: installed dependencies a package manager recreates from the lockfile in the clone.
Functions ¶
func DefaultCacheRoot ¶
DefaultCacheRoot is the machine-local root every fetched source lives under, shared across projects: one fetch of a given pin serves every project on the machine that declares it.
func LoadOverrides ¶
LoadOverrides reads <projectDir>/.forge/source-overrides.yaml. A missing file yields an empty map and no error: having no overrides is the normal state, and the pinned path is what should work by default.
A malformed file IS an error. The failure mode it prevents is the worst one available here — a typo'd override silently ignored, so the build quietly uses the pin while the developer believes they are testing their working copy.
Types ¶
type EvictPolicy ¶
type EvictPolicy struct {
// MaxAge is the unused age beyond which an entry is a candidate.
// Zero means DefaultEvictMaxAge.
MaxAge time.Duration
// KeepPerSlug is the number of most-recently-used entries per
// repository retained regardless of age. Zero means
// DefaultEvictKeepPerSlug; it is never treated as "keep none",
// because a policy that can empty the cache of a repository's only
// pin is a policy nobody wants by accident.
KeepPerSlug int
// InUse reports whether a live process holds the entry directory as
// its cwd or has a file open under it. nil installs the default
// open-file scan (one lsof snapshot, taken once, only when there are
// candidates, and retaining everything when it cannot be taken). Tests
// inject a stub so they never shell out.
InUse func(entry string) bool
// Ctx bounds the eviction: the open-file snapshot, and the removals,
// which stop at the deadline with everything not yet removed retained.
// Nil means unbounded.
Ctx context.Context
// SlimAfter is how long an entry beyond the newest KeepPerSlug of its
// repository may sit unused before its installed dependencies are removed
// (SlimDirs). Zero turns slimming off.
//
// Measured on the machine this was written for: of 17 GB across 37 clones
// of one repository, 14 GB was an installed node_modules (650 MB in 20
// clones) and 1.7 GB was git objects (46 MB per clone). A shared object
// store would save a tenth; the dependency trees are the bytes, and they
// are rebuildable by definition: every consumer of a cached source runs
// the package manager when node_modules is missing (frontenddeps.Ensure),
// so a slimmed clone is a cache hit that costs one install, not a re-fetch.
SlimAfter time.Duration
// BudgetBytes bounds the cache's size. Past it, entries beyond the newest
// KeepPerSlug are evicted least recently used first, down to
// budgetLowWater of it, whatever their age. Zero means no size bound.
//
// The age rule alone cannot bound the cache: a project that bumps its pin
// every few hours produces more clones inside MaxAge than any disk holds.
BudgetBytes int64
// RemoveTree deletes one evicted entry's tree. Machine maintenance passes
// its paced, capped deleter, which may stop partway through a tree; that
// is safe because the entry is renamed out of the cache first (see
// evictingPrefix) and a later Evict finishes the leftover. Nil means an
// unpaced removal of the whole tree.
RemoveTree func(path string) error
}
EvictPolicy bounds what Evict is allowed to remove.
type Eviction ¶
type Eviction struct {
// Removed names the cache entries evicted (or that would be).
Removed []string
// Bytes is the total size of Removed, as measured before removal.
Bytes int64
// Kept is the number of entries retained for any reason.
Kept int
// Held names entries a live process was using, which are always
// retained. Reported separately from Kept's reasons because it is
// the one retention a caller may want to re-run for later.
Held []string
// Slimmed names the entries whose installed dependencies were removed
// (or would be), and SlimBytes is what that frees. The entry itself stays
// a cache hit.
Slimmed []string
SlimBytes int64
// OverBudget is set when the cache is still larger than BudgetBytes after
// everything evictable was: the newest KeepPerSlug clones of each
// repository, and entries a live process holds, are never removed.
OverBudget bool
// TotalBytes is the cache's size before this pass, measured only when a
// budget is set.
TotalBytes int64
}
Eviction reports what Evict did, or would do under apply=false. The counts are returned rather than only printed so a test can assert on the decision instead of parsing prose.
func Evict ¶
func Evict(root string, now time.Time, policy EvictPolicy, apply bool, out io.Writer) (Eviction, error)
Evict removes source-cache entries under root that no project has resolved for longer than the policy's MaxAge, keeping the newest KeepPerSlug entries for every repository regardless of age.
It is deliberately a package-level function over a root rather than a Resolver method: the caller that needs it is machine-wide maintenance (forge storage gc), which has no project and no override file, and making it a method would force that caller to construct a Resolver whose other fields are meaningless to it.
apply=false previews. A missing root is not an error — a machine that has never fetched a source has nothing to reclaim.
Evict FAILS CLOSED on anything it cannot read: an entry whose metadata is missing, unparseable, or unstattable is retained and reported, never removed. Debris from an interrupted fetch is in that set; Resolve clears it the next time that exact pin is resolved, which is a cheaper place to handle it than a reclaimer that would have to guess.
type Fetcher ¶
type Fetcher interface {
// Fetch checks repo out at ref into dst (which does not yet exist).
// It returns the resolved commit sha when it knows one; an empty
// string is acceptable and only costs auditability.
Fetch(ctx context.Context, src Source, dst string) (commit string, err error)
}
Fetcher materializes a source into a destination directory. It exists as an interface so the resolver can be tested without a network: the production implementation shells out to git, and a test injects a fake.
A Fetcher is called only on a cache MISS, and must leave dst either fully populated or absent — the resolver writes the completion marker, so a Fetcher that fails partway is retried rather than trusted.
type GitFetcher ¶
type GitFetcher struct{}
GitFetcher materializes a source with the git CLI.
It fetches ONE ref at depth 1 rather than cloning the repository, which for a large monorepo is the difference between a few megabytes and a few hundred. The shallow fetch is retried unshallowed when the server refuses it: fetching a bare commit sha requires the remote to have uploadpack.allowReachableSHA1InWant enabled, which many hosts do not, and a pin by sha is the MOST reproducible thing a user can write — it must not be the one that fails.
type Metadata ¶
type Metadata struct {
Repo string `json:"repo"`
Ref string `json:"ref"`
// Commit is the sha the ref resolved to at fetch time. This is what
// makes a branch-ref build auditable after the fact — the pin says
// "main", the metadata says which main.
Commit string `json:"commit,omitempty"`
}
Metadata is the record a materialized cache entry carries. Its presence is what marks the entry COMPLETE: a fetch writes it last, so an interrupted fetch leaves a directory no later build treats as a hit.
type MockFetcher ¶
type MockFetcher struct {
contractkit.Recorder
FetchFunc func(context.Context, Source, string) (string, error)
}
MockFetcher is a test mock for the Fetcher interface.
The embedded contractkit.Recorder records every call so tests can assert call counts and captured arguments. Set XxxFunc fields to override per-method behaviour; unset methods return the canonical "MockFetcher.<Method>Func not set" error.
type MockSourceResolver ¶
type MockSourceResolver struct {
contractkit.Recorder
ResolveFunc func(context.Context, Source) (Resolution, error)
RefreshFunc func(Source) error
}
MockSourceResolver is a test mock for the SourceResolver interface.
The embedded contractkit.Recorder records every call so tests can assert call counts and captured arguments. Set XxxFunc fields to override per-method behaviour; unset methods return the canonical "MockSourceResolver.<Method>Func not set" error.
func (*MockSourceResolver) Refresh ¶
func (m *MockSourceResolver) Refresh(src Source) error
Refresh records the call and delegates to RefreshFunc when set.
func (*MockSourceResolver) Resolve ¶
func (m *MockSourceResolver) Resolve(ctx context.Context, src Source) (Resolution, error)
Resolve records the call and delegates to ResolveFunc when set.
type Option ¶
type Option func(*Resolver)
Option configures a Resolver.
func WithCacheRoot ¶
WithCacheRoot overrides the machine-local cache location. Tests point it at a temp dir so they never touch the developer's real cache.
func WithFetcher ¶
WithFetcher replaces the git fetcher. Tests inject a fake; nothing in production needs this.
func WithLogger ¶
WithLogger routes the resolver's one-line-per-source reporting.
func WithOverrides ¶
WithOverrides sets the repo→local-directory override map directly, bypassing the on-disk file. Used by tests and by callers that have already loaded the file themselves.
type Resolution ¶
type Resolution struct {
// Dir is the directory the component's source is at, including any
// subdir. This is what a build shells into.
Dir string
// Overridden is true when a local override stood in for the pin.
Overridden bool
// Cached is true when a fetched pin was already materialized, so no
// network work happened.
Cached bool
// Commit is the resolved sha, when known.
Commit string
}
Resolution reports how a source was resolved, so callers can tell a reproducible pin from a local override in their own output — which is the whole point of surfacing it.
type Resolver ¶
type Resolver struct {
// contains filtered or unexported fields
}
Resolver turns declared sources into directories on disk.
The zero value is not usable; construct one with NewResolver.
func NewResolver ¶
NewResolver builds a resolver for a project rooted at projectDir.
Unless overridden by an Option it uses the real git fetcher, the machine-local cache under the user cache dir, and the override file at <projectDir>/.forge/source-overrides.yaml. A missing or unreadable override file is not an error — the overwhelmingly common case is that there is none.
func (*Resolver) Refresh ¶
Refresh drops the cache entry for a source so the next Resolve re-fetches it — the supported way to pick up a moved branch or tag. A source that was never fetched is a no-op.
func (*Resolver) Resolve ¶
Resolve returns the directory holding the source's code, fetching it into the cache if needed.
Precedence is: local override, then cache, then fetch. An override wins over a pin unconditionally and by design — it is the developer saying "build what is in front of me" — and it is reported, never silent, because a build that quietly stopped honoring its pin is exactly the failure this package removes.
type Source ¶
type Source struct {
// Repo is the repository to fetch. The canonical spelling is
// host/owner/name ("github.com/reliant-labs/reliant"), which forge
// expands to an https clone URL. An explicit https:// or ssh:// URL,
// or scp-style git@host:owner/name, is passed through untouched — a
// project on a private host must not be forced through a spelling
// forge invented.
Repo string `yaml:"repo" json:"repo"`
// Ref is the tag, branch, or full commit sha to check out. Required:
// a source with no ref is the unpinned filesystem dependency this
// type exists to replace, so forge refuses to guess a default branch.
Ref string `yaml:"ref" json:"ref"`
// Subdir is the path WITHIN the repository that this component's
// source lives at ("web" for a repo whose SPA is in web/). Empty
// means the repository root.
Subdir string `yaml:"subdir,omitempty" json:"subdir,omitempty"`
}
Source is a declared cross-repo dependency: a repository, a ref to pin it at, and optionally a subdirectory within it. It mirrors the `forge.GitSource` KCL schema and the `source:` block in forge.yaml.
func (Source) CacheKey ¶
CacheKey is the cache directory NAME for this source: a readable slug plus a digest of the exact repo+ref.
The digest covers repo and ref but NOT subdir — the whole repository is materialized once and every subdir of the same pin is a view into it, so a project depending on two directories of one repo fetches once.
func (Source) CloneURL ¶
CloneURL is the URL forge hands to git. The canonical host/owner/name shorthand expands to https; an explicit URL passes through so a private host, an ssh remote, or a credential-helper-backed URL all keep working.
func (Source) IsZero ¶
IsZero reports whether the source is entirely unset — the common case of a component that declares a filesystem path instead.
type SourceResolver ¶
type SourceResolver interface {
// Resolve returns the directory holding the source's code, fetching it
// into the cache if needed. A local override wins over the pin and is
// reported in the Resolution, never silently.
Resolve(ctx context.Context, src Source) (Resolution, error)
// Refresh drops the cache entry for a source so the next Resolve
// re-fetches it — the supported way to pick up a moved branch or tag.
Refresh(src Source) error
}
SourceResolver turns a declared source into the directory holding its code, and drops a cached copy so the next resolution re-fetches it.
Source Files