lfsscan

package
v0.20.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 12, 2026 License: Apache-2.0 Imports: 13 Imported by: 0

README

internal/lfsscan

Overview

Lenient Git LFS pointer decoder plus a bounded walk that runs three git subprocesses over a repo's refs to collect the sha256 OIDs that reachable pointer blobs reference. On Linux each walk subprocess gets a 2 GiB RLIMIT_DATA cap.

Interface

  • LFSPointerMaxSize — 1024, the prefix length decoded as a pointer [pointer.go:LFSPointerMaxSize]
  • ParseLFSPointer — decodes bytes as an LFS pointer and returns the OID and ok [pointer.go:ParseLFSPointer]
  • ErrLFSScanBudget — returned when the walk charges more work units than the budget [scan.go:ErrLFSScanBudget]
  • ReachableLFSOIDs — walks all refs and returns the OID set, the units charged, and an error [scan.go:ReachableLFSOIDs]

Invariants

  • ReachableLFSOIDs panics if repoPath is empty or the budget is not positive [scan.go:ReachableLFSOIDs]
  • ParseLFSPointer rejects data over LFSPointerMaxSize or containing NUL, needs a known version line and an oid sha256: line in any order, and needs a 64-char lowercase hex OID; it never panics on arbitrary input [pointer.go:ParseLFSPointer]
  • Each object charges 1 + size/4096 units; exceeding the budget returns ErrLFSScanBudget [scan.go:ReachableLFSOIDs]
  • An invalid rev-list OID, a missing batch-check line, a malformed size or header, or a failed subprocess fails the walk with no OID set [scan.go:ReachableLFSOIDs]
  • Every git call runs with GIT_NO_REPLACE_OBJECTS=1 (gitlimit.NoReplaceObjects), and rev-list runs unfiltered so every object is charged [scan.go:ReachableLFSOIDs]

Side Effects

  • Runs git rev-list --objects --no-object-names --all, git cat-file --batch-check, and git cat-file --batch in repoPath, each holding the repository's gitlimit lease until it exits [scan.go:ReachableLFSOIDs] [lease_test.go:TestReachableLFSOIDsLeaseFollowsEveryStage]
  • Starts three goroutines through errgroup connected by pipes, kills started subprocesses on a later start failure, and reaps all with deferred Wait [scan.go:ReachableLFSOIDs]
  • Calls unix.Prlimit on each subprocess PID on Linux; no-op elsewhere [memlimit_linux.go:limitGitHeap] [memlimit_other.go:limitGitHeap]

Verification

  • Run: go test ./internal/lfsscan/... -tags=openstorage_viewcheck -count=1 -race -timeout=30m

Documentation

Overview

Package lfsscan is the leaf reachability layer of unreferenced-LFS-object reclamation (issue #73): a lenient Git LFS pointer-file decoder and a bounded `git rev-list`/`cat-file` walk collecting the OIDs reachable pointer files reference. It deliberately sits below internal/gitread — internal/jobs consumes it, and gitread's githttp dependency would cycle.

Index

Constants

View Source
const LFSPointerMaxSize = 1024

LFSPointerMaxSize is the pointer-decode window: the prefix of a candidate blob the reclamation scan parses, mirroring git-lfs's blobSizeCutoff — its decoder reads at most this many bytes and accepts a blob of ANY total size whose window decodes (a padded pointer smudges fine, Codex security round 19). The scan therefore parses this window of every reachable blob rather than filtering on total size; a false positive only protects an object from retirement, never endangers one.

Variables

View Source
var ErrLFSScanBudget = errors.New("lfs reachability scan budget exceeded")

ErrLFSScanBudget marks a reachability walk aborted because it charged more work units than the caller's budget allows. The partial OID set is unusable for reclamation — an incomplete reachable set must never retire anything — so callers skip the repository and retry when budget allows.

Functions

func ParseLFSPointer

func ParseLFSPointer(data []byte) (oid string, ok bool)

ParseLFSPointer decodes data as a Git LFS pointer file and returns the referenced sha256 OID (64 lowercase hex characters). ok is false when data is not a pointer.

The decoder is deliberately MORE lenient than git-lfs: reclamation deletes registered objects that no reachable pointer references, so treating a dubious blob as a pointer merely spares an object, while rejecting a real pointer could retire live data. It therefore requires only the spec's two load-bearing properties — a `version` line and an `oid sha256:` key line, each ANYWHERE in the file — and ignores line ordering entirely, blank lines, per-line indentation, unknown keys, and a missing size line. Ordering blindness is deliberate (Codex security rounds 8-10): git-lfs itself decodes blank-led files, whitespace-indented version lines, and ext-* metadata lines preceding `version`, and each positional requirement this parser carried became a false negative that could retire live data — order-free matching closes the class instead of the instance. It never panics on arbitrary input.

func ReachableLFSOIDs

func ReachableLFSOIDs(ctx context.Context, repoPath string, budget int64) (map[string]struct{}, int64, error)

ReachableLFSOIDs walks every object reachable from the cache repo's refs (the DB-authority set Materialize just synced), decodes every non-empty reachable blob's first LFSPointerMaxSize bytes as a Git LFS pointer file (the window git-lfs's own decoder reads, whatever the blob's total size), and returns the set of referenced sha256 OIDs plus the work units the walk charged: one per listed object plus one per lfsScanBytesPerUnit of its raw size — a charge that also prices the content streaming.

The walk streams three git subprocesses — rev-list --objects, cat-file --batch-check (type/size), cat-file --batch (candidate contents) — so memory stays O(pointers found), not O(objects). Aborting past budget kills the pipeline and returns ErrLFSScanBudget; any subprocess failure (including an object reported missing) fails the walk, because reclamation must only ever act on a provably complete scan.

Types

This section is empty.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL