objstore

Overview ¶

package objstore provides a minimal, memory-mapped Git object store that resolves objects directly from *.pack files without shelling out to the Git executable.

This file implements two cache layers used during object resolution:

• refCountedDeltaWindow: a bounded, reference-counted LRU cache for intermediate delta bases. It keeps recently resolved objects in memory so that subsequent delta applications can reuse them without re-reading the packfile. The window cooperates with Handle-based reference counting so that actively-used entries are never evicted.

• arcCache: an Adaptive Replacement Cache (ARC) that sits above the delta window and caches fully resolved objects. ARC balances recency and frequency to handle both scan-like (one-pass) and lookup-like (repeated access) workloads.

The two caches serve complementary roles: the delta window is scoped to packfile delta resolution and is consulted during the innermost inflate loop, while the ARC cache is the top-level "have I already resolved this OID?" check used by store.get().

commit_attribution.go

Efficient extraction and caching of Git commit author metadata.

Every secret finding needs to be attributed to a commit author (name, email, timestamp). Parsing the raw commit header each time is expensive, so this file provides metaCache -- a concurrency-safe, read-through cache that stores AuthorInfo keyed by commit OID. When a commit-graph file is available, timestamps are served from the precomputed graph slice instead of re-parsing the header.

commit_order.go

Topological commit ordering using Kahn's algorithm backed by a min-heap.

The primary goal is to produce a deterministic parent-before-child ordering of commits so that every parent is visited before any of its children. This is the natural requirement for incremental secret-scanning: we want to scan a parent's tree changes before its child's, so the "seen" set grows in a predictable order.

The algorithm works in three steps:

1. Build an in-degree map over commits whose parents are in the input set.
2. Seed a min-heap with all root commits (in-degree == 0).
3. Pop the minimum-timestamp commit, decrement children's in-degree, and push newly zero-in-degree children onto the heap.

If the input contains cycles (which can happen with grafted or corrupt history), a deterministic timestamp-then-OID fallback appends the remaining commits so we always return every commit exactly once.

delta.go

Two encodings are supported:

• ref‑deltas identify their base object by its 20‑byte object ID.
• ofs‑deltas locate their base by a backward offset within the same packfile.

Delta chains may be deep or even cyclic. The resolver tracks every hop and enforces a caller‑supplied depth limit to prevent infinite recursion and denial‑of‑service attacks.

Internally the implementation uses a reusable "ping‑pong" arena so that each delta step can decode from one half of the buffer while writing into the other, avoiding heap allocations.

ErrDeltaTargetTooLarge is the only exported symbol. All other symbols below this comment are package‑private in order to keep the public surface minimal and stable.

delta_window_sharded.go

Sharded delta-base window for reducing lock contention during concurrent pack-object resolution.

When multiple goroutines resolve delta-compressed objects simultaneously, a single shared window becomes a serialization bottleneck because every acquire and add operation must take the same lock. Sharding the window by the first byte of the object ID distributes that contention across N independent refCountedDeltaWindow instances, each with its own lock and LRU budget. Because Git object IDs are uniformly distributed, the load across shards is approximately even without any additional hashing.

diff_tree.go implements a streaming, memory-efficient tree-to-tree diff for Git object trees.

The algorithm performs a merge-join over the sorted entries of two trees, emitting per-file change callbacks without materialising the full trees in memory. This design supports arbitrarily large repositories because only one tree level is traversed at a time.

PRECONDITION: Git tree entries are stored in Git tree sort order, which is *not* plain lexicographic order. Directories are compared as if their name had a trailing '/' appended (e.g. "foo" < "foo-bar" < "foo.c" < "foo/" when "foo" is a tree). The TreeIter returned by store.treeIter MUST yield entries in this canonical order for the merge-join comparisons (oln < nln, oln == nln) to be correct. Violating this precondition will produce incorrect diffs silently.

hash.go

Core type for Git object identifiers (SHA-1 hashes).

This file defines the Hash type and its basic operations (formatting, parsing, zero-checking). The Hash type is a fixed-size [20]byte array that represents the raw binary form of a SHA-1 digest.

Thread safety: Hash is a value type (fixed-size array) and is safe to copy, compare, and read concurrently without synchronization. There is no shared mutable state. Functions in this file (String, IsZero, ParseHash) are all pure and safe for concurrent use.

Related files:

• hash_aligned.go: provides Hash.Uint64() for architectures with relaxed alignment (all except 32-bit ARM).
• hash_unaligned.go: provides Hash.Uint64() for 32-bit ARM using safe byte-order-aware decoding.

Package objstore offers a content‑addressable store optimized for Git packfiles and commit‑graph data.

This file defines HistoryScanner, a high‑throughput helper that streams commit‑level information and tree‑to‑tree diffs without inflating full commit objects.

Overview ¶

A HistoryScanner wraps an internal object store plus (optionally) the repository's commit‑graph. It exposes a composable API layer focused on **read‑only** analytics workloads such as:

• Scanning every commit once to extract change hunks.
• Iterating trees to build custom indexes.
• Fetching lightweight commit metadata (author, timestamp) on demand.

Callers should construct exactly one HistoryScanner per repository and reuse it for the lifetime of the program. All methods are safe for concurrent use unless their doc comment states otherwise.

Quick start ¶

// Open an existing repository.
s, err := objstore.NewHistoryScanner(".git")
if err != nil {
    log.Fatal(err)
}
defer s.Close()

s.SetMaxDeltaDepth(100) // Tune delta resolution
s.SetVerifyCRC(true)    // Extra integrity checking

// Stream added hunks from every commit.
hunks, errs := s.DiffHistoryHunks()
go func() {
    for h := range hunks {
        fmt.Println(h)
    }
}()
if err := <-errs; err != nil {
    log.Fatal(err)
}

loose_object.go

Reading Git loose objects from the on-disk object store.

A loose object is a single zlib-compressed file stored at <objects-dir>/<xx>/<yy...> where <xx> are the first two hex characters of the SHA-1 OID and <yy...> the remaining 38 characters. The decompressed content has the format:

<type> <size>\0<body>

where <type> is one of "commit", "tree", "blob", or "tag", <size> is the decimal byte length of <body>, and \0 is a single NUL byte separating the header from the payload.

profiling.go

Optional profiling and execution-tracing support for HistoryScanner.

When enabled via the WithProfiling ScannerOption, this file starts an HTTP server that exposes the standard net/http/pprof endpoints and, optionally, an execution trace that is written to disk for the duration of the scan. Neither capability affects scanning correctness; if anything in this file fails the scanner can still operate normally.

scan_mode.go

Scanning strategy selection for HistoryScanner.

Two modes are supported:

• ScanModeBlob (default) -- iterates every unique blob introduced across the commit history in pack-file offset order, yielding the full blob body exactly once per OID. This is the recommended and fastest path.
• ScanModeHunks (legacy) -- computes per-commit diffs and yields only the added-line hunks. Retained for backward compatibility with callers that require line-level attribution, but significantly slower because it must diff every parent-child commit pair.

scan_plan.go implements the blob-scanning pipeline for HistoryScanner.

The pipeline operates in three phases:

1. Candidate collection -- walkBlobCandidates traverses every commit (via walkCommitsFromRefs) and emits one blobRecord per changed blob. Candidates are partitioned into 256 on-disk buckets keyed by the first byte of the blob OID. Bucketing limits the working-set size during the subsequent dedup pass.

2. Dedup / classification -- the prepare step reads each bucket sequentially, deduplicates by blob OID (keeping the first occurrence), consults the caller-supplied SeenSet for cross-run dedup, and classifies each surviving blob as either "packed" (locatable in an open packfile via the index) or "loose" (must be resolved through the loose-object directory). Packed records include the packfile offset so that the execution phase can sort by offset for sequential I/O.

3. Execution -- packed records are sorted by ascending packfile offset and decoded in batches to maximise sequential read throughput on spinning disks and to benefit from OS read-ahead on SSDs. Loose objects are read individually via store.getNoCache.

Two execution strategies are supported:

• Fast path (in-memory): When the total candidate footprint fits within scanFastPathMaxBytes and the blob count is below scanFastPathMaxBlobs, all records are held in memory (inMemoryScanPlan) and no temporary files are created. This eliminates disk I/O overhead for small-to-medium repositories.

• Spill path (disk-backed): When the fast-path thresholds are exceeded, candidates are spilled to temporary files via spillWriter. The external sort uses fixed-size chunks (scanPackSortChunkSize) that are individually sorted and then merged with a min-heap (packedMergeHeap) to produce a globally offset-ordered stream without requiring the entire dataset to fit in memory.

The streaming executor (streamingPackExecutor) provides a third mode that avoids materializing the full plan by flushing per-pack buffers as they reach scanPackFlushRecords or scanPackFlushBytes, trading some I/O ordering for lower peak memory.

Package objstore provides a minimal, memory-mapped Git object store that resolves objects directly from *.pack files without shelling out to the Git executable.

The store is intended for read-only scenarios—such as code search, indexing, and content serving—where low-latency look-ups are required but a full on-disk checkout is unnecessary.

Implementation ¶

The store memory-maps one or more *.pack / *.idx pairs, builds an in-memory map from SHA-1 object IDs to their pack offsets, and inflates objects on demand. It coordinates between pack-index (IDX) and reverse-index (RIDX) data, and builds a unified in-memory lookup across packfiles with transparent delta chain resolution and caching.

All packfiles are memory-mapped for zero-copy access, with an adaptive replacement cache (ARC) and delta window for hot objects. Delta chains are resolved transparently with bounded depth and cycle detection. A small, size-bounded cache avoids redundant decompression, and optional CRC-32 verification can be enabled for additional integrity checks.

The store is safe for concurrent readers and eliminates the need to shell out to Git while providing high-performance object retrieval for read-only workloads like indexing and content serving.