Documentation
¶
Overview ¶
Package ermitegit implements ports.RepoReader over the ermite-git library.
The adapter wraps the gitcli reader. Each method has a route, a constant below: the methods that match gitcli in the shared conformance table and beat it in the benchmarks are served by ermite-git, the others (and GetCommit) by the wrapped reader. A repository that ermite-git cannot read (ErrUnsupported: SHA-256, reftable, alternates, replace refs...) is served by the wrapped reader for every method.
Any other error of the library is returned, never hidden by a fallback. The exceptions are decisions about the data, not failures: a commit whose message is not in UTF-8 (git converts it, the library does not) and an object larger than the library's in-memory bound are read by the wrapped reader.
Freshness: a read that begins after a push has returned sees the push, whichever path made it, because the library reads refs from disk on each call (loose refs always, packed-refs after a stat) and rescans the packs when an object is missing (TestFreshness proves it with no Refresh at all). The adapter still calls Refresh to release replaced packs, pick up a new commit-graph and notice a repository that stopped being supported: at once when ermite itself wrote (the push and the maintenance tell it through Refresh; a merge request merge waits for its maintenance), and as a safety net at most once per Config.RefreshEvery for each repository while it is read, by one reader at a time. A Refresh stats the files of the repository, too much to pay on every read of a method that itself costs 2 µs.
Memory: the cache of open repositories is bounded (LRU) and drops the ones idle longer than Config.IdleTTL; each repository holds at most Config.DeltaCacheSize of delta bases and reads objects up to Config.MaxObjectSize into memory. gitcli's Trash tells the adapter to drop a deleted repository (see Forget).
Index ¶
- Constants
- Variables
- type Adapter
- func (a *Adapter) Close()
- func (a *Adapter) Forget(dir string)
- func (a *Adapter) GetCommit(ctx context.Context, repo domain.RepoName, rev domain.Revision, ...) (domain.CommitDetail, error)
- func (a *Adapter) GetTree(ctx context.Context, repo domain.RepoName, rev domain.Revision, ...) (domain.Tree, error)
- func (a *Adapter) ListCommits(ctx context.Context, repo domain.RepoName, q ports.CommitQuery) (ports.CommitPage, error)
- func (a *Adapter) ListRefs(ctx context.Context, repo domain.RepoName) ([]domain.Ref, error)
- func (a *Adapter) OpenBlob(ctx context.Context, repo domain.RepoName, rev domain.Revision, ...) (domain.BlobInfo, io.ReadCloser, error)
- func (a *Adapter) OpenRepos() int
- func (a *Adapter) Refresh(ctx context.Context, repo domain.RepoName)
- func (a *Adapter) ResolveRefPath(ctx context.Context, repo domain.RepoName, path string) (domain.RefPath, error)
- func (a *Adapter) ResolveRevision(ctx context.Context, repo domain.RepoName, rev domain.Revision) (domain.ResolvedRevision, error)
- type Backend
- type Config
- type Routes
Constants ¶
const DefaultDeltaCacheSize = 8 << 20
DefaultDeltaCacheSize is the budget of the cache of delta bases of each open repository when Config.DeltaCacheSize is zero (the library's own is 32 MiB).
const DefaultIdleTTL = 10 * time.Minute
DefaultIdleTTL is how long a repository may go unused before it is closed, when Config.IdleTTL is zero.
const DefaultMaxObjectSize = 16 << 20
DefaultMaxObjectSize is the largest object a repository holds in memory when Config.MaxObjectSize is zero (the library's own is 128 MiB). A bigger one is read by the wrapped reader.
const DefaultMaxRepos = 64
DefaultMaxRepos is the number of repositories kept open when Config.MaxRepos is zero.
const DefaultRefreshEvery = 5 * time.Second
DefaultRefreshEvery is how long a repository goes without a Refresh when Config.RefreshEvery is zero. See Config.RefreshEvery for what depends on it. A write that goes through ermite refreshes the repository at once (see Adapter.Refresh), so this is only the safety net for the other writers.
Variables ¶
var ErrClosed = errors.New("ermitegit: reader is closed")
ErrClosed is returned by reads after Close.
Functions ¶
This section is empty.
Types ¶
type Adapter ¶
type Adapter struct {
// contains filtered or unexported fields
}
Adapter is a ports.RepoReader that routes each method to ermite-git or to the wrapped reader.
func (*Adapter) Close ¶
func (a *Adapter) Close()
Close closes the open repositories. Reads after Close fail with ErrClosed; blob streams already returned stay valid until they are closed. The wrapped reader is not closed.
func (*Adapter) GetCommit ¶
func (a *Adapter) GetCommit(ctx context.Context, repo domain.RepoName, rev domain.Revision, limits domain.DiffLimits) (domain.CommitDetail, error)
GetCommit is served by the wrapped reader: the diff stays on git.
func (*Adapter) GetTree ¶
func (a *Adapter) GetTree(ctx context.Context, repo domain.RepoName, rev domain.Revision, path domain.TreePath, includeLastCommit bool) (domain.Tree, error)
GetTree lists a directory.
func (*Adapter) ListCommits ¶
func (a *Adapter) ListCommits(ctx context.Context, repo domain.RepoName, q ports.CommitQuery) (ports.CommitPage, error)
ListCommits returns a page of history.
func (*Adapter) OpenBlob ¶
func (a *Adapter) OpenBlob(ctx context.Context, repo domain.RepoName, rev domain.Revision, path domain.TreePath) (domain.BlobInfo, io.ReadCloser, error)
OpenBlob describes a file and returns its content as a stream.
func (*Adapter) Refresh ¶
Refresh tells the adapter that ermite wrote to the repository (a push, the maintenance): the library refreshes it now, if it is open, instead of at the next read after RefreshEvery. A repository that is not open has nothing to refresh. A failure is logged, not returned: reads do not depend on it.
type Backend ¶
type Backend uint8
Backend says which reader serves a method.
type Config ¶
type Config struct {
// Fallback serves the methods that are not routed to ermite-git and the
// repositories ermite-git cannot read. Required.
Fallback ports.RepoReader
// RepoDir returns the directory of a repository. It does not check that it
// exists. Required.
RepoDir func(domain.RepoName) string
// RefreshEvery is the longest a repository goes without a Refresh of the
// library while it is read. Zero means DefaultRefreshEvery; a negative
// value refreshes before every read. It does not affect what a read
// sees: see the package comment.
RefreshEvery time.Duration
// MaxRepos bounds the repositories kept open; zero means DefaultMaxRepos.
MaxRepos int
// IdleTTL is how long a repository may go unused before it is closed.
// Zero means DefaultIdleTTL; a negative value keeps repositories open
// until evicted.
IdleTTL time.Duration
// DeltaCacheSize is the budget in bytes of the cache of delta bases of
// each open repository. Zero means DefaultDeltaCacheSize; a negative value
// disables the cache.
DeltaCacheSize int64
// MaxObjectSize is the size in bytes above which an object that must be
// held in memory is read by the wrapped reader instead. Zero means
// DefaultMaxObjectSize; a negative value means the library default
// (128 MiB).
MaxObjectSize int64
// Routes overrides DefaultRoutes when not nil.
Routes *Routes
}
Config configures an Adapter.
type Routes ¶
type Routes struct {
ListRefs Backend
// ResolveRevision is a revision that is a name or a full id.
ResolveRevision Backend
// ResolveRevisionAbbrev is a revision that is an abbreviated id.
ResolveRevisionAbbrev Backend
ResolveRefPath Backend
// GetTree is a directory listing without last commits.
GetTree Backend
// GetTreeLastCommits is a listing with the last commit of each entry.
GetTreeLastCommits Backend
OpenBlob Backend
// ListCommits is a page of history of a revision.
ListCommits Backend
// ListCommitsPath is a page of the history of a file or a directory.
ListCommitsPath Backend
}
Routes says which reader serves each method. GetCommit is not here: it is always served by the wrapped reader.
func AllRoutes ¶
AllRoutes returns routes that send every method to b. The tests and the benchmarks use it to exercise a backend whatever the production routes are.
func DefaultRoutes ¶
func DefaultRoutes() Routes
DefaultRoutes returns the routes of production.
The rule: a method is routed to ermite-git only when it passes the shared conformance table (TestConformance, TestCompareWithCLI) and its benchmark beats the CLI adapter in time, in bytes AND in allocations per call, warm AND cold, on the generated fixture AND on lazygit (BenchmarkRepoReader, ERMITE_GIT_BENCH_REPO). Where a method has several shapes, every shape must win: the constants GetTree, GetTreeLastCommits, ResolveRevisionAbbrev and ListCommitsPath cover directories of any size and files of any history. A method that does not win stays on the CLI. GetCommit is not routable: the diff stays on git.
Warm is a repository that is open; cold closes it before every call, the price of the first read after the idle TTL, an eviction or a restart (a cold ermite-git call opens the packs and the commit-graph: about 65 µs and 130 allocations before the call itself). The CLI keeps nothing open, so its cold run is its warm run. Only the allocations of the Go process are counted, which favours the CLI (its work happens in git processes). The parallel runs (b.RunParallel) were also made: they agree with the warm runs and widen the time gap.
The numbers are the median of six runs of `go test -bench RepoReader -benchmem -benchtime 500ms` on an AMD Ryzen 9 7900X3D, git 2.56, ermite-git 5228465, as "time bytes allocations per call", CLI against ermite-git (K and M are KiB and MiB). The fixture has 220 commits, 8 directories of 40 files and a 10 MiB blob; lazygit has 7,550 commits. "loses" names what the library does not win: t time, b bytes, a allocations.
shape repo mode cli ermite-git verdict ListRefs fixture warm 905µs 116.3K 125 24µs 19.0K 73 win ListRefs fixture cold 904µs 116.3K 125 88µs 35.2K 203 loses a ListRefs lazygit warm 901µs 105.2K 88 15µs 2.2K 27 win ListRefs lazygit cold 908µs 105.4K 88 88µs 14.2K 163 loses a ResolveRevision/branch fixture warm 22µs 0.9K 19 4µs 0.4K 8 win ResolveRevision/branch fixture cold 22µs 0.9K 19 68µs 16.6K 138 loses tba ResolveRevision/branch lazygit warm 22µs 1.0K 19 4µs 0.5K 8 win ResolveRevision/branch lazygit cold 23µs 1.0K 19 72µs 12.4K 144 loses tba ResolveRevision/fullid fixture warm 9µs 1.3K 22 2µs 0.4K 6 win ResolveRevision/fullid fixture cold 9µs 1.3K 22 65µs 16.6K 136 loses tba ResolveRevision/fullid lazygit warm 9µs 1.3K 22 2µs 0.4K 6 win ResolveRevision/fullid lazygit cold 9µs 1.3K 22 70µs 12.3K 142 loses tba ResolveRevision/abbrev fixture warm 63µs 1.5K 31 9µs 0.7K 11 win ResolveRevision/abbrev fixture cold 64µs 1.5K 31 71µs 16.8K 141 loses tba ResolveRevision/abbrev lazygit warm 63µs 1.5K 31 9µs 0.6K 11 win ResolveRevision/abbrev lazygit cold 63µs 1.5K 31 76µs 12.6K 147 loses tba ResolveRefPath fixture warm 104µs 2.4K 59 15µs 0.8K 22 win ResolveRefPath fixture cold 105µs 2.4K 59 78µs 17.0K 152 loses ba ResolveRefPath lazygit warm 150µs 4.1K 79 22µs 1.4K 29 win ResolveRefPath lazygit cold 148µs 4.1K 79 89µs 13.4K 165 loses ba GetTree/root fixture warm 717µs 108.4K 114 13µs 5.8K 24 win GetTree/root fixture cold 725µs 108.4K 114 78µs 22.4K 154 loses a GetTree/root lazygit warm 827µs 121.4K 167 29µs 18.2K 56 win GetTree/root lazygit cold 831µs 121.5K 167 376µs 1.6M 200 loses ba GetTree/dir fixture warm 742µs 109.5K 141 16µs 9.2K 26 win GetTree/dir fixture cold 745µs 109.5K 141 109µs 75.8K 158 loses a GetTree/dir lazygit warm 1.4ms 267.9K 1058 193µs 318.4K 73 loses b GetTree/dir lazygit cold 1.4ms 268.0K 1058 4.2ms 2.0M 2490 loses tba GetTree/root+last fixture warm 4.0ms 1.2M 1023 70µs 108.3K 107 win GetTree/root+last fixture cold 4.0ms 1.2M 1024 165µs 127.8K 249 win GetTree/root+last lazygit warm 20.2ms 3.8M 3072 6.1ms 367.0K 510 win GetTree/root+last lazygit cold 20.3ms 3.8M 3073 8.5ms 3.4M 4204 loses a GetTree/dir+last fixture warm 5.3ms 1.8M 1454 190µs 166.1K 142 win GetTree/dir+last fixture cold 5.4ms 1.8M 1455 316µs 247.6K 323 win GetTree/dir+last lazygit warm 195.2ms 20.5M 17116 7.9ms 1.5M 855 win GetTree/dir+last lazygit cold 195.9ms 20.5M 17107 14.8ms 5.1M 6989 win OpenBlob/small fixture warm 46µs 6.1K 25 17µs 5.8K 14 win OpenBlob/small fixture cold 47µs 6.1K 25 87µs 27.7K 147 loses tba OpenBlob/small lazygit warm 51µs 5.4K 25 29µs 7.0K 42 loses ba OpenBlob/small lazygit cold 50µs 5.4K 25 131µs 25.1K 184 loses tba OpenBlob/largest fixture warm 21.4ms 69.7K 64 52.5ms 9.0K 13 loses t OpenBlob/largest fixture cold 21.6ms 70.0K 64 51.4ms 26.2K 143 loses ta OpenBlob/largest lazygit warm 3.3ms 69.9K 64 5.7ms 16.3K 123 loses ta OpenBlob/largest lazygit cold 3.3ms 69.9K 64 5.8ms 31.6K 283 loses ta ListCommits/page1 fixture warm 1.0ms 122.5K 404 61µs 19.1K 108 win ListCommits/page1 fixture cold 1.0ms 122.5K 404 134µs 36.9K 244 win ListCommits/page1 lazygit warm 1.3ms 157.2K 415 361µs 120.1K 224 win ListCommits/page1 lazygit cold 1.3ms 157.2K 415 503µs 136.2K 377 win ListCommits/deep fixture warm 1.0ms 122.1K 404 71µs 19.3K 108 win ListCommits/deep fixture cold 1.0ms 122.1K 404 136µs 36.1K 238 win ListCommits/deep lazygit warm 2.0ms 123.2K 404 424µs 23.9K 112 win ListCommits/deep lazygit cold 2.0ms 123.2K 404 544µs 36.3K 248 win ListCommits/path fixture warm 1.6ms 108.8K 138 443µs 25.4K 42 win ListCommits/path fixture cold 1.6ms 108.9K 138 594µs 93.6K 360 loses a ListCommits/path lazygit warm 12.9ms 108.7K 118 6.0ms 29.2K 58 win ListCommits/path lazygit cold 12.8ms 107.7K 118 9.5ms 2.4M 6498 loses ba ListCommits/dirpath fixture warm 1.3ms 123.0K 403 199µs 33.8K 113 win ListCommits/dirpath fixture cold 1.3ms 123.0K 403 277µs 55.4K 268 win ListCommits/dirpath lazygit warm 6.7ms 127.8K 403 2.0ms 50.2K 181 win ListCommits/dirpath lazygit cold 6.7ms 127.8K 403 3.3ms 887.1K 2019 loses ba
What decides the routes, with ermite-git 5228465 (zlib reader with a reused adler32, header-size cache, Refresh and LookupRefID without allocations):
- The warm runs all win but GetTree of a large directory on lazygit (time is won since the zlib change, bytes are not) and OpenBlob, whose allocations come from the inflate of a small blob and whose time is lost for a 10 MiB blob (inflate speed: 52 ms against 21 ms).
- The cold runs lose in allocations for every method that does little work after the open (ListRefs, GetTree, ResolveRefPath: 150 to 200 against 60 to 140), and in time too for the ones that cost a few microseconds warm (ResolveRevision: 65 µs against 9 to 63 µs).
- Only the history of a revision (ListCommits page 1 and deep) wins every line, so it is the only method routed. GetTree dir+last also wins every line, but GetTreeLastCommits covers GetTree root+last too, which loses cold allocations on lazygit (4204 against 3073): it is the nearest miss, and the first to flip once opening a repository allocates less.
- OpenBlob by size: below 1 MiB the small blob loses cold and, on lazygit, warm bytes and allocations; above, it loses time. No size makes it win, so there is no size-based route.
Once opening a repository allocates less, re-run the benchmark and flip the constants it wins: ListRefs, ResolveRefPath and ResolveRevision win warm by 4 to 60 times in time.
Source Files