githttp

package
v0.19.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 30, 2026 License: Apache-2.0 Imports: 38 Imported by: 0

README

internal/githttp

Overview

The git smart-HTTP and Git LFS server. Server authenticates JWTs, resolves repos, hydrates a bare cache repo from the ref store and blob store, and runs git upload-pack and git receive-pack. The same binary runs as the git hook (RunHook) that enforces ref policy, quota, push holds, signature verification, and the authoritative ref CAS during a push. The package also serves the LFS batch and object APIs with verified uploads, quota accounting, and legacy-key adoption.

Interface

  • Server (fields Store, Blob, CacheDir, Verifier, Org, TestMode, ResolveRepo, MaybeRepack, PackQuotaBytes, PackReservationTTL, BaseURL, MaxPushBytes, VerifySigBudget, LFSStagingBytes, LFSV020WriteCompat), NewServer, (*Server).Handler [server.go:Server]
  • Routes: GET /{repo}.git/info/refs, POST /{repo}.git/git-upload-pack, POST /{repo}.git/git-receive-pack, prefix /{repo}.git/info/lfs/ [server.go:Handler]
  • LFS routes: POST /{repo}.git/info/lfs/objects/batch, GET and PUT /{repo}.git/info/lfs/objects/{oid} [lfs.go:handleLFS]
  • Cache repos: ValidRepoName, BaseRepo, RepoPath, EnsureCacheRepo, EnsureCacheRepoCtx, ValidateCacheDir, HeadTarget [hydrate.go:EnsureCacheRepoCtx]
  • Hydration: Hydrate, HydratePacks, MaterializePack, ErrPackMissing [hydrate.go:Hydrate]
  • Hooks: RunHook (pre-receive, update, post-receive, reference-transaction), WrapHookRefStore [hook.go:RunHook]
  • Push publication: WaitForPushPublication, ErrPushPublicationUnverified [publication.go:WaitForPushPublication]
  • Quota: EnvPackQuota, EnvPackReservationTTL, EnvUsageFloor, MinPackReservationTTL, UsageFloorHandoffMargin, CachedUsageFloor, CachedUsageFloorGen [quota.go:CachedUsageFloor]
  • SetRateLimited — Retry-After and RateLimit headers for 429s [ratelimited.go:SetRateLimited]
  • Hook env names: EnvRepo, EnvSyncClaim, EnvRefsPolicy, EnvOps, EnvScopes, EnvOrg, EnvPushHold, EnvAtomic (OPENSTORAGE_GIT_*, OPENSTORAGE_ORG, OPENSTORAGE_SYNC_CLAIM), EnvVerifyState, EnvVerifySigBudget, EnvVerifySpawnLog [hook.go:EnvRepo] [verifystate.go:EnvVerifyState]
  • Test-mode names: EnvTestMode, EnvCrash, EnvHookDelay, EnvPushDrainMS, EnvCASDrop, EnvCASDropHard, EnvUpdateBarrier, EnvUpdateBarrierEach, EnvPreReceiveBarrier, header CrashHeader (X-Openstorage-Test-Crash) [hook.go:EnvTestMode]
  • Hook store URLs: OPENSTORAGE_PG_URL/PG_URL, OPENSTORAGE_S3_URL/BLOB_URL [hook.go:envHookPGURL]

Invariants

  • assertf panics on programming errors; request-dependent failures map to 4xx [server.go:assertf]
  • Git auth requires basic auth with username t; a bad token is 401; a wrong org, an org-level or mismatched repo claim, or a missing scope is 403 [server.go:verifyClaims]
  • Auth runs before repo resolution, so an unknown repo answers 404 only after auth [server.go:handleServiceRPC]
  • CrashHeader is refused with 403 unless TestMode [server.go:handleServiceRPC]
  • Receive-pack bodies over the push cap and gzip upload-pack bodies over 4 MiB answer 413 [server.go:handleServiceRPC] [uploadcap.go:maxUploadPackBytes]
  • sniffAtomicPush fails closed as atomic when the peek budget is exhausted [atomic.go:sniffAtomicPush]
  • preReceive rejects malformed stdin lines and zero commands, and releases all push holds on error [hook.go:preReceive]
  • Atomic pushes are denied whole, before any hold is reserved, on a policy denial or a D/F conflict inside the pushed commands [hook.go:preReceive] [hook.go:atomicPushDFCheck]; after the holds are reserved and before any update-hook CAS, a stale old SHA or a D/F conflict with refs already in the store also denies an atomic push whole [hook.go:preReceive] [hook.go:authorityDoomedCreations]
  • preReceivePolicyGate re-checks the policy verdicts and refuses pack publication that only denied updates need [hook.go:preReceivePolicyGate]
  • pushRefUpdatable rejects names deeper than 32, with a segment over 200 bytes, or over 900 bytes [hook.go:pushRefUpdatable]
  • updateHook checks policy, reservation, and the sync-claim fence before the CAS; unknown policy ops fail closed [hook.go:updateHook] [hook.go:evalRefPolicy]
  • Hold reservation fails closed on unreadable peer holds and waits at most 10 s for conflicting holds [hydrate.go:reservePushHoldsCovering]
  • Hydrate publishes the generation marker only after a full sync and re-reads the generation under the hydrate lock [hydrate.go:Hydrate]
  • hydratePacksBy retries a missing pack blob a bounded number of times, then returns ErrPackMissing [hydrate.go:hydratePacksBy]
  • Cache repos pin receive.unpackLimit=0 and gc.auto=0; a pinned value that survives repair fails the call [hydrate.go:EnsureCacheRepoCtx]
  • LFS OIDs must match ^[a-f0-9]{64}$; batches are capped at 4 MiB and 1000 objects; only basic transfer and sha256 [lfs.go:handleLFSBatch]
  • LFS PUT needs a Content-Length (411), must hash to the OID (400), answers 413 when its Content-Length exceeds the per-repo staging allowance and 429 when the staging budget is exhausted, and is limited by ref.MaxLFSObjectsPerRepo (507) [lfs.go:lfsProxyUpload]
  • Registered LFS objects are served only from the canonical key [lfs.go:lfsBatchDownload]
  • chargeWalk fails closed through exhaust past the verify walk budget [verifystate.go:chargeWalk]
  • recordCommitGraphOutcome asserts a known outcome [commitgraph.go:recordCommitGraphOutcome]

Side Effects

  • Runs git upload-pack, receive-pack, init --bare, config, index-pack, update-ref --stdin, for-each-ref, symbolic-ref, verify-commit, rev-list, cat-file, merge-base, and commit-graph write subprocesses [server.go:handleServiceRPC] [hydrate.go:runGitCtx]
  • Creates cache repos with hook scripts that re-exec this binary [hydrate.go:hookScript]
  • Downloads pack blobs into objects/pack; uploads packs and LFS objects; deletes and copies LFS blobs [hydrate.go:hydrateOnePack] [lfs.go:lfsProxyUpload]
  • Writes push-hold files and sidecars under codestorage.push-holds, and marker files codestorage.generation and codestorage.generation.held; takes flocks on lock files [hydrate.go:writeGenerationMarker]
  • Starts push-hold refresher, adoption-claim renewal, and repack-probe goroutines [hydrate.go:startPushHoldRefresh] [server.go:dispatchRepackProbe]
  • Reserves quota, registers packs and LFS objects, and CASes refs in the ref store [hook.go:preReceive] [hook.go:updateHook]
  • Spills large receive-pack output to an unlinked temp file [pushcap.go:pushGate.Write]
  • Writes verify-sig state files and imports signing keys with gpg --import in a temp dir [verifystate.go:appendTo] [hook.go:newSigVerifier]
  • Increments metrics.CommitGraphRefresh; logs through log.Printf [commitgraph.go:recordCommitGraphOutcome]
  • Test-mode crash knobs call os.Exit(42) [hook.go:preReceive]

Verification

  • Run: go test ./internal/githttp/... -tags=openstorage_viewcheck -count=1 -race -timeout=30m (default pg lane; needs Docker for Postgres 17 + SeaweedFS through testcontainers)
  • Local lane (no Docker): OPENSTORAGE_TEST_BACKEND=local go test ./internal/githttp -count=1 -race -timeout=15m
  • s3wal lane (needs Docker): OPENSTORAGE_TEST_BACKEND=s3wal go test ./internal/githttp -tags=openstorage_viewcheck -count=1 -race -timeout=25m
  • Floci nightly lane (needs Docker, a Floci container, and STS session credentials; see docs/reference/testing.md): AWS_REGION=us-east-1 OPENSTORAGE_TEST_S3_ENDPOINT=http://127.0.0.1:4566 go test ./internal/githttp -count=1 -race -timeout=25m
  • Fuzz nightly lane, one target per run (FuzzSniffAtomicPush, FuzzLFSBatch, FuzzPushPublicationScan): go test ./internal/githttp -run '^$' -fuzz '^FuzzLFSBatch$' -fuzztime 4m -parallel 8 -timeout 30m
  • Benchmark nightly (.github/workflows/bench-nightly.yml, defaults BENCHTIME=1s, COUNT=1): the whole-repo sweep OPENSTORAGE_TEST_BACKEND=local go test -p 1 ./... -run '^$' -bench . -benchmem -benchtime 1s -count 1 -timeout 50m, then the push-path bench on the s3wal lane (needs Docker) OPENSTORAGE_TEST_BACKEND=s3wal go test ./internal/githttp/ -run '^$' -bench 'BenchmarkPushE2E' -benchtime 1x -count 1 -timeout 50m

Documentation

Overview

commitgraph.go refreshes the split commit-graph of a cache repo after a full hydrate (issue #307, ADR-47). The graph is rebuildable cache state: read-side history walks (`git log -- <path>`, `--topo-order`) use it when present and fall back to parsing commit objects for anything it does not cover, so every failure here is logged and dropped, never returned to a request.

Git LFS Batch API (docs/explanation/git-protocol-and-namespaces.md): the storage host serves the standard LFS discovery path POST /{repo}.git/info/lfs/objects/batch on the same JWT-authenticated remote — no separate LFS server or config. Object bytes live in the BlobStore under blob.LFSKey(repoID, oid); transfers use the "basic" adapter.

DOWNLOADS use presigned S3 URLs when the backend supports them, falling back to proxying through this server (fileblob local mode) via GET /{repo}.git/info/lfs/objects/{oid}. Reads are never quota-gated.

UPLOADS always go through the verified proxy transfer (PUT /{repo}.git/info/lfs/objects/{oid}, issue #51): a presigned PUT binds neither content length nor digest with the current signer (gocloud.dev SignedURLOptions cannot sign Content-Length), so only the server-verified path can anchor the storage-quota ledger and the sha256 content address. Upload admission is a durable reservation (ref.ReservePackBytes) charged at batch time on the DECLARED size and re-charged at PUT time on the request's ACTUAL Content-Length (the declared-size-lie gate); a verified upload settles the reservation into the usage counter via ref.RegisterLFSObject, which is also the registry the LFS orphan sweep classifies against.

Package-level helper for throttled (429) responses. Upstream's OpenAPI documents TWO headers on every 429: Retry-After and RateLimit (issue #150). RateLimit is the current IETF draft shape (draft-ietf-httpapi-ratelimit-headers, §4): a structured-field service limit Item — a quoted quota-policy name with parameters r (available quota) and t (effective window in seconds). On a throttled response r is by definition 0, and t mirrors Retry-After's delay-seconds: the draft (§6) requires Retry-After to reference a point no earlier than the end of the effective window, and equal values are the exact boundary (the draft's own throttled-response example, appendix B.1.4, does the same).

The policy name is opaque to clients: upstream deliberately advertises no RateLimit-Policy field ("the gateway's capacity is dynamic"), so the identifier here is the draft's canonical example name "default".

This helper is the single source of truth for the wire shape across both serving surfaces: the LFS busy/backpressure 429s in this package and the api package's rate-limiter middleware and busy handlers (the api package imports githttp; the reverse edge would cycle).

Package githttp is the git smart HTTP surface (docs/explanation/git-protocol-and-namespaces.md): stateless HTTP nodes with Postgres as ref authority, S3/MinIO as immutable pack storage, and local disk as a rebuildable cache. Authentication follows docs/reference/authentication.md: customer-signed JWTs verified against the issuing org's registered keys.

Index

Constants

View Source
const (
	EnvRepo = "OPENSTORAGE_GIT_REPO"
	// EnvSyncClaim carries the loopback sync push's claim-holder token
	// (auth.Claims.SyncClaim, issue #86): the update hook refuses the ref
	// CAS when the repo's sync claim is no longer held by this holder,
	// fencing a stale run's in-flight force push out of the ref store.
	EnvSyncClaim  = "OPENSTORAGE_SYNC_CLAIM"
	EnvRefsPolicy = "OPENSTORAGE_GIT_REFS_POLICY"
	EnvOps        = "OPENSTORAGE_GIT_OPS"
	// EnvScopes carries the pushing token's scope list (JSON array) into
	// the hook subprocess. The update hook itself never consults it; the
	// hook-side forward_on_push poke does (issue #90): auto-forwarding
	// exports every local head/tag upstream, so a write-only token's push
	// must not trigger it (the same git:read gate the explicit
	// push-upstream route enforces).
	EnvScopes = "OPENSTORAGE_GIT_SCOPES"
	// EnvOrg names the org whose signing keys verify-sig loads. Shared with
	// the serve config on purpose: the server sets it explicitly for hooks.
	EnvOrg = "OPENSTORAGE_ORG"
	// EnvPushHold carries the absolute path of this push's hold file
	// (issue #170, hydrate.go push-hold machinery). pre-receive reserves
	// every ref of the push in it before any update-hook CAS can run;
	// concurrent full syncs skip held refs so they cannot race
	// receive-pack's own local update of a ref it just committed, and
	// D/F-conflicting concurrent pushes serialize on the reservation.
	// Minted per request by the server and stripped from inherited env
	// like the other per-push inputs.
	EnvPushHold = "OPENSTORAGE_GIT_PUSH_HOLD"
	// EnvAtomic marks the push `--atomic` (issue #28). githooks(5) exposes
	// no atomic indicator, so the HTTP layer sniffs the wire capability
	// (atomic.go) and sets this; pre-receive then evaluates the policy
	// across ALL commands and denies before publication when any command
	// is denied — git would abort every ref on the first update-hook
	// denial anyway.
	EnvAtomic    = "OPENSTORAGE_GIT_ATOMIC"
	EnvTestMode  = "OPENSTORAGE_TEST_MODE"
	EnvCrash     = "OPENSTORAGE_TEST_CRASH"
	EnvHookDelay = "OPENSTORAGE_TEST_HOOK_DELAY_MS"
	// EnvPushDrainMS shrinks the post-copy drain read deadline
	// (pushDrainTimeout) so the e2e lane can prove hook phases outlive it
	// (issue #141's receive-pack sibling); ignored outside test mode.
	EnvPushDrainMS = "OPENSTORAGE_TEST_PUSH_DRAIN_MS"
	// EnvCASDrop/EnvCASDropHard hold a marker-file path; the first update
	// hook to create the marker lets its CAS statement commit, then reports
	// a synthetic connection-drop error (issue #2's ambiguous outcome).
	// The hard variant also fails the recovery recheck read, as a truly
	// dead connection would. Later invocations pass through untouched.
	EnvCASDrop     = "OPENSTORAGE_TEST_CAS_DROP"
	EnvCASDropHard = "OPENSTORAGE_TEST_CAS_DROP_HARD"
	// EnvUpdateBarrier holds a handshake-file path prefix; the first update
	// hook to create <path>.ready pauses AFTER its authority CAS commits
	// and before the hook exits — i.e. before receive-pack performs its own
	// local ref update — until the test creates <path>.release (issue #170:
	// this is exactly the window a concurrent hydrate can full-sync the
	// just-committed ref into the shared cache repo). Later invocations
	// pass through untouched.
	EnvUpdateBarrier = "OPENSTORAGE_TEST_UPDATE_BARRIER"
	// EnvUpdateBarrierEach is EnvUpdateBarrier's per-command variant: a
	// handshake-path PREFIX at which EVERY update hook pauses after its
	// committed authority CAS, each under a ref-derived suffix (issue
	// #186 finding 5: the coalescing e2e lane interleaves advertisements
	// BETWEEN a fanout's hook CASes, which the one-shot barrier cannot
	// stage).
	EnvUpdateBarrierEach = "OPENSTORAGE_TEST_UPDATE_BARRIER_EACH"
	// EnvPreReceiveBarrier holds a handshake-file path prefix; the first
	// pre-receive to create <path>.ready pauses AFTER pack publication
	// and BEFORE the deferred hold reservation (Codex round-23 security
	// P2: exactly the window in which a concurrent same-name or
	// D/F-relative push must be able to proceed) until the test creates
	// <path>.release. Later invocations pass through untouched.
	EnvPreReceiveBarrier = "OPENSTORAGE_TEST_PRERECEIVE_BARRIER"
	// CrashHeader lets tests request an EnvCrash injection per push; the
	// server refuses it outside test mode.
	CrashHeader = "X-Openstorage-Test-Crash"
)

Env names used to route request context from the HTTP handler into the git hooks (which re-exec this binary). OPENSTORAGE_TEST_* fault-injection knobs are honored only when OPENSTORAGE_TEST_MODE=1.

View Source
const EnvPackQuota = "OPENSTORAGE_GIT_PACK_QUOTA"

EnvPackQuota carries the resolved per-repo pack-bytes quota from the serve process into the pre-receive hook (set only when the quota is enabled).

View Source
const EnvPackReservationTTL = "OPENSTORAGE_GIT_PACK_RESERVATION_TTL"

EnvPackReservationTTL carries the resolved reservation TTL (issue #29, a Go duration string) from the serve process into the pre-receive hook. Set alongside EnvPackQuota; the hook falls back to defaultReservationTTL when it is absent (a serve binary predating the ledger), so quota enforcement never silently weakens on skew.

View Source
const EnvUsageFloor = "OPENSTORAGE_GIT_USAGE_FLOOR"

EnvUsageFloor carries the repo's UNCOUNTED-bytes admission floor (the packs + LFS listing minus the usage counter at scan start — Codex round 44) from the serve process's shared single-flight cache into the pre-receive hook (Codex round 21): the hook is a fresh process per push, so a hook-side listing cannot be cached across pushes and re-scans both prefixes every time — ~100 pages per push at the LFS object cap. Set alongside EnvPackQuota as "<bytes>:<unix mint time>" (Codex round 30): the value is minted at receive-pack spawn but the hook runs only after the client's transfer, whose duration the CLIENT controls, so the hook self-lists when the floor has aged past hookFloorMaxAge instead of trusting it — beyond the reservation TTL, orphaned uploads whose reservations expired are visible only to a listing.

View Source
const EnvVerifySigBudget = "OPENSTORAGE_GIT_VERIFY_SIG_BUDGET"

EnvVerifySigBudget carries the resolved per-push verified-commit budget (issue #59) from the serve process into the hooks. Set only when the budget is enabled (> 0); malformed values fail closed like EnvPackQuota.

View Source
const EnvVerifySpawnLog = "OPENSTORAGE_TEST_VERIFY_SPAWN_LOG"

EnvVerifySpawnLog is a test-mode-only knob (EnvCASDrop pattern): when set, every actual `git verify-commit` spawn appends its sha to the named file, so tests can assert exact spawn counts across the pre-receive gate and every update hook of one push.

View Source
const EnvVerifyState = "OPENSTORAGE_GIT_VERIFY_STATE"

EnvVerifyState carries the per-push verify-sig state file path from the serve process into the hooks. The file is created empty by the server before receive-pack starts and removed when the request ends; hooks append verdict lines as they verify. Absent (a serve binary predating issue #59, or a bare hook invocation) the hooks fall back to an in-process cache — correctness is unchanged, only cross-process reuse is lost.

View Source
const MinPackReservationTTL = usageCacheTTL + UsageFloorHandoffMargin

MinPackReservationTTL is the exclusive lower bound serve's config enforces on OPENSTORAGE_PACK_RESERVATION_TTL (Codex rounds 36+42 on #51): the reservation TTL must STRICTLY exceed the usage-floor cache lifetime plus the floor-to-reservation handoff margin. CachedUsageFloor feeds LFS and REST admission directly with a listing up to usageCacheTTL stale, and the subsequent reserve call takes up to the margin; with a TTL at or below their sum, an orphaned upload's reservation (published, registration failed) can expire while a floor listed BEFORE the publish still drives an admission — the writer then sees neither the orphaned bytes nor a live reservation and is admitted into headroom that no longer exists. The hook path tolerates such TTLs by self-listing (hookUsageFloor's maxAge goes non-positive), but the direct cache consumers have no fresher source, so the configuration is rejected at startup instead.

View Source
const UsageFloorHandoffMargin = 30 * time.Second

UsageFloorHandoffMargin is the freshness headroom the reservation TTL must leave BEYOND the floor cache lifetime (Codex round 42 on #51): a floor can be served at the very end of its usageCacheTTL window and then sit while ReservePackBytes waits on the database, so the interval from scan start to the admission snapshot is floor age plus handoff time. Charging reserve calls are deadline-bounded by this margin, which makes the chain airtight: an orphan published after its page was scanned (publish p > scan start t0) had its reservation refreshed at the pre-publish admission, so it gates until p+TTL, while any admission using a floor from t0 snapshots by t0 + usageCacheTTL + margin < p+TTL. The reservation stamps and the snapshot share the DATABASE clock, so only the duration — not absolute clocks — must agree.

Variables

View Source
var ErrPackMissing = errors.New("githttp: registered pack missing from blob store")

ErrPackMissing is returned (wrapped, match with errors.Is) when a registered pack's blob stays missing after maxHydrateRaces listing passes: the ref store references bytes that do not exist, which is corruption grade — refs may point at unreachable objects.

View Source
var ErrPushPublicationUnverified = errors.New("push publication wait: live push holds could not be checked; publication stays held")

ErrPushPublicationUnverified marks a WaitForPushPublication that could not prove every live push hold released: a hold or its tombstones were unreadable, or checking them would exceed the wait's resource bounds (issue #331). Publication must then be treated as still pending.

View Source
var WrapHookRefStore = func(ctx context.Context, st ref.RefStore) (ref.RefStore, error) {
	return ref.IdempotentCAS{RefStore: st}, nil
}

WrapHookRefStore decorates the ref store the update hook opens for its CAS. Hooks run in a fresh `openstorage hook` subprocess (git execs the installed hook script), so the serve process's wired store cannot reach them; instead the cmd wiring seam — which runs in the hook subprocess too, it is the same binary — replaces this variable at startup with the events.RefStore decorator so git pushes emit push events (docs/reference/webhooks.md) exactly like API writes (that decorator wraps ref.IdempotentCAS itself). The eventless default keeps hooks working before/without eventing wiring, wrapping just the CAS-recovery decorator so ambiguous outcomes still reconcile. An error refuses the ref update before its CAS (issues #445, #466: the metadata document is in a format this binary cannot write events to, or corrupt).

Functions

func BaseRepo

func BaseRepo(repo string) string

BaseRepo strips the optional namespace suffix: JWTs carry the base repo id in the `repo` claim and are valid for its namespaces (docs/explanation/git-protocol-and-namespaces.md URL scheme).

func EnsureCacheRepo

func EnsureCacheRepo(cacheDir, repo string) (string, error)

EnsureCacheRepo idempotently creates the bare cache repo with the server config and hooks that re-exec this binary (no shell/psql deps in hooks).

Config and hooks are verified on every call, not only at init time: a crash between `git init` and hook installation (or a deleted/tampered hook file) must never leave a repo whose receive-pack silently runs without the hooks that own S3 upload, PG pack/ref registration, and policy enforcement. The hook script also embeds this binary's path, which changes across deployments.

The steady-state verification is memoized (issue #93): the full pass — two `git config --get` forks plus per-hook ReadFile/Stat — used to run on every request and dominated the hydration fast path (~93% of it, PR #97 baseline). Every mutator of the verified state is our own code (`git init` and the `git config` writes below rewrite config via lockfile + rename; ensureHook rewrites hooks via temp + rename; nothing in this codebase deletes cache repos), so a fingerprint of the exact file states the last full pass in THIS process verified proves them unchanged without forking — see cacheRepoMemo.

func EnsureCacheRepoCtx

func EnsureCacheRepoCtx(ctx context.Context, cacheDir, repo string) (repoPath string, retErr error)

EnsureCacheRepoCtx is EnsureCacheRepo honoring ctx in its lock wait and git subprocesses (init, config): the repack fold's materialization runs the setup under the fold's wall-clock budget, and a stalled subprocess or filesystem must not outlive it.

func HeadTarget

func HeadTarget(defaultBranch string) string

HeadTarget qualifies a repo's recorded default branch into the HEAD symref target its cache repo advertises (openapi.yaml PATCH repo: `main` becomes `refs/heads/main`; a value that already starts with `refs/` is used verbatim).

func Hydrate

func Hydrate(ctx context.Context, st ref.RefStore, bl blob.BlobStore, cacheDir, repo, defaultBranch string) (repoPath string, refsErr error, err error)

Hydrate rebuilds the cache repo state for repo from PG + S3 (the stateless node property): ensure repo, hydrate packs, then sync refs. On the receive path a concurrent receive-pack may hold ref locks while we sync; retry briefly, and let the caller decide whether a residual failure is fatal (upload path) or tolerable staleness (receive path, where git re-validates old SHAs itself).

Fast path (issue #4): when the store generation AND the HEAD symref target derived from defaultBranch (issue #18) match the local marker written by the last fully successful hydrate — and the marker is within its TTL — the packs, refs, and HEAD are provably unchanged and Hydrate returns after one point-read, skipping ListPacks, per-pack stats, the paged ListRefsAfter scan, and the for-each-ref/update-ref/ symbolic-ref subprocesses. EnsureCacheRepo still runs unconditionally: hook and config repair must stay verified on every request (the hook script embeds this binary's path, which changes across deployments), and it touches no DB state. After the first full pass its verification is memoized fingerprint stats (issue #93), so the fast path as a whole forks zero subprocesses.

defaultBranch is the repo's default branch as resolved from the meta row for THIS request. A PATCH /repo landing after that read can transiently lose to this hydrate's older value (each full sync records the HEAD it actually set, so the marker never lies), and the next request — which resolves the row fresh — repairs HEAD with one full sync.

Full syncs serialize per repo on <repoPath>/codestorage.hydrate.lock: two overlapping full hydrates spanning a mutation could otherwise interleave so the OLDER ref sync lands after the newer one while the newer marker survives — and the marker would then short-circuit reads onto stale refs for up to a TTL window. Under the lock, each full sync re-reads the generation before listing refs and publishes its marker before releasing, so marker and refs advance monotonically together. The marker read stays lock-free (temp+rename keeps it atomic): the idle fast path must not serialize on an exclusive lock.

func HydratePacks

func HydratePacks(ctx context.Context, st ref.RefStore, bl blob.BlobStore, repoPath, repo string) error

HydratePacks downloads and indexes every registered pack that is not already present in the cache repo. Packs are self-contained (index-pack --fix-thin ran on the receiving node before upload), so each is independently indexable.

Concurrent hydrators (serve requests, re-exec'd hooks) race on the same cache dir, so each pack is materialized under a per-pack flock: download to a uniquely-named temp file, index the temp, then rename pack+idx into place. Only our own temp names are ever removed, and "idx appeared while waiting for the lock" counts as success.

A download hitting blob.ErrNotFound restarts the pass on a fresh ListPacks (bounded by maxHydrateRaces): the pack was folded away by a concurrent repack and the new listing carries its replacement. Restarts are cheap and idempotent — hydrated packs are skipped by the .idx probe. Transport errors keep failing fast; on bound exhaustion the hydrate fails with ErrPackMissing rather than looping.

func MaterializePack

func MaterializePack(ctx context.Context, bl blob.BlobStore, repoPath string, p ref.Pack) error

MaterializePack materializes ONE registered pack into repoPath — the bounded, cancelable alternative to HydratePacks for callers that must not touch any other pack (the issue-#1 repack fold selects its input under byte/object budgets BEFORE downloading; hydrating the whole repo first would defeat them). Safe against concurrent hydrators of the same cache repo: identical per-pack lock, temp naming, and rename ordering. A download hitting blob.ErrNotFound propagates (wrapped, match with errors.Is): for a fold it means a concurrent repack already replaced the snapshotted rows.

func RepoPath

func RepoPath(cacheDir, repo string) string

RepoPath maps a repo name to its bare cache repo directory.

func RunHook

func RunHook(ctx context.Context, args []string) error

RunHook dispatches `openstorage hook <pre-receive|update> [args...]`, invoked by git via the scripts EnsureCacheRepo installs. Arguments and stdin come from outside this process, so malformed input must produce an error (stderr message + non-zero exit denies the push), never a panic.

func SetRateLimited

func SetRateLimited(h http.Header, retryAfterSecs int)

SetRateLimited stamps the two throttling headers carried by every HTTP-level 429 the server emits. retryAfterSecs is the delay-seconds value (>= 1, the Retry-After granularity).

func ValidRepoName

func ValidRepoName(repo string) bool

ValidRepoName guards against path traversal via the URL repo element.

func ValidateCacheDir

func ValidateCacheDir(cacheDir string) error

ValidateCacheDir rejects cache locations too deep for the push-hold bookkeeping this package materializes beside each repository (Codex round-25 P2): the deepest auxiliary path — a hold's unreserved-marker sidecar entry — needs more room than git's own files, so a cache root that leaves hydration and git ref locking working can still fail every push with an opaque mid-push error. Validated once, where the operator's choice is accepted (serve boot), against the target OS PATH_MAX with a generous repo-component budget (repo directory names are server-generated and short); the per-request mint stays the defensive backstop for anything that slips through.

func WaitForPushPublication

func WaitForPushPublication(ctx context.Context, cacheDir, repo string) error

WaitForPushPublication blocks until no live push hold in repo's cache still names an unreleased ref: every accepted authority update that a hold protected has published its cache ref (or its push has exited). A caller that has already claimed its durable intent (issue #266) uses it to avoid acting on a cache ref an in-flight push has not yet published. Nothing on main calls it yet.

Result contract:

  • nil: no live, unreleased held name remained at the last poll.
  • ctx.Err() (unwrapped) on cancellation or deadline.
  • otherwise an error wrapping ErrPushPublicationUnverified: a hold was unreadable or exceeded the resource bounds. Publication was NOT proven. The caller must not start the protected work, and must keep its durable intent, force flag, and retry scheduling exactly as for any failed wait; a later retry re-checks from scratch. The case is also logged here.

Bounds (defaultPublicationLimits): holds are streamed, never collected or sorted. Each poll reads a hold's tombstone sidecar into a bounded set, then its lines until the first live name. Bytes are charged as they are read, so file growth after the listing cannot exceed the budget (at most one probe byte past it), and each record is charged before it is retained. A record longer than pushRefPathMax fails closed. Verdicts are cached per hold by (size, mtime) of the hold and its sidecar, so only changed holds are reread.

Types

type Server

type Server struct {
	Store    ref.RefStore
	Blob     blob.BlobStore
	CacheDir string
	// Verifier authenticates docs/reference/authentication.md JWTs (multi-algorithm, iss-resolved
	// org keys, exp/sub checks). Any Verify failure maps to 401.
	Verifier auth.Verifier
	// Org is the single organization this node serves. A valid token from a
	// *different* registered org must not grant access (403); unknown orgs
	// already fail key resolution inside Verifier (401).
	Org string
	// TestMode gates fault-injection knobs (crash header). Only ever set
	// from OPENSTORAGE_TEST_MODE=1; the crash header is refused otherwise.
	TestMode bool
	// ResolveRepo maps a base path element (docs/explanation/git-protocol-and-namespaces.md: repo_id, but ADR-21
	// accepts name-style urls too) to its live repo record. ok=false when
	// no live repo matches — both handlers then answer 404 (repos must be
	// created via POST /api/v1/repos first). The record's RepoID is the
	// only storage key (repo-key-split fix); it also lets tokens carrying
	// either the url or the repo_id claim (docs/reference/authentication.md) authenticate. When
	// nil, every path element is unresolvable.
	ResolveRepo func(ctx context.Context, base string) (meta.Repo, bool)
	// MaybeRepack, when set, runs after every receive-pack POST with the
	// resolved storage key (issue #1): pre-receive registers pushed packs
	// strictly before receive-pack returns, so this is the git surface's
	// post-RegisterPacks repack-threshold trigger. It must be quick
	// (a pack-count read plus at most one job enqueue) and must never
	// block on the fold itself. Nil disables triggering.
	MaybeRepack func(ctx context.Context, repo string)

	// PackQuotaBytes is the per-repo pack-bytes quota (issue #21, see
	// quota.go); <= 0 disables quota enforcement. Set by the serve wiring
	// from the resolved config, like ResolveRepo.
	PackQuotaBytes int64
	// PackReservationTTL bounds how long a crashed push's quota
	// reservation gates admission (issue #29, ref/ledger.go). Set by the
	// serve wiring; <= 0 leaves the hook on defaultReservationTTL.
	PackReservationTTL time.Duration
	// BaseURL is the externally reachable base URL (scheme://host, no
	// trailing slash) LFS proxy transfer hrefs are built on (issue #51:
	// uploads always proxy, so behind a reverse proxy the href must be
	// the canonical external URL — the backend request's Host header may
	// be internal). Set by the serve wiring from $OPENSTORAGE_BASE_URL
	// (or the bound listener); when empty, hrefs fall back to the
	// request's Host and X-Forwarded-Proto.
	BaseURL string
	// MaxPushBytes caps the DECODED bytes one receive-pack request may
	// stream into git (issue #55, pushcap.go); <= 0 disables the cap. Set
	// by the serve wiring (default: a small multiple of the pack quota).
	MaxPushBytes int64

	// VerifySigBudget caps how many commits one push may signature-verify
	// (issue #59, verifystate.go); <= 0 disables the budget. Set by the
	// serve wiring.
	VerifySigBudget int

	// LFSStagingBytes bounds the aggregate bytes of proxy LFS uploads
	// staging on this replica at once (issue #51, Codex round 4): uploads
	// stage their full Content-Length under $TMPDIR before verification,
	// and the per-repository quota is neither node-wide nor guaranteed
	// enabled, so without this budget concurrent distinct-key uploads
	// could exhaust a replica's disk. Set by the serve wiring
	// ($OPENSTORAGE_LFS_STAGING_BYTES, default 10 GiB); <= 0 disables the
	// bound (the operator's explicit opt-out).
	LFSStagingBytes int64
	// LFSV020WriteCompat keeps the v0.2.0 LFS read surface alive during a
	// rolling upgrade (ADR-41, Codex round 80): old replicas derive only
	// the legacy single-key spelling, so a mixed-version fleet would 404
	// objects that exist only canonically. When set (serve wiring,
	// $OPENSTORAGE_LFS_V020_WRITE_COMPAT), every verified publication
	// also copies the bytes to the legacy key — rowless copies the orphan
	// sweep reclaims after the flag is dropped post-roll. Off by default:
	// the canonical spelling is the secure one, and the compat copies are
	// exactly as forgeable as everything v0.2.0 served.
	LFSV020WriteCompat bool
	// contains filtered or unexported fields
}

Server is one stateless HTTP node: all durable truth lives in PG (refs, pack registry) and S3 (pack bytes); CacheDir is rebuildable at any time.

func NewServer

func NewServer(st ref.RefStore, bl blob.BlobStore, cacheDir string, verifier auth.Verifier, org string) *Server

func (*Server) CachedUsageFloor

func (s *Server) CachedUsageFloor(ctx context.Context, repo string) (int64, error)

CachedUsageFloor returns repo's UNCOUNTED-bytes admission floor — the listing minus the usage counter at that listing's scan start, clamped at zero (Codex round 44: ReservePackBytes SUMS this floor with the live counter, so settlements after the scan count exactly once; the old max() combination hid them whenever pre-registry bytes coexisted with fresh settlements). Serves a ≤usageCacheTTL-stale cached scan and coalesces concurrent misses into one flight (issue #51, Codex round 15: fresh per-transfer listings amplify — a 100-object batch cost ~201 full prefix scans). It feeds every admission surface: LFS transfers here, the pre-receive hook via its spawn env (EnvUsageFloor), and REST pack writers via api.Config.UsageFloor (Codex round 21 — per-push self-listings in those paths re-amplify at the LFS object cap). Staleness is safe FOR THE FLOOR: bytes registered since the scan are in the live counter, in-flight uploads in their reservations, regardless of this value; the floor exists for pre-ledger bytes (fixed) and orphaned uploads, whose admission visibility this cache delays by at most its TTL — a bounded stretch of the ledger's documented orphan-window overshoot. Unlike peekUsage (observational, never blocks), admission needs a value: a cold entry lists inline, concurrent cold callers share the leader's flight.

func (*Server) CachedUsageFloorGen

func (s *Server) CachedUsageFloorGen(ctx context.Context, repo string) (int64, int64, error)

CachedUsageFloorGen is CachedUsageFloor plus the shared floor generation the served value was validated against — the token admission callers forward into ReservePackBytes' transactional recheck (Codex round 60).

func (*Server) Handler

func (s *Server) Handler() http.Handler

Handler routes the smart-HTTP and LFS endpoints per repo (docs/explanation/git-protocol-and-namespaces.md):

GET  /{repo}.git/info/refs?service=git-{upload|receive}-pack
POST /{repo}.git/git-{upload|receive}-pack
POST /{repo}.git/info/lfs/objects/batch
GET  /{repo}.git/info/lfs/objects/{oid}   (proxy transfer, local mode)
PUT  /{repo}.git/info/lfs/objects/{oid}   (proxy transfer, local mode)

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL