gitlimit

package
v0.19.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 30, 2026 License: Apache-2.0 Imports: 10 Imported by: 0

README

internal/gitlimit

Overview

Shared admission and process control for git subprocesses. It provides a slot pool that bounds concurrent git reads and maintenance, a registry that starts, reaps, and stops git child process groups, and a per-child heap cap on Linux through ulimit -d.

Interface

  • HeavyHeapBytes (256 MiB), ReadSlotCount (8), RepoSlotCount (2), ReadSlotWait (1 s), MaintenanceShare (2) — sizing constants [gitlimit.go:HeavyHeapBytes]
  • ErrBusy, ErrMemoryLimit — no slot in time, and git stopped by its own allocation failure [gitlimit.go:ErrBusy]
  • Slots, NewSlots, (*Slots).Acquire, TryAcquire, Release, InUse — slot pool with a per-key cap [gitlimit.go:Slots]
  • RepoKey — per-repository slot key from a canonical repo ID [gitlimit.go:RepoKey]
  • ReadSlots — process-wide pool shared by JSON git reads and admitted maintenance [gitlimit.go:ReadSlots]
  • TryAdmitMaintenance — non-blocking admission of one maintenance job; returns a release func [gitlimit.go:TryAdmitMaintenance]
  • ErrStopping, Start, Wait, Run, Output, CombinedOutput, Stopping, StopChildren — tracked git child lifecycle [children.go:Start]
  • HeapLimitSupported, Command, HeapFailure — heap cap support (Linux: /bin/sh wrapper with ulimit -d; other platforms: no cap) and heap-failure classification [heaplimit_linux.go:Command] [heaplimit_other.go:Command]

Invariants

  • NewSlots panics unless n > 0, 0 < perKey <= n, and wait >= 0 [gitlimit.go:NewSlots]
  • Release panics unless the key holds a slot; RepoKey panics on an empty repo ID [gitlimit.go:Release]
  • TryAdmitMaintenance takes the maintenance permit first, releases it if ReadSlots is full, and panics on a second release call [gitlimit.go:TryAdmitMaintenance]
  • Acquire with wait 0 returns ErrBusy at once when no slot is free [gitlimit.go:Acquire]
  • Output panics if the caller already set Stdout; CombinedOutput panics if the caller already set Stdout or Stderr [children.go:Output] [children.go:CombinedOutput]
  • registry.start panics if SysProcAttr is already set, forces Setpgid, and refuses new children with ErrStopping after stop began [children.go:start]
  • registry.stop sends SIGTERM before SIGKILL so git can clean its lock files [children.go:stop]
  • The Linux wrapper keeps a stricter inherited soft limit and exits 125 if it cannot read or set the limit [heaplimit_linux.go:heapLimitWrapper]
  • HeapFailure ignores SIGKILL and matches git diagnostics only at line start [heaplimit_linux.go:HeapFailure]
  • Non-Linux Command asserts maxHeapBytes == 0 [heaplimit_other.go:Command]

Side Effects

  • Starts git children (through /bin/sh when a heap cap is set), each as its own process group [children.go:start]
  • Sends SIGTERM then SIGKILL to tracked groups in StopChildren, polling every 10 ms until the grace deadline [children.go:stop]
  • Holds mutexes on slot state and on the child registry; release closes and replaces the freed channel to wake waiters [gitlimit.go:Release]
  • Output and CombinedOutput buffer child output in memory [children.go:Output]

Verification

  • Run: go test ./internal/gitlimit/... -tags=openstorage_viewcheck -count=1 -race -timeout=30m

Documentation

Overview

Package gitlimit is the shared chokepoint for git subprocesses that untrusted input can make expensive (#313, #322): the process-wide admission pool, the per-child heap cap, and their error values. The read API (internal/gitread, ADR-049) and cache maintenance (internal/githttp's commit-graph refresh, ADR-47) both use it, so they share one admission pool instead of each bounding itself. It also tracks serve's git children for the shutdown stop (children.go). It is a leaf package: gitread imports githttp, so the shared pieces cannot live in gitread.

Index

Constants

View Source
const (
	// HeavyHeapBytes caps the heap (RLIMIT_DATA, Linux) of a child whose
	// work grows with history length, tree size, blob size, or output
	// size. See gitread.HeavyHeapBytes for the measurements.
	HeavyHeapBytes int64 = 256 << 20

	// ReadSlotCount is the size of ReadSlots: requests with JSON
	// responses that run git, plus admitted maintenance, at once in this
	// process.
	ReadSlotCount = 8
	// RepoSlotCount is how many slots of a pool one repository may hold.
	RepoSlotCount = 2
	// ReadSlotWait is how long a read request waits for a ReadSlots slot.
	ReadSlotWait = time.Second
	// MaintenanceShare is how many ReadSlots slots optional maintenance
	// (commit-graph refreshes) may hold at once, so it can never take the
	// whole pool from reads.
	MaintenanceShare = 2
)

Variables

View Source
var (
	// ErrBusy: no admission slot freed up in time; git was not started.
	ErrBusy = errors.New("git read capacity exhausted")
	// ErrMemoryLimit: git needed more heap than its cap and was stopped by
	// its own allocation failure (Linux only).
	ErrMemoryLimit = errors.New("git exceeded its memory limit")
)
View Source
var ErrStopping = errors.New("server is shutting down")

ErrStopping refuses a git child started after StopChildren began.

ReadSlots is the process-wide pool shared by JSON git reads (the read API's budget middleware) and admitted maintenance.

Functions

func CombinedOutput added in v0.19.0

func CombinedOutput(cmd *exec.Cmd) ([]byte, error)

CombinedOutput is exec.Cmd.CombinedOutput through the registry.

func Command

func Command(ctx context.Context, maxHeapBytes int64, args ...string) *exec.Cmd

Command returns `git args...`. When maxHeapBytes > 0 it runs git behind /bin/sh, which sets RLIMIT_DATA (`ulimit -d`, KiB) and then execs git, so the limit is in force before git allocates anything and the process id stays git's (context kill and Wait still reach git). An inherited stricter limit is kept. If the limit cannot be read or set, the shell exits 125 and git never runs.

RLIMIT_DATA counts the heap and other private writable mappings, which includes every thread's stack and malloc arena; read-only pack mappings (shared page cache) are not counted.

func HeapFailure

func HeapFailure(waitErr error, stderr string) bool

HeapFailure reports whether a git run under a heap cap died of it: git's fatal allocation diagnostics, or, when the cap is hit before or outside git's checked allocators (dynamic loader, libc), a crash by SIGSEGV, SIGBUS, or SIGABRT. SIGKILL (context cancellation or deadline) is not a heap failure.

Only a fatal line that STARTS with a known diagnostic counts: git messages can quote request input (a path named "out of memory"), and such text never forms the start of the line.

func HeapLimitSupported

func HeapLimitSupported() bool

HeapLimitSupported reports whether Run can cap a git child's heap on this platform (RLIMIT_DATA via `ulimit -d`).

func Output added in v0.19.0

func Output(cmd *exec.Cmd) ([]byte, error)

Output is exec.Cmd.Output through the registry. cmd.Stdout must be nil.

func RepoKey

func RepoKey(repoID string) string

RepoKey is the Slots key of a repository: its canonical RepoID, so every spelling of one repository shares one per-repository quota.

func Run added in v0.19.0

func Run(cmd *exec.Cmd) error

Run is Start followed by Wait.

func Start added in v0.19.0

func Start(cmd *exec.Cmd) error

Start starts cmd as the leader of a new process group and tracks it.

func StopChildren added in v0.19.0

func StopChildren(grace time.Duration) int

StopChildren ends every tracked git child and refuses new ones: serve calls it when the shutdown drain times out, so no tracked child outlives serve. It sends SIGTERM to each child's process group, waits up to grace for them to be reaped, then sends SIGKILL to those groups. It returns the number of groups it signaled.

func Stopping added in v0.19.0

func Stopping() bool

Stopping reports whether StopChildren began.

func TryAdmitMaintenance

func TryAdmitMaintenance(key string) (release func(), ok bool)

TryAdmitMaintenance admits one optional maintenance job for the repository key (RepoKey) into ReadSlots without waiting: it needs one of the MaintenanceShare maintenance permits AND one ReadSlots slot within that repository's share, both taken with non-blocking tries (no check-then-act on counts). So at most MaintenanceShare jobs run, reads always keep the rest of the pool, and a repository's maintenance and reads share its per-repository quota. ok=false means skip the job; a caller holding other locks never blocks here. release must be called exactly once when ok.

func Wait added in v0.19.0

func Wait(cmd *exec.Cmd) error

Wait waits for a child that Start started and stops tracking its group.

Types

type Slots

type Slots struct {
	// contains filtered or unexported fields
}

Slots is a pool of admission slots. One key (a repository, RepoKey) may hold at most perKey of them, so slow work on one repository cannot take the whole pool from the others. A holder releases exactly the slots it acquired, under the same key.

func NewSlots

func NewSlots(n, perKey int, wait time.Duration) *Slots

NewSlots returns a pool of n slots, at most perKey of them per key. Acquire waits at most wait for a free slot before failing with ErrBusy.

func (*Slots) Acquire

func (s *Slots) Acquire(ctx context.Context, key string) error

Acquire takes a slot for key, waiting at most the pool's wait (ErrBusy) or until ctx ends (its error).

func (*Slots) InUse

func (s *Slots) InUse() int

InUse reports how many slots are held (tests and diagnostics).

func (*Slots) Release

func (s *Slots) Release(key string)

Release returns one slot taken for key.

func (*Slots) TryAcquire

func (s *Slots) TryAcquire(key string) bool

TryAcquire takes a slot for key if one is free right now.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL