packhorse

Packhorse

The Git caching proxy that protects Gitaly from stampeding herds

Packhorse is a high-performance Git caching service designed to solve Gitaly scaling challenges for ultra-scale customers. When hundreds of CI jobs clone the same repository simultaneously, Packhorse coalesces requests, caches packfiles, and leverages Git's packfile-URI capability to reduce upstream load by up to 99.4%.

Why Packhorse?

Packhorse is to Gitaly what Workhorse is to Rails—a protective layer that handles the heavy lifting. Just as Workhorse shields Rails from expensive file operations, Packhorse shields Gitaly from the crushing load of concurrent CI clones.

The name follows GitLab tradition: Rails ran on Unicorn, and Workhorse was built to protect it by being the opposite—tough and hard-working. Packhorse follows the same philosophy for Git operations. During the initial development, the project was called Donkey but later renamed to Packhorse as a reference to both Workhorse and Packfiles.

The Problem

Modern CI/CD creates a perfect storm for Git servers:

1. Developer pushes to a large monorepo
2. Push triggers hundreds of CI jobs
3. Each job clones the repository
4. All clone requests hit Gitaly simultaneously
5. Gitaly, being effectively single-node per repository, saturates CPU, Memory and IO.

Even with Gitaly Raft providing replication, you're still limited to a small cluster (3-7 nodes) with no elastic scaling. When a "broad commit" triggers hundreds of builds, every node experiences pressure with no way to expand capacity dynamically.

This is the ultra-scale problem that Packhorse looks to help address.

Demos

How Packhorse Works

Packhorse provides two complementary capabilities that work together to eliminate Gitaly bottlenecks:

1. Request Coalescing

When 100 CI jobs request the same commit simultaneously, a naive cache sends 100 requests upstream. Packhorse uses request coalescing to ensure that a single request is sent on cache miss.

How it works:

• Packhorse normalizes incoming fetch requests based on Git protocol "haves" and "wants"
• Generates a unique cache key for each semantic request
• When multiple requests arrive for the same key, Packhorse holds them all
• Issues a single upstream request to Gitaly
• Streams the response to all waiting clients simultaneously

The cache is stored on ephemeral SSD storage (not memory or Redis). Packhorse uses streaming with bounded memory to handle arbitrarily large repositories without exhaustion.

2. Packfile-URIs from Object Storage

Request coalescing and the cache take load off Gitaly. Packfile-URIs move the egress onto object storage, which is CDN-able and cheaper to serve than git-upload-pack traffic out of GitLab.

How it works:

• On a cache miss, Packhorse fetches from Gitaly and streams the pack to the client inline
• In the background it writes that pack to object storage (GCS, S3, Azure) under the cache key, with a metadata record beside it holding the sections that came before the pack, the pack's checksum and the object format
• On a later hit for the same key, a client that accepts packfile-URIs gets a signed URL and an empty inline pack, and downloads the pack from object storage
• A client that reads packs inline is served from the local cache, which Packhorse rebuilds from the offloaded pack once the local entry has expired

Key Features

• Smart HTTP v2 Protocol Support: Git protocol v2 compatibility
• Semantic Request Coalescing: Handles stampeding herd scenarios gracefully
• Disk-Based Caching: LRU eviction with configurable size limits
• Packfile-URI Support: Offload bulk data to object storage
• Streaming Architecture: Bounded memory usage for any repository size
• Observable: Prometheus metrics and structured JSON logging
• Stateless & Horizontally Scalable: Each instance is cattle, not pets
• Single Binary Deployment: No external dependencies required

Quick Start

Build and Run

# Clone the repository
git clone https://gitlab.com/gitlab-org/packhorse.git
cd packhorse

scripts/prepare-dev-env.sh

# Build
go build -o packhorse ./cmd/packhorse

# Create cache directory
mkdir -p /var/cache/packhorse

# Run
./packhorse \
  --upstream-url=https://gitlab.com \
  --listen-addr=:8080 \
  --cache-dir=/var/cache/packhorse \
  --max-cache-size=50GB

With Packfile-URI Support

# Start Packhorse with object storage support
./packhorse \
  --upstream-url=https://gitlab.com \
  --listen-addr=:8080 \
  --cache-dir=/var/cache/packhorse \
  --max-cache-size=50GB \
  --blob-bucket-url=gs://my-packfiles-bucket \
  --gcs-service-account=packhorse@project.iam.gserviceaccount.com

The bucket fills from the fetches Packhorse serves. The first fetch of a given cache key is served inline and offloaded in the background, and fetches after that are served as packfile-URIs.

Kubernetes Deployment

Packhorse includes a Helm chart for Kubernetes deployment:

# Install Packhorse using Helm
helm install packhorse ./chart \
  --set config.upstreamURL=https://gitlab.com \
  --set config.blobBucketURL=gs://my-packfiles-bucket \
  --set config.gcsServiceAccount=packhorse@project.iam.gserviceaccount.com

# Check deployment status
kubectl get pods -l app.kubernetes.io/name=packhorse

See chart/values.yaml for full configuration options.

Configuration

Command-Line Flags

Client Configuration

For packfile-URI support, Git clients must be configured to accept HTTPS URIs:

git clone -c fetch.uriProtocols=https http://packhorse-proxy:8080/gitlab-org/gitlab.git

For CI runners, add to .gitlab-ci.yml:

variables:
  GIT_CLONE_PATH: $CI_BUILDS_DIR/$CI_CONCURRENT_ID/$CI_PROJECT_PATH
  GIT_CONFIG_PARAMETERS: "'fetch.uriProtocols=https'"

Architecture

Packhorse is built with a layered architecture optimized for performance and reliability:

graph TB
    Clients[CI Clients<br/>concurrent shallow clones]

    subgraph Packhorse["Packhorse Proxy"]
        Handler[HTTP Handler<br/>Go 1.22+ ServeMux<br/>- info/refs: advertise packfile-uris<br/>- git-upload-pack: cached fetch]
        Coalescer[Request Coalescer<br/>sync.Map<br/>- Deduplicate requests<br/>- Broadcast results]
        Cache[Cache Manager<br/>Ristretto<br/>- In-memory index with LRU<br/>- SSD-backed storage]
        URICache[URI Cache<br/>- Offload cached packs<br/>- Per-pack metadata<br/>- Signed URLs on hits]

        Handler --> Coalescer
        Coalescer --> Cache
        Cache --> URICache
    end

    Gitaly[Upstream Gitaly]
    Storage[Object Storage<br/>S3/GCS/Azure]

    Clients -->|Git Smart HTTP v2| Handler
    Cache -->|HTTPS on miss| Gitaly
    URICache -->|HTTPS| Storage
    Clients -->|Download packs| Storage

Deployment Models

Near CI Runners (Recommended)

Deploy Packhorse close to your CI infrastructure to maximize cache locality and minimize network hops:

graph LR
    GitLab[GitLab Instance]
    Internet((Internet))
    Packhorse[Packhorse<br/>customer infrastructure]
    Runners[CI Runners]
    Storage[Object Storage<br/>bulk data]

    GitLab --> Internet
    Internet --> Packhorse
    Packhorse --> Runners
    Packhorse --> Storage

Benefits:

• Reduced egress costs
• Lower latency for CI jobs
• Customer-controlled scaling

Behind GitLab.com Endpoints

For GitLab.com customers, Packhorse can be deployed transparently behind GitLab endpoints:

graph LR
    Runners[CI Runners]
    LB[GitLab.com LB]
    Pool[Packhorse Pool]
    Gitaly[Gitaly]
    Storage[Object Storage]

    Runners --> LB
    LB --> Pool
    Pool --> Gitaly
    Pool --> Storage

Benefits:

• Transparent to users
• Elastic scaling
• Shared cache across customers

Monitoring and Observability

Prometheus Metrics

Packhorse exports metrics on :9090/metrics:

# Cache effectiveness (repository label present only when the allowlist bounds
# the cacheable set, otherwise a "_disabled" sentinel keeps cardinality bounded)
packhorse_cache_hits_total{repository}
packhorse_cache_misses_total{repository}
packhorse_non_cacheable_requests_total{reason}

# Request coalescing (a coalesced request waited on an in-flight upstream fetch;
# coalescing ratio = coalesced / (hits + misses + coalesced))
packhorse_coalesced_requests_total{repository}

# Fetch latency and errors
packhorse_fetch_duration_seconds{result}  # result: hit | miss | coalesce | bypass
packhorse_errors_total{type}              # type: upstream | cache | internal

# Cache storage (size/entries against configured maxima for saturation panels)
packhorse_cache_size_bytes      # current on-disk cache size
packhorse_cache_entries         # current number of cached packfiles
packhorse_cache_max_size_bytes  # configured max size
packhorse_cache_max_entries     # configured max entries

Client-facing and upstream request latency and throughput are covered by the LabKit http_* and upstream_* metric families, and the binary's own runtime signals by the standard go_* and process_* collectors. All register to the same /metrics endpoint.

Health Checks

Health endpoints on :8081:

• /health/live - Liveness probe (is Packhorse running?)
• /health/ready - Readiness probe (can Packhorse serve traffic?)

Structured Logging

All logs are JSON-formatted with correlation IDs for request tracing:

{
  "level": "info",
  "component": "coalescer",
  "cache_key": "e3b0c442...",
  "waiters": 47,
  "message": "fetch complete, notified waiters",
  "timestamp": "2025-12-08T16:42:00Z"
}

Use Cases

Ultra-Scale Monorepos

Problem: Very large monorepo customers with high commit frequency and thousands of concurrent CI jobs

Solution: Deploy Packhorse near CI infrastructure, so coalescing and the cache absorb the burst and Gitaly serves one fetch per unique request

Multi-Tenant CI Infrastructure

Problem: Shared CI infrastructure serving many projects, unpredictable load spikes

Solution: Deploy Packhorse pool with request coalescing, handle stampedes gracefully

Bandwidth-Constrained Environments

Problem: Limited network bandwidth between GitLab instance and CI runners

Solution: Deploy Packhorse with local cache, minimize repeated fetches

Object Storage Offloading

Problem: A large share of GitLab's egress is git-upload-pack traffic

Solution: Serve cached packs as packfile-URIs from object storage, moving that egress onto storage and potentially a CDN

Development

Project Structure

packhorse/
├── cmd/
│   ├── packhorse/            # Main application entry point
│   ├── sign-url/             # Generate signed object storage URLs
│   └── testgitserver/        # Test Git server for integration tests
├── internal/
│   ├── cache/                # Cache manager
│   ├── coalesce/             # Request coalescer
│   ├── packmeta/             # Per-pack metadata kept beside each offloaded pack
│   ├── pktlineeditor/        # Split and rebuild fetch responses around the pack
│   ├── parser/               # Git protocol parser
│   ├── proxy/                # HTTP handlers
│   ├── server/               # HTTP server
│   ├── observability/        # Metrics and logging
│   └── blobstorage/          # Object storage abstraction
└── docs/
    ├── architecture.md       # Deployment topology and request flow
    ├── design-doc-poc.md     # Original POC design
    └── demos/                # Demo documentation and assets

Running Tests

# Unit tests
go test ./...

# Integration tests
go test -tags=integration ./...

# With coverage
go test -cover ./...

# Specific package
go test ./internal/coalesce/

Local Development

# Start test Git server
go run ./cmd/testgitserver --listen-addr=:9418

# Run Packhorse against test server
go run ./cmd/packhorse \
  --upstream-url=http://localhost:9418 \
  --listen-addr=:8080 \
  --cache-dir=/tmp/packhorse-cache \
  --log-level=debug

# Test with real Git client
git clone http://localhost:8080/test-repo.git

Contributing

We welcome contributions from the community! Here's how to get started:

1. Fork the repository on GitLab
2. Create a feature branch: git checkout -b feature/your-feature
3. Make your changes with clear commit messages
4. Add tests for new functionality
5. Run tests: go test ./...
6. Submit a merge request with a clear description

Contribution Ideas

• Add support for SSH protocol (currently HTTP/HTTPS only)
• Implement Redis-based distributed coalescing for multi-instance deployments
• Add cache warming via webhooks for proactive optimization
• Implement advanced eviction policies (weighted LRU, access frequency)
• Add authentication integration (OAuth, JWT)
• Create monitoring dashboard templates (Grafana)
• Write additional documentation and examples
• Enhance Helm chart with advanced configurations and examples

Code Standards

• Follow Go standard formatting (go fmt)
• Write tests for new code
• Add structured logging with correlation IDs
• Export Prometheus metrics for observable behavior
• Update documentation for user-facing changes

Documentation

Design Documents

Demos

Roadmap

Phase 1: Production Hardening (Current)

• Request coalescing with stampede protection
• Disk-based caching with LRU eviction
• Packfile-URIs served from object storage
• Prometheus metrics and health checks
• Helm chart for Kubernetes
• Load testing and benchmarking
• Production deployment at GitLab.com

Phase 2: Distributed Deployment

• Redis-based distributed coalescing
• Multi-instance cache consistency
• Cache warming via webhook integration
• Reference advertisement caching
• Authentication and authorization

Phase 3: Advanced Features

• Full clone support (beyond shallow clones)
• Protocol v1 backward compatibility
• Intelligent prefetching based on CI patterns
• Delta optimization for cached packfiles
• Multi-region deployment support

FAQ

Q: Does Packhorse work with non-GitLab Git servers? A: Yes! Packhorse is Git-protocol-agnostic and works with any Git server supporting Smart HTTP v2.

Q: How much disk space do I need? A: It depends on your repository size and clone frequency. For typical usage, 100GB handles ~1000 unique shallow clone packfiles (50-100MB each). Monitor cache hit rates and adjust.

Q: Can I deploy multiple Packhorse instances? A: Yes! Each instance maintains its own cache. For distributed coalescing across instances, Redis support is planned for Phase 2.

Q: What happens if Packhorse crashes? A: CI jobs fall back to direct upstream access. The cache is ephemeral—on restart, it warms naturally from CI traffic.

Q: Does this work with Git LFS? A: Not currently. Packhorse focuses on Git object caching. LFS support is possible but not prioritized.

Q: What do I have to do to populate object storage? A: Nothing. Point --blob-bucket-url at a bucket and it fills from the fetches Packhorse serves. Each cache key's pack is written there in the background after the first fetch for that key has been served, and fetches after that are served as packfile-URIs.

Q: Why not improve Gitaly's existing packfile cache instead of adding another layer? A: Two reasons: (1) Spatial cache locality - caches work best near consumers (like L1/L2 CPU caches, or CDNs). CI jobs are the primary consumers, so caching near runners is more effective than behind Gitaly. (2) Scalability - Gitaly isn't horizontally scalable. When Gitaly is saturated, the cache path is also saturated. Moving the cache outside Gitaly allows elastic scaling independent of Gitaly's constraints.

Q: Can Packhorse cache other artifacts besides Git objects? A: Yes! The same large-object caching techniques could apply to container images (by digest) and CI artifacts (by job ID). This is a planned future enhancement.

Q: Where should Packhorse be deployed? A: Three options: (1) Near CI runners (recommended) - Deploy via Helm chart in customer infrastructure for best cache locality. (2) As a pseudo-runner - Runner Manager can deploy Packhorse as a caching service. (3) Between Workhorse and Gitaly - For transparent caching within GitLab infrastructure.

Q: Does Packhorse support SSH protocol? A: Not currently. Packhorse focuses on HTTP/HTTPS (Git Smart HTTP v2) since that's what CI uses. SSH support is possible but not prioritized.

Q: Does Gitaly already do request coalescing? A: Yes, Gitaly has packfile cache coalescing, but it still requires streaming through Gitaly even on cache hits. Packhorse's coalescing happens before reaching Gitaly, completely removing load from the bottleneck.

Q: What's the difference between packfile-uris and bundle-uris? A: Bundle-URIs happen before have/want negotiation, only work with git clone (not fetch), and aren't shallow-compatible. Packfile-URIs work with both clone and fetch and support shallow operations. The client opts in with fetch.uriProtocols=https, and Packhorse serves the URI from its own cache, so the upstream Git server needs no support for the capability.

License

MIT License - see LICENSE file for details.

Support

• Issues: Report bugs and feature requests at gitlab.com/gitlab-org/packhorse/issues
• Documentation: See docs/ directory
• GitLab Development: This is part of Project Ultra for supporting ultra-scale customers

Acknowledgments

Packhorse is developed as part of GitLab's Project Ultra initiative to support customers with massive monorepos and extreme CI/CD workloads. Special thanks to the Gitaly team for their insights and support.

Packhorse: Protecting Gitaly, one clone at a time. 🫏