packhorse

Packhorse

The Git caching proxy that protects Gitaly from stampeding herds

Packhorse is a high-performance Git caching service designed to solve Gitaly scaling challenges for ultra-scale customers. When hundreds of CI jobs clone the same repository simultaneously, Packhorse coalesces requests, caches packfiles, and leverages Git's packfile-URI capability to reduce upstream load by up to 99.4%.

Why Packhorse?

Packhorse is to Gitaly what Workhorse is to Rails—a protective layer that handles the heavy lifting. Just as Workhorse shields Rails from expensive file operations, Packhorse shields Gitaly from the crushing load of concurrent CI clones.

The name follows GitLab tradition: Rails ran on Unicorn, and Workhorse was built to protect it by being the opposite—tough and hard-working. Packhorse follows the same philosophy for Git operations. During the initial development, the project was called Donkey but later renamed to Packhorse as a reference to both Workhorse and Packfiles.

The Problem

Modern CI/CD creates a perfect storm for Git servers:

1. Developer pushes to a large monorepo
2. Push triggers hundreds of CI jobs
3. Each job clones the repository
4. All clone requests hit Gitaly simultaneously
5. Gitaly, being effectively single-node per repository, saturates CPU, Memory and IO.

Even with Gitaly Raft providing replication, you're still limited to a small cluster (3-7 nodes) with no elastic scaling. When a "broad commit" triggers hundreds of builds, every node experiences pressure with no way to expand capacity dynamically.

This is the ultra-scale problem that Packhorse looks to help address.

Demos

Date	Description	Link
2025-12-04	Git caching for ultra-scale customers - demonstrates request coalescing and pack file URI injection reducing Gitaly load from 169MB to 987 bytes per clone	View Demo

How Packhorse Works

Packhorse provides two complementary capabilities that work together to eliminate Gitaly bottlenecks:

1. Request Coalescing

When 100 CI jobs request the same commit simultaneously, a naive cache sends 100 requests upstream. Packhorse uses request coalescing to ensure that a single request is sent on cache miss.

How it works:

• Packhorse normalizes incoming fetch requests based on Git protocol "haves" and "wants"
• Generates a unique cache key for each semantic request
• When multiple requests arrive for the same key, Packhorse holds them all
• Issues a single upstream request to Gitaly
• Streams the response to all waiting clients simultaneously

The cache is stored on ephemeral SSD storage (not memory or Redis). Packhorse uses streaming with bounded memory to handle arbitrarily large repositories without exhaustion.

2. Packfile-URI Injection

Request coalescing reduces the number of requests. Packfile-URI injection reduces the size of each response.

How it works:

• Pre-generate base packfiles for high-traffic repositories (eg, last N days of commits on default branch)
• Upload base packfiles to object storage (GCS, S3, Azure)
• Packhorse intercepts incoming clone requests
• Injects the base packfile's commits as additional "haves" in the client's request
• Returns a packfile-URI pointing to object storage
• Client fetches base packfile directly from object storage (infinitely scalable)
• Gitaly only sends a small delta (typically <1KB)

Result: Clone that would transfer 169MB from Gitaly instead transfers 987 bytes

Performance Results

Testing against GitLab's own repository (gitlab-org/gitlab):

Scenario	Data from Gitaly	Data from Packhorse Cache	Data from Object Storage
First clone (no base pack)	169 MB	—	—
Subsequent clone (cached)	0	169 MB	—
First clone (with base pack)	987 bytes	—	~217 MB
Subsequent clone (cached + base pack)	0	997 bytes	~217 MB

Key Features

• Smart HTTP v2 Protocol Support: Git protocol v2 compatibility
• Semantic Request Coalescing: Handles stampeding herd scenarios gracefully
• Disk-Based Caching: LRU eviction with configurable size limits
• Packfile-URI Support: Offload bulk data to object storage
• Streaming Architecture: Bounded memory usage for any repository size
• Observable: Prometheus metrics and structured JSON logging
• Stateless & Horizontally Scalable: Each instance is cattle, not pets
• Single Binary Deployment: No external dependencies required

Quick Start

Build and Run

# Clone the repository
git clone https://gitlab.com/gitlab-org/packhorse.git
cd packhorse

scripts/prepare-dev-env.sh

# Build
go build -o packhorse ./cmd/packhorse

# Create cache directory
mkdir -p /var/cache/packhorse

# Run
./packhorse \
  --upstream-url=https://gitlab.com \
  --listen-addr=:8080 \
  --cache-dir=/var/cache/packhorse \
  --max-cache-size=50GB

With Packfile-URI Support

# Start Packhorse with object storage support
./packhorse \
  --upstream-url=https://gitlab.com \
  --listen-addr=:8080 \
  --cache-dir=/var/cache/packhorse \
  --max-cache-size=50GB \
  --blob-bucket-url=gs://my-packfiles-bucket \
  --gcs-service-account=packhorse@project.iam.gserviceaccount.com

# In another terminal, generate and register a base packfile
./scripts/generate-base-packfile.sh /tmp/repos gs://my-packfiles-bucket http://localhost:8080

Kubernetes Deployment

Packhorse includes a Helm chart for Kubernetes deployment:

# Install Packhorse using Helm
helm install packhorse ./chart \
  --set config.upstreamURL=https://gitlab.com \
  --set config.blobBucketURL=gs://my-packfiles-bucket \
  --set config.gcsServiceAccount=packhorse@project.iam.gserviceaccount.com

# Check deployment status
kubectl get pods -l app.kubernetes.io/name=packhorse

See chart/values.yaml for full configuration options.

Configuration

Command-Line Flags

Flag	Default	Description
--upstream-url	(required)	Upstream Git server URL
--listen-addr	:8080	Address to listen on
--cache-dir	/var/cache/packhorse	Cache storage directory
--max-cache-size	10GB	Maximum cache size (supports KB, MB, GB, TB)
--max-entries	10000	Maximum number of cache entries
--metrics-addr	:9090	Prometheus metrics endpoint
--health-addr	:8081	Health check endpoint
--log-level	info	Log level (debug, info, warn, error)
--log-format	json	Log format (json or text)
--blob-bucket-url		Object storage URL (s3://, gs://, azblob://, file://)
--gcs-service-account		GCS service account for signed URLs

Client Configuration

For packfile-URI support, Git clients must be configured to accept HTTPS URIs:

git clone -c fetch.uriProtocols=https http://packhorse-proxy:8080/gitlab-org/gitlab.git

For CI runners, add to .gitlab-ci.yml:

variables:
  GIT_CLONE_PATH: $CI_BUILDS_DIR/$CI_CONCURRENT_ID/$CI_PROJECT_PATH
  GIT_CONFIG_PARAMETERS: "'fetch.uriProtocols=https'"

Architecture

Packhorse is built with a layered architecture optimized for performance and reliability:

graph TB
    Clients[CI Clients<br/>concurrent shallow clones]

    subgraph Packhorse["Packhorse Proxy"]
        Handler[HTTP Handler<br/>Go 1.22+ ServeMux<br/>- info/refs: inject packfile-uris<br/>- git-upload-pack: cached fetch]
        Coalescer[Request Coalescer<br/>sync.Map<br/>- Deduplicate requests<br/>- Broadcast results]
        Cache[Cache Manager<br/>Ristretto<br/>- In-memory index with LRU<br/>- SSD-backed storage]
        PackfileMgr[Packfile Manager<br/>- Base packfile registration API<br/>- Request augmentation<br/>- Response injection]

        Handler --> Coalescer
        Coalescer --> Cache
        Cache --> PackfileMgr
    end

    Gitaly[Upstream Gitaly]
    Storage[Object Storage<br/>S3/GCS/Azure]

    Clients -->|Git Smart HTTP v2| Handler
    PackfileMgr -->|HTTPS| Gitaly
    PackfileMgr -->|HTTPS| Storage
    Clients -.->|Download packfiles| Storage

Deployment Models

Near CI Runners (Recommended)

Deploy Packhorse close to your CI infrastructure to maximize cache locality and minimize network hops:

graph LR
    GitLab[GitLab Instance]
    Internet((Internet))
    Packhorse[Packhorse<br/>customer infrastructure]
    Runners[CI Runners]
    Storage[Object Storage<br/>bulk data]

    GitLab --> Internet
    Internet --> Packhorse
    Packhorse --> Runners
    Packhorse --> Storage

Benefits:

• Reduced egress costs
• Lower latency for CI jobs
• Customer-controlled scaling

Behind GitLab.com Endpoints

For GitLab.com customers, Packhorse can be deployed transparently behind GitLab endpoints:

graph LR
    Runners[CI Runners]
    LB[GitLab.com LB]
    Pool[Packhorse Pool]
    Gitaly[Gitaly]
    Storage[Object Storage]

    Runners --> LB
    LB --> Pool
    Pool --> Gitaly
    Pool --> Storage

Benefits:

• Transparent to users
• Elastic scaling
• Shared cache across customers

Monitoring and Observability

Prometheus Metrics

Packhorse exports metrics on :9090/metrics:

# Cache effectiveness (repository label present only when the allowlist bounds
# the cacheable set, otherwise a "_disabled" sentinel keeps cardinality bounded)
packhorse_cache_hits_total{repository}
packhorse_cache_misses_total{repository}
packhorse_non_cacheable_requests_total{reason}

# Request coalescing (a coalesced request waited on an in-flight upstream fetch;
# coalescing ratio = coalesced / (hits + misses + coalesced))
packhorse_coalesced_requests_total{repository}

# Fetch latency and errors
packhorse_fetch_duration_seconds{result}  # result: hit | miss | coalesce | bypass
packhorse_errors_total{type}              # type: upstream | cache | internal

# Cache storage (size/entries against configured maxima for saturation panels)
packhorse_cache_size_bytes      # current on-disk cache size
packhorse_cache_entries         # current number of cached packfiles
packhorse_cache_max_size_bytes  # configured max size
packhorse_cache_max_entries     # configured max entries

Client-facing and upstream request latency and throughput are covered by the LabKit http_* and upstream_* metric families, and the binary's own runtime signals by the standard go_* and process_* collectors. All register to the same /metrics endpoint.

Health Checks

Health endpoints on :8081:

• /health/live - Liveness probe (is Packhorse running?)
• /health/ready - Readiness probe (can Packhorse serve traffic?)

Structured Logging

All logs are JSON-formatted with correlation IDs for request tracing:

{
  "level": "info",
  "component": "coalescer",
  "cache_key": "e3b0c442...",
  "waiters": 47,
  "message": "fetch complete, notified waiters",
  "timestamp": "2025-12-08T16:42:00Z"
}

Use Cases

Ultra-Scale Monorepos

Problem: Very large monorepo customers with high commit frequency and thousands of concurrent CI jobs

Solution: Deploy Packhorse near CI infrastructure, enable packfile-URI injection, reduce Gitaly load by 99%+

Multi-Tenant CI Infrastructure

Problem: Shared CI infrastructure serving many projects, unpredictable load spikes

Solution: Deploy Packhorse pool with request coalescing, handle stampedes gracefully

Bandwidth-Constrained Environments

Problem: Limited network bandwidth between GitLab instance and CI runners

Solution: Deploy Packhorse with local cache, minimize repeated fetches

Object Storage Offloading

Problem: Gitaly serves same packfiles repeatedly, wasting compute and bandwidth

Solution: Pre-generate base packfiles, upload to object storage, let Packhorse inject URIs

Development

Project Structure

packhorse/
├── cmd/
│   ├── packhorse/               # Main application entry point
│   ├── list-packfile-commits/  # Utility to inspect packfiles
│   ├── sign-url/             # Generate signed object storage URLs
│   └── testgitserver/        # Test Git server for integration tests
├── internal/
│   ├── api/                  # Base packfile registration API
│   ├── cache/                # Cache manager
│   ├── coalesce/             # Request coalescer
│   ├── packfile/             # Packfile parsing and management
│   ├── parser/               # Git protocol parser
│   ├── proxy/                # HTTP handlers
│   ├── server/               # HTTP server
│   ├── observability/        # Metrics and logging
│   └── blobstorage/          # Object storage abstraction
└── docs/
    ├── design-doc-poc.md     # Original POC design
    ├── packfile-uri-injection.md  # Packfile-URI design
    └── demos/                # Demo documentation and assets

Running Tests

# Unit tests
go test ./...

# Integration tests
go test -tags=integration ./...

# With coverage
go test -cover ./...

# Specific package
go test ./internal/coalesce/

Local Development

# Start test Git server
go run ./cmd/testgitserver --listen-addr=:9418

# Run Packhorse against test server
go run ./cmd/packhorse \
  --upstream-url=http://localhost:9418 \
  --listen-addr=:8080 \
  --cache-dir=/tmp/packhorse-cache \
  --log-level=debug

# Test with real Git client
git clone http://localhost:8080/test-repo.git

Contributing

We welcome contributions from the community! Here's how to get started:

1. Fork the repository on GitLab
2. Create a feature branch: git checkout -b feature/your-feature
3. Make your changes with clear commit messages
4. Add tests for new functionality
5. Run tests: go test ./...
6. Submit a merge request with a clear description

Contribution Ideas

• Add support for SSH protocol (currently HTTP/HTTPS only)
• Implement Redis-based distributed coalescing for multi-instance deployments
• Add cache warming via webhooks for proactive optimization
• Implement advanced eviction policies (weighted LRU, access frequency)
• Add authentication integration (OAuth, JWT)
• Create monitoring dashboard templates (Grafana)
• Write additional documentation and examples
• Enhance Helm chart with advanced configurations and examples

Code Standards

• Follow Go standard formatting (go fmt)
• Write tests for new code
• Add structured logging with correlation IDs
• Export Prometheus metrics for observable behavior
• Update documentation for user-facing changes

Documentation

Design Documents

Document	Description	Link
POC Design Document	Original proof-of-concept design covering architecture, request coalescing, cache management, and implementation phases	View Design
Packfile-URI Injection Design	Detailed design for implementing Git Protocol v2's packfile-uris capability, including request augmentation, commit extraction, and object storage integration	View Design
Credential-Aware Caching (credmac)	HMAC-based credential hashing with rotation-safe key management for caching authenticated Git requests without storing credentials	View Design
GitLab Dedicated Deployment	Proposal for deploying Packhorse to GitLab Dedicated tenants: authorizing the cache-hit path without an internal CI gateway, and the chart changes required in this repository	View Design

Demos

Date	Description	Link
2025-12-04	Git caching for ultra-scale customers - demonstrates request coalescing and pack file URI injection reducing Gitaly load from 169MB to 987 bytes per clone	View Demo

Roadmap

Phase 1: Production Hardening (Current)

• Request coalescing with stampede protection
• Disk-based caching with LRU eviction
• Packfile-URI injection
• Prometheus metrics and health checks
• Helm chart for Kubernetes
• Load testing and benchmarking
• Production deployment at GitLab.com

Phase 2: Distributed Deployment

• Redis-based distributed coalescing
• Multi-instance cache consistency
• Cache warming via webhook integration
• Reference advertisement caching
• Authentication and authorization

Phase 3: Advanced Features

• Full clone support (beyond shallow clones)
• Protocol v1 backward compatibility
• Intelligent prefetching based on CI patterns
• Delta optimization for cached packfiles
• Multi-region deployment support

FAQ

Q: Does Packhorse work with non-GitLab Git servers? A: Yes! Packhorse is Git-protocol-agnostic and works with any Git server supporting Smart HTTP v2.

Q: How much disk space do I need? A: It depends on your repository size and clone frequency. For typical usage, 100GB handles ~1000 unique shallow clone packfiles (50-100MB each). Monitor cache hit rates and adjust.

Q: Can I deploy multiple Packhorse instances? A: Yes! Each instance maintains its own cache. For distributed coalescing across instances, Redis support is planned for Phase 2.

Q: What happens if Packhorse crashes? A: CI jobs fall back to direct upstream access. The cache is ephemeral—on restart, it warms naturally from CI traffic.

Q: Does this work with Git LFS? A: Not currently. Packhorse focuses on Git object caching. LFS support is possible but not prioritized.

Q: How do I generate base packfiles? A: Packhorse includes a helper script scripts/generate-base-packfile.sh that automates the process:

# Basic usage (uses defaults)
./scripts/generate-base-packfile.sh

# Specify custom paths
./scripts/generate-base-packfile.sh /path/to/repos gs://my-bucket https://packhorse.example.com

The script:

1. Clones or updates the target repository (default: gitlab-org/gitlab)
2. Creates a packfile from recent commits (last 10,000 commits on master)
3. Verifies the packfile is self-contained and valid
4. Uploads the packfile and index to Google Cloud Storage
5. Outputs the curl command to register it with Packhorse

Prerequisites: git, gsutil (Google Cloud SDK), jq

Manual generation:

# Generate packfile for last 10,000 commits
git rev-list master --max-count=10000 --objects | \
  git pack-objects --shallow --write-bitmap-index \
    --window=250 --depth=50 /tmp/pack

# Upload to object storage
gsutil cp /tmp/pack-*.pack gs://my-bucket/
gsutil cp /tmp/pack-*.idx gs://my-bucket/

# Register with Packhorse
curl -X POST http://packhorse:8080/-/api/git-cache/base-packfiles \
  -H "Content-Type: application/json" \
  -d '{
    "repository": "gitlab-org/gitlab",
    "packfile_uri": "gs://my-bucket/pack-abc123.pack",
    "hash_algorithm": "sha1",
    "packfile_hash": "abc123..."
  }'

In the future, Gitaly will generate these natively.

Q: Why not improve Gitaly's existing packfile cache instead of adding another layer? A: Two reasons: (1) Spatial cache locality - caches work best near consumers (like L1/L2 CPU caches, or CDNs). CI jobs are the primary consumers, so caching near runners is more effective than behind Gitaly. (2) Scalability - Gitaly isn't horizontally scalable. When Gitaly is saturated, the cache path is also saturated. Moving the cache outside Gitaly allows elastic scaling independent of Gitaly's constraints.

Q: Can Packhorse cache other artifacts besides Git objects? A: Yes! The same large-object caching techniques could apply to container images (by digest) and CI artifacts (by job ID). This is a planned future enhancement.

Q: Where should Packhorse be deployed? A: Three options: (1) Near CI runners (recommended) - Deploy via Helm chart in customer infrastructure for best cache locality. (2) As a pseudo-runner - Runner Manager can deploy Packhorse as a caching service. (3) Between Workhorse and Gitaly - For transparent caching within GitLab infrastructure.

Q: Does Packhorse support SSH protocol? A: Not currently. Packhorse focuses on HTTP/HTTPS (Git Smart HTTP v2) since that's what CI uses. SSH support is possible but not prioritized.

Q: Does Gitaly already do request coalescing? A: Yes, Gitaly has packfile cache coalescing, but it still requires streaming through Gitaly even on cache hits. Packhorse's coalescing happens before reaching Gitaly, completely removing load from the bottleneck.

Q: What's the difference between packfile-uris and bundle-uris? A: Bundle-URIs happen before have/want negotiation, only work with git clone (not fetch), and aren't shallow-compatible. Packfile-URIs work with both clone and fetch, support shallow operations, and can be injected transparently by Packhorse without client configuration changes.

Q: How does Packhorse parse packfiles to extract commit IDs? A: Packhorse reads the packfile index to get object offsets, then fetches ranges from the packfile in object storage to extract commit objects. This is parallelized but may download more bytes than the total file size across iterations. For very large repositories (100GB+), we may optimize by downloading the entire packfile as a memory-mapped file.

Q: Do I need to inject every commit from the base packfile into the client's "haves"? A: Currently yes, but this can be optimized. Since shallow fetches don't include parent history, the server can't assume the client has parent commits. Future improvements may use smarter heuristics or multiple packfiles with selection logic.

License

MIT License - see LICENSE file for details.

Support

• Issues: Report bugs and feature requests at gitlab.com/gitlab-org/packhorse/issues
• Documentation: See docs/ directory
• GitLab Development: This is part of Project Ultra for supporting ultra-scale customers

Acknowledgments

Packhorse is developed as part of GitLab's Project Ultra initiative to support customers with massive monorepos and extreme CI/CD workloads. Special thanks to the Gitaly team for their insights and support.

---

Packhorse: Protecting Gitaly, one clone at a time. 🫏