Help us shape the future of Go! Take the 2026 Go Developer Survey

proxy

package
v1.233.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 20, 2026 License: Apache-2.0 Imports: 24 Imported by: 0

Documentation

Overview

Package proxy implements a generic, protocol-agnostic caching HTTP proxy.

The proxy is the reusable core of the Atmos registry cache. It owns nothing Terraform-specific: protocol knowledge lives in Mirror adapters (the provider and module registry mirrors, and a future git mirror) that map an inbound request to a Route — a cache key, an upstream request, an artifact kind, and optional rewrite/verify hooks.

Design tenets:

  • The cache key is authoritative and is the backend object name; the filesystem path IS the key. There is no secondary index.
  • Fetch-once on a miss: no retries, no backoff. Retry/backoff is the caller's (Atmos's) responsibility, keeping failures observable.
  • One downloader, many readers per key, via three tiers of concurrency control (see serveCacheable): 1. A lock-free hit fast path — safe because both the object and its sidecar are written atomically (temp file + rename), so an unlocked reader never sees a torn file. 2. In-process singleflight collapses concurrent cold-key fills in one process (the Atmos bulk case: many terraform child processes share one proxy); waiters block with no timeout and cancel cleanly when their client leaves. 3. A cross-process pkg/cache.FileLock (context-bounded) collapses fills across processes that share a cache directory, held only around fetch + commit — never while streaming to the client. On Windows the file lock degrades to a no-op, so concurrent processes may redundantly download a cold key; the atomic commit still guarantees no corruption.
  • Atomic commit: stream to a temp file, hash, verify, then rename into place.
  • Per-run, in-memory statistics only (hits + bytes saved) for the savings report — no persistent hit/miss store.
  • Credentials and headers are forwarded to upstream so private registries keep working, and Terraform's User-Agent is forwarded verbatim.

Package proxy is a generic, protocol-agnostic caching HTTP proxy. It binds an ephemeral loopback listener, routes each request through a pluggable Mirror to a cache key + upstream request, and either serves a cache hit or fetches the object upstream once (no retries — the caller owns retry), verifies it, stores it atomically, and serves it.

The package deliberately knows nothing about Terraform: the Terraform provider and module registry mirrors are adapters that implement Mirror. It is named for proxying, not caching, so a future git mirror can reuse the same infrastructure.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func BuildUpstreamRequest

func BuildUpstreamRequest(ctx context.Context, inbound *http.Request, up UpstreamRequest) (*http.Request, error)

BuildUpstreamRequest builds a propagated upstream *http.Request from an inbound request. It is exported so Mirror adapters can make their own propagated pre-flight calls (e.g. resolving a provider download URL) using the same credential/header/User-Agent rules the proxy applies on the main fetch path.

Types

type ArtifactKind

type ArtifactKind int

ArtifactKind classifies a routed object for stats and freshness policy.

const (
	// KindMetadata is registry metadata (version listings, download resolution).
	// It honors metadata_ttl with stale-while-revalidate.
	KindMetadata ArtifactKind = iota
	// KindArtifact is immutable content (provider zips, module archives). Once
	// cached it never expires.
	KindArtifact
	// KindPassthrough is streamed straight through and never cached (e.g. a git::
	// module download that the HTTP proxy cannot cache).
	KindPassthrough
)

func (ArtifactKind) String

func (k ArtifactKind) String() string

String renders the kind for sidecar metadata and reporting.

type CommitRequest

type CommitRequest struct {
	// Key is the canonical cache key (and object path under the root).
	Key string
	// Data is the object content to store.
	Data io.Reader
	// Kind classifies the object for freshness policy.
	Kind ArtifactKind
	// ContentType is recorded in the sidecar and used when serving.
	ContentType string
	// Verify, when non-nil, validates the computed SHA-256 before the object is
	// committed; a non-nil error rejects it.
	Verify func(sha256Hex string) error
}

CommitRequest describes an object to commit to the store.

type Doer

type Doer interface {
	Do(req *http.Request) (*http.Response, error)
}

Doer performs an HTTP request. Implemented by pkg/http.Client and *http.Client; injectable so tests can stub upstream.

type Fetcher

type Fetcher func(ctx context.Context, up UpstreamRequest) (*http.Response, error)

Fetcher performs a propagated upstream fetch: it builds the request with the same credential/header/User-Agent propagation the proxy applies, then executes it. Mirrors use it inside Produce to compose multiple upstream calls (e.g. translating the provider registry protocol into a network-mirror response).

type FileStore

type FileStore struct {
	// contains filtered or unexported fields
}

FileStore is the filesystem-backed Store.

func NewFileStore

func NewFileStore(root string) *FileStore

NewFileStore returns a Store rooted at root.

func (*FileStore) Commit

func (s *FileStore) Commit(ctx context.Context, req CommitRequest) (Meta, error)

func (*FileStore) Lock

func (s *FileStore) Lock(key string) cache.FileLock

Lock returns a file lock on the object path (FileLock appends a .lock sidecar). The parent directory is created first so flock can open the lock file even on a cold key whose directory does not yet exist.

func (*FileStore) Open

func (s *FileStore) Open(key string) (io.ReadCloser, Meta, error)

Open returns a reader for the cached object plus its metadata.

func (*FileStore) Root

func (s *FileStore) Root() string

Root returns the cache root.

func (*FileStore) Stat

func (s *FileStore) Stat(key string) (Meta, bool, error)

Stat returns the object's metadata if it exists.

type Meta

type Meta struct {
	Size        int64
	SHA256      string
	FetchedAt   time.Time
	Kind        ArtifactKind
	ContentType string
}

Meta is the proxy's view of a cached object's metadata.

type Mirror

type Mirror interface {
	// Handles reports whether this mirror owns the request path.
	Handles(r *http.Request) bool
	// Route computes the cache key, upstream request, kind, and rewrite/verify hooks.
	Route(r *http.Request) (Route, error)
}

Mirror maps an inbound proxy request to a Route. Adapters (provider/module registry mirrors, a future git mirror) implement it.

type Options

type Options struct {
	// Mirrors are consulted in order; the first whose Handles returns true owns the
	// request.
	Mirrors []Mirror
	// Store is the cache backend.
	Store Store
	// Client performs upstream fetches. Defaults to pkg/http.NewDefaultClient().
	Client Doer
	// MetadataTTL is how long KindMetadata stays fresh. Zero serves metadata as
	// fresh forever (no revalidation).
	MetadataTTL time.Duration
	// StaleWhileRevalidate is the window past MetadataTTL during which stale
	// metadata is served while a background revalidation runs.
	StaleWhileRevalidate time.Duration
	// ReadHeaderTimeout bounds slow-header attacks. Defaults to 30s.
	ReadHeaderTimeout time.Duration
	// TLSCertificate, when set, makes the proxy serve HTTPS with this certificate and
	// the base URL uses the https scheme. Terraform/OpenTofu require provider network
	// mirrors to be https, so the cache always sets this. Nil serves plain HTTP.
	TLSCertificate *tls.Certificate
}

Options configure a Server.

type Route

type Route struct {
	// Key is the canonical cache key, which is also the backend object name. Empty
	// for KindPassthrough.
	Key string
	// Kind classifies the object for caching and freshness.
	Kind ArtifactKind
	// Upstream is the single upstream fetch performed on a miss (or always, for
	// passthrough). Ignored when Produce is set.
	Upstream UpstreamRequest
	// Produce, when non-nil, composes the (KindMetadata) response from one or more
	// upstream calls via the provided Fetcher — used when a single Upstream+Rewrite
	// cannot express the translation (e.g. building a provider <version>.json from
	// per-platform registry download calls). Returns the body and its content type.
	Produce func(ctx context.Context, fetch Fetcher, proxyBaseURL string) (body []byte, contentType string, err error)
	// ProduceArtifact, when non-nil, streams a generated KindArtifact body produced by
	// an operation rather than a single upstream HTTP GET (e.g. taring a go-getter
	// resolved module source tree). The returned reader is committed to the cache and
	// closed by the proxy. Takes precedence over Upstream and Produce.
	ProduceArtifact func(ctx context.Context) (body io.ReadCloser, contentType string, err error)
	// Rewrite, when non-nil, post-processes a KindMetadata body before it is cached
	// and served — e.g. rewriting provider download URLs back through the proxy.
	// proxyBaseURL is the proxy's own base URL ("http://127.0.0.1:<port>/").
	Rewrite func(body []byte, proxyBaseURL string) ([]byte, error)
	// Verify, when non-nil, checks artifact integrity before commit. It receives the
	// hex-encoded SHA-256 of the downloaded bytes (sufficient to verify a provider
	// zip's zh: hash). Returning an error rejects the object.
	Verify func(sha256Hex string) error
	// HeaderRewrite, when non-nil on a KindPassthrough route, may modify the upstream
	// response headers before they are written back — e.g. rewriting a module's
	// HTTP-archive X-Terraform-Get to route the archive back through the proxy while
	// leaving git:: sources untouched.
	HeaderRewrite func(h http.Header, proxyBaseURL string)
	// Serve, when non-nil, fully renders the served response from a cached object
	// instead of the default (200 + body). The mirror receives the cached body and the
	// live proxy base URL, so a value cached in a prior run (when the proxy bound a
	// different ephemeral port) is rewritten to the current run's URL at serve time —
	// e.g. a module download resolution caches the upstream source and emits
	// 204 + X-Terraform-Get pointing at the current-run _source route. Invoked on both
	// the hit and post-fill serve paths.
	Serve func(w http.ResponseWriter, body io.Reader, proxyBaseURL string) error
	// ContentType for the served response. Falls back to the upstream Content-Type.
	ContentType string
}

Route is a Mirror's decision for a single inbound request.

type Server

type Server struct {
	// contains filtered or unexported fields
}

Server is the ephemeral caching proxy.

func NewServer

func NewServer(opts Options) *Server

NewServer constructs a Server. Call Start to bind and serve.

func (*Server) BaseURL

func (s *Server) BaseURL() string

BaseURL returns the proxy base URL (empty before Start).

func (*Server) ServeHTTP

func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request)

ServeHTTP routes the request to the first mirror that handles it, then runs the cache-or-fetch flow.

func (*Server) Shutdown

func (s *Server) Shutdown(ctx context.Context) error

Shutdown gracefully stops the proxy. Safe to call once.

It marks the server closed (so no new fill is started), cancels in-flight fills, drains active HTTP handlers, then waits for the detached fill goroutines to finish. Waiting for the fills guarantees no background goroutine touches the cache directory after Shutdown returns — important for callers that delete the cache dir afterwards.

func (*Server) Start

func (s *Server) Start(_ context.Context) (string, error)

Start binds an ephemeral loopback listener (127.0.0.1:0), serves in a background goroutine, and returns the proxy's base URL. The scheme is https when a TLSCertificate is configured (the cache always configures one), else http.

func (*Server) Stats

func (s *Server) Stats() StatsSnapshot

Stats returns a snapshot of per-run cache statistics.

type Stats

type Stats struct {
	// contains filtered or unexported fields
}

Stats tracks per-run, in-memory cache statistics for the savings report. There is deliberately no persistent hit/miss store: the filesystem is the index, and hit rate is a per-run signal surfaced only by the end-of-run savings report. The report has two halves: bytes served from cache (hits → bandwidth saved) and bytes fetched from upstream and committed (cacheable misses → cache warmed).

func (*Stats) RecordCached

func (s *Stats) RecordCached(size int64)

RecordCached records a cache miss whose upstream response was fetched and committed to the cache: it counts as a miss and adds size bytes to the warmed total.

func (*Stats) RecordHit

func (s *Stats) RecordHit(size int64)

RecordHit records a cache hit, adding size (the bytes actually streamed to the client, not the on-disk object size) to the bandwidth-saved total. A hit is still counted when a transfer is interrupted; size then reflects only the bytes delivered.

func (*Stats) RecordMiss

func (s *Stats) RecordMiss()

RecordMiss records a cache miss whose upstream response was not committed to the cache (e.g. a non-cacheable status replayed to the client).

func (*Stats) Snapshot

func (s *Stats) Snapshot() StatsSnapshot

Snapshot returns a copy of the current statistics.

type StatsSnapshot

type StatsSnapshot struct {
	Hits          int
	Misses        int
	BytesSaved    int64
	ObjectsCached int
	BytesCached   int64
}

StatsSnapshot is an immutable copy of Stats at a point in time.

type Store

type Store interface {
	// Lock returns the per-key file lock (one writer, many readers).
	Lock(key string) cache.FileLock
	// Stat returns metadata for key and whether it exists.
	Stat(key string) (Meta, bool, error)
	// Open returns a reader for the cached object plus its metadata.
	Open(key string) (io.ReadCloser, Meta, error)
	// Commit streams the request data to a temp file (hashing as it goes), runs
	// Verify against the computed SHA-256, then atomically renames it into place and
	// writes the sidecar. A non-nil Verify error rejects the object (nothing is committed).
	Commit(ctx context.Context, req CommitRequest) (Meta, error)
	// Root returns the cache root directory.
	Root() string
}

Store is the proxy's cache backend: a content store keyed by canonical cache keys, with per-key locking. The filesystem implementation (FileStore) writes the object at <root>/<key> and an artifact-compatible <key>.metadata.json sidecar so the object's path IS the key (the source of truth — no separate index).

type UpstreamRequest

type UpstreamRequest struct {
	// URL is the absolute upstream URL to fetch.
	URL string
	// Method defaults to GET when empty.
	Method string
	// Header carries mirror-supplied headers (e.g. Accept). The proxy merges
	// credential and User-Agent headers on top, never dropping the Atmos UA.
	Header http.Header
}

UpstreamRequest describes the upstream fetch for a route. The proxy builds and executes the actual *http.Request, applying credential, header, and User-Agent propagation centrally (see propagation.go) so private registries and the Atmos User-Agent are handled uniformly across all mirrors.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL