Documentation
¶
Overview ¶
Package proxy implements a generic, protocol-agnostic caching HTTP proxy.
The proxy is the reusable core of the Atmos registry cache. It owns nothing Terraform-specific: protocol knowledge lives in Mirror adapters (the provider and module registry mirrors, and a future git mirror) that map an inbound request to a Route — a cache key, an upstream request, an artifact kind, and optional rewrite/verify hooks.
Design tenets:
- The cache key is authoritative and is the backend object name; the filesystem path IS the key. There is no secondary index.
- Fetch-once on a miss: no retries, no backoff. Retry/backoff is the caller's (Atmos's) responsibility, keeping failures observable.
- One downloader, many readers per key, via three tiers of concurrency control (see serveCacheable): 1. A lock-free hit fast path — safe because both the object and its sidecar are written atomically (temp file + rename), so an unlocked reader never sees a torn file. 2. In-process singleflight collapses concurrent cold-key fills in one process (the Atmos bulk case: many terraform child processes share one proxy); waiters block with no timeout and cancel cleanly when their client leaves. 3. A cross-process pkg/cache.FileLock (context-bounded) collapses fills across processes that share a cache directory, held only around fetch + commit — never while streaming to the client. On Windows the file lock degrades to a no-op, so concurrent processes may redundantly download a cold key; the atomic commit still guarantees no corruption.
- Atomic commit: stream to a temp file, hash, verify, then rename into place.
- Per-run, in-memory statistics only (hits + bytes saved) for the savings report — no persistent hit/miss store.
- Credentials and headers are forwarded to upstream so private registries keep working, and Terraform's User-Agent is forwarded verbatim.
Package proxy is a generic, protocol-agnostic caching HTTP proxy. It binds an ephemeral loopback listener, routes each request through a pluggable Mirror to a cache key + upstream request, and either serves a cache hit or fetches the object upstream once (no retries — the caller owns retry), verifies it, stores it atomically, and serves it.
The package deliberately knows nothing about Terraform: the Terraform provider and module registry mirrors are adapters that implement Mirror. It is named for proxying, not caching, so a future git mirror can reuse the same infrastructure.
Index ¶
- func BuildUpstreamRequest(ctx context.Context, inbound *http.Request, up UpstreamRequest) (*http.Request, error)
- type ArtifactKind
- type CommitRequest
- type Doer
- type Fetcher
- type FileStore
- type Meta
- type Mirror
- type Options
- type Route
- type Server
- type Stats
- type StatsSnapshot
- type Store
- type UpstreamRequest
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func BuildUpstreamRequest ¶
func BuildUpstreamRequest(ctx context.Context, inbound *http.Request, up UpstreamRequest) (*http.Request, error)
BuildUpstreamRequest builds a propagated upstream *http.Request from an inbound request. It is exported so Mirror adapters can make their own propagated pre-flight calls (e.g. resolving a provider download URL) using the same credential/header/User-Agent rules the proxy applies on the main fetch path.
Types ¶
type ArtifactKind ¶
type ArtifactKind int
ArtifactKind classifies a routed object for stats and freshness policy.
const ( // KindMetadata is registry metadata (version listings, download resolution). // It honors metadata_ttl with stale-while-revalidate. KindMetadata ArtifactKind = iota // KindArtifact is immutable content (provider zips, module archives). Once // cached it never expires. KindArtifact // KindPassthrough is streamed straight through and never cached (e.g. a git:: // module download that the HTTP proxy cannot cache). KindPassthrough )
func (ArtifactKind) String ¶
func (k ArtifactKind) String() string
String renders the kind for sidecar metadata and reporting.
type CommitRequest ¶
type CommitRequest struct {
// Key is the canonical cache key (and object path under the root).
Key string
// Data is the object content to store.
Data io.Reader
// Kind classifies the object for freshness policy.
Kind ArtifactKind
// ContentType is recorded in the sidecar and used when serving.
ContentType string
// Verify, when non-nil, validates the computed SHA-256 before the object is
// committed; a non-nil error rejects it.
Verify func(sha256Hex string) error
}
CommitRequest describes an object to commit to the store.
type Doer ¶
Doer performs an HTTP request. Implemented by pkg/http.Client and *http.Client; injectable so tests can stub upstream.
type Fetcher ¶
Fetcher performs a propagated upstream fetch: it builds the request with the same credential/header/User-Agent propagation the proxy applies, then executes it. Mirrors use it inside Produce to compose multiple upstream calls (e.g. translating the provider registry protocol into a network-mirror response).
type FileStore ¶
type FileStore struct {
// contains filtered or unexported fields
}
FileStore is the filesystem-backed Store.
func NewFileStore ¶
NewFileStore returns a Store rooted at root.
func (*FileStore) Lock ¶
Lock returns a file lock on the object path (FileLock appends a .lock sidecar). The parent directory is created first so flock can open the lock file even on a cold key whose directory does not yet exist.
type Meta ¶
type Meta struct {
Size int64
SHA256 string
FetchedAt time.Time
Kind ArtifactKind
ContentType string
}
Meta is the proxy's view of a cached object's metadata.
type Mirror ¶
type Mirror interface {
// Handles reports whether this mirror owns the request path.
Handles(r *http.Request) bool
// Route computes the cache key, upstream request, kind, and rewrite/verify hooks.
Route(r *http.Request) (Route, error)
}
Mirror maps an inbound proxy request to a Route. Adapters (provider/module registry mirrors, a future git mirror) implement it.
type Options ¶
type Options struct {
// Mirrors are consulted in order; the first whose Handles returns true owns the
// request.
Mirrors []Mirror
// Store is the cache backend.
Store Store
// Client performs upstream fetches. Defaults to pkg/http.NewDefaultClient().
Client Doer
// MetadataTTL is how long KindMetadata stays fresh. Zero serves metadata as
// fresh forever (no revalidation).
MetadataTTL time.Duration
// StaleWhileRevalidate is the window past MetadataTTL during which stale
// metadata is served while a background revalidation runs.
StaleWhileRevalidate time.Duration
// ReadHeaderTimeout bounds slow-header attacks. Defaults to 30s.
ReadHeaderTimeout time.Duration
// TLSCertificate, when set, makes the proxy serve HTTPS with this certificate and
// the base URL uses the https scheme. Terraform/OpenTofu require provider network
// mirrors to be https, so the cache always sets this. Nil serves plain HTTP.
TLSCertificate *tls.Certificate
}
Options configure a Server.
type Route ¶
type Route struct {
// Key is the canonical cache key, which is also the backend object name. Empty
// for KindPassthrough.
Key string
// Kind classifies the object for caching and freshness.
Kind ArtifactKind
// Upstream is the single upstream fetch performed on a miss (or always, for
// passthrough). Ignored when Produce is set.
Upstream UpstreamRequest
// Produce, when non-nil, composes the (KindMetadata) response from one or more
// upstream calls via the provided Fetcher — used when a single Upstream+Rewrite
// cannot express the translation (e.g. building a provider <version>.json from
// per-platform registry download calls). Returns the body and its content type.
Produce func(ctx context.Context, fetch Fetcher, proxyBaseURL string) (body []byte, contentType string, err error)
// ProduceArtifact, when non-nil, streams a generated KindArtifact body produced by
// an operation rather than a single upstream HTTP GET (e.g. taring a go-getter
// resolved module source tree). The returned reader is committed to the cache and
// closed by the proxy. Takes precedence over Upstream and Produce.
ProduceArtifact func(ctx context.Context) (body io.ReadCloser, contentType string, err error)
// Rewrite, when non-nil, post-processes a KindMetadata body before it is cached
// and served — e.g. rewriting provider download URLs back through the proxy.
// proxyBaseURL is the proxy's own base URL ("http://127.0.0.1:<port>/").
Rewrite func(body []byte, proxyBaseURL string) ([]byte, error)
// Verify, when non-nil, checks artifact integrity before commit. It receives the
// hex-encoded SHA-256 of the downloaded bytes (sufficient to verify a provider
// zip's zh: hash). Returning an error rejects the object.
Verify func(sha256Hex string) error
// HeaderRewrite, when non-nil on a KindPassthrough route, may modify the upstream
// response headers before they are written back — e.g. rewriting a module's
// HTTP-archive X-Terraform-Get to route the archive back through the proxy while
// leaving git:: sources untouched.
HeaderRewrite func(h http.Header, proxyBaseURL string)
// Serve, when non-nil, fully renders the served response from a cached object
// instead of the default (200 + body). The mirror receives the cached body and the
// live proxy base URL, so a value cached in a prior run (when the proxy bound a
// different ephemeral port) is rewritten to the current run's URL at serve time —
// e.g. a module download resolution caches the upstream source and emits
// 204 + X-Terraform-Get pointing at the current-run _source route. Invoked on both
// the hit and post-fill serve paths.
Serve func(w http.ResponseWriter, body io.Reader, proxyBaseURL string) error
// ContentType for the served response. Falls back to the upstream Content-Type.
ContentType string
}
Route is a Mirror's decision for a single inbound request.
type Server ¶
type Server struct {
// contains filtered or unexported fields
}
Server is the ephemeral caching proxy.
func (*Server) ServeHTTP ¶
func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request)
ServeHTTP routes the request to the first mirror that handles it, then runs the cache-or-fetch flow.
func (*Server) Shutdown ¶
Shutdown gracefully stops the proxy. Safe to call once.
It marks the server closed (so no new fill is started), cancels in-flight fills, drains active HTTP handlers, then waits for the detached fill goroutines to finish. Waiting for the fills guarantees no background goroutine touches the cache directory after Shutdown returns — important for callers that delete the cache dir afterwards.
func (*Server) Start ¶
Start binds an ephemeral loopback listener (127.0.0.1:0), serves in a background goroutine, and returns the proxy's base URL. The scheme is https when a TLSCertificate is configured (the cache always configures one), else http.
func (*Server) Stats ¶
func (s *Server) Stats() StatsSnapshot
Stats returns a snapshot of per-run cache statistics.
type Stats ¶
type Stats struct {
// contains filtered or unexported fields
}
Stats tracks per-run, in-memory cache statistics for the savings report. There is deliberately no persistent hit/miss store: the filesystem is the index, and hit rate is a per-run signal surfaced only by the end-of-run savings report. The report has two halves: bytes served from cache (hits → bandwidth saved) and bytes fetched from upstream and committed (cacheable misses → cache warmed).
func (*Stats) RecordCached ¶
RecordCached records a cache miss whose upstream response was fetched and committed to the cache: it counts as a miss and adds size bytes to the warmed total.
func (*Stats) RecordHit ¶
RecordHit records a cache hit, adding size (the bytes actually streamed to the client, not the on-disk object size) to the bandwidth-saved total. A hit is still counted when a transfer is interrupted; size then reflects only the bytes delivered.
func (*Stats) RecordMiss ¶
func (s *Stats) RecordMiss()
RecordMiss records a cache miss whose upstream response was not committed to the cache (e.g. a non-cacheable status replayed to the client).
func (*Stats) Snapshot ¶
func (s *Stats) Snapshot() StatsSnapshot
Snapshot returns a copy of the current statistics.
type StatsSnapshot ¶
type StatsSnapshot struct {
Hits int
Misses int
BytesSaved int64
ObjectsCached int
BytesCached int64
}
StatsSnapshot is an immutable copy of Stats at a point in time.
type Store ¶
type Store interface {
// Lock returns the per-key file lock (one writer, many readers).
Lock(key string) cache.FileLock
// Stat returns metadata for key and whether it exists.
Stat(key string) (Meta, bool, error)
// Open returns a reader for the cached object plus its metadata.
Open(key string) (io.ReadCloser, Meta, error)
// Commit streams the request data to a temp file (hashing as it goes), runs
// Verify against the computed SHA-256, then atomically renames it into place and
// writes the sidecar. A non-nil Verify error rejects the object (nothing is committed).
Commit(ctx context.Context, req CommitRequest) (Meta, error)
// Root returns the cache root directory.
Root() string
}
Store is the proxy's cache backend: a content store keyed by canonical cache keys, with per-key locking. The filesystem implementation (FileStore) writes the object at <root>/<key> and an artifact-compatible <key>.metadata.json sidecar so the object's path IS the key (the source of truth — no separate index).
type UpstreamRequest ¶
type UpstreamRequest struct {
// URL is the absolute upstream URL to fetch.
URL string
// Method defaults to GET when empty.
Method string
// Header carries mirror-supplied headers (e.g. Accept). The proxy merges
// credential and User-Agent headers on top, never dropping the Atmos UA.
Header http.Header
}
UpstreamRequest describes the upstream fetch for a route. The proxy builds and executes the actual *http.Request, applying credential, header, and User-Agent propagation centrally (see propagation.go) so private registries and the Atmos User-Agent are handled uniformly across all mirrors.