proxy

package
v1.222.0-rc.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 14, 2026 License: Apache-2.0 Imports: 23 Imported by: 0

Documentation

Overview

Package proxy implements a generic, protocol-agnostic caching HTTP proxy.

The proxy is the reusable core of the Atmos registry cache. It owns nothing Terraform-specific: protocol knowledge lives in Mirror adapters (the provider and module registry mirrors, and a future git mirror) that map an inbound request to a Route — a cache key, an upstream request, an artifact kind, and optional rewrite/verify hooks.

Design tenets:

  • The cache key is authoritative and is the backend object name; the filesystem path IS the key. There is no secondary index.
  • Fetch-once on a miss: no retries, no backoff. Retry/backoff is the caller's (Atmos's) responsibility, keeping failures observable.
  • One downloader, many readers per key, via three tiers of concurrency control (see serveCacheable): 1. A lock-free hit fast path — safe because both the object and its sidecar are written atomically (temp file + rename), so an unlocked reader never sees a torn file. 2. In-process singleflight collapses concurrent cold-key fills in one process (the Atmos bulk case: many terraform child processes share one proxy); waiters block with no timeout and cancel cleanly when their client leaves. 3. A cross-process pkg/cache.FileLock (context-bounded) collapses fills across processes that share a cache directory, held only around fetch + commit — never while streaming to the client. On Windows the file lock degrades to a no-op, so concurrent processes may redundantly download a cold key; the atomic commit still guarantees no corruption.
  • Atomic commit: stream to a temp file, hash, verify, then rename into place.
  • Per-run, in-memory statistics only (hits + bytes saved) for the savings report — no persistent hit/miss store.
  • Credentials and headers are forwarded to upstream so private registries keep working, and Terraform's User-Agent is forwarded verbatim.

Package proxy is a generic, protocol-agnostic caching HTTP proxy. It binds an ephemeral loopback listener, routes each request through a pluggable Mirror to a cache key + upstream request, and either serves a cache hit or fetches the object upstream once (no retries — the caller owns retry), verifies it, stores it atomically, and serves it.

The package deliberately knows nothing about Terraform: the Terraform provider and module registry mirrors are adapters that implement Mirror. It is named for proxying, not caching, so a future git mirror can reuse the same infrastructure.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func BuildUpstreamRequest

func BuildUpstreamRequest(ctx context.Context, inbound *http.Request, up UpstreamRequest) (*http.Request, error)

BuildUpstreamRequest builds a propagated upstream *http.Request from an inbound request. It is exported so Mirror adapters can make their own propagated pre-flight calls (e.g. resolving a provider download URL) using the same credential/header/User-Agent rules the proxy applies on the main fetch path.

Types

type ArtifactKind

type ArtifactKind int

ArtifactKind classifies a routed object for stats and freshness policy.

const (
	// KindMetadata is registry metadata (version listings, download resolution).
	// It honors metadata_ttl with stale-while-revalidate.
	KindMetadata ArtifactKind = iota
	// KindArtifact is immutable content (provider zips, module archives). Once
	// cached it never expires.
	KindArtifact
	// KindPassthrough is streamed straight through and never cached (e.g. a git::
	// module download that the HTTP proxy cannot cache).
	KindPassthrough
)

func (ArtifactKind) String

func (k ArtifactKind) String() string

String renders the kind for sidecar metadata and reporting.

type CommitRequest

type CommitRequest struct {
	// Key is the canonical cache key (and object path under the root).
	Key string
	// Data is the object content to store.
	Data io.Reader
	// Kind classifies the object for freshness policy.
	Kind ArtifactKind
	// ContentType is recorded in the sidecar and used when serving.
	ContentType string
	// Verify, when non-nil, validates the computed SHA-256 before the object is
	// committed; a non-nil error rejects it.
	Verify func(sha256Hex string) error
}

CommitRequest describes an object to commit to the store.

type Doer

type Doer interface {
	Do(req *http.Request) (*http.Response, error)
}

Doer performs an HTTP request. Implemented by pkg/http.Client and *http.Client; injectable so tests can stub upstream.

type Fetcher

type Fetcher func(ctx context.Context, up UpstreamRequest) (*http.Response, error)

Fetcher performs a propagated upstream fetch: it builds the request with the same credential/header/User-Agent propagation the proxy applies, then executes it. Mirrors use it inside Produce to compose multiple upstream calls (e.g. translating the provider registry protocol into a network-mirror response).

type FileStore

type FileStore struct {
	// contains filtered or unexported fields
}

FileStore is the filesystem-backed Store.

func NewFileStore

func NewFileStore(root string) *FileStore

NewFileStore returns a Store rooted at root.

func (*FileStore) Commit

func (s *FileStore) Commit(ctx context.Context, req CommitRequest) (Meta, error)

func (*FileStore) Lock

func (s *FileStore) Lock(key string) cache.FileLock

Lock returns a file lock on the object path (FileLock appends a .lock sidecar). The parent directory is created first so flock can open the lock file even on a cold key whose directory does not yet exist.

func (*FileStore) Open

func (s *FileStore) Open(key string) (io.ReadCloser, Meta, error)

Open returns a reader for the cached object plus its metadata.

func (*FileStore) Root

func (s *FileStore) Root() string

Root returns the cache root.

func (*FileStore) Stat

func (s *FileStore) Stat(key string) (Meta, bool, error)

Stat returns the object's metadata if it exists.

type Meta

type Meta struct {
	Size        int64
	SHA256      string
	FetchedAt   time.Time
	Kind        ArtifactKind
	ContentType string
}

Meta is the proxy's view of a cached object's metadata.

type Mirror

type Mirror interface {
	// Handles reports whether this mirror owns the request path.
	Handles(r *http.Request) bool
	// Route computes the cache key, upstream request, kind, and rewrite/verify hooks.
	Route(r *http.Request) (Route, error)
}

Mirror maps an inbound proxy request to a Route. Adapters (provider/module registry mirrors, a future git mirror) implement it.

type Options

type Options struct {
	// Mirrors are consulted in order; the first whose Handles returns true owns the
	// request.
	Mirrors []Mirror
	// Store is the cache backend.
	Store Store
	// Client performs upstream fetches. Defaults to pkg/http.NewDefaultClient().
	Client Doer
	// MetadataTTL is how long KindMetadata stays fresh. Zero serves metadata as
	// fresh forever (no revalidation).
	MetadataTTL time.Duration
	// StaleWhileRevalidate is the window past MetadataTTL during which stale
	// metadata is served while a background revalidation runs.
	StaleWhileRevalidate time.Duration
	// ReadHeaderTimeout bounds slow-header attacks. Defaults to 30s.
	ReadHeaderTimeout time.Duration
	// TLSCertificate, when set, makes the proxy serve HTTPS with this certificate and
	// the base URL uses the https scheme. Terraform/OpenTofu require provider network
	// mirrors to be https, so the cache always sets this. Nil serves plain HTTP.
	TLSCertificate *tls.Certificate
}

Options configure a Server.

type Route

type Route struct {
	// Key is the canonical cache key, which is also the backend object name. Empty
	// for KindPassthrough.
	Key string
	// Kind classifies the object for caching and freshness.
	Kind ArtifactKind
	// Upstream is the single upstream fetch performed on a miss (or always, for
	// passthrough). Ignored when Produce is set.
	Upstream UpstreamRequest
	// Produce, when non-nil, composes the (KindMetadata) response from one or more
	// upstream calls via the provided Fetcher — used when a single Upstream+Rewrite
	// cannot express the translation (e.g. building a provider <version>.json from
	// per-platform registry download calls). Returns the body and its content type.
	Produce func(ctx context.Context, fetch Fetcher, proxyBaseURL string) (body []byte, contentType string, err error)
	// ProduceArtifact, when non-nil, streams a generated KindArtifact body produced by
	// an operation rather than a single upstream HTTP GET (e.g. taring a go-getter
	// resolved module source tree). The returned reader is committed to the cache and
	// closed by the proxy. Takes precedence over Upstream and Produce.
	ProduceArtifact func(ctx context.Context) (body io.ReadCloser, contentType string, err error)
	// Rewrite, when non-nil, post-processes a KindMetadata body before it is cached
	// and served — e.g. rewriting provider download URLs back through the proxy.
	// proxyBaseURL is the proxy's own base URL ("http://127.0.0.1:<port>/").
	Rewrite func(body []byte, proxyBaseURL string) ([]byte, error)
	// Verify, when non-nil, checks artifact integrity before commit. It receives the
	// hex-encoded SHA-256 of the downloaded bytes (sufficient to verify a provider
	// zip's zh: hash). Returning an error rejects the object.
	Verify func(sha256Hex string) error
	// HeaderRewrite, when non-nil on a KindPassthrough route, may modify the upstream
	// response headers before they are written back — e.g. rewriting a module's
	// HTTP-archive X-Terraform-Get to route the archive back through the proxy while
	// leaving git:: sources untouched.
	HeaderRewrite func(h http.Header, proxyBaseURL string)
	// Serve, when non-nil, fully renders the served response from a cached object
	// instead of the default (200 + body). The mirror receives the cached body and the
	// live proxy base URL, so a value cached in a prior run (when the proxy bound a
	// different ephemeral port) is rewritten to the current run's URL at serve time —
	// e.g. a module download resolution caches the upstream source and emits
	// 204 + X-Terraform-Get pointing at the current-run _source route. Invoked on both
	// the hit and post-fill serve paths.
	Serve func(w http.ResponseWriter, body io.Reader, proxyBaseURL string) error
	// ContentType for the served response. Falls back to the upstream Content-Type.
	ContentType string
}

Route is a Mirror's decision for a single inbound request.

type Server

type Server struct {
	// contains filtered or unexported fields
}

Server is the ephemeral caching proxy.

func NewServer

func NewServer(opts Options) *Server

NewServer constructs a Server. Call Start to bind and serve.

func (*Server) BaseURL

func (s *Server) BaseURL() string

BaseURL returns the proxy base URL (empty before Start).

func (*Server) ServeHTTP

func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request)

ServeHTTP routes the request to the first mirror that handles it, then runs the cache-or-fetch flow.

func (*Server) Shutdown

func (s *Server) Shutdown(ctx context.Context) error

Shutdown gracefully stops the proxy. Safe to call once.

It marks the server closed (so no new fill is started), cancels in-flight fills, drains active HTTP handlers, then waits for the detached fill goroutines to finish. Waiting for the fills guarantees no background goroutine touches the cache directory after Shutdown returns — important for callers that delete the cache dir afterwards.

func (*Server) Start

func (s *Server) Start(_ context.Context) (string, error)

Start binds an ephemeral loopback listener (127.0.0.1:0), serves in a background goroutine, and returns the proxy's base URL. The scheme is https when a TLSCertificate is configured (the cache always configures one), else http.

func (*Server) Stats

func (s *Server) Stats() StatsSnapshot

Stats returns a snapshot of per-run cache statistics.

type Stats

type Stats struct {
	// contains filtered or unexported fields
}

Stats tracks per-run, in-memory cache statistics for the savings report. There is deliberately no persistent hit/miss store: the filesystem is the index, and hit rate is a per-run signal surfaced only by the end-of-run savings report. The report has two halves: bytes served from cache (hits → bandwidth saved) and bytes fetched from upstream and committed (cacheable misses → cache warmed).

func (*Stats) RecordCached

func (s *Stats) RecordCached(size int64)

RecordCached records a cache miss whose upstream response was fetched and committed to the cache: it counts as a miss and adds size bytes to the warmed total.

func (*Stats) RecordHit

func (s *Stats) RecordHit(size int64)

RecordHit records a cache hit, adding size (the bytes actually streamed to the client, not the on-disk object size) to the bandwidth-saved total. A hit is still counted when a transfer is interrupted; size then reflects only the bytes delivered.

func (*Stats) RecordMiss

func (s *Stats) RecordMiss()

RecordMiss records a cache miss whose upstream response was not committed to the cache (e.g. a non-cacheable status replayed to the client).

func (*Stats) Snapshot

func (s *Stats) Snapshot() StatsSnapshot

Snapshot returns a copy of the current statistics.

type StatsSnapshot

type StatsSnapshot struct {
	Hits          int
	Misses        int
	BytesSaved    int64
	ObjectsCached int
	BytesCached   int64
}

StatsSnapshot is an immutable copy of Stats at a point in time.

type Store

type Store interface {
	// Lock returns the per-key file lock (one writer, many readers).
	Lock(key string) cache.FileLock
	// Stat returns metadata for key and whether it exists.
	Stat(key string) (Meta, bool, error)
	// Open returns a reader for the cached object plus its metadata.
	Open(key string) (io.ReadCloser, Meta, error)
	// Commit streams the request data to a temp file (hashing as it goes), runs
	// Verify against the computed SHA-256, then atomically renames it into place and
	// writes the sidecar. A non-nil Verify error rejects the object (nothing is committed).
	Commit(ctx context.Context, req CommitRequest) (Meta, error)
	// Root returns the cache root directory.
	Root() string
}

Store is the proxy's cache backend: a content store keyed by canonical cache keys, with per-key locking. The filesystem implementation (FileStore) writes the object at <root>/<key> and an artifact-compatible <key>.metadata.json sidecar so the object's path IS the key (the source of truth — no separate index).

type UpstreamRequest

type UpstreamRequest struct {
	// URL is the absolute upstream URL to fetch.
	URL string
	// Method defaults to GET when empty.
	Method string
	// Header carries mirror-supplied headers (e.g. Accept). The proxy merges
	// credential and User-Agent headers on top, never dropping the Atmos UA.
	Header http.Header
}

UpstreamRequest describes the upstream fetch for a route. The proxy builds and executes the actual *http.Request, applying credential, header, and User-Agent propagation centrally (see propagation.go) so private registries and the Atmos User-Agent are handled uniformly across all mirrors.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL