Documentation
¶
Overview ¶
Package pkgcache is the shared engine behind ephemerd's language package caches (npm, pip, pub). It provides three things the individual proxies would otherwise each reinvent:
- a BOUNDED on-disk store: every cache has a byte budget and evicts least-recently-used entries to stay under it, so a warm CI node never becomes the next disk-full incident (see [image_gc] / [buildkit] GC in config.example.toml for the history);
- TRAVERSAL-SAFE key→path mapping, with two independent checks;
- a pull-through fetch that FAILS OPEN: stale-on-error, and a documented hand-off to the caller when nothing is cached at all.
It is deliberately protocol-agnostic. Which URLs are immutable, which are mutable, and how a metadata document is rewritten are decisions each ecosystem's proxy makes.
Index ¶
- Constants
- func ArtifactKey(upstreamURL string) string
- func ArtifactURL(base, upstreamURL string) string
- func IsNotFound(err error) bool
- func MatchesETag(ifNoneMatch, tag string) bool
- func ParseArtifactPath(urlPath string) (string, error)
- func SafeSegments(urlPath string) ([]string, error)
- func WeakETag(body []byte) string
- func WriteDocument(w http.ResponseWriter, r *http.Request, body []byte, contentType string)
- type Cache
- func (c *Cache) Bytes() int64
- func (c *Cache) KeyPath(key string) (string, error)
- func (c *Cache) Len() int
- func (c *Cache) Lock(key string) *sync.Mutex
- func (c *Cache) MaxBytes() int64
- func (c *Cache) Open(key string) (*os.File, Meta, bool)
- func (c *Cache) Read(key string) ([]byte, Meta, bool)
- func (c *Cache) Root() string
- func (c *Cache) Write(key string, body []byte, meta Meta) error
- func (c *Cache) Writer(key string) (*Writer, error)
- type Config
- type Fetcher
- type HostAllowlist
- type Meta
- type NotFoundError
- type Request
- type Server
- func (s *Server) Addr() string
- func (s *Server) AdvertiseBase() string
- func (s *Server) AdvertiseHost() string
- func (s *Server) AdvertiseHostPort() string
- func (s *Server) Cache() *Cache
- func (s *Server) Fetcher() *Fetcher
- func (s *Server) Healthy() bool
- func (s *Server) Log() *slog.Logger
- func (s *Server) Passthrough(w http.ResponseWriter, r *http.Request, upstreamURL string)
- func (s *Server) RedirectUpstream(w http.ResponseWriter, r *http.Request, upstreamURL string)
- func (s *Server) ServeArtifactRoute(w http.ResponseWriter, r *http.Request, allow HostAllowlist, ...)
- func (s *Server) Start() error
- func (s *Server) Stop() error
- type ServerConfig
- type Writer
Constants ¶
const ArtifactRoute = "/_ephemerd/dl"
ArtifactRoute is the URL prefix every proxy serves immutable package artifacts under. Metadata documents (npm packuments, PEP 503 index pages, pub version listings) carry ABSOLUTE download URLs pointing at the upstream CDN; left alone, the client would fetch the bytes — which are the overwhelming bulk of a CI download — straight from the origin and the cache would only ever hold metadata. Each proxy therefore rewrites those URLs to this route, encoding the original URL in the path.
The prefix is namespaced under "_ephemerd" so it cannot collide with a real package name in any of the three ecosystems: npm package names may not begin with an underscore, PEP 503 project names are normalised to [a-z0-9-], and pub package names are [a-z0-9_] but are only ever served under /api/.
const DefaultMaxBytes int64 = 5 << 30 // 5 GiB
DefaultMaxBytes is the per-cache disk budget applied when an operator does not set one. Deliberately modest: three caches at this size is 15 GiB, and these nodes have filled their disks before.
const DefaultTimeout = 10 * time.Minute
DefaultTimeout bounds one upstream request. Generous: a cold PyPI wheel or an npm tarball for a large toolchain can legitimately take minutes on a slow link, and timing it out would turn a slow build into a failed one.
const HealthRoute = "/_ephemerd/healthz"
HealthRoute is the unauthenticated liveness endpoint every proxy serves. main.go probes it before injecting the proxy's env vars into job containers — see the fail-open discussion in each proxy's package doc.
Variables ¶
This section is empty.
Functions ¶
func ArtifactKey ¶
ArtifactKey is the cache key for an upstream artifact URL. Artifacts are keyed by the hash of their URL rather than by its path, so an upstream that serves two different files at colliding paths (or a path we could not safely map to disk) can never collide or escape. Sharded two levels deep to keep directory sizes sane on filesystems that dislike very wide directories.
func ArtifactURL ¶
ArtifactURL builds the proxy-side URL a client should use to download the artifact at upstreamURL. base is the proxy's advertised origin ("http://10.88.0.1:8084").
The upstream URL is base64url-encoded (no padding, so no "=" to escape) into one path segment, and the artifact's real filename is appended as a second, purely cosmetic segment. The filename matters to pip, which parses the distribution name, version and wheel tags out of the LAST path segment of a download URL — an opaque hash there would make every wheel unresolvable.
func IsNotFound ¶
IsNotFound reports whether err is an upstream 404/410.
func MatchesETag ¶
MatchesETag reports whether an If-None-Match header matches tag. Handles the "*" wildcard, comma-separated lists and the W/ weak prefix.
func ParseArtifactPath ¶
ParseArtifactPath decodes an ArtifactRoute request path back into the upstream URL it stands for.
The returned URL is NOT trusted: the caller must still check it against a HostAllowlist. Encoding the upstream URL in a client-controllable path would otherwise turn every proxy into an open relay — a job could ask for http://169.254.169.254/… and read cloud metadata with the daemon's network identity.
func SafeSegments ¶
SafeSegments splits a URL path into segments and rejects anything that could escape a cache root or confuse the host filesystem: traversal, empty segments, absolute/UNC forms, backslashes, drive letters, control characters, and Windows-reserved punctuation.
This is the FIRST of two independent traversal defences; Cache.KeyPath applies a containment check on the resolved path as the second. Issue #129 notes the Go module proxy's equivalent is untested — every branch here is covered by TestSafeSegments and the per-proxy traversal tests.
func WriteDocument ¶
WriteDocument serves a (possibly rewritten) metadata document.
The ETag is computed over the bytes WE serve, not the upstream ones: every proxy rewrites download URLs inside these documents, so replaying the upstream validator would tell the client that a body it has never seen is unchanged. Clients that send If-None-Match still get their 304.
Types ¶
type Cache ¶
type Cache struct {
// contains filtered or unexported fields
}
Cache is a bounded, LRU-evicting on-disk store.
Safe for concurrent use. The index (sizes and access times) lives in memory and is rebuilt by scanning Root at startup, so an operator running `ephemerd cache clear` underneath a live daemon degrades to a slightly wrong byte total, never to a crash.
func New ¶
New creates (or adopts) a cache rooted at cfg.Root and indexes whatever is already on disk.
func (*Cache) KeyPath ¶
KeyPath maps a slash-separated cache key onto an absolute path under the cache root.
TWO independent traversal defences, because this is the one function a hostile URL reaches the filesystem through:
- every segment is validated by SafeSegments, which rejects "..", ".", empty segments, backslashes, drive-letter colons and control bytes;
- the RESOLVED path is then required to be under the root, which catches anything the segment rules might not have anticipated (and is the check cmd/ephemerd/cache.go's resolveCacheRoot applies for the same reason).
func (*Cache) Open ¶
Open returns a readable handle on a cached body plus its metadata. Reports ok=false when the entry is absent or unreadable. The caller MUST close the returned file.
Opening counts as a use for LRU purposes.
func (*Cache) Read ¶
Read returns a cached body in full. Convenience for the small mutable documents (packuments, index pages, version listings) that always have to be rewritten in memory anyway.
type Config ¶
type Config struct {
// Root is the on-disk cache directory. Created if missing.
Root string
// MaxBytes is the disk budget. Zero takes DefaultMaxBytes. A NEGATIVE
// value disables the budget entirely — supported so an operator can opt
// out knowingly, never by accident.
MaxBytes int64
// Log receives eviction and error reporting.
Log *slog.Logger
}
Config configures a Cache.
type Fetcher ¶
Fetcher is the pull-through read path shared by all three proxies.
func NewFetcher ¶
NewFetcher wires a Fetcher with a sane HTTP client.
func (*Fetcher) Document ¶
Document fetches a whole document through the cache and returns its bytes. Intended for the small MUTABLE documents each proxy has to rewrite anyway (npm packuments, PEP 503 index pages, pub version listings).
FAIL-OPEN, layer one: when upstream errors, times out, or answers 5xx and a cached copy exists, the STALE COPY IS SERVED with a warning. A registry outage then shows up as a slightly out-of-date index rather than a red CI job. Callers layer a second fail-open on top (a redirect to the origin) for the case where nothing is cached at all.
func (*Fetcher) ServeArtifact ¶
ServeArtifact streams an IMMUTABLE artifact to the client, populating the cache on the way through.
Nothing is written to w until the upstream response is known good, so a caller can still fail open with a redirect after this returns an error.
The body is streamed, never buffered: a single PyPI wheel can be a gigabyte, and several jobs pull concurrently.
type HostAllowlist ¶
type HostAllowlist []string
HostAllowlist is the set of hosts a proxy is willing to fetch artifacts from. It is the SSRF fence around the ArtifactRoute: without it, the encoded-URL scheme would let any job reach any host the daemon can.
An entry matches the host exactly, or as a parent domain (an entry of "pythonhosted.org" matches "files.pythonhosted.org" but not "evilpythonhosted.org"). Ports are ignored — the fence is about which machine is reached, not which port on it.
func (HostAllowlist) Allows ¶
func (h HostAllowlist) Allows(rawURL string) bool
Allows reports whether rawURL may be fetched. Anything that is not http(s), has no host, or whose host is not covered by the list is denied.
func (HostAllowlist) WithHostsOf ¶
func (h HostAllowlist) WithHostsOf(urls ...string) HostAllowlist
WithHostsOf returns the allowlist extended with the hosts of the given URLs, so a configured upstream override is always permitted to serve its own artifacts without the operator having to restate it.
type Meta ¶
type Meta struct {
// ETag / LastModified are the upstream validators, replayed on
// revalidation and echoed to clients that do their own caching.
ETag string `json:"etag,omitempty"`
LastModified string `json:"last_modified,omitempty"`
// ContentType is the upstream Content-Type, or a proxy-chosen one for
// documents the proxy rewrote.
ContentType string `json:"content_type,omitempty"`
// Fetched is when the body was last known-good from upstream. Drives
// the TTL for mutable entries.
Fetched time.Time `json:"fetched"`
// Size is the body length in bytes.
Size int64 `json:"size,omitempty"`
// URL is the upstream URL this body came from. Not used for lookups —
// artifact keys are hashes, so without this the cache is unreadable by
// a human debugging a node.
URL string `json:"url,omitempty"`
}
Meta is the sidecar record stored beside every cached body. It carries the upstream validators, so a mutable entry that has aged out can be revalidated with a conditional GET (one 304 instead of a re-download), and the content type, so the proxy never has to sniff.
type NotFoundError ¶
type NotFoundError struct{ URL string }
NotFoundError marks an upstream 404/410 so a caller can pass it through as a 404 rather than treating it as an outage. A nonexistent package version must stay distinguishable from a registry being down: the first is a real answer the job should see, the second is what fail-open exists for.
func (*NotFoundError) Error ¶
func (e *NotFoundError) Error() string
type Request ¶
type Request struct {
// Key is the cache key. Must survive Cache.KeyPath.
Key string
// URL is the upstream URL to fetch on a miss.
URL string
// Immutable marks content that, once fetched, is never refetched and
// never revalidated: npm tarballs, PyPI wheels/sdists, pub archives.
// Every one of those is addressed by a (name, version) that a registry
// will not re-publish with different bytes.
Immutable bool
// TTL is how long a MUTABLE entry is served without contacting
// upstream. After it, the entry is revalidated with a conditional GET,
// so the steady-state cost of a stale-but-unchanged document is a 304.
// Zero or negative means "revalidate every time".
TTL time.Duration
// Accept, when set, is forwarded upstream. Content negotiation is part
// of the cache identity for pip (PEP 691 JSON vs PEP 503 HTML) and npm
// (abbreviated vs full packument), so callers that vary on Accept MUST
// also vary their Key.
Accept string
// DefaultContentType is used when upstream does not supply one.
DefaultContentType string
}
Request describes one pull-through fetch.
type Server ¶
type Server struct {
// contains filtered or unexported fields
}
Server is the common lifecycle for a cache proxy: bind, serve, shut down, and answer a health probe. Each ecosystem proxy embeds one and supplies its own routing handler.
func NewServer ¶
NewServer creates the cache and the HTTP scaffolding. handler receives every request that is not the health probe.
func (*Server) AdvertiseBase ¶
AdvertiseBase is the base URL job containers use to reach this proxy.
func (*Server) AdvertiseHost ¶
AdvertiseHost is the host part of AdvertiseBase, for tools (pip) that want a bare host rather than a URL.
func (*Server) AdvertiseHostPort ¶
AdvertiseHostPort is the host:port form of the advertised address.
func (*Server) Healthy ¶
Healthy probes the proxy's own health endpoint over the loopback port it is actually bound to. main.go calls this before injecting the proxy's env vars into job containers: for npm, pip and pub the env var itself has NO fallback (unlike GOPROXY's "|direct"), so the only way to fail open at the tool level is to not point the tool here in the first place when the proxy is not answering.
func (*Server) Passthrough ¶
Passthrough relays a GET to upstream WITHOUT caching it, for endpoints that are neither package metadata nor an artifact (npm's search and ping, pub's advisory feed). Nothing is stored, so nothing can go stale.
Fail-open: an unreachable upstream becomes a 307 to the origin rather than an error, so the client can complete the request itself.
func (*Server) RedirectUpstream ¶
RedirectUpstream is the metadata-path counterpart of the artifact fail-open: when a document cannot be fetched and nothing is cached, send the client to the origin rather than returning an error.
The client then gets the ORIGINAL document, whose download URLs point at the origin CDN, so the whole job proceeds uncached and unbroken. That is strictly better than the alternative every "replace the registry" scheme suffers from (cargo's source.replace-with, notably), where a dead cache is a hard build failure.
func (*Server) ServeArtifactRoute ¶
func (s *Server) ServeArtifactRoute(w http.ResponseWriter, r *http.Request, allow HostAllowlist, defaultContentType string)
ServeArtifactRoute handles a request under ArtifactRoute: decode the upstream URL the proxy itself advertised, check it against the SSRF fence, then stream it through the bounded cache.
FAIL-OPEN, layer two: if the artifact cannot be fetched (upstream down, timeout, 5xx, cache unwritable) the client is 307-redirected to the real origin. npm, pip and pub all follow redirects, so a dead cache costs the job a slower, uncached download — never a failed one. A genuine upstream 404 is passed through as a 404, because "this version does not exist" is a real answer the job must see.
type ServerConfig ¶
type ServerConfig struct {
// Name is the proxy name, used in logs ("npm", "pip", "pub").
Name string
// ListenAddr is the address to bind AND the address advertised to
// containers — normally the bridge gateway, e.g. "10.88.0.1:8084".
ListenAddr string
// CacheDir is the on-disk cache root.
CacheDir string
// MaxBytes is the cache disk budget (see pkgcache.Config).
MaxBytes int64
// Cleanup wipes the cache dir on Stop. Defaults to false: a
// pull-through cache emptied on every restart saves nothing.
Cleanup bool
Log *slog.Logger
}
ServerConfig configures the HTTP scaffolding shared by the npm, pip and pub proxies.
type Writer ¶
type Writer struct {
// contains filtered or unexported fields
}
Writer stages a cache body in a temp file and installs it atomically on Commit. Bodies are streamed rather than buffered: a PyPI wheel or an npm tarball can be hundreds of megabytes and must never be held in RAM, especially with several jobs downloading at once.
func (*Writer) Abort ¶
func (w *Writer) Abort()
Abort discards the staged entry. Safe to call after Commit (no-op).
Source Files