Documentation
¶
Overview ¶
Package cdn signs download URLs on a CDN in front of an object store (decision 0029). A blob store that has a CDN takes one of these signers in its Options and answers blob.ViaCDN with the signer's URL for the blob's object key, so the store is never wrapped and keeps its other interfaces.
Every signer issues URLs on time windows: a URL minted in a window expires when the window ends plus the TTL, so a URL lives at least the TTL and at most twice it, and every client in one window is handed the same URL for a blob when the signature is deterministic.
Role ¶
platform layer: CDN URL signing for the blob stores. No module owns a CDN, so the signers sit in the platform beside platform/blob, whose ViaCDN route they serve. The s3 store takes a CloudFront and the gcs store a CloudCDN as its Signer; the azure store takes a FrontDoor and mints the SAS itself, with the start and expiry that Window gives. The composition root builds them from the blob.<backend>.cdn configuration, which ParseDomain also validates. It imports platform/blob, the standard library and the AWS SDK's CloudFront signer.
Inputs ¶
- Provider configuration, read at start: CloudFrontConfig (the distribution host, the ID of a public key in one of its trusted key groups, the private key and the hash), CloudCDNConfig (the load balancer host, the key name on the backend bucket and the key) and FrontDoorConfig (the endpoint host). Each takes PathPrefix, the CDN origin's path in storage, which its URLs leave out, and the two signers take a clock.
- Keys: ParsePrivateKey reads a CloudFront key in PEM, RSA as PKCS#1 or PKCS#8 or ECDSA as SEC 1 or PKCS#8, and ParseCloudCDNKey a Cloud CDN key file, 16 bytes in base64url.
- When a download redirects, the blob's object key under the bucket root, which the stores export as ObjectKey, and a Request: the TTL, and the file name and media type that travel as response overrides. A FrontDoor takes the storage path "<container>/<blob name>" and the SAS query the azure store minted.
Outputs ¶
- Signer (Provider, Covers, Sign) and its two implementations, CloudFront (NewCloudFront) and CloudCDN (NewCloudCDN).
- FrontDoor (NewFrontDoor; Provider, Covers, URL), which is not a Signer: Front Door checks no signature, so it only moves the SAS that the azure store mints onto the endpoint.
- Expiry and Window: when a URL issued now expires, and the start of its window, which an Azure SAS takes as its start time.
- ParseDomain, a CDN host as a base URL, and SignCloudCDN, Google's signing algorithm on its own, with which tests sign URLs for an edge.
- The constants HashSHA1, HashSHA256, CloudCDNKeySize and CloudFrontMaxSize, the largest file a redirect sends to CloudFront.
- ErrConfig, which every configuration error wraps.
Invariants ¶
- A store with a CDN answers blob.ViaCDN with the signer's URL and is never wrapped in a CDN decorator: a wrapper hides the store's other interfaces, blob.PartSizer among them, which silently changes how the mirror sizes its chunks (issue 39).
- URLs are issued on time windows. Windows are TTL long and start on a multiple of the TTL since the Unix epoch, and a URL issued in one expires at the window's end plus the TTL, in whole seconds, because every CDN states expiry in epoch seconds. So a URL lives between one and two TTLs, and where the signature is deterministic (RSA, HMAC-SHA1 and the window-aligned SAS a FrontDoor carries) every client in a window gets the same URL for a blob, which the edge can cache. An ECDSA signature differs each time. A TTL under a second is refused.
- CloudFront and Cloud CDN sign the response overrides with the rest of the URL, so an edge refuses a URL whose file name was edited. A space in an override is sent as %20, never "+", which object stores read literally.
- Covers refuses a store prefix that the CDN origin's path does not contain, and the stores call it when they are built, so a CDN that cannot reach a store's blobs fails at start. Sign and URL refuse a key outside that path. Both refusals wrap ErrConfig.
- Every error text starts "cdn:" and stays byte-identical.
References ¶
- Decision 0029: docs/research/decisions/0029-cdn-delivery-and-download-redirect.md ("Signing contract", the CDN configuration and the time windows).
- Research 19: docs/research/19-cdn-and-download-redirect.md (the three providers' signing, and CloudFront's 50 GB cacheable size).
- https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-signed-urls.html, https://docs.cloud.google.com/cdn/docs/using-signed-urls and https://learn.microsoft.com/en-us/azure/frontdoor/scenario-storage-blobs.
- Decision 0044: docs/research/decisions/0044-modular-monolith-migration.md, "Dependency rules": the platform imports no module.
- Target layout: docs/research/architecture-options/5-modular-monolith.md, line 152 (platform/cdn) and line 1050 (the mapping row).
- Plan: docs/implementation/plan.md#p4b-reference-deployments-and-cdn-delivery and #p9d-modular-monolith-migration, epic PLT (https://github.com/weaveplatform/artifactweave/issues/938).
- https://github.com/weaveplatform/artifactweave/issues/981
Status ¶
Implemented P4b: the CloudFront signer, the time windows and ParseDomain (issue 120), the Cloud CDN signer (issue 121), FrontDoor and Window (issue 122), and CloudFrontMaxSize, which keeps a file over CloudFront's cacheable size off the edge (issue 402). Moved here in P9d.
Testing ¶
- TestUnitExpiryIsTheWindowEndPlusTheTTL and TestUnitWindow: the window arithmetic, and a TTL under a second refused.
- TestUnitParseDomain: hosts and test edge URLs accepted; a path, query, fragment, user or other scheme refused.
- TestUnitCloudFrontURLsRepeatWithinAWindow and TestUnitCloudCDNURLsRepeatWithinAWindow: one URL per blob in a window, and another in the next.
- TestUnitCloudFrontRefusals and TestUnitCloudCDNRefusals: unusable configuration and keys, a key outside the path prefix and a zero TTL, each with ErrConfig.
- TestUnitSignCloudCDNMatchesGooglesSample and TestUnitCloudCDNSignsTheOverridesAndStripsThePathPrefix: URLs checked against vectors computed with Google's Python sample.
- TestUnitFrontDoorURL and TestUnitFrontDoorOriginPath: the SAS carried unchanged and the origin path stripped.
- TestAccCloudFrontURLsPassTheEdge, for every key form and hash, and TestAccCloudCDNURLsPassTheEdge: signed ranged GETs through a cdntest edge, with the overrides; the Cloud CDN edge refuses unsigned, edited, unknown-key and expired URLs.
The s3, gcs and azure stores sign through these types in their own tests. Through cdntest edges, the acceptance feature cdn.feature runs jf rt dl and crane pull, and cdn_front_door.feature runs jf rt dl.
Index ¶
- Constants
- Variables
- func Expiry(now time.Time, ttl time.Duration) (time.Time, error)
- func ParseCloudCDNKey(file []byte) ([]byte, error)
- func ParseDomain(domain string) (*url.URL, error)
- func ParsePrivateKey(pemBytes []byte) (crypto.Signer, error)
- func SignCloudCDN(rawURL, keyName string, key []byte, expires time.Time) string
- func Window(now time.Time, ttl time.Duration) (start, expiry time.Time, err error)
- type CloudCDN
- type CloudCDNConfig
- type CloudFront
- type CloudFrontConfig
- type FrontDoor
- type FrontDoorConfig
- type Request
- type Signer
Constants ¶
const ( HashSHA1 = "sha1" HashSHA256 = "sha256" )
CloudFront hash algorithms for CloudFrontConfig.Hash.
const CloudCDNKeySize = 16
CloudCDNKeySize is the length of a Cloud CDN signing key: 128 bits.
const CloudFrontMaxSize int64 = 50_000_000_000
CloudFrontMaxSize is the largest GET response CloudFront caches: AWS lists "Maximum cacheable file size per HTTP GET response" as 50 GB. Artifactory answered a larger file requested through CloudFront with 400 until 7.104.5 (RTDEV-49674), so a redirect sends such a file to the object store instead. The decimal reading of 50 GB is the smaller one, so no file over the limit is sent to the edge whichever unit AWS means.
Variables ¶
var ErrConfig = errors.New("cdn: invalid configuration")
ErrConfig reports an unusable CDN configuration.
Functions ¶
func Expiry ¶
Expiry is when a URL issued at now with ttl expires: the end of the current window plus ttl, where windows are ttl long and start on a multiple of ttl since the Unix epoch. Whole seconds, because every CDN expresses expiry in epoch seconds.
func ParseCloudCDNKey ¶
ParseCloudCDNKey decodes a Cloud CDN key file: base64url, with or without padding, around 16 raw bytes.
func ParseDomain ¶
ParseDomain reads a CDN domain: a host such as "d111.cloudfront.net", which means https, or an http or https URL with no path, such as "http://localhost:8080" for a test edge.
func ParsePrivateKey ¶
ParsePrivateKey reads a PEM RSA or ECDSA private key in any of the forms AWS and OpenSSL write: PKCS#1, SEC 1 or PKCS#8.
func SignCloudCDN ¶
SignCloudCDN signs rawURL as Google's documented algorithm does: append Expires and KeyName, take the HMAC-SHA1 of the whole URL with the raw key, and append it as a padded base64url Signature. Any query the URL already has, such as the response overrides, is part of what is signed.
func Window ¶
Window returns the window now falls in for ttl and the expiry of a URL issued in it, as Expiry does: windows are ttl long, start on a multiple of ttl since the Unix epoch, and a URL issued in one expires at its end plus ttl. A signer that fixes its start time to the window start, as an Azure SAS does, then issues the same URL for a blob throughout the window.
Types ¶
type CloudCDN ¶
type CloudCDN struct {
// contains filtered or unexported fields
}
CloudCDN signs URLs for a Cloud CDN backend bucket. It signs each URL whole rather than a URLPrefix: a prefix signature would open every blob under the prefix to whoever holds one URL.
Cloud CDN does not refuse unsigned requests; the bucket must be private, readable only by the cloud-cdn-fill service account, so that the origin refuses them (decision 0029).
func NewCloudCDN ¶
func NewCloudCDN(c CloudCDNConfig) (*CloudCDN, error)
NewCloudCDN returns a Cloud CDN signer.
type CloudCDNConfig ¶
type CloudCDNConfig struct {
// Domain is the load balancer host (see ParseDomain).
Domain string
// KeyName names the key on the backend bucket.
KeyName string
// Key is the key file's contents: 128 random bits, base64url, as
// "gcloud compute backend-buckets add-signed-url-key" reads them.
Key []byte
// PathPrefix is the path inside the bucket that the load balancer maps
// to its root; empty when it serves the bucket root.
PathPrefix string
// Now is the clock; nil means time.Now.
Now func() time.Time
}
CloudCDNConfig configures a Cloud CDN signer.
type CloudFront ¶
type CloudFront struct {
// contains filtered or unexported fields
}
CloudFront signs canned-policy URLs for a CloudFront distribution. The distribution's cache policy must key on the path plus response-content-disposition and response-content-type only: never the signature parameters, or every URL would be a cache miss, and always the overrides, or one item would be served with another's filename.
func NewCloudFront ¶
func NewCloudFront(c CloudFrontConfig) (*CloudFront, error)
NewCloudFront returns a CloudFront signer.
func (*CloudFront) Covers ¶
func (c *CloudFront) Covers(storePrefix string) error
Covers checks the distribution's origin reaches every key under storePrefix.
type CloudFrontConfig ¶
type CloudFrontConfig struct {
// Domain is the distribution host (see ParseDomain).
Domain string
// KeyPairID is the ID of the public key, in one of the distribution's
// trusted key groups, that matches PrivateKey.
KeyPairID string
// PrivateKey is the PEM private key: RSA as PKCS#1 or PKCS#8, or ECDSA
// as SEC 1 or PKCS#8.
PrivateKey []byte
// Hash is HashSHA1 (the default) or HashSHA256.
Hash string
// PathPrefix is the distribution origin's path inside the bucket; empty
// when the origin is the bucket root.
PathPrefix string
// Now is the clock; nil means time.Now.
Now func() time.Time
}
CloudFrontConfig configures a CloudFront signer.
type FrontDoor ¶
type FrontDoor struct {
// contains filtered or unexported fields
}
FrontDoor places read SAS URLs on an Azure Front Door endpoint (decision 0029). Front Door checks no signature of its own: it forwards the request, query string included, to the storage origin, which checks the SAS. So FrontDoor signs nothing and is not a Signer: the azure store mints a window-aligned SAS with its own credentials, which may need a network call for a user delegation key, and FrontDoor moves it onto the endpoint.
The route must key its cache on the full query string (query_string_caching_behavior = UseQueryString), or a cached private blob is served to anyone who asks for its path, and must cap cache expiry at the SAS window, because Front Door keeps serving a cached object after its SAS has expired.
func NewFrontDoor ¶
func NewFrontDoor(c FrontDoorConfig) (*FrontDoor, error)
NewFrontDoor returns a FrontDoor.
func (*FrontDoor) Covers ¶
Covers checks the endpoint reaches every blob under storagePrefix, the store's "<container>/<prefix>".
type FrontDoorConfig ¶
type FrontDoorConfig struct {
// Endpoint is the Front Door endpoint host, such as
// "artifacts-abc123.z01.azurefd.net" (see ParseDomain).
Endpoint string
// PathPrefix is the route's origin path: the leading part of
// "<container>/<blob name>" that the endpoint leaves out, for example the
// container when the origin path is "/<container>". Empty when the
// endpoint path is the whole storage path.
PathPrefix string
}
FrontDoorConfig configures a FrontDoor.
type Request ¶
type Request struct {
// TTL is the least time the URL stays valid once handed out.
TTL time.Duration
// Filename and ContentType, when set, travel as response overrides the
// CDN passes to the origin and keys its cache on.
Filename string
ContentType string
}
Request describes one signed URL.
type Signer ¶
type Signer interface {
// Provider names the CDN for logs and errors, for example "cloudfront".
Provider() string
// Covers returns an error unless every object key under storePrefix
// has a path on the CDN: a CDN whose origin is a folder of the bucket
// reaches only the keys inside it.
Covers(storePrefix string) error
// Sign returns a URL for objectKey, the key under the bucket's root.
Sign(objectKey string, r Request) (string, error)
}
Signer signs URLs for objects behind one CDN distribution.