storage

package
v0.4.20 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 17, 2026 License: Apache-2.0 Imports: 22 Imported by: 0

Documentation

Overview

Package storage provides S3 file storage with per-app prefix isolation and CDN URL generation.

S3 is the source of truth. Cloudflare R2 is the CDN cache layer (Worker handles caching). Production: STS credentials injected per invocation via Lambda payload. Dev uses locally minted, prefix-scoped STS credentials against MinIO. This enforcement is self-imposed because the module process holds MinIO's parent key: it gives dev identical key shapes and code paths, cross-app isolation in one tunnel runner, and an end-to-end test of production's policy shape. The real security wall is the production IAM session policy.

Index

Constants

View Source 
const MaxGetBytes = 8 << 20

MaxGetBytes caps what Get will read into memory. Get exists for small, module-owned control files — HLS playlists, manifests, sidecars — and a cap keeps a mistyped key from turning a 4 GB source video into an OOM. Streaming a large object to a viewer is what PresignGet is for.

Variables

View Source 
var ErrContentTypeRequired = errors.New("mirrorstack/storage: presign put requires a content type")

ErrContentTypeRequired is returned when PresignPut is called without a content type. Fail closed: see PresignPut's own comment for why there is no default.

View Source 
var ErrCredentialExpired = errors.New("mirrorstack/storage: storage credential expired before a safe presigned URL could be created")

ErrCredentialExpired is returned when a vended STS credential has no safe lifetime left for a new presigned URL.

View Source 
var ErrNoCredential = fmt.Errorf("mirrorstack/storage: no storage credentials — presigned URLs require authenticated context")

ErrNoCredential is returned when PresignPut/PresignGet is called without storage credentials. A deployed invocation must receive credentials regardless of route class; public delivery routes may legitimately use storage too.

View Source 
var ErrNotDeclared = errors.New("mirrorstack/storage: storage was not declared — call ms.RequireStorage() during module startup before using ms.Storage(ctx)")

ErrNotDeclared is returned when module code calls ms.Storage without first declaring ms.RequireStorage during startup. Undeclared modules receive no storage credential from the platform.

View Source 
var ErrNotVended = errors.New("mirrorstack/storage: the platform did not vend storage credentials for this invocation (envelope resources.storage was absent) — this is a PLATFORM misconfiguration, not a module bug: the operator must wire dispatch's module-storage vend (MS_MODULE_STORAGE_ROLE_ARN / _BUCKET / _CDN_BASE)")

ErrNotVended is returned when a deployed invocation carries no storage credential.

View Source 
var ErrObjectTooLarge = errors.New("mirrorstack/storage: object exceeds MaxGetBytes")

ErrObjectTooLarge is returned when an object exceeds MaxGetBytes.

Functions

func UnderLambda 

func UnderLambda() bool

UnderLambda reports whether the process is running under AWS Lambda.

func WithCredential 

func WithCredential(ctx context.Context, cred Credential) context.Context

WithCredential returns a context with the storage credential set.

func WithCredentialProvider 

func WithCredentialProvider(ctx context.Context, provider CredentialProvider) context.Context

WithCredentialProvider installs a renewable storage credential source.

Types

type Client 

type Client struct {
	// contains filtered or unexported fields
}

Client wraps an S3 presigner with app-scoped key prefix and CDN base URL.

func NewFromCredential 

func NewFromCredential(cred Credential) (*Client, error)

NewFromCredential creates a Client from platform-injected STS credentials.

func NewFromCredentialForDev 

func NewFromCredentialForDev(cred Credential, cfg DevConfig) (*Client, error)

NewFromCredentialForDev constructs the same credential-based client while separating container and browser endpoints. Credential validation prevents this path (and every other constructor) from ever producing an empty prefix.

func NewFromProvider 

func NewFromProvider(ctx context.Context, provider CredentialProvider) (*Client, error)

NewFromProvider creates a storage client whose AWS credential provider asks for current scoped credentials for operations and presigns. Resource scope fields are immutable; only key material and expiry may rotate.

func Open deprecated 

func Open(context.Context) (*Client, error)

Open is retained for source compatibility with older SDK consumers. An unscoped process-wide client can no longer be constructed safely; module code must declare ms.RequireStorage and resolve ms.Storage per request so the app identity participates in the credential scope.

Deprecated: use ms.RequireStorage during startup and ms.Storage(ctx) in a request handler.

func (*Client) CreateMultipart 

func (c *Client) CreateMultipart(ctx context.Context, key, contentType string) (*MultipartUpload, error)

CreateMultipart starts a multipart upload and returns a MultipartUpload handle. Use this for files larger than ~100MB. 

upload, err := s.CreateMultipart(ctx, "video.mp4", "video/mp4")
part1URL, _ := upload.PresignPart(ctx, 1, 15*time.Minute)
part2URL, _ := upload.PresignPart(ctx, 2, 15*time.Minute)
// browser uploads parts in parallel
upload.Complete(ctx, []CompletedPart{{1, etag1}, {2, etag2}})

func (*Client) DeletePrefix 

func (c *Client) DeletePrefix(ctx context.Context, prefix string) error

DeletePrefix removes every object below prefix. prefix is relative to the credential's app/module scope and must end in "/" so it cannot match a sibling such as "videos/123-backup". A missing prefix succeeds.

func (*Client) ForApp deprecated 

func (c *Client) ForApp(_, _ string) *Client

ForApp is retained for source compatibility but can no longer rescope a vended credential. Prefix and CDN base are platform-owned because the public CDN path must remain inside the same app/module boundary as the IAM policy.

Deprecated: resolve ms.Storage(ctx) separately for each request.

func (*Client) Get 

func (c *Client) Get(ctx context.Context, key string) ([]byte, error)

Get reads one object's bytes through the S3 API.

WHY THIS EXISTS, and when to use it instead of PresignGet. A presigned URL is built for a BROWSER: it is signed against the client-facing endpoint, which in dev is the host-published address of the object store. Fetching one from inside the module is not merely wasteful (sign, then a second HTTP round trip to your own storage) — it is wrong, because the module runs somewhere the client-facing address may not resolve. In the dev container it resolves to the container itself and the read fails with `dial tcp [::1]:9000: connection refused`, which is how this was found: every HLS playlist read 502'd.

Get goes over the S3 API with the module's own credential and the endpoint that credential was minted for, so it is correct from wherever the module runs. Rule of thumb: bytes YOU need, Get; bytes the VIEWER needs, PresignGet or Delivery.

The key is relative to the module's prefix, exactly like every other method here, and is validated the same way.

func (*Client) PresignDownload 

func (c *Client) PresignDownload(ctx context.Context, key, filename string, expires time.Duration) (string, error)

PresignDownload generates a direct presigned GET whose response is an attachment. The signed response override works across origins and keeps large objects on the direct S3 path.

func (*Client) PresignGet 

func (c *Client) PresignGet(ctx context.Context, key string, expires time.Duration) (string, error)

PresignGet generates a presigned S3 GET URL for direct download (bypasses CDN). Requires storage credentials in context (Platform/Internal routes only).

func (*Client) PresignPut 

func (c *Client) PresignPut(ctx context.Context, key, contentType string, expires time.Duration) (string, error)

PresignPut generates a presigned S3 PUT URL for uploading a file.

🔴 contentType IS PART OF THE SIGNATURE, AND THAT IS THE WHOLE POINT OF THIS PARAMETER. It used to be absent, so the type was whatever the CLIENT sent — and a module's allow-list ("only png, jpeg, webp") decided nothing but the extension of the key. An uploader could name a `.png` key and store `image/svg+xml` on it; the CDN then replayed that stored type, cross-origin, from an origin that also serves every module's JS bundle (mirrorstack-ai/mirrorstack-core-v2#1469).

With the type signed, S3 itself refuses a PUT whose Content-Type header does not match — before a byte is stored, with no code of ours in the path. The caller's allow-list becomes the enforced type instead of decoration.

The uploading client MUST send the same `Content-Type` header on the PUT. That is not new work: every browser upload already sets one.

An empty contentType is REFUSED rather than defaulted. A caller that does not know what it is accepting cannot make this decision safely, and a default (octet-stream, or "whatever the client says") is exactly the state this parameter exists to end. Requires storage credentials in context (Platform/Internal routes only).

func (*Client) URL 

func (c *Client) URL(key string) (string, error)

URL returns the CDN URL for a file. The Cloudflare Worker handles R2 caching: R2 hit → serve, R2 miss → fetch from S3 → cache in R2 → serve. Returns an error if the key would escape the prefix scope when normalized by a browser.

type CompletedPart 

type CompletedPart struct {
	PartNumber int32
	ETag       string
}

CompletedPart represents a successfully uploaded part.

type Credential 

type Credential struct {
	Bucket string `json:"bucket"`
	Region string `json:"region"`
	// Prefix is apps/<appUUID>/<moduleUUID>/ in production. Dev uses the
	// platform-minted Config.ID in the module segment because no installed
	// module UUID exists in a local tunnel session.
	Prefix          string `json:"prefix"`
	CDNBase         string `json:"cdnBase"` // "https://media.mirrorstack.ai"
	AccessKeyID     string `json:"accessKeyId"`
	SecretAccessKey string `json:"secretAccessKey"`
	SessionToken    string `json:"sessionToken"`
	// ExpiresAt is the STS credential expiry. Older platform envelopes may omit
	// it; clients preserve that compatibility while newer envelopes use it to
	// keep presigned URLs within the credential lifetime.
	ExpiresAt time.Time `json:"expiresAt,omitzero"`
}

Credential holds S3 access details injected by the platform per invocation. STS temp credentials scoped to the app/module S3 prefix.

func CredentialFrom 

func CredentialFrom(ctx context.Context) *Credential

CredentialFrom reads the storage credential from the context.

type CredentialProvider 

type CredentialProvider interface {
	Credential(context.Context) (Credential, error)
}

CredentialProvider supplies renewable scoped storage credentials. Bucket, Region, Prefix and CDNBase must remain stable for the provider's lifetime.

func CredentialProviderFrom 

func CredentialProviderFrom(ctx context.Context) CredentialProvider

CredentialProviderFrom reads the renewable storage credential source.

type DevConfig 

type DevConfig struct {
	Bucket, Region  string
	Endpoint        string // S3_ENDPOINT — container -> MinIO
	PublicEndpoint  string // S3_PUBLIC_ENDPOINT — browser -> MinIO
	STSEndpoint     string // S3_STS_ENDPOINT — defaults to Endpoint
	CDNBase         string // CDN_BASE_URL
	RoleARN         string // S3_STS_ROLE_ARN
	AccessKeyID     string // AWS_ACCESS_KEY_ID
	SecretAccessKey string // AWS_SECRET_ACCESS_KEY
}

DevConfig is the local S3-compatible (MinIO) plane, read from env.

func DevConfigFromEnv 

func DevConfigFromEnv() (DevConfig, error)

DevConfigFromEnv reads the explicitly configured local plane. Decision 03 forbids an AWS fallback: that fallback previously waited on EC2 IMDS and returned a 502 while also collapsing tenant scope to the bucket root.

type DevMinter 

type DevMinter struct {
	// contains filtered or unexported fields
}

DevMinter exchanges the local parent key for prefix-scoped session keys.

func NewDevMinter 

func NewDevMinter(_ context.Context, cfg DevConfig) (*DevMinter, error)

NewDevMinter constructs the local STS client without invoking network I/O.

func (*DevMinter) Config 

func (m *DevMinter) Config() DevConfig

Config returns the endpoints needed to construct a dev client after Mint.

func (*DevMinter) Mint 

func (m *DevMinter) Mint(ctx context.Context, appID, moduleID string) (Credential, error)

Mint returns a cached credential for the app/module pair or assumes a fresh role.

type MultipartUpload 

type MultipartUpload struct {
	// contains filtered or unexported fields
}

MultipartUpload manages a multipart S3 upload. Browser uploads parts in parallel via presigned URLs, then calls Complete.

The client field is invariant non-nil after construction by CreateMultipart. Do not construct MultipartUpload directly in tests — use CreateMultipart so the client/bucket/key/uploadID invariants hold.

func (*MultipartUpload) Abort 

func (u *MultipartUpload) Abort(ctx context.Context) error

Abort cancels the multipart upload. Call this if the upload fails or is abandoned. S3 will clean up the parts. Also add a lifecycle rule to auto-abort after 24h.

func (*MultipartUpload) Complete 

func (u *MultipartUpload) Complete(ctx context.Context, parts []CompletedPart) error

Complete finalizes the multipart upload with the list of completed parts. Each part must include the PartNumber and the ETag returned by S3 after upload.

func (*MultipartUpload) PresignPart 

func (u *MultipartUpload) PresignPart(ctx context.Context, partNumber int32, expires time.Duration) (string, error)

PresignPart generates a presigned URL for uploading a single part. Part numbers must be in [1, 10000] (S3 limit). Browser PUTs the data to this URL.

type PrefixDeleter 

type PrefixDeleter interface {
	DeletePrefix(ctx context.Context, prefix string) error
}

PrefixDeleter is an optional storage capability for removing every object below a relative directory prefix. Keeping it separate from Storer preserves source compatibility for existing storage implementations and test doubles.

type Storer 

type Storer interface {
	PresignPut(ctx context.Context, key, contentType string, expires time.Duration) (string, error)
	PresignGet(ctx context.Context, key string, expires time.Duration) (string, error)
	// Get reads an object the MODULE needs, over the S3 API. Distinct from
	// PresignGet, which builds a URL for a VIEWER against the client-facing
	// endpoint — an address the module itself may not be able to reach.
	Get(ctx context.Context, key string) ([]byte, error)
	URL(key string) (string, error)
	CreateMultipart(ctx context.Context, key, contentType string) (*MultipartUpload, error)
}

Storer is the interface for storage operations.

Adding a method here is source-breaking for any implementation or test double outside this package, which is why an optional capability stays OFF it and is offered as a separate interface instead — see PrefixDeleter in delete.go. Get is on it deliberately rather than by oversight: ms.Storage hands every module this interface, and reading your own object is as fundamental as writing one, so a module should not need a type assertion to do it. Pre-1.0, and *Client is the only implementation in the workspace.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.