blobstore

package
v1.0.16 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 5, 2026 License: Apache-2.0 Imports: 21 Imported by: 0

Documentation

Overview

Package blobstore is object storage for files a browser uploads and downloads directly: the API hands out short-lived signed URLs, and the bytes never pass through the API process.

The interface is narrow on purpose. A handler presigns an upload for a key it chose (PresignPut), the browser PUTs the file to that URL, and the handler later confirms what landed (Stat), hands out a download URL (PresignGet), lists a prefix or deletes. Nothing here streams bytes.

The main backend speaks the S3 API (NewS3, over minio-go), which covers MinIO in dev, AWS S3, Cloudflare R2, and Google Cloud Storage through its S3 interoperability with HMAC keys. NewMemory is the in-process fake for unit tests.

Keys are validated (ValidateKey): no leading "/", no "." or ".." segments, no empty segments, no backslashes or control characters. A multi-tenant caller wraps the store in Scoped, which joins every key under a fixed prefix, so code holding the scoped store cannot name another tenant's object, whatever key it is handed.

Bounding upload size

A presigned PUT is signed with the object's exact Content-Length (and its Content-Type), so the storage service refuses a body of any other size or type: the caller states the size up front, checks it against its own limit (PutOptions.MaxBytes does that here), and the signature enforces it. The alternative, a presigned POST policy with content-length-range, is not portable: Cloudflare R2 does not implement POST Object at all. After the upload, Stat reports what actually landed; confirm it before recording it.

Index

Constants

View Source
const (
	// DefaultTTL is the lifetime of a presigned URL whose options set none:
	// long enough to upload a large file on a slow link, short enough that a
	// leaked URL goes stale quickly.
	DefaultTTL = 15 * time.Minute
	// MaxTTL is the longest lifetime SigV4 allows a presigned URL.
	MaxTTL = 7 * 24 * time.Hour
	// MaxListLimit is the largest page List returns, S3's own maximum.
	MaxListLimit = 1000
)
View Source
const MaxKeyBytes = 1024

MaxKeyBytes is the longest key S3 accepts.

Variables

View Source
var (
	// ErrNotFound is Stat on a key that holds no object. It is an answer,
	// not a failure: the instrumentation logs it as an expected outcome.
	ErrNotFound = svcerr.NotFound("object")
	// ErrInvalidKey is a key (or prefix) ValidateKey refuses.
	ErrInvalidKey = svcerr.InvalidArgument("invalid object key")
	// ErrInvalidOptions is a PutOptions, GetOptions or ListOptions the store
	// cannot honour: a missing content type, a non-positive length, a TTL
	// past MaxTTL.
	ErrInvalidOptions = svcerr.InvalidArgument("invalid storage request")
	// ErrTooLarge is a PutOptions.ContentLength above its MaxBytes.
	ErrTooLarge = svcerr.InvalidArgument("upload too large")
)

The errors a Store returns, classified for svcerr: a handler that passes one (or an error wrapping one) to svcerr.Wrap answers with the matching Connect code, and the client reads only the short detail below. The context a store wraps around them (the key, the vendor's message) reaches the log, never the client.

Functions

func ValidateKey

func ValidateKey(key string) error

ValidateKey reports whether key is a safe object key: non-empty, valid UTF-8, at most MaxKeyBytes, "/"-separated segments none of which is empty, "." or "..", and no backslash or control character anywhere.

S3 itself accepts all of those, which is the point: a key is often built from user input (a file name), and a key that LOOKS like a path but does not behave like one ("a/../b" is a different object from "b", "/a" from "a") is how a prefix check gets fooled. Refusing them keeps a key's prefix an honest statement of where it lives.

Types

type GetOptions

type GetOptions struct {
	// TTL is how long the URL stays valid. Zero means DefaultTTL; it must be
	// at most MaxTTL.
	TTL time.Duration
	// DownloadName, when set, makes the response a download
	// (Content-Disposition: attachment) saved under this file name.
	DownloadName string
}

GetOptions describe a PresignGet URL.

type ListOptions

type ListOptions struct {
	// After lists only keys after this one: the previous page's Next.
	After string
	// Limit caps the page. Zero means MaxListLimit, and so does anything
	// above it.
	Limit int
}

ListOptions page through List.

type ListPage

type ListPage struct {
	Objects []Object
	// Next is the ListOptions.After that continues the listing; empty when
	// this page is the last.
	Next string
}

ListPage is one page of List.

type Memory

type Memory struct {
	// contains filtered or unexported fields
}

Memory is an in-process Store for unit tests. It validates keys and options exactly as the S3 backend does, and its presigned URLs are "memory://" URLs nothing can fetch: a test plays the browser by calling Put with the bytes the client would have sent.

func NewMemory

func NewMemory() *Memory

NewMemory returns an empty Memory store.

func (*Memory) Delete

func (m *Memory) Delete(_ context.Context, key string) error

Delete implements Store.

func (*Memory) Get

func (m *Memory) Get(key string) ([]byte, bool)

Get returns a copy of the bytes stored at key.

func (*Memory) List

func (m *Memory) List(_ context.Context, prefix string, opts ListOptions) (ListPage, error)

List implements Store.

func (*Memory) PresignGet

func (m *Memory) PresignGet(_ context.Context, key string, opts GetOptions) (PresignedRequest, error)

PresignGet implements Store.

func (*Memory) PresignPut

func (m *Memory) PresignPut(_ context.Context, key string, opts PutOptions) (PresignedRequest, error)

PresignPut implements Store.

func (*Memory) Put

func (m *Memory) Put(key string, data []byte, contentType string) error

Put stores data at key, as a completed upload would. It is the test's stand-in for the browser's PUT.

func (*Memory) Stat

func (m *Memory) Stat(_ context.Context, key string) (Object, error)

Stat implements Store.

type Object

type Object struct {
	// Key is the object's key, as the caller of this store names it (a
	// Scoped store reports keys without its prefix).
	Key          string
	Size         int64
	ContentType  string
	ETag         string
	LastModified time.Time
}

Object is what a store reports about one stored object.

type PresignedRequest

type PresignedRequest struct {
	// Method is the HTTP method: PUT for an upload, GET for a download.
	Method string
	// URL is the signed URL, valid until ExpiresAt.
	URL string
	// Headers are the headers the request must carry, exactly. For an upload
	// that is Content-Type. Content-Length is signed too, but a browser sets
	// it from the body and refuses to let a script set it; send a body of the
	// size that was presigned.
	Headers map[string]string
	// ExpiresAt is when the URL stops working.
	ExpiresAt time.Time
}

PresignedRequest is a request a client can send without credentials.

type PutOptions

type PutOptions struct {
	// ContentType is the upload's media type. Required: it is signed, so the
	// client must send exactly this Content-Type header.
	ContentType string
	// ContentLength is the upload's exact size in bytes. Required and > 0:
	// it is signed, so the storage service refuses a body of any other size.
	ContentLength int64
	// MaxBytes, when > 0, refuses to presign a ContentLength above it
	// (ErrTooLarge). The caller's own limit, checked before anything is
	// signed.
	MaxBytes int64
	// TTL is how long the URL stays valid. Zero means DefaultTTL; it must be
	// at most MaxTTL.
	TTL time.Duration
}

PutOptions describe the one upload a PresignPut URL admits.

type S3

type S3 struct {
	// contains filtered or unexported fields
}

S3 is a Store on an S3-compatible service. Build it with NewS3.

func NewS3

func NewS3(cfg S3Config) (*S3, error)

NewS3 validates cfg and builds the store. It makes no network call: a wrong credential or a missing bucket surfaces on the first Stat, Delete or List, and as a refused upload.

func (*S3) Delete

func (s *S3) Delete(ctx context.Context, key string) error

Delete implements Store.

func (*S3) List

func (s *S3) List(ctx context.Context, prefix string, opts ListOptions) (ListPage, error)

List implements Store.

func (*S3) PresignGet

func (s *S3) PresignGet(ctx context.Context, key string, opts GetOptions) (PresignedRequest, error)

PresignGet implements Store.

func (*S3) PresignPut

func (s *S3) PresignPut(ctx context.Context, key string, opts PutOptions) (PresignedRequest, error)

PresignPut implements Store. The URL signs the Content-Type and the exact Content-Length, so the service refuses any other body.

func (*S3) Stat

func (s *S3) Stat(ctx context.Context, key string) (Object, error)

Stat implements Store.

type S3Config

type S3Config struct {
	// Endpoint is the S3 API's base URL as THIS process reaches it, scheme
	// included: "http://localhost:9000" (MinIO in dev),
	// "https://s3.us-east-1.amazonaws.com", "https://<account>.r2.cloudflarestorage.com",
	// "https://storage.googleapis.com" (GCS interoperability). Required.
	Endpoint string
	// PublicEndpoint is the base URL a BROWSER reaches the same service at,
	// when that differs from Endpoint: a pod dials MinIO at
	// http://host.k3d.internal:9000 while the browser beside it uses
	// http://localhost:9000. Presigned URLs are signed for this host (the
	// host is part of the signature, so a URL cannot be rewritten after
	// signing). Empty means Endpoint.
	PublicEndpoint string
	// Region is the bucket's region. Empty means "us-east-1", which is what
	// MinIO and GCS interoperability accept; R2 takes "auto". Always set
	// explicitly on the client, so presigning never makes a network call to
	// look the region up.
	Region string
	// Bucket is the bucket every key lives in. Required.
	Bucket string
	// AccessKeyID and SecretAccessKey are the S3 credentials (an HMAC key
	// for GCS). Required.
	AccessKeyID     string
	SecretAccessKey string
	// PathStyle addresses the bucket as <endpoint>/<bucket>/<key> rather
	// than <bucket>.<endpoint>/<key>. MinIO needs it; for the others false
	// lets the client choose.
	PathStyle bool

	// Transport is the base RoundTripper (nil: http.DefaultTransport). It is
	// wrapped by observe.NewHTTPTransport, so every request to the storage
	// service is a client span carrying the caller's trace context.
	Transport http.RoundTripper
	// Logger receives the instrumentation's call records. nil: slog.Default.
	Logger *slog.Logger
	// Chain replaces the default instrumentation: a span, the
	// blobstore.{calls,errors,duration} metrics and a log record per call,
	// with ErrNotFound logged as an expected outcome. Build one with
	// observe.NewComponentChain to change it.
	Chain *observe.ComponentChain
}

S3Config configures NewS3. Every field is a value the app reads from its typed config (deploy/kcl's forge.BucketRef fills the env behind it); this package reads no environment of its own.

type Store

type Store interface {
	// PresignPut returns the request a client sends to upload key: a PUT of
	// exactly opts.ContentLength bytes of opts.ContentType. An existing
	// object at key is replaced by the upload.
	PresignPut(ctx context.Context, key string, opts PutOptions) (PresignedRequest, error)
	// PresignGet returns a GET request for key. It does not check that the
	// object exists; the URL answers 404 when it does not.
	PresignGet(ctx context.Context, key string, opts GetOptions) (PresignedRequest, error)
	// Stat reports the object at key, or ErrNotFound.
	Stat(ctx context.Context, key string) (Object, error)
	// Delete removes the object at key. Deleting a missing object is not an
	// error.
	Delete(ctx context.Context, key string) error
	// List returns one page of the objects whose keys start with prefix, in
	// key order. An empty prefix lists everything the store can see.
	List(ctx context.Context, prefix string, opts ListOptions) (ListPage, error)
}

Store is object storage addressed by key. Every method validates its key (ValidateKey) before calling out.

func Scoped

func Scoped(store Store, prefix string) (Store, error)

Scoped returns a Store that keeps every key under prefix: key "a/b" is stored as "<prefix>/a/b", List lists only under the prefix, and the keys it reports have the prefix removed. prefix must be a valid key (typically a tenant id, or "<tenant>/<kind>").

It is the tenant boundary. Key validation refuses ".." and empty segments, and S3 keys are opaque strings (nothing resolves them as paths), so no key handed to the scoped store can name an object outside the prefix. Nor does it trust the wrapped store: a Stat or List answer naming a key (or cursor) outside the prefix is an error, never passed through.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL