storage

storage

File storage abstraction with presigned URL support. Backends are pluggable — swap between local filesystem (development) and AWS S3 (production) without changing application code.

Usage

import (
    "github.com/OpenNSW/core/storage"
    "github.com/OpenNSW/core/storage/drivers"
)

driver, err := storage.NewStorageFromConfig(ctx, storage.Config{
    Type: storage.TypeS3,
    S3: drivers.S3Config{
        Bucket: "my-uploads",
        Region: "ap-southeast-2",
    },
    PresignTTLSeconds: 900,
})
svc := storage.NewService(driver)

Use storage.TypeLocal for development. It stores files under Config.Local.BaseDir and stands in for S3's presigned URLs with URLs of its own, signed by Config.Local.PutSecret. They point at {RoutePrefix}/{key}/content on Config.Local.PublicURL; the prefix defaults to /api/v1/storage. Serve those routes with storage.LocalContentHandler on the server at that origin:

if local, ok := driver.(*drivers.LocalFSDriver); ok {
    storage.NewLocalContentHandler(local).RegisterRoutes(mux) // PUT and GET local.ContentPattern()
}

The routes need no auth middleware. Like an S3 presigned URL, each request has to carry the token the driver signed for that key.

Config embeds each driver's own config type (drivers.LocalConfig, drivers.S3Config) verbatim, rather than flattening every backend's settings into one struct — so each driver keeps ownership of its config shape and validation. All three carry yaml struct tags, so Config can be embedded in a larger application config struct and populated generically (e.g. via yaml.Unmarshal, or configyaml.LoadAndExpand for {{env:}}/{{file:}} secret placeholders):

storage:
  type: s3
  s3:
    bucket: my-uploads
    region: ap-southeast-2
    accessKey: "{{env:AWS_ACCESS_KEY_ID}}"
    secretKey: "{{env:AWS_SECRET_ACCESS_KEY}}"
  presignTTLSeconds: 900

Operations

Upload (presigned)

meta, err := svc.Upload(ctx, "passport.pdf", fileSize, "application/pdf")
// meta.Key       — opaque storage key; persist this to your database
// meta.UploadURL — presigned PUT URL; return to the client for direct upload

The client uploads directly to the storage backend — the file never passes through your application server.

Save (server-side)

For content your service already holds, such as a document it generated or a file it fetched from another system:

meta, err := svc.Save(ctx, "certificate.pdf", "application/pdf", bytes.NewReader(pdf), int64(len(pdf)))
// meta.Key — opaque storage key; persist this to your database. Save stores only the
//            bytes and content type, so the key is the only way to find the file again.

• There's no upload URL. The bytes go straight to the backend through the driver.
• The upload limits (WithAllowedUploadTypes, WithMaxUploadSize) don't apply. They govern what clients may upload, and a service decides for itself what it stores.
• size is recorded in the returned metadata as given; Save doesn't count or check it.
• For S3 over plain HTTP (e.g. a local MinIO), pass a seekable reader such as *bytes.Reader or *os.File.

Download

// Stream the file contents
content, mimeType, err := svc.Download(ctx, fileKey)

// Or get a presigned download URL for the client
url, err := svc.GetDownloadURL(ctx, fileKey)
// url — presigned GET URL, valid for PresignTTLSeconds

Delete

err := svc.Delete(ctx, fileKey)

Upload limits

By default, Upload accepts any MIME type and caps files at 32MB. Pass options to NewService to restrict the types or change the cap:

svc := storage.NewService(driver,
    storage.WithAllowedUploadTypes("application/pdf", "image/png"), // others: ErrContentTypeNotAllowed
    storage.WithMaxUploadSize(10<<20),                              // bytes; default 32MB
)

Upload also returns ErrInvalidSize for a size that isn't positive. The limits are checked when the upload URL is issued, and the URL is signed for that size and type, so a client can't upload something else with it. storage.NewHTTPHandler(svc) maps these errors to 415 and 400. The limits apply to Upload only, not to Save.

Implementing a custom driver

type StorageDriver interface {
    Save(ctx context.Context, key string, body io.Reader, contentType string) error
    Get(ctx context.Context, key string) (io.ReadCloser, string, error)
    Delete(ctx context.Context, key string) error
    GetDownloadURL(ctx context.Context, key string) (string, error)
    GetUploadURL(ctx context.Context, key string, contentType string, maxSizeBytes int64) (string, error)
}

Register your driver by passing it directly to storage.NewService(driver).

Config reference

Config.Type selects the backend (storage.TypeS3 / storage.TypeLocal); only the matching nested config is read. PresignTTLSeconds (top-level, required for both backends) controls how long a presigned upload/download URL stays valid.

S3 (Config.S3, drivers.S3Config, yaml:"s3")

Local filesystem (Config.Local, drivers.LocalConfig, yaml:"local")

Upgrading from the flattened Config

Config used to carry every backend's fields flattened at the top level. If you're updating existing construction code:

Upgrading: the local content routes have their own handler

HTTPHandler.UploadContentLocal and HTTPHandler.DownloadContent are gone. storage.LocalContentHandler serves them instead, on the path the local driver builds its URLs from:

// Before
mux.HandleFunc("PUT /api/v1/storage/{key}/content", handler.UploadContentLocal)
mux.HandleFunc("GET /api/v1/storage/{key}/content", handler.DownloadContent)

// After
storage.NewLocalContentHandler(localDriver).RegisterRoutes(mux)

The routes stay at /api/v1/storage/{key}/content unless you set Local.RoutePrefix. GetDownloadURL on the local driver now returns an error when PublicURL is empty, as GetUploadURL already did. Before, it returned the bare key.

Upgrading: upload content types are no longer restricted by default

The HTTP handler used to accept only these upload types: PDF, JPEG, PNG, GIF, WebP and XLSX. The limits now belong to the Service, and it accepts any type by default. To keep the old behavior, pass that list to NewService:

svc := storage.NewService(driver, storage.WithAllowedUploadTypes(
    "application/pdf",
    "image/jpeg",
    "image/png",
    "image/gif",
    "image/webp",
    // .xlsx only: OOXML workbooks cannot carry VBA macros, unlike legacy .xls.
    "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
))