storage

package
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 13, 2026 License: MIT Imports: 24 Imported by: 0

README

Object storage

Use storage.Storage to write, open, list, and delete complete objects through your chosen driver. The facade preserves driver error identities and adds optional telemetry and mutation notifications. See the file storage guide for application wiring and driver selection.

Write and reopen a report

This focused example uses an existing ctx context.Context and imports errors, fmt, io, arkoren.dev/core/service/storage, and arkoren.dev/core/service/storage/memorystorage. Memory storage needs no lifecycle calls and loses its objects when the process exits.

files := storage.New(memorystorage.New())
_, err := files.PutBytes(ctx, "reports/a.txt", []byte("report"))

object, err := files.Open(ctx, "reports/a.txt")
contents, err := io.ReadAll(object.Contents)
err = errors.Join(err, object.Contents.Close())
fmt.Println(string(contents)) // report

You own an opened body and must close it, including after a read failure. Put borrows its reader and never closes it. The facade does not start or stop a native driver; keep that lifecycle with the code that created the driver.

Keep keys literal and portable

Keys are UTF-8 relative paths of at most 1024 bytes. They are never cleaned, renamed, case-folded, or Unicode-normalized. Validation rejects traversal, empty segments, control characters, Win32 filename metacharacters, trailing dots or spaces, and reserved device names. This includes NUL, CON, COM1–COM9, LPT1–LPT9, superscript port digits ¹, ², ³, and console handles CONIN$/CONOUT$. Device basenames with extensions such as NUL.txt are excluded for Windows versions where they still alias devices, even where Windows 11 permits them. Ordinary names such as COM0, COM10, and console.txt remain valid. Filesystem-specific path-length and case-sensitivity limits still apply.

If an existing backend contains a newly rejected name, migrate that object deliberately through the native backend before changing application references. The portable API cannot access that name, and it never silently renames data. The same restriction applies to configured driver prefixes validated with ValidatePrefix.

Serve downloads without hiding failures

Authorize access before calling Download or Inline. Seekable bodies retain Go's native range, multipart-range, HEAD, and conditional request handling. Non-seekable bodies stream sequentially with HEAD support. Inline keeps its sandbox and safe-type restrictions.

The helpers always close opened bodies. They return observed read, seek, write, short-write, and close failures, joining cleanup errors with transfer errors. An error can arrive after HTTP response commitment: return it through an error-aware handler for observation, but never append a replacement problem body or automatically retry the transfer. Read-only multipart upload sources are closed best-effort; cleanup does not undo a successful storage write.

Cancellation and signed URLs

An already canceled context wins over invalid request input. Memory uploads check cancellation between reads and before publication; listings check during key scanning and metadata copying and return no partial page on cancellation. A blocked reader, mutex acquisition, or synchronous sort cannot be forcibly interrupted by these checks.

Signing keeps the Driver contract's validation order: context, key, TTL including the backend maximum, method, bindings, then capability. The facade delegates backend-sensitive checks together. Memory and filesystem drivers still reject valid signing requests with ErrUnsupported. A signed URL is a bearer credential: authorize its recipient and keep it out of ordinary logs and telemetry.

The storage API documents each operation and ownership rule. The memory driver API documents its snapshots, cancellation limits, and pagination defaults.

Documentation

Overview

Package storage defines portable streaming object storage contracts.

Keys are literal UTF-8 relative paths of at most 1024 bytes with slash-separated segments. ValidateKey rejects traversal/empty segments, control characters, Win32 filename metacharacters, trailing dots/spaces, and reserved device names. Device aliases with extensions are excluded for older Windows compatibility even where Windows 11 permits them. Prefixes are empty or use the same syntax with one optional final slash. Names are never cleaned, renamed, or case-folded; filesystem-specific length and case-sensitivity limits can still apply.

files := storage.New(memorystorage.New())
info, err := files.PutBytes(ctx, "reports/a.txt", []byte("report"))
if err != nil {
	return err
}
fmt.Println(info.Size) // 6

Every Driver implementation maps a missing key from Open or Stat to an error wrapping ErrNotFound. Storage preserves that driver-owned identity; it does not infer missing objects from provider errors. Delete succeeds when a key is missing. Open returns a body that callers must close. Put consumes its reader once and, on success, makes the entire supplied object current. List cursors are opaque, valid only with the same driver and prefix, and have no ordering or snapshot guarantee while objects change.

Storage.Walk flattens pages into a collection.TryLazySlice of ObjectInfo. It defers listing until consumption, follows opaque continuations, and stops requesting pages as soon as the consumer stops. Limit remains a per-page limit; Take limits consumed objects. Each traversal starts fresh I/O with the original options and captured context, without a snapshot guarantee. Errors terminate traversal; Items returns the successful prefix with the error. Cancellation is checked before page requests and between buffered objects. Page preparation can still scan/sort matching keys or perform S3 read-ahead.

objects, err := files.Walk(ctx, storage.ListOptions{Prefix: "reports/", Limit: 100}).Take(10).Items()
if err != nil {
	return err // objects is an incomplete prefix, not a complete listing.
}
for _, info := range objects {
	fmt.Println(info.Key)
}

Driver.SignURL returns a time-limited URL that lets a client with no credentials GET or PUT one key directly against the backend, for example a browser upload or a download link. Signing does not check that the object exists. Drivers whose backend has no presigned access, such as the memory and filesystem drivers, return an error wrapping ErrUnsupported; a TTL that is not positive or exceeds the driver's maximum wraps ErrInvalidTTL. Cancellation precedes malformed input. The facade delegates maximum-TTL, method, binding, and capability checks together so a backend limit cannot be preempted by a later method/binding error. Signed URLs are bearer credentials; authorize their recipients and keep them out of ordinary logs and telemetry.

Storage.PutFile and Storage.PutFileAs store an uploaded *multipart.FileHeader, and Storage.Download and Storage.Inline serve an object as an HTTP response with Range support when the driver's body is seekable. Both helpers close the body and preserve observed transfer and cleanup errors through errors.Is. Errors returned after response commitment are for logging; never append a replacement problem body or automatically retry the transfer. Upload helpers close their read-only multipart sources best-effort without changing an already completed storage write's result.

Telemetry

With Builder.Telemetry, or telemetry.Default when it is not set, each Storage operation emits a client span "storage {operation}" and the arkoren.storage.operations and arkoren.storage.operation.duration metrics, identically for every driver. Builder.Name tells apart several stores. Keys, prefixes, cursors and signed URLs are never recorded, and a missing object is not an error. Disabled telemetry costs nothing. Walk reuses List instrumentation once per requested page, with no separate traversal span or metric.

files := storage.New(driver, func(store *storage.Builder) {
	store.Telemetry(app.Telemetry())
})
_ = files.Driver()

Events

With Builder.Events, Storage.Put and the helpers built on it publish ObjectWritten ("storage.object.written") and Storage.Delete publishes ObjectDeleted ("storage.object.deleted") after success, for every driver, with the store name and key. Reads, lists, and signed URLs publish nothing.

files := storage.New(driver, func(store *storage.Builder) {
	store.Events(app.Events())
})
_ = files.Driver()

New takes Builder policies for Telemetry, Name and Events. They run directly once in order on a fresh draft; nil entries panic and expanded nil/empty slices supply no edits. Publication detaches settings, not the live driver or shared services. Retained drafts cannot reconfigure Storage. Nil telemetry resolves the process default on each operation.

Index

Examples

Constants

View Source
const DefaultName = "default"

DefaultName is the store name reported in telemetry when Builder.Name is empty.

Example:

name := storage.DefaultName
storage.New(driver, func(store *storage.Builder) {
	store.Name(name)
})

Variables

View Source
var (
	// ObjectWritten is published when [Storage.Put], or a helper built on it,
	// stores an object.
	//
	// Example:
	//
	//	unsubscribe, err := bus.Listen(storage.ObjectWritten, func(ctx context.Context, payload storage.ObjectWrittenEvent) error {
	//		fmt.Println(payload.Object.Size)
	//		return nil
	//	})
	//	if err != nil {
	//		return err
	//	}
	//	defer unsubscribe()
	ObjectWritten = event.NewTopic[ObjectWrittenEvent]("storage.object.written")

	// ObjectDeleted is published when [Storage.Delete] succeeds.
	// It reports the API observation even when the key was absent already.
	//
	// Example:
	//
	//	unsubscribe, err := bus.Listen(storage.ObjectDeleted, func(ctx context.Context, payload storage.ObjectDeletedEvent) error {
	//		fmt.Println(payload.Store)
	//		return nil
	//	})
	//	if err != nil {
	//		return err
	//	}
	//	defer unsubscribe()
	ObjectDeleted = event.NewTopic[ObjectDeletedEvent]("storage.object.deleted")
)
View Source
var ErrInvalidKey = errors.New("storage invalid key")

ErrInvalidKey indicates that a key or prefix fails the literal portable syntax. Correct or migrate the name; retries never repair invalid input.

Example:

err := storage.ValidateKey("../private")
fmt.Println(errors.Is(err, storage.ErrInvalidKey)) // true
View Source
var ErrInvalidSignOptions = errors.New("storage invalid sign options")

ErrInvalidSignOptions indicates SignOptions fields that cannot be honored, such as a ContentType or ContentLength on a GET URL or a negative ContentLength. Binding validation is available without contacting a backend.

Example:

err := (storage.SignOptions{ContentLength: -1}).ValidateBinding()
fmt.Println(errors.Is(err, storage.ErrInvalidSignOptions)) // true
View Source
var ErrInvalidTTL = errors.New("storage invalid ttl")

ErrInvalidTTL indicates a signing TTL that is not positive or exceeds the driver's maximum. It does not mean a signed URL has merely expired after successful creation.

Example:

_, err := files.SignURL(ctx, "report.pdf", storage.SignOptions{})
fmt.Println(errors.Is(err, storage.ErrInvalidTTL)) // true
View Source
var ErrNotFound = errors.New("storage object not found")

ErrNotFound indicates that Open or Stat found no logical object. Drivers wrap this identity; the facade preserves it rather than parsing provider messages.

Example:

_, err := files.Stat(ctx, "reports/missing.pdf")
if errors.Is(err, storage.ErrNotFound) {
	return nil // An absent optional report needs no recovery.
}
return err
View Source
var ErrObjectNotFound = problem.New(func(details *problem.Builder) {
	details.Type("tag:arkoren.dev,2026-09:core:service/storage:object-not-found")
	details.Title("File not found")
	details.Detail("The requested file does not exist.")
	details.Status(http.StatusNotFound)
})

ErrObjectNotFound is the 404 problem Storage.Download and Storage.Inline return for a missing object, so handlers can return it directly and the router renders it. The returned error also wraps the driver's error, so errors.Is(err, ErrNotFound) stays true.

Example:

failure := storage.ErrObjectNotFound.Derive(func(details *problem.Builder) {
	details.Detail("The request cannot be completed.")
})
_ = failure.Status()
View Source
var ErrUnsupported = errors.New("storage unsupported operation")

ErrUnsupported indicates that a driver cannot perform the requested operation, such as signing a URL for a backend without presigned access. Invalid signing input keeps its own error before capability rejection.

Example:

_, err := files.SignURL(ctx, "report.pdf", storage.SignOptions{TTL: time.Minute})
if errors.Is(err, storage.ErrUnsupported) {
	http.Redirect(writer, request, "/reports/download", http.StatusSeeOther)
	return nil // The application route streams the authorized object.
}
return err

Functions

func ValidateKey

func ValidateKey(key string) error

ValidateKey checks a relative slash-separated key of at most 1024 UTF-8 bytes. It rejects empty/dot segments, control characters, Windows filename metacharacters, trailing dots/spaces, and reserved device names case-insensitively. Device basenames followed by extensions are also excluded for portability to Windows versions where they remain device aliases; Windows 11 permits some such names. Names are checked literally, never cleaned, escaped, renamed, or case-folded for storage. A rejection wraps ErrInvalidKey. Existing rejected keys need an application-owned migration through the native backend, not automatic repair.

Example:

if err := storage.ValidateKey("reports/annual.pdf"); err != nil {
	return err
}
Example

ExampleValidateKey shows literal key validation without performing object I/O.

package main

import (
	"errors"
	"fmt"

	"arkoren.dev/core/service/storage"
)

func main() {
	fmt.Println(storage.ValidateKey("reports/a.pdf") == nil)
	fmt.Println(errors.Is(storage.ValidateKey("NUL"), storage.ErrInvalidKey))
}
Output:
true
true

func ValidatePrefix

func ValidatePrefix(prefix string) error

ValidatePrefix checks that prefix is empty or a portable key with one optional final slash. It applies the same device-name and character restrictions as ValidateKey, preserving literal prefix bytes instead of cleaning or renaming them. These restrictions also apply to native driver prefixes configured through this validator; migrate affected backend names deliberately before upgrading.

Example:

if err := storage.ValidatePrefix("reports/"); err != nil {
	return err
}

Types

type Builder

type Builder struct {
	// contains filtered or unexported fields
}

Builder configures New before publication. Callbacks run directly once, synchronously in order on one fresh draft. Nil callback entries panic; expanded nil or empty callback slices supply no edits. Retaining the draft cannot change published settings. Methods return the same draft.

Example:

storage.New(driver, func(store *storage.Builder) {
	store.Name("uploads")
})

func (*Builder) Events

func (store *Builder) Events(bus *event.Bus) *Builder

Events selects the shared bus receiving ObjectWritten and ObjectDeleted after successful writes and deletes. Nil publishes nothing; the bus and driver remain live dependencies rather than copied configuration data.

This setting replaces the previous value on the construction draft. Services and loggers remain shared; publication detaches owned settings.

Example:

storage.New(driver, func(store *storage.Builder) {
	store.Events(nil)
})

func (*Builder) Name

func (store *Builder) Name(name string) *Builder

Name sets the store name reported as arkoren.storage.store and in mutation events. Empty selects DefaultName when an operation runs; choose a fixed, non-secret label rather than a user ID or object key.

This setting replaces the previous value on the construction draft. Services and loggers remain shared; publication detaches owned settings.

Example:

storage.New(driver, func(store *storage.Builder) {
	store.Name("primary")
})

func (*Builder) Telemetry

func (store *Builder) Telemetry(t *telemetry.Telemetry) *Builder

Telemetry sets the telemetry that records operations. Without it the store follows telemetry.Default.

This setting replaces the previous value on the construction draft. Services and loggers remain shared; publication detaches owned settings.

Example:

storage.New(driver, func(store *storage.Builder) {
	store.Telemetry(nil)
})

type BuilderFunc

type BuilderFunc = func(*Builder)

BuilderFunc is a construction policy accepted by New. Policies share one draft, run directly once in order, and return no value. A supplied nil policy panics; expanding a nil or empty slice supplies no policies.

Example:

policies := []storage.BuilderFunc{
	func(store *storage.Builder) {
		store.Name("uploads")
	},
}
files := storage.New(driver, policies...)
_ = files.Driver()

type Driver

type Driver interface {
	// Open returns an object body that the caller must close. Missing objects return an error
	// wrapping ErrNotFound.
	//
	// Example:
	//
	//	object, err := driver.Open(ctx, "report.pdf")
	//	if err != nil {
	//		return err
	//	}
	//	_, err = io.Copy(destination, object.Contents)
	//	return errors.Join(err, object.Contents.Close())
	Open(context.Context, string) (Object, error)

	// Put replaces an object with the complete contents from the reader and returns the stored
	// object's metadata. When the reader fails, the previous object, if any, is left intact and
	// no new object appears.
	// The caller retains ownership of the reader; drivers must not close it.
	//
	// Example:
	//
	//	info, err := driver.Put(ctx, "report.txt", strings.NewReader("report"))
	//	_ = info
	//	return err
	Put(context.Context, string, io.Reader) (ObjectInfo, error)

	// Stat returns object metadata without opening its contents. Missing objects return an error
	// wrapping ErrNotFound.
	//
	// Example:
	//
	//	info, err := driver.Stat(ctx, "report.pdf")
	//	if err != nil {
	//		return err
	//	}
	//	fmt.Println(info.Size)
	Stat(context.Context, string) (ObjectInfo, error)

	// Delete removes an object and succeeds if the object is absent.
	//
	// Example:
	//
	//	if err := driver.Delete(ctx, "obsolete/report.pdf"); err != nil {
	//		return err
	//	}
	Delete(context.Context, string) error

	// List returns a page of object metadata for a prefix. Prefix is a plain string prefix of
	// full keys, so "reports" also matches "reports-2024/a". Objects carry full keys and are
	// ordered lexicographically by key. A Limit at or below zero selects a driver default, and
	// a driver may return fewer objects than requested. NextCursor is empty exactly when no
	// further objects match, even if the final page is full.
	//
	// Example:
	//
	//	page, err := driver.List(ctx, storage.ListOptions{Prefix: "reports/", Limit: 20})
	//	_ = page
	//	return err
	List(context.Context, ListOptions) (Page, error)

	// SignURL returns a URL that grants options.Method on key without further credentials until
	// options.TTL elapses. GET returns the object body; PUT replaces the object with the same
	// semantics as Put. Signing does not check existence: a GET URL for a missing key fails
	// when it is fetched.
	//
	// Validation is deterministic for every driver and runs in this order: a canceled ctx
	// returns an error wrapping the context's error; an invalid key returns ErrInvalidKey; a
	// TTL at or below zero, or above the driver's maximum, returns ErrInvalidTTL; a method
	// other than GET or PUT (empty means GET) returns ErrUnsupported; a ContentType or
	// ContentLength on a GET, or a negative ContentLength, returns ErrInvalidSignOptions (see
	// [SignOptions.ValidateBinding]); a driver without signing support returns ErrUnsupported.
	//
	// A driver that can bind ContentType and ContentLength into the signature does so, so the
	// backend rejects uploads that differ. Callers should set both for user uploads.
	//
	// A driver that wraps another must sign the key it would actually store, for example the
	// prefixed one.
	//
	// Example:
	//
	//	signed, err := driver.SignURL(ctx, "report.pdf", storage.SignOptions{TTL: time.Minute})
	//	_ = signed // A bearer credential, not a diagnostic value.
	//	return err
	SignURL(context.Context, string, SignOptions) (*url.URL, error)
}

Driver stores and retrieves portable logical objects. Direct callers must supply keys and prefixes accepted by ValidateKey and ValidatePrefix; drivers reject others with an error wrapping ErrInvalidKey. Implementations must return errors wrapping ErrNotFound when Open or Stat cannot find a valid key.

A key that exists only as the leading path of other keys, such as "dir" when only "dir/a" is stored, is not an object: Open and Stat report it missing and Delete succeeds without touching anything. Storing both "a" and "a/b" is not portable, because a filesystem cannot hold both, so callers should not rely on it.

Every operation fails with an error wrapping the context's error when ctx is already canceled, and then changes nothing.

Example:

var driver storage.Driver = memorystorage.New()
info, err := driver.Put(ctx, "report.txt", strings.NewReader("report"))
if err != nil {
	return err
}
fmt.Println(info.Size) // 6

type ListOptions

type ListOptions struct {
	// Prefix limits the listing to matching keys.
	//
	// Example:
	//
	//	options := storage.ListOptions{Prefix: "reports/"}
	//	page, err := files.List(ctx, options)
	//	_ = page // Check err before using the returned objects.
	//	return err
	Prefix string

	// Cursor continues a prior page from the same driver and prefix.
	//
	// Example:
	//
	//	options := storage.ListOptions{Prefix: "reports/", Cursor: page.NextCursor}
	//	next, err := files.List(ctx, options)
	//	_ = next
	//	return err
	Cursor string

	// Limit caps each page's returned objects, including pages fetched by
	// [Storage.Walk]; it is not a traversal-wide limit. Values at or below zero
	// use a driver default. Use Walk(...).Take(n) to limit consumed objects.
	//
	// Example:
	//
	//	options := storage.ListOptions{Limit: 20}
	//	page, err := files.List(ctx, options)
	//	_ = page
	//	return err
	Limit int
}

ListOptions selects a lexical prefix and one driver-defined page. Cursors are opaque continuations for the same driver and prefix, not snapshot tokens.

Example:

page, err := files.List(ctx, storage.ListOptions{Prefix: "reports/", Limit: 20})
if err != nil {
	return err
}
fmt.Println(len(page.Objects))

type Object

type Object struct {
	// ObjectInfo describes the opened object's metadata.
	//
	// Example:
	//
	//	object := storage.Object{ObjectInfo: storage.ObjectInfo{Key: "report.pdf"}}
	//	fmt.Println(object.ObjectInfo.Key) // report.pdf
	ObjectInfo

	// Contents is the object body. Callers must close it. It may also
	// implement [io.Seeker]; the memory and filesystem drivers do. Helpers
	// such as [Storage.Download] use that to answer range and conditional
	// requests. Seeking is optional: the conformance suite does not require
	// it, so check with a type assertion and never assume it.
	//
	// Example:
	//
	//	object := storage.Object{Contents: io.NopCloser(strings.NewReader("report"))}
	//	_, err := io.Copy(destination, object.Contents)
	//	return errors.Join(err, object.Contents.Close())
	Contents io.ReadCloser
}

Object combines metadata with an open body owned by its caller. Opening does not read all bytes; callers must close Contents after consumption or failure.

Example:

object, err := files.Open(ctx, "report.pdf")
if err != nil {
	return err
}
_, err = io.Copy(destination, object.Contents)
return errors.Join(err, object.Contents.Close())

type ObjectDeletedEvent

type ObjectDeletedEvent struct {
	// Store is [Builder.Name], or [DefaultName] when it is empty.
	//
	// Example:
	//
	//	payload := storage.ObjectDeletedEvent{Store: storage.DefaultName}
	//	fmt.Println(payload.Store) // default
	Store string

	// Key is the deleted key.
	//
	// Example:
	//
	//	payload := storage.ObjectDeletedEvent{Key: "obsolete/report.txt"}
	//	fmt.Println(payload.Key) // obsolete/report.txt
	Key string
}

ObjectDeletedEvent reports successful idempotent deletion through the facade. It is also emitted for an already absent object; it does not prove a prior object existed and is not a durable delivery receipt.

Example:

unsubscribe, err := bus.Listen(storage.ObjectDeleted, func(ctx context.Context, payload storage.ObjectDeletedEvent) error {
	fmt.Println(payload.Store)
	return nil
})
if err != nil {
	return err
}
defer unsubscribe()

type ObjectInfo

type ObjectInfo struct {
	// Key identifies the object with a portable storage key.
	//
	// Example:
	//
	//	info := storage.ObjectInfo{Key: "reports/a.pdf"}
	//	fmt.Println(info.Key) // reports/a.pdf
	Key string

	// Size is the object length in bytes.
	//
	// Example:
	//
	//	info := storage.ObjectInfo{Size: 1024}
	//	fmt.Println(info.Size) // 1024
	Size int64

	// ModifiedAt is the object's most recent modification time.
	//
	// Example:
	//
	//	info := storage.ObjectInfo{ModifiedAt: time.Unix(0, 0).UTC()}
	//	fmt.Println(info.ModifiedAt.Format(time.RFC3339)) // 1970-01-01T00:00:00Z
	ModifiedAt time.Time
}

ObjectInfo describes an object without opening its contents. Keys are logical names, Size counts bytes, and ModifiedAt is supplied by the backend.

Example:

info, err := files.Stat(ctx, "reports/a.pdf")
if err != nil {
	return err
}
fmt.Println(info.Key, info.Size)

type ObjectWrittenEvent

type ObjectWrittenEvent struct {
	// Store is [Builder.Name], or [DefaultName] when it is empty.
	//
	// Example:
	//
	//	payload := storage.ObjectWrittenEvent{Store: storage.DefaultName}
	//	fmt.Println(payload.Store) // default
	Store string

	// Object describes the stored object.
	//
	// Example:
	//
	//	payload := storage.ObjectWrittenEvent{Object: storage.ObjectInfo{Key: "report.txt", Size: 6}}
	//	fmt.Println(payload.Object.Size) // 6
	Object ObjectInfo
}

ObjectWrittenEvent reports metadata after a successful facade write. It contains no contents or signed URL; events are observations, not durable work.

Example:

unsubscribe, err := bus.Listen(storage.ObjectWritten, func(ctx context.Context, payload storage.ObjectWrittenEvent) error {
	fmt.Println(payload.Store, payload.Object.Size)
	return nil
})
if err != nil {
	return err
}
defer unsubscribe()

type Page

type Page struct {
	// Objects contains the objects selected for this page.
	//
	// Example:
	//
	//	page := storage.Page{Objects: []storage.ObjectInfo{{Key: "report.pdf"}}}
	//	fmt.Println(page.Objects[0].Key) // report.pdf
	Objects []ObjectInfo

	// NextCursor continues this listing, or is empty when no further page exists.
	//
	// Example:
	//
	//	if page.NextCursor != "" {
	//		next, err := files.List(ctx, storage.ListOptions{Prefix: "reports/", Cursor: page.NextCursor})
	//		_ = next
	//		return err
	//	}
	NextCursor string
}

Page is an owned page of metadata. Mutating its outer Objects slice does not change storage. A continuation exists only when NextCursor is nonempty.

Example:

page, err := files.List(ctx, storage.ListOptions{Prefix: "reports/"})
if err != nil {
	return err
}
for _, info := range page.Objects {
	fmt.Println(info.Key)
}

type SignOptions

type SignOptions struct {
	// Method is the HTTP method the URL grants. Empty selects http.MethodGet;
	// http.MethodPut is the only other supported value.
	//
	// Example:
	//
	//	options := storage.SignOptions{Method: http.MethodPut, TTL: time.Minute}
	//	_, err := files.SignURL(ctx, "upload.png", options)
	//	return err
	Method string

	// TTL is how long the URL stays valid. It must be positive.
	//
	// Example:
	//
	//	options := storage.SignOptions{TTL: 15 * time.Minute}
	//	_, err := files.SignURL(ctx, "report.pdf", options)
	//	return err
	TTL time.Duration

	// ContentType binds the Content-Type of a PUT: a driver that supports it signs the
	// value into the URL, so an upload with any other Content-Type is rejected by the
	// backend. Empty leaves the type unbound. It is valid only with http.MethodPut.
	//
	// Example:
	//
	//	options := storage.SignOptions{Method: http.MethodPut, ContentType: "image/png"}
	//	return options.ValidateBinding()
	ContentType string

	// ContentLength binds the exact body size in bytes of a PUT. Zero leaves the size
	// unbound; negative values are invalid. It is valid only with http.MethodPut. Backends
	// such as S3 can presign only an exact length, not a maximum; ranges need a different
	// mechanism (S3 presigned POST policies), which this API does not offer.
	//
	// For user uploads, callers should set both ContentType and ContentLength, or the URL
	// holder chooses the type and size of the stored object.
	//
	// Example:
	//
	//	options := storage.SignOptions{Method: http.MethodPut, ContentLength: 1024}
	//	return options.ValidateBinding()
	ContentLength int64
}

SignOptions grants one GET or PUT for a positive TTL, subject to backend limits. Signed URLs are bearer credentials: share them only with authorized recipients, never through ordinary logs or telemetry.

Example:

options := storage.SignOptions{
	Method: http.MethodPut, TTL: 10 * time.Minute,
	ContentType: "image/png", ContentLength: 1024,
}
signed, err := files.SignURL(ctx, "avatars/42.png", options)
_ = signed // Return only to the authorized uploader.
return err

func (SignOptions) ValidateBinding

func (options SignOptions) ValidateBinding() error

ValidateBinding checks ContentType and ContentLength against Method: they are valid only for PUT (empty Method means GET), and ContentLength must not be negative. It returns an error wrapping ErrInvalidSignOptions. Drivers call it after the method check so every driver reports the same error. This method checks bindings only, not TTL, key syntax, or backend support.

Example:

options := storage.SignOptions{Method: http.MethodPut, ContentLength: 1024}
if err := options.ValidateBinding(); err != nil {
	return err
}
Example

ExampleSignOptions_ValidateBinding checks bindings independently of backend limits; it does not claim that a successful check validates the whole request.

package main

import (
	"errors"
	"fmt"
	"net/http"

	"arkoren.dev/core/service/storage"
)

func main() {
	options := storage.SignOptions{Method: http.MethodPut, ContentLength: 1024}
	fmt.Println(options.ValidateBinding() == nil)

	options.Method = http.MethodGet
	fmt.Println(errors.Is(options.ValidateBinding(), storage.ErrInvalidSignOptions))
}
Output:
true
true

type Storage

type Storage struct {
	// contains filtered or unexported fields
}

Storage validates portable requests before delegating to a driver. It preserves driver error identities and does not classify provider errors. It borrows the driver and adds no lifecycle ownership or background workers.

Example:

files := storage.New(memorystorage.New())
info, err := files.PutBytes(ctx, "report.txt", []byte("report"))
if err != nil {
	return err
}
fmt.Println(info.Size) // 6

func New

func New(driver Driver, configure ...BuilderFunc) *Storage

New creates a Storage that delegates to driver. Construction policies configure its telemetry: every operation (open, put, stat, delete, list, sign) emits one client span and the arkoren.storage.operations and arkoren.storage.operation.duration metrics when the telemetry is enabled, and costs nothing otherwise. Keys, prefixes, cursors and signed URLs are never recorded. The duration of open is the time to the response, and of put the time to consume the reader; neither includes reading an opened body.

Example:

files := storage.New(driver, func(store *storage.Builder) {
	store.Telemetry(app.Telemetry())
	store.Name("uploads")
})
_ = files.Driver()

Construction callbacks run directly once, synchronously in order on one fresh draft. Nil entries panic; expanded nil or empty slices omit edits. Publication detaches owned settings, not live dependencies or captures.

func (*Storage) Delete

func (storage *Storage) Delete(ctx context.Context, key string) (err error)

Delete removes key and succeeds when key is absent. On success it publishes ObjectDeleted when Builder.Events is set, also for an absent key. Cancellation and validation failures publish nothing.

Example:

if err := files.Delete(ctx, "obsolete/report.pdf"); err != nil {
	return err
}

func (*Storage) Download

func (storage *Storage) Download(w http.ResponseWriter, r *http.Request, key, filename string) error

Download writes the object at key to w as a file download: the response carries Content-Disposition: attachment so browsers save it instead of showing it. filename is the name offered to the user; empty uses the last segment of key. Handlers return the result, so a missing object becomes a 404 response. Authorize object access before calling; key validation checks syntax, not ownership or permissions.

The Content-Type comes from filename's extension, falling back to the type detected from the first 512 bytes. When the driver's Object.Contents implements io.Seeker (the memory and filesystem drivers do), Download uses http.ServeContent, which adds Range requests (206), HEAD, Last-Modified and If-Modified-Since (304). Otherwise it streams the body with Content-Length from ObjectInfo.Size; HEAD gets headers only, and Range and conditional requests are not supported. Contents is always closed.

Download sets X-Content-Type-Options: nosniff on object responses. Open and initial sniff/rewind failures return before commitment, with ErrObjectNotFound for a missing key. ServeContent retains its native HTTP error responses for invalid ranges or failed size probes. Source, seek, write, short-write, and close failures remain discoverable with errors.Is; premature EOF during a seekable transfer is io.ErrUnexpectedEOF. Cleanup errors join transfer errors. These errors can arrive after response commitment: return them for logging, never append a replacement response or automatically retry a completed download.

Example:

handler := func(w http.ResponseWriter, r *http.Request) error {
	return store.Download(w, r, "reports/2024.pdf", "Q1 report.pdf")
}
_ = handler // Register with the application's error-returning router.
Example

ExampleStorage_Download exercises the declaration's HTTP handler pattern without a server, network request, or wall-clock wait.

package main

import (
	"context"
	"fmt"
	"net/http"
	"net/http/httptest"

	"arkoren.dev/core/service/storage"
	"arkoren.dev/core/service/storage/memorystorage"
)

func main() {
	ctx := context.Background()
	files := storage.New(memorystorage.New())

	if _, err := files.PutBytes(ctx, "report.txt", []byte("report")); err != nil {
		fmt.Println(err)

		return
	}

	handler := func(writer http.ResponseWriter, request *http.Request) error {
		return files.Download(writer, request, "report.txt", "annual.txt")
	}

	request := httptest.NewRequest(http.MethodGet, "/report", nil)
	request.Header.Set("Range", "bytes=2-3")

	response := httptest.NewRecorder()

	if err := handler(response, request); err != nil {
		fmt.Println(err)

		return
	}

	fmt.Println(response.Code, response.Body.String())
}
Output:
206 po

func (*Storage) Driver

func (storage *Storage) Driver() Driver

Driver returns the shared underlying driver. Direct calls bypass facade instrumentation and notifications, but retain the driver's validation rules.

Example:

files := storage.New(memorystorage.New())
info, err := files.Driver().Stat(ctx, "report.pdf")
_ = info
return err

func (*Storage) DriverAs

func (storage *Storage) DriverAs[T any]() (T, bool)

DriverAs returns the first driver of type T in the chain of the underlying driver. A driver that wraps another exposes it with an Unwrap() Driver method, so wrappers such as prefixing or tracing drivers do not hide the concrete driver. The result is shared, not copied; false means no matching driver was found.

Example:

files := storage.New(memorystorage.New())
if driver, ok := files.DriverAs[*memorystorage.Driver](); ok {
	return driver.Delete(ctx, "obsolete.pdf")
}

func (*Storage) Inline

func (storage *Storage) Inline(w http.ResponseWriter, r *http.Request, key string) error

Inline writes the object at key to w for display in the browser (Content-Disposition: inline), such as an image in a page. It behaves like Storage.Download and adds protections for content that could run in your origin: every object response carries Content-Security-Policy: sandbox, and only an allow-list of types is shown inline: raster images (PNG, JPEG, GIF, WebP, AVIF, BMP, ICO), application/pdf, audio/*, video/* and text/plain. Every other type (HTML, SVG, XML, JavaScript, CSS, WebAssembly, JSON, and anything unknown) is forced to attachment disposition, because a browser could run it or load it with <script src> or <link> from your origin. The name offered is the last segment of key. Authorize object access before calling; the sandbox is not access control.

Example:

handler := func(w http.ResponseWriter, r *http.Request) error {
	return store.Inline(w, r, "avatars/"+r.PathValue("id")+".png")
}
_ = handler // Authorize access before exposing the object.

func (*Storage) List

func (storage *Storage) List(ctx context.Context, options ListOptions) (page Page, err error)

List returns a page for options. Cursors are opaque and valid only for the same driver and prefix. Objects remain lexically ordered within a page, but concurrent mutations do not provide a multi-page snapshot. Cancellation precedes prefix validation.

Example:

page, err := files.List(ctx, storage.ListOptions{Prefix: "reports/", Limit: 20})
if err != nil {
	return err
}
for _, info := range page.Objects {
	fmt.Println(info.Key)
}

func (*Storage) Open

func (storage *Storage) Open(ctx context.Context, key string) (object Object, err error)

Open returns an object body that the caller must close. Its ObjectInfo describes that body. Missing-object error identity is supplied by the driver and preserved unchanged. Cancellation precedes key validation. The facade never closes a returned body.

Example:

object, err := files.Open(ctx, "report.pdf")
if err != nil {
	return err
}
_, err = io.Copy(destination, object.Contents)
return errors.Join(err, object.Contents.Close())
Example

ExampleStorage_Open executes the README's owned-body recipe entirely in memory, checking both read and close outcomes before using the bytes.

package main

import (
	"context"
	"errors"
	"fmt"
	"io"

	"arkoren.dev/core/service/storage"
	"arkoren.dev/core/service/storage/memorystorage"
)

func main() {
	ctx := context.Background()
	files := storage.New(memorystorage.New())

	if _, err := files.PutBytes(ctx, "reports/a.txt", []byte("report")); err != nil {
		fmt.Println(err)

		return
	}

	object, err := files.Open(ctx, "reports/a.txt")

	if err != nil {
		fmt.Println(err)

		return
	}

	contents, err := io.ReadAll(object.Contents)

	err = errors.Join(err, object.Contents.Close())

	if err != nil {
		fmt.Println(err)

		return
	}

	fmt.Println(string(contents))
}
Output:
report

func (*Storage) Put

func (storage *Storage) Put(ctx context.Context, key string, contents io.Reader) (info ObjectInfo, err error)

Put consumes contents once and replaces key only when the complete object is current. On success it publishes ObjectWritten when Builder.Events is set. It does not close contents; the caller retains the reader's lifetime.

Example:

info, err := files.Put(ctx, "report.txt", strings.NewReader("report"))
if err != nil {
	return err
}
fmt.Println(info.Size) // 6

func (*Storage) PutBytes

func (storage *Storage) PutBytes(ctx context.Context, key string, contents []byte) (ObjectInfo, error)

PutBytes replaces key with contents without forcing drivers to receive an allocated object. The bytes are consumed synchronously; do not mutate them during the call.

Example:

info, err := files.PutBytes(ctx, "report.txt", []byte("report"))
_ = info
return err

func (*Storage) PutFile

func (storage *Storage) PutFile(ctx context.Context, dir string, file *multipart.FileHeader) (ObjectInfo, error)

PutFile stores an uploaded file under dir with a generated name and returns the stored object's metadata. The key is dir + "/" + name, or just name when dir is empty; a non-empty dir must satisfy ValidateKey (so no trailing slash), otherwise the error wraps ErrInvalidKey and nothing is read.

The name is 26 random lowercase base32 characters, making collisions extremely unlikely; the underlying Put still replaces a colliding key. The client's filename never reaches the key. The extension comes from content detected in the first 512 bytes with http.DetectContentType, not the client's filename or Content-Type header. HTML named photo.png can therefore be stored as .html; sniffing identifies a format, not a safe payload. Serve untrusted files through Download or Inline's restrictive presentation, not an unrestricted static route on your application origin. Unknown or application/octet-stream content gets no extension. Keep the original file.Filename in your own database if needed. file must be non-nil; validate the form field's presence before calling.

The upload is passed to the driver as the file's seekable multipart.File, so a driver that can use a known length, such as the AWS driver, streams from disk. A non-seekable reader handed straight to Storage.Put is buffered in memory there instead.

Cancellation is checked before opening the multipart source. The helper closes its read-only source handle on every path, best-effort: a close failure does not undo an already successful storage write and is not a reason to repeat it.

Example:

With file obtained from an already validated multipart form:

info, err := store.PutFile(ctx, "avatars", file)
if err != nil {
	return err
}
fmt.Println(info.Size)

func (*Storage) PutFileAs

func (storage *Storage) PutFileAs(ctx context.Context, key string, file *multipart.FileHeader) (ObjectInfo, error)

PutFileAs stores an uploaded file at exactly key, replacing any existing object, and returns the stored object's metadata. Use it when the key is derived from your own data, such as "avatars/<user id>.png". The key is not derived from the upload, so choose the extension yourself and do not take it from file.Filename; see Storage.PutFile for why.

Like PutFile it passes the file's seekable multipart.File to the driver and closes that source best-effort. Cancellation is checked before opening it; cleanup cannot undo a successful write and never changes its reported outcome. file must be a non-nil header from a validated multipart form.

Example:

With a validated non-nil file header:

info, err := store.PutFileAs(ctx, "avatars/"+userID+".png", file)
if err != nil {
	return err
}
fmt.Println(info.Key)

func (*Storage) Sign

func (storage *Storage) Sign(ctx context.Context, key string, options SignOptions) (string, error)

Sign is Storage.SignURL returning the URL as a string.

Example:

link, err := files.Sign(ctx, "reports/a.pdf", storage.SignOptions{TTL: time.Hour})
if err != nil {
	return err
}
http.Redirect(writer, request, link, http.StatusSeeOther)

func (*Storage) SignURL

func (storage *Storage) SignURL(ctx context.Context, key string, options SignOptions) (signed *url.URL, err error)

SignURL returns the driver's signed URL. An empty options.Method selects GET. The facade checks cancellation, key syntax, and a positive TTL before calling the driver. The driver checks its maximum TTL before method and binding rules, preserving the complete validation order documented on Driver.SignURL. Drivers without signing support return an error wrapping ErrUnsupported only after validating the request. Signing neither opens nor writes the object. Authorize the caller before granting or returning this bearer credential.

Example:

signed, err := files.SignURL(ctx, "reports/a.pdf", storage.SignOptions{TTL: 15 * time.Minute})
if err != nil {
	return err
}
http.Redirect(writer, request, signed.String(), http.StatusSeeOther)
Example

ExampleStorage_SignURL demonstrates unsupported signing without changing capability behavior or needing a provider credential.

package main

import (
	"context"
	"errors"
	"fmt"
	"time"

	"arkoren.dev/core/service/storage"
	"arkoren.dev/core/service/storage/memorystorage"
)

func main() {
	files := storage.New(memorystorage.New())

	_, err := files.SignURL(context.Background(), "report.pdf", storage.SignOptions{TTL: time.Minute})
	fmt.Println(errors.Is(err, storage.ErrUnsupported))
}
Output:
true

func (*Storage) Stat

func (storage *Storage) Stat(ctx context.Context, key string) (info ObjectInfo, err error)

Stat returns object metadata. Missing-object error identity is supplied by the driver and preserved unchanged.

Example:

info, err := files.Stat(ctx, "report.pdf")
if err != nil {
	return err
}
fmt.Println(info.Size)

func (*Storage) Walk

func (storage *Storage) Walk(ctx context.Context, options ListOptions) collection.TryLazySlice[ObjectInfo]

Walk traverses object metadata in the driver's lexical source order, starting with options.Cursor and retaining options.Prefix and options.Limit for every page. Limit bounds each page, not the total number of objects; nonpositive values retain the driver default. Use Take to bound the number consumed.

Construction performs no listing or validation. Each traversal calls Storage.List afresh with the captured context and original options, so validation and telemetry apply per page. Reusing a walk does not refresh an expired context or provide a snapshot: concurrent storage changes can alter objects observed across pages or traversals. As with List, callers must supply a non-nil context.

Walk follows opaque NextCursor values, including after empty or short pages, until the cursor is empty. It checks cancellation before page requests and between buffered objects. A cancellation or page error is yielded once with a zero ObjectInfo, then traversal ends without retry. Items returns the successful prefix alongside that error; callers must not treat it as a complete listing.

A false yield stops immediately without another page request. This bounds future facade calls, not work already done preparing a page: memory and filesystem drivers can scan and sort matching keys, and S3 can read ahead for continuation.

Example:

files := storage.New(memorystorage.New())
objects, err := files.Walk(ctx, storage.ListOptions{Prefix: "reports/", Limit: 20}).Take(5).Items()
if err != nil {
	return err // objects contains only the successfully traversed prefix.
}
for _, info := range objects {
	fmt.Println(info.Key)
}
Example

ExampleStorage_Walk demonstrates a collection item limit independent of the driver's page size. It uses memory storage and requires no external service.

package main

import (
	"context"
	"fmt"

	"arkoren.dev/core/service/storage"
	"arkoren.dev/core/service/storage/memorystorage"
)

func main() {
	ctx := context.Background()
	files := storage.New(memorystorage.New())

	for _, key := range []string{"reports/c", "reports/a", "reports/b"} {
		if _, err := files.PutBytes(ctx, key, nil); err != nil {
			fmt.Println(err)

			return
		}
	}

	objects, err := files.Walk(ctx, storage.ListOptions{Prefix: "reports/", Limit: 1}).Take(2).Items()

	if err != nil {
		fmt.Println(err)

		return
	}

	for _, info := range objects {
		fmt.Println(info.Key)
	}
}
Output:
reports/a
reports/b

Directories

Path Synopsis
internal
transfer
Package transfer handles HTTP presentation and upload sniffing for storage.
Package transfer handles HTTP presentation and upload sniffing for storage.
Package memorystorage provides a concurrent in-memory storage driver.
Package memorystorage provides a concurrent in-memory storage driver.
Package storagetest provides a conformance suite for storage.Driver implementations, modeled on testing/slogtest and testing/fstest.
Package storagetest provides a conformance suite for storage.Driver implementations, modeled on testing/slogtest and testing/fstest.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL