Documentation
¶
Overview ¶
Package storage provides a durable, provider-agnostic file storage interface for Nucleus applications. It abstracts S3, GCS, Azure Blob, and local filesystem behind a single stable API designed to last through v1.x.
The interface is streaming-native (io.Reader/io.ReadCloser) so large files never need to be held in memory. Multi-tenant applications automatically receive prefix isolation (tenant_a/uploads/file.pdf).
Provider selection is configuration-driven: the application code never changes when switching from local dev to S3 in production.
Index ¶
- func EscapeURLPath(key string) string
- func NormalizeKey(key string) string
- func Register(name string, factory Factory) error
- func Registered() []string
- func Unregister(name string)
- func ValidateKey(key string) error
- func ValidateKeyPrefix(prefix string) error
- type AzureConfig
- type CircuitBreakerConfig
- type CleanupConfig
- type Config
- type CredentialSource
- type ErrInvalidKey
- type ErrNotFound
- type Factory
- type GCSConfig
- type ListOptions
- type ListResult
- type LocalConfig
- type ObjectInfo
- type ProviderType
- type PutOptions
- type S3Config
- type Store
- type URLConfig
- type Visibility
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func EscapeURLPath ¶
func NormalizeKey ¶
func Register ¶
RegisterProvider makes a storage backend selectable by name from configuration (`storage.provider`).
Call it from an init function in the package that implements the backend, then import that package for its side effects — the same shape database/sql drivers use, and the same one this framework already uses for mail providers and quark uses for SQL dialects:
package cephstore
func init() {
storage.RegisterProvider("ceph", New)
}
Registering a name that is already taken is an ERROR rather than a silent replacement: two packages claiming "s3" would otherwise make the effective backend depend on import order, which is the kind of bug that only shows up in someone else's deployment.
func Registered ¶
func Registered() []string
RegisteredProviders returns every selectable provider name, sorted. Built-ins are included, because from the outside they are not special.
func Unregister ¶
func Unregister(name string)
Unregister removes a registered provider.
It exists for tests that register a fake and must not leak it into the next one. Production code has no reason to call it.
func ValidateKey ¶
func ValidateKeyPrefix ¶
Types ¶
type AzureConfig ¶
type AzureConfig struct {
// AccountName is the Azure storage account name.
// env_var: AZURE_ACCOUNT_NAME
AccountName CredentialSource `koanf:"account_name"`
// AccountKey credential source.
// env_var: AZURE_STORAGE_KEY
// secret_manager: env:AZURE_STORAGE_KEY
AccountKey CredentialSource `koanf:"account_key"`
// Container is the default (private) container name.
Container string `koanf:"container"`
// PublicContainer is the container for public objects.
PublicContainer string `koanf:"public_container"`
}
AzureConfig configures Azure Blob Storage.
type CircuitBreakerConfig ¶
type CircuitBreakerConfig struct {
// Enabled turns on circuit-breaker wrapping for the returned Store.
Enabled bool `koanf:"enabled"`
// FailureThreshold is the number of consecutive failures required
// to trip the breaker open. Non-positive falls back to pkg/circuit's
// default (1).
FailureThreshold int `koanf:"failure_threshold"`
// Cooldown is the duration the breaker stays open before admitting
// half-open probes. Non-positive falls back to pkg/circuit's
// default (30s).
Cooldown time.Duration `koanf:"cooldown"`
// HalfOpenMaxConcurrent caps in-flight probes in the half-open
// state. Non-positive falls back to pkg/circuit's default (1).
HalfOpenMaxConcurrent int `koanf:"half_open_max_concurrent"`
}
CircuitBreakerConfig configures the optional circuit breaker that wraps remote storage operations. Zero values fall back to pkg/circuit defaults when Enabled is true; pkg/app applies its own framework defaults before constructing the breaker.
The breaker wraps the network-touching operations of Store (Put, Get, Delete, Exists, List, Copy, SignedURL). PublicURL is pass-through because it is pure string composition. ErrNotFound is treated as a success for the breaker — a missing object is a normal outcome, not a dependency failure.
type CleanupConfig ¶
type CleanupConfig struct {
// Enabled turns on background cleanup.
Enabled bool `koanf:"enabled"`
// Interval is how often to run cleanup (e.g. "1h").
Interval string `koanf:"interval"`
// Prefix is the key prefix for temporary objects (default: "_tmp/").
Prefix string `koanf:"prefix"`
// MaxAge is the maximum age before an object is deleted (e.g. "24h").
MaxAge string `koanf:"max_age"`
}
CleanupConfig configures automatic cleanup of temporary objects.
type Config ¶
type Config struct {
// ProviderConfig carries the `storage.<provider>.*` subtree for a
// provider this package does not know about. The framework fills it;
// a third-party factory reads it with BindProvider.
//
// It is a decoded map rather than a typed field because the framework
// cannot know the shape of a backend it has never seen — which is the
// whole point of the registry. The provider owns the shape and
// declares it in its own struct.
ProviderConfig map[string]any `koanf:"-" json:"-" yaml:"-"`
// Default visibility for new objects (private|public).
DefaultVisibility Visibility `koanf:"default"`
// Provider selects the storage backend (s3|gcs|azure|local).
Provider ProviderType `koanf:"provider"`
// PublicPaths maps public URL paths to storage key prefixes.
// Example: "/media" -> "storage/public/media/"
// Requests to /media/* are served from keys with that prefix.
PublicPaths map[string]string `koanf:"public_paths"`
// PublicURLBase is the base URL for public objects (CDN or direct provider).
// Example: "https://cdn.example.com"
PublicURLBase string `koanf:"public_url_base"`
// S3 configuration
S3 S3Config `koanf:"s3"`
// GCS Configuration
GCS GCSConfig `koanf:"gcs"`
// Azure configuration
Azure AzureConfig `koanf:"azure"`
// Local configuration (development only)
Local LocalConfig `koanf:"local"`
// Cleanup config for temporary objects
Cleanup CleanupConfig `koanf:"cleanup"`
// CircuitBreaker, when Enabled, wraps remote provider operations
// (Put/Get/Delete/Exists/List/SignedURL/Copy) with a pkg/circuit
// breaker. The local provider is never wrapped — filesystem failures
// are not the kind of outage circuit breakers are designed to
// short-circuit. PublicURL is also not wrapped (pure string
// composition).
CircuitBreaker CircuitBreakerConfig `koanf:"circuit_breaker"`
}
Config holds the complete storage configuration.
func DefaultConfig ¶
func DefaultConfig() Config
DefaultConfig returns a sensible default configuration for local development.
func (Config) BindProvider ¶
BindProvider decodes the provider's own configuration subtree into dst, applying `default:` tags to fields the file left unset.
It is what makes a third-party backend a first-class citizen rather than one that has to invent its own configuration channel:
func New(cfg storage.Config) (storage.Store, error) {
var c struct {
Endpoint string `koanf:"endpoint" validate:"required"`
Pool int `koanf:"pool" default:"8"`
}
if err := cfg.BindProvider(&c); err != nil {
return nil, err
}
…
}
A key the destination struct does not declare is an ERROR, not a silently ignored line. Provider configuration is exactly the place where a typo would otherwise sit unnoticed until the day the setting mattered.
type CredentialSource ¶
type CredentialSource struct {
// Value is the literal credential value (for testing or non-sensitive configs).
Value string `koanf:"value"`
// EnvVar reads the credential from an environment variable.
// This is the primary method for production: orchestrators inject
// secrets from Secret Manager into env vars before starting the container.
// Example: "AWS_SECRET_ACCESS_KEY"
EnvVar string `koanf:"env_var"`
// File reads the credential from a file path.
// Used for:
// - GCS service account JSON key mounted as volume
// - Kubernetes secrets mounted as files
// - Azure managed identity token file
// Example: "/etc/secrets/gcs-sa.json"
File string `koanf:"file"`
// SecretManager currently supports only env: references, for example
// "env:MY_SECRET_KEY". Cloud Secret Manager SDK lookups are deliberately
// not implemented yet; inject cloud-managed secrets into env vars or files.
// Resolution happens at startup. The secret value is read once and cached.
SecretManager string `koanf:"secret_manager"`
}
CredentialSource describes where to find a credential value. Supports multiple injection methods used in production environments.
func (*CredentialSource) Resolve ¶
func (cs *CredentialSource) Resolve() (string, error)
Resolve returns the credential value from the configured source. Priority: Value > EnvVar > File > SecretManager. Only one source should be configured; the first non-empty wins.
type ErrInvalidKey ¶
type ErrInvalidKey string
ErrInvalidKey is returned when a key contains invalid characters.
func (ErrInvalidKey) Error ¶
func (e ErrInvalidKey) Error() string
type ErrNotFound ¶
type ErrNotFound string
ErrNotFound is returned when a key does not exist.
func (ErrNotFound) Error ¶
func (e ErrNotFound) Error() string
type Factory ¶
ProviderFactory builds a Store from the resolved storage configuration.
A provider reads the sub-config it owns (`cfg.S3`, `cfg.Local`, …) or, for a third-party backend, whatever it needs from the shared fields. It returns a Store; everything the framework layers on top — the circuit breaker, tenant prefixing, the public-URL mapper — is applied by New around whatever comes back, so a provider never has to reimplement any of it.
type GCSConfig ¶
type GCSConfig struct {
// Bucket is the default (private) bucket name.
Bucket string `koanf:"bucket"`
// CredentialsSource for the GCS service account.
// When empty, uses Application Default Credentials (ADC).
//
// Examples for Cloud Run / GKE:
// env_var: GOOGLE_APPLICATION_CREDENTIALS # Path injected by Secret Manager
// file: /etc/secrets/gcs-sa.json # Volume-mounted secret
//
// Examples for workload identity (no credentials needed):
// (leave empty — ADC uses the GKE service account)
CredentialsSource CredentialSource `koanf:"credentials"`
// PublicBucket is an optional separate bucket for public objects.
PublicBucket string `koanf:"public_bucket"`
}
GCSConfig configures Google Cloud Storage.
type ListOptions ¶
type ListOptions struct {
// Prefix filters objects by key prefix (directory-like listing).
Prefix string
// Delimiter causes keys containing the delimiter after the prefix
// to be rolled up into "common prefixes" (simulating directories).
Delimiter string
// Limit caps the number of results. 0 = provider default (usually 1000).
Limit int
// Marker starts listing after this key (for pagination).
Marker string
}
ListOptions configures object listing.
type ListResult ¶
type ListResult struct {
Objects []ObjectInfo `json:"objects"`
CommonPrefixes []string `json:"common_prefixes,omitempty"`
NextMarker string `json:"next_marker,omitempty"`
Truncated bool `json:"truncated"`
}
ListResult is the response from List().
type LocalConfig ¶
type LocalConfig struct {
// Path is the root directory for all stored files.
Path string `koanf:"path"`
}
LocalConfig configures local filesystem storage (development only).
type ObjectInfo ¶
type ObjectInfo struct {
Key string `json:"key"`
Size int64 `json:"size"`
ContentType string `json:"content_type"`
Visibility Visibility `json:"visibility"`
Metadata map[string]string `json:"metadata,omitempty"`
UpdatedAt time.Time `json:"updated_at"`
}
ObjectInfo describes a stored object.
type ProviderType ¶
type ProviderType string
ProviderType identifies the storage backend.
const ( ProviderS3 ProviderType = "s3" ProviderGCS ProviderType = "gcs" ProviderAzure ProviderType = "azure" ProviderLocal ProviderType = "local" )
type PutOptions ¶
type PutOptions struct {
// Visibility controls public access. Defaults to Private.
Visibility Visibility
// ContentType is the MIME type (e.g. "image/png", "application/pdf").
// Auto-detected from key extension when empty.
ContentType string
// Metadata stores custom key-value pairs on the provider.
Metadata map[string]string
// TenantPrefix overrides the automatic tenant prefix.
// Empty means auto-detect from context. Set to "" explicitly to disable prefixing.
TenantPrefix string
}
PutOptions configures how an object is stored.
type S3Config ¶
type S3Config struct {
// Endpoint is the S3 API endpoint. Empty = AWS S3.
// For MinIO: "http://minio:9000"
// For Cloudflare R2: "https://<account>.r2.cloudflarestorage.com"
Endpoint string `koanf:"endpoint"`
// Bucket is the default (private) bucket name.
Bucket string `koanf:"bucket"`
// Region is the AWS region.
Region string `koanf:"region"`
// AccessKeyID credential source.
// Examples:
// env_var: AWS_ACCESS_KEY_ID # From environment variable
// file: /etc/secrets/aws-access-key # From mounted secret file
// value: AKIA... # Direct (not recommended for production)
AccessKeyID CredentialSource `koanf:"access_key_id"`
// SecretAccessKey credential source.
// Examples:
// env_var: AWS_SECRET_ACCESS_KEY
// file: /etc/secrets/aws-secret-key
SecretAccessKey CredentialSource `koanf:"secret_access_key"`
// SessionToken credential source (for temporary STS credentials).
// env_var: AWS_SESSION_TOKEN
SessionToken CredentialSource `koanf:"session_token"`
// UsePathStyle forces path-style URLs instead of virtual-hosted style.
// Required for MinIO and some other S3-compatible providers.
UsePathStyle bool `koanf:"use_path_style"`
// PublicBucket is an optional separate bucket for public objects.
// When set, public objects are stored here instead of Bucket.
PublicBucket string `koanf:"public_bucket"`
// CreateBucketIfMissing provisions Bucket (and PublicBucket, when set)
// at construction time if they do not exist yet, using Region for the
// bucket location. Opt-in (QCD-FW-2): the default refuses to create
// infrastructure against a production object store — but note that a
// missing bucket now fails the constructor LOUDLY either way, instead
// of booting green and failing on the first Put.
CreateBucketIfMissing bool `koanf:"create_bucket_if_missing"`
}
S3Config configures Amazon S3 or any S3-compatible provider (MinIO, R2, etc.).
type Store ¶
type Store interface {
// Put uploads a file from an io.Reader. The reader is consumed entirely.
// Key is the logical path (e.g. "tenant_a/uploads/image.png").
// Returns the final storage key (with tenant prefix applied) and object info.
Put(ctx context.Context, key string, reader io.Reader, opts PutOptions) (ObjectInfo, error)
// Get retrieves a file by key. Returns an io.ReadCloser that MUST be closed.
// Returns ErrNotFound if the key does not exist.
Get(ctx context.Context, key string) (io.ReadCloser, ObjectInfo, error)
// Delete removes an object by key. Idempotent: no error if key doesn't exist.
Delete(ctx context.Context, key string) error
// Exists checks if a key exists.
Exists(ctx context.Context, key string) (bool, error)
// List returns objects with the given prefix.
List(ctx context.Context, opts ListOptions) (ListResult, error)
// PublicURL returns a publicly accessible URL for a key.
// Returns empty string if the object is private or the provider
// doesn't support public URLs.
PublicURL(ctx context.Context, key string, opts URLConfig) (string, error)
// SignedURL returns a time-limited URL for accessing a private object.
// The URL grants direct access to the object without authentication.
SignedURL(ctx context.Context, key string, expires time.Duration, opts URLConfig) (string, error)
// Copy copies an object from srcKey to dstKey (within the same bucket/container).
Copy(ctx context.Context, srcKey, dstKey string) (ObjectInfo, error)
// Close releases any resources held by the store (connections, background goroutines).
Close() error
}
Store is the durable interface for file storage in Nucleus. All implementations (S3, GCS, Azure, local) must satisfy this interface. It is intentionally minimal: add provider-specific features through type assertions when absolutely necessary.
type URLConfig ¶
type URLConfig struct {
// Expires sets the URL validity duration. Only meaningful for SignedURL.
Expires time.Duration
// ContentType overrides the Content-Type header for the URL response.
ContentType string
// Disposition sets Content-Disposition header ("inline" or "attachment").
Disposition string
}
URLConfig configures URL generation.
type Visibility ¶
type Visibility string
Visibility controls whether an object is publicly accessible or requires a signed URL (or app-layer authentication) to access.
const ( // Private objects are not directly accessible via URL. // Access requires SignedURL() or serving through the app layer. Private Visibility = "private" // Public objects have a direct, unauthenticated URL. Public Visibility = "public" )