provider

package
v1.28.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 12, 2026 License: Apache-2.0 Imports: 11 Imported by: 0

Documentation

Overview

Package storage provides a durable, provider-agnostic file storage interface for Nucleus applications. It abstracts S3, GCS, Azure Blob, and local filesystem behind a single stable API designed to last through v1.x.

The interface is streaming-native (io.Reader/io.ReadCloser) so large files never need to be held in memory. Multi-tenant applications automatically receive prefix isolation (tenant_a/uploads/file.pdf).

Provider selection is configuration-driven: the application code never changes when switching from local dev to S3 in production.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func EscapeURLPath

func EscapeURLPath(key string) string

func NormalizeKey

func NormalizeKey(key string) string

func Register

func Register(name string, factory Factory) error

RegisterProvider makes a storage backend selectable by name from configuration (`storage.provider`).

Call it from an init function in the package that implements the backend, then import that package for its side effects — the same shape database/sql drivers use, and the same one this framework already uses for mail providers and quark uses for SQL dialects:

package cephstore

func init() {
    storage.RegisterProvider("ceph", New)
}

Registering a name that is already taken is an ERROR rather than a silent replacement: two packages claiming "s3" would otherwise make the effective backend depend on import order, which is the kind of bug that only shows up in someone else's deployment.

func Registered

func Registered() []string

RegisteredProviders returns every selectable provider name, sorted. Built-ins are included, because from the outside they are not special.

func Unregister

func Unregister(name string)

Unregister removes a registered provider.

It exists for tests that register a fake and must not leak it into the next one. Production code has no reason to call it.

func ValidateKey

func ValidateKey(key string) error

func ValidateKeyPrefix

func ValidateKeyPrefix(prefix string) error

Types

type AzureConfig

type AzureConfig struct {
	// AccountName is the Azure storage account name.
	//   env_var: AZURE_ACCOUNT_NAME
	AccountName CredentialSource `koanf:"account_name"`

	// AccountKey credential source.
	//   env_var: AZURE_STORAGE_KEY
	//   secret_manager: env:AZURE_STORAGE_KEY
	AccountKey CredentialSource `koanf:"account_key"`

	// Container is the default (private) container name.
	Container string `koanf:"container"`

	// PublicContainer is the container for public objects.
	PublicContainer string `koanf:"public_container"`
}

AzureConfig configures Azure Blob Storage.

type CircuitBreakerConfig

type CircuitBreakerConfig struct {
	// Enabled turns on circuit-breaker wrapping for the returned Store.
	Enabled bool `koanf:"enabled"`

	// FailureThreshold is the number of consecutive failures required
	// to trip the breaker open. Non-positive falls back to pkg/circuit's
	// default (1).
	FailureThreshold int `koanf:"failure_threshold"`

	// Cooldown is the duration the breaker stays open before admitting
	// half-open probes. Non-positive falls back to pkg/circuit's
	// default (30s).
	Cooldown time.Duration `koanf:"cooldown"`

	// HalfOpenMaxConcurrent caps in-flight probes in the half-open
	// state. Non-positive falls back to pkg/circuit's default (1).
	HalfOpenMaxConcurrent int `koanf:"half_open_max_concurrent"`
}

CircuitBreakerConfig configures the optional circuit breaker that wraps remote storage operations. Zero values fall back to pkg/circuit defaults when Enabled is true; pkg/app applies its own framework defaults before constructing the breaker.

The breaker wraps the network-touching operations of Store (Put, Get, Delete, Exists, List, Copy, SignedURL). PublicURL is pass-through because it is pure string composition. ErrNotFound is treated as a success for the breaker — a missing object is a normal outcome, not a dependency failure.

type CleanupConfig

type CleanupConfig struct {
	// Enabled turns on background cleanup.
	Enabled bool `koanf:"enabled"`

	// Interval is how often to run cleanup (e.g. "1h").
	Interval string `koanf:"interval"`

	// Prefix is the key prefix for temporary objects (default: "_tmp/").
	Prefix string `koanf:"prefix"`

	// MaxAge is the maximum age before an object is deleted (e.g. "24h").
	MaxAge string `koanf:"max_age"`
}

CleanupConfig configures automatic cleanup of temporary objects.

type Config

type Config struct {

	// ProviderConfig carries the `storage.<provider>.*` subtree for a
	// provider this package does not know about. The framework fills it;
	// a third-party factory reads it with BindProvider.
	//
	// It is a decoded map rather than a typed field because the framework
	// cannot know the shape of a backend it has never seen — which is the
	// whole point of the registry. The provider owns the shape and
	// declares it in its own struct.
	ProviderConfig map[string]any `koanf:"-" json:"-" yaml:"-"`
	// Default visibility for new objects (private|public).
	DefaultVisibility Visibility `koanf:"default"`

	// Provider selects the storage backend (s3|gcs|azure|local).
	Provider ProviderType `koanf:"provider"`

	// PublicPaths maps public URL paths to storage key prefixes.
	// Example: "/media" -> "storage/public/media/"
	// Requests to /media/* are served from keys with that prefix.
	PublicPaths map[string]string `koanf:"public_paths"`

	// PublicURLBase is the base URL for public objects (CDN or direct provider).
	// Example: "https://cdn.example.com"
	PublicURLBase string `koanf:"public_url_base"`

	// S3 configuration
	S3 S3Config `koanf:"s3"`

	// GCS Configuration
	GCS GCSConfig `koanf:"gcs"`

	// Azure configuration
	Azure AzureConfig `koanf:"azure"`

	// Local configuration (development only)
	Local LocalConfig `koanf:"local"`

	// Cleanup config for temporary objects
	Cleanup CleanupConfig `koanf:"cleanup"`

	// CircuitBreaker, when Enabled, wraps remote provider operations
	// (Put/Get/Delete/Exists/List/SignedURL/Copy) with a pkg/circuit
	// breaker. The local provider is never wrapped — filesystem failures
	// are not the kind of outage circuit breakers are designed to
	// short-circuit. PublicURL is also not wrapped (pure string
	// composition).
	CircuitBreaker CircuitBreakerConfig `koanf:"circuit_breaker"`
}

Config holds the complete storage configuration.

func DefaultConfig

func DefaultConfig() Config

DefaultConfig returns a sensible default configuration for local development.

func (Config) BindProvider

func (c Config) BindProvider(dst any) error

BindProvider decodes the provider's own configuration subtree into dst, applying `default:` tags to fields the file left unset.

It is what makes a third-party backend a first-class citizen rather than one that has to invent its own configuration channel:

func New(cfg storage.Config) (storage.Store, error) {
    var c struct {
        Endpoint string `koanf:"endpoint" validate:"required"`
        Pool     int    `koanf:"pool" default:"8"`
    }
    if err := cfg.BindProvider(&c); err != nil {
        return nil, err
    }
    …
}

A key the destination struct does not declare is an ERROR, not a silently ignored line. Provider configuration is exactly the place where a typo would otherwise sit unnoticed until the day the setting mattered.

func (*Config) Validate

func (c *Config) Validate() error

Validate checks the configuration for required fields.

type CredentialSource

type CredentialSource struct {
	// Value is the literal credential value (for testing or non-sensitive configs).
	Value string `koanf:"value"`

	// EnvVar reads the credential from an environment variable.
	// This is the primary method for production: orchestrators inject
	// secrets from Secret Manager into env vars before starting the container.
	// Example: "AWS_SECRET_ACCESS_KEY"
	EnvVar string `koanf:"env_var"`

	// File reads the credential from a file path.
	// Used for:
	// - GCS service account JSON key mounted as volume
	// - Kubernetes secrets mounted as files
	// - Azure managed identity token file
	// Example: "/etc/secrets/gcs-sa.json"
	File string `koanf:"file"`

	// SecretManager currently supports only env: references, for example
	// "env:MY_SECRET_KEY". Cloud Secret Manager SDK lookups are deliberately
	// not implemented yet; inject cloud-managed secrets into env vars or files.
	// Resolution happens at startup. The secret value is read once and cached.
	SecretManager string `koanf:"secret_manager"`
}

CredentialSource describes where to find a credential value. Supports multiple injection methods used in production environments.

func (*CredentialSource) Resolve

func (cs *CredentialSource) Resolve() (string, error)

Resolve returns the credential value from the configured source. Priority: Value > EnvVar > File > SecretManager. Only one source should be configured; the first non-empty wins.

type ErrInvalidKey

type ErrInvalidKey string

ErrInvalidKey is returned when a key contains invalid characters.

func (ErrInvalidKey) Error

func (e ErrInvalidKey) Error() string

type ErrNotFound

type ErrNotFound string

ErrNotFound is returned when a key does not exist.

func (ErrNotFound) Error

func (e ErrNotFound) Error() string

type Factory

type Factory func(cfg Config) (Store, error)

ProviderFactory builds a Store from the resolved storage configuration.

A provider reads the sub-config it owns (`cfg.S3`, `cfg.Local`, …) or, for a third-party backend, whatever it needs from the shared fields. It returns a Store; everything the framework layers on top — the circuit breaker, tenant prefixing, the public-URL mapper — is applied by New around whatever comes back, so a provider never has to reimplement any of it.

func Lookup

func Lookup(name string) (Factory, bool)

Lookup returns the factory registered under name.

type GCSConfig

type GCSConfig struct {
	// Bucket is the default (private) bucket name.
	Bucket string `koanf:"bucket"`

	// CredentialsSource for the GCS service account.
	// When empty, uses Application Default Credentials (ADC).
	//
	// Examples for Cloud Run / GKE:
	//   env_var: GOOGLE_APPLICATION_CREDENTIALS  # Path injected by Secret Manager
	//   file: /etc/secrets/gcs-sa.json           # Volume-mounted secret
	//
	// Examples for workload identity (no credentials needed):
	//   (leave empty — ADC uses the GKE service account)
	CredentialsSource CredentialSource `koanf:"credentials"`

	// PublicBucket is an optional separate bucket for public objects.
	PublicBucket string `koanf:"public_bucket"`
}

GCSConfig configures Google Cloud Storage.

type ListOptions

type ListOptions struct {
	// Prefix filters objects by key prefix (directory-like listing).
	Prefix string

	// Delimiter causes keys containing the delimiter after the prefix
	// to be rolled up into "common prefixes" (simulating directories).
	Delimiter string

	// Limit caps the number of results. 0 = provider default (usually 1000).
	Limit int

	// Marker starts listing after this key (for pagination).
	Marker string
}

ListOptions configures object listing.

type ListResult

type ListResult struct {
	Objects        []ObjectInfo `json:"objects"`
	CommonPrefixes []string     `json:"common_prefixes,omitempty"`
	NextMarker     string       `json:"next_marker,omitempty"`
	Truncated      bool         `json:"truncated"`
}

ListResult is the response from List().

type LocalConfig

type LocalConfig struct {
	// Path is the root directory for all stored files.
	Path string `koanf:"path"`
}

LocalConfig configures local filesystem storage (development only).

type ObjectInfo

type ObjectInfo struct {
	Key         string            `json:"key"`
	Size        int64             `json:"size"`
	ContentType string            `json:"content_type"`
	Visibility  Visibility        `json:"visibility"`
	Metadata    map[string]string `json:"metadata,omitempty"`
	UpdatedAt   time.Time         `json:"updated_at"`
}

ObjectInfo describes a stored object.

type ProviderType

type ProviderType string

ProviderType identifies the storage backend.

const (
	ProviderS3    ProviderType = "s3"
	ProviderGCS   ProviderType = "gcs"
	ProviderAzure ProviderType = "azure"
	ProviderLocal ProviderType = "local"
)

type PutOptions

type PutOptions struct {
	// Visibility controls public access. Defaults to Private.
	Visibility Visibility

	// ContentType is the MIME type (e.g. "image/png", "application/pdf").
	// Auto-detected from key extension when empty.
	ContentType string

	// Metadata stores custom key-value pairs on the provider.
	Metadata map[string]string

	// TenantPrefix overrides the automatic tenant prefix.
	// Empty means auto-detect from context. Set to "" explicitly to disable prefixing.
	TenantPrefix string
}

PutOptions configures how an object is stored.

type S3Config

type S3Config struct {
	// Endpoint is the S3 API endpoint. Empty = AWS S3.
	// For MinIO: "http://minio:9000"
	// For Cloudflare R2: "https://<account>.r2.cloudflarestorage.com"
	Endpoint string `koanf:"endpoint"`

	// Bucket is the default (private) bucket name.
	Bucket string `koanf:"bucket"`

	// Region is the AWS region.
	Region string `koanf:"region"`

	// AccessKeyID credential source.
	// Examples:
	//   env_var: AWS_ACCESS_KEY_ID          # From environment variable
	//   file: /etc/secrets/aws-access-key   # From mounted secret file
	//   value: AKIA...                       # Direct (not recommended for production)
	AccessKeyID CredentialSource `koanf:"access_key_id"`

	// SecretAccessKey credential source.
	// Examples:
	//   env_var: AWS_SECRET_ACCESS_KEY
	//   file: /etc/secrets/aws-secret-key
	SecretAccessKey CredentialSource `koanf:"secret_access_key"`

	// SessionToken credential source (for temporary STS credentials).
	//   env_var: AWS_SESSION_TOKEN
	SessionToken CredentialSource `koanf:"session_token"`

	// UsePathStyle forces path-style URLs instead of virtual-hosted style.
	// Required for MinIO and some other S3-compatible providers.
	UsePathStyle bool `koanf:"use_path_style"`

	// PublicBucket is an optional separate bucket for public objects.
	// When set, public objects are stored here instead of Bucket.
	PublicBucket string `koanf:"public_bucket"`

	// CreateBucketIfMissing provisions Bucket (and PublicBucket, when set)
	// at construction time if they do not exist yet, using Region for the
	// bucket location. Opt-in (QCD-FW-2): the default refuses to create
	// infrastructure against a production object store — but note that a
	// missing bucket now fails the constructor LOUDLY either way, instead
	// of booting green and failing on the first Put.
	CreateBucketIfMissing bool `koanf:"create_bucket_if_missing"`
}

S3Config configures Amazon S3 or any S3-compatible provider (MinIO, R2, etc.).

type Store

type Store interface {
	// Put uploads a file from an io.Reader. The reader is consumed entirely.
	// Key is the logical path (e.g. "tenant_a/uploads/image.png").
	// Returns the final storage key (with tenant prefix applied) and object info.
	Put(ctx context.Context, key string, reader io.Reader, opts PutOptions) (ObjectInfo, error)

	// Get retrieves a file by key. Returns an io.ReadCloser that MUST be closed.
	// Returns ErrNotFound if the key does not exist.
	Get(ctx context.Context, key string) (io.ReadCloser, ObjectInfo, error)

	// Delete removes an object by key. Idempotent: no error if key doesn't exist.
	Delete(ctx context.Context, key string) error

	// Exists checks if a key exists.
	Exists(ctx context.Context, key string) (bool, error)

	// List returns objects with the given prefix.
	List(ctx context.Context, opts ListOptions) (ListResult, error)

	// PublicURL returns a publicly accessible URL for a key.
	// Returns empty string if the object is private or the provider
	// doesn't support public URLs.
	PublicURL(ctx context.Context, key string, opts URLConfig) (string, error)

	// SignedURL returns a time-limited URL for accessing a private object.
	// The URL grants direct access to the object without authentication.
	SignedURL(ctx context.Context, key string, expires time.Duration, opts URLConfig) (string, error)

	// Copy copies an object from srcKey to dstKey (within the same bucket/container).
	Copy(ctx context.Context, srcKey, dstKey string) (ObjectInfo, error)

	// Close releases any resources held by the store (connections, background goroutines).
	Close() error
}

Store is the durable interface for file storage in Nucleus. All implementations (S3, GCS, Azure, local) must satisfy this interface. It is intentionally minimal: add provider-specific features through type assertions when absolutely necessary.

type URLConfig

type URLConfig struct {
	// Expires sets the URL validity duration. Only meaningful for SignedURL.
	Expires time.Duration

	// ContentType overrides the Content-Type header for the URL response.
	ContentType string

	// Disposition sets Content-Disposition header ("inline" or "attachment").
	Disposition string
}

URLConfig configures URL generation.

type Visibility

type Visibility string

Visibility controls whether an object is publicly accessible or requires a signed URL (or app-layer authentication) to access.

const (
	// Private objects are not directly accessible via URL.
	// Access requires SignedURL() or serving through the app layer.
	Private Visibility = "private"

	// Public objects have a direct, unauthenticated URL.
	Public Visibility = "public"
)

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL