s3

package
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jan 10, 2026 License: MIT Imports: 18 Imported by: 0

Documentation

Overview

Package s3 provides an S3-compatible backend for omnistorage.

This backend works with:

  • AWS S3
  • Cloudflare R2
  • MinIO
  • Wasabi
  • DigitalOcean Spaces
  • Any S3-compatible object storage

Basic usage:

backend, err := s3.New(s3.Config{
    Bucket: "my-bucket",
    Region: "us-east-1",
})

For S3-compatible services:

backend, err := s3.New(s3.Config{
    Bucket:       "my-bucket",
    Endpoint:     "https://play.min.io",
    UsePathStyle: true,
})

Index

Constants

This section is empty.

Variables

View Source
var (
	ErrBucketRequired = errors.New("s3: bucket is required")
)

Errors specific to the S3 backend.

Functions

func NewFromConfig

func NewFromConfig(configMap map[string]string) (omnistorage.Backend, error)

NewFromConfig creates a new S3 backend from a config map. This is used by the omnistorage registry.

Types

type Backend

type Backend struct {
	// contains filtered or unexported fields
}

Backend implements omnistorage.ExtendedBackend for S3-compatible storage.

func New

func New(cfg Config) (*Backend, error)

New creates a new S3 backend with the given configuration.

func (*Backend) Close

func (b *Backend) Close() error

Close releases any resources held by the backend.

func (*Backend) Copy

func (b *Backend) Copy(ctx context.Context, src, dst string) error

Copy copies an object using S3 server-side copy.

func (*Backend) Delete

func (b *Backend) Delete(ctx context.Context, p string) error

Delete removes a path.

func (*Backend) Exists

func (b *Backend) Exists(ctx context.Context, p string) (bool, error)

Exists checks if a path exists.

func (*Backend) Features

func (b *Backend) Features() omnistorage.Features

Features returns the capabilities of the S3 backend.

func (*Backend) List

func (b *Backend) List(ctx context.Context, prefix string) ([]string, error)

List lists paths with the given prefix.

func (*Backend) Mkdir

func (b *Backend) Mkdir(ctx context.Context, p string) error

Mkdir creates a directory (no-op for S3, directories are implicit).

func (*Backend) Move

func (b *Backend) Move(ctx context.Context, src, dst string) error

Move moves an object by copying then deleting.

func (*Backend) NewReader

func (b *Backend) NewReader(ctx context.Context, p string, opts ...omnistorage.ReaderOption) (io.ReadCloser, error)

NewReader creates a reader for the given path.

func (*Backend) NewWriter

func (b *Backend) NewWriter(ctx context.Context, p string, opts ...omnistorage.WriterOption) (io.WriteCloser, error)

NewWriter creates a writer for the given path.

func (*Backend) Rmdir

func (b *Backend) Rmdir(ctx context.Context, p string) error

Rmdir removes a directory (removes directory marker if present).

func (*Backend) Stat

Stat returns metadata about an object.

type Config

type Config struct {
	// Bucket is the S3 bucket name (required).
	Bucket string

	// Region is the AWS region (e.g., "us-east-1").
	// If empty, uses AWS_REGION or AWS_DEFAULT_REGION environment variable.
	Region string

	// Endpoint is a custom endpoint URL for S3-compatible services.
	// Examples:
	//   - MinIO: "http://localhost:9000"
	//   - Cloudflare R2: "https://<account_id>.r2.cloudflarestorage.com"
	//   - Wasabi: "https://s3.wasabisys.com"
	// Leave empty for AWS S3.
	Endpoint string

	// Prefix is an optional prefix for all keys.
	// Useful for organizing data within a bucket.
	Prefix string

	// AccessKeyID is the AWS access key ID.
	// If empty, uses AWS_ACCESS_KEY_ID environment variable or IAM role.
	AccessKeyID string

	// SecretAccessKey is the AWS secret access key.
	// If empty, uses AWS_SECRET_ACCESS_KEY environment variable or IAM role.
	SecretAccessKey string

	// SessionToken is an optional session token for temporary credentials.
	SessionToken string

	// UsePathStyle forces path-style addressing instead of virtual-hosted-style.
	// Required for some S3-compatible services like MinIO.
	// Set to true for: MinIO, some older S3-compatible services.
	// Set to false for: AWS S3, Cloudflare R2, Wasabi.
	UsePathStyle bool

	// DisableSSL disables HTTPS for the endpoint.
	// Only use for local development (e.g., local MinIO).
	DisableSSL bool

	// PartSize is the size in bytes for multipart upload parts.
	// Default: 5MB (minimum for S3).
	// Increase for better performance with large files.
	PartSize int64

	// Concurrency is the number of concurrent upload/download goroutines.
	// Default: 5.
	Concurrency int
}

Config holds configuration for the S3 backend.

func ConfigFromEnv

func ConfigFromEnv() Config

ConfigFromEnv creates a Config from environment variables. Environment variables:

  • OMNISTORAGE_S3_BUCKET or AWS_S3_BUCKET: bucket name
  • OMNISTORAGE_S3_REGION or AWS_REGION or AWS_DEFAULT_REGION: region
  • OMNISTORAGE_S3_ENDPOINT: custom endpoint
  • OMNISTORAGE_S3_PREFIX: key prefix
  • AWS_ACCESS_KEY_ID: access key
  • AWS_SECRET_ACCESS_KEY: secret key
  • AWS_SESSION_TOKEN: session token
  • OMNISTORAGE_S3_USE_PATH_STYLE: "true" for path-style addressing
  • OMNISTORAGE_S3_DISABLE_SSL: "true" to disable SSL

func ConfigFromMap

func ConfigFromMap(m map[string]string) Config

ConfigFromMap creates a Config from a string map. Supported keys:

  • bucket: bucket name (required)
  • region: AWS region
  • endpoint: custom endpoint URL
  • prefix: key prefix
  • access_key_id: AWS access key
  • secret_access_key: AWS secret key
  • session_token: session token
  • use_path_style: "true" for path-style addressing
  • disable_ssl: "true" to disable SSL
  • part_size: multipart upload part size in bytes
  • concurrency: number of concurrent operations

func DefaultConfig

func DefaultConfig() Config

DefaultConfig returns a Config with default values.

func (Config) Validate

func (c Config) Validate() error

Validate checks if the configuration is valid.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL