storage

package
v1.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 2, 2026 License: Apache-2.0 Imports: 20 Imported by: 0

Documentation

Overview

Package storage provides an abstraction for object storage operations.

Index

Constants

This section is empty.

Variables

View Source 
var (
	ErrNotFound     = s3lect.ErrStorageNotFound
	ErrPrecondition = s3lect.ErrStoragePrecondition
)

Common storage errors - use s3lect's errors for compatibility

Functions

This section is empty.

Types

type FileStorage 

type FileStorage struct {
	// contains filtered or unexported fields
}

FileStorage implements Storage interface for local filesystem operations

func NewFileStorage 

func NewFileStorage(baseDir string) (*FileStorage, error)

NewFileStorage creates a new file-based storage

func (*FileStorage) Delete 

func (f *FileStorage) Delete(ctx context.Context, key string) error

Delete removes a file

func (*FileStorage) Exists 

func (f *FileStorage) Exists(ctx context.Context, key string) (bool, error)

Exists checks if a file exists

func (*FileStorage) Get 

func (f *FileStorage) Get(ctx context.Context, key string) ([]byte, string, error)

Get retrieves data from a file and returns its contents and ETag

func (*FileStorage) GetMetadata 

func (f *FileStorage) GetMetadata(ctx context.Context, key string) (*ObjectMetadata, error)

GetMetadata retrieves file metadata

func (*FileStorage) GetStream 

func (f *FileStorage) GetStream(ctx context.Context, key string) (io.ReadCloser, error)

GetStream retrieves a file as a stream

func (*FileStorage) List 

func (f *FileStorage) List(ctx context.Context, prefix string) ([]string, error)

List lists files with the given prefix

func (*FileStorage) Put 

func (f *FileStorage) Put(ctx context.Context, key string, data []byte) error

Put stores data to a file

func (*FileStorage) PutIfMatch 

func (f *FileStorage) PutIfMatch(ctx context.Context, key string, data []byte, etag string) error

PutIfMatch stores data only if ETag matches (optimistic locking) Empty etag means file must not exist

func (*FileStorage) PutStream 

func (f *FileStorage) PutStream(ctx context.Context, key string, reader io.Reader, size int64) error

PutStream stores a file from a stream

type GCSStorage 

type GCSStorage struct {
	// contains filtered or unexported fields
}

GCSStorage implements Storage interface using Google Cloud Storage

func NewGCSStorage 

func NewGCSStorage(client *storage.Client, bucket string) *GCSStorage

NewGCSStorage creates a new GCS storage instance

func (*GCSStorage) Delete 

func (g *GCSStorage) Delete(ctx context.Context, key string) error

Delete removes an object from GCS

func (*GCSStorage) Exists 

func (g *GCSStorage) Exists(ctx context.Context, key string) (bool, error)

Exists checks if an object exists in GCS

func (*GCSStorage) Get 

func (g *GCSStorage) Get(ctx context.Context, key string) ([]byte, string, error)

Get retrieves an object from GCS and returns its contents and ETag (Generation)

func (*GCSStorage) GetMetadata 

func (g *GCSStorage) GetMetadata(ctx context.Context, key string) (*ObjectMetadata, error)

GetMetadata retrieves metadata for an object

func (*GCSStorage) GetStream 

func (g *GCSStorage) GetStream(ctx context.Context, key string) (io.ReadCloser, error)

GetStream retrieves an object as a stream

func (*GCSStorage) List 

func (g *GCSStorage) List(ctx context.Context, prefix string) ([]string, error)

List lists objects with the given prefix

func (*GCSStorage) Put 

func (g *GCSStorage) Put(ctx context.Context, key string, data []byte) error

Put stores an object in GCS

func (*GCSStorage) PutIfMatch 

func (g *GCSStorage) PutIfMatch(ctx context.Context, key string, data []byte, etag string) error

PutIfMatch stores an object only if the generation matches (optimistic locking) Empty etag means object must not exist

func (*GCSStorage) PutStream 

func (g *GCSStorage) PutStream(ctx context.Context, key string, reader io.Reader, size int64) error

PutStream stores an object from a stream

type MockStorage 

type MockStorage struct {
	// contains filtered or unexported fields
}

MockStorage implements Storage interface using local filesystem for testing

func NewMock 

func NewMock() *MockStorage

NewMock creates a new mock storage instance with a temporary directory

func NewMockStorage 

func NewMockStorage(baseDir string) (*MockStorage, error)

NewMockStorage creates a new mock storage instance

func (*MockStorage) Cleanup 

func (m *MockStorage) Cleanup() error

Cleanup removes all files in the mock storage (useful for testing)

func (*MockStorage) Delete 

func (m *MockStorage) Delete(ctx context.Context, key string) error

Delete removes an object from mock storage

func (*MockStorage) Exists 

func (m *MockStorage) Exists(ctx context.Context, key string) (bool, error)

Exists checks if an object exists in mock storage

func (*MockStorage) Get 

func (m *MockStorage) Get(ctx context.Context, key string) ([]byte, string, error)

Get retrieves an object from mock storage and returns its contents and ETag

func (*MockStorage) GetMetadata 

func (m *MockStorage) GetMetadata(ctx context.Context, key string) (*ObjectMetadata, error)

GetMetadata retrieves metadata for an object

func (*MockStorage) GetStream 

func (m *MockStorage) GetStream(ctx context.Context, key string) (io.ReadCloser, error)

GetStream retrieves an object as a stream

func (*MockStorage) List 

func (m *MockStorage) List(ctx context.Context, prefix string) ([]string, error)

List lists objects with the given prefix

func (*MockStorage) Put 

func (m *MockStorage) Put(ctx context.Context, key string, data []byte) error

Put stores an object in mock storage

func (*MockStorage) PutIfMatch 

func (m *MockStorage) PutIfMatch(ctx context.Context, key string, data []byte, etag string) error

PutIfMatch stores an object only if the ETag matches (optimistic locking) Empty etag means object must not exist

func (*MockStorage) PutStream 

func (m *MockStorage) PutStream(ctx context.Context, key string, reader io.Reader, size int64) error

PutStream stores an object from a stream

type ObjectMetadata 

type ObjectMetadata struct {
	Key          string
	Size         int64
	LastModified time.Time
	ETag         string
	ContentType  string
}

ObjectMetadata contains metadata about a stored object

type S3Storage 

type S3Storage struct {
	// contains filtered or unexported fields
}

S3Storage implements Storage interface using AWS S3

func NewS3Storage 

func NewS3Storage(client *s3.Client, bucket string) *S3Storage

NewS3Storage creates a new S3 storage instance

func (*S3Storage) Delete 

func (s *S3Storage) Delete(ctx context.Context, key string) error

Delete removes an object from S3

func (*S3Storage) Exists 

func (s *S3Storage) Exists(ctx context.Context, key string) (bool, error)

Exists checks if an object exists in S3

func (*S3Storage) Get 

func (s *S3Storage) Get(ctx context.Context, key string) ([]byte, string, error)

Get retrieves an object from S3 and returns its contents and ETag

func (*S3Storage) GetMetadata 

func (s *S3Storage) GetMetadata(ctx context.Context, key string) (*ObjectMetadata, error)

GetMetadata retrieves metadata for an object

func (*S3Storage) GetStream 

func (s *S3Storage) GetStream(ctx context.Context, key string) (io.ReadCloser, error)

GetStream retrieves an object as a stream

func (*S3Storage) List 

func (s *S3Storage) List(ctx context.Context, prefix string) ([]string, error)

List lists objects with the given prefix

func (*S3Storage) Put 

func (s *S3Storage) Put(ctx context.Context, key string, data []byte) error

Put stores an object in S3

func (*S3Storage) PutIfMatch 

func (s *S3Storage) PutIfMatch(ctx context.Context, key string, data []byte, etag string) error

PutIfMatch stores an object only if the ETag matches (optimistic locking) Empty etag means object must not exist

func (*S3Storage) PutStream 

func (s *S3Storage) PutStream(ctx context.Context, key string, reader io.Reader, size int64) error

PutStream stores an object from a stream

type ScopedStorage 

type ScopedStorage struct {
	// contains filtered or unexported fields
}

ScopedStorage wraps a Storage interface and prefixes all keys with a given prefix. This enables logical separation of cluster-scoped and shard-scoped data within a single bucket.

func (*ScopedStorage) Delete 

func (s *ScopedStorage) Delete(ctx context.Context, key string) error

Delete removes an object from storage

func (*ScopedStorage) Exists 

func (s *ScopedStorage) Exists(ctx context.Context, key string) (bool, error)

Exists checks if an object exists in storage

func (*ScopedStorage) Get 

func (s *ScopedStorage) Get(ctx context.Context, key string) ([]byte, string, error)

Get retrieves an object from storage and returns its contents and its etag

func (*ScopedStorage) GetMetadata 

func (s *ScopedStorage) GetMetadata(ctx context.Context, key string) (*ObjectMetadata, error)

GetMetadata retrieves metadata for an object, updating the Key field to strip the prefix

func (*ScopedStorage) GetStream 

func (s *ScopedStorage) GetStream(ctx context.Context, key string) (io.ReadCloser, error)

GetStream retrieves an object as a stream

func (*ScopedStorage) List 

func (s *ScopedStorage) List(ctx context.Context, prefix string) ([]string, error)

List lists objects with the given prefix, stripping the scope prefix from returned keys

func (*ScopedStorage) Prefix 

func (s *ScopedStorage) Prefix() string

Prefix returns the configured prefix for this scoped storage

func (*ScopedStorage) Put 

func (s *ScopedStorage) Put(ctx context.Context, key string, data []byte) error

Put stores an object in storage

func (*ScopedStorage) PutIfMatch 

func (s *ScopedStorage) PutIfMatch(ctx context.Context, key string, data []byte, etag string) error

PutIfMatch stores an object only if the ETag matches (empty string = must not exist)

func (*ScopedStorage) PutStream 

func (s *ScopedStorage) PutStream(ctx context.Context, key string, reader io.Reader, size int64) error

PutStream stores an object from a stream

type Storage 

type Storage interface {
	// Get retrieves an object from storage and returns its contents and its etag
	// Returns ErrNotFound if object does not exist
	Get(ctx context.Context, key string) ([]byte, string, error)

	// Put stores an object in storage
	Put(ctx context.Context, key string, data []byte) error

	// PutIfMatch stores an object only if the ETag matches (empty string = must not exist)
	// Returns ErrPrecondition if ETag doesn't match
	PutIfMatch(ctx context.Context, key string, data []byte, etag string) error

	// Delete removes an object from storage
	Delete(ctx context.Context, key string) error

	// Exists checks if an object exists in storage
	Exists(ctx context.Context, key string) (bool, error)

	// List lists objects with the given prefix
	List(ctx context.Context, prefix string) ([]string, error)

	// GetMetadata retrieves metadata for an object
	GetMetadata(ctx context.Context, key string) (*ObjectMetadata, error)

	// GetStream retrieves an object as a stream
	GetStream(ctx context.Context, key string) (io.ReadCloser, error)

	// PutStream stores an object from a stream
	PutStream(ctx context.Context, key string, reader io.Reader, size int64) error
}

Storage provides an abstraction for object storage operations

func New 

func New(ctx context.Context, logger *slog.Logger, provider, bucket string) (Storage, func(), error)

New creates the appropriate storage backend based on the provider string. Valid providers: "s3", "gcs", "file" Returns the storage, a cleanup function, and any error.

func NewScopedStorage 

func NewScopedStorage(underlying Storage, prefix string) Storage

NewScopedStorage creates a new ScopedStorage that prefixes all keys. The prefix should include a trailing slash if keys should be nested (e.g., "cluster/").

func NewWithOptions 

func NewWithOptions(ctx context.Context, logger *slog.Logger, opts StorageOptions) (Storage, func(), error)

NewWithOptions creates a storage backend with full configuration options. This supports custom endpoints for S3-compatible backends like SeaweedFS, MinIO, and Ceph RGW.

type StorageOptions 

type StorageOptions struct {
	Provider  string // Storage provider: "s3", "gcs", "file"
	Bucket    string // Bucket name (or path for file storage)
	Region    string // AWS region override (optional)
	Endpoint  string // Custom endpoint for S3-compatible backends (SeaweedFS, MinIO, Ceph RGW)
	PathStyle bool   // Use path-style access for S3 (default: false, uses virtual-hosted style)
}

StorageOptions contains parameters for creating storage backends with full configuration.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.