Documentation
¶
Overview ¶
Package driver defines the core interfaces and types that every Trove storage backend must implement.
The Driver interface is intentionally minimal — advanced features are exposed via capability interfaces (MultipartDriver, PresignDriver, etc.) and the Unwrap pattern.
Error contract ¶
Callers classify failures with errors.Is, so implementations MUST wrap the sentinels declared in errors.go rather than returning a bare descriptive error. Keep the descriptive message — wrap with %w, do not replace:
return nil, fmt.Errorf("mydriver: object %q not found in bucket %q: %w",
key, bucket, driver.ErrObjectNotFound)
The conditions that MUST be classified:
- a missing object → ErrObjectNotFound (Get, Head, Copy source, GetRange)
- a missing bucket → ErrBucketNotFound (any operation naming a bucket)
- an existing bucket → ErrBucketExists (CreateBucket)
- an unauthorized op → ErrPermissionDenied
- a quota/rate limit → ErrQuotaExceeded
This matters to schedulers built on Trove: a permanently missing object must fail its work item immediately, while a transient backend failure should be retried. Without the sentinel the two are indistinguishable and every retry is burned before the failure is recognized as permanent.
Backends that report these conditions with a typed error or an HTTP status code MUST use that signal rather than matching on message text. Delete is idempotent by convention and returns nil for a missing object.
Index ¶
- Variables
- func Drivers() []string
- func Permanent(err error) bool
- func Register(name string, factory Factory)
- type BucketConfig
- type BucketInfo
- type BucketOption
- type Config
- type CopyConfig
- type CopyOption
- type DSNConfig
- type DeleteConfig
- type DeleteOption
- type Driver
- type Factory
- type GetConfig
- type GetOption
- type KeyPage
- type LifecycleDriver
- type LifecycleRule
- type ListConfig
- type ListOption
- type MultipartDriver
- type NotificationDriver
- type ObjectEvent
- type ObjectEventType
- type ObjectInfo
- type ObjectIterator
- type ObjectReader
- type ObjectRef
- type Option
- type PartInfo
- type PresignDriver
- type PutConfig
- type PutOption
- type RangeDriver
- type ServerCopyDriver
- type VersionInfo
- type VersioningDriver
- type WatchConfig
- type WatchOption
Constants ¶
This section is empty.
Variables ¶
var ( // ErrNotFound reports that a requested resource does not exist. // It is the parent of ErrObjectNotFound and ErrBucketNotFound; match it // when any missing resource should be handled the same way. ErrNotFound = errors.New("trove: not found") // ErrObjectNotFound reports that an object does not exist. // It unwraps to ErrNotFound. ErrObjectNotFound error = &categoryError{msg: "trove: object not found", parent: ErrNotFound} // ErrBucketNotFound reports that a bucket or container does not exist. // It unwraps to ErrNotFound. ErrBucketNotFound error = &categoryError{msg: "trove: bucket not found", parent: ErrNotFound} // ErrBucketExists reports that a bucket being created already exists. ErrBucketExists = errors.New("trove: bucket already exists") // ErrPermissionDenied reports that the credentials in use are not // authorized for the operation. Retrying will not help until the // credentials or the backend's access policy change. ErrPermissionDenied = errors.New("trove: permission denied") // ErrQuotaExceeded reports that a storage quota or rate limit was // exceeded. Unlike ErrPermissionDenied, retrying later may succeed. ErrQuotaExceeded = errors.New("trove: quota exceeded") // ErrInvalidPath reports that a bucket name or object key is not // addressable — it is empty, contains a NUL byte, is rooted, or resolves // outside the container it was addressed to. Drivers that map keys onto a // hierarchical namespace wrap it from their containment guard. // // It is deliberately outside the not-found family. The resource was never // addressable, so nothing was looked up: a caller must be able to answer // "the request is malformed" (HTTP 400) rather than "it isn't there" // (404), which would both misdescribe the failure and let a client use // rejected keys to probe what exists. ErrInvalidPath = errors.New("trove: invalid path") )
Sentinel errors that drivers must wrap so callers can classify failures with errors.Is, independent of which backend produced them.
These live in the driver package rather than the root trove package so that out-of-tree drivers (which live in their own Go modules) can depend on them without importing the whole Trove facade. The root package re-exports every one of them, and since errors.Is compares by identity, trove.ErrObjectNotFound and driver.ErrObjectNotFound are interchangeable.
The not-found sentinels form a small hierarchy: ErrObjectNotFound and ErrBucketNotFound both unwrap to ErrNotFound, so a caller that only cares whether something is missing can match the general sentinel, while a caller that needs to know what was missing can match the specific one.
errors.Is(err, driver.ErrObjectNotFound) // this exact object is gone errors.Is(err, driver.ErrNotFound) // something is gone — also true
Functions ¶
func Permanent ¶
Permanent reports whether err describes a condition that retrying cannot resolve, so the caller should fail the operation now rather than spend its backoff budget on attempts that are certain to fail the same way.
Permanent:
- ErrNotFound, and with it ErrObjectNotFound and ErrBucketNotFound — the resource is gone and will not reappear on its own.
- ErrPermissionDenied — nothing changes until the credentials or the backend's access policy do.
- ErrBucketExists — the bucket will not stop existing because you asked again.
- ErrInvalidPath — a malformed key does not become well-formed on the next attempt.
Not permanent:
- ErrQuotaExceeded — the operation is refused now but may be granted later. Retry it, ideally with a longer backoff than a network failure.
- Everything else, including unclassified errors. A driver that has not classified a condition is assumed to have hit a transient one; treating unrecognized errors as permanent would silently discard work whenever a backend grows a failure mode Trove does not yet map.
The quota check is evaluated first, so an error carrying both a quota signal and a not-found signal is reported as retryable.
if err != nil {
if driver.Permanent(err) {
return job.Fail(err)
}
return job.Retry(err)
}
Types ¶
type BucketConfig ¶
type BucketConfig struct {
// Region overrides the default region for the bucket.
Region string
}
BucketConfig holds options for a CreateBucket operation.
func ApplyBucketOptions ¶
func ApplyBucketOptions(opts ...BucketOption) BucketConfig
ApplyBucketOptions applies a set of BucketOptions to a BucketConfig.
type BucketInfo ¶
BucketInfo holds bucket metadata.
type BucketOption ¶
type BucketOption func(*BucketConfig)
BucketOption configures a CreateBucket operation.
func WithBucketRegion ¶
func WithBucketRegion(region string) BucketOption
WithBucketRegion sets the region for a new bucket.
type Config ¶
type Config struct {
// Region is the cloud region for the storage backend.
Region string
// Endpoint overrides the default service endpoint.
Endpoint string
// ForcePathStyle forces path-style addressing for S3-compatible backends.
ForcePathStyle bool
}
Config holds driver-level configuration applied during Open.
func ApplyOptions ¶
ApplyOptions applies a set of options to a Config.
type CopyConfig ¶
type CopyConfig struct {
// Metadata overrides the source object's metadata on the copy.
Metadata map[string]string
}
CopyConfig holds options for a Copy operation.
func ApplyCopyOptions ¶
func ApplyCopyOptions(opts ...CopyOption) CopyConfig
ApplyCopyOptions applies a set of CopyOptions to a CopyConfig.
type CopyOption ¶
type CopyOption func(*CopyConfig)
CopyOption configures a Copy operation.
func WithCopyMetadata ¶
func WithCopyMetadata(meta map[string]string) CopyOption
WithCopyMetadata overrides the source object's metadata on the copy.
type DSNConfig ¶
type DSNConfig struct {
// Scheme is the protocol scheme (e.g., "s3", "gcs", "file", "mem").
Scheme string
// User is the username or access key.
User string
// Password is the password or secret key.
Password string
// Host is the hostname and optional port.
Host string
// Path is the path component (e.g., bucket, directory).
Path string
// Params holds query string parameters.
Params url.Values
}
DSNConfig holds parsed DSN components.
func ParseDSN ¶
ParseDSN parses a DSN string into its components.
Supported formats:
scheme://host/path?params scheme://user:password@host/path?params scheme:///path (host-less, e.g., file:///tmp/storage)
Its errors never quote the DSN, or any part of it but the scheme. A DSN carries credentials (an access key and secret in the userinfo, an account key or a connection string in the query), and an error travels into logs and onto trace spans: forge records a failed Register's error text on its lifecycle span. An error names the scheme when the DSN starts with a valid one followed by "://", and says what kind of problem it found.
type DeleteConfig ¶
type DeleteConfig struct {
// VersionID deletes a specific version.
VersionID string
}
DeleteConfig holds options for a Delete operation.
func ApplyDeleteOptions ¶
func ApplyDeleteOptions(opts ...DeleteOption) DeleteConfig
ApplyDeleteOptions applies a set of DeleteOptions to a DeleteConfig.
type DeleteOption ¶
type DeleteOption func(*DeleteConfig)
DeleteOption configures a Delete operation.
type Driver ¶
type Driver interface {
// Name returns the driver identifier (e.g., "s3", "gcs", "azure", "local", "mem").
Name() string
// Open initializes the driver connection with the given DSN and options.
Open(ctx context.Context, dsn string, opts ...Option) error
// Close gracefully shuts down the driver, draining active streams.
Close(ctx context.Context) error
// Ping verifies backend connectivity.
Ping(ctx context.Context) error
// Put stores an object from a reader. Returns the stored object metadata.
Put(ctx context.Context, bucket, key string, r io.Reader, opts ...PutOption) (*ObjectInfo, error)
// Get retrieves an object, returning a ReadCloser for the content.
Get(ctx context.Context, bucket, key string, opts ...GetOption) (*ObjectReader, error)
// Delete removes an object.
Delete(ctx context.Context, bucket, key string, opts ...DeleteOption) error
// Head returns object metadata without content.
Head(ctx context.Context, bucket, key string) (*ObjectInfo, error)
// List returns objects matching the given options.
List(ctx context.Context, bucket string, opts ...ListOption) (*ObjectIterator, error)
// Copy copies an object within or across buckets.
Copy(ctx context.Context, srcBucket, srcKey, dstBucket, dstKey string, opts ...CopyOption) (*ObjectInfo, error)
// CreateBucket creates a new bucket/container.
CreateBucket(ctx context.Context, name string, opts ...BucketOption) error
// DeleteBucket removes a bucket.
DeleteBucket(ctx context.Context, name string) error
// ListBuckets returns all accessible buckets.
ListBuckets(ctx context.Context) ([]BucketInfo, error)
}
Driver is the core interface every storage backend implements. It exposes the minimum surface area for object CRUD and streaming.
type GetConfig ¶
type GetConfig struct {
// Range specifies a byte range to retrieve (e.g., "bytes=0-1023").
Range string
// VersionID retrieves a specific version of the object.
VersionID string
}
GetConfig holds options for a Get operation.
func ApplyGetOptions ¶
ApplyGetOptions applies a set of GetOptions to a GetConfig.
type GetOption ¶
type GetOption func(*GetConfig)
GetOption configures a Get operation.
func WithVersionID ¶
WithVersionID retrieves a specific version.
type KeyPage ¶
type KeyPage struct {
// Keys are the object keys on this page, in order.
Keys []string
// Prefixes are the common prefixes on this page, in order.
Prefixes []string
// NextToken is the last item on this page when more items follow,
// and empty when the listing is complete.
NextToken string
}
KeyPage is one page of a listing, computed from a full sorted key set.
func PageKeys ¶
func PageKeys(sorted []string, cfg ListConfig) KeyPage
PageKeys computes one page of a listing for drivers that hold, or can walk, their whole key set. The keys must be sorted lexicographically and free of duplicates.
It applies cfg.Prefix, folds keys into common prefixes when cfg.Delimiter is set, skips every item at or before cfg.Cursor, and stops after cfg.MaxKeys items, counting prefixes and keys together. A MaxKeys of zero or less means 1000. The returned NextToken is the last item emitted, key or prefix, which is what the next call's Cursor must be.
type LifecycleDriver ¶
type LifecycleDriver interface {
Driver
SetLifecycle(ctx context.Context, bucket string, rules []LifecycleRule) error
GetLifecycle(ctx context.Context, bucket string) ([]LifecycleRule, error)
}
LifecycleDriver extends Driver with lifecycle management.
type LifecycleRule ¶
type LifecycleRule struct {
ID string `json:"id"`
Prefix string `json:"prefix"`
ExpirationDays int `json:"expiration_days,omitempty"`
TransitionDays int `json:"transition_days,omitempty"`
TransitionStorage string `json:"transition_storage,omitempty"`
Enabled bool `json:"enabled"`
}
LifecycleRule defines a storage lifecycle transition or expiration rule.
type ListConfig ¶
type ListConfig struct {
// Prefix filters objects by key prefix.
Prefix string
// Delimiter groups keys by a delimiter character (e.g., "/"). Keys that
// contain the delimiter after Prefix are folded into a common prefix and
// come back in ObjectIterator.CommonPrefixes() instead of as objects.
Delimiter string
// MaxKeys is the maximum number of objects and common prefixes returned
// together in one page.
MaxKeys int
// Cursor is the NextToken from a previous List call, passed back
// unchanged. It is opaque: drivers may use a key, a backend
// continuation token or anything else, so never construct one.
Cursor string
}
ListConfig holds options for a List operation.
func ApplyListOptions ¶
func ApplyListOptions(opts ...ListOption) ListConfig
ApplyListOptions applies a set of ListOptions to a ListConfig.
type ListOption ¶
type ListOption func(*ListConfig)
ListOption configures a List operation.
func WithDelimiter ¶
func WithDelimiter(delimiter string) ListOption
WithDelimiter groups keys by a delimiter character.
func WithMaxKeys ¶
func WithMaxKeys(n int) ListOption
WithMaxKeys sets the maximum number of objects to return.
func WithPrefix ¶
func WithPrefix(prefix string) ListOption
WithPrefix filters objects by key prefix.
type MultipartDriver ¶
type MultipartDriver interface {
Driver
InitiateMultipart(ctx context.Context, bucket, key string, opts ...PutOption) (uploadID string, err error)
UploadPart(ctx context.Context, bucket, key, uploadID string, partNum int, r io.Reader) (*PartInfo, error)
CompleteMultipart(ctx context.Context, bucket, key, uploadID string, parts []PartInfo) (*ObjectInfo, error)
AbortMultipart(ctx context.Context, bucket, key, uploadID string) error
}
MultipartDriver extends Driver with multipart upload support.
type NotificationDriver ¶
type NotificationDriver interface {
Driver
Watch(ctx context.Context, bucket string, opts ...WatchOption) (<-chan ObjectEvent, error)
}
NotificationDriver extends Driver with change notification support.
type ObjectEvent ¶
type ObjectEvent struct {
Type ObjectEventType `json:"type"`
Bucket string `json:"bucket"`
Key string `json:"key"`
Info *ObjectInfo `json:"info,omitempty"`
}
ObjectEvent represents a change notification.
type ObjectEventType ¶
type ObjectEventType string
ObjectEventType identifies the type of object change event.
const ( EventCreated ObjectEventType = "object.created" EventDeleted ObjectEventType = "object.deleted" EventUpdated ObjectEventType = "object.updated" )
type ObjectInfo ¶
type ObjectInfo struct {
Key string `json:"key"`
Size int64 `json:"size"`
ContentType string `json:"content_type"`
ETag string `json:"etag"`
LastModified time.Time `json:"last_modified"`
Metadata map[string]string `json:"metadata,omitempty"`
VersionID string `json:"version_id,omitempty"`
StorageClass string `json:"storage_class,omitempty"`
}
ObjectInfo holds the metadata returned by backend operations.
type ObjectIterator ¶
type ObjectIterator struct {
// contains filtered or unexported fields
}
ObjectIterator provides cursor-based listing of objects.
NextToken is opaque. Pass it back unchanged as the Cursor of the next List call; an empty NextToken means the listing is complete. When the List call set a Delimiter, CommonPrefixes holds the prefixes that keys below them were folded into, and MaxKeys counted them alongside objects.
func NewObjectIterator ¶
func NewObjectIterator(objects []ObjectInfo, nextToken string) *ObjectIterator
NewObjectIterator creates an iterator from a slice of objects and an optional continuation token for the next page.
func NewObjectIteratorWithPrefixes ¶
func NewObjectIteratorWithPrefixes(objects []ObjectInfo, prefixes []string, nextToken string) *ObjectIterator
NewObjectIteratorWithPrefixes creates an iterator that also reports the common prefixes a delimiter listing folded keys into.
func (*ObjectIterator) All ¶
func (it *ObjectIterator) All(ctx context.Context) ([]ObjectInfo, error)
All collects all remaining objects from the iterator into a slice.
func (*ObjectIterator) Close ¶
func (it *ObjectIterator) Close() error
Close releases any resources held by the iterator.
func (*ObjectIterator) CommonPrefixes ¶
func (it *ObjectIterator) CommonPrefixes() []string
CommonPrefixes returns the prefixes this page folded keys into. It is empty when the List call set no Delimiter.
func (*ObjectIterator) Next ¶
func (it *ObjectIterator) Next(_ context.Context) (*ObjectInfo, error)
Next returns the next object in the iterator. Returns nil, io.EOF when no more objects are available.
func (*ObjectIterator) NextToken ¶
func (it *ObjectIterator) NextToken() string
NextToken returns the continuation token for fetching the next page. Returns empty string when all results have been returned.
type ObjectReader ¶
type ObjectReader struct {
io.ReadCloser
Info *ObjectInfo
}
ObjectReader wraps content with metadata.
type Option ¶
type Option func(*Config)
Option configures a driver during Open.
func WithEndpoint ¶
WithEndpoint overrides the default service endpoint.
func WithForcePathStyle ¶
WithForcePathStyle forces path-style addressing.
type PartInfo ¶
type PartInfo struct {
PartNumber int `json:"part_number"`
ETag string `json:"etag"`
Size int64 `json:"size"`
}
PartInfo holds metadata about an uploaded part.
type PresignDriver ¶
type PresignDriver interface {
Driver
PresignGet(ctx context.Context, bucket, key string, expires time.Duration) (string, error)
PresignPut(ctx context.Context, bucket, key string, expires time.Duration) (string, error)
}
PresignDriver extends Driver with pre-signed URL generation.
type PutConfig ¶
type PutConfig struct {
ContentType string
Metadata map[string]string
Tags map[string]string
StorageClass string
}
PutConfig holds options for a Put operation.
func ApplyPutOptions ¶
ApplyPutOptions applies a set of PutOptions to a PutConfig.
type PutOption ¶
type PutOption func(*PutConfig)
PutOption configures a Put operation.
func WithContentType ¶
WithContentType sets the object's content type.
func WithMetadata ¶
WithMetadata sets custom metadata on the object.
func WithStorageClass ¶
WithStorageClass sets the storage class for the object.
type RangeDriver ¶
type RangeDriver interface {
Driver
// GetRange retrieves a byte range of an object. The offset is the
// starting byte position and length is the number of bytes to read.
// A length of -1 reads from offset to end.
GetRange(ctx context.Context, bucket, key string, offset, length int64) (*ObjectReader, error)
}
RangeDriver extends Driver with byte-range read support. This enables resumable downloads and partial content retrieval.
type ServerCopyDriver ¶
type ServerCopyDriver interface {
Driver
ServerCopy(ctx context.Context, src, dst ObjectRef) (*ObjectInfo, error)
}
ServerCopyDriver extends Driver with server-side copy (avoids download+reupload).
type VersionInfo ¶
type VersionInfo struct {
VersionID string `json:"version_id"`
Key string `json:"key"`
Size int64 `json:"size"`
LastModified time.Time `json:"last_modified"`
IsLatest bool `json:"is_latest"`
}
VersionInfo holds metadata about an object version.
type VersioningDriver ¶
type VersioningDriver interface {
Driver
ListVersions(ctx context.Context, bucket, key string) ([]VersionInfo, error)
GetVersion(ctx context.Context, bucket, key, versionID string) (*ObjectReader, error)
DeleteVersion(ctx context.Context, bucket, key, versionID string) error
RestoreVersion(ctx context.Context, bucket, key, versionID string) (*ObjectInfo, error)
}
VersioningDriver extends Driver with version management.
type WatchConfig ¶
type WatchConfig struct {
Prefix string
EventTypes []ObjectEventType
}
WatchConfig holds options for a Watch operation.