provider

package
v0.4.7 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 9, 2026 License: Apache-2.0 Imports: 6 Imported by: 0

Documentation

Overview

Package provider defines abstractions for cloud object storage operations.

Providers implement a minimal surface area focused on listing and metadata retrieval. Authentication uses SDK default credential chains - providers should not implement custom auth logic.

API stability: Stable. Breaking changes to exported symbols or documented behavior follow the Library API protocol in docs/api-stability.md.

Index

Constants

This section is empty.

Variables

View Source
var (
	// ErrNotFound indicates the requested object does not exist.
	ErrNotFound = errors.New("object not found")

	// ErrAccessDenied indicates insufficient permissions.
	ErrAccessDenied = errors.New("access denied")

	// ErrAnonymousReadOnly indicates an anonymous provider was asked to perform
	// a mutating operation. It is joined with ErrAccessDenied by providers that
	// support unsigned read-only construction.
	ErrAnonymousReadOnly = errors.New("anonymous provider is read-only")

	// ErrBucketNotFound indicates the bucket does not exist.
	ErrBucketNotFound = errors.New("bucket not found")

	// ErrInvalidCredentials indicates authentication failed.
	ErrInvalidCredentials = errors.New("invalid credentials")

	// ErrCredentialsRefreshFailed indicates a provider credential cache or
	// token provider could not refresh credentials for the operation.
	ErrCredentialsRefreshFailed = errors.New("credentials refresh failed")

	// ErrProviderUnavailable indicates the provider service is unavailable.
	ErrProviderUnavailable = errors.New("provider unavailable")

	// ErrThrottled indicates the request was rate limited by the provider.
	ErrThrottled = errors.New("request throttled")

	// ErrAlreadyExists indicates an atomic create was refused because the object exists.
	ErrAlreadyExists = errors.New("object already exists")

	// ErrPreconditionFailed indicates an atomic write predicate did not hold.
	ErrPreconditionFailed = errors.New("write precondition failed")

	// ErrUnsupportedPrecondition indicates a provider does not support the
	// requested atomic write predicate. This is distinct from
	// ErrPreconditionFailed, which means the provider supports the predicate but
	// the predicate evaluated false.
	ErrUnsupportedPrecondition = errors.New("unsupported write precondition")

	// ErrIncompleteListing indicates a listing page reported more results but
	// gave no way to fetch them (truncated without a continuation token, or a
	// token that repeats). Callers must treat the listing as failed, never as
	// complete: a crawl plan that stops here would attest unobserved keys.
	ErrIncompleteListing = errors.New("incomplete listing")
)

Sentinel errors for provider operations.

View Source
var ErrReplayUnverifiable = errors.New("source replay is unverifiable")

ErrReplayUnverifiable reports that a provider cannot enforce the revision token needed to re-drive an admitted object safely.

View Source
var ErrSourceChanged = errors.New("source changed after admission")

ErrSourceChanged reports that a revision-bound read no longer resolves to the source revision admitted by the run.

Functions

func IsAccessDenied

func IsAccessDenied(err error) bool

IsAccessDenied returns true if the error indicates insufficient permissions.

func IsAlreadyExists

func IsAlreadyExists(err error) bool

IsAlreadyExists returns true if the error indicates the target already exists.

func IsAnonymousReadOnly

func IsAnonymousReadOnly(err error) bool

IsAnonymousReadOnly returns true if the error indicates an anonymous provider rejected a mutating operation.

func IsBucketNotFound

func IsBucketNotFound(err error) bool

IsBucketNotFound returns true if the error indicates the bucket does not exist.

func IsCredentialsRefreshFailed

func IsCredentialsRefreshFailed(err error) bool

IsCredentialsRefreshFailed returns true if the error indicates a provider-side credential refresh failure.

func IsInvalidCredentials

func IsInvalidCredentials(err error) bool

IsInvalidCredentials returns true if the error indicates authentication failed.

func IsNotFound

func IsNotFound(err error) bool

IsNotFound returns true if the error indicates an object was not found.

func IsPreconditionFailed

func IsPreconditionFailed(err error) bool

IsPreconditionFailed returns true if the error indicates a write precondition failed.

func IsProviderUnavailable

func IsProviderUnavailable(err error) bool

IsProviderUnavailable returns true if the error indicates the provider service is unavailable.

func IsReplayUnverifiable

func IsReplayUnverifiable(err error) bool

IsReplayUnverifiable reports whether a revision-bound read cannot be proven.

func IsSourceChanged

func IsSourceChanged(err error) bool

IsSourceChanged reports whether a revision-bound read observed source drift.

func IsThrottled

func IsThrottled(err error) bool

IsThrottled returns true if the error indicates the request was rate limited.

func IsUnsupportedPrecondition

func IsUnsupportedPrecondition(err error) bool

IsUnsupportedPrecondition returns true if the provider explicitly rejected a write precondition it cannot honor atomically.

Types

type ConditionalCapabilityReporter

type ConditionalCapabilityReporter interface {
	ConditionalWriteCapabilities() ConditionalWriteCapabilities
}

ConditionalCapabilityReporter reports which conditional-write predicates the provider honors. Providers that implement ConditionalPutter (and, where applicable, ConditionalMultipartCompleter) should implement this so callers can validate predicate support before touching the destination. A provider that does not implement this interface cannot prove it honors any conditional predicate and must be treated as unable to honor them — the injected adapter's declared capability, not the mere presence of ConditionalPutter, is the authority. A remote endpoint reached through a declaring adapter remains a documented trust boundary.

type ConditionalMultipartCompleter

type ConditionalMultipartCompleter interface {
	CompleteMultipartUploadConditional(ctx context.Context, key, uploadID string, parts []PartETag, precond PutPrecondition) (PutResult, error)
}

ConditionalMultipartCompleter can complete multipart uploads only when a write precondition holds atomically at the provider.

type ConditionalPutter

type ConditionalPutter interface {
	PutObjectConditional(ctx context.Context, key string, body io.Reader, contentLength int64, precond PutPrecondition) (PutResult, error)
}

ConditionalPutter can create/replace objects only when a write precondition holds atomically at the provider. Implementations that expose this capability must honor every supported PutPrecondition predicate atomically. Unsupported validated predicates must fail with ErrUnsupportedPrecondition rather than falling back to an unconditional write or returning ErrPreconditionFailed, which means a supported predicate evaluated false.

type ConditionalWriteCapabilities

type ConditionalWriteCapabilities struct {
	// IfAbsent reports that PutObjectConditional honors an IfAbsent precondition.
	IfAbsent bool
	// IfMatchETag reports that PutObjectConditional honors an IfMatchETag
	// precondition (atomic compare-and-swap against the destination ETag).
	IfMatchETag bool
	// ConditionalMultipartCompletion reports that CompleteMultipartUploadConditional
	// honors its precondition, so a large conditional overwrite completes
	// atomically instead of uploading parts and discovering the predicate is
	// unsupported only at completion time.
	ConditionalMultipartCompletion bool
}

ConditionalWriteCapabilities enumerates the conditional-write predicates a destination provider honors atomically. It exists so a caller can validate predicate support up front — before any read or destination mutation — instead of inferring support from the presence of ConditionalPutter, which cannot distinguish an IfAbsent-only implementation from one that also honors IfMatch or conditional multipart completion. A provider that advertises a predicate here promises to honor it (or return ErrPreconditionFailed when the predicate evaluates false); it must never fall back to an unconditional write or return ErrUnsupportedPrecondition for an advertised predicate.

type ConnectionPoolPolicy

type ConnectionPoolPolicy struct {
	// MaxIdleConnsPerHost is idle retention capacity per host. Zero leaves
	// SDK / transport defaults.
	MaxIdleConnsPerHost int
	// MaxConnsPerHost is the hard cap across active, dialing, and idle
	// connections per host. Zero leaves SDK / transport defaults.
	MaxConnsPerHost int
}

ConnectionPoolPolicy is the numbers-only HTTP connection pool policy for a single provider construction. It does not create clients, mutate http.DefaultTransport or http.DefaultClient, or share state across constructions. Callers apply these integers to concrete provider configs (for example s3.Config / gcs.Config MaxIdleConnsPerHost and MaxConnsPerHost).

API stability: Stable (additive). Breaking changes follow docs/api-stability.md.

func ResolveConnectionPool

func ResolveConnectionPool(admittedN int) (ConnectionPoolPolicy, error)

ResolveConnectionPool returns pool settings for a construction that will run up to admittedN concurrent provider operations that each need at most one simultaneous HTTP connection on this client.

Semantics (locked GON-065 S0):

  • admittedN < 0 → error
  • admittedN 0 or 1 → zero policy (SDK / transport defaults)
  • admittedN >= 2 → both fields equal admittedN (pass-through, including large N)

Callers must resolve engine defaults and clamps before calling this function. Never pass a raw "unset" flag value of 0 when the run will default concurrency to a higher value. Composite admitted-N formulas (transfer source C+L, etc.) must use checked arithmetic outside this resolver and refuse construction on overflow; this function does not apply a second soft cap.

type DelimiterLister

type DelimiterLister interface {
	ListWithDelimiter(ctx context.Context, opts ListWithDelimiterOptions) (*ListWithDelimiterResult, error)
}

DelimiterLister supports delimiter-based listing.

This is used for safe, directory-like operations (tree/du) and prefix discovery. Delimiter listing returns:

  • Objects directly under Prefix (no nested delimiter in the remainder)
  • CommonPrefixes (immediate child prefixes)

Implementations should map to provider-native delimiter listing when available (e.g., S3 ListObjectsV2 with Delimiter).

type ListCommonPrefixesOptions

type ListCommonPrefixesOptions struct {
	Prefix            string
	Delimiter         string
	ContinuationToken string
	MaxKeys           int
}

type ListCommonPrefixesResult

type ListCommonPrefixesResult struct {
	Prefixes          []string
	ContinuationToken string
	IsTruncated       bool
}

type ListOptions

type ListOptions struct {
	// Prefix filters results to keys starting with this value.
	// Empty string lists all objects.
	Prefix string

	// ContinuationToken resumes listing from a previous ListResult.
	// Empty string starts from the beginning.
	ContinuationToken string

	// MaxKeys limits the number of objects returned per page.
	// Zero uses provider default (typically 1000).
	MaxKeys int
}

ListOptions configures a List operation.

type ListResult

type ListResult struct {
	// Objects contains the object summaries for this page.
	Objects []ObjectSummary

	// ContinuationToken is used to retrieve the next page.
	// Empty string indicates no more pages.
	ContinuationToken string

	// IsTruncated indicates whether more results are available.
	IsTruncated bool
}

ListResult contains a page of objects from a List operation.

type ListWithDelimiterOptions

type ListWithDelimiterOptions struct {
	// Prefix filters results to keys starting with this value.
	Prefix string

	// Delimiter groups keys (e.g., "/").
	Delimiter string

	// ContinuationToken resumes listing from a previous ListWithDelimiterResult.
	ContinuationToken string

	// MaxKeys limits the number of keys returned per page.
	MaxKeys int
}

ListWithDelimiterOptions configures a delimiter listing operation.

type ListWithDelimiterResult

type ListWithDelimiterResult struct {
	// Objects are object summaries directly under the requested Prefix.
	Objects []ObjectSummary

	// CommonPrefixes are the immediate child prefixes.
	CommonPrefixes []string

	// ContinuationToken is used to retrieve the next page.
	ContinuationToken string

	// IsTruncated indicates whether more results are available.
	IsTruncated bool
}

ListWithDelimiterResult contains a page of results from a delimiter listing.

type MetadataAwareMultipartUploader

type MetadataAwareMultipartUploader interface {
	CreateMultipartUploadWithOptions(ctx context.Context, key string, opts PutOptions) (uploadID string, err error)
}

MetadataAwareMultipartUploader can start multipart uploads with the same destination object attributes as a single-part PUT.

type MetadataAwarePutter

type MetadataAwarePutter interface {
	PutObjectWithOptions(ctx context.Context, key string, body io.Reader, contentLength int64, opts PutOptions) error
	PutObjectConditionalWithOptions(ctx context.Context, key string, body io.Reader, contentLength int64, precond PutPrecondition, opts PutOptions) (PutResult, error)
}

MetadataAwarePutter can persist caller-controlled metadata, content type, and storage class on both unconditional and conditional PUTs.

type MetadataAwareResultPutter

type MetadataAwareResultPutter interface {
	PutObjectResultWithOptions(ctx context.Context, key string, body io.Reader, contentLength int64, opts PutOptions) (PutResult, error)
}

MetadataAwareResultPutter is the result-bearing unconditional metadata PUT capability. Existing error-only capabilities remain supported.

type MultipartUploader

type MultipartUploader interface {
	CreateMultipartUpload(ctx context.Context, key string) (uploadID string, err error)
	UploadPart(ctx context.Context, key, uploadID string, partNumber int32, body io.Reader, size int64) (PartETag, error)
	CompleteMultipartUpload(ctx context.Context, key, uploadID string, parts []PartETag) (PutResult, error)
	AbortMultipartUpload(ctx context.Context, key, uploadID string) error
}

MultipartUploader can create, upload, complete, and abort multipart uploads.

CreateMultipartUpload+AbortMultipartUpload also provide a low-side-effect write probe when supported.

type ObjectDeleter

type ObjectDeleter interface {
	DeleteObject(ctx context.Context, key string) error
}

ObjectDeleter can delete objects.

For v0.1.x this is primarily used for write-probe preflight operations.

type ObjectGetter

type ObjectGetter interface {
	GetObject(ctx context.Context, key string) (body io.ReadCloser, contentLength int64, err error)
}

ObjectGetter can download objects as a stream.

For v0.1.x this is used for streaming transfer operations and stream helpers.

type ObjectMeta

type ObjectMeta struct {
	ObjectSummary

	// Version is the provider-specific version handle, when available.
	// For S3 this is the VersionId. Providers without native object versions
	// may leave it empty.
	Version string

	// ContentType is the MIME type of the object.
	ContentType string

	// Metadata contains user-defined metadata key-value pairs.
	Metadata map[string]string

	// StorageClass is the provider-native object storage class. Providers may
	// leave it empty when the concept is unsupported or unknown.
	StorageClass string

	// ArchiveStatus is the provider-native archive access status observed via
	// HEAD, when supplied. For S3 this is typically ARCHIVE_ACCESS or
	// DEEP_ARCHIVE_ACCESS.
	ArchiveStatus string

	// RestoreState is the normalized restore state observed via HEAD. Supported
	// values are ongoing, completed, expired, and unknown.
	RestoreState string

	// RestoreExpiry is the provider-reported restore expiry timestamp, when
	// available.
	RestoreExpiry *time.Time
}

ObjectMeta contains full metadata for a single object. Returned by Head operations.

type ObjectPutter

type ObjectPutter interface {
	PutObject(ctx context.Context, key string, body io.Reader, contentLength int64) error
}

ObjectPutter can create/overwrite objects.

For v0.1.x this is primarily used for write-probe preflight operations.

type ObjectRanger

type ObjectRanger interface {
	GetRange(ctx context.Context, key string, start, endInclusive int64) (body io.ReadCloser, contentLength int64, err error)
}

ObjectRanger can download a specific byte range of an object.

This is the foundational primitive for content inspection operations.

start and endInclusive are inclusive offsets following HTTP Range semantics. Implementations SHOULD return the content length of the returned range.

type ObjectSummary

type ObjectSummary struct {
	// Key is the full object key (path) in the bucket.
	Key string

	// Size is the object size in bytes.
	Size int64

	// ETag is the entity tag, typically an MD5 hash of the object.
	ETag string

	// Revision is a provider-native immutable version or generation returned by
	// LIST, when available. It is opaque outside the provider adapter.
	Revision string

	// LastModified is when the object was last modified.
	LastModified time.Time

	// StorageClass is the provider-native storage class returned by LIST, when
	// available. Providers may leave it empty when unsupported or omitted.
	StorageClass string
}

ObjectSummary contains basic metadata returned from List operations.

type PartETag

type PartETag struct {
	PartNumber int32
	ETag       string
}

PartETag identifies one successfully uploaded multipart part.

type PrefixLister

type PrefixLister interface {
	ListCommonPrefixes(ctx context.Context, opts ListCommonPrefixesOptions) (*ListCommonPrefixesResult, error)
}

PrefixLister supports delimiter-based prefix discovery.

For S3 this maps to ListObjectsV2 with a Delimiter.

type Provider

type Provider interface {
	// List returns a page of objects with the given prefix.
	// Use ContinuationToken from ListResult for subsequent pages.
	List(ctx context.Context, opts ListOptions) (*ListResult, error)

	// Head returns metadata for a single object.
	// Returns ErrNotFound if the object does not exist.
	Head(ctx context.Context, key string) (*ObjectMeta, error)

	// Close releases any resources held by the provider.
	Close() error
}

Provider abstracts cloud storage listing operations.

Implementations should:

  • Use SDK default credential chains (AWS default config, GCP ADC)
  • Support pagination via continuation tokens
  • Be safe for concurrent use

type ProviderError

type ProviderError struct {
	// Op is the operation that failed (e.g., "List", "Head").
	Op string

	// Provider is the provider type (e.g., "s3").
	Provider ProviderType

	// Bucket is the bucket name, if applicable.
	Bucket string

	// Key is the object key, if applicable.
	Key string

	// Err is the underlying error.
	Err error
}

ProviderError wraps provider-specific errors with context.

func (*ProviderError) Error

func (e *ProviderError) Error() string

Error implements the error interface.

func (*ProviderError) Unwrap

func (e *ProviderError) Unwrap() error

Unwrap returns the underlying error for errors.Is/As support.

type ProviderType

type ProviderType string

ProviderType identifies a cloud storage provider.

const (
	// ProviderS3 represents AWS S3 or S3-compatible storage.
	ProviderS3 ProviderType = "s3"

	// ProviderGCS represents Google Cloud Storage (future).
	ProviderGCS ProviderType = "gcs"

	// ProviderFile represents local filesystem storage.
	ProviderFile ProviderType = "file"
)

func (ProviderType) String

func (p ProviderType) String() string

String returns the string representation of the provider type.

type PutOptions

type PutOptions struct {
	// UserMetadata contains user-defined object metadata keys without provider
	// wire prefixes such as x-amz-meta-. Keys should already be canonicalized by
	// callers that expose user input.
	UserMetadata map[string]string

	// ContentType is the destination object's content type. Empty means provider
	// default.
	ContentType string

	// StorageClass is the destination object's provider-native storage class.
	// Empty means provider default.
	StorageClass string
}

PutOptions carries optional destination object attributes for providers that can persist caller-controlled metadata on PUT.

func (PutOptions) Empty

func (o PutOptions) Empty() bool

Empty returns true when no optioned destination attributes are requested.

type PutPrecondition

type PutPrecondition struct {
	IfAbsent    bool
	IfMatchETag *string
}

PutPrecondition describes the predicate that must hold for a conditional put.

Exactly one predicate must be set. Callers that want unconditional replacement should use ObjectPutter.PutObject instead.

func (PutPrecondition) Validate

func (p PutPrecondition) Validate() error

Validate checks that exactly one precondition predicate is set.

type PutResult

type PutResult struct {
	ETag    string
	Version string
}

PutResult contains provider version handles returned by a successful put.

type ResultPutter

type ResultPutter interface {
	PutObjectResult(ctx context.Context, key string, body io.Reader, contentLength int64) (PutResult, error)
}

ResultPutter optionally returns handles from the acknowledged unconditional write itself. Callers must not reconstruct these handles with a later HEAD.

type RevisionGetter

type RevisionGetter interface {
	GetObjectRevision(ctx context.Context, key string, revision SourceRevision) (body io.ReadCloser, meta ObjectMeta, err error)
}

RevisionGetter reads exactly the admitted source revision or fails closed. A moved validator returns an error wrapping ErrSourceChanged; an adapter that cannot enforce the supplied token returns ErrReplayUnverifiable.

type RevisionKind

type RevisionKind string

RevisionKind identifies the provider-neutral enforcement shape of an opaque source revision token.

const (
	RevisionNative RevisionKind = "native"
	RevisionETag   RevisionKind = "etag"
)

type SourceRevision

type SourceRevision struct {
	Kind  RevisionKind
	Value string
}

SourceRevision is a non-secret, opaque source revision admitted by a coordinator. Provider adapters interpret Value according to Kind.

func (SourceRevision) Validate

func (r SourceRevision) Validate() error

Validate refuses revision tokens that cannot be enforced.

type VersionedGetter

type VersionedGetter interface {
	GetObjectVersioned(ctx context.Context, key string) (body io.ReadCloser, meta ObjectMeta, err error)
}

VersionedGetter can download an object with a version handle that belongs to the bytes returned by the same read operation.

Directories

Path Synopsis
Package file implements the provider interface for local filesystem paths.
Package file implements the provider interface for local filesystem paths.
Package gcs defines the Google Cloud Storage provider contract.
Package gcs defines the Google Cloud Storage provider contract.
Package s3 implements the provider interface for AWS S3 and S3-compatible object storage.
Package s3 implements the provider interface for AWS S3 and S3-compatible object storage.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL