Help us shape the future of Go! Take the 2026 Go Developer Survey

s3

package
v0.5.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 3, 2026 License: MIT Imports: 41 Imported by: 0

Documentation

Overview

Package s3 implements AWS S3, an explicit Cloudflare R2 profile and a generic S3-compatible profile using the official AWS SDK. Buckets are pre-existing; this package never provisions or deletes buckets, changes ACLs, or changes their access policies.

Index

Constants

View Source 
const MaxPartBytes int64 = 64 << 20
View Source 
const MaxParts = 10000
View Source 
const MaxSingleUploadBytes int64 = 5 << 30

MaxSingleUploadBytes is S3's single-request PutObject limit, which bounds a presigned upload.

View Source 
const MinPartBytes int64 = 5 << 20

Variables

This section is empty.

Functions

func CredentialsModule 

func CredentialsModule(name foundation.ProviderID, key foundation.Key[*Backend], config Config, requires []foundation.ProviderID, provider func(foundation.Resolver) (credentials.Provider, error)) foundation.Module

CredentialsModule resolves a shared framework credential provider during pure construction. Its owner must appear in requires for reverse-order cleanup.

func Module 

func Module(name foundation.ProviderID, key foundation.Key[*Backend], config Config) foundation.Module

Module prepares an adapter without I/O and loads its SDK configuration during boot. Disk modules must require this provider so their readers drain before transport cleanup.

Types

type Backend 

type Backend struct {
	// contains filtered or unexported fields
}

Backend borrows supplied credentials/transports. A Disk owns operation bounds; MaxUploads additionally bounds shared multipart buffers across all disks using this adapter. Close disks and await Done before closing this backend.

func Open 

func Open(ctx context.Context, config Config) (*Backend, error)

Open prepares and starts an adapter, closing it if startup fails.

func Prepare 

func Prepare(config Config) (*Backend, error)

Prepare validates configuration and allocates upload admission without loading credentials or making provider requests.

func (*Backend) AbortUpload 

func (b *Backend) AbortUpload(ctx context.Context, ref UploadReference) error

AbortUpload affects only the referenced staging upload, never a published object. Keys being written by this adapter are rejected, even before an upload ID is known. Other-process owners must be stopped by the operator first; storage age is not a lease.

func (*Backend) Capabilities 

func (b *Backend) Capabilities() storage.Capabilities

Capabilities returns guarantees selected by the configured provider profile.

func (*Backend) Close 

func (b *Backend) Close() error

Close stops upload admission and closes owned idle connections; callers must drain disks first.

func (*Backend) Copy 

func (b *Backend) Copy(ctx context.Context, source, target storage.ObjectKey, options storage.CopyOptions, maximum int64) (storage.ObjectInfo, error)

Copy implements storage.ServerCopier with one CopyObject pinned to the source's ETag (and version). It declines as Unsupported/Unchanged before any mutation when the streamed path is required: a destination condition, a source above 5 GiB, a declared checksum the source metadata cannot prove, or a compatible profile that has not declared ServerCopy. CopyObject is a mutation and is never retried automatically.

func (*Backend) Delete 

func (b *Backend) Delete(ctx context.Context, key storage.ObjectKey, options storage.DeleteOptions) error

Delete sends one validated provider deletion and preserves uncertain outcomes; it does not retry publication.

func (*Backend) DeleteBatch 

func (b *Backend) DeleteBatch(ctx context.Context, keys []storage.ObjectKey) ([]error, error)

DeleteBatch implements storage.BatchDeleter. AWS uses one DeleteObjects request; R2 and compatible profiles delete sequentially because their DeleteObjects checksum support differs. Results align with keys. Deleting a key on a versioned bucket adds a delete marker, as a single Delete does.

func (*Backend) List 

func (b *Backend) List(ctx context.Context, options storage.ListOptions) (storage.Page, error)

List builds entries from one ListObjectsV2 page without per-object requests. Listing metadata has no media type or full SHA-256, so those stay zero; use Stat for complete metadata. Entries that cannot be framework objects (foreign or unparsable keys, oversized objects, missing validators) are skipped and counted in Page.Skipped instead of failing the page. Delimited listings map CommonPrefixes to Page.Directories.

func (*Backend) ListUploads 

func (b *Backend) ListUploads(ctx context.Context, options storage.ListOptions) (UploadPage, error)

ListUploads only inspects bounded staging pages. Age alone does not prove an upload is abandoned. Its owner must be stopped or otherwise reconciled before AbortUpload is called; this API never automatically prunes by age.

func (*Backend) Locate 

func (b *Backend) Locate(key storage.ObjectKey) (storage.ObjectAddress, error)

Locate returns provider, bucket and namespaced object identity without provider I/O.

func (*Backend) Open 

func (b *Backend) Open(ctx context.Context, key storage.ObjectKey, options storage.ReadOptions) (io.ReadCloser, storage.ReadInfo, error)

Open returns an owned response body after validating metadata, selectors and any returned range.

func (*Backend) PublicURL 

func (b *Backend) PublicURL(ctx context.Context, key storage.ObjectKey) (string, error)

PublicURL constructs a stable URL from the explicit public base and namespaced key without changing access policy.

func (*Backend) Put 

func (b *Backend) Put(ctx context.Context, key storage.ObjectKey, source io.Reader, options storage.PutOptions) (storage.ObjectInfo, error)

Put consumes the input exactly once. A declared Size below PartBytes retains only that many bytes for a single PutObject; larger or unknown-length inputs use pooled PartBytes buffers. PartConcurrency 1 uploads multipart parts serially with one buffer; N > 1 keeps up to N parts in flight while one more buffer fills. MaxUploads bounds active uploads across disks and queues briefly before reporting overload. Only replayable UploadPart calls retry; Create, PutObject and Complete never retry ambiguous mutations.

func (*Backend) Start 

func (b *Backend) Start(ctx context.Context) error

Start loads SDK configuration and creates the client once; a closed adapter cannot restart.

func (*Backend) Stat 

func (b *Backend) Stat(ctx context.Context, key storage.ObjectKey, options storage.ReadOptions) (storage.ObjectInfo, error)

Stat reads and validates complete metadata with replayable HEAD retries; byte ranges are rejected.

func (*Backend) TemporaryURL 

func (b *Backend) TemporaryURL(ctx context.Context, key storage.ObjectKey, options storage.LinkOptions) (storage.TemporaryURL, error)

TemporaryURL signs a read URL bounded by the requested lifetime and any known credential expiry.

func (*Backend) TemporaryUploadForm 

func (b *Backend) TemporaryUploadForm(ctx context.Context, key storage.ObjectKey, options storage.UploadFormOptions) (storage.UploadForm, error)

TemporaryUploadForm presigns a browser POST-policy upload. The policy pins the exact bucket and key, the exact Content-Type, a content-length-range and optional checksum metadata; the returned fields include every value the policy requires. R2 does not implement POST-object uploads, and a compatible profile must declare FormUploads.

func (*Backend) TemporaryUploadURL 

func (b *Backend) TemporaryUploadURL(ctx context.Context, key storage.ObjectKey, options storage.UploadLinkOptions) (storage.UploadLink, error)

TemporaryUploadURL presigns one PutObject with its exact length, media type, optional checksum metadata and optional absence condition. Every signed header except Host is returned for the client to send unchanged.

func (*Backend) UploadFromCleanup 

func (b *Backend) UploadFromCleanup(id storage.CleanupID) (UploadReference, error)

UploadFromCleanup validates a reference against this exact endpoint, bucket and namespace. A lost creation response without an upload ID returns Unsupported: inspect ListUploads and establish ownership before aborting.

type CompatibleCapabilities 

type CompatibleCapabilities struct {
	ConditionalCreate, ConditionalReplace, ConditionalDelete bool
	Versions                                                 bool
	SingleRequestConditions                                  bool
	RequiresNFCKeys                                          bool
	ServerCopy                                               bool
	ListingPathEncoding                                      bool
	FormUploads                                              bool
}

CompatibleCapabilities declares what a Compatible provider guarantees. Only declare a feature the service actually enforces; an undeclared option fails as Unsupported before provider I/O. Ranges and conditional reads are always required. SingleRequestConditions limits conditional writes to one PutObject (declared size within PartBytes), as for providers without conditional multipart completion. ListingPathEncoding selects path-style decoding of url-encoded listing keys instead of AWS form encoding ('+' as space). FormUploads declares support for browser POST-policy uploads.

type Config 

type Config struct {
	// PublicBase points to the bucket root; Namespace is appended automatically.
	PublicBase     storage.PublicBase
	Provider       Provider
	Bucket         string
	Region         string
	Endpoint       string
	Namespace      storage.Prefix
	Profile        string
	Credentials    aws.CredentialsProvider
	HTTPClient     aws.HTTPClient
	PathStyle      bool
	PartBytes      int64
	MaxUploads     int
	MaxObjectBytes int64
	ReadAttempts   int
	PartAttempts   int
	AbortTimeout   time.Duration
	// PartConcurrency bounds concurrent multipart part uploads per object.
	PartConcurrency   int
	VerifyPublication bool
	AllowHTTP         bool
	Compatible        CompatibleCapabilities
}

Config describes one bucket namespace. AWS uses its standard credential chain unless Credentials or Profile is supplied. R2 requires explicit Credentials and an HTTPS account endpoint; it never falls back to AWS machine credentials. Compatible requires an explicit Endpoint and explicit Credentials (never a Profile or the AWS default chain, which would disclose the host's AWS machine credentials to another provider); AllowHTTP opts it into a plain-HTTP endpoint for local development only (never use it across a network). HTTPClient, when supplied, is borrowed and must honor request cancellation. Only general-purpose AWS buckets are supported; directory buckets and access point ARNs have different protocol semantics and are rejected.

Each active upload retains one PartBytes buffer when PartConcurrency is 1 and up to PartConcurrency+1 (parts in flight plus the one filling) above that, or a single smaller buffer for a declared Size below PartBytes; MaxUploads bounds active uploads across all disks sharing this backend and queues briefly before failing as overloaded. VerifyPublication pins a HEAD to the acknowledged ETag/version to read the provider's Last-Modified; when disabled or refused (write-only credentials), the acknowledged publication is returned with the response Date as Modified.

func CompatibleConfig 

func CompatibleConfig(bucket, region, endpoint string, capabilities CompatibleCapabilities) Config

CompatibleConfig selects a generic S3-compatible endpoint with explicitly declared capabilities. Path-style addressing is used by default.

func DefaultConfig 

func DefaultConfig(bucket, region string) Config

DefaultConfig returns an AWS profile with bounded multipart uploads, retries and publication verification.

func R2Config 

func R2Config(bucket, endpoint string, credentials aws.CredentialsProvider) Config

R2Config returns an R2 profile with explicit endpoint and credentials and path-style addressing.

func R2WithCredentials 

func R2WithCredentials(bucket, endpoint string, provider credentials.Provider) Config

R2WithCredentials supplies explicit credentials without native SDK types.

func (Config) Validate 

func (c Config) Validate() error

Validate rejects invalid provider, bucket, namespace, credentials, endpoint and resource settings without provider I/O.

func (Config) WithCredentials 

func (c Config) WithCredentials(provider credentials.Provider) Config

WithCredentials selects a borrowed framework provider, shared with other disks/mailers. Keep it alive until this adapter has drained and closed.

type PendingUpload 

type PendingUpload struct {
	Reference UploadReference
	Initiated temporal.DateTime
}

PendingUpload pairs a staging upload reference with its provider initiation time.

type Provider 

type Provider uint8

Provider selects AWS, R2 or explicitly declared S3-compatible behavior. 

const (
	AWS Provider = iota + 1
	R2
	// Compatible is a generic S3-compatible service such as MinIO, DigitalOcean
	// Spaces or Backblaze B2. Nothing is inferred from its endpoint: optional
	// guarantees are declared explicitly in Config.Compatible.
	Compatible
)

type UploadPage 

type UploadPage struct {
	Uploads []PendingUpload
	Next    storage.Cursor
}

UploadPage holds a bounded staging listing and an optional continuation cursor.

type UploadReference 

type UploadReference struct {
	// contains filtered or unexported fields
}

UploadReference identifies one staging upload in this adapter's namespace. It grants no authorization. Obtain it from ListUploads or UploadFromCleanup; it cannot accidentally be passed as an object/version key to Delete.

func (UploadReference) Format 

func (UploadReference) Format(s fmt.State, _ rune)

Format writes a label without exposing the multipart reference.

func (UploadReference) Key 

func (r UploadReference) Key() storage.ObjectKey

Key returns the upload object key relative to the configured namespace.

func (UploadReference) MarshalJSON 

func (UploadReference) MarshalJSON() ([]byte, error)

MarshalJSON rejects implicit multipart-reference export.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.