s3

package
v0.4.7 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 9, 2026 License: Apache-2.0 Imports: 16 Imported by: 0

Documentation

Overview

Package s3 implements the provider interface for AWS S3 and S3-compatible object storage.

Library consumers should construct providers with Config and New. A Config struct literal is passive: it does not read environment variables or shared AWS config files. Provider construction calls the AWS SDK v2 default loader, so environment and shared-config reads happen during New when the SDK resolves credentials, region, profile, and configured endpoints.

Credential resolution is explicit: Anonymous unsigned reads, caller-injected Config.CredentialsProvider, static keys, Profile, then the AWS SDK default chain. Anonymous is read-only, sends no Authorization header, and is rejected by all mutating methods with provider.ErrAnonymousReadOnly joined with provider.ErrAccessDenied. Explicit AccessKeyID and SecretAccessKey values suppress SDK credential-chain lookup for credentials only. They do not suppress ambient region, profile, or endpoint configuration. Consumers that need hermetic endpoint behavior should either pass a non-empty Config.Endpoint or set AWS_IGNORE_CONFIGURED_ENDPOINT_URLS=true in their own process. See docs/library-consumers.md for the full embedded-use contract.

API stability: Stable. Breaking changes to exported symbols or documented behavior follow the Library API protocol in docs/api-stability.md.

Index

Constants

View Source 
const DefaultAWSRegion = "us-east-1"

DefaultAWSRegion is the fallback region for AWS S3 when not specified.

View Source 
const DefaultMaxKeys = 1000

DefaultMaxKeys is the default page size for List operations.

View Source 
const MaxAllowedKeys = 1000

MaxAllowedKeys is the maximum page size allowed by S3.

Variables

This section is empty.

Functions

This section is empty.

Types

type Config 

type Config struct {
	// Bucket is the S3 bucket name (required).
	Bucket string

	// Region is the AWS region.
	// For AWS S3: defaults to us-east-1 if not specified via config or environment.
	// For S3-compatible (when Endpoint is set): no default applied.
	Region string

	// Endpoint is a custom endpoint URL for S3-compatible stores.
	// Leave empty for AWS S3.
	// Examples:
	//   - Wasabi: https://s3.wasabisys.com
	//   - MinIO: http://localhost:9000
	//   - DigitalOcean: https://nyc3.digitaloceanspaces.com
	Endpoint string

	// Profile is the AWS profile name to use from shared config.
	// Leave empty to use the default profile or environment credentials.
	Profile string

	// Anonymous issues unsigned read requests for public buckets. It supports
	// List, Head, GetObject, GetObjectVersioned, and GetRange only. Mutating
	// methods fail closed with provider.ErrAnonymousReadOnly joined with
	// provider.ErrAccessDenied. Anonymous is mutually exclusive with
	// CredentialsProvider, AccessKeyID/SecretAccessKey, and Profile, and it
	// never falls back to ambient environment, profile, or instance credentials.
	Anonymous bool

	// CredentialsProvider overrides credential resolution with caller-managed
	// AWS credentials. It takes precedence over AccessKeyID/SecretAccessKey,
	// Profile, and the SDK default chain. When set, Profile is not loaded for
	// credentials or profile-derived non-credential config such as region; pass
	// Region, Endpoint, and ForcePathStyle directly when those values are needed.
	CredentialsProvider aws.CredentialsProvider

	// AccessKeyID is an explicit access key. If set, SecretAccessKey must also be set.
	// This takes precedence over Profile and the default credential chain.
	AccessKeyID string

	// SecretAccessKey is an explicit secret key. Required if AccessKeyID is set.
	SecretAccessKey string

	// ForcePathStyle forces path-style URLs (bucket in path, not subdomain).
	// Required for most S3-compatible stores and useful for local development.
	// AWS S3 uses virtual-hosted style by default.
	ForcePathStyle bool

	// MaxKeys is the default page size for List operations.
	// Zero uses the provider default (1000). Values over 1000 are clamped.
	MaxKeys int

	// MaxIdleConnsPerHost optionally sizes the per-host idle HTTP connection
	// pool for high-concurrency transfer paths. Zero leaves SDK defaults.
	MaxIdleConnsPerHost int

	// MaxConnsPerHost optionally caps total HTTP connections per host. Zero
	// leaves SDK defaults.
	MaxConnsPerHost int
}

Config configures an S3 provider.

Authentication priority:

  1. Anonymous unsigned reads when Anonymous is true; this is mutually exclusive with every credential source and never falls back to ambient credentials.
  2. Caller-injected CredentialsProvider, if provided.
  3. Explicit AccessKeyID/SecretAccessKey, if provided.
  4. Shared config Profile, if provided.
  5. AWS SDK v2 default credential chain.

Region handling:

  • For AWS S3: If Region is empty and not set via environment/profile, defaults to us-east-1 (standard AWS convention).
  • For S3-compatible stores: Region is typically ignored by the endpoint, but some providers (e.g., Wasabi) may require it. When Endpoint is set, no default region is applied.

For S3-compatible stores (Wasabi, MinIO, DigitalOcean Spaces), set Endpoint and typically ForcePathStyle.

For hermetic embedded use, remember that explicit credentials only suppress credential-chain lookup. If Endpoint is empty, AWS SDK configured endpoint sources such as AWS_ENDPOINT_URL, AWS_ENDPOINT_URL_S3, and shared-config endpoint_url can still redirect requests unless the embedding process sets AWS_IGNORE_CONFIGURED_ENDPOINT_URLS=true.

func (Config) GoString 

func (c Config) GoString() string

GoString returns a redacted representation for %#v formatting.

func (Config) String 

func (c Config) String() string

String returns a redacted representation of the config suitable for logs.

func (*Config) Validate 

func (c *Config) Validate() error

Validate checks that required configuration is present.

type ConfigError 

type ConfigError struct {
	Field   string
	Message string
}

ConfigError represents a configuration validation error.

func (*ConfigError) Error 

func (e *ConfigError) Error() string

Error implements the error interface.

type Provider 

type Provider struct {
	// contains filtered or unexported fields
}

Provider implements provider.Provider for AWS S3 and S3-compatible storage.

func New 

func New(ctx context.Context, cfg Config) (*Provider, error)

New creates a new S3 provider with the given configuration.

The provider uses AWS SDK v2's default credential chain unless explicit credentials are provided in the config.

func (*Provider) AbortMultipartUpload 

func (p *Provider) AbortMultipartUpload(ctx context.Context, key, uploadID string) error

AbortMultipartUpload aborts a multipart upload.

func (*Provider) Close 

func (p *Provider) Close() error

Close releases any resources held by the provider. The S3 client doesn't require explicit cleanup, but this satisfies the interface.

func (*Provider) CompleteMultipartUpload 

func (p *Provider) CompleteMultipartUpload(ctx context.Context, key, uploadID string, parts []provider.PartETag) (provider.PutResult, error)

func (*Provider) CompleteMultipartUploadConditional 

func (p *Provider) CompleteMultipartUploadConditional(ctx context.Context, key, uploadID string, parts []provider.PartETag, precond provider.PutPrecondition) (provider.PutResult, error)

func (*Provider) ConditionalWriteCapabilities 

func (p *Provider) ConditionalWriteCapabilities() provider.ConditionalWriteCapabilities

ConditionalWriteCapabilities declares the conditional-write predicates this S3 adapter honors: IfAbsent (If-None-Match: *), IfMatch (If-Match: etag) on both single-PUT and conditional multipart completion. The remote endpoint behind a custom Endpoint remains the documented trust boundary.

func (*Provider) CreateMultipartUpload 

func (p *Provider) CreateMultipartUpload(ctx context.Context, key string) (string, error)

CreateMultipartUpload starts a multipart upload.

This is used for minimal-side-effect write probes.

func (*Provider) CreateMultipartUploadWithOptions 

func (p *Provider) CreateMultipartUploadWithOptions(ctx context.Context, key string, opts provider.PutOptions) (string, error)

func (*Provider) DeleteObject 

func (p *Provider) DeleteObject(ctx context.Context, key string) error

DeleteObject deletes an object.

This is used for write-probe preflight and future move operations.

func (*Provider) GetObject 

func (p *Provider) GetObject(ctx context.Context, key string) (io.ReadCloser, int64, error)

GetObject downloads an object as a stream.

Caller must close the returned body.

func (*Provider) GetObjectRevision 

func (p *Provider) GetObjectRevision(ctx context.Context, key string, revision provider.SourceRevision) (io.ReadCloser, provider.ObjectMeta, error)

GetObjectRevision downloads exactly the admitted native version or ETag.

func (*Provider) GetObjectVersioned 

func (p *Provider) GetObjectVersioned(ctx context.Context, key string) (io.ReadCloser, provider.ObjectMeta, error)

GetObjectVersioned downloads an object and returns the version metadata from the same S3 GetObject response.

func (*Provider) GetRange 

func (p *Provider) GetRange(ctx context.Context, key string, start, endInclusive int64) (io.ReadCloser, int64, error)

GetRange downloads an object byte range as a stream.

Caller must close the returned body.

func (*Provider) Head 

func (p *Provider) Head(ctx context.Context, key string) (*provider.ObjectMeta, error)

Head returns metadata for a single object.

func (*Provider) List 

func (p *Provider) List(ctx context.Context, opts provider.ListOptions) (*provider.ListResult, error)

List returns a page of objects with the given prefix.

func (*Provider) ListCommonPrefixes 

func (p *Provider) ListCommonPrefixes(ctx context.Context, opts provider.ListCommonPrefixesOptions) (*provider.ListCommonPrefixesResult, error)

func (*Provider) ListWithDelimiter 

func (p *Provider) ListWithDelimiter(ctx context.Context, opts provider.ListWithDelimiterOptions) (*provider.ListWithDelimiterResult, error)

func (*Provider) PutObject 

func (p *Provider) PutObject(ctx context.Context, key string, body io.Reader, contentLength int64) error

PutObject uploads an object.

This is used for write-probe preflight and future transfer operations.

func (*Provider) PutObjectConditional 

func (p *Provider) PutObjectConditional(ctx context.Context, key string, body io.Reader, contentLength int64, precond provider.PutPrecondition) (provider.PutResult, error)

func (*Provider) PutObjectConditionalWithOptions 

func (p *Provider) PutObjectConditionalWithOptions(ctx context.Context, key string, body io.Reader, contentLength int64, precond provider.PutPrecondition, opts provider.PutOptions) (provider.PutResult, error)

func (*Provider) PutObjectEmpty 

func (p *Provider) PutObjectEmpty(ctx context.Context, key string) error

PutObjectEmpty uploads a 0-byte object.

This helper exists for probe operations.

func (*Provider) PutObjectResult 

func (p *Provider) PutObjectResult(ctx context.Context, key string, body io.Reader, contentLength int64) (provider.PutResult, error)

func (*Provider) PutObjectResultWithOptions 

func (p *Provider) PutObjectResultWithOptions(ctx context.Context, key string, body io.Reader, contentLength int64, opts provider.PutOptions) (provider.PutResult, error)

func (*Provider) PutObjectWithOptions 

func (p *Provider) PutObjectWithOptions(ctx context.Context, key string, body io.Reader, contentLength int64, opts provider.PutOptions) error

func (*Provider) UploadPart 

func (p *Provider) UploadPart(ctx context.Context, key, uploadID string, partNumber int32, body io.Reader, size int64) (provider.PartETag, error)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.