s3

package
v0.1.3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 5, 2026 License: MIT Imports: 19 Imported by: 0

Documentation

Overview

Package s3 is the small client of the S3 protocol the replicator uses: PUT, multipart upload, ranged GET, ListObjectsV2 and DELETE, signed with SigV4, against AWS S3 and the stores that speak its protocol (Cloudflare R2, MinIO and the like).

Every body is signed by its SHA-256, which the store checks, and no request asks for the checksums newer AWS SDKs add, which other stores do not all take. Redirects are not followed: a request goes to the endpoint it was given or fails. No error carries a secret.

Index

Constants

This section is empty.

Variables

View Source
var ErrNoAnswer = errors.New("s3: no answer")

ErrNoAnswer is a request that got no answer: the connection failed or the answer was cut. It may have taken effect or not. A PUT with IfNoneMatch sent again after it can fail with ErrPreconditionFailed because of its own first try: Head tells.

View Source
var ErrNotFound = errors.New("s3: no such key")

ErrNotFound is the answer of a GET of a key the bucket does not have. A missing bucket is another error.

View Source
var ErrPreconditionFailed = errors.New("s3: precondition failed")

ErrPreconditionFailed is a 412 answer: an object written with IfNoneMatch is there already.

Functions

func Retryable

func Retryable(err error) bool

Retryable says whether err may pass when the request is sent again: no answer, a 5xx, 429 (slow down), or 409 (a conditional write to the same key at the same time, which AWS asks to retry).

Types

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client talks to one bucket. It is safe for concurrent use.

func New

func New(cfg Config) (*Client, error)

New checks cfg and makes a client. It makes no request.

The bucket goes in the host (virtual-hosted style) on an amazonaws.com endpoint when its name has no dot, and in the path otherwise, which the other stores all take.

func (*Client) CreateBucket

func (c *Client) CreateBucket(ctx context.Context) error

CreateBucket makes the client's bucket, in its region.

func (*Client) Delete

func (c *Client) Delete(ctx context.Context, key string) error

Delete deletes the object at key. A key with no object is no error.

func (Client) Format

func (c Client) Format(f fmt.State, verb rune)

Format prints where the client goes, and nothing of its credentials.

func (*Client) Get

func (c *Client) Get(ctx context.Context, key string, off, n int64) (io.ReadCloser, error)

Get reads the object at key: n bytes from byte off, or all from off when n < 0. The answer is shorter than n only when the object ends first. A range that starts at or past the end, also of an empty object, fails with a 416. The caller closes the reader.

func (*Client) Head

func (c *Client) Head(ctx context.Context, key string) (Info, error)

Head reads the size and metadata of the object at key. Its 404 is ErrNotFound: a HEAD answer has no body to say NoSuchKey, so a missing bucket reads the same.

func (*Client) List

func (c *Client) List(ctx context.Context, prefix, startAfter string) iter.Seq2[Object, error]

List gives the objects whose keys start with prefix and come after startAfter (when not empty), in key order, a page of up to 1000 at a time. It stops at the first error.

func (*Client) Put

func (c *Client) Put(ctx context.Context, key string, body io.ReaderAt, size int64, opt PutOptions) error

Put writes size bytes of body as the object at key. An object larger than a part goes up in parts (a multipart upload), each part hashed and sent from body in turn, so nothing is held in memory. A failed multipart upload is aborted, but for one whose start got no answer, which has no ID to abort: a lifecycle rule of the bucket (AbortIncompleteMultipartUpload) removes its parts. body must not change during the call.

type Config

type Config struct {
	// Endpoint is the scheme and host of the store, such as
	// https://s3.us-east-1.amazonaws.com or
	// https://<account>.r2.cloudflarestorage.com, with no path.
	Endpoint string
	// Region is the region requests are signed for: "auto" for R2, and
	// often "us-east-1" for a store that has no regions.
	Region string
	Bucket string
	// Credentials sign each request.
	Credentials CredentialsProvider
	// AllowHTTP lets the endpoint be plain http, such as a local MinIO.
	// Without it only https is taken.
	AllowHTTP bool
}

Config says where the bucket is and how to sign.

type Credentials

type Credentials struct {
	AccessKeyID     string
	SecretAccessKey string
	SessionToken    string // empty for long-term keys
}

Credentials sign requests. Every fmt verb on them gives the access key ID and hides the rest. fmt calls no method on a field that is not exported, so a struct that keeps them in such a field prints them in full: keep them in a provider (Static), which holds them behind a pointer.

func (Credentials) Format

func (c Credentials) Format(f fmt.State, verb rune)

Format prints the credentials without the secret and the token.

type CredentialsProvider

type CredentialsProvider interface {
	Credentials(ctx context.Context) (Credentials, error)
}

A CredentialsProvider gives the credentials of each request, so that credentials that expire can be renewed. It is called for every request and caches what it fetches.

func Static

Static gives fixed credentials. They are kept behind a pointer, so that a struct that holds the provider prints an address and not the secret, however deep fmt looks.

type EnvCredentials

type EnvCredentials struct{}

EnvCredentials reads AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN at each request.

func (EnvCredentials) Credentials

Credentials reads the variables.

type Error

type Error struct {
	Op         string // GET, PUT, LIST, DELETE, or a step of a multipart upload
	Key        string
	StatusCode int
	Code       string // S3's error code, such as NoSuchKey, if the answer had one
	Message    string
	RequestID  string
}

Error is an answer of the store that is not a success.

func (*Error) Error

func (e *Error) Error() string

func (*Error) Is

func (e *Error) Is(target error) bool

Is matches ErrPreconditionFailed and ErrNotFound.

type Info

type Info struct {
	Size         int64
	LastModified time.Time
	ETag         string
	Metadata     map[string]string // by the names given to Put
}

Info is what Head gives of an object.

type Object

type Object struct {
	Key          string
	Size         int64
	LastModified time.Time
	ETag         string
}

Object is an object as a list gives it.

type PutOptions

type PutOptions struct {
	// IfNoneMatch writes the object only if the key has none: Put fails
	// with ErrPreconditionFailed when it has.
	IfNoneMatch bool
	// Multipart sends the object as a multipart upload even when it fits
	// one PUT, as one part: a probe of whether the store makes the
	// completion of an upload conditional too.
	Multipart bool
	// Metadata is stored with the object as x-amz-meta-<name>. Names are
	// lower case letters, digits and '-', values printable ASCII.
	Metadata map[string]string
}

PutOptions are the options of Put.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL