Documentation
¶
Overview ¶
Package s3 is a storage.Storage on S3-compatible object storage: AWS S3, Cloudflare R2, MinIO, and other services that speak the S3 API.
Objects are stored under Config.Prefix + key in one bucket. Put reads the object in PartSize pieces into one buffer (sized to the object when its declared size is smaller): an object that fits in one piece is a single PutObject, a larger one a multipart upload of PartSize parts sent one after another. A Put holds at most PartSize in memory, whatever the object's size (the largest object is 10,000 parts, about 78 GiB). The object appears only when the upload completes (S3 writes are atomic).
Failed writes ¶
A Put that fails before sending the request that publishes the object (the PutObject, or the CompleteMultipartUpload) leaves the key as it was. "Sent" is observed, not assumed: the driver's HTTP client counts, per call, the requests written out in full and the responses that came back (httptrace). A call that failed before writing its request (no credentials, nothing signed, the connection refused, ctx ending first) sent nothing. Once that request is sent, only S3's answer says whether it took effect: a 4xx answer is a refusal, and the key is as it was; no answer (a dropped connection, a timeout, ctx ending), or a 5xx one, leaves the outcome unknown, and Put fails with storage.ErrUnknownOutcome. The SDK does not retry that request (every other one it does): a retry's answer says nothing about the attempt before it (a Complete that took effect and lost its answer makes its retry fail with NoSuchUpload), so only one attempt's answer is ever classified. The way to retry is to repeat the Put, which is safe. A multipart upload settles that by aborting the upload: an abort that succeeds proves the upload never completed, so the key is as it was; one that finds the upload gone means it completed, and the outcome stays unknown (the new object is most likely in place).
A failed multipart upload is aborted, on a fresh context, which removes the parts S3 has stored. It cannot remove more: a part upload that never got an answer may still be in progress at S3 and be stored after the abort (AWS documents this), and the abort itself can fail (the network still down, s3:AbortMultipartUpload not granted). In either case Put's error also carries an *AbortError naming the upload, whose parts may stay (and be billed). So any failed multipart Put may leave parts behind for a while, and only a lifecycle rule on the bucket that aborts incomplete multipart uploads after a day or so (AbortIncompleteMultipartUpload) guarantees none lingers: give the bucket one.
Permissions ¶
The credentials need s3:GetObject, s3:PutObject, s3:DeleteObject, and s3:AbortMultipartUpload on the objects (arn:aws:s3:::BUCKET/PREFIX*), and s3:ListBucket on the bucket (arn:aws:s3:::BUCKET). Without ListBucket, S3 answers a request for a missing object with 403 Access Denied rather than 404, and the driver cannot tell a missing object from a denied one. s3:ListMultipartUploadParts on the objects is optional: Fence uses it to free parts stored after an abort, and without it leaves them to a lifecycle rule (the fence itself needs only AbortMultipartUpload).
S3 carries user metadata as ASCII HTTP headers. A value that is not printable ASCII is sent RFC 2047 encoded (mime.BEncoding), as S3 expects, and decoded on the way back; storage.ValidateMetadata measures exactly those headers against the 2 KB limit.
Index ¶
- Constants
- type AbortError
- type Config
- type Store
- func (s *Store) Delete(ctx context.Context, key string) error
- func (s *Store) Fence(ctx context.Context, token string) error
- func (s *Store) List(ctx context.Context, prefix string, fn func(storage.ObjectInfo) error) error
- func (s *Store) Open(ctx context.Context, key string) (io.ReadCloser, storage.ObjectInfo, error)
- func (s *Store) PreparePublish(ctx context.Context, src, dst string) (string, error)
- func (s *Store) Publish(ctx context.Context, token string) (storage.ObjectInfo, error)
- func (s *Store) Put(ctx context.Context, key string, r io.Reader, opts storage.PutOptions) (storage.ObjectInfo, error)
- func (s *Store) Stat(ctx context.Context, key string) (storage.ObjectInfo, error)
- func (s *Store) URL(ctx context.Context, key string, opts storage.URLOptions) (string, error)
- func (s *Store) UploadURL(ctx context.Context, key string, opts storage.UploadURLOptions) (storage.UploadRequest, error)
- func (s *Store) VerifyUpload(ctx context.Context, key string) error
Constants ¶
const MaxUploadURLBytes = 5 << 30
MaxUploadURLBytes is the largest direct upload UploadURL grants: a direct upload is one presigned PutObject, and S3's limit for a single PUT is 5 GiB. (Put has no such limit: it uploads larger objects in parts.)
const PartSize = 8 << 20
PartSize is the size of each multipart upload part, and the most a Put holds in memory.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type AbortError ¶
type AbortError struct {
Key string // the S3 object key
UploadID string
Err error // the abort's failure, or errPartInFlight
}
AbortError is a multipart upload that a failed Put may have left parts of: its abort failed, or a part upload was still unanswered when it was aborted (S3 may store such a part afterwards). Err says which. The parts stay stored, and billed, until the bucket's lifecycle rule for incomplete uploads removes them, or someone aborts UploadID again. Put's error carries it (errors.As), alongside the failure it reports; it does not change how that failure is classified.
func (*AbortError) Error ¶
func (e *AbortError) Error() string
type Config ¶
type Config struct {
// Endpoint is the service's base URL; empty means AWS S3.
Endpoint string
// Region signs requests.
Region string
// Bucket holds the objects.
Bucket string
// Prefix is put before every key ("myapp/").
Prefix string
// AccessKeyID and SecretAccessKey are static credentials; both empty
// uses the AWS default credential chain.
AccessKeyID string
SecretAccessKey string
// ForcePathStyle addresses the bucket in the URL path (MinIO and most
// S3-compatible services), not the host name.
ForcePathStyle bool
// PublicPrefix makes the keys under it public (storage.IsPublic); empty
// makes none public. The bucket must serve them to anyone: a bucket
// policy allowing s3:GetObject on Bucket/Prefix+PublicPrefix*, or a
// CDN in front of it.
PublicPrefix string
// PublicURL is the URL public objects are read from, standing for the
// bucket's root: a public object's URL is PublicURL + "/" + Prefix +
// key. A CDN ("https://cdn.example.com"), a custom domain, or the
// bucket's own address. Empty: URL answers storage.ErrUnsupported for a
// public URL (signed URLs still work).
PublicURL string
// Warn, when set, is told about failures that fail no operation, such
// as parts Fence could not free after the upload was already fenced.
Warn func(msg string, err error)
}
Config configures a Store. It mirrors config.S3StorageConfig.
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store is an S3-backed storage.Storage. It is safe for concurrent use.
func New ¶
New returns a store for cfg. It makes no request; the first operation finds out whether the bucket and credentials work.
func (*Store) Delete ¶
Delete implements storage.Storage. S3 deletes are idempotent: deleting a missing object succeeds.
func (*Store) Fence ¶
Fence implements storage.Publisher with AbortMultipartUpload. S3 orders an abort and a Complete of one upload: an abort that succeeds proves the upload never completed and never will; NoSuchUpload means it is over already (completed, or aborted before). Either way nothing more can be published: that, the first abort's result, is what nil proves, and an abort that fails otherwise proves nothing and is returned.
The rest only frees storage, never a reason to fail the fence: a part request still being processed may store its part after the abort, so, as AWS advises, Fence lists the upload's parts (s3:ListMultipartUploadParts, optional) and aborts again while any are listed. A failure there is told to Config.Warn. A listing is a snapshot, and a part can still land after Fence returns; such a part is never an object (it cannot be read or published), and a bucket lifecycle rule aborting incomplete multipart uploads reclaims it.
func (*Store) List ¶
List implements storage.Lister with ListObjectsV2, in key order, a page (up to 1000 objects) at a time. ContentType and Metadata are empty: a listing does not carry them. Objects under the store's prefix whose keys are not valid storage keys (written by something else) are skipped.
func (*Store) Open ¶
func (s *Store) Open(ctx context.Context, key string) (io.ReadCloser, storage.ObjectInfo, error)
Open implements storage.Storage. The object streams from S3 as the caller reads it.
func (*Store) PreparePublish ¶
PreparePublish implements storage.Publisher: it creates a multipart upload of dst with src's content type and metadata, and copies src into its parts server-side (UploadPartCopy; the bytes stay in S3), keeping each part's ETag for the token. Nothing is published until Publish completes the upload. The token never exceeds storage.MaxPublicationToken: its size is checked as each (opaque) ETag arrives, and preparing fails as soon as the next would not fit. If preparing fails, the upload is aborted; if a part request went unanswered (S3 may still store the part after the abort) or the abort failed, the error is returned with a cleanup token (the key and upload ID only, checked to fit before any part is copied), for the caller to record and Fence.
func (*Store) Publish ¶
Publish implements storage.Publisher: it completes the prepared upload, once (a request sent without a definite answer is storage.ErrUnknownOutcome, which Fence settles), with "If-None-Match: *" where S3 honors it.
func (*Store) Put ¶
func (s *Store) Put(ctx context.Context, key string, r io.Reader, opts storage.PutOptions) (storage.ObjectInfo, error)
Put implements storage.Storage.
func (*Store) URL ¶
URL implements storage.Storage. A public URL is PublicURL + "/" + the escaped object key, for a key under PublicPrefix (storage.ErrNotPublic otherwise; storage.ErrUnsupported without a PublicURL). A signed URL is a presigned GetObject request (SigV4 query parameters) valid for opts.Expires counted from its X-Amz-Date, the signing second (see storage.SignedURL); it carries the credentials' authority, so it works for a private object, and stops working when it expires or the credentials are revoked (a URL signed with temporary credentials, such as an IAM role's, also ends when they do).
func (*Store) UploadURL ¶
func (s *Store) UploadURL(ctx context.Context, key string, opts storage.UploadURLOptions) (storage.UploadRequest, error)
UploadURL implements storage.DirectUploader: a presigned PutObject whose signature covers the length, the content type, the metadata headers, and "If-None-Match: *", so S3 refuses (403) a request that differs in any of them, and (412) one to a key that already holds an object. The client sends the returned headers; the bucket needs a CORS rule allowing PUT from the app's origin for a browser to do so.
func (*Store) VerifyUpload ¶
VerifyUpload implements storage.UploadVerifier. A grant signs the length, type, metadata and If-None-Match of the PUT; SigV4 leaves other standard headers unauthenticated, and S3 keeps five of them with the object and serves them back: Cache-Control, Content-Disposition, Content-Encoding, Content-Language and Expires. A grant never sets them, so an object that has one was given it by the client: VerifyUpload fails with storage.ErrInvalidOptions, naming them.
Source Files