Documentation
¶
Overview ¶
Package s3 implements blob.Store on Amazon S3 and S3-compatible object stores (MinIO, Cloudflare R2, Ceph RGW, Backblaze B2, ...).
Credentials come from the default AWS chain (environment, shared config, web identity / IRSA, ECS and EC2 roles) unless static keys are configured. Cloudflare R2 (the bucket option on Cloudflare's free tier) is configured with endpoint https://<account>.r2.cloudflarestorage.com and an R2 API token's access key pair; the region then defaults to "auto".
A custom endpoint switches to path-style addressing and, unless AWS_REQUEST_CHECKSUM_CALCULATION / AWS_RESPONSE_CHECKSUM_VALIDATION say otherwise, only sends checksums where the API requires them (many S3-compatible stores reject the newer flexible-checksum headers).
Uploads of unknown or large size are sent as multipart uploads; ETags are the store's ETags with quotes removed. Writes use SSE-KMS when a key id is configured (presigned uploads then carry the required headers).
Presigned uploads bind the declared size into the SigV4 signature (a signed Content-Length, or for an empty object a signed Content-MD5 of the empty body), so the store rejects a body of any other size. When flexible checksums are enabled (AWS S3 by default; custom endpoints only when AWS_REQUEST_CHECKSUM_CALCULATION asks for them, since many S3-compatible stores reject the headers), a declared SHA-256 is also signed as x-amz-checksum-sha256, which S3 verifies against the body and keeps with the object; StatSHA256 (blob.DigestReporter) reads it back with a HeadObject, so callers learn the digest of such an object without reading it.
Index ¶
- Constants
- type Option
- type Store
- func (s *Store) Client() *awss3.Client
- func (s *Store) Close() error
- func (s *Store) Delete(ctx context.Context, key string) error
- func (s *Store) DeletePrefix(ctx context.Context, prefix string) error
- func (s *Store) Describe() blob.Description
- func (s *Store) Get(ctx context.Context, key string) (io.ReadCloser, *blob.ObjectInfo, error)
- func (s *Store) GetRange(ctx context.Context, key string, offset, length int64) (io.ReadCloser, *blob.ObjectInfo, error)
- func (s *Store) List(ctx context.Context, prefix, delimiter string) (*blob.ListResult, error)
- func (s *Store) Ping(ctx context.Context) error
- func (s *Store) PresignGet(ctx context.Context, key string, ttl time.Duration) (*blob.PresignedRequest, error)
- func (s *Store) PresignPut(ctx context.Context, key string, opts blob.PresignPutOptions, ...) (*blob.PresignedRequest, error)
- func (s *Store) Put(ctx context.Context, key string, r io.Reader, size int64, contentType string) (*blob.ObjectInfo, error)
- func (s *Store) Stat(ctx context.Context, key string) (*blob.ObjectInfo, error)
- func (s *Store) StatSHA256(ctx context.Context, key string) (*blob.ObjectInfo, string, error)
Constants ¶
const ( // DefaultPartSize is the multipart chunk size (and the largest object // sent with a single PutObject). DefaultPartSize = 8 << 20 // MinPartSize is S3's minimum size for all but the last part. MinPartSize = 5 << 20 )
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Option ¶
type Option func(*options)
Option configures a Store.
func WithClientOptions ¶
WithClientOptions adds raw aws-sdk-go-v2 S3 client options (applied last).
func WithHTTPClient ¶
WithHTTPClient sets the HTTP client used for S3 requests.
func WithLogger ¶
WithLogger sets the logger (default slog.Default()).
func WithPartSize ¶
WithPartSize sets the multipart part size (minimum 5 MiB).
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store is a blob.Store backed by an S3 bucket (optionally under a key prefix). It is safe for concurrent use.
func New ¶
New creates a Store from configuration. It does not contact S3; use Ping to verify access.
func (*Store) DeletePrefix ¶
DeletePrefix removes every object whose key starts with prefix using batched DeleteObjects calls. An empty prefix is rejected.
func (*Store) Describe ¶
func (s *Store) Describe() blob.Description
Describe returns the public storage description.
func (*Store) Get ¶
func (s *Store) Get(ctx context.Context, key string) (io.ReadCloser, *blob.ObjectInfo, error)
Get opens an object for reading.
func (*Store) GetRange ¶
func (s *Store) GetRange(ctx context.Context, key string, offset, length int64) (io.ReadCloser, *blob.ObjectInfo, error)
GetRange reads length bytes from offset (length < 0 = to the end). The returned ObjectInfo describes the whole object; ranges past the end yield an empty body.
func (*Store) List ¶
List returns all objects under prefix, sorted by key. With delimiter "/", keys below the next "/" are folded into Prefixes. S3 listings carry no content type, so ObjectInfo.ContentType is empty.
func (*Store) Ping ¶
Ping checks that the bucket is reachable with the configured credentials (HeadBucket).
func (*Store) PresignGet ¶
func (s *Store) PresignGet(ctx context.Context, key string, ttl time.Duration) (*blob.PresignedRequest, error)
PresignGet returns a time-limited download URL.
func (*Store) PresignPut ¶
func (s *Store) PresignPut(ctx context.Context, key string, opts blob.PresignPutOptions, ttl time.Duration) (*blob.PresignedRequest, error)
PresignPut returns a URL accepting a PUT of the object's bytes. The returned Headers are part of the signature and must be sent verbatim: Content-Type; Content-Length (opts.Size > 0) or Content-MD5 of the empty body (opts.Size == 0), so that S3 rejects a body of any other size; X-Amz-Checksum-Sha256 when opts.SHA256 is set and flexible checksums are enabled, so that S3 rejects other content (X-Amz-Sdk-Checksum-Algorithm travels in the signed query string); SSE headers with SSE-KMS. Without a size (-1) the URL accepts a body of any size.
func (*Store) Put ¶
func (s *Store) Put(ctx context.Context, key string, r io.Reader, size int64, contentType string) (*blob.ObjectInfo, error)
Put writes an object. Bodies up to the part size are sent with a single PutObject; larger or unknown-size bodies use a multipart upload. When contentType is empty it is derived from the key extension or content.
func (*Store) StatSHA256 ¶
StatSHA256 implements blob.DigestReporter: a HeadObject with checksum mode enabled returns the full-object SHA-256 that S3 verified when the object was uploaded with x-amz-checksum-sha256 (presigned uploads with a declared digest), and "" for any other object (no SHA-256 checksum, or a composite checksum of a multipart upload). It returns blob.ErrNotSupported when flexible checksums are disabled (custom endpoints by default), since many S3-compatible stores reject the checksum-mode header.
Source Files