Documentation
¶
Overview ¶
Package s3 is a Storage Driver fronting any S3-compatible object store.
Tested against AWS S3, Hetzner Object Storage (path-style endpoint, nbg1.your-objectstorage.com), MinIO, Backblaze B2 (S3 compat), and Cloudflare R2. Multipart uploads are exposed via the optional MultipartUploader interface for browser-direct chunked uploads.
Index ¶
- type Driver
- func (d *Driver) AbortMultipart(ctx context.Context, p, uploadID string) error
- func (d *Driver) Capabilities() storage.Capabilities
- func (d *Driver) CompleteMultipart(ctx context.Context, p, uploadID string, parts []storage.PartCompletion) error
- func (d *Driver) Copy(ctx context.Context, src, dst string) error
- func (d *Driver) Delete(ctx context.Context, p string) error
- func (d *Driver) Init(ctx context.Context, cfg map[string]any) error
- func (d *Driver) InitMultipart(ctx context.Context, p string, _ int64, partCount int) (string, []string, error)
- func (d *Driver) List(ctx context.Context, p string) ([]storage.Object, error)
- func (d *Driver) Mkdir(ctx context.Context, p string) error
- func (d *Driver) Move(ctx context.Context, src, dst string) error
- func (d *Driver) Name() string
- func (d *Driver) PresignDownload(ctx context.Context, p string, ttl time.Duration) (string, error)
- func (d *Driver) PresignUpload(ctx context.Context, p string, _ int64) (storage.PresignedUpload, error)
- func (d *Driver) Read(ctx context.Context, p string) (io.ReadCloser, error)
- func (d *Driver) ReadRange(ctx context.Context, p string, off, length int64) (io.ReadCloser, error)
- func (d *Driver) Stat(ctx context.Context, p string) (storage.Object, error)
- func (d *Driver) UploadPart(ctx context.Context, p, uploadID string, partNumber int, r io.Reader, ...) (string, error)
- func (d *Driver) WalkTree(ctx context.Context, p string, fn func(storage.Object) error) error
- func (d *Driver) Write(ctx context.Context, p string, r io.Reader, size int64) error
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Driver ¶
type Driver struct {
// contains filtered or unexported fields
}
Driver is the S3 storage driver.
func (*Driver) AbortMultipart ¶
AbortMultipart implements storage.MultipartUploader.
func (*Driver) Capabilities ¶
func (d *Driver) Capabilities() storage.Capabilities
Capabilities — S3 supports everything except Watch (notifications go via SQS/EventBridge, not implemented in this skeleton).
func (*Driver) CompleteMultipart ¶
func (d *Driver) CompleteMultipart(ctx context.Context, p, uploadID string, parts []storage.PartCompletion) error
CompleteMultipart implements storage.MultipartUploader.
func (*Driver) Copy ¶
Copy implements storage.Copier (server-side). Directories are copied recursively (same prefix rationale as Move). The CopySource header MUST be URL-encoded — the AWS SDK does not do it for you — or keys with spaces or non-ASCII characters (e.g. Turkish filenames) 404 as NoSuchKey. A genuinely missing source maps to storage.ErrNotFound so a delete can treat it as already-done rather than failing the whole batch.
func (*Driver) Delete ¶
Delete implements storage.Deleter. For a directory it removes every object under the prefix — S3 has no folders, so a single DeleteObject on the bare prefix key would be a no-op and orphan the contents.
func (*Driver) Init ¶
Init configures the driver.
Required: bucket, region, access_key, secret_key. Optional: endpoint (Hetzner: https://nbg1.your-objectstorage.com),
path_style (Hetzner needs true), prefix (storage root prefix).
func (*Driver) InitMultipart ¶
func (d *Driver) InitMultipart(ctx context.Context, p string, _ int64, partCount int) (string, []string, error)
InitMultipart implements storage.MultipartUploader.
func (*Driver) Mkdir ¶
Mkdir writes the hidden .empty keep-marker so an empty folder exists on the object store (and shows as a directory) without any visible child. The marker is filtered from listings and moved/removed with the folder.
func (*Driver) Move ¶
Move implements storage.Mover (copy + delete). A directory is moved as a unit by recursing the prefix: CopyObject cannot operate on a bare prefix, so the old single-object move 404'd on any folder (empty or not) — the S3 folder-delete/rename bug this method fixes.
func (*Driver) PresignDownload ¶
PresignDownload implements storage.Presigner.
func (*Driver) PresignUpload ¶
func (d *Driver) PresignUpload(ctx context.Context, p string, _ int64) (storage.PresignedUpload, error)
PresignUpload implements storage.Presigner — returns a single PUT URL. For multipart, use InitMultipart instead.
func (*Driver) ReadRange ¶
ReadRange implements storage.RangeReader with a `Range:` header on GetObject. The server does the seeking, so no bytes before off ever cross the wire.
An offset at or past the end answers 416 InvalidRange on S3; per the RangeReader contract that is a short read, not a failure, so it maps to an empty (EOF) reader.
func (*Driver) Stat ¶
Stat implements storage.Driver.
404 from HeadObject (NotFound / NoSuchKey) is mapped to storage.ErrNotFound so the manager handler can surface it as a clean 404 instead of a 500. Every other error keeps its original message.
func (*Driver) UploadPart ¶
func (d *Driver) UploadPart(ctx context.Context, p, uploadID string, partNumber int, r io.Reader, size int64) (string, error)
UploadPart implements storage.PartUploader — the server-side half of multipart, used by the staged upload path. (The browser-direct flow uses the presigned URLs from InitMultipart and never comes through here.)
size is always known at this point (it comes from the staging manifest), so the body goes out with a Content-Length instead of chunked.
func (*Driver) WalkTree ¶
WalkTree implements storage.TreeWalker: every object below p in ONE un-delimited ListObjectsV2 pass (1,000 keys a page), with the directories synthesised from the keys the way List synthesises them from CommonPrefixes.
The rules are List's, applied to a whole subtree at once: the bare prefix key is not an object, the hidden `.empty` marker is not a file but does prove its folder exists, and a key ending in "/" (a folder object written by another tool) is a directory, not a file. Each ancestor directory is reported exactly once, before any object under it, so a caller grouping by parent never meets a child whose folder it has not seen.
func (*Driver) Write ¶
Write implements storage.Writer.
The body MUST go out with a Content-Length. When the SDK can't measure it — a plain io.Reader with no Seeker, which is what the upload handler produces after mime-sniffing — it falls back to Transfer-Encoding: chunked. AWS and MinIO accept that, but the S3 spec leaves it to the provider, and DT Cloud S3 answers `411 MissingContentLength`, which broke every browser upload while WebDAV and MCP (both of which hand over a seekable body) kept working. So: declare the length we were given, and when the caller genuinely doesn't know it, measure the body first.
The body must also be REWINDABLE, or the retry budget newRetryer buys is a no-op for uploads: the SDK cannot resend what it cannot rewind, and a transient 503 becomes a permanent failure reported as "failed to rewind transport stream for retry, request stream is not seekable" — a message about our plumbing rather than the outage that caused it. See rewindable.
⚠⚠ Rewinding is not only a retry concern. Over an http:// endpoint the signer hashes the payload and rewinds to send it, so a body that cannot rewind fails the FIRST attempt (issue #27). A body too large to hold in memory and unable to rewind therefore goes out as a multipart upload in parts that can be held — see writeInParts.
Source Files