Documentation
¶
Overview ¶
Package s3 implements the Destination interface for S3 and S3-compatible stores (AWS, MinIO, Wasabi, Backblaze B2, IDrive). It is create-only: there is no delete or overwrite path. Credentials come from the AWS SDK default chain, static keys (S3-compatible) are supplied via AWS_* env, never hand-resolved here.
Index ¶
- Constants
- type Backend
- func (b *Backend) Get(ctx context.Context, key string) (io.ReadCloser, error)
- func (b *Backend) LimitResponses(limit int64)
- func (b *Backend) List(ctx context.Context, prefix string) ([]dest.Object, error)
- func (b *Backend) ListPage(ctx context.Context, prefix string, limit int) ([]dest.Object, bool, error)
- func (b *Backend) ObserveRetention(ctx context.Context, key string) (dest.RetentionObservation, time.Time, error)
- func (b *Backend) PutImmutable(ctx context.Context, key string, r io.Reader, size int64, ret dest.Retention) (dest.PutResult, error)
- func (b *Backend) VerifyWorm(ctx context.Context) (dest.WormStatus, error)
- type Options
Constants ¶
const ( // MinPartSize and MaxPartSize are S3's own bounds on a part, and so on both settings. MinPartSize = 5 << 20 MaxPartSize = 5 << 30 )
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Backend ¶
type Backend struct {
// contains filtered or unexported fields
}
Backend is a create-only S3 Destination.
func (*Backend) LimitResponses ¶ added in v0.1.21
LimitResponses caps every response body b reads at limit bytes from here on: an answer that runs past it fails with dest.ErrResponseTooLarge rather than being read to its end. Call it before b is in use. A limit of zero or less leaves b as it is.
It is for a caller that reads only a store's short answers and must not be held by a long one: `gitdr doctor`, pointed at an endpoint it has no reason to trust yet. Backup never calls it, since a GetObject body is a whole bundle. Uncapped, the SDK reads all of an answer: an error body is copied into memory whole, and every body is drained to its end before it is closed.
The client is rebuilt from its own options, so everything else about it, the endpoint, the credentials, the retries and the transport's timeouts, stays as New made it.
func (*Backend) ListPage ¶ added in v0.1.21
func (b *Backend) ListPage(ctx context.Context, prefix string, limit int) ([]dest.Object, bool, error)
ListPage lists at most limit objects under prefix with one ListObjectsV2 request (read-only), and says whether the store reported more. It never sends the continuation token back, so on a bucket of a million objects it is one request, and on a store that answers every page with another one it is still one.
func (*Backend) ObserveRetention ¶ added in v0.1.16
func (b *Backend) ObserveRetention(ctx context.Context, key string) (dest.RetentionObservation, time.Time, error)
ObserveRetention asks S3 what retention is actually on an object gitdr wrote.
`GetObjectRetention` rather than `HeadObject`, and the difference decides whether this check is usable at all. HeadObject returns `x-amz-object-lock-mode` **only if the caller holds `s3:GetObjectRetention`** - without it the response is a 200 with the headers silently omitted, byte-identical to an object that carries no retention. SPEC tells every operator to scope destination credentials create/put-only, so a HEAD-based check would report "the destination accepted a write it did not retain" about correctly configured, genuinely protected buckets belonging to the operators who followed that advice. That is the unearned negative WormVerdict exists to prevent, and it would teach people to ignore the one warning that matters.
`?retention` fails distinguishably instead:
NoSuchObjectLockConfiguration the store implements the question and this object holds
nothing. An earned negative. AWS answers it with a 404 and
MinIO with a 400.
ObjectLockConfigurationNotFoundError
the same negative from Backblaze B2, a 404. AWS gives this code
to the bucket's lock question, which VerifyWorm asks; here it
answers a question about one object.
AccessDenied / NotImplemented a refusal, and a refusal is not a no.
anything else unclassified, and unclassified is not a no either.
func (*Backend) PutImmutable ¶
func (b *Backend) PutImmutable(ctx context.Context, key string, r io.Reader, size int64, ret dest.Retention) (dest.PutResult, error)
PutImmutable creates key, create-only, never overwriting or deleting. Object Lock retention is applied only when ret carries a retain-until (i.e. the bucket is immutable); on the non-WORM adoption path we write plainly, because sending lock headers to a bucket without Object Lock is a 400. An explicit CRC32 checksum is always sent: AWS and some S3-compatible stores (Backblaze B2) require Content-MD5 or an x-amz-checksum header on Object Lock PutObject requests.
The CRC32 is computed here and sent as a header, so the body goes as it is, with its length. Asked for the algorithm alone, the SDK computes the checksum on the way out, and over TLS it sends the body aws-chunked, the whole object as one chunk, with the checksum in a trailer (service/internal/checksum v1.11.3: middleware_compute_input_checksum.go, lines 163-177, and aws_chunked_encoding.go, lines 90-111). AWS takes a chunk of any size; MinIO refuses one over 16 MiB, so no object over 16 MiB could be written to MinIO over TLS. A checksum header already on the request turns the trailer off (the same file, lines 131-139 and 285-290).
func (*Backend) VerifyWorm ¶
VerifyWorm probes bucket Object Lock. A missing lock config is reported as not-locked once a listing shows the bucket exists (so the operator can override); other failures, an answer that is not S3's among them, are returned as errors.
type Options ¶
type Options struct {
Bucket string
Region string // defaults to us-east-1 (also fine for MinIO)
Endpoint string // empty = AWS; set for MinIO/Wasabi/B2/IDrive
UsePathStyle bool // true for MinIO and most S3-compatible stores
// MultipartThreshold is the size above which an object is written in parts; 0 is 4 GiB.
// PartSize is the smallest part; 0 is 64 MiB, and a larger object gets larger parts so it
// stays under 9,000 of them. Both are bounded by MinPartSize and MaxPartSize. They exist so
// a test can write in parts without writing gigabytes.
MultipartThreshold int64
PartSize int64
}
Options configures the S3 backend. Credentials are intentionally absent: they are resolved by the SDK default chain (env, IRSA/Pod Identity, instance profile, SSO).
Source Files