Documentation
¶
Overview ¶
Package rigs3 keeps a rig project's uploads in S3, or in anything speaking its API.
It is blob.Store and both of its optional halves — blob.Signer and blob.Marker — and it is a module of its own for one reason: the AWS SDK. github.com/simonjanss/rig/files depends on the runtime and a UUID, and every generated application imports it, so a project keeping its uploads in memory must not compile an S3 client to do it. Depending on this module is therefore the same decision as writing `backend: s3` in rig.yaml, and it is made in one place — the `files.gen.go` that `server-go` writes.
The name has an `s3` in it twice on purpose. This package's whole job is to import github.com/aws/aws-sdk-go-v2/service/s3, and a package called `s3` importing `s3` would have to alias one of them in every file.
What the bucket has to allow ¶
Reading and writing objects, and reading and writing their tags: the checksum and the deleted mark are both object tags, for reasons given on Store.Put and Store.Mark. New also reads the bucket's lifecycle configuration once, at startup, and refuses to build a store when a rule would expire objects sooner than the restore window says they are restorable.
Index ¶
- type Config
- type Store
- func (s *Store) Delete(ctx context.Context, key string) error
- func (s *Store) Get(ctx context.Context, key string) (io.ReadSeekCloser, error)
- func (s *Store) Mark(ctx context.Context, key string, state blob.State, at time.Time) error
- func (s *Store) Put(ctx context.Context, key string, r io.Reader, opt blob.PutOptions) (blob.Info, error)
- func (s *Store) SignUpload(ctx context.Context, key string, expires time.Duration) (string, error)
- func (s *Store) Stat(ctx context.Context, key string) (blob.Info, error)
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Config ¶
type Config struct {
// Bucket is the bucket objects go in. Required: there is no default worth
// guessing and a wrong one is somebody else's data.
Bucket string
// Region is the bucket's region. Required for AWS, which will not guess
// one; with an Endpoint set it defaults to us-east-1, which is what
// S3-compatible services that have no regions expect to be told.
Region string
// Endpoint points at something other than AWS — MinIO, R2, a test double.
//
// Setting it also turns on path-style addressing, because virtual-host
// style asks DNS to resolve `<bucket>.<endpoint>` and almost nothing but
// AWS answers that. It is not a separate setting for the same reason: there
// is no combination of the two worth writing down.
Endpoint string
// AccessKeyID and SecretAccessKey are static credentials. Both empty means
// the AWS default chain — an instance profile, IRSA, SSO, the environment —
// which is what a real deployment usually wants and what a laptop with
// `aws configure` already has.
AccessKeyID string
SecretAccessKey string
// RestoreWindow is `files.restore_window`: how long a deleted file stays
// restorable, and therefore how long its bytes have to survive the delete.
//
// It is here so that [New] can refuse a bucket whose own lifecycle rule
// would take them sooner. Zero means the check is skipped, which is what a
// program that is not rig's generated wiring should pass.
RestoreWindow time.Duration
}
Config is what a store needs to know about the bucket it writes to.
The credentials are values rather than the names of environment variables, which is the opposite of how rig.yaml carries them: `files.s3.access_key_env` names a variable because rig.yaml is a file people commit, and reading it is the generated code's job. This struct is the other side of that read, and a program building a store by hand should not have to invent an environment to pass a string through.
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store is a bucket, as blob.Store.
func New ¶
New builds a store, and refuses rather than returning one that cannot keep the promises rig makes about it.
It fails on a missing bucket, on a bucket that names neither a region nor an endpoint, and — the one worth knowing about — on a bucket whose lifecycle configuration would expire objects sooner than Config.RestoreWindow says they are restorable. That last one costs a round trip at startup and buys the failure arriving on the first deploy instead of a month later, when somebody undeletes a file and gets back a row pointing at nothing.
func (*Store) Delete ¶
Delete removes the object and says nothing about a key that was already gone.
S3 answers a delete for a key it does not have with a success, which is the behaviour the interface asks for and the reason the sweeper can be a loop rather than a state machine.
func (*Store) Get ¶
Get opens the object for reading, at any offset.
The reader seeks without asking the bucket anything: http.ServeContent seeks to the end and back again to find out how long the object is before it serves a byte, and a range request that cost two round trips to answer would make scrubbing a video a conversation. The bytes are fetched on the first read after a seek, as one ranged GET.
func (*Store) Mark ¶
Mark records on the object what the row already says.
It reads the tag set before writing it, because S3 replaces the whole set and the checksum Store.Put wrote lives in it. Two calls rather than one, on a path that runs once per delete and once per trashed file per sweep — and the alternative is a store whose own blob.Info.Checksum disappears the first time somebody deletes a file.
The mark is a projection of the row and never the other way round. See blob.Marker for what that costs and why it is the safe direction.
func (*Store) Put ¶
func (s *Store) Put(ctx context.Context, key string, r io.Reader, opt blob.PutOptions) (blob.Info, error)
Put streams the object into the bucket and reports what went past.
The reader is read once, through a hash, which is what makes an upload larger than memory an ordinary upload: nothing here holds the object, and the parts the SDK buffers are bounded by its part size. It also matters that the reader is read exactly once and never rewound — the one rig hands over refuses rather than truncating at the byte cap, and a retry that re-read it would be asking a request body to happen twice.
Why the checksum is a tag ¶
blob.Info.Checksum is a SHA-256 that cannot be known until the last byte has gone past, and object metadata is fixed when the upload begins. So it is written afterwards, as an object tag: one cheap call, no size limit, and it does not rewrite the object.
The two alternatives are worse. Copying the object onto itself to replace its metadata needs a multipart copy past 5 GiB. And S3's own SHA-256 is a checksum of checksums as soon as an upload is multipart, which is the ETag problem blob.Info exists to avoid, stated again in a different field.
A failed tagging fails the Put, rather than leaving an object rig cannot answer questions about. The cost is an upload thrown away; what it buys is that every object in the bucket carries its checksum and its state, which is the whole reason Store.Mark exists. The row it belongs to is still pending at this point, so the sweeper's first rule reaps what is left behind.
func (*Store) SignUpload ¶
SignUpload mints a URL a client can PUT the bytes to itself.
There is no signed download beside it, and that is a decision rather than an omission: a signed URL bypasses the endpoint, and the endpoint is where the permission check lives. See blob.Signer.
An object uploaded this way arrives with none of the tags Store.Put writes, because nothing rig runs saw the bytes. Whatever finalizes such an upload has to write them.
func (*Store) Stat ¶
Stat reports what the bucket knows about the object without opening it.
The checksum comes back from the object's tags, so an object rig did not write has an empty one rather than an invented one. Nothing in github.com/simonjanss/rig/files calls this; it is here because a store that could not answer it would be a store a range request had to guess at.
Source Files