Documentation
¶
Overview ¶
Package files is the half of an upload that is the same in every project.
A file in rig is two things that cannot be written together: a row in rig_file, and an object in a blob.Store. Everything here is about keeping those two in step — which order they are written in, which one leads when they disagree, and what a sweeper is allowed to remove. That is the part every project gets subtly wrong, and it is the part rig takes. What a file *means* — derivatives, retention policy, whether a photo needs a thumbnail — is where products diverge hardest and stays yours.
The generated server calls Service. The generated client never sees this module at all: a program that calls a rig application depends on rigclient, and rigclient knows about multipart bodies and nothing about buckets.
Which way round the writes go ¶
The row leads, always. A file row is inserted with a null uploaded_at and committed alone, the bytes are streamed to the store, and then one transaction finalizes the row and writes the owner. Every read filters `uploaded_at IS NOT NULL`, so a row with no bytes behind it is invisible and reapable with one query. Bytes with no row would need a bucket scan to find.
The single transaction at the end is what keeps the sweeper's two rules sufficient. Finalize the file and write the owner separately and a crash between them leaves a file that is uploaded and referenced by nothing — which neither rule catches, and which would force exactly the unreferenced-file reaper this design refuses to have.
Index ¶
- Constants
- Variables
- type AttachRequest
- type Config
- type Content
- type DB
- type File
- type Owner
- type Pending
- type Service
- func (s *Service) Attach(ctx context.Context, req AttachRequest) (*File, error)
- func (s *Service) Commit(ctx context.Context, pending []*Pending, write func(ctx context.Context) error, ...) error
- func (s *Service) Detach(ctx context.Context, tenantID uuid.UUID, o Owner, fileID uuid.UUID) error
- func (s *Service) Get(ctx context.Context, tenantID, fileID uuid.UUID) (*File, error)
- func (s *Service) InlineTypes() []string
- func (s *Service) MaxBytes() int64
- func (s *Service) Open(ctx context.Context, tenantID, fileID uuid.UUID, name string) (*Content, error)
- func (s *Service) Prepare(ctx context.Context, part string, req AttachRequest) (*Pending, error)
- func (s *Service) Restore(ctx context.Context, tenantID, fileID uuid.UUID) (*File, error)
- func (s *Service) Sweep(ctx context.Context) (SweepReport, error)
- type SweepReport
- type Upload
Constants ¶
const ( // DefaultAbandonedAfter is how long a row with no bytes behind it gets the // benefit of the doubt before it is treated as the remains of a request that // died. DefaultAbandonedAfter = 24 * time.Hour // DefaultRestoreWindow is how long a deleted file stays restorable, and // therefore how long its bytes outlive the delete. DefaultRestoreWindow = 30 * 24 * time.Hour )
Defaults for the two intervals the sweeper reads.
const DefaultMaxBytes int64 = 25 << 20
DefaultMaxBytes is the largest single upload a project accepts unless it says otherwise. It is a hard per-file cap and not a quota: rig does not do storage quotas, and saying so is better than implying one.
const Table = "rig_file"
Table is the managed table every uploaded file has a row in. Spelled once, here, because it is the only name in this package a migration also has to know.
Variables ¶
var DefaultInlineTypes = []string{
"image/png", "image/jpeg", "image/gif", "image/webp", "image/avif",
"application/pdf",
}
DefaultInlineTypes are served without an attachment disposition.
The list is short on purpose, and text/html must never join it. A file served inline from the API origin runs there, and the URL is on a row a client syncs, so it will end up in an <img> or an <a> without anybody thinking about it. Everything not named here downloads.
var ErrNotFound = errors.New("files: no such file")
ErrNotFound is what a lookup answers for a file that is not there, has not finished uploading, or belongs to another tenant.
One error for all three, deliberately. Distinguishing them would tell a caller that a file exists and is somebody else's, which is exactly what a tenant boundary is for.
var ErrTooLarge = errors.New("files: the upload is larger than this endpoint accepts")
ErrTooLarge is what a store's Put sees when an upload runs past the cap. A caller gets rigerr.CodeTooLarge rather than this.
Functions ¶
This section is empty.
Types ¶
type AttachRequest ¶
type AttachRequest struct {
TenantID uuid.UUID
Upload Upload
// Owner is the row the file hangs off. Its identifier is zero for a file
// being uploaded as part of the create that will hold it — see
// [Service.Prepare].
Owner Owner
// URL builds the download path once the file has an identifier and a name.
// The endpoint's own route is what it renders, which is why a generator
// supplies it rather than this package guessing.
URL func(fileID uuid.UUID, name string) string
}
AttachRequest is one upload: what arrived, where it goes, and what its URL will be.
type Config ¶
type Config struct {
// Store is where the bytes go.
Store blob.Store
// DB is the pool the owning tables are in. The same one, deliberately: the
// transaction that finalizes a file and writes its owner has to be one
// transaction, and it cannot be if the two live in different pools.
DB DB
// MaxBytes caps one upload. Zero means [DefaultMaxBytes].
MaxBytes int64
// InlineTypes are the sniffed types served without an attachment
// disposition. Nil means [DefaultInlineTypes]; an explicitly empty slice
// means everything downloads.
InlineTypes []string
// AbandonedAfter and RestoreWindow are the sweeper's two rules. Zero means
// the defaults.
AbandonedAfter time.Duration
RestoreWindow time.Duration
// Now is the clock, so a test can move it. Nil means time.Now.
Now func() time.Time
// Logger is where [Service.Sweep] says what it reaped.
//
// Nil is not silence: it is [log/slog.Default], the reading every other
// Logger in rig gives it. [SweepReport] exists for "the log line the task
// writes" and the generated task discards it, so what a project's
// housekeeping actually did has never been written anywhere.
//
// The line is INFO, unlike the per-interval lines rig's two background
// loops write. Nothing sweeps files in-process — [Service.Sweep] is reached
// from the generated `sweep-files` cron job and nowhere else — so this is
// the whole output of a run that happens when nobody is watching, and
// [log/slog.Default] drops DEBUG. A level that has to have been turned on
// in advance would leave a sweep that did nothing looking exactly like a
// cron entry that never fired.
//
// A sweep that failed is not logged here at all: the error is returned, and
// whatever ran the task reports it once.
Logger *slog.Logger
}
Config is everything a Service needs that is not a request.
It is resolved from the `files:` block in rig.yaml and written into files.gen.go by server-go, so a byte cap or a sweep interval is a line in a file the generated documentation can quote rather than a literal in a main function nobody diffs.
type Content ¶
type Content struct {
File *File
Body io.ReadSeekCloser
}
Content is a file on its way out: the row, and the bytes still in the store.
Body is the store's reader and is the caller's to close. Nothing reads ahead, which is what lets a file larger than memory go straight to a response — and it is why it seeks, because http.ServeContent cannot answer a range request without that.
type DB ¶
DB is the pool the file rows and their owners live in. See dbx.Pool for why it is the two interfaces rather than *pgxpool.Pool.
type File ¶
type File struct {
ID uuid.UUID
TenantID uuid.UUID
StorageKey string
// URL is where the file is served from — the download route, with the
// identifier and the name in it. Stable and unsigned, so holding it grants
// nothing and the endpoint behind it still authorizes.
URL string
Name string
// ContentType is what the bytes were sniffed to be, which is what a download
// announces.
ContentType string
// DeclaredType is what the client claimed, kept beside the sniffed type
// rather than instead of it.
DeclaredType string
Size int64
Checksum string
CreatedAt time.Time
UploadedAt *time.Time
DeletedAt *time.Time
}
File is one row of rig_file, as the server holds it.
It is not the shape a client sees. StorageKey, Checksum and DeclaredType stay here: the storage key is what a signed URL is built from, and putting it anywhere a client can reach is the same class of mistake as syncing a password hash.
type Owner ¶
type Owner struct {
Table string
// IDColumn is the owning table's primary key, normally "id".
IDColumn string
// TenantColumn scopes the update. Empty for a table with no tenant column,
// which is not a shape rig generates but is one a test can build.
TenantColumn string
// FileColumn is the `<role>_file_id` column the identifier is written to.
FileColumn string
ID uuid.UUID
// Forget withdraws whatever is being held of this row, and it exists because
// this is the one place rig writes an application's table from outside that
// table's generated repository.
//
// A table that holds its rows between requests is invalidated by the writes
// its repository makes, and the statement below is not one of them: attaching
// a file changes the row's `<role>_file_id` and every replica holding it would
// go on saying there is no file there — which the download endpoint answers as
// a 404, for a file that was just uploaded successfully.
//
// Generated code fills this in for a table that asked to be held, and it is
// called inside the transaction that writes the column, so the withdrawal is
// atomic with the change and discarded by a rollback. Nil is the ordinary
// case and means nothing is held.
Forget func(ctx context.Context) error
}
Owner names the row a file hangs off: the table, the row, and the column.
The table and column are written by a generator from the document, never by a request, which is what makes it safe for Service to build a statement out of them.
type Pending ¶
type Pending struct {
File *File
// Part is the form part this came from, so a generated create can put each
// identifier on the right column.
Part string
}
Pending is a file whose bytes have landed and whose row is not final: the second half of a create that carried its file with it.
It exists because a create is one transaction and an upload cannot be in it. The bytes go first, outside; then the row and its files are committed together by Service.Commit, which is what makes a not-null file column expressible at all.
type Service ¶
type Service struct {
// contains filtered or unexported fields
}
Service is what a generated service layer calls.
Every method takes the tenant explicitly rather than reading it off the context, because the caller has already resolved the owning row through a repository that scoped the read — and passing the same tenant through means the two cannot come apart in a way only a test with two tenants would notice.
func (*Service) Attach ¶
Attach stores an upload and points an existing row at it.
Three steps, in this order and no other. The row is inserted with no bytes behind it and committed alone; the bytes are streamed to the store; and then one transaction finalizes the row and writes the owning column. A failure anywhere leaves at worst a pending row, which is invisible to every read and which the sweeper reaps.
Replacing a file does not delete the one it replaces. A table that keeps its previous versions still points at the old file from every one of them, and deleting it on replace would corrupt exactly the history the snapshot exists to keep.
That last transaction also withdraws whatever is held of the owning row, when the row is one that is held — see Owner.Forget. It is the same transaction because it has to be.
func (*Service) Commit ¶
func (s *Service) Commit( ctx context.Context, pending []*Pending, write func(ctx context.Context) error, url func(p *Pending) string, ) error
Commit runs a write and finalizes the files it carried, in one transaction.
One transaction is the whole point. Commit the row and the files separately and a crash between them leaves files that are uploaded and referenced by nothing — a state neither sweeper rule catches, and the reason this design refuses to have an unreferenced-file reaper at all. Fail anywhere and the files stay pending, which is invisible and reapable.
write runs first because the URL cannot be built until the owning row has an identifier, which is also why url is a function rather than a field: for a create, the row it points at does not exist when the bytes are stored.
func (*Service) Detach ¶
Detach clears a file column and retires the file it pointed at.
The row leads and the mark follows: the deletion is committed first, then the object is marked, and the mark is best-effort. A failed mark leaves a deleted row beside an unmarked object, which is the safe direction — the sweeper still knows from the row and re-marks anything out of step on its next pass. Marking first would produce an object tagged deleted that the database says is live, and nothing reconciles that back.
func (*Service) InlineTypes ¶
InlineTypes are the sniffed types this service serves without an attachment disposition.
func (*Service) MaxBytes ¶
MaxBytes is the cap this service applies to one upload, so a handler can set up its own limit reader from the same number.
func (*Service) Open ¶
func (s *Service) Open(ctx context.Context, tenantID, fileID uuid.UUID, name string) (*Content, error)
Open reads a file back.
The name is checked against the stored one and a mismatch is a not-found: the identifier is the lookup, and the name in the path is for the browser's save dialog and for cache busting. It never goes near the storage key.
A deleted file inside the restore window still opens, because a restore has to hand back a row pointing at something and the bytes outlive the delete for exactly that reason.
func (*Service) Prepare ¶
Prepare stores an upload that a create is carrying, and hands back the file without attaching it to anything.
The row is not visible yet: uploaded_at is still null, so nothing reads it and the sweeper will take it if the create never lands.
func (*Service) Restore ¶
Restore brings a retired file back, inside the window.
Past the window the bytes are gone, and answering anything but a refusal would hand back a row pointing at nothing.
func (*Service) Sweep ¶
func (s *Service) Sweep(ctx context.Context) (SweepReport, error)
Sweep is the whole of rig's file housekeeping: two rules, and no third.
**Abandoned uploads.** A row with a null uploaded_at is either an upload in flight or the remains of a request that died. Past `files.abandoned_after` it is the second, and both the object and the row go.
**Trash past the window.** A soft-deleted file stays restorable for `files.restore_window`, and its bytes have to outlive the delete for exactly that long — a restore inside the window that handed back a row pointing at nothing would be worse than refusing. Past it, both go.
There is deliberately **no unreferenced-file reaper**. Finding those means enumerating every foreign key pointing at rig_file, and the failure mode of getting that wrong is deleting somebody's data. A file is immutable and is never deleted because the thing referencing it changed: a table with snapshots copies the whole row on every update, so after three picture changes the first file is referenced by three version rows, and reaping it on replace would corrupt the history.
It is a task rather than a goroutine, so it is a subcommand in a cron job rather than something racing itself in every replica.
type SweepReport ¶
type SweepReport struct {
// Abandoned is the number of rows reaped because their bytes never arrived.
Abandoned int
// Expired is the number reaped because their restore window closed.
Expired int
// Remarked is the number of deleted objects the bucket had not been told
// about, brought back into step.
Remarked int
}
SweepReport is what one pass did, for the log line the task writes.
func (SweepReport) String ¶
func (r SweepReport) String() string
String is the one line a sweep is worth in a log: all three counts, including the zeros. A pass that reaped nothing is the ordinary case and still worth seeing, because the absence of a line cannot be told from the job not running.
type Upload ¶
type Upload struct {
// Name is what the file is called. It goes on the row and into the download
// path, and it never becomes the storage key: see [blob.Key].
Name string
// DeclaredType is the type the client claimed. It is recorded and then
// largely ignored — the sniffed type is the one that is stored and served.
DeclaredType string
Body io.Reader
}
Upload is a file on its way in.
Body is read once, streamed straight to the store, and never buffered: an upload larger than memory is an ordinary upload. Whoever produced it closes it — for the generated handler that is the request body, which net/http already owns.
Directories
¶
| Path | Synopsis |
|---|---|
|
Package blob is where the bytes live.
|
Package blob is where the bytes live. |
|
Package filehttp is the HTTP shape of an upload and a download.
|
Package filehttp is the HTTP shape of an upload and a download. |
|
Package foundation carries the schema the files module's table needs.
|
Package foundation carries the schema the files module's table needs. |