Documentation
¶
Overview ¶
Package contracts is everything another module, an app or a test may know about files: the entity, the events, the permissions, the storage this module needs somebody else to satisfy, and the Service interface. The implementation is in ../internal.
A file is a row and a blob, and the two live in different places: the row is in Postgres, under the tenant's own policy, and the bytes are wherever Storage puts them. Everything difficult about this module comes from that split — a blob write cannot be rolled back, and a row that survives a rollback its blob did not is a download that 500s — so where each of the two happens relative to the transaction is written down at every step.
Index ¶
Constants ¶
const ( EventUploaded = "file.uploaded" EventDeleted = "file.deleted" )
The two events this module emits. There is no rest.Spec here, so there is no file.file.created either: a file arrives as bytes and leaves as bytes, and the two routes that do it publish these.
const ( VisibilityPrivate = "private" VisibilityPublic = "public" )
The two visibilities. A private file is served to a caller holding file:read; a public one is served to anybody at the public route, which is what an image in a published page has to be.
const ( MaxName = 255 MaxContentType = 120 )
MaxName is the longest file name kept. A name is a label a person reads, and the bytes are found by a uuid, so nothing depends on it being long.
MaxContentType is the column's width, and it is checked here so that a header somebody padded to a kilobyte is a 422 the caller can act on rather than a constraint violation the database raises as a 500.
const ( PermissionFileRead = "file:read" PermissionFileManage = "file:manage" )
The two permissions files have. Reading is reading a private file's bytes and listing what there is; managing is uploading and deleting. A public file's bytes need neither, which is what "public" means.
The list the manifest declares is in ../module.go, which keeps kit/module out of this package's build graph.
Variables ¶
var ErrNoBlob = errors.New("file: no bytes at this key")
ErrNoBlob is a key Storage has nothing at. It is what a Get answers with when the row says there are bytes and there are not, which is the one inconsistency the split between a row and a blob can produce.
var ErrQuota = errors.New("file: this tenant has no room left")
ErrQuota is an upload that would put this tenant past the disk it is allowed. It is 413 as well, because the caller's remedy is the same one — send something smaller, or delete something first — and a status of its own would tell an anonymous caller how much of somebody else's quota is left.
var ErrTooLarge = errors.New("file: larger than this deployment accepts")
ErrTooLarge is an upload past the limit the deployment set. It is a failure of its own rather than one of kit/crud's three, because the answer is 413 and nothing else in the application has one.
var Events = []string{EventUploaded, EventDeleted}
Events is every event this module emits, for the manifest.
Functions ¶
func Agrees ¶
Agrees refuses an upload whose declared type is one the download would render inline and whose bytes are something else. head is the first 512 bytes, which is what http.DetectContentType reads.
It checks only the renderable types, and the asymmetry is the point: a file served as an attachment is somebody else's problem to open, so what it really is does not matter here, while a file served inline runs on this tenant's origin. The sniffer is allowed to have no opinion — it knows nothing about AVIF, and half the world's images are formats it was never taught — but when it does have one and it disagrees, the upload is refused.
func Renderable ¶
Renderable reports whether a stored media type may be served inline. The parameters — a charset, a boundary — are not part of the decision, so they are stripped first: "text/html; charset=utf-8" is text/html.
Types ¶
type Deleted ¶
type Deleted struct {
FileID uuid.UUID `json:"fileId"`
StorageKey string `json:"storageKey"`
Size int64 `json:"size"`
At time.Time `json:"at"`
}
Deleted is the payload of EventDeleted, and it carries the storage key because by the time anybody handles it the row is gone. That is the whole reason this event exists: removing the bytes is work that has to happen after the transaction that removed the row commits, and an event is the only thing in this architecture that is delivered exactly then.
type File ¶
type File struct {
crud.Base
// Name is what the browser called it, kept for a person to read and used
// for nothing else.
Name string `` /* 160-byte string literal not displayed */
// ContentType is what the upload declared, and it is what the download
// answers with. It is not sniffed: a reference architecture that guessed
// would be teaching a guess, and the download is served with
// X-Content-Type-Options: nosniff so a browser does not guess either.
ContentType string `` /* 190-byte string literal not displayed */
// Size and SHA256 are what actually arrived, counted and hashed in the one
// pass that wrote the bytes.
Size int64 `json:"size" gorm:"not null" doc:"Bytes" readOnly:"true" required:"false"`
SHA256 string `json:"sha256" gorm:"type:char(64);not null" doc:"SHA-256 of the content, hex" readOnly:"true" required:"false"`
// StorageKey is where the bytes are. It is a UUID and nothing else, which
// is the whole of the path-traversal argument: there is no caller-supplied
// component in it to escape with.
StorageKey string `json:"-" gorm:"type:varchar(64);not null"`
// Visibility decides which door serves it. It defaults to private, and it
// is spelled as a closed set rather than a bool so that the third answer
// somebody will want — signed URLs — is a value and not a schema change.
Visibility string `` /* 166-byte string literal not displayed */
// UploaderID is whoever uploaded it, a user id with no foreign key behind
// it. Validate stamps it from the caller on the context, the way content
// stamps an author.
UploaderID uuid.UUID `` /* 137-byte string literal not displayed */
}
File is one uploaded blob's row.
Every field but Name and Visibility is the server's account of what arrived: a caller does not get to say how big their upload was or what its digest is, because those are the two things a reader checks it against.
type Lister ¶
type Lister interface {
// Keys is every key written before before. The bound is the whole safety
// argument: an upload in flight has bytes and no row yet, and a sweep that
// did not exclude it would delete the blob out from under a request that is
// about to commit.
Keys(ctx context.Context, before time.Time) ([]string, error)
}
Lister is the half of Storage a reconciliation can be built on, and it is optional: an implementation that cannot enumerate what it holds — a signed-URL gateway, a store behind somebody else's API — simply does not implement it, and the module mounts no reconciliation job.
It exists because an orphan blob is the one inconsistency this module's own ordering produces on purpose: an upload writes the bytes before the row, so a transaction that then fails leaves bytes nobody references. Nothing in the database records them, which is why the sweep has to start from the store.
type Opener ¶
type Opener interface {
// Open is the row and its bytes. anonymous is the public door: it serves
// only a file whose visibility says anybody may read it, and answers
// ErrNotFound for everything else, so a private file and a file that does
// not exist are the same answer to a caller who is not signed in.
Open(ctx context.Context, tx db.Tx[db.Tenant], id uuid.UUID, anonymous bool) (*File, io.ReadCloser, error)
}
Opener is the one thing a consuming module actually needs: the bytes of a file it knows the id of. A module that renders a page with an image, or a client module that hands a stored document to a printer, takes this and not Service — it cannot upload, cannot delete, and reads the same rows its own transaction can see.
It is separate from Service for the reason every narrow interface here is: what a consumer declares is what a reviewer reads as the dependency, and "this module can delete files" is a different sentence from "this module can read one".
type Service ¶
type Service interface {
Opener
// Upload streams the bytes into Storage while hashing and counting them,
// then opens the caller's transaction, writes the row and publishes
// file.uploaded.
//
// The bytes are written before the row, and the consequence is stated
// rather than hidden: a transaction that fails after this leaves bytes
// nobody references, which costs disk. The other order costs a row that
// references nothing, which is a download that fails forever. Past the
// deployment's limit it answers ErrTooLarge, and past what the tenant's
// quota has room for ErrQuota; either way it removes what it had written,
// because a caller that was refused should not be charged for storage.
//
// The quota is measured inside the transaction and under a lock on the
// tenant, so twenty uploads that start together cannot all read the same
// total and all decide there is room — which is what the review measured at
// 2.9 times the quota.
Upload(ctx context.Context, tx Tx, up Upload) (*File, error)
// Delete removes the row and publishes file.deleted, carrying the storage
// key. The bytes are removed by whoever handles that event, after this
// transaction commits — see the module's subscription. Deleting what is
// already gone is ErrNotFound, because the row is what a caller named.
Delete(ctx context.Context, tx db.Tx[db.Tenant], id uuid.UUID) (*File, error)
}
Service is what a caller does with files. Every command takes the caller's transaction rather than opening one, so the row and its event commit together; the errors are kit/crud's, plus ErrTooLarge.
type Storage ¶
type Storage interface {
// Put writes the bytes at key. size is what the caller declared, or -1 when
// nothing did; an implementation that has to know a length up front may
// refuse -1, and the one here ignores it. Writing a key that already exists
// is an error, because a key is minted per upload and a collision is a bug
// rather than a replacement.
Put(ctx context.Context, key string, r io.Reader, size int64) error
// Get opens the bytes at key, or ErrNoBlob when there are none. The caller
// closes what it is given.
Get(ctx context.Context, key string) (io.ReadCloser, error)
// Delete removes the bytes at key. A key with nothing at it is not an
// error: the worker that calls this retries, and a retry that failed
// because the first attempt succeeded would never stop.
Delete(ctx context.Context, key string) error
}
Storage is where the bytes go. There is one implementation in this module, on local disk, and the ones that speak to an object store live outside this repository: a reference architecture carrying an S3 client would be teaching S3, and the interface is what makes that a wiring decision rather than a rewrite.
It takes no transaction, and that is the shape of the whole module: a blob write cannot be rolled back, so it happens outside one on purpose and the order relative to the commit is chosen at each call site. Every key it is given is a UUID this module generated.
type Tx ¶
Tx opens the caller's transaction. Upload takes one of these rather than an open transaction, and it is the only command here that does.
A body arrives at whatever speed the client chooses to send it, so a transaction opened before the first byte is a database connection one slow client pins for the whole of the server's read timeout — a review trickled an upload at a byte a second and watched it hold one. The bytes are therefore streamed with nothing open, and this is called once they are all in storage. kit/httpx's per-request transaction is already lazy, so the route hands over its accessor and nothing is opened until this module asks for it.
type Upload ¶
type Upload struct {
Name string
ContentType string
Visibility string
// Declared is the length the request declared, or -1. It is a hint for a
// Storage that has to know one up front, and nothing else: the size that is
// stored and the limit that is enforced both come from counting the bytes
// as they go past, because a request that declares a length can be wrong
// about it either by accident or on purpose.
Declared int64
Body io.Reader
}
Upload is one arriving file: what the caller said about it, and the bytes. The reader is streamed straight to Storage while it is hashed and counted, so nothing here is ever held in memory or spooled to a temporary file.
type Uploaded ¶
type Uploaded struct {
FileID uuid.UUID `json:"fileId"`
Name string `json:"name"`
ContentType string `json:"contentType"`
Size int64 `json:"size"`
SHA256 string `json:"sha256"`
Visibility string `json:"visibility"`
At time.Time `json:"at"`
}
Uploaded is the payload of EventUploaded. It carries the digest as well as the size because the subscriber this exists for is whatever indexes or scans an upload, and both are things it would otherwise read the row back for.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Package filetest is the conformance suite for contracts.Service, a Storage in memory that the suite runs against, and a fake Service that passes it.
|
Package filetest is the conformance suite for contracts.Service, a Storage in memory that the suite runs against, and a fake Service that passes it. |