Documentation
¶
Overview ¶
Package media stores uploads — images, videos, and documents: the binary objects on any S3-compatible bucket (AWS, Linode, DigitalOcean, MinIO, R2, ...) and their metadata in the database. Each image upload produces the untouched original plus a ladder of resized variants encoded as lossy WebP — "web" for a full-width slot, "card" for a listing card, "thumb" for a preview grid — which pages combine into a srcset so a browser downloads the size it will actually display. Videos are stored as uploaded (no transcoding) with an optional poster frame; documents are stored as-is. Everything is served directly from the bucket, or proxied by the CMS, which rebuilds a rendition an older upload lacks on the first request for it (see regenerate.go).
Every upload also writes a JSON manifest recording what the object keys cannot — filename, alt text, folder, dimensions, uploader — which makes the bucket self-describing: a database with no media can rebuild its library from one. See manifest.go and restore.go.
Index ¶
- Constants
- Variables
- type AdoptMode
- type Folder
- type Image
- type KeyPrefixer
- type Kind
- type ListOptions
- type Lister
- type Manager
- func (m *Manager) All(ctx context.Context, locale string, opts ListOptions) ([]Media, error)
- func (m *Manager) Count(ctx context.Context) (int, error)
- func (m *Manager) CreateFolder(ctx context.Context, name string, kind Kind) (*Folder, error)
- func (m *Manager) Delete(ctx context.Context, id int64, locale string) error
- func (m *Manager) DeleteFolder(ctx context.Context, id int64) error
- func (m *Manager) EnsureRendition(ctx context.Context, rest string) error
- func (m *Manager) Folders(ctx context.Context) ([]Folder, error)
- func (m *Manager) GetByID(ctx context.Context, id int64, locale string) (*Media, error)
- func (m *Manager) ImageFor(md *Media, prefer string) *Image
- func (m *Manager) IsDark(ctx context.Context, md *Media) (bool, error)
- func (m *Manager) KeyRoot() string
- func (m *Manager) MaxVideoBytes() int64
- func (m *Manager) Move(ctx context.Context, mediaID int64, folderID *int64) error
- func (m *Manager) OpenOriginal(ctx context.Context, md *Media) (io.ReadCloser, string, error)
- func (m *Manager) Rename(ctx context.Context, id int64, locale, name string) (*Media, error)
- func (m *Manager) Restore(ctx context.Context, mode AdoptMode) (RestoreResult, error)
- func (m *Manager) SetMaxVideoBytes(n int64)
- func (m *Manager) SetPoster(ctx context.Context, id int64, locale string, poster []byte) (*Media, error)
- func (m *Manager) SetWebPQuality(q float64)
- func (m *Manager) SyncManifests(ctx context.Context) (int, error)
- func (m *Manager) URL(md *Media, rendition string) string
- func (m *Manager) UpdateAlt(ctx context.Context, id int64, locale, alt string) error
- func (m *Manager) Upload(ctx context.Context, filename string, data []byte, uploadedBy int64, ...) (*Media, error)
- func (m *Manager) UploadFrom(ctx context.Context, filename string, src io.ReadSeeker, size int64, ...) (*Media, error)
- func (m *Manager) Views(items []Media) []View
- type Media
- type ObjectInfo
- type ObjectStore
- type RangeGetter
- type Rendition
- type RestoreResult
- type S3Config
- type S3Store
- func (s *S3Store) ApplyPublicReadPolicy(ctx context.Context) error
- func (s *S3Store) Delete(ctx context.Context, key string) error
- func (s *S3Store) Get(ctx context.Context, key string) (io.ReadCloser, string, error)
- func (s *S3Store) GetRange(ctx context.Context, key, rangeSpec string) (io.ReadCloser, string, string, int64, error)
- func (s *S3Store) KeyPrefix() string
- func (s *S3Store) List(ctx context.Context, prefix string) ([]ObjectInfo, error)
- func (s *S3Store) PublicURL(key string) string
- func (s *S3Store) Put(ctx context.Context, key, contentType string, body io.Reader) error
- type View
Constants ¶
const ( // MaxImageDocBytes caps image and document uploads, which are // buffered in memory for validation and processing. MaxImageDocBytes = 25 << 20 // DefaultMaxVideoBytes caps video uploads unless the host configures // another limit (cms.Config.MediaMaxVideoMB). Videos are streamed to // the object store, so the cap guards storage, not memory. DefaultMaxVideoBytes = 512 << 20 )
const ( // DefaultWebPQuality is the lossy WebP quality, on a 0–1 scale, used // for the derived variants unless the host configures another // (cms.Config.MediaWebPQuality). Deliberately low: variants exist for // fast page loads, and the untouched original is always kept. DefaultWebPQuality = 0.3 )
const ProxyPathPrefix = "/cms/media/"
ProxyPathPrefix is the public route the CMS serves proxied media under.
Variables ¶
var ( // ErrDuplicateFolder is returned by CreateFolder for a name already // used within the same kind. ErrDuplicateFolder = errors.New("media: a folder with that name already exists") // ErrBadFolderName is returned for empty or over-long names. ErrBadFolderName = errors.New("media: folder names must be 1-60 characters") // ErrBadFolderKind is returned for a kind that isn't image, file, or // video. ErrBadFolderKind = errors.New("media: unknown folder kind") // ErrFolderNotEmpty is returned by DeleteFolder for a folder that // still holds media. ErrFolderNotEmpty = errors.New("media: the folder still has files in it") )
var ErrBadFilename = errors.New("media: bad filename")
ErrBadFilename is returned by Rename for a name that is empty (or all path separators) once sanitized, or longer than maxFilenameLen.
var ErrInvalidRange = errors.New("media: invalid range")
ErrInvalidRange is returned by GetRange for unsatisfiable ranges; the media proxy answers it with 416.
var ErrNoRaster = errors.New("media: no raster rendition to measure")
ErrNoRaster is returned when an image's lightness cannot be measured because there are no pixels to measure: a vector, or a record whose derived renditions were never made.
var ErrNoRendition = errors.New("media: not a rebuildable rendition")
ErrNoRendition is returned when a requested object is not a rendition this Manager can rebuild: an unknown name, an item that is not in the library, or a video poster, which has no stored source. It means "this object is legitimately absent", so the proxy answers 404 quietly rather than logging a failure.
var ErrNotFound = errors.New("media: not found")
ErrNotFound is returned when no media matches the query.
var ErrObjectNotFound = errors.New("media: object not found")
ErrObjectNotFound is returned by Get for missing keys.
var ErrTooLarge = errors.New("media: file too large")
ErrTooLarge is returned for uploads over the size cap for their kind: MaxImageDocBytes for images and documents, the Manager's video limit (SetMaxVideoBytes) for videos.
var ErrUnsafeSVG = errors.New("media: svg contains active content")
ErrUnsafeSVG is returned for an SVG upload containing scripts or other active content the scan won't allow.
var ErrUnsupportedType = errors.New("media: unsupported file type")
ErrUnsupportedType is returned for uploads that are neither a supported image (JPEG, PNG, GIF, WebP) nor a whitelisted document type (PDF, office formats, text/CSV, ZIP).
Functions ¶
This section is empty.
Types ¶
type AdoptMode ¶
type AdoptMode int
AdoptMode controls whether Restore rebuilds the media library from the object store.
const ( // AdoptWhenEmpty adopts the bucket's media only when the database has // none — the default, and the case of a fresh deployment pointed at a // bucket that already holds content. Once any media exists it never // runs again, so ordinary startups do no bucket work beyond one list. AdoptWhenEmpty AdoptMode = iota // AdoptOff never touches the bucket. AdoptOff // AdoptReconcile checks on every startup and adopts anything in the // bucket the database is missing. It covers what AdoptWhenEmpty // cannot — restoring a database backup older than the bucket — at the // cost of listing the manifests each time. AdoptReconcile )
type Folder ¶
type Folder struct {
ID int64
Kind Kind
Name string
Count int // number of media items in the folder
}
Folder is a flat organizational bucket for media, scoped to one media kind — a folder made among the images exists only there. Folders live entirely in Postgres; the object store's keys are untouched by foldering.
type Image ¶
type Image struct {
URL string
Srcset string
Width int
Height int
Alt string
Renditions []Rendition
}
Image is one uploaded image ready to put in an <img> element: a default src sized for the slot it fills, the srcset of every distinct rendition so the browser can pick a better one, and the intrinsic size of that default so it can reserve the space before the bytes arrive.
Templates use it as
<img src="{{.URL}}" srcset="{{.Srcset}}" sizes="..."
width="{{.Width}}" height="{{.Height}}" alt="{{.Alt}}">
with sizes written by the template, since only it knows the layout. Every field is safe to use on its own: a legacy or external image with no library record still carries a URL, just no srcset or dimensions.
type KeyPrefixer ¶
type KeyPrefixer interface {
KeyPrefix() string
}
KeyPrefixer is an optional interface an ObjectStore may implement to namespace one deployment's objects inside a bucket shared by several sites: when present, the Manager stores objects under "<KeyPrefix()>/media/..." instead of "media/...". S3Store implements it from S3Config.KeyPrefix.
type Kind ¶
type Kind string
Kind distinguishes images (resized, embeddable) and videos (stored as uploaded, embedded as players) from plain files like PDFs and office documents (stored as-is, linked to).
type ListOptions ¶
type ListOptions struct {
Kind Kind // "" = both images and files
Query string // case-insensitive substring match on filename
FolderID *int64 // only items in this folder
Unfiled bool // only items in no folder (ignored when FolderID set)
}
ListOptions filters All. The zero value lists everything.
type Lister ¶
type Lister interface {
// List returns every object whose key starts with prefix. Order is
// unspecified. An empty result is not an error.
List(ctx context.Context, prefix string) ([]ObjectInfo, error)
}
Lister is an optional ObjectStore interface for enumerating objects under a key prefix. Manager.Restore needs it to find the manifests describing a bucket's media; a store that does not implement it simply cannot be adopted from, and Restore reports that rather than failing. S3Store implements it.
type Manager ¶
type Manager struct {
// contains filtered or unexported fields
}
Manager coordinates the object store and the Postgres metadata.
func NewManager ¶
NewManager returns a Manager storing binaries in objects and metadata in db. If objects implements KeyPrefixer, its prefix namespaces every key, letting several deployments share one bucket.
func (*Manager) All ¶
All returns media records with alt text for locale, newest first, filtered by opts.
func (*Manager) Count ¶
Count returns how many media items exist across every kind — the number the admin shows beside its Media nav entry.
func (*Manager) CreateFolder ¶
CreateFolder makes a new folder of the given kind and returns it.
func (*Manager) Delete ¶
Delete removes a media record, its manifest, and its bucket objects. Pages that still reference the image keep its (now dead) URL; the admin UI warns about this before deleting.
func (*Manager) DeleteFolder ¶
DeleteFolder removes a folder, but only an empty one — a folder still holding media comes back ErrFolderNotEmpty and is left alone. Deleting it used to unfile its contents, which scattered in one click the files an editor had just gathered; emptying it first says the same thing deliberately, and the files stay where they can be seen while you do.
Deleting a folder that is already gone is not an error: two tabs open on the same folder can both submit the delete.
func (*Manager) EnsureRendition ¶
EnsureRendition builds the derived object at rest, a media-root-relative key like "9f2c…/card.webp", and stores it. It returns nil once the object exists, so the caller can go straight back to the object store for it.
Concurrent requests for the same object share one rebuild, and rebuilds across the process share a small pool of workers, so a listing page full of images that all need the same new rung costs one encode each rather than one per visitor.
func (*Manager) Folders ¶
Folders returns all folders — every kind — with their item counts, sorted by name. Callers filter to the kind they present.
func (*Manager) ImageFor ¶
ImageFor builds the <img> data for one library image. prefer names the rung used as the default src — "web" for a full-width slot, "card" for a listing card, "thumb" for a small preview; an unknown name falls back to "web".
The srcset lists every rung, smaller ones included, so a card slot can still take the thumbnail on a narrow phone. Rungs that would come out the same width are listed once: an image narrower than the ladder's bounds is not upscaled, so its larger rungs are all the same picture.
Vectors get a single URL and no srcset — an SVG scales losslessly, so alternate widths would be the same bytes under different names. A nil or non-image record returns nil, which templates render as nothing.
func (*Manager) IsDark ¶
IsDark reports whether an image reads as dark overall — the question to ask before laying text over one, since the answer decides whether that text should be light or dark.
It measures the thumbnail rendition: the smallest object of the set, so the answer costs one small fetch and a decode of a few hundred pixels square. Vectors have no rendition to measure and return ErrNoRaster; callers that only want a hint can treat any error as "not dark", which leaves text in the site's own colour.
func (*Manager) KeyRoot ¶
KeyRoot returns the bucket prefix every object this manager stores lives under: "media/", or "<prefix>/media/" for a store with a deployment prefix.
func (*Manager) MaxVideoBytes ¶
MaxVideoBytes returns the video upload size cap, for request-body limits and user-facing messages.
func (*Manager) Move ¶
Move puts a media item into a folder, or back to unfiled when folderID is nil.
func (*Manager) OpenOriginal ¶ added in v1.1.16
OpenOriginal streams the object md was uploaded as — the untouched original, never a rendition — with the content type the store holds it under. The caller closes the body. It is how the admin serves a download: the public URL cannot, being either cross-origin (where a browser shows the file instead of saving it) or, on a private bucket, not reachable by the browser at all.
func (*Manager) Rename ¶ added in v0.9.0
Rename changes a media item's display name. Purely a metadata change: objects live under an opaque item id (or, for documents, a key cut from the upload-time name), and none of them move — which is what keeps every existing link to the item working. The stored extension is reattached whatever was submitted, so a rename can never make the name contradict the bytes; a trailing copy of it in the new name is absorbed rather than doubled. The extension alone is not a name, so a rename to just ".pdf" comes back ErrBadFilename.
func (*Manager) Restore ¶
Restore rebuilds media rows from the manifests in the object store, according to mode. It returns what it did.
It is safe to call on every startup and safe to call from several instances at once: an advisory lock serializes the run, and items are keyed by store_key, so an interrupted run resumes rather than duplicating. Individual bad manifests are logged and counted, not fatal — one unreadable sidecar must not stop the other thousand items from coming back.
Adoption needs an ObjectStore that implements Lister. With any other store it reports nothing done rather than failing, since a host that supplied its own store can rebuild by whatever means suits it.
func (*Manager) SetMaxVideoBytes ¶
SetMaxVideoBytes overrides the video upload size cap. Values below one are ignored.
func (*Manager) SetPoster ¶ added in v0.9.0
func (m *Manager) SetPoster(ctx context.Context, id int64, locale string, poster []byte) (*Media, error)
SetPoster attaches a poster frame to an existing video, processing the still into the same web/thumb variants an upload-time poster gets and updating the record's variant metadata. The admin uses it to backfill videos uploaded without one — the server can't decode video, so the frame is captured in a browser. Existing poster objects are overwritten in place: the variant keys are deterministic, so a failure part-way leaves nothing a retry won't replace. Non-video media is refused.
func (*Manager) SetWebPQuality ¶
SetWebPQuality overrides the lossy WebP quality used for image variants. Values outside (0, 1] are ignored.
func (*Manager) SyncManifests ¶
SyncManifests rewrites every manifest from the database, and returns how many items it wrote. It is the repair for drift: manifest writes during normal operation are best-effort, so an object store that was briefly unreachable leaves items whose sidecar is missing or stale. Running this makes the bucket a faithful description of the library again.
It is safe to run at any time and costs one small PUT per item.
func (*Manager) URL ¶
URL returns the public URL of one rendition: "original" or any rung of the ladder ("web", "card", "thumb"). Files have a single rendition, returned for any name. Videos have "original" (the video itself; also returned for "web"), plus "poster" and "thumb" — empty when the video has no poster.
func (*Manager) Upload ¶
func (m *Manager) Upload(ctx context.Context, filename string, data []byte, uploadedBy int64, folderID *int64) (*Media, error)
Upload validates and stores a buffered upload, returning its record. It is UploadFrom for callers that already hold the bytes; videos should arrive through UploadFrom so they stream to the object store instead.
func (*Manager) UploadFrom ¶
func (m *Manager) UploadFrom(ctx context.Context, filename string, src io.ReadSeeker, size int64, poster []byte, uploadedBy int64, folderID *int64) (*Media, error)
UploadFrom validates and stores an upload, sniffing its leading bytes to pick the pipeline. Images are buffered and stored untouched alongside resized WebP web and thumb variants; SVGs (selected by extension — sniffing can't identify them) are validated as script-free and stored as their own variants, since vectors need no resizing; whitelisted documents (PDF, office formats, text/CSV, ZIP) are buffered and stored as-is; videos (MP4, WebM) are streamed to the object store as uploaded — no transcoding. size is the upload's byte length (multipart.FileHeader.Size). poster optionally carries a client-captured still image for videos, processed into the same web/thumb variants images get; it is ignored for other kinds and dropped, not fatal, when undecodable. A non-nil folderID files the upload into that folder.
type Media ¶
type Media struct {
ID int64
Kind Kind
// StoreKey locates the item's objects *relative to the media root*, so
// it embeds neither "media/" nor the deployment's S3Config.KeyPrefix:
// images and videos store the bare item id (their objects live at
// StoreKey+"/original.<ext>" under the root), files store
// "<id>/<name>.<ext>", the object itself. Manager.abs composes the
// absolute key. Keeping the root out of the database is what lets a
// deployment change its KeyPrefix, and what lets a bucket be adopted
// into a deployment that uses a different one.
StoreKey string
Filename string
Mime string
Ext string
VariantExt string // extension of the web/thumb objects (".webp"; legacy rows ".jpg"/".png"); empty for a video without a poster
Width int // zero for files, and for videos without a poster
Height int // zero for files, and for videos without a poster
Size int64
FolderID *int64 // nil = unfiled
UploadedBy *int64
CreatedAt time.Time
Alt string
}
Media is one uploaded image, video, or document. Alt is the alt text for the locale it was loaded with (images only).
func (Media) FolderIDValue ¶
FolderIDValue returns the folder id, or 0 when unfiled — convenient in templates, where pointers are awkward to compare.
type ObjectInfo ¶
ObjectInfo describes one object found by a Lister. LastModified is the store's own write timestamp; zero when the backend did not report one.
type ObjectStore ¶
type ObjectStore interface {
// Put stores an object under key.
Put(ctx context.Context, key, contentType string, body io.Reader) error
// Get retrieves the object at key and its content type. The caller
// closes the body.
Get(ctx context.Context, key string) (body io.ReadCloser, contentType string, err error)
// Delete removes the object at key. Deleting a missing key is not an
// error.
Delete(ctx context.Context, key string) error
// PublicURL returns the browser-facing URL for key. It may be
// absolute (bucket or CDN) or app-relative (proxied through the CMS).
PublicURL(key string) string
}
ObjectStore is where media binaries live. The S3 implementation is the default; host applications may substitute their own (e.g. local disk for development) — it is one of the module's extension points.
type RangeGetter ¶
type RangeGetter interface {
// GetRange retrieves part of the object at key, where rangeSpec is a
// verbatim HTTP Range header value ("bytes=0-1023"). contentRange is
// the Content-Range for a 206 response; when the backend ignored the
// range (e.g. a multi-range request) it is empty and body is the
// whole object. length is the byte count of body. Unsatisfiable
// ranges return ErrInvalidRange.
GetRange(ctx context.Context, key, rangeSpec string) (body io.ReadCloser, contentType, contentRange string, length int64, err error)
}
RangeGetter is an optional ObjectStore interface for serving HTTP Range requests. Without it, proxied video cannot seek — and Safari, which probes with a Range request before playing, cannot play it at all. S3Store implements it; custom stores that skip it still work for images and documents.
type RestoreResult ¶
type RestoreResult struct {
// Adopted is how many media items were inserted.
Adopted int
// Folders is how many folders were created.
Folders int
// Skipped counts manifests already present in the database.
Skipped int
// Orphaned counts manifests whose objects are missing from the bucket;
// they are not adopted, since the item could not be served.
Orphaned int
// Failed counts manifests that could not be read or inserted. They are
// logged individually; a re-run retries them.
Failed int
}
RestoreResult reports what an adoption run did.
func (RestoreResult) DidWork ¶
func (r RestoreResult) DidWork() bool
DidWork reports whether the run found anything to act on, for callers deciding whether it is worth logging. Skipped alone does not count: a reconcile that confirms the database already matches the bucket is the quiet, expected case.
type S3Config ¶
type S3Config struct {
// Endpoint is the S3 API host, without scheme, e.g.
// "us-ord-10.linodeobjects.com" or "s3.us-east-1.amazonaws.com".
Endpoint string
// Region for request signing. Defaults to the first label of
// Endpoint (correct for Linode/DO-style endpoints); set explicitly
// for AWS.
Region string
Bucket string
// AccessKey and Secret are the credentials.
AccessKey string
Secret string
// KeyPrefix namespaces this deployment's uploads inside a bucket
// shared by several sites: objects are stored under
// "<KeyPrefix>/media/..." instead of "media/...". Use a short slug
// unique to the deployment — letters, digits, '.', '-', '_' — e.g.
// "acme-hotel". Proxied media URLs (the default) do not expose it;
// direct bucket and CDN URLs include it. A shared bucket pairs well
// with per-deployment credentials restricted to this prefix, and is
// what makes adopting a bucket into an empty database safe (see
// Manager.Restore). Empty — the default — keeps keys under "media/".
//
// Changing it after uploads exist is allowed as far as the database
// is concerned — stored keys are relative to the media root, so no
// row embeds it — but the objects themselves must be moved to the
// new prefix to stay reachable.
KeyPrefix string
// PublicRead marks the bucket as publicly readable, so pages embed
// direct bucket URLs. Leave false — the default — to serve media
// through the CMS itself (the /cms/media/ route on the public
// handler), which works with private buckets and needs no bucket
// policy. Making a bucket public varies by provider: a bucket
// policy allowing s3:GetObject on AWS, the bucket Access setting on
// Linode/DO, or ApplyPublicReadPolicy where permitted.
PublicRead bool
// PublicBaseURL overrides the generated object URL prefix, for
// serving through a CDN or custom domain. No trailing slash. Takes
// precedence over PublicRead.
PublicBaseURL string
// UsePathStyle addresses objects as endpoint/bucket/key instead of
// bucket.endpoint/key. Needed for MinIO and some self-hosted stores.
UsePathStyle bool
// ObjectACL is an optional canned ACL (e.g. "public-read") sent with
// each upload. Leave empty — the default — for buckets whose public
// access comes from a bucket policy; many stores (AWS with ACLs
// disabled, newer Linode clusters) reject per-object ACLs outright.
// See ApplyPublicReadPolicy for setting the bucket policy.
ObjectACL string
// ApplyPublicReadPolicy makes cms.Migrate apply the public-read
// bucket policy (the S3Store method of the same name) when the store
// is built from cms.Config.S3 — one-time setup, idempotent, for
// buckets that weren't created public. It only sets the policy; pair
// it with PublicRead or PublicBaseURL so pages actually embed direct
// bucket URLs.
ApplyPublicReadPolicy bool
}
S3Config configures the S3-compatible object store.
type S3Store ¶
type S3Store struct {
// contains filtered or unexported fields
}
S3Store implements ObjectStore against any S3-compatible service.
func NewS3Store ¶
NewS3Store validates cfg and returns a ready S3Store. It does not call the network.
func (*S3Store) ApplyPublicReadPolicy ¶
ApplyPublicReadPolicy sets a bucket policy that lets anyone GET the media binaries (but not list or write, and not the manifests, which live outside the media root and name uploaders). Call it once during site setup when the bucket was not created public; it is idempotent. This is the supported way to serve uploads publicly on stores that reject per-object ACLs.
The grant is scoped to this deployment's media root, so several sites can share a bucket under different KeyPrefixes without one site's policy publishing another's objects.
func (*S3Store) GetRange ¶
func (s *S3Store) GetRange(ctx context.Context, key, rangeSpec string) (io.ReadCloser, string, string, int64, error)
GetRange implements RangeGetter by forwarding the Range header to S3.
func (*S3Store) KeyPrefix ¶
KeyPrefix returns the configured deployment prefix, implementing KeyPrefixer.