media

package
v0.16.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 23, 2026 License: MIT Imports: 34 Imported by: 0

Documentation

Overview

Package media stores host content files in per-item folders of one private bucket: library-built keys, the generic manifest with conditional-write edits, and the Store port. See media/s3 for the S3 implementation and media/token for access tokens.

Index

Constants

View Source
const (
	HLSContentType = "application/vnd.apple.mpegurl"
	VTTContentType = "text/vtt; charset=utf-8"
)

Playlist content types.

View Source
const (
	AreaManifest  = layout.AreaManifest
	AreaOriginals = layout.AreaOriginals
	AreaBlobs     = layout.AreaBlobs
	AreaPublic    = layout.AreaPublic
)

Folder areas.

View Source
const (
	AccessFull    = "full"    // every file; downloads
	AccessPreview = "preview" // files [0, preview_limit) plus teasers
	AccessNone    = "none"    // teasers only
)

Access levels in ReadResult.

View Source
const (
	MaxSinglePut = 64 << 20
	MinPartSize  = 8 << 20
	MaxPartSize  = 16 << 20
)

Upload size rules. Files up to MaxSinglePut are one checksum-bound PUT to originals/sha256-{hex}; larger ones are multipart to originals/u-{uuid} with parts of MinPartSize growing up to MaxPartSize (the last part may be smaller).

View Source
const (
	OpInsert  = "insert"  // add Name at Index (default: append); a retry with the same Original is a no-op
	OpReplace = "replace" // swap Name's original; variants are regenerated, a stale hls plays until re-encoded
	OpMove    = "move"    // move Name to Index
	OpRename  = "rename"  // rename Name to To
	OpRemove  = "remove"  // drop Name
)

Commit operations. Insert and Replace take an uploaded Original.

View Source
const (
	CodeInvalid     = "invalid_request"   // 400
	CodeForbidden   = "forbidden"         // 403
	CodeNotFound    = "not_found"         // 404: unknown kind, file or upload
	CodeConflict    = "conflict"          // 409: file name taken
	CodeIncomplete  = "incomplete"        // 409: multipart parts missing
	CodeNotUploaded = "not_uploaded"      // 409: commit before the object landed
	CodeTooLarge    = "too_large"         // 413: over the kind's cap
	CodeQuota       = "quota_exceeded"    // 413: owner quota
	CodeType        = "type_not_allowed"  // 415
	CodeChecksum    = "checksum_mismatch" // 422: stored bytes differ from the declared hash
	CodeRate        = "rate_limited"      // 429
)

Stable upload error codes: clients (the browser SDK) branch on Code.

View Source
const CookieName = token.CookieName

CookieName is the access worker's cookie.

View Source
const UserKind = "user"

UserKind is the kind of per-user folders ({tenant}/user/{id}/), erased by EraseUserTx.

Variables

View Source
var (
	ErrUnknownKind = errors.New("media: unknown kind")
	ErrType        = errors.New("media: content type not allowed")
	ErrTooLarge    = errors.New("media: file too large")
)
View Source
var (
	// ErrNotVisible hides an item the viewer may not see, or that does not exist.
	ErrNotVisible = errors.New("media: not found")
	// ErrResolve wraps a resolver failure; it always denies.
	ErrResolve = errors.New("media: resolve failed")
	// ErrInvalidRequest is a malformed read request.
	ErrInvalidRequest = errors.New("media: invalid request")
)
View Source
var (
	ErrNotFound           = errors.New("media: object not found")
	ErrPreconditionFailed = errors.New("media: precondition failed")
	ErrNotModified        = errors.New("media: not modified")
)
View Source
var ErrJobsNotBound = errors.New("media: River jobs are not composed into a client")
View Source
var ErrManifestConflict = errors.New("media: manifest edit kept conflicting")
View Source
var ErrNotAllowed = errors.New("media: not allowed")

ErrNotAllowed is a URL request for a file the viewer may not have, or a blob that file does not reference.

Functions

func NewUploadName

func NewUploadName() string

NewUploadName names a multipart upload whose hash is unknown: "u-{uuid}".

func SHA256Name

func SHA256Name(sum []byte) string

SHA256Name names content-addressed files: "sha256-{hex}".

func UploadHandler

func UploadHandler(u *Uploads, o UploadHandlerOptions) http.Handler

UploadHandler serves the upload API the browser SDK calls. All routes are POST with JSON bodies; SHA-256 values are lowercase hex. Errors are ErrorReply with the status of its code (Retry-After on 429).

POST /presign      PresignBody  -> PresignReply
POST /parts        PartsBody    -> PartsReply     presign multipart parts
POST /parts/list   TicketBody   -> PartsReply     parts that landed (resume)
POST /complete     TicketBody   -> CompleteReply
POST /abort        TicketBody   -> 204
POST /commit       CommitBody   -> CommitReply
POST /commit-slot  SlotBody     -> 204

Types

type AudioTrack

type AudioTrack struct {
	ID        string    `json:"id"`
	Lang      string    `json:"lang,omitempty"`
	Label     string    `json:"label,omitempty"`
	Default   bool      `json:"default,omitempty"`
	Bandwidth int       `json:"bandwidth,omitempty"`
	Codecs    string    `json:"codecs,omitempty"`
	Blob      string    `json:"blob"`
	Segments  []Segment `json:"segments"`
}

type Capabilities

type Capabilities struct {
	ConditionalPut bool // If-Match / If-None-Match on PUT
	ChecksumSHA256 bool // x-amz-checksum-sha256 enforced on PUT
}

Capabilities are backend features the library depends on, established by Probe.

func Probe

func Probe(ctx context.Context, s Store, prefix string) (Capabilities, error)

Probe measures the backend's capabilities with scratch objects under prefix, which it removes. Results depend on the backend release (RGW vs MinIO).

type CommitBody

type CommitBody struct {
	Ref RefBody `json:"ref"`
	Ops []Op    `json:"ops"`
}

type CommitFile

type CommitFile struct {
	Name     string         `json:"name"`
	Original string         `json:"original"`
	Type     string         `json:"type,omitempty"`
	Size     int64          `json:"size,omitempty"`
	Meta     map[string]any `json:"meta,omitempty"`
}

type CommitReply

type CommitReply struct {
	Files []CommitFile `json:"files"`
}

CommitReply is the committed file order.

type CompleteReply

type CompleteReply struct {
	Name string `json:"name"`
	Type string `json:"type"`
	Size int64  `json:"size"`
}

type Deletion

type Deletion struct {
	Ref   contentref.ContentRef
	Owner string
}

Deletion is one item to delete. Owner is its quota owner (UploadGrant.Owner), "" for none; with a Limiter configured the owner's usage is released.

type Delivery

type Delivery struct {
	Mode DeliveryMode
	// BaseURL is the access worker origin for this site, e.g.
	// "https://media.doujins.com".
	BaseURL string
	// CookieDomain is the site's registrable domain, e.g. "doujins.com";
	// required in cookie mode, because a host-only cookie never reaches media.
	CookieDomain string
	// SigningKey is the current key of the ring the access worker verifies.
	SigningKey token.Key
	// TTL is the minimum token lifetime (default 1h); expiries round up to
	// Window (default token.DefaultWindow).
	TTL    time.Duration
	Window time.Duration
}

Delivery is the host's per-site delivery configuration.

type DeliveryMode

type DeliveryMode string

DeliveryMode is how viewers with full access present their token.

const (
	// DeliverCookie (default) returns plain URLs plus a folder cookie scoped to
	// the item's blobs/, so browsers cache immutable blobs normally.
	DeliverCookie DeliveryMode = "cookie"
	// DeliverURL appends ?t= to every URL: apps and clients without cookies.
	DeliverURL DeliveryMode = "url"
)

type Download

type Download struct {
	Blob   string `json:"blob"`
	Type   string `json:"type,omitempty"`
	Size   int64  `json:"size,omitempty"`
	Spec   string `json:"spec,omitempty"`
	Inputs string `json:"inputs,omitempty"` // hash of the ordered input blobs
}

type DownloadInfo

type DownloadInfo struct {
	Key  string `json:"key"`
	Name string `json:"name"`
	Type string `json:"type,omitempty"`
	Size int64  `json:"size,omitempty"`
	URL  string `json:"url"`
}

type ErrorReply

type ErrorReply struct {
	Error      string `json:"error"`
	Code       string `json:"code"`
	RetryAfter int    `json:"retry_after,omitempty"` // seconds, with 429
}

ErrorReply is the error body; Code is one of the media Code* constants, "unauthorized" or "internal_error".

type File

type File struct {
	Name     string             `json:"name"`
	Original string             `json:"original"`
	Master   string             `json:"master,omitempty"`
	Type     string             `json:"type,omitempty"`
	Size     int64              `json:"size,omitempty"`
	Meta     map[string]any     `json:"meta,omitempty"`
	Variants map[string]Variant `json:"variants,omitempty"`
	HLS      *HLS               `json:"hls,omitempty"`
}

File is one manifest entry. Original and Master live in originals/; variants, HLS and downloads in blobs/.

func (File) Source

func (f File) Source() string

Source is the file variants derive from: Master when present, else Original.

func (File) Teaser

func (f File) Teaser() bool

Teaser reports meta.teaser, a file served to any viewer who can see the item.

type FileInfo

type FileInfo struct {
	Index    int     `json:"index"`
	Name     string  `json:"name,omitempty"`
	Type     string  `json:"type,omitempty"`
	Width    int     `json:"w,omitempty"`
	Height   int     `json:"h,omitempty"`
	Duration float64 `json:"duration,omitempty"`
	Teaser   bool    `json:"teaser,omitempty"`
	Locked   bool    `json:"locked,omitempty"`
	HLS      bool    `json:"hls,omitempty"`
	Variant  string  `json:"variant,omitempty"`
	URL      string  `json:"url,omitempty"`
}

type Fit

type Fit string

Fit is how an image spec fits its box.

const (
	FitInside Fit = "inside"
	FitCover  Fit = "cover"
)

type GetOptions

type GetOptions struct {
	IfNoneMatch string
	Range       string
}

GetOptions: IfNoneMatch returns ErrNotModified on a match; Range is an HTTP Range value.

type Grant

type Grant struct {
	Item       Item
	Resolution access.Resolution
	Manifest   *Manifest
	Expires    time.Time
	// contains filtered or unexported fields
}

Grant is one viewer's resolved access to one item or version: the read API and HLS playlists sign every URL through it.

func (*Grant) Allowed

func (g *Grant) Allowed(i int) bool

Allowed reports whether file i is served to this viewer: within the preview cut, or a teaser of a visible item.

func (*Grant) AudioPlaylist

func (g *Grant) AudioPlaylist(file, id string) ([]byte, error)

AudioPlaylist is the byte-range media playlist of one audio track.

func (*Grant) Cookie

func (g *Grant) Cookie() *http.Cookie

Cookie is the folder cookie to set in cookie mode with full access, else nil.

func (*Grant) DownloadURL

func (g *Grant) DownloadURL(ctx context.Context, key string) (name, u string, err error)

DownloadURL signs a manifest download under its display name; full access only, and always a URL token because the worker must see the signed dl=.

func (*Grant) Full

func (g *Grant) Full() bool

Full reports full access: one folder token covers every blob.

func (*Grant) MasterPlaylist

func (g *Grant) MasterPlaylist(file string, o MasterOptions) ([]byte, error)

MasterPlaylist is the multivariant playlist of file: one variant per video rendition, with alternative audio and subtitle groups.

func (*Grant) SpriteVTT

func (g *Grant) SpriteVTT(file string) ([]byte, error)

SpriteVTT is the seek-preview track: one cue per sprite tile, each pointing at its tile with a #xywh fragment.

func (*Grant) SubtitlePlaylist

func (g *Grant) SubtitlePlaylist(file, id string) ([]byte, error)

SubtitlePlaylist is a one-segment playlist over the whole WebVTT blob.

func (*Grant) URL

func (g *Grant) URL(i int, blob string) (string, error)

URL signs blob of file i: plain in cookie mode with full access, the folder token in URL mode, else a token for exactly that key.

func (*Grant) VideoPlaylist

func (g *Grant) VideoPlaylist(file string, height int) ([]byte, error)

VideoPlaylist is the byte-range media playlist of one video rendition.

type HLS

type HLS struct {
	Source string       `json:"source"`
	Spec   string       `json:"spec,omitempty"`
	Video  []Rendition  `json:"video,omitempty"`
	Audio  []AudioTrack `json:"audio,omitempty"`
	Subs   []Subtitle   `json:"subs,omitempty"`
	Sprite *Sprite      `json:"sprite,omitempty"`
}

HLS is a byte-range ladder: each rendition is one fMP4 blob. Source is the original it was encoded from; when it differs from the file's, the ladder is stale but still served until its replacement is promoted.

type HandlerOptions

type HandlerOptions struct {
	Tenant   string
	Identity Identity
	Logger   *slog.Logger
}

HandlerOptions scope the read API to one tenant.

type Hooks

type Hooks struct {
	// DownloadName returns the display name a download is saved under, e.g.
	// "[Artist] Title (English).zip". Default: "{content_id}-{key}{ext}".
	DownloadName func(ctx context.Context, ref contentref.ContentRef, key string, d Download) (string, error)
	// Failed reports a file a processor cannot derive (an undecodable image,
	// say); file is the manifest file name, or the slot name. The job does not
	// retry it; a new commit does.
	Failed func(ctx context.Context, ref contentref.ContentRef, file string, err error)
}

Hooks are optional host callbacks.

type Identity

type Identity interface {
	Actor(ctx context.Context) (access.Actor, bool)
}

Identity reads the authenticated actor the host's middleware put in the context; the same shape as content.Identity.

type Item

type Item struct {
	// contains filtered or unexported fields
}

Item is a validated content item and the keys of its folder:

{tenant}/{kind}/{content_id}/manifest.json | manifests/{version}.json
                            /originals/{sha256-hex | u-uuid | slot}
                            /blobs/{sha256-hex | u-uuid}
                            /public/{output}.webp

func (Item) Blob

func (i Item) Blob(name string) (string, error)

Blob is the key of a served, immutable derivative.

func (Item) BlobsPrefix

func (i Item) BlobsPrefix() string

func (Item) Kind

func (i Item) Kind() Kind

func (Item) ManifestKey

func (i Item) ManifestKey() (string, error)

ManifestKey is manifest.json, or manifests/{version}.json for versioned kinds.

func (Item) ManifestsPrefix

func (i Item) ManifestsPrefix() string

ManifestsPrefix lists every manifest of a versioned kind.

func (Item) Original

func (i Item) Original(name string) (string, error)

Original is the key of an uploaded file or master (never served).

func (Item) OriginalsPrefix

func (i Item) OriginalsPrefix() string

func (Item) Prefix

func (i Item) Prefix() string

Prefix is the folder, "{tenant}/{kind}/{id}/"; versions share it.

func (Item) Public

func (i Item) Public(name string) (string, error)

Public is public/{name}.webp: a slot output or a host-chosen inline image id.

func (Item) PublicPrefix

func (i Item) PublicPrefix() string

func (Item) Ref

func (i Item) Ref() contentref.ContentRef

func (Item) SlotOriginal

func (i Item) SlotOriginal(slot string) (string, error)

SlotOriginal is the fixed, overwritten original of a registered public slot.

type Jobs

type Jobs struct {
	// contains filtered or unexported fields
}

Jobs is media's River contribution: sweep, folder deletion, and the workers other media packages register. Compose RiverJobs once into the host client.

func NewJobs

func NewJobs(cfg JobsConfig) (*Jobs, error)

func (*Jobs) AddProcessor

func (j *Jobs) AddProcessor(p Processor) error

AddProcessor registers a processor for Enqueue'd jobs, before composition.

func (*Jobs) DeleteItemsTx

func (j *Jobs) DeleteItemsTx(ctx context.Context, tx pgx.Tx, items ...Deletion) error

DeleteItemsTx deletes each item's whole folder (every version) through a job enqueued in the host's delete transaction. A second pass after LateUploadWindow removes uploads that land after the first. The quota to release is measured here from the item's manifests, so a retried job settles it once.

func (*Jobs) Enqueue

func (j *Jobs) Enqueue(ctx context.Context, job ProcessJob) error

Enqueue implements ProcessQueue: one pending job per ref and slot, run by every registered processor.

func (*Jobs) EraseUserTx

func (j *Jobs) EraseUserTx(ctx context.Context, tx pgx.Tx, tenant, userID string, items ...Deletion) error

EraseUserTx erases a user's media: the items the host maps to them plus their user folder, {tenant}/user/{id}/.

func (*Jobs) Insert

Insert enqueues a job on the bound client; an empty queue means Queue().

func (*Jobs) InsertTx

func (j *Jobs) InsertTx(ctx context.Context, tx pgx.Tx, args river.JobArgs, o *river.InsertOpts) (*rivertype.JobInsertResult, error)

InsertTx enqueues a job in the host's transaction.

func (*Jobs) Queue

func (j *Jobs) Queue() string

Queue is the shared media queue; registered workers may use it or add their own.

func (*Jobs) Register

func (j *Jobs) Register(fn func(*river.Config) error) error

Register adds workers, queues or periodic jobs to the contribution. Media packages (image variants, video) call it before RiverJobs is composed.

func (*Jobs) RiverJobs

func (j *Jobs) RiverJobs() riverhelpers.Contribution

RiverJobs contributes media's workers, queue and periodic sweep to the host's helpers/river composition. It composes once.

func (*Jobs) ScheduleSweep

func (j *Jobs) ScheduleSweep(ctx context.Context, ref contentref.ContentRef) error

ScheduleSweep sweeps the item's folder after the grace period; a sweep already waiting for the folder absorbs it. Manifests calls it on every edit.

func (*Jobs) Sweep

func (j *Jobs) Sweep(ctx context.Context, ref contentref.ContentRef) (SweepResult, error)

Sweep deletes the item folder's blobs/ and hash-named originals/ that no manifest in the folder references, once every manifest is older than the grace period, and only objects past abandonedAt. Slot originals, public/ and manifests are never swept.

Invariant: the sweep deletes only objects that no manifest references and that no in-flight commit can newly reference. Uploads keeps the second half: presign reuses an existing original, and a commit accepts one, only while it is referenced or well before abandonedAt (see protected).

func (*Jobs) SweepAll

func (j *Jobs) SweepAll(ctx context.Context) error

SweepAll sweeps every folder of the configured tenants whose kind is registered: the periodic backstop for missed schedules and abandoned uploads.

type JobsConfig

type JobsConfig struct {
	Store Store
	Kinds *Registry
	// Tenants are the folders the periodic sweep pass covers; folders of
	// kinds missing from Kinds are skipped.
	Tenants []string
	// Grace protects in-flight uploads, jobs and mid-stream viewers: a folder
	// is swept only when its manifests are this old, and only objects this old
	// are deleted. Default 24 h.
	Grace time.Duration
	// SweepInterval is the periodic pass interval. Default 24 h.
	SweepInterval time.Duration
	// LateUploadWindow delays the second pass of a folder deletion, which
	// removes PUTs and multipart completions that land after the first. It
	// must exceed the longest upload presign TTL and the 1-day multipart
	// abort rule. Default 25 h.
	LateUploadWindow time.Duration
	Limiter          UploadLimiter // releases a deleted item's quota; optional
	Queue            string        // default "contentkit_media"
	MaxWorkers       int           // default 2
	Logger           *slog.Logger
	Now              func() time.Time // clock for grace decisions; default time.Now
}

JobsConfig configures media's River jobs.

type Kind

type Kind struct {
	Name      string
	Versioned bool // manifests live at manifests/{version_id}.json
	Types     []string
	MaxBytes  int64
	Specs     map[string]Spec // variant name → spec
	Slots     map[string]Slot // public slot name → outputs
	Video     bool
	// Zip names the variant packed, in file order, into downloads["zip"];
	// "" offers no zip.
	Zip string
}

Kind is a host's per-kind rule set, registered once at startup.

func (Kind) Allows

func (k Kind) Allows(contentType string, size int64) error

Allows checks a file against the kind's types and size cap.

type Locker

type Locker interface {
	Lock(ctx context.Context, key string) (unlock func(), err error)
}

Locker serializes manifest edits on backends without conditional PUT.

func PGLocker

func PGLocker(pool *pgxpool.Pool) Locker

PGLocker serializes manifest edits with a per-manifest session advisory lock in the host database: the fallback for backends without conditional PUT.

type Manifest

type Manifest struct {
	Files     []File              `json:"files"`
	Meta      map[string]any      `json:"meta,omitempty"`
	Downloads map[string]Download `json:"downloads,omitempty"`
}

Manifest is the ordered file list of an item or version. List order is display order. Blob references are names within the item's folder.

func (*Manifest) Blobs

func (m *Manifest) Blobs() []string

Blobs returns every blobs/ name the manifest references.

func (*Manifest) File

func (m *Manifest) File(name string) int

File returns the index of the named file, or -1.

func (*Manifest) OriginalBytes

func (m *Manifest) OriginalBytes() int64

OriginalBytes is the storage charged for a manifest: the sizes of its distinct originals. Deleting or erasing an item releases it.

func (*Manifest) Originals

func (m *Manifest) Originals() []string

Originals returns every originals/ name the manifest references.

func (*Manifest) Validate

func (m *Manifest) Validate() error

Validate requires unique, non-empty file names and well-formed references.

type ManifestOptions

type ManifestOptions struct {
	// Locker is required when the store lacks ConditionalPut; edits then run
	// under it and write unconditionally. See PGLocker.
	Locker     Locker
	CacheSize  int // manifests kept in process, revalidated by ETag; default 4096
	MaxRetries int // CAS attempts per edit; default 16
	// Jobs, when set, schedules the folder's sweep after every written edit.
	// Scheduling is best-effort (logged); the periodic sweep pass backs it up.
	Jobs *Jobs
}

ManifestOptions configure Manifests.

type Manifests

type Manifests struct {
	// contains filtered or unexported fields
}

Manifests reads and edits item manifests.

func NewManifests

func NewManifests(store Store, kinds *Registry, opts ManifestOptions) (*Manifests, error)

func (*Manifests) Edit

func (m *Manifests) Edit(ctx context.Context, ref contentref.ContentRef, fn func(*Manifest) error) (*Manifest, error)

Edit applies fn to the current manifest (empty if none) and writes it with If-Match on the ETag it read (If-None-Match for a new one), re-reading and re-applying fn on conflict. fn must be safe to run more than once; an error from fn aborts the edit. An unchanged manifest is not written.

func (*Manifests) Get

Get returns the manifest and its ETag, or ErrNotFound. Cached copies are revalidated with a conditional GET, so a read is never stale.

type MasterOptions

type MasterOptions struct {
	Audio, Subs []string
}

MasterOptions filter a master playlist's renditions by id or BCP 47 tag; nil keeps every track, an empty non-nil slice none.

type Multipart

type Multipart struct {
	Ticket      string
	MinPartSize int64
	MaxPartSize int64
	MaxParts    int
}

Multipart carries the opaque ticket for PresignParts, ListParts, Complete and Abort, and the part-size bounds the client adapts within.

type MultipartReply

type MultipartReply struct {
	Ticket      string `json:"ticket"`
	MinPartSize int64  `json:"min_part_size"`
	MaxPartSize int64  `json:"max_part_size"`
	MaxParts    int    `json:"max_parts"`
}

type Object

type Object struct {
	Key            string
	Size           int64
	ETag           string
	ContentType    string
	CacheControl   string
	ContentRange   string
	LastModified   time.Time
	ChecksumSHA256 []byte            // set when the backend stored a full-object SHA-256
	Metadata       map[string]string // user metadata (x-amz-meta-*), lower-case keys; Head and Get
}

Object is object metadata. For a ranged Get, Size is the body length and ContentRange is set.

type Op

type Op struct {
	Op       string         `json:"op"`
	Name     string         `json:"name"`
	Original string         `json:"original,omitempty"`
	Index    *int           `json:"index,omitempty"`
	To       string         `json:"to,omitempty"`
	Meta     map[string]any `json:"meta,omitempty"` // insert, replace: the file's meta
}

Op is one manifest edit.

type PGLimiter

type PGLimiter struct {
	// contains filtered or unexported fields
}

PGLimiter is the default UploadLimiter over ContentKit's Postgres schema: hourly counters per uploader (bytes/day sums the last 24), one usage total per owner and short-lived pending reservations. There are no rows per file.

func NewPGLimiter

func NewPGLimiter(pool *pgxpool.Pool, schema string, limits PGLimits) (*PGLimiter, error)

func (*PGLimiter) Reserve

func (l *PGLimiter) Reserve(ctx context.Context, r Reservation) error

func (*PGLimiter) Settle

func (l *PGLimiter) Settle(ctx context.Context, s Settlement) error

func (*PGLimiter) Usage

func (l *PGLimiter) Usage(ctx context.Context, tenant, owner string) (used, pending int64, err error)

Usage reports an owner's stored bytes and unexpired pending reservations.

type PGLimits

type PGLimits struct {
	FilesPerHour int
	BytesPerDay  int64
	// Quota is the owner's storage cap in bytes (<= 0 unlimited); nil disables quotas.
	Quota func(ctx context.Context, tenant, owner string) (int64, error)
	// ReservationTTL drops unsettled reservations; default 24h.
	ReservationTTL time.Duration
}

PGLimits configure PGLimiter. Zero limits are off.

type Part

type Part struct {
	Number int32
	Size   int64
	ETag   string
	SHA256 []byte
}

Part is one uploaded multipart part.

type PartBody

type PartBody struct {
	Number int32  `json:"number"`
	Size   int64  `json:"size"`
	SHA256 string `json:"sha256"`
}

type PartReply

type PartReply struct {
	Number  int32         `json:"number"`
	Size    int64         `json:"size,omitempty"`
	SHA256  string        `json:"sha256,omitempty"`
	Request *RequestReply `json:"request,omitempty"`
}

PartReply is a presigned part (parts) or a landed part (parts/list).

type PartRequest

type PartRequest struct {
	Number int32
	Size   int64
	SHA256 []byte
}

PartRequest is one part to presign: its exact length and SHA-256.

type PartsBody

type PartsBody struct {
	Ticket string     `json:"ticket"`
	Parts  []PartBody `json:"parts"`
}

type PartsReply

type PartsReply struct {
	Parts []PartReply `json:"parts"`
}

type PresignBody

type PresignBody struct {
	Ref    RefBody `json:"ref"`
	Type   string  `json:"type"`
	Size   int64   `json:"size"`
	SHA256 string  `json:"sha256,omitempty"` // required up to 64 MiB and for slots
	Slot   string  `json:"slot,omitempty"`
}

type PresignPut

type PresignPut struct {
	ContentType string
	Size        int64
	SHA256      []byte
	TTL         time.Duration
}

PresignPut binds a direct upload to its exact type, length and SHA-256.

type PresignReply

type PresignReply struct {
	Name      string          `json:"name"`
	Exists    bool            `json:"exists,omitempty"`
	Put       *RequestReply   `json:"put,omitempty"`
	Multipart *MultipartReply `json:"multipart,omitempty"`
}

PresignReply: exists (commit directly), put (one PUT) or multipart.

type PresignRequest

type PresignRequest struct {
	Ref    contentref.ContentRef
	Type   string
	Size   int64
	SHA256 []byte
	Slot   string
}

PresignRequest declares one file. SHA256 is required up to MaxSinglePut and for slots; Slot targets the kind's fixed slot original.

type Presigned

type Presigned struct {
	Name      string
	Exists    bool
	Put       *PresignedRequest
	Multipart *Multipart
}

Presigned is the upload plan: Exists (already in the folder; commit it), a single Put, or a Multipart upload.

type PresignedPart

type PresignedPart struct {
	Number int32
	PresignedRequest
}

PresignedPart is a part URL.

type PresignedRequest

type PresignedRequest struct {
	Method  string
	URL     string
	Header  http.Header
	Expires time.Time
}

PresignedRequest is a request a browser sends as is, with exactly Header.

type ProcessJob

type ProcessJob struct {
	Ref  contentref.ContentRef
	Slot string
}

ProcessJob asks for derivatives after a commit: the item's manifest, or one public slot when Slot is set.

type ProcessQueue

type ProcessQueue interface {
	Enqueue(ctx context.Context, job ProcessJob) error
}

ProcessQueue enqueues processing (River in the image and video lanes).

type Processor

type Processor func(ctx context.Context, job ProcessJob) error

Processor derives an item's files after a commit (image variants, video encodes). It must be idempotent. An Enqueue while its job runs is absorbed: the job reruns the processors when its input changed during the run.

type PutOptions

type PutOptions struct {
	ContentType    string
	CacheControl   string
	ChecksumSHA256 []byte
	IfMatch        string
	IfNoneMatch    string
	Metadata       map[string]string
}

PutOptions are conditions and headers for Put. IfNoneMatch "*" creates only.

type ReadOptions

type ReadOptions struct {
	// Variants in preference order: each file in range gets a URL for the
	// first one it has. Empty returns metadata only.
	Variants      []string
	Offset, Limit int
}

ReadOptions select the URLs a read returns.

type ReadResult

type ReadResult struct {
	Access       string         `json:"access"`
	Total        int            `json:"total"`
	PreviewLimit int            `json:"preview_limit"`
	Offset       int            `json:"offset"`
	Limit        int            `json:"limit"`
	Expires      int64          `json:"expires"` // unix seconds; URLs and cookie stop working then
	Meta         map[string]any `json:"meta,omitempty"`
	Files        []FileInfo     `json:"files"`
	Downloads    []DownloadInfo `json:"downloads,omitempty"`
	// Cookie must be set on the response (cookie mode, full access).
	Cookie *http.Cookie `json:"-"`
}

ReadResult is the read API response. Files lists every file; only allowed files inside [offset, offset+limit) carry a URL, and files past the cut omit their name.

type Reader

type Reader struct {
	// contains filtered or unexported fields
}

Reader answers the read API: one Resolve per item, metadata for every file, and signed URLs for the requested range.

func NewReader

func NewReader(o ReaderOptions) (*Reader, error)

func (*Reader) Grant

func (r *Reader) Grant(ctx context.Context, ref contentref.ContentRef, actor access.Actor) (*Grant, error)

Grant resolves ref for actor exactly once and loads its manifest. A resolver error denies (ErrResolve); an invisible item is ErrNotVisible. A visible item without a manifest yet has no files.

func (*Reader) Handler

func (r *Reader) Handler(o HandlerOptions) http.Handler

Handler serves the read API. The host mounts it under a prefix such as "/media/" after its auth middleware. {id} is "{content_id}" or "{content_id}@{version_id}" for a versioned kind. Errors are JSON {"error", "code"}: 400 invalid_request, 404 not_found (also for hidden items), 500 internal_error (a resolver error denies this way).

GET /{kind}/{id}?variant=high,thumb&offset=0&limit=50 -> ReadResult (+ Set-Cookie mt)
GET /{kind}/{id}/hls/{file}/master.m3u8?audio=ja&subs=en,s2 (filters optional; empty = none)
GET /{kind}/{id}/hls/{file}/video/{height}.m3u8, audio/{track}.m3u8, subs/{track}.m3u8
GET /{kind}/{id}/hls/{file}/sprite.vtt
GET /{kind}/{id}/download/{key} -> 302 to the signed download URL

Every request resolves the item once; playlists and redirects are "private, no-store" and carry the folder cookie in cookie mode.

func (*Reader) PublicURL

func (r *Reader) PublicURL(ref contentref.ContentRef, name string) (string, error)

PublicURL is the plain URL of a public slot output or inline image; it reads nothing.

func (*Reader) Read

Read resolves ref once and answers the read API.

type ReaderOptions

type ReaderOptions struct {
	Manifests *Manifests
	Kinds     *Registry
	Resolver  access.ContentResolver
	Delivery  Delivery
	Hooks     Hooks
	// MaxLimit caps ReadOptions.Limit (default 200); DefaultLimit is used when
	// Limit is 0 (default 50).
	MaxLimit, DefaultLimit int
	Now                    func() time.Time
}

ReaderOptions configure a Reader.

type RefBody

type RefBody struct {
	Kind    string `json:"kind"`
	ID      string `json:"id"`
	Version string `json:"version,omitempty"`
}

RefBody names an item within the handler's tenant.

type Registry

type Registry struct {
	// contains filtered or unexported fields
}

Registry is the host's set of kinds.

func NewRegistry

func NewRegistry(kinds ...Kind) (*Registry, error)

NewRegistry validates and registers kinds.

func (*Registry) Item

func (r *Registry) Item(ref contentref.ContentRef) (Item, error)

Item validates ref against the registry. A version is required to address a versioned kind's manifest, and refused for an unversioned kind.

func (*Registry) Kind

func (r *Registry) Kind(name string) (Kind, error)

Kind returns a registered kind.

type Rendition

type Rendition struct {
	Height    int       `json:"height"`
	Width     int       `json:"w"`
	Bandwidth int       `json:"bandwidth"`
	Average   int       `json:"avg,omitempty"`
	Codecs    string    `json:"codecs"`
	Blob      string    `json:"blob"`
	Segments  []Segment `json:"segments"`
}

Rendition is one video-only fMP4 blob: its init segment is bytes [0, Segments[0].Offset) and the segments follow contiguously.

type RequestReply

type RequestReply struct {
	Method  string            `json:"method"`
	URL     string            `json:"url"`
	Headers map[string]string `json:"headers"`
	Expires time.Time         `json:"expires"`
}

RequestReply is a presigned request: send exactly these headers (the browser adds Content-Length, which is signed too).

type Reservation

type Reservation struct {
	Tenant   string
	Uploader string
	Owner    string
	Key      string
	Size     int64
}

Reservation is one presigned upload. Uploader is rate-limited; Owner ("" for none) has Size reserved against its quota until the upload is settled or the reservation expires.

type Segment

type Segment struct {
	Offset  int64
	Length  int64
	Seconds float64
}

Segment is one EXT-X-BYTERANGE segment, encoded as [offset, length, seconds].

func (Segment) MarshalJSON

func (s Segment) MarshalJSON() ([]byte, error)

func (*Segment) UnmarshalJSON

func (s *Segment) UnmarshalJSON(b []byte) error

type Settlement

type Settlement struct {
	Tenant string
	Owner  string
	Keys   []string
	Delta  int64
}

Settlement releases reservations and moves an owner's usage.

type Slot

type Slot struct {
	Outputs map[string]Spec
}

Slot is a fixed public image: its original is kept at originals/{slot} and each output is written to public/{output}.webp.

type SlotBody

type SlotBody struct {
	Ref    RefBody `json:"ref"`
	Slot   string  `json:"slot"`
	SHA256 string  `json:"sha256"`
}

type Spec

type Spec struct {
	Width   int
	Height  int
	Fit     Fit
	Quality int
	Blur    float64
}

Spec describes one derived image. Zero Width and Height keep full resolution.

func (Spec) Hash

func (s Spec) Hash() string

Hash is the spec's stable identity; a variant whose recorded spec differs is stale.

type Sprite

type Sprite struct {
	Blob     string  `json:"blob"`
	Cols     int     `json:"cols"`
	Rows     int     `json:"rows"`
	Width    int     `json:"w"`
	Height   int     `json:"h"`
	Interval float64 `json:"interval"`
}

type Store

type Store interface {
	Put(ctx context.Context, key string, body io.Reader, size int64, opts PutOptions) (Object, error)
	Get(ctx context.Context, key string, opts GetOptions) (io.ReadCloser, Object, error)
	Head(ctx context.Context, key string) (Object, error)
	Delete(ctx context.Context, key string) error
	List(ctx context.Context, prefix string) iter.Seq2[Object, error]

	PresignPut(ctx context.Context, key string, p PresignPut) (PresignedRequest, error)
	CreateMultipart(ctx context.Context, key, contentType string) (uploadID string, err error)
	PresignPart(ctx context.Context, key, uploadID string, number int32, size int64, sha256 []byte, ttl time.Duration) (PresignedRequest, error)
	ListParts(ctx context.Context, key, uploadID string) ([]Part, error)
	CompleteMultipart(ctx context.Context, key, uploadID string, parts []Part) (Object, error)
	AbortMultipart(ctx context.Context, key, uploadID string) error

	Capabilities() Capabilities
}

Store is the bucket. Keys are built by Item; implementations do not interpret them.

type Subtitle

type Subtitle struct {
	ID     string `json:"id"`
	Lang   string `json:"lang,omitempty"`
	Label  string `json:"label,omitempty"`
	Forced bool   `json:"forced,omitempty"`
	Blob   string `json:"blob"`
}

type SweepResult

type SweepResult struct {
	Deleted []string
	Wait    time.Duration
}

SweepResult reports one folder sweep. Wait > 0 means a manifest changed within the grace period and nothing was deleted; sweep again after Wait.

type TicketBody

type TicketBody struct {
	Ticket string `json:"ticket"`
}

type UploadAuthorizer

type UploadAuthorizer interface {
	CanUpload(ctx context.Context, actor access.Actor, ref contentref.ContentRef) (UploadGrant, error)
}

UploadAuthorizer is the host's upload permission check (AuthKit), run at presign and commit.

type UploadError

type UploadError struct {
	Code       string
	Message    string
	RetryAfter time.Duration // CodeRate: when the window frees
}

UploadError is a refused upload request. UploadLimiter implementations refuse with CodeRate or CodeQuota.

func AsUploadError

func AsUploadError(err error) (*UploadError, bool)

AsUploadError classifies err: an *UploadError, or the kind and registry sentinels. ok is false for internal failures.

func (*UploadError) Error

func (e *UploadError) Error() string

func (*UploadError) Status

func (e *UploadError) Status() int

Status is the HTTP status for Code.

type UploadGrant

type UploadGrant struct {
	Allowed bool
	Exempt  bool
	Owner   string
}

UploadGrant is the host's verdict. Exempt (trusted roles) skips the UploadLimiter checks; Owner is the quota owner (creator, channel), "" for none.

type UploadHandlerOptions

type UploadHandlerOptions struct {
	Tenant string                                   // every ref is scoped to it
	Actor  func(*http.Request) (access.Actor, bool) // the host's authenticated caller; false answers 401
	Logger *slog.Logger                             // 5xx causes; default slog.Default()
}

UploadHandlerOptions configure UploadHandler.

type UploadLimiter

type UploadLimiter interface {
	Reserve(ctx context.Context, r Reservation) error
	Settle(ctx context.Context, s Settlement) error
}

UploadLimiter is the optional anti-abuse port. Reserve runs at presign (skipped for exempt uploaders) and refuses with an *UploadError coded CodeRate or CodeQuota before any bytes move. Settle runs at commit, abort and item deletion: it drops the reservations of Keys and adds Delta (the change in stored originals, negative on removal) to the owner's usage.

type UploadOptions

type UploadOptions struct {
	Store      Store
	Kinds      *Registry
	Manifests  *Manifests
	Authorizer UploadAuthorizer
	Tickets    *token.Ring   // signs multipart tickets (domain-separated from access tokens); required for files over MaxSinglePut
	Limiter    UploadLimiter // optional
	Queue      ProcessQueue  // optional
	PresignTTL time.Duration // PUT and part URLs; default 15m
	// Grace is the sweep's (JobsConfig.Grace): Queue's when it is *Jobs, else 24h.
	Grace     time.Duration
	TicketTTL time.Duration // multipart ticket; default 24h, the abort-incomplete rule
}

UploadOptions configure Uploads.

type UploadedObject

type UploadedObject struct {
	Name string
	Type string
	Size int64
}

UploadedObject is a completed multipart original.

type Uploads

type Uploads struct {
	// contains filtered or unexported fields
}

Uploads presigns direct-to-bucket uploads and commits them into manifests. It keeps no state: a multipart upload is its S3 UploadId, carried in a signed ticket.

func NewUploads

func NewUploads(o UploadOptions) (*Uploads, error)

func (*Uploads) Abort

func (u *Uploads) Abort(ctx context.Context, actor access.Actor, sealed string) error

Abort cancels a multipart upload and drops its reservation.

func (*Uploads) Commit

func (u *Uploads) Commit(ctx context.Context, actor access.Actor, ref contentref.ContentRef, ops []Op) (*Manifest, error)

Commit applies ops to the manifest in one conditional write. Every new original is HEAD-checked against the kind's type and size cap (and re-hashed when the store does not enforce checksums). Usage is settled by the change in distinct originals the manifest references, then processing is enqueued.

func (*Uploads) CommitSlot

func (u *Uploads) CommitSlot(ctx context.Context, actor access.Actor, ref contentref.ContentRef, slot string, sum []byte) error

CommitSlot validates an uploaded slot original and enqueues the re-encode of its public outputs. sum is the SHA-256 the upload was presigned with.

func (*Uploads) Complete

func (u *Uploads) Complete(ctx context.Context, actor access.Actor, sealed string) (UploadedObject, error)

Complete assembles the parts server-side. Missing parts answer CodeIncomplete and keep the upload; parts that cannot add up to the declared size abort it.

func (*Uploads) ListParts

func (u *Uploads) ListParts(ctx context.Context, actor access.Actor, sealed string) ([]Part, error)

ListParts reports the parts that landed, for resuming.

func (*Uploads) Presign

func (u *Uploads) Presign(ctx context.Context, actor access.Actor, r PresignRequest) (Presigned, error)

func (*Uploads) PresignParts

func (u *Uploads) PresignParts(ctx context.Context, actor access.Actor, sealed string, parts []PartRequest) ([]PresignedPart, error)

PresignParts signs parts of a multipart upload, each bound to its length and SHA-256. Parts re-signed after a failure replace the earlier attempt.

type Variant

type Variant struct {
	Blob string `json:"blob"`
	Spec string `json:"spec,omitempty"`
	Type string `json:"type,omitempty"`
	Size int64  `json:"size,omitempty"`
}

Directories

Path Synopsis
Package accessworker is the media access worker's HTTP handler, run by cmd/media-access: it checks the token for a blob path (URL `?t=` or cookie `mt`), serves public/ paths without one, refuses manifests and originals/, and streams the object from the private bucket with its own read-only key.
Package accessworker is the media access worker's HTTP handler, run by cmd/media-access: it checks the token for a blob path (URL `?t=` or cookie `mt`), serves public/ paths without one, refuses manifests and originals/, and streams the object from the private bucket with its own read-only key.
Package image derives WebP variants, public slots and zip downloads with libvips (CGO).
Package image derives WebP variants, public slots and zip downloads with libvips (CGO).
internal
s3test
Package s3test opens the test bucket from the environment:
Package s3test opens the test bucket from the environment:
uploadtestserver command
Command uploadtestserver serves media.UploadHandler over a fresh MinIO/RGW bucket for the browser SDK's integration tests (sdk/upload/test).
Command uploadtestserver serves media.UploadHandler over a fresh MinIO/RGW bucket for the browser SDK's integration tests (sdk/upload/test).
wirets
Package wirets renders the upload API wire types as TypeScript for the browser SDK (sdk/upload/src/wire.gen.ts).
Package wirets renders the upload API wire types as TypeScript for the browser SDK (sdk/upload/src/wire.gen.ts).
Package layout defines media object keys, dependency-free so the access worker can classify paths without importing the media runtime:
Package layout defines media object keys, dependency-free so the access worker can classify paths without importing the media runtime:
Package s3 implements media.Store over aws-sdk-go-v2 for Ceph RGW (production) and MinIO (tests).
Package s3 implements media.Store over aws-sdk-go-v2 for Ceph RGW (production) and MinIO (tests).
Package tiered is an optional visibility policy: it maps an item's level to entitlement keys and asks a Checker which ones the actor holds.
Package tiered is an optional visibility policy: it maps an item's level to entitlement keys and asks a Checker which ones the actor holds.
Package token signs and verifies media access tokens, shared by the host signer and the access worker so the format cannot drift:
Package token signs and verifies media access tokens, shared by the host signer and the access worker so the format cannot drift:
Package video encodes an item's video files with ffmpeg into a byte-range HLS ladder (one single-file fMP4 blob per rendition and audio track), WebVTT subtitles, a thumbnail sprite and one muxed MP4 download per quality, and records them in the manifest's hls and downloads.
Package video encodes an item's video files with ffmpeg into a byte-range HLS ladder (one single-file fMP4 blob per rendition and audio track), WebVTT subtitles, a thumbnail sprite and one muxed MP4 download per quality, and records them in the manifest's hls and downloads.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL