worker

package
v0.138.12 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 13, 2026 License: MIT Imports: 28 Imported by: 0

Documentation

Overview

Package worker is the media worker: the one process that places uploads and runs every producer, from the host's worker River schema (Config.Schema) in its database: staged uploads hashed and placed at their blobs (media.Manifests.Place), images, zips, public presets and editor views (media/image, libvips), and HLS, MP4, subtitles and frames (media/video, ffmpeg). The host presigns, commits and reads. Each processing attempt records its retained progress/completion fact and wakes version metadata work in one transaction. Stock workers need no host callbacks.

The registry is the host's (media.Config, built by the same code or read from its JSON); cmd/media-worker is the stock build for hosts whose kinds are plain data.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func StoreFromEnv

func StoreFromEnv() (*mediaS3.Store, error)

StoreFromEnv is the media bucket, not dialed:

MEDIA_S3_ENDPOINT            e.g. http://rook-ceph-rgw-external.svc
MEDIA_S3_BUCKET, MEDIA_S3_REGION (default us-east-1), MEDIA_S3_PATH_STYLE (default true)
MEDIA_S3_ACCESS_KEY_ID, MEDIA_S3_SECRET_ACCESS_KEY   read/write key

Types

type Config

type Config struct {
	Pool *pgxpool.Pool // the host database: Schema, manifest locks, progress
	// Schema is the host's worker River schema, the one its workqueue.Queue
	// inserts into; required, and never shared with another host.
	Schema string
	// ContentSchema holds ContentKit's media journal and quota tables. Every
	// writer for this host must use the same schema; migration is host-owned.
	ContentSchema string
	// AccountSchema and AccountWorkerSchema are the shared tenant avatar state
	// and queue from contentkit.Config.AccountMedia. Empty retains legacy mode.
	AccountSchema, AccountWorkerSchema string
	Store                              media.Store
	// Kinds is the host's registry. Failed and the catalog retirement barrier
	// and optional completion callbacks run here.
	Kinds *mediastore.Registry
	// HostSchema is the host's River schema ("" is the connection's search path):
	// metadata snapshots, sweeps and purges run there. Grace is the host's
	// JobsConfig.Grace (default 24 h).
	HostSchema string
	Grace      time.Duration

	TempDir string // scratch for video sources and outputs; default os.TempDir()
	Threads int    // ffmpeg threads; default GOMAXPROCS
	// Preset, TopPreset, VideoEncoder (Encoder) and VideoCodecs (Codecs) are
	// video.Config's.
	Preset, TopPreset, VideoEncoder string
	VideoCodecs                     []media.Codec
	// IngestWorkers and DerivativeWorkers bound concurrent item jobs (defaults
	// 2 and 1). A derivative job handles both images and video for its item.
	IngestWorkers     int
	DerivativeWorkers int
	// ImageSources bounds sources decoded at once within one derivative job.
	ImageSources int
	// PageGroup is image.Config's GroupSize: rendered images recorded and
	// reported per manifest edit (default 1, each on its own).
	PageGroup int
	// MaxPixels, MaxFrames and MaxAnimationSeconds bound decoded images
	// (image.Config's defaults when zero).
	MaxPixels           int
	MaxFrames           int
	MaxAnimationSeconds float64
	// JobTimeout bounds one ingest or derivative attempt (default 1 h).
	JobTimeout time.Duration
	// VideoSlice is how long a derivative attempt keeps encoding its own
	// video's chunks before yielding to the other queued jobs; 0 (the
	// default) yields after every chunk. A slice ends early when its video
	// becomes playable or a job of higher priority waits. At most half of
	// JobTimeout, so the last chunk a slice starts still has the other half.
	VideoSlice time.Duration
	// ShutdownGrace is how long running jobs get to finish on shutdown
	// before they are cancelled and retried elsewhere; default 30 s.
	ShutdownGrace time.Duration
	Logger        *slog.Logger
	// RiverHooks are added to the worker's River client (observability).
	RiverHooks []rivertype.Hook
	// Metrics records work and encodes; the first worker built with it also
	// exports its Schema's queue backlog while it leads that schema.
	Metrics *Metrics
}

Config configures the worker.

func FromEnv

func FromEnv(ctx context.Context) (Config, error)

FromEnv builds a Config's database and bucket (StoreFromEnv) from the environment, with TuningFromEnv; the host then sets Kinds. Neither is dialed. Close the Pool when done.

DATABASE_URL                 host Postgres (holds MEDIA_WORKER_SCHEMA)

func (*Config) TuningFromEnv

func (c *Config) TuningFromEnv() error

TuningFromEnv sets the host queue and tuning fields present in the environment, leaving the others as they are:

MEDIA_WORKER_SCHEMA          the host's worker River schema (required, one per host, e.g. doujins_media_worker)
MEDIA_CONTENT_SCHEMA         the shared ContentKit schema (required; same as the uploader's journal)
MEDIA_HOST_RIVER_SCHEMA      the host's River schema (default: the connection's search path)
MEDIA_HOST_GRACE             the host's sweep grace (default 24h)
MEDIA_WORKER_TMP             scratch dir (default os.TempDir()); size for a video source plus outputs
MEDIA_WORKER_THREADS         ffmpeg threads (default: CPU limit)
MEDIA_WORKER_PRESET          x264 preset of rungs up to 1080 (default fast)
MEDIA_WORKER_TOP_PRESET      x264 preset of the rungs above (default fast)
MEDIA_WORKER_CODECS          the ladder's codecs, preferred first: h264, av1 (default av1,h264)
MEDIA_WORKER_ENCODER         auto (default: per codec NVENC if a probe encode works, else CPU), cpu or nvenc
MEDIA_WORKER_CONCURRENCY     image/video derivative jobs per process (default 1)
MEDIA_WORKER_INGEST_CONCURRENCY  ingest jobs per process (default 2)
MEDIA_WORKER_PAGE_GROUP      rendered images per manifest edit and progress report (default 1; e.g. 32 for bulk imports; retries use 1)
MEDIA_WORKER_MAX_PIXELS      decoded pixels per image source, all frames together (default 100000000)
MEDIA_WORKER_SHUTDOWN_GRACE  time running jobs get on SIGTERM before cancel (default 30s)
MEDIA_WORKER_VIDEO_SLICE     time a derivative job keeps encoding its own video before yielding, e.g. 20m; ends when its video becomes playable (default 0: after every chunk; at most half the job timeout)

type Metrics

type Metrics struct {
	river.MiddlewareDefaults
	// contains filtered or unexported fields
}

Metrics records work attempts, video encode passes and the worker schema's queue backlog. Register it once per worker process. The backlog gauges come only from the worker holding River leadership of its schema.

func NewMetrics

func NewMetrics(registerer prometheus.Registerer) (*Metrics, error)

NewMetrics registers one worker process's metrics with registerer.

func (*Metrics) ObserveEncode

func (m *Metrics) ObserveEncode(o video.EncodeObservation)

func (*Metrics) Work

func (m *Metrics) Work(ctx context.Context, job *rivertype.JobRow, doInner func(context.Context) error) (err error)

Work wraps the full River execution, including WorkBegin and argument decoding, which can fail before WorkEnd is called.

type Worker

type Worker struct {
	// contains filtered or unexported fields
}

Worker is a configured worker; Run drains jobs until shutdown.

func New

func New(ctx context.Context, c Config) (*Worker, error)

New builds the River client with the image and video workers. It needs no DDL rights: the host applies workqueue.Migrate(Schema) in its migration step, so the worker can run as the host's unprivileged app role. Missing encode tables are reported before the worker is built.

func (*Worker) Run

func (w *Worker) Run(ctx context.Context) error

Run waits for the bucket (every job needs it; waiting burns no job attempts), then works jobs until ctx is done. Running jobs get ShutdownGrace before cancellation. A non-nil error may leave jobs unwinding: the caller must not close their database pool until the worker's River client has stopped.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL