syncjobs

package
v0.7.0-dev.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 9, 2026 License: AGPL-3.0 Imports: 13 Imported by: 0

Documentation

Overview

Package syncjobs owns the server-side async sync job machinery (D1, D2): a bounded FIFO dispatcher with an admission lease, the job record registry GET /api/sync/jobs serves, and the sync:done publication seam.

The dispatcher accepts at most MaxJobs top-level entries (queued plus running); overflow is a typed capacity refusal the handler maps to 429, and a same-file dispatch answers the EXISTING job's ids even at cap. Entries run ONE at a time in FIFO order — a queued single behind a season batch waits, visibly — while the syncworker semaphore stays the execution-slot authority (the automatic sync path interleaves there between entries). A season batch (batch.go) is ONE entry holding N item job records.

Locking: mu > activity.Log's lock (Dispatch creates the activity entry under mu so a same-file race cannot observe a half-registered job); the stop registry's callbacks run lock-free (a context cancel). No path takes a foreign lock and then mu.

Index

Constants

View Source
const DefaultRegistryCap = 256

DefaultRegistryCap bounds retained terminal records between prune ticks. Cap eviction removes only records already past retention, so within the retention window every record survives whatever the count.

View Source
const MaxJobs = 8

MaxJobs is the admission lease: at most this many accepted (queued or running) top-level entries; the next dispatch answers a typed 429.

Variables

View Source
var (
	// ErrCapacity: the admission lease is full — mapped to the wire as 429.
	ErrCapacity = errors.New("sync job capacity reached")
	// ErrShuttingDown: the dispatcher has drained; no new work is accepted.
	ErrShuttingDown = errors.New("sync dispatcher is shutting down")
)

Typed dispatch refusals.

Functions

This section is empty.

Types

type Accepted

type Accepted struct {
	ActivityID string
	JobID      int64
	Existing   bool
}

Accepted is a dispatch answer: the two ids the 202 hands the client, and whether they name a pre-existing live job (same-file dedupe).

type BatchAccepted

type BatchAccepted struct {
	// ActivityID is the batch activity entry's id — the 202 body, the
	// registry filter key, and the stop/cancel address.
	ActivityID string
	// Existing reports the idempotent same-scope answer: a live batch for
	// this (series, season) already exists and its id was returned.
	Existing bool
	// SkippedLive counts items not registered because their subtitle file
	// already has a live job (at most one live job per file — the dedupe
	// invariant; that job's own sync:done settles the file).
	SkippedLive int
}

BatchAccepted is a season dispatch answer.

type BatchInput

type BatchInput struct {
	// Detail is the batch activity entry's aggregate detail line.
	Detail string
	// Items in ordinal order; item i is ordinal i+1.
	Items []ExecInput
	// SeriesID and Season scope the batch: a dispatch for a scope that
	// already has a live batch answers that batch's id (idempotent).
	SeriesID int
	Season   int
}

BatchInput is one season batch's work order: pre-resolved items in ordinal order (the handler enumerates and resolves paths at accept time; the executor re-reads file content at run time) plus the aggregate identity the batch activity entry carries.

type CancelOutcome

type CancelOutcome int

CancelOutcome is Cancel's verdict for the activity endpoints.

const (
	// CancelUnknown: the activity id names no sync job; callers fall through
	// to the legacy activity path.
	CancelUnknown CancelOutcome = iota
	// CancelledQueued: a queued entry was cancelled — settled done(cancelled),
	// slot released IMMEDIATELY (the next 202 enters), no success event.
	CancelledQueued
	// CancelConverted: the delete lost the race to admission and converted to
	// a running-cancel through the just-registered stop entry — 204; the job
	// settles done(cancelled) and releases capacity on worker exit.
	CancelConverted
	// CancelTerminal: the job is already done; dismissal is the caller's
	// (the registry is never touched by a terminal row's dismissal).
	CancelTerminal
)

Cancel verdicts.

type Deps

type Deps struct {
	// Exec runs one job's analysis (synchandlers' audio executor).
	Exec ExecFunc
	// Log is the activity log: one entry per job, created queued at accept.
	Log *activity.Log
	// Stops is the live stop registry; the admission hook registers each
	// single-file job's stop entry there BEFORE publishing running.
	Stops *activity.StopRegistry
	// PublishDone publishes the sync:done event (EventBus.PublishSyncDone).
	PublishDone func(ev *events.SyncDoneEvent)
	// OnChange runs after every registry transition GET /api/sync/jobs can
	// observe (accept, admission, settle, batch finish, prune), outside mu;
	// the server mints the jobs digest version from it. Optional.
	OnChange func()
	// RegistryCap overrides DefaultRegistryCap when positive (tests).
	RegistryCap int
}

Deps wires the dispatcher's collaborators.

type Dispatcher

type Dispatcher struct {
	// contains filtered or unexported fields
}

Dispatcher owns the FIFO, the admission lease, the dedupe map, per-job cancels, and the job registry. One Run loop executes entries in order.

func New

func New(deps Deps) *Dispatcher

New builds a Dispatcher over deps. Call Run exactly once to start it.

func (*Dispatcher) Cancel

func (d *Dispatcher) Cancel(activityID string) CancelOutcome

Cancel routes an activity-id cancellation into the dispatcher. See the CancelOutcome constants for the verdict semantics. A batch activity id addresses the WHOLE batch (the batch is the cancellation unit); its item records are settled by the batch arms, never individually.

func (*Dispatcher) Dispatch

func (d *Dispatcher) Dispatch(in *ExecInput) (Accepted, error)

Dispatch enqueues one single-file job: non-blocking, atomically reserving an admission slot. A live job for the same subtitle file answers its EXISTING ids (even at cap); a full lease answers ErrCapacity (429 on the wire, never auto-retried by the client).

func (*Dispatcher) DispatchBatch

func (d *Dispatcher) DispatchBatch(in *BatchInput) (BatchAccepted, error)

DispatchBatch enqueues one season batch: non-blocking, atomically reserving ONE admission slot for the whole batch. All item records are created NOW (state queued, ordinal set, batch_activity_id riding every record), so a registry read before any sync:done lists the full item set. A live batch for the same (series, season) answers its EXISTING id; a full lease answers ErrCapacity (429 on the wire).

func (*Dispatcher) Jobs

func (d *Dispatcher) Jobs(batchActivityID string) []Job

Jobs snapshots the registry in the total order accepted_at DESC, job_id DESC (numeric), optionally filtered by batch_activity_id. Batch item records live here like any job; the batch itself is not a record — its aggregate is the activity entry.

func (*Dispatcher) Prune

func (d *Dispatcher) Prune(maxAge time.Duration)

Prune removes done records older than maxAge (retention = activity.DefaultPruneAge, one owner: the server's prune ticker drives both). Queued and running records are never evicted, and neither is a done item of a still-live batch — the batch finalizers dereference every item id, and the reload read lists the full item set. Terminal batch entries age out with their items.

func (*Dispatcher) Run

func (d *Dispatcher) Run(ctx context.Context)

Run executes queued entries one at a time in FIFO order until ctx (the server lifetime) dies, then drains: queued jobs settle done(cancelled) without ever executing, and the in-flight worker has already been waited for (its context derives from ctx, so the child is killed and reaped).

type ExecFunc

type ExecFunc func(ctx context.Context, in *ExecInput, hook func() bool) ExecResult

ExecFunc runs one job's analysis. The hook MUST be invoked exactly once at execution-slot acquisition (the typed core does this); returning false refuses the run and the executor reports subflux.JobCancelled.

type ExecInput

type ExecInput struct {
	SubtitlePath string
	VideoPath    string
	Ref          resolve.FileRef
	DryRun       bool
}

ExecInput is one job's work order, resolved at accept time (the handler validates and resolves before the 202; the executor reads the file at run time, so the analysis sees current content).

type ExecResult

type ExecResult struct {
	Err        error
	Method     string
	Outcome    subflux.JobOutcome
	OffsetMs   int64
	Confidence float64
	Applied    bool
}

ExecResult is the executor's answer for one job. OffsetMs is CUMULATIVE (the stored offset plus this run's correction).

type Job

type Job struct {
	AcceptedAt      time.Time          `json:"accepted_at"`
	StartedAt       *time.Time         `json:"started_at,omitempty"`
	EndedAt         *time.Time         `json:"ended_at,omitempty"`
	BatchActivityID string             `json:"batch_activity_id,omitempty"`
	Method          string             `json:"method,omitempty"`
	Error           string             `json:"error,omitempty"`
	State           JobState           `json:"state"`
	Outcome         subflux.JobOutcome `json:"outcome,omitempty"`
	ActivityID      string             `json:"activity_id"`
	FileRef         resolve.FileRef    `json:"file_ref"`
	JobID           int64              `json:"job_id"`
	OffsetMs        int64              `json:"offset_ms,omitempty"`
	SeriesID        int                `json:"series_id,omitempty"`
	Season          int                `json:"season,omitempty"`
	Ordinal         int                `json:"ordinal,omitempty"`
	Confidence      float64            `json:"confidence,omitempty"`
	Applied         bool               `json:"applied,omitempty"`
	DryRun          bool               `json:"dry_run,omitempty"`
}

Job is one sync job record: the wire shape GET /api/sync/jobs serves. JobID is a NUMERIC process sequence — distinct from the activity log's string ids, which the record carries beside it. The registry is truth independent of the activity ring; a restart drops it (jobs are not persisted — the client re-attaches via this read and finds nothing).

type JobState

type JobState string

JobState is a job's lifecycle position. (Named for the flat TS wire namespace, where a bare State would collide with the activity vocabulary.)

const (
	StateQueued  JobState = "queued"
	StateRunning JobState = "running"
	StateDone    JobState = "done"
)

Job states: queued → running (at the admission hook) → done.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL