scheduler

package
v0.0.26 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 8, 2026 License: Apache-2.0 Imports: 12 Imported by: 0

Documentation

Overview

Package scheduler is the in-process tick loop for the scheduled-tasks feature (issue #189, Phase 1e). It is the composition-layer owner of:

  • the leader-lease on the well-known `port.SchedulerLeaderLeaseID` ("__scheduler__") so that, in a multi-replica deployment, at most one replica ticks the schedule store at a time;
  • the tick loop that polls `port.ScheduleStore.Due`, applies the misfire policy, claims each due slot via `port.ScheduleStore.Claim` (the at-most-once atomic advance), fires the claimed schedule through the composition-supplied FireFunc seam, and records the outcome via `port.ScheduleStore.RecordFire`.

It is STORAGE-AGNOSTIC: like the run-entry lease renewer on `internal/adapter/server.Service`, the loop NEVER imports `engine/agent` or `internal/adapter/server`. The FireFunc seam is how composition injects the run-entry funnel (Service.CreateSessionWithProfile + Service.StartRunContent with subagent-grade RunRequest) in Phase 1f. A unit test supplies a stub FireFunc that records fires.

The loop reads `now` from an injected `port.Clock` (deterministic tests) and logs through an injected `port.Diagnostics` (NEVER slog — the global-slog ban in engine/ and internal/ applies here too; see ADR 0020).

Index

Constants

This section is empty.

Variables

View Source
var ErrFireNowDisabled = errors.New("scheduler: fire-now rejected (schedule disabled)")

ErrFireNowDisabled is returned by FireNow when the schedule is not Enabled (paused or done). A paused/done schedule cannot be manually fired. The caller maps it to FailedPrecondition.

View Source
var ErrFireNowExhausted = errors.New("scheduler: fire-now rejected (one-shot already fired)")

ErrFireNowExhausted is returned by FireNow when a one-shot schedule has already fired (FireCount > 0). A one-shot fires once; a manual re-fire of a completed one-shot is rejected. The caller maps it to FailedPrecondition.

View Source
var ErrFireNowOverlap = errors.New("scheduler: fire-now skipped (prior fire still running)")

ErrFireNowOverlap is returned by FireNow when the schedule's singleton overlap check found a prior fire still running (its session lease is held by any replica). The fire was SKIPPED, not claimed — the caller (composition's Service.FireNow) maps it to FailedPrecondition / 409 so a client distinguishes "overlapping fire rejected" from a genuine error. It mirrors the tick loop's silent singleton skip, surfaced as an explicit error on the manual path (a manual fire is a client-initiated request that deserves an explicit rejection, unlike the tick loop's best-effort skip).

View Source
var ErrNotLeader = errors.New("scheduler: not the leader (a peer replica is; retry against the leader)")

ErrNotLeader is returned by FireNow when this replica is not the scheduler leader (a multi-replica deployment where a peer holds the `__scheduler__` lease). A standby replica fails a manual fire fast rather than double-firing against the leader's tick loop. The caller maps it to FailedPrecondition and SHOULD surface the current leader (Scheduler.LeaderOwner) so a client can redirect. Nil-lease (single-replica) schedulers are always the leader.

Functions

func LoadLocation

func LoadLocation(tz string) *time.Location

LoadLocation loads the IANA timezone for a schedule's cron expression. An empty or invalid timezone falls back to UTC (fail-safe). It is the exported seam for composition's create-seam (the first NextFireAt computation) so it shares the tick loop's tz loader rather than duplicating it.

Types

type Config

type Config struct {
	// Store is the durable schedule registry. Required.
	Store port.ScheduleStore
	// Lease is the cross-process leader-lease backend for the
	// `port.SchedulerLeaderLeaseID` ("__scheduler__") leader lease. nil means
	// single-replica by affinity: the scheduler runs standalone (no
	// cross-process leader gate) — the byte-identical default when no lease
	// backend is wired, exactly as the run-entry lease defaults off.
	Lease port.SessionLease
	// LeaseOwner is the per-process owner string composition builds once per
	// Build (the same "<hostname>-<pid>-<nonce>" shape the run-entry seam
	// uses). Required when Lease != nil; ignored otherwise.
	LeaseOwner string
	// LeaseTTL is the leader-lease lifetime requested at Acquire. A non-positive
	// value defaults to defaultLeaderLeaseTTL. Ignored when Lease == nil.
	LeaseTTL time.Duration
	// LeaseRenewInterval is how often the leader-lease renewer refreshes the
	// held lease. A non-positive value defaults to LeaseTTL/3. Ignored when
	// Lease == nil.
	LeaseRenewInterval time.Duration
	// Fire is the composition-supplied run-entry callback. Composition wires
	// this in Phase 1f; for Phase 1e's unit tests a stub records fires. Required.
	Fire FireFunc
	// CanProcess admits only schedules this scheduler may touch. A nil function
	// preserves the compatibility path. OIDC composition supplies a predicate that
	// excludes ownerless pre-cutover schedules before Claim, so a background worker
	// cannot adopt, fire, or repeatedly mutate an inaccessible resource.
	CanProcess func(port.Schedule) bool
	// PresentScheduleName projects the authoritative stored schedule to its
	// caller-visible name for lifecycle events and metric labels. It must never be
	// used for store operations; nil preserves Spec.Name.
	PresentScheduleName func(port.Schedule) string
	// Clock supplies `now` for the tick loop and Claim. Required.
	Clock port.Clock
	// Diagnostics is the operational logging seam. A nil value is treated as
	// port.NopDiagnostics so the scheduler is nil-safe by construction.
	Diagnostics port.Diagnostics
	// TickInterval is how often the tick loop polls ScheduleStore.Due. A
	// non-positive value defaults to defaultTickInterval.
	TickInterval time.Duration
	// MinInterval is the frequency floor the composition create-seam enforces at
	// Save time (a schedule whose cadence is tighter than this is rejected,
	// fail-closed). It is NOT read by the tick loop — it lives on Config so a
	// future self-pushing lookahead can consult it without widening the
	// constructor. Documented here to keep it honest.
	MinInterval time.Duration
	// MaxConcurrentFires bounds the per-tick fire fan-out via an errgroup with
	// SetLimit. A non-positive value defaults to defaultMaxConcurrentFires.
	MaxConcurrentFires int
	// StopFireGrace is how long Stop waits for in-flight fires to drain before
	// abandoning them. A non-positive value defaults to stopFireGrace. It is a
	// Config field (not a flag) so a test can shrink it.
	StopFireGrace time.Duration
	// EmitScheduleEvent is the OPTIONAL composition-injected callback the
	// scheduler invokes to emit an EvScheduleFired/Skipped/Failed event. It is
	// nil-safe (nil = no event emitted — the byte-identical no-emit path).
	// Composition wires it to emit into the fire session's event log / the
	// Service's event sink. The scheduler pkg stays EventSink-free (testable, no
	// engine/agent import): the payload is a plain session.SchedulePayload value
	// object, not an EventSink/port import. The scheduler invokes it from
	// fireClaimed (fired/failed) and fireOne/FireNow (skipped) — the caller
	// decides the kind; the callback decides where it lands. The ctx is the
	// firing caller's, so the durable append can attribute the event to whoever
	// acted (a tick fire descends from Start's system-principal root; a manual
	// FireNow keeps its requester) — ADR 0204 decision 5.
	EmitScheduleEvent func(ctx context.Context, payload session.SchedulePayload)
	// ScheduleMetrics is the OPTIONAL composition-injected metrics callback
	// (issue #233, Phase 2b). It is nil-safe (nil = no metrics recorded — the
	// byte-identical no-metrics path). The scheduler invokes it from
	// fireClaimed (fired/failed, with the Claim→terminal duration) and
	// fireOne/FireNow (skipped, duration 0). Composition wires it over the
	// telemetry adapter's Metrics.EmitSchedule — the scheduler pkg stays
	// telemetry-import-free (the metrics seam is a plain callback, mirroring
	// EmitScheduleEvent). The payload's Kind ("fired"/"skipped"/"failed")
	// labels the outcome; duration > 0 only for a fired/failed fire.
	ScheduleMetrics func(payload session.SchedulePayload, duration time.Duration)
	// DeliverFireResult is the OPTIONAL composition-injected callback
	// (ADR 0075, fire-result-delivery) the scheduler invokes from fireClaimed
	// AFTER RecordFire, to route a fire's terminal result back into its origin
	// conversation. It is nil-safe (nil = the byte-identical no-delivery path,
	// matching the pre-ADR-0075 pull-only posture). Composition wires it to
	// deliverFireResult(svc, queue) which: skips an empty OriginSessionID (no
	// delivery), renders the note (renderFireDelivery), enqueues it to the
	// durable DeliveryQueue, and drives a delivery run into an idle/completed/
	// cancelled/failed origin via StartRunContent (loadAndReopen reopens/
	// recovers); a BUSY or AWAITING origin is left queued (the loop's Step 2a
	// drain records it at the next turn boundary); a deleted/child/sched--
	// origin degrades to pull-only with a WARN. A delivery error WARNs and
	// NEVER fails the fire (delivery is decoupled — the fire already recorded).
	// The scheduler invokes it for fired/failed fires alike (a failed fire may
	// still have an origin that should know it errored); the callback decides
	// whether to deliver based on the stop reason (it may skip a StopError
	// fire's delivery, or deliver it — the ADR does not mandate either).
	DeliverFireResult func(ctx context.Context, sched port.Schedule, fire port.ScheduleFire)
	// ReconcileStaleFire is the OPTIONAL composition-injected callback
	// (issue #386 Phase 4b, the stale-fire reconciler) the scheduler invokes
	// from the tick loop's reconcile scan when it DETECTS a stale in-flight
	// fire — a claimed-but-never-terminal fire left behind by a crashed
	// process (acceptance criterion #7). It is nil-safe (nil = the
	// byte-identical no-reconcile path, the pre-Phase-4b posture): detection
	// is store-only (the scheduler CAN do it — it reads ScheduleStore + the
	// leader-lease/isPriorFireLive seam it already has), but the SETTLE
	// (session-load + cancel + RecordFire) needs Service methods the scheduler
	// package must not import (the layering rule: the scheduler is
	// storage-agnostic and must NOT import internal/adapter/server). So the
	// reconcile callback is composition-injected, mirroring Fire/
	// DeliverFireResult/DeliverFireStarted.
	//
	// Two crash cases the detector hands the callback:
	//  1. Crash after Claim, before session creation: LastFireSessionID ==
	//     port.PendingFireSessionID (the sentinel) and stale (LastFireAt older
	//     than the stale window). No session exists. The callback records a
	//     terminal StopError fire ("fire lost: process crashed between claim
	//     and session creation") via RecordFire with a minted id, and clears
	//     the in-flight fields.
	//  2. Crash after session creation, before RecordFire: LastFireSessionID
	//     is a real "sched--" id, the fire record is still in-flight (Stop
	//     empty), the session's lease is released/expired (not live — the
	//     detector's isPriorFireLive trial-lease check acquired freely), and
	//     LastFireStartedAt is older than the stale window. The callback
	//     settles the terminal: marks the session snapshot cancelled
	//     (Interrupt-recoverable) and RecordFire-ing a StopError fire ("fire
	//     lost: process crashed during run").
	//
	// The detector must NOT flag a genuinely-live fire (lease held → skip), and
	// must NOT flag a freshly-claimed fire within the stale window (a fire
	// Claimed moments ago is in flight, not crashed). The callback is
	// idempotent: a fire already terminal is a no-op (RecordFire is idempotent
	// per fire id). Composition wires it over svc.GetSession/svc.Persist/
	// store.RecordFire (reusing settleFireTerminalSnapshot from #388 for the
	// session-settle).
	ReconcileStaleFire func(ctx context.Context, sched port.Schedule)
}

Config wires the scheduler. All fields are set by composition (Phase 1f); the unit tests construct one directly with a stub Fire.

type FireFunc

type FireFunc func(ctx context.Context, sched port.Schedule, now time.Time) (port.ScheduleFire, error)

FireFunc is the composition-supplied callback a scheduler invokes for each claimed fire. It mints a fresh session (the "sched--" top-level session per fire, via Service.CreateSessionWithProfile + Service.StartRunContent with subagent-grade RunRequest), drives it to terminal, and returns the fire record (stop reason + err). The scheduler records the fire outcome via ScheduleStore.RecordFire. A non-nil error from FireFunc is recorded as a failed fire (StopError); the at-most-once Claim already advanced NextFireAt, so a failed fire is NOT retried (the slot is gone — decision #1).

type Scheduler

type Scheduler struct {
	// contains filtered or unexported fields
}

Scheduler is the composition-layer owner of the scheduled-tasks tick loop. Construct one via New, then Start (which acquires the leader lease if a backend is wired and launches the tick + renewer goroutines), and Stop it at shutdown / drain.

func New

func New(cfg Config) *Scheduler

New constructs a Scheduler. It applies Config defaults (TTLs, intervals, fan-out, a NopDiagnostics sink) but does NOT acquire the lease or start any goroutine — call Start. A nil Clock or Store is a programming error at the only construction site (composition); New panics so it surfaces loudly there rather than as a nil-deref in the tick loop. Fire MAY be nil at New (the late-bind seam: composition calls SetFire after NewService, before Start); a nil Fire at Start panics.

func (*Scheduler) Done

func (s *Scheduler) Done() <-chan struct{}

Done returns a channel closed when Stop completes. Tests may wait on it to assert clean shutdown.

func (*Scheduler) Drain

func (s *Scheduler) Drain()

Drain arms the drain gate: in-flight fires complete, but no NEW fires start mid-tick. It mirrors Service.Drain: a shutting-down replica steers its tick-loop work to a survivor. It does NOT cancel in-flight fires (they complete or are cancelled by Stop's grace). It returns immediately.

func (*Scheduler) FireNow

func (s *Scheduler) FireNow(ctx context.Context, name string, now time.Time) (port.ScheduleFire, error)

FireNow manually fires a schedule by name: it Loads the schedule, applies the singleton overlap check (the same trial-lease path fireOne uses), Claims the slot, and runs fireClaimed. It is the manual/ad-hoc fire path (a client or operator triggers a fire out-of-band from the tick loop). It returns the fire record (stop reason + any error) and emits the EvSchedule* event via the callback (fired/failed/skipped) exactly as the tick loop does.

FAIL-CLOSED prelude (the caller's job, NOT the at-most-once Claim):

  • a not-enabled (paused/done) schedule → ErrFireNowDisabled;
  • an already-fired one-shot (FireCount > 0) → ErrFireNowExhausted;
  • a singleton schedule whose prior fire is still running → ErrFireNowOverlap (the slot is NOT claimed — the skip path, mirroring the tick loop's singleton skip).

A cron schedule that is due now or in the future is Claimed and fired. A cron whose NextFireAt is in the past is ALSO fired (the manual path is an explicit request — it does not apply the misfire policy, which is a tick-loop concern for polling cadence). The nextFire handed to ClaimNow is computed via computeNextFire (the same helper the tick loop uses).

func (*Scheduler) IsDraining

func (s *Scheduler) IsDraining() bool

IsDraining reports whether the drain gate is armed.

func (*Scheduler) LeaderOwner

func (s *Scheduler) LeaderOwner() (owner string, leader bool)

LeaderOwner reports whether this replica may fire (it is the scheduler leader, or there is no leader gate at all) and, if so, its lease-owner identity. It backs the FireNow not-leader redirect surface (a standby replica names the leader a client should retry against). A scheduler with no lease backend (single-replica by affinity) has NO leader gate, so it always reports leader=true. A lease-backed scheduler that has not been Started (the unit-test direct-FireNow path) has no standby epoch yet, so it also reports leader=true — the gate bites only for a STARTED lease-backed scheduler currently in standby.

func (*Scheduler) RunOnceForTest

func (s *Scheduler) RunOnceForTest(ctx context.Context)

RunOnceForTest runs a single tick iteration synchronously (no ticker). It is the test seam for driving the loop deterministically: a test advances the fake clock and calls RunOnceForTest to express "one tick elapsed" without a real sleep. Production drives the loop via Start/Stop (the real ticker).

If Start has been called, the iteration runs under the scheduler's OWN tick ctx (so a Stop cancels in-flight fires exactly as the real ticker would); if Start has NOT been called, it runs under the passed ctx (the deterministic at-most-once / misfire tests do not need Start).

func (*Scheduler) SetCanProcess

func (s *Scheduler) SetCanProcess(fn func(port.Schedule) bool)

SetCanProcess replaces the pre-claim schedule eligibility predicate before Start. Composition uses it to add Service-owned authority checks after the Service exists.

func (*Scheduler) SetDeliverFireResult

func (s *Scheduler) SetDeliverFireResult(cb func(ctx context.Context, sched port.Schedule, fire port.ScheduleFire))

SetDeliverFireResult wires the OPTIONAL composition-injected fire-result delivery callback (ADR 0075). Composition calls it after SetFire (so the FireFunc is bound) and before Start. nil is the byte-identical no-delivery path (the pre-ADR-0075 pull-only posture). The scheduler invokes it from fireClaimed AFTER RecordFire, with the schedule + the fire record.

func (*Scheduler) SetEmitScheduleEvent

func (s *Scheduler) SetEmitScheduleEvent(cb func(ctx context.Context, payload session.SchedulePayload))

SetEmitScheduleEvent sets the OPTIONAL composition-injected emit callback. It MUST be called before Start (the late-bind seam, parallel to SetFire). A nil callback is the byte-identical no-emit path (no EvSchedule* events emitted — the scheduler is fully functional, just silent on the schedule lifecycle). Composition calls it after SetFire (so the FireFunc is bound) and before Start (so the callback is in place when the first tick fires).

func (*Scheduler) SetFire

func (s *Scheduler) SetFire(f FireFunc)

SetFire sets the composition-supplied FireFunc. It MUST be called before Start (Start panics if Fire is nil — composition wires the run-entry funnel here). It is the late-bind seam: buildScheduler constructs the Scheduler with the store/lease/clock but no Fire (the Service does not exist yet), Build calls SetFire after NewService, then Start.

func (*Scheduler) SetPresentScheduleName

func (s *Scheduler) SetPresentScheduleName(fn func(port.Schedule) string)

SetPresentScheduleName wires the optional physical-to-literal presentation seam before Start. Nil preserves identity.

func (*Scheduler) SetReconcileStaleFire

func (s *Scheduler) SetReconcileStaleFire(cb func(ctx context.Context, sched port.Schedule))

SetReconcileStaleFire wires the OPTIONAL composition-injected stale-fire reconcile callback (issue #386 Phase 4b). Composition calls it after SetFire (so the FireFunc is bound) and before Start. nil is the byte-identical no-reconcile path (the pre-Phase-4b posture): the reconcile scan is a nil-safe skip when the callback is unwired. The scheduler invokes it from the tick loop's reconcile scan (reconcileStaleFires, called from tickOnce) for each detected stale in-flight fire, with the schedule whose LastFireSessionID/LastFireStartedAt mark it crashed.

func (*Scheduler) SetScheduleMetrics

func (s *Scheduler) SetScheduleMetrics(cb func(payload session.SchedulePayload, duration time.Duration))

SetScheduleMetrics sets the OPTIONAL composition-injected metrics callback (issue #233, Phase 2b). It is the late-bind seam, parallel to SetEmitScheduleEvent: a nil callback is the byte-identical no-metrics path. Composition calls it after SetFire (so the FireFunc is bound) and before Start (so the callback is in place when the first tick fires).

func (*Scheduler) Start

func (s *Scheduler) Start(ctx context.Context) error

Start launches the scheduler's lifecycle. It is INFALLIBLE-AT-LAUNCH: it never returns a leader-lease error. With no lease backend (cfg.Lease == nil, single-replica by affinity) it starts the tick loop directly. With a lease backend it launches a background LEADERSHIP LOOP that acquires the `__scheduler__` leader lease and keeps this replica in one of two states:

  • LEADER: holds the lease, ticks + renews (a leadership EPOCH). On a definitive lease loss (a peer took over) the epoch is torn down and the replica returns to standby.
  • STANDBY (non-leader): not ticking. Retries the acquire on a backoff ticker and PROMOTES when the current leader's lease lapses (crash/TTL) or is released. This is the multi-replica availability contract: a standby replica must SERVE (report ready, answer RPCs) and take over when the leader dies — it must NOT crash (the pre-ADR-0073 on-by-default bug where a non-leader's Start returned ErrLeaseHeld and the process exited, CrashLooping the replica).

A genuine infrastructure fault on the acquire (NOT ErrLeaseHeld / not ErrLeaseUnsupported) is treated like contention: logged and retried on the same backoff — a wedged lease backend degrades scheduling to standby rather than crashing the process. Start is idempotent: a second call is a no-op. FireNow is GATED on leadership (ErrNotLeader) so a standby replica fails a manual fire fast rather than double-firing against the leader.

func (*Scheduler) Stop

func (s *Scheduler) Stop() error

Stop cancels the leadership loop and the current epoch (tick + renewer), waits for in-flight fires to drain (with a grace period), releases the leader lease (if held), and closes done. It is idempotent: a second call is a no-op that returns nil. The leadership-loop and epoch joins are BOUNDED (a goroutine that ignores its cancel ctx is abandoned after stopLeadershipJoinTimeout, never allowed to stall shutdown unboundedly); the fire-grace wait and lease release are likewise bounded.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL