scheduler

package
v0.2.7 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 26, 2026 License: MIT Imports: 39 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func GenerateDispatchToken added in v0.1.4

func GenerateDispatchToken() (string, error)

GenerateDispatchToken returns a fresh 256-bit hex dispatch token. The host mints one when config.toml carries no explicit [dispatch].token and persists it back into config.toml (see config.EnsureDispatchToken) so it rides into the Linux VM through the existing config-delivery channel (initrd on Windows, shared data dir on macOS) and the in-VM server reads the identical value.

func SocketPath

func SocketPath(dataDir string) string

SocketPath returns the path to the gRPC control socket for a given data dir.

Types

type Config

type Config struct {
	Runtime         *runtime.Runtime
	Providers       []providers.Provider
	Artifacts       *artifacts.Extractor // OCI image layer extractor for macOS VM jobs (nil if not available)
	LinuxDispatcher *DispatchClient      // if non-nil, Linux jobs are dispatched to a Linux VM worker via gRPC
	// LinuxJobsDisabled makes canHandleJob refuse Linux-labeled jobs even on
	// a host that could run them ([vm.linux] enabled = false on darwin). Used
	// when a dedicated Linux box serves the same labels: without it, both
	// daemons register a JIT runner per queued job and the loser's runner
	// squats as an orphan until the grace sweep.
	LinuxJobsDisabled bool
	MacOSVMConfig     *vm.MacOSVMConfig // if non-nil, macOS-native jobs are enabled (darwin only)
	// CachePruner reclaims disk held by daemon-managed caches (BuildKit's
	// build cache, the containerd image store) on demand, through the
	// manager that owns each one. Backs the PruneCache RPC that
	// `ephemerd cache clear` uses so an operator no longer has to stop the
	// daemon and delete directories. Nil makes PruneCache return
	// Unimplemented.
	CachePruner       cacheprune.Interface
	DataDir           string // ephemerd data directory (used for artifact extraction paths)
	Version           string // daemon build version (from main.version); surfaced via Status and used by the Upgrade RPC
	MaxConcurrent     int
	MaxMacOSVMs       int // max concurrent macOS VMs (Vz limit; default auto-detected)
	Labels            []string
	PollInterval      time.Duration   // if >0, use polling mode (default)
	ReconcileInterval time.Duration   // webhook mode: periodic catch-up sweep for stranded jobs (0 = disabled)
	WebhookPort       int             // listen port for health/webhook server
	WebhookSecret     string          // webhook signature secret
	TLSCert           string          // TLS certificate path
	TLSKey            string          // TLS private key path
	Tunnel            tunnel.Provider // if non-nil, creates a public tunnel for webhooks
	TunnelMaxRetries  int             // max consecutive reconnect failures before fallback to polling (0 = default 5)

	// ExternalURL is the public base URL of an externally-managed tunnel
	// (tunnel = "external"). When set alongside webhook mode and NO managed
	// Tunnel, the scheduler registers each webhook-capable provider's hook to
	// <ExternalURL>/webhook/<provider> on startup, so the operator doesn't
	// have to hand-add a hook per repo. External hooks are operator-owned and
	// are NOT deregistered on shutdown. Empty means "receiver only, don't
	// touch the platform's webhooks".
	ExternalURL     string
	JobTimeout      time.Duration
	ShutdownTimeout time.Duration
	LogRetention    time.Duration // max age for job log files (default 7d)

	// MacOSProvisionTimeout bounds the pre-registration provisioning phase of
	// a macOS VM job — booting the VM and waiting for its runner to become
	// reachable (handleMacOSJob → MacOSVM.WaitForRunner). If the wait has not
	// returned by this deadline the VM is force-stopped so the reachability
	// wait unblocks, the job is failed, and the single macOS concurrency slot
	// is released. Guards against a hung guest SSH command wedging the wait
	// indefinitely (the VM-internal loop caps itself at ~2 min, but its SSH
	// session calls carry no deadline and ignore ctx). Zero applies
	// defaultMacOSProvisionTimeout.
	MacOSProvisionTimeout time.Duration

	// Retry configures the claim/provision retry queue. When the initial
	// attempt to claim a queued job fails with a retryable error
	// (rate-limit exhausted, transient 5xx, network), the job is
	// enqueued and re-attempted on a backoff ladder rather than lost.
	// GitHub does not re-deliver workflow_job webhooks. Leave zero-valued
	// (Enabled=false) to keep the pre-existing "log and drop" behavior.
	Retry RetryConfig

	// OrphanSweep configures teardown of dispatched runners that were
	// never observed picking up a job. GitHub schedules JIT runners onto
	// ANY queued job with matching labels, so the runner dispatched "for"
	// a job may end up running a different one — leaving the runner that
	// was dispatched for THAT job idle with no job-completion event ever
	// pointing at it. The sweep destroys such runners once they have been
	// idle-unbound for Grace. Only active in webhook mode and only for
	// runners dispatched via providers that report runner assignments
	// (providers.RunnerNameReporter) — otherwise "never observed bound"
	// would just mean "we had no way to observe it".
	OrphanSweep OrphanSweepConfig

	// RunnerImageForRepo resolves the per-repo, per-OS image override
	// configured under [runner.images]. Returns "" when no override is
	// set; the scheduler then falls back to the provider per-OS default
	// and finally the runtime's host-aware default. Nil-safe.
	RunnerImageForRepo func(repo, os string) string

	Log *slog.Logger
}

Config for the scheduler.

type DispatchClient

type DispatchClient struct {
	// contains filtered or unexported fields
}

DispatchClient dispatches Linux jobs to the WSL worker via gRPC.

func NewDispatchClient

func NewDispatchClient(addr, token string) (*DispatchClient, error)

NewDispatchClient connects to the dispatch gRPC server at the given address. When token is non-empty it is attached to every RPC via per-RPC credentials so the server-side interceptor accepts the call; an empty token sends no credential (used by tests against an unauthenticated fake server).

The transport stays insecure (no TLS): the dispatch link is a host<->VM loopback/NAT hop, and the bearer token — not the transport — authenticates the caller.

func (*DispatchClient) Close

func (d *DispatchClient) Close() error

Close closes the gRPC connection.

func (*DispatchClient) ConsumeContainerStats

func (d *DispatchClient) ConsumeContainerStats(ctx context.Context, intervalSeconds uint32, runtimeLabel string, log *slog.Logger) error

ConsumeContainerStats opens the StreamContainerStats stream, asks the in-VM dispatch server for samples at the given interval, and feeds each batch into the host metrics registry under the supplied runtime label (typically metrics.RuntimeLinuxVM). The call returns when ctx is cancelled. On non-fatal stream errors the consumer reconnects with backoff so a transient network blip in the VM doesn't lose metrics for the rest of the daemon's lifetime.

Series are deleted via metrics.DeleteContainerSeries when the consumer sees a container id stop appearing in batches (the in-VM UnregisterSampler removes it from the stream).

func (*DispatchClient) Create

func (d *DispatchClient) Create(ctx context.Context, id, image, jitConfig, provider, repo string) error

Create dispatches a container create to the WSL worker. provider + repo are passed through so the VM-side dind server can scope its per-repo image cache namespace to (provider, repo) and not leak private images across forges or repos.

func (*DispatchClient) Destroy

func (d *DispatchClient) Destroy(ctx context.Context, id string) error

Destroy tears down the dispatched job's container.

func (*DispatchClient) Wait

func (d *DispatchClient) Wait(ctx context.Context, id string) (uint32, error)

Wait blocks until the dispatched job exits and returns its exit code.

type DispatchServer

type DispatchServer struct {
	apiv1.UnimplementedDispatchServer
	// contains filtered or unexported fields
}

DispatchServer implements the Dispatch gRPC service. It runs inside the WSL containerd-only worker and proxies Create/Wait/Destroy calls to the local Linux Runtime. It also serves StreamContainerStats so the host can scrape per-container resource metrics on its own /metrics endpoint without exposing a second listener inside the VM. See docs/arch/container-metrics.md.

func NewDispatchServer

func NewDispatchServer(rt *runtime.Runtime, log *slog.Logger, defaultInterval time.Duration, linuxCPULimit, linuxMemLimitBytes uint64) *DispatchServer

NewDispatchServer constructs a Dispatch service handler. defaultInterval is used when the StreamContainerStats client passes interval_seconds=0; linuxCPULimit / linuxMemLimitBytes are baked into each per-container sampler as the configured cap.

func StartDispatchServer

func StartDispatchServer(cfg DispatchServerConfig) (*DispatchServer, func())

StartDispatchServer starts the dispatch gRPC server on the given TCP port and returns the running server instance plus a cleanup function that gracefully stops it. The returned *DispatchServer exposes RegisterSampler / UnregisterSampler so the local runtime can plumb its OnTaskStarted / OnTaskDestroy hooks into the stats stream surface area.

Binds to cfg.BindAddr (default 0.0.0.0) so the host (outside the VM) can reach it. WSL on Windows shares localhost with the host, so this used to be 127.0.0.1, but the same process is now invoked from inside an Apple Vz VM where the host lives on the NAT side and needs the listener exposed on the VM's external interface.

The gRPC surface (CreateJob/WaitJob/DestroyJob/StreamContainerStats) exposes container lifecycle control with a caller-supplied image + JIT config, so it MUST NOT be reachable unauthenticated by anything sharing the VM's network (notably job containers). When cfg.Token is set, every RPC is gated by a constant-time bearer-token check via unary + stream interceptors. Operators must additionally firewall the dispatch port off from job containers (the worker installs bridge control-port rules for exactly this — see main.go controlPorts).

func (*DispatchServer) CreateJob

func (*DispatchServer) DestroyJob

func (*DispatchServer) RegisterSampler

func (s *DispatchServer) RegisterSampler(id, repo string, sampler metrics.Sampler)

RegisterSampler is called by the in-VM runtime's OnTaskStarted hook to expose a container's sampler to StreamContainerStats subscribers.

func (*DispatchServer) StreamContainerStats

StreamContainerStats serves the long-lived sampling stream that the host uses to surface per-container resource series. The handler ticks at the client-requested cadence and sends one batch per tick covering every registered sampler. Returns when the client cancels the context, when the underlying connection drops, or when Send fails for any reason.

func (*DispatchServer) UnregisterSampler

func (s *DispatchServer) UnregisterSampler(id string)

UnregisterSampler removes a container's sampler from the stream set, called by the runtime's OnTaskDestroy hook.

func (*DispatchServer) WaitJob

type DispatchServerConfig

type DispatchServerConfig struct {
	Port               int
	Runtime            *runtime.Runtime
	Log                *slog.Logger
	StatsInterval      time.Duration // default 10s when zero
	LinuxCPULimit      uint64        // 0 = unlimited
	LinuxMemLimitBytes uint64        // 0 = unlimited

	// Token is the shared bearer token the server requires on every RPC. When
	// non-empty, unary + stream interceptors reject any call that does not
	// present a matching token (constant-time compare). Empty disables auth —
	// which should only happen in tests or misconfigured setups; production
	// callers plumb a token from the shared data dir (see
	// LoadOrCreateDispatchToken). The server logs loudly when it starts
	// unauthenticated so the footgun is visible.
	Token string

	// BindAddr is the interface the listener binds to. Empty defaults to
	// "0.0.0.0" so the Vz/Hyper-V host on the NAT side can reach it (a narrower
	// bind is only safe when the host address is known; see the comment on
	// StartDispatchServer). The token check protects the surface regardless of
	// bind, so 0.0.0.0 is defense-in-depth-behind-auth rather than the sole
	// control.
	BindAddr string
}

DispatchServerConfig configures the in-VM dispatch gRPC server.

type OrphanSweepConfig

type OrphanSweepConfig struct {
	// Enabled toggles the sweep.
	Enabled bool

	// Grace is how long a dispatched runner may remain unbound (never
	// seen in an in_progress event) before it is destroyed. Defaults to
	// 10 minutes when zero.
	Grace time.Duration
}

OrphanSweepConfig tunes the orphaned-runner sweep. Zero-valued = disabled (matching pre-existing behavior); the CLI enables it by default with a 10-minute grace window.

type RetryConfig

type RetryConfig struct {
	// Enabled toggles the entire retry queue. When false, failures still
	// log and drop as before (no behavior change).
	Enabled bool

	// Schedule is the ordered backoff ladder. Each entry is the base
	// delay for that attempt; jitter is applied on top. If nil, defaults
	// to {30s, 1m, 2m, 5m, 10m}.
	Schedule []time.Duration

	// MaxAge is the wall-clock budget from first failure to giving up.
	// Once (now - firstFailure) > MaxAge, we log a WARN and drop. Default 90m.
	MaxAge time.Duration

	// Jitter is the fraction (0-1) of a delay that's randomized +/- around
	// the base value. Set to a NEGATIVE value (e.g. -1) to request the
	// default (0.2 = +/-20%). Literal 0 is honored  -  tests use it for
	// deterministic scheduling.
	Jitter float64

	// RateHint returns the last-observed GitHub rate-limit state. When
	// remaining == 0 and now < reset and updated is fresh (<5m old),
	// the next attempt is snapped to reset + a small jitter instead of
	// the Schedule entry. Nil-safe: nil means "no rate awareness".
	RateHint func() (remaining int64, reset time.Time, updated time.Time)

	// Now is the clock function. Defaults to time.Now. Tests inject
	// a fake clock so backoff scheduling is deterministic.
	Now func() time.Time
}

RetryConfig tunes the claim retry queue.

A zero-valued RetryConfig disables retries entirely, matching the pre-existing "log-and-drop" behavior. In practice New() applies sensible defaults so callers get retries by default.

type Scheduler

type Scheduler struct {
	// contains filtered or unexported fields
}

Scheduler ties CI provider job events to container lifecycle. When a job is queued, it provisions a runner environment. When the job completes, it destroys the environment.

func New

func New(cfg Config) *Scheduler

New creates a scheduler.

func (*Scheduler) ActiveJobs added in v0.1.7

func (s *Scheduler) ActiveJobs() int

ActiveJobs returns the number of jobs currently running. Used by the Upgrade RPC (via the upgrade.Drainer interface) to wait for the scheduler to drain to idle after a Cordon, the same signal `ephemerd drain --wait` polls over the control socket.

func (*Scheduler) Cordon added in v0.1.5

func (s *Scheduler) Cordon() int

Cordon marks the scheduler as draining WITHOUT initiating shutdown: new queued jobs are rejected while running jobs continue undisturbed. Returns the number of jobs still running. Used by the Cordon RPC so `ephemerd drain --wait` can stop claims first and only restart the daemon once the active job count reaches zero.

The flag is enforced at three points, all of which read it live rather than snapshotting it at admission (issue #154):

  • handleQueued — refuses new queued events, whatever their source (webhook, poll, startup catch-up, reconcile sweep, retry-queue fire).
  • admitDispatch — abandons a dispatch that was accepted before the cordon and then sat blocked on a concurrency semaphore. This is the one that was missing: the node kept provisioning for over a minute after the operator was told it had stopped claiming.
  • claimJob — the hard backstop. Nothing can register a JIT runner on a cordoned node, even from a dispatch path that skips the gates above.

Cordon means "stop claiming NEW work". Jobs already running are untouched: their contexts hang off jobsCtx and are never cancelled here.

func (*Scheduler) Run

func (s *Scheduler) Run(ctx context.Context) error

Run starts the scheduler. It discovers jobs via polling (default) or webhooks (when TLS certs are configured), and manages runner lifecycle.

func (*Scheduler) SetMacOSVMConfig

func (s *Scheduler) SetMacOSVMConfig(cfg *vm.MacOSVMConfig)

SetMacOSVMConfig enables macOS job support after startup. This is used when the macOS disk image is being provisioned in the background — the scheduler starts immediately for Linux jobs and picks up macOS jobs once the install finishes.

func (*Scheduler) StartVMSSHServer

func (s *Scheduler) StartVMSSHServer() (func(), error)

StartVMSSHServer starts a small HTTP server on the unix control socket for the VM SSH info endpoint. Called after the gRPC server is set up.

func (*Scheduler) Uncordon added in v0.1.5

func (s *Scheduler) Uncordon() int

Uncordon reverses Cordon: the scheduler resumes claiming queued jobs. Returns the number of jobs currently running. Jobs whose queued events were rejected while cordoned are picked up again by the next poll or reconcile sweep once their seen entry expires (seenTTL).

type VMSSHInfo

type VMSSHInfo struct {
	IP         string `json:"ip"`
	User       string `json:"user"`
	PrivateKey []byte `json:"private_key"` // PEM-encoded ed25519
}

VMSSHInfo contains the information needed to SSH into a macOS VM.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL