Documentation
¶
Index ¶
- func GenerateDispatchToken() (string, error)
- func SocketPath(dataDir string) string
- type Config
- type DispatchClient
- func (d *DispatchClient) Close() error
- func (d *DispatchClient) ConsumeContainerStats(ctx context.Context, intervalSeconds uint32, runtimeLabel string, ...) error
- func (d *DispatchClient) Create(ctx context.Context, id, image, jitConfig, provider, repo string) error
- func (d *DispatchClient) Destroy(ctx context.Context, id string) error
- func (d *DispatchClient) Wait(ctx context.Context, id string) (uint32, error)
- type DispatchServer
- func (s *DispatchServer) CreateJob(ctx context.Context, req *apiv1.CreateJobRequest) (*apiv1.CreateJobResponse, error)
- func (s *DispatchServer) DestroyJob(ctx context.Context, req *apiv1.DestroyJobRequest) (*apiv1.DestroyJobResponse, error)
- func (s *DispatchServer) RegisterSampler(id, repo string, sampler metrics.Sampler)
- func (s *DispatchServer) StreamContainerStats(req *apiv1.StreamContainerStatsRequest, ...) error
- func (s *DispatchServer) UnregisterSampler(id string)
- func (s *DispatchServer) WaitJob(ctx context.Context, req *apiv1.WaitJobRequest) (*apiv1.WaitJobResponse, error)
- type DispatchServerConfig
- type OrphanSweepConfig
- type RetryConfig
- type Scheduler
- type VMSSHInfo
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func GenerateDispatchToken ¶ added in v0.1.4
GenerateDispatchToken returns a fresh 256-bit hex dispatch token. The host mints one when config.toml carries no explicit [dispatch].token and persists it back into config.toml (see config.EnsureDispatchToken) so it rides into the Linux VM through the existing config-delivery channel (initrd on Windows, shared data dir on macOS) and the in-VM server reads the identical value.
func SocketPath ¶
SocketPath returns the path to the gRPC control socket for a given data dir.
Types ¶
type Config ¶
type Config struct {
Runtime *runtime.Runtime
Providers []providers.Provider
Artifacts *artifacts.Extractor // OCI image layer extractor for macOS VM jobs (nil if not available)
LinuxDispatcher *DispatchClient // if non-nil, Linux jobs are dispatched to a Linux VM worker via gRPC
// LinuxJobsDisabled makes canHandleJob refuse Linux-labeled jobs even on
// a host that could run them ([vm.linux] enabled = false on darwin). Used
// when a dedicated Linux box serves the same labels: without it, both
// daemons register a JIT runner per queued job and the loser's runner
// squats as an orphan until the grace sweep.
LinuxJobsDisabled bool
MacOSVMConfig *vm.MacOSVMConfig // if non-nil, macOS-native jobs are enabled (darwin only)
// CachePruner reclaims disk held by daemon-managed caches (BuildKit's
// build cache, the containerd image store) on demand, through the
// manager that owns each one. Backs the PruneCache RPC that
// `ephemerd cache clear` uses so an operator no longer has to stop the
// daemon and delete directories. Nil makes PruneCache return
// Unimplemented.
CachePruner cacheprune.Interface
DataDir string // ephemerd data directory (used for artifact extraction paths)
Version string // daemon build version (from main.version); surfaced via Status and used by the Upgrade RPC
MaxConcurrent int
MaxMacOSVMs int // max concurrent macOS VMs (Vz limit; default auto-detected)
Labels []string
PollInterval time.Duration // if >0, use polling mode (default)
ReconcileInterval time.Duration // webhook mode: periodic catch-up sweep for stranded jobs (0 = disabled)
WebhookPort int // listen port for health/webhook server
WebhookSecret string // webhook signature secret
TLSCert string // TLS certificate path
TLSKey string // TLS private key path
Tunnel tunnel.Provider // if non-nil, creates a public tunnel for webhooks
TunnelMaxRetries int // max consecutive reconnect failures before fallback to polling (0 = default 5)
// ExternalURL is the public base URL of an externally-managed tunnel
// (tunnel = "external"). When set alongside webhook mode and NO managed
// Tunnel, the scheduler registers each webhook-capable provider's hook to
// <ExternalURL>/webhook/<provider> on startup, so the operator doesn't
// have to hand-add a hook per repo. External hooks are operator-owned and
// are NOT deregistered on shutdown. Empty means "receiver only, don't
// touch the platform's webhooks".
ExternalURL string
JobTimeout time.Duration
ShutdownTimeout time.Duration
LogRetention time.Duration // max age for job log files (default 7d)
// MacOSProvisionTimeout bounds the pre-registration provisioning phase of
// a macOS VM job — booting the VM and waiting for its runner to become
// reachable (handleMacOSJob → MacOSVM.WaitForRunner). If the wait has not
// returned by this deadline the VM is force-stopped so the reachability
// wait unblocks, the job is failed, and the single macOS concurrency slot
// is released. Guards against a hung guest SSH command wedging the wait
// indefinitely (the VM-internal loop caps itself at ~2 min, but its SSH
// session calls carry no deadline and ignore ctx). Zero applies
// defaultMacOSProvisionTimeout.
MacOSProvisionTimeout time.Duration
// Retry configures the claim/provision retry queue. When the initial
// attempt to claim a queued job fails with a retryable error
// (rate-limit exhausted, transient 5xx, network), the job is
// enqueued and re-attempted on a backoff ladder rather than lost.
// GitHub does not re-deliver workflow_job webhooks. Leave zero-valued
// (Enabled=false) to keep the pre-existing "log and drop" behavior.
Retry RetryConfig
// OrphanSweep configures teardown of dispatched runners that were
// never observed picking up a job. GitHub schedules JIT runners onto
// ANY queued job with matching labels, so the runner dispatched "for"
// a job may end up running a different one — leaving the runner that
// was dispatched for THAT job idle with no job-completion event ever
// pointing at it. The sweep destroys such runners once they have been
// idle-unbound for Grace. Only active in webhook mode and only for
// runners dispatched via providers that report runner assignments
// (providers.RunnerNameReporter) — otherwise "never observed bound"
// would just mean "we had no way to observe it".
OrphanSweep OrphanSweepConfig
// RunnerImageForRepo resolves the per-repo, per-OS image override
// configured under [runner.images]. Returns "" when no override is
// set; the scheduler then falls back to the provider per-OS default
// and finally the runtime's host-aware default. Nil-safe.
RunnerImageForRepo func(repo, os string) string
Log *slog.Logger
}
Config for the scheduler.
type DispatchClient ¶
type DispatchClient struct {
// contains filtered or unexported fields
}
DispatchClient dispatches Linux jobs to the WSL worker via gRPC.
func NewDispatchClient ¶
func NewDispatchClient(addr, token string) (*DispatchClient, error)
NewDispatchClient connects to the dispatch gRPC server at the given address. When token is non-empty it is attached to every RPC via per-RPC credentials so the server-side interceptor accepts the call; an empty token sends no credential (used by tests against an unauthenticated fake server).
The transport stays insecure (no TLS): the dispatch link is a host<->VM loopback/NAT hop, and the bearer token — not the transport — authenticates the caller.
func (*DispatchClient) Close ¶
func (d *DispatchClient) Close() error
Close closes the gRPC connection.
func (*DispatchClient) ConsumeContainerStats ¶
func (d *DispatchClient) ConsumeContainerStats(ctx context.Context, intervalSeconds uint32, runtimeLabel string, log *slog.Logger) error
ConsumeContainerStats opens the StreamContainerStats stream, asks the in-VM dispatch server for samples at the given interval, and feeds each batch into the host metrics registry under the supplied runtime label (typically metrics.RuntimeLinuxVM). The call returns when ctx is cancelled. On non-fatal stream errors the consumer reconnects with backoff so a transient network blip in the VM doesn't lose metrics for the rest of the daemon's lifetime.
Series are deleted via metrics.DeleteContainerSeries when the consumer sees a container id stop appearing in batches (the in-VM UnregisterSampler removes it from the stream).
func (*DispatchClient) Create ¶
func (d *DispatchClient) Create(ctx context.Context, id, image, jitConfig, provider, repo string) error
Create dispatches a container create to the WSL worker. provider + repo are passed through so the VM-side dind server can scope its per-repo image cache namespace to (provider, repo) and not leak private images across forges or repos.
type DispatchServer ¶
type DispatchServer struct {
apiv1.UnimplementedDispatchServer
// contains filtered or unexported fields
}
DispatchServer implements the Dispatch gRPC service. It runs inside the WSL containerd-only worker and proxies Create/Wait/Destroy calls to the local Linux Runtime. It also serves StreamContainerStats so the host can scrape per-container resource metrics on its own /metrics endpoint without exposing a second listener inside the VM. See docs/arch/container-metrics.md.
func NewDispatchServer ¶
func NewDispatchServer(rt *runtime.Runtime, log *slog.Logger, defaultInterval time.Duration, linuxCPULimit, linuxMemLimitBytes uint64) *DispatchServer
NewDispatchServer constructs a Dispatch service handler. defaultInterval is used when the StreamContainerStats client passes interval_seconds=0; linuxCPULimit / linuxMemLimitBytes are baked into each per-container sampler as the configured cap.
func StartDispatchServer ¶
func StartDispatchServer(cfg DispatchServerConfig) (*DispatchServer, func())
StartDispatchServer starts the dispatch gRPC server on the given TCP port and returns the running server instance plus a cleanup function that gracefully stops it. The returned *DispatchServer exposes RegisterSampler / UnregisterSampler so the local runtime can plumb its OnTaskStarted / OnTaskDestroy hooks into the stats stream surface area.
Binds to cfg.BindAddr (default 0.0.0.0) so the host (outside the VM) can reach it. WSL on Windows shares localhost with the host, so this used to be 127.0.0.1, but the same process is now invoked from inside an Apple Vz VM where the host lives on the NAT side and needs the listener exposed on the VM's external interface.
The gRPC surface (CreateJob/WaitJob/DestroyJob/StreamContainerStats) exposes container lifecycle control with a caller-supplied image + JIT config, so it MUST NOT be reachable unauthenticated by anything sharing the VM's network (notably job containers). When cfg.Token is set, every RPC is gated by a constant-time bearer-token check via unary + stream interceptors. Operators must additionally firewall the dispatch port off from job containers (the worker installs bridge control-port rules for exactly this — see main.go controlPorts).
func (*DispatchServer) CreateJob ¶
func (s *DispatchServer) CreateJob(ctx context.Context, req *apiv1.CreateJobRequest) (*apiv1.CreateJobResponse, error)
func (*DispatchServer) DestroyJob ¶
func (s *DispatchServer) DestroyJob(ctx context.Context, req *apiv1.DestroyJobRequest) (*apiv1.DestroyJobResponse, error)
func (*DispatchServer) RegisterSampler ¶
func (s *DispatchServer) RegisterSampler(id, repo string, sampler metrics.Sampler)
RegisterSampler is called by the in-VM runtime's OnTaskStarted hook to expose a container's sampler to StreamContainerStats subscribers.
func (*DispatchServer) StreamContainerStats ¶
func (s *DispatchServer) StreamContainerStats(req *apiv1.StreamContainerStatsRequest, stream grpc.ServerStreamingServer[apiv1.ContainerStatsBatch]) error
StreamContainerStats serves the long-lived sampling stream that the host uses to surface per-container resource series. The handler ticks at the client-requested cadence and sends one batch per tick covering every registered sampler. Returns when the client cancels the context, when the underlying connection drops, or when Send fails for any reason.
func (*DispatchServer) UnregisterSampler ¶
func (s *DispatchServer) UnregisterSampler(id string)
UnregisterSampler removes a container's sampler from the stream set, called by the runtime's OnTaskDestroy hook.
func (*DispatchServer) WaitJob ¶
func (s *DispatchServer) WaitJob(ctx context.Context, req *apiv1.WaitJobRequest) (*apiv1.WaitJobResponse, error)
type DispatchServerConfig ¶
type DispatchServerConfig struct {
Port int
Runtime *runtime.Runtime
Log *slog.Logger
StatsInterval time.Duration // default 10s when zero
LinuxCPULimit uint64 // 0 = unlimited
LinuxMemLimitBytes uint64 // 0 = unlimited
// Token is the shared bearer token the server requires on every RPC. When
// non-empty, unary + stream interceptors reject any call that does not
// present a matching token (constant-time compare). Empty disables auth —
// which should only happen in tests or misconfigured setups; production
// callers plumb a token from the shared data dir (see
// LoadOrCreateDispatchToken). The server logs loudly when it starts
// unauthenticated so the footgun is visible.
Token string
// BindAddr is the interface the listener binds to. Empty defaults to
// "0.0.0.0" so the Vz/Hyper-V host on the NAT side can reach it (a narrower
// bind is only safe when the host address is known; see the comment on
// StartDispatchServer). The token check protects the surface regardless of
// bind, so 0.0.0.0 is defense-in-depth-behind-auth rather than the sole
// control.
BindAddr string
}
DispatchServerConfig configures the in-VM dispatch gRPC server.
type OrphanSweepConfig ¶
type OrphanSweepConfig struct {
// Enabled toggles the sweep.
Enabled bool
// Grace is how long a dispatched runner may remain unbound (never
// seen in an in_progress event) before it is destroyed. Defaults to
// 10 minutes when zero.
Grace time.Duration
}
OrphanSweepConfig tunes the orphaned-runner sweep. Zero-valued = disabled (matching pre-existing behavior); the CLI enables it by default with a 10-minute grace window.
type RetryConfig ¶
type RetryConfig struct {
// Enabled toggles the entire retry queue. When false, failures still
// log and drop as before (no behavior change).
Enabled bool
// Schedule is the ordered backoff ladder. Each entry is the base
// delay for that attempt; jitter is applied on top. If nil, defaults
// to {30s, 1m, 2m, 5m, 10m}.
Schedule []time.Duration
// MaxAge is the wall-clock budget from first failure to giving up.
// Once (now - firstFailure) > MaxAge, we log a WARN and drop. Default 90m.
MaxAge time.Duration
// Jitter is the fraction (0-1) of a delay that's randomized +/- around
// the base value. Set to a NEGATIVE value (e.g. -1) to request the
// default (0.2 = +/-20%). Literal 0 is honored - tests use it for
// deterministic scheduling.
Jitter float64
// RateHint returns the last-observed GitHub rate-limit state. When
// remaining == 0 and now < reset and updated is fresh (<5m old),
// the next attempt is snapped to reset + a small jitter instead of
// the Schedule entry. Nil-safe: nil means "no rate awareness".
RateHint func() (remaining int64, reset time.Time, updated time.Time)
// Now is the clock function. Defaults to time.Now. Tests inject
// a fake clock so backoff scheduling is deterministic.
Now func() time.Time
}
RetryConfig tunes the claim retry queue.
A zero-valued RetryConfig disables retries entirely, matching the pre-existing "log-and-drop" behavior. In practice New() applies sensible defaults so callers get retries by default.
type Scheduler ¶
type Scheduler struct {
// contains filtered or unexported fields
}
Scheduler ties CI provider job events to container lifecycle. When a job is queued, it provisions a runner environment. When the job completes, it destroys the environment.
func (*Scheduler) ActiveJobs ¶ added in v0.1.7
ActiveJobs returns the number of jobs currently running. Used by the Upgrade RPC (via the upgrade.Drainer interface) to wait for the scheduler to drain to idle after a Cordon, the same signal `ephemerd drain --wait` polls over the control socket.
func (*Scheduler) Cordon ¶ added in v0.1.5
Cordon marks the scheduler as draining WITHOUT initiating shutdown: new queued jobs are rejected while running jobs continue undisturbed. Returns the number of jobs still running. Used by the Cordon RPC so `ephemerd drain --wait` can stop claims first and only restart the daemon once the active job count reaches zero.
The flag is enforced at three points, all of which read it live rather than snapshotting it at admission (issue #154):
- handleQueued — refuses new queued events, whatever their source (webhook, poll, startup catch-up, reconcile sweep, retry-queue fire).
- admitDispatch — abandons a dispatch that was accepted before the cordon and then sat blocked on a concurrency semaphore. This is the one that was missing: the node kept provisioning for over a minute after the operator was told it had stopped claiming.
- claimJob — the hard backstop. Nothing can register a JIT runner on a cordoned node, even from a dispatch path that skips the gates above.
Cordon means "stop claiming NEW work". Jobs already running are untouched: their contexts hang off jobsCtx and are never cancelled here.
func (*Scheduler) Run ¶
Run starts the scheduler. It discovers jobs via polling (default) or webhooks (when TLS certs are configured), and manages runner lifecycle.
func (*Scheduler) SetMacOSVMConfig ¶
func (s *Scheduler) SetMacOSVMConfig(cfg *vm.MacOSVMConfig)
SetMacOSVMConfig enables macOS job support after startup. This is used when the macOS disk image is being provisioned in the background — the scheduler starts immediately for Linux jobs and picks up macOS jobs once the install finishes.
func (*Scheduler) StartVMSSHServer ¶
StartVMSSHServer starts a small HTTP server on the unix control socket for the VM SSH info endpoint. Called after the gRPC server is set up.
func (*Scheduler) Uncordon ¶ added in v0.1.5
Uncordon reverses Cordon: the scheduler resumes claiming queued jobs. Returns the number of jobs currently running. Jobs whose queued events were rejected while cordoned are picked up again by the next poll or reconcile sweep once their seen entry expires (seenTTL).