nodeagent

package
v0.1.18-rc14 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 9, 2026 License: MIT Imports: 48 Imported by: 0

Documentation

Overview

claimer_loop.go contains the claim loop orchestration and backoff logic.

This file owns the Start method that continuously polls for work (jobs) with exponential backoff. Backoff increases when no work is available or on errors, and resets when work is successfully claimed. Nodes claim from a single unified jobs queue (FIFO by next_id); there is no separate Build Gate queue or claim path. Isolating loop mechanics from claim/execution details simplifies backoff testing.

container_job.go contains mig container execution and workspace lifecycle helpers.

Package nodeagent contains the ployd node execution agent.

Responsibilities:

  • Accept run requests from the control plane and orchestrate execution.
  • Hydrate workspaces from Git, run Build Gate validation, execute mig containers, and collect/upload artifacts, diffs, and terminal status.
  • Execute discrete job types from the unified queue (gate, mig).

Key files:

  • execution.go — high level run lifecycle and runtime factories.
  • gate_job.go — gate job execution and failure context persistence.
  • container_job.go — mig job execution and shared container lifecycle.
  • job_reporting.go — centralized diff/status upload helpers.
  • workspace.go — workspace/file utilities.
  • manifest.go — request→manifest translation helpers.
  • job.go — job status types, image name persistence.
  • http.go — base HTTP client, URL builders, compression helpers.

execution.go contains the high-level run lifecycle orchestration.

This file owns executeRun, the main entry point for executing a single run. It coordinates runtime initialization and dispatches to specialized job handlers based on job type. Job implementations live in:

  • container_job.go — mig jobs + standard executor
  • gate_job.go — gate validation jobs

job_reporting.go contains upload, status reporting, diff generation, and artifact helpers used by job executors.

Index

Constants

View Source
const (
	MaxUploadSize = 10 << 20 // 10 MiB

	// SoftUploadSize is the threshold at which log chunks are flushed. The 64-byte
	// margin accounts for gzip footer overhead when finalizing a chunk, ensuring
	// the closed stream stays under MaxUploadSize.
	SoftUploadSize = MaxUploadSize - 64
)
View Source
const MaxConcurrency = 64

Variables

View Source
var ErrPayloadTooLarge = errors.New("payload exceeds size cap")

ErrPayloadTooLarge is returned when compressed data exceeds MaxUploadSize.

Functions

func BuildURL

func BuildURL(base, p string) (string, error)

BuildURL resolves a base URL and a path-only reference, preserving scheme/host.

func MustBuildURL

func MustBuildURL(base, p string) string

MustBuildURL is like BuildURL but panics on error.

Types

type Agent

type Agent struct {
	// contains filtered or unexported fields
}

Agent coordinates the node agent's HTTP server, heartbeat manager, and claim loop.

func New

func New(cfg Config) (*Agent, error)

New constructs a new node agent.

func (*Agent) Run

func (a *Agent) Run(ctx context.Context) error

Run starts the node agent and blocks until the context is canceled.

type ArtifactBundleEntry

type ArtifactBundleEntry struct {
	SourcePath  string
	ArchivePath string
}

ArtifactBundleEntry defines one filesystem source and its archive path inside an uploaded artifact bundle.

type ClaimManager

type ClaimManager struct {
	// contains filtered or unexported fields
}

ClaimManager periodically polls the server for work and executes claimed jobs. Nodes claim from a single unified jobs queue (FIFO by next_id); there is no separate Build Gate queue or claim path. Contains configuration, HTTP client, run controller, and backoff state for polling intervals when no work is available.

func NewClaimManager

func NewClaimManager(cfg Config, controller RunController) (*ClaimManager, error)

NewClaimManager constructs a claim manager for the unified jobs queue. Nodes claim jobs from a single queue (FIFO by next_id); there is no separate Build Gate queue. Initializes backoff parameters for the claim loop polling interval.

func (*ClaimManager) Start

func (c *ClaimManager) Start(ctx context.Context) error

Start begins the claim loop. Continuously polls for work (jobs) using a ticker with exponential backoff. Backoff increases when no work is available or on errors, resets when work is successfully claimed. Nodes claim from a single unified jobs queue (FIFO by next_id).

type Config

type Config struct {
	// HTTP configuration for the node agent API server.
	HTTP HTTPConfig `yaml:"http"`

	// Server URL for the control-plane server.
	ServerURL string `yaml:"server_url"`

	// NodeID identifies this node (NanoID-backed).
	NodeID domaintypes.NodeID `yaml:"node_id"`

	// Concurrency defines the maximum number of concurrent runs.
	Concurrency int `yaml:"concurrency"`

	// Heartbeat configuration.
	Heartbeat HeartbeatConfig `yaml:"heartbeat"`
}

Config holds node agent configuration. Uses domain types for type-safe node identification.

func LoadConfig

func LoadConfig(path string) (Config, error)

LoadConfig reads and parses the YAML configuration file.

type HTTPConfig

type HTTPConfig struct {
	// Listen address (e.g., ":8444").
	Listen string `yaml:"listen"`

	// TLS configuration.
	TLS TLSConfig `yaml:"tls"`

	// Timeouts.
	ReadTimeout  time.Duration `yaml:"read_timeout"`
	WriteTimeout time.Duration `yaml:"write_timeout"`
	IdleTimeout  time.Duration `yaml:"idle_timeout"`
}

HTTPConfig specifies HTTP listener and TLS settings for the node agent.

type HeartbeatConfig

type HeartbeatConfig struct {
	// Interval between heartbeats.
	Interval time.Duration `yaml:"interval"`

	// Timeout for heartbeat requests.
	Timeout time.Duration `yaml:"timeout"`
}

HeartbeatConfig specifies heartbeat interval and timeout.

type HeartbeatManager

type HeartbeatManager struct {
	// contains filtered or unexported fields
}

HeartbeatManager periodically sends resource snapshots to the server.

func NewHeartbeatManager

func NewHeartbeatManager(cfg Config) (*HeartbeatManager, error)

NewHeartbeatManager constructs a heartbeat manager.

func (*HeartbeatManager) Start

func (h *HeartbeatManager) Start(ctx context.Context) error

Start begins sending heartbeats.

type JobDirectories

type JobDirectories struct {
	Root             string
	Cache            string
	Home             string
	In               string
	Out              string
	Staging          string
	Tmp              string
	Stdout           string
	Stderr           string
	Diff             string
	ContainerInspect string
}

JobDirectories is the only authority for job-owned host storage.

type LogHook

type LogHook func(p []byte) ([]byte, error)

LogHook is a function that processes log data before it is compressed and sent to the server. A nil LogHook is treated as a no-op.

type LogStreamer

type LogStreamer struct {
	// contains filtered or unexported fields
}

LogStreamer buffers logs and streams them as gzipped chunks to the server.

func NewLogStreamer

func NewLogStreamer(cfg Config, runID types.RunID, jobID types.JobID, client *http.Client, resumeCount ...int) (*LogStreamer, error)

NewLogStreamer creates a new log streamer for a specific run and (optionally) job. When client is non-nil it is reused; otherwise a new HTTP client is created. Returns an error if HTTP client creation fails (e.g., missing bearer token).

func (*LogStreamer) Close

func (ls *LogStreamer) Close() error

Close flushes any remaining logs and stops the streamer. Returns all errors encountered during close using errors.Join.

func (*LogStreamer) SetHook

func (ls *LogStreamer) SetHook(hook LogHook)

SetHook sets the log processing hook. Must be called before any writes. This method is not safe for concurrent use with Write.

func (*LogStreamer) StderrWriter

func (ls *LogStreamer) StderrWriter() io.Writer

StderrWriter returns an io.Writer that tags all writes as stderr.

func (*LogStreamer) StdoutWriter

func (ls *LogStreamer) StdoutWriter() io.Writer

StdoutWriter returns an io.Writer that tags all writes as stdout.

func (*LogStreamer) Write

func (ls *LogStreamer) Write(p []byte) (n int, err error)

Write implements io.Writer interface for capturing logs.

type RunControlResponse

type RunControlResponse struct {
	RunID  types.RunID `json:"run_id"`
	Status string      `json:"status"`
}

RunControlResponse is returned after a run start or stop request.

type RunController

type RunController interface {
	StartRun(ctx context.Context, req StartRunRequest) error
	StopRun(ctx context.Context, req StopRunRequest) error

	// AcquireSlot blocks until a concurrency slot is available or the context
	// is canceled. Returns nil when a slot is acquired, or ctx.Err() if the
	// context was canceled while waiting.
	//
	// Slot ownership:
	//   - On StartRun success (nil error), the controller is responsible for
	//     releasing the slot when the job completes.
	//   - On StartRun failure (non-nil error), the caller must ReleaseSlot()
	//     before returning.
	AcquireSlot(ctx context.Context) error

	// ReleaseSlot frees a previously acquired concurrency slot.
	// Must be called exactly once for each successful AcquireSlot call.
	ReleaseSlot()
}

RunController manages run lifecycle on the node.

type Server

type Server struct {
	// contains filtered or unexported fields
}

Server exposes the node agent API over HTTPS with mTLS.

func NewServer

func NewServer(cfg Config, controller RunController) (*Server, error)

NewServer constructs a new node agent HTTP server.

func (*Server) Address

func (s *Server) Address() string

Address returns the bound listener address if the server is running.

func (*Server) Start

func (s *Server) Start(ctx context.Context) error

Start begins serving HTTPS requests.

func (*Server) Stop

func (s *Server) Stop(ctx context.Context) error

Stop terminates the HTTP server.

type StartRunRequest

type StartRunRequest struct {
	ResumeCount int             `json:"resume_count,omitempty"`
	RunID       types.RunID     `json:"run_id,omitempty"`
	JobID       types.JobID     `json:"job_id,omitempty"`   // Job ID for artifact/diff uploads
	RepoID      types.RepoID    `json:"repo_id,omitempty"`  // Repo ID for run artifacts (diffs/logs)
	RepoURL     types.RepoURL   `json:"repo_url,omitempty"` // Repository URL for this run
	BaseRef     types.GitRef    `json:"base_ref,omitempty"`
	CommitSHA   types.CommitSHA `json:"commit_sha,omitempty"`
	RepoSHAIn   types.CommitSHA `json:"repo_sha_in,omitempty"`
	JobType     types.JobType   `json:"job_type,omitempty"` // Job type: pre_gate, mig, post_gate
	NextID      *types.JobID    `json:"next_id,omitempty"`  // Linked successor in run chain
	JobName     string          `json:"job_name,omitempty"` // Deprecated: kept for wire compatibility during context rollout.
	// MigContext carries concrete mig step routing.
	MigContext *contracts.MigClaimContext `json:"mig_context,omitempty"`
	// DetectedStack carries the canonical gate-detected stack tuple for this job.
	DetectedStack *contracts.StackExpectation `json:"detected_stack,omitempty"`
	// MigSpec is the validated canonical workflow specification. It is populated
	// from the claimed spec and is not part of the node start wire contract.
	MigSpec   *contracts.MigSpec `json:"-"`
	Env       map[string]string  `json:"env"`
	ServerURL string             `json:"server_url,omitempty"`
}

StartRunRequest describes a run start request from the server.

JobType field:

  • Identifies the job type: "pre_gate", "mig", "post_gate".
  • Used by orchestrator to dispatch to appropriate execution handler.

type StopRunRequest

type StopRunRequest struct {
	RunID  types.RunID `json:"run_id"`
	Reason string      `json:"reason"`
}

StopRunRequest describes a run stop/cancel request.

type TLSConfig

type TLSConfig struct {
	// Enabled indicates whether mTLS is enabled.
	Enabled bool `yaml:"enabled"`

	// CertPath is the path to the node certificate.
	CertPath string `yaml:"cert_path"`

	// KeyPath is the path to the node private key.
	KeyPath string `yaml:"key_path"`

	// CAPath is the path to the CA certificate.
	CAPath string `yaml:"ca_path"`

	// BootstrapCAPath is the path to the CA certificate used to verify
	// the server during bootstrap (before mTLS certificates are obtained).
	// If empty, the CA at CAPath is used if it exists; otherwise
	// system roots are used (for public PKI scenarios).
	BootstrapCAPath string `yaml:"bootstrap_ca_path"`
}

TLSConfig specifies mTLS certificate and key paths.

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL