cohort

type Actuator ¶

type Actuator interface {
	// Launch creates a new entity. The Intent carries the deterministic
	// idempotency token; re-issuing Launch after an Ambiguous fault is safe.
	Launch(ctx context.Context, intent EntityIntent) (Observation, error)
	// Start resumes a Stopped or Hibernated entity. May itself fault to
	// CapacityExhausted — a warm entity is not reserved capacity.
	Start(ctx context.Context, id EntityID) (Observation, error)
	// Stop transitions a Running entity to Stopped or Hibernated per mode.
	Stop(ctx context.Context, id EntityID, mode StopMode) error
	// Terminate destroys an entity. Idempotent.
	Terminate(ctx context.Context, id EntityID) error
}

Actuator drives a single entity toward a desired lifecycle state. It NEVER operates on counts; every call names exactly one entity. This is the non-negotiable consequence of "the named entity is the unit" — a count abstraction structurally cannot express partial-failure correctness.

type Assembler ¶

type Assembler interface {
	Assemble(ctx context.Context, members []Observation) error
}

Assembler is the cohort-scoped action that runs ONCE, after the collective barrier, over a complete and simultaneously-live cohort.

Slurm/MPI domain: PMIx wire-up — publish the hostlist, exchange addresses. Globus domain: collection mesh join.

The core invokes Assemble and learns only pass/fail. It never sees the address list, the topology, or the peer graph. Mechanism is the domain's; the phase-slot is the core's. This is the boundary that keeps cohort a reconciler and not a workflow orchestrator.

type Attempt ¶

type Attempt struct {
	Rung  PlacementRung
	Phase Phase  // phase reached on this rung before the fault (or PhaseReady)
	Fault *Fault // nil if this attempt succeeded
	At    time.Time
}

Attempt records one rung tried for one entity. Rung is the core-visible, provider-agnostic view (PlacementRung) so Explain renders a legible line for any provider — AWS, agent-transport, or otherwise (#1).

type BackoffPolicy ¶

type BackoffPolicy struct {
	Base   time.Duration // duration for attempt 0. Default: 100ms.
	Cap    time.Duration // maximum pre-jitter duration. Default: 30s.
	Jitter float64       // fraction of computed value added as uniform noise. Default: 0.25.
}

BackoffPolicy computes exponential-with-jitter durations for bounded retries. It is provider-agnostic: the reconciler uses it for RetryableConsistency retries; substrate constructs a separate instance (longer cap) for the throttle path that feeds Limiter.Backoff.

This file imports nothing provider- or scheduler-specific.

func DefaultBackoffPolicy() BackoffPolicy

func (p BackoffPolicy) Duration(attempt int) time.Duration

type CapacityModel ¶

type CapacityModel int

const (
	CapacityOnDemand CapacityModel = iota
	CapacitySpot
	CapacityReserved // ODCR / capacity block — should not ICE
)

type Classifier ¶

type Classifier interface {
	Classify(err error) Fault
}

Classifier maps a provider error into exactly one Fault class.

This is the single most provider-specific artifact in queuezero and is NOT portable across clouds. Each provider supplies its own. See docs/ARCHITECTURE.md §5 and §13.

Implementations used with Reconciler MUST NOT return FaultAmbiguous. FaultAmbiguous means "mutation status unknown" and must be resolved by the provider layer (via idempotency-token re-issue) before classification reaches the reconciler. Returning FaultAmbiguous here is a bug; Reconciler flags it loudly as a Terminal fault with code "AmbiguousReachedReconciler".

type Cohort ¶

type Cohort struct {
	ID      CohortID
	Members []EntityIntent

	// Budget bounds each phase with its own deadline, so a failure names the
	// phase it died in. Any individually-zero field fires that phase's deadline
	// at t=0 — always use DefaultBudget() or an explicit budget. The constructors
	// fill every zero field from DefaultBudget() automatically (field-by-field).
	Budget PhaseBudget

	// MinViable is the minimum number of enrolled entities required to satisfy
	// the cohort barrier.
	//
	// ZERO-VALUE CONTRACT: MinViable==0 means "full membership required" —
	// len(Members) is used. It does NOT mean "no quorum required." This is the
	// correct default for MPI (all-or-nothing). For partial success, use
	// NewPartialCohort with an explicit MinViable.
	MinViable int

	// NoAssembly is set by NewPartialCohort. When true, Reconcile refuses to
	// invoke the Assembler even if one is configured on the Reconciler.
	// Partial cohorts have undefined assembly semantics: if 8 of 10 satisfy
	// MinViable, does assembly run over 8 or wait for 10? Rather than guess,
	// partial cohorts explicitly prohibit the assembly phase.
	NoAssembly bool
}

Cohort is a named set of identity-bearing entities that succeed, fail, and fast-fail together. A serial HPC job is the 1-cohort: cardinality one, a trivially-satisfied barrier, and a no-op Assembler. It is the SAME logic, not a special case — which is the evidence the model is the right shape.

Construct with NewSerialCohort, NewMPICohort, or NewPartialCohort rather than a struct literal to avoid zero-value traps (zero Budget fires all deadlines instantly; MinViable==0 means full membership, not no-quorum).

func NewMPICohort(id CohortID, members []EntityIntent, budget PhaseBudget) (Cohort, error)

func NewPartialCohort(id CohortID, members []EntityIntent, budget PhaseBudget, minViable int, asm Assembler) (Cohort, error)

func NewSerialCohort(id CohortID, member EntityIntent, budget PhaseBudget) (Cohort, error)

func (c Cohort) IsCollective() bool

type CohortCancelInfo ¶

type CohortCancelInfo struct {
	// CulpritID is the entity whose fault made the gate unsatisfiable.
	CulpritID EntityID
	// CulpritFault is the verbatim fault (class + provider code) the culprit hit.
	CulpritFault Fault
	// CulpritPhase is the phase the culprit was in when it failed.
	CulpritPhase Phase
	// At is when the fast-fail was triggered.
	At time.Time
	// SurvivorPhase is the phase THIS entity had reached when it was cancelled.
	// This is set per-survivor — not the cohort's phase, each entity's own phase
	// at the instant IT observed the cancellation.
	SurvivorPhase Phase
}

CohortCancelInfo describes why a healthy entity was cancelled by a cohort fast-fail.

type CohortID ¶

type CohortID string

CohortID names a set of entities reconciled as a unit.

type Enroller ¶

type Enroller interface {
	IsEnrolled(ctx context.Context, id EntityID) (Readiness, error)
}

Enroller is the domain probe for the late per-entity phase: "the entity has been accepted by whatever external authority the domain cares about." Slurm domain: slurmd has checked in. Globus domain: endpoint registered with the collection. The core knows the phase exists; not what it means.

type EntityID ¶

type EntityID string

EntityID is the stable, identity-preserving name of a single managed entity. In the Slurm domain this is the node name (e.g. "gpu-042"). It is NEVER a count, an index into an anonymous pool, or anything ASG-shaped.

type EntityIntent ¶

type EntityIntent struct {
	ID         EntityID
	Generation Generation
	Cohort     CohortID

	// Placement is the provider-specific placement payload, opaque to the core
	// (#1). It is the seam parallel to Classifier: the reconciler drives the
	// fallback ladder through it — Advance on a fallback-eligible fault — but
	// never inspects its fields. The AWS provider supplies a RungPlacement
	// (instance type / AZ / capacity model); an agent-transport provider
	// supplies its own (goroutine → session → instance) and never sees a
	// CapacityModel. The core learns only what Current() renders.
	Placement Placement

	// IdempotencyToken is deterministic in (cluster, entity, generation).
	// It collapses FaultAmbiguous at the substrate layer — re-issuing a mutation
	// with the same token returns the existing resource rather than creating a
	// duplicate. MUST NOT be empty or random; use NewEntityIntent which generates
	// it via cohort.Token when not supplied by the caller.
	IdempotencyToken string
}

EntityIntent is the desired specification for one entity. The reconciler's intent for a cohort is a set of these — a set of NAMED slots, never "N".

Construct with NewEntityIntent to validate all fields and auto-generate an idempotency token if one is not supplied.

func NewEntityIntent(cluster string, id EntityID, gen Generation, cohortID CohortID, placement Placement, token string) (EntityIntent, error)

type Fault ¶

type Fault struct {
	Class     FaultClass
	Code      string // verbatim provider error code, never paraphrased
	Retryable bool   // convenience: Class is RetryableConsistency or Throttle
	Message   string
}

Fault is the classified form of a provider error: the class plus the verbatim provider code, preserved for legibility (q0 explain).

type FaultClass ¶

type FaultClass int

const (
	// FaultRetryableConsistency: propagation lag, not failure. Bounded retry,
	// short backoff, TIGHT ceiling (single-digit seconds).
	FaultRetryableConsistency FaultClass = iota
	// FaultThrottle: rate-limited. Exponential backoff + jitter. The fix is
	// slowing the whole client, not waiting — hence distinct from consistency.
	FaultThrottle
	// FaultCapacityExhausted: ICE / no capacity. NEVER retry in place — advance
	// the fallback chain. Purchase-model-independent: on-demand ICEs too.
	FaultCapacityExhausted
	// FaultTerminal: auth, quota, bad parameter. Fail immediately and loud.
	FaultTerminal
	// FaultAmbiguous: timeout / reset / 5xx — mutation status unknown. This
	// class MUST be collapsed into FaultRetryableConsistency by idempotency
	// tokens before it reaches the reconciler. It should never be observed
	// downstream of substrate.Client.
	FaultAmbiguous
)

func (c FaultClass) String() string

type Generation ¶

type Generation string

Generation tags every entity with the spec revision that created it. Instances from a superseded partitions.yaml apply are unambiguously reapable; current-generation instances are protected from the suspend sweeper. See docs/ARCHITECTURE.md §11.

type LifecycleState ¶

type LifecycleState int

const (
	StateUnknown    LifecycleState = iota // Observer could not determine — treat as lag, not absence
	StateAbsent                           // no entity exists for this ID
	StateLaunching                        // Launch/Start acknowledged, not yet Running
	StateRunning                          // provider reports running
	StateStopped                          // warm: EBS persists, instance-store gone
	StateHibernated                       // RAM frozen to EBS: mounts/processes/page-cache survive
	StateDraining                         // marked for teardown
	StateFailed                           // terminal for this generation
)

func (s LifecycleState) String() string

type Observation ¶

type Observation struct {
	ID         EntityID
	Generation Generation
	State      LifecycleState
	ProviderID string // e.g. EC2 instance ID, once known
	Rung       Rung
	Address    string // private address, once Running — domain may need it
	ObservedAt time.Time
}

Observation is one entity's infrastructure-truth state as seen by an Observer. It is advisory: a StateUnknown is lag, and the idempotency token is consulted for ground truth.

type Observer ¶

type Observer interface {
	Observe(ctx context.Context, ids []EntityID) ([]Observation, error)
}

Observer reports the actual, infrastructure-truth state of named entities.

Implementations MUST tolerate eventual consistency: a Describe-miss on a freshly created entity is reported as StateUnknown, never StateAbsent. The reconciler decides what a miss means using the idempotency token as ground truth (see substrate.Client) — the Observer only reports what it sees.

type Outcome ¶

type Outcome struct {
	Cohort  CohortID
	Ready   bool
	Records map[EntityID]Record
}

Outcome is the result of reconciling one cohort. Whether success or failure, every member carries a Record (see explain.go) — "it didn't work" is never an acceptable answer; "ICE on p5.48xlarge in us-east-1a, chain exhausted at 14:32:07" is.

type ParentCancelInfo ¶

type ParentCancelInfo struct {
	// SurvivorPhase is the phase this entity had reached when cancelled.
	SurvivorPhase Phase
	// Cause is the parent context error string ("context canceled" or
	// "context deadline exceeded"). Preserved verbatim for legibility.
	Cause string
}

ParentCancelInfo describes a reconcile aborted by the parent context. There is no culprit entity.

type Phase ¶

type Phase int

const (
	PhaseLaunchAcked    Phase = iota // 1: Launch/Start returned. Blowing it => throttle/API, NOT capacity.
	PhaseRunning                     // 2: provider reports running. Capacity faults surface here or in 1.
	PhaseEnrolled                    // 3: entity accepted by its authority; readiness probe (incl. mount) passes.
	PhaseCohortBarrier               // 4: ALL members reached PhaseEnrolled, or gate unsatisfiable => fast-fail set.
	PhaseCohortAssembly              // 5: domain Assembler succeeded over the complete cohort.
	PhaseReady                       // terminal-success: cohort is usable.
)

func (p Phase) String() string

type PhaseBudget ¶

type PhaseBudget struct {
	LaunchAcked    time.Duration // ~10-15s
	Running        time.Duration // ~minutes; capacity faults surface here
	Enrolled       time.Duration // bootstrap + authority check-in + operational probe
	CohortBarrier  time.Duration // how long to wait for stragglers before fast-failing the set
	CohortAssembly time.Duration // domain wire-up
}

PhaseBudget is the deadline for each phase of reconciliation. Phase 1 is deliberately tight — blowing it means throttling or an API problem, never a capacity problem, and that distinction is load-bearing for legibility.

func DefaultBudget() PhaseBudget

type Placement ¶ added in v0.2.0

type Placement interface {
	// Current returns the legible identity of the rung currently selected —
	// what Record/Attempt/Explain render. It must have a non-empty Name.
	Current() PlacementRung
	// Advance returns the next placement in the approved fallback chain and true,
	// or false when the chain is exhausted. It must NEVER substitute a rung
	// outside the approved chain. The returned Placement is a new value; the
	// receiver is not mutated.
	Advance() (Placement, bool)
}

Placement is the provider-specific placement payload and fallback ladder, opaque to the core (#1). It is the seam that keeps cohort provider-agnostic: the reconciler advances the ladder on a fallback-eligible fault but never inspects what a rung means. AWS supplies RungPlacement; an agent-transport provider supplies its own (goroutine → session → instance).

type PlacementRung ¶ added in v0.2.0

type PlacementRung struct {
	// Name is the legible identifier rendered in Record/Explain, e.g.
	// "p5.48xlarge/us-east-1a/spot" (AWS) or "a2a-session" / "goroutine" (Telos).
	Name string
	// Class is a provider-defined category, e.g. AWS capacity model ("spot") or
	// a transport rung kind. Free-form; the core only displays it.
	Class string
	// WarmStart means "resume a Stopped/Hibernated entity" rather than cold-launch
	// — the one placement property the core acts on (it picks Actuator.Start vs
	// Launch). A provider with no warm state simply always returns false.
	WarmStart bool
}

PlacementRung is the core-visible, provider-agnostic view of one rung: a human-legible name, a provider-defined class, and whether selecting it means resuming a warm entity vs. a cold launch. This is all the core needs — the provider's real placement fields (instance type, AZ, capacity model, …) stay inside the provider's Placement implementation.

type RateLimiter ¶

type RateLimiter interface {
	Acquire(ctx context.Context) error
	Backoff(d time.Duration)
}

RateLimiter is the account-shared client-side throttle. On a FaultThrottle the whole limiter backs off, not just one caller.

type Readiness ¶

type Readiness struct {
	Enrolled    bool
	Operational bool   // domain-defined: fully functional, not merely running
	Detail      string // human-readable, surfaced by q0 explain
}

Readiness is the result of a domain Enroller probe.

Two fields beyond "enrolled?":

• Operational: the entity is fully functional, not merely running. What "operational" means is domain-defined: a Slurm Enroller checks mount health; an MPI Enroller checks EFA health; a Globus Enroller checks endpoint health. cohort does not define WHAT operational means — only that the domain reports it. A node can be running+idle with a dead mount; Operational=false catches that case regardless of domain.
• Detail: human-readable, propagated to q0 explain via Record.

func (r Readiness) OK() bool

type Reconciler ¶

type Reconciler struct {
	Actuator   Actuator
	Observer   Observer
	Classifier Classifier
	Enroller   Enroller
	Assembler  Assembler

	// Limiter gates outbound provider mutations. It is account-shared: all
	// cohorts reconciling against one account share one budget, because
	// throttling is a property of the account, not the call site.
	Limiter RateLimiter

	// Clock overrides time.Now for deterministic tests. Nil uses time.Now.
	// TEST-ONLY: do not set in production code.
	Clock func() time.Time
}

Reconciler converges one Cohort against eventually-consistent infrastructure.

It is the core loop: declare intent (a set of named entities), observe actual state tolerating consistency gaps, diff PER-ENTITY, correct, repeat until the cohort converges or a phase budget runs out. This is ASG done right and visible — and its unit is the named entity, never a count.

The Reconciler holds NO provider or domain knowledge. Everything it needs arrives through the ports interfaces.

Construct with NewReconciler rather than a struct literal; the field set may grow in v0.x without notice.

func NewReconciler(act Actuator, obs Observer, clf Classifier, enr Enroller, asm Assembler, lim RateLimiter) *Reconciler

func (r *Reconciler) Drain(ctx context.Context, ids []EntityID) error

func (r *Reconciler) Reconcile(ctx context.Context, c Cohort) (Outcome, error)

type Record ¶

type Record struct {
	Entity     EntityID
	Generation Generation
	Cohort     CohortID

	// ReachedPhase is the furthest phase the entity got to.
	ReachedPhase Phase

	// Attempts is every rung tried, in order.
	Attempts []Attempt

	// Terminal, if set, is the fault that ended reconciliation for THIS entity.
	// Nil on success, nil on CohortCancelled, nil on ParentCancelled.
	Terminal *Fault

	// CohortCancelled is set when this entity was healthy and in-flight but
	// the cohort fast-failed around it because ANOTHER entity became unsalvageable.
	// The entity did not fail; the cohort died around it.
	// Mutually exclusive with Terminal and ParentCancelled.
	//
	// q0 explain on a 64-node cohort must distinguish the ONE entity that
	// caused the fast-fail from the 63 cancelled because of it.
	CohortCancelled *CohortCancelInfo

	// ParentCancelled is set when the PARENT context was cancelled (operator
	// shutdown, q0 process exiting, an outer deadline on the whole reconcile).
	// There is no culprit entity — the reconcile was aborted externally.
	// Mutually exclusive with Terminal and CohortCancelled.
	//
	// This is distinct from CohortCancelled: no cohort-level cause was recorded.
	// Reading it as CohortCancelled-with-empty-culprit would be a fabricated story.
	ParentCancelled *ParentCancelInfo

	// EnrollDetail is the human-readable Readiness.Detail from a successful
	// enrollment (e.g. "efa ok"), surfaced in Explain(). It is a DISPLAY string
	// — distinct from Observation.Address, which carries the entity's actual
	// address for the Assembler.
	EnrollDetail string

	StartedAt  time.Time
	FinishedAt time.Time
}

Record is the structured legibility artifact every reconciled entity carries. Legibility is a deep requirement, not polish: queuezero must be crystal clear about WHY when something fails, or fall back in a legible and approved manner. See docs/ARCHITECTURE.md §10.

Exactly one of {Succeeded(), WasCohortCancelled(), WasParentCancelled(), Terminal != nil} is true for every completed Record. Never more than one.

func (r Record) Explain() string

func (r Record) Succeeded() bool

func (r Record) Summary() string

func (r Record) WasCohortCancelled() bool

func (r Record) WasParentCancelled() bool

type Rung ¶

type Rung struct {
	InstanceType  string
	AvailZone     string
	CapacityModel CapacityModel
	AccountID     string // execution account for this rung (multi-account, §3)

	// WarmStart means resume a Stopped/Hibernated entity rather than cold-launch.
	// It is a RUNG property, not a pre-check: if warm-start ICEs, the chain
	// advances to the next rung exactly like any other ICE.
	WarmStart bool
}

Rung is one option in an AWS capacity fallback chain. There is no "safe baseline" — on-demand and spot are both rungs that can fault to capacity; they differ only in ICE probability and price. ODCR/capacity-block rungs are the one kind genuinely reserved against ICE.

Rung is the AWS provider's placement vocabulary, not core vocabulary — it is carried into the core via RungPlacement, which adapts it to the Placement seam. A non-AWS provider never constructs a Rung.

type RungPlacement ¶ added in v0.2.0

type RungPlacement struct {
	Rung  Rung
	Chain []Rung
}

RungPlacement is the built-in Placement backed by an AWS Rung and its approved fallback chain — the adapter that lets the AWS/MPI consumers migrate to the Placement seam near-mechanically (`Placement: cohort.RungPlacement{Rung: r, Chain: chain}`). The chain is the ordered list of approved rungs; Advance walks it and never substitutes outside it. An empty chain means single-rung (no fallback).

func (p RungPlacement) Advance() (Placement, bool)

func (p RungPlacement) Current() PlacementRung

func (p RungPlacement) Validate() error

type StopMode ¶

type StopMode int

StopMode selects warm-stop vs hibernate when suspending an entity.

const (
	StopWarm      StopMode = iota // EBS persists
	StopHibernate                 // RAM frozen to EBS
)