node

package
v0.6.4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 27, 2026 License: Apache-2.0 Imports: 17 Imported by: 0

Documentation

Overview

Package node is the control plane's authority over enrolled Mercator nodes.

A Node is the runtime Mercator controls on one Rental generation. It is what makes capacity reusable: without a Node there is no host runtime to hand a second workload to. This package owns node identity, leases and fencing, command delivery and deduplication, and the reconciliation a node performs after either side restarts. It implements capability.NodeRuntime, so the rest of the control plane talks to nodes through the same contract a simulated node satisfies in the Lab.

Nodes connect outbound. Nothing here dials a node, and no node exposes a listener or a Docker socket.

Index

Constants

View Source 
const AdapterType = "node"

AdapterType names node-backed capacity in a Booking Decision. It reads differently from a provider's name on purpose: this offer is a machine Mercator holds through its own runtime, not a product a provider sells.

View Source 
const ConnectionID = "connection:nodes"

ConnectionID is the pseudo-connection every node offer is aggregated under. Nodes have no provider credential: their authorization is the enrollment they already completed.

View Source 
const DefaultInvitation = 30 * time.Minute

DefaultInvitation is how long an enrollment token stays redeemable. It bounds the window in which a leaked bootstrap could join, and is long enough for a slow provider to finish booting a machine.

View Source 
const DefaultLease = 90 * time.Second

DefaultLease is how long the control plane believes a node absent a heartbeat. It is deliberately several heartbeat intervals, so one slow network round does not declare a healthy machine lost.

View Source 
const DefaultSession = 30 * time.Minute

DefaultSession is how long one session credential stays valid. A node renews by reconnecting, which is also when it reconciles.

Variables

View Source 
var ErrEnrollmentInvalid = errors.New("node: enrollment token is not valid")

ErrEnrollmentInvalid is returned when the material presented is not this node's invitation, or is one whose window has closed. It is a different answer from ErrEnrollmentSpent on purpose: the two are independent doors, and which one refused a machine is the difference between a bootstrap that arrived too late and a bootstrap somebody is replaying.

View Source 
var ErrEnrollmentSpent = errors.New("node: enrollment token already redeemed")

ErrEnrollmentSpent is returned when an invitation has already been redeemed. An agent that lost its session credential needs a fresh invitation rather than a replay of the old one.

View Source 
var ErrFenced = errors.New("node: fencing token superseded")

ErrFenced is returned when a command carries a superseded fencing token. It is the durable half of the fencing guarantee: even if a partitioned session reaches the control plane, its work is refused rather than applied late.

View Source 
var ErrIdentityExists = errors.New("node: identity already exists")

ErrIdentityExists is returned when an invitation names a node identity that is already reserved. Identity is immutable, so reusing one would let a second machine claim the first one's history.

View Source 
var ErrNotFound = errors.New("node: not found")

ErrNotFound is returned for a node identity the control plane never invited.

View Source 
var ErrRetired = errors.New("node: retired")

ErrRetired is returned to a node whose Rental generation is over at every door that would give it standing again: enrolling, opening a session, renewing a lease, or being asked to do anything further. Retirement is terminal: the machine the identity was minted for is gone, and the way back is a fresh generation with a fresh identity rather than this one coming round again.

It is never the answer to a machine reporting what it already did. An exit code and the result of a command the agent applied are the node's own authority, and a generation ends while a container is running every time a provider reclaims a machine, so a retirement that refused those would lose the only account of what the process did.

Functions

func DeriveKey 

func DeriveKey(masterKey []byte) []byte

DeriveKey produces a domain-separated subkey for node credentials from the master key, so node identity never shares signing material with run reporting or anything else. An empty master key leaves the signer disabled, and a disabled signer refuses to mint or verify rather than accepting everything.

func TokenID 

func TokenID(token string) string

TokenID is the stable identity of one minted token, safe to persist. It is a digest, so recording which invitation was redeemed never stores the credential itself.

Types

type Command 

type Command struct {
	OperationID  string      `json:"operation_id"`
	NodeID       string      `json:"node_id"`
	Kind         CommandKind `json:"kind"`
	FencingToken uint64      `json:"fencing_token"`
	IssuedAt     time.Time   `json:"issued_at"`
	// Payload is the typed command from the capability contract, encoded for
	// the wire.
	Payload []byte `json:"payload"`
}

Command is one instruction in flight to a node. OperationID makes it idempotent: however many times it is delivered, the node applies it once and reports Duplicate afterwards.

type CommandKind 

type CommandKind string

CommandKind names one instruction a node can be asked to perform. The wire carries the kind beside an opaque payload so the transport stays narrow enough for a second runtime implementation to reuse. 

const (
	CommandPrepareImage    CommandKind = "prepare_image"
	CommandPrepareArtifact CommandKind = "prepare_artifact"
	CommandLaunchWorkload  CommandKind = "launch_workload"
	CommandStopWorkload    CommandKind = "stop_workload"
)

func (CommandKind) MayLeaveEffectOnFailure 

func (kind CommandKind) MayLeaveEffectOnFailure() bool

MayLeaveEffectOnFailure reports whether a command that returned an error might still have changed the machine. Launching and stopping a workload might: the container may exist, or may already have been signalled, and doing either twice is worse than not retrying. Preparing content cannot: a failed pull or fetch leaves nothing, and it is idempotent, so retrying is both safe and necessary.

func (CommandKind) Valid 

func (kind CommandKind) Valid() bool

type DiskReport 

type DiskReport string

DiskReport is what Mercator can say about one node's disk, which is three answers and not two. Each sends an operator somewhere different, and a boolean can only carry two, so the third gets told as one of the others. 

const (
	// DiskNeverReported is an identity nobody has heard from: invited and not
	// yet enrolled, or enrolled by an agent whose first heartbeat has not
	// landed. Nothing has been measured because nothing has been asked, and
	// saying "this node could not measure its disk" about a machine that has
	// never spoken states a fact about a daemon Mercator has never seen.
	DiskNeverReported DiskReport = "never_reported"
	// DiskUnmeasurable is a node that reported and could not measure: its
	// daemon keeps content somewhere its agent cannot see. It stays in the
	// fleet, it reports its containers and their exits, and it wins no
	// placement that declares a disk floor.
	DiskUnmeasurable DiskReport = "unmeasurable"
	// DiskMeasured is a number this node established. Zero free bytes is a full
	// machine, which is a thing to go and clear out rather than a thing nobody
	// could read.
	DiskMeasured DiskReport = "measured"
)

type Enrollment 

type Enrollment struct {
	EnrollmentTokenID string
	AgentVersion      string
	Facts             capability.NodeFacts
	EnrolledAt        time.Time
	LeaseExpires      time.Time
}

Enrollment is the durable outcome of redeeming one invitation.

type Event 

type Event struct {
	// ID deduplicates a spooled event replayed after a reconnection.
	ID     string `json:"id"`
	NodeID string `json:"node_id"`

	Kind       EventKind                       `json:"kind"`
	ObservedAt time.Time                       `json:"observed_at"`
	Facts      *capability.NodeFacts           `json:"facts,omitempty"`
	Workload   *capability.WorkloadObservation `json:"workload,omitempty"`
}

Event is one fact a node reports on its own authority: its liveness and inventory, or a container's lifecycle. Application semantics arrive separately, through the run's own reporting path.

func (Event) Validate 

func (event Event) Validate() error

type EventKind 

type EventKind string

EventKind names what a node is reporting. 

const (
	// EventHeartbeat carries the node's current host and inventory facts and
	// renews its lease.
	EventHeartbeat EventKind = "heartbeat"
	// EventWorkload carries a container lifecycle transition the node observed
	// directly, independently of anything the application reports.
	EventWorkload EventKind = "workload"
)

type Invitation 

type Invitation struct {
	NodeID     string
	RentalID   string
	Generation uint64
	// ShadowPriceUSDPerHour is what this machine costs to hold. Placement needs
	// a price to weigh a node against fresh capacity, and a node without one is
	// refused rather than treated as free.
	ShadowPriceUSDPerHour float64
	// RedeemableThrough is the moment the caller stops waiting for this machine.
	// The invitation lasts exactly that long, because the two are the same
	// question asked from either end: a host is handed its material once, when it
	// is created, and nothing can tell it anything afterwards, so material that
	// lapses first is a paid machine booting into a fleet it can never join, and
	// material that outlives the wait is a bearer credential still redeemable for
	// a machine nobody is expecting any more.
	//
	// Stated by whoever knows the wait, which is provisioning. An operator
	// inviting a machine by hand is waiting on a person rather than on a
	// deadline, and gets DefaultInvitation.
	RedeemableThrough time.Time
	// Purchase is the rest of what this machine is bought on: the block of time it
	// is billed in, the kinds of work its operator holds it for, and the moment it
	// stops being Mercator's.
	Purchase Purchase
}

Invitation is what an operator or a capacity provider states about a machine before it exists: which node it will be, which Rental generation it belongs to, and what holding it costs.

type Operation 

type Operation struct {
	OperationID string `json:"operation_id"`
	NodeID      string `json:"node_id"`

	Kind         CommandKind    `json:"kind"`
	FencingToken uint64         `json:"fencing_token"`
	State        OperationState `json:"state"`
	IssuedAt     time.Time      `json:"issued_at"`
	SettledAt    time.Time      `json:"settled_at,omitzero"`
	Payload      []byte         `json:"payload"`
	// Failure is the node's own explanation when the operation did not succeed.
	// It never carries credential material.
	Failure string `json:"failure,omitempty"`
}

Operation is the durable record of one command and what became of it.

func (Operation) Reissuable 

func (operation Operation) Reissuable() bool

Reissuable reports whether this identity may be asked of the machine again. Only a refusal ever is: a pending command is still in flight and an applied one already happened, and answering either with anything but Duplicate is how a workload gets launched twice.

A refusal is reissuable only for a command that cannot have changed the machine on its way to failing, which is the distinction CommandKind already draws. A pull that failed left nothing, so the same content can be asked for again; a launch that failed may have created the container, so its identity is spent whatever the machine said about it.

Without this the identity was terminal in either direction: a node whose pull failed answered Duplicate for that content from then on, delivered nothing, and never appeared as applied, so the content was neither there nor askable while the node agent was deliberately not remembering the failure so that a retry could happen.

type OperationState 

type OperationState string

OperationState is how far one command has travelled. 

const (
	// OperationPending is recorded and not yet acknowledged by the node. It
	// survives a control-plane restart, which is what lets the node be told
	// again rather than the work being lost.
	OperationPending OperationState = "pending"
	// OperationApplied is acknowledged by the node with a result.
	OperationApplied OperationState = "applied"
	// OperationRefused is a command the node declined, most often because its
	// fencing token was superseded.
	OperationRefused OperationState = "refused"
)

type Option 

type Option func(*Registry)

Option configures a Registry.

func WithAgentVersion 

func WithAgentVersion(version string) Option

WithAgentVersion pins the node agent build a bootstrap asks for. There is no default and there deliberately cannot be one: the version is what a capacity provider substitutes into the operator's download URL, so a value invented here would be a pin nobody chose, and every machine ever rented by every deployment would install whatever is behind that same name on the day it booted. Left unstated, a bootstrap names no build and a provider that needs one refuses before a machine is paid for.

func WithClock 

func WithClock(now func() time.Time) Option

WithClock replaces the wall clock. The Lab injects a deterministic one so lease expiry and token windows are reproducible.

func WithIdentitySource 

func WithIdentitySource(identity func() string) Option

WithIdentitySource replaces how the registry mints identities for machines an operator did not name. Production uses random material; the Lab and tests inject a deterministic source so an invitation replays identically.

func WithLease 

func WithLease(lease time.Duration) Option

WithLease sets how long a node is believed absent a heartbeat.

func WithSession 

func WithSession(session time.Duration) Option

WithSession sets how long one session credential stays valid. Tests shorten it so a machine outliving its first session is stated rather than waited out: at the production thirty minutes, every case in the tree finishes inside the window and none of them can see what happens after it.

type Purchase 

type Purchase struct {
	// BillingIntervalSeconds is the block of time this machine is bought in. It is
	// what makes an hour Mercator has committed to an hour it pays for whether or not
	// a Run uses the rest of it: work that runs past the end of one block commits
	// Mercator to the next whole one, and the part of that nothing uses is charged to
	// the placement that bought it rather than to nobody.
	BillingIntervalSeconds int64 `json:"billing_interval_seconds,omitempty"`
	// EligibleClasses is the kinds of work this machine may be used for. Placement
	// refuses every other class outright rather than pricing it, because a
	// reservation is a statement about what the machine is for and no amount of
	// waiting changes it.
	EligibleClasses []domain.ServiceClass `json:"eligible_classes,omitempty"`
	// AvailableUntil is the moment this machine stops being Mercator's to use. It is
	// a window somebody stated rather than capacity that can vanish without notice,
	// so work that could still be running then is refused before it starts and work
	// that finishes inside it is never at risk.
	AvailableUntil time.Time `json:"available_until,omitzero"`
}

Purchase is the sale one enrolled machine sits under, as its operator stated it. It is one type rather than three fields because it is one statement: an operator answering "what am I buying, for whom, and until when" answers all of it at once, and it is what Placement reads as the terms of the capacity.

Every part of it is optional and every absence is an answer rather than a default. No increment is a machine bought in no blocks at all, which is an operator's own hardware: Mercator holds it continuously, so no second of it is a fresh commitment and there is no tail of a block to charge. No classes is a machine held for nobody in particular. No moment is a machine with no window.

type Record 

type Record struct {
	ID string `json:"id"`

	RentalID string `json:"rental_id"`
	// Generation is the Rental lifecycle cycle this node belongs to. Capacity
	// that stops and resumes comes back as a new generation, so a node from the
	// previous one can never be mistaken for the current runtime.
	Generation uint64 `json:"generation"`
	State      State  `json:"state"`
	// FencingToken increases on every enrollment. A command carrying a lower
	// token is refused, which is what stops a partitioned old session from
	// acting after a new one took over.
	FencingToken uint64 `json:"fencing_token"`
	// EnrollmentTokenID identifies the invitation this node may redeem, exactly
	// once.
	EnrollmentTokenID string               `json:"enrollment_token_id"`
	EnrollmentExpires time.Time            `json:"enrollment_expires"`
	EnrolledAt        time.Time            `json:"enrolled_at,omitzero"`
	LeaseExpires      time.Time            `json:"lease_expires,omitzero"`
	LastHeartbeatAt   time.Time            `json:"last_heartbeat_at,omitzero"`
	AgentVersion      string               `json:"agent_version,omitempty"`
	Facts             capability.NodeFacts `json:"facts"`
	// ShadowPriceUSDPerHour is what holding this machine costs, as the operator
	// configured it. Placement needs a price to weigh a node against fresh
	// capacity; a node with none has unknown pricing and is refused rather than
	// treated as free.
	ShadowPriceUSDPerHour float64 `json:"shadow_price_usd_per_hour,omitempty"`
	// Purchase is what an operator says this machine is bought on beyond its price:
	// the block of time it is billed in, the kinds of work they hold it for, and the
	// moment it stops being Mercator's.
	Purchase Purchase `json:"purchase,omitzero"`
}

Record is one node's durable identity and latest reported facts. Identity is immutable: a node does not choose its ID, and it cannot claim a generation it was not invited for.

func (Record) Alive 

func (record Record) Alive(now time.Time) bool

Alive reports whether the control plane still believes this node at now. An expired lease is not a death certificate: it means Mercator has stopped hearing from the node and must reconcile rather than assume.

func (Record) CommittedUntil 

func (record Record) CommittedUntil(now time.Time) time.Time

CommittedUntil is the end of the billing interval this machine is inside right now, and nothing for a machine bought in no increments: Mercator holds such a machine continuously, so there is no interval whose end is a decision.

The intervals are counted from enrollment, because that is the moment Mercator started paying for this generation of this machine. A node that has not enrolled is not being paid for and is not offered either.

func (Record) Disk 

func (record Record) Disk() DiskReport

Disk is which of the three this node's latest facts amount to. A node states its own observation time on every report, which is what separates a machine that has said nothing from one that answered.

func (Record) Ref 

func (record Record) Ref() capability.NodeRef

Ref is this record's address in the capability contract.

func (Record) Retired 

func (record Record) Retired() bool

Retired reports whether this node's Rental generation is over. It is the only terminal state, and what it withdraws is standing: this identity enrolls no more, opens no session, renews no lease, is published as capacity to nobody, and is asked to do nothing further. It withdraws nothing the machine says about what it already did, because the node is the only authority on that and a generation that ended while a container was still running is exactly when Mercator most needs to hear it.

func (Record) Terms 

func (record Record) Terms(now time.Time) domain.CapacityTerms

Terms is what this machine was sold to Mercator on, as the offer built from this record publishes them. They are derived from the operator's own configuration and from the moment Mercator started paying, because a billing interval is a repeating block anchored to the beginning of the lease and nothing else in the record can say where the current one ends.

type Registry 

type Registry struct {
	// contains filtered or unexported fields
}

Registry is the control plane's authority over enrolled nodes. It implements capability.NodeRuntime by recording every command durably and delivering it over the node's own outbound session, so a command survives a control-plane restart and a node that reconnects is told again rather than losing the work.

func NewRegistry 

func NewRegistry(store Store, signer *Signer, controlPlaneURL string, opts ...Option) *Registry

func (*Registry) CloseSession 

func (registry *Registry) CloseSession(session *Session)

CloseSession ends one node's session, which a transport calls when its connection drops.

func (*Registry) Drain 

func (registry *Registry) Drain()

Drain ends every open session and refuses to open another. It is what a control plane shutting down says to the object that owns the sessions.

It exists because a session is a long-lived read: the node holds the connection open and the control plane writes commands down it, so the request is active for as long as the machine is healthy. http.Server.Shutdown waits for active requests and cancels none of them, which means a control plane with one enrolled machine had nothing in the tree that could end the read and waited out its whole shutdown window on a drain that could never finish.

A drained node loses nothing. Commands are durable before they reach a session, so the machine reconnects to whatever control plane comes back and is told again about everything it never acknowledged.

func (*Registry) Enroll 

func (registry *Registry) Enroll(ctx context.Context, request capability.EnrollmentRequest) (capability.Enrollment, error)

Enroll redeems an invitation for an authenticated session. Identity is not negotiable: the request must name the node and generation the invitation was minted for, and the invitation is spent by redeeming it.

Two independent doors refuse a replay, and they refuse it for two different reasons. The signer answers whether this material is this node's invitation and whether its window is still open, which needs nothing durable and holds even against a store that lost the record. The store answers whether this exact invitation has already been redeemed, which holds inside the window, where the signature is still perfectly good. A machine whose session lapsed presents material both of them refuse eventually and only the second refuses at once, so collapsing them would leave a replayed bootstrap accepted for the rest of its window.

The way back for a machine with nothing left to present is Reinvite, which is a fresh invitation for the same identity, and never this route being made forgiving.

func (*Registry) EnrolledAt 

func (registry *Registry) EnrolledAt(ctx context.Context, ref capability.NodeRef) (time.Time, error)

EnrolledAt is when the machine invited under this identity opened its session, and the zero time while none has. It is the one authority on the question: a provider can say a machine is active and an operator can say an image has an agent in it, and neither is a session Mercator can create a container through.

It answers with the moment rather than with a yes because the moment is a fact this registry holds and nobody else does. The agent calls Mercator to enrol, so the session is dated when it opens; a caller told only that a node is ready would have to date the arrival from its own next look, and then how long a machine took to become usable would be a property of how often it was asked about.

An identity nobody has heard from is not an error. A node invited and never filled is exactly the state provisioning waits in, and reporting it as a failure would make every look at a machine still booting an incident.

func (*Registry) ExpireLeases 

func (registry *Registry) ExpireLeases(ctx context.Context) ([]Record, error)

ExpireLeases marks nodes the control plane has stopped hearing from as lost and returns them. A lost node is unobserved, not dead: its workloads need reconciliation, and only the node or its provider can say what happened.

func (*Registry) Facts 

func (registry *Registry) Facts(ctx context.Context, ref capability.NodeRef) (capability.NodeFacts, error)

Facts returns the node's latest reported host and inventory facts. It is an observation with an age, not a live read: callers weigh staleness rather than assuming currency.

func (*Registry) Invite 

func (registry *Registry) Invite(ctx context.Context, invitation Invitation) (capability.NodeBootstrap, error)

Invite reserves a node identity and mints the bootstrap material a machine needs to reach the control plane. The identity exists before the machine does, so capacity provisioned by a request whose response was lost is still reconcilable.

func (*Registry) LaunchWorkload 

func (registry *Registry) LaunchWorkload(ctx context.Context, command capability.LaunchWorkloadCommand) (capability.OperationReceipt, error)

func (*Registry) LeaseWindow 

func (registry *Registry) LeaseWindow() time.Duration

LeaseWindow is how long the registry believes a node absent a heartbeat. Agents use it to choose a heartbeat interval that keeps them inside it.

func (*Registry) List 

func (registry *Registry) List(ctx context.Context) ([]Record, error)

List returns every node identity in the deployment, whatever its state, so operators and reconciliation can see capacity that never enrolled as readily as capacity that did.

func (*Registry) NodeSupport 

func (registry *Registry) NodeSupport() capability.NodeSupport

NodeSupport reports what this runtime performs, which is the whole point of a negotiated capability set: a declaration is a promise the control plane will place work against, so anything declared here that the Docker runtime does not do is Placement believing in locality nothing produces. Four of the five are earned today: the agent enumerates the images and layers it unpacked and reports the platform of each, it attaches and enumerates deployment-scoped cache volumes across workloads, it pulls an image it was not asked to launch and answers Duplicate on a redelivery, and it replicates an immutable Artifact from a location the control plane minted and reports the digest those bytes actually hashed to. It still reclaims no disk, and garbage collection becomes true in the slice that implements it and not before.

func (*Registry) ObserveWorkload 

func (registry *Registry) ObserveWorkload(ctx context.Context, ref capability.WorkloadRef) (capability.WorkloadObservation, error)

ObserveWorkload returns what the node last reported about one workload. A node that has never mentioned it answers absent, which after a launch is materially different from an exit and must not be read as one.

func (*Registry) Offers 

func (registry *Registry) Offers(ctx context.Context) ([]domain.OfferSnapshot, error)

Offers presents every ready node as reusable capacity. This is the only source of reusable-lane offers today: a node has, by definition, a host runtime that can execute a second workload, which is exactly what the lane claims.

A node that has gone quiet is not offered. Its workloads still need reconciling, but nothing new should be sent to a machine Mercator has stopped hearing from.

func (*Registry) OpenSession 

func (registry *Registry) OpenSession(ctx context.Context, nodeID, sessionToken string) (*Session, error)

OpenSession authenticates a node's outbound connection and returns its command stream, beginning with every command it has not acknowledged. A node that was disconnected, or that reconnected to a restarted control plane, therefore receives the work it missed instead of the work being lost.

A retired machine gets no session, which is what makes ending one worth anything. The credential outlives the machine and an agent's transport reconnects the instant its connection drops, so closing the session of a machine Mercator gave up buys nothing on its own: the same token would open the next one, preloaded with every command the last one never acknowledged. This is also the door through which such an agent learns it is retired.

func (*Registry) PrepareArtifact 

func (registry *Registry) PrepareArtifact(ctx context.Context, command capability.PrepareArtifactCommand) (capability.OperationReceipt, error)

func (*Registry) PrepareImage 

func (registry *Registry) PrepareImage(ctx context.Context, command capability.PrepareImageCommand) (capability.OperationReceipt, error)

func (*Registry) Reconcile 

func (registry *Registry) Reconcile(ctx context.Context, ref capability.NodeRef) (capability.Reconciliation, error)

Reconcile reports what the node actually holds: which operations it has applied and which workloads it knows about. It is how a restarted control plane learns it must not launch again.

func (*Registry) RecordEvents 

func (registry *Registry) RecordEvents(ctx context.Context, nodeID, sessionToken string, events []Event) error

RecordEvents accepts facts a node reports on its own authority: its liveness and inventory, and container lifecycle transitions. Events carry IDs so a spool replayed after a reconnection changes nothing, which is what lets an agent keep reporting while disconnected.

A retired machine still reports here. The node owns container lifecycle and exit codes, and a generation ends while a container is running every time a provider reclaims a machine inside its interruption window: refusing the report would leave the exit nowhere, and the only other account of what the process did is an application callback, which must never be the sole way Mercator learns a process exited. What retirement does withdraw is the lease renewal below.

func (*Registry) RecordResult 

func (registry *Registry) RecordResult(ctx context.Context, nodeID, sessionToken string, result Result) error

RecordResult accepts a node's answer about one command. A node that already applied the operation reports Duplicate, which is what makes redelivery after a lost response safe rather than doubling the effect.

A retired machine settles here too. The command was dispatched while the generation stood, the agent applied it, and refusing the answer would strand an operation the machine really performed as pending forever, leaving the control plane's record permanently disagreeing with the machine's over work that actually happened.

func (*Registry) Ref 

func (registry *Registry) Ref(ctx context.Context, nodeID string) (capability.NodeRef, error)

Ref resolves a node's current identity, including the generation a command must be stamped with. A node that has gone quiet is refused here rather than sent work, because the control plane cannot say what it is doing.

func (*Registry) Reinvite 

func (registry *Registry) Reinvite(ctx context.Context, nodeID string, redeemableThrough time.Time) (capability.NodeBootstrap, error)

Reinvite is the invitation an identity that already exists is redeemable on. An agent whose machine came back without its session credential, or a Rental generation that restarted, joins through this rather than replaying a spent invitation.

An invitation still outstanding is handed back rather than replaced, and that is the whole of what makes this safe to ask for again. Redeeming is an exact match on the token the record names, so minting a fresh one invalidates the one already out there and the machine holding it is a paid host that can no longer enrol. Provisioning asks for this on every attempt, before it can know whether a machine from an earlier attempt exists, so an identity's material has to be stable across attempts for a machine adopted from one of them to be able to join at all.

A fresh one is minted only where nothing can still be holding a usable copy of the old one: an invitation already redeemed, which the record clears, one whose window has closed, and one that closes before the caller stops waiting. The last is not a machine given up on: an invitation that lapses inside the wait cannot end in an enrolment whoever is holding it, so replacing it costs the machine already out there nothing and gives the one about to be built material that will still be good when it boots.

func (*Registry) RenewSession 

func (registry *Registry) RenewSession(ctx context.Context, nodeID, sessionToken string) (capability.SessionRenewal, error)

RenewSession mints a fresh credential for a node that still holds a valid one. It is how a machine keeps working past the thirty minutes its first credential is good for, and it is the only way: the invitation the agent joined with is spent, so an agent that let its session lapse has nothing left to present and stops being able to speak to a control plane whose machine is still running.

It writes nothing. A session credential is signed rather than stored, and renewing changes no fact about the node: the fencing token does not move, the lease is not touched, and the node is the node it was a moment before. That is what makes this a different act from enrolling rather than a gentler one.

A retired machine renews nothing, which is the rule every door in this file keeps: asking for something is refused, reporting what already happened is kept. Retirement is also the whole of what bounds a stolen credential here. A bearer token can be presented by whoever holds it, so the answer to a leak is ending the generation, and a renewal that outlived the generation would take that answer away.

func (*Registry) Retire 

func (registry *Registry) Retire(ctx context.Context, nodeID string) error

Retire ends this node's working life because the Rental generation it was invited for is over. The identity stays in the fleet as history: what it was told and what it reported is what a later reconciliation reads, and deleting it would leave the machine's last word nowhere.

The record is written before the session is ended, in that order. The record is what refuses the machine everything: closing the session only drops the connection it happens to hold, and an agent redials in milliseconds, so a control plane that ended the session and failed to write would hand the same credential a fresh session preloaded with every command the last one never acknowledged. Written first, a control plane that stops in between still offers the machine to nobody and answers its next connection with ErrRetired.

func (*Registry) StopWorkload 

func (registry *Registry) StopWorkload(ctx context.Context, command capability.StopWorkloadCommand) (capability.OperationReceipt, error)

type Result 

type Result struct {
	OperationID string `json:"operation_id"`
	// Applied is false only when the node refused the command outright.
	Applied bool   `json:"applied"`
	Failure string `json:"failure,omitempty"`
	// Duplicate reports that the node had already applied this operation,
	// which is the answer that makes retry after a lost response safe.
	Duplicate  bool      `json:"duplicate"`
	ReportedAt time.Time `json:"reported_at"`
}

Result is what a node reports back about one operation.

type Session 

type Session struct {
	NodeID string

	FencingToken uint64
	// contains filtered or unexported fields
}

Session is one node's open outbound connection. The control plane never dials a node: the node opens this, and commands travel back down it.

func (*Session) Commands 

func (session *Session) Commands() <-chan Command

Commands is the stream a transport writes to the node.

func (*Session) Done 

func (session *Session) Done() <-chan struct{}

Done closes when the control plane supersedes or ends this session, which is the transport's signal to close the connection.

type Signer 

type Signer struct {
	// contains filtered or unexported fields
}

Signer mints the two short-lived credentials a node ever holds. An enrollment token proves a machine was invited to be one specific node generation, and is redeemable once. A session token authenticates one authenticated outbound session and expires with it. Neither is a long-lived credential, and neither authorizes anything beyond its own node.

func NewSigner 

func NewSigner(key []byte) *Signer

func (*Signer) Enabled 

func (signer *Signer) Enabled() bool

func (*Signer) Enrollment 

func (signer *Signer) Enrollment(nodeID, rentalID string, generation uint64, expires time.Time) (string, error)

Enrollment mints the invitation a machine presents to join. The token binds the node identity, its Rental generation, and an expiry, so a leaked token cannot enroll a different node, a different generation, or enroll late.

func (*Signer) Expiry 

func (signer *Signer) Expiry(at time.Time) time.Time

Expiry is when a credential asked to last until at really stops being accepted. A token carries whole seconds, so the honest answer is the second the credential actually has rather than the fraction it was asked for.

It matters because the node is told this moment and renews against it. Told a moment later than its credential's own, an agent renews after the credential died, presents material the signer refuses, and is locked out of a control plane whose machine is still running. Rounding the other way would be a credential outliving what the control plane believes it granted.

func (*Signer) Session 

func (signer *Signer) Session(nodeID string, fencingToken uint64, expires time.Time) (string, error)

Session mints the credential for one authenticated session. It binds the fencing token, so a session token from a superseded enrollment cannot authenticate against the current one.

func (*Signer) VerifyEnrollment 

func (signer *Signer) VerifyEnrollment(nodeID, rentalID string, generation uint64, token string, now time.Time) bool

VerifyEnrollment reports whether token invites this exact node identity and generation, and has not expired at now.

func (*Signer) VerifySession 

func (signer *Signer) VerifySession(nodeID string, fencingToken uint64, token string, now time.Time) bool

VerifySession reports whether token authenticates this node at this fencing token, and has not expired at now.

type State 

type State string

State is what the control plane believes about a node right now. 

const (
	// StateEnrolling is an identity that has been invited but has not yet
	// redeemed its enrollment token.
	StateEnrolling State = "enrolling"
	// StateReady is an enrolled node inside its lease.
	StateReady State = "ready"
	// StateLost is an enrolled node whose lease expired without a heartbeat.
	// Its workloads are unobserved, not dead: only the node or the provider can
	// say what actually happened.
	StateLost State = "lost"
	// StateRetired is a node whose Rental generation is over. It can never
	// enroll again under the same generation.
	StateRetired State = "retired"
)

func (State) Valid 

func (state State) Valid() bool

type Store 

type Store interface {
	// Invite records a node identity before any machine exists to fill it, so
	// an accepted-but-lost provision is still reconcilable.
	Invite(ctx context.Context, record Record) error
	Get(ctx context.Context, nodeID string) (Record, error)
	// Find resolves a node by identity alone. An enrolling machine knows which
	// node it is and nothing else, so enrollment is deployment-scoped.
	Find(ctx context.Context, nodeID string) (Record, error)
	List(ctx context.Context) ([]Record, error)
	// Enroll redeems an invitation exactly once, bumping the fencing token and
	// opening a lease. Redeeming a spent invitation returns ErrEnrollmentSpent.
	Enroll(ctx context.Context, nodeID string, enrollment Enrollment) (Record, error)
	// Reinvite replaces an existing identity's redeemable invitation without
	// disturbing its current enrollment.
	Reinvite(ctx context.Context, nodeID, enrollmentTokenID string, expires time.Time) error
	// Retire ends a node's working life, because the Rental generation it was
	// invited for is over. It can never enroll again, it renews no lease, and it
	// is offered as capacity no more, whatever state it was in: a generation can
	// end before its machine ever answered.
	//
	// Retiring a retired node changes nothing. A generation's end is reached
	// again by any reconciliation that lost its answer, and telling the second
	// pass it failed would leave a caller retrying work already done.
	Retire(ctx context.Context, nodeID string) error
	// Heartbeat renews a lease and stores the node's latest facts.
	Heartbeat(ctx context.Context, nodeID string, facts capability.NodeFacts, leaseExpires time.Time) (Record, error)
	// RecordEvent stores one node-authored fact, reporting false when this
	// event ID was already recorded so a replayed spool changes nothing.
	RecordEvent(ctx context.Context, event Event) (bool, error)
	// LatestWorkload returns the node's most recent observation of one
	// workload, and whether the node has ever reported it.
	LatestWorkload(ctx context.Context, nodeID, runID, attemptID string) (capability.WorkloadObservation, bool, error)
	// Workloads returns the node's latest observation of every workload it has
	// reported, which is what reconciliation compares against.
	Workloads(ctx context.Context, nodeID string) ([]capability.WorkloadObservation, error)
	// AppendOperation records one command durably, reporting true when this
	// operation ID was already recorded and is not reissuable. What the stored
	// operation became decides that: a refused command that cannot have left an
	// effect behind is recorded afresh and reported as new, so the machine is asked
	// again instead of being answered from a record of the pull that failed.
	AppendOperation(ctx context.Context, operation Operation) (Operation, bool, error)
	// SettleOperation records what the node did with one command.
	SettleOperation(ctx context.Context, nodeID string, result Result) error
	// PendingOperations returns commands the node has not acknowledged, oldest
	// first. They are redelivered when a node reconnects.
	PendingOperations(ctx context.Context, nodeID string) ([]Operation, error)
	// AppliedOperationIDs returns every operation the node acknowledged, which
	// is what tells a restarted control plane not to act twice.
	AppliedOperationIDs(ctx context.Context, nodeID string) ([]string, error)
	// ExpireLeases marks every node whose lease elapsed as lost and returns
	// them, so a caller can reconcile what those nodes were running.
	ExpireLeases(ctx context.Context, now time.Time) ([]Record, error)
}

Store is the durable half of the node registry. Everything a restart must not forget lives here: who a node is, what it has been told, and what it reported back. Session channels are deliberately not in this interface, because a session cannot survive a restart and pretending otherwise is how duplicate launches happen.

func NewMemoryStore 

func NewMemoryStore() Store

NewMemoryStore returns an in-memory Store for focused tests and local compositions. Production uses the SQLite store, because a node registry that forgets across a restart cannot promise not to launch twice.

Source Files

View all Source files

Directories

Path Synopsis
Package nodetest is the shared conformance suite every node.Store must pass.
 Package nodetest is the shared conformance suite every node.Store must pass.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.