README
¶
federation — cross-cluster SandboxEnv capacity sharing
Each cluster owns its own SandboxEnv objects. When SandboxEnvs of the same
(namespace, name) exist in more than one cluster, this package shares only
their runtime capacity (idle / running / pending / desired per scaling
group) so a Worker's router can tell whether a same-named Env in another
cluster currently has idle room. Env spec is never synced — management
stays local to each cluster.
Shape
Worker A ─ReportFederation─▶ Hub (ws-proxy) ─WatchFederation─▶ Worker A/B/…
Reporter federationStore Registry
(reads local Env status) (soft state, TTL) (soft state, TTL)
- Registry (
registry.go) — the Worker's in-memory store ofCapacityrecords, aged out on a TTL. Injectable clock; safe for concurrent use.ForeignForEnv/ForeignIdledeliberately exclude the local cluster. - CapacitySource (
reporter.go) — reads back the per-Env aggregates the Env reconciler already writes intostatus.scalingGroupsand the status rollups, one record per scaling group plus a whole-Env row (group""). - PublishMetrics (
metrics.go) — exports the registry snapshot asagentbox_federation_env_{idle,running,desired}{cluster,namespace,env,group}.
Transport
Federation rides the existing /v1/ws/sync gRPC connection as
FederationService (pkg/proto/sandbox/sync/v1/sync.proto). The Worker is the
gRPC client (report + watch); the Hub is the server and relays each batch to
every connected Worker. Wire freshness is relative (observed_for_ms) so
TTL expiry is immune to cross-cluster clock skew.
Enablement
Active only in multi-cluster mode — both a sync secret and LOCAL_CLUSTER_ID
must be set. Single-cluster deployments never start the report/watch
goroutines. A Worker connected to a Hub that predates FederationService gets
Unimplemented, logs once, and continues without federation (no disruption to
key / template / cluster-config sync).
Documentation
¶
Overview ¶
Package federation holds the Worker-side soft-state view of cross-cluster SandboxEnv capacity. Same-named SandboxEnvs living in different clusters are independent objects; the federation shares only their per-member runtime capacity so a Worker's Env router can pin a create to an exact member pool in another cluster (forwarding "<cluster>::<pool>"). Env spec is never synced — management stays local to each cluster.
The registry is pure soft state: every record has an observation time and is aged out on a TTL, so a cluster that stops reporting simply disappears. No explicit delete is ever needed.
Index ¶
- func PublishMetrics(snapshot []Capacity)
- type Capacity
- type CapacitySource
- type Registry
- func (r *Registry) BestForeignMember(namespace, env, group string) (clusterID, memberPool string, idle int32, ok bool)
- func (r *Registry) ForeignMembers(namespace, env string) []Capacity
- func (r *Registry) LocalCanGrow(namespace, env, group string) bool
- func (r *Registry) LocalIdle(namespace, env, group string) int32
- func (r *Registry) SetClock(now func() time.Time)
- func (r *Registry) Snapshot() []Capacity
- func (r *Registry) Upsert(items []Capacity)
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func PublishMetrics ¶
func PublishMetrics(snapshot []Capacity)
PublishMetrics rewrites the federation gauges from a full registry snapshot. Resetting first ensures records that aged out since the last publish stop being exported instead of lingering at a stale value.
Types ¶
type Capacity ¶
type Capacity struct {
ClusterID string
Namespace string
EnvName string
MemberPool string
ScalingGroup string
Idle int32
Running int32
Pending int32
Desired int32
// AutoscalingEnabled reports whether this member's scaling group has the
// autoscaler on in its owning cluster. Because each cluster scales
// independently, this is a per-member, per-cluster fact; it gates whether
// Capacity below is meaningful.
AutoscalingEnabled bool
// Capacity is the member's remaining scale-up headroom on its owning
// cluster, meaningful only when AutoscalingEnabled: -1 = enabled but
// unbounded (no finite ceiling), 0 = enabled but at ceiling (cannot grow
// now), >0 = enabled with this much room. When AutoscalingEnabled is
// false it is 0 and ignored.
Capacity int32
// SaturatedFor is the remaining saturation window; zero means not saturated.
SaturatedFor time.Duration
// ObservedAt is when the sample was taken at its source cluster. The
// consumer derives it from the wire's relative age so TTL expiry is
// measured from the real observation instant, immune to clock skew.
ObservedAt time.Time
}
Capacity is one cluster's runtime capacity for a single member pool of a same-named SandboxEnv. The (ClusterID, Namespace, EnvName, MemberPool) tuple is the record's identity; ScalingGroup lets the router honour a requested group.
func (Capacity) CanGrow ¶
CanGrow reports whether the member could accept a new sandbox by scaling up: autoscaling is on and it is not already at its ceiling (Capacity != 0). Capacity == -1 (unbounded) and Capacity > 0 both count as room.
func (Capacity) Schedulable ¶
Schedulable reports whether a create routed here would be served without an open-ended wait: either an idle Pod is ready now, or the member can scale up to make one. A member with no idle and no scale-up room is excluded — routing there would park the request until it times out.
type CapacitySource ¶
CapacitySource produces this cluster's current per-Env capacity, one record per (Env, scalingGroup) plus a whole-Env aggregate row.
func NewCapacitySource ¶
func NewCapacitySource(c client.Client, localClusterID string) CapacitySource
NewCapacitySource builds a CapacitySource backed by the SandboxEnv informer cache.
type Registry ¶
type Registry struct {
// contains filtered or unexported fields
}
Registry is the Worker's in-memory federation store. It is safe for concurrent use. The clock is injectable so tests exercise TTL expiry without sleeping.
func NewRegistry ¶
NewRegistry builds a Registry that ignores its own cluster's records when answering foreign-capacity queries and ages every record out after ttl.
func (*Registry) BestForeignMember ¶
func (r *Registry) BestForeignMember(namespace, env, group string) (clusterID, memberPool string, idle int32, ok bool)
BestForeignMember returns the cluster ID and member pool of the best schedulable member in another cluster for the Env and requested scaling group. A member is schedulable when it can serve without an open-ended wait: an idle Pod ready now, or autoscaling room to make one (see Schedulable). Members that are neither (no idle, cannot grow) are excluded — routing there would park the request until it times out.
Ranking prefers immediacy: any member with idle > 0 outranks any pure scale-up member (an idle Pod serves instantly; a scale-up incurs a cold start). Within the idle tier, more idle wins; within the scale-up tier, more headroom wins (unbounded ranks highest). Ties break on (clusterID, memberPool) for determinism.
The returned idle is the chosen member's idle count — 0 when it was chosen via scale-up headroom. ok is false when no foreign member is schedulable.
func (*Registry) ForeignMembers ¶
ForeignMembers returns every non-expired member record for the Env that belongs to a cluster other than the local one, sorted by (cluster, pool). Used to mirror the cross-cluster view into SandboxEnv.status.clusters so it is visible via kubectl.
func (*Registry) LocalCanGrow ¶
LocalCanGrow reports whether any local member of the Env in the requested group could scale up (autoscaling on and not at ceiling). Lets the router keep a create local — letting the local autoscaler react — instead of spilling to a foreign cluster that can also only scale up.
func (*Registry) LocalIdle ¶
LocalIdle sums this cluster's idle capacity for the Env across members that match the requested scaling group (empty group = all members).
Source Files