Documentation
¶
Overview ¶
Package members - self-announce: write this agent's libp2p peer.ID, multiaddrs and transfer endpoint into its own Pod's annotations so other agents can discover the libp2p identity without operator-supplied bootstrap config. See the design doc (libp2p bootstrap) and the design doc (membership view).
Package members provides Gantry's cluster-membership view, sourced from Kubernetes informers.
Design :
- A label-selected Pod informer enumerates Gantry DaemonSet pods. The selector is operator-configurable (`members_label_selector`) so this package does not assume a particular DaemonSet manifest. - A cluster-scoped Node informer is joined on `pod.spec.nodeName` so each peer carries its zone label (default `topology.kubernetes.io/zone`). HRW reads `Node.Zone` directly - Members owns the join so HRW does not re-fetch. - `Self` is set from the Downward API (`spec.nodeName` -> `GANTRY_NODE_NAME`); this is the stable identifier HRW uses to score digests.
Failure modes:
- In-cluster: a missing/invalid service-account token fails fast at New; the agent cannot run without a Kubernetes API view. - Out-of-cluster: explicit `members_kubeconfig` is honored; this path is the developer/test path, not production.
Tests use `client-go/kubernetes/fake` via the Options.Clientset escape hatch so the package does not require a real cluster.
Index ¶
- Constants
- type Manager
- func (m *Manager) AnnounceSelf(ctx context.Context, podName string, ann SelfAnnouncement) error
- func (m *Manager) RunningMatchingPodCount() int
- func (m *Manager) Self() ifaces.NodeID
- func (m *Manager) Snapshot() []ifaces.Node
- func (m *Manager) SnapshotForBootstrap() []ifaces.Node
- func (m *Manager) Start()
- func (m *Manager) Stop()
- func (m *Manager) WaitForSync(ctx context.Context) error
- type Options
- type SelfAnnouncement
Constants ¶
const ( AnnotationPeerID = "gantry.io/peer-id" AnnotationP2PAddrs = "gantry.io/p2p-addrs" // comma-separated multiaddrs AnnotationTransferAddr = "gantry.io/transfer-addr" // host:port )
Annotation keys agents publish on their own pod so peers can discover libp2p identity and the transfer endpoint without operator-supplied bootstrap config.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Manager ¶
type Manager struct {
// contains filtered or unexported fields
}
Manager owns the Pod+Node informers and exposes ifaces.Members.
func New ¶
New builds a Manager. The factory is constructed but informers are not started until Start is called.
func (*Manager) AnnounceSelf ¶
AnnounceSelf writes the announcement into the named Pod's annotations via a JSON-merge-patch. PodName MUST identify the agent's own pod (typically sourced from the Downward API as $POD_NAME); Namespace defaults to the namespace the informer was scoped to.
The patch is a strict overwrite: any prior gantry.io/* annotation values are replaced. Other annotations on the pod are left untouched.
AnnounceSelf requires `pods` `patch` verb on the agent's own namespace; see deploy/gantry/serviceaccount.yaml.
func (*Manager) RunningMatchingPodCount ¶
RunningMatchingPodCount returns the number of pods the informer sees in Running phase with a populated PodIP, REGARDLESS of Ready condition or any annotation. The informer's label selector and namespace scope already constrain the set to Gantry peers; this method intentionally does not apply any further filter.
Contract (do not tighten):
- Requires Status.Phase == PodRunning AND Status.PodIP != "". - Requires NOT terminating (DeletionTimestamp == nil) - see below for the rolling-update deadlock this avoids. - Does NOT require: PodReady=True, gantry.io/p2p-addrs, gantry.io/peer-id, or any other readiness/announcement signal.
This is the strict superset that Snapshot (Ready) and SnapshotForBootstrap (Running + p2p-addrs annotated) are filtered subsets of. It exists so /readyz can distinguish "this is a real single-node cluster" (count == 1) from "multi-node cluster, peers just haven't self-announced yet" (count > 1, bootstrap view ≤ 1). Surfacing the latter as "peer self-announcements pending" rather than "dht routing table empty" tells operators the real cause; it also keeps the readiness probe at 503 during the first-rollout window where every pod is Running but none has yet published its libp2p multiaddrs - without this gate, /readyz would race to green before any peer was actually reachable.
Terminating-pod exclusion is the rolling-update / drain guard: a terminating pod stays Phase=Running for the grace period but will never re-AnnounceSelf with fresh annotations. Counting it here while SnapshotForBootstrap rightly excludes it (stale or absent p2p-addrs) would deadlock the surviving pods at "peer self-announcements pending" - they would wait forever for an announcement from a peer that has no opportunity to make one.
func (*Manager) Snapshot ¶
Snapshot returns the current peer view: one Node per Ready pod matching the selector, joined on spec.nodeName for zone labels. The returned slice is sorted by NodeID for deterministic HRW input.
PeerID, P2PAddrs and a transfer-addr override are read from pod annotations (gantry.io/peer-id, gantry.io/p2p-addrs, gantry.io/transfer-addr) populated by each agent's AnnounceSelf call at startup. Pods that have not yet published these annotations still appear in the snapshot - Addr falls back to podIP[:TransferPort], PeerID/P2PAddrs are empty until the announcement arrives.
Terminating pods (DeletionTimestamp set) are excluded even if they are still Phase=Running and Ready=True - kubelet leaves Ready=True for the duration of the grace period, and routing HRW/transfer traffic to a pod that is about to disappear is a connection-refused waiting to happen. The exclusion is enforced both directly (the explicit podTerminating check below) and via podReady, which short-circuits on the same predicate.
func (*Manager) SnapshotForBootstrap ¶
SnapshotForBootstrap returns the membership view used for libp2p bootstrap dialing: every Running pod matching the selector that has published a gantry.io/p2p-addrs annotation, regardless of Ready status. This intentionally diverges from Snapshot because the readiness probe depends on a populated DHT routing table, which depends on libp2p bootstrap, which (under Snapshot's filter) would depend on at least one peer already being Ready - a deadlock on first-cluster boot and full-cluster restart. Bootstrap-time peers are a strictly larger set than serving-time peers; we want any peer we can reach, even a NotReady one, just to seed the DHT.
The serving path (HRW choice, transfer destinations) MUST keep using Snapshot so unready peers don't receive request traffic.
Terminating pods (DeletionTimestamp set) are excluded even if they still have valid p2p-addrs annotations - they will never re-announce, so bootstrap dials to them are wasted at best and at worst keep the connection pool half-open until the termination grace period expires.
func (*Manager) Start ¶
func (m *Manager) Start()
Start begins the informers' list-and-watch in the background. It does NOT wait for initial sync - callers MUST follow Start with a WaitForSync(ctx) call under a bounded context so they own the sync-deadline policy (production mode treats a timeout as fatal, dev mode warns and continues; see cmd/gantry/main.go.buildMembers for the canonical use). Previously Start blocked on WaitForSync(ctx) using the long-lived app context, which made the 10s bounded WaitForSync in buildMembers dead code - an RBAC / permissions failure could pin startup indefinitely instead of reaching the deadline branch.
type Options ¶
type Options struct {
// NodeName is the Kubernetes node this agent runs on. Required - used
// as Self and as the join key against the Node informer for zone
// resolution.
NodeName string
// Namespace restricts the pod informer; empty means cluster-wide.
Namespace string
// LabelSelector is the K8s label selector identifying peer agents.
// Required (empty selector would enrol every pod in the cluster).
LabelSelector string
// ZoneLabelKey is the node label that exposes the topology zone.
// Defaults to `topology.kubernetes.io/zone` if empty.
ZoneLabelKey string
// Kubeconfig is the path to an out-of-cluster kubeconfig. Empty means
// in-cluster service-account discovery.
Kubeconfig string
// Clientset is an optional pre-built clientset (typically the fake
// clientset in tests). When non-nil, Kubeconfig is ignored.
Clientset kubernetes.Interface
// ResyncPeriod is the informer resync interval. Zero means "no resync"
// (rely on watch events). Default 30s when zero.
ResyncPeriod time.Duration
// TransferPort is the TCP port each agent's transfer endpoint
// listens on (the design doc). When non-zero, Snapshot fills Node.Addr as
// "podIP:TransferPort"; when zero, Snapshot returns the bare pod
// IP (back-compat). Production deployments MUST set this so
// peer-fetch URLs are reachable.
TransferPort int
}
Options configures a Manager.
type SelfAnnouncement ¶
type SelfAnnouncement struct {
// PeerID is the libp2p peer.ID string form.
PeerID string
// P2PAddrs is the agent's libp2p listen multiaddrs. Each entry is
// a full /ip4/.../tcp/<port>/p2p/<peerID> multiaddr.
P2PAddrs []string
// TransferAddr is the host:port for the HTTP/2 transfer endpoint
// reachable from peer pods. When empty, peers reconstruct
// "podIP:TransferPort" from members.Options.TransferPort instead.
TransferAddr string
}
SelfAnnouncement is the data each agent publishes onto its own Pod. Empty fields produce empty annotations (which clears any stale value from a previous incarnation of the pod IP).
Source Files