Documentation
¶
Overview ¶
Package cluster turns N cadish nodes in a region into a sharded / cooperative cache. It implements two interlocking features driven by one `cluster { … }` Cadishfile membership block:
PEER READ-THROUGH (#7): on a LOCAL cache miss, consult peer cadish nodes (same region) before going to origin; if a peer has the object, stream it locally (same tee contract as origin). Modeled as an origin.Origin (internal/origin/peerorigin) composed BEFORE the real origin in a chain.
OWNERSHIP ROUTING (#8): one node OWNS each cache key via a consistent-hash ring over the cluster peers (the SAME lb ring used for upstream sharding). A request landing on a non-owner is reverse-proxied to the owner so the object is cached once per region, not N times.
Both share cluster membership + peer health (reused from internal/lb) and a hop guard (the X-Cadish-Peer header) that prevents forward loops/storms: a request already forwarded to a peer is never re-forwarded.
ZERO COST WHEN ABSENT. Everything here is gated by the presence of a `cluster` membership block; a non-clustered cadish constructs no Membership and behaves exactly as before.
Index ¶
- Constants
- func IsMembershipBlock(d *cadishfile.Directive) bool
- type Config
- type Fallback
- type Membership
- func (m *Membership) Close()
- func (m *Membership) Config() Config
- func (m *Membership) Fallback() Fallback
- func (m *Membership) IntendedOwner(key string) (string, bool)
- func (m *Membership) IsForwardedHop(h http.Header) bool
- func (m *Membership) IsSelf(peerURL string) bool
- func (m *Membership) ListenAddr() string
- func (m *Membership) Mode() Mode
- func (m *Membership) Owner(key string) (string, bool)
- func (m *Membership) OwnsKey(key string) bool
- func (m *Membership) PeerCount() int
- func (m *Membership) PeerOrigin() origin.Origin
- func (m *Membership) Region() string
- func (m *Membership) Self() string
- func (m *Membership) Start(ctx context.Context)
- func (m *Membership) StorePeerReads() bool
- type Mode
- type PeerStore
Constants ¶
const ( DefaultDataHTTPPort = 80 DefaultDataHTTPSPort = 443 )
DefaultDataHTTPPort / DefaultDataHTTPSPort are the ports the server binds for the data plane when `cadish run` is started with the default -addr / -https-addr flags. `cadish check` has no access to those flags, so ValidatePeerReachable is invoked at check time with these defaults; `cadish run` re-validates against the ACTUAL flag values in server.NewServer (the authoritative check). Kept here so the check side and the run side share one source of truth for "the default listener ports".
const HopHeader = "X-Cadish-Peer"
HopHeader marks a request as already forwarded to a peer cadish node. A node that sees it MUST NOT forward again (read-through or owner-route), which is what makes the cluster loop-safe and storm-safe. Its value is the originating region, used to ignore cross-region hops.
Variables ¶
This section is empty.
Functions ¶
func IsMembershipBlock ¶
func IsMembershipBlock(d *cadishfile.Directive) bool
IsMembershipBlock reports whether a `cluster` directive is the membership form (this package) rather than the pre-existing `cluster NAME { to … }` LB-pool form (internal/lb). The membership block has NO name argument and carries a `peers` directive; the LB-pool form has a name and `to` backends. Distinguishing by shape lets both coexist without a keyword clash.
Types ¶
type Config ¶
type Config struct {
// Self is this node's own peer URL (must appear in Peers). It identifies which
// ring node is "us" for ownership decisions and is excluded from peer fetches.
Self string
// Listen is the OPTIONAL `listen <ip:port>` address of a DEDICATED plain-HTTP peer
// listener this node opens for inbound peer read-through / owner-route traffic. It
// exists for the standalone-TLS bare-metal topology, where the data plane is a :80
// HTTP→HTTPS redirect + a :443 TLS listener and NEITHER can carry plain peer reads:
// a peer dialing :80 is 301'd, and a peer dialing https://IP:443 fails cert
// verification against an IP. `self`/`peers` then point at this dedicated port. Empty
// (the default / k8s topology) means peers reach this node on the plain data listener
// (`-addr`), which only works when the node does NOT terminate TLS. See
// ValidatePeerReachable. The listener serves ONLY marker-carrying peer traffic
// (server.peerListenHandler); it is NOT a general TLS bypass.
Listen string
// Peers are the peer cadish node targets (static URLs and/or dns://, k8s://
// discovery), parsed with the same lb.Target syntax as upstream `to`. At least
// one required; Self must be among them.
Peers []lb.Target
// Region scopes the cluster; it is the value of the X-Cadish-Peer hop header so
// a node ignores hops stamped by a different region.
Region string
// Mode selects #7/#8 coexistence (default read_through).
Mode Mode
// Fallback selects owner-mode behavior on owner-down (default degraded).
Fallback Fallback
// PeerStore selects whether a peer-sourced read-through response is stored locally
// (default PeerStoreLocal — the historical tee-into-local-cache behavior). It only
// affects the read-through path (#7); owner mode reverse-proxies to the owner and
// never tees a peer response, so the knob is inert there. See PeerStore.
PeerStore PeerStore
// Health is the active peer-health probe spec (reused verbatim from lb). Nil
// disables active probing (peers are then always eligible).
Health *lb.HealthSpec
// Replicas is the consistent-hash virtual-node count per peer (0 ⇒ lb default).
Replicas int
// Pos is the directive's source position.
Pos cadishfile.Pos
}
Config is a parsed `cluster { … }` membership block.
func Parse ¶
func Parse(d *cadishfile.Directive) (Config, error)
Parse builds a Config from a `cluster { … }` membership directive. The block has NO name argument (that form is the lb pool) and accepts:
self URL (this node; ∈ peers) peers URL... (repeatable; ≥1) listen IP:PORT (dedicated peer listener; opt) region NAME mode read_through | owner (default read_through) fallback strict | degraded (default degraded) peer_store local | none (default local; read_through tee) health METHOD PATH expect CODE interval D window N threshold T replicas N (ring vnodes; tests)
Errors are positioned *cadishfile.ParseError (file:line:col: message).
func (Config) ValidatePeerReachable ¶ added in v0.2.3
ValidatePeerReachable reports an error when this node's `self` peer port is served by NO listener that carries PLAIN peer traffic — the standalone-TLS trap CAD-43 closes.
Peers reach a node by dialing its `self` URL over plain HTTP (the peer client does not follow redirects and cannot verify a cert against a bare IP). A port is peer-reachable only if a listener answers plain, un-redirected HTTP there:
- the dedicated peer listener (`cluster { listen … }`) — always plain, and gated to ONLY peer traffic (server.peerListenHandler); or
- the data HTTP listener (`-addr`, default :80) — but ONLY when this node does NOT terminate TLS. When it does, :80 is the HTTP→HTTPS redirect (it 301s a peer read) and :443 fails cert verification against an IP, so the data plane cannot carry peer traffic at all — a `listen` port is then mandatory.
httpPort/httpsPort are the data-plane ports the process binds; tlsTerminated marks TLS mode. `cadish check` passes the DEFAULT ports (it has no -addr flag); server.NewServer passes the ACTUAL flag values (authoritative). It also rejects a `listen` port that collides with a data-plane listener (that would fail to bind with a confusing EADDRINUSE).
type Fallback ¶
type Fallback int
Fallback selects owner-mode behavior when the computed owner is unavailable.
const ( // FallbackDegraded (the default) serves the request from the next eligible // ring node, then peer read-through, then local origin — availability over // strict single-ownership. A flapping owner degrades to "cached on a neighbor" // rather than failing. FallbackDegraded Fallback = iota // FallbackStrict serves the request locally (this node's cache→origin) when // the owner is down, accepting a transient duplicate cache entry rather than // chaining proxies. No second hop. FallbackStrict )
type Membership ¶
type Membership struct {
// contains filtered or unexported fields
}
Membership is the live cluster state for one node: the peer pool (an lb.Upstream that consistent-hash-shards a cache key to its owning peer, with reused health / failover / discovery), this node's identity, and the #7/#8 policy. It is built once from a Config and is safe for concurrent use. A non-clustered cadish never constructs one (zero cost when absent).
func New ¶
func New(cfg Config) (*Membership, error)
New builds a Membership from cfg. The peer pool is a shard-by-key lb.Upstream so a cache key routes to its owning peer; the same pool answers Owner() for #8 and backs the read-through PeerOrigin for #7. Background health probing / dns re-resolution start on Start(ctx). New itself does no network I/O beyond lb's initial resolution.
func (*Membership) Close ¶
func (m *Membership) Close()
Close is a no-op placeholder for symmetry (the peer pool stops with its Start context). Present so callers can `defer m.Close()` uniformly.
func (*Membership) Config ¶ added in v0.2.3
func (m *Membership) Config() Config
Config returns a copy of the membership's parsed Config, so callers (the server's startup reachability check) can re-run ValidatePeerReachable against the real listener ports without reaching into unexported state.
func (*Membership) Fallback ¶
func (m *Membership) Fallback() Fallback
Fallback reports the owner-mode fallback policy.
func (*Membership) IntendedOwner ¶
func (m *Membership) IntendedOwner(key string) (string, bool)
IntendedOwner returns the ring owner IGNORING health — the topological owner even when it is down. Used to detect "the owner is unavailable" so the caller can apply the strict-vs-degraded fallback.
func (*Membership) IsForwardedHop ¶
func (m *Membership) IsForwardedHop(h http.Header) bool
IsForwardedHop reports whether an inbound request was already forwarded to us by a peer in OUR region (the X-Cadish-Peer loop guard). Such a request must NOT be re-forwarded (read-through or owner-route) — it is served locally. A hop stamped by a DIFFERENT region is foreign (a separate cluster) and is treated as a fresh client request.
func (*Membership) IsSelf ¶
func (m *Membership) IsSelf(peerURL string) bool
IsSelf reports whether a peer base URL is this node.
func (*Membership) ListenAddr ¶ added in v0.2.3
func (m *Membership) ListenAddr() string
ListenAddr reports the dedicated peer-listener address (`cluster { listen … }`), or "" when none is configured (peers reach this node on the plain data listener). The server opens a plain-HTTP listener on this address, gated to peer traffic only.
func (*Membership) Mode ¶
func (m *Membership) Mode() Mode
Mode reports the configured coexistence mode.
func (*Membership) Owner ¶
func (m *Membership) Owner(key string) (string, bool)
Owner returns the base URL of the HEALTHY peer that owns key on the ring, and whether one exists. Walks past unhealthy/ejected peers (lb's health-aware ring walk), so the result is the node a sharded key currently lives on. Used by #8 to decide owner-vs-self.
func (*Membership) OwnsKey ¶
func (m *Membership) OwnsKey(key string) bool
OwnsKey reports whether THIS node is the (healthy) owner of key. When no healthy owner exists it returns false (the caller then applies fallback).
func (*Membership) PeerCount ¶
func (m *Membership) PeerCount() int
PeerCount reports the number of currently-known peer endpoints (post-resolution).
func (*Membership) PeerOrigin ¶
func (m *Membership) PeerOrigin() origin.Origin
PeerOrigin returns the read-through origin (#7) to compose BEFORE the real origin in a chain. Never nil.
func (*Membership) Region ¶
func (m *Membership) Region() string
Region reports the cluster region (the hop-header value).
func (*Membership) Self ¶
func (m *Membership) Self() string
Self reports this node's normalized peer URL.
func (*Membership) Start ¶
func (m *Membership) Start(ctx context.Context)
Start launches the peer pool's background workers (active health probing + dynamic re-resolution), bound to ctx. Idempotent.
func (*Membership) StorePeerReads ¶ added in v0.2.3
func (m *Membership) StorePeerReads() bool
StorePeerReads reports whether a peer-sourced read-through response (#7) should be TEED into this node's local cache. It is true for the default `peer_store local` (the historical replicate-locally behavior) and false for `peer_store none` (serve-through, keep the object once per region on its owning peer — CAD-52). A nil Membership (non-clustered) reports true so the plain, non-peer store path is unchanged.
type Mode ¶
type Mode int
Mode selects how #7 (read-through) and #8 (ownership routing) coexist.
const ( // ModeReadThrough (the default) is #7 ONLY: opportunistic peer read-through. // On a local miss, the owning peer is asked for the key (hop-guarded); a peer // hit is streamed-and-stored locally, a peer miss falls through to origin. // Requests are never re-routed — every node may serve any key. ModeReadThrough Mode = iota // ModeOwner is #8 PRIMARY with #7 as the fallback: each key has one owner on // the ring. A request landing on a non-owner is reverse-proxied to the owner // so the object is cached once per region. If the owner is down, Fallback // decides between serving locally (strict) or the next ring node (degraded); // the degraded path also tries peer read-through before origin. ModeOwner )
type PeerStore ¶ added in v0.2.3
type PeerStore int
PeerStore selects whether a peer-sourced read-through response (#7) is TEED into this node's local cache. It exists to resolve the read_through replication tradeoff (CAD-52): client load balancing spreads requests across nodes (not hash-by-key), so any node that serves key K after a peer read stores K locally — a popular key replicates toward every node and a region's effective cache capacity trends to 1× per node instead of "cached once per region".
const ( // PeerStoreLocal (the DEFAULT) tees a peer read-through hit into the local // tiers — the historical behavior. Every node that serves a key ends up holding // it: lowest latency on a repeat (a local HIT, no peer hop) at the cost of N× // storage for a hot key. Changing away from this default is a topology decision // the operator makes consciously; the knob only makes the alternative possible. PeerStoreLocal PeerStore = iota // PeerStoreNone serves a peer read-through response THROUGH to the client without // committing it locally — the owning peer keeps the only copy, so region cache // capacity stays ~1× (no replication). The cost is an extra peer hop on every // subsequent hit for that key on a non-owner node (it never becomes a local HIT). // It reuses the server's existing non-stored serve path (the same one a // non-cacheable response takes), so no in-flight cache writer is opened. PeerStoreNone )
Source Files