Documentation
¶
Overview ¶
Package cluster is the part of Wegweiser that several servers share: the transport members talk over (docs/decisions/d43-the-cluster-transport.md), and the Raft node that carries the replicated log across it and applies each entry to the store (docs/decisions/d24-what-the-cluster-replicates.md).
None of it runs unless a cluster is configured. A single node never listens on the cluster port and never dials anybody.
Index ¶
- Constants
- Variables
- func Identity(ctx context.Context, st store.Store, configured string) (string, error)
- func IsWitness(id string) bool
- type Config
- type JoinRequest
- type Loader
- type MemberState
- type Mux
- type Node
- func (n *Node) AddNonvoter(id, addr string) error
- func (n *Node) AddVoter(id, addr string) error
- func (n *Node) Bootstrap() error
- func (n *Node) Close() error
- func (n *Node) DialForward(ctx context.Context, addr string) (net.Conn, error)
- func (n *Node) Init(ctx context.Context) error
- func (n *Node) IsLeader() bool
- func (n *Node) Join(ctx context.Context, addr string, role Role) error
- func (n *Node) Leader() (id, addr string)
- func (n *Node) Left() error
- func (n *Node) Member() (id, addr string)
- func (n *Node) Propose(ctx context.Context, b *apply.Batch) error
- func (n *Node) Remove(id string) error
- func (n *Node) Replicating() bool
- func (n *Node) Settle(ctx context.Context) error
- func (n *Node) Stalled() (Stall, bool)
- func (n *Node) Status(ctx context.Context) (Status, error)
- type NodeConfig
- type Replication
- type Role
- type Stall
- type Status
- type StreamKind
- type Transport
Constants ¶
const ( // ALPN names the protocol and its version. A peer that names anything // else, or nothing at all, is refused. ALPN = "wegweiser-cluster/1" // ExporterLabel is the label the keying material a proof is bound to is // drawn under (RFC 5705). ExporterLabel = "EXPORTER-wegweiser-cluster-proof" )
The protocol's constants. They are the contract with every other implementation of it, wegwitness included, and D43 points here rather than repeating them.
const DefaultHandshakeTimeout = 10 * time.Second
DefaultHandshakeTimeout bounds how long a connection has to prove itself.
const MinSecret = 32
MinSecret is the shortest secret accepted, in octets: 256 bits. A member that dials an impostor hands it a proof, and a proof is an offline test of any guess at the secret.
const WitnessPrefix = "witness-"
WitnessPrefix begins the identifier of every witness, and of nothing else (docs/decisions/d48-a-witness-is-known-by-its-identifier.md). wegwitness holds the same constant and points at the same record.
Variables ¶
var ( // ErrSecretMismatch is a proof that does not verify: the other end holds // another secret, or none. ErrSecretMismatch = errors.New("cluster: the other end's proof does not match this member's secret") // ErrRefused is the other end hanging up rather than proving itself. A // member that does not accept a proof says nothing, so this is most often // a different secret; it is also what another version, or a stream the // other end does not take, looks like from here. ErrRefused = errors.New( "cluster: the other end hung up instead of proving itself; it holds another secret, " + "speaks another version, or does not take this stream") )
var ErrBehind = errors.New("cluster: this member has left the cluster and is behind it")
ErrBehind is a member that has stopped applying the log and left the cluster. What it holds is correct as of where it stopped, and the cluster may well be fine without it.
var ErrLastVoter = errors.New("cluster: that member is the cluster's only voter, and a cluster needs one")
ErrLastVoter is the removal of the only voter. Raft refuses it too: a cluster without a voter can neither lead nor commit anything again.
var ErrMember = errors.New("cluster: this node is a cluster member already")
ErrMember is a cluster started on a node that is in one already. A cluster is started once, by one node (docs/decisions/d44-starting-and-joining.md).
var ErrNoLeader = errors.New("cluster: the member asked knows of no leader")
ErrNoLeader is a member that could not say who leads.
var ErrNoMember = errors.New("cluster: the cluster has no member by that identifier")
ErrNoMember is a member named for removal that the cluster does not have.
var ErrNotLeader = errors.New("cluster: this member is not the leader")
ErrNotLeader is a proposal made to a member that is not leading. It is the cue to forward the write (docs/decisions/d40-a-write-reaches-the-leader.md).
var ErrRemoved = errors.New("cluster: this member has been taken out of the cluster")
ErrRemoved is a member that has been taken out of the cluster, by leaving or by being removed (docs/decisions/d46-a-member-that-has-left.md).
var ErrWitnessMajority = errors.New("cluster: witnesses have to stay fewer than half of the voters")
ErrWitnessMajority is a change after which witnesses would no longer be fewer than half of the voters (docs/decisions/d39-the-witness.md).
Functions ¶
func Identity ¶
Identity returns the identifier this node is a member by, minting one the first time it is asked (docs/decisions/d42-membership-lives-in-the-log.md).
configured is what the configuration file names, and may be empty. The first time, a named identifier is taken as it stands; after that the file is held to it. A member whose identifier changed would graft one member's history onto another's, so a file that disagrees with what the store holds is refused rather than obeyed.
Types ¶
type Config ¶
type Config struct {
// Secret is the cluster's shared secret, at least [MinSecret] octets.
Secret []byte
// HandshakeTimeout bounds TLS and the proof together. Zero picks
// [DefaultHandshakeTimeout].
HandshakeTimeout time.Duration
// OnRefused hears about a connection turned away, and why. A port that a
// network can reach is probed by strangers, so this is an observation
// rather than a fault, and the wiring decides whether to count it or log
// it. May be nil.
OnRefused func(remote net.Addr, err error)
}
Config is what a Transport needs.
type JoinRequest ¶
type JoinRequest struct {
// ID is the identifier the node is a member by (D42).
ID string `json:"id"`
// Address is where the other members reach it, as host:port.
Address string `json:"address"`
// Role is what it joins as.
Role Role `json:"role"`
// HoldsLog says the node has been sent the log and applied it. A voter
// asks once without it and is added without a vote, and asks again with
// it to be given one.
HoldsLog bool `json:"holdsLog,omitempty"`
}
JoinRequest is what a node sends to be made a member (docs/decisions/d44-starting-and-joining.md).
type Loader ¶
Loader copies the whole store into the query path. A restored log snapshot replaces the store wholesale, so what follows it is a load rather than a rebuild of the zones one batch named (docs/decisions/d41-what-follows-applying-a-batch.md). A *publish.Publisher is one.
type MemberState ¶
MemberState is one member, as this node's copy of the configuration lists it.
type Mux ¶
type Mux struct {
// contains filtered or unexported fields
}
Mux is the cluster port: one listener, sorted into one stream of connections per kind.
type Node ¶
type Node struct {
// contains filtered or unexported fields
}
Node is this server's member of a cluster.
func Start ¶
func Start(cfg NodeConfig) (_ *Node, err error)
Start brings the member up. A member that has never been part of a cluster holds no Raft state, and waits to be added by one that is, or to be told to start one with Node.Bootstrap.
func (*Node) AddNonvoter ¶
AddNonvoter makes the member at addr one that receives the log and does not vote (docs/decisions/d25-cluster-shape.md).
func (*Node) Bootstrap ¶
Bootstrap starts a cluster with this member as its only voter. It is done once, by one member, and refused by one that already holds Raft state (docs/decisions/d42-membership-lives-in-the-log.md).
func (*Node) Close ¶
Close stops taking part in the cluster. The query path is not touched: a member that stops is one that stops voting, not one that stops answering.
func (*Node) DialForward ¶
DialForward opens a stream to the member at addr that carries a write forwarded to it (docs/decisions/d40-a-write-reaches-the-leader.md).
func (*Node) Init ¶
Init makes this member the first of a new cluster, with everything its store holds (docs/decisions/d44-starting-and-joining.md).
What the store held until now is in no entry of the log, so a member that joins later could not replay it. Init therefore ends with a log snapshot and nothing of the log left in front of it, and every member that joins starts from that snapshot. Run it through apply.Applier.Exclusive, so that no write lands between the log beginning and the snapshot being taken.
func (*Node) Join ¶
Join asks the member at addr to make this node a member, follows it to the leader when it does not lead, and returns once this node is one and has applied the log it was sent.
A voter is added without a vote first, and asks for one once the log has reached it, so that a node the leader cannot reach never counts towards quorum (docs/decisions/d44-starting-and-joining.md, where it stands).
func (*Node) Leader ¶
Leader names the member leading, as far as this one knows. Both are empty while there is none.
func (*Node) Left ¶
Left says why this member no longer takes part in the cluster, and is nil while it does. It stopped over an entry it could not apply (docs/decisions/d29-a-node-that-cannot-apply.md), or it was taken out (docs/decisions/d46-a-member-that-has-left.md). Either way it answers queries with what it held and refuses writes with this error.
func (*Node) Member ¶
Member is the identifier this node is a member by, and the address the others reach it at.
func (*Node) Propose ¶
Propose puts a batch into the log and waits until this member has applied it. Only the leader takes one; anywhere else it fails with ErrNotLeader.
A member that has stalled refuses it with an error saying that this member is behind, rather than that the cluster is.
func (*Node) Remove ¶
Remove takes a member out of the cluster. Leaving is an act rather than an absence, and this is the act. A leader may remove itself, and stops taking part once the change is committed (docs/decisions/d46-a-member-that-has-left.md).
func (*Node) Replicating ¶
Replicating reports whether this member holds Raft state: it has started a cluster or been added to one. Until then it is an ordinary server (D44).
A member that has stalled counts as replicating, so that a write made there is refused as one made on a member that is behind, rather than landing in its store unseen by any other.
func (*Node) Settle ¶
Settle returns once this member may plan a write: it leads, and everything committed before now is applied here. A follower fails with ErrNotLeader before anything is planned (docs/decisions/d24-what-the-cluster-replicates.md).
func (*Node) Status ¶
Status reports what this member knows of the cluster and of itself.
How far it has got is Raft's count, which includes the entries Raft writes for itself and never hands over: one each time a leader is elected, and a barrier wherever a write waits for the log. The store's own count skips those, so a member at rest would read as a few entries behind for ever. A member that has stopped is the exception: Raft went on counting entries it refused, and the store says where it really is.
type NodeConfig ¶
type NodeConfig struct {
// ID identifies this member for as long as it is one
// (docs/decisions/d42-membership-lives-in-the-log.md).
ID string
// Advertise is where the other members reach this one, as host:port.
Advertise string
// Dir is where Raft keeps its log and its snapshots. The store is kept
// elsewhere; D24 says why Raft's log is not in it.
Dir string
Transport *Transport
Mux *Mux
Store store.Store
Applier *apply.Applier
Loader Loader
// Logger is where Raft's lines go. Nil discards them.
Logger *slog.Logger
// OnError hears about an entry this member could not carry out, and about
// it leaving the cluster over one. May be nil.
OnError func(error)
// contains filtered or unexported fields
}
NodeConfig is what a Node needs.
type Replication ¶
type Replication struct {
// contains filtered or unexported fields
}
Replication is the applier's way to this member (apply.Replication), bound once the member is up. The applier is built before the member, because the member applies the log through it, so the two meet here.
Until it is bound, and while the member holds no Raft state, the node is an ordinary server and writes to its own store (docs/decisions/d44-starting-and-joining.md).
func (*Replication) Bind ¶
func (r *Replication) Bind(n *Node)
Bind connects the member the applier's writes go through from now on.
func (*Replication) Propose ¶
Propose implements apply.Replication.
func (*Replication) Replicating ¶
func (r *Replication) Replicating() bool
Replicating implements apply.Replication.
func (*Replication) Settle ¶
func (r *Replication) Settle(ctx context.Context) error
Settle implements apply.Replication.
type Role ¶
type Role string
Role is what a node joins as (docs/decisions/d25-cluster-shape.md).
const ( // RoleVoter counts towards quorum and can lead. RoleVoter Role = "voter" // RoleNonvoter receives the whole log and answers queries like any member, // and neither votes nor is waited for. RoleNonvoter Role = "nonvoter" // RoleWitness votes and keeps the log, and applies none of it // (docs/decisions/d39-the-witness.md). Only wegwitness joins as one. RoleWitness Role = "witness" )
type Stall ¶
type Stall struct {
// Entry is the log position it could not get past. Zero is a log
// snapshot that could not be read far enough to say where it stood.
Entry uint64
// Reason is what went wrong there.
Reason error
// At is when this member gave up. It is this member's clock, and says
// nothing about the cluster, which is why it may be read here.
At time.Time
}
Stall is where a member stopped applying the log, and why.
type Status ¶
type Status struct {
// Members is empty on a member that has stopped taking part, whose copy
// of the configuration Raft no longer hands out.
Members []MemberState
// Applied is the last entry this member has carried out, and Committed
// the last one it knows to be committed. A member that keeps up has the
// two equal, or nearly.
Applied uint64
Committed uint64
// Stall is where this member stopped, if it has left the cluster over an
// entry it could not apply (docs/decisions/d29-a-node-that-cannot-apply.md).
Stall *Stall
// Removed says this member has been taken out of the cluster. Members is
// then who it last knew to be in it
// (docs/decisions/d46-a-member-that-has-left.md).
Removed bool
}
Status is what this member knows: who the members are, as far as its copy of the configuration says, and how far it has got through the log itself. How far the others have got is theirs to say, and each one is asked (docs/decisions/d42-membership-lives-in-the-log.md).
type StreamKind ¶
type StreamKind byte
StreamKind says what an authenticated stream carries.
const ( // StreamRaft is Raft's own traffic: votes, entries and snapshots. StreamRaft StreamKind = 'r' // StreamForward is a write forwarded to the leader // (docs/decisions/d40-a-write-reaches-the-leader.md). StreamForward StreamKind = 'f' // StreamJoin is a node asking to be made a member // (docs/decisions/d44-starting-and-joining.md). StreamJoin StreamKind = 'j' )
func (StreamKind) String ¶
func (k StreamKind) String() string
String names the kind for a message somebody reads.
type Transport ¶
type Transport struct {
// contains filtered or unexported fields
}
Transport opens and accepts connections between members of one cluster.
Source Files