Documentation
¶
Overview ¶
Package fsmwire defines the §6.3 / §11 binary wire format for the FSM-internal encryption Raft entry types (opcodes 0x03 / 0x04 / 0x05). The encoding is hand-rolled binary with a leading version byte rather than proto3 because:
Each replica's apply path must reproduce the bytes exactly to keep state machine determinism. proto3's lenient unknown-field handling would let a future leader-built field silently drop on a stale follower, leaving registry rows or active-pointer entries missing on that follower — exactly the silent divergence the encryption tag was added to detect.
The schema is small (~30 lines of encode/decode per opcode), so the maintenance cost of hand-rolled binary is negligible while keeping every state-machine-affecting bit explicit.
It matches the in-house style: internal/encryption/envelope.go, raft_envelope.go, and kv/raft_payload_wrapper.go all hand-roll binary with a leading version byte.
Version byte. Every payload starts with WireVersionV1 (0x01). Future schema changes bump the version and the decoder fails closed (ErrFSMWireVersion) — operators upgrade clusters fully before flipping flags that produce the new format. There is no "skip unknown fields" path because skipping a field on the apply path is the divergence we are guarding against.
Index ¶
Constants ¶
const ( OpRegistration byte = 0x03 OpBootstrap byte = 0x04 OpRotation byte = 0x05 )
Opcode tags consumed by kv/fsm.go's Apply dispatch. Reserved per design §11.3:
0x00 raftEncodeSingle (kv legacy) 0x01 raftEncodeBatch (kv legacy) 0x02 raftEncodeHLCLease (kv HLC ceiling) 0x03 OpRegistration (writer-registry register, §4.1) 0x04 OpBootstrap (initial DEK install, §5.6) 0x05 OpRotation (DEK rotation / cluster flag flip, §5.2 + §7.1)
const ( // RotateSubRotateDEK is a fresh DEK install + active-pointer // update. Ships in Stage 4. See §5.2. RotateSubRotateDEK byte = 0x01 // RotateSubEnableStorageEnvelope is the one-shot storage-layer // cutover (§2.2 / §7.1 Phase 1). Ships in Stage 6D. The wire // shape reuses the rotate-dek triple but with `Wrapped` empty // (`len(Wrapped) == 0`) and `DEKID == sidecar.Active.Storage`; // the applier re-validates both at apply time. The byte value // `0x04` matches the original `0x02..0x0F` reservation block // so older binaries that pre-date Stage 6D continue to halt on // the existing "unknown sub-tag" branch. RotateSubEnableStorageEnvelope byte = 0x04 // RotateSubEnableRaftEnvelope is the one-shot raft-layer cutover // (§7.1 Phase 2). Ships in Stage 6E-1. The wire shape mirrors // RotateSubEnableStorageEnvelope but with the raft slot: // `Wrapped` empty (`len(Wrapped) == 0`), `DEKID == // sidecar.Active.Raft`, `Purpose == PurposeRaft`. The applier // re-validates all three at apply time. On a fresh-success // apply the sidecar's `RaftEnvelopeCutoverIndex` is set to // `raftIdx` (the cutover entry's own apply index, which Stage // 6E-2's engine apply-hook compares against `entry.Index` via // strict `>` to decide unwrap-or-not). RotateSubEnableRaftEnvelope byte = 0x05 )
Sub-tags used by OpRotation. Encoded as the first byte after the version byte.
const ( OpEncryptionMin byte = 0x03 OpEncryptionMax byte = 0x07 )
OpEncryptionMin / OpEncryptionMax delimit the FSM-internal opcode range RESERVED for the encryption subsystem. kv/fsm.go's Apply dispatcher routes EVERY byte in the closed range [OpEncryptionMin, OpEncryptionMax] through applyEncryption, which fails closed via ErrEncryptionApply for any byte that is not yet implemented (Stage 4 implements only 0x03/0x04/0x05; 0x06/0x07 are reserved for later stages). The widened range is the codex P1 fix for PR748: previously only the three implemented opcodes were routed, so a future leader emitting 0x06 against a stale follower would fall through to the legacy proto3 decoder rather than halting the apply loop — the same divergence shape the §6.3 fail-closed halt was added to prevent.
Upper bound 0x07 (NOT 0x0F): proto3 wire tags for field 1 occupy 0x08..0x0D (field 1 with wire types varint/fixed64/length-delim/ start-group/end-group/fixed32). Routing those through the encryption dispatcher would short-circuit the legacy proto3 fallback in `decodeLegacyRaftRequest` for any RaftCommand/Request payload whose first encoded field is field 1 (e.g. `Request.is_txn` = true → first bytes `0x08 0x01`). Bytes 0x03..0x07 are SAFE because they encode either field 0 (proto3 disallows field 0) or reserved/invalid wire types (0x06/0x07 = wire types 6/7 which proto3 marks reserved), so no valid proto3 marshal output starts with them. Future encryption opcodes 0x08+ would need a different dispatch shape (e.g. a 2-byte sentinel) before they could be routed safely.
const WireVersionV1 byte = 0x01
WireVersionV1 is the current FSM-wire version. Carried as the second byte of every payload (after the opcode tag, which is stripped before fsmwire sees the payload).
Variables ¶
var ( // ErrFSMWireMalformed indicates the payload bytes do not // match the fixed-shape decoder for the supplied opcode. // Length mismatch, missing length-prefixed body, or trailing // garbage all surface here. ErrFSMWireMalformed = errors.New("fsmwire: payload is malformed") // ErrFSMWireVersion indicates the payload version byte is // not WireVersionV1. The decoder fails closed rather than // trying to interpret an unknown version under the v1 // layout. ErrFSMWireVersion = errors.New("fsmwire: unknown wire version") // ErrFSMWireSubtag indicates the rotation opcode (0x05) // carries an unknown sub-tag. Reserved for forward-compat: // future opcodes that share the rotation tag (rewrap-deks, // retire-dek, enable-storage-envelope, enable-raft-envelope) // will allocate distinct sub-tags. ErrFSMWireSubtag = errors.New("fsmwire: unknown rotation sub-tag") )
Errors returned by fsmwire's encoders/decoders. All three are terminal in the apply path — kv/fsm.go marks them with ErrEncryptionApply so internal/raftengine/etcd halts the apply loop instead of silently advancing setApplied.
Functions ¶
func EncodeBootstrap ¶
func EncodeBootstrap(p BootstrapPayload) []byte
EncodeBootstrap serialises p as the OpBootstrap payload (no leading opcode tag).
Wire layout:
[ver 1] [storage_dek_id 4] [storage_wrapped_len 4] [storage_wrapped ...] [raft_dek_id 4] [raft_wrapped_len 4] [raft_wrapped ...] [batch_count 4] [registration_payload]*batch_count
The two wrapped-DEK length prefixes are independent; the concrete length is whatever the configured KEK produced (the reference FileWrapper produces 60 bytes per 32-byte DEK).
func EncodeRegistration ¶
func EncodeRegistration(p RegistrationPayload) []byte
EncodeRegistration serialises p as the OpRegistration payload (without the leading 0x03 opcode tag — that is added by the kv/fsm.go dispatch layer).
Wire layout:
[ver 1] [dek_id 4 BE] [full_node_id 8 BE] [local_epoch 2 BE]
func EncodeRotation ¶
func EncodeRotation(p RotationPayload) []byte
EncodeRotation serialises p as the OpRotation payload (no leading opcode tag).
Wire layout (RotateSubRotateDEK):
[ver 1] [subtag 1] [dek_id 4] [purpose 1] [wrapped_len 4] [wrapped ...] [proposer_dek_id 4] [proposer_full_node_id 8] [proposer_local_epoch 2]
Types ¶
type BootstrapPayload ¶
type BootstrapPayload struct {
StorageDEKID uint32
WrappedStorage []byte
RaftDEKID uint32
WrappedRaft []byte
BatchRegistry []RegistrationPayload
}
BootstrapPayload is the OpBootstrap body: §5.6 step 1a's "install the initial wrapped DEK pair plus a batch of writer-registry rows for every member that passed the capability pre-check".
The wrapped DEKs are KEK-output blobs the leader produced; the FSM apply layer installs them into the local keystore via Keystore.Set after the KEK Unwrap.
func DecodeBootstrap ¶
func DecodeBootstrap(raw []byte) (BootstrapPayload, error)
DecodeBootstrap reverses EncodeBootstrap. Fails closed on any length-prefix overrun or version mismatch.
type Purpose ¶
type Purpose byte
Purpose is the §5.1 sidecar purpose tag. The Cipher itself does not enforce purpose — that contract is maintained by the sidecar loader and the FSM apply handlers in kv/fsm_encryption.go. The fsmwire codec carries the purpose alongside each DEK so the applier can install it under the correct sidecar slot.
type RegistrationPayload ¶
RegistrationPayload is the OpRegistration body: a single (dek_id, full_node_id, local_epoch) triple inserted by FSM apply into the §4.1 writer registry. The kv/fsm_encryption.go handler decides between case 1 (insert), 2 (re-register / monotonic bump), 3 (rollback → ErrLocalEpochRollback) and 4 (collision → ErrNodeIDCollision).
func DecodeRegistration ¶
func DecodeRegistration(raw []byte) (RegistrationPayload, error)
DecodeRegistration reverses EncodeRegistration. Fails closed on length mismatch or unknown version byte.
type RotationPayload ¶
type RotationPayload struct {
SubTag byte
DEKID uint32
Purpose Purpose
Wrapped []byte
ProposerRegistration RegistrationPayload
}
RotationPayload is the OpRotation body. SubTag selects between rotate-dek / rewrap / enable-flag variants. Stage 4 ships RotateSubRotateDEK; Stage 6D adds RotateSubEnableStorageEnvelope.
For RotateSubRotateDEK the payload is:
- DEKID: the new key id being installed
- Purpose: which sidecar slot to update (PurposeStorage / PurposeRaft)
- Wrapped: the KEK-wrapped DEK bytes
- ProposerRegistration: the proposing node's writer-registry row, inserted in the same FSM apply transaction so the proposer's first encrypted write under the new DEK is covered by the §4.1 case 2 epoch monotonicity check.
For RotateSubEnableStorageEnvelope the payload reuses the same triple but with stricter constraints (validated in both the cutover RPC mutator on the propose side and `ApplyRotation` on the apply side, per the 6D design's §2.1):
- DEKID: MUST equal sidecar.Active.Storage at apply time
- Purpose: MUST be PurposeStorage
- Wrapped: MUST be empty (`len(Wrapped) == 0`, NOT nil — the wire decoder materialises zero-length payloads as an allocated empty slice)
- ProposerRegistration: covers the proposer's first write under the now-active envelope
func DecodeRotation ¶
func DecodeRotation(raw []byte) (RotationPayload, error)
DecodeRotation reverses EncodeRotation. Fails closed on length prefix overrun, version mismatch, or unknown sub-tag.