placementprobe

package
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 4, 2026 License: Apache-2.0 Imports: 14 Imported by: 0

Documentation

Overview

Package placementprobe implements the multi-node campaign's placement probe (H5) and ballast/control writer (H6).

The probe lists every physical key of a Record Layer store's record subspace prefix+(1,) and index subspace prefix+(2,), the store's shard boundaries and the storage servers holding each key, as the fdb-bench-placement-probe/v3 document that tools/benchcampaign/multinode/placement_receipt.py judges. Store metadata (header (0,), index state and other subspaces (3..10,)) is out of scope by design and is never listed.

One snapshot. Everything the placement claim rests on is read in ONE transaction pinned at read version V1 with READ_SYSTEM_KEYS: the data-distribution mode, the moveKeysLock owner and write token, the \xff/keyServers entries of the store range (shard boundaries and team identities, compared as raw bytes, never decoded), every key of the two data subspaces, and every key's storage addresses. The production backend is libfdb_c (cmd/placementprobe, build tag libfdbc): its getAddressesForKey (NativeAPI.actor.cpp getAddressesForKeyActor) reads \xff/serverTag, \xff/keyServers and \xff/serverList transactionally, so addresses are those of the shard map at V1. The pure-Go client's LocalityGetAddressesForKey answers from its location cache instead (a divergence from libfdb_c); that backend is DIAGNOSTIC ONLY: its documents carry addresses_basis kind "location-cache (diagnostic)", the probe refuses it without --diagnostic, and placement_receipt.py fails such a document's integrity.

Stability over [V1, V2] is established by the moveKeysLock and the raw shard map, not by boundary identity (FoundationDB 7.3.77):

  • `datadistribution off|on` (ManagementAPI setDDMode; SpecialKeySpace DataDistributionImpl::commit) writes the mode, the lock owner (dataDistributionModeLock) AND a fresh random write token;
  • takeMoveKeysLock (MoveKeys.actor.cpp), which DataDistributor::init runs on every DD (re)start BEFORE it reads the mode, writes ONLY a random owner and leaves the write token unchanged; a healthy frozen cluster may therefore show any owner, and an owner-only change is caught solely by the owner-identity check;
  • the relocation paths startMoveKeys, finishMoveKeys, startMoveShards, finishMoveShards, removeStorageServer, removeKeysFromFailedServer and cleanUpDataMove{Background,Core} call checkMoveKeysLock in their transaction, which writes a new token (and, when it takes the lock from the previous owner, the owner too). Not every keyServers write is guarded: cleanUpSingleShardDataMove (MoveKeys.actor.cpp:630-703) writes it without checkMoveKeysLock, though only with SHARD_ENCODE_LOCATION_METADATA, which is off by default.

The probe requires the mode key present and 0 at V1 and V2, the owner present and identical at V1 and V2 (any value, recorded), the write token present and identical, and the raw keyServers bytes identical, and emits nothing otherwise: a toggle writes a new token, a DD restart writes a new owner, a guarded relocation commits a new token, and any shard-map write, guarded or not, changes the raw keyServers bytes.

The ballast tool writes only into [prefix+(1,1000000), strinc(prefix+(1,))) and only while data distribution is on; it clears only that range and only while it is off, reading \xff/dataDistributionMode inside the committing transaction so a concurrent mode change conflicts the commit. Every destructive operation takes an OwnedStore, whose prefix can only come from the store's fixture receipt (OpenStore) or the fixed scratch prefix (OpenControlStore) and is re-validated at every call.

Index

Constants

View Source 
const (
	ProbeFormat   = "fdb-bench-placement-probe/v3"
	ReceiptFormat = "fdb-bench-ballast-receipt/v2"
	Tool          = "placementprobe/v2"

	// AddressTransactional is the only address basis placement receipts accept.
	AddressTransactional = "transactional"
	// AddressLocationCache marks diagnostic documents from the pure-Go backend.
	AddressLocationCache = "location-cache (diagnostic)"

	RecordKey = 1
	IndexKey  = 2
	// BallastFirstID is the first record primary key the ballast uses; every
	// fixture ID is below it.
	BallastFirstID = 1_000_000
	// BallastValueBytes is the size of one ballast value.
	BallastValueBytes = 100_000
	// BallastKeysPerTransaction keeps each write transaction near 5 MB, below
	// FoundationDB's 10 MB transaction limit.
	BallastKeysPerTransaction = 50
	// ControlMutationID is the record the on-cluster mutation controls damage.
	ControlMutationID = 7
	// IndexName is the fixtures' VALUE index on price (= order ID).
	IndexName = "price_idx"
)

Formats and fixed layout constants. They mirror placement_receipt.py.

Variables

View Source 
var (
	// DDModeKey is dataDistributionModeKey: a BinaryWriter int (4 bytes, little
	// endian); absent means 1 (on).
	DDModeKey = []byte("\xff/dataDistributionMode")
	// MoveKeysLockOwnerKey and MoveKeysLockWriteKey hold BinaryWriter UIDs.
	MoveKeysLockOwnerKey = []byte("\xff/moveKeysLock/Owner")
	MoveKeysLockWriteKey = []byte("\xff/moveKeysLock/Write")
	// KeyServersPrefix is keyServersKeys.begin.
	KeyServersPrefix = []byte("\xff/keyServers/")
	// DDModeLock is dataDistributionModeLock = UID(6345, 3425), serialized as
	// two little-endian uint64 (UID::serialize_unversioned): the owner setDDMode
	// writes. Informational only: a DD restart replaces it with a random owner.
	DDModeLock = uidBytes(6345, 3425)
)

FoundationDB 7.3.77 system keys (fdbclient/SystemData.cpp).

View Source 
var ErrDDMode = errors.New("data-distribution mode forbids this operation")

ErrDDMode reports a data-distribution mode that forbids the operation.

Functions

func BallastKey 

func BallastKey(prefix []byte, i int) []byte

BallastKey is ballast entry i: prefix+(1, 1000000+i).

func BallastRange 

func BallastRange(prefix []byte) (begin, end []byte, err error)

BallastRange is [prefix+(1,1000000), strinc(prefix+(1,))).

func BallastValue 

func BallastValue(i int) []byte

BallastValue is ballast entry i's deterministic value.

func DDMode 

func DDMode(value []byte, present bool) (int32, error)

DDMode decodes a dataDistributionMode value; absent means 1 (on).

func InsideGitWorktree 

func InsideGitWorktree(path string) (bool, error)

InsideGitWorktree reports whether path (existing or not) lies inside a directory holding .git, after resolving symlinks of its parent directory.

func ResolvePrefix 

func ResolvePrefix(store string, r FixtureReceipt) ([]byte, error)

ResolvePrefix returns the owned store's key prefix, cross-checked against the fixture receipt. Unknown stores and any mismatch are refused.

func Stores 

func Stores() []string

Stores returns the owned store names, sorted.

func Subspaces 

func Subspaces(prefix []byte) (record, index []byte)

Subspaces returns the record and index data subspaces of prefix.

func WriteControlFixture 

func WriteControlFixture(c Cluster, s OwnedStore, rows int) (*BallastReceipt, FixtureReceipt, error)

WriteControlFixture writes a canonical fixture into an empty scratch control store and returns its fixture receipt for the probe.

func WriteJSONExclusive 

func WriteJSONExclusive(path string, v any) error

WriteJSONExclusive writes v as indented JSON to a new file outside any Git work tree; it never overwrites.

Types

type AddressBasis 

type AddressBasis struct {
	Kind        string `json:"kind"`
	ReadVersion int64  `json:"read_version"`
	Method      string `json:"method"`
}

AddressBasis records how the addresses were obtained.

type BallastReceipt 

type BallastReceipt struct {
	Format            string          `json:"format"`
	Tool              string          `json:"tool"`
	Action            string          `json:"action"`
	Store             string          `json:"store"`
	StorePrefixHex    string          `json:"store_prefix_hex"`
	Cluster           ClusterIdentity `json:"cluster"`
	RangeHex          [2]string       `json:"range_hex"`
	DDModeRequired    string          `json:"dd_mode_required"`
	Defect            string          `json:"defect,omitempty"`
	KeysWritten       int             `json:"keys_written,omitempty"`
	ValueBytes        int             `json:"value_bytes,omitempty"`
	TotalBytes        int64           `json:"total_bytes,omitempty"`
	Transactions      int             `json:"transactions,omitempty"`
	CommittedVersions []int64         `json:"committed_versions,omitempty"`
	ReadVersion       int64           `json:"read_version,omitempty"`
	KeysFound         *int            `json:"keys_found,omitempty"`
	Empty             *bool           `json:"empty,omitempty"`
}

BallastReceipt is the H6 receipt for every write, clear and verification.

func ClearBallast 

func ClearBallast(c Cluster, s OwnedStore) (*BallastReceipt, error)

ClearBallast clears exactly the ballast range, only while data distribution is off (mode key present and 0, read in the clearing transaction).

func ClearControl 

func ClearControl(c Cluster, s OwnedStore) (*BallastReceipt, error)

ClearControl clears a whole scratch control store and verifies it is empty.

func MutateControl 

func MutateControl(c Cluster, s OwnedStore, defect string) (*BallastReceipt, error)

MutateControl applies one defect to the split control store.

func VerifyBallastEmpty 

func VerifyBallastEmpty(c Cluster, s OwnedStore) (*BallastReceipt, error)

VerifyBallastEmpty reads the whole ballast range at a fresh read version. It returns the receipt even when the range is not empty (with an error).

func WriteBallast 

func WriteBallast(c Cluster, s OwnedStore, totalBytes int64) (*BallastReceipt, error)

WriteBallast writes ceil(totalBytes / 100000) ballast values into the ballast range of an owned benchmark store, only while data distribution is on and only into an empty ballast range.

type Cluster 

type Cluster interface {
	// ReadVersion returns a fresh read version.
	ReadVersion() (int64, error)
	// At opens one transaction at version.
	At(version int64) (Txn, error)
	// AddressBasis is AddressTransactional or AddressLocationCache.
	AddressBasis() string
	// Commit runs guard and then applies writes in one transaction, retrying
	// retryable errors; guard errors abort without retry. It returns the
	// committed version.
	Commit(guard func(read Reader) error, writes []Mutation) (int64, error)
}

Cluster is everything the probe and the ballast tool need from FoundationDB. Production backends: cmd/placementprobe backend_cgo.go (libfdb_c, transactional addresses) and GoCluster (pure Go, diagnostic addresses). Tests use a fake.

type ClusterIdentity 

type ClusterIdentity struct {
	ConnectionStringID string `json:"connection_string_id"`
	ClusterFileSHA256  string `json:"cluster_file_sha256"`
}

ClusterIdentity names the cluster a document was taken on.

func ParseClusterIdentity 

func ParseClusterIdentity(data []byte) (ClusterIdentity, error)

ParseClusterIdentity derives the identity from cluster file bytes: the SHA-256 of the file and the description:id of its connection string.

type DDModeRead 

type DDModeRead struct {
	ReadVersion int64 `json:"read_version"`
	Mode        int32 `json:"mode"`
	KeyPresent  bool  `json:"key_present"`
}

DDModeRead is one observation of the data-distribution mode.

type DDModes 

type DDModes struct {
	RequiredOff bool       `json:"required_off"`
	AtV1        DDModeRead `json:"at_v1"`
	AtV2        DDModeRead `json:"at_v2"`
}

DDModes records the data-distribution mode at V1 and V2.

type FixtureReceipt 

type FixtureReceipt struct {
	Phase          string `json:"phase"`
	Payload        string `json:"payload,omitempty"`
	PrefixBytes    *int   `json:"prefix_bytes"`
	Rows           *int   `json:"rows"`
	Chunks         *int   `json:"chunks,omitempty"`
	StorePrefixHex string `json:"store_prefix_hex,omitempty"`
	Source         string `json:"source,omitempty"`
}

FixtureReceipt is the benchmark's untimed fixture receipt (the JSON after FDB_BENCH_AUDIT with phase "fixture"), or the H6 control equivalent.

func ParseFixtureReceipt 

func ParseFixtureReceipt(data []byte, store string) (FixtureReceipt, error)

ParseFixtureReceipt accepts either one JSON object or a benchmark log, from which it takes the FDB_BENCH_AUDIT lines with phase "fixture" (and, for stores that have one, the matching payload). Exactly one distinct receipt must remain; the per-benchmark "case" field is ignored for that comparison.

type GoCluster 

type GoCluster struct {
	DB fdb.Database
	// TimeoutMillis bounds each transaction (FDB transaction timeout option).
	TimeoutMillis int64
	// MaxAttempts bounds Commit retries.
	MaxAttempts int
}

GoCluster implements Cluster over the pure-Go client. Its LocalityGetAddressesForKey answers from the client location cache, unlike libfdb_c's transactional getAddressesForKey, so its address basis is diagnostic: probes from it are refused without --diagnostic and never pass placement_receipt integrity. Ballast and control operations are exact.

func (GoCluster) AddressBasis 

func (c GoCluster) AddressBasis() string

AddressBasis implements Cluster.

func (GoCluster) At 

func (c GoCluster) At(version int64) (Txn, error)

At implements Cluster.

func (GoCluster) Commit 

func (c GoCluster) Commit(guard func(read Reader) error, writes []Mutation) (int64, error)

Commit implements Cluster with an explicit OnError loop so the committed version of the successful attempt is available.

func (GoCluster) ReadVersion 

func (c GoCluster) ReadVersion() (int64, error)

ReadVersion implements Cluster.

type KV 

type KV struct {
	Key, Value []byte
}

KV is one key-value pair.

type KeyServers 

type KeyServers struct {
	BeginHex    string `json:"begin_hex"`
	EndHex      string `json:"end_hex"`
	Entries     int    `json:"entries"`
	SHA256AtV1  string `json:"sha256_at_v1"`
	SHA256AtV2  string `json:"sha256_at_v2"`
	Unchanged   bool   `json:"unchanged"`
	Description string `json:"description"`
}

KeyServers records the raw \xff/keyServers entries covering the store range: from the last entry <= keyServers+prefix to keyServers+strinc(prefix).

type MoveKeysLock 

type MoveKeysLock struct {
	OwnerHexAtV1        string `json:"owner_hex_at_v1"`
	OwnerHexAtV2        string `json:"owner_hex_at_v2"`
	OwnerUnchanged      bool   `json:"owner_unchanged"`
	WriteTokenHexAtV1   string `json:"write_token_hex_at_v1"`
	WriteTokenHexAtV2   string `json:"write_token_hex_at_v2"`
	WriteTokenUnchanged bool   `json:"write_token_unchanged"`
}

MoveKeysLock records the moveKeysLock owner and write token at V1 and V2.

type Mutation 

type Mutation struct {
	Op    Op
	Key   []byte
	End   []byte // OpClearRange only
	Value []byte // OpSet only
}

Mutation is one write applied by Cluster.Commit.

func ControlKeys 

func ControlKeys(s OwnedStore, rows int) ([]Mutation, int, error)

ControlKeys returns the canonical fixture keys of a control store: index prefix+(2,"price_idx",i,i) and records prefix+(1,i,0) (small) or prefix+(1,i,1), prefix+(1,i,2) (split).

type Op 

type Op int

Op is a mutation kind. 

const (
	OpSet Op = iota
	OpClear
	OpClearRange
)

Mutation kinds.

type OwnedStore 

type OwnedStore struct {
	// contains filtered or unexported fields
}

OwnedStore is an owned store with its validated prefix. Its fields are unexported: outside this package it can only be obtained from OpenStore or OpenControlStore, and every operation re-validates it (check).

func OpenControlStore 

func OpenControlStore(store string) (OwnedStore, error)

OpenControlStore returns a scratch control store with its fixed tuple prefix. Control stores have no benchmark fixture receipt until control-write emits one.

func OpenStore 

func OpenStore(store string, r FixtureReceipt) (OwnedStore, error)

OpenStore resolves a store's prefix from its fixture receipt.

func (OwnedStore) Name 

func (s OwnedStore) Name() string

Name returns the store name.

func (OwnedStore) Prefix 

func (s OwnedStore) Prefix() []byte

Prefix returns a copy of the store prefix.

type ProbeDoc 

type ProbeDoc struct {
	Format                string          `json:"format"`
	Tool                  string          `json:"tool"`
	Store                 string          `json:"store"`
	StorePrefixHex        string          `json:"store_prefix_hex"`
	FixtureReceiptSHA256  string          `json:"fixture_receipt_sha256"`
	Cluster               ClusterIdentity `json:"cluster"`
	ReadVersion           int64           `json:"read_version"`
	BoundariesReadVersion int64           `json:"boundaries_read_version"`
	BoundaryRangeHex      [2]string       `json:"boundary_range_hex"`
	BoundariesComplete    bool            `json:"boundaries_complete"`
	BoundariesHex         []string        `json:"boundaries_hex"`
	ScannedSubspacesHex   [2]string       `json:"scanned_subspaces_hex"`
	KeysComplete          bool            `json:"keys_complete"`
	Keys                  []ProbeKey      `json:"keys"`
	AddressesBasis        AddressBasis    `json:"addresses_basis"`
	Recheck               Recheck         `json:"recheck"`
	DDMode                DDModes         `json:"dd_mode"`
	MoveKeysLock          MoveKeysLock    `json:"move_keys_lock"`
	KeyServers            KeyServers      `json:"key_servers"`
}

ProbeDoc is the fdb-bench-placement-probe/v3 document.

func Probe 

func Probe(c Cluster, in ProbeInput) (*ProbeDoc, error)

Probe builds the placement probe document from one snapshot at V1 and a recheck snapshot at V2 >= V1. It never judges placement; the receipt does. It emits nothing when the recheck differs (lock owner, write token, keyServers bytes, boundaries), when data distribution is not off at V1 and V2, or when the backend's addresses are not transactional (unless Diagnostic).

type ProbeInput 

type ProbeInput struct {
	Store         OwnedStore
	FixtureSHA256 string
	Cluster       ClusterIdentity
	// Diagnostic permits a location-cache address source and data
	// distribution on; such documents always fail placement_receipt integrity.
	Diagnostic bool
}

ProbeInput selects the store to probe.

type ProbeKey 

type ProbeKey struct {
	Kind      string   `json:"kind"`
	ID        *int64   `json:"id"`
	KeyHex    string   `json:"key_hex"`
	Addresses []string `json:"addresses"`
}

ProbeKey is one physical key with its storage addresses.

type Reader 

type Reader func(key []byte) (value []byte, present bool, err error)

Reader reads a key inside the committing transaction (non-snapshot, so the key joins the read-conflict set; system keys are readable).

type Recheck 

type Recheck struct {
	ReadVersion         int64 `json:"read_version"`
	BoundariesIdentical bool  `json:"boundaries_identical"`
}

Recheck records the second snapshot.

type Txn 

type Txn interface {
	// Get reads one key.
	Get(key []byte) (value []byte, present bool, err error)
	// Range returns every pair of [begin, end), without a limit. With
	// fromLastLessOrEqual it starts at the last key <= begin instead (the
	// keyServers entry covering begin).
	Range(begin, end []byte, fromLastLessOrEqual bool) ([]KV, error)
	// Addresses returns each key's storage addresses (LocalityGetAddressesForKey
	// in this transaction), in the order of keys.
	Addresses(keys [][]byte) ([][]string, error)
}

Txn is ONE backend transaction pinned at a read version with READ_SYSTEM_KEYS. Every method reads at that version.

Source Files

View all Source files

Directories

Path Synopsis
cmd
placementprobe command
Command placementprobe is the multi-node campaign's placement probe (H5) and ballast/control writer (H6).
 Command placementprobe is the multi-node campaign's placement probe (H5) and ballast/control writer (H6).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.