integration/

NeoHA Integration Tests

Integration tests live here and are not run by default make test or GitHub Actions CI. They use the Go integration build tag and require a local MySQL 8.0 build (plus optional Xtrabackup / SSH).

Why not CI? Installing upstream MySQL/Xtrabackup RPMs and warming datadirs adds many minutes per run; unit tests + coverage in CI are sufficient for merge gates. Run IT locally before releases or after HA-related changes. See docs/operations.md for drill procedures aligned with these tests.

Design

Layer	Responsibility
Harness (thin)	Process lifecycle only: init datadir, my.cnf, start/stop mysqld, start/stop neoha binary, ports, cleanup
neorpc / neohactl	Cluster ops via neohactl binary (same RPC as production); assertions via read-only SQL
Read-only SQL	Assert MGR / semi-sync state after NeoHA has configured MySQL

NeoHA behaviour (repl user, MGR bootstrap, Raft election) is exercised through agent startup + neorpc, not duplicate SQL in the harness.

Raft timers in integration tests

Mode	heartbeat-timeout	admit-defeat-hearbeat-count	Primary fault detection
Semi-sync	2000 ms (2 s)	5	~10 s (5 consecutive failed heartbeats)
MGR	2000 ms (2 s)	5	~10 s (same as semi-sync; MGR join is separate)

Semi-sync and MGR IT share Xenon-style Raft timing (applySemiSyncRaftIT / applyMGRRaftIT in harness/neoha.go). MGR cluster formation and START GROUP_REPLICATION may still take 20s+ per node; see ha-failover.md.

Constants live in harness/neoha.go (semiSyncIT*).

Prerequisites

• MySQL 8.0 debug build with group_replication plugin (MGR tests) and semi-sync plugins (semisync_master.so, semisync_slave.so)
• Xtrabackup 8.0 (xtrabackup, xbstream) for backup tests
• SSH to localhost (key-based or sshpass + password) for xtrabackup stream

Tool paths (env → IT config file → PATH → OS default)

Copy test/integration/it.local.yaml.example to test/integration/it.local.yaml, or export env vars. Priority per field: environment variable → it.local.yaml → PATH → Ubuntu 22.04 default install path.

Setting	Env var	IT config key	Ubuntu 22.04 default
MySQL base	NEOHA_IT_MYSQL_BASE	mysql-base	/usr/local/mysql (tarball) or /usr via mysqld in PATH (/usr/sbin/mysqld)
PostgreSQL base	NEOHA_IT_PG_BASE	pg-base	/usr/lib/postgresql/14
etcd binary	NEOHA_ETCD_BIN	etcd-bin	/usr/bin/etcd
Xtrabackup bin dir	NEOHA_IT_XTRABACKUP_BINDIR	xtrabackup-bindir	/usr/bin (xtrabackup, xbstream)
IT config file	NEOHA_IT_CONFIG	—	auto: test/integration/it.local.yaml
Work dir	NEOHA_IT_WORKDIR	workdir	$TMPDIR/neoha-it
SSH port	NEOHA_IT_SSH_PORT	ssh-port	22

Datadir, my.cnf, and etcd data dirs are always created under the IT workdir or /tmp — not under system package paths.

Optional: symlink dev builds to defaults so IT works without it.local.yaml:

sudo ln -sfn /home/wslu/work/mysql/mysql80-debug /usr/local/mysql
sudo ln -sfn /home/wslu/work/pg/pgsql /usr/local/pgsql
sudo ln -sf /home/wslu/work/github/db/etcd/etcd/bin/etcd /usr/bin/etcd
sudo ln -sf /home/wslu/work/mysql/xtrabackup-8.0.35/bin/xtrabackup /usr/bin/xtrabackup
sudo ln -sf /home/wslu/work/mysql/xtrabackup-8.0.35/bin/xbstream /usr/bin/xbstream

cp test/integration/it.local.yaml.example test/integration/it.local.yaml
# edit paths only if not using defaults or symlinks above

Run

Quick start

# 1. Local config (gitignored)
cp test/integration/it.local.yaml.example test/integration/it.local.yaml
# Edit mysql-base, xtrabackup-bindir, ssh-port (WSL often 2222), ssh-user if needed

# 2. Full integration suite (~6–8 min; global timeout 12m via IT_TIMEOUT)
make test-integration

# One or more tests (comma- or space-separated names match go test -run regex)
make test-integration TestNeoHASemiSyncWarmSuite
make test-integration TestNeoHASemiSyncWarmSuite,TestNeoHAMGRWarmSuite
make test-integration TestNeoHAMGRWarmSuite TestNeoHAMGRFailoverMajorityLoss

# Same via TESTS= (useful in scripts)
make test-integration TESTS=TestNeoHASemiSyncWarmSuite,TestNeoHAMGRWarmSuite

# Subtest (regex)
make test-integration 'TestNeoHAMGRFailoverMajorityLoss/RejoinThenWritable'

make test-integration pre-builds bin/neoha-it.test, then runs all tests with a 12-minute global timeout (-test.timeout=12m, override with IT_TIMEOUT=8m make test-integration). MGR warm formation fails fast at 2 minutes if 3 nodes are not ONLINE. It passes through NEOHA_IT_MYSQL_BASE and NEOHA_IT_XTRABACKUP_BINDIR when set; otherwise it uses the defaults shown in the Makefile.

Pre-build neoha / neohactl to avoid implicit compile during IT (also done automatically by make test-integration):

make build
# or rely on defaults: ./bin/neoha and ./bin/neohactl when present
make test-integration

make test-integration sets NEOHA_IT_BIN / NEOHA_IT_CTL_BIN to $(pwd)/bin/neoha and bin/neohactl when unset. The harness also picks up ./bin/* from the repo root when env vars are empty.

One-time datadir prep (warm clusters)

Cold MySQL datadir init is the slowest part of the first IT run (~20–30s per node). Semi-sync warm tests reuse datadirs across runs when ibdata1 already exists; MGR warm tests always reset datadirs (group replication state is persisted on disk).

make test-integration-prep   # optional: pre-init semi-sync datadirs only
make test-integration

make test-integration-prep creates datadirs under $NEOHA_IT_WORKDIR for:

Cluster name	MySQL ports	Mode	Datadir reuse
neoha-mgr-warm	13306–13308	MGR	No — reset every run (SetupReset)
neoha-semisync-warm	13316–13318	semi-sync	Yes — stop mysqld, keep workdir

By default, warm tests stop mysqld but keep workdirs (semi-sync skips init on the next run; MGR still re-inits because SetupReset clears datadirs). To remove artifacts after a run:

NEOHA_IT_TEARDOWN=1 make test-integration
# or
rm -rf "${NEOHA_IT_WORKDIR:-/tmp/neoha-it}/neoha-semisync-warm"

Environment variables

Variable	Purpose
NEOHA_IT_MYSQL_BASE	MySQL 8.0 build root (bin/mysqld)
NEOHA_IT_XTRABACKUP_BINDIR	Directory with xtrabackup and xbstream
NEOHA_IT_CONFIG	Override IT config file path
NEOHA_IT_WORKDIR	Artifact directory (default $TMPDIR/neoha-it)
NEOHA_IT_BIN / NEOHA_IT_CTL_BIN	Pre-built neoha / neohactl (default: ./bin/neoha, ./bin/neohactl from make build)
NEOHA_IT_KEEP_WORKDIR	1 (default): stop processes but retain datadirs; 0 removes workdir after warm tests
NEOHA_IT_TEARDOWN	1: always remove cluster workdir after warm tests (overrides keep)
NEOHA_IT_SSH_PORT	SSH port when not using it.local.yaml

Example:

export NEOHA_IT_MYSQL_BASE=/home/wslu/work/mysql/mysql80-debug
export NEOHA_IT_XTRABACKUP_BINDIR=/home/wslu/work/mysql/xtrabackup-8.0.35
export NEOHA_IT_WORKDIR=/tmp/neoha-it
make test-integration

go test (single test or custom flags)

Build tag integration is required.

# All integration tests (match Makefile global timeout)
go test -tags=integration -v -timeout=12m -count=1 ./test/integration

# MGR warm cluster (formation + minority failover subtests)
go test -tags=integration -v -timeout=12m -count=1 -run TestNeoHAMGRWarmSuite ./test/integration

# MGR majority loss (sole survivor + rejoin)
go test -tags=integration -v -timeout=12m -count=1 -run TestNeoHAMGRFailoverMajorityLoss ./test/integration

# Semi-sync warm cluster
go test -tags=integration -v -timeout=12m -count=1 -run TestNeoHASemiSyncWarmSuite ./test/integration

# Xtrabackup rebuildme (requires SSH + xtrabackup; see below)
go test -tags=integration -v -timeout=12m -count=1 -run TestNeoHAXtrabackupRebuildMe ./test/integration

Pre-compile to avoid a long silent compile phase:

go test -tags=integration -c -o bin/neoha-it.test ./test/integration
NEOHA_IT_MYSQL_BASE=... NEOHA_IT_XTRABACKUP_BINDIR=... \
  NEOHA_IT_BIN=$PWD/bin/neoha NEOHA_IT_CTL_BIN=$PWD/bin/neohactl \
  bin/neoha-it.test -test.v -test.timeout=15m -test.count=1

Xtrabackup / rebuildme test

TestNeoHAXtrabackupRebuildMe exercises the production path:

1. Two-node semi-sync cluster (MySQL 13326/13327, NeoHA Raft 18131/18132)
2. Data sync to replica, then neohactl mysql rebuildme --from=<leader> --force on the follower
3. Assert replication and row data after rebuild

Extra prerequisites:

• xtrabackup and xbstream on PATH or under xtrabackup-bindir
• SSH to localhost for the xtrabackup | ssh … xbstream pipeline (key-based or sshpass)

WSL example (it.local.yaml):

ssh-host: 127.0.0.1
ssh-port: 2222
ssh-user: ""          # defaults to $USER
ssh-passwd: ""        # empty = key auth

Verify SSH before running:

ssh -p 2222 -o BatchMode=yes $(whoami)@127.0.0.1 echo ok

Run only the rebuildme test:

go test -tags=integration -v -timeout=12m -count=1 \
  -run TestNeoHAXtrabackupRebuildMe ./test/integration

Ports used by tests

Scenario	MySQL ports	GR ports	NeoHA Raft ports
MySQL scaffold (TestMySQL3NodeScaffold)	13346–13348	13401–13403	—
MGR warm (TestNeoHAMGRWarmSuite)	13306–13308	13361–13363	18081–18083
MGR majority loss (TestNeoHAMGRFailoverMajorityLoss)	13326–13328	13381–13383	18101–18103
Semi-sync warm	13316–13318	—	18111–18113
Xtrabackup rebuildme	13336–13337	—	18141–18142
PostgreSQL	15432–15433	—	etcd / agent per test

Majority-loss and xtrabackup use disjoint MySQL ports. TestMain kills stale processes under $NEOHA_IT_WORKDIR at start and end; warm suites also call FreePorts before boot.

First run vs later runs

The harness initializes MySQL datadirs on first use (roughly 20–30s per node, parallelized across nodes).

Suite	Datadir behavior
Semi-sync warm	Reuses existing datadirs under $NEOHA_IT_WORKDIR/<cluster-name>/ when ibdata1 is present
MGR warm / majority-loss	Always resets datadirs (SetupReset) — MGR group state on disk is unsafe to reuse after failover subtests

IT my.cnf puts socket and pid-file inside datadir so rebuildme’s datadir wipe does not leave stale socket files.

To force a clean slate:

NEOHA_IT_TEARDOWN=1 make test-integration
# or manually:
rm -rf "${NEOHA_IT_WORKDIR:-/tmp/neoha-it}"
pkill -f 'defaults-file=/tmp/neoha-it' || true

Failover segment baseline

Warm suite logs distinguish end-to-end test time from the failover segment (fault injection → new primary ready):

Log prefix	Meaning	Typical range (warm, WSL)
timing: init datadirs	mysqld datadir + my.cnf	semi-sync: 0s (reuse) or ~30–90s (cold); MGR: always cold (~30–90s)
timing: mysqld ready	3× mysqld up	~5–15s
timing: neoha ready	NeoHA agents + CLI wire	~10–30s
timing: MGR 3 online / timing: semi-sync formation	Cluster healthy before fault	~15–60s
timing: failover segment	Stop old primary → new primary + replication	semi-sync ~10–50s; MGR ~30–78s
timing: sole-survivor bootstrap	MGR majority-loss phase 1	~17–50s
timing: rejoin then writable	MGR majority-loss phase 2	~5–30s

Semi-sync target after warm formation: ~10–15s failover segment (2s heartbeat × 5 admit-defeat). MGR IT uses the same Raft timers; formation/rejoin latency is dominated by START GROUP_REPLICATION (often 20s+ per node).

See also docs/TODO.md for full-suite timing (~6–8 min with warm datadirs + pre-built bins).

Scenarios

Test	What it verifies
TestMySQL3NodeScaffold	3 mysqld instances start with MGR plugin loaded
TestNeoHAMGRWarmSuite	Warm 3-node MGR: formation + minority failover (failover subtest times only the fault segment)
TestNeoHAMGRFailoverMajorityLoss	2 mysqld down: sole-survivor read-only PRIMARY, then rejoin → 2+ ONLINE → writable
TestPostgreSQLApplyReplicaPgRewind	ApplyReplica with pg_rewind after promote
TestPostgreSQL2NodeScaffold	2-node PG primary + streaming standby
TestNeoHAPGEtcdFailoverMinority	PG + etcd DCS: minority failover promote
TestNeoHASemiSyncWarmSuite	Warm 3-node semi-sync: formation + minority failover
TestNeoHAXtrabackupRebuildMe	2-node semi-sync: neohactl mysql rebuildme --from=<leader> --force

MGR / semi-sync warm suites pre-write peers.json and skip neohactl raft enable + cluster add (production CLI wire is still covered by the xtrabackup test).

Repository layout

cmd/
  neoha/                 # neoha daemon main
  neohactl/              # CLI main
internal/                # application-private packages
  config/ server/ election/ database/ manager/ base/ neohactl/ neorpc/
api/                     # REST API for external callers (e.g. Kubernetes)
configs/examples/        # sample YAML (see configs/README.md)
test/integration/        # integration tests (build tag: integration)
build/                   # ldflags / build metadata
pkg/                     # reserved for future public libraries

Layout

test/integration/
├── harness/
│   backend.go       # Cluster, Node, Backend interface
│   mysql80.go       # MySQL 8.0 start/stop
│   mysql_assert.go  # Read-only MGR assertions
│   neoha.go         # NeoHA config + process lifecycle
│   neorpc.go        # neorpc helpers (neohactl-equivalent)
├── mysql_warm_test.go   # warm fixture helpers (MGR / semi-sync)
├── mgr_3node_test.go
├── mgr_neoha_test.go
├── semisync_neoha_test.go
├── xtrabackup_neoha_test.go
├── postgresql_apply_replica_test.go
├── postgresql_etcd_neoha_test.go
├── it.local.yaml.example
└── README.md

Troubleshooting

go test appears stuck with no output

The first go test compiles the whole module and can take 30s+ with no logs. Prefer:

make test-integration
# or
go test -tags=integration -c -o bin/neoha-it.test ./test/integration && bin/neoha-it.test -test.v ...

Stale processes / ports

pkill -f 'defaults-file=/tmp/neoha-it' || true
rm -rf "${NEOHA_IT_WORKDIR:-/tmp/neoha-it}/<cluster-name>"

MGR warm suites reset datadirs automatically; if a run was interrupted mid-test, clear the whole workdir or kill stale mysqld processes before retrying.

Common ports: scaffold 13346–13348, MGR warm 13306–13308, MGR majority-loss 13326–13328, semi-sync 13316–13318, xtrabackup 13336–13337.

Xtrabackup test hangs on rebuildme

• Confirm SSH: ssh -p <port> $(whoami)@127.0.0.1 echo ok
• Confirm tools: ls "$NEOHA_IT_XTRABACKUP_BINDIR"/xtrabackup "$NEOHA_IT_XTRABACKUP_BINDIR"/xbstream
• Check NeoHA logs under $NEOHA_IT_WORKDIR/<cluster>/neoha/*/neoha.log for backup or socket errors