harness

package
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 4, 2026 License: Apache-2.0 Imports: 23 Imported by: 0

Documentation

Overview

Package harness orchestrates databases and NeoHA nodes for integration tests. Backend is intentionally database-agnostic so PostgreSQL support can plug in later.

Index

Constants

View Source
const (
	EnvMySQLBase = "NEOHA_IT_MYSQL_BASE"
	EnvWorkDir   = "NEOHA_IT_WORKDIR"
)
View Source
const (
	DefaultMySQLBaseLinux         = "/usr/local/mysql"       // tarball layout: bin/mysqld; apt: see ResolveMysqld
	DefaultPostgreSQLBaseLinux    = "/usr/lib/postgresql/14" // apt; tarball often at /usr/local/pgsql
	DefaultPostgreSQLTarballLinux = "/usr/local/pgsql"
	DefaultEtcdBinLinux           = "/usr/bin/etcd"
	DefaultXtrabackupDirLinux     = "/usr/bin"
	DefaultConsulBinLinux         = "/usr/bin/consul"
	DefaultZookeeperBinLinux      = "/usr/bin/zookeeper-server"
)

Default install locations on Ubuntu 22.04 (apt/deb). IT datadir/config stay under workdir or /tmp. Tarball installs often use /usr/local/*; apt packages use paths below.

View Source
const (
	EnvITConfig         = "NEOHA_IT_CONFIG"
	EnvXtrabackupBinDir = "NEOHA_IT_XTRABACKUP_BINDIR"
	EnvEtcdBin          = "NEOHA_ETCD_BIN"
	EnvSSHPort          = "NEOHA_IT_SSH_PORT"
	EnvKeepWorkDir      = "NEOHA_IT_KEEP_WORKDIR"
	EnvTeardownWorkDir  = "NEOHA_IT_TEARDOWN"
)
View Source
const (
	EnvNeoHABin = "NEOHA_IT_BIN"

	// ITGlobalTimeout matches Makefile -test.timeout for make test-integration.
	ITGlobalTimeout = 10 * time.Minute
	// ITWarmClusterTimeout caps one warm fixture lifecycle (setup through subtests on warm.ctx).
	ITWarmClusterTimeout = 5 * time.Minute
)
View Source
const (
	EnvPGBase   = "NEOHA_IT_PG_BASE"
	PGSuperUser = "postgres"
	PGSuperPass = "NeoHAIT1!Pass"
	PGReplUser  = "replicator"
	PGReplPass  = "NeoHAIT1!Repl"
)
View Source
const (
	EnvNeoHACtlBin = "NEOHA_IT_CTL_BIN"
)

Variables

This section is empty.

Functions

func AssertBackupArtifacts

func AssertBackupArtifacts(t *testing.T, dir string)

AssertBackupArtifacts checks xtrabackup stream output exists after backup+prepare.

func BuildNeoHA

func BuildNeoHA(ctx context.Context, repoRoot, outPath string) error

BuildNeoHA compiles the neoha daemon to outPath (go build uses the module cache; always invoke it so dependency changes are picked up).

func BuildNeoHActl

func BuildNeoHActl(ctx context.Context, repoRoot, outPath string) error

BuildNeoHActl compiles neohactl to outPath.

func EndpointsForClusterNodes

func EndpointsForClusterNodes(neoNodes []*NeoHANode, cluster *Cluster, nodes []*Node) []string

EndpointsForClusterNodes maps surviving MySQL cluster nodes to NeoHA RPC endpoints.

func EnsurePortsFree

func EnsurePortsFree(ports []int) error

EnsurePortsFree fails fast when MySQL/RPC ports are still held by a stale run.

func EtcdBin added in v0.2.0

func EtcdBin() string

EtcdBin returns the etcd binary path for integration tests.

func ExecSQL

func ExecSQL(port int, query string) error

ExecSQL runs a statement on local mysqld (root, no password).

func FormatGRSeeds added in v0.2.0

func FormatGRSeeds(ports []int) string

FormatGRSeeds builds a group_replication_group_seeds value from GR ports.

func FreePorts added in v0.2.0

func FreePorts(ports []int) error

FreePorts kills stale listeners on ports and waits until they are available.

func GetRaftStates added in v0.2.0

func GetRaftStates(endpoints []string) map[string]string

GetRaftStates returns the current Raft state per endpoint (for diagnostics).

func KeepWorkDir added in v0.3.0

func KeepWorkDir() bool

KeepWorkDir reports whether integration tests should retain cluster workdirs after stop (reuse datadirs on the next run). Default true unless NEOHA_IT_TEARDOWN=1.

func KillProcessesOnPorts added in v0.2.0

func KillProcessesOnPorts(ports []int)

KillProcessesOnPorts sends SIGKILL to processes listening on the given TCP ports.

func KillProcessesOnWorkDir

func KillProcessesOnWorkDir(workDir string)

KillProcessesOnWorkDir sends SIGTERM to processes whose cmdline references workDir.

func MySQLBaseFromEnv

func MySQLBaseFromEnv() string

MySQLBaseFromEnv returns the MySQL installation root or empty if unset.

func NeoHABinFromEnv

func NeoHABinFromEnv() string

NeoHABinFromEnv returns the neoha binary path or empty.

func NeoHACtlBinFromEnv

func NeoHACtlBinFromEnv() string

NeoHACtlBinFromEnv returns the neohactl binary path or empty.

func PGBaseFromEnv added in v0.2.0

func PGBaseFromEnv() string

PGBaseFromEnv returns the PostgreSQL installation root.

func PrepareBackupDir

func PrepareBackupDir(dir string) error

func QueryScalar

func QueryScalar(port int, query string) (int, error)

QueryScalar runs a query returning a single int value on local mysqld.

func ReadyPollInterval added in v0.2.0

func ReadyPollInterval() time.Duration

ReadyPollInterval is the poll interval used by harness readiness loops.

func RunNeoHActl

func RunNeoHActl(ctx context.Context, ctlBin, ctlDir string, args ...string) error

RunNeoHActl runs neohactl in ctlDir (must contain config.path).

func RunNeoHActlMysqlRebuildMe

func RunNeoHActlMysqlRebuildMe(ctx context.Context, ctlBin, ctlDir, from string, force bool) error

RunNeoHActlMysqlRebuildMe runs neohactl mysql rebuildme on a replica node.

func StartTestEtcd added in v0.2.0

func StartTestEtcd(t *testing.T) (endpoint string, stop func())

StartTestEtcd launches a single-node etcd for integration tests.

func WaitNeoHAReady

func WaitNeoHAReady(ctx context.Context, endpoints []string) error

WaitNeoHAReady waits until all endpoints accept ServerPingRPC (same RPC as neohactl neoha ping).

func WaitNeoHAReadyViaCLI

func WaitNeoHAReadyViaCLI(ctx context.Context, ctlBin string, nodes []*NeoHANode) error

WaitNeoHAReadyViaCLI waits until neohactl neoha ping succeeds on every node.

func WaitPortFree added in v0.2.0

func WaitPortFree(ctx context.Context, host string, port int) error

func WaitRaftLeader

func WaitRaftLeader(ctx context.Context, endpoints []string) (string, error)

WaitRaftLeader polls RaftStatusRPC until one endpoint reports LEADER.

func WaitRaftNotState added in v0.2.0

func WaitRaftNotState(ctx context.Context, endpoint, avoid string) error

WaitRaftNotState waits until endpoint reports a Raft state other than avoid.

func WaitRaftState added in v0.2.0

func WaitRaftState(ctx context.Context, endpoint, want string) error

WaitRaftState waits until endpoint reports the wanted Raft state.

func WireNeoHAClusterViaCLI

func WireNeoHAClusterViaCLI(ctx context.Context, ctlBin string, nodes []*NeoHANode, endpoints []string) error

WireNeoHAClusterViaCLI registers peers using neohactl.

func WorkDirFromEnv

func WorkDirFromEnv() string

WorkDirFromEnv returns the integration test work directory.

func WritePeersJSON

func WritePeersJSON(path string, peers []string) error

WritePeersJSON writes raft peers metadata consumed on NeoHA startup.

Types

type Backend

type Backend interface {
	Name() string
	InitNode(ctx context.Context, node *Node) error
	StartNode(ctx context.Context, node *Node) error
	StopNode(ctx context.Context, node *Node) error
	Ready(ctx context.Context, node *Node) error
}

Backend manages lifecycle of a database engine (MySQL, PostgreSQL, …).

type Cluster

type Cluster struct {
	Name    string
	WorkDir string
	Backend Backend
	Nodes   []*Node
}

Cluster groups nodes sharing a work directory.

func NewCluster

func NewCluster(name, workDir string, backend Backend) *Cluster

NewCluster creates a cluster under workDir/name.

func (*Cluster) AddNode

func (c *Cluster) AddNode(name string, port, grPort int) *Node

AddNode registers a node; ports are chosen by the caller.

func (*Cluster) Setup

func (c *Cluster) Setup(ctx context.Context) error

Setup initializes datadirs and configs for all nodes.

func (*Cluster) SetupFresh

func (c *Cluster) SetupFresh(ctx context.Context, clean bool) error

SetupFresh initializes nodes; when clean is true, stale processes are stopped and the workdir is removed only if datadirs are missing or incomplete.

func (*Cluster) SetupReset added in v0.2.0

func (c *Cluster) SetupReset(ctx context.Context) error

SetupReset stops stale processes, removes the workdir, and re-initializes all nodes. Use for tests that leave MGR/replication in a state unsafe to reuse (e.g. majority loss).

func (*Cluster) StartAll

func (c *Cluster) StartAll(ctx context.Context) error

StartAll starts every node in parallel and waits until ready.

func (*Cluster) StopAll

func (c *Cluster) StopAll(ctx context.Context)

StopAll stops all nodes (best effort).

func (*Cluster) StopAndMaybeTeardown added in v0.3.0

func (c *Cluster) StopAndMaybeTeardown(ctx context.Context, keepWorkDir bool) error

StopAndMaybeTeardown stops nodes; removes the workdir only when keepWorkDir is false.

func (*Cluster) Teardown

func (c *Cluster) Teardown(ctx context.Context) error

Teardown stops nodes and removes the work directory.

type DatadirChecker added in v0.2.0

type DatadirChecker interface {
	NodeDatadirReady(node *Node) bool
}

DatadirChecker optionally reports whether a node's data directory is initialized.

type ITConfigFile

type ITConfigFile struct {
	MySQLBase        string `yaml:"mysql-base" json:"mysql-base"`
	XtrabackupBinDir string `yaml:"xtrabackup-bindir" json:"xtrabackup-bindir"`
	WorkDir          string `yaml:"workdir" json:"workdir"`
	NeoHABin         string `yaml:"neoha-bin" json:"neoha-bin"`
	NeoHACtlBin      string `yaml:"neohactl-bin" json:"neohactl-bin"`
	PGBase           string `yaml:"pg-base" json:"pg-base"`
	EtcdBin          string `yaml:"etcd-bin" json:"etcd-bin"`
	SSHHost          string `yaml:"ssh-host" json:"ssh-host"`
	SSHPort          int    `yaml:"ssh-port" json:"ssh-port"`
	SSHUser          string `yaml:"ssh-user" json:"ssh-user"`
	SSHPasswd        string `yaml:"ssh-passwd" json:"ssh-passwd"`
}

ITConfigFile is the optional integration-test settings file (YAML or JSON).

type IntegrationSettings

type IntegrationSettings struct {
	ConfigPath string
	// contains filtered or unexported fields
}

IntegrationSettings resolves tool paths for integration tests. Priority per field: environment variable → IT config file → PATH lookup → OS default path.

func LoadIntegrationSettings

func LoadIntegrationSettings() *IntegrationSettings

LoadIntegrationSettings returns cached integration settings.

func (*IntegrationSettings) ApplyBackupConfig

func (s *IntegrationSettings) ApplyBackupConfig(conf *config.Config, mysqlPort int, mysqlBase, defaultsFile, mysqlDataDir string)

ApplyBackupConfig fills database.mysql.backup for integration tests. mysqlDataDir is the local mysqld datadir (rebuildme xbstream target).

func (*IntegrationSettings) EtcdBin added in v0.3.0

func (s *IntegrationSettings) EtcdBin() string

EtcdBin returns the etcd server binary path.

func (*IntegrationSettings) MySQLBase

func (s *IntegrationSettings) MySQLBase() string

MySQLBase returns the MySQL installation root (contains bin/mysqld or sbin/mysqld).

func (*IntegrationSettings) NeoHABinPath

func (s *IntegrationSettings) NeoHABinPath() string

NeoHABinPath returns a pre-built neoha binary path or empty. Priority: NEOHA_IT_BIN → IT config → ./bin/neoha (when present).

func (*IntegrationSettings) NeoHACtlBinPath

func (s *IntegrationSettings) NeoHACtlBinPath() string

NeoHACtlBinPath returns a pre-built neohactl binary path or empty. Priority: NEOHA_IT_CTL_BIN → IT config → ./bin/neohactl (when present).

func (*IntegrationSettings) PostgreSQLBase added in v0.3.0

func (s *IntegrationSettings) PostgreSQLBase() string

PostgreSQLBase returns the PostgreSQL installation root (contains bin/pg_ctl).

func (*IntegrationSettings) RequireMySQL80

func (s *IntegrationSettings) RequireMySQL80(t *testing.T) (string, string)

RequireMySQL80 skips the test when mysqld is unavailable.

func (*IntegrationSettings) RequireSSH

func (s *IntegrationSettings) RequireSSH(t *testing.T)

RequireSSH skips when localhost SSH is unavailable for xbstream.

func (*IntegrationSettings) RequireXtrabackup

func (s *IntegrationSettings) RequireXtrabackup(t *testing.T) string

RequireXtrabackup skips when xtrabackup/xbstream are unavailable.

func (*IntegrationSettings) ResolveMysqld added in v0.3.0

func (s *IntegrationSettings) ResolveMysqld() (base, mysqld string)

ResolveMysqld returns MySQL base dir and mysqld binary path.

func (*IntegrationSettings) SSHAvailable

func (s *IntegrationSettings) SSHAvailable(ctx context.Context) bool

SSHAvailable reports whether backup SSH preflight would succeed.

func (*IntegrationSettings) WorkDir

func (s *IntegrationSettings) WorkDir() string

WorkDir returns the integration test work directory.

func (*IntegrationSettings) XtrabackupBinDir

func (s *IntegrationSettings) XtrabackupBinDir() string

XtrabackupBinDir returns the directory containing xtrabackup and xbstream.

func (*IntegrationSettings) XtrabackupToolsPresent

func (s *IntegrationSettings) XtrabackupToolsPresent() bool

XtrabackupToolsPresent reports whether backup binaries exist (no test skip).

type MySQL80

type MySQL80 struct {
	BaseDir  string
	SemiSync bool // when true, load semi-sync plugins instead of group_replication
	Plain    bool // when true, minimal mysqld without replication plugins (backup tests)
}

MySQL80 implements Backend for a debug/source-tree MySQL 8.0 build.

func NewMySQL80

func NewMySQL80(baseDir string) *MySQL80

NewMySQL80 returns a backend using baseDir (mysqld lives in baseDir/bin/mysqld).

func (*MySQL80) FindMGRPrimaryNode

func (b *MySQL80) FindMGRPrimaryNode(nodes []*Node) (*Node, error)

FindMGRPrimaryNode returns the cluster node whose local mysqld reports PRIMARY.

func (*MySQL80) FindWritableNode

func (b *MySQL80) FindWritableNode(nodes []*Node) (*Node, error)

FindWritableNode returns the first node with @@read_only=0.

func (*MySQL80) InitNode

func (b *MySQL80) InitNode(ctx context.Context, node *Node) error

func (*MySQL80) MGRMemberRole

func (b *MySQL80) MGRMemberRole(node *Node) (string, error)

MGRMemberRole returns MEMBER_ROLE for the local node (PRIMARY/SECONDARY). Read-only assertion helper.

func (*MySQL80) MySQLReadOnly

func (b *MySQL80) MySQLReadOnly(node *Node) (bool, error)

MySQLReadOnly returns @@read_only for node.

func (*MySQL80) Name

func (b *MySQL80) Name() string

func (*MySQL80) NodeDatadirReady added in v0.2.0

func (b *MySQL80) NodeDatadirReady(node *Node) bool

func (*MySQL80) OnlineMGRMembers

func (b *MySQL80) OnlineMGRMembers(node *Node) (int, error)

OnlineMGRMembers counts ONLINE members visible from node. Read-only assertion helper.

func (*MySQL80) Ready

func (b *MySQL80) Ready(ctx context.Context, node *Node) error

func (*MySQL80) ReplicaStatus

func (b *MySQL80) ReplicaStatus(node *Node) (*ReplicaStatus, error)

ReplicaStatus returns replication thread state for node. Read-only assertion helper.

func (*MySQL80) StartNode

func (b *MySQL80) StartNode(ctx context.Context, node *Node) error

func (*MySQL80) StopNode

func (b *MySQL80) StopNode(ctx context.Context, node *Node) error

func (*MySQL80) TailLog

func (b *MySQL80) TailLog(node *Node, maxBytes int) string

TailLog returns the last n bytes of a node's mysqld log (for diagnostics).

func (*MySQL80) WaitMGROnlineMembers

func (b *MySQL80) WaitMGROnlineMembers(ctx context.Context, node *Node, want int) error

WaitMGROnlineMembers waits until at least want ONLINE MGR members are visible from node.

func (*MySQL80) WaitMGROnlineMembersBelow

func (b *MySQL80) WaitMGROnlineMembersBelow(ctx context.Context, node *Node, maxExclusive int) error

WaitMGROnlineMembersBelow waits until fewer than maxExclusive ONLINE members are visible from node.

func (*MySQL80) WaitMGRPrimaryOnAny

func (b *MySQL80) WaitMGRPrimaryOnAny(ctx context.Context, nodes []*Node) (*Node, error)

func (*MySQL80) WaitMySQLWritable

func (b *MySQL80) WaitMySQLWritable(ctx context.Context, node *Node) error

WaitMySQLWritable waits until @@read_only=0 on node.

func (*MySQL80) WaitReplicaConnectedTo

func (b *MySQL80) WaitReplicaConnectedTo(ctx context.Context, node *Node, masterPort int) error

WaitReplicaConnectedTo waits until node replicates from masterPort with both threads running.

func (*MySQL80) WaitReplicaRunning

func (b *MySQL80) WaitReplicaRunning(ctx context.Context, node *Node) error

WaitReplicaRunning waits until IO and SQL replica threads are Yes on node.

type NeoHANode

type NeoHANode struct {
	Name       string
	Endpoint   string
	MetaDir    string
	ConfigPath string
	LogPath    string
	MySQLPort  int
	// contains filtered or unexported fields
}

NeoHANode is one NeoHA agent process in an integration cluster.

func NewNeoHANode

func NewNeoHANode(name, workDir, endpoint string, mysqlPort int) *NeoHANode

NewNeoHANode prepares directories for a NeoHA agent.

func (*NeoHANode) Start

func (n *NeoHANode) Start(ctx context.Context, neohaBin, role string) error

Start launches the neoha daemon. role is LEADER, FOLLOWER, or IDLE (default FOLLOWER).

func (*NeoHANode) Stop

func (n *NeoHANode) Stop(ctx context.Context) error

Stop terminates the neoha process.

func (*NeoHANode) TailNeoHALog

func (n *NeoHANode) TailNeoHALog(maxBytes int) string

TailNeoHALog returns the tail of a NeoHA agent log file.

func (*NeoHANode) WriteConfig

func (n *NeoHANode) WriteConfig(mysqlBase, defaultsFile, clusterWorkDir, mysqlDataDir string, peers []string) error

WriteConfig writes a NeoHA config for MGR integration tests.

func (*NeoHANode) WriteEtcdPGConfig added in v0.2.0

func (n *NeoHANode) WriteEtcdPGConfig(pgBase, dataDir, etcdEndpoint string, pgPort int) error

WriteEtcdPGConfig writes NeoHA config for PostgreSQL + etcd integration tests.

func (*NeoHANode) WriteMGRConfig added in v0.2.0

func (n *NeoHANode) WriteMGRConfig(mysqlBase, defaultsFile, clusterWorkDir, mysqlDataDir string, peers []string, grLocalPort int, grSeeds string) error

WriteMGRConfig writes NeoHA config for MGR integration tests.

func (*NeoHANode) WritePlainConfig

func (n *NeoHANode) WritePlainConfig(mysqlBase, defaultsFile, clusterWorkDir, mysqlDataDir string, peers []string) error

WritePlainConfig writes a minimal NeoHA config for backup / xtrabackup tests.

func (*NeoHANode) WriteSemiSyncConfig

func (n *NeoHANode) WriteSemiSyncConfig(mysqlBase, defaultsFile, clusterWorkDir, mysqlDataDir string, peers []string) error

WriteSemiSyncConfig writes a NeoHA config for semi-sync integration tests.

type Node

type Node struct {
	Name    string
	Port    int
	GRPort  int    // group_replication local address port (MySQL MGR)
	GRSeeds string // group_replication_group_seeds for this cluster
	DataDir string
	Config  string // path to my.cnf or postgresql.conf
	Socket  string
	WorkDir string
	// contains filtered or unexported fields
}

Node is one database (+ optional NeoHA agent) in a test cluster.

type PostgreSQL added in v0.2.0

type PostgreSQL struct {
	BaseDir string
}

PostgreSQL implements Backend for a source-tree or package PostgreSQL build.

func NewPostgreSQL added in v0.2.0

func NewPostgreSQL(baseDir string) *PostgreSQL

NewPostgreSQL returns a backend using baseDir (pg_ctl lives in baseDir/bin/pg_ctl).

func RequirePostgreSQL added in v0.2.0

func RequirePostgreSQL(t testingT, base string) *PostgreSQL

RequirePostgreSQL skips the test when pg_ctl is missing.

func (*PostgreSQL) BootstrapPrimary added in v0.2.0

func (b *PostgreSQL) BootstrapPrimary(ctx context.Context, node *Node) error

BootstrapPrimary creates replication users and sets superuser password on a fresh primary.

func (*PostgreSQL) CloneStandby added in v0.2.0

func (b *PostgreSQL) CloneStandby(ctx context.Context, primary, standby *Node) error

CloneStandby runs pg_basebackup from primary into standby node.

func (*PostgreSQL) ExecPrimary added in v0.2.0

func (b *PostgreSQL) ExecPrimary(ctx context.Context, node *Node, query string) (int64, error)

ExecPrimary runs SQL on a node using the superuser.

func (*PostgreSQL) InitNode added in v0.2.0

func (b *PostgreSQL) InitNode(ctx context.Context, node *Node) error

func (*PostgreSQL) Name added in v0.2.0

func (b *PostgreSQL) Name() string

func (*PostgreSQL) NodeDatadirReady added in v0.2.0

func (b *PostgreSQL) NodeDatadirReady(node *Node) bool

func (*PostgreSQL) OpenAdmin added in v0.2.0

func (b *PostgreSQL) OpenAdmin(ctx context.Context, node *Node) (*sql.DB, error)

OpenAdmin connects as the integration superuser.

func (*PostgreSQL) Promote added in v0.2.0

func (b *PostgreSQL) Promote(ctx context.Context, node *Node) error

Promote runs pg_promote on standby.

func (*PostgreSQL) Ready added in v0.2.0

func (b *PostgreSQL) Ready(ctx context.Context, node *Node) error

func (*PostgreSQL) StartNode added in v0.2.0

func (b *PostgreSQL) StartNode(ctx context.Context, node *Node) error

func (*PostgreSQL) StopNode added in v0.2.0

func (b *PostgreSQL) StopNode(ctx context.Context, node *Node) error

func (*PostgreSQL) WaitInRecovery added in v0.2.0

func (b *PostgreSQL) WaitInRecovery(ctx context.Context, node *Node, inRecovery bool) error

WaitInRecovery polls until node reports pg_is_in_recovery().

func (*PostgreSQL) WaitReplicationConnected added in v0.2.0

func (b *PostgreSQL) WaitReplicationConnected(ctx context.Context, primary, standby *Node) error

WaitReplicationConnected waits until primary shows a connected standby.

type ReplicaStatus

type ReplicaStatus struct {
	MasterHost      string
	MasterPort      int
	SlaveIORunning  string
	SlaveSQLRunning string
}

ReplicaStatus holds read-only fields from SHOW REPLICA STATUS.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL