globalkilltest

GlobalKillTest

GlobalKillTest is a test command tool for TiDB "Global Kill" feature.

(About "Global Kill", see design doc for detail.)

Usage: ./run-tests.sh [options]

    -h: Print this help message.

    -L <info|warn|error>: Log level of testing. Defaults to "info".

    --server_log_level <info|warn|error>: Log level of TiDB server. Defaults to "info".
    
    --tmp <temporary path>: Temporary files path. Defaults to "/tmp/tidb_globalkilltest".

    -s <tidb-server-path>: Use tidb-server in <tidb-server-path> for testing.
                           Defaults to "bin/globalkilltest_tidb-server".

    --tidb_start_port <port>: First TiDB server listening port. port ~ port+2 will be used.
                              Defaults to "5000".

    --tidb_status_port <port>: First TiDB server status listening port. port ~ port+2 will be used.
                               Defaults to "8000".

    --pd <pd-client-path>: PD client path, ip:port list separated by comma.
                           Defaults to "127.0.0.1:2379".

    --pd_proxy_port <port>: PD proxy port. PD proxy is used to simulate lost connection between TiDB and PD.
                            Defaults to "3379".

    --conn_lost <timeout in seconds>: Lost connection to PD timeout,
                                      should be the same as TiDB ldflag <ldflagLostConnectionToPDTimeout>.
                                      See tidb/Makefile for detail.
                                      Defaults to "5".

    --conn_restored <timeout in seconds>: Time to check PD connection restored,
                                          should be the same as TiDB ldflag 
                                          <ldflagServerIDTimeToCheckPDConnectionRestored>.
                                          See tidb/Makefile for detail.
                                          Defaults to "1".

Prerequisite

1. Build TiDB binary for test. See Makefile for detail.

2. Prepare pd-server and tikv-server to setup a cluster for tests. You can download the binaries by TiUP

cd tests/globalkilltest
mkdir -p bin
tiup install pd:nightly tikv:nightly
cp ~/.tiup/components/pd/$(ls ~/.tiup/components/pd | tail -1)/pd-server bin/
cp ~/.tiup/components/tikv/$(ls ~/.tiup/components/tikv | tail -1)/tikv-server bin/

Alternatively, if you have Docker environment, you can run up.sh, which will prepare binaries & run make for you:

cd tests/globalkilltest
./up.sh

Test Scenarios

1. A TiDB without PD, killed by Ctrl+C, and killed by KILL.

2. One TiDB with PD, killed by Ctrl+C, and killed by KILL.

3. Multiple TiDB nodes, killed {local,remote} by {Ctrl-C,KILL}.

4. TiDB with PD, existing connections are killed after PD lost connection for long time.

5. TiDB with PD, new connections are not accepted after PD lost connection for long time.

6. TiDB with PD, new connections are accepted after PD lost connection for long time and then recovered.

7. TiDB with PD, connections can be killed (see 3) after PD lost connection for long time and then recovered.

How it works

• TiDB is built by Makefile, to hack some timeout variables, as the default value of these variables are too long (several hours) for automated testing.

• Execute SELECT SLEEP(x) as payload, and kill the query before x expired. If the query had no error and elapsed less than x, the test is PASSED.

Usage

Regression Execute in Integration Test

In Integration Test after commit and before merge, run these commands under TiDB tests/globalkilltest folder.

cd tests/globalkilltest
make
./run-tests.sh

Manual Test

Run a single test manually (take TestMultipleTiDB as example):

cd tests/globalkilltest
make
go test -check.f TestMultipleTiDB -args --pd=<pd client path>