runn

runn ( means "Run N". is pronounced /rʌ́n én/. ) is a package/tool for running operations following a scenario.

Key features of runn are:

• As a tool for scenario based testing.
• As a test helper package for the Go language.
• As a tool for workflow automation.
• Support HTTP request, gRPC request, DB query, Chrome DevTools Protocol, and SSH/Local command execution
• OpenAPI Document-like syntax for HTTP request testing.
• Single binary = CI-Friendly.

Online book

• runn Tutorial (Japanese)
• runn cookbook (Japanese)

Quickstart

You can use the runn new command to quickly start creating scenarios (runbooks).

🚀 Create and run scenario using curl or grpcurl commands:

$ curl https://httpbin.org/json -H "accept: application/json"
{
  "slideshow": {
    "author": "Yours Truly",
    "date": "date of publication",
    "slides": [
      {
        "title": "Wake up to WonderWidgets!",
        "type": "all"
      },
      {
        "items": [
          "Why <em>WonderWidgets</em> are great",
          "Who <em>buys</em> WonderWidgets"
        ],
        "title": "Overview",
        "type": "all"
      }
    ],
    "title": "Sample Slide Show"
  }
}
$ runn new --and-run --desc 'httpbin.org GET' --out http.yml -- curl https://httpbin.org/json -H "accept: application/json"
$ grpcurl -d '{"greeting": "alice"}' grpcb.in:9001 hello.HelloService/SayHello
{
  "reply": "hello alice"
}
$ runn new --and-run --desc 'grpcb.in Call' --out grpc.yml -- grpcurl -d '{"greeting": "alice"}' grpcb.in:9001 hello.HelloService/SayHello
$ runn list *.yml
  Desc             Path      If
---------------------------------
  grpcb.in Call    grpc.yml
  httpbin.org GET  http.yml
$ runn run *.yml
..

2 scenarios, 0 skipped, 0 failures

🚀 Create scenario using access log:

$ cat access_log
183.87.255.54 - - [18/May/2019:05:37:09 +0200] "GET /?post=%3script%3ealert(1); HTTP/1.0" 200 42433
62.109.16.162 - - [18/May/2019:05:37:12 +0200] "GET /core/files/js/editor.js/?form=\xeb\x2a\x5e\x89\x76\x08\xc6\x46\x07\x00\xc7\x46\x0c\x00\x00\x00\x80\xe8\xdc\xff\xff\xff/bin/sh HTTP/1.0" 200 81956
87.251.81.179 - - [18/May/2019:05:37:13 +0200] "GET /login.php/?user=admin&amount=100000 HTTP/1.0" 400 4797
103.36.79.144 - - [18/May/2019:05:37:14 +0200] "GET /authorize.php/.well-known/assetlinks.json HTTP/1.0" 200 9436
$ cat access_log| runn new --out axslog.yml
$ cat axslog.yml| yq
desc: Generated by `runn new`
runners:
  req: https://dummy.example.com
steps:
  - req:
      /?post=%3script%3ealert(1);:
        get:
          body: null
  - req:
      /core/files/js/editor.js/?form=xebx2ax5ex89x76x08xc6x46x07x00xc7x46x0cx00x00x00x80xe8xdcxffxffxff/bin/sh:
        get:
          body: null
  - req:
      /login.php/?user=admin&amount=100000:
        get:
          body: null
  - req:
      /authorize.php/.well-known/assetlinks.json:
        get:
          body: null
$

Usage

runn can run a multi-step scenario following a runbook written in YAML format.

As a tool for scenario based testing / As a tool for automation.

runn can run one or more runbooks as a CLI tool.

$ runn list path/to/**/*.yml
  id:      desc:             if:       steps:  path
-------------------------------------------------------------------------
  a1b7b02  Only if included  included       2  p/t/only_if_included.yml
  85ccd5f  List projects.                   4  p/t/p/list.yml
  47d7ef7  List users.                      3  p/t/u/list.yml
  97f9884  Login                            2  p/t/u/login.yml
  2249d1b  Logout                           3  p/t/u/logout.yml
$ runn run path/to/**/*.yml
S....

5 scenarios, 1 skipped, 0 failures

As a test helper package for the Go language.

runn can also behave as a test helper for the Go language.

Run N runbooks using httptest.Server and sql.DB

func TestRouter(t *testing.T) {
	ctx := context.Background()
	dsn := "username:password@tcp(localhost:3306)/testdb"
	db, err := sql.Open("mysql", dsn)
	if err != nil {
		log.Fatal(err)
	}
	dbr, err := sql.Open("mysql", dsn)
	if err != nil {
		log.Fatal(err)
	}
	ts := httptest.NewServer(NewRouter(db))
	t.Cleanup(func() {
		ts.Close()
		db.Close()
		dbr.Close()
	})
	opts := []runn.Option{
		runn.T(t),
		runn.Runner("req", ts.URL),
		runn.DBRunner("db", dbr),
	}
	o, err := runn.Load("testdata/books/**/*.yml", opts...)
	if err != nil {
		t.Fatal(err)
	}
	if err := o.RunN(ctx); err != nil {
		t.Fatal(err)
	}
}

Run single runbook using httptest.Server and sql.DB

func TestRouter(t *testing.T) {
	ctx := context.Background()
	dsn := "username:password@tcp(localhost:3306)/testdb"
	db, err := sql.Open("mysql", dsn)
	if err != nil {
		log.Fatal(err)
	}
	dbr, err := sql.Open("mysql", dsn)
	if err != nil {
		log.Fatal(err)
	}
	ts := httptest.NewServer(NewRouter(db))
	t.Cleanup(func() {
		ts.Close()
		db.Close()
		dbr.Close()
	})
	opts := []runn.Option{
		runn.T(t),
		runn.Book("testdata/books/login.yml"),
		runn.Runner("req", ts.URL),
		runn.DBRunner("db", dbr),
	}
	o, err := runn.New(opts...)
	if err != nil {
		t.Fatal(err)
	}
	if err := o.Run(ctx); err != nil {
		t.Fatal(err)
	}
}

Run N runbooks using grpc.Server

func TestServer(t *testing.T) {
	addr := "127.0.0.1:8080"
	l, err := net.Listen("tcp", addr)
	if err != nil {
		t.Fatal(err)
	}
	ts := grpc.NewServer()
	myapppb.RegisterMyappServiceServer(s, NewMyappServer())
	reflection.Register(s)
	go func() {
		ts.Serve(l)
	}()
	t.Cleanup(func() {
		ts.GracefulStop()
	})
	opts := []runn.Option{
		runn.T(t),
		runn.Runner("greq", fmt.Sprintf("grpc://%s", addr),
	}
	o, err := runn.Load("testdata/books/**/*.yml", opts...)
	if err != nil {
		t.Fatal(err)
	}
	if err := o.RunN(ctx); err != nil {
		t.Fatal(err)
	}
}

Run N runbooks with http.Handler and sql.DB

func TestRouter(t *testing.T) {
	ctx := context.Background()
	dsn := "username:password@tcp(localhost:3306)/testdb"
	db, err := sql.Open("mysql", dsn)
	if err != nil {
		log.Fatal(err)
	}
	dbr, err := sql.Open("mysql", dsn)
	if err != nil {
		log.Fatal(err)
	}
	t.Cleanup(func() {
		db.Close()
		dbr.Close()
	})
	opts := []runn.Option{
		runn.T(t),
		runn.HTTPRunnerWithHandler("req", NewRouter(db)),
		runn.DBRunner("db", dbr),
	}
	o, err := runn.Load("testdata/books/**/*.yml", opts...)
	if err != nil {
		t.Fatal(err)
	}
	if err := o.RunN(ctx); err != nil {
		t.Fatal(err)
	}
}

Examples

See the details

Runbook ( runn scenario file )

The runbook file has the following format.

step: section accepts list or ordered map.

List:

desc: Login and get projects.
runners:
  req: https://example.com/api/v1
  db: mysql://root:mypass@localhost:3306/testdb
vars:
  username: alice
  password: ${TEST_PASS}
steps:
  -
    db:
      query: SELECT * FROM users WHERE name = '{{ vars.username }}'
  -
    req:
      /login:
        post:
          body:
            application/json:
              email: "{{ steps[0].rows[0].email }}"
              password: "{{ vars.password }}"
    test: steps[1].res.status == 200
  -
    req:
      /projects:
        get:
          headers:
            Authorization: "token {{ steps[1].res.body.session_token }}"
          body: null
    test: steps[2].res.status == 200
  -
    test: len(steps[2].res.body.projects) > 0

Map:

desc: Login and get projects.
runners:
  req: https://example.com/api/v1
  db: mysql://root:mypass@localhost:3306/testdb
vars:
  username: alice
  password: ${TEST_PASS}
steps:
  find_user:
    db:
      query: SELECT * FROM users WHERE name = '{{ vars.username }}'
  login:
    req:
      /login:
        post:
          body:
            application/json:
              email: "{{ steps.find_user.rows[0].email }}"
              password: "{{ vars.password }}"
    test: steps.login.res.status == 200
  list_projects:
    req:
      /projects:
        get:
          headers:
            Authorization: "token {{ steps.login.res.body.session_token }}"
          body: null
    test: steps.list_projects.res.status == 200
  count_projects:
    test: len(steps.list_projects.res.body.projects) > 0

Grouping of related parts by color

List:

Map:

desc:

Description of runbook.

desc: Login and get projects.
runners:
  req: https://example.com/api/v1
vars:
  username: alice
steps:
[...]

labels:

Labels of runbook.

desc: Login
runners:
  req: https://example.com/api/v1
labels:
  - users
  - auth
steps:
[...]

Runbooks to be run can be filtered by labels.

$ runn run path/to/**/*.yml --label users --label projects

$ runn run path/to/**/*.yml --label 'users and auth'

runners:

Mapping of runners that run steps: of runbook.

In the steps: section, call the runner with the key specified in the runners: section.

Built-in runners such as test runner do not need to be specified in this section.

runners:
  ghapi: ${GITHUB_API_ENDPOINT}
  idp: https://auth.example.com
  db: my:dbuser:${DB_PASS}@hostname:3306/dbname

In the example, each runner can be called by ghapi:, idp: or db: in steps:.

hostRules:

Allows remapping any request hostname to another hostname, IP address in HTTP/gRPC/DB/CDP/SSH runners.

hostRules:
  example.com: 127.0.0.1:8080
  '*.example.test': 192.168.0.16

vars:

Mapping of variables available in the steps: of runbook.

vars:
  username: alice@example.com
  token: ${SECRET_TOKEN}

In the example, each variable can be used in {{ vars.username }} or {{ vars.token }} in steps:.

secrets:

List of secret var names to be masked.

secrets:
  - vars.secret_token
  - binded_password
  - current.res.message.token

debug:

Enable debug output for runn.

debug: true

interval:

Interval between steps.

interval: 1

if:

Conditions for skip all steps.

if: included # Run steps only if included

skipTest:

Skip all test: sections

skipTest: true

force:

Force all steps to run.

force: true

trace:

Add tokens for tracing to headers and queries by default.

Currently, HTTP runner, gRPC runner and DB runner are supported.

trace: true

loop:

Loop setting for runbook.

Simple loop runbook

loop: 10
steps:
  [...]

or

loop:
  count: 10
steps:
  [...]

Retry runbook

It can be used as a retry mechanism by setting a condition in the until: section.

If the condition of until: is met, the loop is broken without waiting for the number of count: to be run.

Also, if the run of the number of count: completes but does not satisfy the condition of until:, then the step is considered to be failed.

loop:
  count: 10
  until: 'outcome == "success"' # until the runbook outcome is successful.
  minInterval: 0.5 # sec
  maxInterval: 10  # sec
  # jitter: 0.0
  # interval: 5
  # multiplier: 1.5
steps:
  waitingroom:
    req:
      /cart/in:
        post:
          body:
[...]

• outcome ... the result of a completed (success, failure, skipped).

concurrency:

Runbooks with the same key are assured of a single run at the same time.

concurrency: use-shared-db

or

concurrency:
  - use-shared-db
  - use-shared-api

needs:

It is possible to identify runbooks that must be pre-run.

needs:
  prebook: path/to/prebook.yml
  prebook2: path/to/prebook2.yml

Values bound by the bind runner can be referenced by needs.<key>. *.

steps:

Steps to run in runbook.

The steps are invoked in order from top to bottom.

Any return values are recorded for each step.

When steps: is array, recorded values can be retrieved with {{ steps[*].* }}.

steps:
  -
    db:
      query: SELECT * FROM users WHERE name = '{{ vars.username }}'
  -
    req:
      /users/{{ steps[0].rows[0].id }}:
        get:
          body: null

When steps: is map, recorded values can be retrieved with {{ steps.<key>.* }}.

steps:
  find_user:
    db:
      query: SELECT * FROM users WHERE name = '{{ vars.username }}'
  user_info:
    req:
      /users/{{ steps.find_user.rows[0].id }}:
        get:
          body: null

steps[*].desc: steps.<key>.desc:

Description of step.

steps:
  -
    desc: Login
    req:
      /login:
        post:
          body:
[...]

steps[*].if: steps.<key>.if:

Conditions for skip step.

steps:
  login:
    if: 'len(vars.token) == 0' # Run step only if var.token is not set
    req:
      /login:
        post:
          body:
[...]

steps[*].loop: steps.<key>.loop:

Loop setting for step.

Simple loop step

steps:
  multicartin:
    loop: 10
    req:
      /cart/in:
        post:
          body:
            application/json:
              product_id: "{{ i }}" # The loop count (0..9) is assigned to `i`.
[...]

or

steps:
  multicartin:
    loop:
      count: 10
    req:
      /cart/in:
        post:
          body:
            application/json:
              product_id: "{{ i }}" # The loop count (0..9) is assigned to `i`.
[...]

Retry step

It can be used as a retry mechanism by setting a condition in the until: section.

If the condition of until: is met, the loop is broken without waiting for the number of count: to be run.

Also, if the run of the number of count: completes but does not satisfy the condition of until:, then the step is considered to be failed.

steps:
  waitingroom:
    loop:
      count: 10
      until: 'steps.waitingroom.res.status == "201"' # Store values of latest loop
      minInterval: 500ms
      maxInterval: 10 # sec
      # jitter: 0.0
      # interval: 5
      # multiplier: 1.5
    req:
      /cart/in:
        post:
          body:
[...]

Execution order within a loop

When using loop:, it's important to understand the execution order of runners within each iteration:

1. Main runner (HTTP, DB, gRPC, CDP, SSH, Exec, Include, or Runner) - Only one is executed
2. Dump runner - Records request/response data (if specified)
3. Bind runner - Binds variables (if specified)
4. Test runner - Performs assertions (if specified)

This sequence repeats for each loop iteration.

⚠️ Warning: Avoid using test: with loop.until:

When using loop: with an until: condition, avoid also using test: in the same step. The test: assertion runs on every loop iteration, which can cause the step to fail on the first iteration before the retry logic has a chance to work.

❌ Incorrect usage:

steps:
  retry_until_success:
    req:
      /status:
        get:
    loop:
      count: 10
      until: 'current.res.status == 200'  # Retry until successful
    test: 'current.res.status == 200'     # This runs every iteration and may fail early!

✅ Correct usage:

steps:
  retry_until_success:
    req:
      /status:
        get:
    loop:
      count: 10
      until: 'current.res.status == 200'  # This is sufficient for retry logic

The until: condition already serves as your test condition for retry scenarios.

steps[*].defer: steps.<key>.defer: [THIS IS EXPERIMENT]

Deferring setting for step.

steps:
  -
    defer: true
    req:
      /cart:
        delete:
          body: null
[...]

The step marked defer behaves as follows.

• If defer: true is set, run of the step is deferred until finish of the runbook.
• Steps marked with defer are always run even if the running of intermediate steps fails.
• If there are multiple steps marked with defer, they are run in LIFO order.
  • Also, the included steps are added to run sequence of the parent runbook's deferred steps.

steps[*].force: steps.<key>.force:

Force step to run.

steps:
[...]
  -
     force: true
     dump: previous.res.body
[...]

Variables to be stored

runn can use variables and functions when running step.

Also, after step runs, HTTP responses, DB query results, etc. are automatically stored in variables.

The values are stored in predefined variables.

Variable Expansion

runn uses {{ }} syntax for variable expansion.

Object variable expansion

When an object variable is used directly in headers: or body:, runn expands it as a map structure, not as a JSON string.

vars:
  auth_headers:
    X-Token: xxx
    X-Api-Key: yyy
steps:
  - req:
      /api:
        get:
          headers: "{{vars.auth_headers}}"

In the example, vars.auth_headers expands to multiple headers.

Converting object to JSON string

To convert an object to a JSON string, wrap the entire {{...}} template expression in single quotes. This signals runn to serialize the object to JSON.

vars:
  metadata:
    user: alice
    role: admin
steps:
  - req:
      /api:
        get:
          headers:
            X-Metadata: "'{{vars.metadata}}'"

## Runner

### HTTP Runner: Do HTTP request

Use `https://` or `http://` scheme to specify HTTP Runner.

When the step is invoked, it sends the specified HTTP Request and records the response.

``` yaml
runners:
  req: https://example.com
steps:
  -
    desc: Post /users                     # description of step
    req:                                  # key to identify the runner. In this case, it is HTTP Runner.
      /users:                             # path of http request
        post:                             # method of http request
          headers:                        # headers of http request
            Authorization: 'Bearer xxxxx'
          body:                           # body of http request
            application/json:             # Content-Type specification. In this case, it is "Content-Type: application/json"
              username: alice
              password: passw0rd
          trace: false                    # add `X-Runn-Trace` header to HTTP request for tracing
    test: |                               # test for current step
      current.res.status == 201

See testdata/book/http.yml and testdata/book/http_multipart.yml.

Structure of recorded responses

The following response

HTTP/1.1 200 OK
Content-Length: 29
Content-Type: application/json
Date: Wed, 07 Sep 2022 06:28:20 GMT
Set-Cookie: cookie-name=cookie-value

{"data":{"username":"alice"}}

is recorded with the following structure.

[`step key` or `current` or `previous`]:
  res:
    status: 200                              # current.res.status
    headers:
      Content-Length:
        - '29'                               # current.res.headers["Content-Length"][0]
      Content-Type:
        - 'application/json'                 # current.res.headers["Content-Type"][0]
      Date:
        - 'Wed, 07 Sep 2022 06:28:20 GMT'    # current.res.headers["Date"][0]
      Set-Cookie:
        - 'cookie-name=cookie-value'         # current.res.headers["Set-Cookie"][0]
    cookies:
      cookie-name: *http.Cookie              # current.res.cookies["cookie-name"].Value
    body:
      data:
        username: 'alice'                    # current.res.body.data.username
    rawBody: '{"data":{"username":"alice"}}' # current.res.rawBody

Do not follow redirect

The HTTP Runner interprets HTTP responses and automatically redirects. To disable this, set notFollowRedirect to true.

runners:
  req:
    endpoint: https://example.com
    notFollowRedirect: true

Enable Cookie Sending

The HTTP Runner automatically saves cookies by interpreting HTTP responses. To enable cookie sending during requests, set useCookie to true.

runners:
  req:
    endpoint: https://example.com
    useCookie: true

See testdata/book/cookie.yml and testdata/book/cookie_in_requests_automatically.yml.

Validation of HTTP request and HTTP response

HTTP requests sent by runn and their HTTP responses can be validated.

OpenAPI v3:

runners:
  myapi:
    endpoint: https://api.example.com
    openapi3: path/to/openapi.yaml
    # skipValidateRequest: false
    # skipValidateResponse: false
    # skipCircularReferenceCheck: false # skip checking circular references in OpenAPIv3 document.

Custom CA and Certificates

runners:
  myapi:
    endpoint: https://api.github.com
    cacert: path/to/cacert.pem
    cert: path/to/cert.pem
    key: path/to/key.pem
    # skipVerify: false

Add X-Runn-Trace header to HTTP request for tracing

runners:
  myapi:
    endpoint: https://api.github.com
    trace: true

gRPC Runner: Do gRPC request

Use grpc:// scheme to specify gRPC Runner.

When the step is invoked, it sends the specified gRPC Request and records the response.

runners:
  greq: grpc://grpc.example.com:80
steps:
  -
    desc: Request using Unary RPC                     # description of step
    greq:                                             # key to identify the runner. In this case, it is gRPC Runner.
      grpctest.GrpcTestService/Hello:                 # package.Service/Method of rpc
        headers:                                      # headers of rpc
          authentication: tokenhello
        message:                                      # message of rpc
          name: alice
          num: 3
          request_time: 2022-06-25T05:24:43.861872Z
        trace: false                                  # add `x-runn-trace` header to gRPC request for tracing
  -
    desc: Request using Server streaming RPC
    greq:
      grpctest.GrpcTestService/ListHello:
        headers:
          authentication: tokenlisthello
        message:
          name: bob
          num: 4
          request_time: 2022-06-25T05:24:43.861872Z
        timeout: 3sec                                 # timeout for rpc
    test: |
      steps.server_streaming.res.status == 0 && len(steps.server_streaming.res.messages) > 0
  -
    desc: Request using Client streaming RPC
    greq:
      grpctest.GrpcTestService/MultiHello:
        headers:
          authentication: tokenmultihello
        messages:                                     # messages of rpc
          -
            name: alice
            num: 5
            request_time: 2022-06-25T05:24:43.861872Z
          -
            name: bob
            num: 6
            request_time: 2022-06-25T05:24:43.861872Z

runners:
  greq:
    addr: grpc.example.com:8080
    tls: true
    cacert: path/to/cacert.pem
    cert: path/to/cert.pem
    key: path/to/key.pem
    # skipVerify: false
    # importPaths:
    #   - protobuf/proto
    # protos:
    #   - general/health.proto
    #   - myapp/**/*.proto

See testdata/book/grpc.yml.

Structure of recorded responses

The following response

message HelloResponse {
  string message = 1;

  int32 num = 2;

  google.protobuf.Timestamp create_time = 3;
}

{"create_time":"2022-06-25T05:24:43.861872Z","message":"hello","num":32}

and headers

content-type: ["application/grpc"]
hello: ["this is header"]

and trailers

hello: ["this is trailer"]

are recorded with the following structure.

[`step key` or `current` or `previous`]:
  res:
    status: 0                                      # current.res.status
    headers:
      content-type:
        - 'application/grpc'                       # current.res.headers[0].content-type
      hello:
        - 'this is header'                         # current.res.headers[0].hello
    trailers:
      hello:
        - 'this is trailer'                        # current.res.trailers[0].hello
    message:
      create_time: '2022-06-25T05:24:43.861872Z'   # current.res.message.create_time
      message: 'hello'                             # current.res.message.message
      num: 32                                      # current.res.message.num
    messages:
      -
        create_time: '2022-06-25T05:24:43.861872Z' # current.res.messages[0].create_time
        message: 'hello'                           # current.res.messages[0].message
        num: 32                                    # current.res.messages[0].num

Add x-runn-trace header to gRPC request for tracing

runners:
  greq:
    addr: grpc.example.com:8080
    trace: true

Buf

gRPC Runner supports Buf ecosystem includes Buf Schema Registry.

It can use the buf modules ( and protos ) it depends on.

runners:
  greq:
    addr: grpc.example.com:8080
    bufDirs:
      - path/to # Set buf directories for registering buf modules and protos

runners:
  greq:
    addr: grpc.example.com:8080
    bufLocks:
      - path/to/buf.lock # Register buf modules using buf.lock

runners:
  greq:
    addr: grpc.example.com:8080
    bufConfigs:
      - path/to/buf.yaml # Register buf modules using buf.yaml

runners:
  greq:
    addr: grpc.example.com:8080
    bufModules:
        - buf.build/owner/repository
        - buf.build/owner2/repository2

DB Runner: Query a database

Use dsn (Data Source Name) to specify DB Runner.

When step is invoked, it executes the specified query the database.

runners:
  db: postgres://dbuser:dbpass@hostname:5432/dbname
steps:
  -
    desc: Select users            # description of step
    db:                           # key to identify the runner. In this case, it is DB Runner.
      query: SELECT * FROM users; # query to execute
      trace: false                # add comment with trace token to query for tracing

See testdata/book/db.yml.

Structure of recorded responses

If the query is a SELECT clause, it records the selected rows,

[`step key` or `current` or `previous`]:
  rows:
    -
      id: 1                           # current.rows[0].id
      username: 'alice'               # current.rows[0].username
      password: 'passw0rd'            # current.rows[0].password
      email: 'alice@example.com'      # current.rows[0].email
      created: '2017-12-05T00:00:00Z' # current.rows[0].created
    -
      id: 2                           # current.rows[1].id
      username: 'bob'                 # current.rows[1].username
      password: 'passw0rd'            # current.rows[1].password
      email: 'bob@example.com'        # current.rows[1].email
      created: '2022-02-22T00:00:00Z' # current.rows[1].created

otherwise it records last_insert_id and rows_affected .

[`step key` or `current` or `previous`]:
  last_insert_id: 3 # current.last_insert_id
  rows_affected: 1  # current.rows_affected

Add comment with trace token to query for tracing

runners:
  db:
    dsn: mysql://dbuser:dbpass@hostname:3306/dbname
    trace: true

Support Databases

PostgreSQL:

runners:
  mydb: postgres://dbuser:dbpass@hostname:5432/dbname

runners:
  db: pg://dbuser:dbpass@hostname:5432/dbname

MySQL:

runners:
  testdb: mysql://dbuser:dbpass@hostname:3306/dbname

runners:
  db: my://dbuser:dbpass@hostname:3306/dbname

SQLite3:

runners:
  db: sqlite:///path/to/dbname.db

runners:
  local: sq://dbname.db

Cloud Spanner:

runners:
  testdb: spanner://test-project/test-instance/test-database

runners:
  db: sp://test-project/test-instance/test-database

CDP Runner: Control browser using Chrome DevTools Protocol (CDP)

Use cdp:// or chrome:// scheme to specify CDP Runner.

When the step is invoked, it controls browser via Chrome DevTools Protocol.

runners:
  cc: chrome://new
steps:
  -
    desc: Navigate, click and get h1 using CDP  # description of step
    cc:                                         # key to identify the runner. In this case, it is CDP Runner.
      actions:                                  # actions to control browser
        - navigate: https://pkg.go.dev/time
        - click: 'body > header > div.go-Header-inner > nav > div > ul > li:nth-child(2) > a'
        - waitVisible: 'body > footer'
        - text: 'h1'
  -
    test: |
      previous.text == 'Install the latest version of Go'

CDP Configuration Options

The CDP runner supports additional configuration options:

runners:
  cc:
    addr: chrome://new  # or cdp://new
    timeout: 120sec     # Timeout for each CDP action (default: 60s)
    flags:              # Chrome browser flags
      headless: true
      disable-gpu: true
      no-sandbox: true

Configuration parameters:

• addr: Chrome DevTools Protocol address. Use chrome://new or cdp://new to launch a new browser instance
• timeout: Timeout duration for each CDP action/step (e.g., "30s", "2m", "1m30s"). Default is 60 seconds
• flags: Chrome browser launch flags as key-value pairs

See testdata/book/cdp.yml.

Functions for action to control browser

attributes (aliases: getAttributes, attrs, getAttrs)

Get the element attributes for the first element node matching the selector (sel).

actions:
  - attributes:
      sel: "h1"
# record to current.attrs:

or

actions:
  - attributes: "h1"

click

Send a mouse click event to the first element node matching the selector (sel).

actions:
  - click:
      sel: "nav > div > a"

or

actions:
  - click: "nav > div > a"

doubleClick

Send a mouse double click event to the first element node matching the selector (sel).

actions:
  - doubleClick:
      sel: "nav > div > li"

or

actions:
  - doubleClick: "nav > div > li"

evaluate (aliases: eval)

Evaluate the Javascript expression (expr).

actions:
  - evaluate:
      expr: "document.querySelector(\"h1\").textContent = \"hello\""

or

actions:
  - evaluate: "document.querySelector(\"h1\").textContent = \"hello\""

fullHTML (aliases: getFullHTML, getHTML, html)

Get the full html of page.

actions:
  - fullHTML
# record to current.html:

innerHTML (aliases: getInnerHTML)

Get the inner html of the first element node matching the selector (sel).

actions:
  - innerHTML:
      sel: "h1"
# record to current.html:

or

actions:
  - innerHTML: "h1"

localStorage (aliases: getLocalStorage)

Get localStorage items.

actions:
  - localStorage:
      origin: "https://github.com"
# record to current.items:

or

actions:
  - localStorage: "https://github.com"

location (aliases: getLocation)

Get the document location.

actions:
  - location
# record to current.url:

navigate

Navigate the current frame to url page.

actions:
  - navigate:
      url: "https://pkg.go.dev/time"

or

actions:
  - navigate: "https://pkg.go.dev/time"

outerHTML (aliases: getOuterHTML)

Get the outer html of the first element node matching the selector (sel).

actions:
  - outerHTML:
      sel: "h1"
# record to current.html:

or

actions:
  - outerHTML: "h1"

screenshot (aliases: getScreenshot)

Take a full screenshot of the entire browser viewport.

actions:
  - screenshot
# record to current.png:

scroll (aliases: scrollIntoView)

Scroll the window to the first element node matching the selector (sel).

actions:
  - scroll:
      sel: "body > footer"

or

actions:
  - scroll: "body > footer"

sendKeys

Send keys (value) to the first element node matching the selector (sel).

actions:
  - sendKeys:
      sel: "input[name=username]"
      value: "k1lowxb@gmail.com"

sessionStorage (aliases: getSessionStorage)

Get sessionStorage items.

actions:
  - sessionStorage:
      origin: "https://github.com"
# record to current.items:

or

actions:
  - sessionStorage: "https://github.com"

setUploadFile (aliases: setUpload)

Set upload file (path) to the first element node matching the selector (sel).

actions:
  - setUploadFile:
      sel: "input[name=avator]"
      path: "/path/to/image.png"

setUserAgent (aliases: setUA, ua, userAgent)

Set the default User-Agent

actions:
  - setUserAgent:
      userAgent: "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36"

or

actions:
  - setUserAgent: "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36"

submit

Submit the parent form of the first element node matching the selector (sel).

actions:
  - submit:
      sel: "form.login"

or

actions:
  - submit: "form.login"

tabTo

Change current frame to the tab with the specified url.

actions:
  - tabTo:
      url: "https://pkg.go.dev/time"

or

actions:
  - tabTo: "https://pkg.go.dev/time"

text (aliases: getText)

Get the visible text of the first element node matching the selector (sel).

actions:
  - text:
      sel: "h1"
# record to current.text:

or

actions:
  - text: "h1"

textContent (aliases: getTextContent)

Get the text content of the first element node matching the selector (sel).

actions:
  - textContent:
      sel: "h1"
# record to current.text:

or

actions:
  - textContent: "h1"

title (aliases: getTitle)

Get the document title.

actions:
  - title
# record to current.title:

value (aliases: getValue)

Get the Javascript value field of the first element node matching the selector (sel).

actions:
  - value:
      sel: "input[name=address]"
# record to current.value:

or

actions:
  - value: "input[name=address]"

wait (aliases: sleep)

Wait for the specified time.

actions:
  - wait:
      time: "10sec"

or

actions:
  - wait: "10sec"

waitReady

Wait until the element matching the selector (sel) is ready.

actions:
  - waitReady:
      sel: "body > footer"

or

actions:
  - waitReady: "body > footer"

waitVisible

Wait until the element matching the selector (sel) is visible.

actions:
  - waitVisible:
      sel: "body > footer"

or

actions:
  - waitVisible: "body > footer"

SSH Runner: execute commands on a remote server connected via SSH

Use ssh:// scheme to specify SSH Runner.

When step is invoked, it executes commands on a remote server connected via SSH.

runners:
  sc: ssh://username@hostname:port
steps:
  -
    desc: 'execute `hostname`' # description of step
    sc:
      command: hostname

runners:
  sc:
    hostname: hostname
    user: username
    port: 22
    # host: myserver
    # sshConfig: path/to/ssh_config
    # keepSession: false
    # localForward: '33306:127.0.0.1:3306'
    # keyboardInteractive:
    #   - match: Username
    #     answer: k1low
    #   - match: OTP
    #     answer: ${MY_OTP}

See testdata/book/sshd.yml.

Structure of recorded responses

The response to the run command is always stdout and stderr.

[`step key` or `current` or `previous`]:
  stdout: 'hello world' # current.stdout
  stderr: ''            # current.stderr

Exec Runner: execute command

Note Exec runner requires run:exec scope to run.

The exec runner is a built-in runner, so there is no need to specify it in the runners: section.

It executes command using command:, stdin:, shell:, background:, liveOutput: and env:.

-
  exec:
    command: grep hello
    stdin: '{{ steps[3].res.rawBody }}'

-
  exec:
    command: echo $0
    shell: bash

background: set to true to run the command in the background.

-
  exec:
    command: kubectl port-forward svc/nginx 8080:80
    liveOutput: true

liveOutput: set to true to output the command output live.

-
  exec:
    command: |
      echo "Start heavy commands"
      sleep 5
      echo "Heavy command finished"
    liveOutput: true

env: sets additional environment variables for the command execution.

-
  exec:
    command: printenv MY_VAR
    env:
      MY_VAR: hello
      ANOTHER_VAR: "{{ vars.value }}"

See testdata/book/exec.yml.

Structure of recorded responses

The response to the run command is always stdout, stderr and exit_code.

[`step key` or `current` or `previous`]:
  stdout: 'hello world' # current.stdout
  stderr: ''            # current.stderr
  exit_code: 0          # current.exit_code

exec.shell:

Use shell: to define the shell and options to be used by the Exec runner.

Test Runner: test using recorded values

The test runner is a built-in runner, so there is no need to specify it in the runners: section.

It evaluates the conditional expression using the recorded values.

-
  test: steps[3].res.status == 200

The test runner can run in the same steps as the other runners.

Dump Runner: dump recorded values

The dump runner is a built-in runner, so there is no need to specify it in the runners: section.

It dumps the specified recorded values.

-
  dump: steps[4].rows

or

-
  dump:
    expr: steps[4].rows
    out: path/to/dump.out
    disableTrailingNewline: true # disable trailing newline. default is false
    disableMaskingSecrets: true  # disable masking secrets. default is false

The dump runner can run in the same steps as the other runners.

Include Runner: include other runbook

The include runner is a built-in runner, so there is no need to specify it in the runners: section.

Include runner reads and runs the runbook in the specified path.

Recorded values are nested.

-
  include: path/to/get_token.yml

It is also possible to override vars: of included runbook.

-
  include:
    path: path/to/login.yml
    vars:
      username: alice
      password: alicepass
-
  include:
    path: path/to/login.yml
    vars:
      username: bob
      password: bobpass

It is also possible to skip all test: sections in the included runbook.

-
  include:
    path: path/to/signup.yml
    skipTest: true

It is also possible to force all steps in the included runbook to run.

-
  include:
    path: path/to/signup.yml
    force: true

Bind Runner: bind variables

The bind runner is a built-in runner, so there is no need to specify it in the runners: section.

It bind runner binds any values with another key.

  -
    req:
      /users/k1low:
        get:
          body: null
  -
    bind:
      user_id: steps[0].res.body.data.id
  -
    dump: user_id

The bind runner can run in the same steps as the other runners.

Runner Runner: Define runner in the middle of steps.

The runner runner is a built-in runner, so there is no need to specify it in the runners: section.

It defines a runner in the middle of steps.

  -
    runner:
      sc: ssh://username@hostname:port
  -
    sc:
      command: hostname

The runner runner can not run in the same steps as the other runners.

Expression evaluation engine

runn has embedded expr-lang/expr as the evaluation engine for the expression.

See Language Definition.

Additional built-in functions

• urlencode ... url.QueryEscape
• bool ... cast.ToBool
• compare ... Compare two values ( func(x, y any, ignorePaths ...string) bool ). The optional ignorePaths argument is a list of jq syntax path expressions to ignore when comparing two values.
• diff ... Difference between two values ( func(x, y any, ignorePaths ...string) string ). The optional ignorePaths argument is a list of jq syntax path expressions to ignore when comparing two values.
• pick ... Returns same map type filtered by given keys left lo.PickByKeys.
• omit ... Returns same map type filtered by given keys excluded lo.OmitByKeys.
• merge ... Merges multiple maps from left to right lo.Assign.
• input ... prompter.Prompt
• intersect ... Find the intersection of two iterable values ( func(x, y any) any ).
• secret ... prompter.Password
• select ... Select from candidates. func(message string, candidates []string, default string) string
• basename ... filepath.Base
• time ... Converts the given string or number to time.Time{}.
• faker.* ... Generate fake data using Faker ).
• file ... Read the file as a string. Returns nil if it does not exist.
• jwt.* ... Generate and parse JSON Web Tokens (JWT) using the specified claims and signature algorithm. jwt.Sign, jwt.Parse.
  See testdata/book/http_bearer.yml for a complete example.
  Note: This function currently supports JWS (JSON Web Signature) only. JWE (JSON Web Encryption) is not supported.

Option

See https://pkg.go.dev/github.com/k1LoW/runn#Option

Example: Run as a test helper ( func T )

https://pkg.go.dev/github.com/k1LoW/runn#T

o, err := runn.Load("testdata/**/*.yml", runn.T(t))
if err != nil {
	t.Fatal(err)
}
if err := o.RunN(ctx); err != nil {
	t.Fatal(err)
}

Example: Add custom function ( func Func )

https://pkg.go.dev/github.com/k1LoW/runn#Func

desc: Test using GitHub
runners:
  req:
    endpoint: https://github.com
steps:
  -
    req:
      /search?l={{ urlencode('C++') }}&q=runn&type=Repositories:
        get:
          body:
            application/json:
              null
    test: 'steps[0].res.status == 200'

o, err := runn.Load("testdata/**/*.yml", runn.Func("urlencode", url.QueryEscape))
if err != nil {
	t.Fatal(err)
}
if err := o.RunN(ctx); err != nil {
	t.Fatal(err)
}

Scope

runn requires explicit specification of scope for some features.

runn has the following scopes.

To specify scopes, using the --scopes option or the environment variable RUNN_SCOPES.

$ runn run path/to/**/*.yml --scopes read:parent,read:remote

$ env RUNN_SCOPES=read:parent,read:remote runn run path/to/**/*.yml

Also, runn.Scopes can be used in the code

o, err := runn.Load("path/to/**/*.yml", runn.Scopes(runn.AllowReadParent, runn.AllowReadRemote))
if err != nil {
	t.Fatal(err)
}
if err := o.RunN(ctx); err != nil {
	t.Fatal(err)
}

To disable scope, can use !read:* instead of read:*

$ runn run path/to/**/*.yml --scopes '!read:parent'

Filter runbooks to be executed by the environment variable RUNN_RUN

Run only runbooks matching the filename "login".

$ env RUNN_RUN=login go test ./... -run TestRouter

Measure elapsed time as profile

opts := []runn.Option{
	runn.T(t),
	runn.Book("testdata/books/login.yml"),
	runn.Profile(true),
}
o, err := runn.New(opts...)
if err != nil {
	t.Fatal(err)
}
if err := o.Run(ctx); err != nil {
	t.Fatal(err)
}
f, err := os.Open("profile.json")
if err != nil {
	t.Fatal(err)
}
if err := o.DumpProfile(f); err != nil {
	t.Fatal(err)
}

or

$ runn run testdata/books/login.yml --profile

The runbook run profile can be read with runn rprof command.

$ runn rprof runn.prof
  runbook[login site](t/b/login.yml)           2995.72ms
    steps[0].req                                747.67ms
    steps[1].req                                185.69ms
    steps[2].req                                192.65ms
    steps[3].req                                188.23ms
    steps[4].req                                569.53ms
    steps[5].req                                299.88ms
    steps[6].test                                 0.14ms
    steps[7].include                            620.88ms
      runbook[include](t/b/login_include.yml)   605.56ms
        steps[0].req                            605.54ms
    steps[8].req                                190.92ms
  [total]                                      2995.84ms

Capture runbook runs

opts := []runn.Option{
	runn.T(t),
	runn.Capture(capture.Runbook("path/to/dir")),
}
o, err := runn.Load("testdata/books/**/*.yml", opts...)
if err != nil {
	t.Fatal(err)
}
if err := o.RunN(ctx); err != nil {
	t.Fatal(err)
}

or

$ runn run path/to/**/*.yml --capture path/to/dir

Load test using runbooks

You can use the runn loadt command for load testing using runbooks.

$ runn loadt --load-concurrent 2 --max-rps 0 path/to/*.yml

Number of runbooks per RunN....: 15
Warm up time (--warm-up).......: 5s
Duration (--duration)..........: 10s
Concurrent (--load-concurrent).: 2
Max RunN per second (--max-rps): 0

Total..........................: 12
Succeeded......................: 12
Failed.........................: 0
Error rate.....................: 0%
RunN per seconds...............: 1.2
Latency .......................: max=1,835.1ms min=1,451.3ms avg=1,627.8ms med=1,619.8ms p(90)=1,741.5ms p(99)=1,788.4ms

It also checks the results of the load test with the --threshold option. If the condition is not met, it returns exit status 1.

$ runn loadt --load-concurrent 2 --max-rps 0 --threshold 'error_rate < 10' path/to/*.yml

Number of runbooks per RunN...: 15
Warm up time (--warm-up)......: 5s
Duration (--duration).........: 10s
Concurrent (--load-concurrent): 2

Total.........................: 13
Succeeded.....................: 12
Failed........................: 1
Error rate....................: 7.6%
RunN per seconds..............: 1.3
Latency ......................: max=1,790.2ms min=95.0ms avg=1,541.4ms med=1,640.4ms p(90)=1,749.7ms p(99)=1,786.5ms

Error: (error_rate < 10) is not true
error_rate < 10
├── error_rate => 14.285714285714285
└── 10 => 10

Variables for threshold

Install

As a CLI tool

deb:

$ export RUNN_VERSION=X.X.X
$ curl -o runn.deb -L https://github.com/k1LoW/runn/releases/download/v$RUNN_VERSION/runn_$RUNN_VERSION-1_amd64.deb
$ dpkg -i runn.deb

RPM:

$ export RUNN_VERSION=X.X.X
$ yum install https://github.com/k1LoW/runn/releases/download/v$RUNN_VERSION/runn_$RUNN_VERSION-1_amd64.rpm

apk:

$ export RUNN_VERSION=X.X.X
$ curl -o runn.apk -L https://github.com/k1LoW/runn/releases/download/v$RUNN_VERSION/runn_$RUNN_VERSION-1_amd64.apk
$ apk add runn.apk

homebrew tap:

$ brew install k1LoW/tap/runn

aqua:

$ aqua g -i k1LoW/runn

manually:

Download binary from releases page

docker:

$ docker container run -it --rm --name runn -v $PWD:/books ghcr.io/k1low/runn:latest list /books/*.yml

go install:

$ go install github.com/k1LoW/runn/cmd/runn@latest

As a test helper

$ go get github.com/k1LoW/runn

Alternatives

• zoncoen/scenarigo: An end-to-end scenario testing tool for HTTP/gRPC server.

References

• zoncoen/scenarigo: An end-to-end scenario testing tool for HTTP/gRPC server.
• fullstorydev/grpcurl: Like cURL, but for gRPC: Command-line tool for interacting with gRPC servers
• ktr0731/evans: Evans: more expressive universal gRPC client

License

• MIT License
  • Include logo as well as source code.
  • Only logo license can be selected CC BY 4.0.
  • Also, if there is no alteration to the logo and it is used for technical information about runn, I would not say anything if the copyright notice is omitted.