rush

rush

rush -- parallelly execute shell commands.

rush is a tool similar to GNU parallel and gargs. rush borrows some idea from them and has some unique features, e.g., more advanced embeded strings replacement than parallel.

Source code: https://github.com/shenwei356/rush

Latest version:

Features

Major:

• record delimiter (-D, default \n), records sending to every command (-n, default 1), and field delimiter (-d, default \s+).
• keep output order, may use temporary file
• support timeout and retry
• exit on fist error
• support continue, save status after capturing ctrl+c
• support positional replacement strings: {n}
  • columns in delimiter-delimited data
  • matches of regular expression
• GNU parallel like replacement strings:
  • {#}, job number
  • {}, full line
  • {.}, input line without the last extension
  • {:}, input line without any extension (GNU parallel does not have)
  • {/}, dirname ({//} in GNU parallel)
  • {%}, basename ({/} in GNU parallel)
  • possible combinations:
    • {%.}, {%,}
    • {n.}, {n/} ...
• awk -v like defined variables
• appropriate quoting

Minor:

• logging
• dry run
• trim arguments
• verbose

Workflow

1. read data from STDIN (default) or files (-i)
2. split N (-n, default 1) records by record delimiter (-D, default \n) into Chunks
3. create commands with splitted data
   1. optinally split input data by field delimiter (-d, default \s+)
   2. replaces placeholders in command (by joining arguments) with input data
4. run commands in parallel (max jobs: -j, default #. CPUs)
   1. optionally (--dry-run) print command and not run
   2. retry if fail to run, give up when reached the max retry times (-r)
   3. cancel if time out (-t)
   4. optionally (-e) stop all commands and exit on first error
   5. output failed comands to file (failed.txt), so we can redo them (rush -i failed.txt)
   6. optionally (-c) save finished commands to file (finished.txt), so we can ignore them when run in "continue" mode (-c)
5. show STDOUT of commands to STDOUT, optionally (-k) keep order according to the input data

Performance

See on release page

Acknowledgements

Specially thank @brentp and his gargs, from which rush borrows lots of ideas.

Contact

Create an issue to report bugs, propose new functions or ask for help.

License

MIT License