rush

rush

rush -- parallelly execute shell commands.

rush is a tool similar to GNU parallel and gargs. rush borrows some idea from them and has some unique features, e.g., more advanced embeded strings replacement than parallel.

• Features
• Performance
• Examples
• Usage
• Installation
• Download
• Acknowledgements
• Contact
• License

Features

Major:

• record delimiter (-D, default \n), records sending to every command (-n, default 1), and field delimiter (-d, default \s+).
• keep output order, may use temporary file
• support timeout and retry
• support exit on fist error(s)
• support continue, save status after capturing ctrl+c
• Replacement strings (like GNU parallel):
  • {#}, job ID
  • {}, full data
  • support positional replacement strings: {n}
    • nth field in delimiter-delimited data
    • nth matche of regular expression
  • direcotry and file
    • {/}, dirname ({//} in GNU parallel)
    • {%}, basename ({/} in GNU parallel)
    • {.}, remove the last extension
    • {:}, remove any extension (GNU parallel does not have)
  • combinations:
    • {%.}, {%:}, basename without extension
    • {n.}, {n/}, manipulate nth field
• awk -v like defined variables
• appropriate quoting

Minor:

• logging
• dry run
• trim arguments
• verbose

Performance

Performance of rush is similar to gargs, and they are both slightly faster than parallel (Perl). See on release page.

Examples

1. Simple run, quoting is not necessary

    # seq 1 3 | rush 'echo {}'
    $ seq 1 3 | rush echo {}
    3
    1
    2
   

2. Read data from file (-i)

    $ rush echo {} -i data1.txt -i data2.txt
   

3. Keep output order (-k)

    $ seq 1 3 | rush 'echo {}' -k
    1
    2
    3
   

4. Timeout (-t)

    $ time seq 1 | rush 'sleep 2; echo {}' -t 1
    [ERRO] run command #1: sleep 2; echo 1: time out
   
    real    0m1.010s
    user    0m0.005s
    sys     0m0.007s
   

5. Retry (-r)

    $ seq 1 | rush 'python unexisted_script.py' -r 1
    python: can't open file 'unexisted_script.py': [Errno 2] No such file or directory
    [WARN] wait command: python unexisted_script.py: exit status 2
    python: can't open file 'unexisted_script.py': [Errno 2] No such file or directory
    [ERRO] wait command: python unexisted_script.py: exit status 2
   

6. Dirname ({/}) and basename ({%})

    $ echo dir/file.txt.gz | rush 'echo {/} {%}'
    dir file.txt.gz
   

7. Get basename, and remove last ({.}) or any ({:}) extension

    $ echo dir/file.txt.gz | rush 'echo {%.} {%:}'
    file.txt file
   

8. Job ID, combine fields index and other replacement strings

    $ echo 123 file.txt | rush 'echo job {#}: {2} {2.}'
    job 1: file.txt file
   

9. Custom field delimiter (-d)

    $ echo a=b=c | rush 'echo {1} {2} {3}' -d =
    a b c
   

10. Send multi-lines to every command (-n)

     $ seq 1 5 | rush -j 3 'echo {1}' -n 2 -k
     1
     3
     5
    

11. Custom record delimiter (-D)

     $ echo -ne ">seq1\nactg\n>seq2\nAAAA\n>seq3\nCCCC"
     >seq1
     actg
     >seq2
     AAAA
     >seq3
     CCCC
    
     $ echo -ne ">seq1\nactg\n>seq2\nAAAA\n>seq3\nCCCC" | rush -D ">" 'echo FASTA record {#}: name: {1} sequence: {2}' -k -d "\n"
     FASTA record 2: name: seq1 sequence: actg
     FASTA record 3: name: seq2 sequence: AAAA
     FASTA record 4: name: seq3 sequence: CCCC
    

12. Assign value to variable, like awk -v (-v)

     $ seq 1  | rush 'echo Hello, {fname} {lname}!' -v fname=Wei -v lname=Shen
     Hello, Wei Shen!
    
     $ for var in a b; do \
     $   seq 1 3 | rush -k -v var=$var 'echo var: {var}, data: {}'; \
     $ done
     var: a, data: 1
     var: a, data: 2
     var: a, data: 3
     var: b, data: 1
     var: b, data: 2
     var: b, data: 3
    

Usage

rush -- parallelly execute shell commands

Version: 0.0.4

Author: Wei Shen <shenwei356@gmail.com>

Source code: https://github.com/shenwei356/rush

Usage:
  rush [flags] [command] [args of command...]

Examples:
  1. simple run, quoting is not necessary
      $ seq 1 10 | rush echo {}
  2. keep order
      $ seq 1 10 | rush 'echo {}' -k
  3. timeout
      $ seq 1 | rush 'sleep 2; echo {}' -t 1
  4. retry
      $ seq 1 | rush 'python script.py' -r 3
  5. dirname & basename
      $ echo dir/file.txt.gz | rush 'echo {/} {%}'
      dir file.txt.gz
  6. basename without last or any extension
      $ echo dir/file.txt.gz | rush 'echo {%.} {%:}'
      file.txt file
  7. job ID, combine fields and other replacement strings
      $ echo 123 file.txt | rush 'echo job {#}: {2} {2.}'
      job 1: file.txt file
  8. custom field delimiter
      $ echo a=b=c | rush 'echo {1} {2} {3}' -d =
      a b c
  9. assign value to variable, like "awk -v"
      $ seq 1 | rush 'echo Hello, {fname} {lname}!' -v fname=Wei -v lname=Shen
      Hello, Wei Shen!
  More examples: https://github.com/shenwei356/rush

Flags:
  -v, --assign stringSlice        assign the value val to the variable var (format: var=val)
      --dry-run                   print command but not run
  -d, --field-delimiter string    field delimiter in records, support regular expression (default "\s+")
  -i, --infile stringSlice        input data file, multi-values supported
  -j, --jobs int                  run n jobs in parallel (default value depends on your device) (default 4)
  -k, --keep-order                keep output in order of input
  -n, --nrecords int              number of records sent to a command (default 1)
  -o, --out-file string           out file ("-" for stdout) (default "-")
  -D, --record-delimiter string   record delimiter, supports regular expression (default is "\n") (default "
")
  -r, --retries int               maximum retries (default 0)
      --retry-interval int        retry interval (unit: second) (default 0)
  -e, --stop-on-error             stop all processes on first error(s)
  -t, --timeout int               timeout of a command (unit: second, 0 for no timeout) (default 0)
      --trim string               trim white space in input (available values: "l" for left, "r" for right, "lr", "rl", "b" for both side)
      --verbose                   print verbose information
  -V, --version                   print version information and check for update

Installation

rush is implemented in Go programming language, executable binary files for most popular operating systems are freely available in release page.

Just download compressed executable file of your operating system, and uncompress it with tar -zxvf xxx.tar.gz command or other tools. And then:

1. For Unix-like systems

   1. If you have root privilege simply copy it to /usr/local/bin:

       sudo cp rush /usr/local/bin/
      

   2. Or add the current directory of the executable file to environment variable PATH:

       echo export PATH=\"$(pwd)\":\$PATH >> ~/.bashrc
       source ~/.bashrc
      

2. For windows, just copy rush.exe to C:\WINDOWS\system32.

For Go developer, just one command:

go get -u github.com/shenwei356/rush/rush

Download

rush v0.0.4

OS	Arch	File	Download Count
Linux	32-bit	rush_linux_386.tar.gz
Linux	64-bit	rush_linux_amd64.tar.gz
OS X	32-bit	rush_darwin_386.tar.gz
OS X	64-bit	rush_darwin_amd64.tar.gz
Windows	32-bit	rush_windows_386.exe.tar.gz
Windows	64-bit	rush_windows_amd64.exe.tar.gz

Acknowledgements

Specially thank @brentp and his gargs, from which rush borrows lots of ideas.

Contact

Create an issue to report bugs, propose new functions or ask for help.

License

MIT License