rush

rush

rush -- parallelly execute shell commands.

rush is a tool similar to GNU parallel and gargs. rush borrows some idea from them and has some unique features, e.g., more advanced embeded strings replacement than parallel.

• Features
• Performance
• Examples
• Usage
• Acknowledgements
• Contact
• License

Features

Major:

• record delimiter (-D, default \n), records sending to every command (-n, default 1), and field delimiter (-d, default \s+).
• keep output order, may use temporary file
• support timeout and retry
• support exit on fist error(s)
• support continue, save status after capturing ctrl+c
• Replacement strings (like GNU parallel):
  • {#}, job number
  • {}, full line
  • support positional replacement strings: {n}
    • nth field in delimiter-delimited data
    • nth matches of regular expression
  • direcotry and file
    • {/}, dirname ({//} in GNU parallel)
    • {%}, basename ({/} in GNU parallel)
    • {.}, remove the last extension
    • {:}, remove any extension (GNU parallel does not have)
  • combinations:
    • {%.}, {%:}, basename without extension
    • {n.}, {n/}, manipulate nth field
• awk -v like defined variables
• appropriate quoting

Minor:

• logging
• dry run
• trim arguments
• verbose

Performance

Performance of rush is similar to gargs, and they are both slightly faster than parallel (Perl). See on release page.

Examples

1. Simple run, quoting is not necessary

    # seq 1 3 | rush 'echo {}'
    $ seq 1 3 | rush echo {}
    3
    1
    2
   

2. Read data from file (-i)

    $ rush echo {} -i data.txt
   

3. Keep output order (-k)

    $ seq 1 3 | rush 'echo {}' -k
    1
    2
    3
   

4. Timeout (-t)

    $ time seq 1 | rush 'sleep 2; echo {}' -t 1
    [ERRO] run command #1: sleep 2; echo 1: time out
   
    real    0m1.010s
    user    0m0.005s
    sys     0m0.007s
   

5. Retry (-r)

    $ seq 1 | rush 'python unexisted_script.py' -r 1
    python: can't open file 'unexisted_script.py': [Errno 2] No such file or directory
    [WARN] wait command: python unexisted_script.py: exit status 2
    python: can't open file 'unexisted_script.py': [Errno 2] No such file or directory
    [ERRO] wait command: python unexisted_script.py: exit status 2
   

6. Directory ({/}) and basename ({%})

    $ echo dir/file.txt.gz | rush 'echo {/} {%}'
    dir file.txt.gz
   

7. Get basename, and remove last ({.}) or any ({:}) extension

    $ echo dir/file.txt.gz | rush 'echo {%.} {%:}'
    file.txt file
   

8. Job ID, combine fields index and other replacement strings

    $ echo 123 file.txt | rush 'echo job {#}: {2} {2.} {1}'
    job 1: file.txt file 123
   

9. Custom field delimiter (-d)

    $ echo a=b=c | rush 'echo {1} {2} {3}' -d =
    a b c
   

10. Send multi-lines to every command (-n)

     $ seq 1 5 | rush -j 3 'echo {1}' -n 2 -k
     1
     3
     5
    

11. Custom record delimiter (-D)

     $ echo -ne ">seq1\nactg\n>seq2\nAAAA\n>seq3\nCCCC"
     >seq1
     actg
     >seq2
     AAAA
     >seq3
     CCCC
    
     $ echo -ne ">seq1\nactg\n>seq2\nAAAA\n>seq3\nCCCC" | rush -D ">" 'echo FASTA record {#}: name: {1} sequence: {2}' -k -d "\n"
     FASTA record 1: name: sequence:
     FASTA record 2: name: seq1 sequence: actg
     FASTA record 3: name: seq2 sequence: AAAA
     FASTA record 4: name: seq3 sequence: CCCC
    

Usage

rush -- parallelly execute shell commands

Version: 0.0.3

Author: Wei Shen <shenwei356@gmail.com>

Source code: https://github.com/shenwei356/rush

Usage:
  rush [flags] [command] [args of command...]

Examples:
  1. simple run  : seq 1 10 | rush echo {}  # quoting is not necessary
  2. keep order  : seq 1 10 | rush 'echo {}' -k
  3. with timeout: seq 1 | rush 'sleep 2; echo {}' -t 1
  4. retry       : seq 1 | rush 'python script.py' -r 3
  5. basename    : echo dir/file.txt.gz | rush 'echo {%}'　　# file.txt.gz
  6. dirname     : echo dir/file.txt.gz | rush 'echo {/}'    # dir
  7. basename without last extension
                 : echo dir/file.txt.gz | rush 'echo {%.}'   # file.txt
  8. basename without last extension
                 : echo dir/file.txt.gz | rush 'echo {%:}'   # file
  9. job ID, combine fields and other replacement string
                 : echo 123 file.txt | rush 'echo job {#}: {2} {2.} {1}'
                 # job 1: file.txt file 123
  More examples: https://github.com/shenwei356/rush

Flags:
  -v, --assign stringSlice        assign the value val to the variable var (format: var=val)
      --dry-run                   print command but not run
  -d, --field-delimiter string    field delimiter in records (default "\s+")
  -i, --infile stringSlice        input data file
  -j, --jobs int                  run n jobs in parallel (default 4)
  -k, --keep-order                keep output in order of input
  -n, --nrecords int              number of records sent to a command (default 1)
  -o, --out-file string           out file ("-" for stdout) (default "-")
  -D, --record-delimiter string   record delimiter (default is "\n") (default "
")
  -r, --retries int               maximum retries
      --retry-interval int        retry interval (unit: second)
  -e, --stop-on-error             stop all processes on first error(s)
  -t, --timeout int               timeout of a command (unit: second, 0 for no timeout)
      --trim string               trim white space in input (available values: "l" for left, "r" for right, "lr", "rl", "b" for both side)
      --verbose                   print verbose information
  -V, --version                   print version information and check for update

Acknowledgements

Specially thank @brentp and his gargs, from which rush borrows lots of ideas.

Contact

Create an issue to report bugs, propose new functions or ask for help.

License

MIT License