bitfan

Bitfan

Bitfan is an open source data processing pipeline.

Install

Download binary

linux, windows, osx available here : https://github.com/vjeantet/bitfan/releases

Or compile from sources

$ go get -u github.com/vjeantet/bitfan

Run

Example with a remote configuration file which ingest data from stdin and output a tranformation to stdout.

$ bitfan run https://raw.githubusercontent.com/vjeantet/bitfan/master/examples.d/simple.conf

copy/paste this in your console

127.0.0.1 - - [11/Dec/2013:00:01:45 -0800] "GET /xampp/status.php HTTP/1.1" 200 3891 "http://cadenza/xampp/navi.php" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0"

Other commands

type bitfan help to display usage information

Usage:
  bitfan [flags]
  bitfan [command]

Available Commands:
  doc         Display documentation about plugins
  list        List running pipelines
  run         Run bitfan
  service     Install and manage bitfan service
  start       Start a new pipeline in a running bitfan
  stop        Stop a running pipeline
  test        Test configurations (files, url, directories)
  version     Display version informations

Flags:
  -f, --config string       Load the Logstash config from a file a directory or a url
  -t, --configtest          Test config file or directory
      --debug               Increase verbosity to the last level (trace), more verbose.
  -e, --eval string         Use the given string as the configuration data.
  -w, --filterworkers int   number of workers (default 4)
  -h, --help                help for bitfan
  -l, --log string          Log to a given path. Default is to log to stdout.
      --settings string     Set the directory containing the bitfan.toml settings (default "current dir, then ~/.bitfan/ then /etc/bitfan/")
      --verbose             Increase verbosity to the first level (info), less verbose.
  -V, --version             Display version info.

Use "bitfan [command] --help" for more information about a command.

logstash flags works as well -f, -e, --configtest, ...

TODO

• parse logstash config file
• support command line flags "à la logstash"
• generic input support
• generic filter support
• generic output support
• configuration condition (if else) support
• dynamic %{field.key} support in config file
• gracefully stop
• gracefully start
• name all contributors, imported packages, similar projects
• use remote configuration file
• include local and remote files from configuration files
• codec support
• log to file
• plugins autodocumentation
• install bitfan as a system daemon / service
• list currently runnnung pipelines
• start new pipelines in a running instance
• stop a pipeline without stopping other
• import external configuration from configuration (use)
• dispatch message to another configuration from configuration (fork)

Supported inputs, filters and outputs in config file

type bitfan doc to list all available plugins

INPUT bitfan doc --type input

type bitfan doc pluginname to get more information about plugin configuration and usage

FILTER bitfan doc --type filter

type bitfan doc pluginname to get more information about plugin configuration and usage

OUTPUT bitfan doc --type output

type bitfan doc pluginname to get more information about plugin configuration and usage

Similar projets in go

• tsaikd/gogstash - Logstash like, written in golang
• packetzoom/logzoom - A lightweight replacement for logstash indexer in Go
• hailocab/logslam - A lightweight lumberjack protocol compliant logstash indexer

Credits

logo "hand fan" by lastspark from the Noun Project

Contributors

• @vjeantet - Valere JEANTET
• @mirdhyn - Merlin Gaillard
• @AlexAkulov - Alexander AKULOV

Used packages

• mitchellh/mapstructure

• ChimeraCoder/anaconda

• etrepat/postman/watch

• go-fsnotify/fsnotify

• hpcloud/tail

• nu7hatch/gouuid

• parnurzeal/gorequest

• vjeantet/go.enmime

• Knetic/govaluate

• vjeantet/grok

• vjeantet/jodaTime

• streadway/amqp

• ShowMax/go-fqdn

• oschwald/geoip2-golang

• gopkg.in/fsnotify.v1

• gopkg.in/go-playground/validator.v8

• gopkg.in/mgo.v2

• gopkg.in/olivere/elastic.v2

• gopkg.in/olivere/elastic.v3

• gopkg.in/alexcesaro/statsd.v2

• kardianos/govendor

• spf13/cobra

• bbuck/go-lexer

• k0kubun/pp (debug)