mapreduce

mrkit-go

mrkit-go is a MapReduce toolkit in Go with a config-driven batch runner.

Primary path for new users:

• define source / transform / sink in JSON
• run ./cmd/batch with that config
• support MySQL/Redis source and sink combinations

Quickstart (Config-Driven)

1) Prepare demo MySQL source data

MYSQL_HOST=localhost MYSQL_PORT=3306 MYSQL_USER=root MYSQL_PASSWORD=123456 \
MYSQL_DB=mysql SOURCE_TABLE=source_events TARGET_TABLE=agg_results \
ROWS=5000 KEY_MOD=100 \
go run ./cmd/batch -mode prepare

2) Validate config schema

go run ./cmd/batch -check -config example/batch-minimal/mysqlbatch/flow.mysql.count.json

Expected output:

config check pass

3) Run a flow

go run ./cmd/batch -config example/batch-minimal/mysqlbatch/flow.mysql.count.json

Cross-DB examples:

# mysql -> redis
go run ./cmd/batch -config example/batch-minimal/mysqlbatch/flow.mysql_to_redis.count.json

# prepare redis source keys for redis->* examples (db0/event:*)
sed "s/\"db\": 1/\"db\": 0/; s/\"key_prefix\": \"mr:count:\"/\"key_prefix\": \"event:\"/; s/\"value_field\": \"metric_sum\"/\"value_field\": \"metric\"/" \
  example/batch-minimal/mysqlbatch/flow.mysql_to_redis.count.json > /tmp/flow.mysql_to_redis.seed_event.json
go run ./cmd/batch -config /tmp/flow.mysql_to_redis.seed_event.json

# redis -> mysql
go run ./cmd/batch -config example/batch-minimal/redisbatch/flow.redis_to_mysql.count.json

# redis -> redis
go run ./cmd/batch -config example/batch-minimal/redisbatch/flow.redis_to_redis.count.json

Expected log includes:

flow done

Recommended Structure

• source: mysql or redis extraction
• transform: builtin (count / minmax / topN) or mapreduce plugin
• sink: mysql or redis write-back

For built-in transforms, no prebuilt .so is required.

Documentation Map

• Config-driven schema, production template, and rerun guidance: docs/config-driven-flow.md
• Built-in transforms and plugin mode: docs/transforms.md
• Go library usage (batch.RunPipeline): docs/library-usage.md
• Benchmark usage: docs/benchmark.md
• Legacy MapReduce entrypoints (cmd/main.go, cmd/master.go, cmd/worker.go): docs/legacy-mapreduce.md
• Minimal end-to-end examples: example/batch-minimal/README.md

Contributions

Pull requests are always welcome.