GolangWorkerPerformance

module
v0.0.0-...-05a0e05 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 23, 2019 License: MIT

README

GolangWorkerPerformance

Clone
git clone https://gitlab.com/SenseyeFastExamples/GolangWorkerPerformance.git ~/go/src/workerperformance

Протестувати ефективність MongoDB & MySQL & Redis & ScyllaDB & Aerospike для збереження та оновлення підпищиків

Дано

####### алгоритм:

  • Є воркери які роблять запити в БД, отримуть підпищиків
  • Відправляють підпищикам повідомлення
  • Оновлюють в БД дату після якої повторно можна відправити підпищику повідомлення

####### структура підпищика з полями

type Subscriber struct {
	ID          uint32
	NotifyAfter uint32
	Partner     uint32
	Stream      uint32
	Widget      uint32
	WorkerID    uint8
	Timezone    uint8
	OS          uint8
	Browser     uint8
	Country     string
	Language    string
	Endpoint    string
}

####### параметри для налаштування

  • A — кількість підпищиків в БД
  • B — кількість воркерів
  • C — кількість підпищиків за один запит до БД
  • D — кількість часових поясів
  • E — час для відправлення повідемлень
  • F — час перед повторним запитом до БД
  • G — кількість ітерацій
Результат:

Провести тести для різних комбінацій параметрів та відобразити на графіку зміни часу запитів до БД

Resources:
Result
Match benchmark
Database documents ns/op B/op allocs/op
MySQL 100 000 2 098 259 963 557 10 526
MySQL 1 000 000 74 543 949 4 377 291 104 906
MongoDB 100 000 10 630 428 5 187 138 42 678
MongoDB 1 000 000 76 433 153 49 004 393 428 886
Redis 100 000 962 512 212 295 2 494
Redis 1 000 000 8 277 857 2 167 039 25 002
Dynomite Redis 100 000 1 179 160 213 251 2 504
Dynomite Redis 1 000 000 9 143 460 2 179 032 25 093
ScyllaDB 100 000 7 011 458 1 317 351 4 530
ScyllaDB 1 000 000 38 801 484 5 888 443 32 820
Match for MongoDB with buffer, short and full strcture
Format documents ns/op B/op allocs/op
short 100 000 12 954 067 6 372 560 48 324
short 1 000 000 71 101 093 48 847 126 344 353
full 100 000 19 312 950 9 337 833 86 108
full 1 000 000 108 516 790 70 047 797 616 357
Worker match & update
Database COUNT(*) match 0-150 ms % match 150-250 ms % match 250+ ms % match min s match max s match avg s documents processed N
MongoDB 100 000 90.00 9.00 0.00 0.00 0.20 0.02 70557 200
MongoDB 1 000 000 79.00 2.00 18.00 0.03 0.83 0.16 708378 200
MongoDB 10 000 000 58.00 16.00 25.00 0.10 0.83 0.23 1600000 200
Redis 100 000 100.00 0.00 0.00 0.00 0.03 0.01 41000 200
Redis 1 000 000 100.00 0.00 0.00 0.01 0.08 0.03 416550 200
Redis 10 000 000 57.00 21.00 22.00 0.03 0.38 0.17 1600000 200
ScyllaDB 100 000 100.00 0.00 0.00 0.00 0.10 0.02 71453 200
ScyllaDB 1 000 000 0.00 2.00 97.00 0.17 3.23 1.29 1567325 200
ScyllaDB 10 000 000
Database COUNT(*) update 0-500 ms % update 500-1000 ms % update 1000+ ms % update min s update max s update avg s documents processed N
MongoDB 100 000 100.00 0.00 0.00 0.01 0.23 0.03 70557 200
MongoDB 1 000 000 82.00 9.00 8.00 0.05 1.12 0.23 708378 200
MongoDB 10 000 000 77.00 16.00 7.00 0.19 1.15 0.39 1600000 200
Redis 100 000 100.00 0.00 0.00 0.00 0.01 0.00 41000 200
Redis 1 000 000 100.00 0.00 0.00 0.00 0.02 0.01 416550 200
Redis 10 000 000 100.00 0.00 0.00 0.01 0.08 0.03 1600000 200
ScyllaDB 100 000 90.00 0.00 10.00 0.01 7.07 0.69 71453 200
ScyllaDB 1 000 000 0.00 0.00 100.00 2.52 22.60 10.87 39990 6
ScyllaDB 10 000 000 
mongodb-fixtures-filler -reset -n=100000 -index
mongodb-worker -batch=8000 -notify_time=5 -rest_time=2 -update_time=50 -n=10

mongodb-fixtures-filler -reset -n=1000000 -index
mongodb-worker -batch=8000 -notify_time=5 -rest_time=2 -update_time=50 -n=10

mongodb-fixtures-filler -reset -n=10000000 -index
mongodb-worker -batch=8000 -notify_time=5 -rest_time=2 -update_time=450 -n=10

redis-fixtures-filler -reset -n=100000
redis-worker -batch=8000 -notify_time=5 -rest_time=2 -update_time=50 -n=10

redis-fixtures-filler -reset -n=1000000
redis-worker -batch=8000 -notify_time=5 -rest_time=2 -update_time=50 -n=10

redis-fixtures-filler -reset -n=10000000
redis-worker -batch=8000 -notify_time=5 -rest_time=2 -update_time=450 -n=10

scylladb-fixtures-filler -reset -n=100000 -index
scylladb-worker -batch=8000 -notify_time=5 -rest_time=2 -update_time=50 -n=10

scylladb-fixtures-filler -reset -n=1000000 -index
scylladb-worker -batch=8000 -notify_time=5 -rest_time=2 -update_time=50 -n=10
On DigitalOcean
Compute-optimized 32 CPUs, 64 GB Memory / 400 GB Disk / AMS3 - Ubuntu 18.04 (LTS) x64
3 node in Scylla cluster

processors count : 32
models : [ Intel(R) Xeon(R) Platinum 8168 CPU @ 2.70GHz 
hypervisors  kvm 
filesystems EXT4

####### prepare for 1e6

# first
scylladb-fixtures-filler -reset -n=400000
# command complete by 66678340102 nanoseconds

# parallel
scylladb-fixtures-filler -n=300000 -shift=400000
# command complete by 48340913305 nanoseconds
scylladb-fixtures-filler -n=300000 -shift=700000
# command complete by 45319268895 nanoseconds

# index
scylladb-fixtures-filler -index
# command complete by 2618557231 nanoseconds

####### run worker

scylladb-worker -batch=8000 -notify_time=5 -rest_time=2 -update_time=50 -n=10
# worker complete by 77882508087 nanoseconds
Database COUNT(*) match 0-150 ms % match 150-250 ms % match 250+ ms % match min s match max s match avg s documents processed N
ScyllaDB 1 000 000 75.00 6.00 18.00 0.03 0.54 0.13 758396 200
Database COUNT(*) update 0-500 ms % update 500-1000 ms % update 1000+ ms % update min s update max s update avg s documents processed N
ScyllaDB 1 000 000 66.00 0.00 33.00 0.02 10.77 1.66 758396 200

####### prepare for 1e7

# first
scylladb-fixtures-filler -reset -n=4000000
# command complete by 630058809980 nanoseconds

# parallel
scylladb-fixtures-filler -n=3000000 -shift=4000000
# command complete by 490291884582 nanoseconds
scylladb-fixtures-filler -n=3000000 -shift=7000000
# command complete by 463316731873 nanoseconds

# index
scylladb-fixtures-filler -index
# command complete by 9916748807 nanoseconds

####### run worker

scylladb-worker -batch=8000 -notify_time=5 -rest_time=2 -update_time=450 -n=10
# worker complete by 78840690374 nanoseconds
scylladb-worker -batch=8000 -notify_time=5 -rest_time=2 -update_time=450 -n=100
# worker complete by 821244125712 nanoseconds
Database COUNT(*) match 0-150 ms % match 150-250 ms % match 250+ ms % match min s match max s match avg s documents processed N
ScyllaDB 10 000 000 63.00 22.00 14.00 0.07 0.57 0.17 1600000 200
ScyllaDB 10 000 000 13.00 10.00 75.00 0.08 3.63 0.45 14600502 1889
Database COUNT(*) update 0-500 ms % update 500-1000 ms % update 1000+ ms % update min s update max s update avg s documents processed N
ScyllaDB 10 000 000 59.00 18.00 22.00 0.09 2.65 0.60 1600000 200
ScyllaDB 10 000 000 49.00 14.00 36.00 0.01 8.20 1.12 13158590 1703

####### prepare for 1e8

# first
scylladb-fixtures-filler -reset -n=10000000
# too long

# parallel
scylladb-fixtures-filler -n=10000000 -shift=10000000
# 
scylladb-fixtures-filler -n=10000000 -shift=20000000
# 
scylladb-fixtures-filler -n=10000000 -shift=30000000
# 
scylladb-fixtures-filler -n=10000000 -shift=40000000
# 
scylladb-fixtures-filler -n=10000000 -shift=50000000
# 

# index
scylladb-fixtures-filler -index
# failed
Explain
MongoDB

for 1 000 000 documents

use rtb;

db.subscriber.find({
    "worker_id": 1,
    "timezone": 1,
    "notify_after": {
        "$lte": 1564671368
    }
}).explain();

{
	"queryPlanner" : {
		"plannerVersion" : 1,
		"namespace" : "rtb.subscriber",
		"indexFilterSet" : false,
		"parsedQuery" : {
			"$and" : [
				{
					"timezone" : {
						"$eq" : 1
					}
				},
				{
					"worker_id" : {
						"$eq" : 1
					}
				},
				{
					"notify_after" : {
						"$lte" : 1564671368
					}
				}
			]
		},
		"winningPlan" : {
			"stage" : "FETCH",
			"inputStage" : {
				"stage" : "IXSCAN",
				"keyPattern" : {
					"worker_id" : 1,
					"timezone" : 1,
					"notify_after" : -1
				},
				"indexName" : "worker_id_1_timezone_1_notify_after_-1",
				"isMultiKey" : false,
				"multiKeyPaths" : {
					"worker_id" : [ ],
					"timezone" : [ ],
					"notify_after" : [ ]
				},
				"isUnique" : false,
				"isSparse" : false,
				"isPartial" : false,
				"indexVersion" : 2,
				"direction" : "forward",
				"indexBounds" : {
					"worker_id" : [
						"[1.0, 1.0]"
					],
					"timezone" : [
						"[1.0, 1.0]"
					],
					"notify_after" : [
						"[1564671368.0, -inf.0]"
					]
				}
			}
		},
		"rejectedPlans" : [ ]
	},
	"serverInfo" : {
		"host" : "5bc6e6b6d9d9",
		"port" : 27017,
		"version" : "4.0.11",
		"gitVersion" : "417d1a712e9f040d54beca8e4943edce218e9a8c"
	},
	"ok" : 1
}

for 10 000 000 documents

db.subscriber.find({
    "worker_id": 1,
    "timezone": 1,
    "notify_after": {
        "$lte": 1564672706
    }
}).limit(8000).explain("executionStats");

{
	"queryPlanner" : {
		"plannerVersion" : 1,
		"namespace" : "rtb.subscriber",
		"indexFilterSet" : false,
		"parsedQuery" : {
			"$and" : [
				{
					"timezone" : {
						"$eq" : 1
					}
				},
				{
					"worker_id" : {
						"$eq" : 1
					}
				},
				{
					"notify_after" : {
						"$lte" : 1564672706
					}
				}
			]
		},
		"winningPlan" : {
			"stage" : "LIMIT",
			"limitAmount" : 8000,
			"inputStage" : {
				"stage" : "FETCH",
				"inputStage" : {
					"stage" : "IXSCAN",
					"keyPattern" : {
						"worker_id" : 1,
						"timezone" : 1,
						"notify_after" : -1
					},
					"indexName" : "worker_id_1_timezone_1_notify_after_-1",
					"isMultiKey" : false,
					"multiKeyPaths" : {
						"worker_id" : [ ],
						"timezone" : [ ],
						"notify_after" : [ ]
					},
					"isUnique" : false,
					"isSparse" : false,
					"isPartial" : false,
					"indexVersion" : 2,
					"direction" : "forward",
					"indexBounds" : {
						"worker_id" : [
							"[1.0, 1.0]"
						],
						"timezone" : [
							"[1.0, 1.0]"
						],
						"notify_after" : [
							"[1564672706.0, -inf.0]"
						]
					}
				}
			}
		},
		"rejectedPlans" : [ ]
	},
	"executionStats" : {
		"executionSuccess" : true,
		"nReturned" : 8000,
		"executionTimeMillis" : 14,
		"totalKeysExamined" : 8000,
		"totalDocsExamined" : 8000,
		"executionStages" : {
			"stage" : "LIMIT",
			"nReturned" : 8000,
			"executionTimeMillisEstimate" : 1,
			"works" : 8001,
			"advanced" : 8000,
			"needTime" : 0,
			"needYield" : 0,
			"saveState" : 62,
			"restoreState" : 62,
			"isEOF" : 1,
			"invalidates" : 0,
			"limitAmount" : 8000,
			"inputStage" : {
				"stage" : "FETCH",
				"nReturned" : 8000,
				"executionTimeMillisEstimate" : 1,
				"works" : 8000,
				"advanced" : 8000,
				"needTime" : 0,
				"needYield" : 0,
				"saveState" : 62,
				"restoreState" : 62,
				"isEOF" : 0,
				"invalidates" : 0,
				"docsExamined" : 8000,
				"alreadyHasObj" : 0,
				"inputStage" : {
					"stage" : "IXSCAN",
					"nReturned" : 8000,
					"executionTimeMillisEstimate" : 0,
					"works" : 8000,
					"advanced" : 8000,
					"needTime" : 0,
					"needYield" : 0,
					"saveState" : 62,
					"restoreState" : 62,
					"isEOF" : 0,
					"invalidates" : 0,
					"keyPattern" : {
						"worker_id" : 1,
						"timezone" : 1,
						"notify_after" : -1
					},
					"indexName" : "worker_id_1_timezone_1_notify_after_-1",
					"isMultiKey" : false,
					"multiKeyPaths" : {
						"worker_id" : [ ],
						"timezone" : [ ],
						"notify_after" : [ ]
					},
					"isUnique" : false,
					"isSparse" : false,
					"isPartial" : false,
					"indexVersion" : 2,
					"direction" : "forward",
					"indexBounds" : {
						"worker_id" : [
							"[1.0, 1.0]"
						],
						"timezone" : [
							"[1.0, 1.0]"
						],
						"notify_after" : [
							"[1564672706.0, -inf.0]"
						]
					},
					"keysExamined" : 8000,
					"seeks" : 1,
					"dupsTested" : 0,
					"dupsDropped" : 0,
					"seenInvalidated" : 0
				}
			}
		}
	},
	"serverInfo" : {
		"host" : "5bc6e6b6d9d9",
		"port" : 27017,
		"version" : "4.0.11",
		"gitVersion" : "417d1a712e9f040d54beca8e4943edce218e9a8c"
	},
	"ok" : 1
}
MySQL
EXPLAIN SELECT id, worker_id, timezone, notify_after, endpoint
        FROM subscribers
        WHERE worker_id = 1
          AND timezone = 1
          AND notify_after <= 1;
id select_type table partitions type possible_keys key key_len ref rows filtered Extra
1 SIMPLE subscribers NULL range WORKER_TIMEZONE_NOTIFY WORKER_TIMEZONE_NOTIFY 6 NULL 1 100.00 Using index condition
Aerospike skipped, reason missing batch update
Native method

Keep subscribers for worker in memory and update without change index For 1e6 by 20 workers and 1e7 to 2 workers

go run ./development/mysql/command/fixtures/fill.go -reset -n=1000000
# on generate use WorkerCount = 20
go run ./development/mysql/command/fixtures/fill.go -reset -n=10000000
# on generate use WorkerCount = 2

Let's select index

CREATE TABLE subscribers
(
  id           INT(11) UNSIGNED PRIMARY KEY AUTO_INCREMENT,
  notify_after INT(11) UNSIGNED    NOT NULL,
  partner      INT(11) UNSIGNED    NOT NULL,
  stream       INT(11) UNSIGNED    NOT NULL,
  widget       INT(11) UNSIGNED    NOT NULL,
  worker_id    TINYINT(3) UNSIGNED NOT NULL,
  timezone     TINYINT(3) UNSIGNED NOT NULL,
  os           TINYINT(3) UNSIGNED NOT NULL,
  browser      TINYINT(3) UNSIGNED NOT NULL,
  country      CHAR(2)             NOT NULL,
  language     CHAR(2)             NOT NULL,
  endpoint     VARCHAR(255)        NOT NULL
) ENGINE = InnoDB;

Without index

EXPLAIN SELECT id, notify_after, partner, stream, widget, worker_id, timezone, os, browser, country, language, endpoint
        FROM subscribers
        WHERE worker_id = 1
          AND id > 8000
        ORDER BY id ASC
        LIMIT 8000;
id select_type table partitions type possible_keys key key_len ref rows filtered Extra
1 SIMPLE subscribers NULL range PRIMARY PRIMARY 4 NULL 492094 10.00 Using where
1 SIMPLE subscribers NULL range PRIMARY PRIMARY 4 NULL 4925265 10.00 Using where

With worker id index:

ALTER TABLE subscribers
  ADD INDEX WORKER_INDEX (worker_id);

EXPLAIN SELECT id, notify_after, partner, stream, widget, worker_id, timezone, os, browser, country, language, endpoint
        FROM subscribers
        WHERE worker_id = 1
          AND id > 8000
        ORDER BY id ASC
        LIMIT 8000;
id select_type table partitions type possible_keys key key_len ref rows filtered Extra
1 SIMPLE subscribers NULL index_merge PRIMARY,WORKER_INDEX WORKER_INDEX,PRIMARY 5,4 NULL 47842 100.00 Using intersect(WORKER_INDEX,PRIMARY); Using where; Using filesort
1 SIMPLE subscribers NULL range PRIMARY,WORKER_INDEX PRIMARY 4 NULL 4925265 100.00 Using where

With worker_id, id index:

DROP INDEX WORKER_INDEX ON subscribers;
ALTER TABLE subscribers
  ADD INDEX WORKER_PRIMARY_INDEX (worker_id, id);

EXPLAIN SELECT id, notify_after, partner, stream, widget, worker_id, timezone, os, browser, country, language, endpoint
        FROM subscribers
        WHERE worker_id = 1
          AND id > 8000
        ORDER BY id ASC
        LIMIT 8000;
id select_type table partitions type possible_keys key key_len ref rows filtered Extra
1 SIMPLE subscribers NULL range PRIMARY,WORKER_PRIMARY_INDEX PRIMARY 4 NULL 492094 5.26 Using where
1 SIMPLE subscribers NULL range PRIMARY,WORKER_PRIMARY_INDEX PRIMARY 4 NULL 4925265 100.00 Using where

Conclusion: need benchmark it

Benchmark for 1e6 by 20 workers & 10e6 by 2 workers
go test ./components/native/... -v -run=$^ -bench=Run -benchmem -benchtime=10s
go test ./components/native/... -v -run=$^ -bench=Run -benchmem -benchtime=50s
Without index
BenchmarkRun   	      20	 625788182 ns/op	28507189 B/op	 1033430 allocs/op
--- BENCH: BenchmarkRun
    benchmark_test.go:21: documents 49895
    benchmark_test.go:21: documents 49895
PASS
ok  	workerperformance/components/native	13.121s
BenchmarkRun   	       5	13741979681 ns/op	2789578715 B/op	103560544 allocs/op
--- BENCH: BenchmarkRun
    benchmark_test.go:21: documents 4999864
    benchmark_test.go:21: documents 4999864
    benchmark_test.go:21: documents 4999864
PASS
ok  	workerperformance/components/native	124.799s
With worker id index
BenchmarkRun   	       5	2742468246 ns/op	28509600 B/op	 1033451 allocs/op
--- BENCH: BenchmarkRun
    benchmark_test.go:21: documents 49895
    benchmark_test.go:21: documents 49895
    benchmark_test.go:21: documents 49895
PASS
ok  	workerperformance/components/native	24.350s
BenchmarkRun   	       5	13720915920 ns/op	2789579009 B/op	103560551 allocs/op
--- BENCH: BenchmarkRun
    benchmark_test.go:21: documents 4999864
    benchmark_test.go:21: documents 4999864
    benchmark_test.go:21: documents 4999864
PASS
ok  	workerperformance/components/native	123.635s

With worker_id, id index:

BenchmarkRun   	      20	 679537963 ns/op	28507044 B/op	 1033430 allocs/op
--- BENCH: BenchmarkRun
    benchmark_test.go:21: documents 49895
    benchmark_test.go:21: documents 49895
PASS
ok  	workerperformance/components/native	14.220s
BenchmarkRun   	       5	13771612881 ns/op	2789577878 B/op	103560538 allocs/op
--- BENCH: BenchmarkRun
    benchmark_test.go:21: documents 4999864
    benchmark_test.go:21: documents 4999864
    benchmark_test.go:21: documents 4999864
PASS
ok  	workerperformance/components/native	123.947s

Directories

Path Synopsis
components
aerospike
bson
common-redis-command
consts
dataprovider
dynomite-redis
env
etalon
iterator
logger
mongodb
mysql
native
oplogger
redis
redis-test
roundtime
scylladb
stats
development
dynomite-redis/command/fixtures command
dynomite-redis/command/worker command
mongodb/command/fixtures command
mongodb/command/worker command
mongodb/worker command
mysql/command/fixtures command
mysql/worker command
native/command/fixtures command
native/command/worker command
native/testing command
redis/command/fixtures command
redis/command/worker command
scylladb/command/fixtures command
scylladb/command/worker command
models
aerospike
etalon
mongodb
mysql
native
protos/redis
redis
skylladb

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.