README
¶
worker-pool
intro:
worker pool is a simple go library to intergrate a limited routine process in to your golang application. Jobs added to the worker pool will be assigned to a worker depending on a preffered hash algorithm calculated by the key. This ensures that the Jobs with same key will be processed by the same worker, thus ensuring race conditions in concurrent processes.
Warning:if shared resources are used by the process function, the library does not gurantee the above claims, in which case you must implement your own logic to avoid race conditions.
usage:
-
import the library to development environment using go-get
go-get github.com/wishperera/worker-pool -
creating a new pool and initializing with a process function
conf := worker_pool.NewPoolConfig() conf.Workers = 100 conf.WorkerBufferSize = 10 conf.Metrics.NameSpace = "test_namespace" conf.Metrics.SubSystem = "test_subsystem" conf.HashFunc = worker_pool.SHA256 pool,err := worker_pool.NewPool(conf) if err != nil{ //todo- handle error as preffered } pool.Init(context.Background(), func(ctx context.Context, in interface{}) (out interface{}, err error) { //todo - process function body return out,nil }) -
conf.Workersrefers to the number of workers(spawned in go routines) running in parallel, andconf.WorkerBufferSizerefers to the size of the workers buffered channel. -
conf.HashFuncrefers to the bucket hashing algorithm used by job manager -
conf.Metricsis used to set the prometheus metric name space and subsystem for pool metrics -
worker_pool.DefaultConfigcan be passed as the config for worker_pool.NewPool() function in which case the defaults will be as followsWorkers = 100 WorkerBufferSize = 10 Metrics.NameSpace = "worker_pool" Metrics.SubSystem = "worker_pool" HashFunc = worker_pool.SHA256 -
adding jobs to the pool
val := 300 jobID := pool.AddNewJob(context.Background(),val,key) -
value is the input to the process function, which can be a primitive type or a struct if multiple input parameters are required
-
key is can be a string or an integer (8,16,32,64). Jobs with same key will be processed by the same worker.
-
output can be retreived from the
pool.Outputchannel which returns aJobtype structure containg the original input,output and the possible errors. TheGetIDmethod returns a unique id to the Job that can be used to align the input with output. -
more documentation can be found here
Documentation
¶
Overview ¶
worker pool is a simple go library to intergrate a limited routine process in to your golang application.
Index ¶
Constants ¶
const ( SHA256 = iota + 1 SHA512 MD5 )
Variables ¶
var DefaultConfig poolConfig
default pool config
var (
PoolMap map[uuid.UUID](*Pool)
)
Reduntant for now.Will be useful for a multiple pool scenario
Functions ¶
Types ¶
type Job ¶
type Job struct {
// contains filtered or unexported fields
}
type Pool ¶
type Pool struct {
Output chan Job
// contains filtered or unexported fields
}
func NewPool ¶
returns a pointer to pool object with the input,output and error channels size set to the buffersize and routines count limited to workers
func (*Pool) AddNewJob ¶
adds a new job to the process queue. will panic if the pool is not initialized using pool.Init(). Returns the job id for future use.
func (*Pool) Init ¶
func (p *Pool) Init(ctx context.Context, processFunc func(ctx context.Context, in interface{}) (out interface{}, err error))
intiailize the pool with a process function that accepts a context and the function parameters as a interface. parameter can be a single value or a structure in case of multiple expected inputs, same goes for output.
Source Files
Directories