README
¶
batchgo
batchgo is a Golang library for batch processing data. It provides a simple way to process large amounts of data in batches.
Features
- Asynchronous batch processing
- concurrency-safe mechanism to add item to batch
Slicerinterface empowers users to merge or append data
Go Docs
Documentation at pkg.go.dev
Installation
go get github.com/WheeskyJack/batchgo
Usage
The user needs to implement the Slicer interface on the batching object for this package to work. It has Append, Len Export and OnFailure methods.
Append method is called to merge the item into existing batch.
Len method is called to get the length of current batch.
Export method is called once batch is full and ready for processing.
OnFailure method is called if Export method fails.
Please refer to the example directory for a working demonstration. It showcases how Add() can be called concurrently for batching.
Contributing
Contributions are welcome! Please feel free to create an issue and submit a pull request.
License
This project is licensed under the MIT License - see the LICENSE file for details.
Documentation
¶
Overview ¶
Package batchgo provides batch provides batch processing functionality. It reads the data using Add method and processes it as per the Slicer interface defined. It launches the background goroutine to collect and export batches. Batch supports both size and time based batching.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Batch ¶
type Batch struct {
// Size is item count after which a batch will be sent regardless of the Timeout
// must be non zero
Size int
// Timeout is time duration after which a batch will be sent regardless of Size
// must be non zero
Timeout time.Duration
// NewSlice is the method using which new slice can be craeted for export purpose.
// All the items received will be merged into this.
NewSlice func() Slicer
// ReqBufferSize is the buffer size of req channel used in Add().
// defualt value is 10.
ReqBufferSize int
// contains filtered or unexported fields
}
Batch supports both size and time based batching. Size : item count after which a batch will be sent regardless of the Timeout. Timeout : time duration after which a batch will be sent regardless of Size.
func New ¶
func New(size int, timeout time.Duration, createSlice func() Slicer, opts ...Option) (*Batch, error)
New creates new batcher.
func (*Batch) Add ¶
Add puts up the item for merging and is async merge operation. this is concurrency-safe call.
Source Files
Directories