Documentation
¶
Overview ¶
Copyright 2015 by Leipzig University Library, http://ub.uni-leipzig.de
The Finc Authors, http://finc.info
Martin Czygan, <martin.czygan@uni-leipzig.de>
This file is part of some open source application.
Some open source application is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
Some open source application is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with Foobar. If not, see <http://www.gnu.org/licenses/>.
@license GPL-3.0+ <http://spdx.org/licenses/GPL-3.0+>
Package bytebatch implements a []byte batch and parallel processing facilities for line oriented input.
Basic usage:
p := bytebatch.NewLineProcessor(os.Stdin, os.Stdout, func(b []byte) ([]byte, error) {
// implement your business logic here
return b, nil
})
if err := p.Run(); err != nil {
// handle error
}
More self contained examples (parallel wc -l, grep -F):
* https://gist.github.com/miku/0687690c68e6104893a7c328d2900ca8.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type ByteBatch ¶
type ByteBatch struct {
// contains filtered or unexported fields
}
ByteBatch is a batch of byte slices.
func NewByteBatch ¶
NewByteBatch creates a new ByteBatch with a given capacity.
type ByteFunc ¶
ByteFunc is a function, that transforms a byte slice into another byte slice (plus error).
type LineProcessor ¶
type LineProcessor struct {
BatchSize int
LineSeparator byte
NumWorkers int
// contains filtered or unexported fields
}
LineProcessor reads bytes from a reader, feeds them into a ByteFunc and writes the result to a writer. The default unit of work is one line.
func NewLineProcessor ¶
NewLineProcessor creates a new LineProcessor. Default batch size is 10000, default line separator is `\n`, default number of workers equals the cpu count.
func (LineProcessor) Run ¶
func (p LineProcessor) Run() error
Run starts executing workers, crunching trough the input.
Source Files