README
¶
pgzip
Go parallel gzip compression. This is a fully gzip compatible drop in replacement for "compress/gzip".
This will split compression into blocks that are compressed in parallel. This can be useful for compressing big amounts of data.
The gzip decompression has not been modified, but remains in the package, so you can use it as a complete replacement for "compress/gzip".
You should only use this if you are compressing big amounts of data, say more than 1MB at the time, otherwise you will not see any benefit.
A variant of this is bgzf, which has the same feature, as well as seeking in the resulting file. The only drawback is a slightly bigger overhead compared to gzip.
Installation
go get github.com/klauspost/pgzip
Usage
To use as a replacement for gzip, exchange import compress/gzip with import gzip github.com/klauspost/pgzip. The API should be compatible.
To change the block size, use the added (*pgzip.Writer).SetConcurrency(blockSize, blocks int) function. With this you can control the approximate size of your blocks, as well as how many you want to be processing in parallel. Default values for this is SetConcurrency(250000, 16), meaning blocks are split at 250000 bytes and up to 16 blocks can be processing at once before the writer blocks.
Example:
var b bytes.Buffer
w := gzip.NewWriter(&b)
w.SetConcurrency(100000, 10)
w.Write([]byte("hello, world\n"))
w.Close()
To get any performance gains, you should at least be compressing more than 1 megabyte of data at the time.
You should at least have a block size of 100k and at least a number of blocks that match the number of cores your would like to utilize, but about twice the number of blocks would be the best.
Another side effect of this is, that it is likely to speed up your other code, since writes to the compressor only blocks if the compressor is already compressing the number of blocks you have specified. This also means you don't have worry about buffering input to the compressor.
Performance
Compression cost is usually about 0.2% with default settings with a block size of 250k.
Example with GOMAXPROC set to 4 (dual core with 2 hyperthreads)
| Compressor | MB/sec | speedup | size | size overhead |
|---|---|---|---|---|
| gzip (golang) | 15.082MB/s | 1.0x | 6.405.193 | 0% |
| pgzip (golang) | 26.736MB/s | 1.8x | 6.421.585 | 0.2% |
| bgzf (golang) | 29.525MB/s | 1.9x | 6.875.913 | 7.3% |
Documentation
¶
Overview ¶
Package pgzip implements reading and writing of gzip format compressed files, as specified in RFC 1952.
This is a drop in replacement for "compress/gzip". This will split compression into blocks that are compressed in parallel. This can be useful for compressing big amounts of data. The gzip decompression has not been modified, but remains in the package, so you can use it as a complete replacement for "compress/gzip".
See more at https://github.com/klauspost/pgzip
Index ¶
Constants ¶
const ( NoCompression = flate.NoCompression BestSpeed = flate.BestSpeed BestCompression = flate.BestCompression DefaultCompression = flate.DefaultCompression )
These constants are copied from the flate package, so that code that imports "compress/gzip" does not also have to import "compress/flate".
Variables ¶
var ( // ErrChecksum is returned when reading GZIP data that has an invalid checksum. ErrChecksum = errors.New("gzip: invalid checksum") // ErrHeader is returned when reading GZIP data that has an invalid header. ErrHeader = errors.New("gzip: invalid header") )
Functions ¶
This section is empty.
Types ¶
type Header ¶
type Header struct {
Comment string // comment
Extra []byte // "extra data"
ModTime time.Time // modification time
Name string // file name
OS byte // operating system type
}
The gzip file stores a header giving metadata about the compressed file. That header is exposed as the fields of the Writer and Reader structs.
type Reader ¶
type Reader struct {
Header
// contains filtered or unexported fields
}
A Reader is an io.Reader that can be read to retrieve uncompressed data from a gzip-format compressed file.
In general, a gzip file can be a concatenation of gzip files, each with its own header. Reads from the Reader return the concatenation of the uncompressed data of each. Only the first header is recorded in the Reader fields.
Gzip files store a length and checksum of the uncompressed data. The Reader will return a ErrChecksum when Read reaches the end of the uncompressed data if it does not have the expected length or checksum. Clients should treat data returned by Read as tentative until they receive the io.EOF marking the end of the data.
func NewReader ¶
NewReader creates a new Reader reading the given reader. The implementation buffers input and may read more data than necessary from r. It is the caller's responsibility to call Close on the Reader when done.
type Writer ¶
type Writer struct {
Header
// contains filtered or unexported fields
}
A Writer is an io.WriteCloser. Writes to a Writer are compressed and written to w.
func NewWriter ¶
NewWriter returns a new Writer. Writes to the returned writer are compressed and written to w.
It is the caller's responsibility to call Close on the WriteCloser when done. Writes may be buffered and not flushed until Close.
Callers that wish to set the fields in Writer.Header must do so before the first call to Write or Close. The Comment and Name header fields are UTF-8 strings in Go, but the underlying format requires NUL-terminated ISO 8859-1 (Latin-1). NUL or non-Latin-1 runes in those strings will lead to an error on Write.
func NewWriterLevel ¶
NewWriterLevel is like NewWriter but specifies the compression level instead of assuming DefaultCompression.
The compression level can be DefaultCompression, NoCompression, or any integer value between BestSpeed and BestCompression inclusive. The error returned will be nil if the level is valid.
func (*Writer) Close ¶
Close closes the Writer, flushing any unwritten data to the underlying io.Writer, but does not close the underlying io.Writer.
func (*Writer) Flush ¶
Flush flushes any pending compressed data to the underlying writer.
It is useful mainly in compressed network protocols, to ensure that a remote reader has enough data to reconstruct a packet. Flush does not return until the data has been written. If the underlying writer returns an error, Flush returns that error.
In the terminology of the zlib library, Flush is equivalent to Z_SYNC_FLUSH.
func (*Writer) Reset ¶
Reset discards the Writer z's state and makes it equivalent to the result of its original state from NewWriter or NewWriterLevel, but writing to w instead. This permits reusing a Writer rather than allocating a new one.
func (*Writer) SetConcurrency ¶
Use SetConcurrency to finetune the concurrency level if needed.
With this you can control the approximate size of your blocks, as well as how many you want to be processing in parallel.
Default values for this is SetConcurrency(250000, 16), meaning blocks are split at 250000 bytes and up to 16 blocks can be processing at once before the writer blocks.
Source Files