minlz

MinLZ

MinLZ is a LZ77-type compressor with a fixed byte-aligned encoding, in the similar class to Snappy and LZ4.

The goal of MinLZ is to provide a fast, low memory compression algorithm that can be used for fast compression of data, where encoding and/or decoding speed is the primary concern.

MinLZ is designed to operate faster than IO for both compression and decompression.

It is a viable "always on" option, even if some content already is compressed.

If slow compression is acceptable, MinLZ can be configured to produce a high compression ratio while retaining a high decompression speed.

• Best in class compression
• Block or Streaming interfaces
• Very fast decompression, even as pure Go
• AMD64 encoder+decoder assembly
• ARM64 decoder assembly
• Adjustable Compression (4 levels)
• Concurrent stream Compression
• Concurrent stream Decompression
• Skip forward in compressed streams via independent blocks
• Random seeking with optional index
• Stream EOF validation
• Automatic stream size padding
• Custom encoders for small blocks
• Skippable/Non-skippable user blocks
• Detailed control of memory under decompression
• Fast detection of pre-compressed data
• Powerful commandline utility
• Customizable Bloom Filter Stream Search

This package implements the MinLZ specification v1.0 in Go.

For format specification see the included SPEC.md.

Changelog

• v1.1.0

  • Added SuperFast compression mode. See LevelSuperFast below.
  • Added ARM64 decompression assembly.
  • Added -follow to mz command to allow reading files while they are being written.
  • Improve default compression.

• v1.0.0

  • Initial Release Blog Post.

Usage

MinLZ can operate on blocks up to 8 MB or streams with unlimited length.

Blocks are the simplest, but do not provide any output validation. Blocks are mainly useful for small data sizes.

Streams are a collection of independent blocks, which each have checksums and EOF checks, which ensures against corruption and truncation.

4 compression levels are provided:

• Level -1, "SuperFast": Provides the fastest compression, but at a very reduced compression ratio.
• Level 1, "Fastest": Provides the fastest compression with reasonable compression.
• Level 2, "Balanced": Provides a good balance between compression and speed. ~50% the speed of the fastest level.
• Level 3, "Smallest": Provides the smallest output possible. Not tuned for speed.

A secondary option to control speed/compression is adjusting the block size. See "Writer Block Size" section below.

Blocks

MinLZ provides a block encoding interface with blocks up to 8MB. Blocks do not perform any data integrity check of the content, so additional checksum is recommended.

A basic roundtrip looks like this:

   compressed, err := minlz.Encode(nil, src, minlz.LevelBalanced)
   if err != nil {
       // Handle error
   }
   
    decompressed, err := minlz.Decode(nil, compressed)
    if err != nil {
        // Handle error 
    }

In both cases, a destination buffer can be provided, which will be overwritten. If the destination buffer is too small, an appropriately sized buffer will be allocated and returned.

It is possible to get the decompressed buffer size by using minlz.DecodedLen(block []byte) (int, error).

You can use the predefined LevelFastest, LevelBalanced or LevelSmallest which correspond to levels 1,2 and 3 respectively.

MinLZ does not track the compressed size of buffers and the decode input must match the output exactly. Extra bytes given to decompression will return an error.

It is possible to use minlz.TryEncode, which will only return compressed bytes if the output size is strictly less than input. Use minlz.AppendEncoded and minlz.AppendDecoded to append to existing slices.

Streams

Streams provide much more safety and allow for unlimited length encoding, as well as seeking and concurrent encoding/decoding.

Generally, you do not need buffering on the input or output side as reads and writes are done in rather big blocks. Reading and writing data on streams are buffered, and only non-concurrent will block for input/output.

When dealing with many streams, it is recommended to re-use the Readers and Writers. If you are dealing with short streams, consider limiting the concurrency, so block_size * concurrency doesn't exceed the expected stream size.

Encoding

Streams are the recommended way to use MinLZ. They provide end-to-end validation against corruption and truncation.

    // Create a new stream encoder.
    // The encoder will write to the provided io.Writer.
    enc := minlz.NewWriter(output)
	
	// We defer a call to Close.
	// This will flush any pending data and indicate we have reached the end of the stream.
	defer enc.Close()

	// Write data to the encoder.
	// The encoder will write the compressed data to the underlying io.Writer.
	js := json.NewEncoder(enc)
	err := js.Encode(data)

Encoders can be reused by calling Reset on them with another output. This will reset the encoder to its initial state.

The encoder supports the io.ReaderFrom interface, which can be used for encoding data from an io.Reader. This will typically be faster than writing data to the encoder, since it avoids a memory copy.

If you have a single big buffer to encode, you can use the EncodeBuffer([]byte) error to encode it. This will encode the buffer with minimal overhead. If you plan to do multiple writes, use the regular Write function.

Options

There are various options that can be set on the stream encoder. This can be used to control resource usage on compression and some aspects of decompression. If invalid options are set, the encoder will return an error when used.

We will cover the most common options here. Refer to the godoc for a complete list.

Writer Compression Level

The WriterLevel option controls the compression level of the stream encoder.

You can use the predefined LevelFastest, LevelBalanced or LevelSmallest which correspond to levels 1,2 and 3 respectively.

Setting level 0 will disable compression and write the data as an uncompressed stream.

The default level is LevelBalanced.

Typical speeds are LevelFastest is 2x the speed of LevelBalanced and LevelSmallest is at least an order of magnitude slower.

LevelSuperFast

Furthermore LevelSuperFast is provided. This compression mode is aimed purely at reducing slowdown when compressing hard-to-compress data. In practice that is short matches.

The compression ratio can greatly suffer in this mode, but in these cases it can be faster. So if you have a very high throughput (> 1GB/core/s) and a time-sensitive use case, this can be used to ensure the compression doesn't take too much longer on this content.

For these cases the speed difference can be up to 2x over LevelFastest, but also with a much worse compression ratio. However, most often it will be around 15% faster with a similar drop in the compression ratio.

Typically, the performance difference will also be the same for decompression.

Writer Block Size

The WriterBlockSize allows to set the maximum size of each block on the stream encoder. The blocksize - rounded up to a power of 2 - is communicated in the stream, and the decoder will use this to allocate memory during decompression.

Smaller blocks will take up less memory on both compression and decompression, but will result in a larger output.

Block size further allows trading off speed vs. size; Here is a sample chart of how speed and block size can correlate, using the fastest encoder setting:

Input is a 3,325,605,752 byte CSV file compressed on a 16 core CPU.

The actual scaling mostly depends on the amount of CPU L2 cache (speed) and the nature of the compressed data (size).

Decompression speed is affected similarly, but less predictably, since it is more likely to be limited by memory throughput, and larger output also tends to affect it more negatively.

If your software is very sensitive to GC stoppages, also note that with assembly single block de/compression cannot be pre-empted, so stop-the-world events may take longer on bigger blocks.

The default block size is 2 MB.

Writer Concurrency

The WriterConcurrency option allows setting the number of concurrent blocks that can be compressed. Higher concurrency will increase the throughput of the encoder, but will also increase memory usage.

If WriterConcurrency(1) is used no async goroutines will be used and the encoder will run in the calling goroutine.

The default concurrency is GOMAXPROCS.

Decoding

Decoding streams mostly just involves sending the compressed stream to a Reader.

Anything accepting an io.Reader as input will then be able to read the decompressed data.

	// Create a new stream decoder. 
	// The encoder will read from the provided io.Reader.
	dec := minlz.NewReader(input)
	
	// Read decompressed input.
	js := json.NewDecoder(dec)
	err := js.Decode(&data)

If you would like the output to be written to an io.Writer, the easiest is to use the WriteTo functionality.

	// Our input and output
	in, _ := os.Create("input.mz")
	out, _ := os.Create("output.txt")
	
	// Create a new stream decoder
	dec := minlz.NewReader(in)

	// Write all decompressed data to output
	n, err := dec.WriteTo(out)
	fmt.Println("Wrote", n, "bytes. Error:", err)

The DecompressConcurrent has similar functionality to WriteTo, but allows specifying the concurrency. By default WriteTo uses runtime.NumCPU() or at most 8 concurrent decompressors. Besides offering higher throughput using DecompressConcurrent will also make input reads async when used.

For memory-sensitive systems, the maximum block size can be set below 8MB. For this use the ReaderMaxBlockSize(int) option.

Skipping and Seeking

Streams can be skipped forward by calling (*Reader).Skip(n int64) error. This will skip forward in the stream by n bytes. Intermediate blocks be read, but will not be decompressed unless the skip ends inside the block.

Full random seeking is supported by using an index. An index can be created when the stream is encoded. The index can either be added to the stream or stored separately. For existing streams the IndexStream(r io.Reader) ([]byte, error) function can be used to create an index.

To add an index at the end of streams, use the WriterAddIndex() option when creating the writer, then the index will be added to the stream when it is closed. To keep the index separate, use the (*Writer).CloseIndex() ([]byte, error) method to retrieve the index when finishing a stream.

To get a fully seekable reader use (*Reader).ReadSeeker(index []byte) (*ReadSeeker, error). The returned reader will implement io.Seeker, io.ReaderAt in addition to the existing Reader methods and can be used to seek to any position in the stream.

If an index is not provided in the call, the reader will attempt to read the index from the end of the stream. If the input stream does not support io.Seeker an error will be returned.

Custom User Data

Streams can contain user-defined data, that isn't part of the stream. Each "chunk" has an ID, which allows for processing of different types.

This data can either be "skippable" - meaning it is ignored if the user hasn't provided a handler for these. If the chunk is non-skippable, the encoder will error out if this chunk isn't handled by the user.

MinUserSkippableChunk is the minimum chunk id with user data and MaxUserSkippableChunk is the maximum.

MinUserNonSkippableChunk is the minimum ID that will not automatically be skipped if unhandled by the user. Finally MaxUserNonSkippableChunk is the final ID that can be used for this.

The custom data will not be compressed or modified in any way.

func ExampleWriterAddUserChunk() {
	var buf bytes.Buffer
	w := minlz.NewWriter(&buf)
	// Add a skippable chunk
	w.AddUserChunk(minlz.MinUserSkippableChunk, []byte("Chunk Custom Data"))
	// Write content to stream.
	w.Write([]byte("some data"))
	w.Close()

	// Read back what we wrote.
	r := minlz.NewReader(&buf)
	r.SkippableCB(minlz.MinUserSkippableChunk, func(sr io.Reader) error {
		b, err := io.ReadAll(sr)
		fmt.Println("Callback:", string(b), err)
		return err
	})

	// Read stream data
	b, err := io.ReadAll(r)
	fmt.Println("Stream data:", string(b))

	//OUTPUT:
	//Callback: Chunk Custom Data <nil>
	//Stream data: some data
}

The maximum single chunk size is 16MB, but as many chunks as needed can be added.

Block Search Tables

MinLZ streams can include optional per-block hash tables that enable fast pattern searching without decompressing every block. Blocks that definitely don't contain the search pattern are skipped entirely.

// Compression: add search tables.
cfg := minlz.NewSearchTableConfig() // matchLen defaults to 6
w := minlz.NewWriter(output, minlz.WriterSearchTable(cfg))

// Searching: skip non-matching blocks automatically.
searcher := minlz.NewBlockSearcher(input)
err := searcher.Search([]byte("pattern"), func(r minlz.SearchResult) error {
    fmt.Printf("match at stream offset %d\n", r.StreamOffset)
    return nil
})

Prefix filtering can reduce table size by only indexing positions after specific bytes (e.g. " and : for JSON). See SEARCH.md for parameter tuning, prefix configuration, API details, and CLI usage.

Build Tags

The following build tags can be used to control which speed improvements are used:

• noasm disables all assembly.
• nounsafe disables all use of unsafe package.
• purego disables assembly and unsafe usage.

Using assembly/non-assembly versions will often produce slightly different output.

We will support 2 releases prior to current Go release version.

This package has been extensively fuzz tested to ensure that no data input can cause crashes or excessive memory usage.

When doing fuzz testing, use -tags=nounsafe. Non-assembly functions will also be tested, but for completeness also test with -tags=purego.

Performance

BLOCKS

Individual block benchmarks should be considered carefully - and can be hard to generalize, since they tend to overemphasize specific characteristics of the content.

Therefore, it will be easy to find counter-examples to the benchmarks, where specific patterns suit a specific compressor better than others. We present a few examples from the Snappy benchmark set. As a benchmark, this set has an over-emphasis on text files.

Blocks are compressed/decompress using 16 concurrent threads on an AMD Ryzen 9 3950X 16-Core Processor. Click below to see some sample benchmarks compared to Snappy and LZ4:

Protobuf Sample

Source file: https://github.com/google/snappy/blob/main/testdata/geo.protodata

HTML Sample

URL List Sample

Serialized GEO data Sample

In overall terms, we typically observe that:

• The fastest mode typically beats LZ4 both in speed and output size.
• The fastest mode is typically equal to Snappy in speed, but significantly smaller.
• The "balanced" mode typically beats the best possible LZ4 compression, but much faster.
• Without assembler MinLZ is mostly the fastest option for compression.
• LZ4 is decompression speed king.
• Snappy decompression is usually slowest — especially without assembly.

We encourage you to do your own testing with realistic blocks.

You can use λ mz c -block -bench=10 -verify -cpu=16 -1 file.ext with our commandline tool to test speed of block encoding/decoding.

Visualizer

You can visualize individual blocks using our block visualizer.

STREAMS

For fair stream comparisons, we run each encoder at its maximum block size or max 4MB, while maintaining independent blocks where it is an option. We use the concurrency offered by the package.

This means there may be further speed/size tradeoffs possible for each, so experiment with fine-tuning for your needs.

Blocks are compressed/decompressed using 16 core AMD Ryzen 9 3950X 16-Core Processor.

JSON Stream

Input Size: 6,273,951,764 bytes

Source file: https://files.klauspost.com/compress/github-june-2days-2019.json.zst

CSV Stream

Log data

Serialized Data

Backup (Mixed) Data

Our conclusion is that the new compression algorithm provides a good compression increase, while retaining the ability to saturate pretty much any IO either with compression or decompression given a moderate number of CPU cores.

Why are concurrent block and stream speeds so different?

In most cases, MinLZ will be limited by memory bandwidth.

Since streams consist of mostly "unseen" data, it will often mean that memory reads are outside any CPU cache.

Contrast that to blocks, where data has often just been read/produced and therefore already is in one of the CPU caches. Therefore, block (de)compression will more often take place with data read from cache rather than a stream, where data can be coming from memory.

Even if data is streamed into cache, the "penalty" will still have to paid at some place in the chain. So streams will mostly appear slower in benchmarks.

Commandline utility

Official releases can be downloaded from the releases section with binaries for most platforms.

To install from source execute go install github.com/minio/minlz/cmd/mz@latest.

Usage

λ mz
MinLZ compression tool vx.x built at home, (c) 2025 MinIO Inc.
Homepage: https://github.com/minio/minlz

Usage:
Compress:     mz c [options] <input>
Decompress:   mz d [options] <input>
 (cat)    :   mz cat [options] <input>
 (tail)   :   mz tail [options] <input>

Without options 'c' and 'd' can be omitted. Extension decides if decompressing.

Compress file:    mz file.txt
Compress stdin:   mz -
Decompress file:  mz file.txt.mz
Decompress stdin: mz d -

Note that all option sizes KB, MB, etc. are base 1024 in the commandline tool.

Speed indications are base 10.

Compressing

Usage: mz c [options] <input>

Compresses all files supplied as input separately.
Output files are written as 'filename.ext.mz.
By default output files will be overwritten.
Use - as the only file name to read from stdin and write to stdout.

Wildcards are accepted: testdir/*.txt will compress all files in testdir ending with .txt
Directories can be wildcards as well. testdir/*/*.txt will match testdir/subdir/b.txt

File names beginning with 'http://' and 'https://' will be downloaded and compressed.
Only http response code 200 is accepted.

Options:
  -0    Perform no compression
  -1    Compress faster, but with a minor compression loss
  -2    Default compression speed (default true)
  -3    Compress more, but a lot slower
  -bench int
        Run benchmark n times. No output will be written
  -block
        Use as a single block. Will load content into memory. Max 8MB.
  -bs string
        Max block size. Examples: 64K, 256K, 1M, 8M. Must be power of two and <= 8MB (default "8M")
  -c    Write all output to stdout. Multiple input files will be concatenated
  -cpu int
        Maximum number of threads to use (default 32)
  -help
        Display help
  -index
        Add seek index (default true)
  -o string
        Write output to another file. Single input file only
  -pad string
        Pad size to a multiple of this value, Examples: 500, 64K, 256K, 1M, 4M, etc (default "1")
  -q    Don't write any output to terminal, except errors
  -recomp
        Recompress MinLZ, Snappy or S2 input
  -rm
        Delete source file(s) after success
  -safe
        Do not overwrite output files
  -verify
        Verify files, but do not write output
  -xfast
        Compress fastest, with a major compression loss

Example:

λ mz c apache.log
Compressing apache.log -> apache.log.mz 2622574440 -> 170960982 [6.52%]; 4155.2MB/s

Decompressing

Usage: mz d [options] <input>

Decompresses all files supplied as input. Input files must end with '.mz', '.s2' or '.sz'.
Output file names have the extension removed. By default output files will be overwritten.
Use - as the only file name to read from stdin and write to stdout.

Wildcards are accepted: testdir/*.txt will decompress all files in testdir ending with .txt
Directories can be wildcards as well. testdir/*/*.txt will match testdir/subdir/b.txt

File names beginning with 'http://' and 'https://' will be downloaded and decompressed.
Extensions on downloaded files are ignored. Only http response code 200 is accepted.

Options:
  -bench int
        Run benchmark n times. No output will be written
  -block
        Decompress single block. Will load content into memory. Max 8MB.
  -block-debug
        Print block encoding
  -c    Write all output to stdout. Multiple input files will be concatenated
  -cpu int
        Maximum number of threads to use (default 32)
  -follow
        Follow file like tail -f, reopening when EOF is reached
  -help
        Display help
  -limit string
        Return at most this much data. Examples: 92, 64K, 256K, 1M, 4M
  -o string
        Write output to another file. Single input file only
  -offset string
        Start at offset. Examples: 92, 64K, 256K, 1M, 4M. Requires Index
  -q    Don't write any output to terminal, except errors
  -rm
        Delete source file(s) after success
  -safe
        Do not overwrite output files
  -tail string
        Return last of compressed file. Examples: 92, 64K, 256K, 1M, 4M. Requires Index
  -verify
        Verify files, but do not write output

Example:

λ mz d apache.log.mz
Decompressing apache.log.mz -> apache.log 170960982 -> 2622574440 [1534.02%]; 2660.2MB/s

Tail, Offset and Limit can be made to forward to the next newline by adding +nl.

For example mz d -c -offset=50MB+nl -limit=1KB+nl enwik9.mz will skip 50MB, search for the next newline, start outputting data. After 1KB, it will stop at the next newline.

Partial files - decoded with tail, offset or limit will have .part extension.

Searching

The search (or find or s) command searches for a literal pattern in compressed streams, using search tables to skip non-matching blocks:

mz search [options] <pattern> <file...>

Options:

• -l Print matching lines (default: show match offsets)
• -c Print match count only
• -n Include line numbers (with -l)
• -q Quiet mode, exit code only (0=found, 1=not found)
• -bail Error if search tables are not available

Example:

$ mz search -l "error_pattern" server.log.mz
10506612227:{"message":"error_pattern found in processing"}

Files must be compressed with -search to include search tables:

$ mz c -search=6 server.log
$ mz c -search=4 -search.prefixes="," data.csv

Without search tables, the search command falls back to decompressing every block.

Snappy/S2 Compatibility

MinLZ is designed to be easily upgradable from Snappy and S2.

Both the streaming and block interfaces in the Go port provide seamless compatibility with existing Snappy and S2 content. This means that any content encoded with either will be decoded correctly by MinLZ.

Content encoded with MinLZ cannot be decoded by Snappy or S2.

(*) MinLZ decoders may implement fallback to S2/Snappy. This is however not required and ports may not support this.

License

MinLZ is Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Based on code from snappy-go project.

Ports

Reference code is provided in the internal/reference folder. This provides simplified, but explicit versions of the block de/encoder; stream and index decoders with minimal dependencies.

There are official Go and Rust ports, plus an experimental C port; see the table below. If you are interested in porting MinLZ to another language, open a discussion topic.

If you do a port, feel free to send in a PR for this table:

Indicated features must support all parts of each feature as described in the specification. However, it is up to the implementation to decide the encoding implementation(s).