zlib

zlib — go-ruby-zlib

A pure-Go (no cgo) reimplementation of Ruby's zlib standard library — the MRI 4.0.5 Zlib module. The DEFLATE engine is klauspost/compress (its drop-in compress/flate, compress/zlib and compress/gzip packages) — a pure-Go, CGO=0, build-from-source dependency that is far faster than the standard library's flate (≈10× on Deflate); the checksums are SIMD-accelerated — CRC-32 folds with a carryless-multiply kernel (go-simd/crc32, arm64 PMULL fold-by-eight; amd64/ppc64le/s390x defer to the hardware-assisted standard library) and Adler-32 uses go-simd/adler32, each bit-identical to hash/crc32 / hash/adler32. It offers deflate / inflate, gzip, the CRC-32 and Adler-32 checksums (and their combine forms), and a streaming compressor / decompressor, so a host such as go-embedded-ruby can serve require "zlib" with no C extension and a static, CGO=0 binary.

It is the zlib backend for go-embedded-ruby but is a standalone, reusable module with no dependency on the Ruby runtime — a sibling of go-ruby-regexp (the Onigmo engine), go-ruby-yaml (Psych), and go-ruby-erb (the ERB compiler).

Byte-exactness — what is and isn't guaranteed. The checksums (Crc32, Adler32, and their combine forms) are byte-exact with MRI: a value computed here equals the integer MRI prints for the same input. The compressors round-trip and interoperate with MRI in both directions (we inflate MRI's deflate output, MRI inflates ours), but zlib never promises a canonical encoding, so the exact deflate byte stream is implementation-defined and need not equal MRI's. For gzip the same holds, and additionally the header carries an mtime/OS byte; GzipCompress fixes the mtime to zero for determinism, but you should compare the decompressed payload (and its CRC), not the raw gzip bytes.

Features

Faithful port of the Zlib surface, validated against the ruby binary on every platform that has one:

• Deflate / Inflate — zlib-stream compression at any level (Zlib::Deflate.deflate / Zlib::Inflate.inflate).
• GzipCompress / GzipDecompress — gzip round trip (Zlib::GzipWriter / Zlib::GzipReader), with a deterministic (zero) header mtime.
• Crc32 / Adler32 and Crc32Combine / Adler32Combine — byte-exact with MRI, with seeded (running) checksums.
• Streaming Deflater / Inflater — the Zlib::Deflate.new(level).deflate(s, flush).finish / Zlib::Inflate.new.inflate(s) idiom, with the flush modes (NoFlush/SyncFlush/FullFlush/Finish) and the TotalIn / TotalOut / Adler / Finished accessors.
• Constants — compression levels (NoCompression … DefaultCompression), strategies, flush modes, and Version / ZlibVersion.
• Errors — an *Error family carrying MRI's class names (Zlib::StreamError, Zlib::BufError, Zlib::DataError, Zlib::GzipFile::Error) so a host maps them straight onto the Ruby exceptions; all wrap to Error via errors.Is.

CGO-free, dependency-free, 100% test coverage, gofmt + go vet clean, and green across the six 64-bit Go targets (amd64, arm64, riscv64, loong64, ppc64le, s390x) and three OSes (Linux, macOS, Windows).

Install

go get github.com/go-ruby-zlib/zlib

Usage

package main

import (
	"fmt"

	"github.com/go-ruby-zlib/zlib"
)

func main() {
	// Deflate / Inflate — Zlib::Inflate.inflate(Zlib::Deflate.deflate(s)) == s.
	comp, _ := zlib.Deflate([]byte("hello world"), zlib.BestCompression)
	out, _ := zlib.Inflate(comp)
	fmt.Printf("%s\n", out) // hello world

	// Byte-exact checksums (seed 0 for CRC, 1 for Adler — the MRI identities).
	fmt.Println(zlib.Crc32([]byte("hello world"), 0))   // 222957957
	fmt.Println(zlib.Adler32([]byte("hello world"), 1)) // 436929629

	// Streaming, like Zlib::Deflate.new(level).deflate(a, SYNC_FLUSH)…finish.
	d := zlib.NewDeflater(zlib.BestCompression)
	p1, _ := d.Deflate([]byte("hello "), zlib.SyncFlush)
	p2, _ := d.Deflate([]byte("world"), zlib.NoFlush)
	tail, _ := d.Finish()
	stream := append(append(p1, p2...), tail...)
	got, _ := zlib.Inflate(stream)
	fmt.Printf("%s in=%d out=%d adler=%d\n", got, d.TotalIn(), d.TotalOut(), d.Adler())
}

API

// One-shot
func Deflate(data []byte, level int) ([]byte, error) // Zlib::Deflate.deflate
func Inflate(data []byte) ([]byte, error)            // Zlib::Inflate.inflate
func GzipCompress(data []byte, level int) ([]byte, error)
func GzipDecompress(data []byte) ([]byte, error)

// Checksums (byte-exact with MRI)
func Crc32(data []byte, seed uint32) uint32
func Adler32(data []byte, seed uint32) uint32
func Crc32Combine(crc1, crc2 uint32, len2 int64) uint32
func Adler32Combine(adler1, adler2 uint32, len2 int64) uint32
func Crc32Table() *crc32.Table

// Streaming
type Deflater struct{ /* … */ }
func NewDeflater(level int) *Deflater
func NewDeflaterLevel(level int) (*Deflater, error)
func (d *Deflater) Deflate(data []byte, flush int) ([]byte, error)
func (d *Deflater) Finish() ([]byte, error)
func (d *Deflater) TotalIn() int64
func (d *Deflater) TotalOut() int64
func (d *Deflater) Adler() uint32
func (d *Deflater) Finished() bool

type Inflater struct{ /* … */ }
func NewInflater() *Inflater
func (i *Inflater) Inflate(data []byte) ([]byte, error)
func (i *Inflater) Finish() ([]byte, error)
func (i *Inflater) TotalIn() int64
func (i *Inflater) TotalOut() int64
func (i *Inflater) Adler() uint32
func (i *Inflater) Finished() bool

// Errors — *Error carries the MRI exception Class name; all wrap to Error.
type Error struct{ Class, Msg string /* … */ }
var ErrStream, ErrBuf, ErrData, ErrGzipFile *Error

Constants

¹ Version is the Ruby binding version (stable). ZlibVersion stands in for the linked C zlib library version, which in MRI varies by host build ("1.2.12", "1.3", …); this pure-Go port has no C zlib and reports a representative constant.

Performance

The DEFLATE engine is klauspost/compress rather than the standard library's compress/flate, whose DefaultCompression match-finder is markedly slower on realistic data. Go-level throughput on a 1 MiB semi-compressible payload (go test -bench, Apple M-class, Go 1.26.4):

The combined deflate+inflate+crc32 workload — the one previously measured ~6.8× slower than MRI from rbgo — is now ≈6.7× faster than the old stdlib path, which closes essentially all of that gap. CRC-32 additionally gained a carryless-multiply kernel: on arm64 the standard library's IEEE path is a latency-bound serial CRC32X, and the fold-by-eight PMULL kernel here runs ≈4.3× faster (≈51 GB/s), now beating MRI's C zlib and YJIT on the library-level benchmark (see docs/performance). Wire compatibility is unaffected: the output is still a standard zlib/gzip/raw-DEFLATE stream that MRI's Zlib inflates, validated by the differential MRI oracle below.

Tests & coverage

The suite pairs deterministic, ruby-free tests (which alone hold coverage at 100%, so the qemu cross-arch and Windows lanes pass the gate) with a differential MRI oracle (gated to RUBY_VERSION >= "4.0"): checksums and combine values are compared byte-exact with the system ruby, and the deflate / gzip / streaming paths are round-tripped through MRI in both directions — we inflate MRI's output and MRI inflates ours, with gzip compared by decompressed payload + CRC (not raw bytes). The oracle scripts $stdout.binmode and $stdin.binmode so Windows text-mode never pollutes the binary payloads, and skip themselves where ruby is absent or older than 4.0.

COVERPKG=$(go list ./... | paste -sd, -)
go test -race -coverpkg="$COVERPKG" -coverprofile=cover.out ./...
go tool cover -func=cover.out | tail -1   # 100.0%

License

BSD-3-Clause — see LICENSE. Copyright the go-ruby-zlib/zlib authors.

WebAssembly

Being pure Go (CGO=0), this library also compiles to WebAssembly — both GOOS=js GOARCH=wasm (browser / Node.js) and GOOS=wasip1 GOARCH=wasm (WASI). CI builds both targets on every push, alongside the six 64-bit native/qemu arches.

GOOS=js     GOARCH=wasm go build ./...   # browser / Node
GOOS=wasip1 GOARCH=wasm go build ./...   # WASI (wasmtime, wasmer, wasmedge, …)