Retracted: these held pkg/ and internal/, now a module per pkg/v1 family and github.com/kitsunium/sdk/internal (ADR 0171): go get github.com/kitsunium/sdk@latest && go mod tidy
README
¶
internal/service/data/transform
Concrete core/data/transform.Compressor implementations over the Go standard
library: gzip (compress/gzip), flate (raw DEFLATE, compress/flate)
and zlib (the RFC 1950 envelope, compress/zlib). One package, three
self-registering schemes.
var GzipCompressor = transform.Register(gzipCompressor{}) // Algorithm "gzip"
var FlateCompressor = transform.Register(flateCompressor{}) // Algorithm "flate"
var ZlibCompressor = transform.Register(zlibCompressor{level: zlib.DefaultCompression}) // Algorithm "zlib"
Blank-importing the package registers all three with internal/core/data/transform,
so a consumer that only needs transform.Lookup("gzip") adds a single blank
import.
flate is not deflate
| Scheme | RFC | Wire shape |
|---|---|---|
flate |
RFC 1951 | bare DEFLATE bit-stream — no header, no checksum |
zlib |
RFC 1950 | 2-byte header + the same DEFLATE body + Adler-32 trailer |
gzip |
RFC 1952 | 10-byte header + DEFLATE body + CRC-32 and size trailer |
HTTP's Content-Encoding: deflate (RFC 9110 §8.4.1) names the zlib
envelope, not the raw DEFLATE stream. A peer advertising deflate wants
zlib; flate is correct only where the surrounding format supplies the
framing. The two are not wire-compatible in either direction and carry distinct
error codes.
Compression level
NewZlibCompressor(level) builds a zlib compressor at an explicit level and
clamps to zlib.DefaultCompression anything that would not compress — both a
level the stdlib writer rejects and zlib.NoCompression, which it accepts and
which stores bytes verbatim. No configuration yields a compressor that silently
does not compress (ADR 0031). The result is not registered; the registry entry
is the default-level ZlibCompressor.
Bounded decompression
Decompress drains its stdlib reader through io.LimitReader capped at 256 MiB
(maxDecompressedBytes). An over-cap stream returns the scheme's failure
sentinel instead of driving an OOM — the conservative layer-local backstop. The
full ratio-aware decompression-bomb guard lives in the pkg/v1/data/codec frame
layer (a later commit). This package ships stdlib only; the vendor
compressors (zstd, s2) live in third-party/transform behind their own opt-in
import, with their own ceiling (ADR 0066).
Error codes (range 0.3.26.*)
Declared in internal/core/data/transform since ADR 0160, under the values
this package allocated; the schemes here wrap a stdlib cause in them.
| Code | Var | Trigger |
|---|---|---|
0.3.26.1 |
GzipFailed |
compress/gzip returned an error (Compress or Decompress) |
0.3.26.2 |
FlateFailed |
compress/flate returned an error (Compress or Decompress) |
0.3.26.3 |
ZlibFailed |
compress/zlib returned an error (Compress or Decompress), including a failed Adler-32 check |
Documentation
¶
Overview ¶
Package transform — shared decompression bound for the stdlib gzip, flate and zlib schemes. Each Decompress reads through an io.LimitReader so a malformed or hostile stream cannot drive an unbounded allocation at this layer. The full self-describing decompression-bomb guard (max output size + max expansion ratio keyed to the frame header) lives in the pkg/v1/data/codec frame layer (a later commit, CodeCompressedFrameInvalid); this is the conservative layer-local backstop ADR 0014 D1 asks for.
Package transform — the flate (raw DEFLATE) Compressor over compress/flate. Shares the package with the gzip scheme; both self-register at import.
Package transform wraps the stdlib compress/gzip and compress/flate codecs as core/data/transform.Compressor implementations. Blank-importing this package is enough to make "gzip" and "flate" resolvable via the core/data/transform registry.
Package transform — the recycled stdlib codecs behind Compress and Decompress. Neither direction's stdlib object is small: at the default level an encoder carries two hash tables and a 320 KiB history window, and a decoder carries a 32 KiB dictionary window. Constructing one per call put compress/flate at 99 % of the bytes a 256-byte gzip allocated and runtime.memclrNoHeapPointers at 18.7 % of its CPU — zeroing tables the call was about to overwrite. Recycling moves that off the per-call path; the before/after medians are in BENCH.md.
A pooled object is reachable until the next take on the same P or the next GC, whichever comes first, and it is large. That is why the two directions unbind differently, and the line is what the SDK created versus what the caller lent us: an encoder is rebound to io.Discard before it goes back, because it would otherwise pin the output buffer this package built and handed to the caller as exclusively theirs; a decoder is not rebound, because what it holds is the caller's own compressed input, which the caller passed in and still has — and because every decoder Reset re-reads a header and so returns an error that an unbind would have to discard, while every encoder Reset returns nothing at all.
Every take checks the pooled decoder rather than trusting it, and the check is in two parts because one of them cannot do the job alone: a type assertion establishes the Reset CAPABILITY, and a tag on the box establishes the SCHEME. readerScheme carries the measurement that forces the split — the flate and zlib Resetter interfaces declare the identical method, so each pool's assertion accepts the other pool's decoder and detects nothing.
A box holding another scheme's decoder is a corrupted pool, which is this package's own defect and not the caller's stream: nobody who called Decompress could have caused it and nobody could act on being told. So it is never an error. It fails the same condition an empty box fails, takes the same constructor path, and the fresh decoder overwrites the foreign one — so the fault costs exactly the allocation the pool exists to avoid, once, and does not outlive the take that found it.
Package transform — the WrapParams the three schemes attach to a stdlib cause. Each mirrors its sentinel — GzipFailed, FlateFailed, ZlibFailed, declared with their codes in internal/core/data/transform (ADR 0160) — so a wrapped compress/* failure carries the same Code/Reason/Public on the wire as the bare sentinel.
Package transform — the zlib Compressor over compress/zlib (RFC 1950): a two-byte header, a raw DEFLATE body, and an Adler-32 trailer. This is NOT the `flate` scheme: `flate` is the bare DEFLATE stream of RFC 1951, with no header and no checksum, so the two are not wire-compatible in either direction. HTTP's `Content-Encoding: deflate` (RFC 9110 §8.4.1) names the zlib envelope, so this scheme — not `flate` — is the one that interoperates with it. Shares the package with the gzip and flate schemes; all three self-register at import.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var FlateCompressor = coretransform.Register(flateCompressor{})
FlateCompressor is the flate (raw DEFLATE) singleton, registered with core/data/transform at package load. Binding the registration result to a named var keeps us clear of init().
var GzipCompressor = coretransform.Register(gzipCompressor{})
GzipCompressor is the gzip singleton, registered with core/data/transform at package load. Binding the registration result to a named var is idiomatic and keeps us clear of init().
var ZlibCompressor = coretransform.Register(zlibCompressor{level: zlib.DefaultCompression})
ZlibCompressor is the zlib singleton, registered with core/data/transform at package load at the stdlib default level. Binding the registration result to a named var is idiomatic and keeps us clear of init().
Functions ¶
func NewZlibCompressor ¶
func NewZlibCompressor(level int) coretransform.Compressor
NewZlibCompressor returns a zlib Compressor encoding at level, which may be zlib.DefaultCompression, zlib.HuffmanOnly, or any value from zlib.BestSpeed to zlib.BestCompression. Anything else — including zlib.NoCompression, which the stdlib accepts but which stores the payload verbatim — is clamped to zlib.DefaultCompression, so no configuration can hand back a compressor that silently does not compress (ADR 0031). The result is NOT registered; the registry entry is the default-level ZlibCompressor singleton.
Types ¶
This section is empty.
Source Files