transform

package
v0.12.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 3, 2026 License: MIT Imports: 4 Imported by: 0

README

transform

import "github.com/kitsunium/sdk/pkg/v1/data/transform"

Package transform compresses and decompresses bytes with the SDK's compressors, without the codec package (ADR 0014). Importing it registers three, which the standard library implements:

box, err := transform.Compress(transform.Gzip, nil, payload)
plain, err := transform.DecompressBounded(transform.Gzip, nil, box, 1<<20)
Algorithms

An Algorithm names a Compressor in a registry of its own — compression is a transform of bytes, never a codec Format. Gzip is RFC 1952, Flate the raw DEFLATE stream of RFC 1951, and Zlib the RFC 1950 envelope HTTP misnames "deflate": three wire formats, not aliases of one another. A compressor another package registers — the module github.com/kitsunium/sdk/third-party/transform adds zstd and s2 — is reached by its Algorithm the same way once that package is imported; Available lists what is registered.

Every function appends to dst, which may be nil, and src must not overlap dst's spare capacity: a scheme writes there while it still reads src.

Bounds

Decompress is bounded by the scheme's own backstop — 256 MiB for the three registered here — so a bomb cannot exhaust memory, but it is the scheme's number, not the caller's. DecompressBounded takes the caller's ceiling: a scheme implementing BoundedDecompressor stops at it, so the ceiling bounds the work and not only the verdict; a scheme that does not is decompressed under its own backstop and judged afterwards. Either way no more than limit bytes are returned, and a stream that would exceed it is refused with DecompressedTooLarge rather than cut short — a truncated success would turn a bomb into a parsing bug further up.

Streams

NewReader and NewWriter take an io.Reader or an io.Writer of any length and never hold it whole, through a scheme implementing StreamingCompressor — the three registered here, and the third-party module's zstd and s2:

w, err := transform.NewWriter(transform.Gzip, conn)
r, err := transform.NewReader(transform.Gzip, body, 64<<20)

The reader's limit is the stream's ceiling: it yields at most limit bytes, and the read past them fails with DecompressedTooLarge. Close the writer, or the stream has no end; close the reader, or its decoder is not reused.

Errors

Match them with errs.HasCode, or the sentinels with errors.Is:

CodeUnknownCompressor       0.2.5.1   no compressor is registered under the Algorithm
CodeCompressedFrameInvalid  0.2.5.4   the codec package's compressed frame is malformed, or its bomb guard tripped
CodeDecompressedTooLarge    0.2.5.6   the plaintext would exceed DecompressBounded's or NewReader's ceiling
CodeStreamingUnsupported    0.2.5.7   the registered scheme has no stream form
CodeGzipFailed              0.3.26.1  compress/gzip refused the stream, compressing or decompressing
CodeFlateFailed             0.3.26.2  compress/flate refused the stream
CodeZlibFailed              0.3.26.3  compress/zlib refused the stream

A malformed stream is the scheme's own refusal, with the scheme's code: GzipFailed, FlateFailed or ZlibFailed for the three registered here, the stdlib's own error kept in the chain beneath it.

Index

Constants

const CodeCompressedFrameInvalid errs.Code = 132356

CodeCompressedFrameInvalid identifies a malformed compressed frame of the codec package, or a frame its decompression-bomb guard refused (0.2.5.4).

Initialised with github.com/kitsunium/sdk/internal/core/data/transform.CodeCompressedFrameInvalid.

Code 0.2.5.4

const CodeDecompressedTooLarge errs.Code = 132358

CodeDecompressedTooLarge identifies a stream whose plaintext would exceed the ceiling its caller gave (0.2.5.6).

Initialised with github.com/kitsunium/sdk/internal/core/data/transform.CodeDecompressedTooLarge.

Code 0.2.5.6

const CodeFlateFailed errs.Code = 203266

CodeFlateFailed identifies a stream compress/flate refused (0.3.26.2).

Initialised with github.com/kitsunium/sdk/internal/core/data/transform.CodeFlateFailed.

Code 0.3.26.2

const CodeGzipFailed errs.Code = 203265

CodeGzipFailed identifies a stream compress/gzip refused, compressing or decompressing (0.3.26.1).

Initialised with github.com/kitsunium/sdk/internal/core/data/transform.CodeGzipFailed.

Code 0.3.26.1

const CodeStreamingUnsupported errs.Code = 132359

CodeStreamingUnsupported identifies a stream asked of a registered scheme that has no stream form (0.2.5.7).

Initialised with github.com/kitsunium/sdk/internal/core/data/transform.CodeStreamingUnsupported.

Code 0.2.5.7

const CodeUnknownCompressor errs.Code = 132353

CodeUnknownCompressor identifies an Algorithm no imported package has registered (0.2.5.1).

Initialised with github.com/kitsunium/sdk/internal/core/data/transform.CodeUnknownCompressor.

Code 0.2.5.1

const CodeZlibFailed errs.Code = 203267

CodeZlibFailed identifies a stream compress/zlib refused (0.3.26.3): the RFC 1950 envelope, told apart from raw DEFLATE because a stream one of them refuses is often valid under the other.

Initialised with github.com/kitsunium/sdk/internal/core/data/transform.CodeZlibFailed.

Code 0.3.26.3

const Flate Algorithm = "flate"

Flate is the raw DEFLATE stream of RFC 1951, with no envelope.

const Gzip Algorithm = "gzip"

Gzip is RFC 1952: a DEFLATE stream in a gzip envelope with a CRC-32.

const Zlib Algorithm = "zlib"

Zlib is the RFC 1950 envelope with an Adler-32 — what HTTP calls "deflate".

Variables

var CompressedFrameInvalid *errs.Error

CompressedFrameInvalid is the sentinel of CodeCompressedFrameInvalid.

Initialised with github.com/kitsunium/sdk/internal/core/data/transform.CompressedFrameInvalid.

Code 0.2.5.4 · reason COMPRESSED_FRAME_INVALID · public "Compressed frame is malformed or exceeds the safety bound"

var DecompressedTooLarge *errs.Error

DecompressedTooLarge is the sentinel of CodeDecompressedTooLarge, returned as is.

Initialised with github.com/kitsunium/sdk/internal/core/data/transform.DecompressedTooLarge.

Code 0.2.5.6 · reason DECOMPRESSED_TOO_LARGE · public "Decompressed payload exceeds the caller's ceiling"

var FlateFailed *errs.Error

FlateFailed is the sentinel of CodeFlateFailed.

Initialised with github.com/kitsunium/sdk/internal/core/data/transform.FlateFailed.

Code 0.3.26.2 · reason FLATE_FAILED · public "flate transform failed"

var GzipFailed *errs.Error

GzipFailed is the sentinel of CodeGzipFailed; the stdlib's error is kept beneath it.

Initialised with github.com/kitsunium/sdk/internal/core/data/transform.GzipFailed.

Code 0.3.26.1 · reason GZIP_FAILED · public "gzip transform failed"

var StreamingUnsupported *errs.Error

StreamingUnsupported is the sentinel of CodeStreamingUnsupported, returned as is.

Initialised with github.com/kitsunium/sdk/internal/core/data/transform.StreamingUnsupported.

Code 0.2.5.7 · reason STREAMING_UNSUPPORTED · public "The compressor has no streaming form"

var UnknownCompressor *errs.Error

UnknownCompressor is the sentinel of CodeUnknownCompressor, returned as is.

Initialised with github.com/kitsunium/sdk/internal/core/data/transform.UnknownCompressor.

Code 0.2.5.1 · reason UNKNOWN_COMPRESSOR · public "No compressor is registered under that algorithm"

var ZlibFailed *errs.Error

ZlibFailed is the sentinel of CodeZlibFailed.

Initialised with github.com/kitsunium/sdk/internal/core/data/transform.ZlibFailed.

Code 0.3.26.3 · reason ZLIB_FAILED · public "zlib transform failed"

func Available

func Available() []Algorithm

Available returns the registered Algorithms, sorted.

func Compress

func Compress(algo Algorithm, dst []byte, src []byte) (encoded []byte, err error)

Compress compresses src with the scheme registered under algo and appends the result to dst. An unregistered algo is refused with UnknownCompressor and dst comes back as it was.

func Decompress

func Decompress(algo Algorithm, dst []byte, src []byte) (decoded []byte, err error)

Decompress decompresses src with the scheme registered under algo and appends the result to dst, bounded by the scheme's own backstop. To bound it by a number of your own, call DecompressBounded. An unregistered algo is refused with UnknownCompressor and dst comes back as it was.

func DecompressBounded

func DecompressBounded(algo Algorithm, dst []byte, src []byte, limit int64) (decoded []byte, err error)

DecompressBounded decompresses src with the scheme registered under algo and appends the result to dst, producing at most limit bytes of plaintext: a stream that would produce more is refused with DecompressedTooLarge and dst comes back as it was. A limit of zero admits an empty stream and nothing else, and a negative limit is read as zero rather than given a meaning of its own.

A scheme implementing BoundedDecompressor stops at limit, so the work is bounded too. A scheme that does not — a consumer's own, or one of the third-party module's — is decompressed under its own backstop and judged afterwards: declining to tighten the work can never change the verdict, only the bytes touched reaching it.

func Lookup

func Lookup(algo Algorithm) (c Compressor, ok bool)

Lookup returns the Compressor registered under algo.

func NewFlateCompressor

func NewFlateCompressor(level int) Compressor

NewFlateCompressor returns a raw-DEFLATE Compressor encoding at level, a compress/flate constant, clamped as NewGzipCompressor's is. The result is not registered.

func NewGzipCompressor

func NewGzipCompressor(level int) Compressor

NewGzipCompressor returns a gzip Compressor encoding at level — a compress/gzip constant, BestSpeed to BestCompression, HuffmanOnly or DefaultCompression. Any other level, NoCompression included, is read as DefaultCompression: no level gives a compressor that does not compress. The result is not registered; pass it where a Compressor is taken.

func NewReader

func NewReader(algo Algorithm, r io.Reader, limit int64) (rc io.ReadCloser, err error)

NewReader returns a reader of r's stream decompressed by the scheme registered under algo, yielding at most limit bytes: the read that would deliver the byte past limit fails with DecompressedTooLarge, after limit bytes and never instead of them. A limit of zero admits an empty stream and a negative one is read as zero, as in DecompressBounded. Nothing is held whole — an HTTP body, a file or a log shipment is read through the decoder's window — so the limit bounds what the caller receives, and the decoder's own memory is its window. Close the reader to release the decoder; it does not close r.

An unregistered algo is refused with UnknownCompressor, a registered one with no stream form with StreamingUnsupported.

func NewWriter

func NewWriter(algo Algorithm, w io.Writer) (wc io.WriteCloser, err error)

NewWriter returns a writer that compresses into w with the scheme registered under algo. Close it to write the stream's end — a stream not closed is incomplete — and release the encoder; it does not close w. What it writes NewReader reads back; for every scheme but s2 it is also what Compress writes and Decompress reads.

An unregistered algo is refused with UnknownCompressor, a registered one with no stream form with StreamingUnsupported.

func NewZlibCompressor

func NewZlibCompressor(level int) Compressor

NewZlibCompressor returns a zlib Compressor encoding at level, a compress/zlib constant, clamped as NewGzipCompressor's is. The result is not registered.

type Algorithm

Algorithm is the name a Compressor registers under. The zero value is reserved as invalid.

type Algorithm = coretransform.Algorithm

Algorithm is an alias of github.com/kitsunium/sdk/internal/core/data/transform.Algorithm, declared as:

type Algorithm string

func (Algorithm) Known
func (Algorithm) Known() bool

Known reports whether a has been registered in the Compressor registry.

func (Algorithm) String
func (Algorithm) String() string

String implements fmt.Stringer and returns the raw identifier.

type BoundedDecompressor

BoundedDecompressor is the optional extension of a Compressor that can be told how much plaintext it may produce; DecompressBounded uses it when the scheme has it.

type BoundedDecompressor = coretransform.BoundedDecompressor

BoundedDecompressor is an alias of github.com/kitsunium/sdk/internal/core/data/transform.BoundedDecompressor, declared as:

type BoundedDecompressor interface {
	DecompressBounded(dst []byte, src []byte, max int64) (out []byte, err error)
	Compressor
}

func (BoundedDecompressor) DecompressBounded
func (BoundedDecompressor) DecompressBounded(dst []byte, src []byte, max int64) (out []byte, err error)

type Compressor

Compressor is one compression scheme: Compress and Decompress append to dst. Implementations are safe for concurrent use.

type Compressor = coretransform.Compressor

Compressor is an alias of github.com/kitsunium/sdk/internal/core/data/transform.Compressor, declared as:

type Compressor interface {
	Algorithm() Algorithm
	Compress(dst []byte, src []byte) (out []byte, err error)
	Decompress(dst []byte, src []byte) (out []byte, err error)
}

func (Compressor) Algorithm
func (Compressor) Algorithm() Algorithm

func (Compressor) Compress
func (Compressor) Compress(dst []byte, src []byte) (out []byte, err error)

func (Compressor) Decompress
func (Compressor) Decompress(dst []byte, src []byte) (out []byte, err error)

type StreamingCompressor

StreamingCompressor is the optional extension of a Compressor that compresses and decompresses streams; NewReader and NewWriter use it.

type StreamingCompressor = coretransform.StreamingCompressor

StreamingCompressor is an alias of github.com/kitsunium/sdk/internal/core/data/transform.StreamingCompressor, declared as:

type StreamingCompressor interface {
	NewReader(r io.Reader, max int64) (rc io.ReadCloser, err error)
	NewWriter(w io.Writer) (wc io.WriteCloser, err error)
	Compressor
}

func (StreamingCompressor) NewReader
func (StreamingCompressor) NewReader(r io.Reader, max int64) (rc io.ReadCloser, err error)

func (StreamingCompressor) NewWriter
func (StreamingCompressor) NewWriter(w io.Writer) (wc io.WriteCloser, err error)

Generated from docs/api by tools/genindex.

Documentation

Overview

Package transform compresses and decompresses bytes with the SDK's compressors, without the codec package (ADR 0014). Importing it registers three, which the standard library implements: 

box, err := transform.Compress(transform.Gzip, nil, payload)
plain, err := transform.DecompressBounded(transform.Gzip, nil, box, 1<<20)

Algorithms

An Algorithm names a Compressor in a registry of its own — compression is a transform of bytes, never a codec Format. Gzip is RFC 1952, Flate the raw DEFLATE stream of RFC 1951, and Zlib the RFC 1950 envelope HTTP misnames "deflate": three wire formats, not aliases of one another. A compressor another package registers — the module github.com/kitsunium/sdk/third-party/transform adds zstd and s2 — is reached by its Algorithm the same way once that package is imported; Available lists what is registered.

Every function appends to dst, which may be nil, and src must not overlap dst's spare capacity: a scheme writes there while it still reads src.

Bounds

Decompress is bounded by the scheme's own backstop — 256 MiB for the three registered here — so a bomb cannot exhaust memory, but it is the scheme's number, not the caller's. DecompressBounded takes the caller's ceiling: a scheme implementing BoundedDecompressor stops at it, so the ceiling bounds the work and not only the verdict; a scheme that does not is decompressed under its own backstop and judged afterwards. Either way no more than limit bytes are returned, and a stream that would exceed it is refused with DecompressedTooLarge rather than cut short — a truncated success would turn a bomb into a parsing bug further up.

Streams

NewReader and NewWriter take an io.Reader or an io.Writer of any length and never hold it whole, through a scheme implementing StreamingCompressor — the three registered here, and the third-party module's zstd and s2: 

w, err := transform.NewWriter(transform.Gzip, conn)
r, err := transform.NewReader(transform.Gzip, body, 64<<20)

The reader's limit is the stream's ceiling: it yields at most limit bytes, and the read past them fails with DecompressedTooLarge. Close the writer, or the stream has no end; close the reader, or its decoder is not reused.

Errors

Match them with errs.HasCode, or the sentinels with errors.Is: 

CodeUnknownCompressor       0.2.5.1   no compressor is registered under the Algorithm
CodeCompressedFrameInvalid  0.2.5.4   the codec package's compressed frame is malformed, or its bomb guard tripped
CodeDecompressedTooLarge    0.2.5.6   the plaintext would exceed DecompressBounded's or NewReader's ceiling
CodeStreamingUnsupported    0.2.5.7   the registered scheme has no stream form
CodeGzipFailed              0.3.26.1  compress/gzip refused the stream, compressing or decompressing
CodeFlateFailed             0.3.26.2  compress/flate refused the stream
CodeZlibFailed              0.3.26.3  compress/zlib refused the stream

A malformed stream is the scheme's own refusal, with the scheme's code: GzipFailed, FlateFailed or ZlibFailed for the three registered here, the stdlib's own error kept in the chain beneath it.

Index

Constants

View Source 
const (
	// CodeUnknownCompressor identifies an Algorithm no imported package has
	// registered (0.2.5.1).
	CodeUnknownCompressor errs.Code = coretransform.CodeUnknownCompressor
	// CodeCompressedFrameInvalid identifies a malformed compressed frame of the
	// codec package, or a frame its decompression-bomb guard refused (0.2.5.4).
	CodeCompressedFrameInvalid errs.Code = coretransform.CodeCompressedFrameInvalid
	// CodeDecompressedTooLarge identifies a stream whose plaintext would exceed
	// the ceiling its caller gave (0.2.5.6).
	CodeDecompressedTooLarge errs.Code = coretransform.CodeDecompressedTooLarge
	// CodeStreamingUnsupported identifies a stream asked of a registered scheme
	// that has no stream form (0.2.5.7).
	CodeStreamingUnsupported errs.Code = coretransform.CodeStreamingUnsupported
	// CodeGzipFailed identifies a stream compress/gzip refused, compressing or
	// decompressing (0.3.26.1).
	CodeGzipFailed errs.Code = coretransform.CodeGzipFailed
	// CodeFlateFailed identifies a stream compress/flate refused (0.3.26.2).
	CodeFlateFailed errs.Code = coretransform.CodeFlateFailed
	// CodeZlibFailed identifies a stream compress/zlib refused (0.3.26.3): the
	// RFC 1950 envelope, told apart from raw DEFLATE because a stream one of
	// them refuses is often valid under the other.
	CodeZlibFailed errs.Code = coretransform.CodeZlibFailed
)

The error codes, ranges 0.2.5.* and 0.3.26.*, declared in the core.

Variables

View Source 
var (
	// UnknownCompressor is the sentinel of [CodeUnknownCompressor], returned
	// as is.
	UnknownCompressor = coretransform.UnknownCompressor
	// CompressedFrameInvalid is the sentinel of [CodeCompressedFrameInvalid].
	CompressedFrameInvalid = coretransform.CompressedFrameInvalid
	// DecompressedTooLarge is the sentinel of [CodeDecompressedTooLarge],
	// returned as is.
	DecompressedTooLarge = coretransform.DecompressedTooLarge
	// StreamingUnsupported is the sentinel of [CodeStreamingUnsupported],
	// returned as is.
	StreamingUnsupported = coretransform.StreamingUnsupported
	// GzipFailed is the sentinel of [CodeGzipFailed]; the stdlib's error is
	// kept beneath it.
	GzipFailed = coretransform.GzipFailed
	// FlateFailed is the sentinel of [CodeFlateFailed].
	FlateFailed = coretransform.FlateFailed
	// ZlibFailed is the sentinel of [CodeZlibFailed].
	ZlibFailed = coretransform.ZlibFailed
)

The sentinels, for errors.Is: each carries the code of the same name.

Functions

func Compress 

func Compress(algo Algorithm, dst, src []byte) (encoded []byte, err error)

Compress compresses src with the scheme registered under algo and appends the result to dst. An unregistered algo is refused with UnknownCompressor and dst comes back as it was.

func Decompress 

func Decompress(algo Algorithm, dst, src []byte) (decoded []byte, err error)

Decompress decompresses src with the scheme registered under algo and appends the result to dst, bounded by the scheme's own backstop. To bound it by a number of your own, call DecompressBounded. An unregistered algo is refused with UnknownCompressor and dst comes back as it was.

func DecompressBounded 

func DecompressBounded(algo Algorithm, dst, src []byte, limit int64) (decoded []byte, err error)

DecompressBounded decompresses src with the scheme registered under algo and appends the result to dst, producing at most limit bytes of plaintext: a stream that would produce more is refused with DecompressedTooLarge and dst comes back as it was. A limit of zero admits an empty stream and nothing else, and a negative limit is read as zero rather than given a meaning of its own.

A scheme implementing BoundedDecompressor stops at limit, so the work is bounded too. A scheme that does not — a consumer's own, or one of the third-party module's — is decompressed under its own backstop and judged afterwards: declining to tighten the work can never change the verdict, only the bytes touched reaching it.

func NewReader added in v0.12.0 

func NewReader(algo Algorithm, r io.Reader, limit int64) (rc io.ReadCloser, err error)

NewReader returns a reader of r's stream decompressed by the scheme registered under algo, yielding at most limit bytes: the read that would deliver the byte past limit fails with DecompressedTooLarge, after limit bytes and never instead of them. A limit of zero admits an empty stream and a negative one is read as zero, as in DecompressBounded. Nothing is held whole — an HTTP body, a file or a log shipment is read through the decoder's window — so the limit bounds what the caller receives, and the decoder's own memory is its window. Close the reader to release the decoder; it does not close r.

An unregistered algo is refused with UnknownCompressor, a registered one with no stream form with StreamingUnsupported.

func NewWriter added in v0.12.0 

func NewWriter(algo Algorithm, w io.Writer) (wc io.WriteCloser, err error)

NewWriter returns a writer that compresses into w with the scheme registered under algo. Close it to write the stream's end — a stream not closed is incomplete — and release the encoder; it does not close w. What it writes NewReader reads back; for every scheme but s2 it is also what Compress writes and Decompress reads.

An unregistered algo is refused with UnknownCompressor, a registered one with no stream form with StreamingUnsupported.

Types

type Algorithm 

type Algorithm = coretransform.Algorithm

Algorithm is the name a Compressor registers under. The zero value is reserved as invalid. 

const (
	// Gzip is RFC 1952: a DEFLATE stream in a gzip envelope with a CRC-32.
	Gzip Algorithm = "gzip"
	// Flate is the raw DEFLATE stream of RFC 1951, with no envelope.
	Flate Algorithm = "flate"
	// Zlib is the RFC 1950 envelope with an Adler-32 — what HTTP calls
	// "deflate".
	Zlib Algorithm = "zlib"
)

The Algorithms registered by importing this package.

func Available 

func Available() []Algorithm

Available returns the registered Algorithms, sorted.

type BoundedDecompressor 

type BoundedDecompressor = coretransform.BoundedDecompressor

BoundedDecompressor is the optional extension of a Compressor that can be told how much plaintext it may produce; DecompressBounded uses it when the scheme has it.

type Compressor 

type Compressor = coretransform.Compressor

Compressor is one compression scheme: Compress and Decompress append to dst. Implementations are safe for concurrent use.

func Lookup 

func Lookup(algo Algorithm) (c Compressor, ok bool)

Lookup returns the Compressor registered under algo.

func NewFlateCompressor added in v0.12.0 

func NewFlateCompressor(level int) Compressor

NewFlateCompressor returns a raw-DEFLATE Compressor encoding at level, a compress/flate constant, clamped as NewGzipCompressor's is. The result is not registered.

func NewGzipCompressor added in v0.12.0 

func NewGzipCompressor(level int) Compressor

NewGzipCompressor returns a gzip Compressor encoding at level — a compress/gzip constant, BestSpeed to BestCompression, HuffmanOnly or DefaultCompression. Any other level, NoCompression included, is read as DefaultCompression: no level gives a compressor that does not compress. The result is not registered; pass it where a Compressor is taken.

func NewZlibCompressor added in v0.12.0 

func NewZlibCompressor(level int) Compressor

NewZlibCompressor returns a zlib Compressor encoding at level, a compress/zlib constant, clamped as NewGzipCompressor's is. The result is not registered.

type StreamingCompressor added in v0.12.0 

type StreamingCompressor = coretransform.StreamingCompressor

StreamingCompressor is the optional extension of a Compressor that compresses and decompresses streams; NewReader and NewWriter use it.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.