zstd

package
v1.1.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 2, 2026 License: MIT, Apache-2.0, BSD-3-Clause, + 1 more Imports: 17 Imported by: 0

Documentation

Overview

Package zstd provides decompression of zstandard files.

For advanced usage and examples, go to the README: https://github.com/klauspost/compress/tree/master/zstd#zstd

Index

Constants

View Source
const (
	// MinWindowSize is the minimum Window Size, which is 1 KB.
	MinWindowSize = 1 << 10

	// MaxWindowSize is the maximum encoder window size
	// and the default decoder maximum window size.
	MaxWindowSize = 1 << 29
)

Variables

View Source
var (
	// ErrReservedBlockType is returned when a reserved block type is found.
	// Typically this indicates wrong or corrupted input.
	ErrReservedBlockType = errors.New("invalid input: reserved block type encountered")

	// ErrCompressedSizeTooBig is returned when a block is bigger than allowed.
	// Typically this indicates wrong or corrupted input.
	ErrCompressedSizeTooBig = errors.New("invalid input: compressed size too big")

	// ErrBlockTooSmall is returned when a block is too small to be decoded.
	// Typically returned on invalid input.
	ErrBlockTooSmall = errors.New("block too small")

	// ErrUnexpectedBlockSize is returned when a block has unexpected size.
	// Typically returned on invalid input.
	ErrUnexpectedBlockSize = errors.New("unexpected block size")

	// ErrMagicMismatch is returned when a "magic" number isn't what is expected.
	// Typically this indicates wrong or corrupted input.
	ErrMagicMismatch = errors.New("invalid input: magic number mismatch")

	// ErrWindowSizeExceeded is returned when a reference exceeds the valid window size.
	// Typically this indicates wrong or corrupted input.
	ErrWindowSizeExceeded = errors.New("window size exceeded")

	// ErrWindowSizeTooSmall is returned when no window size is specified.
	// Typically this indicates wrong or corrupted input.
	ErrWindowSizeTooSmall = errors.New("invalid input: window size was too small")

	// ErrDecoderSizeExceeded is returned if decompressed size exceeds the configured limit.
	ErrDecoderSizeExceeded = errors.New("decompressed size exceeds configured limit")

	// ErrUnknownDictionary is returned if the dictionary ID is unknown.
	ErrUnknownDictionary = errors.New("unknown dictionary")

	// ErrFrameSizeExceeded is returned if the stated frame size is exceeded.
	// This is only returned if SingleSegment is specified on the frame.
	ErrFrameSizeExceeded = errors.New("frame size exceeded")

	// ErrFrameSizeMismatch is returned if the stated frame size does not match the expected size.
	// This is only returned if SingleSegment is specified on the frame.
	ErrFrameSizeMismatch = errors.New("frame size does not match size on stream")

	// ErrCRCMismatch is returned if CRC mismatches.
	ErrCRCMismatch = errors.New("CRC check failed")

	// ErrDecoderClosed will be returned if the Decoder was used after
	// Close has been called.
	ErrDecoderClosed = errors.New("decoder used after Close")

	// ErrEncoderClosed will be returned if the Encoder was used after
	// Close has been called.
	ErrEncoderClosed = errors.New("encoder used after Close")

	// ErrDecoderNilInput is returned when a nil Reader was provided
	// and an operation other than Reset/DecodeAll/Close was attempted.
	ErrDecoderNilInput = errors.New("nil input provided as reader")
)

Functions

This section is empty.

Types

type BuildDictOptions

type BuildDictOptions struct {
	// Dictionary ID.
	ID uint32

	// Content to use to create dictionary tables.
	Contents [][]byte

	// History to use for all blocks.
	History []byte

	// Offsets to use.
	Offsets [3]int

	// CompatV155 will make the dictionary compatible with Zstd v1.5.5 and earlier.
	// See https://github.com/facebook/zstd/issues/3724
	CompatV155 bool

	// Use the specified encoder level.
	// The dictionary will be built using the specified encoder level,
	// which will reflect speed and make the dictionary tailored for that level.
	// If not set SpeedBestCompression will be used.
	Level EncoderLevel

	// DebugOut will write stats and other details here if set.
	DebugOut io.Writer
}

type EOption

type EOption func(*encoderOptions) error

EOption is an option for creating a encoder.

func WithEncoderCRC

func WithEncoderCRC(b bool) EOption

WithEncoderCRC will add CRC value to output. Output will be 4 bytes larger. Can be changed with ResetWithOptions.

func WithEncoderLevel

func WithEncoderLevel(l EncoderLevel) EOption

WithEncoderLevel specifies a predefined compression level. Cannot be changed with ResetWithOptions.

func WithSingleSegment

func WithSingleSegment(b bool) EOption

WithSingleSegment will set the "single segment" flag when EncodeAll is used. If this flag is set, data must be regenerated within a single continuous memory segment. In this case, Window_Descriptor byte is skipped, but Frame_Content_Size is necessarily present. As a consequence, the decoder must allocate a memory segment of size equal or larger than size of your content. In order to preserve the decoder from unreasonable memory requirements, a decoder is allowed to reject a compressed frame which requests a memory size beyond decoder's authorized range. For broader compatibility, decoders are recommended to support memory sizes of at least 8 MB. This is only a recommendation, each decoder is free to support higher or lower limits, depending on local limitations. If this is not specified, block encodes will automatically choose this based on the input size and the window size. This setting has no effect on streamed encodes. Can be changed with ResetWithOptions.

func WithZeroFrames

func WithZeroFrames(b bool) EOption

WithZeroFrames will encode 0 length input as full frames. This can be needed for compatibility with zstandard usage, but is not needed for this package. Can be changed with ResetWithOptions.

type Encoder

type Encoder struct {
	// contains filtered or unexported fields
}

Encoder provides encoding to Zstandard. An Encoder can be used for either compressing a stream via the io.WriteCloser interface supported by the Encoder or as multiple independent tasks via the EncodeAll function. Smaller encodes are encouraged to use the EncodeAll function. Use NewWriter to create a new instance.

func NewWriter

func NewWriter(w io.Writer, opts ...EOption) (*Encoder, error)

NewWriter will create a new Zstandard encoder. If the encoder will be used for encoding blocks a nil writer can be used.

func (*Encoder) EncodeAll

func (e *Encoder) EncodeAll(src, dst []byte) []byte

EncodeAll will encode all input in src and append it to dst. This function can be called concurrently, but each call will only run on a single goroutine. If empty input is given, nothing is returned, unless WithZeroFrames is specified. Encoded blocks can be concatenated and the result will be the combined input stream. Data compressed with EncodeAll can be decoded with the Decoder, using either a stream or DecodeAll.

func (*Encoder) Reset

func (e *Encoder) Reset(w io.Writer)

Reset will re-initialize the writer and new writes will encode to the supplied writer as a new, independent stream.

type EncoderLevel

type EncoderLevel int

EncoderLevel predefines encoder compression levels. Only use the constants made available, since the actual mapping of these values are very likely to change and your compression could change unpredictably when upgrading the library.

const (

	// SpeedFastest will choose the fastest reasonable compression.
	// This is roughly equivalent to the fastest Zstandard mode.
	SpeedFastest EncoderLevel

	// SpeedDefault is the default "pretty fast" compression option.
	// This is roughly equivalent to the default Zstandard mode (level 3).
	SpeedDefault

	// SpeedBetterCompression will yield better compression than the default.
	// Currently it is about zstd level 7-8 with ~ 2x-3x the default CPU usage.
	// By using this, notice that CPU usage may go up in the future.
	SpeedBetterCompression

	// SpeedBestCompression will choose the best available compression option.
	// This will offer the best compression no matter the CPU cost.
	SpeedBestCompression
)

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL