Documentation
¶
Overview ¶
Package compress provides pluggable compression codecs used for both block-volume chunk frames (EncodeFrame) and per-file filesystem-volume entries (EncodeStream). EncodeFrame produces independent frames that may be concatenated into a valid multi-frame stream. EncodeStream writes one self-contained compressed stream per call and is the path used for per-file FS compression inside data.tar.
Index ¶
- Constants
- Variables
- func IsUserSelectable(name string) bool
- func Names() []string
- func NewReader(ext string, src io.Reader) (io.ReadCloser, error)
- func SkipZstdFrames(rs io.ReadSeeker, n int) (int64, error)
- func UserSelectableNames() []string
- func ZstdDecodedSize(rs io.ReadSeeker) (int64, error)
- type Codec
- type Encoder
- type Level
- type ZstdRawOffset
Constants ¶
const ( LevelFastest = zstd.SpeedFastest LevelDefault = zstd.SpeedDefault LevelBetter = zstd.SpeedBetterCompression LevelBest = zstd.SpeedBestCompression )
Predefined compression levels.
const DefaultCodecName = "zstd"
DefaultCodecName is the codec used when no --volume-compression flag is provided.
Variables ¶
var ErrCorruptZstdFrame = errors.New("corrupt zstd frame")
ErrCorruptZstdFrame is returned (wrapped) by SkipZstdFrames when the stream deviates from the subset of RFC 8878 this walker accepts: a non-Zstandard or skippable magic, a reserved/unused frame-header bit, a reserved block type, a malformed header shape, an arithmetic overflow, or any truncation. Compare with errors.Is.
var ErrUnknownCodec = errors.New("unknown codec")
ErrUnknownCodec is returned by New when the requested codec name is not registered.
Functions ¶
func IsUserSelectable ¶
IsUserSelectable reports whether name is currently in the user-selectable codec allow-list (see UserSelectableNames).
func NewReader ¶
NewReader returns a streaming decompressing io.ReadCloser for src, selecting the codec by ext (the file-extension convention used by Codec.Ext: ".zst", ".gz", ".lz4", or "" for no compression). It is the decode-side counterpart to New/Codec's Encode* methods.
All four readers decode a concatenation of independent codec frames/members written by EncodeFrame in one continuous Read stream — no per-frame loop is needed by the caller, matching how block-volume chunks are stored. ".lz4" is the odd one out internally: pierrec's lz4.Reader does not auto-continue past one frame's end marker into a concatenated next frame, so NewReader wraps it in a stateful frame-swap reader (see lz4FrameReader) that reproduces that behavior; ".zst" and ".gz" already auto-concatenate in the underlying library.
Callers MUST call Close on the returned io.ReadCloser once done reading: the zstd and gzip decoders hold internal buffers (and, for zstd, background goroutines) that are only released on Close.
NewReader returns an error wrapping ErrUnknownCodec for any ext it does not recognize.
func SkipZstdFrames ¶
func SkipZstdFrames(rs io.ReadSeeker, n int) (int64, error)
SkipZstdFrames returns the absolute byte offset, relative to the ReadSeeker's current position, at which frame n begins in a concatenation of independent Zstandard frames — i.e. the sum of the physical lengths of frames 0..n-1.
It walks the stream by reading only frame/block headers (and the optional content checksum) into small fixed buffers with io.ReadFull, and skipping every block payload with Seek. It never wraps the reader in a buffered reader and never decodes payloads, so the reader's offset stays exactly aligned with the walker's own accounting.
Because seeking past EOF is NOT an error for an io.ReadSeeker, existence of a byte range cannot be inferred from a successful Seek. SkipZstdFrames therefore establishes the real end bound up front (Seek to io.SeekEnd) and checks that enough bytes remain before every header read and every payload skip; a short stream fails with ErrCorruptZstdFrame rather than silently seeking into the void.
Before returning, it validates the magic and complete header of the target frame n (so a returned offset always points at a well-formed frame start), then restores the reader to the position it held on entry — the call is non-destructive, including for n == 0. n must be a valid frame index in [0, frameCount); asking for n == frameCount fails because frame n's magic cannot be read. A negative n is an argument error and does not wrap ErrCorruptZstdFrame.
func UserSelectableNames ¶
func UserSelectableNames() []string
UserSelectableNames returns the codec names a user may pass on the CLI (e.g. via --volume-compression), in the order they should be presented. See the userSelectableNames doc comment for why this differs from Names() and why it is expected to change over time. The returned slice is a copy; callers may not mutate the package's allow-list through it.
func ZstdDecodedSize ¶
func ZstdDecodedSize(rs io.ReadSeeker) (int64, error)
ZstdDecodedSize returns the sum of the mandatory Frame_Content_Size fields in every frame from the ReadSeeker's current position through EOF.
The proof reads only fixed-size frame and block metadata. Compressed block payloads are skipped with Seek, so memory and bytes read are independent of the decoded volume size. Every frame must declare its content size; a content-size-less, malformed, truncated, empty, or overflowing stream wraps ErrCorruptZstdFrame. This is a size preflight, not a payload-integrity proof: skipped payloads and frame checksums are not decoded or validated. The reader position is restored before return.
Types ¶
type Codec ¶
type Codec interface {
// Name returns the codec identifier, e.g. "zstd", "none".
Name() string
// Ext returns the file-extension suffix including the leading dot, e.g. ".zst".
// Returns "" for no compression so the output file carries no extension.
Ext() string
// EncodeFrame compresses src into a single self-contained frame.
// Multiple frames may be concatenated to form a valid multi-frame stream.
EncodeFrame(src []byte) ([]byte, error)
// EncodeStream compresses src into dst as one self-contained stream.
// It is safe to call concurrently from multiple goroutines.
EncodeStream(dst io.Writer, src io.Reader) error
// EncodeFrameStream writes one independent frame from src to dst. size is the
// expected raw length used by codecs that encode content-size metadata; its
// encoding need not be byte-identical to EncodeFrame. Implementations that
// genuinely stream bound memory to codec windows and buffers independent of
// size.
EncodeFrameStream(dst io.Writer, src io.Reader, size int64) error
}
Codec compresses data for snapshot volume payloads.
EncodeFrame is used for block-volume chunks: each call produces one independent frame; concatenated frames form a valid multi-frame stream decodable by standard tools (e.g. zstd, gunzip, lz4 -d).
EncodeStream is used for per-file FS compression: it streams src into dst as one self-contained compressed stream. Each call is safe to invoke concurrently. For codec=none, EncodeStream is a byte-identical passthrough.
EncodeFrameStream produces one independent frame from src, but its encoded bytes need not match EncodeFrame for the same input. size supplies the known raw length to codecs with content-size metadata. It is the finalize-time path for a block chunk whose raw bytes already live in a durable file on disk (see volume.downloadChunk), allowing codecs to keep memory bounded independently of the chunk size.
type Encoder ¶
type Encoder struct {
// contains filtered or unexported fields
}
Encoder compresses data using the zstd algorithm with CRC always enabled. Concurrent calls to EncodeFrame are safe. EncodeStream creates a new internal writer per call and is also safe to call concurrently.
func NewEncoder ¶
NewEncoder creates a new Encoder at the given compression level. CRC is always enabled to allow integrity checking at decode time.
func (*Encoder) EncodeFrame ¶
EncodeFrame compresses src into a single independent zstd frame and returns the compressed bytes. Multiple frames produced by EncodeFrame can be concatenated; the result is a valid multi-frame zstd stream that decodes to the concatenation of all original inputs. Use this to encode individual block-volume chunks before merging them into data.bin.zst.
type Level ¶
type Level = zstd.EncoderLevel
Level is the zstd compression level. Re-exported as a type alias so callers need not import the underlying library.
type ZstdRawOffset ¶
ZstdRawOffset describes a decoded offset's position in an independent-frame zstd stream. CompressedOffset is the start of the frame containing the requested raw byte, RawDiscard is the decoded prefix within that frame, and DecodedSize is the sum of every frame's mandatory Frame_Content_Size.
func LocateZstdRawOffset ¶
func LocateZstdRawOffset(ctx context.Context, rs io.ReadSeeker, rawOffset int64) (ZstdRawOffset, error)
LocateZstdRawOffset locates rawOffset in a concatenation of independent zstd frames without decoding payload bytes. It walks the complete stream so the returned location also proves that every frame is structurally complete, declares Frame_Content_Size, and that their decoded-size sum does not overflow int64.
An offset aligned to a frame starts at that frame with zero RawDiscard. An offset inside a frame starts at the preceding compressed boundary and discards only that frame's decoded prefix. EOF returns the compressed end with zero discard. The reader position is restored on every return path.
Source Files