Documentation
¶
Overview ¶
Package v3 implements the C1Z3 envelope format: 5-byte magic, a length-prefixed proto manifest, and a payload (typically a zstd-tar of a Pebble directory).
The envelope is independent of the engine it carries. v3 readers inspect the manifest's `engine` field to dispatch; new engines plug in without an envelope change. The descriptor sidecar makes every v3 file self-describing — any reader can resolve every stored record's fields without compiled-in protos for that record type.
Index ¶
- Variables
- func BuildDescriptorClosure() (*descriptorpb.FileDescriptorSet, error)
- func ExtractZstdTar(r io.Reader, destDir string) error
- func MarshalManifest(m *c1zv3.C1ZManifestV3) ([]byte, error)
- func ReadIndexedFrameIndex(f *os.File) (*c1zv3.IndexedFrameIndex, error)
- func ReadManifestHeader(r io.Reader) (*c1zv3.C1ZManifestV3, error)
- func UnmarshalManifest(b []byte) (*c1zv3.C1ZManifestV3, error)
- func VerifyDescriptorClosure(set *descriptorpb.FileDescriptorSet) error
- func WriteEnvelope(w io.Writer, m *c1zv3.C1ZManifestV3, payloadDir string) error
- type DecoderPool
- type Envelope
- type PayloadOption
- type PayloadReuse
- type ReuseEntry
- type SpliceStats
Constants ¶
This section is empty.
Variables ¶
var ( ErrManifestInvalid = errors.New("c1z v3: manifest unmarshal failed") ErrManifestIncompleteDescriptors = errors.New("c1z v3: manifest descriptor closure incomplete") ErrManifestEmptyEngine = errors.New("c1z v3: manifest engine field is empty") )
Sentinel errors for manifest operations. These are duplicated from pkg/dotc1z/engine/pebble's sentinel set because importing the engine package from format/v3 would create a cycle (engine imports format/v3 for the save protocol). The two sentinels are kept structurally equivalent; callers can use errors.Is against either.
var C1Z3Magic = []byte("C1Z3\x00")
C1Z3Magic is the 5-byte header that identifies a v3 c1z file. Same value as dotc1z.C1Z3FileHeader; duplicated here so the format package doesn't import its parent (it's the layer below).
var ErrEnvelopeTruncated = errors.New("c1z v3: envelope truncated")
ErrEnvelopeTruncated is returned when the reader hits EOF mid-header or mid-manifest. This is a corruption-class error; the C1 corruption classifier should auto-invalidate the live c1z and trigger a re-sync.
var ErrInvalidV3Magic = errors.New("c1z v3: invalid magic; not a v3 file")
ErrInvalidV3Magic is returned by ReadEnvelope when the first 5 bytes don't match C1Z3Magic. Callers that want to dispatch across v1 and v3 should use dotc1z.ReadHeaderFormat (which delegates to this package for the v3 detection only after a v1/v3 magic-byte choice).
var ErrMaxSizeExceeded = fmt.Errorf("c1z v3: max decoded payload size exceeded, increase via the %s environment variable", maxDecodedSizeEnvVar)
ErrMaxSizeExceeded is returned (wrapped) when a payload decodes to more bytes than the configured cap. Guards against decompression bombs in untrusted v3 c1z files.
Functions ¶
func BuildDescriptorClosure ¶
func BuildDescriptorClosure() (*descriptorpb.FileDescriptorSet, error)
BuildDescriptorClosure walks every c1.storage.v3 file currently registered in protoregistry.GlobalFiles and returns a transitively- closed FileDescriptorSet containing them plus every file they transitively import. The result is what gets pinned into a v3 manifest's `descriptors` field at save time.
The closure invariant: for every file F in the result, every file F imports is also in the result. Reader-side verification can detect any missing import and return ErrManifestIncompleteDescriptors.
func ExtractZstdTar ¶
ExtractZstdTar reads a zstd-tar payload stream from r and unpacks it into destDir. destDir must exist. Used by the engine to rematerialize a Pebble directory at open time.
Despite the name, the function accepts ANY tar stream — the Envelope's payload reader has already transparently decoded the outer zstd layer when the encoding was TAR_ZSTD, or is a plain tar reader when the encoding was TAR. Same downstream code path.
Parallelism: the tar reader pulls bytes from the underlying stream serially in this goroutine (tar framing is sequential). Regular-file entries up to inlineCopyThresholdBytes are read into a freshly-allocated buffer and dispatched (target, mode, buffer) to a writer worker pool; workers perform the per-file open/write/close syscalls in parallel. Larger entries are streamed straight to disk on this goroutine so a hostile archive full of multi-GiB entries can't drive memory up with the worker fan-out. Memory peak is bounded by (extractWorkerCount + channel buffer) × inlineCopyThresholdBytes; at Pebble's typical 2 MiB FlushSplitBytes nearly every entry takes the parallel path — the per-entry parallelism win compounds at production-scale c1z files (100s GB).
Aggregate extraction is bounded too: when r is an Envelope's PayloadReader, the decoded-byte budget (file contents AND tar headers, so entry count as well) fails the extraction with ErrMaxSizeExceeded once exceeded.
Directory creation stays on the main goroutine because tar entries are emitted in walk order — a TypeDir must finish before a TypeReg child can be written.
func MarshalManifest ¶
func MarshalManifest(m *c1zv3.C1ZManifestV3) ([]byte, error)
MarshalManifest serializes m to deterministic proto bytes.
func ReadIndexedFrameIndex ¶ added in v0.13.5
func ReadIndexedFrameIndex(f *os.File) (*c1zv3.IndexedFrameIndex, error)
ReadIndexedFrameIndex returns the frame table of an INDEXED_ZSTD envelope using three small preads (head, footer, index) — no frame bytes are read and the manifest body is never parsed. This is the planning primitive for chunked object storage: every frame's name, absolute byte range, sizes, and content identity (SHA-256 of raw bytes), plus payload totals, without rematerializing the store.
Frame offsets are absolute file offsets, so entries can drive ranged reads (or object reassembly) directly. Returns ErrEnvelopeTruncated-wrapped errors for missing trailers and explicit errors for corrupt ones; returns an error for non-v3 files. The caller retains ownership of f.
func ReadManifestHeader ¶ added in v0.13.2
func ReadManifestHeader(r io.Reader) (*c1zv3.C1ZManifestV3, error)
ReadManifestHeader parses only the cheap manifest fields and stops before the payload: no zstd decoder is constructed and not a single payload byte is read. This is the "shallow unpack" path for callers that want envelope metadata — sync run summaries with their cached stats — without rematerializing the Pebble directory (compaction source selection, overlay bucket planning, tooling).
func UnmarshalManifest ¶
func UnmarshalManifest(b []byte) (*c1zv3.C1ZManifestV3, error)
UnmarshalManifest parses bytes into a fresh C1ZManifestV3.
func VerifyDescriptorClosure ¶
func VerifyDescriptorClosure(set *descriptorpb.FileDescriptorSet) error
VerifyDescriptorClosure checks that every file referenced by every other file in the set is itself in the set. Returns ErrManifestIncompleteDescriptors on the first missing import.
func WriteEnvelope ¶
WriteEnvelope writes a complete v3 envelope to w:
- The 5-byte C1Z3 magic.
- A uint32 BE length prefix for the marshaled manifest.
- The marshaled manifest bytes.
- The payload, per the manifest's PayloadEncoding.
The manifest's PayloadEncoding field selects the payload format:
- PAYLOAD_ENCODING_TAR_ZSTD (1): default; tar then zstd. The manifest can leave PayloadEncoding as the zero value (UNSPECIFIED) and WriteEnvelope will write TAR_ZSTD and patch the manifest in place so the reader sees the same value.
- PAYLOAD_ENCODING_TAR (2): uncompressed tar.
- PAYLOAD_ENCODING_INDEXED_ZSTD (5): per-file zstd frames with a trailing frame index (see indexed.go for the layout).
- PAYLOAD_ENCODING_UNSPECIFIED (0): treated as TAR_ZSTD.
Any other value (including the reserved 3 and 4) returns an error before any bytes are written to w.
payloadDir is walked in sorted lexical order; file mtimes are NOT normalized (the RFC documents tar as not byte-stable). w is typically a *os.File created via os.CreateTemp in the same directory as the final destination so an atomic rename can finalize the write.
Types ¶
type DecoderPool ¶ added in v0.13.2
type DecoderPool struct {
// contains filtered or unexported fields
}
DecoderPool reuses zstd payload decoders across envelope reads. A decoder retains its window/history buffers across Reset, so reuse avoids both the construction cost (worker goroutine spin-up, buffer allocation) and the per-open garbage when many envelopes are read in a loop — the compactor opens one envelope per source per merge.
The pool is deliberately NOT process-global: a pooled decoder keeps whatever buffers its largest stream grew, so the owner scopes the pool to the operation that benefits (one pool per compaction) and calls Close when done, releasing everything deterministically instead of waiting out sync.Pool's GC-paced eviction in a long-lived worker process. Callers that open a single envelope pass a nil pool and get a one-shot decoder destroyed at Envelope.Close.
func NewDecoderPool ¶ added in v0.13.2
func NewDecoderPool() *DecoderPool
NewDecoderPool returns an empty pool. The caller owns its lifetime and must call Close to release idle decoders.
func (*DecoderPool) Close ¶ added in v0.13.2
func (p *DecoderPool) Close()
Close releases every idle decoder and marks the pool closed; later puts destroy their decoders. Safe to call on a nil pool.
type Envelope ¶
type Envelope struct {
Manifest *c1zv3.C1ZManifestV3
PayloadReader io.Reader
// contains filtered or unexported fields
}
Envelope is the parsed result of ReadEnvelope. Manifest holds the decoded manifest; PayloadReader yields the zstd-tar payload bytes (or whatever PayloadEncoding the manifest declared). Callers must Close the envelope when done.
func ReadEnvelope ¶
ReadEnvelope reads a v3 envelope from r. r must be positioned at the start of the file (the C1Z3 magic). Returns an Envelope whose PayloadReader streams the uncompressed tar bytes for both PAYLOAD_ENCODING_TAR_ZSTD (transparently decoding zstd first) and PAYLOAD_ENCODING_TAR (passing through unchanged). For PAYLOAD_ENCODING_INDEXED_ZSTD the payload is not a tar stream and PayloadReader is nil — use ExtractEnvelopePayload (random access over the frame table) or ReadIndexedFrameIndex instead. The reserved values 3 and 4 return an error.
The payload is treated as untrusted: the zstd decoder's window memory is capped (BATON_DECODER_MAX_MEMORY_MB, default 128 MiB) and PayloadReader fails with ErrMaxSizeExceeded once the decoded payload exceeds the configured budget (BATON_DECODER_MAX_DECODED_SIZE_MB, default 10 GiB) — the same knobs the v1 decoder honors.
func ReadEnvelopeHeader ¶ added in v0.13.2
ReadEnvelopeHeader is the low-allocation open path for engine dispatch. It parses only the cheap manifest fields (engine, engine_schema_version, payload_encoding, sync_runs) and skips the descriptor closure. Call ReadEnvelope when callers need the full self-describing descriptor set.
func ReadEnvelopeHeaderWithPool ¶ added in v0.13.2
func ReadEnvelopeHeaderWithPool(r io.Reader, pool *DecoderPool) (*Envelope, error)
ReadEnvelopeHeaderWithPool is ReadEnvelopeHeader with a caller-scoped payload decoder pool: the envelope draws its zstd decoder from pool and returns it on Close instead of destroying it. Used by callers that open many envelopes in a loop (compaction source opens). A nil pool behaves exactly like ReadEnvelopeHeader.
type PayloadOption ¶ added in v0.13.5
type PayloadOption func(*payloadOptions)
func WithMaxDecodedPayloadBytes ¶ added in v0.13.5
func WithMaxDecodedPayloadBytes(n uint64) PayloadOption
WithMaxDecodedPayloadBytes sets the decoded payload byte cap for v3 payload extraction. A zero value means use BATON_DECODER_MAX_DECODED_SIZE_MB or the built-in default.
func WithMaxDecoderMemoryBytes ¶ added in v0.13.5
func WithMaxDecoderMemoryBytes(n uint64) PayloadOption
WithMaxDecoderMemoryBytes sets the zstd decoder memory cap for v3 payload extraction. A zero value means use BATON_DECODER_MAX_MEMORY_MB or the built-in default.
func WithPayloadDecoderPool ¶ added in v0.13.5
func WithPayloadDecoderPool(pool *DecoderPool) PayloadOption
WithPayloadDecoderPool scopes zstd payload-decoder reuse to the caller's pool for the tar_zstd encoding (the single-stream decode that dominates when many envelopes are opened in a loop, e.g. compaction source opens).
The pool only engages when the resolved decoder memory cap equals the pool's standard construction cap: a pooled decoder's WithDecoderMaxMemory is baked in at construction, so a caller-specific cap must construct a fresh decoder to be honored. The indexed encoding never draws from the pool — its parallel frame workers use cheap concurrency-1 decoders whose settings don't match the pool's. Nil is valid and means no reuse.
type PayloadReuse ¶ added in v0.13.5
type PayloadReuse struct {
// contains filtered or unexported fields
}
PayloadReuse indexes a source envelope's frames for splice-at-save. Returned by ExtractEnvelopePayload for indexed payloads; passed to WriteEnvelopeWithReuse when rewriting a store opened from that source.
func ExtractEnvelopePayload ¶ added in v0.13.5
func ExtractEnvelopePayload(f *os.File, destDir string, opts ...PayloadOption) (*c1zv3.C1ZManifestV3, *PayloadReuse, error)
ExtractEnvelopePayload opens, validates, and extracts a v3 envelope's payload from f into destDir, dispatching on the manifest's payload encoding. For indexed payloads it decodes frames in parallel and returns a PayloadReuse for splice-at-save; the tar encodings extract sequentially and return a nil reuse.
f must be positioned anywhere (it is rewound); it stays open and owned by the caller.
func (*PayloadReuse) SourcePath ¶ added in v0.13.5
func (r *PayloadReuse) SourcePath() string
SourcePath returns the envelope file the reuse entries point into.
type ReuseEntry ¶ added in v0.13.5
type ReuseEntry struct {
Name string
FrameOffset int64 // offset of the compressed bytes in the source file
CompressedSize int64
RawSize int64
XXH64 uint64
// SHA256 is the content-address identity of the raw bytes, carried
// from the source envelope's trailer index. The splice writer
// recomputes it from the local payload file if a caller hands it an
// entry without one.
SHA256 []byte
// ExtractedPath is where extraction wrote the raw bytes. Used for
// the zero-read os.SameFile identity check at save time: a
// checkpoint hard-link of the extracted file is provably the same
// content. Empty when unknown.
ExtractedPath string
}
ReuseEntry describes one frame of a source envelope: where its compressed bytes live in the source file, what they decode to, and where the extractor materialized them.
type SpliceStats ¶ added in v0.13.5
type SpliceStats struct {
SplicedFrames int
SplicedBytes int64 // compressed bytes copied verbatim
EncodedFrames int
EncodedBytes int64 // raw bytes freshly compressed
}
SpliceStats reports how a WriteEnvelopeWithReuse call split between verbatim frame copies and fresh compression. Exposed so callers can log reuse effectiveness in production.
func WriteEnvelopeWithReuse ¶ added in v0.13.5
func WriteEnvelopeWithReuse(w io.Writer, m *c1zv3.C1ZManifestV3, payloadDir string, reuse *PayloadReuse) (SpliceStats, error)
WriteEnvelopeWithReuse is WriteEnvelope plus frame splicing for the INDEXED_ZSTD encoding: payload files proven byte-identical to a frame in reuse's source envelope are copied as compressed bytes instead of re-encoded. reuse is ignored (and stats zero) for the tar encodings. Every encoding writes in a single pass, so w may be any io.Writer — including a non-seekable network sink.
Source Files