Documentation
¶
Overview ¶
Package amberpack defines the Amber-Store pack format. The content-addressed record codec (record.go) is shared by packstore's on-disk segments and the remote-sync wire packs defined here.
A wire pack is a possibly-partial, unordered set of CAS objects (like a git pack) carrying no root key. Layout:
Magic "AMBERPK\x04" 8 bytes (plaintext)
Records repeat: one EncodeRecordWith output each — a 46-byte header
(tag 0x01 + key[32] + flags + ulen + slen + CRC) followed by the payload
End 0x00
Each record is the same self-describing, CRC-protected, individually compressed unit packstore writes on disk (see record.go); a wire pack is just those records framed by a magic and an explicit end marker, so a truncated stream is detected rather than read as a clean EOF. The Reader validates framing, CRC, and key canonicality and decodes each payload (All), or hands the validated records over undecoded (Records); it does NOT verify the payload hash — that happens in the storage path (packstore WriteParallel with Verify).
Versions 1 and 2 ("AMBERPK\x01" / "AMBERPK\x02") were the older uncompressed and whole-stream-zstd stream formats, and version 3 was this layout with keys in their earlier byte order (header byte first); they are no longer produced and are rejected by the Reader.
Index ¶
- Constants
- Variables
- func DecodePayload(flags byte, ulen uint32, stored []byte) ([]byte, error)
- func EncodeRecord(k key.Key, data []byte) ([]byte, error)
- func EncodeRecordWith(k key.Key, data []byte, c Compression) ([]byte, error)
- type Algorithm
- type Compression
- type RawRecord
- type Reader
- type Record
- type Writer
- type WriterOption
Constants ¶
const ( // RecHeaderSize is the fixed record-header length: // tag(1) + key(32) + flags(1) + ulen(4) + slen(4) + crc(4). Payload follows. // The flags byte holds the payload's codec id, an Algorithm value. RecHeaderSize = 46 // MaxPayload bounds one object's payload, stored or decoded. The length // fields are untrusted and size allocations. Real objects are ~1 MiB. MaxPayload = 256 << 20 )
Variables ¶
var ErrCorrupt = errors.New("amberpack: corrupt pack data")
ErrCorrupt wraps every record-level corruption error surfaced by ParseRecord and DecodePayload (bad framing, bad flags, CRC mismatch, length inconsistency, non-canonical key). It is the record-level counterpart to the stream-level ErrMalformed; distinguish either with errors.Is. The packstore package aliases this sentinel for its footer- and scrub-level corruption too, so the message stays deliberately general rather than naming "record".
var ErrInvalidCompression = errors.New("amberpack: invalid compression")
ErrInvalidCompression wraps every rejection of a Compression value, and of text that does not name one.
var ErrMalformed = errors.New("amberpack: malformed pack stream")
ErrMalformed wraps every error from a structurally invalid wire pack (bad or legacy magic, truncation, an oversized or bad record, or a corrupt record). Callers distinguish it with errors.Is to map to a client error.
Functions ¶
func DecodePayload ¶
DecodePayload returns caller-owned payload bytes from a record's stored payload, decoded as the record's flags byte says. stored may be a read-only mmap slice and is never retained.
func EncodeRecord ¶
EncodeRecord serializes (k, data) into a complete record with the payload stored raw. It is EncodeRecordWith with no compression.
func EncodeRecordWith ¶ added in v0.10.0
EncodeRecordWith serializes (k, data) into a complete record, compressing the payload as c says when that makes it strictly smaller and storing it raw otherwise. An invalid c is an error wrapping ErrInvalidCompression. k is written as given; canonical-form validation happens on the read side.
Types ¶
type Algorithm ¶ added in v0.10.0
type Algorithm uint8
Algorithm names a compression algorithm. Its values are the codec ids a record carries in its flags byte, so they never change.
type Compression ¶ added in v0.10.0
Compression is an algorithm and its level. The zero value is no compression. Level 0 selects the algorithm's default: 3 for zstd, the fast compressor for lz4. zstd takes levels up to 22 and lz4 up to 12, where 1 and above are its high-compression levels.
This package's zstd encoder has four tiers, not 22 levels, and its lz4 encoder stops at level 9; a level is mapped to the nearest one there is. The level changes what is written and never how it is read.
func ParseCompression ¶ added in v0.10.0
func ParseCompression(s string) (Compression, error)
ParseCompression reads the text form: none, zstd, zstd:LEVEL, lz4 or lz4:LEVEL. The result is valid.
func (Compression) String ¶ added in v0.10.0
func (c Compression) String() string
String returns the text form ParseCompression reads: "none", "zstd", "zstd:19", "lz4", "lz4:9". A level of 0 is left out.
func (Compression) Validate ¶ added in v0.10.0
func (c Compression) Validate() error
Validate reports whether c names a known algorithm at a level it takes.
type RawRecord ¶
RawRecord is one record of a pack as it was read: its parsed header and its complete bytes, header and stored payload, exactly as EncodeRecord produced them. Bytes is caller-owned.
type Reader ¶
type Reader struct {
// contains filtered or unexported fields
}
Reader decodes a wire pack stream.
func (*Reader) All ¶
All iterates over the objects in the stream: Records with every payload decoded. It yields exactly one error (and stops) on any structural problem; on a clean stream it yields every object and returns after the end marker. All must be called at most once per Reader because the underlying stream position is not reset between calls.
func (*Reader) Records ¶
Records iterates over the records in the stream without decoding them. Every record is validated exactly as All validates it (framing, the size bound, CRC, key canonicality) but its payload stays as stored, so a consumer that appends records verbatim (packstore's Object.Record, Writer.AddRecord) skips the decompress/recompress round trip; it is the read-side counterpart of AddRecord. It yields exactly one error (and stops) on any structural problem; on a clean stream it yields every record and returns after the end marker. Records must be called at most once per Reader because the underlying stream position is not reset between calls.
type Record ¶
type Record struct {
Key key.Key
// Flags is the raw record flag byte; pass it to DecodePayload unchanged.
Flags byte
Ulen uint32
Slen uint32
}
Record describes a parsed record header. The payload lives at [RecHeaderSize : RecHeaderSize+Slen] within the record's bytes.
func ParseRecord ¶
ParseRecord validates the record at the start of b (which may extend past it) and returns its header. It checks framing, flags, key canonicality, and the CRC, without mutating b (b may be a read-only mmap).
type Writer ¶
type Writer struct {
// contains filtered or unexported fields
}
Writer serializes fstree.Objects into the wire pack format. It is not safe for concurrent use; a client wanting parallel uploads creates one Writer per pack.
func NewWriter ¶
func NewWriter(w io.Writer, opts ...WriterOption) *Writer
NewWriter returns a Writer emitting to w. The caller owns w and must close it; Writer.Close only writes the end marker and flushes.
func (*Writer) AddRecord ¶
AddRecord appends a pre-encoded record (an EncodeRecord output, as stored verbatim on disk) without decoding or re-encoding it. It is the zero-copy counterpart to Add: the push path reads a record straight from the local store and writes it to the wire, skipping the decompress/recompress round trip. rec is written as given; its framing and CRC are validated by the receiving Reader.
type WriterOption ¶ added in v0.10.0
type WriterOption func(*Writer)
WriterOption configures a Writer.
func WithCompression ¶ added in v0.10.0
func WithCompression(c Compression) WriterOption
WithCompression sets the compression Add encodes with. The default is no compression. An invalid value fails every Add with an error wrapping ErrInvalidCompression. AddRecord is unaffected: it writes records as given.
Source Files