Documentation
¶
Index ¶
- Constants
- func AppendRemoteBlockRefChunk(dst []byte, refs []RemoteBlockRef) []byte
- func AppendSearchInfoChunk(dst []byte, cfg SearchConfig) []byte
- func AppendSearchTableChunk(dst []byte, cfg SearchConfig, reductions uint8, table []byte) []byte
- func AppendSearchTableCompressedChunk(dst []byte, cfg SearchConfig, reductions uint8, table []byte) []byte
- func BuildSearchTable(cfg SearchConfig, blockData, overlap []byte) (table []byte, reductions uint8)
- func DecodeBlock(src []byte) (dst []byte, err error)
- func EncodeBlock(src []byte) ([]byte, error)
- func HashValue(val uint64, tableSize, matchLen uint8) uint32
- func MaxEncodedLen(srcLen int) int
- func ReadStream(r io.Reader, debugOut io.Writer) error
- type Index
- type ParsedSearchTable
- type RemoteBlockRef
- type SearchConfig
Constants ¶
const ( IndexHeader = "s2idx\x00" IndexTrailer = "\x00xdi2s" MaxIndexEntries = 1 << 16 )
const ( ChunkSearchInfo = 0x44 ChunkSearchTable = 0x45 ChunkSearchTableCompressed = 0x46 ChunkRemoteBlockRef = 0x47 )
Chunk types from SPEC_SEARCH.md section 2.
const ( TableTypeNoPrefix = 1 TableTypeBytePrefix = 2 TableTypeMaskPrefix = 3 TableTypeLongPrefix = 4 )
Table types from SPEC_SEARCH.md section 3.0.
const ( ChunkLegacyCompressed = 0x00 ChunkUncompressed = 0x01 ChunkMinLZBlock = 0x02 ChunkMinLZCompCRC = 0x03 ChunkIndex = 0x40 ChunkEOF = 0x20 ChunkPadding = 0xfe ChunkStreamID = 0xff )
Variables ¶
This section is empty.
Functions ¶
func AppendRemoteBlockRefChunk ¶ added in v1.2.0
func AppendRemoteBlockRefChunk(dst []byte, refs []RemoteBlockRef) []byte
AppendRemoteBlockRefChunk emits a complete 0x47 chunk containing one or more remote block references. refs[i].Offset is the absolute stream offset; the wire encoding stores the first offset absolutely and subsequent offsets as positive deltas (SPEC_SEARCH.md section 2.3). Offsets must be strictly ascending.
func AppendSearchInfoChunk ¶ added in v1.2.0
func AppendSearchInfoChunk(dst []byte, cfg SearchConfig) []byte
AppendSearchInfoChunk appends a complete 0x44 chunk (header + payload).
func AppendSearchTableChunk ¶ added in v1.2.0
func AppendSearchTableChunk(dst []byte, cfg SearchConfig, reductions uint8, table []byte) []byte
AppendSearchTableChunk appends a complete 0x45 chunk: header, config, reductions byte, CRC32 of the bitmap, then the bitmap bytes.
func AppendSearchTableCompressedChunk ¶ added in v1.2.0
func AppendSearchTableCompressedChunk(dst []byte, cfg SearchConfig, reductions uint8, table []byte) []byte
AppendSearchTableCompressedChunk appends a complete 0x46 chunk. The bitmap is split into one or more huff0 sub-blocks; each sub-block carries its own huff0 table when compression helps, otherwise disposition 16 (raw) is used. RLE and sparse dispositions are not produced by this reference, but the decoder accepts them.
func BuildSearchTable ¶ added in v1.2.0
func BuildSearchTable(cfg SearchConfig, blockData, overlap []byte) (table []byte, reductions uint8)
BuildSearchTable produces the packed bitmap and the chosen reductions count for blockData. overlap is up to MatchLen+Extras bytes from the start of the next block (empty for the last block in a stream).
The returned table length is 1 << (BaseTableSize - reductions - 3) bytes, rounded up to the 32-byte minimum.
func DecodeBlock ¶
DecodeBlock is a reference implementation of the MinLZ block decoder. This implementation is not optimized for speed, but for readability with no dependencies.
func EncodeBlock ¶
EncodeBlock is a reference implementation of the MinLZ block decoder. This implementation is not optimized for speed nor efficiency, but for readability with no dependencies. See 'encodeBlockGo' for a more optimal encoder.
func HashValue ¶ added in v1.2.0
HashValue returns a table index for the lowest matchLen bytes of val. tableSize is the number of output bits (8..23). matchLen must be 1..8. Direct translation of SPEC_SEARCH.md section 3.1.
func MaxEncodedLen ¶
MaxEncodedLen returns the maximum length of a snappy block, given its uncompressed length.
It will return a negative value if srcLen is too large to encode.
Types ¶
type Index ¶
type Index struct {
TotalUncompressed int64 // Total Uncompressed size.
TotalCompressed int64 // Total Compressed size if known. Will be -1 if unknown.
// Compressed -> Uncompressed map of block start positions.
// Not all blocks will be in here
Blocks []struct {
CompressedOffset int64
UncompressedOffset int64
}
// contains filtered or unexported fields
}
Index represents a MinLZ index.
func LoadIndexAfterHeader ¶
LoadIndexAfterHeader will load and parse an index, after the stream header has been parsed.
type ParsedSearchTable ¶ added in v1.2.0
type ParsedSearchTable struct {
Cfg SearchConfig
Reductions uint8
Table []byte
}
ParsedSearchTable is the result of decoding a 0x45 or 0x46 chunk.
Table holds the reduced packed bitmap (length = 1 << (BaseTableSize - Reductions - 3)). Look up a candidate value with the Contains method.
func ParseSearchTableChunk ¶ added in v1.2.0
func ParseSearchTableChunk(payload []byte) (ParsedSearchTable, error)
ParseSearchTableChunk parses a 0x45 chunk payload (the bytes after the 4-byte chunk header). The bitmap CRC is verified.
func ParseSearchTableCompressedChunk ¶ added in v1.2.0
func ParseSearchTableCompressedChunk(payload []byte) (ParsedSearchTable, error)
ParseSearchTableCompressedChunk parses a 0x46 chunk payload. All four dispositions (huff0 table 0..15, raw 16, RLE 17, sparse 18) are accepted. The bitmap CRC is verified after reconstruction.
func (ParsedSearchTable) Contains ¶ added in v1.2.0
func (p ParsedSearchTable) Contains(needle []byte) bool
Contains reports whether the MatchLen-byte prefix of needle could be present in the indexed block. Returns false if needle is shorter than MatchLen, or when the bit is definitely absent. A true result allows for false positives (the table is a Bloom-style summary, not a membership proof).
type RemoteBlockRef ¶ added in v1.2.0
type RemoteBlockRef struct {
Offset uint64 // stream byte offset of the referenced block
MaxMinusActualLen uint64 // maxBlockSize - actualUncompressedSize
}
RemoteBlockRef references one remote data block (chunk 0x47).
func ParseRemoteBlockRefChunk ¶ added in v1.2.0
func ParseRemoteBlockRefChunk(payload []byte) ([]RemoteBlockRef, error)
ParseRemoteBlockRefChunk parses a 0x47 chunk payload (the bytes after the 4-byte chunk header). The first wire offset is absolute, subsequent ones are positive deltas (SPEC_SEARCH.md section 2.3). Returned Offset values are absolute.
type SearchConfig ¶ added in v1.2.0
type SearchConfig struct {
MatchLen uint8 // 1..8
BaseTableSize uint8 // 8..23 (log2 of bit count)
TableType uint8 // 1..4
Extras uint8 // type 4 only: E+1 hashes per prefix occurrence; MatchLen+Extras ≤ 16
PrefixBytes []byte // type 2: 1..8 bytes (will be padded to 8 on the wire)
PrefixMask [32]byte // type 3: 256-bit set
LongPrefix []byte // type 4: 1..256 bytes
}
SearchConfig is the per-block search-table wire descriptor.
Wire fields only — tunable thresholds are package constants, not config.
func ParseSearchInfoChunk ¶ added in v1.2.0
func ParseSearchInfoChunk(payload []byte) (SearchConfig, error)
ParseSearchInfoChunk parses a 0x44 chunk payload (the bytes after the 4-byte chunk header).
Source Files