Documentation
¶
Index ¶
- Constants
- Variables
- func BlockCeilIdx(off, blockSize int64) int64
- func BlockIdx(off, blockSize int64) int64
- func BlockOffset(idx, blockSize int64) int64
- func BlocksOffsets(size, blockSize int64) []int64
- func Equal(a, b []BuildMap) bool
- func IsZero(b []byte) bool
- func Layers(mappings []BuildMap) *map[uuid.UUID]struct{}
- func SerializeHeader(h *Header) ([]byte, error)
- func StoreHeader(ctx context.Context, s storage.StorageProvider, path string, h *Header, ...) (cfg storage.CompressConfig, stored, uncompressed int64, err error)
- func TotalBlocks(size, blockSize int64) int64
- func ValidateHeader(h *Header) error
- func ValidateMappings(mappings []BuildMap, size, blockSize uint64) error
- func Visualize(mappings []BuildMap, size, blockSize, cols uint64, ...) string
- type BuildData
- type BuildMap
- type DiffMetadata
- type DiffMetadataBuilder
- type Header
- func (t *Header) CloneForUpload(version uint64) *Header
- func (t *Header) GetBuildFrameData(buildID uuid.UUID) *storage.FrameTable
- func (t *Header) GetShiftedMapping(ctx context.Context, offset int64) (BuildMap, error)
- func (t *Header) IsNormalizeFixApplied() bool
- func (t *Header) SelfBuildData() (int64, *storage.FullFrameTable, error)
- func (t *Header) SetBuild(buildID uuid.UUID, bd BuildData)
- func (t *Header) String() string
- type Mapping
- func (m Mapping) All() iter.Seq2[int, BuildMap]
- func (m Mapping) At(i int) BuildMap
- func (m Mapping) BlockSize() uint64
- func (m Mapping) Builds() []uuid.UUID
- func (m Mapping) BytesByBuild() map[uuid.UUID]uint64
- func (m Mapping) Len() int
- func (m Mapping) SearchOffset(off int64) int
- func (m Mapping) Slice() []BuildMap
- func (m Mapping) Validate(size, blockSize uint64) error
- type Metadata
Constants ¶
const ( PageSize = 4 << 10 // 4 KiB HugepageSize = 2 << 20 // 2 MiB RootfsBlockSize = 4 << 10 // 4 KiB )
const ( SkippedBlockChar = '░' DirtyBlockChar1 = '▓' DirtyBlockChar2 = '█' )
const ( // MetadataVersionV4 is used for compressed builds (V4 headers with FrameTables). MetadataVersionV4 = 4 // MetadataVersionV5 is V4 with a columnar, varint-encoded mapping section. // Same semantics as V4 (Builds map + FrameTables); only the on-disk mapping // layout differs. Far smaller and far more compressible for the large, // fragmented headers produced by page-granular memfile dedup. MetadataVersionV5 = 5 )
const NormalizeFixVersion = 3
Variables ¶
var EmptyHugePage = make([]byte, HugepageSize)
Functions ¶
func BlockCeilIdx ¶
BlockCeilIdx returns the index of the first block after byte offset off (ceiling division).
func BlockIdx ¶
BlockIdx returns the index of the block containing byte offset off (floor division).
func BlockOffset ¶
func BlocksOffsets ¶
func IsZero ¶
IsZero reports whether b is all-zero. Samples first/middle/last byte to reject most non-zero buffers from one cache line, then falls back to bytes.Equal(b[:n-1], b[1:]): true exactly when every adjacent pair of bytes is equal, i.e. all bytes equal b[0] (which the sample already proved is zero).
func SerializeHeader ¶
SerializeHeader serializes a header, dispatching to the version-specific format.
V3 (Version <= 3): Metadata [v3 mappings…] V4 (Version >= 4): Metadata [uint8 flags] [uint32 uncompressedSize] [LZ4( Builds + v4 mappings )]
func StoreHeader ¶
func StoreHeader(ctx context.Context, s storage.StorageProvider, path string, h *Header, opts ...storage.PutOption) (cfg storage.CompressConfig, stored, uncompressed int64, err error)
StoreHeader serializes a header, uploads it, and returns the effective compression config plus the stored and pre-compression byte counts. V3 has no inner compression so the counts match and cfg is the zero value. Refuses to persist a header still flagged as in-flight.
func TotalBlocks ¶
func ValidateHeader ¶
ValidateHeader checks header integrity and returns an error if corruption is detected. This verifies: 1. Header and metadata are valid 2. Mappings cover the entire file [0, Size) with no gaps 3. Mappings don't extend beyond file size (with block alignment tolerance)
func ValidateMappings ¶
ValidateMappings validates the mappings. It is used to check if the mappings are valid.
It checks if the mappings are contiguous and if the length of each mapping is a multiple of the block size. It also checks if the mappings cover the whole size.
func Visualize ¶
func Visualize(mappings []BuildMap, size, blockSize, cols uint64, bottomGroup, topGroup *map[uuid.UUID]struct{}) string
Visualize returns a string representation of the mappings as a grid of blocks. It is used for debugging and visualization.
You can pass maps to visualize different groups of buildIds.
Types ¶
type BuildData ¶
type BuildData struct {
Size int64 // uncompressed file size
Checksum [32]byte // SHA-256 of uncompressed data; zero value means unknown
FrameData *storage.FrameTable // nil for uncompressed builds
}
BuildData holds per-build metadata stored in V4 headers. Each layer's header carries a Builds map; child headers inherit parent entries for still-referenced build IDs via newDiffHeader.
type BuildMap ¶
type BuildMap struct {
// Offset is the starting position of this range in the block device.
Offset uint64
Length uint64
BuildId uuid.UUID
BuildStorageOffset uint64
}
BuildMap maps a byte range in the block device to a region in a build's storage. Offset, Length, and BuildStorageOffset are in bytes.
func CreateMapping ¶
func MergeMappings ¶
MergeMappings merges two sets of mappings.
The mapping are stored in a sorted order. The baseMapping must cover the whole size.
It returns a new set of mappings that covers the whole size.
func NormalizeMappings ¶
NormalizeMappings joins adjacent mappings that have the same buildId.
Same-build runs are only merged when their storage is contiguous, i.e. mp.BuildStorageOffset == current end. The read path resolves bytes as BuildStorageOffset + shift, so merging a run across a storage discontinuity would silently remap reads to the wrong bytes. Empty (uuid.Nil) regions are served as zeros and never read from storage (see build.ReadAt), so their storage offset is irrelevant and they always merge.
Clone the result so the oversized intermediate (cap == len(input)) is released; the merged Header is cached for up to 25h. slices.Clip will not do — it only retightens cap on the same backing array.
type DiffMetadata ¶
func NewDiffMetadata ¶
func NewDiffMetadata(blockSize int64, dirty, empty *roaring.Bitmap) *DiffMetadata
func (*DiffMetadata) ToDiffHeader ¶
type DiffMetadataBuilder ¶
type DiffMetadataBuilder struct {
// contains filtered or unexported fields
}
func NewDiffMetadataBuilder ¶
func NewDiffMetadataBuilder(blockSize int64) *DiffMetadataBuilder
func (*DiffMetadataBuilder) Build ¶
func (b *DiffMetadataBuilder) Build() *DiffMetadata
type Header ¶
type Header struct {
Metadata *Metadata
// Builds maps build IDs to per-build metadata (size, checksum, FrameTable).
// nil for V3 (uncompressed) headers; the read path falls back to a Size()
// RPC and reads uncompressed data when nil.
Builds map[uuid.UUID]BuildData
// Mapping is the per-block source map. Stored compactly (~14 B/entry vs 40
// for a BuildMap) so long-lived cached headers don't dominate orchestrator
// heap. Read via At / All / Slice / Len, not indexing.
Mapping Mapping
// IncompletePendingUpload is set on diff headers produced by ToDiffHeader and
// cleared on the finalized headers swapped in by the upload pipeline. It
// is in-memory only (never serialized), and signals that the build's data
// has not yet reached object storage — readers must serve from the local
// cache and skip FrameTable lookups for the still-missing self entry.
IncompletePendingUpload bool
}
func Deserialize ¶
Deserialize reads a header from a storage Blob (legacy API).
func DeserializeBytes ¶
DeserializeBytes auto-detects the header version and deserializes accordingly. See SerializeHeader for the binary layout.
func LoadHeader ¶
LoadHeader fetches a serialized header from storage and deserializes it. Returns the on-wire byte count alongside the header so callers can attribute it to throughput telemetry. Errors (including storage.ErrObjectNotExist) are returned as-is.
func (*Header) CloneForUpload ¶
CloneForUpload returns a clone the upload path can mutate for serialization without racing with concurrent readers of the original. The version is set on the clone and the Builds map is copied (the upload path adds the self entry). Mapping is shared by reference: it is immutable once a Header is constructed.
func (*Header) GetBuildFrameData ¶
func (t *Header) GetBuildFrameData(buildID uuid.UUID) *storage.FrameTable
GetBuildFrameData returns the FrameTable for a build: nil = no entry, storage.UncompressedFrameTable = authoritatively uncompressed, else compressed.
func (*Header) GetShiftedMapping ¶
GetShiftedMapping resolves a virtual offset to a build-local range. The read path uses this to find which build owns the data, then calls GetBuildFrameData to get the FrameTable for C-space lookup.
func (*Header) IsNormalizeFixApplied ¶
IsNormalizeFixApplied is a helper method to soft fail for older versions of the header where fix for normalization was not applied. This should be removed in the future.
func (*Header) SelfBuildData ¶
func (t *Header) SelfBuildData() (int64, *storage.FullFrameTable, error)
SelfBuildData returns the size and full FrameTable for the header's own build (t.Metadata.BuildId). Errors when the self entry is missing — only possible with peer-served incomplete headers, never with storage-uploaded ones (build_upload_v4 always populates self before publish).
This is the *only* place the FullFrameTable upcast happens in production: Builds[id].FrameData is typed *FrameTable to match the trimmed-FT case where our header carries only the frames we mapped from an ancestor. For the self entry, build_upload_v4 always stores the complete table, so the upcast via storage.FullFromTable is sound.
type Mapping ¶
type Mapping struct {
// contains filtered or unexported fields
}
Mapping is the compact in-memory representation of a Header's mapping list. A merged Header is cached for up to 25h, so on snapshot-heavy nodes these slices dominate host RAM. It shrinks each entry from 40 bytes (a BuildMap) to 14 by encoding offset/length/storage as uint32 block indices, deduplicating BuildId into a per-header table addressed by uint16, and storing the columns as parallel slices. Immutable; read via At / All / Slice.
func NewMapping ¶
NewMapping packs src into the compact representation. blockSize is the unit for the block indices and must divide every Offset, Length, and BuildStorageOffset in src (callers pass PageSize, the universal granularity).
func (Mapping) All ¶
All iterates the mapping, materializing each entry as a BuildMap. This is the preferred read path for callers that don't need a backing []BuildMap.
func (Mapping) At ¶
At materializes the i-th entry as a BuildMap. Panics if i is out of range, matching `mapping[i]` semantics.
func (Mapping) Builds ¶
Builds returns the deduplicated build IDs referenced by the mapping. The returned slice is shared with the Mapping; callers must not mutate it.
func (Mapping) BytesByBuild ¶
BytesByBuild sums the bytes attributed to each referenced build. It scans the length and buildIdx columns directly (no BuildMap materialization or per-entry uuid hashing), accumulating into a small per-build slice, so it stays cheap even for mappings with millions of entries. Empty (nil-build) regions are skipped. The returned map is non-nil and addressable by the caller.
func (Mapping) SearchOffset ¶
SearchOffset returns the first index i whose byte offset is strictly greater than off, matching sort.Search semantics on Offset. It compares in block units, so entries are never materialized: entry.OffsetBlocks*blockSize > off iff entry.OffsetBlocks > off/blockSize (integer division).
Source Files