valuelog

package
v0.6.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 1, 2026 License: MIT Imports: 31 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	Version = 1

	HeaderSize = 4 + 1 + 1 + 2 + 8 + 4
)
View Source 
const (
	FrameVersion    = 1
	FrameHeaderSize = 12
	MaxFrameK       = 255
)
View Source 
const DefaultKeepSafetyMargin = 0.10
View Source 
const (
	FrameFlagCompressed byte = 1 << 0
)
View Source 
const (

	// ReservedLeafLogLaneID is the dedicated lane used for outer-leaf storage.
	ReservedLeafLogLaneID = maxLaneID
)

Variables

View Source 
var (
	CurrentWritableMmapTargetBytes       = int64(defaultCurrentWritableMapTarget)
	MaxMappedSealedSegments              = defaultMaxMappedSealed
	MaxMappedSealedBytes           int64 = defaultMaxMappedSealedBytes
	MaxMappedLeafSealedSegments          = defaultMaxMappedLeafSealed
	MaxMappedLeafSealedBytes       int64 = defaultMaxMappedLeafSealedBytes
)
View Source 
var (
	ErrCorrupt         = errors.New("valuelog: corrupt record")
	ErrRecordTooLarge  = errors.New("valuelog: record too large")
	ErrMissingDict     = errors.New("valuelog: missing dict bytes")
	ErrMissingTemplate = errors.New("valuelog: missing template bytes")
)
View Source 
var ErrFileNotFound = errors.New("valuelog: file not found")
View Source 
var ErrFilePinned = errors.New("valuelog: file pinned")

ErrFilePinned reports that a value-log segment is still pinned by a live snapshot and cannot be removed yet.

View Source 
var ErrSegmentIDRange = errors.New("valuelog: segment id out of range")
View Source 
var MaxDeadMappings = defaultMaxDeadMappings

MaxDeadMappings is the base cap for old mmaps retained to avoid exhausting vm.max_map_count. Unless explicitly configured, the effective cap can grow with mapped size up to maxAdaptiveDeadMappings. Set <= 0 to disable the cap.

Each remap retains the previous mapping until the file is closed to avoid use-after-unmap with concurrent readers.

Functions

func AppendDecodeScratchStats added in v0.6.0 

func AppendDecodeScratchStats(out map[string]string, prefix string, stats DecodeScratchStats)

func AppendWriterAppendBufferStats added in v0.6.0 

func AppendWriterAppendBufferStats(out map[string]string, prefix string, stats WriterAppendBufferStats)

func ChooseBlockGroupK added in v0.3.0 

func ChooseBlockGroupK(records, rawPayloadBytes, targetCompressedBytes int, observedRatio float64) int

ChooseBlockGroupK estimates grouped frame size K for block compression.

observedRatio is stored/raw payload ratio. Values >= 0.98 are treated as expansion-risk/incompressible and force k=1.

func DecodeFileID 

func DecodeFileID(fileID uint32) (lane uint32, seq uint32)

func DecodeSegmentID 

func DecodeSegmentID(id uint32) (lane uint32, seq uint32)

func EncodeFileID 

func EncodeFileID(lane uint32, seq uint32) (uint32, error)

func EncodeSegmentID 

func EncodeSegmentID(lane uint32, seq uint32) (uint32, error)

func GenerateAutotuneValues 

func GenerateAutotuneValues(workload AutotuneWorkload, size, count int, seed int64) [][]byte

GenerateAutotuneValues builds deterministic samples for a workload.

func HasCompactLeafLogPayload added in v0.5.0 

func HasCompactLeafLogPayload(payload []byte) bool

func MaybeAppendCompactLeafLogPayloadTo added in v0.6.0 

func MaybeAppendCompactLeafLogPayloadTo(dst, leafPage []byte) ([]byte, []byte, bool, error)

func MaybeCompactLeafLogPayload added in v0.5.0 

func MaybeCompactLeafLogPayload(leafPage []byte) ([]byte, bool, error)

func MaybeCompactLeafLogPayloadLength added in v0.6.0 

func MaybeCompactLeafLogPayloadLength(leafPage []byte) (compactLen int, compacted bool)

MaybeCompactLeafLogPayloadLength returns the compact payload length that the MaybeCompact* helpers would produce for leafPage, without materializing the payload. A false compacted result means callers should use leafPage as-is.

func MaybeCompactLeafLogPayloadTo added in v0.6.0 

func MaybeCompactLeafLogPayloadTo(dst, leafPage []byte) ([]byte, bool, error)

func NormalizeBlockTargetCompressedBytes added in v0.3.0 

func NormalizeBlockTargetCompressedBytes(v int) int

NormalizeBlockTargetCompressedBytes returns a bounded grouped block target.

func ReadAt 

func ReadAt(f *os.File, ptr page.ValuePtr, verifyCRC bool) ([]byte, error)

func ReadAtTo added in v0.4.0 

func ReadAtTo(f *os.File, ptr page.ValuePtr, verifyCRC bool, dst []byte) ([]byte, bool, error)

ReadAtTo decodes ptr from f and, when possible, writes decoded bytes into dst. It returns the decoded value, whether dst backed the returned slice, and an error.

Callers must treat the returned bytes as immutable. When usedDst is true, the returned slice aliases dst.

func ReadAtWithDict 

func ReadAtWithDict(f *os.File, ptr page.ValuePtr, verifyCRC bool, dictLookup DictLookup, templateLookup TemplateLookup, templateCache *templateDefCache, templateOpts templ.DecodeOptions) ([]byte, error)

func ReadAtWithDictTo added in v0.4.0 

func ReadAtWithDictTo(f *os.File, ptr page.ValuePtr, verifyCRC bool, dictLookup DictLookup, templateLookup TemplateLookup, templateCache *templateDefCache, templateOpts templ.DecodeOptions, dst []byte) ([]byte, bool, error)

ReadAtWithDictTo is ReadAtWithDict with caller-provided decode storage.

The returned slice may alias dst when usedDst is true. The returned bytes are immutable from the caller perspective.

func ReadRIDAtUnverified added in v0.6.0 

func ReadRIDAtUnverified(f *os.File, fileID uint32, ptr page.ValuePtr) (uint64, error)

ReadRIDAtUnverified reads only the RID metadata for ptr and does not verify the record CRC. It avoids full segment scans when the caller's ValuePtr was just produced by this process's value-log write path and only the append RID is needed for a command-WAL external-reference fence.

It MUST NOT be used on pointers loaded from durable index pages or any other untrusted source; use Reader for integrity-checked reads from durable pointers.

func SegmentPath added in v0.6.0 

func SegmentPath(dir string, id uint32) string

SegmentPath returns the canonical on-disk path for an encoded value-log file ID, such as an ID produced by EncodeFileID.

func ShouldKeepCompressed 

func ShouldKeepCompressed(rawBytes, encodedBytes int, encodeNs int64, ioNsPerStoredByte, safetyMargin float64) bool

ShouldKeepCompressed decides whether to keep compressed bytes for a frame. It compares estimated IO savings against encode cost with a safety margin.

Types

type AutotuneMode 

type AutotuneMode uint8
const (
	// AutotuneUnset indicates the caller did not explicitly configure the
	// autotuner mode. NormalizeAutotuneOptions maps this to a sensible default
	// based on whether the value log is enabled.
	AutotuneUnset AutotuneMode = iota
	AutotuneOff
	AutotuneMedium
	AutotuneAggressive
)

type AutotuneOptions 

type AutotuneOptions struct {
	Mode AutotuneMode

	CandidateK            []int
	CandidateHistoryBytes []int
	CandidateDictBytes    []int

	MinGainToSwitch float64
	MinDwellFrames  uint64

	SampleStride     uint64
	MaxSampleBytes   uint64
	TrainCPUFraction float64

	ProbeBytes uint64
	PauseBytes uint64

	DisableBelowValueBytes int
}

func NormalizeAutotuneOptions 

func NormalizeAutotuneOptions(opts AutotuneOptions, valueLogEnabled bool) AutotuneOptions

type AutotuneWorkload 

type AutotuneWorkload struct {
	Name string
	Make func(rng *rand.Rand, size int) []byte
}

AutotuneWorkload describes a deterministic value generator.

func AutotuneWorkloads 

func AutotuneWorkloads() []AutotuneWorkload

AutotuneWorkloads returns the standard deterministic workloads used by the autotune bench suite.

func LookupAutotuneWorkload 

func LookupAutotuneWorkload(name string) (AutotuneWorkload, bool)

LookupAutotuneWorkload returns the workload with the given name.

type BlockCodec added in v0.3.0 

type BlockCodec uint8

BlockCodec identifies non-dictionary value-log frame compression codecs. 

const (
	BlockCodecNone BlockCodec = iota
	BlockCodecSnappy
	BlockCodecLZ4
	BlockCodecZSTD
)

type Clock 

type Clock interface {
	Now() time.Time
}

type DecodeScratchStats added in v0.6.0 

type DecodeScratchStats struct {
	SmallPoolMaxEntries      int
	SmallPoolEntries         int
	SmallPoolRetainedBuffers uint64
	SmallPoolRetainedBytes   uint64
	SmallPoolGetsTotal       uint64
	SmallPoolHitsTotal       uint64
	SmallPoolMissesTotal     uint64
	SmallPoolTooSmallTotal   uint64
	SmallPoolPutsTotal       uint64
	SmallPoolDropsTotal      uint64
	SmallPoolDroppedBytes    uint64
	SmallAllocCallsTotal     uint64
	SmallAllocatedBytesTotal uint64

	LargePoolMaxEntries      int
	LargePoolEntries         int
	LargePoolRetainedBuffers uint64
	LargePoolRetainedBytes   uint64
	LargePoolGetsTotal       uint64
	LargePoolHitsTotal       uint64
	LargePoolMissesTotal     uint64
	LargePoolTooSmallTotal   uint64
	LargePoolPutsTotal       uint64
	LargePoolDropsTotal      uint64
	LargePoolDroppedBytes    uint64
	LargeAllocCallsTotal     uint64
	LargeAllocatedBytesTotal uint64

	OversizeAllocCallsTotal     uint64
	OversizeAllocatedBytesTotal uint64
	OversizeDropsTotal          uint64
	OversizeDroppedBytesTotal   uint64

	FileRetainMaxBytes  uint64
	FileSpareMaxCount   uint64
	FileRetainedFiles   uint64
	FileRetainedBuffers uint64
	FileRetainedBytes   uint64
}

DecodeScratchStats reports process-global decode scratch pool activity plus optional per-manager file stash retention filled by Manager.DecodeScratchStats.

func DecodeScratchStatsSnapshot added in v0.6.0 

func DecodeScratchStatsSnapshot() DecodeScratchStats

type DictLookup 

type DictLookup func(dictID uint64) ([]byte, error)

type EncodeCostModel 

type EncodeCostModel interface {
	EncodeNs(rawPayloadBytes int, records int) int64
}

EncodeCostModel provides deterministic encode cost estimates (ns) for tests/benches. When set on a Writer, it overrides wall-time measurement for sampled frames.

type EncodeCostModelFunc 

type EncodeCostModelFunc func(rawPayloadBytes int, records int) int64

EncodeCostModelFunc adapts a function to the EncodeCostModel interface.

func (EncodeCostModelFunc) EncodeNs 

func (f EncodeCostModelFunc) EncodeNs(rawPayloadBytes int, records int) int64

type File 

type File struct {
	ID   uint32
	Path string
	File *os.File

	RefCount atomic.Int64
	IsZombie atomic.Bool
	// contains filtered or unexported fields
}

File represents a value-log segment on disk.

func (*File) Close 

func (f *File) Close() error

func (*File) EnsureCurrentWritableReadableFor added in v0.6.0 

func (f *File) EnsureCurrentWritableReadableFor(ptr page.ValuePtr) error

func (*File) Read 

func (f *File) Read(ptr page.ValuePtr, verifyCRC bool) ([]byte, error)

func (*File) ReadAppend 

func (f *File) ReadAppend(ptr page.ValuePtr, verifyCRC bool, dst []byte) ([]byte, error)

func (*File) ReadStats added in v0.6.0 

func (f *File) ReadStats() ReadStats

ReadStats returns this file's read-path counters.

func (*File) ReadUnsafe 

func (f *File) ReadUnsafe(ptr page.ValuePtr, verifyCRC bool) ([]byte, error)

func (*File) ReadUnsafeAppend 

func (f *File) ReadUnsafeAppend(ptr page.ValuePtr, verifyCRC bool, dst []byte) ([]byte, error)

func (*File) ReadUnsafeTo added in v0.4.0 

func (f *File) ReadUnsafeTo(ptr page.ValuePtr, verifyCRC bool, dst []byte) ([]byte, bool, error)

ReadUnsafeTo is like ReadUnsafe, but it may return a slice backed by dst when dst has enough capacity for the decoded value. Callers must keep dst alive and avoid reusing it while they use the returned slice when usedDst is true.

func (*File) SizeBestEffort added in v0.4.0 

func (f *File) SizeBestEffort() int64

SizeBestEffort returns the current best-effort segment size in bytes. It prefers cached size/mmap metadata and only falls back to stat when needed.

type FrameHeader 

type FrameHeader struct {
	Version byte
	Flags   byte
	K       uint8
	// Reserved stores per-frame metadata. For compressed block frames
	// (Flags&FrameFlagCompressed, DictID==0), this holds the BlockCodec ID.
	Reserved uint8
	DictID   uint64
}

func DecodeFrame 

func DecodeFrame(body []byte) (FrameHeader, []uint64, []uint32, []byte, error)

func EncodeFrame 

func EncodeFrame(dictID uint64, dict []byte, records []Record) ([]byte, FrameHeader, error)

func EncodeFrameWithOptions added in v0.3.0 

func EncodeFrameWithOptions(dictID uint64, dict []byte, records []Record, level zstd.EncoderLevel, enableEntropy bool) ([]byte, FrameHeader, error)

type FramePreparer added in v0.3.0 

type FramePreparer struct {
	// contains filtered or unexported fields
}

FramePreparer reuses dict codecs and scratch buffers to prepare grouped frame bodies outside the value-log write lock.

It is not safe for concurrent use.

func NewFramePreparer added in v0.3.0 

func NewFramePreparer() *FramePreparer

func (*FramePreparer) PrepareFrame added in v0.3.0 

func (p *FramePreparer) PrepareFrame(dictID uint64, dict []byte, records []Record) ([]byte, FrameStats, error)

func (*FramePreparer) PrepareFrameInto added in v0.3.0 

func (p *FramePreparer) PrepareFrameInto(dst []byte, dictID uint64, dict []byte, records []Record) ([]byte, FrameStats, error)

PrepareFrameInto behaves like PrepareFrame, but writes the frame body into dst when capacity allows. Callers can reuse large frame buffers across requests.

func (*FramePreparer) ResetCompressionHints added in v0.3.0 

func (p *FramePreparer) ResetCompressionHints()

func (*FramePreparer) ResetForReuse added in v0.6.0 

func (p *FramePreparer) ResetForReuse()

ResetForReuse restores the default policy/hint state while retaining bounded scratch buffers. It is intended for callers that pool FramePreparers between independent append batches without changing compression semantics from a freshly-created preparer.

func (*FramePreparer) SetBlockCompression added in v0.3.0 

func (p *FramePreparer) SetBlockCompression(codec BlockCodec, enabled bool)

func (*FramePreparer) SetClock added in v0.3.0 

func (p *FramePreparer) SetClock(clock Clock)

func (*FramePreparer) SetDictFrameEncoderOptions added in v0.3.0 

func (p *FramePreparer) SetDictFrameEncoderOptions(level zstd.EncoderLevel, enableEntropy bool)

func (*FramePreparer) SetEncodeCostModel added in v0.3.0 

func (p *FramePreparer) SetEncodeCostModel(model EncodeCostModel)

func (*FramePreparer) SetEncodeSampleStride added in v0.3.0 

func (p *FramePreparer) SetEncodeSampleStride(stride uint64)

func (*FramePreparer) SetKeepPolicy added in v0.3.0 

func (p *FramePreparer) SetKeepPolicy(ioNsPerStoredByte, encodeNsPerRawByte, safetyMargin float64)

func (*FramePreparer) TrimScratchForReuse added in v0.6.0 

func (p *FramePreparer) TrimScratchForReuse()

TrimScratchForReuse drops oversized temporary buffers while preserving codec and compression-hint state. Long-lived prepare workers call this between tasks so an occasional large frame does not pin unbounded scratch memory.

type FrameStats 

type FrameStats struct {
	Records            int
	RawPayloadBytes    int
	StoredPayloadBytes int
	// Attempted is true when zstd encoding was executed (even if we fell back to raw).
	Attempted bool
	// Kept is true when compressed bytes were stored.
	Kept bool
	// EncodeNs is the sampled encode time (monotonic ns); 0 when not measured.
	EncodeNs int64
}

type GroupedFrameCacheStats added in v0.6.0 

type GroupedFrameCacheStats struct {
	Hits              uint64
	Misses            uint64
	Stores            uint64
	Evictions         uint64
	Releases          uint64
	RetainedBytes     uint64
	BudgetBytes       uint64
	SkippedDisabled   uint64
	SkippedOversize   uint64
	SkippedBudget     uint64
	SkippedContention uint64
	Entries           int
	Capacity          int
	AllocatedShards   int
	AllocatedSlots    int
}

GroupedFrameCacheStats captures decoded grouped-frame cache behavior. RetainedBytes is the currently retained decoded-raw bytes; BudgetBytes is the configured manager/file budget (0 means no byte budget beyond entry limits).

type GrowBufferStats added in v0.4.0 

type GrowBufferStats struct {
	CallsTotal                                            uint64
	ReallocCallsTotal                                     uint64
	RequestedBytesTotal                                   uint64
	AllocatedBytesTotal                                   uint64
	CopiedBytesTotal                                      uint64
	CapacityWasteBytesTotal                               uint64
	ReadAppendCompressedFallbackCallsTotal                uint64
	ReadAppendCompressedFallbackRequestedBytesTotal       uint64
	ReadAppendCompressedFallbackDstPresentCallsTotal      uint64
	ReadAppendCompressedFallbackDstFitCallsTotal          uint64
	ReadAppendCompressedFallbackDstFitRequestedBytesTotal uint64
	ReadAppendPayloadCallsTotal                           uint64
	ReadAppendPayloadRequestedBytesTotal                  uint64
	ReadAppendCurrentMmapDirectDecodeCallsTotal           uint64
	ReadAppendCurrentMmapDirectDecodeRequestedBytesTotal  uint64
	ReadAppendDecodedPayloadCallsTotal                    uint64
	ReadAppendDecodedPayloadRequestedBytesTotal           uint64
	ReadAppendDecodedPayloadDstPresentCallsTotal          uint64
	ReadAppendDecodedPayloadDstFitCallsTotal              uint64
	ReadAppendDecodedPayloadDstFitRequestedBytesTotal     uint64
	ReadAppendTemplateEncodedPayloadCallsTotal            uint64
	ReadAppendTemplateEncodedPayloadRequestedBytesTotal   uint64
}

func GrowBufferStatsSnapshot added in v0.4.0 

func GrowBufferStatsSnapshot() GrowBufferStats

type Manager 

type Manager struct {
	// contains filtered or unexported fields
}

func NewManager 

func NewManager(dir string) (*Manager, error)

func (*Manager) Acquire 

func (m *Manager) Acquire(set *Set)

Acquire increments the Set refcount (O(1)).

func (*Manager) AddScanDir added in v0.5.0 

func (m *Manager) AddScanDir(dir string) error

func (*Manager) Close 

func (m *Manager) Close() error

func (*Manager) CurrentSet 

func (m *Manager) CurrentSet() *Set

CurrentSet returns a snapshot of the current value-log files.

func (*Manager) CurrentSetNoRefresh added in v0.4.0 

func (m *Manager) CurrentSetNoRefresh() *Set

CurrentSetNoRefresh returns a snapshot of the currently registered value-log files without scanning the filesystem for newly created segments.

Callers in hot paths should prefer this when they know no new value-log segments can be referenced by the state being published.

func (*Manager) CurrentWritableFileIDs added in v0.6.0 

func (m *Manager) CurrentWritableFileIDs() []uint32

CurrentWritableFileIDs returns the registered current writable value-log segment IDs. These files may still be held open or mapped and must not be physically removed by maintenance cleanup.

func (*Manager) DecodeScratchStats added in v0.6.0 

func (m *Manager) DecodeScratchStats() DecodeScratchStats

func (*Manager) DemoteCurrentWritable added in v0.6.0 

func (m *Manager) DemoteCurrentWritable(fileID uint32)

DemoteCurrentWritable seals fileID if it is currently tracked as writable.

func (*Manager) EvictSegment added in v0.3.0 

func (m *Manager) EvictSegment(id uint32) error

EvictSegment closes and forgets a segment without deleting it from disk. This is useful when another component owns lifecycle/deletion.

func (*Manager) GroupedFrameCacheDetailedStats added in v0.6.0 

func (m *Manager) GroupedFrameCacheDetailedStats() GroupedFrameCacheStats

func (*Manager) GroupedFrameCacheStats added in v0.4.0 

func (m *Manager) GroupedFrameCacheStats() (hits, misses uint64, entries, capacity int)

func (*Manager) HasSegment added in v0.4.0 

func (m *Manager) HasSegment(id uint32) bool

HasSegment reports whether id is already registered and not marked zombie.

func (*Manager) MarkZombie 

func (m *Manager) MarkZombie(id uint32) error

func (*Manager) MarkZombieIfTracked added in v0.6.0 

func (m *Manager) MarkZombieIfTracked(id uint32) (tracked bool, newlyMarked bool, err error)

MarkZombieIfTracked marks id zombie when the manager still tracks it, including files that were already marked zombie by a previous cleanup pass.

func (*Manager) MmapReadStats added in v0.4.0 

func (m *Manager) MmapReadStats() (hits uint64, missesOutOfRange uint64, missesNoMapping uint64, missesDeadMappingCap uint64, fallbacksReadAt uint64)

func (*Manager) MmapResidencyStats added in v0.4.0 

func (m *Manager) MmapResidencyStats() (currentSegments uint64, currentBytes uint64, sealedSegments uint64, sealedBytes uint64, deadMappings uint64, deadBytes uint64)

MmapResidencyStats reports aggregate mmap residency split by segment type: current writable segments, sealed segments, and dead mappings/bytes.

func (*Manager) PromoteCurrentWritable added in v0.4.0 

func (m *Manager) PromoteCurrentWritable(fileID uint32) error

PromoteCurrentWritable marks fileID as current writable. By default this seals the previous current segment in the same encoded lane. Lanes enabled via SetMultiCurrentWritableLane instead use one current slot per file id unless a caller provides a previous file id with PromoteCurrentWritableReplacing.

func (*Manager) PromoteCurrentWritableReplacing added in v0.6.0 

func (m *Manager) PromoteCurrentWritableReplacing(fileID, previousFileID uint32) error

PromoteCurrentWritableReplacing marks fileID current-writable and, when previousFileID is non-zero, seals that writer's prior current segment even for lanes that allow multiple current writers.

func (*Manager) Read 

func (m *Manager) Read(ptr page.ValuePtr) ([]byte, error)

func (*Manager) ReadAppend 

func (m *Manager) ReadAppend(ptr page.ValuePtr, dst []byte) ([]byte, error)

func (*Manager) ReadChecksumEnabled added in v0.4.0 

func (m *Manager) ReadChecksumEnabled() bool

func (*Manager) ReadStats added in v0.6.0 

func (m *Manager) ReadStats() ReadStats

ReadStats returns aggregate read-path counters for currently tracked files.

func (*Manager) ReadUnsafe 

func (m *Manager) ReadUnsafe(ptr page.ValuePtr) ([]byte, error)

func (*Manager) ReadUnsafeAppend 

func (m *Manager) ReadUnsafeAppend(ptr page.ValuePtr, dst []byte) ([]byte, error)

func (*Manager) ReadUnsafeAppendBatch added in v0.4.0 

func (m *Manager) ReadUnsafeAppendBatch(ptrs []page.ValuePtr, dst [][]byte) ([][]byte, error)

func (*Manager) ReadUnsafeTo added in v0.4.0 

func (m *Manager) ReadUnsafeTo(ptr page.ValuePtr, dst []byte) ([]byte, bool, error)

func (*Manager) Refresh 

func (m *Manager) Refresh() error

Refresh scans the directory and registers any new segments.

func (*Manager) RefreshScanCount added in v0.4.0 

func (m *Manager) RefreshScanCount() uint64

RefreshScanCount returns the number of directory scans performed by Refresh(). This is used in tests and profiling to guard against accidental rescan loops.

func (*Manager) RegisterSegment added in v0.4.0 

func (m *Manager) RegisterSegment(path string, id uint32) error

RegisterSegment registers a newly created segment without scanning the filesystem. Callers that create value-log segments in-process should use this on the hot path so CurrentSetNoRefresh remains sufficient for immediate publish/read-after-write visibility.

func (*Manager) Release 

func (m *Manager) Release(set *Set) error

Release decrements the Set refcount and removes zombie files once unpinned.

func (*Manager) RemapStats 

func (m *Manager) RemapStats() (remaps uint64, deadMappings uint64)

RemapStats reports aggregate remap executions and tracked dead mappings.

func (*Manager) RemoveSegment 

func (m *Manager) RemoveSegment(id uint32) error

func (*Manager) RemoveSegmentForce 

func (m *Manager) RemoveSegmentForce(id uint32) error

RemoveSegmentForce removes a segment without refcount checks. Intended for recovery cleanup before any snapshots are live.

func (*Manager) RemoveSegmentIfUnpinned added in v0.6.0 

func (m *Manager) RemoveSegmentIfUnpinned(id uint32) (bool, error)

RemoveSegmentIfUnpinned removes a tracked segment only when no live snapshot pins it. It returns false with no error when the segment is absent or still pinned.

func (*Manager) RewriteLaneHint added in v0.4.0 

func (m *Manager) RewriteLaneHint() (lane uint32, startSeq uint32, ok bool)

RewriteLaneHint returns a best-effort lane/start-seq hint for creating new rewrite segments without scanning the filesystem.

It considers all currently tracked segments (including zombies) to avoid immediate lane/seq reuse while deletes are pending.

func (*Manager) SealedMapDeniedByReasonStats added in v0.4.0 

func (m *Manager) SealedMapDeniedByReasonStats() (countCap uint64, bytesCap uint64)

SealedMapDeniedByReasonStats reports sealed lazy-map denials split by budget: count-cap denials and byte-cap denials.

func (*Manager) SealedMapDeniedCount added in v0.4.0 

func (m *Manager) SealedMapDeniedCount() uint64

SealedMapDeniedCount is an alias for SealedMapDeniedStats.

func (*Manager) SealedMapDeniedStats added in v0.4.0 

func (m *Manager) SealedMapDeniedStats() uint64

SealedMapDeniedStats reports the total sealed-lazy-map deny count.

func (*Manager) SegmentPath 

func (m *Manager) SegmentPath(id uint32) string

func (*Manager) SetCurrentWritableMmapEnabled added in v0.6.0 

func (m *Manager) SetCurrentWritableMmapEnabled(enabled bool)

SetCurrentWritableMmapEnabled enables persistent read-only mmap views for current writable value-log segments owned by this manager.

func (*Manager) SetCurrentWritableReadBarrier added in v0.4.0 

func (m *Manager) SetCurrentWritableReadBarrier(fn func(fileID uint32) error)

SetCurrentWritableReadBarrier installs an optional callback that is invoked before reading a segment still marked currentWritable. Cached mode uses this to flush the owning writer so backend-internal tree reads do not observe EOF from still-buffered grouped records.

func (*Manager) SetCurrentWritableReadBarrierWithSize added in v0.6.0 

func (m *Manager) SetCurrentWritableReadBarrierWithSize(fn func(fileID uint32) (int64, error))

SetCurrentWritableReadBarrierWithSize is like SetCurrentWritableReadBarrier, but lets the callback return the current on-disk size after its flush barrier. A positive size is used as a file-size hint for current-writable mmap reads.

func (*Manager) SetDictLookup 

func (m *Manager) SetDictLookup(lookup DictLookup)

func (*Manager) SetDisableReadChecksum 

func (m *Manager) SetDisableReadChecksum(disable bool)

func (*Manager) SetGroupedFrameCacheEntries added in v0.4.0 

func (m *Manager) SetGroupedFrameCacheEntries(entries int)

func (*Manager) SetGroupedFrameCacheMaxBytes added in v0.6.0 

func (m *Manager) SetGroupedFrameCacheMaxBytes(maxBytes int64)

SetGroupedFrameCacheMaxBytes configures the manager-wide decoded grouped-frame byte budget. A value <= 0 disables the byte budget while preserving entry and per-frame raw-size bounds.

func (*Manager) SetGroupedFrameCacheMaxRawBytes added in v0.4.0 

func (m *Manager) SetGroupedFrameCacheMaxRawBytes(maxRaw int)

func (*Manager) SetMultiCurrentWritableLane added in v0.6.0 

func (m *Manager) SetMultiCurrentWritableLane(lane uint32, enabled bool)

SetMultiCurrentWritableLane allows multiple physical files with the same encoded lane id to remain current-writable at the same time. This is used for TreeDB leaf-log append lanes, where the reserved lane id identifies the leaf log class while file ids still identify independent writer streams.

func (*Manager) SetTemplateLookup added in v0.3.0 

func (m *Manager) SetTemplateLookup(lookup TemplateLookup, opts templ.DecodeOptions)

func (*Manager) TemplateDefCacheStats added in v0.3.0 

func (m *Manager) TemplateDefCacheStats() (hits, misses uint64, entries, capacity int)

func (*Manager) ZombieStats added in v0.4.0 

func (m *Manager) ZombieStats() (segments uint64, bytes uint64, pinnedSegments uint64, pinnedBytes uint64, unpinnedSegments uint64, unpinnedBytes uint64)

ZombieStats reports tracked zombie segments and their approximate byte totals. A zombie remains on disk until all snapshots release it (RefCount reaches 0).

type RawWriteStats added in v0.4.0 

type RawWriteStats struct {
	Syscalls uint64
	Bytes    uint64
	Calls    uint64
}

RawWriteStats captures plain write(2)-based append behavior.

type RawWritevStats added in v0.4.0 

type RawWritevStats struct {
	Syscalls uint64
	Bytes    uint64
	Iovecs   uint64
	Flushes  uint64
}

RawWritevStats captures low-level writev syscall behavior for grouped raw frame appends.

type ReadStats added in v0.6.0 

type ReadStats struct {
	// RecordCRCChecks counts value-log record CRC32 computations performed while
	// checksum verification is enabled. A grouped-frame read should increment this
	// once per verified grouped record read, not once per subvalue served from a
	// reused verified frame.
	RecordCRCChecks uint64
}

ReadStats captures value-log read-path counters.

type Reader 

type Reader struct {
	// contains filtered or unexported fields
}

func NewReader 

func NewReader(path string, fileID uint32) (*Reader, error)

func NewReaderFromFileWithBufferSize added in v0.6.0 

func NewReaderFromFileWithBufferSize(f *os.File, fileID uint32, bufferSize int) *Reader

NewReaderFromFileWithBufferSize creates a metadata/value reader from an already-open segment handle without taking ownership of that file. The file is read via a section reader so the caller's current offset is not disturbed.

func NewReaderWithBufferSize added in v0.4.0 

func NewReaderWithBufferSize(path string, fileID uint32, bufferSize int) (*Reader, error)

NewReaderWithBufferSize creates a value-log reader with an explicit bufio read buffer size. Callers doing metadata-only scans can use a smaller buffer to reduce transient allocation footprint.

func (*Reader) Close 

func (r *Reader) Close() error

func (*Reader) DisableChecksum 

func (r *Reader) DisableChecksum()

func (*Reader) DisableValueDecode 

func (r *Reader) DisableValueDecode()

func (*Reader) ReadNext 

func (r *Reader) ReadNext() (uint64, []byte, page.ValuePtr, error)

func (*Reader) ReadNextMeta added in v0.4.0 

func (r *Reader) ReadNextMeta() (uint64, page.ValuePtr, error)

ReadNextMeta streams the next RID + pointer without allocating or returning the record payload bytes.

It mirrors ReadNext's RID/pointer semantics, including grouped-record pending expansion, but always returns a nil value payload.

func (*Reader) SetDictLookup 

func (r *Reader) SetDictLookup(lookup DictLookup)

func (*Reader) SetTemplateLookup added in v0.3.0 

func (r *Reader) SetTemplateLookup(lookup TemplateLookup, opts templ.DecodeOptions)

func (*Reader) ValidateDicts 

func (r *Reader) ValidateDicts()

ValidateDicts enables dictionary existence checks even when value decoding is disabled. This provides a low-cost "fail fast" validation pass during WAL replay and open.

type RealClock 

type RealClock struct{}

func (RealClock) Now 

func (RealClock) Now() time.Time

type Record 

type Record struct {
	RID   uint64
	Value []byte
}

type Set 

type Set struct {
	Files    map[uint32]*File
	RefCount atomic.Int64
	// contains filtered or unexported fields
}

Set is an immutable snapshot of value-log files for snapshot isolation.

func (*Set) Read 

func (s *Set) Read(ptr page.ValuePtr) ([]byte, error)

func (*Set) ReadAppend 

func (s *Set) ReadAppend(ptr page.ValuePtr, dst []byte) ([]byte, error)

func (*Set) ReadChecksumEnabled added in v0.4.0 

func (s *Set) ReadChecksumEnabled() bool

func (*Set) ReadStats added in v0.6.0 

func (s *Set) ReadStats() ReadStats

ReadStats returns aggregate read-path counters for this snapshot set.

func (*Set) ReadUnsafe 

func (s *Set) ReadUnsafe(ptr page.ValuePtr) ([]byte, error)

func (*Set) ReadUnsafeAppend 

func (s *Set) ReadUnsafeAppend(ptr page.ValuePtr, dst []byte) ([]byte, error)

func (*Set) ReadUnsafeAppendBatch added in v0.4.0 

func (s *Set) ReadUnsafeAppendBatch(ptrs []page.ValuePtr, dst [][]byte) ([][]byte, error)

ReadUnsafeAppendBatch resolves pointers in order, reusing file lookups for contiguous same-file runs to reduce scan-path overhead.

func (*Set) ReadUnsafeTo added in v0.4.0 

func (s *Set) ReadUnsafeTo(ptr page.ValuePtr, dst []byte) ([]byte, bool, error)

type TemplateLookup added in v0.3.0 

type TemplateLookup func(templateID uint64) ([]byte, error)

type UnsupportedBlockCodecError added in v0.3.0 

type UnsupportedBlockCodecError struct {
	CodecID uint8
}

UnsupportedBlockCodecError reports an unknown per-frame block codec ID.

func (UnsupportedBlockCodecError) Error added in v0.3.0 

func (e UnsupportedBlockCodecError) Error() string

type VirtualClock 

type VirtualClock struct {
	// contains filtered or unexported fields
}

VirtualClock provides deterministic time control for tests.

func NewVirtualClock 

func NewVirtualClock(start time.Time) *VirtualClock

func (*VirtualClock) Advance 

func (c *VirtualClock) Advance(ns int64)

func (*VirtualClock) Now 

func (c *VirtualClock) Now() time.Time

type VirtualSink 

type VirtualSink struct {
	Clock          interface{ Advance(int64) }
	NsPerByte      int64
	SyncPenaltyNs  int64
	FlushPenaltyNs int64
}

VirtualSink advances a clock based on written bytes to simulate IO wall time. Intended for deterministic tests/benchmarks.

func (*VirtualSink) Flush 

func (s *VirtualSink) Flush() error

func (*VirtualSink) Sync 

func (s *VirtualSink) Sync() error

func (*VirtualSink) Write 

func (s *VirtualSink) Write(p []byte) (int, error)

type Writer 

type Writer struct {
	// contains filtered or unexported fields
}

func NewWriter 

func NewWriter(path string, fileID uint32) (*Writer, error)

func NewWriterWithSink 

func NewWriterWithSink(sink io.Writer, fileID uint32) *Writer

NewWriterWithSink creates a value-log writer that writes to the provided sink. Use this for sink-backed/custom outputs that do not provide file-backed durability.

func (*Writer) Append 

func (w *Writer) Append(dictID uint64, dict []byte, rid uint64, value []byte) (page.ValuePtr, error)

func (*Writer) AppendEncodedFrameInto added in v0.3.0 

func (w *Writer) AppendEncodedFrameInto(body []byte, k int, dst []page.ValuePtr) ([]page.ValuePtr, error)

AppendEncodedFrameInto appends a pre-encoded grouped frame body and fills dst with grouped value pointers (dst must be at least k long).

The body must be produced by EncodeFrame/EncodeFrameWithOptions.

func (*Writer) AppendEncodedFrameOne added in v0.6.0 

func (w *Writer) AppendEncodedFrameOne(body []byte) (page.ValuePtr, error)

AppendEncodedFrameOne appends a pre-encoded single-record grouped frame body and returns its value pointer without requiring caller-owned pointer-slice scratch.

func (*Writer) AppendFrame 

func (w *Writer) AppendFrame(dictID uint64, dict []byte, records []Record) ([]page.ValuePtr, error)

func (*Writer) AppendFrameWithStats 

func (w *Writer) AppendFrameWithStats(dictID uint64, dict []byte, records []Record) ([]page.ValuePtr, FrameStats, error)

func (*Writer) AppendFrameWithStatsInto 

func (w *Writer) AppendFrameWithStatsInto(dictID uint64, dict []byte, records []Record, dst []page.ValuePtr) ([]page.ValuePtr, FrameStats, error)

AppendFrameWithStatsInto appends a grouped frame and fills dst with the returned pointers (dst must be at least len(records) long).

This is a performance-oriented helper to avoid allocating a new pointer slice on every frame append.

func (*Writer) AppendOneFrameWithStats added in v0.4.0 

func (w *Writer) AppendOneFrameWithStats(dictID uint64, dict []byte, rid uint64, value []byte) (page.ValuePtr, FrameStats, error)

func (*Writer) AppendRawFramesBufferedInto added in v0.4.0 

func (w *Writer) AppendRawFramesBufferedInto(records []Record, k int, dst []page.ValuePtr) ([]page.ValuePtr, FrameStats, error)

AppendRawFramesBufferedInto appends raw grouped frames through the writer's append buffer, allowing multiple calls to coalesce into larger write syscalls.

Compared to AppendRawFramesWritevInto, this path stages frame bytes in the writer append buffer and flushes later, so independent append calls can merge into fewer, larger writes.

Prefer this for high-throughput append streams where delayed flush is acceptable and syscall coalescing across calls is desirable.

dst must be at least len(records) long.

func (*Writer) AppendRawFramesWritevInto 

func (w *Writer) AppendRawFramesWritevInto(records []Record, k int, dst []page.ValuePtr) ([]page.ValuePtr, FrameStats, error)

AppendRawFramesWritevInto appends raw (uncompressed) grouped frames using a writev batching strategy.

This avoids concatenating payloads into a contiguous frame buffer, reducing user-space copying for large value workloads. It always flushes the writev queue before returning, so it does not retain references to the input slices.

Prefer this when the caller needs pointers durably visible at function return and cannot rely on future appends to coalesce writes.

dst must be at least len(records) long.

func (*Writer) AppendRawRecord added in v0.3.0 

func (w *Writer) AppendRawRecord(raw []byte, length uint32) (page.ValuePtr, error)

AppendRawRecord appends a raw value-log record (CRC + header + payload) without re-encoding. The length argument should include any pointer flags.

func (*Writer) Close 

func (w *Writer) Close() error

func (*Writer) FileID 

func (w *Writer) FileID() uint32

func (*Writer) Flush 

func (w *Writer) Flush() error

func (*Writer) PendingBytes added in v0.4.0 

func (w *Writer) PendingBytes() int

PendingBytes reports bytes accepted by the writer but not yet flushed to the underlying file descriptor. For file-backed writers this is the append-buffer tail that same-process readers cannot see through ReadAt until flushed.

func (*Writer) RawWriteStats added in v0.4.0 

func (w *Writer) RawWriteStats() RawWriteStats

RawWriteStats returns cumulative plain write(2) counters for this writer.

func (*Writer) RawWritevStats added in v0.4.0 

func (w *Writer) RawWritevStats() RawWritevStats

RawWritevStats returns cumulative writev syscall counters for this writer.

func (*Writer) ResetCompressionHints 

func (w *Writer) ResetCompressionHints()

ResetCompressionHints clears skip/backoff state for deterministic benches.

func (*Writer) RotateTo 

func (w *Writer) RotateTo(path string, fileID uint32) error

RotateTo flushes and closes the current file, then opens (or creates) the provided path and reuses the writer's buffers for future appends.

func (*Writer) RotateToWithSync added in v0.6.0 

func (w *Writer) RotateToWithSync(path string, fileID uint32, syncCurrent bool) error

RotateToWithSync flushes and closes the current file, then opens (or creates) the provided path and reuses the writer's buffers for future appends. When syncCurrent is false, the current file is flushed to the OS but not fsynced.

func (*Writer) SetBlockCompression added in v0.3.0 

func (w *Writer) SetBlockCompression(codec BlockCodec, enabled bool)

SetBlockCompression configures grouped frame block compression for dictID=0 append paths.

func (*Writer) SetClock 

func (w *Writer) SetClock(clock Clock)

func (*Writer) SetDictFrameEncoderOptions added in v0.3.0 

func (w *Writer) SetDictFrameEncoderOptions(level zstd.EncoderLevel, enableEntropy bool)

func (*Writer) SetEncodeCostModel 

func (w *Writer) SetEncodeCostModel(model EncodeCostModel)

func (*Writer) SetEncodeSampleStride 

func (w *Writer) SetEncodeSampleStride(stride uint64)

func (*Writer) SetKeepPolicy 

func (w *Writer) SetKeepPolicy(ioNsPerStoredByte, encodeNsPerRawByte, safetyMargin float64)

func (*Writer) SetRawWritevStrategy added in v0.3.0 

func (w *Writer) SetRawWritevStrategy(minAvgBytes, minBatchRecs int)

SetRawWritevStrategy configures grouped raw writev usage heuristics.

minAvgBytes:

  • 0 enables adaptive mode (no average-bytes floor).
  • >0 requires average payload bytes/record to meet this floor.

minBatchRecs:

  • <=0 uses the default.

func (*Writer) Size 

func (w *Writer) Size() int64

func (*Writer) Sync 

func (w *Writer) Sync() error

type WriterAppendBufferStats added in v0.6.0 

type WriterAppendBufferStats struct {
	PoolMaxEntries      int
	PoolEntries         int
	PoolRetainedBuffers uint64
	PoolRetainedBytes   uint64
	GetsTotal           uint64
	HitsTotal           uint64
	MissesTotal         uint64
	PutsTotal           uint64
	DropsTotal          uint64
	DroppedBytesTotal   uint64
	AllocCallsTotal     uint64
	AllocatedBytesTotal uint64
}

WriterAppendBufferStats reports process-global writer append-buffer pool activity. Live Writer.appendBuf ownership is intentionally not inferred here; pprof plus pool/drop counters distinguish pooled retention from live writers.

func WriterAppendBufferStatsSnapshot added in v0.6.0 

func WriterAppendBufferStatsSnapshot() WriterAppendBufferStats

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.