valuelog

package
v0.5.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 17, 2026 License: MIT Imports: 30 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	Version = 1

	HeaderSize = 4 + 1 + 1 + 2 + 8 + 4
)
View Source 
const (
	FrameVersion    = 1
	FrameHeaderSize = 12
	MaxFrameK       = 128
)
View Source 
const DefaultKeepSafetyMargin = 0.10
View Source 
const (
	FrameFlagCompressed byte = 1 << 0
)
View Source 
const (

	// ReservedLeafLogLaneID is the dedicated lane used for outer-leaf storage.
	ReservedLeafLogLaneID = maxLaneID
)

Variables

View Source 
var (
	MaxMappedSealedSegments           = defaultMaxMappedSealed
	MaxMappedSealedBytes        int64 = defaultMaxMappedSealedBytes
	MaxMappedLeafSealedSegments       = defaultMaxMappedLeafSealed
	MaxMappedLeafSealedBytes    int64 = defaultMaxMappedLeafSealedBytes
)
View Source 
var (
	ErrCorrupt         = errors.New("valuelog: corrupt record")
	ErrRecordTooLarge  = errors.New("valuelog: record too large")
	ErrMissingDict     = errors.New("valuelog: missing dict bytes")
	ErrMissingTemplate = errors.New("valuelog: missing template bytes")
)
View Source 
var ErrFileNotFound = errors.New("valuelog: file not found")
View Source 
var ErrSegmentIDRange = errors.New("valuelog: segment id out of range")
View Source 
var MaxDeadMappings = defaultMaxDeadMappings

MaxDeadMappings is the base cap for old mmaps retained to avoid exhausting vm.max_map_count. Unless explicitly configured, the effective cap can grow with mapped size up to maxAdaptiveDeadMappings. Set <= 0 to disable the cap.

Each remap retains the previous mapping until the file is closed to avoid use-after-unmap with concurrent readers.

Functions

func ChooseBlockGroupK added in v0.3.0 

func ChooseBlockGroupK(records, rawPayloadBytes, targetCompressedBytes int, observedRatio float64) int

ChooseBlockGroupK estimates grouped frame size K for block compression.

observedRatio is stored/raw payload ratio. Values >= 0.98 are treated as expansion-risk/incompressible and force k=1.

func DecodeFileID 

func DecodeFileID(fileID uint32) (lane uint32, seq uint32)

func DecodeSegmentID 

func DecodeSegmentID(id uint32) (lane uint32, seq uint32)

func EncodeFileID 

func EncodeFileID(lane uint32, seq uint32) (uint32, error)

func EncodeSegmentID 

func EncodeSegmentID(lane uint32, seq uint32) (uint32, error)

func GenerateAutotuneValues 

func GenerateAutotuneValues(workload AutotuneWorkload, size, count int, seed int64) [][]byte

GenerateAutotuneValues builds deterministic samples for a workload.

func HasCompactLeafLogPayload added in v0.5.0 

func HasCompactLeafLogPayload(payload []byte) bool

func MaybeCompactLeafLogPayload added in v0.5.0 

func MaybeCompactLeafLogPayload(leafPage []byte) ([]byte, bool, error)

func NormalizeBlockTargetCompressedBytes added in v0.3.0 

func NormalizeBlockTargetCompressedBytes(v int) int

NormalizeBlockTargetCompressedBytes returns a bounded grouped block target.

func ReadAt 

func ReadAt(f *os.File, ptr page.ValuePtr, verifyCRC bool) ([]byte, error)

func ReadAtTo added in v0.4.0 

func ReadAtTo(f *os.File, ptr page.ValuePtr, verifyCRC bool, dst []byte) ([]byte, bool, error)

ReadAtTo decodes ptr from f and, when possible, writes decoded bytes into dst. It returns the decoded value, whether dst backed the returned slice, and an error.

Callers must treat the returned bytes as immutable. When usedDst is true, the returned slice aliases dst.

func ReadAtWithDict 

func ReadAtWithDict(f *os.File, ptr page.ValuePtr, verifyCRC bool, dictLookup DictLookup, templateLookup TemplateLookup, templateCache *templateDefCache, templateOpts templ.DecodeOptions) ([]byte, error)

func ReadAtWithDictTo added in v0.4.0 

func ReadAtWithDictTo(f *os.File, ptr page.ValuePtr, verifyCRC bool, dictLookup DictLookup, templateLookup TemplateLookup, templateCache *templateDefCache, templateOpts templ.DecodeOptions, dst []byte) ([]byte, bool, error)

ReadAtWithDictTo is ReadAtWithDict with caller-provided decode storage.

The returned slice may alias dst when usedDst is true. The returned bytes are immutable from the caller perspective.

func ShouldKeepCompressed 

func ShouldKeepCompressed(rawBytes, encodedBytes int, encodeNs int64, ioNsPerStoredByte, safetyMargin float64) bool

ShouldKeepCompressed decides whether to keep compressed bytes for a frame. It compares estimated IO savings against encode cost with a safety margin.

Types

type AutotuneMode 

type AutotuneMode uint8
const (
	// AutotuneUnset indicates the caller did not explicitly configure the
	// autotuner mode. NormalizeAutotuneOptions maps this to a sensible default
	// based on whether the value log is enabled.
	AutotuneUnset AutotuneMode = iota
	AutotuneOff
	AutotuneMedium
	AutotuneAggressive
)

type AutotuneOptions 

type AutotuneOptions struct {
	Mode AutotuneMode

	CandidateK            []int
	CandidateHistoryBytes []int
	CandidateDictBytes    []int

	MinGainToSwitch float64
	MinDwellFrames  uint64

	SampleStride     uint64
	MaxSampleBytes   uint64
	TrainCPUFraction float64

	ProbeBytes uint64
	PauseBytes uint64

	DisableBelowValueBytes int
}

func NormalizeAutotuneOptions 

func NormalizeAutotuneOptions(opts AutotuneOptions, valueLogEnabled bool) AutotuneOptions

type AutotuneWorkload 

type AutotuneWorkload struct {
	Name string
	Make func(rng *rand.Rand, size int) []byte
}

AutotuneWorkload describes a deterministic value generator.

func AutotuneWorkloads 

func AutotuneWorkloads() []AutotuneWorkload

AutotuneWorkloads returns the standard deterministic workloads used by the autotune bench suite.

func LookupAutotuneWorkload 

func LookupAutotuneWorkload(name string) (AutotuneWorkload, bool)

LookupAutotuneWorkload returns the workload with the given name.

type BlockCodec added in v0.3.0 

type BlockCodec uint8

BlockCodec identifies non-dictionary value-log frame compression codecs. 

const (
	BlockCodecNone BlockCodec = iota
	BlockCodecSnappy
	BlockCodecLZ4
)

type Clock 

type Clock interface {
	Now() time.Time
}

type DictLookup 

type DictLookup func(dictID uint64) ([]byte, error)

type EncodeCostModel 

type EncodeCostModel interface {
	EncodeNs(rawPayloadBytes int, records int) int64
}

EncodeCostModel provides deterministic encode cost estimates (ns) for tests/benches. When set on a Writer, it overrides wall-time measurement for sampled frames.

type EncodeCostModelFunc 

type EncodeCostModelFunc func(rawPayloadBytes int, records int) int64

EncodeCostModelFunc adapts a function to the EncodeCostModel interface.

func (EncodeCostModelFunc) EncodeNs 

func (f EncodeCostModelFunc) EncodeNs(rawPayloadBytes int, records int) int64

type File 

type File struct {
	ID   uint32
	Path string
	File *os.File

	RefCount atomic.Int64
	IsZombie atomic.Bool
	// contains filtered or unexported fields
}

File represents a value-log segment on disk.

func (*File) Close 

func (f *File) Close() error

func (*File) Read 

func (f *File) Read(ptr page.ValuePtr, verifyCRC bool) ([]byte, error)

func (*File) ReadAppend 

func (f *File) ReadAppend(ptr page.ValuePtr, verifyCRC bool, dst []byte) ([]byte, error)

func (*File) ReadUnsafe 

func (f *File) ReadUnsafe(ptr page.ValuePtr, verifyCRC bool) ([]byte, error)

func (*File) ReadUnsafeAppend 

func (f *File) ReadUnsafeAppend(ptr page.ValuePtr, verifyCRC bool, dst []byte) ([]byte, error)

func (*File) ReadUnsafeTo added in v0.4.0 

func (f *File) ReadUnsafeTo(ptr page.ValuePtr, verifyCRC bool, dst []byte) ([]byte, bool, error)

ReadUnsafeTo is like ReadUnsafe, but it may return a slice backed by dst when dst has enough capacity for the decoded value. Callers must keep dst alive and avoid reusing it while they use the returned slice when usedDst is true.

func (*File) SizeBestEffort added in v0.4.0 

func (f *File) SizeBestEffort() int64

SizeBestEffort returns the current best-effort segment size in bytes. It prefers cached size/mmap metadata and only falls back to stat when needed.

type FrameHeader 

type FrameHeader struct {
	Version byte
	Flags   byte
	K       uint8
	// Reserved stores per-frame metadata. For compressed block frames
	// (Flags&FrameFlagCompressed, DictID==0), this holds the BlockCodec ID.
	Reserved uint8
	DictID   uint64
}

func DecodeFrame 

func DecodeFrame(body []byte) (FrameHeader, []uint64, []uint32, []byte, error)

func EncodeFrame 

func EncodeFrame(dictID uint64, dict []byte, records []Record) ([]byte, FrameHeader, error)

func EncodeFrameWithOptions added in v0.3.0 

func EncodeFrameWithOptions(dictID uint64, dict []byte, records []Record, level zstd.EncoderLevel, enableEntropy bool) ([]byte, FrameHeader, error)

type FramePreparer added in v0.3.0 

type FramePreparer struct {
	// contains filtered or unexported fields
}

FramePreparer reuses dict codecs and scratch buffers to prepare grouped frame bodies outside the value-log write lock.

It is not safe for concurrent use.

func NewFramePreparer added in v0.3.0 

func NewFramePreparer() *FramePreparer

func (*FramePreparer) PrepareFrame added in v0.3.0 

func (p *FramePreparer) PrepareFrame(dictID uint64, dict []byte, records []Record) ([]byte, FrameStats, error)

func (*FramePreparer) PrepareFrameInto added in v0.3.0 

func (p *FramePreparer) PrepareFrameInto(dst []byte, dictID uint64, dict []byte, records []Record) ([]byte, FrameStats, error)

PrepareFrameInto behaves like PrepareFrame, but writes the frame body into dst when capacity allows. Callers can reuse large frame buffers across requests.

func (*FramePreparer) ResetCompressionHints added in v0.3.0 

func (p *FramePreparer) ResetCompressionHints()

func (*FramePreparer) SetBlockCompression added in v0.3.0 

func (p *FramePreparer) SetBlockCompression(codec BlockCodec, enabled bool)

func (*FramePreparer) SetClock added in v0.3.0 

func (p *FramePreparer) SetClock(clock Clock)

func (*FramePreparer) SetDictFrameEncoderOptions added in v0.3.0 

func (p *FramePreparer) SetDictFrameEncoderOptions(level zstd.EncoderLevel, enableEntropy bool)

func (*FramePreparer) SetEncodeCostModel added in v0.3.0 

func (p *FramePreparer) SetEncodeCostModel(model EncodeCostModel)

func (*FramePreparer) SetEncodeSampleStride added in v0.3.0 

func (p *FramePreparer) SetEncodeSampleStride(stride uint64)

func (*FramePreparer) SetKeepPolicy added in v0.3.0 

func (p *FramePreparer) SetKeepPolicy(ioNsPerStoredByte, encodeNsPerRawByte, safetyMargin float64)

type FrameStats 

type FrameStats struct {
	Records            int
	RawPayloadBytes    int
	StoredPayloadBytes int
	// Attempted is true when zstd encoding was executed (even if we fell back to raw).
	Attempted bool
	// Kept is true when compressed bytes were stored.
	Kept bool
	// EncodeNs is the sampled encode time (monotonic ns); 0 when not measured.
	EncodeNs int64
}

type GrowBufferStats added in v0.4.0 

type GrowBufferStats struct {
	CallsTotal                                            uint64
	ReallocCallsTotal                                     uint64
	RequestedBytesTotal                                   uint64
	AllocatedBytesTotal                                   uint64
	CopiedBytesTotal                                      uint64
	CapacityWasteBytesTotal                               uint64
	ReadAppendCompressedFallbackCallsTotal                uint64
	ReadAppendCompressedFallbackRequestedBytesTotal       uint64
	ReadAppendCompressedFallbackDstPresentCallsTotal      uint64
	ReadAppendCompressedFallbackDstFitCallsTotal          uint64
	ReadAppendCompressedFallbackDstFitRequestedBytesTotal uint64
	ReadAppendPayloadCallsTotal                           uint64
	ReadAppendPayloadRequestedBytesTotal                  uint64
	ReadAppendCurrentMmapDirectDecodeCallsTotal           uint64
	ReadAppendCurrentMmapDirectDecodeRequestedBytesTotal  uint64
}

func GrowBufferStatsSnapshot added in v0.4.0 

func GrowBufferStatsSnapshot() GrowBufferStats

type Manager 

type Manager struct {
	// contains filtered or unexported fields
}

func NewManager 

func NewManager(dir string) (*Manager, error)

func (*Manager) Acquire 

func (m *Manager) Acquire(set *Set)

Acquire increments the Set refcount (O(1)).

func (*Manager) AddScanDir added in v0.5.0 

func (m *Manager) AddScanDir(dir string) error

func (*Manager) Close 

func (m *Manager) Close() error

func (*Manager) CurrentSet 

func (m *Manager) CurrentSet() *Set

CurrentSet returns a snapshot of the current value-log files.

func (*Manager) CurrentSetNoRefresh added in v0.4.0 

func (m *Manager) CurrentSetNoRefresh() *Set

CurrentSetNoRefresh returns a snapshot of the currently registered value-log files without scanning the filesystem for newly created segments.

Callers in hot paths should prefer this when they know no new value-log segments can be referenced by the state being published.

func (*Manager) EvictSegment added in v0.3.0 

func (m *Manager) EvictSegment(id uint32) error

EvictSegment closes and forgets a segment without deleting it from disk. This is useful when another component owns lifecycle/deletion.

func (*Manager) GroupedFrameCacheStats added in v0.4.0 

func (m *Manager) GroupedFrameCacheStats() (hits, misses uint64, entries, capacity int)

func (*Manager) HasSegment added in v0.4.0 

func (m *Manager) HasSegment(id uint32) bool

HasSegment reports whether id is already registered and not marked zombie.

func (*Manager) MarkZombie 

func (m *Manager) MarkZombie(id uint32) error

func (*Manager) MmapReadStats added in v0.4.0 

func (m *Manager) MmapReadStats() (hits uint64, missesOutOfRange uint64, missesNoMapping uint64, missesDeadMappingCap uint64, fallbacksReadAt uint64)

func (*Manager) MmapResidencyStats added in v0.4.0 

func (m *Manager) MmapResidencyStats() (currentSegments uint64, currentBytes uint64, sealedSegments uint64, sealedBytes uint64, deadMappings uint64, deadBytes uint64)

MmapResidencyStats reports aggregate mmap residency split by segment type: current writable segments, sealed segments, and dead mappings/bytes.

func (*Manager) PromoteCurrentWritable added in v0.4.0 

func (m *Manager) PromoteCurrentWritable(fileID uint32) error

PromoteCurrentWritable marks fileID as the current writable segment for its lane and seals the previous current segment in that lane.

func (*Manager) Read 

func (m *Manager) Read(ptr page.ValuePtr) ([]byte, error)

func (*Manager) ReadAppend 

func (m *Manager) ReadAppend(ptr page.ValuePtr, dst []byte) ([]byte, error)

func (*Manager) ReadChecksumEnabled added in v0.4.0 

func (m *Manager) ReadChecksumEnabled() bool

func (*Manager) ReadUnsafe 

func (m *Manager) ReadUnsafe(ptr page.ValuePtr) ([]byte, error)

func (*Manager) ReadUnsafeAppend 

func (m *Manager) ReadUnsafeAppend(ptr page.ValuePtr, dst []byte) ([]byte, error)

func (*Manager) ReadUnsafeAppendBatch added in v0.4.0 

func (m *Manager) ReadUnsafeAppendBatch(ptrs []page.ValuePtr, dst [][]byte) ([][]byte, error)

func (*Manager) ReadUnsafeTo added in v0.4.0 

func (m *Manager) ReadUnsafeTo(ptr page.ValuePtr, dst []byte) ([]byte, bool, error)

func (*Manager) Refresh 

func (m *Manager) Refresh() error

Refresh scans the directory and registers any new segments.

func (*Manager) RefreshScanCount added in v0.4.0 

func (m *Manager) RefreshScanCount() uint64

RefreshScanCount returns the number of directory scans performed by Refresh(). This is used in tests and profiling to guard against accidental rescan loops.

func (*Manager) RegisterSegment added in v0.4.0 

func (m *Manager) RegisterSegment(path string, id uint32) error

RegisterSegment registers a newly created segment without scanning the filesystem. Callers that create value-log segments in-process should use this on the hot path so CurrentSetNoRefresh remains sufficient for immediate publish/read-after-write visibility.

func (*Manager) Release 

func (m *Manager) Release(set *Set) error

Release decrements the Set refcount and removes zombie files once unpinned.

func (*Manager) RemapStats 

func (m *Manager) RemapStats() (remaps uint64, deadMappings uint64)

RemapStats reports aggregate remap executions and tracked dead mappings.

func (*Manager) RemoveSegment 

func (m *Manager) RemoveSegment(id uint32) error

func (*Manager) RemoveSegmentForce 

func (m *Manager) RemoveSegmentForce(id uint32) error

RemoveSegmentForce removes a segment without refcount checks. Intended for recovery cleanup before any snapshots are live.

func (*Manager) RewriteLaneHint added in v0.4.0 

func (m *Manager) RewriteLaneHint() (lane uint32, startSeq uint32, ok bool)

RewriteLaneHint returns a best-effort lane/start-seq hint for creating new rewrite segments without scanning the filesystem.

It considers all currently tracked segments (including zombies) to avoid immediate lane/seq reuse while deletes are pending.

func (*Manager) SealedMapDeniedByReasonStats added in v0.4.0 

func (m *Manager) SealedMapDeniedByReasonStats() (countCap uint64, bytesCap uint64)

SealedMapDeniedByReasonStats reports sealed lazy-map denials split by budget: count-cap denials and byte-cap denials.

func (*Manager) SealedMapDeniedCount added in v0.4.0 

func (m *Manager) SealedMapDeniedCount() uint64

SealedMapDeniedCount is an alias for SealedMapDeniedStats.

func (*Manager) SealedMapDeniedStats added in v0.4.0 

func (m *Manager) SealedMapDeniedStats() uint64

SealedMapDeniedStats reports the total sealed-lazy-map deny count.

func (*Manager) SegmentPath 

func (m *Manager) SegmentPath(id uint32) string

func (*Manager) SetCurrentWritableReadBarrier added in v0.4.0 

func (m *Manager) SetCurrentWritableReadBarrier(fn func(fileID uint32) error)

SetCurrentWritableReadBarrier installs an optional callback that is invoked before reading a segment still marked currentWritable. Cached mode uses this to flush the owning writer so backend-internal tree reads do not observe EOF from still-buffered grouped records.

func (*Manager) SetDictLookup 

func (m *Manager) SetDictLookup(lookup DictLookup)

func (*Manager) SetDisableReadChecksum 

func (m *Manager) SetDisableReadChecksum(disable bool)

func (*Manager) SetGroupedFrameCacheEntries added in v0.4.0 

func (m *Manager) SetGroupedFrameCacheEntries(entries int)

func (*Manager) SetGroupedFrameCacheMaxRawBytes added in v0.4.0 

func (m *Manager) SetGroupedFrameCacheMaxRawBytes(maxRaw int)

func (*Manager) SetTemplateLookup added in v0.3.0 

func (m *Manager) SetTemplateLookup(lookup TemplateLookup, opts templ.DecodeOptions)

func (*Manager) TemplateDefCacheStats added in v0.3.0 

func (m *Manager) TemplateDefCacheStats() (hits, misses uint64, entries, capacity int)

func (*Manager) ZombieStats added in v0.4.0 

func (m *Manager) ZombieStats() (segments uint64, bytes uint64, pinnedSegments uint64, pinnedBytes uint64, unpinnedSegments uint64, unpinnedBytes uint64)

ZombieStats reports tracked zombie segments and their approximate byte totals. A zombie remains on disk until all snapshots release it (RefCount reaches 0).

type RawWriteStats added in v0.4.0 

type RawWriteStats struct {
	Syscalls uint64
	Bytes    uint64
	Calls    uint64
}

RawWriteStats captures plain write(2)-based append behavior.

type RawWritevStats added in v0.4.0 

type RawWritevStats struct {
	Syscalls uint64
	Bytes    uint64
	Iovecs   uint64
	Flushes  uint64
}

RawWritevStats captures low-level writev syscall behavior for grouped raw frame appends.

type Reader 

type Reader struct {
	// contains filtered or unexported fields
}

func NewReader 

func NewReader(path string, fileID uint32) (*Reader, error)

func NewReaderWithBufferSize added in v0.4.0 

func NewReaderWithBufferSize(path string, fileID uint32, bufferSize int) (*Reader, error)

NewReaderWithBufferSize creates a value-log reader with an explicit bufio read buffer size. Callers doing metadata-only scans can use a smaller buffer to reduce transient allocation footprint.

func (*Reader) Close 

func (r *Reader) Close() error

func (*Reader) DisableChecksum 

func (r *Reader) DisableChecksum()

func (*Reader) DisableValueDecode 

func (r *Reader) DisableValueDecode()

func (*Reader) ReadNext 

func (r *Reader) ReadNext() (uint64, []byte, page.ValuePtr, error)

func (*Reader) ReadNextMeta added in v0.4.0 

func (r *Reader) ReadNextMeta() (uint64, page.ValuePtr, error)

ReadNextMeta streams the next RID + pointer without allocating or returning the record payload bytes.

It mirrors ReadNext's RID/pointer semantics, including grouped-record pending expansion, but always returns a nil value payload.

func (*Reader) SetDictLookup 

func (r *Reader) SetDictLookup(lookup DictLookup)

func (*Reader) SetTemplateLookup added in v0.3.0 

func (r *Reader) SetTemplateLookup(lookup TemplateLookup, opts templ.DecodeOptions)

func (*Reader) ValidateDicts 

func (r *Reader) ValidateDicts()

ValidateDicts enables dictionary existence checks even when value decoding is disabled. This provides a low-cost "fail fast" validation pass during WAL replay and open.

type RealClock 

type RealClock struct{}

func (RealClock) Now 

func (RealClock) Now() time.Time

type Record 

type Record struct {
	RID   uint64
	Value []byte
}

type Set 

type Set struct {
	Files    map[uint32]*File
	RefCount atomic.Int64
	// contains filtered or unexported fields
}

Set is an immutable snapshot of value-log files for snapshot isolation.

func (*Set) Read 

func (s *Set) Read(ptr page.ValuePtr) ([]byte, error)

func (*Set) ReadAppend 

func (s *Set) ReadAppend(ptr page.ValuePtr, dst []byte) ([]byte, error)

func (*Set) ReadChecksumEnabled added in v0.4.0 

func (s *Set) ReadChecksumEnabled() bool

func (*Set) ReadUnsafe 

func (s *Set) ReadUnsafe(ptr page.ValuePtr) ([]byte, error)

func (*Set) ReadUnsafeAppend 

func (s *Set) ReadUnsafeAppend(ptr page.ValuePtr, dst []byte) ([]byte, error)

func (*Set) ReadUnsafeAppendBatch added in v0.4.0 

func (s *Set) ReadUnsafeAppendBatch(ptrs []page.ValuePtr, dst [][]byte) ([][]byte, error)

ReadUnsafeAppendBatch resolves pointers in order, reusing file lookups for contiguous same-file runs to reduce scan-path overhead.

func (*Set) ReadUnsafeTo added in v0.4.0 

func (s *Set) ReadUnsafeTo(ptr page.ValuePtr, dst []byte) ([]byte, bool, error)

type TemplateLookup added in v0.3.0 

type TemplateLookup func(templateID uint64) ([]byte, error)

type UnsupportedBlockCodecError added in v0.3.0 

type UnsupportedBlockCodecError struct {
	CodecID uint8
}

UnsupportedBlockCodecError reports an unknown per-frame block codec ID.

func (UnsupportedBlockCodecError) Error added in v0.3.0 

func (e UnsupportedBlockCodecError) Error() string

type VirtualClock 

type VirtualClock struct {
	// contains filtered or unexported fields
}

VirtualClock provides deterministic time control for tests.

func NewVirtualClock 

func NewVirtualClock(start time.Time) *VirtualClock

func (*VirtualClock) Advance 

func (c *VirtualClock) Advance(ns int64)

func (*VirtualClock) Now 

func (c *VirtualClock) Now() time.Time

type VirtualSink 

type VirtualSink struct {
	Clock          interface{ Advance(int64) }
	NsPerByte      int64
	SyncPenaltyNs  int64
	FlushPenaltyNs int64
}

VirtualSink advances a clock based on written bytes to simulate IO wall time. Intended for deterministic tests/benchmarks.

func (*VirtualSink) Flush 

func (s *VirtualSink) Flush() error

func (*VirtualSink) Sync 

func (s *VirtualSink) Sync() error

func (*VirtualSink) Write 

func (s *VirtualSink) Write(p []byte) (int, error)

type Writer 

type Writer struct {
	// contains filtered or unexported fields
}

func NewWriter 

func NewWriter(path string, fileID uint32) (*Writer, error)

func NewWriterWithSink 

func NewWriterWithSink(sink io.Writer, fileID uint32) *Writer

NewWriterWithSink creates a value-log writer that writes to the provided sink. Use this for sink-backed/custom outputs that do not provide file-backed durability.

func (*Writer) Append 

func (w *Writer) Append(dictID uint64, dict []byte, rid uint64, value []byte) (page.ValuePtr, error)

func (*Writer) AppendEncodedFrameInto added in v0.3.0 

func (w *Writer) AppendEncodedFrameInto(body []byte, k int, dst []page.ValuePtr) ([]page.ValuePtr, error)

AppendEncodedFrameInto appends a pre-encoded grouped frame body and fills dst with grouped value pointers (dst must be at least k long).

The body must be produced by EncodeFrame/EncodeFrameWithOptions.

func (*Writer) AppendFrame 

func (w *Writer) AppendFrame(dictID uint64, dict []byte, records []Record) ([]page.ValuePtr, error)

func (*Writer) AppendFrameWithStats 

func (w *Writer) AppendFrameWithStats(dictID uint64, dict []byte, records []Record) ([]page.ValuePtr, FrameStats, error)

func (*Writer) AppendFrameWithStatsInto 

func (w *Writer) AppendFrameWithStatsInto(dictID uint64, dict []byte, records []Record, dst []page.ValuePtr) ([]page.ValuePtr, FrameStats, error)

AppendFrameWithStatsInto appends a grouped frame and fills dst with the returned pointers (dst must be at least len(records) long).

This is a performance-oriented helper to avoid allocating a new pointer slice on every frame append.

func (*Writer) AppendOneFrameWithStats added in v0.4.0 

func (w *Writer) AppendOneFrameWithStats(dictID uint64, dict []byte, rid uint64, value []byte) (page.ValuePtr, FrameStats, error)

func (*Writer) AppendRawFramesBufferedInto added in v0.4.0 

func (w *Writer) AppendRawFramesBufferedInto(records []Record, k int, dst []page.ValuePtr) ([]page.ValuePtr, FrameStats, error)

AppendRawFramesBufferedInto appends raw grouped frames through the writer's append buffer, allowing multiple calls to coalesce into larger write syscalls.

Compared to AppendRawFramesWritevInto, this path stages frame bytes in the writer append buffer and flushes later, so independent append calls can merge into fewer, larger writes.

Prefer this for high-throughput append streams where delayed flush is acceptable and syscall coalescing across calls is desirable.

dst must be at least len(records) long.

func (*Writer) AppendRawFramesWritevInto 

func (w *Writer) AppendRawFramesWritevInto(records []Record, k int, dst []page.ValuePtr) ([]page.ValuePtr, FrameStats, error)

AppendRawFramesWritevInto appends raw (uncompressed) grouped frames using a writev batching strategy.

This avoids concatenating payloads into a contiguous frame buffer, reducing user-space copying for large value workloads. It always flushes the writev queue before returning, so it does not retain references to the input slices.

Prefer this when the caller needs pointers durably visible at function return and cannot rely on future appends to coalesce writes.

dst must be at least len(records) long.

func (*Writer) AppendRawRecord added in v0.3.0 

func (w *Writer) AppendRawRecord(raw []byte, length uint32) (page.ValuePtr, error)

AppendRawRecord appends a raw value-log record (CRC + header + payload) without re-encoding. The length argument should include any pointer flags.

func (*Writer) Close 

func (w *Writer) Close() error

func (*Writer) FileID 

func (w *Writer) FileID() uint32

func (*Writer) Flush 

func (w *Writer) Flush() error

func (*Writer) PendingBytes added in v0.4.0 

func (w *Writer) PendingBytes() int

PendingBytes reports bytes accepted by the writer but not yet flushed to the underlying file descriptor. For file-backed writers this is the append-buffer tail that same-process readers cannot see through ReadAt until flushed.

func (*Writer) RawWriteStats added in v0.4.0 

func (w *Writer) RawWriteStats() RawWriteStats

RawWriteStats returns cumulative plain write(2) counters for this writer.

func (*Writer) RawWritevStats added in v0.4.0 

func (w *Writer) RawWritevStats() RawWritevStats

RawWritevStats returns cumulative writev syscall counters for this writer.

func (*Writer) ResetCompressionHints 

func (w *Writer) ResetCompressionHints()

ResetCompressionHints clears skip/backoff state for deterministic benches.

func (*Writer) RotateTo 

func (w *Writer) RotateTo(path string, fileID uint32) error

func (*Writer) SetBlockCompression added in v0.3.0 

func (w *Writer) SetBlockCompression(codec BlockCodec, enabled bool)

SetBlockCompression configures grouped frame block compression for dictID=0 append paths.

func (*Writer) SetClock 

func (w *Writer) SetClock(clock Clock)

func (*Writer) SetDictFrameEncoderOptions added in v0.3.0 

func (w *Writer) SetDictFrameEncoderOptions(level zstd.EncoderLevel, enableEntropy bool)

func (*Writer) SetEncodeCostModel 

func (w *Writer) SetEncodeCostModel(model EncodeCostModel)

func (*Writer) SetEncodeSampleStride 

func (w *Writer) SetEncodeSampleStride(stride uint64)

func (*Writer) SetKeepPolicy 

func (w *Writer) SetKeepPolicy(ioNsPerStoredByte, encodeNsPerRawByte, safetyMargin float64)

func (*Writer) SetRawWritevStrategy added in v0.3.0 

func (w *Writer) SetRawWritevStrategy(minAvgBytes, minBatchRecs int)

SetRawWritevStrategy configures grouped raw writev usage heuristics.

minAvgBytes:

  • 0 enables adaptive mode (no average-bytes floor).
  • >0 requires average payload bytes/record to meet this floor.

minBatchRecs:

  • <=0 uses the default.

func (*Writer) Size 

func (w *Writer) Size() int64

func (*Writer) Sync 

func (w *Writer) Sync() error

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.