Help us shape the future of Go! Take the 2026 Go Developer Survey

store

package
v0.57.21 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 2, 2026 License: Apache-2.0 Imports: 27 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var ErrPackReaderClosed = errors.New("packfile store: pack reader closed")

ErrPackReaderClosed is returned by a read on a pack reader that was closed, because the store shut down or a manifest change removed its pack.

View Source 
var ErrPackfileStoreClosed = errors.New("packfile store: closed")

ErrPackfileStoreClosed is returned when a read opens a pack after shutdown.

Functions

This section is empty.

Types

type IndexCache 

type IndexCache interface {
	// Get returns cached raw index-tail bytes for a packfile.
	Get(ctx context.Context, packID string) ([]byte, bool, error)
	// Set stores raw index-tail bytes for a packfile.
	Set(ctx context.Context, packID string, data []byte) error
}

IndexCache stores raw kvfile index-tail bytes per packfile.

Implementations are expected to be durable (kvtx-backed) in production and ephemeral in tests. The engine parses and validates cached tail bytes into runtime-only index views before serving block data.

type ManifestSnapshot 

type ManifestSnapshot struct {
	// contains filtered or unexported fields
}

ManifestSnapshot pins catalog membership without copying retained entries. It borrows the store's lifetime and its shared reader/index caches.

func (*ManifestSnapshot) GetBlockExistsBatch 

func (v *ManifestSnapshot) GetBlockExistsBatch(ctx context.Context, refs []*block.BlockRef) ([]bool, error)

GetBlockExistsBatch probes only packs present when the snapshot was captured. Paging a flush therefore never fetches indexes of packs it just uploaded.

func (*ManifestSnapshot) GetEntries 

func (v *ManifestSnapshot) GetEntries() []*packfile.PackfileEntry

GetEntries materializes an explicitly requested complete catalog snapshot.

type Opener 

type Opener func(packID string, size int64) (*PackReader, error)

Opener returns a per-pack access engine for a remote packfile of the given size.

The size is taken from the manifest entry so the opener does not need to issue a separate metadata request. Implementations typically wrap a Transport via NewPackReader or NewHTTPRangeReader.

type PackBlock 

type PackBlock struct {
	// Hash is the block hash the pack stores the block under.
	Hash *hash.Hash
	// Block is the block data with its refs.
	Block *block.StoredBlock
}

PackBlock is one verified block read from a pack.

type PackReader 

type PackReader struct {
	// contains filtered or unexported fields
}

PackReader is the per-pack access engine.

The engine composes two projections over one shared byte substrate:

  • Span Store: resident raw packfile bytes, LRU eviction, uncovered-gap planning for transport fetches.
  • Block Index: the kvfile index entries of the pack, and the set of blocks handed to the writeback target.

Reads verify the requested block inline. Each fetched span hands every block it completes to one batched writeback job, so a block is published at most once.

The engine is addressable as an io.ReaderAt (backed by the span store) and exposes higher-level GetBlock operations keyed by kvfile index entries.

func NewHTTPRangeReader 

func NewHTTPRangeReader(
	cli *http.Client,
	url string,
	size int64,
	readAheadSize int,
	signReq func(*http.Request) error,
	observeResp func(*http.Response),
) *PackReader

NewHTTPRangeReader builds a per-pack engine backed by HTTP range requests.

readAheadSize sets the engine's minimum transport window (and alignment quantum); zero accepts the default. signReq optionally mutates the outgoing request (for signed CDN access).

func NewPackReader 

func NewPackReader(packID string, size int64, transport Transport) *PackReader

NewPackReader builds a per-pack access engine wrapping a transport.

func (*PackReader) Close 

func (e *PackReader) Close()

Close cancels and drains every job admitted by the engine.

func (*PackReader) ReaderAt 

func (e *PackReader) ReaderAt(ctx context.Context) io.ReaderAt

ReaderAt returns an io.ReaderAt bound to ctx.

Reads issued through the returned ReaderAt fall through the span store: resident bytes are served from memory, and misses trigger transport fetches via the uncovered-window planner.

func (*PackReader) SetExpectedBlockCount 

func (e *PackReader) SetExpectedBlockCount(blockCount uint64)

SetExpectedBlockCount configures the manifest block count used to validate index tails.

func (*PackReader) SetIndexCache 

func (e *PackReader) SetIndexCache(cache IndexCache)

SetIndexCache configures persistent storage for raw kvfile index-tail bytes.

func (*PackReader) SetStatsChangedCallback 

func (e *PackReader) SetStatsChangedCallback(fn func())

SetStatsChangedCallback sets a callback invoked after observable stats change.

func (*PackReader) SetWriteback 

func (e *PackReader) SetWriteback(ctx context.Context, target block.StoreOps, windowBytes int64)

SetWriteback configures co-block publication to a target store.

ctx scopes background writeback work. target receives verified block copies when non-nil. windowBytes is the neighborhood fetched around a miss so its co-blocks are published with it. Pass 0 to use the default.

func (*PackReader) SnapshotStats 

func (e *PackReader) SnapshotStats() PackReaderStats

SnapshotStats returns a snapshot of engine state and counters.

type PackReaderStats 

type PackReaderStats struct {
	ResidentBytes             int64
	SpanCount                 int
	InFlightFetches           int
	FetchCount                uint64
	FetchedBytes              int64
	RangeRequestCount         uint64
	RangeResponseBytes        int64
	IndexTailFetchCount       uint64
	IndexTailFetchBytes       int64
	IndexTailResponseBytes    int64
	FullResponseFallbackCount uint64
	FullResponseFallbackBytes int64
	LastFullResponseFallback  int64
	LastFetchAt               time.Time
	LastFetchBytes            int
	CurrentWindow             int
	PublishedBlocks           int
	WritebackRunning          int
	VerifyFailures            uint64
	WritebackCount            uint64
	WritebackErrors           uint64
	IndexLoaded               bool
	IndexCacheHits            uint64
	IndexCacheMisses          uint64
	IndexCacheReadErrors      uint64
	IndexCacheWriteErrors     uint64
	RemoteIndexLoads          uint64
	RemoteIndexBytes          int64
	LastRemoteIndexBytes      int64
}

PackReaderStats describes the current engine state and counters.

type PackfileStore 

type PackfileStore struct {
	// contains filtered or unexported fields
}

PackfileStore is a read-only block.StoreOps over a set of remote packfiles.

The store fans reads out to per-pack engines: it handles manifest-wide concerns (bloom lookup, engine registry, writeback and index cache configuration, the shared resident budget) while the engines own per-pack spans, indexes, and writeback.

func NewPackfileStore 

func NewPackfileStore(opener Opener, cache IndexCache) *PackfileStore

NewPackfileStore creates a new packfile store.

func (*PackfileStore) ApplyManifestDelta 

func (s *PackfileStore) ApplyManifestDelta(entries []*packfile.PackfileEntry, removed []string)

ApplyManifestDelta publishes accepted entries and removes only named packs.

func (*PackfileStore) BeginReadOperation 

func (s *PackfileStore) BeginReadOperation(context.Context) (block.StoreOps, func(), error)

BeginReadOperation returns the packfile store as the scoped read handle.

func (*PackfileStore) Close 

func (s *PackfileStore) Close()

Close cancels transport work and releases every open pack reader.

func (*PackfileStore) GetBlock 

func (s *PackfileStore) GetBlock(ctx context.Context, ref *block.BlockRef) ([]byte, bool, error)

GetBlock gets a block by reference from the packfile store.

func (*PackfileStore) GetBlockExists 

func (s *PackfileStore) GetBlockExists(ctx context.Context, ref *block.BlockRef) (bool, error)

GetBlockExists reports whether a block exists in the store.

func (*PackfileStore) GetBlockExistsBatch 

func (s *PackfileStore) GetBlockExistsBatch(ctx context.Context, refs []*block.BlockRef) ([]bool, error)

GetBlockExistsBatch checks whether each block exists. A manifest change that closes a probed reader restarts the batch on the new catalog.

func (*PackfileStore) GetHashType 

func (s *PackfileStore) GetHashType() hash.HashType

GetHashType returns the hash type for the store.

func (*PackfileStore) GetStoredBlock 

func (s *PackfileStore) GetStoredBlock(ctx context.Context, ref *block.BlockRef) (*block.StoredBlock, error)

GetStoredBlock gets a block with its refs from the packfile store.

Each pack whose bloom filter may hold the block is consulted in manifest order. The first pack that finds the block returns it. Returns nil when no pack holds the block.

func (*PackfileStore) GetSupportedFeatures 

func (s *PackfileStore) GetSupportedFeatures() block.StoreFeature

GetSupportedFeatures returns the native feature bitset.

func (*PackfileStore) PutBlock 

func (s *PackfileStore) PutBlock(_ context.Context, _ []byte, _ *block.PutOpts) (*block.BlockRef, bool, error)

PutBlock is not supported on a read-only store.

func (*PackfileStore) PutBlockBatch 

func (s *PackfileStore) PutBlockBatch(_ context.Context, entries []*block.PutBatchEntry) error

PutBlockBatch is not supported on a read-only store.

func (*PackfileStore) ReadPackBlocks 

func (s *PackfileStore) ReadPackBlocks(ctx context.Context, packID string, size int64) ([]PackBlock, error)

ReadPackBlocks fetches a whole pack with one transport read and returns its blocks in the order the pack stores their values. Every value is checked against its key hash, as a served read is. The read uses a fresh reader instead of the pack's engine, so it neither evicts nor fills the resident spans that serve block reads.

func (*PackfileStore) RmBlock 

func (s *PackfileStore) RmBlock(_ context.Context, _ *block.BlockRef) error

RmBlock is not supported on a read-only store.

func (*PackfileStore) SetRangeCacheMaxBytes 

func (s *PackfileStore) SetRangeCacheMaxBytes(maxBytes int64)

SetRangeCacheMaxBytes sets the resident-byte budget shared by every engine.

func (*PackfileStore) SetStatsChangedCallback 

func (s *PackfileStore) SetStatsChangedCallback(fn func())

SetStatsChangedCallback sets a callback invoked after observable stats change.

func (*PackfileStore) SetWriteback 

func (s *PackfileStore) SetWriteback(ctx context.Context, target block.StoreOps, windowBytes int64)

SetWriteback enables co-block persistence to a target store.

When a block is fetched from a remote packfile the engine also fetches every other block that fully fits within windowBytes of the target, and writes each fetched block to target in an asynchronous batch after verifying it. ctx scopes the background work. Pass nil target to disable persistence.

func (*PackfileStore) SnapshotEngineStats 

func (s *PackfileStore) SnapshotEngineStats() map[string]PackReaderStats

SnapshotEngineStats returns per-engine stats keyed by manifest pack id.

func (*PackfileStore) SnapshotManifest 

func (s *PackfileStore) SnapshotManifest() *ManifestSnapshot

SnapshotManifest captures the current catalog in constant space and time. The result is safe for concurrent reads until the store is closed.

func (*PackfileStore) SnapshotStats 

func (s *PackfileStore) SnapshotStats() PackfileStoreStats

SnapshotStats returns aggregate store state across all open engines.

func (*PackfileStore) StatBlock 

func (s *PackfileStore) StatBlock(ctx context.Context, ref *block.BlockRef) (*block.BlockStat, error)

StatBlock returns metadata about a block without reading its data. Returns nil, nil if the block does not exist.

func (*PackfileStore) Sync 

func (s *PackfileStore) Sync(_ context.Context) (bool, error)

Sync reports always-durable: the read-only packfile store holds no buffered writes.

func (*PackfileStore) UpdateManifest 

func (s *PackfileStore) UpdateManifest(entries []*packfile.PackfileEntry)

UpdateManifest replaces a complete catalog, for startup and remote snapshots. Incremental changes use ApplyManifestDelta instead.

type PackfileStoreStats 

type PackfileStoreStats struct {
	EngineCount               int
	ResidentBytes             int64
	SpanCount                 int
	InFlightFetches           int
	FetchCount                uint64
	FetchedBytes              int64
	RangeRequestCount         uint64
	RangeResponseBytes        int64
	IndexTailFetchCount       uint64
	IndexTailFetchBytes       int64
	IndexTailResponseBytes    int64
	FullResponseFallbackCount uint64
	FullResponseFallbackBytes int64
	LastFullResponseFallback  int64
	LastFetchAt               time.Time
	LastFetchBytes            int
	PublishedBlocks           int
	WritebackRunning          int
	VerifyFailures            uint64
	WritebackCount            uint64
	WritebackErrors           uint64
	ManifestEntries           int
	PackBlockCountTotal       uint64
	PackBlockCountMin         uint64
	PackBlockCountMax         uint64
	PackSizeBytesTotal        uint64
	PackSizeBytesMin          uint64
	PackSizeBytesMax          uint64
	BloomFilterCount          int
	BloomMissingCount         int
	BloomInvalidCount         int
	BloomMaxFalsePositiveRate float64
	BloomRiskPackCount        int
	WritebackWindow           int64
	ResidentByteBudget        int64
	LookupCount               uint64
	CandidatePacks            uint64
	OpenedPacks               uint64
	NegativePacks             uint64
	TargetHits                uint64
	LastCandidatePacks        int
	LastOpenedPacks           int
	LastNegativePacks         int
	LastTargetHit             bool
	IndexCacheHits            uint64
	IndexCacheMisses          uint64
	IndexCacheReadErrors      uint64
	IndexCacheWriteErrors     uint64
	RemoteIndexLoads          uint64
	RemoteIndexBytes          int64
	LastRemoteIndexBytes      int64
}

PackfileStoreStats describes aggregate store state across all open engines.

type Transport 

type Transport interface {
	// Fetch returns bytes [off, off+length) from the pack in a single call.
	// The returned slice has len <= length; a short read signals end-of-pack.
	Fetch(ctx context.Context, off int64, length int) ([]byte, error)
}

Transport fetches raw byte ranges from a remote packfile.

Implementations are the lowest layer in the pack access pipeline: they do not cache, deduplicate, or verify anything. The engine wraps a transport in a span store (resident bytes), block catalog (block-level publication state), and publication queue (verify + writeback).

type TransportStats 

type TransportStats struct {
	FullResponseFallbackCount uint64
	FullResponseFallbackBytes int64
	LastFullResponseFallback  int64
}

TransportStats describes transport-specific fetch counters.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.