Documentation
¶
Overview ¶
Package disk implements a disk-based cache for file data.
Index ¶
- Constants
- type DiskCache
- func (dc *DiskCache) Clear() error
- func (dc *DiskCache) Commit(path string, meta cache.EntryMetadata) error
- func (dc *DiskCache) Delete(path string) bool
- func (dc *DiskCache) DeletePartial(path string) bool
- func (dc *DiskCache) FlushRanges(path string, meta cache.EntryMetadata) error
- func (dc *DiskCache) Invalidate() (*InvalidatedEntries, error)
- func (dc *DiskCache) InvalidateRanges(path string) error
- func (dc *DiskCache) MissingRanges(path string, meta cache.EntryMetadata, requested cache.ByteRange) ([]cache.ByteRange, error)
- func (dc *DiskCache) Pin(path string)
- func (dc *DiskCache) PinPartial(path string)
- func (dc *DiskCache) RangeEntryComplete(path string, meta cache.EntryMetadata) (bool, error)
- func (dc *DiskCache) RangeMetadata(path string, meta cache.EntryMetadata) (cache.EntryMetadata, bool, error)
- func (dc *DiskCache) Read(path string, buff []byte, ofst int64) (n int, err error)
- func (dc *DiskCache) ReadComplete(path string, meta cache.EntryMetadata, buff []byte, ofst int64) (n int, err error)
- func (dc *DiskCache) ReadPartial(path string, buff []byte, ofst int64) (n int, err error)
- func (dc *DiskCache) ReadRange(path string, meta cache.EntryMetadata, buff []byte, ofst int64) (int, error)
- func (dc *DiskCache) SizeBytes() int64
- func (dc *DiskCache) StartMaintenance()
- func (dc *DiskCache) Stats() *cache.Stats
- func (dc *DiskCache) StopMaintenance()
- func (dc *DiskCache) Unpin(path string)
- func (dc *DiskCache) UnpinPartial(path string)
- func (dc *DiskCache) Write(path string, buff []byte, ofst int64) (n int, err error)
- func (dc *DiskCache) WriteCompleteRange(path string, meta cache.EntryMetadata, buff []byte, ofst int64) (int, error)
- func (dc *DiskCache) WritePartial(path string, buff []byte, ofst int64) (n int, err error)
- func (dc *DiskCache) WriteRange(path string, meta cache.EntryMetadata, buff []byte, ofst int64) (int, error)
- type InvalidatedEntries
- type Option
- func WithCapacityBytes(n int64) Option
- func WithDisabled(disabled bool) Option
- func WithLogger(logger log.Logger) Option
- func WithLruFlushInterval(d time.Duration) Option
- func WithMaintenanceInterval(d time.Duration) Option
- func WithMaxAge(d time.Duration) Option
- func WithMaxFileCount(n int64) Option
- func WithRangeFlushBytes(n int64) Option
Constants ¶
const ( // DefaultCacheCapacity is the maximum size of the local cache in bytes if not provided // at cache initialization. 0 means unbounded. Equivalent to 2GiB. DefaultCapacity = 2 * (1 << 30) // (1 << 30) == 1GiB // DefaultMaxAge is the maximum age of cache files, if not provided // at cache initialization. 0 means no expiration. Equivalent to 7 days. DefaultMaxAge = 7 * 24 * time.Hour // DefaultMaxFileCount is the maximum number of files in the cache if not provided // at cache initialization. 0 means unbounded. DefaultMaxFileCount = 10_000 // DefaultMaintenanceInterval is the interval at which cache maintenance tasks are performed, // if not provided at cache initialization. DefaultMaintenanceInterval = 5 * time.Minute // DefaultLruFlushInterval is the interval at which the LRU state is persisted to disk, // if not provided at cache initialization. DefaultLruFlushInterval = 1 * time.Minute // DefaultRangeFlushBytes is how many newly written bytes an entry may hold // before they are synced and recorded as durable. Downloads also flush when // they stop, so this only bounds the progress a crash can lose. DefaultRangeFlushBytes = 256 * (1 << 20) )
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type DiskCache ¶
type DiskCache struct {
// CacheRoot is the root directory for the cache.
CacheRoot string
// Capacity is the maximum size of the cache in bytes. 0 means unbounded.
Capacity int64
// Disabled indicates whether the cache is disabled.
//
// A disabled cache:
// - performs no I/O and returns no data.
// - allows the caller to operate without caching, while not requiring conditional code around
// all cache operations.
Disabled bool
// MaxAge is the maximum age of cache files. 0 means no expiration.
MaxAge time.Duration
// MaxFileCount is the maximum number of files in the cache. 0 means unbounded.
MaxFileCount int64
// MaintenanceInterval is the interval at which cache maintenance tasks are performed.
//
// Maintenance tasks include:
// - Stats are reloaded from disk
// - Old files are deleted based on MaxAge
// - Unpinned data files without completed metadata are deleted
MaintenanceInterval time.Duration
// LruFlushInterval is the interval at which the LRU state is persisted to disk.
LruFlushInterval time.Duration
// RangeFlushBytes is how many newly written bytes an entry may hold before
// they are synced and recorded as durable.
RangeFlushBytes int64
// contains filtered or unexported fields
}
DiskCache implements a simple disk-based cache for file data with an interface that roughly matches the FUSE Read/Write methods.
TODO:
- allow concurrent reads/writes to different files in the cache by moving lock coordination out of the caller and lock by path within the cache with a RWMutex map. This will require maintaining a reservation of bytes so that concurrent goroutines can't exhaust memory that a different goroutine evicted for its Write operation.
- consider limiting the max size of individual cache files so that a single large file cannot evict the entire cache
- allow callers to distiguish between "fatal" vs "non fatal" errors with custom error types e.g. if a read fails, the caller can just read from the source instead of treating it as a fatal error but a write failure because the cache is at capacity and could not make space is more serious
- move Stats#MarshalJSON to a separate file to and apply the filescomfs_debug tag
func NewDiskCache ¶
NewDiskCache creates a DiskCache rooted at path and applies any options. The provided path must be an absolute path to a directory that already exists and is writable by the current process.
If not disabled, it ensures the directory exists and initializes stats by scanning it.
The cache root itself belongs to the caller and is left as it is. Everything the cache stores below it (file data, partial data, metadata and LRU state) is readable only by the current user: the data, partial and state directories the cache owns are created private, and an existing cache is made private before any of its state is read, so a cache written by an earlier version that created world-readable entries is protected as well. See lib/privatefile for what private means on each platform. A root that another user could replace entries in is refused.
Defaults:
- Disabled: false
- Capacity: DefaultCapacity
- MaxAge: DefaultMaxAge
- MaxFileCount: DefaultMaxFileCount
func (*DiskCache) Clear ¶ added in v3.3.218
Clear removes all unpinned file data from the cache. Pinned entries are invalidated immediately and removed when their final open handle closes.
func (*DiskCache) Commit ¶ added in v3.3.109
func (dc *DiskCache) Commit(path string, meta cache.EntryMetadata) error
func (*DiskCache) Delete ¶
Delete removes the cached file from the cache. It returns true if the file was deleted.
func (*DiskCache) DeletePartial ¶ added in v3.3.109
func (*DiskCache) FlushRanges ¶ added in v3.3.285
func (dc *DiskCache) FlushRanges(path string, meta cache.EntryMetadata) error
FlushRanges makes all currently readable bytes durable at request completion.
func (*DiskCache) Invalidate ¶ added in v3.3.285
func (dc *DiskCache) Invalidate() (*InvalidatedEntries, error)
Invalidate makes every cached entry unreadable in one short exclusive step: download progress is forgotten and the metadata directory is replaced with an empty one. Reads and writes can continue right away, and Remove then deletes the invalidated data without blocking them.
func (*DiskCache) InvalidateRanges ¶ added in v3.3.285
InvalidateRanges waits for readers before removing metadata. Windows can reject deletion while another reader has the metadata file open.
func (*DiskCache) MissingRanges ¶ added in v3.3.285
func (dc *DiskCache) MissingRanges(path string, meta cache.EntryMetadata, requested cache.ByteRange) ([]cache.ByteRange, error)
MissingRanges returns the requested ranges not present for the remote file version.
func (*DiskCache) Pin ¶ added in v3.2.264
Pin increments the reference count for a file, preventing it from being evicted. This should be called when a file handle is opened.
func (*DiskCache) PinPartial ¶ added in v3.3.109
func (*DiskCache) RangeEntryComplete ¶ added in v3.3.285
RangeEntryComplete reports whether the cached ranges cover the remote file.
func (*DiskCache) RangeMetadata ¶ added in v3.3.285
func (dc *DiskCache) RangeMetadata(path string, meta cache.EntryMetadata) (cache.EntryMetadata, bool, error)
RangeMetadata returns the validator and coverage belonging to a listing identity.
func (*DiskCache) Read ¶
Read reads data from the cached file at the given path into buff starting at the provided offset.
It returns the number of bytes read, or 0 if the file is not in the cache.
func (*DiskCache) ReadComplete ¶ added in v3.3.109
func (*DiskCache) ReadPartial ¶ added in v3.3.109
func (*DiskCache) ReadRange ¶ added in v3.3.285
func (dc *DiskCache) ReadRange(path string, meta cache.EntryMetadata, buff []byte, ofst int64) (int, error)
ReadRange reads only when every requested byte is valid for the remote file version.
func (*DiskCache) SizeBytes ¶ added in v3.3.218
SizeBytes returns the cached byte count, once the startup scan has accounted the entries already on disk.
func (*DiskCache) StartMaintenance ¶
func (dc *DiskCache) StartMaintenance()
StartMaintenance starts the maintenance goroutine if it is not already running.
func (*DiskCache) Stats ¶
Stats returns the current cache statistics, once the startup scan has accounted the entries already on disk.
func (*DiskCache) StopMaintenance ¶
func (dc *DiskCache) StopMaintenance()
StopMaintenance stops the maintenance goroutine if it is running, and closes data files kept open for downloads. Files reopen if the cache is used again.
func (*DiskCache) Unpin ¶ added in v3.2.264
Unpin decrements the reference count for a file. When the count reaches zero, the file becomes eligible for eviction. This should be called when a file handle is closed.
func (*DiskCache) UnpinPartial ¶ added in v3.3.109
func (*DiskCache) Write ¶
Write writes data from buff to the cached file at the given path starting at offset ofst. Writing at an offset past the end of the file will grow the file and fill the gap with zeros.
It returns the number of bytes written, or 0 if the cache is not enabled.
func (*DiskCache) WriteCompleteRange ¶ added in v3.3.285
func (dc *DiskCache) WriteCompleteRange(path string, meta cache.EntryMetadata, buff []byte, ofst int64) (int, error)
WriteCompleteRange shares the publication policy without requiring sparse file support, since a complete response fills every byte through EOF.
func (*DiskCache) WritePartial ¶ added in v3.3.109
func (*DiskCache) WriteRange ¶ added in v3.3.285
func (dc *DiskCache) WriteRange(path string, meta cache.EntryMetadata, buff []byte, ofst int64) (int, error)
WriteRange makes received bytes readable immediately. Durable coverage is published every RangeFlushBytes of new data; FlushRanges publishes the rest when a download stops.
type InvalidatedEntries ¶ added in v3.3.285
type InvalidatedEntries struct {
// contains filtered or unexported fields
}
InvalidatedEntries is the data Invalidate made unreadable, which Remove deletes.
func (*InvalidatedEntries) Remove ¶ added in v3.3.285
func (invalidated *InvalidatedEntries) Remove() error
Remove deletes the data files Invalidate made unreadable. A file with an open handle is deleted when its last handle closes, and a file written since the invalidation is kept.
type Option ¶
type Option func(*DiskCache)
Option configures a DiskCache.
func WithCapacityBytes ¶
WithCapacityBytes sets the maximum total size (in bytes) for the cache.
func WithDisabled ¶
WithDisabled disables the cache (no I/O, no directory creation).
func WithLogger ¶
WithLogger sets the logger for the cache.
func WithLruFlushInterval ¶
WithLruFlushInterval sets the interval for cache maintenance operations.
func WithMaintenanceInterval ¶
WithMaintenanceInterval sets the interval for cache maintenance operations.
func WithMaxAge ¶
WithMaxAge sets the maximum age for cache files.
func WithMaxFileCount ¶
WithMaxFileCount sets the maximum number of files for the cache.
func WithRangeFlushBytes ¶ added in v3.3.285
WithRangeFlushBytes sets how many newly written bytes an entry may hold before they are synced and recorded as durable.
Source Files