Documentation
¶
Index ¶
- Variables
- type Config
- type File
- func (f *File) Cancel() error
- func (f *File) Close() error
- func (f *File) Commit() error
- func (f *File) Off() int64
- func (f *File) Read(p []byte) (n int, err error)
- func (f *File) ReadAt(p []byte, off int64) (n int, err error)
- func (f *File) Seek(off int64, whence int) (int64, error)
- func (f *File) Size() int64
- func (f *File) Write(p []byte) (n int, err error)
- func (f *File) WriteAt(p []byte, off int64) (n int, err error)
- type Store
- func (s *Store) BanEviction(key string) error
- func (s *Store) Create(key string, sizeBytes uint64) (*File, error)
- func (s *Store) Delete(key string) error
- func (s *Store) DeleteMetadata(key string, mdSuffix string) error
- func (s *Store) GetMetadata(key string, md metadata.Metadata) (ok bool, err error)
- func (s *Store) Has(key string) (inStore bool, inScope bool)
- func (s *Store) List() []string
- func (s *Store) ListMetadata(key string) ([]metadata.Metadata, error)
- func (s *Store) MarkComplete(key string) error
- func (s *Store) Open(key string) (*File, error)
- func (s *Store) ScopeComplete() *Store
- func (s *Store) ScopeIncomplete() *Store
- func (s *Store) Scoped(scope storelib.BlobScope) *Store
- func (s *Store) SetMetadata(key string, md metadata.Metadata) error
- func (s *Store) Stat(key string) (size int64, err error)
- func (s *Store) UnbanEviction(key string) error
Constants ¶
This section is empty.
Variables ¶
var ErrEvicted error = errors.New("the blob has been evicted or deleted from the store") // TODO - rename to ErrGone
ErrEvicted is returned when a user tries operating on a blob's *File after the blob has been evicted or deleted.
var ErrNoSpace error = errors.New("cannot free enough memory for new entry")
ErrNoSpace means the store could not free enough space for a new entry.
Functions ¶
This section is empty.
Types ¶
type Config ¶
type Config struct {
// GOMEMLIMITBytes and GOGC tune the Go GC. In short, it is advised to set GOMEMLIMIT to 90-95% of the container's reserved memory
// and to turn GOGC off, assuming the container has reserved memory. More info on tuning the Go GC: https://go.dev/doc/gc-guide.
//
// If set, they overwrite any previous configurations for GOMEMLIMIT and GOGC.
// GOMEMLIMIT *must* be configured either through [Config] or another way (e.g. an env var).
// GOGC defaults to the Go default (100), if not set and is turned off if a negative value is provided.
GOMEMLIMITBytes int64 `yaml:"gomemlimit_bytes"`
// Check the comment at [Config.GOMEMLIMITBytes].
GOGC int `yaml:"gogc"`
// CapacityBytes sets a *hard* limit on the memory [Store] can keep reachable (i.e. unreclaimable by the GC) to store blobs.
// Any other memory that [Store] uses (e.g. storing metadata) is NOT capped by CapacityBytes.
CapacityBytes uint64 `yaml:"capacity_bytes"`
}
Config configures Store.
type File ¶
type File struct {
// contains filtered or unexported fields
}
File represents an open handle to a blob in Store, similar to how an os.File is an open file descriptor to a file on disk. As soon as the blob is evicted, File's APIs starts returning ErrEvicted, as File no longer has a reference to its data, ensuring GC can clean it.
func (*File) Off ¶
Off returns the offset for the next Read or Write operation. Thread-safe with other File APIs ONLY after File's blob is evicted.
func (*File) ReadAt ¶
ReadAt implements io.ReaderAt. Thread-safe.
func (*File) Size ¶
Stat returns the blob's actual size, even if it differs from the size reported during Create. A return value of -1 represents ErrEvicted.
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store is an in-memory, thread-safe, LRU cache for blobs and their metadata.Metadata.
Supports pagination of blobs during reading/writing, such that blobs don't need to be fully loaded into memory.
New blobs are considered 'incomplete', which unlists them from LRU eviction. The store can be scoped to work on only (in-)complete blobs.
The store prioritizes writing new blobs over reading existing ones. Therefore, blobs may get evicted while clients hold a *File to them. In such cases, ErrEvicted is returned.
All APIs are thread-safe. Parallel access to a single blob is allowed but clients must ensure they don't intervene with one another.
Supports (un-)marking blobs as non-evictable (needed when we want to ensure an entry does not get evicted before the client flushes it to disk).
func (*Store) BanEviction ¶
BanEviction marks a blob as unevictable by LRU eviction. It is idempotent. Usually used by clients to ensure a blob is not evicted before being flushed to disk.
func (*Store) Create ¶
Create initializes a new, incomplete blob, reserves space for it, and returns a *File pointing to it. Incomplete entries cannot be automatically evicted. MarkComplete must be called once the blob is complete. The store uses `sizeBytes` for its eviction logic even if the blob's real size differs (which is tolerated).
func (*Store) Delete ¶
Delete removes a blob and its metadata from the store. Returns os.ErrNotExist on missing entry.
func (*Store) DeleteMetadata ¶
DeleteMetadata removes a blob's metadata. No error returned if the metadata is not present.
func (*Store) GetMetadata ¶
GetMetadata populates `md` if the metadata is present. Returns os.ErrNotExist if key is not in store.
func (*Store) ListMetadata ¶
ListMetadata returns all metadata.Metadata of key.
func (*Store) MarkComplete ¶
MarkComplete marks the blob as fully written, which enlists it for LRU eviction (unless BanEviction has been called). It is idempotent. Additionally, other store APIs may filter blobs based on completeness.
func (*Store) Open ¶
Open returns a *File pointing to the blob. *File APIs returns ErrEvicted once the blob gets evicted.
func (*Store) ScopeComplete ¶
ScopeComplete scopes Store's APIs such that they can only operate on complete blobs. storelib.ErrOutOfScope is returned if the user tries to operate on an incomplete blob.
func (*Store) ScopeIncomplete ¶
ScopeIncomplete scopes Store's APIs such that they can only operate on incomplete blobs. storelib.ErrOutOfScope is returned if the user tries to operate on a complete blob.
func (*Store) Scoped ¶
Scoped scopes Store's APIs, such that storelib.ErrOutOfScope is returned upon attempting to operate on blobs out of scope.
func (*Store) SetMetadata ¶
SetMetadata sets the respective metadata of the blob.
func (*Store) Stat ¶
Stat returns the blob's actual size, even if it differs from the size reported during Create.
func (*Store) UnbanEviction ¶
UnbanEviction removes the effect of BanEviction for a blob. It is idempotent.
Source Files