Documentation
¶
Index ¶
- type Config
- type File
- func (f *File) Cancel() error
- func (f *File) Close() error
- func (f *File) Commit() error
- func (f *File) Read(p []byte) (n int, err error)
- func (f *File) ReadAt(p []byte, off int64) (n int, err error)
- func (f *File) Seek(off int64, whence int) (int64, error)
- func (f *File) Size() int64
- func (f *File) Write(p []byte) (n int, err error)
- func (f *File) WriteAt(p []byte, off int64) (n int, err error)
- type Store
- func (s *Store) BanEviction(key string) error
- func (s *Store) Clean(targetUtilPercent int, respectEvictionBan bool) (newUtil int, err error)
- func (s *Store) Create(key string, sizeBytes uint64) (*File, error)
- func (s *Store) Delete(key string) error
- func (s *Store) DeleteMetadata(key string, mdSuffix string) error
- func (s *Store) GetMetadata(key string, md metadata.Metadata) (ok bool, err error)
- func (s *Store) Has(key string) (inStore bool, inScope bool)
- func (s *Store) List() []string
- func (s *Store) ListMetadata(key string) ([]metadata.Metadata, error)
- func (s *Store) MarkComplete(key string) error
- func (s *Store) Open(key string) (*File, error)
- func (s *Store) ScopeComplete() *Store
- func (s *Store) ScopeIncomplete() *Store
- func (s *Store) Scoped(scope storelib.BlobScope) *Store
- func (s *Store) SetMetadata(key string, md metadata.Metadata) error
- func (s *Store) Stat(key string) (os.FileInfo, error)
- func (s *Store) UnbanEviction(key string) error
- func (s *Store) WriteAtMetadata(key string, md metadata.Metadata, p []byte, off int64) error
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Config ¶
type Config struct {
// The capacity of the store in bytes. When breached, LRU eviction is used.
CapacityBytes uint64 `yaml:"capacity_bytes"`
// The root directory under which [Store]'s blobs and state are stored.
RootDir string `yaml:"root_dir"`
// Whether after crash/restart, the Store removes incomplete files from disk (usually to prevent leaks) OR
// reboots any incomplete files from disk (allowing users to continue the blob download/upload, where it was left off before the crash).
RebootIncompleteBlobs bool `yaml:"reboot_incomplete_blobs"`
// If > 0, directory sharding is used to speed up performance, where ShardLength denotes
// 1) the length of each directory shard's name and 2) the number of shards.
// A value of 0 denotes no sharding.
ShardLength int `yaml:"shard_length"`
}
Config configures Store.
type File ¶
type File struct {
// contains filtered or unexported fields
}
File represends an open file descriptor to a blob in Store.
func (*File) Cancel ¶
Cancel is supposed to remove any written content. In this implementation file is not actually removed, but rather closed, which is fine, as there won't be key collisions when creating new files.
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store is a key-value, persistent, thread-safe, LRU store for blobs and their metadata.Metadata.
Supports pagination of blobs during reading/writing, such that blobs don't need to be fully loaded into memory.
New blobs are considered 'incomplete', which unlists them from LRU eviction. The store can be scoped to work on only (in-)complete blobs.
All APIs are thread-safe. Parallel access to a single file is allowed but clients must ensure they don't intervene with one another.
Supports (un-)marking blobs as non-evictable (may be needed when that data must be written back to remote storage).
Crash-resistant - all state is restored upon restart (check [newStore] for details).
Supports directory sharding to speed up disk performance.
func NewStore ¶
NewStore initializes a new *Store. If the store has been initialized in the same directory before, its state is recovered from disk with the following caveats:
`rebootIncompleteBlobs` configures whether incomplete blobs are evicted or rebooted on restart.
If the store's size is bigger than its capacity (e.g. configured capacity has been reduced or files have been leaked), it evicts blobs until size is within capacity.
func (*Store) BanEviction ¶
BanEviction marks a blob as unevictable by LRU eviction. It is idempotent. Needed when e.g. blobs must be written back to GCS/S3 and eviction before that is unacceptable.
func (*Store) Clean ¶
Clean deletes blobs from the store until targetUtilPercent is hit. Intended for manual use for incident mitigation/benchmarking. Blobs are deleted in the following order:
- complete, non-eviction-banned blobs, in LRU order.
- incomplete, non-eviction-banned blobs, in a random order.
- If respectEvictionBan is true, Clean will not delete blobs banned from eviction. Else, it will delete them, in a random order.
func (*Store) Create ¶
Create adds a new, incomplete blob to the store and reserves space for it. Incomplete entries cannot be automatically evicted. MarkComplete must be called once the blob is complete. The store uses `sizeBytes` for its eviction logic even if the blob's real size differs.
func (*Store) Delete ¶
Delete removes a blob and its metadata.Metadata from the store. Returns os.ErrNotExist on missing blob.
func (*Store) DeleteMetadata ¶
DeleteMetadata removes a blob's metadata. No error returned if the metadata is not present.
func (*Store) GetMetadata ¶
GetMetadata populates `md` if the metadata is present. Returns os.ErrNotExist if key is not in store.
func (*Store) ListMetadata ¶
ListMetadata returns all metadata.Metadata of key.
func (*Store) MarkComplete ¶
MarkComplete marks a blob as fully written. It enlists the blob for LRU eviction (unless BanEviction has been called). Additionally, other store APIs may filter blobs based on completeness.
func (*Store) Open ¶
Open returns an FD to a file in the store. os.ErrNotExist is returned on missing entry.
func (*Store) ScopeComplete ¶
ScopeComplete scopes Store's APIs such that they can only operate on complete blobs (except MarkComplete and Create). storelib.ErrOutOfScope is returned if the user tries to operate on an incomplete blob.
func (*Store) ScopeIncomplete ¶
ScopeIncomplete scopes Store's APIs such that they can only operate on incomplete blobs. storelib.ErrOutOfScope is returned if the user tries to operate on a complete blob.
func (*Store) Scoped ¶
Scoped scopes Store's APIs, such that storelib.ErrOutOfScope is returned upon attempting to operate on blobs out of scope.
func (*Store) SetMetadata ¶
SetMetadata atomically sets the respective metadata for a blob.
func (*Store) Stat ¶
Stat returns os.FileInfo about the blob. Returns os.ErrNotExist if the blob is not found.
func (*Store) UnbanEviction ¶
UnbanEviction removes the effect of BanEviction for a blob. It is idempotent.
func (*Store) WriteAtMetadata ¶
WriteAtMetadata implements io.WriterAt for the metadata file on disk.
Source Files