Documentation
¶
Overview ¶
Package flatfs is a Datastore implementation that stores all objects in a two-level directory structure in the local file system, regardless of the hierarchy of the keys.
Index ¶
- Constants
- Variables
- func DirIsEmpty(name string) (bool, error)
- func WriteReadme(dir string, id *ShardIdV1) error
- func WriteShardFunc(dir string, id *ShardIdV1) error
- type FileObject
- type ShardFunc
- type ShardIdV1
- type Store
- func (fs *Store) Close() error
- func (fs *Store) Delete(ctx context.Context, key string) error
- func (fs *Store) Get(ctx context.Context, key string, opts ...objectstore.GetOption) (objectstore.Object, error)
- func (fs *Store) Put(ctx context.Context, key string, size uint64, value io.Reader) error
- func (fs *Store) ShardStr() string
Constants ¶
const PREFIX = "/repo/flatfs/shard/"
const README_FN = "_README"
const SHARDING_FN = "SHARDING"
const SyncThreadsMax = 16
don't block more than 16 threads on sync opearation 16 should be able to saturate most RAIDs in case of two used disks per write (RAID 1, 5) and queue depth of 2, 16 concurrent Sync calls should be able to saturate 16 HDDs RAID TODO: benchmark it out, maybe provide tweak parmeter
Variables ¶
var ( // RetryDelay is a timeout for a backoff on retrying operations // that fail due to transient errors like too many file descriptors open. RetryDelay = time.Millisecond * 200 // RetryAttempts is the maximum number of retries that will be attempted // before giving up. RetryAttempts = 6 )
var ( ErrStoreExists = errors.New("datastore already exists") ErrStoreDoesNotExist = errors.New("datastore directory does not exist") ErrShardingFileMissing = fmt.Errorf("%s file not found in datastore", SHARDING_FN) ErrClosed = errors.New("datastore closed") ErrInvalidKey = errors.New("key not supported by flatfs") )
var NEXT_TO_LAST2_DEF_SHARD = NextToLast(2)
var README_NEXT_TO_LAST2_DEF_SHARD = `` /* 925-byte string literal not displayed */
Functions ¶
func DirIsEmpty ¶
From: http://stackoverflow.com/questions/30697324/how-to-check-if-directory-on-path-is-empty
func WriteReadme ¶
func WriteShardFunc ¶
Types ¶
type FileObject ¶
type FileObject struct {
// contains filtered or unexported fields
}
func (FileObject) Body ¶
func (o FileObject) Body() io.ReadCloser
func (FileObject) Size ¶
func (o FileObject) Size() int64
type ShardIdV1 ¶
type ShardIdV1 struct {
// contains filtered or unexported fields
}
func NextToLast ¶
Prefix returns a sharding function taking the suffixLen characters of the key before the very last character. If too short, the key is padded with "_".
func ParseShardFunc ¶
func Prefix ¶
Prefix returns a sharding function taking the first prefixLen characters of the key. If too short, the key is padded with "_".
func ReadShardFunc ¶
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store implements objectstore.Store. Note this datastore cannot guarantee order of concurrent write operations to the same key. See the explanation in Put().
func (*Store) Delete ¶
Delete removes a key/value from the Datastore. Please read the Put() explanation about the handling of concurrent write operations to the same key.
func (*Store) Get ¶
func (fs *Store) Get(ctx context.Context, key string, opts ...objectstore.GetOption) (objectstore.Object, error)
func (*Store) Put ¶
Put stores a key/value in the datastore.
Note, that we do not guarantee order of write operations (Put or Delete) to the same key in this datastore.
For example. i.e. in the case of two concurrent Put, we only guarantee that one of them will come through, but cannot assure which one even if one arrived slightly later than the other. In the case of a concurrent Put and a Delete operation, we cannot guarantee which one will win.
Source Files