Documentation
¶
Overview ¶
Package shard implements a single Shard in the database A Shard is a single directory on disk, called the Shard's "root". All of the data for a Shard is stored as files, which are the immediate children of the directory. Shards may have subdirectories, which are entirely ignored by the Shard. In practice, the organizing layer above will place child Shard directories beneath the the parent in the filesystem.
Index ¶
- Constants
- func Copy(srcLen uint32, srcTab Table, srcPack Pack, dstTab Table, dstPack Pack) (int, error)
- func CreatePackFile(root *os.Root, gen uint32, maxSize uint32) (*os.File, error)
- func CreateTableFile(shardRoot *os.Root, gen uint32, maxSize uint32) (*os.File, error)
- func IsErrShardFull(err error) bool
- func KeyCompare(a, b Key) int
- func LoadPackFile(root *os.Root, gen uint32) (*os.File, error)
- func LoadTableFile(shardRoot *os.Root, gen uint32) (*os.File, error)
- func PackFilename(gen uint32) string
- func SaveManifest(shardRoot *os.Root, manifest Manifest) error
- func TableFilename(gen uint32) string
- type Entry
- type ErrShardFull
- type Key
- type Manifest
- type Pack
- type Shard
- func (s *Shard) Close() error
- func (sh *Shard) Flush() error
- func (s *Shard) HasSpace(dataLen int) bool
- func (sh *Shard) Hydrate(maxTableSize, maxPackSize uint32) error
- func (s *Shard) LocalAppend(key Key, data []byte) (bool, error)
- func (s *Shard) LocalDelete(key Key) (bool, error)
- func (sh *Shard) LocalExists(key Key) bool
- func (sh *Shard) LocalGet(key Key, fn func(data []byte)) (bool, error)
- func (sh *Shard) TableLen() uint32
- type Table
Constants ¶
const ( TableFileExt = ".slot" PackFileExt = ".pack" )
const ( // TableEntrySize is the size of a row in the table. // It actually uses less space than this, but we want to align to 4096 bytes. TableEntrySize = 32 PageSize = 4096 EntriesPerPage = PageSize / TableEntrySize // DefaultMaxTableLen is the maximum length of a table in rows. DefaultMaxTableLen = (1 << 20) / TableEntrySize )
const DefaultMaxPackSize = 1 << 26
const ManifestFilename = "MF"
ManifestFilename is the filename for the manifest. This does not include the shard id, which will be in the directory path.
const ManifestSize = 32
ManifestSize is the size of the manifest in bytes.
Variables ¶
This section is empty.
Functions ¶
func Copy ¶
Copy copies entries from src{Tab,Pack} to dst{Tab,Pack}. If there is not enough space available in either the table or the pack, then an error is returned. Copy will not copy tombstones, and will sort all the entries by key before copying.
func CreatePackFile ¶
CreatePackFile creates a file configured for a pack in the filesystem, and returns it.
func CreateTableFile ¶
CreateTableFile creates a file configured for a table in the filesystem, and returns it. maxSize is the maximum size of the table in bytes, NOT the number of rows.
func IsErrShardFull ¶
func KeyCompare ¶
func PackFilename ¶
func TableFilename ¶
TableFilename returns the filename for a table. This is just the filename, the directory will contain the shard prefix.
Types ¶
type ErrShardFull ¶
type ErrShardFull struct{}
func (ErrShardFull) Error ¶
func (e ErrShardFull) Error() string
type Key ¶
type Key [3]uint64
Key is a 192 bit key. The 0th bit is considered the first bit, and that is at k[0] & (1 << 0).
func KeyFromBytes ¶
func (Key) RotateAway ¶
RotateAway specifies the amount of bits to rotate the key away from the 0th bit. The rotation is performed away from zero (towards more significant bit positions) modulo the key length.
type Manifest ¶
type Manifest struct {
// Nonce is the number of times the manifest has been saved.
Nonce uint64
// Gen is the Shard's current generation.
Gen uint32
// Count is the number of entries in the table with the current generation.
TableLen uint32
}
Manifest is the source of truth for a Shard's state.
type Pack ¶
type Pack struct {
// contains filtered or unexported fields
}
Pack is an append-only file on disk.
type Shard ¶
type Shard struct {
// contains filtered or unexported fields
}
Shard is a directory on disk containing table and pack files, potentially across multiple generations. Each shard is independent of every other shard, and there are no consistency guarantees between shards. The Shard has no information about where it is in the trie. Shards only deal with files in their immediate directory, subdirectories (which correspond to children in the trie) are ignored by the Shard.
func (*Shard) Flush ¶
Flush causes the Shard's current state to be written to disk. First the pack and table are flushed concurrently. Then once both of them have flushed successfully, the manifest is saved.
func (*Shard) LocalAppend ¶
LocalAppend appends the key and data to the table. It returns an error if the table is full or the pack is full. It returns (false, nil) if the data already exists. It returns (true, nil) if the data was appended successfully.
func (*Shard) LocalDelete ¶
LocalDelete deletes the key from the table. It returns (true, nil) if the key was deleted successfully.
func (*Shard) LocalExists ¶
LocalExists checks if the key exists in this shards local data. Local means that the children and grandchildren are not checked.
type Table ¶
type Table struct {
// contains filtered or unexported fields
}
Table is an unordered append-only list of entries. Each entry points into a pack, and all entries are the same size.
func NewTable ¶
NewTable mmaps a file and returns a Table. count should be the number of rows in the table.
func (*Table) Close ¶
Close unmaps the table and closes the file. It DOES NOT flush the mmap to disk.
func (*Table) SlotOffset ¶
SlotOffset returns the offset of the i-th slot in the table. Slot 0 starts at 4 bytes, to make room for the row count at the beginning.
Source Files