cloudspanner

type SequenceOpts ¶

type SequenceOpts struct {
	BatchSize uint64
	Shards    uint64
}

SequenceOpts configures the sequence storage sharding mechanism.

Log entries are split into the configured number of Shards, where each shard stores a periodic sub-sequence of batches of BatchSize. For example, if Shards is 3, and BatchSize is 2 then the entries are sharded as follows:

0 0 1 1 2 2 0 0 1 1 ...

Such schema optimizes for the case when entries are written in a nearly sequential way. If many concurrent writes are happening, all shards will be involved in parallel, and Cloud Spanner will add splits in between.

TODO(pavelkalinnikov): Store the parameters in per-tree metadata.

type SequenceStorage ¶

type SequenceStorage struct {
	// contains filtered or unexported fields
}

SequenceStorage allows reading from and writing to a sequence storage.

func NewSequenceStorage(c *spanner.Client, treeID int64, opts SequenceOpts) *SequenceStorage

func (s *SequenceStorage) Read(ctx context.Context, begin, end uint64) ([]storage.Entry, error)

func (s *SequenceStorage) Write(ctx context.Context, begin uint64, entries []storage.Entry) error

type TreeOpts ¶

type TreeOpts struct {
	ShardLevels uint // Between 1 and 65.
	LeafShards  int64
}

TreeOpts stores sharding parameters for tree storage.

The sharding scheme is as follows. The lower ShardLevels levels are split into LeafShards shards, where each shard stores a periodic sub-structure of perfect subtrees. For example, if ShardLevels is 2, and LeafShards is 3 then the lower 2 levels are sharded as shown below:

 0   1   2   0   1
/ \ / \ / \ / \ / \
0 0 1 1 2 2 0 0 1 1 ...

Additionally, a single shard number 3 is created for all the nodes from the levels above.

Such schema optimizes for the case when nodes are written to the tree in a nearly sequential way. If many concurrent writes are happening, all shards will be involved in parallel, and Cloud Spanner will add splits in between.

TODO(pavelkalinnikov): Shard higher levels as well for more scalability. TODO(pavelkalinnikov): Achieve better vertical locality with stratification. TODO(pavelkalinnikov): Store the parameters in per-tree metadata.

type TreeStorage ¶

type TreeStorage struct {
	// contains filtered or unexported fields
}

TreeStorage allows reading from and writing to a tree storage.

func NewTreeStorage(c *spanner.Client, treeID int64, opts TreeOpts) *TreeStorage

func (t *TreeStorage) Read(ctx context.Context, ids []compact.NodeID) ([][]byte, error)

func (t *TreeStorage) Write(ctx context.Context, nodes []storage.Node) error