Documentation
¶
Overview ¶
This package contains several useful readymade sharding functions, which should plug nicely into most storage implementations.
The API contract for a sharding function is:
func(key string, shards *[]string)
In other words, the return is actually by a pointer to a slice which will be mutated. This API allows the calling code to hand in a slice with existing capacity, and thus allows for sharding functions to work without allocations.
There is not a named type for this contract, because we prefer that packages implementing the storage APIs should be possible to write without being required to import any code from the go-ipld-prime module. However, the function type definition above can be seen in many packages.
Not all packages use this API convention. The `fsstore` package does; some other storage implementations don't use sharding functions because they don't need them; most of the adapter packages which target older code do not, because those modules have their own sharding APIs already.
Example (Shard_R133) ¶
printShard(Shard_r133, "abcdefgh") printShard(Shard_r133, "abcdefg") printShard(Shard_r133, "abcdef") printShard(Shard_r133, "abcde") printShard(Shard_r133, "abcd") printShard(Shard_r133, "abc")
Output: abcdefgh => bcd/efg/abcdefgh abcdefg => abc/def/abcdefg abcdef => 000/cde/abcdef abcde => 000/bcd/abcde abcd => 000/abc/abcd abc => 000/000/abc
Example (Shard_r12) ¶
printShard(Shard_r12, "abcde") printShard(Shard_r12, "abcd") printShard(Shard_r12, "abc") printShard(Shard_r12, "ab")
Output: abcde => cd/abcde abcd => bc/abcd abc => ab/abc ab => 00/ab
Example (Shard_r122) ¶
printShard(Shard_r122, "abcdefgh") printShard(Shard_r122, "abcdefg") printShard(Shard_r122, "abcdef") printShard(Shard_r122, "abcde") printShard(Shard_r122, "abcd") printShard(Shard_r122, "abc")
Output: abcdefgh => de/fg/abcdefgh abcdefg => cd/ef/abcdefg abcdef => bc/de/abcdef abcde => ab/cd/abcde abcd => 00/bc/abcd abc => 00/ab/abc
Index ¶
Examples ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Shard_r12 ¶
Shard_r12 is a sharding function which will return two hunks. The last hunk is the full original key, and the first hunk is two bytes long. The prefix is are taken from the end of the original key, after skipping one byte. If the key is too short, the first hunk is just the ascii characters "00" instead.
If the shards parameter is a pointer to a slice that starts at zero length and a capacity of at least 2, this function will operate with no allocations.
Shard_r122 is functionally equivalent to "flatfs/shard/v1/next-to-last/2", as it's known in some other code -- it may be familiar as the default for block storage in go-ipfs.
func Shard_r122 ¶
Shard_r133 is a sharding function which will return three hunks. It is very similar to Shard_r133, but with shorter hunks. The last hunk is the full original key, and the first two hunks are two bytes long each. The prefix hunks are taken from the end of the original key, after skipping one byte. If the key is too short, padding of the ascii "0" character is used.
If the shards parameter is a pointer to a slice that starts at zero length and a capacity of at least 3, this function will operate with no allocations.
Supposing the key is a base32 string (where each byte effectively contains 2^5 bits), if a sufficient range of keys is present that all shards are seen, each group of shards will contain (2^5)^2=1024 entries. (This is often a useful number in practice, because if one is mapping shards onto filesystem directories, 1024 entries is almost certainly going to fit efficiently within any filesystem format you're likely to encounter; 1024-within-1024 also means you'll see about a billion entries before directories on the second layer of sharding will contain more than 1024 files. (If we're assuming 1MB blocks of data asthe actual contents, that would be quite a few terabytes of storage, so this is a very nice balanced trade for most practical systems.))
func Shard_r133 ¶
Shard_r133 is a sharding function which will return three hunks, the last of which is the full original key, and the first two of which are three bytes long. The prefix hunks are taken from the end of the original key, after skipping one byte. If the key is too short, padding of the ascii "0" character is used.
(This somewhat odd-sounding procedure is a useful one in practice, because if applying it on a base32 string that's a CID or multihash (which is the typical usage), it avoids the uneven distribution of the trailing characters of a base32 string, and also avoids the uneven distribution of the prefixes of CIDs or mulithashes.)
If the shards parameter is a pointer to a slice that starts at zero length and a capacity of at least 3, this function will operate with no allocations.
Supposing the key is a base32 string (where each byte effectively contains 2^5 bits), if a sufficient range of keys is present that all shards are seen, each group of shards will contain (2^5)^3=32768 entries.
Types ¶
This section is empty.
Source Files