Documentation
¶
Overview ¶
Package cluster implements the L0 distribution layer (DESIGN.md §3, §11, §14 M6–M7): rendezvous (HRW) hashing with spread-minimizing tokens, etcd-backed ring state and leases, RF=3 quorum replication, and rebalancing.
This is a scaffold stub: the ring, etcd integration, replication, and rebalancing are filled in at M6. Single-node users skip this package entirely ([Options.Cluster] == nil ⇒ no cluster layer).
Index ¶
- Constants
- func DecodeBatches(data []byte) ([]*fetch.Batch, error)
- func DecodeFetchRequest(data []byte) (sig signal.Signal, tenant string, start, end int64, eq []fetch.EqualMatcher, ...)
- func DecodeLogBatches(data []byte) ([]*fetch.Batch, error)
- func DecodeSeriesList(data []byte) ([]signal.Series, error)
- func DecodeSideTables(data []byte) (map[string][]byte, error)
- func DecodeWrite(data []byte) (sig signal.Signal, tenant string, walBytes []byte, err error)
- func EncodeBatches(batches []*fetch.Batch) []byte
- func EncodeFetchRequest(sig signal.Signal, tenant string, start, end int64, eq []fetch.EqualMatcher) []byte
- func EncodeLogBatches(batches []*fetch.Batch) []byte
- func EncodeSeriesList(series []signal.Series) []byte
- func EncodeSideTables(tables map[string][]byte) []byte
- func EncodeWrite(sig signal.Signal, tenant string, walBytes []byte) []byte
- func FetchSeries(ctx context.Context, client *http.Client, addr string, sig signal.Signal, ...) ([]signal.Series, error)
- func FetchSide(ctx context.Context, client *http.Client, addr string, sig signal.Signal, ...) (map[string][]byte, error)
- func ReadHandler(metricFn, logFn, traceFn, profileFn FetchFunc) http.Handler
- func SeriesHandler(fn SeriesFunc) http.Handler
- func SideHandler(fn SideFunc) http.Handler
- type Config
- type FetchFunc
- type RemoteFetcher
- type SeriesFunc
- type SideFunc
Constants ¶
const ( SeriesPath = "/internal/series" SidePath = "/internal/side" )
SeriesPath and SidePath are the HTTP paths of the series-listing and side-store servers.
const DefaultRF = 3
DefaultRF is the replication factor used when Config.RF is unset.
const DefaultRoot = "/oteldb"
DefaultRoot is the etcd key prefix used when Config.Root is empty.
const ReadPath = "/internal/fetch"
ReadPath is the HTTP path the cluster read (fetch fan-out) server serves.
Variables ¶
This section is empty.
Functions ¶
func DecodeBatches ¶
DecodeBatches parses EncodeBatches output, recomputing each batch's id from its identity.
func DecodeFetchRequest ¶
func DecodeFetchRequest(data []byte) (sig signal.Signal, tenant string, start, end int64, eq []fetch.EqualMatcher, err error)
DecodeFetchRequest parses a request made by EncodeFetchRequest.
func DecodeLogBatches ¶ added in v0.2.0
DecodeLogBatches parses EncodeLogBatches output, recomputing each batch's id from its identity.
func DecodeSeriesList ¶ added in v0.2.0
DecodeSeriesList parses EncodeSeriesList output.
func DecodeSideTables ¶ added in v0.2.0
DecodeSideTables parses EncodeSideTables output.
func DecodeWrite ¶
DecodeWrite splits a payload made by EncodeWrite into the signal, tenant id, and WAL bytes.
func EncodeBatches ¶
EncodeBatches serializes fetch batches: each series' identity (reversible hash pre-image) followed by its (timestamp, value) samples. The id is recomputed from the identity on decode, so it is not sent.
func EncodeFetchRequest ¶
func EncodeFetchRequest(sig signal.Signal, tenant string, start, end int64, eq []fetch.EqualMatcher) []byte
EncodeFetchRequest frames a fetch request: the signal, tenant, window, and any serializable equality matchers to push down to the peer (other predicates are re-checked by the requester).
func EncodeLogBatches ¶ added in v0.2.0
EncodeLogBatches serializes log fetch batches: each stream's identity, its record timestamps, and its named per-record columns (each tagged by physical kind). The id is recomputed from the identity on decode, so it is not sent.
func EncodeSeriesList ¶ added in v0.2.0
EncodeSeriesList serializes stream identities as length-prefixed reversible hash pre-images.
func EncodeSideTables ¶ added in v0.2.0
EncodeSideTables serializes a side-store table set (sorted by name for determinism).
func EncodeWrite ¶
EncodeWrite frames signal ‖ tenant ‖ walBytes into a replication payload.
func FetchSeries ¶ added in v0.2.0
func FetchSeries( ctx context.Context, client *http.Client, addr string, sig signal.Signal, tenant string, start, end int64, eq []fetch.EqualMatcher, ) ([]signal.Series, error)
FetchSeries lists a peer's stream identities for the signal+tenant+window, pushing down the serializable (equality) matchers; the caller re-applies any non-equality matchers.
func FetchSide ¶ added in v0.2.0
func FetchSide(ctx context.Context, client *http.Client, addr string, sig signal.Signal, tenant string) (map[string][]byte, error)
FetchSide returns a peer's side-store tables for the signal+tenant.
func ReadHandler ¶
ReadHandler returns the HTTP handler that serves fetches from the local store, reconstructing the pushed-down equality matchers and dispatching to the metric, log, trace, or profile fetch by the request's signal (encoding the result with the matching batch codec — samples for metrics, columns for the record signals). Mount it at ReadPath.
func SeriesHandler ¶ added in v0.2.0
func SeriesHandler(fn SeriesFunc) http.Handler
SeriesHandler serves SeriesPath: it reconstructs the pushed-down equality matchers and lists the matching stream identities via fn.
Types ¶
type Config ¶
type Config struct {
// Etcd is the etcd endpoint list for membership coordination.
Etcd []string
// Self is this node's identity: ID (ring identity), Zone (failure domain), and Addr
// (host:port the node listens on for replication and reaches peers at).
Self etcd.Member
// RF is the replication factor (replicas per write). Zero ⇒ 3.
RF int
// ShardsPerTenant splits each tenant's metric series into this many independently-placed
// shards (series → shard = hash(seriesID) % N), so a single large tenant spreads its ingest,
// storage, and compaction across up to N nodes instead of being pinned to one owner set. Zero
// or one ⇒ a single shard (the tenant is the shard; on-disk layout and placement are identical
// to the unsharded path). Applies to metrics only; the record signals are a single shard.
ShardsPerTenant int
// Root is the etcd key prefix for this cluster's state. Empty ⇒ "/oteldb".
Root string
}
Config is the cluster configuration. It is optional: a nil [storage.Options].Cluster means single-node mode (the cluster layer is absent). When set, the storage facade joins the etcd-coordinated cluster, runs the replica server on Config.Self.Addr, and routes writes to their ring-owners at replication factor Config.RF.
type FetchFunc ¶
type FetchFunc func(ctx context.Context, tenant string, start, end int64, matchers []fetch.Matcher) ([]*fetch.Batch, error)
FetchFunc fetches a tenant's series within [start, end] from the local store, applying the pushed-down matchers. It is what ReadHandler serves.
type RemoteFetcher ¶
type RemoteFetcher struct {
// contains filtered or unexported fields
}
RemoteFetcher is a fetch.Fetcher over a peer node's ReadHandler. It forwards only the request's tenant and window (matchers are re-applied by the caller), so it returns the peer's full window — a superset the fetch contract permits.
func NewRemoteFetcher ¶
NewRemoteFetcher returns a fetcher that reads the given signal from the peer at addr. A nil client uses http.DefaultClient. The zero signal value reads metrics.
type SeriesFunc ¶ added in v0.2.0
type SeriesFunc func(ctx context.Context, tenant string, start, end int64, matchers []fetch.Matcher) ([]signal.Series, error)
SeriesFunc lists the local store's stream identities for a tenant matching matchers within the window (a zero window disables the time filter).
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Package etcd backs the L0 cluster ring with etcd: a node registers itself under a lease and watches the member set, so membership is live and self-healing — a crashed node's lease expires and it drops out of every other node's ring within the TTL, with no manual deregistration.
|
Package etcd backs the L0 cluster ring with etcd: a node registers itself under a lease and watches the member set, so membership is live and self-healing — a crashed node's lease expires and it drops out of every other node's ring within the TTL, with no manual deregistration. |
|
Package rebalance computes the minimal ownership changes to apply a cluster membership change (DESIGN.md §11).
|
Package rebalance computes the minimal ownership changes to apply a cluster membership change (DESIGN.md §11). |
|
Package replica is the L0 write-replication layer: it fans an opaque write payload out to the ring-owners of a key and returns once a write **quorum** has durably applied it, so the unflushed head survives the loss of a minority of replicas (DESIGN.md §11; RF=3, quorum (RF/2)+1=2).
|
Package replica is the L0 write-replication layer: it fans an opaque write payload out to the ring-owners of a key and returns once a write **quorum** has durably applied it, so the unflushed head survives the loss of a minority of replicas (DESIGN.md §11; RF=3, quorum (RF/2)+1=2). |
|
Package ring implements rendezvous (highest-random-weight, HRW) hashing — the L0 sharding primitive (DESIGN.md §11).
|
Package ring implements rendezvous (highest-random-weight, HRW) hashing — the L0 sharding primitive (DESIGN.md §11). |