Documentation
¶
Overview ¶
Package cluster implements rendezvous hashing (a.k.a. highest random weight hashing). See http://en.wikipedia.org/wiki/Rendezvous_hashing for more information. This is based off of https://github.com/tysonmote/rendezvous
Index ¶
- Constants
- Variables
- func FairQueue(ctx context.Context, input <-chan *Batch, opts FairQueueOpts) chan *Batch
- func GetOutboundIP() (net.IP, error)
- func IsInitReady(pod *v1.Pod) bool
- func IsPodReady(pod *v1.Pod) bool
- func ReservedWorkers(n, percent int) int
- func RunWorkers(ctx context.Context, q PriorityQueues, n, percent int, fn func(*Batch), ...)
- type Batch
- type Batcher
- type BatcherOpts
- type Client
- type ClientOpts
- type Coordinator
- type CoordinatorOpts
- type ErrBadRequest
- type FairQueueOpts
- type FakeReplicator
- type Hash
- type Health
- func (h *Health) Close() error
- func (h *Health) IsHealthy() bool
- func (h *Health) IsPeerHealthy(peer string) bool
- func (h *Health) MaxSegmentAge() time.Duration
- func (h *Health) Open(ctx context.Context) error
- func (h *Health) SegmentsSize() int64
- func (h *Health) SegmentsTotal() int64
- func (h *Health) SetPeerHealthy(peer string)
- func (h *Health) SetPeerUnhealthy(peer string)
- func (h *Health) SetPeerUnhealthyFor(peer string, retryAfter time.Duration)
- func (h *Health) TransferQueueSize() int
- func (h *Health) UnhealthyReason() string
- func (h *Health) UploadQueueSize() int
- type HealthOpts
- type HealthStatus
- type MetricPartitioner
- type OTLPLogsWriter
- type Partitioner
- type PeerHealthReporter
- type PeerInfo
- type PeerOverloadedError
- type PriorityQueues
- type QueueSizer
- type RealtimeBatchOpts
- type Replicator
- type ReplicatorOpts
- type SegmentExistsError
- type SegmentLockedError
- type SegmentRemover
- type SegmentUsage
- type Segmenter
Constants ¶
const ( ReasonLargeUploadQueue = "LargeUploadQueue" ReasonLargeTransferQueue = "LargeTransferQueue" ReasonMaxSegmentsExceeded = "MaxSegmentsExceeded" ReasonMaxDiskUsageExceeded = "MaxDiskUsageExceeded" )
const DefaultMaxSegmentCount = 25
DefaultMaxSegmentCount is the default maximum number of segments to include in a batch before creating a new batch.
const ( // DefaultQueuedLinger is the default time a closed queued segment waits to be batched with others. It matches // the interval of the periodic scan it replaces so batches are similar. DefaultQueuedLinger = 5 * time.Second )
const DefaultQueuedReservedWorkersPercent = 10
DefaultQueuedReservedWorkersPercent is the default percentage of workers reserved for queued batches.
const MaxTransferBytesHeader = "X-Adxmon-Max-Transfer-Bytes"
MaxTransferBytesHeader advertises how many bytes a destination can accept, so a sender holding an oversized batch can split instead of retrying it.
const OverloadReasonHeader = "X-Adxmon-Overload-Reason"
const OverloadScopeHeader = "X-Adxmon-Overload-Scope"
const PeerTransferHeader = "X-Adxmon-Peer-Transfer"
PeerTransferHeader marks transfers sent by an ingestor rather than a collector. They are already routed to their owner, so the receiver stores them instead of forwarding them again.
Variables ¶
var ErrPeerOverloaded error = &PeerOverloadedError{}
var ErrSegmentExists error = &SegmentExistsError{}
var ErrSegmentLocked error = &SegmentLockedError{}
Functions ¶
func FairQueue ¶ added in v0.4.0
func FairQueue(ctx context.Context, input <-chan *Batch, opts FairQueueOpts) chan *Batch
FairQueue drains a compatibility input channel into bounded per-table queues and schedules them with deficit round robin, so a destination's share of throughput reflects bytes rather than batch count.
A full table queue releases its claims for a later WAL scan, never its data. Batches must be released by workers. A destination may use the whole pool when it is the only one with work; as other destinations become active each one keeps a worker in reserve, so no destination can monopolise the pool.
The returned channel plugs into PriorityQueues as the queued (bulk) path. Realtime work is deliberately not scheduled here: it is latency critical and already protected by reserved workers.
func IsInitReady ¶
IsInitReady returns true if all init containers are in a ready state
func IsPodReady ¶
IsPodReady returns true if all containers in a pod are in a ready state
func ReservedWorkers ¶ added in v0.4.0
ReservedWorkers returns how many of n workers are reserved for queued batches. At least one worker is reserved and at least one worker is left to process realtime batches. A single worker is never reserved.
func RunWorkers ¶ added in v0.4.0
func RunWorkers(ctx context.Context, q PriorityQueues, n, percent int, fn func(*Batch), done func())
RunWorkers starts n workers that call fn for each batch from q until ctx is done. The first ReservedWorkers(n, percent) workers only process queued batches. done is called when each worker exits.
Types ¶
type Batch ¶
type Batch struct {
Segments []wal.SegmentInfo
Database string
Table string
Prefix string
// Priority is the ingestion priority of the batch's table.
Priority ingestpolicy.Priority
// contains filtered or unexported fields
}
func (*Batch) IsReleased ¶
func (*Batch) Release ¶
func (b *Batch) Release()
Release releases the segments in the batch so they can be processed again.
func (*Batch) RemoveSegments ¶ added in v0.4.0
func (b *Batch) RemoveSegments(segments []wal.SegmentInfo) error
RemoveSegments removes a subset of the batch from disk. It is used when an oversized transfer is split and only part of the batch was delivered.
func (*Batch) RetryAfter ¶ added in v0.4.0
RetryAfter releases the worker promptly while deferring this table's next work.
type Batcher ¶
type Batcher interface {
service.Component
BatchSegments() error
UploadQueueSize() int
TransferQueueSize() int
SegmentsTotal() int64
SegmentsSize() int64
Release(batch *Batch)
Remove(batch *Batch) error
MaxSegmentAge() time.Duration
}
func NewBatcher ¶
func NewBatcher(opts BatcherOpts) (Batcher, error)
type BatcherOpts ¶
type BatcherOpts struct {
StorageDir string
MinUploadSize int64
MaxSegmentAge time.Duration
MaxTransferSize int64
MaxTransferAge time.Duration
MaxBatchSegments int
MaxTableBatches int
Partitioner MetricPartitioner
Segmenter Segmenter
UploadQueue chan *Batch
TransferQueue chan *Batch
PeerHealthReporter PeerHealthReporter
// RealtimeUploadQueue and RealtimeTransferQueue receive realtime batches. When nil, realtime batches are sent to
// UploadQueue and TransferQueue.
RealtimeUploadQueue chan *Batch
RealtimeTransferQueue chan *Batch
// Realtime enables event driven batching of realtime segments. When disabled, realtime segments are batched by
// the periodic scan.
Realtime RealtimeBatchOpts
// QueuedLinger is the maximum time a closed queued segment waits to be batched with others. Defaults to
// DefaultQueuedLinger.
QueuedLinger time.Duration
TransfersDisabled bool
// Usage is optional. When set, SegmentsTotal and SegmentsSize, and the metrics derived from them, report every
// segment file the store accounts for, including active segments, rather than only the closed segments in the
// Segmenter. This keeps the metrics consistent with the health status and with what is on disk.
Usage SegmentUsage
// Metrics for observability
SegmentsCountMetric prometheus.Gauge
SegmentsSizeBytesMetric prometheus.Gauge
SegmentsMaxAgeMetric prometheus.Gauge
// SegmentsSizeByPriorityMetric is optional. When set, it reports segment sizes by ingestion priority if the
// Segmenter tracks them.
SegmentsSizeByPriorityMetric *prometheus.GaugeVec
}
type Client ¶
type Client struct {
// contains filtered or unexported fields
}
func NewClient ¶
func NewClient(opts ClientOpts) (*Client, error)
type ClientOpts ¶
type ClientOpts struct {
// Close controls whether the client closes the connection after each request.
Close bool
// Timeout is the timeout for the http client and the http request.
Timeout time.Duration
// InsecureSkipVerify controls whether the client verifies the server's certificate chain and host name.
InsecureSkipVerify bool
// IdleConnTimeout is the maximum amount of time an idle (keep-alive) connection
// will remain idle before closing itself.
IdleConnTimeout time.Duration
// ResponseHeaderTimeout is the amount of time to wait for a server's response headers
// after fully writing the request (including its body, if any).
ResponseHeaderTimeout time.Duration
// MaxIdleConns controls the maximum number of idle (keep-alive) connections across all hosts.
MaxIdleConns int
// MaxIdleConnsPerHost, if non-zero, controls the maximum idle (keep-alive) per host.
MaxIdleConnsPerHost int
// MaxConnsPerHost, if non-zero, controls the maximum connections per host.
MaxConnsPerHost int
// TLSHandshakeTimeout specifies the maximum amount of time to
// wait for a TLS handshake. Zero means no timeout.
TLSHandshakeTimeout time.Duration
// DisableHTTP2 controls whether the client disables HTTP/2 support.
DisableHTTP2 bool
// DisableKeepAlives controls whether the client disables HTTP keep-alives.
DisableKeepAlives bool
// DisableGzip controls whether the client disables gzip compression.
DisableGzip bool
// PeerTransfer marks requests with PeerTransferHeader.
PeerTransfer bool
}
func (ClientOpts) WithDefaults ¶
func (c ClientOpts) WithDefaults() ClientOpts
type Coordinator ¶
type Coordinator interface {
MetricPartitioner
service.Component
// IsLeader returns true if the current node is the leader.
IsLeader() bool
// Peers returns the number of ingestor peers and this node's rank among them.
Peers() PeerInfo
// SubscribePeers registers fn to be called when the peer count or this node's rank changes. fn is called
// synchronously and must return quickly. The returned func unsubscribes fn.
SubscribePeers(fn func(PeerInfo)) (unsubscribe func())
}
func NewCoordinator ¶
func NewCoordinator(opts *CoordinatorOpts) (Coordinator, error)
type CoordinatorOpts ¶
type CoordinatorOpts struct {
K8sCli kubernetes.Interface
// Namespace is the namespace used to discover peers. If not specified, the coordinator will
// try to use the namespace of the current pod.
Namespace string
// Hostname is the hostname of the current node. This should be the statefulset hostname format
// in order to discover peers correctly
Hostname string
// InsecureSkipVerify controls whether a client verifies the server's certificate chain and host name.
InsecureSkipVerify bool
// PartitionSize is the max size of the group of nodes forming a partition. A partition is a set of nodes where
// keys are distributed.
// NOTE: This is not used in the current implementation.
PartitionSize int
}
type ErrBadRequest ¶
type ErrBadRequest struct {
Msg string
// contains filtered or unexported fields
}
func (*ErrBadRequest) Error ¶
func (e *ErrBadRequest) Error() string
func (*ErrBadRequest) Is ¶
func (e *ErrBadRequest) Is(target error) bool
func (*ErrBadRequest) WithRequestId ¶ added in v0.2.0
func (e *ErrBadRequest) WithRequestId(id string) error
type FairQueueOpts ¶ added in v0.4.0
type FairQueueOpts struct {
// Workers is the size of the consuming worker pool.
Workers int
// Pipeline labels the exported scheduling metrics. Empty disables them.
Pipeline string
// Quantum is the byte credit a destination earns per round. Larger values
// favour throughput, smaller values favour fairness between destinations
// with very different batch sizes.
Quantum int64
// QueueDepth bounds how many batches are held per destination. Excess
// batches are released and rediscovered by the next WAL scan.
QueueDepth int
// ReservedWorkers is how many workers stay available for destinations that
// are not already running. Without a reserve, a destination with enough
// queued work saturates the pool before a new destination appears and the
// newcomer waits for an in-flight batch to finish. Defaults to one when the
// pool has more than one worker.
ReservedWorkers int
}
FairQueueOpts configures how queued work is shared between destinations.
type FakeReplicator ¶
type FakeReplicator struct {
// contains filtered or unexported fields
}
func NewFakeReplicator ¶
func NewFakeReplicator() *FakeReplicator
func (*FakeReplicator) Close ¶
func (f *FakeReplicator) Close() error
func (*FakeReplicator) RealtimeTransferQueue ¶ added in v0.4.0
func (f *FakeReplicator) RealtimeTransferQueue() chan *Batch
func (*FakeReplicator) TransferQueue ¶
func (f *FakeReplicator) TransferQueue() chan *Batch
type Hash ¶
type Hash struct {
// contains filtered or unexported fields
}
func NewRendezvous ¶
New returns a new Hash ready for use with the given nodes.
type Health ¶
type Health struct {
QueueSizer QueueSizer
// contains filtered or unexported fields
}
Health tracks the health of peers in the cluster. If a peer is overloaded, it will be marked as unhealthy which will cause the service to stop sending writes to that peer for timeout period. Similarly, the of the current peer is tracked here and if it is unhealthy, the service will stop accepting writes.
func NewHealth ¶
func NewHealth(opts HealthOpts) *Health
func (*Health) IsPeerHealthy ¶
func (*Health) MaxSegmentAge ¶ added in v0.2.0
func (*Health) SegmentsSize ¶
func (*Health) SegmentsTotal ¶
func (*Health) SetPeerHealthy ¶
func (*Health) SetPeerUnhealthy ¶
func (*Health) SetPeerUnhealthyFor ¶ added in v0.4.0
SetPeerUnhealthyFor marks the peer unhealthy for at least retryAfter, capped at UnhealthyTimeout.
func (*Health) TransferQueueSize ¶
func (*Health) UnhealthyReason ¶
func (*Health) UploadQueueSize ¶
type HealthOpts ¶
type HealthOpts struct {
// UnhealthyTimeout is the maximum amount of time to wait before marking a peer as healthy.
UnhealthyTimeout time.Duration
// InitialUnhealthyTimeout is how long a peer is skipped after its first failure. Each consecutive failure
// doubles it, up to UnhealthyTimeout, and a success resets it. Defaults to UnhealthyTimeout, which skips the
// peer for the full UnhealthyTimeout on every failure.
InitialUnhealthyTimeout time.Duration
QueueSizer QueueSizer
MaxSegmentCount int64
MaxDiskUsage int64
}
type HealthStatus ¶
type MetricPartitioner ¶
type OTLPLogsWriter ¶
type Partitioner ¶
type Partitioner struct {
// contains filtered or unexported fields
}
Partitioner manages the distribution of metrics across nodes. It uses rendezvous hashing to distribute metrics roughly evenly. When nodes are added or removed, the distribution of metrics will change, but only by a proportional amount for each node. For example, if four nodes exists and a fifth is added, only 20% of the metrics will be reassigned to the new node.
func NewPartition ¶
func NewPartition(nodes map[string]string) (*Partitioner, error)
type PeerHealthReporter ¶
type PeerInfo ¶ added in v0.4.0
type PeerInfo struct {
// Count is the number of ready peers including this node.
Count int
// Rank is this node's zero based index among the peers sorted by name. Every peer computes the same ordering so
// ranks are unique across peers that observe the same peer set.
Rank int
}
PeerInfo describes this node's position among its ingestor peers.
type PeerOverloadedError ¶ added in v0.2.0
type PeerOverloadedError struct {
// RetryAfter is how long the peer asked senders to wait before retrying.
RetryAfter time.Duration
// contains filtered or unexported fields
}
PeerOverloadedError represents a peer overloaded error with optional requestId.
func (*PeerOverloadedError) Error ¶ added in v0.2.0
func (e *PeerOverloadedError) Error() string
func (*PeerOverloadedError) Is ¶ added in v0.2.0
func (e *PeerOverloadedError) Is(target error) bool
func (*PeerOverloadedError) WithRequestId ¶ added in v0.2.0
func (e *PeerOverloadedError) WithRequestId(id string) error
type PriorityQueues ¶ added in v0.4.0
PriorityQueues are the work queues for realtime and queued batches.
type QueueSizer ¶
type RealtimeBatchOpts ¶ added in v0.4.0
type RealtimeBatchOpts struct {
// MaxBatchLatency is the maximum time a closed realtime segment waits to be batched with others.
MaxBatchLatency time.Duration
// MaxBatchBytes is the maximum size of a realtime batch. A single larger segment is batched alone.
MaxBatchBytes int64
}
RealtimeBatchOpts configures event driven batching of realtime segments.
type Replicator ¶
type Replicator interface {
service.Component
// TransferQueue returns a channel that can be used to transfer files to other nodes.
TransferQueue() chan *Batch
// RealtimeTransferQueue returns a channel for realtime batches. They are transferred before queued batches.
RealtimeTransferQueue() chan *Batch
}
Replicator manages the transfer of local segments to other nodes.
func NewReplicator ¶
func NewReplicator(opts ReplicatorOpts) (Replicator, error)
type ReplicatorOpts ¶
type ReplicatorOpts struct {
// Partitioner is used to determine which node owns a given metric.
Partitioner MetricPartitioner
// Health is used to report the health of the peer replication.
Health PeerHealthReporter
// SegmentRemover is used to remove segments after they have been replicated.
SegmentRemover SegmentRemover
// InsecureSkipVerify controls whether a client verifies the server's certificate chain and host name.
InsecureSkipVerify bool
// Hostname is the name of the current node.
Hostname string
// MaxTransferConcurrency is the maximum number of concurrent transfer requests to in flight at a time.
// Default is 5.
MaxTransferConcurrency int
// DisableGzip controls whether the client uses gzip compression for transfer requests.
DisableGzip bool
// QueuedReservedWorkersPercent is the percentage of transfer workers reserved for queued batches. Defaults to
// DefaultQueuedReservedWorkersPercent.
QueuedReservedWorkersPercent int
// PeerTransfer marks transfers as sent by an ingestor so the receiver does not forward them.
PeerTransfer bool
// TransferLatency, if set, records the age of the oldest segment in each transferred batch by database.
TransferLatency *prometheus.HistogramVec
}
type SegmentExistsError ¶ added in v0.2.0
type SegmentExistsError struct {
// contains filtered or unexported fields
}
SegmentExistsError represents a segment already exists error with optional requestId.
func (*SegmentExistsError) Error ¶ added in v0.2.0
func (e *SegmentExistsError) Error() string
func (*SegmentExistsError) Is ¶ added in v0.2.0
func (e *SegmentExistsError) Is(target error) bool
func (*SegmentExistsError) WithRequestId ¶ added in v0.2.0
func (e *SegmentExistsError) WithRequestId(id string) error
type SegmentLockedError ¶ added in v0.2.0
type SegmentLockedError struct {
// contains filtered or unexported fields
}
SegmentLockedError represents a segment locked error with optional requestId.
func (*SegmentLockedError) Error ¶ added in v0.2.0
func (e *SegmentLockedError) Error() string
func (*SegmentLockedError) Is ¶ added in v0.2.0
func (e *SegmentLockedError) Is(target error) bool
func (*SegmentLockedError) WithRequestId ¶ added in v0.2.0
func (e *SegmentLockedError) WithRequestId(id string) error
type SegmentRemover ¶
type SegmentUsage ¶ added in v0.4.0
SegmentUsage reports the number and total size of WAL segment files, both active and closed.
type Segmenter ¶
type Segmenter interface {
Get(infos []wal.SegmentInfo, prefix string) []wal.SegmentInfo
PrefixesByAge() []string
Remove(si wal.SegmentInfo)
TotalSegments() int
TotalSize() int64
OldestSegmentAge() time.Duration
}