Documentation
¶
Overview ¶
Package ch2storagebackend migrates data out of chstorage's ClickHouse tables into the embedded storagebackend engine, by scanning ClickHouse directly (bypassing chstorage's selector-oriented queriers) and re-ingesting the decoded records as OTLP pdata.
Logs, traces, and metrics are supported. Metrics are migrated verbatim: chstorage already stores them as decomposed Prometheus-style series (histograms/summaries exploded into _count/_sum/_bucket{le}/{quantile} series), so each stored series is re-ingested 1:1 as a gauge number point, and exponential histograms (stored natively) are reconstructed as OTLP exponential-histogram datapoints. Exemplars are not migrated (the target engine drops them).
Index ¶
- type LogsStats
- type MetricsStats
- type Migrator
- func (m *Migrator) MigrateLogs(ctx context.Context, since time.Duration, batchSize int) (LogsStats, error)
- func (m *Migrator) MigrateMetrics(ctx context.Context, since time.Duration, batchSize int) (MetricsStats, error)
- func (m *Migrator) MigrateTraces(ctx context.Context, since time.Duration, batchSize int) (TracesStats, error)
- type Option
- type TracesStats
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type LogsStats ¶
type LogsStats struct {
// Records is the total number of log records migrated.
Records int
// Batches is the number of batches ConsumeLogs was called with.
Batches int
}
LogsStats reports the outcome of a Migrator.MigrateLogs run.
type MetricsStats ¶
type MetricsStats struct {
// Points is the total number of decomposed number points migrated.
Points int
// ExpHistograms is the total number of exponential-histogram datapoints migrated.
ExpHistograms int
// Batches is the number of batches ConsumeMetrics was called with.
Batches int
}
MetricsStats reports the outcome of a Migrator.MigrateMetrics run.
type Migrator ¶
type Migrator struct {
// contains filtered or unexported fields
}
Migrator copies data from chstorage's ClickHouse tables into a storagebackend.Backend.
func NewMigrator ¶
func NewMigrator(client chstorage.ClickHouseClient, tables chstorage.Tables, back *storagebackend.Backend, logger *zap.Logger, opts ...Option) *Migrator
NewMigrator creates a new Migrator.
func (*Migrator) MigrateLogs ¶
func (m *Migrator) MigrateLogs(ctx context.Context, since time.Duration, batchSize int) (LogsStats, error)
MigrateLogs migrates log records stored in ClickHouse into the storagebackend engine, converting and ingesting up to batchSize records at a time. When since is positive, only the last since of data (relative to the most recent record) is migrated.
func (*Migrator) MigrateMetrics ¶
func (m *Migrator) MigrateMetrics(ctx context.Context, since time.Duration, batchSize int) (MetricsStats, error)
MigrateMetrics migrates metrics stored in ClickHouse into the storagebackend engine. Decomposed number series (gauges/sums and histogram/summary components) are ingested verbatim as gauge datapoints; exponential histograms are reconstructed natively. When since is positive, only the last since of data (relative to the most recent point) is migrated.
func (*Migrator) MigrateTraces ¶
func (m *Migrator) MigrateTraces(ctx context.Context, since time.Duration, batchSize int) (TracesStats, error)
MigrateTraces migrates spans stored in ClickHouse into the storagebackend engine, converting and ingesting up to batchSize spans at a time. When since is positive, only the last since of data (relative to the most recent span) is migrated.
type Option ¶
type Option func(*Migrator)
Option configures a Migrator.
func WithThrottle ¶
WithThrottle sleeps d after every ConsumeLogs/ConsumeTraces batch. A bulk migration can ingest orders of magnitude faster than the storage engine's background flush/compaction loop can drain, so without a cap the head grows unbounded in RAM until the process OOMs (see storagebackend, the FlushInterval/FlushThresholdBytes options alone do not apply backpressure on the write path). Zero (the default) applies no throttling.
type TracesStats ¶
type TracesStats struct {
// Spans is the total number of spans migrated.
Spans int
// Batches is the number of batches ConsumeTraces was called with.
Batches int
}
TracesStats reports the outcome of a Migrator.MigrateTraces run.
Source Files