Documentation
¶
Overview ¶
Package backfill imports historical samples into the persistent tier and the index, bypassing the hot tier. The hot tier accepts samples only while their chunk window is open (chunk_range + grace_period), so this is the only way to load history: data migrated from Prometheus, samples lost in an outage longer than the grace period, or data from xtsdb 0.x.
Every chunk window of an imported series is written as a version of that window (see storage.ColdChunk), next to any version flushed from the hot tier; reads merge the versions.
Index ¶
Constants ¶
const DefaultMaxBuffered = 10_000_000
DefaultMaxBuffered bounds the samples ImportOpenMetrics holds in memory (about 32 bytes each, with their share of the per-series buffers) before importing what it has read so far.
const Fragment = 0
Fragment is the fragment of imported chunk versions. Chunks flushed from the hot tier have fragments that are timestamps in microseconds, so where an imported sample and an ingested one have the same timestamp, the ingested one wins: an import fills gaps but never overrides what xtsdb received itself. Being constant, it also makes imports idempotent: importing the same samples again writes the same rows.
Variables ¶
This section is empty.
Functions ¶
func Blocks ¶
Blocks returns the Prometheus TSDB blocks in dir: dir itself if it is a block, otherwise its subdirectories that are blocks, in name (ULID, hence time) order. The head of a Prometheus data directory (its WAL, about the last two to three hours) is not a block, even after Prometheus stopped; a snapshot writes it as a block. See HasHead.
Types ¶
type Importer ¶
type Importer struct {
Stats Stats
// contains filtered or unexported fields
}
Importer writes series to the cold tier and the index.
func (*Importer) Import ¶
Import imports the samples of one series and takes ownership of samples. They need not be sorted; of several samples with the same timestamp, the last one wins. Samples that cannot be imported are counted in Stats; an error means that a backend failed, and importing the series again is safe. Import is safe for concurrent use.
func (*Importer) ImportBlock ¶
ImportBlock imports the float samples of every series of a Prometheus TSDB block, up to concurrency series at a time, native histograms included. Deleted samples (tombstones) are left out.
The block is read with the index, chunks and tombstones packages of the Prometheus TSDB rather than the tsdb package, which would link in the Prometheus configuration and the cloud SDKs it depends on.
func (*Importer) ImportOpenMetrics ¶
func (im *Importer) ImportOpenMetrics(ctx context.Context, data []byte, concurrency, maxBuffered int) error
ImportOpenMetrics imports OpenMetrics text in which every sample has a timestamp, the input of `promtool tsdb create-blocks-from openmetrics`. Samples are collected per series and imported when maxBuffered samples have been read, and at the end; a window split this way is stored as several versions, which reads merge. _created series are skipped.
type Options ¶
type Options struct {
Layout model.Layout
// Retention skips samples older than this; zero keeps all of them.
Retention time.Duration
// MaxFuture skips samples further than this in the future.
MaxFuture time.Duration
Logger *slog.Logger
}
Options configures an Importer.
type State ¶
type State struct {
// contains filtered or unexported fields
}
State records the sources that were imported completely, one per line, so that an interrupted import can be resumed without reading them again. Imports are idempotent, so a source that was being imported when the import stopped is simply imported again.
func OpenState ¶
OpenState opens or creates a state file. With an empty path the state is kept in memory only.
type Stats ¶
type Stats struct {
// Series counts series imports with at least one sample: a series
// imported from several blocks, or from OpenMetrics input imported in
// several slices, counts once per block or slice.
Series atomic.Int64
Samples atomic.Int64 // samples imported
Chunks atomic.Int64 // chunk window versions written
Bytes atomic.Int64 // compressed bytes written
TooOld atomic.Int64 // samples older than the retention, skipped
TooNew atomic.Int64 // samples too far in the future, skipped
Invalid atomic.Int64 // samples of series with invalid labels, skipped
Collisions atomic.Int64 // samples of series whose ID collides, skipped
Histograms atomic.Int64 // native histogram samples imported (part of Samples)
}
Stats counts what an import did.