backfill

package
v0.0.0-...-0398336 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 15, 2026 License: Apache-2.0 Imports: 28 Imported by: 0

Documentation

Overview

Package backfill imports historical samples into the persistent tier and the index, bypassing the hot tier. The hot tier accepts samples only while their chunk window is open (chunk_range + grace_period), so this is the only way to load history: data migrated from Prometheus, samples lost in an outage longer than the grace period, or data from xtsdb 0.x.

Every chunk window of an imported series is written as a version of that window (see storage.ColdChunk), next to any version flushed from the hot tier; reads merge the versions.

Index

Constants

View Source
const DefaultMaxBuffered = 10_000_000

DefaultMaxBuffered bounds the samples ImportOpenMetrics holds in memory (about 32 bytes each, with their share of the per-series buffers) before importing what it has read so far.

View Source
const Fragment = 0

Fragment is the fragment of imported chunk versions. Chunks flushed from the hot tier have fragments that are timestamps in microseconds, so where an imported sample and an ingested one have the same timestamp, the ingested one wins: an import fills gaps but never overrides what xtsdb received itself. Being constant, it also makes imports idempotent: importing the same samples again writes the same rows.

Variables

This section is empty.

Functions

func BlockID

func BlockID(dir string) (string, error)

BlockID returns the ULID of a block, which identifies its contents.

func Blocks

func Blocks(dir string) ([]string, error)

Blocks returns the Prometheus TSDB blocks in dir: dir itself if it is a block, otherwise its subdirectories that are blocks, in name (ULID, hence time) order. The head of a Prometheus data directory (its WAL, about the last two to three hours) is not a block, even after Prometheus stopped; a snapshot writes it as a block. See HasHead.

func FileID

func FileID(path string) (string, error)

FileID identifies the content of an input file by its absolute path, size and modification time.

func HasHead

func HasHead(dir string) bool

HasHead reports whether dir is a Prometheus data directory with a head (wal/) that Blocks does not return.

Types

type Importer

type Importer struct {
	Stats Stats
	// contains filtered or unexported fields
}

Importer writes series to the cold tier and the index.

func New

func New(cold storage.ColdStore, index storage.IndexWriter, opts Options) (*Importer, error)

New creates an Importer.

func (*Importer) Import

func (im *Importer) Import(ctx context.Context, lset labels.Labels, samples []model.Sample) error

Import imports the samples of one series and takes ownership of samples. They need not be sorted; of several samples with the same timestamp, the last one wins. Samples that cannot be imported are counted in Stats; an error means that a backend failed, and importing the series again is safe. Import is safe for concurrent use.

func (*Importer) ImportBlock

func (im *Importer) ImportBlock(ctx context.Context, dir string, concurrency int) error

ImportBlock imports the float samples of every series of a Prometheus TSDB block, up to concurrency series at a time, native histograms included. Deleted samples (tombstones) are left out.

The block is read with the index, chunks and tombstones packages of the Prometheus TSDB rather than the tsdb package, which would link in the Prometheus configuration and the cloud SDKs it depends on.

func (*Importer) ImportOpenMetrics

func (im *Importer) ImportOpenMetrics(ctx context.Context, data []byte, concurrency, maxBuffered int) error

ImportOpenMetrics imports OpenMetrics text in which every sample has a timestamp, the input of `promtool tsdb create-blocks-from openmetrics`. Samples are collected per series and imported when maxBuffered samples have been read, and at the end; a window split this way is stored as several versions, which reads merge. _created series are skipped.

type Options

type Options struct {
	Layout model.Layout
	// Retention skips samples older than this; zero keeps all of them.
	Retention time.Duration
	// MaxFuture skips samples further than this in the future.
	MaxFuture time.Duration
	Logger    *slog.Logger
}

Options configures an Importer.

type State

type State struct {
	// contains filtered or unexported fields
}

State records the sources that were imported completely, one per line, so that an interrupted import can be resumed without reading them again. Imports are idempotent, so a source that was being imported when the import stopped is simply imported again.

func OpenState

func OpenState(path string) (*State, error)

OpenState opens or creates a state file. With an empty path the state is kept in memory only.

func (*State) Close

func (s *State) Close() error

Close closes the state file.

func (*State) Done

func (s *State) Done(id string) bool

Done reports whether a source was imported completely.

func (*State) MarkDone

func (s *State) MarkDone(id string) error

MarkDone records that a source was imported completely.

type Stats

type Stats struct {
	// Series counts series imports with at least one sample: a series
	// imported from several blocks, or from OpenMetrics input imported in
	// several slices, counts once per block or slice.
	Series     atomic.Int64
	Samples    atomic.Int64 // samples imported
	Chunks     atomic.Int64 // chunk window versions written
	Bytes      atomic.Int64 // compressed bytes written
	TooOld     atomic.Int64 // samples older than the retention, skipped
	TooNew     atomic.Int64 // samples too far in the future, skipped
	Invalid    atomic.Int64 // samples of series with invalid labels, skipped
	Collisions atomic.Int64 // samples of series whose ID collides, skipped
	Histograms atomic.Int64 // native histogram samples imported (part of Samples)
}

Stats counts what an import did.

func (*Stats) String

func (s *Stats) String() string

String summarizes the stats.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL