insertprogress

package
v6.4.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 17, 2026 License: Apache-2.0 Imports: 9 Imported by: 0

Documentation

Overview

Package insertprogress provides driver-agnostic insert operation progress tracking.

Index

Constants

View Source
const (
	// StageRuntimeNext means a worker is pulling rows from the datagen runtime.
	StageRuntimeNext = "runtime_next"
	// StageSQLBulkInsertExec means a SQL driver is executing a multi-row INSERT.
	StageSQLBulkInsertExec = "sql_bulk_insert_exec"
	// StagePostgresCopyFrom means the PostgreSQL driver is inside pgx.CopyFrom.
	StagePostgresCopyFrom = "postgres_copy_from"
	// StagePostgresBulkInsertExec means the PostgreSQL driver is executing a VALUES INSERT.
	StagePostgresBulkInsertExec = "postgres_bulk_insert_exec"
	// StagePostgresColumnarExec means the PostgreSQL driver is executing an unnest INSERT.
	StagePostgresColumnarExec = "postgres_columnar_exec"
	// StageYDBBulkUpsert means the YDB driver is flushing a BulkUpsert batch.
	StageYDBBulkUpsert = "ydb_bulk_upsert"
	// StageCSVWrite means the CSV driver is writing generated rows to a shard file.
	StageCSVWrite = "csv_write"
	// StageNoopDrain means the noop driver is draining generated rows without I/O.
	StageNoopDrain = "noop_drain"
)
View Source
const (
	// DefaultInterval is the default cadence for insert operation progress samples.
	DefaultInterval = 10 * time.Second
	// DefaultStallAfter is the default no-progress duration before warning.
	DefaultStallAfter = time.Minute
)

Variables

View Source
var ErrInvalidMode = errors.New("insert progress: invalid mode")

ErrInvalidMode is returned when progress mode is not one of off/log/metrics/both.

Functions

func AddBatch

func AddBatch(ctx context.Context, rows int64, elapsed time.Duration)

AddBatch records one completed driver write unit for the tracker attached to ctx.

func AddConfirmed

func AddConfirmed(ctx context.Context, rows int64)

AddConfirmed records confirmed rows on the tracker attached to ctx.

func AddGenerated

func AddGenerated(ctx context.Context, rows int64)

AddGenerated records generated rows on the tracker attached to ctx.

func Canceled

func Canceled(ctx context.Context) error

Canceled reports whether ctx has been canceled. Insert workers call it inside their row-drain loops so cancellation propagates through errgroup.WithContext and unblocks the worker instead of letting it drain the whole table. The non-blocking select keeps the per-row cost negligible.

func ContextWithTracker

func ContextWithTracker(ctx context.Context, tracker *Tracker) context.Context

ContextWithTracker attaches an insert operation progress tracker to ctx.

func ContextWithWorker

func ContextWithWorker(ctx context.Context, workerIndex int) context.Context

ContextWithWorker attaches the current insert operation worker index to ctx.

func SetStage

func SetStage(ctx context.Context, stage string)

SetStage records the current stage for the tracker attached to ctx.

func SetTotal

func SetTotal(ctx context.Context, rows int64)

SetTotal records the actual runtime row count for the tracker attached to ctx.

func SetWorkers

func SetWorkers(ctx context.Context, workers int)

SetWorkers records the effective worker count for the tracker attached to ctx.

func WorkerFromContext

func WorkerFromContext(ctx context.Context) int

WorkerFromContext returns the current insert operation worker index.

Types

type Config

type Config struct {
	Enabled    bool
	Interval   time.Duration
	StallAfter time.Duration
	Mode       Mode
	Table      string
	Method     string
	Workers    int
	Logger     *zap.Logger
	OnSample   func(Snapshot)
}

Config defines one insert operation progress tracker.

func DefaultConfig

func DefaultConfig() Config

DefaultConfig returns the default insert operation progress configuration.

type Event

type Event string

Event describes the lifecycle stage of a progress sample.

const (
	// EventProgress is a regular interval sample.
	EventProgress Event = "progress"
	// EventStall is an interval sample emitted after no row progress for StallAfter.
	EventStall Event = "stall"
	// EventCompleted is the final sample for a successful insert operation.
	EventCompleted Event = "completed"
	// EventFailed is the final sample for a failed insert operation.
	EventFailed Event = "failed"
)

type Mode

type Mode string

Mode controls where insert operation progress samples are emitted.

const (
	// ModeOff disables the insert operation progress watcher.
	ModeOff Mode = "off"
	// ModeLog emits progress samples to the logger.
	ModeLog Mode = "log"
	// ModeMetrics emits progress samples to the metrics callback.
	ModeMetrics Mode = "metrics"
	// ModeBoth emits progress samples to both logger and metrics callback.
	ModeBoth Mode = "both"
)

func ParseMode

func ParseMode(raw string) (Mode, error)

ParseMode parses a user-facing progress mode.

type RowCounter

type RowCounter struct {
	// contains filtered or unexported fields
}

RowCounter batches per-row progress updates to avoid an atomic operation per generated row.

func NewConfirmedRowCounter

func NewConfirmedRowCounter(ctx context.Context) RowCounter

NewConfirmedRowCounter returns a row counter that reports confirmed rows.

func NewGeneratedRowCounter

func NewGeneratedRowCounter(ctx context.Context) RowCounter

NewGeneratedRowCounter returns a row counter that reports generated rows.

func (*RowCounter) Add

func (counter *RowCounter) Add(rows int64)

Add records rows and flushes to the tracker when the local buffer is large enough.

func (*RowCounter) Flush

func (counter *RowCounter) Flush()

Flush sends any pending rows to the tracker.

type Snapshot

type Snapshot struct {
	Event                Event
	Table                string
	Method               string
	RowKind              string
	Stage                string
	Workers              int
	TotalRows            int64
	GeneratedRows        int64
	ConfirmedRows        int64
	Rows                 int64
	InflightRows         int64
	DeltaRows            int64
	Batches              int64
	Elapsed              time.Duration
	Interval             time.Duration
	StallDuration        time.Duration
	ETA                  time.Duration
	AvgBatchDuration     time.Duration
	CurrentRowsPerSecond float64
	AvgRowsPerSecond     float64
	Percent              float64
	Stalled              bool
}

Snapshot is one progress sample emitted by the watcher.

type Tracker

type Tracker struct {
	// contains filtered or unexported fields
}

Tracker stores mutable insert operation progress state and owns the watcher goroutine.

func FromContext

func FromContext(ctx context.Context) *Tracker

FromContext returns the insert operation progress tracker attached to ctx, if any.

func NewTracker

func NewTracker(config *Config) *Tracker

NewTracker creates an insert operation progress tracker.

func (*Tracker) AddBatch

func (tracker *Tracker) AddBatch(_ int, rows int64, elapsed time.Duration)

AddBatch records one completed driver write unit.

func (*Tracker) AddConfirmed

func (tracker *Tracker) AddConfirmed(_ int, rows int64)

AddConfirmed records rows accepted by a completed driver write unit.

func (*Tracker) AddGenerated

func (tracker *Tracker) AddGenerated(_ int, rows int64)

AddGenerated records rows produced by datagen/runtime.

func (*Tracker) Enabled

func (tracker *Tracker) Enabled() bool

Enabled reports whether this tracker should emit progress samples.

func (*Tracker) Finish

func (tracker *Tracker) Finish(operationErr error) Snapshot

Finish stops the watcher and emits a final sample.

func (*Tracker) SetStage

func (tracker *Tracker) SetStage(workerIndex int, stage string)

SetStage records the current stage for one insert operation worker.

func (*Tracker) SetTotal

func (tracker *Tracker) SetTotal(rows int64)

SetTotal records the actual runtime row count for this insert operation.

func (*Tracker) SetWorkers

func (tracker *Tracker) SetWorkers(workers int)

SetWorkers records the effective worker count for this insert operation.

func (*Tracker) Start

func (tracker *Tracker) Start(ctx context.Context)

Start starts the watcher goroutine.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL