common

package
v6.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 20, 2026 License: Apache-2.0 Imports: 8 Imported by: 0

Documentation

Overview

Package common hosts driver-agnostic building blocks shared by every Stroppy database driver. The within-table parallel insert orchestrator lives here so pg, mysql, native, and future drivers stay free of their own chunking and worker lifecycle logic.

Package common hosts driver-agnostic building blocks shared by every Stroppy database driver. The within-table parallel insert orchestrator lives here so pg, mysql, native, and future drivers stay free of their own chunking and worker lifecycle logic.

Index

Constants

This section is empty.

Variables

View Source
var ErrNilBatchFn = errors.New("common: RunParallelBatch requires a non-nil BatchFn")

ErrNilBatchFn is returned when the per-chunk callback is nil.

View Source
var ErrNilBatchSource = errors.New("common: RunParallelBatch requires a non-nil source")

ErrNilBatchSource is returned by the typed parallel helpers when the gen.BatchSource argument is nil.

Functions

func RunParallelBatch

func RunParallelBatch(
	ctx context.Context,
	src gen.BatchSource,
	workers int,
	batchRows int,
	fn BatchFn,
) (int64, error)

RunParallelBatch carves src into `workers` contiguous chunks by entity index, prepares each partition's cursor, and drains them concurrently through fn. It returns the source's total row count (which may differ from Units for fan-out generators).

batchRows is the per-batch row capacity passed to gen.BatchSource.Prepare; each cursor allocates one batch of that capacity and refills it on every Next, so generation after preparation allocates nothing.

Types

type BatchFn

type BatchFn func(ctx context.Context, chunk Chunk, cur gen.Cursor) error

BatchFn consumes one prepared partition's cursor. The cursor is already positioned at chunk.Start and bounded to chunk.Count rows, so the callback drains it to io.EOF. It must honor ctx.Done: the parallel helpers cancel sibling workers on the first error.

type BatchRowSource

type BatchRowSource struct {
	// contains filtered or unexported fields
}

BatchRowSource adapts a gen.Cursor (typed columnar batches) to the legacy source.RowSource interface (one []any row per Next) so the existing per-driver drain logic — COPY, multi-row INSERT, BulkUpsert, CSV encode, noop discard — runs unchanged on the typed path.

It is a coexistence bridge: the typed generation layer produces reusable batches, and this adapter materializes one row at a time into a reused []any scratch slice for the driver's encoding layer. Driver- side encoding (the row copy into a driver batch, dialect.Convert, the actual network write) may allocate; generation before this adapter does not. When the legacy source package is removed, the drain logic is rewritten to consume *gen.Batch directly and this adapter goes too.

The scratch slice is returned from Next and overwritten on the next call; drain code that needs a row across calls must copy it (the existing drain paths already do).

func NewBatchRowSource

func NewBatchRowSource(cur gen.Cursor, columns []string, nCols int) *BatchRowSource

NewBatchRowSource returns a RowSource backed by cur. columns is the per-emission-order column name list (from gen.Schema.ColumnNames); nCols is the column count, used once to size the scratch slice.

func (*BatchRowSource) Columns

func (s *BatchRowSource) Columns() []string

Columns returns the emission-order column names.

func (*BatchRowSource) Next

func (s *BatchRowSource) Next() ([]any, error)

Next pulls the next row from the cursor's current batch, refilling from the cursor when exhausted, and materializes it into the scratch slice. Returns (nil, io.EOF) when the partition is drained.

type Chunk

type Chunk struct {
	Index int
	Start int64
	Count int64
}

Chunk describes one worker's slice of a population's row range. Start is inclusive; Count is the number of rows the worker must emit. Index identifies the worker for logging and error attribution and runs from 0 to len(chunks)-1.

func SplitChunks

func SplitChunks(total int64, workers int) []Chunk

SplitChunks carves the row range [0, total) into exactly max(workers, 1) contiguous chunks. Every chunk has floor(total/workers) rows except the last, which absorbs the remainder so the total count is preserved exactly.

total == 0 yields a single zero-count chunk: this lets callers treat empty populations uniformly without a special-case branch.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL