Documentation
¶
Overview ¶
Package common hosts driver-agnostic building blocks shared by every Stroppy database driver. The within-table parallel insert orchestrator lives here so pg, mysql, native, and future drivers stay free of their own chunking and worker lifecycle logic.
Package common hosts driver-agnostic building blocks shared by every Stroppy database driver. The within-table parallel insert orchestrator lives here so pg, mysql, native, and future drivers stay free of their own chunking and worker lifecycle logic.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ErrNilBatchFn = errors.New("common: RunParallelBatch requires a non-nil BatchFn")
ErrNilBatchFn is returned when the per-chunk callback is nil.
var ErrNilBatchSource = errors.New("common: RunParallelBatch requires a non-nil source")
ErrNilBatchSource is returned by the typed parallel helpers when the gen.BatchSource argument is nil.
Functions ¶
func RunParallelBatch ¶
func RunParallelBatch( ctx context.Context, src gen.BatchSource, workers int, batchRows int, fn BatchFn, ) (int64, error)
RunParallelBatch carves src into `workers` contiguous chunks by entity index, prepares each partition's cursor, and drains them concurrently through fn. It returns the source's total row count (which may differ from Units for fan-out generators).
batchRows is the per-batch row capacity passed to gen.BatchSource.Prepare; each cursor allocates one batch of that capacity and refills it on every Next, so generation after preparation allocates nothing.
Types ¶
type BatchFn ¶
BatchFn consumes one prepared partition's cursor. The cursor is already positioned at chunk.Start and bounded to chunk.Count rows, so the callback drains it to io.EOF. It must honor ctx.Done: the parallel helpers cancel sibling workers on the first error.
type BatchRowSource ¶
type BatchRowSource struct {
// contains filtered or unexported fields
}
BatchRowSource adapts a gen.Cursor (typed columnar batches) to the legacy source.RowSource interface (one []any row per Next) so the existing per-driver drain logic — COPY, multi-row INSERT, BulkUpsert, CSV encode, noop discard — runs unchanged on the typed path.
It is a coexistence bridge: the typed generation layer produces reusable batches, and this adapter materializes one row at a time into a reused []any scratch slice for the driver's encoding layer. Driver- side encoding (the row copy into a driver batch, dialect.Convert, the actual network write) may allocate; generation before this adapter does not. When the legacy source package is removed, the drain logic is rewritten to consume *gen.Batch directly and this adapter goes too.
The scratch slice is returned from Next and overwritten on the next call; drain code that needs a row across calls must copy it (the existing drain paths already do).
func NewBatchRowSource ¶
func NewBatchRowSource(cur gen.Cursor, columns []string, nCols int) *BatchRowSource
NewBatchRowSource returns a RowSource backed by cur. columns is the per-emission-order column name list (from gen.Schema.ColumnNames); nCols is the column count, used once to size the scratch slice.
func (*BatchRowSource) Columns ¶
func (s *BatchRowSource) Columns() []string
Columns returns the emission-order column names.
func (*BatchRowSource) Next ¶
func (s *BatchRowSource) Next() ([]any, error)
Next pulls the next row from the cursor's current batch, refilling from the cursor when exhausted, and materializes it into the scratch slice. Returns (nil, io.EOF) when the partition is drained.
type Chunk ¶
Chunk describes one worker's slice of a population's row range. Start is inclusive; Count is the number of rows the worker must emit. Index identifies the worker for logging and error attribution and runs from 0 to len(chunks)-1.
func SplitChunks ¶
SplitChunks carves the row range [0, total) into exactly max(workers, 1) contiguous chunks. Every chunk has floor(total/workers) rows except the last, which absorbs the remainder so the total count is preserved exactly.
total == 0 yields a single zero-count chunk: this lets callers treat empty populations uniformly without a special-case branch.