gen

package
v6.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 20, 2026 License: Apache-2.0 Imports: 7 Imported by: 0

Documentation

Overview

Package gen provides deterministic data-generation primitives for stroppy workloads.

A value generated by this package is a pure function of a root seed, a semantic domain name, a field name, an entity index, and (for values that need more than one 64-bit word) a sub-draw index. There is no mutable per-draw state and no shared global PRNG: any entity of any field is reachable in O(1) without affecting any other entity.

Coordinates

The derivation hierarchy is:

Root   = New(seed)
Domain = root.Domain("tpcc/item@1")   // workload-owned, versioned name
Field  = domain.Field("i_im_id")      // one logical column's stream
Draw   = field.At(entityIndex)        // cursor for one entity's draws

Domain names are chosen by the workload and carry an explicit version suffix (the "@1" above). Renaming or renumbering a domain changes the data; inserting an unrelated domain does not. Field names partition one domain's stream so editing one field never shifts another field's sequence.

Worker identity, chunk boundaries, batch sizes, and the executor step name never enter the coordinate. A dataset generated with workers=N is therefore byte-identical to the same dataset generated with workers=1, and resuming a load at entity E produces the same suffix as a full run from entity E.

Seeds

A seed of 0 is a valid, deterministic seed. Operators wanting a non-deterministic run pass a random seed explicitly; the package never silently substitutes one.

Allocation

Preparation (constructing Root, Domain, Field, and prepared batches and tables) may allocate. Generating a value at a coordinate — scalar draws, alphabet fills, weighted picks over prepared tables — allocates nothing on the hot path.

Example

Example models three columns of a TPC-C-style item row with plain imperative Go. The formula reads top-to-bottom with no seed plumbing, no RNG object, and no positional column index: field namespaces are derived once, scalars are pure one-liners, and variable-length text is drawn and filled into a caller-owned buffer in one call.

package main

import (
	"fmt"

	"github.com/stroppy-io/stroppy/v6/pkg/gen"
)

func main() {
	root := gen.New(0xC0FFEE)
	items := root.Domain("tpcc/item@1")
	imID := items.Field("i_im_id")
	name := items.Field("i_name")
	price := items.Field("i_price")

	var buf [24]byte

	for _, id := range []uint64{0, 1, 2} {
		im := imID.Int64(id, 1, 10000)
		nd := name.At(id)
		n := gen.AlphaNumeric.Bytes(&nd, buf[:], 14, 24)
		pr := price.Decimal(id, 1, 100, 2)
		fmt.Printf("id=%d im_id=%d name=%q price=%.2f\n", id+1, im, buf[:n], pr)
	}
}
Output:
id=1 im_id=4539 name="dgHhw9Gvmr6DowPtD" price=40.03
id=2 im_id=4504 name="abMjYAOwfPIrKkiHoP" price=74.11
id=3 im_id=3428 name="GHx2EYJZBrRA1CUpqEO" price=90.34

Index

Examples

Constants

This section is empty.

Variables

View Source
var (
	ErrBytesNegative = errors.New("gen: bytes column negative length")
	ErrBytesOverflow = errors.New("gen: bytes column length exceeds budget")
)

Sentinel errors for bytes-column length violations. Defined statically so callers can errors.Is them; the dynamic detail (column name, length) is added at the return site by wrapping.

View Source
var (
	// ASCII is the 94 printable ASCII characters [asciiPrintableLo, asciiPrintableHi].
	ASCII = NewASCII(asciiPrintableLo, asciiPrintableHi)
	// AlphaNumeric is [A-Za-z0-9], the TPC-C a-string alphabet.
	AlphaNumeric = NewAlphabet("ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789")
	// Numeric is [0-9], the TPC-C n-string alphabet.
	Numeric = NewAlphabet("0123456789")
	// Alpha is [A-Za-z].
	Alpha = NewAlphabet("ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz")
	// AlphaUpper is [A-Z].
	AlphaUpper = NewAlphabet("ABCDEFGHIJKLMNOPQRSTUVWXYZ")
)
View Source
var ErrEmptySchema = errors.New("gen: empty schema")

ErrEmptySchema is returned by a driver when an InsertRequest carries a schema with zero columns.

View Source
var ErrPrepareRange = errors.New("gen: prepare start out of range")

ErrPrepareRange is returned when Prepare's start is outside [0, TotalRows).

Functions

func Permute

func Permute(seedVal, idx, n int64) (int64, error)

Permute returns the image of idx under a bijective permutation of [0, n) that is deterministic in (seed, n). Iterating idx across [0, n) yields every element of [0, n) exactly once; different seeds yield uncorrelated permutations. Stateless by construction — parallel workers may call it for disjoint idx ranges without coordination.

func SplitMix64

func SplitMix64(x uint64) uint64

SplitMix64 is the full splitmix64 bit-mixer (5 XORs + 2 multiplies), the single mixing primitive for both derivation and the counter core. It is a bijection with BigCrush-passing avalanche. Exported so imperative generators can mix a derived key directly (for example TPC-C NURand).

Types

type Alphabet

type Alphabet struct {
	// contains filtered or unexported fields
}

Alphabet is an immutable byte alphabet for Fill and Bytes. Construct one with NewAlphabet at plan time; picking a byte at any position allocates nothing. Alphabets are safe to share across goroutines.

func NewASCII

func NewASCII(lo, hi byte) Alphabet

NewASCII returns an alphabet of every printable byte in [lo, hi] inclusive. lo > hi panics.

func NewAlphabet

func NewAlphabet(s string) Alphabet

NewAlphabet returns an alphabet over s. s must be non-empty; an empty alphabet panics because a fill from it has no valid output. The bytes are copied so later mutation of s does not affect the alphabet.

func (Alphabet) Bytes

func (a Alphabet) Bytes(d *Draw, dst []byte, minLen, maxLen int) int

Bytes draws a length uniformly in [minLen, maxLen] inclusive, fills dst[:n] from a, and returns n. It draws the length at the cursor's next sub-draw and each position at successive sub-draws, so a single field yields both a random length and its content in one sequenced pass. dst must have room for at least maxLen bytes. minLen > maxLen yields minLen; minLen < 0 is clamped to 0.

func (Alphabet) Fill

func (a Alphabet) Fill(d *Draw, dst []byte) []byte

Fill writes len(dst) bytes drawn from a into dst, one sub-draw per position, and returns dst unchanged. dst is caller-owned and controls the length: draw the length separately (from another field), slice dst, then fill. Exactly uniform via rejection sampling.

type Batch

type Batch struct {
	// contains filtered or unexported fields
}

Batch is a reusable, typed, columnar batch of generated rows. It is allocated once when a Cursor is prepared and refilled on every Cursor.Next call; generation after preparation allocates nothing.

A driver must consume or copy a batch's contents before requesting the next one: the same backing storage is reused. Row is the author-facing facade over one row of a batch.

func NewBatch

func NewBatch(schema Schema, capacity int) *Batch

NewBatch allocates a batch for schema with row capacity cap. Called once at cursor preparation; subsequent fills reuse it. It is exported for direct tests and ad-hoc cursors, but production sources obtain batches through [Cursor.Prepare].

func (*Batch) AddRow

func (b *Batch) AddRow() Row

AddRow reserves the next unfilled row slot, advances the filled count by one, and returns a facade over it. It is the stateful-cursor entry point for generators that fill rows one at a time from a non-indexed source (for example a canonical dbgen stream): the caller drains one upstream row, calls AddRow, and writes its cells through the returned Row before the next call. Panics if the batch is already full; callers bound the loop with Batch.Len < Batch.Cap.

func (*Batch) Cap

func (b *Batch) Cap() int

Cap reports the batch's row capacity.

func (*Batch) Len

func (b *Batch) Len() int

Len reports the number of rows currently filled in this batch.

func (*Batch) MaterializeRow

func (b *Batch) MaterializeRow(i int, dst []any)

MaterializeRow writes row i's column values into dst, which must have length at least Schema.Columns. It is the bridge from typed columnar storage to the []any-per-row shape the driver encoding layer consumes: generation stays allocation-free inside a cursor, and materialization (this call, on the driver boundary) is allowed to allocate.

NULL cells are written as untyped nil. Scalar kinds copy by value. A bytes column is written as a string (the text representation every dialect driver expects for variable-length columns; binary bytea is not represented by the current kind set). The string is freshly allocated so dst outlives the batch slab, which a cursor may reuse for the next batch before the driver flushes.

i must be a valid row index in [0, b.Len()); the caller (the driver adapter) guarantees this by draining the cursor in order.

func (*Batch) Reset

func (b *Batch) Reset()

Reset clears the batch to zero rows, preserving capacity. Null flags are zeroed (every cell starts non-NULL); byte offsets are reset so the next fill rewrites the slab from the top. Generated content is overwritten, so stale data beyond the new length is not visible.

func (*Batch) Row

func (b *Batch) Row(i int) Row

Row returns a facade over row i of the batch. The facade is a small stack value; setters write through it into the batch's backing storage.

type BatchSource

type BatchSource interface {
	Schema() Schema
	Units() int64
	TotalRows() int64
	Prepare(start, count int64, batchRows int) (Cursor, error)
}

BatchSource is the driver-independent contract for a typed, partitionable row generator. It is the typed successor to the legacy pkg/datagen/source.Partitionable: it produces reusable Batch values instead of []any rows, so generation after preparation allocates nothing.

Units is the number of partitionable units (entities); the loader carves [0, Units) into chunks. TotalRows is the number of output rows the source emits in full, used for progress and stats; for a fan-out source it may be an estimate (as the legacy contract allows). Prepare returns a Cursor positioned at start, bounded to count units.

Implementations must be safe to Prepare and drain concurrently across disjoint ranges.

type Column

type Column struct {
	// contains filtered or unexported fields
}

Column is a name-bound handle into a Schema. A workload binds handles once at plan time and passes them to Row setters; the handle is a small immutable value safe to copy.

type Cursor

type Cursor interface {
	Next() (*Batch, error)
}

Cursor fills reusable batches from one prepared partition. Next writes the next batch (up to the configured batchRows) into the same Batch backing storage and returns it; it returns (nil, io.EOF) when the partition is exhausted. A driver must consume or copy the returned batch before the next call.

type Domain

type Domain struct {
	// contains filtered or unexported fields
}

Domain is a namespace within a Root. Fields derived from a domain are independent of fields derived from any other domain or from the root.

The zero Domain is a valid domain for the empty name under seed 0.

func (Domain) Field

func (d Domain) Field(name string) Field

Field derives a per-field stream from the domain. The same (domain, name) pair always yields the same field; distinct names yield independent streams. Editing one field's name never shifts another field's sequence (the way a hand-numbered stream-id list would).

Field is immutable and safe to copy and share across goroutines.

type Draw

type Draw struct {
	// contains filtered or unexported fields
}

Draw is a stack-only cursor for the values of one field at one entity.

Multi-word kernels (Fill, Bytes) advance the cursor's sub-draw across positions; a Draw is therefore not safe for concurrent use by multiple goroutines. Copy it per goroutine (copies are cheap and independent).

type Field

type Field struct {
	// contains filtered or unexported fields
}

Field is a derived, read-only random sequence scoped to one domain and one logical field. It is a small immutable value: safe to copy and share.

One-word kernels (Uint64, Int64, Int, Float64, Decimal, Chance) are pure methods on Field taking an explicit entity index; no cursor, no mutable state. Multi-word kernels (Fill, Bytes) advance a *Draw cursor across the entity's sub-draws; use them for a single field that must yield more than one word (a filled string). Do not mix the two on one field: a one-word kernel draws on a local copy and does not advance the caller's cursor, so draw a random length from a separate field, or use Bytes which draws the length and fills in one sequenced pass.

func (Field) At

func (f Field) At(entity uint64) Draw

At returns a Draw positioned at entity. A Draw is the stack-only cursor a multi-word kernel uses to take several values for a single entity of this field. Distinct entities of the same field are independent; seeking is selecting a different entity, not replaying prior draws.

func (Field) Chance

func (f Field) Chance(entity uint64, p float64) bool

Chance reports a Bernoulli trial for entity: true with probability p in [0, 1]. p <= 0 is always false; p >= 1 is always true.

func (Field) Decimal

func (f Field) Decimal(entity uint64, lo, hi float64, scale int) float64

Decimal returns a float64 uniformly in [lo, hi], rounded to scale decimal places, for entity. The draw is over the integer range [round(lo*10^scale), round(hi*10^scale)] (exactly uniform via rejection sampling) and scaled back down, so every value lands exactly on a representable decimal at the given scale — no float rounding noise in the generated data and no reliance on the DB column to truncate. scale <= 0 yields whole-number draws over [round(lo), round(hi)].

func (Field) Float64

func (f Field) Float64(entity uint64) float64

Float64 returns a float64 uniformly in [0, 1) for entity.

func (Field) Int

func (f Field) Int(entity uint64, lo, hi int) int

Int returns an int uniformly in [lo, hi] inclusive for entity. hi < lo yields lo.

func (Field) Int64

func (f Field) Int64(entity uint64, lo, hi int64) int64

Int64 returns an int64 uniformly in [lo, hi] inclusive for entity. hi < lo yields lo. The draw is exactly uniform via rejection sampling.

func (Field) Uint64

func (f Field) Uint64(entity uint64) uint64

Uint64 returns a raw 64-bit value for entity (sub 0).

type IndexedSource

type IndexedSource struct {
	// contains filtered or unexported fields
}

IndexedSource is a BatchSource for pure f(seed, entity) -> row generation: every row is a pure function of an entity index, so any entity is reachable in O(1), partitions are independent, and worker-count changes do not alter the dataset. Construct with NewIndexedSource.

func FromRows

func FromRows[T any](totalRows int64, fn func(uint64) (T, error), options ...RowsOption) *IndexedSource

FromRows adapts an indexed ordinary struct function into reusable typed batches. Callback content must be independent of partitioning. Arbitrary callback allocations are permitted; strings/bytes are copied into bounded batch storage.

func NewIndexedSource

func NewIndexedSource(
	schema Schema, root Root, domain string, totalRows int64, batchRows int, fn RowFunc,
) *IndexedSource

NewIndexedSource returns a source over schema whose rows are produced by fn at entity indices [0, totalRows). totalRows must be >= 0. batchRows is the per-batch row capacity each cursor allocates once; 0 clamps to 1. The domain name is the versioned namespace (for example "tpcc/item@1") under which fn's fields derive; it must be workload-owned and stable.

func (*IndexedSource) Prepare

func (s *IndexedSource) Prepare(start, count int64, batchRows int) (Cursor, error)

Prepare returns a cursor over entities [start, start+count). count < 0 means "from start to the end". The cursor allocates one Batch of batchRows capacity; subsequent fills allocate nothing.

func (*IndexedSource) Schema

func (s *IndexedSource) Schema() Schema

Schema returns the source's schema.

func (*IndexedSource) TotalRows

func (s *IndexedSource) TotalRows() int64

TotalRows returns the row count.

func (*IndexedSource) Units

func (s *IndexedSource) Units() int64

Units returns the entity count; for an indexed source Units == TotalRows.

type Kind

type Kind int

Kind is the concrete storage type of a generated column. A column's kind fixes which Row setter applies to it; a bytes column also carries a per-row byte budget.

const (
	KindInt64 Kind = iota
	KindFloat64
	KindBool
	KindBytes
	KindTime
)

func (Kind) String

func (k Kind) String() string

String reports the kind name, for schema errors and probe output.

type Root

type Root struct {
	// contains filtered or unexported fields
}

Root is the deterministic base for a run. It is a small immutable value: it is safe to copy and to share across goroutines.

Construct one with New at the start of a workload and derive domains and fields from it. The zero Root is a valid deterministic root for seed 0.

func New

func New(seed uint64) Root

New returns the Root for seed. seed 0 is valid and deterministic.

func (Root) Domain

func (r Root) Domain(name string) Domain

Domain derives a named namespace from the root. The same (root, name) pair always yields the same domain, independent of declaration order or of how many other domains exist. name should be workload-owned and carry a version suffix (for example "tpcc/item@1") so that a workload's data is stable across execution-graph edits and so an intentional formula change can increment the version.

Domain is immutable and safe to copy and share.

func (Root) Seed

func (r Root) Seed() uint64

Seed returns the root seed this Root was constructed from.

type Row

type Row struct {
	// contains filtered or unexported fields
}

Row is the author-facing facade over one row of a Batch. A workload's row callback receives a *Row and writes each column through a bound Column handle; no seed, RNG, buffer, or positional index appears at the call site.

func (Row) BoolVal

func (r Row) BoolVal(c Column) bool

BoolVal reads back the bool column c.

func (Row) Bytes

func (r Row) Bytes(c Column, n int) ([]byte, error)

Bytes returns the destination slice for the bytes column c at this row, of length exactly n and backed by the batch's slab. The caller writes into it directly (e.g. via Alphabet.Fill) and n becomes the stored length. n must not exceed the column's maxBytes; if it does, Bytes returns nil and an error rather than growing the slab (which would allocate).

The returned slice aliases the batch slab; it is valid only until the next Cursor.Next or Batch.Reset on the owning batch.

func (Row) BytesLen

func (r Row) BytesLen(c Column) int

BytesLen returns the stored length of bytes column c at this row.

func (Row) BytesVal

func (r Row) BytesVal(c Column) []byte

BytesVal returns the stored bytes slice for column c (valid until the next fill/reset), without copying.

func (Row) ClearNull

func (r Row) ClearNull(c Column)

ClearNull marks column c non-NULL (the default). Useful when a batch is reused across fills and a row that was NULL in one fill is non-NULL in the next.

func (Row) Float64Val

func (r Row) Float64Val(c Column) float64

Float64Val reads back the float64 column c.

func (Row) Int64Val

func (r Row) Int64Val(c Column) int64

Int64Val reads back the int64 column c at this row (for tests and drivers).

func (Row) IsNull

func (r Row) IsNull(c Column) bool

IsNull reports whether column c is NULL for this row.

func (Row) SetBool

func (r Row) SetBool(c Column, v bool)

SetBool writes v to the bool column c.

func (Row) SetFloat64

func (r Row) SetFloat64(c Column, v float64)

SetFloat64 writes v to the float64 column c.

func (Row) SetInt64

func (r Row) SetInt64(c Column, v int64)

SetInt64 writes v to the int64 column c at this row.

func (Row) SetNull

func (r Row) SetNull(c Column)

SetNull marks column c NULL for this row. A NULL overrides any value set for the same column in the same row; drivers read the null flag before the typed value.

For bytes columns it also reserves zero bytes at this row's offset, so the sequential byte-offset invariant (boff[R+1] == boff[R] + len(R)) holds even when a row writes no bytes; otherwise the next non-null row would read a stale start offset.

func (Row) SetTime

func (r Row) SetTime(c Column, v time.Time)

SetTime writes v to the time column c.

func (Row) TimeVal

func (r Row) TimeVal(c Column) time.Time

TimeVal reads back the time column c.

type RowFunc

type RowFunc func(r Row, entity uint64) error

RowFunc generates one row at entity into r. It is the author-facing entry point for an indexed source: a pure function of (root-derived fields, entity) that writes through bound Column handles. It must not depend on worker identity, chunk boundary, or call order.

r is passed by value: it carries a pointer to the batch's backing storage, so setters persist; passing it by value (not pointer) keeps Row on the stack and the hot path allocation-free.

type RowsOption

type RowsOption func(*rowsOptions)

RowsOption bounds storage used by an ordinary struct-row adapter.

func BatchRows

func BatchRows(n int) RowsOption

func MaxBytes

func MaxBytes(n int) RowsOption

MaxBytes bounds each variable-length column per row; default is 4096.

type Schema

type Schema struct {
	// contains filtered or unexported fields
}

Schema is an immutable, ordered set of column definitions. Column handles (Column) are positions into this schema; binding them once at plan time keeps row formulas free of positional numbers and hot-path name lookups.

Build one with [NewSchema].

func (Schema) ColumnNames

func (s Schema) ColumnNames() []string

ColumnNames reports the emission-order names, for a driver's column list.

func (Schema) Columns

func (s Schema) Columns() int

Columns reports the column count.

type SchemaBuilder

type SchemaBuilder struct {
	// contains filtered or unexported fields
}

SchemaBuilder accumulates column definitions.

func NewSchemaBuilder

func NewSchemaBuilder() *SchemaBuilder

NewSchemaBuilder returns a builder that appends columns in emission order and validates names. Call SchemaBuilder.Build to freeze the schema.

func (*SchemaBuilder) Bool

func (b *SchemaBuilder) Bool(name string) Column

Bool declares a bool column.

func (*SchemaBuilder) Build

func (b *SchemaBuilder) Build() Schema

Build freezes the schema. A schema with zero columns is valid (an empty load) but useless.

func (*SchemaBuilder) Bytes

func (b *SchemaBuilder) Bytes(name string, maxBytes int) Column

Bytes declares a variable-length bytes column with a per-row byte budget maxBytes. A row may store fewer bytes; storing more returns an error.

func (*SchemaBuilder) Float64

func (b *SchemaBuilder) Float64(name string) Column

Float64 declares a float64 column.

func (*SchemaBuilder) Int64

func (b *SchemaBuilder) Int64(name string) Column

Int64 declares an int64 column and returns its bound handle.

func (*SchemaBuilder) Time

func (b *SchemaBuilder) Time(name string) Column

Time declares a time.Time column.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL