Documentation
¶
Overview ¶
Package gen provides deterministic data-generation primitives for stroppy workloads.
A value generated by this package is a pure function of a root seed, a semantic domain name, a field name, an entity index, and (for values that need more than one 64-bit word) a sub-draw index. There is no mutable per-draw state and no shared global PRNG: any entity of any field is reachable in O(1) without affecting any other entity.
Coordinates ¶
The derivation hierarchy is:
Root = New(seed)
Domain = root.Domain("tpcc/item@1") // workload-owned, versioned name
Field = domain.Field("i_im_id") // one logical column's stream
Draw = field.At(entityIndex) // cursor for one entity's draws
Domain names are chosen by the workload and carry an explicit version suffix (the "@1" above). Renaming or renumbering a domain changes the data; inserting an unrelated domain does not. Field names partition one domain's stream so editing one field never shifts another field's sequence.
Worker identity, chunk boundaries, batch sizes, and the executor step name never enter the coordinate. A dataset generated with workers=N is therefore byte-identical to the same dataset generated with workers=1, and resuming a load at entity E produces the same suffix as a full run from entity E.
Seeds ¶
A seed of 0 is a valid, deterministic seed. Operators wanting a non-deterministic run pass a random seed explicitly; the package never silently substitutes one.
Allocation ¶
Preparation (constructing Root, Domain, Field, and prepared batches and tables) may allocate. Generating a value at a coordinate — scalar draws, alphabet fills, weighted picks over prepared tables — allocates nothing on the hot path.
Example ¶
Example models three columns of a TPC-C-style item row with plain imperative Go. The formula reads top-to-bottom with no seed plumbing, no RNG object, and no positional column index: field namespaces are derived once, scalars are pure one-liners, and variable-length text is drawn and filled into a caller-owned buffer in one call.
package main
import (
"fmt"
"github.com/stroppy-io/stroppy/v6/pkg/gen"
)
func main() {
root := gen.New(0xC0FFEE)
items := root.Domain("tpcc/item@1")
imID := items.Field("i_im_id")
name := items.Field("i_name")
price := items.Field("i_price")
var buf [24]byte
for _, id := range []uint64{0, 1, 2} {
im := imID.Int64(id, 1, 10000)
nd := name.At(id)
n := gen.AlphaNumeric.Bytes(&nd, buf[:], 14, 24)
pr := price.Decimal(id, 1, 100, 2)
fmt.Printf("id=%d im_id=%d name=%q price=%.2f\n", id+1, im, buf[:n], pr)
}
}
Output: id=1 im_id=4539 name="dgHhw9Gvmr6DowPtD" price=40.03 id=2 im_id=4504 name="abMjYAOwfPIrKkiHoP" price=74.11 id=3 im_id=3428 name="GHx2EYJZBrRA1CUpqEO" price=90.34
Index ¶
- Variables
- func Permute(seedVal, idx, n int64) (int64, error)
- func SplitMix64(x uint64) uint64
- type Alphabet
- type Batch
- type BatchSource
- type Column
- type Cursor
- type Domain
- type Draw
- type Field
- func (f Field) At(entity uint64) Draw
- func (f Field) Chance(entity uint64, p float64) bool
- func (f Field) Decimal(entity uint64, lo, hi float64, scale int) float64
- func (f Field) Float64(entity uint64) float64
- func (f Field) Int(entity uint64, lo, hi int) int
- func (f Field) Int64(entity uint64, lo, hi int64) int64
- func (f Field) Uint64(entity uint64) uint64
- type IndexedSource
- type Kind
- type Root
- type Row
- func (r Row) BoolVal(c Column) bool
- func (r Row) Bytes(c Column, n int) ([]byte, error)
- func (r Row) BytesLen(c Column) int
- func (r Row) BytesVal(c Column) []byte
- func (r Row) ClearNull(c Column)
- func (r Row) Float64Val(c Column) float64
- func (r Row) Int64Val(c Column) int64
- func (r Row) IsNull(c Column) bool
- func (r Row) SetBool(c Column, v bool)
- func (r Row) SetFloat64(c Column, v float64)
- func (r Row) SetInt64(c Column, v int64)
- func (r Row) SetNull(c Column)
- func (r Row) SetTime(c Column, v time.Time)
- func (r Row) TimeVal(c Column) time.Time
- type RowFunc
- type RowsOption
- type Schema
- type SchemaBuilder
- func (b *SchemaBuilder) Bool(name string) Column
- func (b *SchemaBuilder) Build() Schema
- func (b *SchemaBuilder) Bytes(name string, maxBytes int) Column
- func (b *SchemaBuilder) Float64(name string) Column
- func (b *SchemaBuilder) Int64(name string) Column
- func (b *SchemaBuilder) Time(name string) Column
Examples ¶
Constants ¶
This section is empty.
Variables ¶
var ( ErrBytesNegative = errors.New("gen: bytes column negative length") ErrBytesOverflow = errors.New("gen: bytes column length exceeds budget") )
Sentinel errors for bytes-column length violations. Defined statically so callers can errors.Is them; the dynamic detail (column name, length) is added at the return site by wrapping.
var ( // ASCII is the 94 printable ASCII characters [asciiPrintableLo, asciiPrintableHi]. ASCII = NewASCII(asciiPrintableLo, asciiPrintableHi) // AlphaNumeric is [A-Za-z0-9], the TPC-C a-string alphabet. AlphaNumeric = NewAlphabet("ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789") // Numeric is [0-9], the TPC-C n-string alphabet. Numeric = NewAlphabet("0123456789") // Alpha is [A-Za-z]. Alpha = NewAlphabet("ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz") // AlphaUpper is [A-Z]. AlphaUpper = NewAlphabet("ABCDEFGHIJKLMNOPQRSTUVWXYZ") )
var ErrEmptySchema = errors.New("gen: empty schema")
ErrEmptySchema is returned by a driver when an InsertRequest carries a schema with zero columns.
var ErrPrepareRange = errors.New("gen: prepare start out of range")
ErrPrepareRange is returned when Prepare's start is outside [0, TotalRows).
Functions ¶
func Permute ¶
Permute returns the image of idx under a bijective permutation of [0, n) that is deterministic in (seed, n). Iterating idx across [0, n) yields every element of [0, n) exactly once; different seeds yield uncorrelated permutations. Stateless by construction — parallel workers may call it for disjoint idx ranges without coordination.
func SplitMix64 ¶
SplitMix64 is the full splitmix64 bit-mixer (5 XORs + 2 multiplies), the single mixing primitive for both derivation and the counter core. It is a bijection with BigCrush-passing avalanche. Exported so imperative generators can mix a derived key directly (for example TPC-C NURand).
Types ¶
type Alphabet ¶
type Alphabet struct {
// contains filtered or unexported fields
}
Alphabet is an immutable byte alphabet for Fill and Bytes. Construct one with NewAlphabet at plan time; picking a byte at any position allocates nothing. Alphabets are safe to share across goroutines.
func NewASCII ¶
NewASCII returns an alphabet of every printable byte in [lo, hi] inclusive. lo > hi panics.
func NewAlphabet ¶
NewAlphabet returns an alphabet over s. s must be non-empty; an empty alphabet panics because a fill from it has no valid output. The bytes are copied so later mutation of s does not affect the alphabet.
func (Alphabet) Bytes ¶
Bytes draws a length uniformly in [minLen, maxLen] inclusive, fills dst[:n] from a, and returns n. It draws the length at the cursor's next sub-draw and each position at successive sub-draws, so a single field yields both a random length and its content in one sequenced pass. dst must have room for at least maxLen bytes. minLen > maxLen yields minLen; minLen < 0 is clamped to 0.
type Batch ¶
type Batch struct {
// contains filtered or unexported fields
}
Batch is a reusable, typed, columnar batch of generated rows. It is allocated once when a Cursor is prepared and refilled on every Cursor.Next call; generation after preparation allocates nothing.
A driver must consume or copy a batch's contents before requesting the next one: the same backing storage is reused. Row is the author-facing facade over one row of a batch.
func NewBatch ¶
NewBatch allocates a batch for schema with row capacity cap. Called once at cursor preparation; subsequent fills reuse it. It is exported for direct tests and ad-hoc cursors, but production sources obtain batches through [Cursor.Prepare].
func (*Batch) AddRow ¶
AddRow reserves the next unfilled row slot, advances the filled count by one, and returns a facade over it. It is the stateful-cursor entry point for generators that fill rows one at a time from a non-indexed source (for example a canonical dbgen stream): the caller drains one upstream row, calls AddRow, and writes its cells through the returned Row before the next call. Panics if the batch is already full; callers bound the loop with Batch.Len < Batch.Cap.
func (*Batch) MaterializeRow ¶
MaterializeRow writes row i's column values into dst, which must have length at least Schema.Columns. It is the bridge from typed columnar storage to the []any-per-row shape the driver encoding layer consumes: generation stays allocation-free inside a cursor, and materialization (this call, on the driver boundary) is allowed to allocate.
NULL cells are written as untyped nil. Scalar kinds copy by value. A bytes column is written as a string (the text representation every dialect driver expects for variable-length columns; binary bytea is not represented by the current kind set). The string is freshly allocated so dst outlives the batch slab, which a cursor may reuse for the next batch before the driver flushes.
i must be a valid row index in [0, b.Len()); the caller (the driver adapter) guarantees this by draining the cursor in order.
func (*Batch) Reset ¶
func (b *Batch) Reset()
Reset clears the batch to zero rows, preserving capacity. Null flags are zeroed (every cell starts non-NULL); byte offsets are reset so the next fill rewrites the slab from the top. Generated content is overwritten, so stale data beyond the new length is not visible.
type BatchSource ¶
type BatchSource interface {
Schema() Schema
Units() int64
TotalRows() int64
Prepare(start, count int64, batchRows int) (Cursor, error)
}
BatchSource is the driver-independent contract for a typed, partitionable row generator. It is the typed successor to the legacy pkg/datagen/source.Partitionable: it produces reusable Batch values instead of []any rows, so generation after preparation allocates nothing.
Units is the number of partitionable units (entities); the loader carves [0, Units) into chunks. TotalRows is the number of output rows the source emits in full, used for progress and stats; for a fan-out source it may be an estimate (as the legacy contract allows). Prepare returns a Cursor positioned at start, bounded to count units.
Implementations must be safe to Prepare and drain concurrently across disjoint ranges.
type Column ¶
type Column struct {
// contains filtered or unexported fields
}
Column is a name-bound handle into a Schema. A workload binds handles once at plan time and passes them to Row setters; the handle is a small immutable value safe to copy.
type Cursor ¶
Cursor fills reusable batches from one prepared partition. Next writes the next batch (up to the configured batchRows) into the same Batch backing storage and returns it; it returns (nil, io.EOF) when the partition is exhausted. A driver must consume or copy the returned batch before the next call.
type Domain ¶
type Domain struct {
// contains filtered or unexported fields
}
Domain is a namespace within a Root. Fields derived from a domain are independent of fields derived from any other domain or from the root.
The zero Domain is a valid domain for the empty name under seed 0.
func (Domain) Field ¶
Field derives a per-field stream from the domain. The same (domain, name) pair always yields the same field; distinct names yield independent streams. Editing one field's name never shifts another field's sequence (the way a hand-numbered stream-id list would).
Field is immutable and safe to copy and share across goroutines.
type Draw ¶
type Draw struct {
// contains filtered or unexported fields
}
Draw is a stack-only cursor for the values of one field at one entity.
Multi-word kernels (Fill, Bytes) advance the cursor's sub-draw across positions; a Draw is therefore not safe for concurrent use by multiple goroutines. Copy it per goroutine (copies are cheap and independent).
type Field ¶
type Field struct {
// contains filtered or unexported fields
}
Field is a derived, read-only random sequence scoped to one domain and one logical field. It is a small immutable value: safe to copy and share.
One-word kernels (Uint64, Int64, Int, Float64, Decimal, Chance) are pure methods on Field taking an explicit entity index; no cursor, no mutable state. Multi-word kernels (Fill, Bytes) advance a *Draw cursor across the entity's sub-draws; use them for a single field that must yield more than one word (a filled string). Do not mix the two on one field: a one-word kernel draws on a local copy and does not advance the caller's cursor, so draw a random length from a separate field, or use Bytes which draws the length and fills in one sequenced pass.
func (Field) At ¶
At returns a Draw positioned at entity. A Draw is the stack-only cursor a multi-word kernel uses to take several values for a single entity of this field. Distinct entities of the same field are independent; seeking is selecting a different entity, not replaying prior draws.
func (Field) Chance ¶
Chance reports a Bernoulli trial for entity: true with probability p in [0, 1]. p <= 0 is always false; p >= 1 is always true.
func (Field) Decimal ¶
Decimal returns a float64 uniformly in [lo, hi], rounded to scale decimal places, for entity. The draw is over the integer range [round(lo*10^scale), round(hi*10^scale)] (exactly uniform via rejection sampling) and scaled back down, so every value lands exactly on a representable decimal at the given scale — no float rounding noise in the generated data and no reliance on the DB column to truncate. scale <= 0 yields whole-number draws over [round(lo), round(hi)].
func (Field) Int ¶
Int returns an int uniformly in [lo, hi] inclusive for entity. hi < lo yields lo.
type IndexedSource ¶
type IndexedSource struct {
// contains filtered or unexported fields
}
IndexedSource is a BatchSource for pure f(seed, entity) -> row generation: every row is a pure function of an entity index, so any entity is reachable in O(1), partitions are independent, and worker-count changes do not alter the dataset. Construct with NewIndexedSource.
func FromRows ¶
func FromRows[T any](totalRows int64, fn func(uint64) (T, error), options ...RowsOption) *IndexedSource
FromRows adapts an indexed ordinary struct function into reusable typed batches. Callback content must be independent of partitioning. Arbitrary callback allocations are permitted; strings/bytes are copied into bounded batch storage.
func NewIndexedSource ¶
func NewIndexedSource( schema Schema, root Root, domain string, totalRows int64, batchRows int, fn RowFunc, ) *IndexedSource
NewIndexedSource returns a source over schema whose rows are produced by fn at entity indices [0, totalRows). totalRows must be >= 0. batchRows is the per-batch row capacity each cursor allocates once; 0 clamps to 1. The domain name is the versioned namespace (for example "tpcc/item@1") under which fn's fields derive; it must be workload-owned and stable.
func (*IndexedSource) Prepare ¶
func (s *IndexedSource) Prepare(start, count int64, batchRows int) (Cursor, error)
Prepare returns a cursor over entities [start, start+count). count < 0 means "from start to the end". The cursor allocates one Batch of batchRows capacity; subsequent fills allocate nothing.
func (*IndexedSource) Schema ¶
func (s *IndexedSource) Schema() Schema
Schema returns the source's schema.
func (*IndexedSource) TotalRows ¶
func (s *IndexedSource) TotalRows() int64
TotalRows returns the row count.
func (*IndexedSource) Units ¶
func (s *IndexedSource) Units() int64
Units returns the entity count; for an indexed source Units == TotalRows.
type Kind ¶
type Kind int
Kind is the concrete storage type of a generated column. A column's kind fixes which Row setter applies to it; a bytes column also carries a per-row byte budget.
type Root ¶
type Root struct {
// contains filtered or unexported fields
}
Root is the deterministic base for a run. It is a small immutable value: it is safe to copy and to share across goroutines.
Construct one with New at the start of a workload and derive domains and fields from it. The zero Root is a valid deterministic root for seed 0.
func (Root) Domain ¶
Domain derives a named namespace from the root. The same (root, name) pair always yields the same domain, independent of declaration order or of how many other domains exist. name should be workload-owned and carry a version suffix (for example "tpcc/item@1") so that a workload's data is stable across execution-graph edits and so an intentional formula change can increment the version.
Domain is immutable and safe to copy and share.
type Row ¶
type Row struct {
// contains filtered or unexported fields
}
Row is the author-facing facade over one row of a Batch. A workload's row callback receives a *Row and writes each column through a bound Column handle; no seed, RNG, buffer, or positional index appears at the call site.
func (Row) Bytes ¶
Bytes returns the destination slice for the bytes column c at this row, of length exactly n and backed by the batch's slab. The caller writes into it directly (e.g. via Alphabet.Fill) and n becomes the stored length. n must not exceed the column's maxBytes; if it does, Bytes returns nil and an error rather than growing the slab (which would allocate).
The returned slice aliases the batch slab; it is valid only until the next Cursor.Next or Batch.Reset on the owning batch.
func (Row) BytesVal ¶
BytesVal returns the stored bytes slice for column c (valid until the next fill/reset), without copying.
func (Row) ClearNull ¶
ClearNull marks column c non-NULL (the default). Useful when a batch is reused across fills and a row that was NULL in one fill is non-NULL in the next.
func (Row) Float64Val ¶
Float64Val reads back the float64 column c.
func (Row) SetFloat64 ¶
SetFloat64 writes v to the float64 column c.
func (Row) SetNull ¶
SetNull marks column c NULL for this row. A NULL overrides any value set for the same column in the same row; drivers read the null flag before the typed value.
For bytes columns it also reserves zero bytes at this row's offset, so the sequential byte-offset invariant (boff[R+1] == boff[R] + len(R)) holds even when a row writes no bytes; otherwise the next non-null row would read a stale start offset.
type RowFunc ¶
RowFunc generates one row at entity into r. It is the author-facing entry point for an indexed source: a pure function of (root-derived fields, entity) that writes through bound Column handles. It must not depend on worker identity, chunk boundary, or call order.
r is passed by value: it carries a pointer to the batch's backing storage, so setters persist; passing it by value (not pointer) keeps Row on the stack and the hot path allocation-free.
type RowsOption ¶
type RowsOption func(*rowsOptions)
RowsOption bounds storage used by an ordinary struct-row adapter.
func BatchRows ¶
func BatchRows(n int) RowsOption
func MaxBytes ¶
func MaxBytes(n int) RowsOption
MaxBytes bounds each variable-length column per row; default is 4096.
type Schema ¶
type Schema struct {
// contains filtered or unexported fields
}
Schema is an immutable, ordered set of column definitions. Column handles (Column) are positions into this schema; binding them once at plan time keeps row formulas free of positional numbers and hot-path name lookups.
Build one with [NewSchema].
func (Schema) ColumnNames ¶
ColumnNames reports the emission-order names, for a driver's column list.
type SchemaBuilder ¶
type SchemaBuilder struct {
// contains filtered or unexported fields
}
SchemaBuilder accumulates column definitions.
func NewSchemaBuilder ¶
func NewSchemaBuilder() *SchemaBuilder
NewSchemaBuilder returns a builder that appends columns in emission order and validates names. Call SchemaBuilder.Build to freeze the schema.
func (*SchemaBuilder) Bool ¶
func (b *SchemaBuilder) Bool(name string) Column
Bool declares a bool column.
func (*SchemaBuilder) Build ¶
func (b *SchemaBuilder) Build() Schema
Build freezes the schema. A schema with zero columns is valid (an empty load) but useless.
func (*SchemaBuilder) Bytes ¶
func (b *SchemaBuilder) Bytes(name string, maxBytes int) Column
Bytes declares a variable-length bytes column with a per-row byte budget maxBytes. A row may store fewer bytes; storing more returns an error.
func (*SchemaBuilder) Float64 ¶
func (b *SchemaBuilder) Float64(name string) Column
Float64 declares a float64 column.
func (*SchemaBuilder) Int64 ¶
func (b *SchemaBuilder) Int64(name string) Column
Int64 declares an int64 column and returns its bound handle.
func (*SchemaBuilder) Time ¶
func (b *SchemaBuilder) Time(name string) Column
Time declares a time.Time column.