reflowthroughput

package
v0.4.6 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 8, 2026 License: Apache-2.0 Imports: 28 Imported by: 0

Documentation

Overview

Package reflowthroughput is a non-CI, on-demand measurement harness for transfer reflow throughput and parallelism behavior.

It is test/tooling only: it executes a built gonimbus binary as a child process, never calls Cobra command internals, and must not introduce production package seams, product record schema, or provider runtime changes.

Entry point: make test-reflow-throughput PROFILE=<name> PROVIDER=<class> Default PROFILE=smoke, PROVIDER=file (credential-free local backend).

BYO cloud transport reuses the existing non-CI lanes in test/cloudtest and make test-cloud-real (GONIMBUS_S3_TEST_* / GONIMBUS_GCS_TEST_* opt-in, unique minted prefixes, ambient credentials). Moto uses the same endpoint and test-key pattern as make test-cloud. Local/moto results are correctness evidence only; real-provider claims require PROVIDER=s3-compatible with BYO env and are labeled provider_opt_in in the report.

Index

Constants

View Source
const (
	BindFBlkMin   = 0.05 // blocked fraction floor for arm writer pressure
	BindROccMin   = 0.50 // occupancy ratio floor
	BindFBarMin   = 0.10 // barrier-wait capacity fraction floor
	BindTauE2E    = 0.15 // min |Δe2e| for material pair
	BindEpsTie    = 0.02 // |Δe2e| < ε ⇒ tie (not material)
	BindPairCount = 3
)

Frozen numeric constants (checkpoint measure bind rule — pre-registered).

View Source
const (
	DefaultRunBudget   = 25 * time.Minute
	DefaultPointBudget = 10 * time.Minute
	// DefaultCleanupAllowance leaves room after the run context expires for
	// verified teardown. At the default deletion fan-out, the observed
	// ~94,000-object teardown takes roughly 10 minutes; 15 minutes preserves
	// margin for service variance.
	DefaultCleanupAllowance = 15 * time.Minute
)

Default run and per-point budgets. These hold the smoke profile comfortably. They are overridable because they are sized for the default corpus, not for the corpus an operator asks for: a scale override that raises the object count without a matching budget expires mid-run, and the failure surfaces as a context deadline during staging rather than as a scale error.

View Source
const (
	RunBudgetEnv     = "GONIMBUS_THROUGHPUT_RUN_BUDGET"
	RunBudgetAlias   = "RUN_BUDGET"
	PointBudgetEnv   = "GONIMBUS_THROUGHPUT_POINT_TIMEOUT"
	PointBudgetAlias = "POINT_TIMEOUT"
)

Env names for the budgets, with the bare aliases the Make target also accepts.

View Source
const (
	ProviderFile         = "file"
	ProviderMoto         = "moto"
	ProviderS3Compatible = "s3-compatible"
	ProviderGCS          = "gcs"
)

Provider class names used in Options and sanitized reports.

View Source
const (
	ProfileSmoke            = "smoke"
	ProfileReflowSaturation = "reflow-saturation"
	ProfileCeilingLift      = "ceiling-lift"
	ProfileCheckpoint       = "checkpoint"
	ProfileCheckpointScale  = "checkpoint-scale"
	ProfileFullPipe         = "fullpipe-ab"
	ProfileProbeSaturation  = "probe-saturation"
)

Known profile names. Unknown names hard-reject; empty defaults to smoke.

View Source
const (
	// MemoryArmProbeBound applies no memory override — the product's detection
	// chain binds the limit (cgroup / physical RAM), and the budget is derived.
	MemoryArmProbeBound = "probe_bound"
	// MemoryArmGOMEMLIMIT constrains the limit with an operator GOMEMLIMIT,
	// which the chain selects when it is the lowest candidate.
	MemoryArmGOMEMLIMIT = "gomemlimit_constrained"
	// MemoryArmOperatorBudget sets the budget directly with --memory-budget,
	// leaving limit detection alone.
	MemoryArmOperatorBudget = "operator_budget"
)

Memory-arm labels. These name which lever bounds the run's memory, not a direction of change: the product binds the LOWEST detected limit, so an operator GOMEMLIMIT constrains the envelope and the absence of any override lets the physical-memory probe bind it.

View Source
const (
	DefaultSmokeSeed       int64 = 42
	DefaultSmokeObjects    int   = 32
	DefaultSmokeSizeBytes  int   = 256
	DefaultSmokePartitions int   = 2
)

Default smoke sizes stay cheap for local credential-free runs.

View Source
const (
	DefaultScaleObjects    int = 20000
	DefaultScaleSizeBytes  int = 256
	DefaultScalePartitions int = 8
)

Scale corpus defaults for the checkpoint-scale profile. Many tiny objects so per-object checkpoint-write cost dominates copy cost: at a lifted concurrency ceiling the single-connection checkpoint writer — not the copy path — is the binding wall, and the disk-vs-tmpfs class gap becomes measurable. Operators scale the corpus further via the OBJECT_COUNT / SIZE_BYTES / PARTITIONS overrides; these are only the starting point.

View Source
const (
	MaxRecipeObjectCount int   = 5_000_000
	MaxRecipeSizeBytes   int   = 64 << 20 // 64 MiB
	MaxRecipePartitions  int   = 4096
	MaxTotalCorpusBytes  int64 = 64 << 30 // 64 GiB: aggregate cap on materialized source corpus
)

Recipe override bounds. Overrides fail closed on absurd values rather than silently materializing a corpus that would exhaust local disk or memory. The per-dimension maxima are not independently sufficient — their product is astronomically large — so an aggregate byte bound caps the materialized corpus regardless of how the dimensions combine.

View Source
const (
	MeasurementScopeSmallScaleRelative = "small_scale_relative"
	AbsoluteEnvelopeNotEvaluated       = "not_evaluated"
)

Measurement-scope labels (public, product-safe — no private planning IDs).

View Source
const (
	ScheduleDiskFirstOdd  = "disk_first_odd"  // pair order D/T, T/D, D/T
	ScheduleTmpfsFirstOdd = "tmpfs_first_odd" // reverse: T/D, D/T, T/D
)

Frozen schedule IDs for checkpoint-scale counterbalancing.

View Source
const (
	// ServiceDemandAdmissionsPerUnit is Mark + Note + Upsert (three admissions).
	ServiceDemandAdmissionsPerUnit = 3
	// ServiceDemandAdmissionsPerArm is the formal work quantum (100k units).
	ServiceDemandAdmissionsPerArm int64 = 300_000
	// ServiceDemandMixID identifies the terminal mutation mix contract.
	ServiceDemandMixID = "mark_note_upsert_v1"
	// ServiceDemandMedium is the quorum medium for formal sets on this host class.
	ServiceDemandMedium = "apfs_disk"
	// ServiceDemandReportSchema is the sterile set-report schema version.
	ServiceDemandReportSchema = "gonimbus.service_demand_set.v1"
	// ServiceDemandProductMaxBatch is the product MaxBatch (unchanged).
	ServiceDemandProductMaxBatch = 256
)
View Source
const (
	ServiceDemandUtilMinHighN   = 0.85 // util median floor at each of 64..512
	ServiceDemandPlateauRatio   = 1.05 // mut/s@512 ≤ 1.05 × mut/s@128
	ServiceDemandBarrierMeanMul = 2.0  // mean barrier-wait/admission @512 ≥ 2× @64
	ServiceDemandMutsCVMax      = 0.10 // mut/s CV ceiling on high-N cells
	ServiceDemandDoesNotUtilMax = 0.50 // util median < 0.50 at 256 and 512
	ServiceDemandDoesNotScale   = 1.5  // mut/s increases ≥1.5× from 64→256
)

service-demand classifier constants (formal service-demand plan).

View Source
const DefaultCorpusUploadConcurrency = 16

DefaultCorpusUploadConcurrency bounds the corpus upload fan-out.

Corpus upload is setup, not measurement, but it is priced in round trips: a serial Put+Head loop costs two round trips per object and cannot exceed 1/(2*RTT) no matter how much capacity the lane has. Against a real S3 endpoint that measured ~4.5 objects/s while the same lane sustained ~85/s at concurrency 16 -- so a 50k corpus spent roughly three hours uploading before the first measurement point, and a run budget sized for the measurement expired during setup.

Concurrency here changes only how fast the fixture is staged. It does not touch the measured child, which is a separate process invoked afterwards.

View Source
const DefaultMaxRecordBytes = 4 << 20 // 4 MiB

DefaultMaxRecordBytes bounds a single JSONL line retained for inspection.

View Source
const MaxCapturedStderr = 64 << 10 // 64 KiB

MaxCapturedStderr bounds retained stderr per stage for structural diagnostics.

View Source
const RecipeVersion = "reflowthroughput-hive-v1"

RecipeVersion identifies the synthetic corpus shape for envelope comparison.

View Source
const ReportSchemaVersion = "reflowthroughput.report.v1"

ReportSchemaVersion is the test-tool report schema (not a gonimbus.* product type).

View Source
const SyntheticProbeConfigYAML = `` /* 162-byte string literal not displayed */

SyntheticProbeConfigYAML is the sterile probe config matching ObjectContent.

Variables

View Source
var ServiceDemandFormalSchedules = map[ServiceDemandScheduleID][]int{
	ServiceDemandScheduleR1: {8, 32, 64, 128, 256, 512},
	ServiceDemandScheduleR2: {512, 256, 128, 64, 32, 8},
	ServiceDemandScheduleR3: {64, 256, 8, 512, 32, 128},
}

ServiceDemandFormalSchedules are the exact three formal rep orders (Day-0 AC). Rep 1→R1, rep 2→R2, rep 3→provenance gate. Pilot rep 0 may reuse R1 and is excluded from formal classifier input.

View Source
var ServiceDemandSubmitterGrid = []int{8, 32, 64, 128, 256, 512}

ServiceDemandSubmitterGrid is the freeze-v2 concurrency steps (canonical order).

Functions

func ArmHasWriterPressure

func ArmHasWriterPressure(a ArmPressureInputs) bool

ArmHasWriterPressure implements the frozen per-arm pressure predicate. Pressure requires: StatsOK, A>=1, r_occ >= r_occ^min, and (f_blk >= f_blk^min OR f_bar >= f_bar^min).

func AssertNoSensitiveTokens

func AssertNoSensitiveTokens(s string, forbidden []string) error

AssertNoSensitiveTokens fails if s contains obvious path/bucket/credential tells. Used by the report sterility sentinel.

func BarrierWaitCapacityFraction

func BarrierWaitCapacityFraction(barrierWaitNanos, e2eWallNanos int64, effectiveConcurrency int) float64

BarrierWaitCapacityFraction is f_bar = W_b / (T_ns * C_eff).

BarrierWaitNanos is aggregate concurrent waiter-time; dividing by wall alone yields average concurrent waiters (can exceed 1). Capacity normalization by effective concurrency yields a unitless fraction of arm capacity with no silent cap. C_eff must be >= 1.

func BuildBinary

func BuildBinary(ctx context.Context, repoRoot, outPath string) (string, string, error)

BuildBinary builds ./cmd/gonimbus into outPath and returns absolute path + sha.

func CLIProbeProviderFlags

func CLIProbeProviderFlags(cfg BYOS3Config) []string

CLIProbeProviderFlags returns the provider flags `content probe` accepts.

Deliberately separate from CLIProviderFlags rather than reusing it. That helper emits the transfer command's paired --src-*/--dest-* flags, which probe does not accept: probe reads one source and has no destination. Handing the transfer set to a probe child would fail on an unknown flag, and handing it a union would give it flags that mean nothing there.

func CLIProviderFlags

func CLIProviderFlags(cfg BYOS3Config) []string

CLIProviderFlags returns transfer reflow --src-* / --dest-* args for BYO S3. Empty strings are omitted (same as release-stress real-cloud pattern).

func CheckCheckpointWriterStatsAdmission

func CheckCheckpointWriterStatsAdmission(p ParsedReflowOutput) error

CheckCheckpointWriterStatsAdmission enforces the measurement-harness fail-closed contract for retained checkpoint-scale evidence (retained-measure admission):

  • exactly one gonimbus.reflow.checkpoint_writer_stats.v1 record
  • that record appears only after the single summary
  • structural integrity of the aggregate snapshot

Product emission may remain best-effort; this gate is the strict enforcement boundary for checkpoint-scale (and any other profile that opts in).

func CheckCounts

func CheckCounts(generated int, tapRows int64, completed int64, summaryErrors int64, invalid int64) error

CheckCounts enforces corpus/tap/terminal agreement for hard structural gates.

func ChildEnv

func ChildEnv(base []string, gomemlimit string, extra ...string) []string

ChildEnv builds a per-child environment copy. It never calls os.Setenv. When gomemlimit is non-empty it replaces/sets GOMEMLIMIT on the copy only. extra pairs (e.g. moto AWS keys) are appended/replaced without ambient mutation.

func CompareLandedMultisets

func CompareLandedMultisets(a, b []LandedObjectID) error

CompareLandedMultisets returns nil when the ordered identity sets are equal.

func ContentDigest

func ContentDigest(b []byte) string

ContentDigest returns SHA-256 hex of object bytes.

func CorpusUploadConcurrency

func CorpusUploadConcurrency() int

CorpusUploadConcurrency resolves the upload fan-out, allowing an operator override for lanes that want a gentler staging rate.

func CountFilesRecursive

func CountFilesRecursive(root string) (int64, error)

CountFilesRecursive counts regular files under root (post-run object count).

func CountS3Prefix

func CountS3Prefix(ctx context.Context, p *providers3.Provider, prefix string) (int64, error)

CountS3Prefix counts objects under prefix (for post-run dest object count).

func CreateMotoBucket

func CreateMotoBucket(ctx context.Context, name string) error

CreateMotoBucket creates a unique moto bucket using the existing cloudtest S3 client (same credentials/endpoint as the moto integration lane).

func DeleteS3PrefixVerified

func DeleteS3PrefixVerified(ctx context.Context, p *providers3.Provider, prefix string) error

DeleteS3PrefixVerified deletes all objects under prefix, then lists again and fails if any remain. Stricter than cloudtest's log-on-cleanup-failure helper (measurement harness AC: verified cleanup).

func DigestManifestEntries

func DigestManifestEntries(entries []ManifestEntry) string

DigestManifestEntries returns the ordered entry digest used for envelope identity.

func E2EDelta

func E2EDelta(tDisk, tTmpfs float64) float64

E2EDelta computes the frozen pair e2e relative shift. Either arm with T <= 0 is non-comparable (NaN), matching the freeze.

func EnsureAbsent

func EnsureAbsent(path string) error

EnsureAbsent fails if path already exists (fresh checkpoint identity).

func EnsureEmptyDir

func EnsureEmptyDir(path string) error

EnsureEmptyDir fails if path exists and is non-empty, or creates it empty.

func FileURI

func FileURI(absPath string) string

FileURI builds a file:// URI for an absolute path (unix-style slash form). Callers must pass an absolute path; filepath.ToSlash normalization is applied at generation call sites via fileURIFromAbs.

func FullPipeReflowArgs

func FullPipeReflowArgs(opts FullPipeOpts) []string

FullPipeReflowArgs builds the reflow child's argv for a full-pipe point.

func HashFile

func HashFile(path string) (string, error)

HashFile returns SHA-256 hex of a file (binary provenance).

func ListProfiles

func ListProfiles() []string

ListProfiles returns known profile names for help text.

func MarshalIndentJSON

func MarshalIndentJSON(v any) (string, error)

MarshalIndentJSON is a helper for golden-style tests.

func MarshalJSONReport

func MarshalJSONReport(r Report) ([]byte, error)

MarshalJSONReport returns the report JSON.

func MotoAvailable

func MotoAvailable() bool

MotoAvailable reports whether a local moto endpoint is reachable (cloudtest).

func NormalizeMeasuredBinaryCommit

func NormalizeMeasuredBinaryCommit(probeCommit string) string

NormalizeMeasuredBinaryCommit maps probe output to the measured-binary commit field. Empty becomes "unknown". Worktree/instrument identity must never be passed here (retained-measure provenance) — this helper intentionally has no fallback args.

func OpenS3Provider

func OpenS3Provider(ctx context.Context, cfg BYOS3Config) (*providers3.Provider, error)

OpenS3Provider constructs an S3-compatible provider from BYO config.

func PairIsMaterial

func PairIsMaterial(delta float64) bool

PairIsMaterial reports |Δ| >= τ_e2e.

func PairIsTie

func PairIsTie(delta float64) bool

PairIsTie reports |Δ| < ε_tie (and not NaN).

func PrepareSourcePrefix

func PrepareSourcePrefix(ledger *CleanupLedger, mint func() string, upload func(prefix string) error) (string, error)

PrepareSourcePrefix mints a source prefix, registers it for cleanup, and only then uploads into it.

The registration sits between the two deliberately. Minting and the first object are separated by an entire corpus upload, and a failure inside that window is the one case a post-upload registration cannot cover.

func ProbeArgs

func ProbeArgs(sourcePrefix, probeConfig string, probeConc int, extra []string) []string

ProbeArgs builds the content-probe child's argv.

Pure and exported so a control can pin the exact argv without executing a child or reaching a provider. Lifting the cloud guards without proving what each child is actually invoked with would produce an instrument that looks runnable and is misconfigured.

func ResolveProviderClass

func ResolveProviderClass(name string) (string, error)

ResolveProviderClass normalizes Options.Provider / env. Empty → file. Unknown → error. s3/byo-s3 → s3-compatible.

func SortedHistogramKeys

func SortedHistogramKeys(h map[string]int) []string

SortedHistogramKeys returns histogram keys in stable order (for tests).

func SterilityForbidden

func SterilityForbidden(extra ...string) []string

SterilityForbidden returns tokens that must never appear in a serialized report.

func UploadCorpusToS3

func UploadCorpusToS3(ctx context.Context, p *providers3.Provider, cfg BYOS3Config, corpus GeneratedCorpus, sourcePrefix string, outInputPath string) error

UploadCorpusToS3 puts each local corpus object under sourcePrefix and returns a rewritten reflow.input.jsonl path using s3:// source URIs + dest_rel_key.

Uploads run concurrently; the emitted input file stays in manifest order so the reflow input is byte-identical to what the serial form produced. Order is preserved by writing each result into its own slot rather than appending, so completion order cannot leak into the fixture.

func ValidateArmMatrix

func ValidateArmMatrix(spec ProfileSpec, r Report) error

ValidateArmMatrix checks that a report contains exactly the arm × execution shape × parallel × checkpoint cells its profile declares — each one once. Per-point validation cannot see a whole cell that went missing, and "a declared arm is never silently dropped" is precisely what these profiles promise.

func ValidateCheckpointScaleSchedule

func ValidateCheckpointScaleSchedule(steps []ScheduleStep) error

ValidateCheckpointScaleSchedule checks counterbalancing invariants on a plan.

func ValidateCheckpointWriterStatsStructure

func ValidateCheckpointWriterStatsStructure(st reflow.CheckpointWriterStatsRecord) error

ValidateCheckpointWriterStatsStructure pins the sterile aggregate invariants required for an honest retained arm. A successful comparable arm must not carry writer-fatal/closed/canceled barrier outcomes.

func ValidateReportEnvelope

func ValidateReportEnvelope(r Report) error

ValidateReportEnvelope performs light structural checks on the report itself.

func ValidateServiceDemandFormalSet

func ValidateServiceDemandFormalSet(cells []ServiceDemandCell) error

ValidateServiceDemandFormalSet is the pure retained-set schedule/honesty validator (E1). A malformed set must never be treated as formal.

func VerifyCorpusImmutable

func VerifyCorpusImmutable(c GeneratedCorpus) error

VerifyCorpusImmutable re-hashes every source object and checks the manifest digest.

func VerifyManifestUnchanged

func VerifyManifestUnchanged(path string, wantDigest string) error

VerifyManifestUnchanged re-reads the on-disk manifest and checks digest identity.

func WriteServiceDemandSetReport

func WriteServiceDemandSetReport(path string, rep ServiceDemandSetReport) error

WriteServiceDemandSetReport writes the sterile set report as JSON. Uses O_CREATE|O_EXCL|O_WRONLY so the final path cannot replace existing retained evidence (POSIX rename is replace-capable and is not used here). Non-blocking nit: a crash mid-write may leave a partial exclusive file that blocks re-run until the operator removes it.

Types

type ArmPressureInputs

type ArmPressureInputs struct {
	Admissions           int64
	AdmissionBlocked     int64
	QueueDepthPeak       int64
	MaxBatch             int
	BarrierWaitNanos     int64
	E2EWallNanos         int64
	EffectiveConcurrency int
	// StatsOK is true when admission (exactly-one post-summary + structure) passed.
	StatsOK bool
}

ArmPressureInputs is the synthetic/measure surface for one arm's writer pressure.

type BYOS3Config

type BYOS3Config struct {
	Bucket         string
	Endpoint       string
	Region         string
	Profile        string
	RootPrefix     string
	ForcePathStyle bool
	// Optional static keys for moto only (cloudtest.TestAccessKeyID pattern).
	// Never used for real BYO — real-cloud uses ambient/profile chain only.
	AccessKeyID     string
	SecretAccessKey string
}

BYOS3Config is the harness view of the existing bring-your-own S3-compatible lane (same env vars as test/cloudtest and make test-cloud-real). Bucket/endpoint/profile never enter the sanitized report.

func LoadBYOS3Config

func LoadBYOS3Config() (cfg BYOS3Config, ok bool)

LoadBYOS3Config reads GONIMBUS_S3_TEST_* (cloudtest constants). Returns ok=false when the opt-in bucket is unset — callers skip, they do not fail.

func (BYOS3Config) ChildAWSEnv

func (c BYOS3Config) ChildAWSEnv() []string

ChildAWSEnv returns extra KEY=value pairs for the gonimbus child when static moto credentials are required. Empty for real BYO (ambient chain / profile).

func (BYOS3Config) MintUniquePrefix

func (c BYOS3Config) MintUniquePrefix(slug string) string

MintUniquePrefix returns a unique object prefix under the operator root, matching cloudtest.CreateS3ObjectPrefix naming shape without testing.T.

func (BYOS3Config) ObjectURI

func (c BYOS3Config) ObjectURI(key string) string

ObjectURI builds s3://bucket/key (not written into reports).

func (BYOS3Config) ProviderConfig

func (c BYOS3Config) ProviderConfig() providers3.Config

ProviderConfig maps to the S3 provider constructor (ambient credential chain for real BYO; optional static keys for moto only).

The connection pool follows the same public policy as product constructions: provider.ResolveConnectionPool(admittedN) with admittedN = corpus upload fan-out. For N >= 2 both fields equal N so concurrent staging is not bound by net/http.DefaultMaxIdleConnsPerHost (2). For N < 2 the zero policy leaves SDK / transport defaults (same as product). Staging must not invent a second transport policy pattern.

type BindDisposition

type BindDisposition string

BindDisposition is the frozen set-level outcome.

const (
	BindBinds        BindDisposition = "binds"
	BindDoesNotBind  BindDisposition = "does_not_bind"
	BindInconclusive BindDisposition = "inconclusive"
)

func ClassifyBindSet

func ClassifyBindSet(pairs []PairInputs) BindDisposition

ClassifyBindSet implements the frozen three-pair disposition.

binds:         both arms pressured on all 3 pairs + material same-sign e2e
does_not_bind: neither arm pressured on all 3 pairs + material same-sign e2e
inconclusive:  any invalid pair, any asymmetric (one-arm) pressure, mixed
               pressure quorum, or e2e material/direction failure

Asymmetric one-arm pressure must never collapse into does_not_bind.

type Budgets

type Budgets struct {
	Run   time.Duration
	Point time.Duration
}

Budgets is the resolved time envelope for one harness invocation.

func ResolveBudgets

func ResolveBudgets(getenv func(string) string) (Budgets, error)

ResolveBudgets reads both budgets through the supplied lookup so the resolution is testable without mutating process environment, and so the wiring — not just the parsing helper — can be pinned by a control.

A set-but-unparseable or non-positive value is an error rather than a silent fallback: falling back would hand the operator a budget that cannot hold the scale they asked for, which is the failure this was introduced to remove.

type CleanupLedger

type CleanupLedger struct {
	// Keep suppresses removal entirely. It is an operator choice to retain
	// artifacts, not a failure, so residue under Keep is expected.
	Keep bool
	// DeletePrefix is nil when the run has no object-store residue.
	DeletePrefix PrefixDeleter
	// contains filtered or unexported fields
}

CleanupLedger owns everything a run mints, so that whatever fails, there is exactly one place that knows what has to be removed.

The ordering rule it exists to enforce: a prefix is registered when it is minted, not when it has been successfully populated. Registering after a successful upload leaves the whole upload window uncovered — a failure partway through strands objects that nothing is responsible for removing.

func (*CleanupLedger) Owned

func (l *CleanupLedger) Owned() (source string, dests []string)

Owned reports the prefixes currently registered, for controls and diagnostics.

func (*CleanupLedger) RegisterDestPrefix

func (l *CleanupLedger) RegisterDestPrefix(prefix string)

RegisterDestPrefix takes ownership of a minted destination prefix.

func (*CleanupLedger) RegisterPoint

func (l *CleanupLedger) RegisterPoint(p MintedPoint)

RegisterPoint takes ownership of a point's local residue.

func (*CleanupLedger) RegisterSourcePrefix

func (l *CleanupLedger) RegisterSourcePrefix(prefix string)

RegisterSourcePrefix takes ownership of the minted source prefix. Call it immediately after minting and before the first upload.

func (*CleanupLedger) Run

func (l *CleanupLedger) Run() error

Run removes everything registered and verifies it is gone.

Every failure is collected rather than the first one returned alone: a destination that would not delete must not hide a source that also would not, because the operator needs to know the full extent of what was left behind.

type CompactManifest

type CompactManifest struct {
	RecipeVersion string         `json:"recipe_version"`
	Seed          int64          `json:"seed"`
	ObjectCount   int            `json:"object_count"`
	SizeBytes     int            `json:"size_bytes"`
	Partitions    int            `json:"partitions"`
	TotalBytes    int64          `json:"total_bytes"`
	SizeHistogram map[string]int `json:"size_histogram"`
	Digest        string         `json:"digest"`
}

CompactManifest is the report-safe subset (no path material).

type FullPipeOpts

type FullPipeOpts struct {
	Binary         string
	SourcePrefix   string
	ProbeConfig    string
	DestURI        string
	ProbeConc      int
	ReflowParallel int
	CheckpointPath string
	GOMEMLIMIT     string
	// MemoryBudget is the operator --memory-budget size string; empty omits
	// the flag so the product derives the budget from the detected limit.
	MemoryBudget string
	StdoutPath   string

	// ProviderClass labels the point in the report. Carried rather than
	// assumed, so a cloud run is not reported as a local one.
	ProviderClass string
	// ProbeExtraArgs are provider flags for the probe child only. Kept separate
	// from ReflowExtraArgs because the two commands do not accept the same
	// flags: probe takes --region/--profile/--endpoint, transfer takes the
	// paired --src-*/--dest-* set. One shared slice would hand each child flags
	// the other needs and the command would refuse them.
	ProbeExtraArgs []string
	// ReflowExtraArgs are provider flags for the reflow child only.
	ReflowExtraArgs []string
	// ChildExtraEnv is appended to both children's environment. Static
	// credentials belong here only for a backend that requires them; a real BYO
	// backend stays ambient or profile-based.
	ChildExtraEnv []string
	// RewriteFrom/RewriteTo map probe output, which carries no dest_rel_key,
	// onto destination keys. They are caller-supplied because the source key
	// shape is not the same on every backend: a local corpus root yields the
	// bare hive key, while an object store prefixes it with the minted run
	// prefix, so a fixed template matches on one and rejects every record on
	// the other. Empty falls back to the bare four-segment identity.
	RewriteFrom string
	RewriteTo   string
}

FullPipeOpts configures a supervised probe | tap | reflow point. Named fields rather than positional arguments: the several adjacent string parameters (checkpoint path, GOMEMLIMIT, memory budget, stdout path) are otherwise trivially transposable at the call site.

type GenerateOptions

type GenerateOptions struct {
	Recipe Recipe
	// RunRoot is the external run directory (ownership ledger lives here).
	RunRoot string
	// SourceDirName is the relative name under RunRoot for the immutable source.
	SourceDirName string
}

GenerateOptions controls corpus materialization.

type GeneratedCorpus

type GeneratedCorpus struct {
	Recipe          Recipe
	Root            string // absolute source root (operator-supplied test root subdir)
	Manifest        Manifest
	ManifestPath    string
	ReflowInputPath string
	ProbeConfigPath string
	ObjectCount     int
}

GeneratedCorpus is one verified immutable source corpus for a harness invocation.

func Generate

func Generate(opts GenerateOptions) (GeneratedCorpus, error)

Generate materializes a deterministic corpus under RunRoot and writes reflow-input JSONL + sterile probe config. Source objects are never mutated after return; callers treat the corpus as immutable for the invocation.

type HonestyResult

type HonestyResult struct {
	OK      bool
	Message string
}

HonestyResult is the universal ceiling/reason check for one point.

func CheckHonesty

func CheckHonesty(p ParsedReflowOutput, requested int) HonestyResult

CheckHonesty enforces:

0 <= max_active <= sourceCap+destCap+probeCap
effective <= requested

Source and probe domains cap at effective. Dest is bounded by the *recorded* dest_domain_ceiling_effective from the product run/summary (not a hard-coded multiplier). Multiplier is cross-checked when present so a non-default ConcurrencyConfig.DestDomainMultiplier cannot silently over-admit relative to the gate. Equality effective==requested requires reason exactly "requested". Lower effective requires resource_capped:* reason AND exactly one clamp warning. Honest caps are valid results, not harness failures.

type LandedObjectID

type LandedObjectID struct {
	RelKey string `json:"rel_key"`
	Size   int64  `json:"size_bytes"`
	Digest string `json:"content_digest"`
}

LandedObjectID is a content-identity record for fullpipe A/B parity (relative key + size + content digest). Paths stay relative/sanitized.

func SnapshotDestTree

func SnapshotDestTree(destRoot string) ([]LandedObjectID, error)

SnapshotDestTree walks destRoot and returns a sorted multiset of landed objects. Uses ReadDir recursion (not filepath.Walk) so each open is of a directory entry under the harness-owned dest root, avoiding Walk callback path races.

func SnapshotS3DestPrefix

func SnapshotS3DestPrefix(ctx context.Context, p *providers3.Provider, prefix string) ([]LandedObjectID, error)

SnapshotS3DestPrefix returns a sorted content-identity multiset for objects under prefix, matching SnapshotDestTree's shape.

It exists so the fullpipe A/B parity check keeps its meaning on an object store. The local check walks a destination directory, which a cloud point does not have; skipping it there would quietly drop the guarantee the "ab" in the profile name refers to, on exactly the runs the attribution depends on.

Identity is the provider ETag rather than a recomputed content hash. Both arms are listed the same way and compared only with each other, so the ETag is a sufficient identity for parity without re-downloading the corpus twice.

type Manifest

type Manifest struct {
	RecipeVersion string `json:"recipe_version"`
	Seed          int64  `json:"seed"`
	ObjectCount   int    `json:"object_count"`
	// SizeBytes and Partitions are the resolved recipe knobs, recorded so a
	// reader can name the exact corpus shape that produced the evidence.
	// Partitions in particular cannot be reconstructed from the entry digest.
	SizeBytes     int             `json:"size_bytes"`
	Partitions    int             `json:"partitions"`
	TotalBytes    int64           `json:"total_bytes"`
	SizeHistogram map[string]int  `json:"size_histogram"`
	Entries       []ManifestEntry `json:"entries"`
	// Digest is SHA-256 over ordered (relative_key, size, content_digest) lines.
	Digest string `json:"digest"`
}

Manifest is the canonical corpus identity bound into every report envelope.

func BuildManifest

func BuildManifest(recipe Recipe, entries []ManifestEntry) (Manifest, error)

BuildManifest constructs and digests a manifest from ordered entries.

func (Manifest) Compact

func (m Manifest) Compact() CompactManifest

Compact returns the allowlisted report view of the manifest.

type ManifestEntry

type ManifestEntry struct {
	RelativeKey   string `json:"relative_key"`
	SizeBytes     int64  `json:"size_bytes"`
	ContentDigest string `json:"content_digest"`
}

ManifestEntry is one object in the ordered corpus identity set.

type MemoryArm

type MemoryArm struct {
	// Label is the reported memory_envelope value.
	Label string
	// UseGOMEMLIMIT sources the arm's GOMEMLIMIT from operator options.
	UseGOMEMLIMIT bool
	// UseMemoryBudget sources the arm's --memory-budget from operator options.
	UseMemoryBudget bool
}

MemoryArm declares one memory envelope for a profile's points. The concrete GOMEMLIMIT / budget values are operator-supplied at invocation; the harness never invents them.

type MemoryResolution

type MemoryResolution struct {
	LimitBytes           int64
	LimitSource          string
	BudgetRequestedBytes int64
	BudgetEffectiveBytes int64
	BudgetSource         string
	RetryBufferCapBytes  int64
}

MemoryResolution is the startup-fixed memory arithmetic one record reports.

type MintedPoint

type MintedPoint struct {
	DestDir        string
	CheckpointPath string
}

MintedPoint is the local residue one measurement point owns.

type OccupancyResult

type OccupancyResult struct {
	OK           bool
	Samples      []int
	Floor        int
	PassedNeeded int
	Passed       int
	Message      string
}

OccupancyResult is the fixed-mode reflow-saturation occupancy check.

func CheckOccupancy

func CheckOccupancy(samples []int, effective int, passCount int) OccupancyResult

CheckOccupancy applies the serial-dispatch detector for fixed-mode reflow saturation.

type Options

type Options struct {
	// Binary is the absolute path to the built gonimbus binary.
	Binary string
	// Profile name (empty → smoke).
	Profile string
	// Provider is file (default), moto, s3-compatible, or gcs.
	// BYO cloud uses the same GONIMBUS_S3_TEST_* / GONIMBUS_GCS_TEST_* env
	// conventions as test/cloudtest and make test-cloud-real (opt-in; skip/fail
	// clearly when unset for a non-file provider).
	Provider string
	// RunRoot is the operator-supplied external test root (created if needed).
	RunRoot string
	// GOMEMLIMIT is operator-supplied only; never auto-set by the harness.
	// Under the product's minimum-selection limit chain a GOMEMLIMIT binds
	// only when it is the lowest candidate, so this constrains an envelope
	// rather than raising one.
	GOMEMLIMIT string
	// MemoryBudget is the operator --memory-budget value for arms that set the
	// budget directly. Operator-supplied only; never invented by the harness.
	MemoryBudget string
	// TmpfsCheckpointRoot when set is used for checkpoint_class=tmpfs points.
	// The path itself is never written into the report.
	// Also accepted via env TMPFS_CHECKPOINT_ROOT / GONIMBUS_THROUGHPUT_TMPFS_CHECKPOINT_ROOT.
	TmpfsCheckpointRoot string
	// Keep retains minted roots (report only retained invocation id).
	Keep bool
	// PointTimeout bounds each measured point.
	PointTimeout time.Duration
	// ConstrainedGOMEMLIMIT is the GOMEMLIMIT used by constrained-envelope arms
	// of the ceiling-lift and checkpoint profiles. When empty it falls back to
	// GOMEMLIMIT. Accepted via CONSTRAINED_GOMEMLIMIT, or the older
	// CEILING_LIFT_GOMEMLIMIT spelling.
	ConstrainedGOMEMLIMIT string
	// WorktreeCommit is optional instrument commit override (harness identity).
	// It is NEVER applied to measured BinaryCommit (retained-measure provenance).
	WorktreeCommit string
	// ScheduleID selects checkpoint-scale counterbalancing plan
	// (disk_first_odd default; tmpfs_first_odd reverse). Ignored for other profiles.
	ScheduleID string

	// Recipe overrides scale the profile's synthetic corpus at invocation. Zero
	// means keep the profile default. Accepted via OBJECT_COUNT / SIZE_BYTES /
	// PARTITIONS (GONIMBUS_THROUGHPUT_OBJECT_COUNT / _SIZE_BYTES / _PARTITIONS).
	// Absurd values fail closed in Recipe.Validate; the effective corpus is
	// recorded in the report's corpus block so evidence names what was measured.
	RecipeObjectCount int
	RecipeSizeBytes   int
	RecipePartitions  int
}

Options configures a harness invocation.

type PairInputs

type PairInputs struct {
	// Invalid marks failed/honesty/missing-stats/non-comparable pairs.
	Invalid bool
	// DiskPressure is true when the disk arm shows writer pressure.
	DiskPressure bool
	// TmpfsPressure is true when the tmpfs arm shows writer pressure.
	TmpfsPressure bool
	// DeltaE2E is (T_disk - T_tmpfs) / max(T_disk, T_tmpfs).
	DeltaE2E float64
}

PairInputs is one matched disk/tmpfs pair at the disposition layer.

DiskPressure and TmpfsPressure are independent so asymmetric (exactly-one-arm) evidence cannot collapse into "no pair pressure" and falsely yield does_not_bind.

func (PairInputs) AsymmetricArmPressure

func (p PairInputs) AsymmetricArmPressure() bool

AsymmetricArmPressure is true when exactly one arm shows writer pressure.

func (PairInputs) BothArmsPressure

func (p PairInputs) BothArmsPressure() bool

BothArmsPressure is true when each arm independently shows writer pressure.

func (PairInputs) NeitherArmPressure

func (p PairInputs) NeitherArmPressure() bool

NeitherArmPressure is true when neither arm shows writer pressure.

type ParsedReflowOutput

type ParsedReflowOutput struct {
	RunCount          int
	SummaryCount      int
	WarningClampCount int
	ObjectComplete    int64

	ObjectError   int64
	InvalidInputs int64
	SummaryErrors int64

	// Run-record concurrency (required agreement with summary).
	RunRequested int
	RunEffective int
	RunReason    string
	RunMaxActive int
	RunFinal     int
	RunAdaptive  bool

	// Summary-record concurrency.
	SummaryRequested int
	SummaryEffective int
	SummaryReason    string
	SummaryMaxActive int
	SummaryFinal     int
	SummaryAdaptive  bool

	// Dest-domain policy (GON-067 PR2): static for the run; run and summary must
	// agree. Used by CheckHonesty so the gate asserts resolved output, not an
	// assumed default multiplier.
	RunDestDomainMultiplier           int
	RunDestDomainCeilingEffective     int
	SummaryDestDomainMultiplier       int
	SummaryDestDomainCeilingEffective int

	// Memory resolution as the product reported it. The whole tuple is fixed at
	// startup, so run and summary must agree on every field — comparing only
	// the source labels would let a differing limit, budget, or retry cap
	// through and publish the run's values as if both records agreed. Values
	// stay aggregate (bytes and source labels) — no host identity is retained.
	RunMemory     MemoryResolution
	SummaryMemory MemoryResolution

	// Consensus memory resolution after the agreement check.
	MemoryLimitBytes           int64
	MemoryLimitSource          string
	MemoryBudgetRequestedBytes int64
	MemoryBudgetEffectiveBytes int64
	MemoryBudgetSource         string
	RetryBufferCapBytes        int64
	// SummaryTimeAvgActive is the completed-run occupancy diagnostic. The run
	// record's startup sample is always zero by contract and is not retained.
	SummaryTimeAvgActive float64

	// CheckpointWriterStats is the sterile end-of-run checkpoint writer
	// diagnostic snapshot when the child emitted
	// gonimbus.reflow.checkpoint_writer_stats.v1 (checkpoint measure). Nil when absent.
	// Measure-first only — not product throughput evidence.
	CheckpointWriterStats *reflow.CheckpointWriterStatsRecord
	// CheckpointWriterStatsCount is how many stats records appeared (exactly
	// one is required for retained checkpoint-scale arms).
	CheckpointWriterStatsCount int
	// CheckpointWriterStatsBeforeSummary counts stats records observed before
	// the single summary (must be zero for retained arms).
	CheckpointWriterStatsBeforeSummary int

	// ObjectPathStageStats is the sterile object-path stage attribution snapshot
	// when the child emitted gonimbus.reflow.object_path_stage_stats.v1.
	// Optional; measure-first only — not product throughput marketing.
	ObjectPathStageStats *reflow.ObjectPathStageStatsRecord
	// ObjectPathStageStatsCount is how many stage-stats records appeared.
	ObjectPathStageStatsCount int

	// Consensus fields after agreement check (filled by Finalize).
	Requested       int
	Effective       int
	Reason          string
	MaxActive       int
	Final           int
	AdaptiveEnabled bool
	Initial         int
	Floor           int

	// Consensus dest-domain policy after run/summary agreement.
	DestDomainMultiplier       int
	DestDomainCeilingEffective int

	ThrottleBackoffs  int64
	AdditiveIncreases int64
	ConnectionFreezes int64
	ClampWarningOK    bool
	// Clamp warning details for three-way agreement with run/summary.
	ClampWarnRequested int
	ClampWarnEffective int
	ClampWarnReason    string
	ClampWarnAdaptive  *bool
	ParseErrors        []string

	UnknownTypes []string
	// contains filtered or unexported fields
}

ParsedReflowOutput is the allowlist-extracted view of reflow child stdout. It never retains destination URI or checkpoint path from product records.

func ParseReflowFile

func ParseReflowFile(path string) (ParsedReflowOutput, error)

ParseReflowFile opens path and parses incrementally.

func ParseReflowReader

func ParseReflowReader(r io.Reader) (ParsedReflowOutput, error)

ParseReflowReader streams JSONL from r with bounded line reading.

func ParseReflowStdout

func ParseReflowStdout(stdout []byte) (ParsedReflowOutput, error)

ParseReflowStdout extracts concurrency and count fields by record type. Malformed JSONL objects and unknown types are hard failures.

type PointReport

type PointReport struct {
	PointID          string `json:"point_id"`
	ExecutionShape   string `json:"execution_shape"` // reflow_only|full_pipe|probe_drain
	Parallel         int    `json:"reflow_parallel_requested,omitempty"`
	ProbeConcurrency int    `json:"probe_concurrency,omitempty"`
	// AdaptiveMode is reflow adaptive/fixed when reflow ran; empty for probe_drain.
	AdaptiveMode    string `json:"adaptive_mode,omitempty"`
	GOMEMLIMITSet   bool   `json:"gomemlimit_set"`
	GOMEMLIMITValue string `json:"gomemlimit_value,omitempty"`
	// MemoryBudgetRequested is the operator --memory-budget passed to the child
	// (empty when the arm let the product derive the budget).
	MemoryBudgetRequested string `json:"memory_budget_requested,omitempty"`
	// MemoryLimitSource is product memory-limit provenance when reflow ran; omit for probe_drain.
	MemoryLimitSource string `json:"memory_limit_source,omitempty"`
	// MemoryBudgetSource is product budget provenance: derived|operator|operator_clamped_to_limit.
	MemoryBudgetSource string `json:"memory_budget_source,omitempty"`
	// Resolved memory arithmetic as the product reported it (aggregate bytes only).
	MemoryLimitBytes           int64 `json:"memory_limit_bytes,omitempty"`
	MemoryBudgetEffectiveBytes int64 `json:"memory_budget_effective_bytes,omitempty"`
	RetryBufferCapBytes        int64 `json:"retry_buffer_cap_bytes,omitempty"`
	// MemoryEnvelope names which lever bound this arm's memory:
	// gomemlimit_constrained | probe_bound | operator_budget | "".
	MemoryEnvelope string `json:"memory_envelope,omitempty"`
	// ConcurrencyTimeAvgActive is the completed-run occupancy diagnostic.
	ConcurrencyTimeAvgActive float64 `json:"concurrency_time_avg_active,omitempty"`
	CheckpointClass          string  `json:"checkpoint_class,omitempty"`

	// Product reflow concurrency telemetry (omit when not applicable).
	ConcurrencyRequested *int    `json:"concurrency_ceiling_requested,omitempty"`
	ConcurrencyEffective *int    `json:"concurrency_ceiling_effective,omitempty"`
	ConcurrencyReason    *string `json:"concurrency_ceiling_reason,omitempty"`
	ConcurrencyMaxActive *int    `json:"concurrency_max_active,omitempty"`
	ConcurrencyFinal     *int    `json:"concurrency_final,omitempty"`
	AdaptiveEnabled      *bool   `json:"adaptive_enabled,omitempty"`

	// CheckpointWriterStats is sterile checkpoint-writer diagnostics from the
	// child when emitted (checkpoint measure). Measure-first only — not product
	// throughput marketing. Omitted when the child did not emit the record.
	CheckpointWriterStats *reflow.CheckpointWriterStatsRecord `json:"checkpoint_writer_stats,omitempty"`

	ElapsedSeconds      float64 `json:"elapsed_seconds"`
	CompletedObjects    int64   `json:"completed_objects"`
	EndToEndRate        float64 `json:"end_to_end_rate_objects_per_s,omitempty"`
	ProbeDeliveredRate  float64 `json:"probe_delivered_rate_objects_per_s,omitempty"`
	ProbeSaturationRate float64 `json:"probe_saturation_rate_objects_per_s,omitempty"`
	TapValidRows        int64   `json:"tap_valid_reflow_input_rows,omitempty"`
	// TapCopyIntervalSeconds is the wall time of the tap Copy loop (includes
	// producer pacing and downstream backpressure). Not calibrated overhead.
	TapCopyIntervalSeconds float64 `json:"tap_copy_interval_seconds,omitempty"`

	// HonestyOK is nil when honesty is not applicable (e.g. probe_drain).
	HonestyOK      *bool  `json:"honesty_ok,omitempty"`
	HonestyMessage string `json:"honesty_message,omitempty"`

	OccupancySamples []int  `json:"occupancy_samples,omitempty"`
	OccupancyOK      *bool  `json:"occupancy_ok,omitempty"`
	OccupancyMessage string `json:"occupancy_message,omitempty"`

	// ContentParityOK is true when landed key+size+content-digest multisets match.
	ContentParityOK *bool `json:"content_parity_ok,omitempty"`

	StageExitCodes map[string]int `json:"stage_exit_codes"`
}

PointReport is one measured arm with allowlisted aggregate fields only. Reflow concurrency fields are pointers so probe_drain omits them (no product reflow telemetry exists for that shape).

type PointResult

type PointResult struct {
	Profile          string
	PointID          string
	Parallel         int
	ProbeConcurrency int
	GOMEMLIMIT       string
	// MemoryBudget is the operator --memory-budget value passed to the child
	// (empty when the arm let the product derive the budget).
	MemoryBudget    string
	CheckpointClass string // disk | tmpfs — never the path
	ProviderClass   string // file | moto | s3-compatible | gcs
	Stages          map[string]StageResult
	Tap             TapStats
	Stdout          []byte // small in-memory only; prefer StdoutPath for scale
	StdoutPath      string // external run artifact for reflow JSONL
	Elapsed         time.Duration
	// CompletedObjects is derived from summary/object counts when available.
	CompletedObjects int64
	WallRate         float64 // objects / elapsed seconds (0 if incomplete)
}

PointResult aggregates one measured sweep point.

func RunFullPipe

func RunFullPipe(ctx context.Context, opts FullPipeOpts) (PointResult, error)

RunFullPipe supervises probe | tap | reflow without a shell. Shared cancel on any stage/tap failure; rewrite templates supply dest mapping for probe-emitted reflow.input (no dest_rel_key).

func RunProbeDrain

func RunProbeDrain(ctx context.Context, opts ProbeDrainOpts) (PointResult, error)

RunProbeDrain runs content probe into a draining tap sink (no reflow). This is the methodology-faithful producer-only probe saturation shape.

func RunReflowOnly

func RunReflowOnly(ctx context.Context, opts StageRunOpts) (PointResult, error)

RunReflowOnly executes: binary transfer reflow --stdin ... with pre-materialized input.

type PrefixDeleter

type PrefixDeleter func(ctx context.Context, prefix string) error

PrefixDeleter removes every object under a minted object-store prefix and verifies none remain. Injected so the cleanup lifecycle can be controlled without a provider: the failure modes that matter are ordering and error propagation, and neither needs a bucket to exercise.

type ProbeDrainOpts

type ProbeDrainOpts struct {
	Binary         string
	SourcePrefix   string
	ProbeConfig    string
	ProbeConc      int
	GOMEMLIMIT     string
	ProviderClass  string
	ProbeExtraArgs []string
	ChildExtraEnv  []string
}

ProbeDrainOpts configures a probe-only point with a draining sink.

Named fields for the same reason as FullPipeOpts: the positional form had grown a run of adjacent strings a caller can transpose without the compiler noticing, and provider support adds more of them.

type ProfileSpec

type ProfileSpec struct {
	Name   string
	Recipe Recipe
	// ExecutionShape: reflow_only | full_pipe
	ExecutionShape string
	// ParallelPoints for reflow --parallel sweeps.
	ParallelPoints []int
	// ProbeConcurrencyPoints for probe sweeps / full-pipe pairs.
	ProbeConcurrencyPoints []int
	// FullPipePairs is (probe, reflow) pairs for full-pipe profiles.
	FullPipePairs [][2]int
	// RequireOccupancy enables fixed-mode saturation occupancy sampling.
	RequireOccupancy bool
	// NoAdaptive forces --no-adaptive.
	NoAdaptive bool
	// MemoryArms declares the memory envelopes each parallel point is run
	// under. Empty means a single arm with no memory override. See MemoryArm.
	MemoryArms []MemoryArm
	// CheckpointClasses lists disk and/or tmpfs discriminators.
	CheckpointClasses []string
	// Description is operator-facing sterile help.
	Description string
}

ProfileSpec declares a named measurement profile.

func ResolveProfile

func ResolveProfile(name string) (ProfileSpec, error)

ResolveProfile maps a name to a spec. Empty name → smoke.

type Recipe

type Recipe struct {
	Version     string `json:"recipe_version"`
	Seed        int64  `json:"seed"`
	ObjectCount int    `json:"object_count"`
	SizeBytes   int    `json:"size_bytes"`
	Partitions  int    `json:"partitions"`
	// FixedDate is a sterile calendar token embedded in keys (not a field identity).
	FixedDate string `json:"fixed_date"`
}

Recipe describes a deterministic synthetic hive-shaped corpus. Keys are generic (entity/device/date) with no client keyspace identity.

func DefaultSmokeRecipe

func DefaultSmokeRecipe() Recipe

DefaultSmokeRecipe returns the cheapest credential-free local smoke recipe.

func SaturationRecipe

func SaturationRecipe() Recipe

SaturationRecipe returns a larger local recipe for reflow occupancy checks. Object count targets roughly 32× a modest effective ceiling of 8 for smoke-scale saturation; operators select scale profiles explicitly.

func ScaleRecipe

func ScaleRecipe() Recipe

ScaleRecipe returns the checkpoint-scale corpus (see the scale defaults above).

func (Recipe) ObjectContent

func (r Recipe) ObjectContent(i int) []byte

ObjectContent returns byte-deterministic XML content for object index i. A probeable <Marker> field is near the head for full-pipe profiles.

func (Recipe) RelativeKey

func (r Recipe) RelativeKey(i int) string

RelativeKey returns the sterile relative key for object index i. Shape: entity=%04d/device=%04d/date=<fixed>/object-%08d.xml

func (Recipe) Validate

func (r Recipe) Validate() error

Validate checks recipe bounds.

type Report

type Report struct {
	SchemaVersion string `json:"schema_version"`

	BinaryVersion string `json:"binary_version,omitempty"`
	BinaryCommit  string `json:"binary_commit,omitempty"`
	BinarySHA256  string `json:"binary_sha256"`

	// Instrument identity is the harness (test binary / worktree), distinct from
	// the measured child Binary* fields (checkpoint measure / provenance gate). Never backfill
	// BinaryCommit from instrument HEAD.
	InstrumentCommit string `json:"instrument_commit,omitempty"`
	InstrumentSHA256 string `json:"instrument_sha256,omitempty"`
	InstrumentDirty  bool   `json:"instrument_dirty"` // always present: false=clean, true=dirty (probe fail aborts run)

	// Checkpoint-scale schedule provenance (empty for other profiles).
	ScheduleID    string   `json:"schedule_id,omitempty"`
	ScheduleOrder []string `json:"schedule_order,omitempty"` // e.g. "warm:disk:64", "p1:disk:64"

	Corpus CompactManifest `json:"corpus"`

	Profile                 string `json:"profile"`
	ProviderClass           string `json:"provider_class"`
	ThroughputEvidenceClass string `json:"throughput_evidence_class"`
	// MeasurementScope declares this cut's intent (small-scale honesty/relative).
	MeasurementScope string `json:"measurement_scope"`
	// AbsoluteEnvelopeValidation is not_evaluated until an opt-in BYO-cloud
	// scale lane is used; avoids presenting local/moto as a throughput oracle.
	AbsoluteEnvelopeValidation string `json:"absolute_envelope_validation"`

	OS   string `json:"os"`
	Arch string `json:"arch"`

	InvocationID string    `json:"invocation_id"`
	StartedAt    time.Time `json:"started_at"`

	Keep bool `json:"keep"`

	Points []PointReport `json:"points"`
}

Report is the sanitized, versioned measurement report. It must never include endpoints, credentials, bucket names, signed URLs, checkpoint paths, or local absolute paths.

func NewReport

func NewReport(profile, providerClass, invocationID, binarySHA string, corpus CompactManifest, keep bool) Report

NewReport seeds a report shell.

func Run

func Run(ctx context.Context, opts Options) (report Report, runErr error)

Run executes the named profile and returns a sanitized report.

type ScheduleStep

type ScheduleStep struct {
	// PairIndex is 0 for warmups; 1..N for retained matched pairs.
	PairIndex int
	// Warmup marks discarded arms (not retained in report / Δe2e).
	Warmup          bool
	Parallel        int
	CheckpointClass string // disk | tmpfs
	MemoryEnvelope  string
	// Ordinal is 0-based position in the full execution order (including warmups).
	Ordinal int
}

ScheduleStep is one planned arm in a checkpoint-scale run.

func CheckpointScaleSchedule

func CheckpointScaleSchedule(spec ProfileSpec, scheduleID string) ([]ScheduleStep, error)

CheckpointScaleSchedule builds the frozen counterbalanced plan for a profile. scheduleID must be ScheduleDiskFirstOdd or ScheduleTmpfsFirstOdd.

Layout:

  1. one discarded warmup per class (first parallel point, first memory arm)
  2. three matched pairs over ParallelPoints in order, alternating first-class as D/T, T/D, D/T (or reverse for tmpfs_first_odd)

type ServiceDemandArmConfig

type ServiceDemandArmConfig struct {
	// StorePath is the APFS (or other quorum medium) checkpoint path.
	StorePath string
	// Submitters is fixed concurrent mutation callers.
	Submitters int
	// Units is the number of terminal units (each = 3 admissions). 0 → formal quantum.
	Units int
	// MinAdmissions overrides the formal work quantum when > 0 (tests only).
	MinAdmissions int64
	// ElideRawExecSavepoints enables experimental Phase A elision for this arm.
	ElideRawExecSavepoints bool
}

ServiceDemandArmConfig configures one timed arm.

type ServiceDemandArmReport

type ServiceDemandArmReport struct {
	Submitters           int                                `json:"submitters"`
	Units                int                                `json:"units"`
	AdmissionsTarget     int64                              `json:"admissions_target"`
	WallSeconds          float64                            `json:"wall_seconds"`
	MutationsPerSec      float64                            `json:"mutations_per_sec"`
	TransactionsPerSec   float64                            `json:"transactions_per_sec"`
	WriterUtilization    float64                            `json:"writer_utilization"`
	BarrierWaitLoad      float64                            `json:"barrier_wait_load"`
	BarrierWaitMeanNanos float64                            `json:"barrier_wait_mean_nanos"`
	BarrierWaitMaxNanos  int64                              `json:"barrier_wait_max_nanos"`
	AdmissionsPerBatch   float64                            `json:"admissions_per_batch"`
	ReachableOccupancy   float64                            `json:"reachable_occupancy"`
	AdmissionWaitMaxNs   int64                              `json:"admission_wait_max_nanos"`
	AdmissionBlocked     int64                              `json:"admission_blocked"`
	Honest               bool                               `json:"honest"`
	HonestyMessage       string                             `json:"honesty_message,omitempty"`
	WriterStats          reflow.CheckpointWriterStatsRecord `json:"writer_stats"`
}

ServiceDemandArmReport is a sterile arm result (no paths in JSON export helpers).

func RunServiceDemandArm

func RunServiceDemandArm(ctx context.Context, cfg ServiceDemandArmConfig) (ServiceDemandArmReport, error)

RunServiceDemandArm opens a fresh store, runs the terminal mutation mix under fixed submitter concurrency, and returns sterile metrics from WriterStats.

Timed region: MarkDestKeyObserved → NoteDestKeySource → UpsertItem(complete) only. Key formatting is outside the timed loop (Day-0 mix-order AC).

type ServiceDemandCell

type ServiceDemandCell struct {
	Rep        int                     `json:"rep"`
	Ordinal    int                     `json:"ordinal"`
	ScheduleID ServiceDemandScheduleID `json:"schedule_id"`
	Submitters int                     `json:"submitters"`
	ServiceDemandArmReport
}

ServiceDemandCell is one arm inside a multi-rep set (sterile; no store path).

type ServiceDemandDisposition

type ServiceDemandDisposition string

ServiceDemandDisposition is the pure classifier outcome for a formal set.

const (
	ServiceDemandMechanismBinds   ServiceDemandDisposition = "mechanism_binds"
	ServiceDemandMechanismDoesNot ServiceDemandDisposition = "mechanism_does_not_bind"
	ServiceDemandInconclusive     ServiceDemandDisposition = "inconclusive"
)

func ClassifyServiceDemandV2

func ClassifyServiceDemandV2(cells []ServiceDemandCell) (ServiceDemandDisposition, string)

ClassifyServiceDemandV2 is a pure classifier over formal set cells.

mechanism_binds requires formal shape (3 reps × full grid × exact 300k admissions on every cell) plus all of:

  1. util median ≥ 0.85 at each of {64,128,256,512}
  2. mut/s@512 ≤ 1.05 × mut/s@128 (plateau) OR decline across two adjacent of {128,256,512}
  3. mean barrier-wait/admission @512 ≥ 2 × mean @64
  4. mut/s CV ≤ 0.10 at each high-N point with n≥3 samples (else inconclusive)

mechanism_does_not_bind: same formal shape; util median < 0.50 at 256 and 512, and mut/s increases ≥1.5× from 64→256. Otherwise: inconclusive.

type ServiceDemandScheduleID

type ServiceDemandScheduleID string

ServiceDemandScheduleID names a fixed formal schedule.

const (
	ServiceDemandScheduleR1 ServiceDemandScheduleID = "R1"
	ServiceDemandScheduleR2 ServiceDemandScheduleID = "R2"
	ServiceDemandScheduleR3 ServiceDemandScheduleID = "provenance gate"
)

func FormalScheduleForRep

func FormalScheduleForRep(rep int) (ServiceDemandScheduleID, error)

FormalScheduleForRep maps formal rep index (1..3) to schedule ID.

type ServiceDemandSetConfig

type ServiceDemandSetConfig struct {
	// RootDir holds per-arm SQLite files (fresh unique DB per arm).
	RootDir string
	// FormalReps is the list of formal rep indices (default 1,2,3). Pilot 0 excluded.
	FormalReps []int
	// Units overrides formal quantum for tests (0 → formal 100k units).
	Units int
	// MinAdmissions overrides formal quantum for tests.
	MinAdmissions int64
	// WorktreeCommitOverride is optional instrument commit override (tests).
	// When set, the set report is always non-formal (test-only identity).
	WorktreeCommitOverride string
	// MediumOverride labels reduced/non-formal structural runs (e.g. "test_tmp").
	// Empty + formal path → verified APFS medium (or set fails formal admission).
	MediumOverride string
	// SkipAPFSCheck allows structural tests on non-APFS roots; forces non-formal.
	SkipAPFSCheck bool
	// ElideRawExecSavepoints enables Phase A elision for every arm in the set.
	// Sets with elision are never formal baseline authority (wantFormalShape false).
	ElideRawExecSavepoints bool
}

ServiceDemandSetConfig configures RunServiceDemandSet.

type ServiceDemandSetReport

type ServiceDemandSetReport struct {
	SchemaVersion    string `json:"schema_version"`
	Medium           string `json:"medium"`
	MixID            string `json:"mix_id"`
	AdmissionsPerArm int64  `json:"admissions_per_arm"`
	// Formal is true only for the default Day-0 shape: reps {1,2,3}, full
	// submitter grid, exact ServiceDemandAdmissionsPerArm on every cell.
	// Reduced Units/MinAdmissions/FormalReps sets are non-formal (structure only).
	Formal                 bool                     `json:"formal"`
	InstrumentSHABefore    string                   `json:"instrument_sha_before"`
	InstrumentCommitBefore string                   `json:"instrument_commit_before"`
	InstrumentDirtyBefore  bool                     `json:"instrument_dirty_before"`
	InstrumentSHAAfter     string                   `json:"instrument_sha_after"`
	InstrumentCommitAfter  string                   `json:"instrument_commit_after"`
	InstrumentDirtyAfter   bool                     `json:"instrument_dirty_after"`
	Cells                  []ServiceDemandCell      `json:"cells"`
	Disposition            ServiceDemandDisposition `json:"disposition"`
	DispositionMessage     string                   `json:"disposition_message,omitempty"`
}

ServiceDemandSetReport is the versioned sterile multi-rep report.

func RunFormalServiceDemandBaseline

func RunFormalServiceDemandBaseline(ctx context.Context, rootDir, reportPath string) (ServiceDemandSetReport, error)

RunFormalServiceDemandBaseline is the gated formal-run entry: default 3×6×300k under RootDir, writes the sterile set report to reportPath (required, no overwrite).

func RunServiceDemandSet

func RunServiceDemandSet(ctx context.Context, cfg ServiceDemandSetConfig) (ServiceDemandSetReport, error)

RunServiceDemandSet runs formal reps under fixed schedules (18 cells for 3×6). Captures instrument identity before/after; refuses set if instrument drifts.

type StageResult

type StageResult struct {
	Name       string        `json:"name"`
	ExitCode   int           `json:"exit_code"`
	Err        string        `json:"error,omitempty"`
	Stderr     string        `json:"stderr_excerpt,omitempty"`
	Duration   time.Duration `json:"duration_ns"`
	StartedAt  time.Time     `json:"-"`
	FinishedAt time.Time     `json:"-"`
}

StageResult is the independent disposition of one supervised child.

type StageRunOpts

type StageRunOpts struct {
	Binary         string
	InputPath      string
	DestURI        string
	Parallel       int
	CheckpointPath string
	GOMEMLIMIT     string
	// MemoryBudget is the operator --memory-budget size string. Empty omits the
	// flag entirely, so the product derives the budget from the detected limit.
	MemoryBudget  string
	NoAdaptive    bool
	ProviderClass string
	ExtraArgs     []string
	ChildExtraEnv []string
	// StdoutPath streams reflow JSONL to a file (bounded memory). Required for scale.
	StdoutPath string
	// RewriteFrom/To map probe output without dest_rel_key onto destinations.
	RewriteFrom string
	RewriteTo   string
}

StageRunOpts configures a supervised child stage.

type Tap

type Tap struct {
	MaxRecordBytes        int
	AcceptReflowInputOnly bool
	// contains filtered or unexported fields
}

Tap is a bounded raw-byte pass-through: forwards fragments immediately with backpressure, frames on newline for counting, and hard-fails on oversize/malformed/unexpected records via strict JSON decode.

func (*Tap) Copy

func (t *Tap) Copy(w io.Writer, r io.Reader) error

Copy runs the transparent tap from r to w until EOF or error.

func (*Tap) Err

func (t *Tap) Err() error

Err returns the first hard-fail error, if any.

func (*Tap) Stats

func (t *Tap) Stats() TapStats

Stats returns a copy of tap statistics.

type TapStats

type TapStats struct {
	ValidReflowInputRows int64
	UnexpectedRows       int64
	BytesForwarded       int64
	FirstForward         time.Time
	LastForward          time.Time
	// CopyDuration is wall time of the tap Copy loop (includes producer pacing
	// and downstream backpressure). Not calibrated isolated overhead.
	CopyDuration time.Duration
}

TapStats records transparent pass-through metrics for the probe→reflow pipe.

func (TapStats) ActiveInterval

func (s TapStats) ActiveInterval() time.Duration

ActiveInterval returns last-first duration for probe-delivered rate windows.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL