Documentation
¶
Overview ¶
Package reflowthroughput is a non-CI, on-demand measurement harness for transfer reflow throughput and parallelism behavior.
It is test/tooling only: it executes a built gonimbus binary as a child process, never calls Cobra command internals, and must not introduce production package seams, product record schema, or provider runtime changes.
Entry point: make test-reflow-throughput PROFILE=<name> PROVIDER=<class> Default PROFILE=smoke, PROVIDER=file (credential-free local backend).
BYO cloud transport reuses the existing non-CI lanes in test/cloudtest and make test-cloud-real (GONIMBUS_S3_TEST_* / GONIMBUS_GCS_TEST_* opt-in, unique minted prefixes, ambient credentials). Moto uses the same endpoint and test-key pattern as make test-cloud. Local/moto results are correctness evidence only; real-provider claims require PROVIDER=s3-compatible with BYO env and are labeled provider_opt_in in the report.
Index ¶
- Constants
- Variables
- func ArmHasWriterPressure(a ArmPressureInputs) bool
- func AssertNoSensitiveTokens(s string, forbidden []string) error
- func BarrierWaitCapacityFraction(barrierWaitNanos, e2eWallNanos int64, effectiveConcurrency int) float64
- func BuildBinary(ctx context.Context, repoRoot, outPath string) (string, string, error)
- func CLIProbeProviderFlags(cfg BYOS3Config) []string
- func CLIProviderFlags(cfg BYOS3Config) []string
- func CheckCheckpointWriterStatsAdmission(p ParsedReflowOutput) error
- func CheckCounts(generated int, tapRows int64, completed int64, summaryErrors int64, ...) error
- func ChildEnv(base []string, gomemlimit string, extra ...string) []string
- func CompareLandedMultisets(a, b []LandedObjectID) error
- func ContentDigest(b []byte) string
- func CorpusUploadConcurrency() int
- func CountFilesRecursive(root string) (int64, error)
- func CountS3Prefix(ctx context.Context, p *providers3.Provider, prefix string) (int64, error)
- func CreateMotoBucket(ctx context.Context, name string) error
- func DeleteS3PrefixVerified(ctx context.Context, p *providers3.Provider, prefix string) error
- func DigestManifestEntries(entries []ManifestEntry) string
- func E2EDelta(tDisk, tTmpfs float64) float64
- func EnsureAbsent(path string) error
- func EnsureEmptyDir(path string) error
- func FileURI(absPath string) string
- func FullPipeReflowArgs(opts FullPipeOpts) []string
- func HashFile(path string) (string, error)
- func ListProfiles() []string
- func MarshalIndentJSON(v any) (string, error)
- func MarshalJSONReport(r Report) ([]byte, error)
- func MotoAvailable() bool
- func NormalizeMeasuredBinaryCommit(probeCommit string) string
- func OpenS3Provider(ctx context.Context, cfg BYOS3Config) (*providers3.Provider, error)
- func PairIsMaterial(delta float64) bool
- func PairIsTie(delta float64) bool
- func PrepareSourcePrefix(ledger *CleanupLedger, mint func() string, upload func(prefix string) error) (string, error)
- func ProbeArgs(sourcePrefix, probeConfig string, probeConc int, extra []string) []string
- func ResolveProviderClass(name string) (string, error)
- func SortedHistogramKeys(h map[string]int) []string
- func SterilityForbidden(extra ...string) []string
- func UploadCorpusToS3(ctx context.Context, p *providers3.Provider, cfg BYOS3Config, ...) error
- func ValidateArmMatrix(spec ProfileSpec, r Report) error
- func ValidateCheckpointScaleSchedule(steps []ScheduleStep) error
- func ValidateCheckpointWriterStatsStructure(st reflow.CheckpointWriterStatsRecord) error
- func ValidateReportEnvelope(r Report) error
- func ValidateServiceDemandFormalSet(cells []ServiceDemandCell) error
- func VerifyCorpusImmutable(c GeneratedCorpus) error
- func VerifyManifestUnchanged(path string, wantDigest string) error
- func WriteServiceDemandSetReport(path string, rep ServiceDemandSetReport) error
- type ArmPressureInputs
- type BYOS3Config
- type BindDisposition
- type Budgets
- type CleanupLedger
- type CompactManifest
- type FullPipeOpts
- type GenerateOptions
- type GeneratedCorpus
- type HonestyResult
- type LandedObjectID
- type Manifest
- type ManifestEntry
- type MemoryArm
- type MemoryResolution
- type MintedPoint
- type OccupancyResult
- type Options
- type PairInputs
- type ParsedReflowOutput
- type PointReport
- type PointResult
- type PrefixDeleter
- type ProbeDrainOpts
- type ProfileSpec
- type Recipe
- type Report
- type ScheduleStep
- type ServiceDemandArmConfig
- type ServiceDemandArmReport
- type ServiceDemandCell
- type ServiceDemandDisposition
- type ServiceDemandScheduleID
- type ServiceDemandSetConfig
- type ServiceDemandSetReport
- type StageResult
- type StageRunOpts
- type Tap
- type TapStats
Constants ¶
const ( BindFBlkMin = 0.05 // blocked fraction floor for arm writer pressure BindROccMin = 0.50 // occupancy ratio floor BindFBarMin = 0.10 // barrier-wait capacity fraction floor BindTauE2E = 0.15 // min |Δe2e| for material pair BindEpsTie = 0.02 // |Δe2e| < ε ⇒ tie (not material) BindPairCount = 3 )
Frozen numeric constants (checkpoint measure bind rule — pre-registered).
const ( DefaultRunBudget = 25 * time.Minute DefaultPointBudget = 10 * time.Minute // DefaultCleanupAllowance leaves room after the run context expires for // verified teardown. At the default deletion fan-out, the observed // ~94,000-object teardown takes roughly 10 minutes; 15 minutes preserves // margin for service variance. DefaultCleanupAllowance = 15 * time.Minute )
Default run and per-point budgets. These hold the smoke profile comfortably. They are overridable because they are sized for the default corpus, not for the corpus an operator asks for: a scale override that raises the object count without a matching budget expires mid-run, and the failure surfaces as a context deadline during staging rather than as a scale error.
const ( RunBudgetEnv = "GONIMBUS_THROUGHPUT_RUN_BUDGET" RunBudgetAlias = "RUN_BUDGET" PointBudgetEnv = "GONIMBUS_THROUGHPUT_POINT_TIMEOUT" PointBudgetAlias = "POINT_TIMEOUT" )
Env names for the budgets, with the bare aliases the Make target also accepts.
const ( ProviderFile = "file" ProviderMoto = "moto" ProviderS3Compatible = "s3-compatible" ProviderGCS = "gcs" )
Provider class names used in Options and sanitized reports.
const ( ProfileSmoke = "smoke" ProfileReflowSaturation = "reflow-saturation" ProfileCeilingLift = "ceiling-lift" ProfileCheckpoint = "checkpoint" ProfileCheckpointScale = "checkpoint-scale" ProfileFullPipe = "fullpipe-ab" ProfileProbeSaturation = "probe-saturation" )
Known profile names. Unknown names hard-reject; empty defaults to smoke.
const ( // MemoryArmProbeBound applies no memory override — the product's detection // chain binds the limit (cgroup / physical RAM), and the budget is derived. MemoryArmProbeBound = "probe_bound" // MemoryArmGOMEMLIMIT constrains the limit with an operator GOMEMLIMIT, // which the chain selects when it is the lowest candidate. MemoryArmGOMEMLIMIT = "gomemlimit_constrained" // MemoryArmOperatorBudget sets the budget directly with --memory-budget, // leaving limit detection alone. MemoryArmOperatorBudget = "operator_budget" )
Memory-arm labels. These name which lever bounds the run's memory, not a direction of change: the product binds the LOWEST detected limit, so an operator GOMEMLIMIT constrains the envelope and the absence of any override lets the physical-memory probe bind it.
const ( DefaultSmokeSeed int64 = 42 DefaultSmokeObjects int = 32 DefaultSmokeSizeBytes int = 256 DefaultSmokePartitions int = 2 )
Default smoke sizes stay cheap for local credential-free runs.
const ( DefaultScaleObjects int = 20000 DefaultScaleSizeBytes int = 256 DefaultScalePartitions int = 8 )
Scale corpus defaults for the checkpoint-scale profile. Many tiny objects so per-object checkpoint-write cost dominates copy cost: at a lifted concurrency ceiling the single-connection checkpoint writer — not the copy path — is the binding wall, and the disk-vs-tmpfs class gap becomes measurable. Operators scale the corpus further via the OBJECT_COUNT / SIZE_BYTES / PARTITIONS overrides; these are only the starting point.
const ( MaxRecipeObjectCount int = 5_000_000 MaxRecipeSizeBytes int = 64 << 20 // 64 MiB MaxRecipePartitions int = 4096 MaxTotalCorpusBytes int64 = 64 << 30 // 64 GiB: aggregate cap on materialized source corpus )
Recipe override bounds. Overrides fail closed on absurd values rather than silently materializing a corpus that would exhaust local disk or memory. The per-dimension maxima are not independently sufficient — their product is astronomically large — so an aggregate byte bound caps the materialized corpus regardless of how the dimensions combine.
const ( MeasurementScopeSmallScaleRelative = "small_scale_relative" AbsoluteEnvelopeNotEvaluated = "not_evaluated" )
Measurement-scope labels (public, product-safe — no private planning IDs).
const ( ScheduleDiskFirstOdd = "disk_first_odd" // pair order D/T, T/D, D/T ScheduleTmpfsFirstOdd = "tmpfs_first_odd" // reverse: T/D, D/T, T/D )
Frozen schedule IDs for checkpoint-scale counterbalancing.
const ( // ServiceDemandAdmissionsPerUnit is Mark + Note + Upsert (three admissions). ServiceDemandAdmissionsPerUnit = 3 // ServiceDemandAdmissionsPerArm is the formal work quantum (100k units). ServiceDemandAdmissionsPerArm int64 = 300_000 // ServiceDemandMixID identifies the terminal mutation mix contract. ServiceDemandMixID = "mark_note_upsert_v1" // ServiceDemandMedium is the quorum medium for formal sets on this host class. ServiceDemandMedium = "apfs_disk" // ServiceDemandReportSchema is the sterile set-report schema version. ServiceDemandReportSchema = "gonimbus.service_demand_set.v1" // ServiceDemandProductMaxBatch is the product MaxBatch (unchanged). ServiceDemandProductMaxBatch = 256 )
const ( ServiceDemandUtilMinHighN = 0.85 // util median floor at each of 64..512 ServiceDemandPlateauRatio = 1.05 // mut/s@512 ≤ 1.05 × mut/s@128 ServiceDemandBarrierMeanMul = 2.0 // mean barrier-wait/admission @512 ≥ 2× @64 ServiceDemandMutsCVMax = 0.10 // mut/s CV ceiling on high-N cells ServiceDemandDoesNotUtilMax = 0.50 // util median < 0.50 at 256 and 512 ServiceDemandDoesNotScale = 1.5 // mut/s increases ≥1.5× from 64→256 )
service-demand classifier constants (formal service-demand plan).
const DefaultCorpusUploadConcurrency = 16
DefaultCorpusUploadConcurrency bounds the corpus upload fan-out.
Corpus upload is setup, not measurement, but it is priced in round trips: a serial Put+Head loop costs two round trips per object and cannot exceed 1/(2*RTT) no matter how much capacity the lane has. Against a real S3 endpoint that measured ~4.5 objects/s while the same lane sustained ~85/s at concurrency 16 -- so a 50k corpus spent roughly three hours uploading before the first measurement point, and a run budget sized for the measurement expired during setup.
Concurrency here changes only how fast the fixture is staged. It does not touch the measured child, which is a separate process invoked afterwards.
const DefaultMaxRecordBytes = 4 << 20 // 4 MiB
DefaultMaxRecordBytes bounds a single JSONL line retained for inspection.
const MaxCapturedStderr = 64 << 10 // 64 KiB
MaxCapturedStderr bounds retained stderr per stage for structural diagnostics.
const RecipeVersion = "reflowthroughput-hive-v1"
RecipeVersion identifies the synthetic corpus shape for envelope comparison.
const ReportSchemaVersion = "reflowthroughput.report.v1"
ReportSchemaVersion is the test-tool report schema (not a gonimbus.* product type).
const SyntheticProbeConfigYAML = `` /* 162-byte string literal not displayed */
SyntheticProbeConfigYAML is the sterile probe config matching ObjectContent.
Variables ¶
var ServiceDemandFormalSchedules = map[ServiceDemandScheduleID][]int{ ServiceDemandScheduleR1: {8, 32, 64, 128, 256, 512}, ServiceDemandScheduleR2: {512, 256, 128, 64, 32, 8}, ServiceDemandScheduleR3: {64, 256, 8, 512, 32, 128}, }
ServiceDemandFormalSchedules are the exact three formal rep orders (Day-0 AC). Rep 1→R1, rep 2→R2, rep 3→provenance gate. Pilot rep 0 may reuse R1 and is excluded from formal classifier input.
var ServiceDemandSubmitterGrid = []int{8, 32, 64, 128, 256, 512}
ServiceDemandSubmitterGrid is the freeze-v2 concurrency steps (canonical order).
Functions ¶
func ArmHasWriterPressure ¶
func ArmHasWriterPressure(a ArmPressureInputs) bool
ArmHasWriterPressure implements the frozen per-arm pressure predicate. Pressure requires: StatsOK, A>=1, r_occ >= r_occ^min, and (f_blk >= f_blk^min OR f_bar >= f_bar^min).
func AssertNoSensitiveTokens ¶
AssertNoSensitiveTokens fails if s contains obvious path/bucket/credential tells. Used by the report sterility sentinel.
func BarrierWaitCapacityFraction ¶
func BarrierWaitCapacityFraction(barrierWaitNanos, e2eWallNanos int64, effectiveConcurrency int) float64
BarrierWaitCapacityFraction is f_bar = W_b / (T_ns * C_eff).
BarrierWaitNanos is aggregate concurrent waiter-time; dividing by wall alone yields average concurrent waiters (can exceed 1). Capacity normalization by effective concurrency yields a unitless fraction of arm capacity with no silent cap. C_eff must be >= 1.
func BuildBinary ¶
BuildBinary builds ./cmd/gonimbus into outPath and returns absolute path + sha.
func CLIProbeProviderFlags ¶
func CLIProbeProviderFlags(cfg BYOS3Config) []string
CLIProbeProviderFlags returns the provider flags `content probe` accepts.
Deliberately separate from CLIProviderFlags rather than reusing it. That helper emits the transfer command's paired --src-*/--dest-* flags, which probe does not accept: probe reads one source and has no destination. Handing the transfer set to a probe child would fail on an unknown flag, and handing it a union would give it flags that mean nothing there.
func CLIProviderFlags ¶
func CLIProviderFlags(cfg BYOS3Config) []string
CLIProviderFlags returns transfer reflow --src-* / --dest-* args for BYO S3. Empty strings are omitted (same as release-stress real-cloud pattern).
func CheckCheckpointWriterStatsAdmission ¶
func CheckCheckpointWriterStatsAdmission(p ParsedReflowOutput) error
CheckCheckpointWriterStatsAdmission enforces the measurement-harness fail-closed contract for retained checkpoint-scale evidence (retained-measure admission):
- exactly one gonimbus.reflow.checkpoint_writer_stats.v1 record
- that record appears only after the single summary
- structural integrity of the aggregate snapshot
Product emission may remain best-effort; this gate is the strict enforcement boundary for checkpoint-scale (and any other profile that opts in).
func CheckCounts ¶
func CheckCounts(generated int, tapRows int64, completed int64, summaryErrors int64, invalid int64) error
CheckCounts enforces corpus/tap/terminal agreement for hard structural gates.
func ChildEnv ¶
ChildEnv builds a per-child environment copy. It never calls os.Setenv. When gomemlimit is non-empty it replaces/sets GOMEMLIMIT on the copy only. extra pairs (e.g. moto AWS keys) are appended/replaced without ambient mutation.
func CompareLandedMultisets ¶
func CompareLandedMultisets(a, b []LandedObjectID) error
CompareLandedMultisets returns nil when the ordered identity sets are equal.
func ContentDigest ¶
ContentDigest returns SHA-256 hex of object bytes.
func CorpusUploadConcurrency ¶
func CorpusUploadConcurrency() int
CorpusUploadConcurrency resolves the upload fan-out, allowing an operator override for lanes that want a gentler staging rate.
func CountFilesRecursive ¶
CountFilesRecursive counts regular files under root (post-run object count).
func CountS3Prefix ¶
CountS3Prefix counts objects under prefix (for post-run dest object count).
func CreateMotoBucket ¶
CreateMotoBucket creates a unique moto bucket using the existing cloudtest S3 client (same credentials/endpoint as the moto integration lane).
func DeleteS3PrefixVerified ¶
DeleteS3PrefixVerified deletes all objects under prefix, then lists again and fails if any remain. Stricter than cloudtest's log-on-cleanup-failure helper (measurement harness AC: verified cleanup).
func DigestManifestEntries ¶
func DigestManifestEntries(entries []ManifestEntry) string
DigestManifestEntries returns the ordered entry digest used for envelope identity.
func E2EDelta ¶
E2EDelta computes the frozen pair e2e relative shift. Either arm with T <= 0 is non-comparable (NaN), matching the freeze.
func EnsureAbsent ¶
EnsureAbsent fails if path already exists (fresh checkpoint identity).
func EnsureEmptyDir ¶
EnsureEmptyDir fails if path exists and is non-empty, or creates it empty.
func FileURI ¶
FileURI builds a file:// URI for an absolute path (unix-style slash form). Callers must pass an absolute path; filepath.ToSlash normalization is applied at generation call sites via fileURIFromAbs.
func FullPipeReflowArgs ¶
func FullPipeReflowArgs(opts FullPipeOpts) []string
FullPipeReflowArgs builds the reflow child's argv for a full-pipe point.
func ListProfiles ¶
func ListProfiles() []string
ListProfiles returns known profile names for help text.
func MarshalIndentJSON ¶
MarshalIndentJSON is a helper for golden-style tests.
func MarshalJSONReport ¶
MarshalJSONReport returns the report JSON.
func MotoAvailable ¶
func MotoAvailable() bool
MotoAvailable reports whether a local moto endpoint is reachable (cloudtest).
func NormalizeMeasuredBinaryCommit ¶
NormalizeMeasuredBinaryCommit maps probe output to the measured-binary commit field. Empty becomes "unknown". Worktree/instrument identity must never be passed here (retained-measure provenance) — this helper intentionally has no fallback args.
func OpenS3Provider ¶
func OpenS3Provider(ctx context.Context, cfg BYOS3Config) (*providers3.Provider, error)
OpenS3Provider constructs an S3-compatible provider from BYO config.
func PrepareSourcePrefix ¶
func PrepareSourcePrefix(ledger *CleanupLedger, mint func() string, upload func(prefix string) error) (string, error)
PrepareSourcePrefix mints a source prefix, registers it for cleanup, and only then uploads into it.
The registration sits between the two deliberately. Minting and the first object are separated by an entire corpus upload, and a failure inside that window is the one case a post-upload registration cannot cover.
func ProbeArgs ¶
ProbeArgs builds the content-probe child's argv.
Pure and exported so a control can pin the exact argv without executing a child or reaching a provider. Lifting the cloud guards without proving what each child is actually invoked with would produce an instrument that looks runnable and is misconfigured.
func ResolveProviderClass ¶
ResolveProviderClass normalizes Options.Provider / env. Empty → file. Unknown → error. s3/byo-s3 → s3-compatible.
func SortedHistogramKeys ¶
SortedHistogramKeys returns histogram keys in stable order (for tests).
func SterilityForbidden ¶
SterilityForbidden returns tokens that must never appear in a serialized report.
func UploadCorpusToS3 ¶
func UploadCorpusToS3(ctx context.Context, p *providers3.Provider, cfg BYOS3Config, corpus GeneratedCorpus, sourcePrefix string, outInputPath string) error
UploadCorpusToS3 puts each local corpus object under sourcePrefix and returns a rewritten reflow.input.jsonl path using s3:// source URIs + dest_rel_key.
Uploads run concurrently; the emitted input file stays in manifest order so the reflow input is byte-identical to what the serial form produced. Order is preserved by writing each result into its own slot rather than appending, so completion order cannot leak into the fixture.
func ValidateArmMatrix ¶
func ValidateArmMatrix(spec ProfileSpec, r Report) error
ValidateArmMatrix checks that a report contains exactly the arm × execution shape × parallel × checkpoint cells its profile declares — each one once. Per-point validation cannot see a whole cell that went missing, and "a declared arm is never silently dropped" is precisely what these profiles promise.
func ValidateCheckpointScaleSchedule ¶
func ValidateCheckpointScaleSchedule(steps []ScheduleStep) error
ValidateCheckpointScaleSchedule checks counterbalancing invariants on a plan.
func ValidateCheckpointWriterStatsStructure ¶
func ValidateCheckpointWriterStatsStructure(st reflow.CheckpointWriterStatsRecord) error
ValidateCheckpointWriterStatsStructure pins the sterile aggregate invariants required for an honest retained arm. A successful comparable arm must not carry writer-fatal/closed/canceled barrier outcomes.
func ValidateReportEnvelope ¶
ValidateReportEnvelope performs light structural checks on the report itself.
func ValidateServiceDemandFormalSet ¶
func ValidateServiceDemandFormalSet(cells []ServiceDemandCell) error
ValidateServiceDemandFormalSet is the pure retained-set schedule/honesty validator (E1). A malformed set must never be treated as formal.
func VerifyCorpusImmutable ¶
func VerifyCorpusImmutable(c GeneratedCorpus) error
VerifyCorpusImmutable re-hashes every source object and checks the manifest digest.
func VerifyManifestUnchanged ¶
VerifyManifestUnchanged re-reads the on-disk manifest and checks digest identity.
func WriteServiceDemandSetReport ¶
func WriteServiceDemandSetReport(path string, rep ServiceDemandSetReport) error
WriteServiceDemandSetReport writes the sterile set report as JSON. Uses O_CREATE|O_EXCL|O_WRONLY so the final path cannot replace existing retained evidence (POSIX rename is replace-capable and is not used here). Non-blocking nit: a crash mid-write may leave a partial exclusive file that blocks re-run until the operator removes it.
Types ¶
type ArmPressureInputs ¶
type ArmPressureInputs struct {
Admissions int64
AdmissionBlocked int64
QueueDepthPeak int64
MaxBatch int
BarrierWaitNanos int64
E2EWallNanos int64
EffectiveConcurrency int
// StatsOK is true when admission (exactly-one post-summary + structure) passed.
StatsOK bool
}
ArmPressureInputs is the synthetic/measure surface for one arm's writer pressure.
type BYOS3Config ¶
type BYOS3Config struct {
Bucket string
Endpoint string
Region string
Profile string
RootPrefix string
ForcePathStyle bool
// Optional static keys for moto only (cloudtest.TestAccessKeyID pattern).
// Never used for real BYO — real-cloud uses ambient/profile chain only.
AccessKeyID string
SecretAccessKey string
}
BYOS3Config is the harness view of the existing bring-your-own S3-compatible lane (same env vars as test/cloudtest and make test-cloud-real). Bucket/endpoint/profile never enter the sanitized report.
func LoadBYOS3Config ¶
func LoadBYOS3Config() (cfg BYOS3Config, ok bool)
LoadBYOS3Config reads GONIMBUS_S3_TEST_* (cloudtest constants). Returns ok=false when the opt-in bucket is unset — callers skip, they do not fail.
func (BYOS3Config) ChildAWSEnv ¶
func (c BYOS3Config) ChildAWSEnv() []string
ChildAWSEnv returns extra KEY=value pairs for the gonimbus child when static moto credentials are required. Empty for real BYO (ambient chain / profile).
func (BYOS3Config) MintUniquePrefix ¶
func (c BYOS3Config) MintUniquePrefix(slug string) string
MintUniquePrefix returns a unique object prefix under the operator root, matching cloudtest.CreateS3ObjectPrefix naming shape without testing.T.
func (BYOS3Config) ObjectURI ¶
func (c BYOS3Config) ObjectURI(key string) string
ObjectURI builds s3://bucket/key (not written into reports).
func (BYOS3Config) ProviderConfig ¶
func (c BYOS3Config) ProviderConfig() providers3.Config
ProviderConfig maps to the S3 provider constructor (ambient credential chain for real BYO; optional static keys for moto only).
The connection pool follows the same public policy as product constructions: provider.ResolveConnectionPool(admittedN) with admittedN = corpus upload fan-out. For N >= 2 both fields equal N so concurrent staging is not bound by net/http.DefaultMaxIdleConnsPerHost (2). For N < 2 the zero policy leaves SDK / transport defaults (same as product). Staging must not invent a second transport policy pattern.
type BindDisposition ¶
type BindDisposition string
BindDisposition is the frozen set-level outcome.
const ( BindBinds BindDisposition = "binds" BindDoesNotBind BindDisposition = "does_not_bind" BindInconclusive BindDisposition = "inconclusive" )
func ClassifyBindSet ¶
func ClassifyBindSet(pairs []PairInputs) BindDisposition
ClassifyBindSet implements the frozen three-pair disposition.
binds: both arms pressured on all 3 pairs + material same-sign e2e
does_not_bind: neither arm pressured on all 3 pairs + material same-sign e2e
inconclusive: any invalid pair, any asymmetric (one-arm) pressure, mixed
pressure quorum, or e2e material/direction failure
Asymmetric one-arm pressure must never collapse into does_not_bind.
type Budgets ¶
Budgets is the resolved time envelope for one harness invocation.
func ResolveBudgets ¶
ResolveBudgets reads both budgets through the supplied lookup so the resolution is testable without mutating process environment, and so the wiring — not just the parsing helper — can be pinned by a control.
A set-but-unparseable or non-positive value is an error rather than a silent fallback: falling back would hand the operator a budget that cannot hold the scale they asked for, which is the failure this was introduced to remove.
type CleanupLedger ¶
type CleanupLedger struct {
// Keep suppresses removal entirely. It is an operator choice to retain
// artifacts, not a failure, so residue under Keep is expected.
Keep bool
// DeletePrefix is nil when the run has no object-store residue.
DeletePrefix PrefixDeleter
// contains filtered or unexported fields
}
CleanupLedger owns everything a run mints, so that whatever fails, there is exactly one place that knows what has to be removed.
The ordering rule it exists to enforce: a prefix is registered when it is minted, not when it has been successfully populated. Registering after a successful upload leaves the whole upload window uncovered — a failure partway through strands objects that nothing is responsible for removing.
func (*CleanupLedger) Owned ¶
func (l *CleanupLedger) Owned() (source string, dests []string)
Owned reports the prefixes currently registered, for controls and diagnostics.
func (*CleanupLedger) RegisterDestPrefix ¶
func (l *CleanupLedger) RegisterDestPrefix(prefix string)
RegisterDestPrefix takes ownership of a minted destination prefix.
func (*CleanupLedger) RegisterPoint ¶
func (l *CleanupLedger) RegisterPoint(p MintedPoint)
RegisterPoint takes ownership of a point's local residue.
func (*CleanupLedger) RegisterSourcePrefix ¶
func (l *CleanupLedger) RegisterSourcePrefix(prefix string)
RegisterSourcePrefix takes ownership of the minted source prefix. Call it immediately after minting and before the first upload.
func (*CleanupLedger) Run ¶
func (l *CleanupLedger) Run() error
Run removes everything registered and verifies it is gone.
Every failure is collected rather than the first one returned alone: a destination that would not delete must not hide a source that also would not, because the operator needs to know the full extent of what was left behind.
type CompactManifest ¶
type CompactManifest struct {
RecipeVersion string `json:"recipe_version"`
Seed int64 `json:"seed"`
ObjectCount int `json:"object_count"`
SizeBytes int `json:"size_bytes"`
Partitions int `json:"partitions"`
TotalBytes int64 `json:"total_bytes"`
SizeHistogram map[string]int `json:"size_histogram"`
Digest string `json:"digest"`
}
CompactManifest is the report-safe subset (no path material).
type FullPipeOpts ¶
type FullPipeOpts struct {
Binary string
SourcePrefix string
ProbeConfig string
DestURI string
ProbeConc int
ReflowParallel int
CheckpointPath string
GOMEMLIMIT string
// MemoryBudget is the operator --memory-budget size string; empty omits
// the flag so the product derives the budget from the detected limit.
MemoryBudget string
StdoutPath string
// ProviderClass labels the point in the report. Carried rather than
// assumed, so a cloud run is not reported as a local one.
ProviderClass string
// ProbeExtraArgs are provider flags for the probe child only. Kept separate
// from ReflowExtraArgs because the two commands do not accept the same
// flags: probe takes --region/--profile/--endpoint, transfer takes the
// paired --src-*/--dest-* set. One shared slice would hand each child flags
// the other needs and the command would refuse them.
ProbeExtraArgs []string
// ReflowExtraArgs are provider flags for the reflow child only.
ReflowExtraArgs []string
// ChildExtraEnv is appended to both children's environment. Static
// credentials belong here only for a backend that requires them; a real BYO
// backend stays ambient or profile-based.
ChildExtraEnv []string
// RewriteFrom/RewriteTo map probe output, which carries no dest_rel_key,
// onto destination keys. They are caller-supplied because the source key
// shape is not the same on every backend: a local corpus root yields the
// bare hive key, while an object store prefixes it with the minted run
// prefix, so a fixed template matches on one and rejects every record on
// the other. Empty falls back to the bare four-segment identity.
RewriteFrom string
RewriteTo string
}
FullPipeOpts configures a supervised probe | tap | reflow point. Named fields rather than positional arguments: the several adjacent string parameters (checkpoint path, GOMEMLIMIT, memory budget, stdout path) are otherwise trivially transposable at the call site.
type GenerateOptions ¶
type GenerateOptions struct {
Recipe Recipe
// RunRoot is the external run directory (ownership ledger lives here).
RunRoot string
// SourceDirName is the relative name under RunRoot for the immutable source.
SourceDirName string
}
GenerateOptions controls corpus materialization.
type GeneratedCorpus ¶
type GeneratedCorpus struct {
Recipe Recipe
Root string // absolute source root (operator-supplied test root subdir)
Manifest Manifest
ManifestPath string
ReflowInputPath string
ProbeConfigPath string
ObjectCount int
}
GeneratedCorpus is one verified immutable source corpus for a harness invocation.
func Generate ¶
func Generate(opts GenerateOptions) (GeneratedCorpus, error)
Generate materializes a deterministic corpus under RunRoot and writes reflow-input JSONL + sterile probe config. Source objects are never mutated after return; callers treat the corpus as immutable for the invocation.
type HonestyResult ¶
HonestyResult is the universal ceiling/reason check for one point.
func CheckHonesty ¶
func CheckHonesty(p ParsedReflowOutput, requested int) HonestyResult
CheckHonesty enforces:
0 <= max_active <= sourceCap+destCap+probeCap effective <= requested
Source and probe domains cap at effective. Dest is bounded by the *recorded* dest_domain_ceiling_effective from the product run/summary (not a hard-coded multiplier). Multiplier is cross-checked when present so a non-default ConcurrencyConfig.DestDomainMultiplier cannot silently over-admit relative to the gate. Equality effective==requested requires reason exactly "requested". Lower effective requires resource_capped:* reason AND exactly one clamp warning. Honest caps are valid results, not harness failures.
type LandedObjectID ¶
type LandedObjectID struct {
RelKey string `json:"rel_key"`
Size int64 `json:"size_bytes"`
Digest string `json:"content_digest"`
}
LandedObjectID is a content-identity record for fullpipe A/B parity (relative key + size + content digest). Paths stay relative/sanitized.
func SnapshotDestTree ¶
func SnapshotDestTree(destRoot string) ([]LandedObjectID, error)
SnapshotDestTree walks destRoot and returns a sorted multiset of landed objects. Uses ReadDir recursion (not filepath.Walk) so each open is of a directory entry under the harness-owned dest root, avoiding Walk callback path races.
func SnapshotS3DestPrefix ¶
func SnapshotS3DestPrefix(ctx context.Context, p *providers3.Provider, prefix string) ([]LandedObjectID, error)
SnapshotS3DestPrefix returns a sorted content-identity multiset for objects under prefix, matching SnapshotDestTree's shape.
It exists so the fullpipe A/B parity check keeps its meaning on an object store. The local check walks a destination directory, which a cloud point does not have; skipping it there would quietly drop the guarantee the "ab" in the profile name refers to, on exactly the runs the attribution depends on.
Identity is the provider ETag rather than a recomputed content hash. Both arms are listed the same way and compared only with each other, so the ETag is a sufficient identity for parity without re-downloading the corpus twice.
type Manifest ¶
type Manifest struct {
RecipeVersion string `json:"recipe_version"`
Seed int64 `json:"seed"`
ObjectCount int `json:"object_count"`
// SizeBytes and Partitions are the resolved recipe knobs, recorded so a
// reader can name the exact corpus shape that produced the evidence.
// Partitions in particular cannot be reconstructed from the entry digest.
SizeBytes int `json:"size_bytes"`
Partitions int `json:"partitions"`
TotalBytes int64 `json:"total_bytes"`
SizeHistogram map[string]int `json:"size_histogram"`
Entries []ManifestEntry `json:"entries"`
// Digest is SHA-256 over ordered (relative_key, size, content_digest) lines.
Digest string `json:"digest"`
}
Manifest is the canonical corpus identity bound into every report envelope.
func BuildManifest ¶
func BuildManifest(recipe Recipe, entries []ManifestEntry) (Manifest, error)
BuildManifest constructs and digests a manifest from ordered entries.
func (Manifest) Compact ¶
func (m Manifest) Compact() CompactManifest
Compact returns the allowlisted report view of the manifest.
type ManifestEntry ¶
type ManifestEntry struct {
RelativeKey string `json:"relative_key"`
SizeBytes int64 `json:"size_bytes"`
ContentDigest string `json:"content_digest"`
}
ManifestEntry is one object in the ordered corpus identity set.
type MemoryArm ¶
type MemoryArm struct {
// Label is the reported memory_envelope value.
Label string
// UseGOMEMLIMIT sources the arm's GOMEMLIMIT from operator options.
UseGOMEMLIMIT bool
// UseMemoryBudget sources the arm's --memory-budget from operator options.
UseMemoryBudget bool
}
MemoryArm declares one memory envelope for a profile's points. The concrete GOMEMLIMIT / budget values are operator-supplied at invocation; the harness never invents them.
type MemoryResolution ¶
type MemoryResolution struct {
LimitBytes int64
LimitSource string
BudgetRequestedBytes int64
BudgetEffectiveBytes int64
BudgetSource string
RetryBufferCapBytes int64
}
MemoryResolution is the startup-fixed memory arithmetic one record reports.
type MintedPoint ¶
MintedPoint is the local residue one measurement point owns.
type OccupancyResult ¶
type OccupancyResult struct {
OK bool
Samples []int
Floor int
PassedNeeded int
Passed int
Message string
}
OccupancyResult is the fixed-mode reflow-saturation occupancy check.
func CheckOccupancy ¶
func CheckOccupancy(samples []int, effective int, passCount int) OccupancyResult
CheckOccupancy applies the serial-dispatch detector for fixed-mode reflow saturation.
type Options ¶
type Options struct {
// Binary is the absolute path to the built gonimbus binary.
Binary string
// Profile name (empty → smoke).
Profile string
// Provider is file (default), moto, s3-compatible, or gcs.
// BYO cloud uses the same GONIMBUS_S3_TEST_* / GONIMBUS_GCS_TEST_* env
// conventions as test/cloudtest and make test-cloud-real (opt-in; skip/fail
// clearly when unset for a non-file provider).
Provider string
// RunRoot is the operator-supplied external test root (created if needed).
RunRoot string
// GOMEMLIMIT is operator-supplied only; never auto-set by the harness.
// Under the product's minimum-selection limit chain a GOMEMLIMIT binds
// only when it is the lowest candidate, so this constrains an envelope
// rather than raising one.
GOMEMLIMIT string
// MemoryBudget is the operator --memory-budget value for arms that set the
// budget directly. Operator-supplied only; never invented by the harness.
MemoryBudget string
// TmpfsCheckpointRoot when set is used for checkpoint_class=tmpfs points.
// The path itself is never written into the report.
// Also accepted via env TMPFS_CHECKPOINT_ROOT / GONIMBUS_THROUGHPUT_TMPFS_CHECKPOINT_ROOT.
TmpfsCheckpointRoot string
// Keep retains minted roots (report only retained invocation id).
Keep bool
// PointTimeout bounds each measured point.
PointTimeout time.Duration
// ConstrainedGOMEMLIMIT is the GOMEMLIMIT used by constrained-envelope arms
// of the ceiling-lift and checkpoint profiles. When empty it falls back to
// GOMEMLIMIT. Accepted via CONSTRAINED_GOMEMLIMIT, or the older
// CEILING_LIFT_GOMEMLIMIT spelling.
ConstrainedGOMEMLIMIT string
// WorktreeCommit is optional instrument commit override (harness identity).
// It is NEVER applied to measured BinaryCommit (retained-measure provenance).
WorktreeCommit string
// ScheduleID selects checkpoint-scale counterbalancing plan
// (disk_first_odd default; tmpfs_first_odd reverse). Ignored for other profiles.
ScheduleID string
// Recipe overrides scale the profile's synthetic corpus at invocation. Zero
// means keep the profile default. Accepted via OBJECT_COUNT / SIZE_BYTES /
// PARTITIONS (GONIMBUS_THROUGHPUT_OBJECT_COUNT / _SIZE_BYTES / _PARTITIONS).
// Absurd values fail closed in Recipe.Validate; the effective corpus is
// recorded in the report's corpus block so evidence names what was measured.
RecipeObjectCount int
RecipeSizeBytes int
RecipePartitions int
}
Options configures a harness invocation.
type PairInputs ¶
type PairInputs struct {
// Invalid marks failed/honesty/missing-stats/non-comparable pairs.
Invalid bool
// DiskPressure is true when the disk arm shows writer pressure.
DiskPressure bool
// TmpfsPressure is true when the tmpfs arm shows writer pressure.
TmpfsPressure bool
// DeltaE2E is (T_disk - T_tmpfs) / max(T_disk, T_tmpfs).
DeltaE2E float64
}
PairInputs is one matched disk/tmpfs pair at the disposition layer.
DiskPressure and TmpfsPressure are independent so asymmetric (exactly-one-arm) evidence cannot collapse into "no pair pressure" and falsely yield does_not_bind.
func (PairInputs) AsymmetricArmPressure ¶
func (p PairInputs) AsymmetricArmPressure() bool
AsymmetricArmPressure is true when exactly one arm shows writer pressure.
func (PairInputs) BothArmsPressure ¶
func (p PairInputs) BothArmsPressure() bool
BothArmsPressure is true when each arm independently shows writer pressure.
func (PairInputs) NeitherArmPressure ¶
func (p PairInputs) NeitherArmPressure() bool
NeitherArmPressure is true when neither arm shows writer pressure.
type ParsedReflowOutput ¶
type ParsedReflowOutput struct {
RunCount int
SummaryCount int
WarningClampCount int
ObjectComplete int64
ObjectError int64
InvalidInputs int64
SummaryErrors int64
// Run-record concurrency (required agreement with summary).
RunRequested int
RunEffective int
RunReason string
RunMaxActive int
RunFinal int
RunAdaptive bool
// Summary-record concurrency.
SummaryRequested int
SummaryEffective int
SummaryReason string
SummaryMaxActive int
SummaryFinal int
SummaryAdaptive bool
// Dest-domain policy (GON-067 PR2): static for the run; run and summary must
// agree. Used by CheckHonesty so the gate asserts resolved output, not an
// assumed default multiplier.
RunDestDomainMultiplier int
RunDestDomainCeilingEffective int
SummaryDestDomainMultiplier int
SummaryDestDomainCeilingEffective int
// Memory resolution as the product reported it. The whole tuple is fixed at
// startup, so run and summary must agree on every field — comparing only
// the source labels would let a differing limit, budget, or retry cap
// through and publish the run's values as if both records agreed. Values
// stay aggregate (bytes and source labels) — no host identity is retained.
RunMemory MemoryResolution
SummaryMemory MemoryResolution
// Consensus memory resolution after the agreement check.
MemoryLimitBytes int64
MemoryLimitSource string
MemoryBudgetRequestedBytes int64
MemoryBudgetEffectiveBytes int64
MemoryBudgetSource string
RetryBufferCapBytes int64
// SummaryTimeAvgActive is the completed-run occupancy diagnostic. The run
// record's startup sample is always zero by contract and is not retained.
SummaryTimeAvgActive float64
// CheckpointWriterStats is the sterile end-of-run checkpoint writer
// diagnostic snapshot when the child emitted
// gonimbus.reflow.checkpoint_writer_stats.v1 (checkpoint measure). Nil when absent.
// Measure-first only — not product throughput evidence.
CheckpointWriterStats *reflow.CheckpointWriterStatsRecord
// CheckpointWriterStatsCount is how many stats records appeared (exactly
// one is required for retained checkpoint-scale arms).
CheckpointWriterStatsCount int
// CheckpointWriterStatsBeforeSummary counts stats records observed before
// the single summary (must be zero for retained arms).
CheckpointWriterStatsBeforeSummary int
// ObjectPathStageStats is the sterile object-path stage attribution snapshot
// when the child emitted gonimbus.reflow.object_path_stage_stats.v1.
// Optional; measure-first only — not product throughput marketing.
ObjectPathStageStats *reflow.ObjectPathStageStatsRecord
// ObjectPathStageStatsCount is how many stage-stats records appeared.
ObjectPathStageStatsCount int
// Consensus fields after agreement check (filled by Finalize).
Requested int
Effective int
Reason string
MaxActive int
Final int
AdaptiveEnabled bool
Initial int
Floor int
// Consensus dest-domain policy after run/summary agreement.
DestDomainMultiplier int
DestDomainCeilingEffective int
ThrottleBackoffs int64
AdditiveIncreases int64
ConnectionFreezes int64
ClampWarningOK bool
// Clamp warning details for three-way agreement with run/summary.
ClampWarnRequested int
ClampWarnEffective int
ClampWarnReason string
ClampWarnAdaptive *bool
ParseErrors []string
UnknownTypes []string
// contains filtered or unexported fields
}
ParsedReflowOutput is the allowlist-extracted view of reflow child stdout. It never retains destination URI or checkpoint path from product records.
func ParseReflowFile ¶
func ParseReflowFile(path string) (ParsedReflowOutput, error)
ParseReflowFile opens path and parses incrementally.
func ParseReflowReader ¶
func ParseReflowReader(r io.Reader) (ParsedReflowOutput, error)
ParseReflowReader streams JSONL from r with bounded line reading.
func ParseReflowStdout ¶
func ParseReflowStdout(stdout []byte) (ParsedReflowOutput, error)
ParseReflowStdout extracts concurrency and count fields by record type. Malformed JSONL objects and unknown types are hard failures.
type PointReport ¶
type PointReport struct {
PointID string `json:"point_id"`
ExecutionShape string `json:"execution_shape"` // reflow_only|full_pipe|probe_drain
Parallel int `json:"reflow_parallel_requested,omitempty"`
ProbeConcurrency int `json:"probe_concurrency,omitempty"`
// AdaptiveMode is reflow adaptive/fixed when reflow ran; empty for probe_drain.
AdaptiveMode string `json:"adaptive_mode,omitempty"`
GOMEMLIMITSet bool `json:"gomemlimit_set"`
GOMEMLIMITValue string `json:"gomemlimit_value,omitempty"`
// MemoryBudgetRequested is the operator --memory-budget passed to the child
// (empty when the arm let the product derive the budget).
MemoryBudgetRequested string `json:"memory_budget_requested,omitempty"`
// MemoryLimitSource is product memory-limit provenance when reflow ran; omit for probe_drain.
MemoryLimitSource string `json:"memory_limit_source,omitempty"`
// MemoryBudgetSource is product budget provenance: derived|operator|operator_clamped_to_limit.
MemoryBudgetSource string `json:"memory_budget_source,omitempty"`
// Resolved memory arithmetic as the product reported it (aggregate bytes only).
MemoryLimitBytes int64 `json:"memory_limit_bytes,omitempty"`
MemoryBudgetEffectiveBytes int64 `json:"memory_budget_effective_bytes,omitempty"`
RetryBufferCapBytes int64 `json:"retry_buffer_cap_bytes,omitempty"`
// MemoryEnvelope names which lever bound this arm's memory:
// gomemlimit_constrained | probe_bound | operator_budget | "".
MemoryEnvelope string `json:"memory_envelope,omitempty"`
// ConcurrencyTimeAvgActive is the completed-run occupancy diagnostic.
ConcurrencyTimeAvgActive float64 `json:"concurrency_time_avg_active,omitempty"`
CheckpointClass string `json:"checkpoint_class,omitempty"`
// Product reflow concurrency telemetry (omit when not applicable).
ConcurrencyRequested *int `json:"concurrency_ceiling_requested,omitempty"`
ConcurrencyEffective *int `json:"concurrency_ceiling_effective,omitempty"`
ConcurrencyReason *string `json:"concurrency_ceiling_reason,omitempty"`
ConcurrencyMaxActive *int `json:"concurrency_max_active,omitempty"`
ConcurrencyFinal *int `json:"concurrency_final,omitempty"`
AdaptiveEnabled *bool `json:"adaptive_enabled,omitempty"`
// CheckpointWriterStats is sterile checkpoint-writer diagnostics from the
// child when emitted (checkpoint measure). Measure-first only — not product
// throughput marketing. Omitted when the child did not emit the record.
CheckpointWriterStats *reflow.CheckpointWriterStatsRecord `json:"checkpoint_writer_stats,omitempty"`
ElapsedSeconds float64 `json:"elapsed_seconds"`
CompletedObjects int64 `json:"completed_objects"`
EndToEndRate float64 `json:"end_to_end_rate_objects_per_s,omitempty"`
ProbeDeliveredRate float64 `json:"probe_delivered_rate_objects_per_s,omitempty"`
ProbeSaturationRate float64 `json:"probe_saturation_rate_objects_per_s,omitempty"`
TapValidRows int64 `json:"tap_valid_reflow_input_rows,omitempty"`
// TapCopyIntervalSeconds is the wall time of the tap Copy loop (includes
// producer pacing and downstream backpressure). Not calibrated overhead.
TapCopyIntervalSeconds float64 `json:"tap_copy_interval_seconds,omitempty"`
// HonestyOK is nil when honesty is not applicable (e.g. probe_drain).
HonestyOK *bool `json:"honesty_ok,omitempty"`
HonestyMessage string `json:"honesty_message,omitempty"`
OccupancySamples []int `json:"occupancy_samples,omitempty"`
OccupancyOK *bool `json:"occupancy_ok,omitempty"`
OccupancyMessage string `json:"occupancy_message,omitempty"`
// ContentParityOK is true when landed key+size+content-digest multisets match.
ContentParityOK *bool `json:"content_parity_ok,omitempty"`
StageExitCodes map[string]int `json:"stage_exit_codes"`
}
PointReport is one measured arm with allowlisted aggregate fields only. Reflow concurrency fields are pointers so probe_drain omits them (no product reflow telemetry exists for that shape).
type PointResult ¶
type PointResult struct {
Profile string
PointID string
Parallel int
ProbeConcurrency int
GOMEMLIMIT string
// MemoryBudget is the operator --memory-budget value passed to the child
// (empty when the arm let the product derive the budget).
MemoryBudget string
CheckpointClass string // disk | tmpfs — never the path
ProviderClass string // file | moto | s3-compatible | gcs
Stages map[string]StageResult
Tap TapStats
Stdout []byte // small in-memory only; prefer StdoutPath for scale
StdoutPath string // external run artifact for reflow JSONL
Elapsed time.Duration
// CompletedObjects is derived from summary/object counts when available.
CompletedObjects int64
WallRate float64 // objects / elapsed seconds (0 if incomplete)
}
PointResult aggregates one measured sweep point.
func RunFullPipe ¶
func RunFullPipe(ctx context.Context, opts FullPipeOpts) (PointResult, error)
RunFullPipe supervises probe | tap | reflow without a shell. Shared cancel on any stage/tap failure; rewrite templates supply dest mapping for probe-emitted reflow.input (no dest_rel_key).
func RunProbeDrain ¶
func RunProbeDrain(ctx context.Context, opts ProbeDrainOpts) (PointResult, error)
RunProbeDrain runs content probe into a draining tap sink (no reflow). This is the methodology-faithful producer-only probe saturation shape.
func RunReflowOnly ¶
func RunReflowOnly(ctx context.Context, opts StageRunOpts) (PointResult, error)
RunReflowOnly executes: binary transfer reflow --stdin ... with pre-materialized input.
type PrefixDeleter ¶
PrefixDeleter removes every object under a minted object-store prefix and verifies none remain. Injected so the cleanup lifecycle can be controlled without a provider: the failure modes that matter are ordering and error propagation, and neither needs a bucket to exercise.
type ProbeDrainOpts ¶
type ProbeDrainOpts struct {
Binary string
SourcePrefix string
ProbeConfig string
ProbeConc int
GOMEMLIMIT string
ProviderClass string
ProbeExtraArgs []string
ChildExtraEnv []string
}
ProbeDrainOpts configures a probe-only point with a draining sink.
Named fields for the same reason as FullPipeOpts: the positional form had grown a run of adjacent strings a caller can transpose without the compiler noticing, and provider support adds more of them.
type ProfileSpec ¶
type ProfileSpec struct {
Name string
Recipe Recipe
// ExecutionShape: reflow_only | full_pipe
ExecutionShape string
// ParallelPoints for reflow --parallel sweeps.
ParallelPoints []int
// ProbeConcurrencyPoints for probe sweeps / full-pipe pairs.
ProbeConcurrencyPoints []int
// FullPipePairs is (probe, reflow) pairs for full-pipe profiles.
FullPipePairs [][2]int
// RequireOccupancy enables fixed-mode saturation occupancy sampling.
RequireOccupancy bool
// NoAdaptive forces --no-adaptive.
NoAdaptive bool
// MemoryArms declares the memory envelopes each parallel point is run
// under. Empty means a single arm with no memory override. See MemoryArm.
MemoryArms []MemoryArm
// CheckpointClasses lists disk and/or tmpfs discriminators.
CheckpointClasses []string
// Description is operator-facing sterile help.
Description string
}
ProfileSpec declares a named measurement profile.
func ResolveProfile ¶
func ResolveProfile(name string) (ProfileSpec, error)
ResolveProfile maps a name to a spec. Empty name → smoke.
type Recipe ¶
type Recipe struct {
Version string `json:"recipe_version"`
Seed int64 `json:"seed"`
ObjectCount int `json:"object_count"`
SizeBytes int `json:"size_bytes"`
Partitions int `json:"partitions"`
// FixedDate is a sterile calendar token embedded in keys (not a field identity).
FixedDate string `json:"fixed_date"`
}
Recipe describes a deterministic synthetic hive-shaped corpus. Keys are generic (entity/device/date) with no client keyspace identity.
func DefaultSmokeRecipe ¶
func DefaultSmokeRecipe() Recipe
DefaultSmokeRecipe returns the cheapest credential-free local smoke recipe.
func SaturationRecipe ¶
func SaturationRecipe() Recipe
SaturationRecipe returns a larger local recipe for reflow occupancy checks. Object count targets roughly 32× a modest effective ceiling of 8 for smoke-scale saturation; operators select scale profiles explicitly.
func ScaleRecipe ¶
func ScaleRecipe() Recipe
ScaleRecipe returns the checkpoint-scale corpus (see the scale defaults above).
func (Recipe) ObjectContent ¶
ObjectContent returns byte-deterministic XML content for object index i. A probeable <Marker> field is near the head for full-pipe profiles.
func (Recipe) RelativeKey ¶
RelativeKey returns the sterile relative key for object index i. Shape: entity=%04d/device=%04d/date=<fixed>/object-%08d.xml
type Report ¶
type Report struct {
SchemaVersion string `json:"schema_version"`
BinaryVersion string `json:"binary_version,omitempty"`
BinaryCommit string `json:"binary_commit,omitempty"`
BinarySHA256 string `json:"binary_sha256"`
// Instrument identity is the harness (test binary / worktree), distinct from
// the measured child Binary* fields (checkpoint measure / provenance gate). Never backfill
// BinaryCommit from instrument HEAD.
InstrumentCommit string `json:"instrument_commit,omitempty"`
InstrumentSHA256 string `json:"instrument_sha256,omitempty"`
InstrumentDirty bool `json:"instrument_dirty"` // always present: false=clean, true=dirty (probe fail aborts run)
// Checkpoint-scale schedule provenance (empty for other profiles).
ScheduleID string `json:"schedule_id,omitempty"`
ScheduleOrder []string `json:"schedule_order,omitempty"` // e.g. "warm:disk:64", "p1:disk:64"
Corpus CompactManifest `json:"corpus"`
Profile string `json:"profile"`
ProviderClass string `json:"provider_class"`
ThroughputEvidenceClass string `json:"throughput_evidence_class"`
// MeasurementScope declares this cut's intent (small-scale honesty/relative).
MeasurementScope string `json:"measurement_scope"`
// AbsoluteEnvelopeValidation is not_evaluated until an opt-in BYO-cloud
// scale lane is used; avoids presenting local/moto as a throughput oracle.
AbsoluteEnvelopeValidation string `json:"absolute_envelope_validation"`
OS string `json:"os"`
Arch string `json:"arch"`
InvocationID string `json:"invocation_id"`
StartedAt time.Time `json:"started_at"`
Keep bool `json:"keep"`
Points []PointReport `json:"points"`
}
Report is the sanitized, versioned measurement report. It must never include endpoints, credentials, bucket names, signed URLs, checkpoint paths, or local absolute paths.
type ScheduleStep ¶
type ScheduleStep struct {
// PairIndex is 0 for warmups; 1..N for retained matched pairs.
PairIndex int
// Warmup marks discarded arms (not retained in report / Δe2e).
Warmup bool
Parallel int
CheckpointClass string // disk | tmpfs
MemoryEnvelope string
// Ordinal is 0-based position in the full execution order (including warmups).
Ordinal int
}
ScheduleStep is one planned arm in a checkpoint-scale run.
func CheckpointScaleSchedule ¶
func CheckpointScaleSchedule(spec ProfileSpec, scheduleID string) ([]ScheduleStep, error)
CheckpointScaleSchedule builds the frozen counterbalanced plan for a profile. scheduleID must be ScheduleDiskFirstOdd or ScheduleTmpfsFirstOdd.
Layout:
- one discarded warmup per class (first parallel point, first memory arm)
- three matched pairs over ParallelPoints in order, alternating first-class as D/T, T/D, D/T (or reverse for tmpfs_first_odd)
type ServiceDemandArmConfig ¶
type ServiceDemandArmConfig struct {
// StorePath is the APFS (or other quorum medium) checkpoint path.
StorePath string
// Submitters is fixed concurrent mutation callers.
Submitters int
// Units is the number of terminal units (each = 3 admissions). 0 → formal quantum.
Units int
// MinAdmissions overrides the formal work quantum when > 0 (tests only).
MinAdmissions int64
// ElideRawExecSavepoints enables experimental Phase A elision for this arm.
ElideRawExecSavepoints bool
}
ServiceDemandArmConfig configures one timed arm.
type ServiceDemandArmReport ¶
type ServiceDemandArmReport struct {
Submitters int `json:"submitters"`
Units int `json:"units"`
AdmissionsTarget int64 `json:"admissions_target"`
WallSeconds float64 `json:"wall_seconds"`
MutationsPerSec float64 `json:"mutations_per_sec"`
TransactionsPerSec float64 `json:"transactions_per_sec"`
WriterUtilization float64 `json:"writer_utilization"`
BarrierWaitLoad float64 `json:"barrier_wait_load"`
BarrierWaitMeanNanos float64 `json:"barrier_wait_mean_nanos"`
BarrierWaitMaxNanos int64 `json:"barrier_wait_max_nanos"`
AdmissionsPerBatch float64 `json:"admissions_per_batch"`
ReachableOccupancy float64 `json:"reachable_occupancy"`
AdmissionWaitMaxNs int64 `json:"admission_wait_max_nanos"`
AdmissionBlocked int64 `json:"admission_blocked"`
Honest bool `json:"honest"`
HonestyMessage string `json:"honesty_message,omitempty"`
WriterStats reflow.CheckpointWriterStatsRecord `json:"writer_stats"`
}
ServiceDemandArmReport is a sterile arm result (no paths in JSON export helpers).
func RunServiceDemandArm ¶
func RunServiceDemandArm(ctx context.Context, cfg ServiceDemandArmConfig) (ServiceDemandArmReport, error)
RunServiceDemandArm opens a fresh store, runs the terminal mutation mix under fixed submitter concurrency, and returns sterile metrics from WriterStats.
Timed region: MarkDestKeyObserved → NoteDestKeySource → UpsertItem(complete) only. Key formatting is outside the timed loop (Day-0 mix-order AC).
type ServiceDemandCell ¶
type ServiceDemandCell struct {
Rep int `json:"rep"`
Ordinal int `json:"ordinal"`
ScheduleID ServiceDemandScheduleID `json:"schedule_id"`
Submitters int `json:"submitters"`
ServiceDemandArmReport
}
ServiceDemandCell is one arm inside a multi-rep set (sterile; no store path).
type ServiceDemandDisposition ¶
type ServiceDemandDisposition string
ServiceDemandDisposition is the pure classifier outcome for a formal set.
const ( ServiceDemandMechanismBinds ServiceDemandDisposition = "mechanism_binds" ServiceDemandMechanismDoesNot ServiceDemandDisposition = "mechanism_does_not_bind" ServiceDemandInconclusive ServiceDemandDisposition = "inconclusive" )
func ClassifyServiceDemandV2 ¶
func ClassifyServiceDemandV2(cells []ServiceDemandCell) (ServiceDemandDisposition, string)
ClassifyServiceDemandV2 is a pure classifier over formal set cells.
mechanism_binds requires formal shape (3 reps × full grid × exact 300k admissions on every cell) plus all of:
- util median ≥ 0.85 at each of {64,128,256,512}
- mut/s@512 ≤ 1.05 × mut/s@128 (plateau) OR decline across two adjacent of {128,256,512}
- mean barrier-wait/admission @512 ≥ 2 × mean @64
- mut/s CV ≤ 0.10 at each high-N point with n≥3 samples (else inconclusive)
mechanism_does_not_bind: same formal shape; util median < 0.50 at 256 and 512, and mut/s increases ≥1.5× from 64→256. Otherwise: inconclusive.
type ServiceDemandScheduleID ¶
type ServiceDemandScheduleID string
ServiceDemandScheduleID names a fixed formal schedule.
const ( ServiceDemandScheduleR1 ServiceDemandScheduleID = "R1" ServiceDemandScheduleR2 ServiceDemandScheduleID = "R2" ServiceDemandScheduleR3 ServiceDemandScheduleID = "provenance gate" )
func FormalScheduleForRep ¶
func FormalScheduleForRep(rep int) (ServiceDemandScheduleID, error)
FormalScheduleForRep maps formal rep index (1..3) to schedule ID.
type ServiceDemandSetConfig ¶
type ServiceDemandSetConfig struct {
// RootDir holds per-arm SQLite files (fresh unique DB per arm).
RootDir string
// FormalReps is the list of formal rep indices (default 1,2,3). Pilot 0 excluded.
FormalReps []int
// Units overrides formal quantum for tests (0 → formal 100k units).
Units int
// MinAdmissions overrides formal quantum for tests.
MinAdmissions int64
// WorktreeCommitOverride is optional instrument commit override (tests).
// When set, the set report is always non-formal (test-only identity).
WorktreeCommitOverride string
// MediumOverride labels reduced/non-formal structural runs (e.g. "test_tmp").
// Empty + formal path → verified APFS medium (or set fails formal admission).
MediumOverride string
// SkipAPFSCheck allows structural tests on non-APFS roots; forces non-formal.
SkipAPFSCheck bool
// ElideRawExecSavepoints enables Phase A elision for every arm in the set.
// Sets with elision are never formal baseline authority (wantFormalShape false).
ElideRawExecSavepoints bool
}
ServiceDemandSetConfig configures RunServiceDemandSet.
type ServiceDemandSetReport ¶
type ServiceDemandSetReport struct {
SchemaVersion string `json:"schema_version"`
Medium string `json:"medium"`
MixID string `json:"mix_id"`
AdmissionsPerArm int64 `json:"admissions_per_arm"`
// Formal is true only for the default Day-0 shape: reps {1,2,3}, full
// submitter grid, exact ServiceDemandAdmissionsPerArm on every cell.
// Reduced Units/MinAdmissions/FormalReps sets are non-formal (structure only).
Formal bool `json:"formal"`
InstrumentSHABefore string `json:"instrument_sha_before"`
InstrumentCommitBefore string `json:"instrument_commit_before"`
InstrumentDirtyBefore bool `json:"instrument_dirty_before"`
InstrumentSHAAfter string `json:"instrument_sha_after"`
InstrumentCommitAfter string `json:"instrument_commit_after"`
InstrumentDirtyAfter bool `json:"instrument_dirty_after"`
Cells []ServiceDemandCell `json:"cells"`
Disposition ServiceDemandDisposition `json:"disposition"`
DispositionMessage string `json:"disposition_message,omitempty"`
}
ServiceDemandSetReport is the versioned sterile multi-rep report.
func RunFormalServiceDemandBaseline ¶
func RunFormalServiceDemandBaseline(ctx context.Context, rootDir, reportPath string) (ServiceDemandSetReport, error)
RunFormalServiceDemandBaseline is the gated formal-run entry: default 3×6×300k under RootDir, writes the sterile set report to reportPath (required, no overwrite).
func RunServiceDemandSet ¶
func RunServiceDemandSet(ctx context.Context, cfg ServiceDemandSetConfig) (ServiceDemandSetReport, error)
RunServiceDemandSet runs formal reps under fixed schedules (18 cells for 3×6). Captures instrument identity before/after; refuses set if instrument drifts.
type StageResult ¶
type StageResult struct {
Name string `json:"name"`
ExitCode int `json:"exit_code"`
Err string `json:"error,omitempty"`
Stderr string `json:"stderr_excerpt,omitempty"`
Duration time.Duration `json:"duration_ns"`
StartedAt time.Time `json:"-"`
FinishedAt time.Time `json:"-"`
}
StageResult is the independent disposition of one supervised child.
type StageRunOpts ¶
type StageRunOpts struct {
Binary string
InputPath string
DestURI string
Parallel int
CheckpointPath string
GOMEMLIMIT string
// MemoryBudget is the operator --memory-budget size string. Empty omits the
// flag entirely, so the product derives the budget from the detected limit.
MemoryBudget string
NoAdaptive bool
ProviderClass string
ExtraArgs []string
ChildExtraEnv []string
// StdoutPath streams reflow JSONL to a file (bounded memory). Required for scale.
StdoutPath string
// RewriteFrom/To map probe output without dest_rel_key onto destinations.
RewriteFrom string
RewriteTo string
}
StageRunOpts configures a supervised child stage.
type Tap ¶
type Tap struct {
MaxRecordBytes int
AcceptReflowInputOnly bool
// contains filtered or unexported fields
}
Tap is a bounded raw-byte pass-through: forwards fragments immediately with backpressure, frames on newline for counting, and hard-fails on oversize/malformed/unexpected records via strict JSON decode.
type TapStats ¶
type TapStats struct {
ValidReflowInputRows int64
UnexpectedRows int64
BytesForwarded int64
FirstForward time.Time
LastForward time.Time
// CopyDuration is wall time of the tap Copy loop (includes producer pacing
// and downstream backpressure). Not calibrated isolated overhead.
CopyDuration time.Duration
}
TapStats records transparent pass-through metrics for the probe→reflow pipe.
func (TapStats) ActiveInterval ¶
ActiveInterval returns last-first duration for probe-delivered rate windows.