README
¶
Native Context Evaluation
Run the deterministic four-mode harness from the repository root:
go run ./internal/testkit/nativecontext/cmd/evaluate -json-only
The command emits one machine-readable JSON report on stdout. Without
-json-only, it also writes a concise summary to stderr. The report uses fixed
seed 104729, the native-context-fixed-v1 emulator fixture, paired
baseline, reasoning_only, compaction_only, and full results, numeric
baseline/full comparisons, and one-time compaction break-even turns.
The fixture is evaluation plumbing only. quality_claim is always
observed_only; it does not make provider or task-quality claims.
Documentation
¶
Overview ¶
Package nativecontext contains deterministic, content-safe evidence helpers for the experimental Codex native-context integration and quality suites.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var Modes = [...]Mode{ModeBaseline, ModeReasoningOnly, ModeCompactionOnly, ModeFull}
Functions ¶
func BreakEvenTurns ¶
BreakEvenTurns reports the first turn where full-mode compaction savings cover its one-time cost against baseline. It returns 0 when the deterministic sample does not break even; this is evidence, not a quality claim.
Types ¶
type Metrics ¶
type Metrics struct {
TaskQualityPass bool `json:"task_quality_pass"`
TestsPassed bool `json:"tests_passed"`
Repeats int `json:"repeats"`
Contradictions int `json:"contradictions"`
Rediscovery int `json:"rediscovery"`
InvalidatedDecisions int `json:"invalidated_decisions"`
Turns int `json:"turns"`
ToolCalls int `json:"tool_calls"`
ProviderInputTokens int64 `json:"provider_input_tokens"`
ProviderOutputTokens int64 `json:"provider_output_tokens"`
ProviderReasoningTok int64 `json:"provider_reasoning_tokens"`
CacheReadTokens int64 `json:"cache_read_tokens"`
CacheWriteTokens int64 `json:"cache_write_tokens"`
CompactionCostTokens int64 `json:"compaction_cost_tokens"`
LatencyMS float64 `json:"latency_ms"`
TTFTMS float64 `json:"ttft_ms"`
ContextBytes int64 `json:"context_bytes"`
CheckpointReuse int `json:"checkpoint_reuse"`
Failures int `json:"failures"`
}
Metrics is intentionally numeric and content-free. It is suitable for JSON output and paired comparisons without retaining prompts or opaque state.
type PairedComparison ¶
type PairedComparison struct {
TaskID string `json:"task_id"`
BaselineInputTokens int64 `json:"baseline_input_tokens"`
FullInputTokens int64 `json:"full_input_tokens"`
InputSavingsPerTurn int64 `json:"input_savings_per_turn"`
CompactionCostTokens int64 `json:"compaction_cost_tokens"`
BreakEvenTurns int `json:"break_even_turns"`
}
PairedComparison contains numeric baseline/full deltas for one task.
type Report ¶
type Report struct {
SchemaVersion string `json:"schema_version"`
Seed int64 `json:"seed"`
Environment map[string]string `json:"environment"`
Results []Result `json:"results"`
Comparisons []PairedComparison `json:"comparisons"`
Summary Summary `json:"summary"`
Live bool `json:"live"`
}
Report is the machine-readable harness output. Summary is kept concise and uses fixed keys so CI can consume it without parsing prose.
func DeterministicReport ¶
func DeterministicReport() Report
DeterministicReport returns a fixed-seed, emulator-only ablation report. The values are harness fixtures, not provider observations or quality claims.
Source Files
Directories