common

package
v2.3.4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 2, 2026 License: MIT Imports: 19 Imported by: 0

Documentation

Overview

Package common — LLM-as-Judge evaluator (Ollama-only).

This file replaces the legacy cloud-provider judge (OpenAI / Anthropic) with a strict Ollama-only implementation. The rewrite closes all 10 gaps catalogued in the G1 spike (docs/q1-2026/spikes/g1-judge-model.md §6):

  1. Provider architecture → Ollama only; no cloud fallback.
  2. OLLAMA_ENDPOINT env wired (default http://localhost:11434).
  3. format:"json" + structured JudgeResult{Verdict, Score, Reasoning}.
  4. temperature=0 + seed=42 always set for determinism.
  5. Typed JudgeResult public return + backward-compat JudgeAnswer(float64).
  6. Test surface (llm_judge_test.go) covers HTTP, retry, timeout, streaming.
  7. Retry once on transient 5xx; no retry on 4xx.
  8. Configurable judge model via OLLAMA_JUDGE_MODEL (default qwen2.5:7b-instruct).
  9. Cross-platform: pure stdlib, no OS-specific path/scheme assumptions.
  10. Ollama NDJSON streaming responses are accumulated, not just first chunk.

Function signatures preserved for backward compatibility with the benchmark runners (bench/locomo, bench/longmemeval, bench/dmr):

  • JudgeConfig (struct; callers only check != nil)
  • DefaultJudgeConfig() *JudgeConfig
  • JudgeAnswer(cfg *JudgeConfig, q, expected, got string) (float64, error)

New preferred API:

  • JudgeResult{Verdict string, Score float64, Reasoning string}
  • Judge(ctx context.Context, cfg *JudgeConfig, q, expected, got string) (JudgeResult, error)

Package common provides shared utilities for benchmark evaluation.

Index

Constants

View Source 
const (
	// CapabilityExecutedCurrentPath marks an authority dimension enforced by the
	// production retrieval path exercised by this corpus.
	CapabilityExecutedCurrentPath = "executed_current_path"
	// CapabilityNotExecuted prevents future authority labels from being reported
	// as current-path conformance evidence.
	CapabilityNotExecuted = "not_executed_capability"

	CoverageRanking           = "ranking"
	CoverageFilterCollision   = "filter_collision"
	CoverageLifecycleTemporal = "lifecycle_temporal"
	CoverageProvenance        = "provenance"
	CoverageNoAnswer          = "no_answer"
)
View Source 
const (
	// DirectionHigherIsBetter and DirectionLowerIsBetter are the only supported
	// numeric gate directions.
	DirectionHigherIsBetter = "higher_is_better"
	DirectionLowerIsBetter  = "lower_is_better"

	// Blocking policies distinguish non-negotiable correctness from
	// profile-specific release criteria and advisory observations.
	BlockingUniversal = "universal"
	BlockingProfile   = "profile"
	BlockingAdvisory  = "advisory"

	// Fixed correctness rules are predicates, not baseline-derived thresholds.
	RuleZeroViolations   = "zero_violations"
	RuleExactFilterMatch = "exact_filter_match"

	GateIDZeroIsolationLeakage   = "zero-isolation-leakage"
	GateIDExactFilterCorrectness = "exact-filter-correctness"
)

Variables

This section is empty.

Functions

func EvidenceRecall 

func EvidenceRecall(retrieved, relevant []EvidenceRef) float64

EvidenceRecall returns exact labelled-evidence recall. Stable episode and fact IDs match by ID; spans match by episode ID and exact half-open offsets.

func F1Score 

func F1Score(prediction, reference string) float64

F1Score computes token-level F1 between prediction and reference.

func FilterEligibilityExact 

func FilterEligibilityExact(returned, eligible []string) bool

FilterEligibilityExact reports exact equality between returned and eligible stable-ID sets. Ordering and duplicate occurrences do not change set parity.

func IsolationViolationCount 

func IsolationViolationCount(retrieved, excluded []string) int

IsolationViolationCount counts every returned occurrence of an excluded ID. It is an absolute blocking correctness count, not an averaged quality score.

func JudgeAnswer 

func JudgeAnswer(cfg *JudgeConfig, question, expected, got string) (float64, error)

JudgeAnswer is the backward-compatible float64-returning wrapper used by the benchmark runners (bench/locomo, bench/longmemeval, bench/dmr). Runners gate on `judgeScore > 0.5` for "correct", which keeps working because partial verdicts map to the model's own score field.

Sentinel preserved: returns (-1, err) when cfg is nil, matching the legacy behaviour that callers use as a signal to fall back to F1.

func MRR 

func MRR(retrieved, relevant []string) float64

MRR returns the reciprocal rank of the first unique relevant result. Duplicate results occupy ranks but cannot receive relevance credit twice.

func MarshalReproAnalysis 

func MarshalReproAnalysis(analysis ReproAnalysis) ([]byte, error)

MarshalReproAnalysis emits deterministic JSON while preserving run order.

func NDCGAtK 

func NDCGAtK(retrieved []string, relevance map[string]float64, k int) float64

NDCGAtK computes normalized discounted cumulative gain using exponential gain for positive graded relevance. Duplicate results occupy ranks and gain credit only at their first occurrence.

func NoAnswerCorrect 

func NoAnswerCorrect(noAnswer bool, retrieved []string, abstained bool) bool

NoAnswerCorrect reports whether no-answer/abstention behavior matches the label. A no-answer query requires an explicit abstention and zero results.

func RecallAtK 

func RecallAtK(retrieved, relevant []string, k int) float64

RecallAtK returns the fraction of unique relevant IDs present in the first k results. Duplicate results occupy ranks but receive credit only once.

func RougeL 

func RougeL(prediction, reference string) float64

RougeL computes ROUGE-L F1 score using longest common subsequence.

func SerializeEvidenceReport 

func SerializeEvidenceReport(report EvidenceReport) ([]byte, error)

SerializeEvidenceReport validates and emits stable, human-readable JSON. Order-insensitive report collections are sorted on a copy; ranked outputs retain their observed order.

Types

type BenchStores 

type BenchStores struct {
	App      *app.App
	Embedder embedding.Service
}

BenchStores wraps a Cortex app instance for benchmark evaluation.

func NewBenchStores 

func NewBenchStores() (*BenchStores, error)

NewBenchStores creates an in-memory Cortex instance for benchmarking. If useEmbeddings is true, tries to connect to Ollama for vector search.

func NewBenchStoresWithEmbeddings 

func NewBenchStoresWithEmbeddings(cfg embedding.Config) (*BenchStores, error)

NewBenchStoresWithEmbeddings creates a bench store with embedding support.

func (*BenchStores) Close 

func (bs *BenchStores) Close() error

Close cleans up the benchmark database and embedding service idle HTTP connections. The embedding backend (if configured) implements io.Closer; type-asserting here reaps its Transport's persistConn goroutines without bloating the embedding.Service interface.

func (*BenchStores) EmbedQuery 

func (bs *BenchStores) EmbedQuery(ctx context.Context, query string) []float32

EmbedQuery generates an embedding for a search query. Returns nil if embeddings are not enabled.

func (*BenchStores) IngestSession 

func (bs *BenchStores) IngestSession(ctx context.Context, sessionID, project string, observations []domain.Observation) error

IngestSession creates a session and its observations. If embeddings are enabled, each observation is also embedded.

type BenchmarkResult 

type BenchmarkResult struct {
	Benchmark string             `json:"benchmark"`
	Overall   float64            `json:"overall_accuracy"`
	ByType    map[string]float64 `json:"by_type"`
	Total     int                `json:"total_questions"`
	Correct   int                `json:"correct"`
	Details   []QuestionResult   `json:"details,omitempty"`
}

BenchmarkResult holds aggregated benchmark results.

func Aggregate 

func Aggregate(results []QuestionResult) BenchmarkResult

Aggregate computes overall and per-type accuracy from question results.

type BuildMetadata 

type BuildMetadata struct {
	Commit string `json:"commit"`
	Dirty  bool   `json:"dirty"`
}

BuildMetadata binds corpus evidence to the evaluated source revision.

type Corpus 

type Corpus struct {
	SchemaVersion string           `json:"schema_version"`
	Version       string           `json:"version"`
	Identity      CorpusIdentity   `json:"identity"`
	Records       []CorpusRecord   `json:"records"`
	Queries       []CorpusQuery    `json:"queries"`
	SplitPolicy   SplitPolicy      `json:"split_policy"`
	Build         BuildMetadata    `json:"build"`
	Hardware      HardwareMetadata `json:"hardware"`
}

Corpus is the versioned, reproducible input contract for retrieval evaluation.

func (Corpus) Validate 

func (c Corpus) Validate() error

Validate rejects corpus data that cannot serve as reproducible release evidence.

type CorpusIdentity 

type CorpusIdentity struct {
	Origin        string `json:"origin"`
	License       string `json:"license"`
	PrivacyReview string `json:"privacy_review"`
}

CorpusIdentity records the authorship, redistribution, and privacy basis of the immutable benchmark input.

type CorpusQuery 

type CorpusQuery struct {
	ID           string            `json:"id"`
	Text         string            `json:"text"`
	ProfileClass string            `json:"profile_class"`
	QueryClass   string            `json:"query_class"`
	Split        string            `json:"split"`
	Coverage     []string          `json:"coverage"`
	Authority    map[string]string `json:"authority"`
	Labels       EvidenceLabels    `json:"labels"`
}

CorpusQuery identifies one immutable query and its complete evidence labels.

type CorpusRecord 

type CorpusRecord struct {
	ID                string `json:"id"`
	Project           string `json:"project"`
	Type              string `json:"type"`
	Kind              string `json:"kind"`
	Scope             string `json:"scope"`
	Privacy           string `json:"privacy"`
	PrincipalID       string `json:"principal_id,omitempty"`
	TopicKey          string `json:"topic_key"`
	Content           string `json:"content"`
	Lifecycle         string `json:"lifecycle"`
	RecordedAt        string `json:"recorded_at"`
	ValidFrom         string `json:"valid_from"`
	ValidUntil        string `json:"valid_until,omitempty"`
	SourceEpisodeID   string `json:"source_episode_id,omitempty"`
	DerivationID      string `json:"derivation_id,omitempty"`
	DerivationVersion string `json:"derivation_version,omitempty"`
}

CorpusRecord is the labelled retrieval truth ingested by a baseline run.

type EvidenceLabels 

type EvidenceLabels struct {
	Relevant        []EvidenceRef    `json:"relevant"`
	HardNegativeIDs []string         `json:"hard_negative_ids"`
	NoAnswer        *bool            `json:"no_answer"`
	Temporal        *TemporalLabels  `json:"temporal"`
	Isolation       *IsolationLabels `json:"isolation"`
}

EvidenceLabels records relevance, negative, abstention, temporal, and isolation truth.

type EvidenceRef 

type EvidenceRef struct {
	EpisodeID string        `json:"episode_id,omitempty"`
	FactID    string        `json:"fact_id,omitempty"`
	Span      *EvidenceSpan `json:"span,omitempty"`
}

EvidenceRef points to an immutable episode, fact, or byte span within an episode.

func (EvidenceRef) Validate 

func (r EvidenceRef) Validate() error

Validate requires exactly one complete immutable evidence locator.

type EvidenceReport 

type EvidenceReport struct {
	SchemaVersion     string             `json:"schema_version"`
	RunID             string             `json:"run_id,omitempty"`
	ReportID          string             `json:"report_id"`
	CorpusVersion     string             `json:"corpus_version"`
	ProtocolVersion   string             `json:"protocol_version"`
	Build             BuildMetadata      `json:"build"`
	Hardware          HardwareMetadata   `json:"hardware"`
	MetricDefinitions []MetricDefinition `json:"metric_definitions"`
	Profiles          []ProfileReport    `json:"profiles"`
	Queries           []QueryReport      `json:"queries"`
	Resources         ResourceReport     `json:"resources"`
	Uncertainty       UncertaintyReport  `json:"uncertainty"`
	Limitations       []string           `json:"limitations"`
}

EvidenceReport is the versioned release-evidence contract for retrieval runs. It is intentionally separate from BenchmarkResult, which remains the legacy answer-evaluation format used by existing benchmark runners.

func (EvidenceReport) Validate 

func (r EvidenceReport) Validate() error

Validate rejects reports that cannot support reproducible retrieval claims.

type EvidenceSpan 

type EvidenceSpan struct {
	EpisodeID string `json:"episode_id"`
	StartByte int    `json:"start_byte"`
	EndByte   int    `json:"end_byte"`
}

EvidenceSpan identifies a half-open byte range in an immutable episode.

type GateApproval 

type GateApproval struct {
	Reviewers  []string `json:"reviewers"`
	Rationale  string   `json:"rationale"`
	ApprovedAt string   `json:"approved_at"`
}

GateApproval records reviewer sign-off and its written rationale.

type GateBaseline 

type GateBaseline struct {
	CorpusVersion        string           `json:"corpus_version"`
	Hardware             HardwareMetadata `json:"hardware"`
	BaselineReportIDs    []string         `json:"baseline_report_ids"`
	BaselineReportSHA256 []string         `json:"baseline_report_sha256"`
	ReproSHA256          string           `json:"repro_sha256"`
	EvidenceCompletedAt  string           `json:"evidence_completed_at"`
	Representative       bool             `json:"representative"`
	VarianceAnalyzed     bool             `json:"variance_analyzed"`
	CorpusApprovedBy     string           `json:"corpus_approved_by"`
	HardwareApprovedBy   string           `json:"hardware_approved_by"`
}

GateBaseline binds a protocol to representative, independently preserved baseline evidence and its approved corpus and hardware envelope.

type GateDecision 

type GateDecision struct {
	Passed           bool     `json:"passed"`
	BlockingFailures []string `json:"blocking_failures"`
}

GateDecision records every blocking failure; aggregate improvements cannot hide a universal or critical-class regression.

type GateDefinition 

type GateDefinition struct {
	ID            string           `json:"id"`
	Metric        MetricDefinition `json:"metric"`
	QueryClass    string           `json:"query_class"`
	SampleSize    int              `json:"sample_size"`
	Blocking      string           `json:"blocking"`
	CriticalClass bool             `json:"critical_class"`
	Threshold     *float64         `json:"threshold,omitempty"`
	Rule          string           `json:"rule,omitempty"`
}

GateDefinition declares one preregistered metric or correctness predicate. Threshold remains nil until representative baseline evidence is approved.

type GateObservation 

type GateObservation struct {
	IsolationViolations int                `json:"isolation_violations"`
	ExactFilterMatch    bool               `json:"exact_filter_match"`
	Metrics             map[string]float64 `json:"metrics"`
}

GateObservation contains only the evidence needed to evaluate registered gates. Correctness predicates remain independent of aggregate quality gains.

type GateProtocol 

type GateProtocol struct {
	Version                          string           `json:"version"`
	RegisteredBeforeCandidateResults bool             `json:"registered_before_candidate_results"`
	Baseline                         GateBaseline     `json:"baseline"`
	Approval                         GateApproval     `json:"approval"`
	Gates                            []GateDefinition `json:"gates"`
}

GateProtocol is the input used to create an immutable gate registry version.

type GateRegistry 

type GateRegistry struct {
	// contains filtered or unexported fields
}

GateRegistry is immutable from outside this package. Construction and accessors defensively copy all slices and threshold pointers.

func RegisterGateRegistry 

func RegisterGateRegistry(protocol GateProtocol) (GateRegistry, error)

RegisterGateRegistry validates and freezes a versioned preregistered gate protocol. Universal correctness gates are installed by Cortex and cannot be supplied, removed, or relaxed by callers.

func ReviseGateRegistry 

func ReviseGateRegistry(previous GateRegistry, next GateProtocol, revision GateRevision) (GateRegistry, error)

ReviseGateRegistry creates a separately versioned registry. Once candidate results exist, written approval and fresh held-out evidence are mandatory.

func (GateRegistry) Evaluate 

func (r GateRegistry) Evaluate(observation GateObservation) (GateDecision, error)

Evaluate applies universal predicates before all numeric gates. Every numeric release gate must have an explicitly registered threshold.

func (GateRegistry) Gates 

func (r GateRegistry) Gates() []GateDefinition

Gates returns a defensive copy of all registered gates.

func (GateRegistry) Version 

func (r GateRegistry) Version() string

Version returns the immutable protocol version.

func (GateRegistry) WithCandidateResultsObserved 

func (r GateRegistry) WithCandidateResultsObserved() GateRegistry

WithCandidateResultsObserved returns a sealed copy. It never mutates the original registry, so a caller cannot edit a protocol in place after results.

type GateRevision 

type GateRevision struct {
	Rationale              string `json:"rationale"`
	ApprovedBy             string `json:"approved_by"`
	FreshHeldOutEvaluation bool   `json:"fresh_held_out_evaluation"`
	ReusesDecisionEvidence bool   `json:"reuses_decision_evidence"`
}

GateRevision is mandatory evidence for changing a protocol after candidate results have been observed.

type HardwareMetadata 

type HardwareMetadata struct {
	ProfileID string `json:"profile_id"`
	OS        string `json:"os"`
	Arch      string `json:"arch"`
	CPU       string `json:"cpu"`
	MemoryMB  int    `json:"memory_mb"`
}

HardwareMetadata describes the representative execution envelope.

type IndependentRun 

type IndependentRun struct {
	RunID                string              `json:"run_id"`
	Seed                 string              `json:"seed"`
	BinarySHA256         string              `json:"binary_sha256"`
	Report               EvidenceReport      `json:"report"`
	HeapAllocBytes       uint64              `json:"heap_alloc_bytes"`
	TotalAllocBytes      uint64              `json:"total_alloc_bytes"`
	AllocationsAvailable bool                `json:"allocations_available"`
	Outliers             []OutlierDisclosure `json:"outliers,omitempty"`
}

IndependentRun preserves one complete, independently executed evidence report, its binary identity, and process allocation samples.

type IsolationLabels 

type IsolationLabels struct {
	PrincipalProject string   `json:"principal_project"`
	EligibleIDs      []string `json:"eligible_ids"`
	ExcludedIDs      []string `json:"excluded_ids"`
}

IsolationLabels record the principal project and exact eligible/excluded ID sets.

type JudgeConfig 

type JudgeConfig struct {
	// Endpoint is the Ollama HTTP root (e.g. "http://localhost:11434").
	// Populated from OLLAMA_ENDPOINT; defaults to localhost:11434.
	Endpoint string

	// Model is the Ollama model name (e.g. "qwen2.5:7b-instruct").
	// Populated from OLLAMA_JUDGE_MODEL; defaults to qwen2.5:7b-instruct.
	Model string

	// Timeout is the per-request HTTP timeout. Defaults to 30s when zero.
	// Used both for connect and overall request lifetime.
	Timeout time.Duration

	// HTTPClient is an optional pre-built *http.Client used for the
	// underlying call. When nil, the judge constructs one from Timeout.
	// Tests inject httptest servers by setting Endpoint to server.URL and
	// leaving HTTPClient nil — the default client honours Timeout correctly.
	HTTPClient *http.Client

	// Logger receives one structured record per call. When nil, a
	// default slog logger writing to stderr is used.
	Logger *slog.Logger
}

JudgeConfig configures the Ollama-backed LLM judge.

Callers (bench/locomo, bench/longmemeval, bench/dmr) treat this as an opaque pointer — they only check `cfg != nil` — so adding new fields is safe. The legacy Provider/APIKey fields from the prior cloud-provider implementation are intentionally absent: this judge MUST NOT call any cloud provider.

func DefaultJudgeConfig 

func DefaultJudgeConfig() *JudgeConfig

DefaultJudgeConfig returns a config wired from environment variables. Unlike the legacy implementation, this never returns nil — Ollama is always the judge path. Callers should not gate on nil.

Env vars (all optional):

type JudgeResult 

type JudgeResult struct {
	Verdict   string  `json:"verdict"`
	Score     float64 `json:"score"`
	Reasoning string  `json:"reasoning"`
}

JudgeResult is the typed, structured return for Judge.

Verdict — "correct" | "incorrect" | "partial" (case-insensitive, lower-cased) Score — float in [0.0, 1.0]; reflects the model's confidence in Verdict Reasoning — short natural-language justification from the model

func Judge 

func Judge(ctx context.Context, cfg *JudgeConfig, question, expected, got string) (JudgeResult, error)

Judge evaluates a single question/expected/got triple and returns the typed JudgeResult. This is the preferred API for new code; existing runners continue to call JudgeAnswer below.

type LatencyReport 

type LatencyReport struct {
	Unit string  `json:"unit"`
	P50  float64 `json:"p50"`
	P95  float64 `json:"p95"`
	P99  float64 `json:"p99"`
}

LatencyReport records required latency quantiles with an explicit unit.

type MetricDefinition 

type MetricDefinition struct {
	Name        string `json:"name"`
	Unit        string `json:"unit"`
	Direction   string `json:"direction"`
	Description string `json:"description"`
}

MetricDefinition records how a reported metric is interpreted before candidate results are evaluated.

type MetricTolerance 

type MetricTolerance struct {
	MetricKey  string  `json:"metric_key"`
	MaxRange   float64 `json:"max_range"`
	ApprovedBy string  `json:"approved_by"`
}

MetricTolerance is an explicitly approved maximum observed range. Cortex never supplies a default because thresholds require baseline evidence.

type MetricVariance 

type MetricVariance struct {
	MetricKey               string    `json:"metric_key"`
	SampleSize              int       `json:"sample_size"`
	Samples                 []float64 `json:"samples"`
	Minimum                 float64   `json:"minimum"`
	Maximum                 float64   `json:"maximum"`
	Mean                    float64   `json:"mean"`
	Range                   float64   `json:"range"`
	SampleStandardDeviation float64   `json:"sample_standard_deviation"`
	DispersionMethod        string    `json:"dispersion_method"`
	DispersionApprovedBy    string    `json:"dispersion_approved_by"`
}

MetricVariance reports descriptive dispersion without assigning an unregistered release threshold.

type OutlierDisclosure 

type OutlierDisclosure struct {
	RunID     string `json:"run_id,omitempty"`
	MetricKey string `json:"metric_key"`
	Reason    string `json:"reason"`
}

OutlierDisclosure records an observed anomaly without silently excluding the associated run from variance calculations.

type ProfileReport 

type ProfileReport struct {
	ProfileID      string             `json:"profile_id"`
	ProfileVersion string             `json:"profile_version"`
	QueryClass     string             `json:"query_class"`
	Metrics        map[string]float64 `json:"metrics"`
	Latency        LatencyReport      `json:"latency"`
	Throughput     ThroughputReport   `json:"throughput"`
}

ProfileReport contains aggregate metrics and performance distributions for one immutable profile version and query class.

type QueryReport 

type QueryReport struct {
	QueryID         string             `json:"query_id"`
	ProfileID       string             `json:"profile_id"`
	ProfileVersion  string             `json:"profile_version"`
	QueryClass      string             `json:"query_class"`
	Metrics         map[string]float64 `json:"metrics"`
	CurrentOutput   []RankedOutput     `json:"current_output"`
	CandidateOutput []RankedOutput     `json:"candidate_output"`
}

QueryReport preserves traceable current and candidate ranked outputs for one immutable query/profile execution.

type QuestionResult 

type QuestionResult struct {
	ID       string  `json:"id"`
	Type     string  `json:"type"`
	Query    string  `json:"query"`
	Expected string  `json:"expected"`
	Got      string  `json:"got"`
	Score    float64 `json:"score"`
	Correct  bool    `json:"correct"`
}

QuestionResult holds the result of a single question evaluation.

type RankedOutput 

type RankedOutput struct {
	StableID string  `json:"stable_id"`
	Rank     int     `json:"rank"`
	Score    float64 `json:"score"`
}

RankedOutput identifies one stable result in an observed ranking.

type ReproAnalysis 

type ReproAnalysis struct {
	Identity                 ReproIdentity         `json:"identity"`
	Runs                     []IndependentRun      `json:"runs"`
	DeterministicMatch       bool                  `json:"deterministic_match"`
	DeterministicDifferences []string              `json:"deterministic_differences"`
	Variance                 []MetricVariance      `json:"variance"`
	Outliers                 []OutlierDisclosure   `json:"outliers"`
	ToleranceEvaluations     []ToleranceEvaluation `json:"tolerance_evaluations"`
}

ReproAnalysis preserves raw independent runs alongside exact deterministic comparison, descriptive variance, outlier disclosure, and tolerance results.

func AnalyzeReproducibility 

func AnalyzeReproducibility(runs []IndependentRun, protocol ReproProtocol) (ReproAnalysis, error)

AnalyzeReproducibility compares independent baseline runs only when their seed, build, corpus, hardware, and evaluated protocol identities match.

type ReproIdentity 

type ReproIdentity struct {
	Seed            string           `json:"seed"`
	BinarySHA256    string           `json:"binary_sha256"`
	Build           BuildMetadata    `json:"build"`
	CorpusVersion   string           `json:"corpus_version"`
	Hardware        HardwareMetadata `json:"hardware"`
	ProtocolVersion string           `json:"protocol_version"`
}

ReproIdentity contains every semantic field that must match before runs are compared.

type ReproProtocol 

type ReproProtocol struct {
	Version          string            `json:"version"`
	DispersionMethod string            `json:"dispersion_method"`
	ApprovedBy       string            `json:"approved_by"`
	Tolerances       []MetricTolerance `json:"tolerances,omitempty"`
}

ReproProtocol identifies the reviewer-approved dispersion method and any numeric tolerances registered before candidate evaluation.

type ResourceReport 

type ResourceReport struct {
	CPUSeconds       float64 `json:"cpu_seconds"`
	CPUUnit          string  `json:"cpu_unit"`
	CPUAvailable     *bool   `json:"cpu_available,omitempty"`
	PeakRSSBytes     int64   `json:"peak_rss_bytes"`
	PeakRSSUnit      string  `json:"peak_rss_unit"`
	PeakRSSAvailable *bool   `json:"peak_rss_available,omitempty"`
	StorageBytes     int64   `json:"storage_bytes"`
	StorageUnit      string  `json:"storage_unit"`
	StorageAvailable *bool   `json:"storage_available,omitempty"`
	IndexBytes       int64   `json:"index_bytes"`
	IndexUnit        string  `json:"index_unit"`
	IndexAvailable   *bool   `json:"index_available,omitempty"`
}

ResourceReport records CPU, peak RSS, corpus storage, and retrieval-index costs. Units are explicit so reports cannot silently reinterpret values.

type SplitPolicy 

type SplitPolicy struct {
	Version  string `json:"version"`
	Strategy string `json:"strategy"`
}

SplitPolicy describes how immutable query IDs are assigned to evaluation splits.

type TemporalLabels 

type TemporalLabels struct {
	ValidAt             string   `json:"valid_at"`
	EligibleEvidenceIDs []string `json:"eligible_evidence_ids"`
}

TemporalLabels record the query time and exact evidence eligible at that time.

type ThroughputReport 

type ThroughputReport struct {
	Unit             string  `json:"unit"`
	QueriesPerSecond float64 `json:"queries_per_second"`
}

ThroughputReport records completed queries per second with an explicit unit.

type ToleranceEvaluation 

type ToleranceEvaluation struct {
	MetricKey         string  `json:"metric_key"`
	ObservedRange     float64 `json:"observed_range"`
	RegisteredMaximum float64 `json:"registered_maximum"`
	ApprovedBy        string  `json:"approved_by"`
	ProtocolVersion   string  `json:"protocol_version"`
	Passed            bool    `json:"passed"`
}

ToleranceEvaluation compares observed dispersion only with a caller-supplied, reviewer-approved preregistered tolerance.

type UncertaintyReport 

type UncertaintyReport struct {
	Method          string  `json:"method"`
	ConfidenceLevel float64 `json:"confidence_level"`
	SampleSize      int     `json:"sample_size"`
	Notes           string  `json:"notes"`
}

UncertaintyReport discloses the uncertainty method and sampling basis.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.