Documentation
¶
Overview ¶
Package research is every experiment you have ever run, kept and comparable.
The R&D EVIDENCE plane (HIP-0512 §"Hanzo Research"). Every experiment across every product — a benchmark run is ONE kind; kernel-perf (hanzo-engine), training (hanzo-ml), ablations, and policy-evals are the others — accrues here as VERSIONED, append-only evidence under one discriminator, kind ∈ benchmark | kernel-perf | training | ablation | policy-eval.
Two planes, never one (HIP-0512): each org's transactional SQLite (store.go, per HIP-0302, physically file-isolated) is the local source of truth; it rolls up into hanzoai/datastore — the column-oriented OLAP aggregate plane (datastore.go) — for the unified cross-project query surface. Durability today is VERSIONED · APPEND-ONLY; cloud mirroring is ROLLING OUT (the roll-up is best-effort; retry + reconciliation + replication + backup/restore are still on the critical path, so the stronger "immutable · replicated · recovery-tested" claim is NOT yet made).
Versioned, not write-once: a correction APPENDS a new version under the same stable id and the prior version is RETAINED (superseded), never mutated. RETAINED is the full history (the truth); CANONICAL is the deterministic deduped view over it — so dedup never reads as loss. faulted/failed runs are retained (negative results are evidence). Ingest is idempotent by content (measurement + provenance) — re-running the backfill appends nothing.
Provenance is first-class + queryable: project, git (sha/branch/dirty), and lib_versions travel as columns/structured fields, so the board answers "which lib version regressed LiveCodeBench" over the longitudinal record.
Private-by-default: an upload records visibility='private' and grants NO training or commons-publication rights. Public board visibility, training rights, and commons publication are each a SEPARATE authorized grant (POST /v1/research/grants), never implied by uploading a run.
Surface (every route org-scoped by the validated principal; project is a sub- dimension column, so the org's ops board reads across its projects in one query):
POST /v1/research/experiments ingest a batch (idempotent) → SQLite → roll up GET /v1/research/experiments list canonical (?project= ?kind=) GET /v1/research/projects every project + real totals (canonical + retained) GET /v1/research/totals headline aggregate + per-kind (?project=) POST /v1/research/grants set visibility/consent for a stable id (separate auth) POST /v1/research/artifacts record a diary artifact, content-addressed GET /v1/research/artifacts the diary feed, newest first (metadata only) GET /v1/research/artifacts/:sha256 the artifact's bytes, by content hash
Its own binary (plugin/research) states Name/Price/Mount; the host learns the prefix from its manifest row.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Record ¶
func Record(ctx context.Context, org, project string, exps []Experiment) error
Record writes experiment evidence rows for (org, project) idempotently into the org's durable research plane (latest-run-canonical, exactly as POST /v1/research/experiments) AND ships them fenced before returning. It is the in-process seam the experiments primitive composes to persist per-variant A/B samples. No SSRF gate is needed: the caller sets no BYO endpoint (these are in-process samples, not a measured probe).
Ship-before-return is the SAME durability contract the HTTP ingest path has: an evidence row is "recorded" only once fenced to the durable object, so a takeover hydrates it and no acknowledged write is lost. A ship that is not acknowledged — this pod is not the org's elected writer, or was deposed — is an ERROR, so the caller never treats a stale local write on a non-owner as recorded (it logs and leaves the recomputable evidence unpersisted). On a local-only deployment (no Durability) Sync acks trivially, so this is a no-op there.
Types ¶
type Artifact ¶
type Artifact struct {
SHA256 string `json:"sha256"` // SERVER-derived on write; the identity
Content string `json:"content,omitempty"` // base64 bytes on write; the server hashes + stores them (never returned)
Kind string `json:"kind"`
Ref string `json:"ref"` // server-derived content address (sha256:<hash>)
RunID string `json:"run_id"`
Project string `json:"project"`
Visibility string `json:"visibility"`
RetentionClass string `json:"retention_class"`
GitSHA string `json:"git_sha"`
GitBranch string `json:"git_branch"`
GitDirty bool `json:"git_dirty"`
LibVersions json.RawMessage `json:"lib_versions,omitempty"`
TS int64 `json:"ts"`
}
Artifact is one research-diary record — a raw-artifact-class record tied to a run: a snapshot (a PNG of the canonical board) or a report (a generated page). The caller submits the bytes (base64 content); the SERVER hashes them and that sha256 is the identity + ref (sha256:<hash>) — never a client-asserted hash or ref, so it is genuinely content-addressed and un-poisonable. A re-POST of the same bytes is a no-op. Private by default; public only via the separate visibility grant. Carries the same provenance as a run.
type Attempt ¶
type Attempt struct {
Benchmark string `json:"benchmark"`
Item string `json:"item"`
Model string `json:"model"`
Revision string `json:"revision"`
Status string `json:"status"`
Gold string `json:"gold"`
Answer string `json:"answer"`
Correct bool `json:"correct"`
Response string `json:"response"`
Source string `json:"source"`
TS int64 `json:"ts"`
}
Attempt is one versioned measured attempt on one item, keyed by (project, benchmark, item, model). A re-scored/re-run attempt is a new version; the prior is retained. response is the raw artifact (own retention class; may carry personal/licensed material). status=faulted retains a negative result.
type Counts ¶
type Counts struct {
ExperimentsRetained int `json:"experiments_retained"`
ExperimentsCanonical int `json:"canonical_experiments"`
AttemptsRetained int `json:"attempts_retained"`
AttemptsCanonical int `json:"canonical_attempts"`
}
Counts carries both axes for both kinds. retained is the full versioned history; canonical is the distinct-stable-id answered deduped view. retained ≥ canonical always, and the gap is superseded/faulted versions — never lost data.
type Experiment ¶
type Experiment struct {
Project string `json:"project"`
ID string `json:"id"`
Revision string `json:"revision"` // original | corrected | retracted
Status string `json:"status"` // planning | running | complete | faulted
Canonical bool `json:"canonical"`
Visibility string `json:"visibility"`
Trainable bool `json:"trainable"`
Publishable bool `json:"publishable"`
Kind string `json:"kind"`
Subject string `json:"subject"`
Task string `json:"task"`
Metric string `json:"metric"`
Value float64 `json:"value"`
N int `json:"n"`
NTotal int `json:"n_total"`
CostUSD float64 `json:"cost_usd"`
Meta json.RawMessage `json:"meta,omitempty"`
GitSHA string `json:"git_sha"`
GitBranch string `json:"git_branch"`
GitDirty bool `json:"git_dirty"`
LibVersions json.RawMessage `json:"lib_versions,omitempty"`
TS int64 `json:"ts"`
Endpoint string `json:"endpoint,omitempty"`
}
Experiment is one versioned run record, keyed by its stable id (<kind>:<subject>: <task>) within its project. Ingest reads the measurement + revision/status + provenance (git*, lib_versions); it IGNORES visibility/trainable/publishable (forced private/withheld — a grant is separate). Reads return the canonical version with its derived Canonical flag and current visibility/consent. TS (unix seconds) orders versions. Endpoint is the OPTIONAL BYO arm the run measured — SSRF-gated at ingest.
func List ¶
func List(ctx context.Context, org, project, kind string) ([]Experiment, error)
List reads (org, project, kind) evidence rows from the durable research plane — the in-process read the experiments primitive composes to fetch an experiment's per-variant samples back. project "" reads the org's whole set; kind narrows to a discriminator (e.g. "ab").
type GrantRequest ¶
type GrantRequest struct {
Project string `json:"project"`
ID string `json:"id"` // an experiment (run) stable id
SHA256 string `json:"sha256"` // OR an artifact content hash
Visibility *string `json:"visibility"`
Trainable *bool `json:"trainable"`
Publishable *bool `json:"publishable"`
}
GrantRequest is a SEPARATE authorization decision for a stable id's records: visibility (private/org/public) and the training/commons consent flags. A nil field is left unchanged. This is the only path that elevates a record beyond private.
type IngestRequest ¶
type IngestRequest struct {
Experiments []Experiment `json:"experiments"`
Attempts []Attempt `json:"attempts"`
}
IngestRequest is one upload batch: experiments and their attempts as two flat arrays — the shape of the two tables, so the uploader streams rows through with no reshaping.
type KindTotal ¶
type KindTotal struct {
Kind string `json:"kind"`
Experiments int `json:"experiments"`
CostUSD float64 `json:"cost_usd"`
}
KindTotal is the per-kind slice of a totals aggregate (canonical experiments + cost).
type ProjectSummary ¶
type ProjectSummary struct {
Project string `json:"project"`
Experiments int `json:"experiments"` // canonical
ExperimentsRetained int `json:"experiments_retained"`
Attempts int `json:"attempts"` // canonical
AttemptsRetained int `json:"attempts_retained"`
Models int `json:"models"`
Benchmarks int `json:"benchmarks"`
CostUSD float64 `json:"cost_usd"`
Kinds []string `json:"kinds"`
}
ProjectSummary is one project's real totals — canonical (answered, deduped) plus *_retained (full versioned history) so dedup never reads as loss. Tokens are intentionally absent (the corpus carries no per-attempt token usage); cost_usd is the real spend.
type ResearchTotals ¶ added in v1.801.350
type ResearchTotals struct {
Project string `json:"project,omitempty"`
Projects int `json:"projects"`
Experiments int `json:"experiments"` // canonical
ExperimentsRetained int `json:"experiments_retained"`
Attempts int `json:"attempts"` // canonical
AttemptsRetained int `json:"attempts_retained"`
Models int `json:"models"`
Benchmarks int `json:"benchmarks"`
CostUSD float64 `json:"cost_usd"`
ByKind []KindTotal `json:"by_kind"`
}
ResearchTotals is the observatory's headline aggregate: canonical (answered) counts plus *_retained, and a per-kind breakdown. Deterministic from the store.
The name carries its own product because the SCHEMA namespace is FLAT: zip keys a component schema on the Go type's bare name (typeName drops the package), so the fleet weave refuses two apps that mean different things by one name. It was `Totals`, and apps/admin publishes a `Totals` of its own about volumes and clusters — the weave named the collision the moment this type became an op's Out.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Package ui embeds the Hanzo Research R&D Ops Board — the dashboard over the /v1/research evidence plane (HIP-0512) — directly into the cloud binary and serves it at /research (cloud.hanzo.ai/research + console.hanzo.ai/research).
|
Package ui embeds the Hanzo Research R&D Ops Board — the dashboard over the /v1/research evidence plane (HIP-0512) — directly into the cloud binary and serves it at /research (cloud.hanzo.ai/research + console.hanzo.ai/research). |