Help us shape the future of Go! Take the 2026 Go Developer Survey

experiment

package
v0.10.10 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 12, 2026 License: MIT Imports: 20 Imported by: 0

Documentation

Overview

Package experiment provides the SQLite-backed database for VaultMind experiment data.

Index

Constants

View Source 
const (
	// SuppressedByCaller: --pointers-only. The caller asked for ids.
	SuppressedByCaller = "pointers_only"
	// SuppressedBelowFloor: the top hit sat at or under the off-topic noise
	// floor, so the body was judged not worth the tokens.
	SuppressedBelowFloor = "below_floor"
	// SuppressedLowContrast: a tight vault, where the embedder cannot spread
	// scores and a "weak" top hit is often the best available correct match.
	// This is the reason most likely to be withholding something useful.
	SuppressedLowContrast = "low_contrast"
)

Why bodies were withheld from an ask result. These name the branches that already exist in the formatter; recording which one fired turns "the agent ignored it" and "the tool never handed it over" into separate, countable facts.

View Source 
const (
	EnvCaller        = "VAULTMIND_CALLER"
	EnvUserSessionID = "VAULTMIND_USER_SESSION_ID"
)

EnvCaller names the invoking agent; EnvUserSessionID carries the harness's real conversation id into the session row.

View Source 
const (
	EnvClaudeCodeSessionID = "CLAUDE_CODE_SESSION_ID"
	EnvCodexSessionID      = "CODEX_SESSION_ID"
)

The harnesses' own conversation ids, which each exposes to the agent's shell (Claude Code: CLAUDE_CODE_SESSION_ID; Codex 0.156: CODEX_SESSION_ID, the root session, see codex-rs protocol/shell_environment.rs). They are the same ids the harness puts in every hook payload, so an agent's own CLI call and the hooks' searches land in one user session.

View Source 
const (
	EventSearch      = "search"
	EventAsk         = "ask"
	EventContextPack = "context_pack"
	EventNoteAccess  = "note_access"
	EventIndexEmbed  = "index_embed"
	EventUseEvidence = "use_evidence"
)

Event type constants.

View Source 
const (
	TelemetryOff       = "off"
	TelemetryAnonymous = "anonymous"
	TelemetryFull      = "full"
)

Telemetry tier constants.

Write-path contract (as of this writing):

  • TelemetryOff → the experiment DB is never opened; no session is started; no events are written. Gated in cmd/root.go.
  • TelemetryAnonymous and TelemetryFull → identical write paths. Both write everything locally. The distinction is future-proofing for an uploader that does not yet exist.

Upload-path contract (for future implementation): 

When a data-return mechanism is built (not yet), the uploader MUST
filter outbound payloads by tier. Under Anonymous, the following fields
MUST be stripped before transmission:
  - event_data.variants.*.results[].note_id
  - event_data.variants.*.results[].path
  - event_data.query.text (if/when query text is added to events)
  - vault_path
Aggregate structure (ranks, scores, counts, timestamps, variant names,
note_type) MAY be sent under Anonymous. Under Full, the uploader MAY
send everything.

If you're adding the uploader: verify this contract still matches what the README's "The local usage log" section promises, and what `export --preview` prints. Those are the two places the promise is now made — the consent prompt this used to point at was deleted for asking about a transmission that does not happen.

View Source 
const CallerRecite = "vaultmind-recite"

CallerRecite tags accesses written by `arc recite` — the whole-layer read.

It lives on the LEDGER event, not on a vault-local access row. `self` selects on body_delivered rather than caller (migration 008 retired the caller proxy once hooks began delivering real bodies), so a vault-local recite row would appear in the hot list however it was tagged — and 27 arcs on every session start would bury the deliberate reads the view exists to show. Tagging the ledger row keeps recitations filterable at no cost to `self`.

View Source 
const DefaultSpreadingActivationDelta = 0.2

DefaultSpreadingActivationDelta is the Delta weight applied when query similarities are available to the scorer (hybrid retrieval on). The value is ACT-R's standard spreading-activation contribution; use DefaultActivationParamsWithSimilarity to construct a params set with this value set, or DefaultActivationParams when similarities are intentionally absent (Delta stays 0).

View Source 
const MemoryPath = ":memory:"

MemoryPath opens a database that exists only in RAM. Used by the read path when no log is on disk, so an empty export runs through the real readers.

View Source 
const MetaUserSessionID = "user_session_id"

MetaUserSessionID is the caller-meta key carrying the harness's REAL conversation id. Present ⇒ used verbatim; absent ⇒ the time heuristic. Named once here so the hook, the detector and the resolver cannot disagree.

View Source 
const MinElapsedHours = 1.0 / 3600.0

MinElapsedHours is the minimum effective-elapsed value substituted when a note access's compressed elapsed time rounds to zero or below. Expressed as 1 second in hours so it's a soft floor that still makes the log(time) term in retrieval strength well-defined. Changing this shifts the retrieval score for very recent accesses.

View Source 
const UserSessionThreshold = 30 * time.Minute

UserSessionThreshold defines how long after the last matching invocation a new one still counts as part of the same user-session. 30 minutes is the default; invocations beyond this gap mint a new user-session id. "Matching" means same caller + same user + same host — companion persona loads and cli queries stay in separate groupings even when close in time.

Variables

View Source 
var ErrNoUsageLog = errors.New("no usage log")

ErrNoUsageLog reports that no usage log exists at the path asked for.

Functions

func BuildAskEventData 

func BuildAskEventData(p AskEventParams) map[string]any

BuildAskEventData composes the event_data payload for an ask event. Retrieval hits are carried as a variant under variants.{RetrievalMode}; shadow variants are merged into the same map. A collision between a shadow variant name and the retrieval mode emits a warn log and the shadow payload wins (deterministic, documented behavior — rename one set if it matters).

func BuildRetrievalEventData 

func BuildRetrievalEventData(variants map[string]any, resultCount int, err error) map[string]any

BuildRetrievalEventData composes the base event_data payload for a retrieval event. resultCount is the total number of hits the retrieval reported (may differ from len(variants.*.results) when limits truncate). When err is non-nil, an "error" field is added so downstream consumers can distinguish genuine zero-hit retrievals (curiosity signal) from crashes (error signal). The error field is omitted on success, including the zero-hit success case.

func BuildShadowVariantResults 

func BuildShadowVariantResults(session *Session, actDef ExperimentDef, noteIDs []string) map[string]any

BuildShadowVariantResults computes activation features for every variant in actDef (primary + shadows) over the given ranked note IDs. Returns a map shaped like the event's `variants` substructure: 

{ variant: { results: [ { note_id, rank, features? } ] } }

Rank is 1-indexed from the position of each note ID in noteIDs. Unknown variant names are skipped (logged at debug). Used by ask and context-pack events to emit shadow-scoring comparisons alongside primary retrieval.

func BuildVariantPayload 

func BuildVariantPayload(variant string, hits []RetrievalHit) map[string]any

BuildVariantPayload produces the event_data.variants substructure for one retrieval variant. The shape matches what LinkOutcomes walks: 

{ variant: { results: [ { note_id, rank, score_final, note_type, path, scores? } ] } }

scores is omitted when the hit has no per-component breakdown.

func CombinedScore 

func CombinedScore(retrieval, storage, similarity, alpha, beta, delta float64) float64

CombinedScore returns score = alpha * retrieval + beta * storage + delta * similarity. This implements the full ACT-R model: base-level activation (retrieval), dual-strength storage (Bjork), and spreading activation (similarity).

func CompressedElapsed 

func CompressedElapsed(active, idle time.Duration, gamma float64) time.Duration

CompressedElapsed returns tj_effective = active + gamma * idle.

func ComputeApproxRetrieval 

func ComputeApproxRetrieval(accessCount int, lastAccessedAt, now time.Time, windows []SessionWindow, gamma, d float64) float64

ComputeApproxRetrieval is the live-retrieval-path approximation of ComputeRetrieval. The notes table stores only (access_count, last_accessed_at) — a scalar count and a single timestamp — not the full per-event access history that ComputeRetrieval consumes. To use the same ACT-R math without a per-event access-times table, this approximator treats every recorded access as if it had happened at last_accessed_at, then defers to ComputeRetrieval. With all N timestamps equal to t, sum(t_k^(-d)) reduces to N*t^(-d), so the score collapses to ln(N) - d*ln(t) — preserving both the count-amplifies and elapsed-decays monotonic properties the ranking layer needs (Track A.4, slice 5b'). Returns 0 for the degenerate cases (no accesses, or no timestamp) so the caller doesn't have to guard against -Inf from log of zero or non-positive elapsed.

func ComputeBatchScores 

func ComputeBatchScores(db *DB, noteIDs []string, params ActivationParams, similarities map[string]float64) (map[string]float64, map[string]map[string]float64, error)

ComputeBatchScores computes activation scores for a batch of notes. Returns noteID->score and noteID->features maps. similarities is optional (nil = no spreading activation). When provided, it maps noteID -> cosine similarity with the current query.

func ComputeBatchScoresAt 

func ComputeBatchScoresAt(db *DB, noteIDs []string, params ActivationParams, similarities map[string]float64, now time.Time) (map[string]float64, map[string]map[string]float64, error)

ComputeBatchScoresAt is ComputeBatchScores with an explicit reference time. Production passes time.Now(); tests pass a fixed instant so activation ranking is deterministic instead of depending on sub-millisecond wall-clock differences between the access events and the scoring call.

func ComputeRetrieval 

func ComputeRetrieval(accessTimes []time.Time, now time.Time, windows []SessionWindow, gamma, d float64) float64

ComputeRetrieval computes Bi = ln(sum(tj_effective^(-d))). Uses session windows to partition each access-to-now interval. Returns 0.0 for empty accessTimes.

The windows are merged once here, not once per access: a note's access history can run to hundreds of entries, and every ask scores every accessed note several times.

func ComputeStorage 

func ComputeStorage(accessCount int) float64

ComputeStorage returns Si = ln(1 + access_count). Never decays. Returns 0.0 for non-positive counts.

func DetectCaller 

func DetectCaller() (string, map[string]any)

DetectCaller resolves the caller identity for a new experiment session. Precedence:

  1. VAULTMIND_CALLER env var — explicit, wins unconditionally. Hooks and scripts set this ("companion-persona-hook", "vaultmind-persona-hook").
  2. CLAUDE_PROJECT_DIR set — we were invoked from a Claude Code session, probably by the agent running bash. Label as "claude-code" so we can distinguish from raw CLI use.
  3. Default to "cli" — a human at a terminal, or an unknown script.

The companion caller_meta captures $USER, $HOSTNAME, and CLAUDE_PROJECT_DIR so we can later answer "whose laptop" and "which project context" without having to encode them into the caller label itself.

func ExportToJSONL 

func ExportToJSONL(db *DB, tier string, w io.Writer) error

ExportToJSONL writes a sanitized snapshot of the experiment DB to w as a JSONL stream: one manifest record on line 1, then one record per session, event, and outcome. Each record is a JSON object with a "kind" discriminator ("manifest" / "session" / "event" / "outcome").

Tier handling:

  • TelemetryOff → returns an error and writes nothing. The user opted out; producing a file implies we collected data we shouldn't have.
  • TelemetryAnonymous → strips vault_path on sessions and events, query_text on events, caller_meta on sessions, and the documented fields inside event_data via SanitizeEventData.
  • TelemetryFull → preserves everything.

func IsActivationSignal added in v0.7.0 

func IsActivationSignal(source AccessSource, bodyDelivered *bool) bool

IsActivationSignal reports whether a note_access event is evidence that the agent actually encountered a note's content, and may therefore raise its activation.

WHY THIS EXISTS — the feedback loop it closes: 

a note ranks first
  → an access is logged (gated on the relevance floor, not on delivery)
  → the activation scorer counts it as a retrieval
  → the note is boosted
  → it ranks first more reliably

Under pointers-only, no body was ever delivered, so every turn of that loop was driven by a read that did not happen. On a live log `principle-how-to-write-arcs` shows 222 such accesses against 181 rank-1 injections; the mid-task hook's 98% repeat rate is this loop, not a deduplication bug — which is why deduplicating would have hidden it.

The activation scorer reads the whole history with no time window, so the pre-fix events cannot simply age out. Requiring positive proof of delivery rather than the absence of a denial is what makes the historical phantoms stop counting: they carry no body_delivered at all, so they read as not-delivered and drop out on their own.

Default-deny, so a future logging site cannot quietly reopen the loop by inventing a source nobody added here.

bodyDelivered is TRI-state and nil means "this event predates the field", which is not the same as "no body was delivered". Collapsing the two is what the plain-bool version did, and it is the NULL-read-as-zero shape that produced the bug this function exists to fix.

func JaccardAtK 

func JaccardAtK(a, b []string, k int) float64

JaccardAtK computes the Jaccard similarity of the top-K items in a and b. Order within each list does not affect the result — only set membership after truncation to K. Two empty lists return 1.0 (conventional). One empty and one non-empty returns 0.0.

func KendallTauShared 

func KendallTauShared(a, b []string) (float64, int)

KendallTauShared computes Kendall's tau-a rank correlation restricted to items appearing in both a and b. Ranks are derived from list position (earlier = smaller rank). Returns (tau, sharedCount). If fewer than 2 items are shared, returns (NaN, sharedCount) — rank correlation is undefined on <2 pairs and callers should treat NaN as "insufficient data."

func ParseExperiments 

func ParseExperiments(raw map[string]any) map[string]ExperimentDef

ParseExperiments parses experiment definitions from Viper's raw map. Reserved keys ("telemetry", "outcome_window_sessions") and non-map values are skipped. For each map entry it extracts enabled (bool), primary (string), and shadows ([]string from []any).

func PartitionTime 

func PartitionTime(start, end time.Time, windows []SessionWindow) (active, idle time.Duration)

PartitionTime splits the duration [start, end] into active (overlapping sessions) and idle (gaps between sessions). Windows need not be sorted.

func SanitizeEventData 

func SanitizeEventData(data map[string]any, tier string) map[string]any

SanitizeEventData returns a sanitized copy of event_data for the given telemetry tier. Under TelemetryAnonymous, the contract documented in telemetry.go is enforced: variants[*].results[*].note_id and .path are stripped; aggregate fields (rank, score, scores, type, …) survive. Under TelemetryFull the data is returned as-is.

SanitizeEventData does NOT mutate its input. It walks the structure and rebuilds the parts it touches, so callers can reuse the original map without surprise.

func ScoreFromData 

func ScoreFromData(noteIDs []string, accessMap map[string][]time.Time, windows []SessionWindow, now time.Time, params ActivationParams, similarities map[string]float64) (map[string]float64, map[string]map[string]float64)

ScoreFromData computes activation scores from pre-fetched data. Returns (scores, features). Use this to avoid redundant DB queries when computing multiple variants over the same data. similarities is optional (nil = no similarity component). When provided, it maps noteID -> cosine similarity with the current query, implementing ACT-R spreading activation.

func VariantGamma 

func VariantGamma(variant string) (float64, bool)

VariantGamma returns the gamma for a known variant name.

func VariantPerformance 

func VariantPerformance(db *DB) (map[string]*VariantStats, error)

VariantPerformance computes per-variant stats over the entire experiment DB. Returns one entry per distinct primary_variant observed in events. The map is keyed by variant name; iteration order is not guaranteed — callers that need deterministic output must sort by name themselves.

func WithSession 

func WithSession(ctx context.Context, s *Session) context.Context

WithSession stores s in the context and returns the updated context.

func WriteRollup 

func WriteRollup(r *Rollup, w io.Writer) error

WriteRollup emits a single Rollup record as one indented JSON object. Unlike ExportToJSONL (which emits a stream of records), the rollup IS the unit of analysis — one record per vault — so pretty-printed JSON is the right shape for both machines and humans inspecting the payload before transmission.

Types

type AccessSource added in v0.7.0 

type AccessSource string

AccessSource names what kind of event produced a note_access row.

A defined type rather than a bare string because IsActivationSignal default-denies anything it does not recognise: a typo does not fail, it silently makes the event invisible to activation. That already happened — a fixture seeding "search" asserted against an empty score map while appearing to exercise the path, and nothing reported it. The compiler catches that now.

The values are on disk in every existing log, so they are pinned by a test. 

const (
	// AccessSourceRead is written by `note get`: the agent named a note and its
	// text was rendered. The single strongest signal the tool emits, and the
	// only source trusted without also proving delivery.
	AccessSourceRead AccessSource = "note_get"

	// AccessSourceAsk is written for a query's top hit. It means "this note
	// ranked first and cleared the relevance floor" — NOT "the agent read it".
	// Under pointers-only no body is handed over at all.
	AccessSourceAsk AccessSource = "ask"

	// AccessSourceNeighbors is written when the context pack selects a
	// neighbour. The agent sees a title at most.
	AccessSourceNeighbors AccessSource = "neighbors"

	// AccessSourceRecall was written by `memory recall`, which 835472a renamed
	// to `memory neighbors` — same code, same formatter, printing
	// `id [type] "title" depth N` and no body at all. It is the literal ancestor
	// of AccessSourceNeighbors above.
	//
	// Nothing writes it today. It is kept, and gated on delivery like ask,
	// because deleting it would silently reclassify the events already in
	// people's logs, and pre-trusting it would let a future recall path reopen
	// the phantom loop the moment someone rewires it.
	AccessSourceRecall AccessSource = "recall"

	// AccessSourceRecite is written by `arc recite`: the WHOLE layer of a type,
	// delivered unconditionally at session start.
	//
	// It is recorded because the bodies genuinely reach the agent, and a
	// tracked bulk-read path is the point of issue #53 — the untracked
	// `cat arcs/*.md` bypass kept winning precisely because no tracked
	// equivalent existed. A read that leaves no trace is how the ledger came to
	// disagree with reality in the first place.
	AccessSourceRecite AccessSource = "recite"
)

type ActivationParams 

type ActivationParams struct {
	Gamma float64 // idle time compression (0.0-1.0)
	D     float64 // decay exponent (ACT-R default: 0.5)
	Alpha float64 // retrieval strength weight
	Beta  float64 // storage strength weight
	Delta float64 // spreading activation / similarity weight
}

ActivationParams holds tunable parameters for activation scoring.

func DefaultActivationParams 

func DefaultActivationParams(gamma float64) ActivationParams

DefaultActivationParams returns params with research-based defaults. Delta defaults to 0.0 (no similarity component) for backward compatibility.

func DefaultActivationParamsWithSimilarity 

func DefaultActivationParamsWithSimilarity(gamma float64) ActivationParams

DefaultActivationParamsWithSimilarity returns a params set for the similarity-available case. Only Delta differs from DefaultActivationParams; everything else is shared so the two constructors are swappable without cascading tuning changes.

type AggregateRow 

type AggregateRow struct {
	PrimaryVariant    string   `json:"primary_variant"`
	ShadowVariant     string   `json:"shadow_variant"`
	EventCount        int      `json:"event_count"`
	MeanJaccardAtK    float64  `json:"mean_jaccard_at_k"`
	MeanKendallTau    *float64 `json:"mean_kendall_tau"`
	KendallEventCount int      `json:"kendall_event_count"`
}

AggregateRow is the per-(primary, shadow) pair summary across many events. MeanKendallTau is a pointer because tau is undefined when no event in the pair had >=2 shared items; nil here means "insufficient data" and serializes to JSON null. Storing NaN here would break encoding/json (NaN is unsupported).

func AggregateComparisons 

func AggregateComparisons(events []ComparableEvent, kCap int) []AggregateRow

AggregateComparisons collapses per-event EventPairs into per-pair aggregates. Jaccard is averaged over every event that contributed a pair. Kendall's tau averages only over events where the pair had >=2 shared items (others would contribute NaN). KendallEventCount reports how many events actually contributed to the tau average. Output rows are sorted by (primary, shadow) for deterministic display.

type AskEventParams 

type AskEventParams struct {
	// RetrievalMode is the retriever label used as the variant key for the
	// actual retrieval hits (e.g. "hybrid", "keyword").
	RetrievalMode string
	// TopHits are the retrieval results, in rank order.
	TopHits []RetrievalHit
	// ShadowVariants are the shadow-scored variant results from
	// BuildShadowVariantResults. Merged into the event's variants map.
	ShadowVariants map[string]any
	// PrimaryVariant is the activation experiment's chosen variant name
	// (recorded only when ActivationOn is true).
	PrimaryVariant string
	// ActivationOn reports whether the activation experiment is enabled for
	// this event. When false, PrimaryVariant is omitted from the payload.
	ActivationOn bool
	// BodyDelivered reports whether the caller actually received note BODIES,
	// as opposed to a list of ids it would have to go and fetch.
	//
	// The distinction is the whole question. A hook that surfaces five perfect
	// pointers and no text has not delivered memory — it has delivered
	// homework, and the agent mid-task does not do homework. Without this
	// field, "surfaced but unused" cannot be told apart from "never actually
	// handed over", and those two have opposite fixes.
	BodyDelivered bool
	// SuppressedReason names why bodies were withheld, when they were. Empty
	// when they were delivered. See SuppressedReason* constants.
	SuppressedReason string
	// RetrievalErr, when non-nil, causes BuildRetrievalEventData to populate
	// the "error" field so failed retrievals are distinguishable from
	// zero-hit successes.
	RetrievalErr error
	// RelevanceZ is the top hit's band-normalized relevance when the noise
	// floor was applied; nil when it was never measured. A pointer because
	// z == 0 is a real measurement (the top hit sat exactly on the floor)
	// and must stay distinguishable from absence — a window criterion on
	// median z was once committed and then found uncomputable because this
	// value lived only in the CLI output, never in the event.
	RelevanceZ *float64
}

AskEventParams is the boundary-clean input for composing an ask event payload. The cmd layer converts retrieval/result types into this shape so the experiment package stays agnostic of retrieval implementation details (no import of internal/query).

type CalibrationSnapshot 

type CalibrationSnapshot struct {
	CalibrationID    string  `json:"calibration_id"`
	CreatedAt        string  `json:"created_at"` // RFC3339
	EmbedderLabel    string  `json:"embedder_label"`
	EmbeddingDims    int     `json:"embedding_dims"`
	NoteCount        int     `json:"note_count"`
	NoiseFloor       float64 `json:"noise_floor"`
	NoiseFloorProbes int     `json:"noise_floor_probes"`
	ProbeSetVersion  int     `json:"probe_set_version"`
	NTNCosineMu      float64 `json:"ntn_cosine_mu"`
	NTNCosineSigma   float64 `json:"ntn_cosine_sigma"`
	NTNSampleCount   int     `json:"ntn_sample_count"`
	// VaultPath scopes the snapshot to one vault so the ask path reads THIS
	// vault's floor (LatestCalibrationForVault), not the globally-most-recent
	// one. `json:"-"` — it is a storage discriminator, NOT part of the
	// content-free federated-export surface, and must never be serialized.
	VaultPath string `json:"-"`
}

CalibrationSnapshot is a per-vault retrieval-calibration measurement, stored in the experiment DB. Every field is a content-free scalar — no note ids, no content, no query text — so the snapshot doubles as the vault_features the federated study (Paper #2) aggregates, and can be exported as-is.

  • NoiseFloor (N): the cosine an off-topic probe query gets to any note — the relevance floor. Relevance R = top_cosine - N.
  • NTNCosineMu/Sigma: the vault's note-to-note cosine dispersion (how tight or loose the embedding space is). NTNSampleCount is the number of pairs sampled.

type CallerUse added in v0.7.0 

type CallerUse struct {
	Caller     string
	Injections int
	Consumed   int
	Opened     int
	Cited      int
	Unknown    int
}

CallerUse is one hook's slice of MemoryUse.

type ComparableEvent 

type ComparableEvent struct {
	EventID string
	Pairs   []EventPair
}

ComparableEvent is one DB event after pair extraction. Events without shadow variants (or without a primary) should be filtered out before this slice is passed to AggregateComparisons.

type ComparableEventFilter 

type ComparableEventFilter struct {
	SessionID    string
	Caller       string
	SinceRFC3339 string
	EventTypes   []string
}

ComparableEventFilter narrows which events LoadComparableEvents returns. Zero-value fields mean "no filter." EventTypes empty → default to ask/search/context_pack.

type DB 

type DB struct {
	// contains filtered or unexported fields
}

DB wraps *sql.DB with schema initialization and experiment-specific helpers.

func Open 

func Open(dbPath string) (*DB, error)

Open opens (or creates) a SQLite database at dbPath, creates the parent directory if needed, applies pragmas (WAL mode, foreign key enforcement), and runs pending migrations using PRAGMA user_version for versioning.

func OpenExisting added in v0.7.0 

func OpenExisting(dbPath string) (*DB, error)

OpenExisting opens the usage log only if it is already there.

Open creates the file, runs migrations, and leaves a schema behind. That is correct for the write path and wrong for every READ command: asking "what is in the log?" must not be the thing that creates one.

It also made a documented promise false. `experiments.telemetry: off` says nothing is written — and `vaultmind experiment trace` under off would write a fresh database as a side effect of reporting that there was nothing to report. A read that creates its own subject is the same silent-write shape this codebase keeps closing.

Reading an EXISTING log under off stays allowed on purpose: turning telemetry off is a decision about new data, not a reason to lock the operator out of what was already collected.

func OpenMemory added in v0.7.0 

func OpenMemory() (*DB, error)

OpenMemory returns an empty, migrated log that exists only in memory.

For the read path when no log is on disk: the caller gets a database with the real schema and zero rows, so every reader, manifest and rollup behaves exactly as it would against an empty file — without creating one. The alternative, a separate "there is nothing to export" code path, is a second implementation of the output format that drifts from the first.

func (*DB) AccessedNoteIDs 

func (d *DB) AccessedNoteIDs() ([]string, error)

AccessedNoteIDs returns all unique note IDs from note_access events that count as activation signal, in the order each was first accessed.

func (*DB) BatchNoteAccessTimes 

func (d *DB) BatchNoteAccessTimes(noteIDs []string) (map[string][]time.Time, error)

BatchNoteAccessTimes returns access times for multiple notes in one pass.

THIS is the function the activation scorer calls (activation_scorer.go, shadow_variants.go) — not NoteAccessTimes. When the phantom-read filter was first added it was applied to NoteAccessTimes and AccessedNoteIDs and missed here, so a note with even one genuine read stayed in the candidate set and kept every phantom alongside it: the fix was a no-op on the exact note it cited as evidence.

Any new query over EventNoteAccess must call activationSignalFrom before treating a row as a retrieval. There are three such queries in this file and they are the entire input surface of the scorer.

func (*DB) Begin 

func (d *DB) Begin() (*sql.Tx, error)

Begin starts a transaction.

func (*DB) Close 

func (d *DB) Close() error

Close closes the underlying database connection.

func (*DB) CountEvents 

func (d *DB) CountEvents() (int, error)

CountEvents returns the total number of event rows.

func (*DB) CountOutcomes 

func (d *DB) CountOutcomes() (int, error)

CountOutcomes returns the total number of outcome rows.

func (*DB) CountSessions 

func (d *DB) CountSessions() (int, error)

CountSessions returns the total number of session rows. Used by rollup assembly so the receiver can sanity-check the aggregation horizon (e.g. is this vault active or stale?).

func (*DB) DistinctVariants 

func (d *DB) DistinctVariants() ([]string, error)

DistinctVariants returns the sorted list of distinct variant names in the outcomes table.

func (*DB) EndSession 

func (d *DB) EndSession(sessionID string) error

EndSession sets ended_at for the given session to the current time.

func (*DB) Exec 

func (d *DB) Exec(query string, args ...any) (sql.Result, error)

Exec executes a query that doesn't return rows.

func (*DB) GetSessionCaller 

func (d *DB) GetSessionCaller(sessionID string) (SessionCaller, error)

GetSessionCaller returns the caller attribution for the given session. Returns zero values (empty caller, nil meta, empty UserSessionID) for sessions that predate attribution or were started via the older StartSession.

func (*DB) LatestCalibration 

func (d *DB) LatestCalibration() (*CalibrationSnapshot, error)

LatestCalibration returns the most recently created snapshot across ALL vaults, or (nil, nil) when none exists. Use this only for vault-agnostic checks; the ask path must use LatestCalibrationForVault to avoid reading another vault's floor.

func (*DB) LatestCalibrationForVault 

func (d *DB) LatestCalibrationForVault(vaultPath string) (*CalibrationSnapshot, error)

LatestCalibrationForVault returns the most recent snapshot for one vault, or (nil, nil) when that vault has none. This is the lookup the ask path uses: the experiment DB is global (many vaults), so scoping by vault_path is what keeps a query against vault B from reading vault A's noise floor. Callers should pass a filepath.Clean'd path so "/a/b" and "/a/b/" match the stored key.

func (*DB) LinkOutcomes 

func (d *DB) LinkOutcomes(currentSessionID, noteID string, outcomeWindow int) (int, error)

LinkOutcomes looks back at recent search/ask/context_pack events (within the outcome window) and creates outcome rows for any variant where noteID appears in the results. Returns the number of outcome rows created.

func (*DB) LoadComparableEvents 

func (d *DB) LoadComparableEvents(f ComparableEventFilter) ([]ComparableEvent, error)

LoadComparableEvents returns events that carry at least one shadow variant (i.e., a variants map with primary plus another) with their pairs pre-extracted. Rows whose primary_variant is NULL/empty are skipped.

func (*DB) LogEvent 

func (d *DB) LogEvent(evt Event) (string, error)

LogEvent inserts an event row into the events table and returns the generated event ID. Optional string fields (QueryText, QueryMode, PrimaryVariant) are stored as NULL when empty.

func (*DB) MemoryUseSince added in v0.7.0 

func (d *DB) MemoryUseSince(windowDays int) (*MemoryUse, error)

MemoryUseSince computes the meter over the last windowDays.

Both sides of the window comparison go through datetime(). Events are stored as RFC3339 ("...T20:00:00Z") while SQLite's datetime() returns a space-separated form, and comparing the two as strings makes 'T' > ' ' — so the upper bound never matches and every access looks like it fell outside the window. The first hand-run of this measurement had that bug and under-reported consumption; the tests below are what caught it.

func (*DB) NoteAccessTimes 

func (d *DB) NoteAccessTimes(noteID string) ([]time.Time, error)

NoteAccessTimes returns timestamps of all note_access events for noteID, ordered ascending. Filters by parsing event_data JSON for matching note_id.

func (*DB) NoteRetrievals 

func (d *DB) NoteRetrievals(noteID string) ([]SessionHit, error)

NoteRetrievals returns every (session, event, rank, timestamp) tuple where the given note appeared in a retrieval. Ordered chronologically. Best rank across variants wins when a note appears under multiple variants of one event — so one event contributes one row per distinct note, not per variant.

func (*DB) PerNoteStats 

func (d *DB) PerNoteStats() ([]NoteStat, error)

PerNoteStats returns one row per note that has appeared in a retrieval event, ordered by retrieval_count_total desc (most-often-retrieved first), then by last_retrieved_ts desc to break count ties with recency.

Note: the source-of-truth for which notes exist is the vault index DB, not the experiment DB. This query only knows about notes that have been retrieved at least once; notes that were never retrieved do not appear.

func (*DB) Query 

func (d *DB) Query(query string, args ...any) (*sql.Rows, error)

Query executes a query that returns rows.

func (*DB) QueryRow 

func (d *DB) QueryRow(query string, args ...any) *sql.Row

QueryRow executes a query that returns at most one row.

func (*DB) RecentSessionWindows 

func (d *DB) RecentSessionWindows(limit int) ([]SessionWindow, error)

RecentSessionWindows returns the N most recent completed sessions as SessionWindows. Orphans (NULL ended_at) are excluded.

func (*DB) RecordUseEvidence added in v0.10.10 

func (d *DB) RecordUseEvidence(transcriptPath string) error

RecordUseEvidence runs during episode capture, off the recall/prompt path. It records observations, never an inference that a note was used or unused. Re-capture upserts one event per (surfaced event, note), allowing a late read or a fuller transcript to correct evidence without inflating doctor's counts. Work is bounded to two seconds; timeout leaves the transaction uncommitted and is best effort so episode capture can continue.

func (*DB) RecoverOrphans 

func (d *DB) RecoverOrphans() (int, error)

RecoverOrphans finds sessions with NULL ended_at and sets ended_at to the timestamp of their last event, or started_at + 1 minute if no events exist. Returns the number of sessions recovered.

func (*DB) Report 

func (d *DB) Report(variants []string, k int) (*ReportResult, error)

Report computes Hit@K and MRR for the given variants across all linkable events. variants is the list of variant names to report on; k is the rank cutoff for Hit@K.

func (*DB) SessionGaps 

func (d *DB) SessionGaps() ([]SessionGap, error)

SessionGaps returns all sessions with their inter-session gaps, ordered by started_at ascending. Derived from the session_gaps SQL view.

func (*DB) SessionRetrievals 

func (d *DB) SessionRetrievals(sessionID string) ([]RetrievalEventSummary, error)

SessionRetrievals returns every retrieval event in the given session, chronologically ascending. Events of type search / ask / context_pack are included; note_access and index_embed are not (they are not retrievals). An unknown session ID returns an empty slice without error.

func (*DB) SessionsByUserSession 

func (d *DB) SessionsByUserSession(userSessionID string) ([]string, error)

SessionsByUserSession returns all session_ids (invocation-sessions) that belong to the given user_session_id. Useful for reconstructing "everything that happened in this working session" across multiple invocations.

func (*DB) StartSession 

func (d *DB) StartSession(vaultPath string) (string, error)

StartSession inserts a new session row and returns the generated session ID.

func (*DB) StartSessionWithCaller 

func (d *DB) StartSessionWithCaller(vaultPath, caller string, meta map[string]any) (string, error)

StartSessionWithCaller starts a session and records caller attribution. meta is serialized as JSON; nil meta is stored as NULL. A user_session_id is computed from (caller, user, host) against recent sessions — reused when the last matching session is within UserSessionThreshold, otherwise a new id is minted.

func (*DB) StoreCalibration 

func (d *DB) StoreCalibration(snap *CalibrationSnapshot) error

StoreCalibration inserts a calibration snapshot. Snapshots are append-only (history is kept); LatestCalibration reads the most recent.

func (*DB) UpdateSessionVaultPath 

func (d *DB) UpdateSessionVaultPath(sessionID, vaultPath string) error

UpdateSessionVaultPath sets the vault_path for the given session. Called when the vault path becomes known (after command flag resolution).

func (*DB) UsageSummary 

func (d *DB) UsageSummary(topN int) (*UsageSummary, error)

UsageSummary computes the aggregate metrics in one pass across the DB. topN caps the TopNotes slice; 0 means "all recalled notes." topN is applied after the per_note_stats view's own ordering (count desc, recency desc).

type Dispatcher 

type Dispatcher struct {
	// contains filtered or unexported fields
}

Dispatcher routes scoring requests to registered variant scorers.

func NewDispatcher 

func NewDispatcher() *Dispatcher

NewDispatcher creates a dispatcher with the built-in "none" scorer registered.

func (*Dispatcher) Register 

func (d *Dispatcher) Register(variant string, s Scorer)

Register adds a scorer for the given variant name.

func (*Dispatcher) RunAll 

func (d *Dispatcher) RunAll(variants []string, notes []ScoredNote) (map[string][]ScoredNote, error)

RunAll runs all specified variants and returns map[variant][]ScoredNote.

func (*Dispatcher) Score 

func (d *Dispatcher) Score(variant string, notes []ScoredNote) ([]ScoredNote, error)

Score runs the scorer for the given variant. Returns error for unknown variants.

type Event 

type Event struct {
	SessionID      string
	Type           string
	VaultPath      string
	QueryText      string
	QueryMode      string
	PrimaryVariant string
	Data           map[string]any
}

Event holds the data for a single experiment event to be logged.

type EventPair 

type EventPair struct {
	PrimaryVariant string
	ShadowVariant  string
	PrimaryList    []string
	ShadowList     []string
}

EventPair is one (primary, shadow) comparison extracted from a single event's variants map. The two lists are note IDs in rank order (rank 1 first).

func ExtractEventPairs 

func ExtractEventPairs(eventData map[string]any, primaryName string) ([]EventPair, error)

ExtractEventPairs walks an event's decoded event_data JSON and returns one EventPair per shadow variant. Shadow order is deterministic (sorted by name). Returns an error if the primary variant is absent.

type ExperimentDef 

type ExperimentDef struct {
	Enabled bool
	Primary string
	Shadows []string
}

ExperimentDef holds the parsed definition for a single experiment.

func (ExperimentDef) AllVariants 

func (e ExperimentDef) AllVariants() []string

AllVariants returns the primary variant followed by all shadow variants.

type GapStats 

type GapStats struct {
	Count         int
	MedianSeconds int64
	P90Seconds    int64
	MaxSeconds    int64
}

GapStats summarizes inter-session intervals (seconds). Count is the number of gaps — always TotalSessions-1, or 0 when fewer than 2 sessions exist.

type MemoryUse added in v0.7.0 

type MemoryUse struct {
	// WindowDays is the period measured, for rendering.
	WindowDays int

	// Injections counts notes surfaced by hook-driven retrievals — the rows an
	// agent was shown, not the number of queries.
	Injections int

	// Consumed preserves the legacy access-within-window count for JSON callers.
	// It predates conversation/vault scoping; use Opened for read evidence.
	Consumed int
	Opened   int
	Cited    int
	Unknown  int

	// BodiesDelivered counts injections whose event recorded that a body was
	// actually included. Zero for every event logged before that field existed,
	// which is most of them — so a low number here means "unknown" for old data
	// and "withheld" for new.
	BodiesDelivered int

	// PerCaller breaks the same numbers down by hook, because the recall hook
	// (every prompt) and the reach hook (decision points only) are different
	// bets and deserve separate observations.
	PerCaller []CallerUse
}

MemoryUse measures observable evidence for hook-surfaced notes. Opened is a read event in the same conversation and vault; Cited is an exact id in the turn's assistant output or tool inputs, excluding injected hook context. Unknown includes missing transcript observations: it never means unused.

func (MemoryUse) ConsumedRate added in v0.7.0 

func (m MemoryUse) ConsumedRate() float64

ConsumedRate returns consumed/injections as a fraction, 0 when nothing was surfaced. Callers render it; this does not decide how many decimals a number this small deserves.

type NoteStat 

type NoteStat struct {
	NoteID              string
	RetrievalCountTotal int
	FirstRetrievedTs    string
	LastRetrievedTs     string
}

NoteStat reports aggregate retrieval metrics for a single note, derived from the per_note_stats SQL view over event_data.variants.*.results[]. Counts are per distinct retrieval event, not per variant occurrence — so a note appearing under 3 shadow variants of one event counts once.

type ReportResult 

type ReportResult struct {
	SessionCount int                       `json:"session_count"`
	EventCount   int                       `json:"event_count"`
	OutcomeCount int                       `json:"outcome_count"`
	K            int                       `json:"k"`
	Variants     map[string]VariantMetrics `json:"variants"`
}

ReportResult holds the full experiment report.

type RetrievalEventHit 

type RetrievalEventHit struct {
	NoteID string
	Rank   int
}

RetrievalEventHit is one note surfaced by one event, with its rank in the event's result set. Rank is the minimum across all variants that contained the note (best placement wins).

type RetrievalEventSummary 

type RetrievalEventSummary struct {
	EventID   string
	EventType string
	Timestamp string
	Query     string
	Hits      []RetrievalEventHit
}

RetrievalEventSummary describes one retrieval event — its type, timestamp, query, and the deduplicated list of notes returned. Multiple variants of the same event collapse: a note appearing under "hybrid" and three shadow variants is reported once with its best (lowest) rank across them.

type RetrievalHit 

type RetrievalHit struct {
	NoteID   string
	Rank     int
	Score    float64
	NoteType string
	Path     string
	Scores   map[string]float64
}

RetrievalHit is the input shape for building a variant payload: a single retrieved note with its rank, aggregated score, and optional per-component scores (e.g. fts/dense/sparse/colbert for hybrid retrieval).

Storage/privacy note: payloads built from RetrievalHit are written to the local experiment DB only. The anonymous/full telemetry tiers are identical in the current (local-only) write path — see cmd/root.go. A future uploader MUST filter by tier: anonymous should strip note_id/path; full may send everything.

type Rollup 

type Rollup struct {
	Kind             string                   `json:"kind"`
	SchemaVersion    int                      `json:"schema_version"`
	Tier             string                   `json:"tier"`
	Fingerprint      string                   `json:"vault_fingerprint"`
	NoteCount        int                      `json:"note_count"`
	TypeDistribution map[string]int           `json:"type_distribution"`
	LinkCount        int                      `json:"link_count"`
	AliasCount       int                      `json:"alias_count"`
	EmbeddingCount   int                      `json:"embedding_count"`
	EmbeddingDims    int                      `json:"embedding_dims"`
	VariantStats     map[string]*VariantStats `json:"variant_stats"`
	ExportedAt       string                   `json:"exported_at"`
	SessionCount     int                      `json:"session_count"`
	EventCount       int                      `json:"event_count"`
	OutcomeCount     int                      `json:"outcome_count"`
}

Rollup is the federated-aggregator-shaped payload — one record per vault that captures everything the paper (Paper #2 from the federated-paper design note) needs without exposing content.

Designed to be small, content-free, and stable across schema versions. SchemaVersion tracks payload shape so receivers can dispatch on it.

func (*Rollup) MarshalJSON 

func (r *Rollup) MarshalJSON() ([]byte, error)

MarshalJSON pins the field order via custom marshaling for human-readable preview output. The standard library marshaler would alphabetize; we want kind/version/tier first, fingerprint next, then aggregate features, then variants. Receivers that care about order can rely on the JSON itself; receivers that don't, ignore.

type ScoredNote 

type ScoredNote struct {
	NoteID   string             `json:"note_id"`
	Score    float64            `json:"score"`
	Features map[string]float64 `json:"features,omitempty"`
}

ScoredNote is a note with a computed score, used as input/output for variant scorers.

type Scorer 

type Scorer interface {
	Score(notes []ScoredNote) []ScoredNote
}

Scorer computes variant-specific scores for a list of notes.

type ScorerFunc 

type ScorerFunc func([]ScoredNote) []ScoredNote

ScorerFunc adapts a plain function to the Scorer interface.

func (ScorerFunc) Score 

func (f ScorerFunc) Score(notes []ScoredNote) []ScoredNote

Score implements Scorer for ScorerFunc.

type Session 

type Session struct {
	DB            *DB
	ID            string
	VaultPath     string
	OutcomeWindow int // configurable outcome window; 0 uses default (2)
}

Session holds the active experiment session and its associated database.

func FromContext 

func FromContext(ctx context.Context) *Session

FromContext retrieves the Session from ctx. Returns nil if no Session is stored in the context.

func (*Session) LogAskEvent 

func (s *Session) LogAskEvent(query, primaryVariant string, data map[string]any) (string, error)

LogAskEvent logs an ask event from this session. primaryVariant is the activation experiment's chosen variant name (empty when the activation experiment is disabled or the event has no context pack); it populates the events.primary_variant column so per-variant rollups can group on it without parsing event_data JSON. Without this, the activation experiment's variant choice landed only in the event_data blob and the column stayed NULL — silently breaking VariantPerformance and `vaultmind export --rollup`.

func (*Session) LogContextPackEvent 

func (s *Session) LogContextPackEvent(data map[string]any) (string, error)

LogContextPackEvent logs a context_pack event from this session.

func (*Session) LogNoteAccessEvent 

func (s *Session) LogNoteAccessEvent(noteID string, source AccessSource, bodyDelivered bool) (string, error)

LogNoteAccessEvent logs a note_access event and triggers outcome linkage.

bodyDelivered records whether the note's CONTENT actually reached the agent, which is not the same question as whether the note was surfaced. It is a parameter rather than an inference so every call site has to answer it: the ask path surfaced a top hit on every query and recorded an access for it even when the body was withheld, and the activation scorer then counted those as retrievals. See IsActivationSignal for the loop that produced.

func (*Session) LogNoteAccessEventAs added in v0.8.0 

func (s *Session) LogNoteAccessEventAs(noteID string, source AccessSource, bodyDelivered bool, caller string) (string, error)

LogNoteAccessEventAs is LogNoteAccessEvent with an explicit caller tag.

An empty caller writes NO caller key at all, rather than an empty string: absence and "" are different facts, and a reader that cannot tell them apart is how a missing field becomes a confident zero.

func (*Session) LogSearchEvent 

func (s *Session) LogSearchEvent(query, mode string, data map[string]any) (string, error)

LogSearchEvent logs a search event from this session.

func (*Session) SetVaultPath 

func (s *Session) SetVaultPath(vaultPath string)

SetVaultPath updates the vault path on the session and persists it to the DB.

type SessionCaller 

type SessionCaller struct {
	Caller        string
	Meta          map[string]any
	UserSessionID string
}

SessionCaller describes the invoking agent for a session. Caller is a short label ("companion-persona-hook", "claude-code", "cli", or empty if the session pre-dates attribution). Meta is a flexible JSON blob — project_dir, pid, env-var snapshot — parsed back into a map so callers can read it without knowing the shape. UserSessionID groups multiple invocations into one working session via the UserSessionThreshold time heuristic.

type SessionGap 

type SessionGap struct {
	SessionID     string
	StartedAt     string
	PrevSessionID sql.NullString
	PrevStartedAt sql.NullString
	GapSeconds    sql.NullInt64
}

SessionGap reports one session and its gap from the previous session. Sessions are ordered chronologically ascending. The first session has no predecessor — PrevSessionID, PrevStartedAt, and GapSeconds are NULL.

Primary consumer: compressed-idle-time analysis (gamma parameter fitting).

type SessionHit 

type SessionHit struct {
	SessionID string
	EventID   string
	EventType string
	Timestamp string
	Rank      int
}

SessionHit is one occurrence of a note being retrieved: which session saw it, when, at what rank, under which event. Used by NoteRetrievals to build a cross-session history for a single note.

type SessionWindow 

type SessionWindow struct {
	Start time.Time
	End   time.Time
}

SessionWindow represents a period when VaultMind was actively in use.

type UsageSummary 

type UsageSummary struct {
	TotalSessions       int
	RetrievalEventCount int
	UniqueNotesRecalled int
	TopNotes            []NoteStat
	GapStats            GapStats
}

UsageSummary reports aggregate memory-usage metrics derived from the experiment DB — intended as the "weekly readout" view for someone dogfooding VaultMind: which notes are being recalled, how often, and how sessions are spaced over time.

type VariantMetrics 

type VariantMetrics struct {
	HitAtK     float64 `json:"hit_at_k"`
	MRR        float64 `json:"mrr"`
	EventCount int     `json:"event_count"`
}

VariantMetrics holds computed metrics for a single variant.

type VariantStats 

type VariantStats struct {
	Name         string  `json:"name"`
	EventCount   int     `json:"event_count"`
	OutcomeCount int     `json:"outcome_count"`
	HitAt5       float64 `json:"hit_at_5"`
	HitAt10      float64 `json:"hit_at_10"`
	MRR          float64 `json:"mrr"`
}

VariantStats summarizes one variant's performance across all retrieval events in the experiment DB. The fields are exactly what the federated paper (the federated-paper design note) needs to answer H2.5: which retrieval constants vary meaningfully across vaults vs which are stable.

Reciprocal-rank semantics:

  • For every event whose primary_variant equals this variant AND for which an outcome row was logged (i.e. a note that was retrieved was later actually accessed), we contribute (1 / rank) to the reciprocal-rank sum. MRR = sum / event_count.
  • Hit@K is the fraction of primary-variant events where the accessed-note's rank is ≤ K.

This is the conservative interpretation: shadow variants do not get MRR/Hit@K credit because we don't know what the user "would have" accessed under that variant — only what they actually accessed under the primary. The federated dataset still recovers shadow signal via the side-by-side rank distributions in the variants payload.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.