store

package
v0.5.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 30, 2026 License: MIT Imports: 14 Imported by: 0

Documentation

Overview

Package store owns the SQLite index. Knowledge-card search data can be rebuilt from files; runtime injection history is preserved by explicit schema migrations.

Index

Constants

View Source 
const (
	EffHelpful   = "helpful"   // pulled or referenced after injection
	EffNoisy     = "noisy"     // downvoted or repeatedly ignored as a pointer
	EffExpensive = "expensive" // heavy token spend, no observed positive signal
	EffUncertain = "uncertain" // too few observations to judge
)

Effectiveness buckets — a per-card verdict derived from the decayed feedback counters plus the card's footprint in the injection-log retention window. Purely read-side: buckets never feed ranking (UtilityMultipliers does that); they exist so the operator can see which cards earn their tokens.

View Source 
const (
	GranHook    = "hook"
	GranSummary = "summary"
	GranBody    = "body"
)

Granularity levels, ordered: hook < summary < body. Session dedup is monotonic — a card is only re-sent at a strictly higher level (14.43).

View Source 
const (
	FeedbackInjected   = "injected"
	FeedbackExpanded   = "expanded"   // expand_card pull — strong positive
	FeedbackReferenced = "referenced" // cited by a saved lesson — strongest positive; recorded by the Phase 4 miner
	FeedbackDownvoted  = "downvoted"  // explicit `culi down` (weight 1.0) or abandoned pointer (0.1)
)

Feedback kinds — event counters on card_stats. Weights are applied at multiplier time (utilityWeights), so counters stay interpretable as decayed event counts.

Variables

This section is empty.

Functions

func GranLevel 

func GranLevel(g string) int

GranLevel maps a granularity name to its ordering level.

Types

type BM25Hit 

type BM25Hit struct {
	Rowid int64
	Rank  float64 // bm25(): lower (more negative) = better
}

SearchBM25 runs an FTS5 MATCH over the corpus and returns (rowid, rank) pairs, best first. matchExpr MUST be pre-sanitized by the caller (internal/retrieve owns the sanitizer) — raw prompt text here is a bug.

type CardStats 

type CardStats struct {
	Injected, Expanded, Referenced, Downvoted float64
	LastDecayAt                               time.Time
}

CardStats is one card's decayed counters.

type Effectiveness added in v0.3.0 

type Effectiveness struct {
	Bucket   string  `json:"bucket"`
	Positive float64 `json:"positive"`  // weighted: 3·expanded + 5·referenced
	Negative float64 `json:"negative"`  // weighted: 5·downvoted
	PullRate float64 `json:"pull_rate"` // (expanded+referenced) per observed injection
}

Effectiveness is one card's verdict with the rates behind it.

func ClassifyEffectiveness added in v0.3.0 

func ClassifyEffectiveness(cs CardStats, usage WindowUsage) Effectiveness

ClassifyEffectiveness buckets one card from its decayed feedback counters and its retention-window usage.

Deliberate asymmetry: full-body pushes are unobservable (plan §9 — an injected body can help without ever being expanded), so "no signal" alone never marks a card noisy. Negative inference comes only from explicit downvotes and the abandoned-pointer penalty, both already in Downvoted; signal-free cards are at most "expensive" when their token cost is high.

The asymmetry only holds because of effNoisyFloor. A single unexpanded pointer is barely distinguishable from "no signal" — expansion runs at about 1% — so without the floor the noisy bucket degenerates into "was ever shown as a pointer", which is exactly the silence this comment promises not to punish.

type EmbedTarget 

type EmbedTarget struct {
	Rowid       int64
	ContentHash string
	Text        string // title + summary — what gets embedded
}

EmbedTarget is a card needing (re-)embedding.

type FileState 

type FileState struct {
	Path  string
	MTime int64
	Size  int64
	Hash  string
}

FileState is the (mtime, size) fingerprint the indexer compares before re-hashing a file.

type InjectionAgg 

type InjectionAgg struct {
	Event       string `json:"event"`
	Granularity string `json:"granularity"`
	Count       int    `json:"count"`
	Tokens      int    `json:"tokens"`
}

InjectionAgg is one (event, granularity) bucket of the injection history. Only the retention window exists — PruneStale bounds it to ~7 days. JSON tags serve `culi stats --json` (snake_case like the rest of that schema).

type InjectionRecord 

type InjectionRecord struct {
	CardID      string
	Granularity string
	Tokens      int
}

InjectionRecord is one emitted card at one granularity.

type InjectionRow 

type InjectionRow struct {
	SessionID   string
	TS          string
	Event       string
	CardID      string
	Granularity string
	Tokens      int
	Cwd         string
	Harness     string // source agent: claude | codex
}

InjectionRow is one logged card injection with its timestamp — the raw material the review console groups into the Activity "Injections" view. Read-only, off the hot path.

type Store 

type Store struct {
	// contains filtered or unexported fields
}

Store wraps the SQLite handle. Safe for concurrent use (database/sql pools; WAL allows one writer + many readers).

func Open 

func Open(ctx context.Context, path string) (*Store, error)

Open opens (creating if needed) the index at path. WAL + busy_timeout are set per-connection via the DSN so every pooled connection gets them.

func (*Store) AddFeedback 

func (s *Store) AddFeedback(ctx context.Context, cardID, kind string, delta float64) error

AddFeedback folds decay-to-now into a card's counters and bumps one of them. kind must be a Feedback* constant; delta is the event count (0.1 for an abandoned pointer, 1 otherwise).

func (*Store) AllCardStats 

func (s *Store) AllCardStats(ctx context.Context, now time.Time) (map[string]CardStats, error)

AllCardStats returns every card's counters decayed to now — the raw material for stats' top-pulled / noisy lists. Off the hot path.

func (*Store) AllCards 

func (s *Store) AllCards(ctx context.Context) ([]StoredCard, error)

AllCards returns every card (corpus is small; callers filter in Go).

func (*Store) AllCardsMeta 

func (s *Store) AllCardsMeta(ctx context.Context) ([]StoredCard, error)

AllCardsMeta returns every card WITHOUT bodies — the hot-path scan. Bodies are hydrated per-candidate via CardsByRowid only for the few cards the packer actually emits.

func (*Store) CardByID 

func (s *Store) CardByID(ctx context.Context, id string) (StoredCard, error)

CardByID fetches one card by full ID or 4-hex short ID.

func (*Store) CardWindowUsage added in v0.3.0 

func (s *Store) CardWindowUsage(ctx context.Context) (map[string]WindowUsage, error)

CardWindowUsage aggregates the injection log per card over its retention window — the exposure and cost half of the effectiveness verdict. Off the hot path.

func (*Store) CardsByRowid 

func (s *Store) CardsByRowid(ctx context.Context, rowids []int64) ([]StoredCard, error)

CardsByRowid hydrates specific rows (e.g. FTS hits), preserving no order.

func (*Store) ClaimAttribution added in v0.5.0 

func (s *Store) ClaimAttribution(ctx context.Context, sessionID string, now time.Time) (bool, error)

ClaimAttribution reserves the right to credit this session's card usage, returning true exactly once per session. Usage credit is additive, so it must not be applied twice — and the job label cannot be trusted for that: the Codex rollout scanner stamps trigger "session-end" on every rescan of a growing rollout, and a resumed Claude session reaches SessionEnd more than once. Both would otherwise re-credit the whole session from offset 0.

One statement, so two workers racing the same session cannot both win: the conflicting UPDATE is skipped when the row was already claimed, leaving RowsAffected at 0. The marker rides in session_state, which PruneStale already bounds to the retention window.

func (*Store) Close 

func (s *Store) Close() error

Close releases the underlying pool.

func (*Store) DailyTokens 

func (s *Store) DailyTokens(ctx context.Context, days int) ([]int, error)

DailyTokens returns injected-token totals per calendar day (UTC), oldest first, for up to the last `days` days present in the retention window — the Overview hero sparkline. Empty when nothing has been injected yet.

func (*Store) DeleteCardByPath 

func (s *Store) DeleteCardByPath(ctx context.Context, path string) error

DeleteCardByPath removes a vanished file's card + FTS row.

func (*Store) Embeddings 

func (s *Store) Embeddings(ctx context.Context, model string) (map[int64][]float32, error)

Embeddings returns rowid → vector for every card whose stored vector is FRESH: same model and same content hash as the current card row. Stale vectors are simply absent — the funnel degrades per-card, never errors.

func (*Store) FileStates 

func (s *Store) FileStates(ctx context.Context) (map[string]FileState, error)

FileStates returns path → state for every indexed card.

func (*Store) InjectedCardIDs 

func (s *Store) InjectedCardIDs(ctx context.Context) (map[string]bool, error)

InjectedCardIDs returns the set of card IDs with at least one row in the injection log's retention window — the "was this card ever shown" half of the staleness report.

func (*Store) InjectedLevels 

func (s *Store) InjectedLevels(ctx context.Context, sessionID string) (map[string]int, error)

InjectedLevels returns card_id → highest granularity level already injected in this session.

func (*Store) InjectionAggs 

func (s *Store) InjectionAggs(ctx context.Context) ([]InjectionAgg, error)

InjectionAggs aggregates the injection log for `culi stats`.

func (*Store) InjectionsSince added in v0.3.0 

func (s *Store) InjectionsSince(ctx context.Context, cutoff time.Time) ([]InjectionRow, error)

InjectionsSince returns every retained injection at or after cutoff, oldest first. It is intentionally uncapped: analytics needs the complete retention window, while RecentInjections is the bounded feed used by the Activity UI. This query is only used by the local console and never runs on a hook path.

func (*Store) MetaGet 

func (s *Store) MetaGet(ctx context.Context, key string) (string, error)

MetaGet reads one meta key ("" when absent).

func (*Store) MetaSet 

func (s *Store) MetaSet(ctx context.Context, key, value string) error

MetaSet writes one meta key.

func (*Store) MissingEmbeddings 

func (s *Store) MissingEmbeddings(ctx context.Context, model string) ([]EmbedTarget, error)

MissingEmbeddings lists cards with no fresh vector for model.

func (*Store) PenalizeAbandonedPointers 

func (s *Store) PenalizeAbandonedPointers(ctx context.Context, sessionID string) error

PenalizeAbandonedPointers applies the −0.5-equivalent nudge (0.1 downvote events × weight 5) to cards injected at hook granularity this session and never expanded. Only pointers earn negative inference: pushed bodies are unobservable (plan §9). Runs at session end, off the per-prompt hot path.

Claims the session first so the penalty lands at most once. SessionEnd is not once-per-session — a resumed Claude session reaches it again on the same session_id — and the penalty is additive, so a second pass would push the same pointers toward "noisy" twice as fast as their evidence warrants. Claiming inside keeps the invariant with the operation rather than relying on every caller to remember it.

The trade this makes: pointers injected *after* a resumed session's first end are never penalized. That is the deliberate direction — negative inference here is already conservative (see ClassifyEffectiveness: silence alone never marks a card noisy), so under-penalizing costs a slower verdict while over-penalizing manufactures one the evidence does not support.

func (*Store) PruneStale 

func (s *Store) PruneStale(ctx context.Context, retainDays int) error

PruneStale deletes injection/session rows older than the retention window. Cheap, runs at SessionStart — keeps the disposable history bounded.

func (*Store) Rebuild 

func (s *Store) Rebuild(ctx context.Context) error

Rebuild recreates only the file-backed card search index. Runtime-only injections, card stats, session state, and metadata are preserved.

func (*Store) RecentInjections 

func (s *Store) RecentInjections(ctx context.Context, limit int) ([]InjectionRow, error)

RecentInjections returns injection-log rows newest first, capped at limit. The console groups them by session then by (ts, event) in Go; keeping the grouping out of SQL avoids a bespoke query the hot path never needs.

func (*Store) RecordInjections 

func (s *Store) RecordInjections(ctx context.Context, sessionID, event, promptHash, cwd string, h harness.Harness, recs []InjectionRecord) error

RecordInjections logs emitted cards for dedup + stats in one transaction. cwd is the working directory at injection time, stored for repo attribution in the review console ("" when unknown — e.g. a non-git prompt). h attributes the injection to its source agent for the console's harness filter.

func (*Store) ResetSession 

func (s *Store) ResetSession(ctx context.Context, sessionID string) error

ResetSession clears dedup state for a session (SessionStart with source=compact: the context window was rebuilt, prior injections are gone).

func (*Store) SearchBM25 

func (s *Store) SearchBM25(ctx context.Context, matchExpr string, limit int) ([]BM25Hit, error)

func (*Store) SessionContentCards added in v0.5.0 

func (s *Store) SessionContentCards(ctx context.Context, sessionID string) ([]string, error)

SessionContentCards lists the cards whose actual content reached the model this session — summary or body granularity, never bare pointers. It is the candidate set for usage attribution: only a card the model could read can be credited for having been used. Pointers keep the opposite contract (expand or be penalized, see PenalizeAbandonedPointers), so excluding them here prevents one line of teaser text earning the same credit as a full body.

func (*Store) SessionCount 

func (s *Store) SessionCount(ctx context.Context) (int, error)

SessionCount returns the number of distinct sessions in the injection log.

func (*Store) SessionTokens 

func (s *Store) SessionTokens(ctx context.Context, sessionID string) (int, error)

SessionTokens sums the tokens injected into one session — the statusline's "what did this session cost so far" number.

func (*Store) SwapLastPrompt 

func (s *Store) SwapLastPrompt(ctx context.Context, sessionID, prompt string) (string, error)

SwapLastPrompt returns the previous prompt for the gate's novelty check ("" if none) and stores the new one, bounded to maxStoredPrompt bytes.

func (*Store) UpsertCard 

func (s *Store) UpsertCard(ctx context.Context, c knowledge.Card, mtime, size int64) error

UpsertCard inserts or replaces a card and its FTS row atomically.

func (*Store) UpsertEmbedding 

func (s *Store) UpsertEmbedding(ctx context.Context, rowid int64, model string, vec []float32, contentHash string) error

UpsertEmbedding stores one card's vector, replacing any prior row. The content hash ties the vector to the card revision it was computed from.

func (*Store) UtilityMultipliers 

func (s *Store) UtilityMultipliers(ctx context.Context) (map[string]float64, error)

UtilityMultipliers returns card_id → ranking multiplier, decayed read-side (no writes on the hot path). Laplace-smoothed and clamped to [0.5, 1.5]: feedback tunes ordering but can never bury a card (plan §9). Cards without stats are simply absent — callers treat missing as 1.0.

type StoredCard 

type StoredCard struct {
	knowledge.Card
	Rowid   int64
	ShortID string
}

StoredCard is a card row hydrated from the DB.

type WindowUsage added in v0.3.0 

type WindowUsage struct {
	Injections int
	Tokens     int
}

WindowUsage is one card's footprint in the injection-log retention window (~7 days, bounded by PruneStale): how often it was delivered and what it cost. This — not CardStats.Injected — is the ground truth for exposure: no code path bumps the injected counter (the hook path stays write-light), so classification must read the log.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.