Documentation
¶
Overview ¶
Package sqlite is the derived-cache store described in ARCHITECTURE § 4.
The SQLite database mirrors JSONL columns and adds writer-computed columns (model, tokens, key_hash, prefix_canonical_hash, parent_id, session_root_id). Everything in here is fully rebuildable from data/**/*.jsonl{,.gz}; SQLite is a rebuildable derived cache.
v0 uses modernc.org/sqlite (pure Go, no cgo). Slightly slower than mattn/go-sqlite3 but trivial to build / cross-compile.
Index ¶
- Variables
- type ListFilters
- type ListPage
- type Row
- type SessionSummary
- type Store
- func (s *Store) AllMatching(filters ListFilters, hardCap int) ([]Row, error)
- func (s *Store) AppendTrace(r Row, sessionPrefix []json.RawMessage) error
- func (s *Store) Close() error
- func (s *Store) CountMatching(ctx context.Context, filters ListFilters, hardCap int) (int, error)
- func (s *Store) CountRows() (int64, error)
- func (s *Store) DeleteByJSONLPath(ctx context.Context, canonicalPath string) (int64, error)
- func (s *Store) FindParent(keyHash string, strictHashes []string) (parentID, sessionRootID string, prefixLen int, ok bool, err error)
- func (s *Store) GetByID(id string) (Row, error)
- func (s *Store) List(filters ListFilters) (ListPage, error)
- func (s *Store) ListDistinctJSONLPaths(ctx context.Context) ([]string, error)
- func (s *Store) ListSessions(since time.Time, limit int) ([]SessionSummary, error)
- func (s *Store) StreamMatching(ctx context.Context, filters ListFilters, visit func(Row) error) error
Constants ¶
This section is empty.
Variables ¶
var ErrNotFound = errors.New("trace not found")
ErrNotFound is returned by GetByID when no row matches the given id.
Functions ¶
This section is empty.
Types ¶
type ListFilters ¶
type ListFilters struct {
Since time.Time // ts_start >= Since (zero = no bound)
Until time.Time // ts_start < Until (zero = no bound)
Status *int // exact match if set
StatusBucket int // 0 = no constraint; 2/4/5 = match 2xx/4xx/5xx range
Model string // exact match if set
Path string // exact match if set
PathPrefix string // matched as `LIKE <prefix>%`; mutually exclusive with Path
KeyHashPrefix string // accepts 8- or 16-char prefix; matches by LIKE
SessionRootID string // exact match if set
Project string // exact match on client_project if set
Limit int // 1..500; <=0 means default 100
// Cursor is opaque base64 of "<ts_start_ms>:<id>"; rows strictly older
// than the cursor (ts_start DESC, id DESC ordering) are returned.
// Empty = first page.
CursorTsStart int64
CursorID string
}
ListFilters bounds and filters /api/traces queries per ARCHITECTURE § 6.2. All fields are optional; zero values mean "no constraint".
type ListPage ¶
type ListPage struct {
Rows []Row
NextCursorMs int64 // 0 if no next page
NextCursorID string // empty if no next page
}
ListPage is the paginated result.
type Row ¶
type Row struct {
ID string
TsStart time.Time
TsEnd time.Time
Client string
Method string
Path string
Upstream string
Status int
Disconnected bool
TruncatedReq bool
TruncatedResp bool
// Protocol field copies (may be NULL if absent). Populated from
// trace.Trace by the writer; pass nil for unknown.
Model *string
Stream *bool
PromptTokens *int64
CompletionTokens *int64
TotalTokens *int64
FinishReason *string
// Extracted usage fields. Deterministic copies of named protocol usage
// fields. Nil when the upstream response did not carry the field; nil is
// distinct from a real zero.
CachedTokens *int64 // sum of cache-hit tokens across protocols
CacheCreationTokens *int64 // Anthropic cache_creation_input_tokens (cache miss writes)
ReasoningTokens *int64 // OpenAI Responses reasoning tokens from output
// Client-identity fields. Deterministic copies of named request-header
// fields routed through ExtractClient's taxonomy table; no heuristic UA
// parsing.
ClientKind *string // taxonomy row key (e.g. "claude-code-desktop")
ClientVersion *string // version string from the matching header (if any)
// Project-context field. Deterministic copy of the project name parsed from
// the request body's system/instructions text. Nil when the trace carried
// no project signal.
ClientProject *string // project name (e.g. "my-repo" from `# my-repo` heading)
// Identifiers.
KeyHash string
// Session fields. Populated by AppendTrace (writer side) or by
// scanRow (read side). Callers should NOT set these on the writer
// path — they are derived inside the transaction.
PrefixLen *int
PrefixCanonicalHash *string
ParentID *string
SessionRootID string
// JSONL location.
JSONLPath string
JSONLOffset int64
// MediaCount is the number of extracted media files for this trace. Filled
// by the writer after extraction runs; default 0 when extraction is disabled
// or the trace has no extractable media.
// Deterministic count of named protocol fields only.
MediaCount int
}
Row is the Go-side projection of one traces row. Session-inference columns (PrefixLen, PrefixCanonicalHash, ParentID, SessionRootID) are filled by AppendTrace, not by the caller — pass them empty.
type SessionSummary ¶
type SessionSummary struct {
SessionRootID string
NTurns int64
FirstTs time.Time
LastTs time.Time
LastPath string
LastStatus int
LastModel string
DistinctKeyCount int64
OkCount int64
ErrCount int64
}
SessionSummary is one row of the /api/sessions response — an aggregate over a session_root_id. Useful for "what conversations are happening" without paging through every per-turn trace.
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store is the writer-side handle into index.sqlite. One Writer holds one Store. Read-API handlers will get their own Store handles in M4 (separate connections, WAL allows concurrent reads).
func Open ¶
Open opens or creates index.sqlite at path, runs migration, and applies the pragmas from ARCHITECTURE § 4.
func (*Store) AllMatching ¶
func (s *Store) AllMatching(filters ListFilters, hardCap int) ([]Row, error)
AllMatching returns every row matching filters, ordered chronologically (ts_start ASC, id ASC for a deterministic tiebreak).
Unlike List, this method ignores filters.Limit (the per-page cap) and any cursor fields, and orders ASC so callers (the exporter) can stream results into a reproducible zip. The hardCap argument optionally bounds the total rows fetched; pass 0 (or any non-positive value) for unlimited — the export path uses this. The cap parameter is kept on the signature for future callers that want a guard (rate-limit, preview).
The WHERE-clause is built from the same fields as List (status, status bucket, model, path/prefix, key_hash prefix, session_root_id, since, until) — keep them in sync if List grows new filters.
func (*Store) AppendTrace ¶
func (s *Store) AppendTrace(r Row, sessionPrefix []json.RawMessage) error
AppendTrace inserts a row and runs session inference atomically.
sessionPrefix is the result of session.Build(path, req.body). Pass nil if the trace has no session concept (embeddings, image gen, etc.); the row's session_root_id defaults to id (self-root) and parent_id stays NULL.
Everything happens in one transaction: this gives us one fsync per trace (instead of two for INSERT + UPDATE) and atomic session-inference reads. On commit failure the row is not inserted; the caller's JSONL line is still on disk and will be picked up by startup rebuild.
func (*Store) CountMatching ¶ added in v0.1.1
CountMatching returns the number of rows that match filters, capped at hardCap+1. The subquery pattern (`SELECT COUNT(*) FROM (SELECT 1 ... LIMIT ?)`) lets SQLite stop counting once the cap is reached rather than scanning every matching row — important for the export pre-flight check on multi-million-row stores.
Pass hardCap = 0 (or any non-positive value) to count without a cap. Callers (api/export.go) use the result to short-circuit oversized exports with a 413 BEFORE any zip bytes hit the wire. Returns the raw count; callers compare against their own cap.
Reuses buildListConds so the predicate stays in lockstep with List and StreamMatching — adding a new ListFilters field only requires editing buildListConds.
func (*Store) DeleteByJSONLPath ¶ added in v0.1.1
DeleteByJSONLPath removes every traces row whose jsonl_path column equals canonicalPath. Returns the affected row count.
Called by the storage coordinator's eviction sequence (see internal/storage/eviction.go.deleteIfIdle) after the on-disk JSONL + media subtree have been removed. Backed by idx_jsonl_path so the DELETE plan is a single index seek even with millions of rows.
Context-aware: a canceled ctx surfaces as an error from ExecContext and the caller (deleteIfIdle) returns it up the stack to the monitor tick, which logs and continues. Zero rows affected is NOT an error — it's the "already cleaned up by a previous partial-failure tick" case; deleteIfIdle's defensive-delete model assumes this can happen.
func (*Store) FindParent ¶
func (s *Store) FindParent(keyHash string, strictHashes []string) (parentID, sessionRootID string, prefixLen int, ok bool, err error)
FindParent runs the session-inference IN-clause query described in ARCHITECTURE § 5.2. Given a trace's strict-prefix canonical hashes (in descending-length order), it returns the longest-matching prior trace's (id, session_root_id, prefix_len). If no prior trace matches, ok is false.
strictHashes MUST be ordered by length descending so the SQL's ORDER BY prefix_len DESC stably returns the longest match. The caller's responsibility — sqlite has no information about prefix semantics, only the hashes.
func (*Store) GetByID ¶
GetByID returns the row with the given trace ID. Returns ErrNotFound when no such row exists.
func (*Store) List ¶
func (s *Store) List(filters ListFilters) (ListPage, error)
List returns at most filters.Limit (default 100, max 500) rows ordered by (ts_start DESC, id DESC) that match the filters.
Always orders by ts_start DESC for cursor stability. To page, pass the previous page's NextCursorMs / NextCursorID back as filters.CursorTsStart / filters.CursorID.
func (*Store) ListDistinctJSONLPaths ¶ added in v0.1.1
ListDistinctJSONLPaths returns every distinct jsonl_path stored in the traces table, ordered ASC for deterministic test output. With idx_jsonl_path SQLite can satisfy the DISTINCT scan from the index alone — no table-row visits.
Used by the storage coordinator's reconcileOrphans path to find rows referencing files that no longer exist on disk (post-crash gap between os.Remove + DELETE). Returned slice is safe for the caller to mutate; nothing else in the package shares it.
func (*Store) ListSessions ¶
ListSessions returns the most recent sessions, latest-activity first. limit defaults to 100, max 500. Optional since filter applies to the session's last-activity timestamp.
func (*Store) StreamMatching ¶ added in v0.1.1
func (s *Store) StreamMatching(ctx context.Context, filters ListFilters, visit func(Row) error) error
StreamMatching iterates every row matching filters in the same order AllMatching returns them (ts_start ASC, id ASC), invoking visit on each row. If visit returns an error, iteration stops and that error is returned to the caller. Used by the streaming export path so the exporter never has to hold the full result set in memory.
One database connection is borrowed via db.Conn(ctx) for the entire cursor lifetime — the WAL snapshot stays consistent across all visit calls without blocking the writer. The conn is released on return so the standard pool can serve subsequent queries.
No internal LIMIT: callers gate oversize exports via CountMatching BEFORE calling StreamMatching. The cursor itself is unbounded so adopters opting into `?all=1` see every row.
Context propagation: a canceled ctx surfaces as the visit boundary (next QueryContext call returns ctx.Err()) — partial-write callers (the zip exporter) recover from that mid-stream.
Source Files