Documentation
¶
Overview ¶
Package storage
Index ¶
- Constants
- type BackfillSessionStatusResult
- type Chain
- type Cursor
- type Driver
- type IngestTurnRequest
- type IngestTurnResult
- type ListOpts
- type ModelTokenStats
- type NotFoundError
- type Page
- type ParentRef
- type ParentRefLister
- type SessionBackfillRequest
- type SessionBackfillResult
- type SessionBackfiller
- type SessionIdentity
- type SessionIngester
- type SessionRecord
- type SessionStats
- type SessionStatusBackfiller
Constants ¶
const DefaultListLimit = 50
DefaultListLimit is the page size used when ListOpts.Limit is zero.
const MaxListLimit = 5000
MaxListLimit is the maximum permitted page size. Drivers clamp ListOpts.Limit to this value.
Set high (5000) because the AncestryChains hot path has a large fixed cost per request (the recursive CTE setup) and a tiny incremental cost per leaf — measured at ~260ms regardless of whether limit is 200 or 1000 against the brian_large store. Forcing callers to paginate at small page sizes multiplies the fixed cost. The deck's full Overview load drops from ~17s @ limit=200 (50 round trips) to ~3s @ limit=2000 (5 round trips) just from the math.
Memory impact: each row carries ~200 bytes (the CTE doesn't ship the heavy `content` blob — see the label_hint extraction in pkg/storage/ent/driver/driver.go). 5000 leaves × ~30 avg depth × 200 bytes ≈ 30 MB peak per request, which is fine for an API server backing one deck instance.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type BackfillSessionStatusResult ¶ added in v0.13.0
type Chain ¶ added in v0.4.0
type Chain struct {
// Nodes is the walk output in node-first order: index 0 is the node
// the walk started from, and the last element is either a real root
// or the last resolvable node whose parent could not be found.
Nodes []*merkle.Node
// Incomplete is true when the walk stopped short of a real root,
// whether because a parent_hash could not be resolved (see
// MissingParent) or because a cycle was detected (see CycleDetected).
Incomplete bool
// MissingParent is the parent_hash that could not be resolved in this
// store. Only set when Incomplete is true due to a dangling pointer;
// left empty for cycle-detected incompletes (the cycle-triggering
// hash is present in Nodes already).
MissingParent string
// CycleDetected is true when the walk stopped because it was about
// to re-visit a hash already in Nodes. Drivers guard every walk with
// a per-chain seen-set so a corrupt parent edge can never spin
// forever; when this flag trips, the chain is the largest
// acyclic prefix reachable from the starting hash.
//
// In practice this never trips on a healthy store. It exists because
// a single corrupted parent_hash would otherwise hang any endpoint
// that walks ancestry, which is a blast radius out of proportion to
// the likelihood.
CycleDetected bool
}
Chain is the result of walking a node's parent edges back toward a root.
A Chain whose Incomplete field is false reached a legitimate root (a node whose parent_hash is nil or empty). A Chain whose Incomplete field is true stopped because a parent_hash pointed at a node that is not currently present in this store — see MissingParent. The nodes in Nodes are still valid; the store simply can't resolve any higher ancestors from here.
This state is expected on large or long-lived stores that trim/offload older data, merge content from foreign sources, or receive chains whose ancestors live in another store altogether. Callers should treat it as informational (a signal to render a marker, or trigger a future "thaw" lookup against another source), not as corruption.
type Cursor ¶ added in v0.4.0
type Cursor struct {
// CreatedAt is the head-node timestamp of the last item on the prior page.
CreatedAt time.Time `json:"t"`
// Hash is the head-node hash of the last item on the prior page.
// Used as a tiebreaker when multiple nodes share a CreatedAt.
Hash string `json:"h"`
}
Cursor is the decoded form of an opaque ListOpts.Cursor token. It is exported for driver implementations; clients should treat the encoded string as opaque.
func DecodeCursor ¶ added in v0.4.0
DecodeCursor parses an opaque cursor token. An empty token returns the zero Cursor without error, meaning "start from the most recent".
type Driver ¶
type Driver interface {
// Get retrieves a node by its hash.
Get(ctx context.Context, hash string) (*merkle.Node, error)
// GetByParent retrieves all nodes that have the given parent hash.
// Pass nil to get root nodes.
GetByParent(ctx context.Context, parentHash *string) ([]*merkle.Node, error)
// Put stores a node. Returns true if the node was newly inserted,
// false if it already exists. If the node already exists, this should be
// a no-op. Put provides automatic deduplication via content-addressing in the dag.
Put(ctx context.Context, node *merkle.Node) (bool, error)
// Has checks if a node exists by its hash.
Has(ctx context.Context, hash string) (bool, error)
// List returns all nodes in the store.
List(ctx context.Context) ([]*merkle.Node, error)
// Roots returns all root nodes (nodes with no parent).
Roots(ctx context.Context) ([]*merkle.Node, error)
// Leaves returns all leaf nodes (nodes with no children).
Leaves(ctx context.Context) ([]*merkle.Node, error)
// ListSessions returns a page of leaf nodes ordered by created_at descending,
// optionally filtered by ListOpts. The returned Page.NextCursor is empty
// when there are no further pages.
//
// "Session" here is the API-layer concept: a leaf node identifies the head
// of a conversation chain. Filters apply to the leaf node itself, not to
// any ancestor in the chain.
ListSessions(ctx context.Context, opts ListOpts) (*Page[*merkle.Node], error)
// CountSessions returns aggregate counts for the slice of data matching
// the filter in opts. Pagination fields on opts (Limit, Cursor) are ignored.
CountSessions(ctx context.Context, opts ListOpts) (SessionStats, error)
// Ancestry returns the path from a node back to its root (node first, root last).
Ancestry(ctx context.Context, hash string) ([]*merkle.Node, error)
// AncestryChain is Ancestry with a marker describing how the walk
// terminated. When the walk stops at a parent_hash whose target is not
// present in this store, the returned Chain has Incomplete=true and
// MissingParent set to that parent_hash. The nodes in Chain.Nodes are
// still valid; this state is expected on stores that trim older data or
// merge content from foreign sources, and is not an error.
AncestryChain(ctx context.Context, hash string) (*Chain, error)
// AncestryChains returns a Chain for each input hash, batched per depth
// level so the cost scales with maximum chain depth rather than the
// product of starting-node count and depth. Shared ancestors across
// starts are fetched once.
//
// The returned map is keyed by each starting hash. Starts that are not
// present in the store are omitted from the map rather than surfaced as
// errors — callers that need a strict "every start must resolve" check
// should compare the map's keys against their input slice.
//
// Use this instead of looping over AncestryChain when walking many
// leaves (e.g. the /v1/sessions/summary handler): the per-leaf loop
// issues O(N_starts × depth) queries, which on a real store with tens
// of thousands of leaves will not complete in any reasonable time.
AncestryChains(ctx context.Context, hashes []string) (map[string]*Chain, error)
// LoadDag takes a node hash and returns the full graph.
LoadDag(ctx context.Context, hash string) (*merkle.Dag, error)
// Open initializes the backing store and makes it ready for use.
Open(ctx context.Context) error
// UpdateUsage updates token / duration usage metadata on an existing node.
UpdateUsage(ctx context.Context, hash string, usage *llm.Usage) error
// Depth returns the depth of a node (0 for roots).
Depth(ctx context.Context, hash string) (int, error)
// Close closes the store and releases any resources.
Close() error
}
Driver defines the interface for persisting and retrieving nodes in a storage backend. The Driver is the primary interface for working with pkg/merkle - it handles storage, retrieval, and traversal of nodes per the storage implementor.
type IngestTurnRequest ¶ added in v0.10.0
type IngestTurnRequest struct {
// Session is the optional session-tracking envelope. nil is a
// legitimate value (legacy clients): implementations fall back
// to a Merkle-derived synthetic harness_session_id.
Session *sessions.IngestEnvelope
// Nodes is the chain of nodes for this turn, ordered root-to-leaf.
// The first element is the conversation root; the last is the
// assistant response. ParentHash linkage between successive
// elements must already be set by NewNode at the call site.
Nodes []*merkle.Node
// InputTokens / OutputTokens / CostUSD are the per-turn deltas
// applied to the sessions counters when at least one node was
// newly inserted. Pre-computed by the worker (which owns the
// pricing lookup) so this layer stays free of llm/pricing
// dependencies.
InputTokens int64
OutputTokens int64
CostUSD float64
}
IngestTurnRequest bundles every input the SessionIngester needs. Built by the worker pool from a single Job + the chain of content-addressed nodes derived from that Job's request/response pair.
type IngestTurnResult ¶ added in v0.10.0
type IngestTurnResult struct {
// SessionID is the resolved/created sessions row id as a 36-char
// canonical UUID string. Stable across retries (idempotent on
// natural key).
SessionID string
// NewNodes lists the nodes that were actually inserted on this
// call (i.e. nodes.hash was not already present). May be shorter
// than the input chain on retries.
NewNodes []*merkle.Node
// CountersUpdated is true when the call bumped sessions counters.
// False for pure-retry calls where every node hash already
// existed (idempotent retries must not double-count).
CountersUpdated bool
}
IngestTurnResult reports what the call actually did. Used by the worker to drive downstream side effects (publisher events, vector embeddings) for newly-inserted nodes only.
type ListOpts ¶ added in v0.4.0
type ListOpts struct {
// Limit is the maximum number of items to return. If zero, DefaultListLimit
// is used. Values larger than MaxListLimit are clamped.
Limit int
// Cursor is an opaque pagination token from a prior Page.NextCursor.
// Empty means start from the most recent.
Cursor string
// Filters. Empty / nil values mean "no filter on this field".
Project string
Agent string
Model string
Provider string
Since *time.Time
Until *time.Time
}
ListOpts controls filtering and cursor pagination for session listings.
All filter fields are AND-combined and apply to the head (leaf) node of each session. Empty string and nil pointer fields are treated as "no filter".
Pagination is keyset-based on (CreatedAt DESC, Hash DESC). Callers should treat Cursor as opaque; use the NextCursor returned in Page.
type ModelTokenStats ¶ added in v0.8.0
type ModelTokenStats struct {
InputTokens int64
OutputTokens int64
CacheCreationTokens int64
CacheReadTokens int64
}
ModelTokenStats is the per-model token rollup returned inside SessionStats. Cost is intentionally not computed here — pricing lives in pkg/sessions and is applied by the API handler.
type NotFoundError ¶
type NotFoundError struct {
Hash string
}
NotFoundError is returned when a node doesn't exist in the store.
func (NotFoundError) Error ¶
func (e NotFoundError) Error() string
type Page ¶ added in v0.4.0
type Page[T any] struct { Items []T // NextCursor is empty when there are no more pages. NextCursor string }
Page is a generic paginated result envelope.
type ParentRef ¶ added in v0.4.0
ParentRef is a lightweight (hash, parent_hash) tuple used by integrity checks and bulk traversal code that only needs the DAG edges, not the full node bucket JSON.
type ParentRefLister ¶ added in v0.4.0
ParentRefLister is an optional capability for drivers that can return every node's parent edge in a single efficient query. Drivers that do not implement it are assumed to be unsuitable for bulk integrity checks over large stores.
type SessionBackfillRequest ¶ added in v0.10.0
type SessionBackfillResult ¶ added in v0.10.0
type SessionBackfiller ¶ added in v0.10.0
type SessionBackfiller interface {
BackfillSession(ctx context.Context, req SessionBackfillRequest) (SessionBackfillResult, error)
}
SessionBackfiller is an optional Driver capability for linking legacy node rows to a session table row after the fact. It is deliberately separate from SessionIngester: backfill does not insert nodes, it only UPSERTs session identity and stamps existing nodes that still have no session_id.
type SessionIdentity ¶ added in v0.10.0
type SessionIdentity struct {
HarnessID string `json:"harness_id,omitempty"`
HarnessSessionID string `json:"harness_session_id,omitempty"`
}
SessionIdentity is the harness-level identity attached to a persisted sessions row. For Claude Code captures, HarnessID is "claude" and HarnessSessionID is Claude's own session id.
type SessionIngester ¶ added in v0.10.0
type SessionIngester interface {
// IngestTurn stores every node in `nodes` (root first, leaf
// last; ParentHash must already chain correctly) inside a single
// Tx that also resolves/UPSERTs the sessions row, optionally
// resolves the parent_session_id FK, stamps session_id onto each
// newly-inserted node, and rolls up the counters.
IngestTurn(ctx context.Context, req IngestTurnRequest) (IngestTurnResult, error)
}
SessionIngester is an optional capability for a Driver: it folds session resolution, node insert, and counter rollup into a single transaction.
Only drivers that can host the sessions table implement this. The Postgres driver does; the in-memory driver intentionally does not (it has no concept of sessions, and tests that exercise the classic Put-only path still want the legacy behavior). Callers like the worker pool MUST type-assert against this interface before invoking it.
Implementations MUST satisfy these invariants:
- All side effects (sessions UPSERT, parent placeholder insert, nodes inserts, counter UPDATE) commit atomically. A panic or error on any step rolls back every other step.
- The resolved sessions row is keyed by (org_id, harness_id, harness_session_id). When the inbound envelope is nil, or HarnessID is "unknown", or HarnessSessionID is empty, implementations derive a synthetic harness_session_id from the captured turn's Merkle root prefix.
- Counters (turn_count, total_input_tokens, total_output_tokens, total_cost_usd) are incremented ONLY when at least one new node was actually inserted in this call. A duplicate envelope (every node hash already present) is a true no-op on counters — this preserves end-to-end idempotency for retried POSTs.
type SessionRecord ¶ added in v0.12.0
type SessionRecord struct {
ID string
HarnessID string
HarnessSessionID string
Name string // empty when not set
Cwd string // empty when not set
HarnessVersion string // empty when not set
ParentSessionID string // empty when not set
StartedAt time.Time
LastSeenAt time.Time
EndedAt *time.Time // nil when session is still live
HarnessMetadata map[string]any
TotalInputTokens int64
TotalOutputTokens int64
TotalCostUsd float64
TurnCount int
// DerivedStatus is the chain-aware session status (completed / failed /
// abandoned / unknown), denormalized at ingest. 'unknown' until the first
// turn lands or, for pre-feature rows, until the status backfill runs.
DerivedStatus string
Preview string // first user turn text, truncated; empty when unavailable
}
SessionRecord is the flat sessions-table row surfaced by GET /v1/sessions. Fields absent in the DB (NULL) are represented as empty strings or nil/zero values so API callers never have to unwrap optional pgtype wrappers.
type SessionStats ¶ added in v0.4.0
type SessionStats struct {
// SessionCount is the number of distinct first-class sessions touched by
// the matching nodes (keyed on nodes.session_id). Nodes with no
// session_id — legacy or non-session-tracked writers, including the
// in-memory driver — do not contribute, so a store with no session rows
// reports 0. See StemCount for the leaf-based metric.
SessionCount int
// StemCount is the number of leaf nodes (Merkle chains) matching the
// filter — one per /v1/stems entry. This is the value SessionCount
// reported before the sessions table existed.
StemCount int
// TurnCount is the number of nodes (turns) matching the filter.
TurnCount int
// RootCount is the number of root nodes (no parent) matching the filter.
RootCount int
// CompletedCount is the number of distinct sessions whose chain-aware
// derived_status is "completed" (computed at ingest via
// pkg/sessions.DetermineStatus, read from the sessions table). Like
// SessionCount it is session-grained: a store with no session rows
// reports 0.
CompletedCount int
// InputTokens / OutputTokens are SUMs over the matching node set,
// taken from prompt_tokens / completion_tokens columns.
InputTokens int64
OutputTokens int64
// CacheCreationTokens / CacheReadTokens are SUMs of the cache-aware
// token columns. Surfaced so a caller (typically the API handler) can
// fold cost via pkg/sessions.CostForTokensWithCache.
CacheCreationTokens int64
CacheReadTokens int64
// TotalDurationNs is the wall-clock span MAX(created_at) − MIN(created_at)
// over the matching node set, in nanoseconds. It is NOT a sum of per-call
// durations: nodes.total_duration_ns is currently never populated by the
// proxy (see PCC-514), so SUMming the column would always return 0.
// Wall-clock span is meaningful for a dashboard "Agent Time" card and is
// the same shape that pkg/sessions.BuildSummary uses per session.
TotalDurationNs int64
// ToolCalls is the number of tool_use content blocks across the
// matching node set.
ToolCalls int
// PerModel breaks tokens down by (normalized) model so the API layer
// can apply per-model pricing without the storage driver having to
// know about pricing tables. Keys are normalized model names; nodes
// with no model are excluded.
PerModel map[string]ModelTokenStats
}
SessionStats is the aggregate result of CountSessions for a given filter.
All numeric aggregates are computed over the set of nodes matching the supplied ListOpts filter; they are not restricted to nodes that are part of a matching leaf session. This mirrors the long-standing TurnCount semantic: the filter is per-node, not per-chain.
type SessionStatusBackfiller ¶ added in v0.13.0
type SessionStatusBackfiller interface {
BackfillSessionStatus(ctx context.Context) (BackfillSessionStatusResult, error)
}
SessionStatusBackfiller is an optional Driver capability that recomputes the denormalized derived_status (and the sticky has_git_activity flag, tool_result_count, and tool_error_count) for sessions whose rows predate the ingest-time status computation. It walks each session's nodes with the same signal helpers ingest uses, so a backfilled store matches what live ingest would have written. Idempotent and safe to run online.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
Package postgres
|
Package postgres |
|
Package storagetest provides shared Ginkgo specs that any storage.Driver implementation can run against.
|
Package storagetest provides shared Ginkgo specs that any storage.Driver implementation can run against. |