storage

package
v0.13.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 9, 2026 License: AGPL-3.0 Imports: 9 Imported by: 0

Documentation

Overview

Package storage

Index

Constants

View Source
const DefaultListLimit = 50

DefaultListLimit is the page size used when ListOpts.Limit is zero.

View Source
const MaxListLimit = 5000

MaxListLimit is the maximum permitted page size. Drivers clamp ListOpts.Limit to this value.

Set high (5000) because the AncestryChains hot path has a large fixed cost per request (the recursive CTE setup) and a tiny incremental cost per leaf — measured at ~260ms regardless of whether limit is 200 or 1000 against the brian_large store. Forcing callers to paginate at small page sizes multiplies the fixed cost. The deck's full Overview load drops from ~17s @ limit=200 (50 round trips) to ~3s @ limit=2000 (5 round trips) just from the math.

Memory impact: each row carries ~200 bytes (the CTE doesn't ship the heavy `content` blob — see the label_hint extraction in pkg/storage/ent/driver/driver.go). 5000 leaves × ~30 avg depth × 200 bytes ≈ 30 MB peak per request, which is fine for an API server backing one deck instance.

Variables

This section is empty.

Functions

This section is empty.

Types

type BackfillSessionStatusResult added in v0.13.0

type BackfillSessionStatusResult struct {
	// Scanned is the number of session rows visited.
	Scanned int
	// Updated is the number whose status row was (re)written.
	Updated int
}

type Chain added in v0.4.0

type Chain struct {
	// Nodes is the walk output in node-first order: index 0 is the node
	// the walk started from, and the last element is either a real root
	// or the last resolvable node whose parent could not be found.
	Nodes []*merkle.Node

	// Incomplete is true when the walk stopped short of a real root,
	// whether because a parent_hash could not be resolved (see
	// MissingParent) or because a cycle was detected (see CycleDetected).
	Incomplete bool

	// MissingParent is the parent_hash that could not be resolved in this
	// store. Only set when Incomplete is true due to a dangling pointer;
	// left empty for cycle-detected incompletes (the cycle-triggering
	// hash is present in Nodes already).
	MissingParent string

	// CycleDetected is true when the walk stopped because it was about
	// to re-visit a hash already in Nodes. Drivers guard every walk with
	// a per-chain seen-set so a corrupt parent edge can never spin
	// forever; when this flag trips, the chain is the largest
	// acyclic prefix reachable from the starting hash.
	//
	// In practice this never trips on a healthy store. It exists because
	// a single corrupted parent_hash would otherwise hang any endpoint
	// that walks ancestry, which is a blast radius out of proportion to
	// the likelihood.
	CycleDetected bool
}

Chain is the result of walking a node's parent edges back toward a root.

A Chain whose Incomplete field is false reached a legitimate root (a node whose parent_hash is nil or empty). A Chain whose Incomplete field is true stopped because a parent_hash pointed at a node that is not currently present in this store — see MissingParent. The nodes in Nodes are still valid; the store simply can't resolve any higher ancestors from here.

This state is expected on large or long-lived stores that trim/offload older data, merge content from foreign sources, or receive chains whose ancestors live in another store altogether. Callers should treat it as informational (a signal to render a marker, or trigger a future "thaw" lookup against another source), not as corruption.

func (*Chain) Complete added in v0.4.0

func (c *Chain) Complete() bool

Complete reports whether the walk reached a real root.

type Cursor added in v0.4.0

type Cursor struct {
	// CreatedAt is the head-node timestamp of the last item on the prior page.
	CreatedAt time.Time `json:"t"`

	// Hash is the head-node hash of the last item on the prior page.
	// Used as a tiebreaker when multiple nodes share a CreatedAt.
	Hash string `json:"h"`
}

Cursor is the decoded form of an opaque ListOpts.Cursor token. It is exported for driver implementations; clients should treat the encoded string as opaque.

func DecodeCursor added in v0.4.0

func DecodeCursor(token string) (Cursor, error)

DecodeCursor parses an opaque cursor token. An empty token returns the zero Cursor without error, meaning "start from the most recent".

func (Cursor) Encode added in v0.4.0

func (c Cursor) Encode() string

Encode returns the opaque base64 representation of the cursor.

type Driver

type Driver interface {
	// Get retrieves a node by its hash.
	Get(ctx context.Context, hash string) (*merkle.Node, error)

	// GetByParent retrieves all nodes that have the given parent hash.
	// Pass nil to get root nodes.
	GetByParent(ctx context.Context, parentHash *string) ([]*merkle.Node, error)

	// Put stores a node. Returns true if the node was newly inserted,
	// false if it already exists. If the node already exists, this should be
	// a no-op. Put provides automatic deduplication via content-addressing in the dag.
	Put(ctx context.Context, node *merkle.Node) (bool, error)

	// Has checks if a node exists by its hash.
	Has(ctx context.Context, hash string) (bool, error)

	// List returns all nodes in the store.
	List(ctx context.Context) ([]*merkle.Node, error)

	// Roots returns all root nodes (nodes with no parent).
	Roots(ctx context.Context) ([]*merkle.Node, error)

	// Leaves returns all leaf nodes (nodes with no children).
	Leaves(ctx context.Context) ([]*merkle.Node, error)

	// ListSessions returns a page of leaf nodes ordered by created_at descending,
	// optionally filtered by ListOpts. The returned Page.NextCursor is empty
	// when there are no further pages.
	//
	// "Session" here is the API-layer concept: a leaf node identifies the head
	// of a conversation chain. Filters apply to the leaf node itself, not to
	// any ancestor in the chain.
	ListSessions(ctx context.Context, opts ListOpts) (*Page[*merkle.Node], error)

	// CountSessions returns aggregate counts for the slice of data matching
	// the filter in opts. Pagination fields on opts (Limit, Cursor) are ignored.
	CountSessions(ctx context.Context, opts ListOpts) (SessionStats, error)

	// Ancestry returns the path from a node back to its root (node first, root last).
	Ancestry(ctx context.Context, hash string) ([]*merkle.Node, error)

	// AncestryChain is Ancestry with a marker describing how the walk
	// terminated. When the walk stops at a parent_hash whose target is not
	// present in this store, the returned Chain has Incomplete=true and
	// MissingParent set to that parent_hash. The nodes in Chain.Nodes are
	// still valid; this state is expected on stores that trim older data or
	// merge content from foreign sources, and is not an error.
	AncestryChain(ctx context.Context, hash string) (*Chain, error)

	// AncestryChains returns a Chain for each input hash, batched per depth
	// level so the cost scales with maximum chain depth rather than the
	// product of starting-node count and depth. Shared ancestors across
	// starts are fetched once.
	//
	// The returned map is keyed by each starting hash. Starts that are not
	// present in the store are omitted from the map rather than surfaced as
	// errors — callers that need a strict "every start must resolve" check
	// should compare the map's keys against their input slice.
	//
	// Use this instead of looping over AncestryChain when walking many
	// leaves (e.g. the /v1/sessions/summary handler): the per-leaf loop
	// issues O(N_starts × depth) queries, which on a real store with tens
	// of thousands of leaves will not complete in any reasonable time.
	AncestryChains(ctx context.Context, hashes []string) (map[string]*Chain, error)

	// LoadDag takes a node hash and returns the full graph.
	LoadDag(ctx context.Context, hash string) (*merkle.Dag, error)

	// Open initializes the backing store and makes it ready for use.
	Open(ctx context.Context) error

	// UpdateUsage updates token / duration usage metadata on an existing node.
	UpdateUsage(ctx context.Context, hash string, usage *llm.Usage) error

	// Depth returns the depth of a node (0 for roots).
	Depth(ctx context.Context, hash string) (int, error)

	// Close closes the store and releases any resources.
	Close() error
}

Driver defines the interface for persisting and retrieving nodes in a storage backend. The Driver is the primary interface for working with pkg/merkle - it handles storage, retrieval, and traversal of nodes per the storage implementor.

type IngestTurnRequest added in v0.10.0

type IngestTurnRequest struct {
	// Session is the optional session-tracking envelope. nil is a
	// legitimate value (legacy clients): implementations fall back
	// to a Merkle-derived synthetic harness_session_id.
	Session *sessions.IngestEnvelope

	// Nodes is the chain of nodes for this turn, ordered root-to-leaf.
	// The first element is the conversation root; the last is the
	// assistant response. ParentHash linkage between successive
	// elements must already be set by NewNode at the call site.
	Nodes []*merkle.Node

	// InputTokens / OutputTokens / CostUSD are the per-turn deltas
	// applied to the sessions counters when at least one node was
	// newly inserted. Pre-computed by the worker (which owns the
	// pricing lookup) so this layer stays free of llm/pricing
	// dependencies.
	InputTokens  int64
	OutputTokens int64
	CostUSD      float64
}

IngestTurnRequest bundles every input the SessionIngester needs. Built by the worker pool from a single Job + the chain of content-addressed nodes derived from that Job's request/response pair.

type IngestTurnResult added in v0.10.0

type IngestTurnResult struct {
	// SessionID is the resolved/created sessions row id as a 36-char
	// canonical UUID string. Stable across retries (idempotent on
	// natural key).
	SessionID string

	// NewNodes lists the nodes that were actually inserted on this
	// call (i.e. nodes.hash was not already present). May be shorter
	// than the input chain on retries.
	NewNodes []*merkle.Node

	// CountersUpdated is true when the call bumped sessions counters.
	// False for pure-retry calls where every node hash already
	// existed (idempotent retries must not double-count).
	CountersUpdated bool
}

IngestTurnResult reports what the call actually did. Used by the worker to drive downstream side effects (publisher events, vector embeddings) for newly-inserted nodes only.

type ListOpts added in v0.4.0

type ListOpts struct {
	// Limit is the maximum number of items to return. If zero, DefaultListLimit
	// is used. Values larger than MaxListLimit are clamped.
	Limit int

	// Cursor is an opaque pagination token from a prior Page.NextCursor.
	// Empty means start from the most recent.
	Cursor string

	// Filters. Empty / nil values mean "no filter on this field".
	Project  string
	Agent    string
	Model    string
	Provider string
	Since    *time.Time
	Until    *time.Time
}

ListOpts controls filtering and cursor pagination for session listings.

All filter fields are AND-combined and apply to the head (leaf) node of each session. Empty string and nil pointer fields are treated as "no filter".

Pagination is keyset-based on (CreatedAt DESC, Hash DESC). Callers should treat Cursor as opaque; use the NextCursor returned in Page.

func (ListOpts) Normalize added in v0.4.0

func (o ListOpts) Normalize() ListOpts

Normalize returns a copy of opts with Limit clamped to [1, MaxListLimit]. A zero Limit is replaced with DefaultListLimit.

type ModelTokenStats added in v0.8.0

type ModelTokenStats struct {
	InputTokens         int64
	OutputTokens        int64
	CacheCreationTokens int64
	CacheReadTokens     int64
}

ModelTokenStats is the per-model token rollup returned inside SessionStats. Cost is intentionally not computed here — pricing lives in pkg/sessions and is applied by the API handler.

type NotFoundError

type NotFoundError struct {
	Hash string
}

NotFoundError is returned when a node doesn't exist in the store.

func (NotFoundError) Error

func (e NotFoundError) Error() string

type Page added in v0.4.0

type Page[T any] struct {
	Items []T

	// NextCursor is empty when there are no more pages.
	NextCursor string
}

Page is a generic paginated result envelope.

type ParentRef added in v0.4.0

type ParentRef struct {
	Hash       string
	ParentHash *string
}

ParentRef is a lightweight (hash, parent_hash) tuple used by integrity checks and bulk traversal code that only needs the DAG edges, not the full node bucket JSON.

type ParentRefLister added in v0.4.0

type ParentRefLister interface {
	ListParentRefs(ctx context.Context) ([]ParentRef, error)
}

ParentRefLister is an optional capability for drivers that can return every node's parent edge in a single efficient query. Drivers that do not implement it are assumed to be unsuitable for bulk integrity checks over large stores.

type SessionBackfillRequest added in v0.10.0

type SessionBackfillRequest struct {
	Session      *sessions.IngestEnvelope
	NodeHashes   []string
	StartedAt    time.Time
	LastSeenAt   time.Time
	InputTokens  int64
	OutputTokens int64
	TurnCount    int64
}

type SessionBackfillResult added in v0.10.0

type SessionBackfillResult struct {
	SessionID   string
	NodesLinked int
}

type SessionBackfiller added in v0.10.0

type SessionBackfiller interface {
	BackfillSession(ctx context.Context, req SessionBackfillRequest) (SessionBackfillResult, error)
}

SessionBackfiller is an optional Driver capability for linking legacy node rows to a session table row after the fact. It is deliberately separate from SessionIngester: backfill does not insert nodes, it only UPSERTs session identity and stamps existing nodes that still have no session_id.

type SessionIdentity added in v0.10.0

type SessionIdentity struct {
	HarnessID        string `json:"harness_id,omitempty"`
	HarnessSessionID string `json:"harness_session_id,omitempty"`
}

SessionIdentity is the harness-level identity attached to a persisted sessions row. For Claude Code captures, HarnessID is "claude" and HarnessSessionID is Claude's own session id.

type SessionIngester added in v0.10.0

type SessionIngester interface {
	// IngestTurn stores every node in `nodes` (root first, leaf
	// last; ParentHash must already chain correctly) inside a single
	// Tx that also resolves/UPSERTs the sessions row, optionally
	// resolves the parent_session_id FK, stamps session_id onto each
	// newly-inserted node, and rolls up the counters.
	IngestTurn(ctx context.Context, req IngestTurnRequest) (IngestTurnResult, error)
}

SessionIngester is an optional capability for a Driver: it folds session resolution, node insert, and counter rollup into a single transaction.

Only drivers that can host the sessions table implement this. The Postgres driver does; the in-memory driver intentionally does not (it has no concept of sessions, and tests that exercise the classic Put-only path still want the legacy behavior). Callers like the worker pool MUST type-assert against this interface before invoking it.

Implementations MUST satisfy these invariants:

  • All side effects (sessions UPSERT, parent placeholder insert, nodes inserts, counter UPDATE) commit atomically. A panic or error on any step rolls back every other step.
  • The resolved sessions row is keyed by (org_id, harness_id, harness_session_id). When the inbound envelope is nil, or HarnessID is "unknown", or HarnessSessionID is empty, implementations derive a synthetic harness_session_id from the captured turn's Merkle root prefix.
  • Counters (turn_count, total_input_tokens, total_output_tokens, total_cost_usd) are incremented ONLY when at least one new node was actually inserted in this call. A duplicate envelope (every node hash already present) is a true no-op on counters — this preserves end-to-end idempotency for retried POSTs.

type SessionRecord added in v0.12.0

type SessionRecord struct {
	ID                string
	HarnessID         string
	HarnessSessionID  string
	Name              string // empty when not set
	Cwd               string // empty when not set
	HarnessVersion    string // empty when not set
	ParentSessionID   string // empty when not set
	StartedAt         time.Time
	LastSeenAt        time.Time
	EndedAt           *time.Time // nil when session is still live
	HarnessMetadata   map[string]any
	TotalInputTokens  int64
	TotalOutputTokens int64
	TotalCostUsd      float64
	TurnCount         int
	// DerivedStatus is the chain-aware session status (completed / failed /
	// abandoned / unknown), denormalized at ingest. 'unknown' until the first
	// turn lands or, for pre-feature rows, until the status backfill runs.
	DerivedStatus string
	Preview       string // first user turn text, truncated; empty when unavailable
}

SessionRecord is the flat sessions-table row surfaced by GET /v1/sessions. Fields absent in the DB (NULL) are represented as empty strings or nil/zero values so API callers never have to unwrap optional pgtype wrappers.

type SessionStats added in v0.4.0

type SessionStats struct {
	// SessionCount is the number of distinct first-class sessions touched by
	// the matching nodes (keyed on nodes.session_id). Nodes with no
	// session_id — legacy or non-session-tracked writers, including the
	// in-memory driver — do not contribute, so a store with no session rows
	// reports 0. See StemCount for the leaf-based metric.
	SessionCount int

	// StemCount is the number of leaf nodes (Merkle chains) matching the
	// filter — one per /v1/stems entry. This is the value SessionCount
	// reported before the sessions table existed.
	StemCount int

	// TurnCount is the number of nodes (turns) matching the filter.
	TurnCount int

	// RootCount is the number of root nodes (no parent) matching the filter.
	RootCount int

	// CompletedCount is the number of distinct sessions whose chain-aware
	// derived_status is "completed" (computed at ingest via
	// pkg/sessions.DetermineStatus, read from the sessions table). Like
	// SessionCount it is session-grained: a store with no session rows
	// reports 0.
	CompletedCount int

	// InputTokens / OutputTokens are SUMs over the matching node set,
	// taken from prompt_tokens / completion_tokens columns.
	InputTokens  int64
	OutputTokens int64

	// CacheCreationTokens / CacheReadTokens are SUMs of the cache-aware
	// token columns. Surfaced so a caller (typically the API handler) can
	// fold cost via pkg/sessions.CostForTokensWithCache.
	CacheCreationTokens int64
	CacheReadTokens     int64

	// TotalDurationNs is the wall-clock span MAX(created_at) − MIN(created_at)
	// over the matching node set, in nanoseconds. It is NOT a sum of per-call
	// durations: nodes.total_duration_ns is currently never populated by the
	// proxy (see PCC-514), so SUMming the column would always return 0.
	// Wall-clock span is meaningful for a dashboard "Agent Time" card and is
	// the same shape that pkg/sessions.BuildSummary uses per session.
	TotalDurationNs int64

	// ToolCalls is the number of tool_use content blocks across the
	// matching node set.
	ToolCalls int

	// PerModel breaks tokens down by (normalized) model so the API layer
	// can apply per-model pricing without the storage driver having to
	// know about pricing tables. Keys are normalized model names; nodes
	// with no model are excluded.
	PerModel map[string]ModelTokenStats
}

SessionStats is the aggregate result of CountSessions for a given filter.

All numeric aggregates are computed over the set of nodes matching the supplied ListOpts filter; they are not restricted to nodes that are part of a matching leaf session. This mirrors the long-standing TurnCount semantic: the filter is per-node, not per-chain.

type SessionStatusBackfiller added in v0.13.0

type SessionStatusBackfiller interface {
	BackfillSessionStatus(ctx context.Context) (BackfillSessionStatusResult, error)
}

SessionStatusBackfiller is an optional Driver capability that recomputes the denormalized derived_status (and the sticky has_git_activity flag, tool_result_count, and tool_error_count) for sessions whose rows predate the ingest-time status computation. It walks each session's nodes with the same signal helpers ingest uses, so a backfilled store matches what live ingest would have written. Idempotent and safe to run online.

Directories

Path Synopsis
Package postgres
Package postgres
Package storagetest provides shared Ginkgo specs that any storage.Driver implementation can run against.
Package storagetest provides shared Ginkgo specs that any storage.Driver implementation can run against.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL