storage

package
v0.9.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: May 19, 2026 License: AGPL-3.0 Imports: 8 Imported by: 0

Documentation

Overview

Package storage

Index

Constants

View Source
const DefaultListLimit = 50

DefaultListLimit is the page size used when ListOpts.Limit is zero.

View Source
const MaxListLimit = 5000

MaxListLimit is the maximum permitted page size. Drivers clamp ListOpts.Limit to this value.

Set high (5000) because the AncestryChains hot path has a large fixed cost per request (the recursive CTE setup) and a tiny incremental cost per leaf — measured at ~260ms regardless of whether limit is 200 or 1000 against the brian_large store. Forcing callers to paginate at small page sizes multiplies the fixed cost. The deck's full Overview load drops from ~17s @ limit=200 (50 round trips) to ~3s @ limit=2000 (5 round trips) just from the math.

Memory impact: each row carries ~200 bytes (the CTE doesn't ship the heavy `content` blob — see the label_hint extraction in pkg/storage/ent/driver/driver.go). 5000 leaves × ~30 avg depth × 200 bytes ≈ 30 MB peak per request, which is fine for an API server backing one deck instance.

Variables

This section is empty.

Functions

This section is empty.

Types

type Chain added in v0.4.0

type Chain struct {
	// Nodes is the walk output in node-first order: index 0 is the node
	// the walk started from, and the last element is either a real root
	// or the last resolvable node whose parent could not be found.
	Nodes []*merkle.Node

	// Incomplete is true when the walk stopped short of a real root,
	// whether because a parent_hash could not be resolved (see
	// MissingParent) or because a cycle was detected (see CycleDetected).
	Incomplete bool

	// MissingParent is the parent_hash that could not be resolved in this
	// store. Only set when Incomplete is true due to a dangling pointer;
	// left empty for cycle-detected incompletes (the cycle-triggering
	// hash is present in Nodes already).
	MissingParent string

	// CycleDetected is true when the walk stopped because it was about
	// to re-visit a hash already in Nodes. Drivers guard every walk with
	// a per-chain seen-set so a corrupt parent edge can never spin
	// forever; when this flag trips, the chain is the largest
	// acyclic prefix reachable from the starting hash.
	//
	// In practice this never trips on a healthy store. It exists because
	// a single corrupted parent_hash would otherwise hang any endpoint
	// that walks ancestry, which is a blast radius out of proportion to
	// the likelihood.
	CycleDetected bool
}

Chain is the result of walking a node's parent edges back toward a root.

A Chain whose Incomplete field is false reached a legitimate root (a node whose parent_hash is nil or empty). A Chain whose Incomplete field is true stopped because a parent_hash pointed at a node that is not currently present in this store — see MissingParent. The nodes in Nodes are still valid; the store simply can't resolve any higher ancestors from here.

This state is expected on large or long-lived stores that trim/offload older data, merge content from foreign sources, or receive chains whose ancestors live in another store altogether. Callers should treat it as informational (a signal to render a marker, or trigger a future "thaw" lookup against another source), not as corruption.

func (*Chain) Complete added in v0.4.0

func (c *Chain) Complete() bool

Complete reports whether the walk reached a real root.

type Cursor added in v0.4.0

type Cursor struct {
	// CreatedAt is the head-node timestamp of the last item on the prior page.
	CreatedAt time.Time `json:"t"`

	// Hash is the head-node hash of the last item on the prior page.
	// Used as a tiebreaker when multiple nodes share a CreatedAt.
	Hash string `json:"h"`
}

Cursor is the decoded form of an opaque ListOpts.Cursor token. It is exported for driver implementations; clients should treat the encoded string as opaque.

func DecodeCursor added in v0.4.0

func DecodeCursor(token string) (Cursor, error)

DecodeCursor parses an opaque cursor token. An empty token returns the zero Cursor without error, meaning "start from the most recent".

func (Cursor) Encode added in v0.4.0

func (c Cursor) Encode() string

Encode returns the opaque base64 representation of the cursor.

type Driver

type Driver interface {
	// Get retrieves a node by its hash.
	Get(ctx context.Context, hash string) (*merkle.Node, error)

	// GetByParent retrieves all nodes that have the given parent hash.
	// Pass nil to get root nodes.
	GetByParent(ctx context.Context, parentHash *string) ([]*merkle.Node, error)

	// Put stores a node. Returns true if the node was newly inserted,
	// false if it already exists. If the node already exists, this should be
	// a no-op. Put provides automatic deduplication via content-addressing in the dag.
	Put(ctx context.Context, node *merkle.Node) (bool, error)

	// Has checks if a node exists by its hash.
	Has(ctx context.Context, hash string) (bool, error)

	// List returns all nodes in the store.
	List(ctx context.Context) ([]*merkle.Node, error)

	// Roots returns all root nodes (nodes with no parent).
	Roots(ctx context.Context) ([]*merkle.Node, error)

	// Leaves returns all leaf nodes (nodes with no children).
	Leaves(ctx context.Context) ([]*merkle.Node, error)

	// ListSessions returns a page of leaf nodes ordered by created_at descending,
	// optionally filtered by ListOpts. The returned Page.NextCursor is empty
	// when there are no further pages.
	//
	// "Session" here is the API-layer concept: a leaf node identifies the head
	// of a conversation chain. Filters apply to the leaf node itself, not to
	// any ancestor in the chain.
	ListSessions(ctx context.Context, opts ListOpts) (*Page[*merkle.Node], error)

	// CountSessions returns aggregate counts for the slice of data matching
	// the filter in opts. Pagination fields on opts (Limit, Cursor) are ignored.
	CountSessions(ctx context.Context, opts ListOpts) (SessionStats, error)

	// Ancestry returns the path from a node back to its root (node first, root last).
	Ancestry(ctx context.Context, hash string) ([]*merkle.Node, error)

	// AncestryChain is Ancestry with a marker describing how the walk
	// terminated. When the walk stops at a parent_hash whose target is not
	// present in this store, the returned Chain has Incomplete=true and
	// MissingParent set to that parent_hash. The nodes in Chain.Nodes are
	// still valid; this state is expected on stores that trim older data or
	// merge content from foreign sources, and is not an error.
	AncestryChain(ctx context.Context, hash string) (*Chain, error)

	// AncestryChains returns a Chain for each input hash, batched per depth
	// level so the cost scales with maximum chain depth rather than the
	// product of starting-node count and depth. Shared ancestors across
	// starts are fetched once.
	//
	// The returned map is keyed by each starting hash. Starts that are not
	// present in the store are omitted from the map rather than surfaced as
	// errors — callers that need a strict "every start must resolve" check
	// should compare the map's keys against their input slice.
	//
	// Use this instead of looping over AncestryChain when walking many
	// leaves (e.g. the /v1/sessions/summary handler): the per-leaf loop
	// issues O(N_starts × depth) queries, which on a real store with tens
	// of thousands of leaves will not complete in any reasonable time.
	AncestryChains(ctx context.Context, hashes []string) (map[string]*Chain, error)

	// LoadDag takes a node hash and returns the full graph.
	LoadDag(ctx context.Context, hash string) (*merkle.Dag, error)

	// Open initializes the backing store and makes it ready for use.
	Open(ctx context.Context) error

	// UpdateUsage updates token / duration usage metadata on an existing node.
	UpdateUsage(ctx context.Context, hash string, usage *llm.Usage) error

	// Depth returns the depth of a node (0 for roots).
	Depth(ctx context.Context, hash string) (int, error)

	// Close closes the store and releases any resources.
	Close() error
}

Driver defines the interface for persisting and retrieving nodes in a storage backend. The Driver is the primary interface for working with pkg/merkle - it handles storage, retrieval, and traversal of nodes per the storage implementor.

type ListOpts added in v0.4.0

type ListOpts struct {
	// Limit is the maximum number of items to return. If zero, DefaultListLimit
	// is used. Values larger than MaxListLimit are clamped.
	Limit int

	// Cursor is an opaque pagination token from a prior Page.NextCursor.
	// Empty means start from the most recent.
	Cursor string

	// Filters. Empty / nil values mean "no filter on this field".
	Project  string
	Agent    string
	Model    string
	Provider string
	Since    *time.Time
	Until    *time.Time
}

ListOpts controls filtering and cursor pagination for session listings.

All filter fields are AND-combined and apply to the head (leaf) node of each session. Empty string and nil pointer fields are treated as "no filter".

Pagination is keyset-based on (CreatedAt DESC, Hash DESC). Callers should treat Cursor as opaque; use the NextCursor returned in Page.

func (ListOpts) Normalize added in v0.4.0

func (o ListOpts) Normalize() ListOpts

Normalize returns a copy of opts with Limit clamped to [1, MaxListLimit]. A zero Limit is replaced with DefaultListLimit.

type ModelTokenStats added in v0.8.0

type ModelTokenStats struct {
	InputTokens         int64
	OutputTokens        int64
	CacheCreationTokens int64
	CacheReadTokens     int64
}

ModelTokenStats is the per-model token rollup returned inside SessionStats. Cost is intentionally not computed here — pricing lives in pkg/sessions and is applied by the API handler.

type NotFoundError

type NotFoundError struct {
	Hash string
}

NotFoundError is returned when a node doesn't exist in the store.

func (NotFoundError) Error

func (e NotFoundError) Error() string

type Page added in v0.4.0

type Page[T any] struct {
	Items []T

	// NextCursor is empty when there are no more pages.
	NextCursor string
}

Page is a generic paginated result envelope.

type ParentRef added in v0.4.0

type ParentRef struct {
	Hash       string
	ParentHash *string
}

ParentRef is a lightweight (hash, parent_hash) tuple used by integrity checks and bulk traversal code that only needs the DAG edges, not the full node bucket JSON.

type ParentRefLister added in v0.4.0

type ParentRefLister interface {
	ListParentRefs(ctx context.Context) ([]ParentRef, error)
}

ParentRefLister is an optional capability for drivers that can return every node's parent edge in a single efficient query. Drivers that do not implement it are assumed to be unsuitable for bulk integrity checks over large stores.

type SessionStats added in v0.4.0

type SessionStats struct {
	// SessionCount is the number of leaf nodes matching the filter.
	SessionCount int

	// TurnCount is the number of nodes (turns) matching the filter.
	TurnCount int

	// RootCount is the number of root nodes (no parent) matching the filter.
	RootCount int

	// CompletedCount is the number of leaf nodes whose terminal classification
	// is "completed" using leaf-only inputs (assistant role + a terminal
	// stop_reason). This intentionally diverges from
	// pkg/sessions.DetermineStatus, which also walks the chain to detect
	// tool errors and git activity. The leaf-only form is fast and
	// satisfiable in a single SQL aggregate.
	CompletedCount int

	// InputTokens / OutputTokens are SUMs over the matching node set,
	// taken from prompt_tokens / completion_tokens columns.
	InputTokens  int64
	OutputTokens int64

	// CacheCreationTokens / CacheReadTokens are SUMs of the cache-aware
	// token columns. Surfaced so a caller (typically the API handler) can
	// fold cost via pkg/sessions.CostForTokensWithCache.
	CacheCreationTokens int64
	CacheReadTokens     int64

	// TotalDurationNs is the wall-clock span MAX(created_at) − MIN(created_at)
	// over the matching node set, in nanoseconds. It is NOT a sum of per-call
	// durations: nodes.total_duration_ns is currently never populated by the
	// proxy (see PCC-514), so SUMming the column would always return 0.
	// Wall-clock span is meaningful for a dashboard "Agent Time" card and is
	// the same shape that pkg/sessions.BuildSummary uses per session.
	TotalDurationNs int64

	// ToolCalls is the number of tool_use content blocks across the
	// matching node set.
	ToolCalls int

	// PerModel breaks tokens down by (normalized) model so the API layer
	// can apply per-model pricing without the storage driver having to
	// know about pricing tables. Keys are normalized model names; nodes
	// with no model are excluded.
	PerModel map[string]ModelTokenStats
}

SessionStats is the aggregate result of CountSessions for a given filter.

All numeric aggregates are computed over the set of nodes matching the supplied ListOpts filter; they are not restricted to nodes that are part of a matching leaf session. This mirrors the long-standing TurnCount semantic: the filter is per-node, not per-chain.

Directories

Path Synopsis
Package postgres
Package postgres
Package storagetest provides shared Ginkgo specs that any storage.Driver implementation can run against.
Package storagetest provides shared Ginkgo specs that any storage.Driver implementation can run against.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL