derive

package
v0.24.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 14, 2026 License: AGPL-3.0 Imports: 21 Imported by: 0

Documentation

Overview

Package derive turns immutable raw captures into the derived layer: content-addressed node chains today; semantic typing (node_kind), fork edges, and offshoot classification as the reconciled conversation tree lands (design/agent-session-reconciliation.md).

Everything in this package is a pure, re-runnable function of the raw capture. The ingest worker uses the same TurnChain at capture time that the re-deriver uses offline, so "re-deriving nodes from raw_turns reproduces the captured nodes" holds by construction.

Index

Constants

View Source
const (
	// KindMain is the conversation spine: full tool set, streaming,
	// high max_tokens.
	KindMain = "main"

	// KindUnknown marks calls that match no cataloged tell. A non-zero
	// unknown count is either a genuinely new harness side-call to
	// catalog or a classifier regression — surfaced prominently, never
	// silently bucketed.
	KindUnknown = "unknown"

	// Shadow model calls — separate API requests that never appear in
	// the harness transcript.
	KindCheckStage1 = "offshoot:permission-check:stage1"
	KindCheckStage2 = "offshoot:permission-check:stage2"
	KindTitleGen    = "offshoot:title-gen"
	KindPlanNameGen = "offshoot:plan-name-gen"
	KindSuggestion  = "offshoot:suggestion"
	KindWebSummary  = "offshoot:web-summary"
	KindProbe       = "offshoot:probe"
	KindCompaction  = "offshoot:compaction"

	// Injected context — whole messages the harness prepends inside
	// otherwise-normal calls. They drift between turns (server lists
	// change, modes toggle), so they are kept OFF the main chain as
	// side-branch nodes (see TurnChain) and marked with these kinds.
	KindInjectedMCPInstructions = "injected:mcp-instructions"
	KindInjectedSkillsList      = "injected:skills-list"
	KindInjectedModeBanner      = "injected:mode-banner"
	// KindInjectedClaudeMD is the user-context blob the harness
	// prepends to its security-monitor checks (<user_claude_md>…).
	// Every check in a session shares it byte-for-byte, so left on the
	// chain it fuses all checks into one fan rooted at the blob.
	KindInjectedClaudeMD = "injected:claude-md"

	// KindInjectedSystemInsert marks mid-spine system-role inserts
	// (task reminders, CLAUDE.md re-injections, post-compaction
	// replays). Currently minted only by the span emit stage — the
	// node classifier still types these "main" because they ride
	// inside main calls; a dedicated classifier kind is a candidate
	// follow-up.
	KindInjectedSystemInsert = "injected:system-insert"
)

Node kinds — the design doc's §2g taxonomy. A "session" on the wire is many API calls of different kinds: the conversation spine plus shadow calls the harness fires on the user's behalf, plus injected context blocks. The set is OPEN: new kinds get cataloged here and a re-derive reclassifies all existing raw data.

View Source
const (
	SpanKindAgent = "agent"
	SpanKindLLM   = "llm"
	SpanKindTool  = "tool"
	SpanKindEvent = "event"
)

Span kinds — the RFD 00007 vocabulary (step is reserved, unused).

View Source
const (
	LinkEmits          = "emits"           // llm output -> tool input
	LinkFeeds          = "feeds"           // tool output -> llm input
	LinkRejoin         = "rejoin"          // subagent agent -> Task tool
	LinkVerdict        = "verdict"         // shadow llm -> judged tool
	LinkCompactionSeam = "compaction-seam" // compaction llm -> next trace's first llm
)

Span link kinds — dataflow edges between spans. Containment is the parent_span_id tree; links carry causality that containment cannot.

Variables

This section is empty.

Functions

func CapturedAt

func CapturedAt(rec *storage.RawTurnRecord) time.Time

CapturedAt resolves a raw record's original capture time: the adapter's request timestamp when the meta block carries one, otherwise the ingest receive time.

func ClassifyCall

func ClassifyCall(req *llm.ChatRequest, resp *llm.ChatResponse) string

ClassifyCall determines the kind of a captured API call from its parsed request and reduced response. Request-envelope parameters (system prompt, max_tokens, tool count, stream) are the definitive discriminators; content shape is the fallback. Tells are grounded in observed traffic — see the design doc §2g and the golden-session measurements.

func ClassifyInjected

func ClassifyInjected(msg llm.Message) string

ClassifyInjected reports the injected-context kind for one request message, or "" when the message is ordinary conversation. Only whole messages that ARE an injected block qualify — a user turn that merely mentions skills is untouched. These messages drift between turns of the same conversation (an MCP server connects, a mode toggles), so TurnChain keeps them off the hashed spine.

func IsTranscriptMeta

func IsTranscriptMeta(meta json.RawMessage) bool

IsTranscriptMeta reports whether a raw row's meta marks it as a transcript file.

func LoadCorpus

func LoadCorpus(r io.Reader) (wire, transcripts []storage.RawTurnRecord, err error)

LoadCorpus reads a gzipped-JSONL corpus dump from r and returns its rows split by source: wire captures and transcript pushes. The io.Reader form serves embedded corpora (the demo seed bundles corpus files into the binary); LoadCorpusFile wraps it for on-disk dumps.

func LoadCorpusFile

func LoadCorpusFile(path string) (wire, transcripts []storage.RawTurnRecord, err error)

LoadCorpusFile reads a gzipped-JSONL corpus dump and returns its rows split by source: wire captures and transcript pushes.

func SessionTitle

func SessionTitle(kind string, resp *llm.ChatResponse) string

SessionTitle extracts the generated title from a title-gen call's response ({"title": "…"}), or "" when the call isn't a title-gen or the response doesn't parse. Tolerates prose around the JSON object (models occasionally wrap it).

func TurnChain

func TurnChain(call CallContext, req *llm.ChatRequest, resp *llm.ChatResponse) []*merkle.Node

Types

type CallContext

type CallContext struct {
	Provider  string
	AgentName string
	ThreadID  string
	Project   string
}

TurnChain materializes the ordered (root → leaf) chain of nodes for a single captured call: one node per request message, followed by the assistant response node. ParentHash linkage is set via merkle.NewNode so every node's hash is stable before any I/O.

Every node is stamped with the call's classified kind (ClassifyCall) and the request-envelope parameters — both as non-hashed metadata. On insert, nodes already present from an earlier call keep that call's stamp (ON CONFLICT DO NOTHING), so each stored node carries the classification of the call that first captured it.

Injected-context messages (MCP instructions, skills lists, mode banners — see ClassifyInjected) become SIDE-BRANCH nodes: they hang off the spine at the position they appeared but the next message's parent bypasses them. These blocks drift between turns of the same conversation, so chaining them would fork the spine at every drift; as side branches they are preserved, marked injected:*, and inert. CallContext carries the capture-side identity of one API call into chain construction: who routed it (provider/agent), which harness sub-thread fired it, and the project tag. All non-hashed metadata.

type DerivedNode

type DerivedNode struct {
	Node       *merkle.Node
	Session    SessionKey
	CapturedAt time.Time
}

DerivedNode is one node of the rebuilt derived layer plus its attribution and provenance.

type DerivedSet

type DerivedSet struct {
	// Nodes in capture order (chain order within a turn). Deduplicated
	// by hash — the first capturing call wins, mirroring ingest's
	// ON CONFLICT DO NOTHING semantics.
	Nodes []*DerivedNode

	// Sessions are the harness keys covered by the raw layer. The
	// store prunes stale derived rows only within these sessions.
	Sessions []SessionKey

	// SessionTitles carries the folded title-gen output per session
	// (latest call wins — the harness regenerates titles as the
	// session evolves).
	SessionTitles map[SessionKey]string

	// SpanSources retains one slim record per parsed raw turn so the
	// span emit stage (EmitSpans) can re-walk the capture in call
	// order after the attach/reconcile passes have stamped fork and
	// offshoot anchors. Holds only pointers to retained nodes plus
	// scalar call identity — never payload copies.
	SpanSources []*SpanSource

	Report RederiveReport
}

DerivedSet is the deriver's output for one org: the complete node set re-derived from the raw layer, ready to upsert.

func BuildDerivedSet

func BuildDerivedSet(rawTurns []storage.RawTurnRecord, project string) (*DerivedSet, error)

BuildDerivedSet derives a complete node set from an in-memory slice of raw turns, in the order given. Convenience wrapper around the streaming Deriver for tests and small batches; callers with a real store should stream records in capture order instead.

type Deriver

type Deriver struct {
	// contains filtered or unexported fields
}

Deriver streams raw turns (in capture order) into a deduplicated derived node set. Memory stays proportional to the UNIQUE content in the raw layer, not to the sum of every turn's re-sent history.

func NewDeriver

func NewDeriver(project string) (*Deriver, error)

NewDeriver creates a streaming deriver. Feed turns with AddTurn in chronological order, then call Finish exactly once.

func (*Deriver) AddTurn

func (dv *Deriver) AddTurn(rec *storage.RawTurnRecord)

AddTurn parses, classifies, and chains one raw turn, folding its nodes into the deduplicated set. The record is not retained.

func (*Deriver) Finish

func (dv *Deriver) Finish() *DerivedSet

Finish runs the cross-call attach passes and returns the completed set. The deriver must not be reused afterwards.

type ModelUsage

type ModelUsage struct {
	Model        string  `json:"model"`
	Calls        int64   `json:"calls"`
	InputTokens  int64   `json:"input_tokens"`
	OutputTokens int64   `json:"output_tokens"`
	CostUSD      float64 `json:"cost_usd"`
}

ModelUsage is one model's contribution to a session: how many llm calls ran on it and what they spent. Cost is priced at derive time (like the trace fold), so a re-derive reprices history.

type ReconcileStats

type ReconcileStats struct {
	TranscriptFiles  int `json:"transcript_files"`
	SubagentForks    int `json:"subagent_forks"`
	ForkedChains     int `json:"forked_chains"`
	MainChainsJoined int `json:"main_chains_joined"`

	// ConversationJoined / ConversationTotal measure how many
	// conversation-spine nodes' content appears in a transcript — the
	// Go-native version of the prototype's join-rate oracle.
	ConversationJoined int `json:"conversation_joined"`
	ConversationTotal  int `json:"conversation_total"`
}

ReconcileStats reports the transcript↔wire fusion for one org.

func ReconcileTranscripts

func ReconcileTranscripts(set *DerivedSet, files []*TranscriptFile) *ReconcileStats

ReconcileTranscripts assigns each wire-derived conversation chain to the transcript file whose content it matches, stamping subagent chain roots with their fork edge. Operates on the in-memory derived set after the wire pass and before the store write — pure and re-runnable like everything else in the deriver.

type RederiveReport

type RederiveReport struct {
	RawTurns      int            `json:"raw_turns"`
	ParsedTurns   int            `json:"parsed_turns"`
	RawOnlyTurns  int            `json:"raw_only_turns"`
	ParseFailures []string       `json:"parse_failures,omitempty"`
	Nodes         int            `json:"nodes"`
	CallKinds     map[string]int `json:"call_kinds"`
	NodeKinds     map[string]int `json:"node_kinds"`

	// Verdict attach: judged actions grouped across stages, and how
	// many attached one-to-one to a captured tool_use.
	JudgedActions    int `json:"judged_actions"`
	AttachedVerdicts int `json:"attached_verdicts"`

	// UnattachedActions samples judged actions that found no matching
	// tool_use (capped) — expected for non-tool events like subagent
	// handbacks; anything else is matcher signal worth reading.
	UnattachedActions []string `json:"unattached_actions,omitempty"`

	// WebSummaryAttached counts web-summary calls linked back to their
	// WebFetch/WebSearch tool_use.
	WebSummaryAttached int `json:"web_summary_attached"`

	// PlansAttached counts plan-name-gen calls linked to the
	// ExitPlanMode tool_use that accepted the plan.
	PlansAttached int `json:"plans_attached"`

	// Upserted/Pruned are filled by the store after the write pass.
	Upserted int `json:"upserted"`
	Pruned   int `json:"pruned"`

	// Reconcile reports the transcript↔wire fusion when transcript
	// rows are present in the raw layer.
	Reconcile *ReconcileStats `json:"reconcile,omitempty"`
}

RederiveReport summarizes one derive pass.

type SessionKey

type SessionKey struct {
	HarnessID        string
	HarnessSessionID string
}

SessionKey is the natural key a raw turn attributes its nodes to.

type Span

type Span struct {
	SpanID       string
	ParentSpanID string
	Kind         string
	Name         string
	Status       string
	StartedAt    time.Time
	DurationNS   int64

	// Seq is the emit ordinal within the trace — presentation order.
	// started_at cannot order spans inside one llm call (a parallel
	// tool batch shares a single instant), so readers sort by seq.
	Seq int64

	// Input: llm — fresh request content blocks; tool — the tool_use
	// input rendered as a single tool_use block.
	Input []llm.ContentBlock
	// Output: llm — response content blocks; tool — the tool_result
	// block, once it arrives.
	Output []llm.ContentBlock

	// CallKind is the §2g taxonomy verbatim ("main", "offshoot:…",
	// "injected:…") on llm and event spans.
	CallKind string
	ThreadID string
	Model    string

	StopReason string
	Usage      *llm.Usage

	// RawTurnID is the raw row whose call produced this span (0 for
	// tool/agent spans, which are assembled across calls).
	RawTurnID int64
	// NodeHash joins the span back to the merkle layer node that
	// carries the same content ("" for agent spans).
	NodeHash string
}

Span is one observed unit of work. Payloads are delta-only: an llm span's Input carries the content blocks NEW to that call, never the re-sent history; tool results live solely on their tool span's Output. RawTurnID references the capturing raw row — provenance by reference, not copy.

type SpanLink struct {
	FromTraceID string
	FromSpanID  string
	FromIO      string
	ToTraceID   string
	ToSpanID    string
	ToIO        string
	Kind        string
}

SpanLink is a dataflow edge. From/To trace ids differ on cross-trace causality (compaction seams) — single-trace link keys cannot represent those.

type SpanReport

type SpanReport struct {
	Traces       int            `json:"traces"`
	Spans        int            `json:"spans"`
	SpanKinds    map[string]int `json:"span_kinds"`
	CallKinds    map[string]int `json:"call_kinds"`
	LinkKinds    map[string]int `json:"link_kinds"`
	Synthetic    int            `json:"synthetic_traces"`
	OrphanShadow int            `json:"orphan_shadow_calls"`
}

SpanReport summarizes one emit pass for gates and operators.

type SpanSet

type SpanSet struct {
	Turns []*SpanTurn

	// Links holds CROSS-trace links only (compaction seams, the odd
	// interrupted tool feed); intra-trace links live on their turn.
	Links []*SpanLink

	// ModelUsage is the per-session, per-model spend breakdown folded at
	// derive time across ALL threads (subagent models included). The
	// session detail surfaces it so the UI can show a dominant model and
	// per-model share; the share basis is cost, not call count, so a
	// fan-out of cheap subagent calls never out-votes the expensive
	// main-spine model (#28).
	ModelUsage map[SessionKey][]ModelUsage

	Report SpanReport
}

SpanSet is the emit stage's output for one derive pass: traces in capture order plus the cross-trace links between them.

func EmitSpans

func EmitSpans(set *DerivedSet) *SpanSet

EmitSpans projects a finished, reconciled DerivedSet into the span model. Pure; safe to call repeatedly.

The walk is phased because wire order races structure: a permission check completes before the main call whose tool_use it judges finishes streaming, and a subagent's first call can land before its Task tool_use is captured. The spine pass builds every trace and tool span; threads then anchor to completed Task spans; shadow calls anchor last, when every candidate tool span exists. Capture order is preserved within each phase, and span ordering inside a trace is by StartedAt, so phasing never reorders time.

type SpanSource

type SpanSource struct {
	RawTurnID  int64
	RequestID  string
	CapturedAt time.Time
	Kind       string
	ThreadID   string
	Session    SessionKey

	// Chain holds the retained node for every chain position of this
	// call (root → leaf; last is the response). New marks positions
	// first captured by THIS call.
	Chain []*DerivedNode
	New   []bool

	// Anchor is the tool_use id this call attaches to, recorded
	// per-call by the attach passes. Node stamps (ParentToolUseID)
	// cannot carry this: checks share deduped prefix nodes, and a
	// shared node holds only the last writer's edge.
	Anchor string
}

SpanSource is the per-call input to the span emit stage: the call's wire identity, its classified kind, and which chain positions this call captured first (the delta vs. re-sent history).

type SpanTurn

type SpanTurn struct {
	TraceID string
	Session SessionKey

	// UserPrompt is the text of the genuine user prompt that opened
	// the turn ("" for synthetic openers).
	UserPrompt string

	// ResponsePreview is the text the closing conversation-spine llm
	// call answered with, truncated for the turn card — the response
	// counterpart of UserPrompt, folded at derive time so collapsed
	// renderings never need span payloads.
	ResponsePreview string

	// Synthetic marks traces not opened by a human prompt
	// ("post-compaction" for compaction continuations).
	Synthetic string

	StartedAt time.Time
	EndedAt   time.Time

	// Token totals summed over every llm span in the trace — shadow
	// calls included, because the turn really spent them. The Main*
	// pair counts only call_kind=main spans, so shadow spend is the
	// difference and both numbers are visible to the read layer.
	TotalInputTokens    int64
	TotalOutputTokens   int64
	MainInputTokens     int64
	MainOutputTokens    int64
	CacheReadTokens     int64
	CacheCreationTokens int64

	// TotalCostUSD prices every llm span's usage at derive time, so a
	// re-derive reprices history when the table updates.
	TotalCostUSD float64

	Spans []*Span
	Links []*SpanLink
}

SpanTurn is one user-visible turn: a trace. Everything the harness did inside the turn — subagent runs and shadow calls included — lives here.

type TranscriptFile

type TranscriptFile struct {
	Session     SessionKey
	AgentID     string // "" for the main transcript
	AgentType   string
	Description string
	ToolUseID   string
	// contains filtered or unexported fields
}

TranscriptFile is one parsed harness transcript: the main session file or one subagent's. ToolUseID (from the harness's subagent meta.json) is the Task tool_use that forked the agent — the causal fork edge reconciliation attaches to the wire capture.

func ParseTranscriptFile

func ParseTranscriptFile(rec *storage.RawTurnRecord) (*TranscriptFile, error)

ParseTranscriptFile decodes one transcript raw row into the reconciler's working shape, building its projected-content signature set.

type VerifyResult

type VerifyResult struct {
	// RawTurns is the number of raw rows scanned.
	RawTurns int `json:"raw_turns"`

	// ParsedTurns is how many raw rows parsed cleanly into a
	// request/response pair.
	ParsedTurns int `json:"parsed_turns"`

	// ParseFailures lists raw rows whose request or response no longer
	// parses (capped). Non-empty means a provider parser regressed OR
	// a raw row predates a parser fix — either way the raw row is
	// intact and re-derivable later.
	ParseFailures []string `json:"parse_failures,omitempty"`

	// RawOnlyTurns counts raw rows whose response fails the same
	// validation ingest applies (missing role/content — error
	// captures, aborted streams). Ingest never derived nodes for
	// these, so neither does the verifier; the raw row is their only
	// representation, by design.
	RawOnlyTurns int `json:"raw_only_turns"`

	// DerivedNodes is the number of distinct node hashes produced by
	// re-deriving every parsed raw turn.
	DerivedNodes int `json:"derived_nodes"`

	// PresentNodes is how many derived hashes exist in the node store.
	PresentNodes int `json:"present_nodes"`

	// MissingNodes samples derived hashes absent from the node store
	// (capped at 20). Empty means the round-trip reproduces the
	// captured nodes exactly.
	MissingNodes []string `json:"missing_nodes,omitempty"`
}

VerifyResult reports the raw→derived round-trip: re-deriving node chains from the immutable raw layer and checking every derived hash against the node store. This is the Phase-1 definition-of-done for the raw-capture layer — when MissingNodes is empty, the derived layer is a pure function of raw and can be rebuilt at will.

func VerifyRederive

func VerifyRederive(ctx context.Context, raw storage.RawTurnStore, nodes storage.Driver, project string) (*VerifyResult, error)

VerifyRederive scans the entire raw layer, re-derives each turn's node chain with the same construction the ingest worker uses, and checks every derived hash against the node store.

project mirrors the ingest worker's configured project tag; it does not participate in node hashes, so any value verifies identically.

func (*VerifyResult) Verified

func (r *VerifyResult) Verified() bool

Verified reports whether the round-trip was lossless.

Directories

Path Synopsis
Package worker is the production derive loop: it polls the dirty- session queue ingest feeds (see storage.DeriveQueue), debounces bursts, and re-derives one session at a time under a per-session advisory lock.
Package worker is the production derive loop: it polls the dirty- session queue ingest feeds (see storage.DeriveQueue), debounces bursts, and re-derives one session at a time under a per-session advisory lock.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL