Documentation
¶
Overview ¶
Package derive turns immutable raw captures into the derived layer: content-addressed node chains today; semantic typing (node_kind), fork edges, and offshoot classification as the reconciled conversation tree lands (design/agent-session-reconciliation.md).
Everything in this package is a pure, re-runnable function of the raw capture. The ingest worker uses the same TurnChain at capture time that the re-deriver uses offline, so "re-deriving nodes from raw_turns reproduces the captured nodes" holds by construction.
Index ¶
- Constants
- func CapturedAt(rec *storage.RawTurnRecord) time.Time
- func ClassifyCall(req *llm.ChatRequest, resp *llm.ChatResponse) string
- func ClassifyInjected(msg llm.Message) string
- func IsTranscriptMeta(meta json.RawMessage) bool
- func LoadCorpus(r io.Reader) (wire, transcripts []storage.RawTurnRecord, err error)
- func LoadCorpusFile(path string) (wire, transcripts []storage.RawTurnRecord, err error)
- func SessionTitle(kind string, resp *llm.ChatResponse) string
- func TurnChain(call CallContext, req *llm.ChatRequest, resp *llm.ChatResponse) []*merkle.Node
- func WriteCorpus(w io.Writer, rows []storage.RawTurnRecord) error
- type CallContext
- type CorpusWriter
- type DerivedNode
- type DerivedSet
- type Deriver
- type ModelUsage
- type ReconcileStats
- type RederiveReport
- type SessionKey
- type SessionStatus
- type Span
- type SpanLink
- type SpanReport
- type SpanSet
- type SpanSource
- type SpanTurn
- type Task
- type TranscriptFile
- type Verdict
Constants ¶
const ( // KindMain is the conversation spine: full tool set, streaming, // high max_tokens. KindMain = "main" // KindUnknown marks calls that match no cataloged tell. A non-zero // unknown count is either a genuinely new harness side-call to // catalog or a classifier regression — surfaced prominently, never // silently bucketed. KindUnknown = "unknown" // Shadow model calls — separate API requests that never appear in // the harness transcript. KindCheckStage1 = "offshoot:permission-check:stage1" KindCheckStage2 = "offshoot:permission-check:stage2" KindTitleGen = "offshoot:title-gen" KindPlanNameGen = "offshoot:plan-name-gen" KindSuggestion = "offshoot:suggestion" KindWebSummary = "offshoot:web-summary" KindProbe = "offshoot:probe" KindCompaction = "offshoot:compaction" // Injected context — whole messages the harness prepends inside // otherwise-normal calls. They drift between turns (server lists // change, modes toggle), so they are kept OFF the main chain as // side-branch nodes (see TurnChain) and marked with these kinds. KindInjectedMCPInstructions = "injected:mcp-instructions" KindInjectedSkillsList = "injected:skills-list" KindInjectedModeBanner = "injected:mode-banner" // KindInjectedClaudeMD is the user-context blob the harness // prepends to its security-monitor checks (<user_claude_md>…). // Every check in a session shares it byte-for-byte, so left on the // chain it fuses all checks into one fan rooted at the blob. KindInjectedClaudeMD = "injected:claude-md" // KindInjectedSystemInsert marks mid-spine system-role inserts // (task reminders, CLAUDE.md re-injections, post-compaction // replays). Currently minted only by the span emit stage — the // node classifier still types these "main" because they ride // inside main calls; a dedicated classifier kind is a candidate // follow-up. KindInjectedSystemInsert = "injected:system-insert" )
Node kinds — the design doc's §2g taxonomy. A "session" on the wire is many API calls of different kinds: the conversation spine plus shadow calls the harness fires on the user's behalf, plus injected context blocks. The set is OPEN: new kinds get cataloged here and a re-derive reclassifies all existing raw data.
const ( SpanKindAgent = "agent" SpanKindLLM = "llm" SpanKindTool = "tool" SpanKindEvent = "event" )
Span kinds — the RFD 00007 vocabulary (step is reserved, unused).
const ( LinkEmits = "emits" // llm output -> tool input LinkFeeds = "feeds" // tool output -> llm input LinkRejoin = "rejoin" // subagent agent -> Task tool LinkVerdict = "verdict" // shadow llm -> judged tool LinkCompactionSeam = "compaction-seam" // compaction llm -> next trace's first llm )
Span link kinds — dataflow edges between spans. Containment is the parent_span_id tree; links carry causality that containment cannot.
Variables ¶
This section is empty.
Functions ¶
func CapturedAt ¶
func CapturedAt(rec *storage.RawTurnRecord) time.Time
CapturedAt resolves a raw record's original capture time: the adapter's request timestamp when the meta block carries one, otherwise the ingest receive time.
func ClassifyCall ¶
func ClassifyCall(req *llm.ChatRequest, resp *llm.ChatResponse) string
ClassifyCall determines the kind of a captured API call from its parsed request and reduced response. Request-envelope parameters (system prompt, max_tokens, tool count, stream) are the definitive discriminators; content shape is the fallback. Tells are grounded in observed traffic — see the design doc §2g and the golden-session measurements.
func ClassifyInjected ¶
ClassifyInjected reports the injected-context kind for one request message, or "" when the message is ordinary conversation. Only whole messages that ARE an injected block qualify — a user turn that merely mentions skills is untouched. These messages drift between turns of the same conversation (an MCP server connects, a mode toggles), so TurnChain keeps them off the hashed spine.
func IsTranscriptMeta ¶
func IsTranscriptMeta(meta json.RawMessage) bool
IsTranscriptMeta reports whether a raw row's meta marks it as a transcript file.
func LoadCorpus ¶
func LoadCorpus(r io.Reader) (wire, transcripts []storage.RawTurnRecord, err error)
LoadCorpus reads a gzipped-JSONL corpus dump from r and returns its rows split by source: wire captures and transcript pushes. The io.Reader form serves embedded corpora (the demo seed bundles corpus files into the binary); LoadCorpusFile wraps it for on-disk dumps.
func LoadCorpusFile ¶
func LoadCorpusFile(path string) (wire, transcripts []storage.RawTurnRecord, err error)
LoadCorpusFile reads a gzipped-JSONL corpus dump and returns its rows split by source: wire captures and transcript pushes.
func SessionTitle ¶
func SessionTitle(kind string, resp *llm.ChatResponse) string
SessionTitle extracts the generated title from a title-gen call's response ({"title": "…"}), or "" when the call isn't a title-gen or the response doesn't parse. Tolerates prose around the JSON object (models occasionally wrap it).
func TurnChain ¶
func TurnChain(call CallContext, req *llm.ChatRequest, resp *llm.ChatResponse) []*merkle.Node
func WriteCorpus ¶ added in v0.25.0
func WriteCorpus(w io.Writer, rows []storage.RawTurnRecord) error
WriteCorpus encodes a slice of raw-turn rows as a corpus dump in one call — the batch convenience over CorpusWriter.
Types ¶
type CallContext ¶
TurnChain materializes the ordered (root → leaf) chain of nodes for a single captured call: one node per request message, followed by the assistant response node. ParentHash linkage is set via merkle.NewNode so every node's hash is stable before any I/O.
Every node is stamped with the call's classified kind (ClassifyCall) and the request-envelope parameters — both as non-hashed metadata. On insert, nodes already present from an earlier call keep that call's stamp (ON CONFLICT DO NOTHING), so each stored node carries the classification of the call that first captured it.
Injected-context messages (MCP instructions, skills lists, mode banners — see ClassifyInjected) become SIDE-BRANCH nodes: they hang off the spine at the position they appeared but the next message's parent bypasses them. These blocks drift between turns of the same conversation, so chaining them would fork the spine at every drift; as side branches they are preserved, marked injected:*, and inert. CallContext carries the capture-side identity of one API call into chain construction: who routed it (provider/agent), which harness sub-thread fired it, and the project tag. All non-hashed metadata.
type CorpusWriter ¶ added in v0.25.0
type CorpusWriter struct {
// contains filtered or unexported fields
}
CorpusWriter streams raw-turn rows into a gzipped-JSONL corpus dump — the inverse of LoadCorpus, and the on-disk shape `tapes dev dump-corpus` emits. Rows are appended one at a time so a scan of the interleaved raw layer (many sessions, insertion-ordered by id) can fan out to one open writer per session without buffering payloads. Each row carries its own `source`, so wire + transcript rows written to the same file round-trip back through LoadCorpus's source split. Close flushes the gzip trailer. The writer lives beside the loader so the two formats can never drift.
func NewCorpusWriter ¶ added in v0.25.0
func NewCorpusWriter(w io.Writer) *CorpusWriter
NewCorpusWriter wraps w with the gzip + JSONL encoding the corpus format uses.
func (*CorpusWriter) Close ¶ added in v0.25.0
func (c *CorpusWriter) Close() error
Close flushes and finalizes the gzip stream.
func (*CorpusWriter) Write ¶ added in v0.25.0
func (c *CorpusWriter) Write(rec *storage.RawTurnRecord) error
Write appends one raw-turn row, verbatim.
type DerivedNode ¶
type DerivedNode struct {
Node *merkle.Node
Session SessionKey
CapturedAt time.Time
}
DerivedNode is one node of the rebuilt derived layer plus its attribution and provenance.
type DerivedSet ¶
type DerivedSet struct {
// Nodes in capture order (chain order within a turn). Deduplicated
// by hash — the first capturing call wins, mirroring ingest's
// ON CONFLICT DO NOTHING semantics.
Nodes []*DerivedNode
// Sessions are the harness keys covered by the raw layer. The
// store prunes stale derived rows only within these sessions.
Sessions []SessionKey
// SessionTitles carries the folded title-gen output per session
// (latest call wins — the harness regenerates titles as the
// session evolves).
SessionTitles map[SessionKey]string
// SpanSources retains one slim record per parsed raw turn so the
// span emit stage (EmitSpans) can re-walk the capture in call
// order after the attach/reconcile passes have stamped fork and
// offshoot anchors. Holds only pointers to retained nodes plus
// scalar call identity — never payload copies.
SpanSources []*SpanSource
Report RederiveReport
}
DerivedSet is the deriver's output for one org: the complete node set re-derived from the raw layer, ready to upsert.
func BuildDerivedSet ¶
func BuildDerivedSet(rawTurns []storage.RawTurnRecord, project string) (*DerivedSet, error)
BuildDerivedSet derives a complete node set from an in-memory slice of raw turns, in the order given. Convenience wrapper around the streaming Deriver for tests and small batches; callers with a real store should stream records in capture order instead.
type Deriver ¶
type Deriver struct {
// contains filtered or unexported fields
}
Deriver streams raw turns (in capture order) into a deduplicated derived node set. Memory stays proportional to the UNIQUE content in the raw layer, not to the sum of every turn's re-sent history.
func NewDeriver ¶
NewDeriver creates a streaming deriver. Feed turns with AddTurn in chronological order, then call Finish exactly once.
func (*Deriver) AddTurn ¶
func (dv *Deriver) AddTurn(rec *storage.RawTurnRecord)
AddTurn parses, classifies, and chains one raw turn, folding its nodes into the deduplicated set. The record is not retained.
func (*Deriver) Finish ¶
func (dv *Deriver) Finish() *DerivedSet
Finish runs the cross-call attach passes and returns the completed set. The deriver must not be reused afterwards.
type ModelUsage ¶
type ModelUsage struct {
Model string `json:"model"`
Calls int64 `json:"calls"`
InputTokens int64 `json:"input_tokens"`
OutputTokens int64 `json:"output_tokens"`
CostUSD float64 `json:"cost_usd"`
}
ModelUsage is one model's contribution to a session: how many llm calls ran on it and what they spent. Cost is priced at derive time (like the trace fold), so a re-derive reprices history.
type ReconcileStats ¶
type ReconcileStats struct {
TranscriptFiles int `json:"transcript_files"`
SubagentForks int `json:"subagent_forks"`
ForkedChains int `json:"forked_chains"`
MainChainsJoined int `json:"main_chains_joined"`
// ConversationJoined / ConversationTotal measure how many
// conversation-spine nodes' content appears in a transcript — the
// Go-native version of the prototype's join-rate oracle.
ConversationJoined int `json:"conversation_joined"`
ConversationTotal int `json:"conversation_total"`
}
ReconcileStats reports the transcript↔wire fusion for one org.
func ReconcileTranscripts ¶
func ReconcileTranscripts(set *DerivedSet, files []*TranscriptFile) *ReconcileStats
ReconcileTranscripts assigns each wire-derived conversation chain to the transcript file whose content it matches, stamping subagent chain roots with their fork edge. Operates on the in-memory derived set after the wire pass and before the store write — pure and re-runnable like everything else in the deriver.
type RederiveReport ¶
type RederiveReport struct {
RawTurns int `json:"raw_turns"`
ParsedTurns int `json:"parsed_turns"`
RawOnlyTurns int `json:"raw_only_turns"`
ParseFailures []string `json:"parse_failures,omitempty"`
Nodes int `json:"nodes"`
CallKinds map[string]int `json:"call_kinds"`
NodeKinds map[string]int `json:"node_kinds"`
// Verdict attach: judged actions grouped across stages, and how
// many attached one-to-one to a captured tool_use.
JudgedActions int `json:"judged_actions"`
AttachedVerdicts int `json:"attached_verdicts"`
// UnattachedActions samples judged actions that found no matching
// tool_use (capped) — expected for non-tool events like subagent
// handbacks; anything else is matcher signal worth reading.
UnattachedActions []string `json:"unattached_actions,omitempty"`
// WebSummaryAttached counts web-summary calls linked back to their
// WebFetch/WebSearch tool_use.
WebSummaryAttached int `json:"web_summary_attached"`
// PlansAttached counts plan-name-gen calls linked to the
// ExitPlanMode tool_use that accepted the plan.
PlansAttached int `json:"plans_attached"`
// Reconcile reports the transcript↔wire fusion when transcript
// rows are present in the raw layer.
Reconcile *ReconcileStats `json:"reconcile,omitempty"`
}
RederiveReport summarizes one derive pass.
type SessionKey ¶
SessionKey is the natural key a raw turn attributes its nodes to.
type SessionStatus ¶ added in v0.25.0
type SessionStatus struct {
DerivedStatus string
HasGitActivity bool
ToolResultCount int
ToolErrorCount int
}
SessionStatus is the deriver's chain-aware session outcome plus the signals that produced it. It was computed at ingest from the re-sent conversation nodes; the deriver now folds it from the delta-span projection so status is a pure, re-derivable output (Goals 1 & 5), and ingest touches session identity only.
func FoldSessionStatus ¶ added in v0.25.0
func FoldSessionStatus(tools []*Span, terminal *Span) SessionStatus
FoldSessionStatus reproduces the ingest-time status computation from a session's spans: git activity and tool result/error tallies over the session's tool spans (all threads, matching the ingest count over the re-sent conversation), and sessions.DetermineStatus over the session's terminal main-spine response.
A tool span carries its tool_result once its Output is filled, and its Status is "error" for an error result — so counting spans is the delta-view equivalent of the ingest count over tool_result blocks, without the re-sent-history double-count the node path had to dedup.
type Span ¶
type Span struct {
SpanID string
ParentSpanID string
Kind string
Name string
Status string
StartedAt time.Time
DurationNS int64
// Seq is the emit ordinal within the trace — presentation order.
// started_at cannot order spans inside one llm call (a parallel
// tool batch shares a single instant), so readers sort by seq.
Seq int64
// Input: llm — fresh request content blocks; tool — the tool_use
// input rendered as a single tool_use block.
Input []llm.ContentBlock
// Output: llm — response content blocks; tool — the tool_result
// block, once it arrives.
Output []llm.ContentBlock
// CallKind is the §2g taxonomy verbatim ("main", "offshoot:…",
// "injected:…") on llm and event spans.
CallKind string
ThreadID string
Model string
StopReason string
Usage *llm.Usage
// Verdict is the security-monitor disposition on a permission-check
// span (nil elsewhere), extracted at derive time by ClassifyVerdict.
Verdict *Verdict
// RawTurnID is the raw row whose call produced this span (0 for
// tool/agent spans, which are assembled across calls).
RawTurnID int64
// NodeHash joins the span back to the merkle layer node that
// carries the same content ("" for agent spans).
NodeHash string
}
Span is one observed unit of work. Payloads are delta-only: an llm span's Input carries the content blocks NEW to that call, never the re-sent history; tool results live solely on their tool span's Output. RawTurnID references the capturing raw row — provenance by reference, not copy.
type SpanLink ¶
type SpanLink struct {
FromTraceID string
FromSpanID string
FromIO string
ToTraceID string
ToSpanID string
ToIO string
Kind string
}
SpanLink is a dataflow edge. From/To trace ids differ on cross-trace causality (compaction seams) — single-trace link keys cannot represent those.
type SpanReport ¶
type SpanReport struct {
Traces int `json:"traces"`
Spans int `json:"spans"`
SpanKinds map[string]int `json:"span_kinds"`
CallKinds map[string]int `json:"call_kinds"`
LinkKinds map[string]int `json:"link_kinds"`
Synthetic int `json:"synthetic_traces"`
OrphanShadow int `json:"orphan_shadow_calls"`
}
SpanReport summarizes one emit pass for gates and operators.
type SpanSet ¶
type SpanSet struct {
Turns []*SpanTurn
// Links holds CROSS-trace links only (compaction seams, the odd
// interrupted tool feed); intra-trace links live on their turn.
Links []*SpanLink
// ModelUsage is the per-session, per-model spend breakdown folded at
// derive time across ALL threads (subagent models included). The
// session detail surfaces it so the UI can show a dominant model and
// per-model share; the share basis is cost, not call count, so a
// fan-out of cheap subagent calls never out-votes the expensive
// main-spine model (#28).
ModelUsage map[SessionKey][]ModelUsage
// Tasks is the per-session TaskCreate/TaskUpdate fold, replayed in
// event order. KindCounts is the per-session tally of span call_kinds.
// Both are session-rollup facts the deriver owns; the API serves them
// on the composite traces response without re-folding.
Tasks map[SessionKey][]Task
KindCounts map[SessionKey]map[string]int
// Status is the per-session chain-aware outcome + signals, folded from
// the spans at derive time (moved off the ingest hot path).
Status map[SessionKey]SessionStatus
Report SpanReport
}
SpanSet is the emit stage's output for one derive pass: traces in capture order plus the cross-trace links between them.
func EmitSpans ¶
func EmitSpans(set *DerivedSet) *SpanSet
EmitSpans projects a finished, reconciled DerivedSet into the span model. Pure; safe to call repeatedly.
The walk is phased because wire order races structure: a permission check completes before the main call whose tool_use it judges finishes streaming, and a subagent's first call can land before its Task tool_use is captured. The spine pass builds every trace and tool span; threads then anchor to completed Task spans; shadow calls anchor last, when every candidate tool span exists. Capture order is preserved within each phase, and span ordering inside a trace is by StartedAt, so phasing never reorders time.
type SpanSource ¶
type SpanSource struct {
RawTurnID int64
RequestID string
CapturedAt time.Time
Kind string
ThreadID string
Session SessionKey
// Source is the capture source of the raw turn this call came from
// ('wire' | 'transcript') — provenance carried onto the trace.
Source string
// Chain holds the retained node for every chain position of this
// call (root → leaf; last is the response). New marks positions
// first captured by THIS call.
Chain []*DerivedNode
New []bool
// Anchor is the tool_use id this call attaches to, recorded
// per-call by the attach passes. Node stamps (ParentToolUseID)
// cannot carry this: checks share deduped prefix nodes, and a
// shared node holds only the last writer's edge.
Anchor string
}
SpanSource is the per-call input to the span emit stage: the call's wire identity, its classified kind, and which chain positions this call captured first (the delta vs. re-sent history).
type SpanTurn ¶
type SpanTurn struct {
TraceID string
Session SessionKey
// UserPrompt is the text of the genuine user prompt that opened
// the turn ("" for synthetic openers).
UserPrompt string
// ResponsePreview is the text the closing conversation-spine llm
// call answered with, truncated for the turn card — the response
// counterpart of UserPrompt, folded at derive time so collapsed
// renderings never need span payloads.
ResponsePreview string
// Synthetic marks traces not opened by a human prompt
// ("post-compaction" for compaction continuations).
Synthetic string
// Source is the capture source of the raw turns that produced this
// trace ('wire' | 'transcript'), promoted from raw_turns.source.
Source string
StartedAt time.Time
EndedAt time.Time
// Token totals summed over every llm span in the trace — shadow
// calls included, because the turn really spent them. The Main*
// pair counts only call_kind=main spans, so shadow spend is the
// difference and both numbers are visible to the read layer.
TotalInputTokens int64
TotalOutputTokens int64
MainInputTokens int64
MainOutputTokens int64
CacheReadTokens int64
CacheCreationTokens int64
// TotalCostUSD prices every llm span's usage at derive time, so a
// re-derive reprices history when the table updates.
TotalCostUSD float64
Spans []*Span
Links []*SpanLink
}
SpanTurn is one user-visible turn: a trace. Everything the harness did inside the turn — subagent runs and shadow calls included — lives here.
type Task ¶ added in v0.25.0
type Task struct {
ID string `json:"id"`
Subject string `json:"subject"`
Description string `json:"description,omitempty"`
Status string `json:"status"`
Updates int `json:"updates"`
}
Task is one harness todo item, folded from TaskCreate/TaskUpdate tool calls at derive time. It is a session-scoped rollup fact persisted on sessions.tasks; the API serves it verbatim.
func FoldSessionTasks ¶ added in v0.25.0
FoldSessionTasks replays a session's TaskCreate/TaskUpdate tool spans in event order into the current task list. The replay must run in event order, so tool spans are sorted by StartedAt then Seq (storage hands spans back sorted by trace_id, which is lexicographic, not chronological — the same reason the old read-time fold sorted).
type TranscriptFile ¶
type TranscriptFile struct {
Session SessionKey
AgentID string // "" for the main transcript
AgentType string
Description string
ToolUseID string
// contains filtered or unexported fields
}
TranscriptFile is one parsed harness transcript: the main session file or one subagent's. ToolUseID (from the harness's subagent meta.json) is the Task tool_use that forked the agent — the causal fork edge reconciliation attaches to the wire capture.
func ParseTranscriptFile ¶
func ParseTranscriptFile(rec *storage.RawTurnRecord) (*TranscriptFile, error)
ParseTranscriptFile decodes one transcript raw row into the reconciler's working shape, building its projected-content signature set.
type Verdict ¶ added in v0.25.0
type Verdict struct {
Disposition string `json:"disposition"` // ALLOW | BLOCK
Stage int `json:"stage"`
Reasoned bool `json:"reasoned"`
}
Verdict is the security monitor's disposition on a permission-check span, extracted at derive time from the check response. It is a typed derived field persisted on the span (spans.verdict), not read-time text parsing.
func ClassifyVerdict ¶ added in v0.25.0
func ClassifyVerdict(callKind string, blocks []llm.ContentBlock) *Verdict
ClassifyVerdict extracts the security-monitor disposition from a permission-check span's output blocks. It returns nil for any span that is not a permission check, or for a check that carries no <block> marker. This is the (wire, claude-code) adapter for verdicts, mirroring ClassifyCall: harness-specific tells stay here, the wire type is harness-neutral.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
Package worker is the production derive loop: it polls the dirty- session queue ingest feeds (see storage.DeriveQueue), debounces bursts, and re-derives one session at a time under a per-session advisory lock.
|
Package worker is the production derive loop: it polls the dirty- session queue ingest feeds (see storage.DeriveQueue), debounces bursts, and re-derives one session at a time under a per-session advisory lock. |