parser

type Agent ¶

type Agent string

Agent identifies which on-disk format a session uses.

const (
	AgentClaude Agent = "claude"
	AgentCodex  Agent = "codex"
	AgentPi     Agent = "pi"
)

type Attachment ¶

type Attachment struct {
	MessageOrdinal int
	SHA256         string
	Bytes          int
	MediaType      string
	Filename       string
	Content        string
}

Attachment is one binary blob attached to a message: today a lifted image (a Codex image-generation result or a pasted image). The bytes live in the CAS keyed by SHA256; Bytes is the raw (decoded) size and MediaType its semantic type, so the UI can render it without fetching. Content carries the decoded bytes only on the inline path (a small image with no sentinel, used by the batch/test parser); when the client lifted the image it is empty and SHA256 names the already-uploaded blob.

type AttachmentOp ¶

type AttachmentOp struct {
	MessageOrdinal int
	SHA256         string
	Content        string
	Bytes          int
	MediaType      string
	Filename       string
}

AttachmentOp records one binary attachment (today a lifted image) against a message. Like a tool body it reaches the CAS by one of two paths: when the client lifted the image and left a sentinel, SHA256 is set and the server records the reference with no blob write; otherwise Content holds the decoded bytes inline for the server to store. Bytes and MediaType describe the decoded image either way, so the row's metadata is the same whichever path delivered the bytes.

type Body ¶

type Body struct {
	SHA256      string
	Bytes       int
	MediaType   string
	Stored      []byte
	ContentType string
	Kind        string
}

Body is one tool body the client lifts out of the transcript. Stored holds the exact bytes the CAS keeps, which are the raw canonical bytes for a small body and the zstd-compressed form for a large one; ContentType says which. SHA256 is the key, the sha256 of Stored. Bytes is the RAW (uncompressed) canonical length, the size the transcript sentinel and the tool_calls row record (what a reader thinks of as the body's size), independent of how the bytes are stored. Kind is "input" or "result", for diagnostics and tests.

func ExtractBodies(agent Agent, region []byte, enc BodyEncoder) ([]byte, []Body, error)

func RewriteLine(agent Agent, line []byte, enc BodyEncoder) ([]byte, []Body)

type BodyEncoder ¶

type BodyEncoder interface {
	// EncodeBody returns the CAS key (sha256 of the stored bytes), the stored bytes,
	// and their storage content type (ContentRaw or ContentZstd) for a body's raw
	// canonical bytes.
	EncodeBody(raw []byte) (sha string, stored []byte, contentType string)
}

BodyEncoder turns a tool body's canonical raw bytes into the bytes the CAS actually stores, deciding per body whether compression is worth it. The CAS key is the sha256 of the STORED bytes, so the same raw body must always encode to the same stored bytes and key; the client supplies a deterministic implementation. The parser depends only on this interface, never on a compression library, which is what keeps the server (which links the parser but never compresses) free of one. The streaming big-line path lives in the client and drives the same encoder directly, so a body encodes identically whether buffered or streamed.

type BodyKind ¶

type BodyKind int

BodyKind selects how the raw located value is canonicalized into the bytes the CAS stores. The three kinds mirror the branches of bodyContent: a value copied verbatim, a JSON string whose decoded contents are the body, and an array of blocks flattened to its text.

const (
	// BodyRaw copies the value's raw bytes verbatim. This is the canonical form
	// for genuine objects (a result whose body is a JSON object) and for the
	// claude/pi tool input, where the stored body is exactly input.Raw.
	BodyRaw BodyKind = iota
	// BodyJSONString treats the value as a JSON string and emits its decoded
	// contents, matching gjson .String(). This is the canonical form for a string
	// result body and for codex arguments (a JSON-encoded string).
	BodyJSONString
	// BodyArrayText treats the value as an array of typed blocks and emits the
	// blockText flattening: the decoded text of each contributing element joined
	// by a single newline.
	BodyArrayText
	// BodyBase64 treats the value as a JSON string carrying a base64-encoded binary
	// blob (optionally wrapped in a `data:<media>;base64,` URI) and emits the decoded
	// binary bytes. This is the canonical form for the image payloads Codex inlines
	// (image_generation results, data-URI image_url blocks, pasted images): the CAS
	// stores the real PNG/JPEG bytes, not the base64 text, so a reader serves the blob
	// directly under its image media type. The decode ignores \r and \n exactly as
	// encoding/base64 does, so the streamed result is byte-identical to the buffered
	// base64.StdEncoding.DecodeString the extractor uses.
	BodyBase64
)

func ClassifyResultBody(firstByte byte) (BodyKind, string)

type BodyLocation ¶

type BodyLocation struct {
	Span     ValueSpan
	Kind     BodyKind
	Media    string
	FilePath string
	Detail   string
}

BodyLocation is one tool body found in a transcript line by streaming, ready to be lifted to the CAS without ever buffering the body. Span is the raw byte range of the value within the line (relative to the line's first byte), the bytes the sentinel replaces. Kind and Media say how to canonicalize the raw value into the bytes the CAS stores (CanonicalBodyReader), so the streamed body is byte identical to what the server records inline today. FilePath is the top-level file_path string of a JSON tool input (empty otherwise), and Detail is the input's short human-scannable summary (a command, pattern, URL, or description; empty otherwise), both extracted here so the sentinel the caller builds carries them, exactly as the buffered RewriteLine path does via sentinelFilePath and sentinelDetail.

type Delta ¶

type Delta struct {
	Messages    []MessageOp
	ToolCalls   []ToolCall
	ToolResults []ToolResultOp
	Usage       []Usage
	Attachments []AttachmentOp
	Fallbacks   []FallbackOp

	Started time.Time
	Ended   time.Time

	// Cwd and GitBranch are the last values seen in the region. The store ignores
	// them (announce owns those columns); the test-facing Parse wrapper uses them.
	Cwd       string
	GitBranch string
}

Delta is everything one parse of a session produces: the rows to write and the session's timestamp span. It deliberately carries no message/token counters: the session rollups are derived downstream from the deduped set the rebuild actually writes, so a counter here would only risk drifting from it.

type FallbackOp ¶ added in v0.2.5

type FallbackOp struct {
	// MessageOrdinal ties an assistant-side op to the message op the same entry produced
	// (the same way Usage ties to its message). It is nil on a system-side op, which
	// produces no message row and must not disturb ordinals.
	MessageOrdinal *int
	FromModel      string
	ToModel        string
	// Trigger, RefusalCategory, and RefusalExplanation come only from the system entry.
	Trigger            string
	RefusalCategory    string
	RefusalExplanation string
	// Declined* are the token counts of the declined attempt (the type=="message"
	// iteration entries), meaningful only on an assistant-side op that saw a
	// fallback_message entry. Zero elsewhere.
	DeclinedInput      int
	DeclinedOutput     int
	DeclinedCacheWrite int
	DeclinedCacheRead  int
	// DeclinedObserved is true only when the declined spend was actually summed from
	// fallback_message iteration entries. An assistant entry that carries a fallback
	// content block but no usage.iterations is a real fallback whose declined counts
	// were never reported, so it leaves this false and the zero Declined* stay
	// "unmeasured" rather than reading as a measured zero-token attempt.
	DeclinedObserved bool
	OccurredAt       time.Time
	// DedupKey is the top-level requestId when present, else the assistant message id.
	// Every line of one logical fallback repeats it, so the store dedups and merges on it.
	DedupKey string
}

FallbackOp records that a Claude Fable turn was declined by the safety classifier and re-served by a lower model. It is emitted only from explicit Claude Code markers (a "fallback" content block, a usage.iterations "fallback_message" entry, or a "model_refusal_fallback" system entry): never from a bare model-string change, which is an intentional switch (a /model command, a resume under a different default, a subagent on a smaller model), not a fallback.

One logical fallback surfaces across several JSONL lines that the store merges by DedupKey: Claude splits one API message into several assistant entries sharing the requestId (each repeating the same iterations), and a separate system entry carries the refusal category. The assistant side brings MessageOrdinal and the declined attempt's token counts; the system side brings Trigger, RefusalCategory, and RefusalExplanation. Either can be the first to arrive, so the store's merge fills a field from whichever line carries it. A field the source did not observe is left zero (MessageOrdinal nil, token counts zero, strings empty) so the merge can tell "unset" from a real value.

type Idx ¶

type Idx int

Idx selects an array element by its 0-based position.

type Key ¶

type Key string

Key selects an object member by name.

type LocatedSpan ¶

type LocatedSpan struct {
	PathIndex int
	Span      ValueSpan
}

LocatedSpan pairs a located value's span with the index of the path that produced it, so callers can correlate results back to their request even when some paths are absent and skipped.

func LocateValues(ctx context.Context, paths [][]Step, next func() ([]byte, error)) ([]LocatedSpan, error)

type Message ¶

type Message struct {
	Ordinal       int
	Role          Role
	Content       string
	ThinkingText  string
	ThinkingBytes int
	Model         string
	Timestamp     time.Time
	HasThinking   bool
	HasToolUse    bool
}

Message is one turn. Content holds the conversational text (stored inline and searchable); ThinkingText holds the concatenated reasoning plaintext, which is empty when the agent redacts it (see ThinkingBytes).

ThinkingBytes is the reasoning-trace weight: the byte size of the turn's reasoning material, whether or not its plaintext survived. Current Claude Code and Codex ship the reasoning encrypted (Claude leaves a "signature", Codex an "encrypted_content" blob) and drop the plaintext, so ThinkingText is empty while the reasoning still happened; the encrypted payload's length tracks the hidden reasoning volume closely (measured r=0.97 for Claude signatures against the rare blocks that kept their plaintext, r=0.997 for Codex encrypted_content against the reasoning-token count it reports). pi keeps its thinking in the clear, so there the weight is just the plaintext length. It is the per-turn volume the observed-thinking signal ranks per model; HasThinking is true whenever a reasoning block was present, decoupled from whether its text was redacted.

type MessageOp ¶

type MessageOp struct {
	Ordinal       int
	Role          Role
	Content       string
	ThinkingText  string
	ThinkingBytes int
	Model         string
	HasThinking   bool
	HasToolUse    bool
	Timestamp     time.Time
}

MessageOp is one message write in a Delta. Each ordinal is written exactly once, carrying the turn's complete text: the reducer folds a whole session in one pass, so a turn's fragments are joined before the op is emitted, never appended in place. ThinkingBytes is the turn's reasoning-trace weight (see Message.ThinkingBytes): plaintext length where the agent logs it, else the encrypted payload length, so a redacted turn still records how much it thought.

type Reducer ¶ added in v0.3.0

type Reducer struct {
	// contains filtered or unexported fields
}

Reducer folds one session's raw bytes into a Delta. A parse always covers the whole session from byte zero: construct a Reducer, Feed the stored regions in offset order (each must contain only complete lines; the ingest protocol guarantees every stored byte ends on a newline), and Finish to flush the open turn and take the Delta. Because the whole parse is one Reducer, an open turn folds freely across region boundaries and no state is ever serialized. Malformed individual lines are skipped; a Feed error means the region could not be processed.

func NewReducer(agent Agent) (*Reducer, error)

func (x *Reducer) Feed(region []byte, baseOffset int64) error

func (x *Reducer) Finish() Delta

type Role ¶

type Role string

Role is a normalized message role.

const (
	RoleUser      Role = "user"
	RoleAssistant Role = "assistant"
	RoleSystem    Role = "system"
	RoleTool      Role = "tool"
	// RoleContext marks an injected-context turn: agent framing (project
	// instructions, the environment block) that an agent prepends to a session
	// rather than a human prompt. It is a distinct role so every role='user'
	// reader (the session title, user_message_count, the prompt-hygiene
	// aggregate) excludes it, and the transcript renders it in its own Context
	// section instead of as the opening turn. Only Codex records such framing as a
	// transcript message today (Claude and pi keep their framing in the system
	// prompt, which akari never ingests); the role is agent-agnostic so any
	// reducer can classify into it.
	RoleContext Role = "context"
)

type Session ¶

type Session struct {
	Cwd         string
	GitBranch   string
	StartedAt   time.Time
	EndedAt     time.Time
	Messages    []Message
	ToolCalls   []ToolCall
	UsageEvent  []Usage
	Attachments []Attachment
	Fallbacks   []FallbackOp
}

Session is the parsed projection of one session file.

func Parse(agent Agent, raw []byte) (Session, error)

type Step ¶

type Step interface {
	// contains filtered or unexported methods
}

Step is one segment of a path into a JSON document: either an object Key or an array Idx. The marker method keeps the two step kinds in a single closed set so callers cannot smuggle in an arbitrary type.

type ToolCall ¶

type ToolCall struct {
	MessageOrdinal  int
	CallIndex       int
	ToolName        string
	Category        string
	FilePath        string
	Detail          string
	InputJSON       string
	InputSHA256     string
	InputBytes      int
	InputMediaType  string
	ResultBody      string
	ResultSHA256    string
	ResultBytes     int
	ResultMediaType string
	ResultStatus    string // "ok" | "error" | "" (pending)
	CallUID         string
}

ToolCall is one tool invocation attached to a message. InputJSON and ResultBody carry the bulky bodies the CAS stores; ResultBytes and ResultMediaType describe ResultBody exactly, so the recorded size and media type always match the stored content. InputJSON is the raw tool-input JSON; ResultBody is the result body (a tool result that is an array of text blocks is flattened to its text).

CallUID is the agent's own call id; the incremental pipeline records it on the row so a tool result arriving in a later line (Claude delivers tool_result in the following user entry, which may land in a later region) is back-patched by an UPDATE keyed on it rather than by a parser-held id->row map. It is carried on inserts only; a Session assembled for tests ignores it. InputSHA256, InputBytes, and InputMediaType carry a CAS reference when the client lifted the input body to the CAS and left a sentinel in the transcript: the server records the reference instead of re-storing the body. InputJSON is empty in that case. When the body travels inline, InputJSON holds it and the SHA/bytes/media fields are unset (the server hashes and sizes the inline body).

Detail is a bounded, human-scannable summary of the input the UI shows when a call has no file_path: a shell command, a search pattern, a fetched URL, or an agent's description, derived from the input's top-level JSON keys. On the inline path it is derived here from the raw input; on the sentinel path the body is no longer readable, so it rides the sentinel and comes back through the casRef, exactly the way FilePath does.

type ToolResultOp ¶

type ToolResultOp struct {
	CallUID    string
	Body       string
	BodySHA256 string
	Bytes      int
	MediaType  string
	Status     string
}

ToolResultOp back-patches a tool call's result, matched by the agent's call id.

A result body normally travels inline (Body holds it) and the server writes it to the CAS. When the client has already lifted the body to the CAS and left a sentinel in the transcript, BodySHA256 is set instead: the server records the reference and writes no blob, but Bytes and MediaType still describe the body exactly so the row's metadata is unchanged.

type Usage ¶

type Usage struct {
	MessageOrdinal *int
	Model          string
	Input          int
	Output         int
	CacheWrite     int
	CacheRead      int
	Reasoning      int
	OccurredAt     time.Time
	DedupKey       string
	SourceOffset   int64
	SourceIndex    int
}

Usage is one token-accounting record. MessageOrdinal is nil for session-level totals not tied to a single message. SourceOffset and SourceIndex identify the originating line (and its position within it) so incremental inserts are idempotent even for agents whose usage carries no native dedup key.

type ValueSpan ¶

type ValueSpan struct {
	Start int64
	End   int64
}

ValueSpan is the byte range [Start,End) of one located JSON value within a single JSONL line, relative to the line's first byte (offset 0 = first byte of the line). It matches gjson's value Index and Index+len(Raw) exactly.

func LocateValue(ctx context.Context, path []Step, next func() ([]byte, error)) (ValueSpan, bool, error)