Documentation
¶
Overview ¶
Package parse is the server-side pipeline that turns a session's stored raw bytes into the queryable projection. It runs the per-agent reducer over the unparsed tail of a session, prices each usage event from the compiled-in pricing table, and applies the result incrementally: each chunk does work proportional to its own bytes, not to the whole session.
Index ¶
Constants ¶
const Epoch = 3
Epoch is the fleet-wide reparse signal: a binary constant the running server compares against parse_meta.reparsed_epoch to decide whether already-ingested data needs rebuilding. When Epoch != reparsed_epoch the server reparses every session in the background and, on success, writes Epoch back, so a deploy is all it takes to roll a parser improvement out to old data. Bump it in the same commit as any change that alters parser or reducer output: new or removed rows, changed field values, a different fold, or a pricing-table change that re-prices stored usage (the projection carries cost, so a reprice is an output change too).
Why a constant and not a migration: parser behavior lives in the binary, not the schema. The most recent parser change (PR #18, "Lift Codex image payloads to the CAS; stop refusing bodyless big lines") shipped with no database migration at all, so a migration-versioned trigger would have missed it and the maintainer had to reparse by hand over SSH. The epoch travels with the binary, so the signal fires exactly when the code that produces the projection changes, migration or not.
Relationship to Version: Version (see parse.go) is the per-session incremental-resume marker stored on each session_raw row; it stops two parser versions' output from blending on the live append path. Epoch is the global "has the whole corpus been reparsed since the last output change" marker. They move together in practice, so bump both when output changes. The golden-fixtures test (epoch_test.go) is the guardrail that makes the Epoch bump impossible to forget: it snapshots the projection for representative fixtures and fails, by name, when that output drifts.
An Epoch bump is also a way to backfill a derived projection table that did not exist before. A caught-up session never re-enters AdvanceProjection, and the append path does not fill session_signals (the settle pass does, once a session settles), so on the first deploy of a new derived table an Epoch bump reparses the corpus to populate it in one pass, independent of whether the settle loop is enabled. The reparse also rebuilds session_signals at the running quality.Version, so a scoring-model change rides the same signal. Such a bump leaves the parser's projection delta byte-for-byte identical, so the golden fixtures do not move; the bump is intentional and stands on its own.
Epoch 1 -> 2: introduce session_signals, the per-session derived behavioral signals (an outcome classification, a quality score and grade, tool-health counts, prompt-hygiene counts, and context-health figures). They are materialized by the settle pass as sessions settle and rebuilt on reparse; this one-time Epoch bump backfills the table across the existing (already-settled) corpus on first deploy and stamps every row at the running quality.Version. session_signals is a new derived table rather than a change to the parser's output, so this bump leaves the projection delta byte-for-byte identical and the golden fixtures do not move; it stands on its own.
Epoch 2 -> 3: materialize per-message prompt-hygiene facts on the messages row (prompt_short, prompt_no_code, prompt_bare_greeting, prompt_digest; see migration 0022_prompt_hygiene_facts and the message insert in store/projection.go). They are derived at insert so the settle pass aggregates fixed-size columns instead of reading prompt bodies back. Migration 0022 adds the columns but, unlike a generated column, cannot backfill them; this bump reparses the corpus so every existing message re-inserts through the classifier and the columns fill in one pass. The facts are store-side derived columns, not parser output, so the reducer's projection delta is byte-for-byte identical and the golden fixtures do not move; the bump is the backfill signal and stands on its own.
const Version = 3
Version is the parser projection version. Bump it when parsing changes so a reparse can be told which sessions are stale. A session parsed past byte 0 by a different version cannot be resumed incrementally; reparse rewinds and replays it from scratch.
Version 2 changed how the session rollups are folded: they now count only the usage and message rows that survive their ON CONFLICT dedup, where version 1 added every per-region occurrence and so inflated Claude sessions (which stream one assistant message across several lines that share a usage block). A version-1 session keeps its inflated rollup until a reparse rewinds it, which is why the fix ships with a version bump: an incremental advance over a still version-1 session would fold a correct delta onto a wrong base. Run `akari-server reparse` to correct the live data.
Version 3 added Codex custom_tool_call bodies and binary image attachments (image generation results and pasted images) to the projection, so a reparse backfills those rows on already-ingested sessions.
Variables ¶
This section is empty.
Functions ¶
func Advance ¶
Advance parses any not-yet-parsed bytes of a session and applies them to the projection, looping until the parse cursor catches up to the stored length. It returns the session's message count. The raw bytes are never modified; a parser error leaves the cursor where it was for the next chunk or a reparse to retry.
func Reparse ¶
Reparse rebuilds a session's projection from its stored raw bytes by clearing the derived rows and replaying the whole session through the same reducer the live path uses, atomically (see store.ReparseSession): on any failure the prior projection is left intact rather than a cleared session. This is how a parser improvement reaches already-ingested data without re-uploading anything.
Types ¶
type ParserError ¶
type ParserError struct {
// contains filtered or unexported fields
}
ParserError marks a failure that came from the parser reducer itself: malformed transcript bytes the reducer cannot turn into a projection. It is distinct from an operational error (a store query, a CAS read, a cancelled context), which travels up un-wrapped. The reparse service uses this distinction: a parser error is per-session and deterministic (re-running fails the same way), so it is counted and the run still completes; an operational error is treated as transient and aborts the run without stamping the epoch, so the next start retries rather than masking it.
func (*ParserError) Error ¶
func (e *ParserError) Error() string
func (*ParserError) Unwrap ¶
func (e *ParserError) Unwrap() error