tokens

package
v1.52.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 7, 2026 License: AGPL-3.0 Imports: 15 Imported by: 0

Documentation

Overview

Package tokens provides JSONL-based token usage parsing and aggregation for Claude Code sessions.

Privacy guarantee: ParseResult and all derived types carry only token counts, tool names (e.g. "Bash", "mcp__datadog__search_logs"), skill names (short /command strings), and aggregated statistics. The actual text of user prompts, assistant responses, file contents, or command outputs is NEVER stored.

Architecture:

  • Parser.ParseFile reads a Claude JSONL transcript file line-by-line using a 10MB bufio.Scanner buffer. Each line is a JSON object; malformed lines are skipped without returning an error.
  • TokenStore caches parsed results keyed by file path, invalidating on modtime change. A background walker pre-parses all JSONL files on startup; fsnotify callbacks keep the cache fresh for active sessions.
  • PricingTable maps normalized model family names to USD-per-MTok rates and computes estimated cost from a ParseResult.
  • Associator links a ParseResult to a stapler-squad session by conversation UUID, project path prefix, or timestamp proximity.

Index

Constants

View Source 
const (
	ActivityDebugging   = sessionv1.ActivityType_ACTIVITY_TYPE_DEBUGGING
	ActivityRefactoring = sessionv1.ActivityType_ACTIVITY_TYPE_REFACTORING
	ActivityFeatureDev  = sessionv1.ActivityType_ACTIVITY_TYPE_FEATURE_DEV
	ActivityExploratory = sessionv1.ActivityType_ACTIVITY_TYPE_EXPLORATORY
	ActivityOther       = sessionv1.ActivityType_ACTIVITY_TYPE_OTHER
)

ActivityType constants — the 5 non-zero values ActivityType ships in v1.

View Source 
const (
	FindingCacheHitFloorBreach   = sessionv1.FindingType_FINDING_TYPE_CACHE_HIT_FLOOR_BREACH
	FindingSessionTokenCeiling   = sessionv1.FindingType_FINDING_TYPE_SESSION_TOKEN_CEILING
	FindingModelSwitchCacheBust  = sessionv1.FindingType_FINDING_TYPE_MODEL_SWITCH_CACHE_BUST
	FindingOversizedStartContext = sessionv1.FindingType_FINDING_TYPE_OVERSIZED_START_CONTEXT
)

FindingType constants — 4 non-zero values ship in v1. Two more heuristics (redundant/large-file-reads, tool-failure-rate) are deferred — see plan.md's "Detector scope cut" note — and are intentionally not aliased here yet.

View Source 
const (
	SeverityInfo     = sessionv1.Severity_SEVERITY_INFO
	SeverityWarn     = sessionv1.Severity_SEVERITY_WARN
	SeverityCritical = sessionv1.Severity_SEVERITY_CRITICAL
)

Severity constants.

Variables

This section is empty.

Functions

func AttributeToolCosts added in v1.52.0 

func AttributeToolCosts(r *ParseResult, pt *PricingTable) (costs map[string]float64, doubleCounted map[string]bool, unpriced map[string]bool)

AttributeToolCosts attributes each turn's whole cost once to every distinct tool name that appeared in it (never split, never per-call), marking any turn with more than one distinct tool doubleCounted since its cost is now double-booked across tool buckets — so costs must never be summed across tools. See project_plans/insights-cost-intelligence/decisions/ADR-001-per-tool-cost-attribution.md for the full attribution method and the unpriced/doubleCounted semantics.

func ComputeCacheHitRate added in v1.52.0 

func ComputeCacheHitRate(input, cacheRead int64) float64

ComputeCacheHitRate returns cache_read / (input + cache_read), or 0 when both are zero.

func ComputeCacheROI added in v1.52.0 

func ComputeCacheROI(r *ParseResult, pt *PricingTable) (roi float64, ok bool)

ComputeCacheROI returns the signed USD amount saved (or lost) by using prompt caching versus paying for every cache-read token as fresh input: 

roi = cacheRead*(inputPrice-cacheReadPrice)/1e6 - cacheCreation*cacheWritePrice/1e6

A negative result (cache written but never read back) is a real outcome, not an error. Returns (0, false) — never a fake $0.00 — when r's model has no PricingTable entry; see EstimateCost's doc comment for the "abstain rather than guess" rule.

func NormalizeModelFamily 

func NormalizeModelFamily(modelID string) string

NormalizeModelFamily strips date suffixes and normalizes a raw model ID to a pricing-table key.

Examples: 

"claude-sonnet-4-6-20250514" → "claude-sonnet-4"
"claude-sonnet-4-6"          → "claude-sonnet-4"
"claude-opus-4-7"            → "claude-opus-4"
"claude-3-opus-20240229"     → "claude-opus-3"
"claude-haiku-4"             → "claude-haiku-4"
"unknown-model-xyz"          → "unknown-model-xyz"

Types

type ActivityType added in v1.52.0 

type ActivityType = sessionv1.ActivityType

ActivityType classifies the kind of work a session performed. Maps to session.v1.ActivityType enum in Go, same alias pattern as FindingType/Severity (session/tokens/finding_types.go).

func ClassifyActivity added in v1.52.0 

func ClassifyActivity(r *ParseResult) ActivityType

ClassifyActivity classifies a session's activity type: a skill-name substring match ("debug"/"refactor") takes priority, falling back to the writeRatio/readRatio tool-call-ratio heuristic above when no skill matched. See plan.md's Story 1.2.3 acceptance criteria for the full priority rules.

type Associator 

type Associator struct {
	// contains filtered or unexported fields
}

Associator links ParseResult values to stapler-squad sessions.

func NewAssociator 

func NewAssociator(storage SessionStorage) *Associator

NewAssociator creates a new Associator backed by the given storage.

func (*Associator) Associate 

func (a *Associator) Associate(result *ParseResult) (sessionID string, isOrphan bool)

Associate returns the stapler-squad session ID that best matches the given ParseResult, and whether the result is an orphan (no match found).

Lookup priority:

  1. Exact conversation UUID match (ParseResult.SessionUUID == session.ConversationID)
  2. Project path prefix match (ParseResult.ProjectPath is a prefix of session.Path)
  3. Timestamp proximity (file mod time within ±5 minutes of session.CreatedAt)

func (*Associator) AssociateWithSnapshot added in v1.44.0 

func (a *Associator) AssociateWithSnapshot(result *ParseResult, sessions []SessionRecord) (sessionID string, isOrphan bool)

AssociateWithSnapshot is Associate against a pre-fetched session snapshot (see Snapshot), instead of re-querying storage on every call.

func (*Associator) Snapshot added in v1.44.0 

func (a *Associator) Snapshot() []SessionRecord

Snapshot fetches the current session records once, for callers that need to call AssociateWithSnapshot for many results without re-querying storage per call. Both ListInstancesFiltered-style loops in InsightsService previously called Associate per result, each paying a fresh ListSessionRecords() -> ListInstanceData() full-repository scan.

type DollarImpact added in v1.52.0 

type DollarImpact float64

DollarImpact is a modeled USD estimate for one finding's waste. There is no exported Sum/Add helper anywhere in this codebase — dollar impacts across findings are not summable (a session-token-ceiling breach and a cache-hit-floor breach on the same session double-count the same wasted tokens from different angles). See ADR-002-findings-non-summable-dollar-impact.md. This is a review-convention guardrail, not a compiler guarantee: Go cannot forbid `+` on a numeric newtype.

type Finding added in v1.52.0 

type Finding struct {
	Type         FindingType
	Severity     Severity
	DollarImpact DollarImpact
	Message      string
}

Finding is one waste-pattern verdict for a single session, produced by a detector in findings.go and translated to the sessionv1.WasteFinding proto message by the caller (server/services/insights_service.go).

func ComputeFindings added in v1.52.0 

func ComputeFindings(r *ParseResult, pt *PricingTable) []Finding

ComputeFindings runs all 4 shipped detectors against r and returns whatever fires, in detector-declaration order (the caller, insights_service.go, sorts the request-wide accumulation by dollar impact separately). Each detector call is isolated with a recover() so one detector panicking on a malformed session can never take down the rest of the batch, or the caller's other sessions — matching the Observability Requirement that a computation error shows up as an empty/error state, not a page.

type FindingType added in v1.52.0 

type FindingType = sessionv1.FindingType

FindingType identifies which waste-pattern heuristic produced a Finding. Maps to session.v1.FindingType enum in Go — see proto/session/v1/types.proto:423's precedent for this alias pattern.

type ModelPricing 

type ModelPricing struct {
	ModelFamily        string  // normalized key, e.g. "claude-sonnet-4"
	InputPricePerMTok  float64 // USD per 1M input tokens
	OutputPricePerMTok float64 // USD per 1M output tokens
	CacheWritePerMTok  float64 // USD per 1M cache-write tokens
	CacheReadPerMTok   float64 // USD per 1M cache-read tokens
	EffectiveDate      string  // ISO date of last price update
}

ModelPricing holds per-model token prices in USD per million tokens.

type ParseResult 

type ParseResult struct {
	SessionUUID  string
	ProjectPath  string   // decoded from project dir name (best-effort)
	PrimaryModel string   // most-used model in this session
	Models       []string // all distinct models observed

	TotalInput    int64
	TotalOutput   int64
	CacheCreation int64
	CacheRead     int64
	MessageCount  int

	TurnTimeline     []TurnStats // per-assistant-message stats for burn rate chart
	ToolUsage        map[string]ToolTokenStats
	SkillActivations []SkillActivation

	ParsedAt    time.Time
	FileModTime time.Time // used for cache invalidation
}

ParseResult holds aggregated token data extracted from one JSONL file. Privacy: only tool names, skill names (short strings), and token counts. Message content is never stored.

type Parser 

type Parser struct{}

Parser parses Claude Code JSONL transcript files into ParseResult values.

func NewParser 

func NewParser() *Parser

NewParser creates a new Parser.

func (*Parser) ParseFile 

func (p *Parser) ParseFile(filePath string) (*ParseResult, error)

ParseFile reads a JSONL transcript file and returns an aggregated ParseResult. Malformed or truncated lines are skipped without returning an error. The caller must not retain message content — ParseResult only holds aggregates.

func (*Parser) ParseReader 

func (p *Parser) ParseReader(r io.Reader) (*ParseResult, error)

ParseReader parses JSONL from an io.Reader. Suitable for tests that pass in strings via strings.NewReader.

type PricingTable 

type PricingTable struct {
	Prices     map[string]ModelPricing
	LoadedAt   time.Time
	ConfigPath string // empty = hardcoded only
}

PricingTable maps normalized model family names to pricing. Hardcoded defaults; overridable via config JSON.

func DefaultPricingTable 

func DefaultPricingTable() *PricingTable

DefaultPricingTable returns a PricingTable with hardcoded defaults as of 2026-07-27. Prices are in USD per million tokens.

func LoadPricingOverride 

func LoadPricingOverride(configPath string) (*PricingTable, error)

LoadPricingOverride loads pricing from a JSON file and merges it over the hardcoded defaults. Unknown fields are ignored. The file must be a JSON object mapping model family names to ModelPricing objects.

func (*PricingTable) EstimateCost 

func (pt *PricingTable) EstimateCost(r *ParseResult) (cost float64, unpriced []string)

EstimateCost computes USD cost for a ParseResult using the PricingTable. Returns 0.0 for the cost of any model family not found in the table, and reports those skipped families in unpriced (sorted) so the caller can distinguish "genuinely zero usage" from "usage present but unpriced" — see ADR-001-unpriced-signal-return-shape.md. A family with zero usage across every counter (e.g. Claude Code's internal "<synthetic>" turns, which the parser already filters out of TurnTimeline in production — see parser.go's syntheticModelSentinel) is never flagged unpriced, since there is nothing to price and nothing to warn about.

func (*PricingTable) EstimateTurnCost added in v1.52.0 

func (pt *PricingTable) EstimateTurnCost(turn TurnStats) (cost float64, priced bool)

EstimateTurnCost computes USD cost for a single turn's token counts under its own model, mirroring EstimateCost's per-family arithmetic but at turn granularity (a turn has exactly one model, so no per-family map is needed). priced is false when turn.Model normalizes to a family absent from the PricingTable — callers must treat that as "unknown," never as a $0.00 cost.

func (*PricingTable) IsStale 

func (pt *PricingTable) IsStale() bool

IsStale returns true when any entry in the table has an EffectiveDate older than 30 days, indicating the pricing data may be outdated.

func (*PricingTable) LookupByModel 

func (pt *PricingTable) LookupByModel(modelID string) (ModelPricing, bool)

LookupByModel returns the ModelPricing for a raw model ID (normalizes first). Returns zero-value ModelPricing and false if not found.

func (*PricingTable) ModelFamilyCost 

func (pt *PricingTable) ModelFamilyCost(r *ParseResult) (costs map[string]float64, unpriced map[string]bool)

ModelFamilyCost returns a breakdown of estimated cost per model family, plus the set of families that had usage but no PricingTable entry (unpriced). A family with zero usage across every counter is never flagged unpriced — see EstimateCost's doc comment and ADR-001-unpriced-signal-return-shape.md for why.

type SessionRecord 

type SessionRecord struct {
	SessionID      string
	ConversationID string // matches ParseResult.SessionUUID
	Path           string // working directory
	CreatedAt      time.Time
}

SessionRecord is a minimal snapshot of a stapler-squad session used for matching against ParseResult values. This avoids importing the full session package and prevents circular dependencies.

type SessionStorage 

type SessionStorage interface {
	// ListSessionRecords returns a snapshot of all sessions for association.
	ListSessionRecords() []SessionRecord
}

SessionStorage is the interface Associator uses to look up sessions. Implemented by session.Storage (or a test stub).

type Severity added in v1.52.0 

type Severity = sessionv1.Severity

Severity classifies how urgently a Finding should be acted on. Maps to session.v1.Severity enum in Go, same alias pattern as FindingType.

type SkillActivation 

type SkillActivation struct {
	Name      string // e.g. "code-review", "/plan:feature"
	TurnIndex int    // which human turn triggered it
	IsCommand bool   // true for /command, false for skill name
}

SkillActivation records a detected skill or command invocation.

type TokenStore 

type TokenStore struct {
	// contains filtered or unexported fields
}

TokenStore caches parsed JSONL results keyed by file path. It pre-parses all JSONL files in a directory on startup and keeps the cache fresh via fsnotify callbacks.

func NewTokenStore 

func NewTokenStore(historyDir string) *TokenStore

NewTokenStore creates a TokenStore that will pre-parse all JSONL files in historyDir on startup.

func (*TokenStore) GetAll 

func (ts *TokenStore) GetAll() []*ParseResult

GetAll returns a snapshot of all cached ParseResult values under read lock.

func (*TokenStore) GetByUUID 

func (ts *TokenStore) GetByUUID(uuid string) *ParseResult

GetByUUID returns the ParseResult for a given conversation UUID, or nil.

func (*TokenStore) IsLoading 

func (ts *TokenStore) IsLoading() bool

IsLoading returns true while the background walk is still in progress.

func (*TokenStore) OnHistoryFileChanged 

func (ts *TokenStore) OnHistoryFileChanged(filePath string)

OnHistoryFileChanged is called by the HistoryFileWatcher callback when a file is created or modified. It enqueues the file for re-parsing.

func (*TokenStore) Start 

func (ts *TokenStore) Start(ctx context.Context)

Start launches background workers and the initial directory walker. It stops when ctx is cancelled. Call this once after creating the store.

func (*TokenStore) Stop 

func (ts *TokenStore) Stop()

Stop cancels the background context, stopping all goroutines.

func (*TokenStore) Subscribe 

func (ts *TokenStore) Subscribe() <-chan *ParseResult

Subscribe returns a channel that receives the changed file's *ParseResult whenever the store is updated by a single-file reparse, or nil when the initial directory walk completes. The caller should drain the channel promptly to avoid blocking notifications.

func (*TokenStore) Unsubscribe 

func (ts *TokenStore) Unsubscribe(ch <-chan *ParseResult)

Unsubscribe removes a subscriber channel.

type TokenStoreReader 

type TokenStoreReader interface {
	GetAll() []*ParseResult
	GetByUUID(uuid string) *ParseResult
	IsLoading() bool
	Subscribe() <-chan *ParseResult
	Unsubscribe(ch <-chan *ParseResult)
}

TokenStoreReader is the read-only interface InsightsService needs from a TokenStore. Defined as an interface so test fakes can be injected without constructing a real store.

type ToolTokenStats 

type ToolTokenStats struct {
	ToolName  string
	CallCount int
	// MCPServer is non-empty when tool follows mcp__<server>__<tool> pattern.
	MCPServer string
	// CostUsd is populated by AttributeToolCosts (tool-type-level session sum,
	// per ADR-001), not by the parser. Zero until a caller explicitly attributes
	// costs onto this struct.
	CostUsd float64
}

ToolTokenStats aggregates attribution for one tool name. Token attribution is message-level (not per-tool-call); CallCount is exact.

type TurnStats 

type TurnStats struct {
	Timestamp     time.Time
	Model         string
	Input         int64
	Output        int64
	CacheCreation int64
	CacheRead     int64
	ToolNames     []string // tool_use block names in this message
}

TurnStats is per-assistant-message token data (for timeline/burn-rate chart).

type WasteScore added in v1.52.0 

type WasteScore float64

WasteScore is a single sortable 0-100 badness number for a session, computed by ComputeWasteScore. It is NOT a sum of finding dollar impacts — it's a weighted blend of ratios (cache-hit shortfall, ceiling proximity, oversized start-context proximity). See ADR-002-findings-non-summable-dollar-impact.md.

func ComputeWasteScore added in v1.52.0 

func ComputeWasteScore(r *ParseResult, pt *PricingTable) *WasteScore

ComputeWasteScore returns a single sortable 0-100 badness number for a session, or nil when the session is too sparse to evaluate meaningfully (fewer than minTurnsForCacheFloor turns) — nil, not 0, so "not evaluated" is never confused with "evaluated and clean". This is a weighted blend of ratios, NOT a sum of finding dollar impacts — see ADR-002.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.