tokens

package
v1.2.13 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 11, 2026 License: MIT Imports: 25 Imported by: 0

Documentation

Overview

Package tokens is the accounting layer: the one message shape every source produces, the one fold over a stream of them, the day file, and the readers for the five declared source kinds.

The shape of the thing is one sentence. Each source hands the fold a stream of messages — {day, basis, model, repo, usage} — and the fold is ONE function over that stream, keyed exactly by (day, model, repo), with the five token types kept apart and a type no source reported left as a dash. What differs per source kind is only how the stream is produced, which is why the attribution rule, the dash rule and the sum rule are each written once here rather than once per reader: the prototype this replaces had two copies of the attribution table in two scripts and they disagreed about three repos.

Nothing in this package removes a file, and nothing in it runs a program except the one subprocess the OpenCode reader declares.

Index

Constants

View Source 
const (
	PartialBlended   = "blended"
	PartialCollision = "collision"
)

PartialBlended and PartialCollision are the two reasons a row of the file on disk can be neither kept nor recomputed by this fold.

View Source 
const (
	Unknown      = "unknown"
	Other        = "other"
	Unattributed = "unattributed"
)

The bucket names, which are repo names like any other and are never invented per source.

View Source 
const (
	KindClaude   = "claude"
	KindOpenCode = "opencode"
	KindSwarm    = "swarm"
	KindBus      = "bus"
	KindProvider = "provider"
)

The five source kinds, as `kind=` prints them.

View Source 
const BusDateLayout = "Mon Jan  2 15:04:05 UTC 2006"

BusDateLayout is how nova-bus writes a note's Date line.

View Source 
const Dash = "-"

Dash is the cell of a type the source did not report. It is not a zero: a zero is a measurement and a dash is an absence, and a zero that meant "not measured" would sum into a month claiming to be complete.

View Source 
const DefaultMaxClaudeFiles = 20000

DefaultMaxClaudeFiles is the ceiling one --claude tree starts under.

View Source 
const DefaultTimeout = 120 * time.Second

DefaultTimeout is the per-subprocess budget. It is the one flag with a default, for the reason SPEC.md gives nova-bus --git-timeout: it is how long the tool waits before saying so, not a fact about anybody's data.

View Source 
const FileSuffix = ".tsv"

FileSuffix is a day file's extension.

View Source 
const LockName = "fold.lock"

LockName is the lock file, inside the output directory: the thing being protected is that directory's day files, so the lock belongs beside them.

View Source 
const LockWait = 10 * time.Second

LockWait is how long a second fold waits before refusing. It is bounded on purpose: a wait with no end is a tool that has stopped saying anything.

View Source 
const SQLiteBinary = "sqlite3"

SQLiteBinary is the one program this family requires on PATH.

View Source 
const SessionLabel = "claude-session"

SessionLabel is the source label the row names, so every number stays traceable to the flag of the run that wrote it.

View Source 
const SubjectPrefix = "tokens "

SubjectPrefix opens every tokens note's subject, exactly: lower case, one blank.

View Source 
const TempSuffix = ".tsv.tmp"

TempSuffix is the fixed temp name of the legacy day-file temporary, a file shape check still steps over: a temporary of this shape is a live input, and `check` reads it as a step-over rather than reporting it as a stray. The writer goes through pkg/atomicfile and produces unique `.<day>.tsv.tmp-<rand>` siblings, so nothing this tool writes carries this plain suffix.

View Source 
const UTC = "utc"

UTC is the day basis of every row dated from a stamp. Anything else is the zone a provider's export declares for its own per-day totals, and a row that is not UTC says so in its own column rather than being quietly counted as one.

View Source 
const Version = "nova-tokens v1"

Version is the first token of every day file's first line. A file whose first line is not this is refused by `sum` and named by `check`, and the repair is `fold --day <d>`.

Variables

View Source 
var AllTypes = []Type{Input, Output, CacheWrite, CacheRead, Reasoning}

AllTypes is every type, for a source that reports all five.

View Source 
var ClaudeTypes = []Type{Input, Output, CacheWrite, CacheRead}

ClaudeTypes is what a Claude Code transcript carries: no reasoning count exists in it.

View Source 
var Columns = []string{"date", "model", "repo", "input", "output", "cache_write", "cache_read", "reasoning", "rough", "day_basis", "sources"}

Columns are the eleven, in order, and the file's second line is exactly these.

A day file written while a twelfth column, `units`, existed carries that column in its header and on every row. The reader takes that file and ignores the column (LegacyColumns); the writer never writes it, so the day after a fold the file is eleven wide. The version line is unchanged, `nova-tokens v1`: the eleven columns mean exactly what they always did.

View Source 
var DefaultWeights = Weights{Input: 1, CacheWrite: 1.25, CacheRead: 0.1, Output: 5}

DefaultWeights is a comparison, not a price: the ratios of one vendor's published list prices (cache write 1.25x an input token, cache read 0.1x, output 5x). The session verb's --weights flag carries these four numbers and takes yours.

View Source 
var HeaderLine = strings.Join(Columns, "\t")

HeaderLine is the second line of every day file this tool writes.

View Source 
var LegacyColumns = append(append([]string(nil), Columns...), "units")

LegacyColumns is the twelve-column header of a day file written while the `units` column existed. Only the reader knows it.

View Source 
var LegacyHeaderLine = strings.Join(LegacyColumns, "\t")

LegacyHeaderLine is that file's second line.

View Source 
var Parsers = []string{"google", "openai", "xai"}

Parsers are the export shapes this tool knows, one per provider. A label that is not one of them is a bad invocation rather than an unreadable file: the caller named a parser that does not exist.

View Source 
var SwarmColumns = []string{
	"job", "attempt", "from", "started", "ended", "end", "rc", "provider",
	"model", "repo", "tokens_in", "tokens_out", "cache_write", "cache_read", "reasoning", "usd",
}

SwarmColumns are the sixteen usage-file columns, in the order the swarm writes them: `job`, `attempt`, `from`, `started`, `ended`, `end`, `rc`, `provider`, `model`, `repo`, `tokens_in`, `tokens_out`, `cache_write`, `cache_read`, `reasoning`, `usd`. A header that is not these is refused by name. The reader looks every cell up BY NAME, so this list is the whole contract with the swarm.

View Source 
var TypeNames = [NTypes]string{"input", "output", "cache_write", "cache_read", "reasoning"}

TypeNames are the five names as a source, a bus line and a day file spell them.

Functions

func AllowedNote added in v1.2.3 

func AllowedNote(name string, dir bool) bool

AllowedNote is the allowlist: the non-day entries a person keeps beside their day files. It is deliberately a list of SHAPES and not of names — a tool with `README.md` compiled into it would be a tool that special-cases one bench's directory — and every shape here is one this repository's own reports/tokens holds: a README and a session note (`*.md`), a collator's log (`*.log`), and the archive of what the day files replaced (`pre-*/`).

func Applies 

func Applies(kind, field string) bool

Applies reports whether a Stat field is a measurement for this kind.

func BodyLine 

func BodyLine(day, who, model, repo string, t Type, count int64, basis string) string

BodyLine renders one body line, which is what fold --bus parses. Every stored field goes through oneline.Field for the reason the day file's cells do: six tab-separated fields is a promise, and a model id holding a tab would make it seven.

func DayOfStamp 

func DayOfStamp(stamp string) (string, bool)

DayOfStamp is the UTC day of an RFC 3339 stamp. The stamp is PARSED and converted, never sliced: its first ten characters are the day in whatever zone it was printed in, so a stamp offset from UTC can belong to a different UTC day. A stamp this tool cannot read is not a day, is not dated by a guess, and is not dropped: every caller counts and prints it.

func HaveSQLite 

func HaveSQLite() error

HaveSQLite reports whether the one required program is on PATH.

func HolderPID 

func HolderPID(path string) string

HolderPID is the pid in the lock file's holder stamp, or a dash when it holds none. It reads filelock's stamp and the bare pid an earlier binary wrote alike. It is read rather than scanned for: this tool never looks at another process's command line.

func KnownParser 

func KnownParser(name string) bool

KnownParser reports whether a --provider kind names a parser.

func Label 

func Label(kind, name string) string

Label is how a source names itself in a row's sources column and on its own line: the kind, a colon, and the flag's label — or, for a bus lane, the lane owner's name, whatever the `who` field of a line inside it says.

func MergeDay added in v1.2.3 

func MergeDay(old, fresh []DayRow, declared []string) (rows []DayRow, retained int, partials []Partial)

MergeDay merges this run's recomputed rows into the rows already in the day file.

declared is the set of source labels this run read, the same labels that land in a row's sources column. For each row of old:

  • every source outside declared: RETAINED, cell for cell, because no source this run read contributed to it and this run has nothing truer to say about it;
  • every source inside declared: REPLACED by this run's row for that (model, repo), and never summed with it, so no source is counted twice;
  • some inside and some outside: a Partial, PartialBlended. Its cells are already a sum over both and nothing on disk takes them apart.

A retained row and a recomputed row with the same (model, repo, unit) is a Partial too, PartialCollision: the day file's rows are unique by (model, repo, unit), and summing the two would blend two runs' arithmetic into one cell no later fold could undo.

The returned rows are sorted by (model, repo), which is what ParseDayFile demands. When any Partial is returned the caller writes NOTHING: the rows are what the merge would have been, and are not a file.

func MissingDays 

func MissingDays(days []string) []string

MissingDays is every day between the first and the last with no file. A missing day is NAMED and never filled: nobody folded it, and the tool will not invent what nobody measured.

func ParseDayFile 

func ParseDayFile(name, text string) (DayFile, []Finding)

ParseDayFile reads a day file and returns everything wrong with it rather than the first thing: a run over a month wants one line per finding, and a reader sent back once per problem is a reader sent back eight times.

name is the file's own name without its suffix, which the `date` column must equal.

func ParseMicro added in v1.2.3 

func ParseMicro(s string) (int64, bool)

ParseMicro reads a decimal dollar amount ("1.23", "0.000001", the int64 micro-dollars down to six fractional digits) into micro-dollars. Any other text -- a dash, an empty cell, an exponent that no usage column writes -- is the second return false, an absence rather than a zero.

func ParseSubject 

func ParseSubject(subject string) (parsedSubject, bool)

ParseSubject reads a tokens note's subject. The second return is false when the subject is not a tokens note's at all — a different case from a tokens note whose trailer is wrong, which is a note refused by name.

func ParserColumns 

func ParserColumns(kind string) []string

ParserColumns is the sorted list of column names one parser reads, for a refusal that says what the file WANTS and not only what was wrong.

func Path 

func Path(out, day string) string

Path is where a day file lives under an output directory.

func PathStem added in v1.2.3 

func PathStem(tok string) string

PathStem is the leading directory a token names, to stemDepth elements. It is what the unattributed tally counts, and it is written here rather than in the command so that the listing and the rule it suggests are cut from the same string.

func PathTokens 

func PathTokens(inputs []string) []string

PathTokens is every path-like token in the strings given: one starting with `/`, `~`, or a `<host>:<org>/<repo>` remote, and continuing over path characters.

func Percent 

func Percent(part, whole int64) string

Percent renders a bucket's share of a day to one decimal, so a day that is 40% unknown says so on the line a person reads.

func ReadDayFile 

func ReadDayFile(path string) (DayFile, []Finding, error)

ReadDayFile reads and parses one day file from disk.

func ReadNoSpendFile added in v1.2.3 

func ReadNoSpendFile(path string) (map[string]bool, error)

ReadNoSpendFile reads --no-spend: one YYYY-MM-DD per line, with a blank line and a `#` comment skipped and anything after the day on a line taken as its note, so the list can carry WHY a day had no spend. A line that is not a day is an error naming it, because a list half-read is a gate half-shut.

func SourcesOf added in v1.2.3 

func SourcesOf(rows []DayRow) []string

SourcesOf is the union of every row's sources, sorted: what the version line's `sources=` says about the file that was actually written, retained rows included.

func Subject 

func Subject(day, at, build string, supersedes []string) string

Subject renders a tokens note's subject: the date, and the tool's one trailer, which is where the stamp, the build id and a correction's predecessor set travel.

func TakeFoldLock 

func TakeFoldLock(out string, wait time.Duration) (func(), error)

TakeFoldLock takes the output directory's lock, waiting up to wait, and returns the release. The release is safe to call more than once.

func Usd added in v1.2.3 

func Usd(micro int64) string

Usd renders micro-dollars as dollars, trimming trailing zeros after the point: 1234001 is "1.234001" and 2000000 is "2". A negative cost keeps its sign.

func UsdPerMtok added in v1.2.3 

func UsdPerMtok(micro, tokens int64) string

UsdPerMtok is the average cost, in dollars per million tokens, to four decimals. Zero tokens is the dash, never a division: there is no average over nothing.

func ValidDay 

func ValidDay(s string) bool

ValidDay reports whether s is a YYYY-MM-DD day ON THE CALENDAR. The shape alone is not enough: an impossible day that passes the shape check would be written as a day file, pass `check`, and leave MissingDays walking from a day that does not exist. time.Parse is the range check, and the round trip refuses what it normalises (2026-02-30 -> 2026-03-02).

func ValidMonth added in v1.2.3 

func ValidMonth(s string) bool

ValidMonth reports whether s is a YYYY-MM calendar month.

func ValidNoteID 

func ValidNoteID(id string) bool

ValidNoteID reports whether an id has the shape nova-bus assigns. `report --supersedes` checks the shape and nothing else, because the lane is not on that machine.

func ValidZone 

func ValidZone(s string) bool

ValidZone reports whether s is a day_basis a day file may carry: a zone name with no whitespace, and never the word utc, which the six-field form already says.

Types

type Agg 

type Agg struct {
	Totals [NTypes]int64
	Dashes [NTypes]int
	Rows   int
	Rough  int
	NonUTC int
	// contains filtered or unexported fields
}

Agg is one grouping's totals.

func (*Agg) Cell 

func (a *Agg) Cell(t Type) string

Cell is one type's total as a month prints it: the number, or `-` when NO row in this grouping reported that type at all. A type the source did not report is `-` in the cell, never 0 -- a zero there would be summed into a month claiming to be complete -- and the dashes= tuple beside it is a correction a reader has to know the column order of.

func (*Agg) Days 

func (a *Agg) Days() int

Days is how many days fed this grouping.

func (*Agg) Keys 

func (a *Agg) Keys() int

Keys is how many distinct models or repos fed it.

type BadDayFile 

type BadDayFile struct{ Path, Reason string }

BadDayFile is a file under --out that `sum` will not read.

func (*BadDayFile) Error 

func (e *BadDayFile) Error() string

type CheckOptions added in v1.2.3 

type CheckOptions struct {
	// Strict names every calendar gap as missing and every non-day entry as a stray.
	Strict bool
	// NoSpend is the days a person has recorded as having had no spend, from
	// --no-spend. When it is non-nil, every gap day NOT in it is missing: the caller has
	// claimed to know the whole range, so a gap the list does not name is a day nobody
	// folded.
	NoSpend map[string]bool
	// Through is the day the ledger is expected to be current through, from --through.
	// When set, a last folded day before it marks the result Stale.
	Through string
}

CheckOptions is how a caller reads the two judgement calls above. The zero value is the default reading: gaps and notes are counted, not named.

type CheckResult 

type CheckResult struct {
	Files    int
	Rows     int
	First    string
	Last     string
	Findings []FileFinding
	// Missing is the days named as findings; Gaps is every calendar day in the range
	// with no file, whether or not it was named. Gaps is the larger fact, and is what
	// gap=<n> prints.
	Missing []string
	Gaps    []string
	// Strays is what is named; Notes is the non-day entries the allowlist stepped over.
	Strays []string
	Notes  []string
	Stale  bool
}

CheckResult is a whole output directory, checked.

func Check 

func Check(out string, opt CheckOptions) (*CheckResult, error)

Check walks every file under out. A day file is parsed; recognized day-file temporaries and the lock are stepped over, because the wreckage of a killed fold is not a stray; a non-day entry the allowlist admits is counted as a note; anything else is named and LEFT ALONE — this tool removes nothing, and a person removes a stray.

type ClaudeBound added in v1.2.3 

type ClaudeBound struct {
	MaxFiles int
	Exclude  []string
}

ClaudeBound is the ceiling and the excludes one --claude tree is walked under: at most MaxFiles transcript files, with every path matching an Exclude glob (or lying under one) left out. The zero value reads the whole tree, the behavior before the bound existed. DefaultMaxClaudeFiles is the ceiling the tool states in its help.

The largest plausible state is 3,000 transcript files a day (docs/SPEC-TOKENS.md rule 11), so the 20,000-file default sits far above a real tree and still stops the very large cost a temporary tree included by mistake produced for a cold reader.

type Conflict 

type Conflict struct {
	Label, Day string
	Notes      []string
}

Conflict is a lane-day with more than one tip: no winner is inferred, nothing folds.

type Counts 

type Counts struct {
	// contains filtered or unexported fields
}

Counts is the five types, each either a count or absent.

func (*Counts) Add 

func (c *Counts) Add(o Counts)

Add folds one Counts into another, per type, over the types the other reported.

func (Counts) Billed added in v1.2.3 

func (c Counts) Billed() int64

Total is the five types summed, for the shares on a day line. A dash adds nothing. Billed is the tokens a cost covers: input, output, cache write and cache read, the four types a provider prices. Reasoning is its own column and in none of them.

func (Counts) Cell 

func (c Counts) Cell(t Type) string

Cell is the day-file cell: the number, or a dash where nothing reported it.

func (Counts) Dashes 

func (c Counts) Dashes() int

Dashes is how many of the five cells are a dash.

func (Counts) Get 

func (c Counts) Get(t Type) (int64, bool)

Get is the count and whether any source reported it.

func (*Counts) Set 

func (c *Counts) Set(t Type, v int64)

Set records a measurement. A second Set on one type ADDS, which is how a row fed by two sources sums per type over the sources that reported that type. Saturates at math.MaxInt64 instead of wrapping (security#75 finding 1).

func (Counts) Total 

func (c Counts) Total() int64

type DayFile 

type DayFile struct {
	Day     string
	At      string
	Build   string
	Turns   string // a decimal integer, or a dash
	Sources []string
	Rows    []DayRow
}

DayFile is a whole day file, parsed or about to be written.

func (*DayFile) Render 

func (d *DayFile) Render() string

Render is the file's bytes.

EVERY STORED CELL GOES THROUGH oneline.Field, and that is what makes eleven tab-separated columns a promise rather than a hope: a model id or a repo name holding a tab would otherwise write a twelve-cell row, and one holding a newline would write two rows, from a value this tool copied out of somebody's transcript. Field escapes whitespace and "=", so an ordinary name is untouched and a hostile one is one token.

func (*DayFile) Save 

func (d *DayFile) Save(out string) error

Save writes the file whole atomically via pkg/atomicfile (exclusive temporary file beside target, explicit mode, fsync to media, atomic rename). Nothing is appended and nothing is edited in place. It refuses to write a file whose rendered count cell is neither a non-negative integer nor "-" (security#75 finding 1).

func (*DayFile) Totals 

func (d *DayFile) Totals() Counts

Totals is the day's per-type totals, and whether any row reported each type. It is what the shrink comparison compares.

type DayRow 

type DayRow struct {
	Date, Model, Repo string
	Counts            Counts
	Rough             int
	Basis             string
	Sources           []string
}

DayRow is one written row.

type FileFinding 

type FileFinding struct {
	Path   string
	Line   int
	Reason string
}

FileFinding is one thing wrong with one file.

type Finding 

type Finding struct {
	Line   int
	Reason string
}

Finding is one thing wrong with a file on disk, with the line it is on (0 for the file as a whole).

type Folder 

type Folder struct {
	// contains filtered or unexported fields
}

Folder is the fold: the one function over the stream, and the only place a row is made.

func NewFolder 

func NewFolder() *Folder

NewFolder returns an empty fold.

func (*Folder) Add 

func (f *Folder) Add(label string, m Message)

Add folds one message from the source named by label. A repeated id from another source of the same provider is recorded and kept: the write is refused later, the message is not dropped (TokenFold, OverlapRefusedBeforeWrite and UnscopedIDNotDeduped).

func (*Folder) DayRows 

func (f *Folder) DayRows(day string) (rows []*Row, mixed []Mixed)

DayRows is the day's rows, sorted by (model, repo), and separately the rows of that day that mixed two bases. A mixed row is in neither the file nor the counts.

func (*Folder) Days 

func (f *Folder) Days() []string

Days is every day the sources named, sorted.

func (*Folder) Overlaps 

func (f *Folder) Overlaps() []Overlap

Overlaps is every pair of sources that shared an id, sorted, so the remedy can name one.

func (*Folder) RefuseWrite added in v1.2.3 

func (f *Folder) RefuseWrite() bool

RefuseWrite reports a detected overlap. TokenFold invariant OverlapRefusedBeforeWrite: the fold is refused before any day file is written.

func (*Folder) Turns 

func (f *Folder) Turns(day string) (int, bool)

Turns is the message count for a day across the sources that count messages, and whether any of them fed it. A day fed by a bus note alone has none, and its version line says `-` rather than 0: a count nobody took is not a count of zero.

type Key 

type Key struct{ Day, Model, Repo string }

Key is exactly (day, model, repo). Nobody's name is in it: the `who` of a bus line and the window-or-child mark of a transcript are not columns, because a model on a repo on a day is one row whoever drove it.

type Message 

type Message struct {
	ID       string // the source's own message id, for the overlap check; not part of Key
	Day      string // YYYY-MM-DD
	Basis    string // UTC, or the zone a provider export declares
	Model    string
	Repo     string // already attributed by the reader, through repo.go's one function
	Counts   Counts
	Rough    int    // how many `~` bus lines this message stands for
	Turn     bool   // counted into turns= (the sources that count messages)
	Usd      int64  // micro-dollars, from a usage `usd` column or a cost tick; 0 where absent
	Priced   bool   // a source reported Usd for this message; false is "no cost given", never a zero cost
	Provider string // the provider prefix for model= on an AVG line; "" where unknown
}

Message is the one unit every source hands the fold.

type Mixed 

type Mixed struct {
	Key
	Bases  []string
	Labels []string
}

Mixed is a row fed by two day bases: not written, and named. Labels is WHICH sources fed it: "if a row mixed two day bases it names the two labels" (SPEC-TOKENS, the TOKENS NOTE paragraph). Telling a caller to declare one export for that day without saying which two are competing is the one remedy that names nothing.

type ModelSum 

type ModelSum struct {
	Model string
	Agg   *Agg
}

ModelSum is one model grouping.

type Overlap 

type Overlap struct {
	A, B string
	IDs  int
}

Overlap is two declared sources of one provider that fed the same message ids. TokenFold invariant OverlapRefusedBeforeWrite: it is a refusal before any day file is written, not a drop of either message.

type Pair 

type Pair struct {
	Model, Repo string
	Agg         *Agg
}

Pair is one (model, repo) grouping.

type Partial added in v1.2.3 

type Partial struct {
	Day, Model, Repo string
	Sources          []string // the row's own sources cell
	Folded           []string // the labels this run declared, sorted
	Why              string   // PartialBlended or PartialCollision
}

Partial is one row this fold can neither retain nor replace. It is a refusal, not a warning: nothing of the day is written while one stands.

type Row 

type Row struct {
	Key
	Counts Counts
	Rough  int
	Usd    int64 // micro-dollars summed over the messages that fed the row
	Priced bool  // some message that fed the row reported a cost; false: Usd is no measurement
	// PricedTokens is the Billed tokens of the messages that reported a cost: what Usd
	// covers, and the only tokens a rate over Usd may divide by.
	PricedTokens int64
	Provider     string // the provider prefix of the messages that fed the row; "" where unknown
	// contains filtered or unexported fields
}

Row is one line of a day file while it is still being accumulated.

func (*Row) Bases 

func (r *Row) Bases() []string

Bases is the day bases that fed this row, sorted. More than one is a row that is not written: a row must carry one day basis, and a mixed row is named and left out of the file and the counts.

func (*Row) Basis 

func (r *Row) Basis() string

Basis is the row's one basis, or the empty string when it has two.

func (*Row) Sources 

func (r *Row) Sources() []string

Sources is the sorted, comma-joinable labels that fed this row.

type Rules 

type Rules struct {
	// contains filtered or unexported fields
}

Rules is a rules file: lines of <name><TAB><regexp>, in priority order.

It optionally keeps the tally behind `sources --unattributed`: the path stems that were SEEN and matched no rule, which is the evidence a person needs to improve the file. `other=81%` on a day line says the rules are not good enough; it does not say which line to add, and until this tally existed nobody could find out without grepping transcripts by hand.

func LoadRules 

func LoadRules(path string) (*Rules, error)

LoadRules reads the caller's rules file. A malformed line is an error naming the line, because a rules file half-read is a repo column nobody can defend.

func (*Rules) Attribute 

func (r *Rules) Attribute(tokens []string, prev string) string

Attribute is THE statement of the rule, and every reader reaches a repo name through it: 

take every path-like token in the message's tool-call inputs
for each token, in order: the first rule whose regexp matches names the repo; stop
a token that matches no rule counts as "seen"
no token matched a rule, at least one token seen  -> other
no token at all                                   -> the stream's previous repo
no previous repo                                  -> unknown

prev is the stream's previous repo — per transcript file, per OpenCode session, and never across sources.

The `other` arm is also where the unattributed tally is taken, when a caller asked for it: the tokens that reached it are exactly the ones a better rules file would name.

func (*Rules) AttributeInputs 

func (r *Rules) AttributeInputs(inputs []string, prev string) string

AttributeInputs is Attribute over the raw strings of a message's tool-call inputs, with the path-like tokens pulled out of them first. A reader that carried its own regexp would be the second copy of the rule, so the extraction is here too.

func (*Rules) FilterDay added in v1.2.3 

func (r *Rules) FilterDay(day string)

FilterDay limits unattributed tallying to messages on the named day. When day is empty, all days are tallied.

func (*Rules) SetDay added in v1.2.3 

func (r *Rules) SetDay(day string)

SetDay sets the day of the message currently being attributed.

func (*Rules) TotalUnattributed added in v1.2.3 

func (r *Rules) TotalUnattributed() int

TotalUnattributed is every path token that fell to `other`, counted whether or not its stem got a place in the tally.

func (*Rules) Unattributed added in v1.2.3 

func (r *Rules) Unattributed() []UnattributedStem

Unattributed is the tally, heaviest first and then by stem, so two runs over one tree print the same list in the same order.

func (*Rules) WatchUnattributed added in v1.2.3 

func (r *Rules) WatchUnattributed()

WatchUnattributed turns the tally on. It is called before any source is read, and never by fold: `sources --unattributed` is a verb that only looks.

type SessionSum added in v1.2.3 

type SessionSum struct {
	Turns      int
	Input      int64
	CacheWrite int64
	CacheRead  int64
	Output     int64

	// Days is the per-day split, keyed by the UTC day of each turn's stamp. A session that
	// crosses midnight is two rows, never one row dated by the file.
	Days map[string]*SessionSum

	// Unstamped counts the turns whose stamp this tool could not read. They are in the
	// totals and in no day: a turn measured but not dated is named, never dropped and
	// never dated by a guess.
	Unstamped int

	// Models is every model the transcript names, sorted, and Unnamed counts the turns that
	// name none. A turn with no model cannot be booked under one, and is never booked under
	// a guess: the fold refuses while Unnamed is above zero.
	Models  []string
	Unnamed int

	// DayModels is the split of Days by the model of each turn: day, then model. A session
	// that changes model mid-window is one row per model, never one row under the last.
	DayModels map[string]map[string]*SessionSum
}

SessionSum is one session's usage: the turns, the four counts, and the days the turns fell on so a fold can write the right day file.

func ReadClaudeSession added in v1.2.3 

func ReadClaudeSession(path string) (SessionSum, error)

ReadClaudeSession sums one Claude Code session jsonl.

A STREAMED assistant message writes its id on many lines, each with a usage block that grows; the LAST line for an id is the message and every earlier one is a partial. So the sum is per message id, last write wins -- the same rule ReadClaude keeps -- and a reader that added every line would report a session several times its own size.

func (SessionSum) AvgContext added in v1.2.3 

func (s SessionSum) AvgContext() int64

AvgContext is the average context a turn carried: everything read (input, cache write, cache read) over the turns. It is the number that says whether the window is the cost, and a session with no turn has none.

func (SessionSum) DayList added in v1.2.3 

func (s SessionSum) DayList() []string

DayList is the days this session touched, sorted.

func (SessionSum) Line added in v1.2.3 

func (s SessionSum) Line(w Weights) string

Line is the one line the session verb prints, WEIGHTED under w.

func (SessionSum) Rows added in v1.2.3 

func (s SessionSum) Rows(day, role string) []DayRow

Rows is the day file rows one day of this session folds into: one per model the turns of that day name. The row names the model the TRANSCRIPT names and, when the caller's --role flag names a role, the seat too: `<model>/<role>`, with the role in the repo cell. The same model driving another line of work is a different row, because the question the ledger answers is what that work cost, not what the model cost; a window's turns are not one repo's work, and a row attributed to whichever repo a tool call happened to name would move the cost around from day to day, so with no role the model stands alone and the repo cell is the fixed word `unattributed`. A day the session has no turn on is a row of zeros for every model the session names, because "this window spent nothing that day" is a measurement; a session that names no model has no rows.

func (SessionSum) UnbookableReason added in v1.2.3 

func (s SessionSum) UnbookableReason() string

UnbookableReason is why this session cannot be folded into the ledger, or "" when it can: every row is booked under the model the transcript names, so a transcript that names none (or names it on only some turns) has no honest row. The caller refuses with this text.

func (SessionSum) Weighted added in v1.2.3 

func (s SessionSum) Weighted(w Weights) int64

Weighted is the fresh-input-equivalent under w: w.Input x input + w.CacheWrite x cache write + w.CacheRead x cache read + w.Output x output, rounded down to a whole token.

type Shrink 

type Shrink struct {
	Day  string
	Type Type
	File string // the number in the file
	Now  string // the number now, or a dash where the source went quiet
}

Shrink is one type that would go backwards.

func Shrinks 

func Shrinks(old, now Counts, day string) []Shrink

Shrinks compares what is on disk with what a fold has just computed. A type that is lower, or that was a number and is a dash now, is a source that went quiet, and a source that went quiet must never silently lower a day's spend.

A dash in the file that is a number now is NOT a shrink: that is coverage arriving.

type Source 

type Source struct {
	Label   string // as it is printed and as it appears in a row's sources column
	Kind    string
	Path    string
	Reports []Type
	Basis   string // utc, a zone, or "mixed" for a bus lane whose lines carry more than one
	Stat    Stat
	Stream  []Message

	Unreadables []Unreadable
	Unparseds   []Unparsed
	Supersededs []Superseded
	Conflicts   []Conflict
	Toucheds    []Touched
	// contains filtered or unexported fields
}

Source is one declared source after it has been read.

func ReadBus 

func ReadBus(dir string, fsys fs.FS, rules *Rules, at time.Time) []*Source

ReadBus reads every lane the roster names and returns ONE source per lane, because the label of a self-report is the lane owner's name whatever the `who` field of a line inside it says. The lanes are read through the caller's fs.FS, rooted at dir (os.DirFS(dir) in main, fstest.MapFS in a test), and dir is the path the report names.

func ReadClaude 

func ReadClaude(label, dir string, fsys fs.FS, rules *Rules, bounds ...ClaudeBound) *Source

ReadClaude walks a transcript tree and returns its stream and its accounting.

Every message carries no unit, which the fold writes as `-`: a transcript names a repo, never a piece of work. The tree is read through the caller's fs.FS, rooted at dir (os.DirFS(dir) in main, fstest.MapFS in a test), and dir is the path the report names.

The walk runs to completion BEFORE the first file is opened, so a tree over MaxFiles is refused with the totals it found and not one byte of it is read (SPEC-TOKENS rule 11: the state is bounded, and a source is a claim that the report covers it). An Exclude glob is the remedy for the temporary tree that made the tree overrun.

func ReadOpenCode 

func ReadOpenCode(label, dbPath, scratch string, timeout time.Duration, rules *Rules) *Source

ReadOpenCode copies the database into scratch and reads it there.

func ReadProvider 

func ReadProvider(kind, name, path string, _ *Rules) *Source

ReadProvider reads one billing export with the parser its kind names. The label on every row it feeds is `<kind>:<name>`, a per-account label, as the day-file example writes it, so two exports from one provider are two sources.

func ReadSwarm 

func ReadSwarm(label, pool string, fsys fs.FS, rules *Rules) *Source

ReadSwarm reads every <pool>/usage/*.tsv and nothing else. The pool is read through the caller's fs.FS, rooted at pool (os.DirFS(pool) in main, fstest.MapFS in a test), and pool is the path the report names.

func (*Source) AddMessage 

func (s *Source) AddMessage(id string, m Message)

AddMessage puts one message into this source's stream, collapsed onto its id.

A streamed transcript repeats a message id on every line; the last line for an id carries the message's final usage, and that is the one counted. Within one source, a second occurrence of an id is dup=<n>, never a second count. A message with no id is counted in noid=<n> and not folded. The rule is written ONCE, here, and every reader collapses through it: two copies of one rule are two rules.

func (*Source) Collapse 

func (s *Source) Collapse()

Collapse lays the collapsed messages into Stream, in first-seen order, and counts them. Every reader that calls AddMessage ends with it.

func (*Source) ReportsList 

func (s *Source) ReportsList() string

ReportsList is the comma-joined type names this source reports at all, so a reader of a mixed row can see which source could not have covered which cell.

func (*Source) StatField 

func (s *Source) StatField(field string) string

StatField renders one Stat field for the source line: the number, or a dash where the field is not a measurement for this kind.

type Stat 

type Stat struct {
	Files      int
	Unreadable int
	Messages   int
	Dup        int
	NoID       int
	NoUsage    int
	Unparsed   int
	Comments   int
	Redated    int
	Superseded int
	Rows       int
}

Stat is what one declared source yielded. A field that does not apply to the kind prints a dash rather than a zero: a transcript has no unparsed lines and a bus lane has no duplicate ids, and a dash is an absence where a zero is a measurement.

type Sum 

type Sum struct {
	Month    string
	Days     []string
	Missing  []string
	Rows     int
	Turns    int
	HaveTurn bool
	Pairs    []Pair
	Models   []ModelSum
	Total    *Agg
}

Sum is a whole month.

func SumMonth 

func SumMonth(out, month string) (*Sum, error)

SumMonth walks <out>/<month>-??.tsv in name order. A file whose stamp line is not `nova-tokens v1` is an error rather than a finding: `sum` reads day files, and a file that is not one is not a thing it can answer about.

type Superseded 

type Superseded struct{ Label, Note, By, Day string }

Superseded is a bus note a later note in the same lane replaced by name.

type Touched 

type Touched struct {
	Label, Day string
	Repos      []string
}

Touched is the one comment shape a friend's note may carry, and it adds no numbers.

type Type 

type Type int

Type is one of the five token types. They are kept apart everywhere: nothing in this package folds one into another, and a provider that does not expose reasoning or cache usage is not evidence that none occurred. 

const (
	Input Type = iota
	Output
	CacheWrite
	CacheRead
	Reasoning
	NTypes
)

The five, in the order every row prints them.

func TypeByName 

func TypeByName(name string) (Type, bool)

TypeByName resolves one of the five names.

type UnattributedStem added in v1.2.3 

type UnattributedStem struct {
	Stem  string
	Count int
}

UnattributedStem is one path stem no rule named, with how many path tokens fell onto it.

type Unparsed 

type Unparsed struct {
	Label, Note string
	Line        int
	Text        string
	// Remedy, when set, is the one act for THIS unparsed rather than for its kind: a
	// near-miss subject is a bus unparsed, and "a body line is date<TAB>who..." is not
	// what its writer has to change.
	Remedy string
}

Unparsed is a bus line, a whole bus note, or an export the parser could not read.

type Unreadable 

type Unreadable struct {
	Label, Path, Why string
	Line             int
}

Unreadable is a declared source the tool could not read: counted, printed on its own line, and the reason the run exits 1, because a declared source is a claim that the report covers it. Line is the 1-based first line that could not be read (a badline); 0 when the failure is not a line, so a permission problem is not given a line number.

type Weights added in v1.2.3 

type Weights struct{ Input, CacheWrite, CacheRead, Output float64 }

Weights is the four ratios the WEIGHTED equivalent is built from: how many fresh input tokens one input, one cache write, one cache read and one output token are worth.

func ParseWeights added in v1.2.3 

func ParseWeights(s string) (Weights, error)

ParseWeights reads the in,cw,cr,out four numbers --weights takes. A value that is not four finite numbers is a refusal saying what the flag WANTS, never a guess (docs/STANDARD.md section 3, onboarding point 2).

func (Weights) Flag added in v1.2.3 

func (w Weights) Flag() string

Flag renders w as the in,cw,cr,out four comma-separated numbers the session verb's --weights flag takes, so the flag's default is DefaultWeights printed and never a second copy of it.

type XaiUsageMissingError added in v1.2.3 

type XaiUsageMissingError struct {
	Path string
}

XaiUsageMissingError is the refusal when --provider xai names a path that is not there. The path is the one the flag gave. No other path is opened.

func (*XaiUsageMissingError) Error added in v1.2.3 

func (e *XaiUsageMissingError) Error() string

func (*XaiUsageMissingError) Unwrap added in v1.2.3 

func (e *XaiUsageMissingError) Unwrap() error

type XaiUsageNotFileError added in v1.2.3 

type XaiUsageNotFileError struct {
	Path string
}

XaiUsageNotFileError is the refusal when --provider xai names something other than one regular file. A directory is not walked for usage.json files.

func (*XaiUsageNotFileError) Error added in v1.2.3 

func (e *XaiUsageNotFileError) Error() string

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.