records

package
v1.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 28, 2026 License: MIT Imports: 14 Imported by: 0

Documentation

Overview

Package records is the core of the retained token record format: the canonical envelope, exact numeric strings with presence kept, and the structural validation the publisher boundary performs.

The contract is docs/PROPOSAL-TOKENS-FORMAT.md and docs/PROPOSAL-TOKENS-RECORDS.md at 71ea08b (Stella's #124). Three sentences carry most of this package:

  • An object envelope is exactly {"id":"sha256:<64 lowercase hex>","body":{...}}, and the ID is SHA-256 of the body's RFC 8785 canonical JSON with no trailing newline. The envelope ID is excluded from its own digest because only the body is hashed.
  • Usage values are JSON strings. A float64 round trip is never performed on them, so an integer above 2^53 and a decimal lexeme survive byte-for-byte. Presence is present, absent or unavailable; absent and unavailable require value:null and a bounded reason code; a present zero survives as a present zero.
  • Validation here is STRUCTURAL. A record that fails is refused with the field named and is never repaired: no key is dropped, no value normalised, no missing field defaulted. A refusal is a question for the table, not a thing this package fixes.

What this package deliberately is not: it has no adapters, no batch or path joins, and no publication. Those are separate owners (see the PR that introduced this package).

Index

Constants

View Source
const (
	SchemaObservation = "nova.tokens.observation/2"
	SchemaMapping     = "nova.tokens.mapping/2"
	SchemaCoverage    = "nova.tokens.coverage/2"
)

Schema constants for the three retained record bodies.

View Source
const (
	RuleInvalidUTF8                 = "invalid_utf8"
	RuleLoneSurrogate               = "lone_surrogate"
	RuleNotJSON                     = "not_json"
	RuleDuplicateKey                = "duplicate_key"
	RuleRawJSONNumber               = "raw_json_number"
	RuleEnvelopeShape               = "envelope_shape"
	RuleEnvelopeIDSyntax            = "envelope_id_syntax"
	RuleEnvelopeIDMismatch          = "envelope_id_mismatch"
	RuleUnknownSchema               = "unknown_schema"
	RuleUnknownField                = "unknown_field"
	RuleMissingField                = "missing_field"
	RuleWrongType                   = "wrong_type"
	RuleUnknownSourceKind           = "unknown_source_kind"
	RuleUnknownEnum                 = "unknown_enum"
	RuleEmptyString                 = "empty_string"
	RuleLabelSyntax                 = "label_syntax"
	RuleNamespaceSyntax             = "namespace_syntax"
	RuleContentIDSyntax             = "content_id_syntax"
	RuleEmptyArray                  = "empty_array"
	RuleNotSorted                   = "not_sorted"
	RuleDuplicateElement            = "duplicate_element"
	RuleTimestampSyntax             = "timestamp_syntax"
	RuleTimestampNoOffset           = "timestamp_no_offset"
	RuleIntervalIncomplete          = "interval_incomplete"
	RuleIntervalOrder               = "interval_order"
	RuleUnknownPresence             = "unknown_presence"
	RuleUnknownNumberKind           = "unknown_number_kind"
	RuleUnitSyntax                  = "unit_syntax"
	RulePresentValueNull            = "present_value_null"
	RulePresentWithReason           = "present_with_reason"
	RuleAbsentValueNotNull          = "absent_value_not_null"
	RuleAbsentReasonMissing         = "absent_reason_missing"
	RuleUnknownReasonCode           = "unknown_reason_code"
	RuleIntegerLexeme               = "integer_lexeme"
	RuleDecimalLexeme               = "decimal_lexeme"
	RuleNegativeValue               = "negative_value"
	RuleFieldNotAllowlisted         = "field_not_allowlisted"
	RuleReceiptNotAllowed           = "receipt_field_not_allowlisted"
	RuleMappingMissingZeroSemantics = "mapping_missing_zero_semantics"
	RuleMappingRuleShape            = "mapping_rule_shape"
	RuleShardReference              = "shard_reference"
	RuleCoverageReasonSource        = "coverage_reason_source"
)

The rule names. Every refusal this package can produce names exactly one of them, and every one of them has a fixture under testdata/refused that fails only that rule. The names are stable: a refusal is reported to a person and quoted back in a note.

Variables

AllRules is every rule name, for the test that proves each one has a fixture and is load-bearing. A rule added without a fixture fails that test rather than shipping unexercised.

View Source
var ErrSealed = errors.New("records: this object came from a validated envelope and is sealed; its digest is an identity somebody holds, so build a new body with NewObject")

ErrSealed is what the exported Set answers on an object that came from a validated envelope. It is a plain error and not a Refusal on purpose: a Refusal names one of the Rule* constants, every one of those owes testdata a fixture that fails only that rule, and this is an API misuse rather than a defect in a record's bytes.

View Source
var ReasonCodes = []string{
	"not_supplied", "not_supported_by_source", "source_unavailable",
	"unsupported_semantics", "parse_failed", "redacted", "omitted_from_wire",
}

ReasonCodes is the bounded set an absent or unavailable field may give. Bounded is the requirement the format states; these are the seven distinctions the documents draw between a field a source never had, one it had and would not give, and one this collection could not read.

View Source
var SourceKinds = []string{
	"antigravity", "claude_code", "codex_desktop", "grok", "nova_swarm", "opencode", "v1_aggregate",
}

SourceKinds is the allowlist of source.kind values, and it is exactly the harness shapes the two proposal documents name at 71ea08b -- claude_code and opencode and nova_swarm from the v1 readers and the wrapper-overlap paragraph, codex_desktop and antigravity and grok from the counter-semantics paragraph, v1_aggregate from "V1 imports ... are labelled aggregate observations". A kind outside it is a bad declaration by the caller rather than an unreadable record, the same distinction internal/tokens draws for a --provider label that names no parser.

Functions

func Canonicalize

func Canonicalize(v Value) ([]byte, error)

Canonicalize writes the RFC 8785 canonical JSON of a parsed value: members sorted by the UTF-16 code units of their names, no insignificant whitespace, and the minimal string escaping ECMAScript's JSON.stringify produces. There is no trailing newline, because the digest is over exactly these bytes.

RFC 8785's number serialisation is deliberately absent. This format's bodies carry no raw JSON numbers at all -- every usage value is a string, so no lexeme ever makes a float64 round trip -- so a number reaching here is refused by parseStrict before it can reach the digest. That removes the shortest-round-trip float printing that is the one genuinely hard corner of RFC 8785, and it removes it by contract rather than by hope.

func ContentID

func ContentID(v Value) (string, error)

ContentID is the identity the format names: "sha256:" and 64 lowercase hex digits of the SHA-256 of the value's canonical JSON. Applied to a body it is the envelope's ID, which is therefore excluded from its own digest by construction: nothing but the body is hashed, so there is no fixed point to solve and no circular dependency to break.

func CoverageMembers added in v0.15.0

func CoverageMembers() []string

CoverageMembers returns a copy of the 14 members of nova.tokens.coverage/2.

func KnownZeroSemantics

func KnownZeroSemantics(z ZeroSemantics) bool

KnownZeroSemantics reports whether a mapping's declared value is one of the three.

func MappingMembers added in v0.15.0

func MappingMembers() []string

MappingMembers returns a copy of the 12 members of nova.tokens.mapping/2.

func ObservationMembers added in v0.15.0

func ObservationMembers() []string

ObservationMembers returns the twelve members of a nova.tokens.observation/2 body, in the format's table order. It returns a COPY: writing through the result changes nothing, so a caller can read the schema's shape without being able to widen or shrink what the validator enforces.

func Seal

func Seal(body Value) ([]byte, string, error)

Seal returns the envelope bytes for a body: the ID derived from the body's canonical JSON, and the body itself in canonical form. It is the only writer here, and it writes no trailing newline.

func SealCoverage added in v0.15.0

func SealCoverage(c Coverage) ([]byte, string, error)

SealCoverage seals a Coverage record and strictly validates its own bytes.

func SealMapping added in v0.15.0

func SealMapping(m Mapping) ([]byte, string, error)

SealMapping seals a Mapping and strictly validates its own bytes.

Types

type Allowlists

type Allowlists struct {
	RawUsageFields []string
	ReceiptFields  []string
}

Allowlists carries the two sets that belong to a MAPPING rather than to the format: the source field names raw_usage may carry, and the native locator fields a receipt may carry. The format calls both "mapping-allowlisted", so core cannot know them; it enforces that something named them. An empty set means no field is allowed, which is the safe direction: a record whose mapping was not supplied is refused, not accepted whole.

type Coverage added in v0.15.0

type Coverage struct {
	Schema          string
	ScopeID         string
	SourceIDs       []string
	Interval        CoverageInterval
	Status          string
	Reasons         []CoverageReason
	CollectedAt     string
	CollectorBuild  string
	CollectorFriend *string
	CollectionBench *string
	MappingIDs      []string
	Shards          []ShardRef
	Predecessors    []string
	Counts          CoverageCounts
}

Coverage is a validated nova.tokens.coverage/2 body.

func (Coverage) Body added in v0.15.0

func (c Coverage) Body() (Value, error)

Body encodes the coverage record to canonical JSON value representation.

type CoverageCounts added in v0.15.0

type CoverageCounts struct {
	SourceCandidates string
	RecordsEmitted   string
	Observations     string
	Conflicts        string
	Gaps             string
}

CoverageCounts defines the five non-negative integer string counts.

type CoverageInterval added in v0.15.0

type CoverageInterval struct {
	Start string
	End   string
}

CoverageInterval defines the interval start and end timestamps.

type CoverageReason added in v0.15.0

type CoverageReason struct {
	Code   string
	Source *string
}

CoverageReason defines one reason entry.

type Envelope

type Envelope struct {
	ID          string
	Body        *Object
	Observation *Observation
	Mapping     *Mapping
	Coverage    *Coverage
}

Envelope is a validated {"id":..., "body":...}.

type Mapping added in v0.15.0

type Mapping struct {
	Schema           string
	Name             string
	Revision         string
	SourceShapes     []string
	FieldRules       map[string]MappingFieldRule
	IdentityRule     MappingIdentityRule
	RevisionRule     MappingRevisionRule
	TimeRule         MappingTimeRule
	ModelRule        MappingModelRule
	OverlapRule      MappingOverlapRule
	FixtureDigests   map[string]string
	ImplementationID *string
}

Mapping is a validated nova.tokens.mapping/2 body.

func (Mapping) Body added in v0.15.0

func (m Mapping) Body() (Value, error)

Body encodes the mapping to canonical JSON value representation.

type MappingFieldRule added in v0.15.0

type MappingFieldRule struct {
	NumberKind      string
	ZeroSemantics   string
	Unit            string
	SpendRole       string
	AbsentPresence  string
	AbsentReason    string
	InvalidPresence string
	InvalidReason   string
}

MappingFieldRule defines the 8 members of each numeric source field rule.

type MappingIdentityRule added in v0.15.0

type MappingIdentityRule struct {
	SourceKind               string
	Namespace                string
	EventKey                 []string
	ObservationKind          string
	NormalizedSpendSupported bool
	ProducerVersionFrom      string
	SessionID                string
	ReceiptFields            []string
	ReceiptValueType         string
	// Codex-specific:
	ContainingOrForkedSessionInEventKey *bool
	// Grok-specific:
	CollectionTimestampSubstitution *bool
	ContainingSessionSubstitution   *bool
	UnsupportedReason               *string
}

MappingIdentityRule defines the closed identity rule for reviewed adapters.

type MappingMissingTimestamp added in v0.15.0

type MappingMissingTimestamp struct {
	Basis      string
	OccurredAt *string
}

MappingMissingTimestamp represents {basis, occurred_at}.

type MappingModelBasisID added in v0.15.0

type MappingModelBasisID struct {
	Basis string
	ID    *string
}

MappingModelBasisID represents nested {basis, id}.

type MappingModelRule added in v0.15.0

type MappingModelRule struct {
	ForbiddenWireKeys []string
	// Codex-specific:
	Basis                              *string
	ID                                 *string
	Absent                             *MappingModelBasisID
	LaterModelRelabelsEarlierResponses *bool
	ModelUsage                         []string
	// Grok-specific:
	SingleReportedID              *MappingModelBasisID
	AbsentID                      *MappingModelBasisID
	MultipleModelIDs              *MappingModelBasisID
	SplitNormalized               *bool
	AggregateIsCountingCandidate  *bool
	DetailRetainedNotCountedAgain *bool
	ModelUsageFields              []string
	ModelUsageSortedBy            *string
}

MappingModelRule defines model rule members.

type MappingOverlapRule added in v0.15.0

type MappingOverlapRule struct {
	OwedCoverageTasks []string
	// Codex-specific:
	CountingSource                            *string
	ArithmeticMismatch                        *string
	MissingRawTotal                           *string
	ThreadTokenUsage                          *string
	TurnTokenUsage                            *string
	TokenCountSnapshots                       *string
	SummedOrMixedWithPreferred                *bool
	ReportMustNameUnhandledTokenCountCoverage *bool
	// Grok-specific:
	CountingGrain                                     *string
	ArithmeticViolation                               *string
	EvidencedOverlappingGrain                         *string
	RequestGrainMapping                               *string
	SessionTotals                                     *string
	ForbiddenUnions                                   []string
	ChecksumRequiresEvidencedCompleteMatchingCoverage *bool
	SessionTotalsFillMissingTurnSpend                 *bool
}

MappingOverlapRule defines overlap rule members.

type MappingRevisionRule added in v0.15.0

type MappingRevisionRule struct {
	Basis                             string
	Native                            *string
	Supersedes                        []string
	IdenticalCopy                     string
	ChangedSameKey                    string
	InferredOrderFromExportUpdateTime bool
	NewestWins                        bool
}

MappingRevisionRule defines the 7 revision rule members.

type MappingTimeRule added in v0.15.0

type MappingTimeRule struct {
	Basis            string
	OccurredAt       string
	Offsets          string
	DayAllocation    string
	Start            *string
	End              *string
	MissingTimestamp *MappingMissingTimestamp
	// Grok-specific:
	CrossMidnight *string
}

MappingTimeRule defines time rule members.

type Measurement

type Measurement struct {
	// Measured is true only when the raw evidence supports a measurement.
	Measured bool
	// SubsetComplete is the format's "subset completeness": false whenever a present zero
	// might be a synthesised default rather than an observation.
	SubsetComplete bool
	// Why is a fixed phrase for a diagnostic, never source data.
	Why string
}

Measurement is what a normaliser may believe about one raw field once the mapping's zero_semantics is applied. It is NOT a number: producing one is normalisation, which is a separate owner. This is the presence-preserving half, and it is here because getting it wrong is how a "not measured" becomes a zero that sums into a month claiming to be complete.

func Normalize

func Normalize(f RawField, z ZeroSemantics) Measurement

Normalize applies field_rules.zero_semantics exactly as the format writes it:

  • "Missing/unavailable raw fields are never normalized to measured 0." An absent or unavailable field yields no measurement at all, whatever the zero rule says.
  • "For a present zero under either latter rule, normalized measurement is unknown and subset completeness is false." A present zero is measured only under `measured`.
  • "a positive value is normalized only when its other counter semantics are supported" -- that second gate belongs to the mapping and is the caller's, so a positive value is reported as measured here and the caller withholds it if its semantics are not.

A mapping that declares no zero rule for the field is treated as `unknown` and said so; it is not treated as measured.

type Model

type Model struct {
	ID    *string
	Basis string
}

Model is the source model ID with the basis that says how well it is known.

type ModelUsage

type ModelUsage struct {
	ModelID  string
	RawUsage map[string]RawField
}

ModelUsage is one entry of the source-provided per-model split.

type Object

type Object struct {
	// contains filtered or unexported fields
}

Object is a JSON object with its keys in source order.

func NewObject added in v0.15.0

func NewObject() *Object

NewObject returns an empty object to build a body in. It is the only exported way to make one, and it exists so a caller can construct the Value that Canonicalize, ContentID and Seal have always taken.

func (*Object) Get

func (o *Object) Get(k string) (Value, bool)

Get returns a member and whether it was present. Present-with-null and absent are different answers, which is the whole point of the presence rules further down.

On a SEALED object -- one the strict parser built, so one whose digest is an identity somebody holds -- an array member is returned as a copy, for the same reason: the read measured on this API was `src.Get("event_key").([]Value)[0] = sentinel`. A nested *Object needs no copy because it is sealed too and defends itself. An object a caller built with NewObject is theirs, and Get hands back exactly what they put in.

func (*Object) Keys

func (o *Object) Keys() []string

Keys returns the member names in source order, as a COPY.

It used to return o.keys itself, and that made the seal below a half-seal: a whole-PR read of #146 measured `env.Body.Keys()[0] = sentinel` moving ContentID(env.Body) away from env.ID with no error, which is the live-handle shape one method over from the exported member list. A reader of a validated body gets the names, not the array they live in.

func (*Object) Set added in v0.15.0

func (o *Object) Set(name string, v Value) error

Set adds one member and returns a refusal rather than overwriting.

A name already present is RuleDuplicateKey, the same rule the strict parser applies to the bytes: a writer that kept the last of two would produce a body its own reader refuses, and the two sides of one grammar would disagree about a record that already had an ID. The first value stands; a refused Set changes nothing.

A value with no wire form -- a Go map, a struct, an int -- is refused HERE, at the member, rather than three levels down in the canonicaliser as "the value has no canonical form" about a path the caller cannot place.

A refusal names the member's ORDINAL and never the name itself, the way exactKeys, rawUsage and receipt all report indexPath(path, i). The name is the caller's string: it can hold a prompt or a private path, a carriage return that renders one refusal as two lines, or a forged "refused: ..." token, and a diagnostic is a shared file too (#146's adversarial read, finding 3: this was the one place in the package where a caller's string reached a rendered refusal).

An object that came from a validated envelope is sealed and answers ErrSealed: its digest is already an identity. An object from NewObject is the caller's own and stays writable.

type Observation

type Observation struct {
	Schema     string
	Source     Source
	Kind       string
	Revision   Revision
	Time       Times
	Origin     Origin
	Model      Model
	Repository Repository
	RawUsage   map[string]RawField
	ModelUsage []ModelUsage
	MappingID  string
	Receipt    map[string]string
}

Observation is a validated nova.tokens.observation/2 body.

func (Observation) Body added in v0.15.0

func (o Observation) Body() (Value, error)

Body returns the observation's body, the value Seal and ContentID take. It is the inverse of the validator's read: every member the schema names is written, present even when null, and nothing else is.

type Origin

type Origin struct {
	Friend    *string
	Bench     *string
	Basis     string
	BindingID *string
}

Origin is whose work and where it ran. The collector host never supplies it.

type RawField

type RawField struct {
	Presence   string  // present, absent, unavailable
	Value      *string // the original lexeme; nil exactly when presence is not present
	NumberKind string  // integer or decimal
	Unit       string
	Reason     *string // a bounded code; nil exactly when presence is present
}

A RawField is one allowlisted source usage field as it was observed. The value is the ORIGINAL lexeme, kept as a string: an integer above 2^53 and a decimal with trailing digits both survive byte-for-byte because nothing here parses them into a float64.

func (RawField) IsZero

func (f RawField) IsZero() bool

IsZero reports a present zero by its lexeme, not by arithmetic. "0", "0.0" and "0e0" are all zero; the comparison is over the decimal value the lexeme names, and no other lexeme is rewritten in the process.

func (RawField) Present

func (f RawField) Present() bool

Present reports whether the wire carried the field at all. A present zero is present: "Raw present zero survives even when measurement semantics are unknown."

type Refusal

type Refusal struct {
	Rule  string // one of the Rule* constants
	Field string // dotted path into the envelope, e.g. body.raw_usage.input.value
	Note  string // a fixed phrase, never source data
}

A Refusal is the only failure this package produces. It names the rule and the field, and it deliberately does NOT carry the offending value: the format says to report the field name and location without echoing its value, because a prompt or a private path can sit in an unsupported source field and a diagnostic is a shared file too.

func (*Refusal) Error

func (r *Refusal) Error() string

Error renders the one line. It cleans again rather than trusting the fields, because Refusal is an exported struct: a caller can build one with a newline in it, and Error is the render point that cannot be bypassed.

type Repository

type Repository struct {
	ID       *string
	Basis    string
	PolicyID *string
	Touched  []string
}

Repository is the repository identity and the evidence that assigned it.

type Revision

type Revision struct {
	Native     *string
	Supersedes []string
	Basis      string
}

Revision is revision: the predecessors this observation supersedes and why.

type ShardRef added in v0.15.0

type ShardRef struct {
	ShardID       string
	RecordCount   string
	InlineIDs     []string
	InventoryFile *string
}

ShardRef defines a referenced shard and its disjoint tagged inventory.

type Source

type Source struct {
	Kind            string
	ProducerVersion *string
	Namespace       string
	SessionID       string
	EventKey        []string
}

Source is source: the declared shape and the identity within it.

type Times

type Times struct {
	OccurredAt *string
	Start      *string
	End        *string
	Basis      string
}

Times keeps the ORIGINAL offset lexemes. Reporting converts known instants to UTC; this package does not, because a converted timestamp has lost the offset that said which day the source meant.

type Validator

type Validator struct {
	// contains filtered or unexported fields
}

A Validator performs the publisher boundary's structural validation. It holds the mapping-supplied allowlists and nothing else: it opens no file, resolves no path and loads no code, which is the property the format asks of the publisher.

func NewValidator

func NewValidator(a Allowlists) *Validator

NewValidator returns a validator over one mapping's allowlists.

The two slices are COPIED. Finding 1 of #146's adversarial read was the schema's closed set held in mutable state a caller could reach; a validator that kept the caller's own backing arrays is the same shape one level down -- what the boundary enforces would change under it whenever the caller appended to the slice it had handed over, including between a SealObservation's seal and its strict read-back. The allowlist is the caller's DECLARATION, made once when the validator is built; it is not a live handle.

func (*Validator) SealObservation added in v0.15.0

func (v *Validator) SealObservation(o Observation) ([]byte, string, error)

SealObservation is the one call an adapter needs: the envelope bytes and the ID for an observation it has assembled.

It seals and then VALIDATES ITS OWN BYTES through ValidateEnvelope, under this validator's mapping allowlists. So a record that would be refused at the publisher boundary is refused here instead, with the same rule and the same field named, before it reaches a shard -- and the writer cannot drift from the reader, because the writer runs the reader. The bytes returned have already passed the boundary; nothing else needs to trust them.

The returned bytes are the envelope exactly as it goes on the wire: canonical, with no trailing newline. A shard writer adds the newline that separates JSONL records; it is not part of the digest.

func (*Validator) ValidateEnvelope

func (v *Validator) ValidateEnvelope(raw []byte) (*Envelope, error)

ValidateEnvelope is the boundary. It takes the bytes as they were written, refuses with the field named, and returns the accepted record otherwise. It NEVER returns a repaired record: there is no path through this function that edits a value.

func (*Validator) Without

func (v *Validator) Without(rules ...string) *Validator

Without returns a copy of the validator with the named rules not enforced. It exists for exactly one caller: the test that proves each refusal rule is load-bearing by showing the fixture that fails only because of it. It is never used by the publisher; a record that fails is refused with the field named, never repaired and never waved through.

type Value

type Value any

Value is nil, bool, string, json.Number, *Object or []Value.

func Strings added in v0.15.0

func Strings(ss []string) []Value

Strings is the array member a caller wants most: event_key, supersedes and touched are all arrays of strings. An empty one seals as [] and never as null -- the format says "Empty means no split supplied, not no usage", and a null where an array belongs is a different fact. (A nil []Value would also canonicalise as [], because the dynamic type is what the canonicaliser switches on; the slice is returned non-nil for the caller who inspects it, not to fix the bytes, and a test measured that before this sentence was written.)

type ZeroSemantics

type ZeroSemantics string

ZeroSemantics is field_rules.zero_semantics from the mapping, as written: measured, default_may_mask_absence, or unknown.

const (
	ZeroMeasured      ZeroSemantics = "measured"
	ZeroMayMaskAbsent ZeroSemantics = "default_may_mask_absence"
	ZeroUnknown       ZeroSemantics = "unknown"
)

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL