Documentation
¶
Overview ¶
Package records is the core of the retained token record format: the canonical envelope, exact numeric strings with presence kept, and the structural validation the publisher boundary performs.
The contract is docs/PROPOSAL-TOKENS-FORMAT.md and docs/PROPOSAL-TOKENS-RECORDS.md at 71ea08b (Stella's #124). Three sentences carry most of this package:
- An object envelope is exactly {"id":"sha256:<64 lowercase hex>","body":{...}}, and the ID is SHA-256 of the body's RFC 8785 canonical JSON with no trailing newline. The envelope ID is excluded from its own digest because only the body is hashed.
- Usage values are JSON strings. A float64 round trip is never performed on them, so an integer above 2^53 and a decimal lexeme survive byte-for-byte. Presence is present, absent or unavailable; absent and unavailable require value:null and a bounded reason code; a present zero survives as a present zero.
- Validation here is STRUCTURAL. A record that fails is refused with the field named and is never repaired: no key is dropped, no value normalised, no missing field defaulted. A refusal is a question for the table, not a thing this package fixes.
What this package deliberately is not: it has no adapters, no batch or path joins, and no publication. Those are separate owners (see the PR that introduced this package).
Index ¶
- Constants
- Variables
- func Canonicalize(v Value) ([]byte, error)
- func ContentID(v Value) (string, error)
- func CoverageMembers() []string
- func KnownZeroSemantics(z ZeroSemantics) bool
- func MappingMembers() []string
- func ObservationMembers() []string
- func Seal(body Value) ([]byte, string, error)
- func SealCoverage(c Coverage) ([]byte, string, error)
- func SealMapping(m Mapping) ([]byte, string, error)
- type Allowlists
- type Coverage
- type CoverageCounts
- type CoverageInterval
- type CoverageReason
- type Envelope
- type Mapping
- type MappingFieldRule
- type MappingIdentityRule
- type MappingMissingTimestamp
- type MappingModelBasisID
- type MappingModelRule
- type MappingOverlapRule
- type MappingRevisionRule
- type MappingTimeRule
- type Measurement
- type Model
- type ModelUsage
- type Object
- type Observation
- type Origin
- type RawField
- type Refusal
- type Repository
- type Revision
- type ShardRef
- type Source
- type Times
- type Validator
- type Value
- type ZeroSemantics
Constants ¶
const ( SchemaObservation = "nova.tokens.observation/2" SchemaMapping = "nova.tokens.mapping/2" SchemaCoverage = "nova.tokens.coverage/2" )
Schema constants for the three retained record bodies.
const ( RuleInvalidUTF8 = "invalid_utf8" RuleLoneSurrogate = "lone_surrogate" RuleNotJSON = "not_json" RuleDuplicateKey = "duplicate_key" RuleRawJSONNumber = "raw_json_number" RuleEnvelopeShape = "envelope_shape" RuleEnvelopeIDSyntax = "envelope_id_syntax" RuleEnvelopeIDMismatch = "envelope_id_mismatch" RuleUnknownSchema = "unknown_schema" RuleUnknownField = "unknown_field" RuleMissingField = "missing_field" RuleWrongType = "wrong_type" RuleUnknownSourceKind = "unknown_source_kind" RuleUnknownEnum = "unknown_enum" RuleEmptyString = "empty_string" RuleLabelSyntax = "label_syntax" RuleNamespaceSyntax = "namespace_syntax" RuleContentIDSyntax = "content_id_syntax" RuleEmptyArray = "empty_array" RuleNotSorted = "not_sorted" RuleDuplicateElement = "duplicate_element" RuleTimestampSyntax = "timestamp_syntax" RuleTimestampNoOffset = "timestamp_no_offset" RuleIntervalIncomplete = "interval_incomplete" RuleIntervalOrder = "interval_order" RuleUnknownPresence = "unknown_presence" RuleUnknownNumberKind = "unknown_number_kind" RuleUnitSyntax = "unit_syntax" RulePresentValueNull = "present_value_null" RulePresentWithReason = "present_with_reason" RuleAbsentValueNotNull = "absent_value_not_null" RuleAbsentReasonMissing = "absent_reason_missing" RuleUnknownReasonCode = "unknown_reason_code" RuleIntegerLexeme = "integer_lexeme" RuleDecimalLexeme = "decimal_lexeme" RuleNegativeValue = "negative_value" RuleFieldNotAllowlisted = "field_not_allowlisted" RuleReceiptNotAllowed = "receipt_field_not_allowlisted" RuleMappingMissingZeroSemantics = "mapping_missing_zero_semantics" RuleMappingRuleShape = "mapping_rule_shape" RuleShardReference = "shard_reference" RuleCoverageReasonSource = "coverage_reason_source" )
The rule names. Every refusal this package can produce names exactly one of them, and every one of them has a fixture under testdata/refused that fails only that rule. The names are stable: a refusal is reported to a person and quoted back in a note.
Variables ¶
var AllRules = []string{ RuleInvalidUTF8, RuleLoneSurrogate, RuleNotJSON, RuleDuplicateKey, RuleRawJSONNumber, RuleEnvelopeShape, RuleEnvelopeIDSyntax, RuleEnvelopeIDMismatch, RuleUnknownSchema, RuleUnknownField, RuleMissingField, RuleWrongType, RuleUnknownSourceKind, RuleUnknownEnum, RuleEmptyString, RuleLabelSyntax, RuleNamespaceSyntax, RuleContentIDSyntax, RuleEmptyArray, RuleNotSorted, RuleDuplicateElement, RuleTimestampSyntax, RuleTimestampNoOffset, RuleIntervalIncomplete, RuleIntervalOrder, RuleUnknownPresence, RuleUnknownNumberKind, RuleUnitSyntax, RulePresentValueNull, RulePresentWithReason, RuleAbsentValueNotNull, RuleAbsentReasonMissing, RuleUnknownReasonCode, RuleIntegerLexeme, RuleDecimalLexeme, RuleNegativeValue, RuleFieldNotAllowlisted, RuleReceiptNotAllowed, RuleMappingMissingZeroSemantics, RuleMappingRuleShape, RuleShardReference, RuleCoverageReasonSource, }
AllRules is every rule name, for the test that proves each one has a fixture and is load-bearing. A rule added without a fixture fails that test rather than shipping unexercised.
var ErrSealed = errors.New("records: this object came from a validated envelope and is sealed; its digest is an identity somebody holds, so build a new body with NewObject")
ErrSealed is what the exported Set answers on an object that came from a validated envelope. It is a plain error and not a Refusal on purpose: a Refusal names one of the Rule* constants, every one of those owes testdata a fixture that fails only that rule, and this is an API misuse rather than a defect in a record's bytes.
var ReasonCodes = []string{
"not_supplied", "not_supported_by_source", "source_unavailable",
"unsupported_semantics", "parse_failed", "redacted", "omitted_from_wire",
}
ReasonCodes is the bounded set an absent or unavailable field may give. Bounded is the requirement the format states; these are the seven distinctions the documents draw between a field a source never had, one it had and would not give, and one this collection could not read.
var SourceKinds = []string{
"antigravity", "claude_code", "codex_desktop", "grok", "nova_swarm", "opencode", "v1_aggregate",
}
SourceKinds is the allowlist of source.kind values, and it is exactly the harness shapes the two proposal documents name at 71ea08b -- claude_code and opencode and nova_swarm from the v1 readers and the wrapper-overlap paragraph, codex_desktop and antigravity and grok from the counter-semantics paragraph, v1_aggregate from "V1 imports ... are labelled aggregate observations". A kind outside it is a bad declaration by the caller rather than an unreadable record, the same distinction internal/tokens draws for a --provider label that names no parser.
Functions ¶
func Canonicalize ¶
Canonicalize writes the RFC 8785 canonical JSON of a parsed value: members sorted by the UTF-16 code units of their names, no insignificant whitespace, and the minimal string escaping ECMAScript's JSON.stringify produces. There is no trailing newline, because the digest is over exactly these bytes.
RFC 8785's number serialisation is deliberately absent. This format's bodies carry no raw JSON numbers at all -- every usage value is a string, so no lexeme ever makes a float64 round trip -- so a number reaching here is refused by parseStrict before it can reach the digest. That removes the shortest-round-trip float printing that is the one genuinely hard corner of RFC 8785, and it removes it by contract rather than by hope.
func ContentID ¶
ContentID is the identity the format names: "sha256:" and 64 lowercase hex digits of the SHA-256 of the value's canonical JSON. Applied to a body it is the envelope's ID, which is therefore excluded from its own digest by construction: nothing but the body is hashed, so there is no fixed point to solve and no circular dependency to break.
func CoverageMembers ¶ added in v0.15.0
func CoverageMembers() []string
CoverageMembers returns a copy of the 14 members of nova.tokens.coverage/2.
func KnownZeroSemantics ¶
func KnownZeroSemantics(z ZeroSemantics) bool
KnownZeroSemantics reports whether a mapping's declared value is one of the three.
func MappingMembers ¶ added in v0.15.0
func MappingMembers() []string
MappingMembers returns a copy of the 12 members of nova.tokens.mapping/2.
func ObservationMembers ¶ added in v0.15.0
func ObservationMembers() []string
ObservationMembers returns the twelve members of a nova.tokens.observation/2 body, in the format's table order. It returns a COPY: writing through the result changes nothing, so a caller can read the schema's shape without being able to widen or shrink what the validator enforces.
func Seal ¶
Seal returns the envelope bytes for a body: the ID derived from the body's canonical JSON, and the body itself in canonical form. It is the only writer here, and it writes no trailing newline.
func SealCoverage ¶ added in v0.15.0
SealCoverage seals a Coverage record and strictly validates its own bytes.
Types ¶
type Allowlists ¶
Allowlists carries the two sets that belong to a MAPPING rather than to the format: the source field names raw_usage may carry, and the native locator fields a receipt may carry. The format calls both "mapping-allowlisted", so core cannot know them; it enforces that something named them. An empty set means no field is allowed, which is the safe direction: a record whose mapping was not supplied is refused, not accepted whole.
type Coverage ¶ added in v0.15.0
type Coverage struct {
Schema string
ScopeID string
SourceIDs []string
Interval CoverageInterval
Status string
Reasons []CoverageReason
CollectedAt string
CollectorBuild string
CollectorFriend *string
CollectionBench *string
MappingIDs []string
Shards []ShardRef
Predecessors []string
Counts CoverageCounts
}
Coverage is a validated nova.tokens.coverage/2 body.
type CoverageCounts ¶ added in v0.15.0
type CoverageCounts struct {
SourceCandidates string
RecordsEmitted string
Observations string
Conflicts string
Gaps string
}
CoverageCounts defines the five non-negative integer string counts.
type CoverageInterval ¶ added in v0.15.0
CoverageInterval defines the interval start and end timestamps.
type CoverageReason ¶ added in v0.15.0
CoverageReason defines one reason entry.
type Envelope ¶
type Envelope struct {
ID string
Body *Object
Observation *Observation
Mapping *Mapping
Coverage *Coverage
}
Envelope is a validated {"id":..., "body":...}.
type Mapping ¶ added in v0.15.0
type Mapping struct {
Schema string
Name string
Revision string
SourceShapes []string
FieldRules map[string]MappingFieldRule
IdentityRule MappingIdentityRule
RevisionRule MappingRevisionRule
TimeRule MappingTimeRule
ModelRule MappingModelRule
OverlapRule MappingOverlapRule
FixtureDigests map[string]string
ImplementationID *string
}
Mapping is a validated nova.tokens.mapping/2 body.
type MappingFieldRule ¶ added in v0.15.0
type MappingFieldRule struct {
NumberKind string
ZeroSemantics string
Unit string
SpendRole string
AbsentPresence string
AbsentReason string
InvalidPresence string
InvalidReason string
}
MappingFieldRule defines the 8 members of each numeric source field rule.
type MappingIdentityRule ¶ added in v0.15.0
type MappingIdentityRule struct {
SourceKind string
Namespace string
EventKey []string
ObservationKind string
NormalizedSpendSupported bool
ProducerVersionFrom string
SessionID string
ReceiptFields []string
ReceiptValueType string
// Codex-specific:
ContainingOrForkedSessionInEventKey *bool
// Grok-specific:
CollectionTimestampSubstitution *bool
ContainingSessionSubstitution *bool
UnsupportedReason *string
}
MappingIdentityRule defines the closed identity rule for reviewed adapters.
type MappingMissingTimestamp ¶ added in v0.15.0
MappingMissingTimestamp represents {basis, occurred_at}.
type MappingModelBasisID ¶ added in v0.15.0
MappingModelBasisID represents nested {basis, id}.
type MappingModelRule ¶ added in v0.15.0
type MappingModelRule struct {
ForbiddenWireKeys []string
// Codex-specific:
Basis *string
ID *string
Absent *MappingModelBasisID
LaterModelRelabelsEarlierResponses *bool
ModelUsage []string
// Grok-specific:
SingleReportedID *MappingModelBasisID
AbsentID *MappingModelBasisID
MultipleModelIDs *MappingModelBasisID
SplitNormalized *bool
AggregateIsCountingCandidate *bool
DetailRetainedNotCountedAgain *bool
ModelUsageFields []string
ModelUsageSortedBy *string
}
MappingModelRule defines model rule members.
type MappingOverlapRule ¶ added in v0.15.0
type MappingOverlapRule struct {
OwedCoverageTasks []string
// Codex-specific:
CountingSource *string
ArithmeticMismatch *string
MissingRawTotal *string
ThreadTokenUsage *string
TurnTokenUsage *string
TokenCountSnapshots *string
SummedOrMixedWithPreferred *bool
ReportMustNameUnhandledTokenCountCoverage *bool
// Grok-specific:
CountingGrain *string
ArithmeticViolation *string
EvidencedOverlappingGrain *string
RequestGrainMapping *string
SessionTotals *string
ForbiddenUnions []string
ChecksumRequiresEvidencedCompleteMatchingCoverage *bool
SessionTotalsFillMissingTurnSpend *bool
}
MappingOverlapRule defines overlap rule members.
type MappingRevisionRule ¶ added in v0.15.0
type MappingRevisionRule struct {
Basis string
Native *string
Supersedes []string
IdenticalCopy string
ChangedSameKey string
InferredOrderFromExportUpdateTime bool
NewestWins bool
}
MappingRevisionRule defines the 7 revision rule members.
type MappingTimeRule ¶ added in v0.15.0
type MappingTimeRule struct {
Basis string
OccurredAt string
Offsets string
DayAllocation string
Start *string
End *string
MissingTimestamp *MappingMissingTimestamp
// Grok-specific:
CrossMidnight *string
}
MappingTimeRule defines time rule members.
type Measurement ¶
type Measurement struct {
// Measured is true only when the raw evidence supports a measurement.
Measured bool
// SubsetComplete is the format's "subset completeness": false whenever a present zero
// might be a synthesised default rather than an observation.
SubsetComplete bool
// Why is a fixed phrase for a diagnostic, never source data.
Why string
}
Measurement is what a normaliser may believe about one raw field once the mapping's zero_semantics is applied. It is NOT a number: producing one is normalisation, which is a separate owner. This is the presence-preserving half, and it is here because getting it wrong is how a "not measured" becomes a zero that sums into a month claiming to be complete.
func Normalize ¶
func Normalize(f RawField, z ZeroSemantics) Measurement
Normalize applies field_rules.zero_semantics exactly as the format writes it:
- "Missing/unavailable raw fields are never normalized to measured 0." An absent or unavailable field yields no measurement at all, whatever the zero rule says.
- "For a present zero under either latter rule, normalized measurement is unknown and subset completeness is false." A present zero is measured only under `measured`.
- "a positive value is normalized only when its other counter semantics are supported" -- that second gate belongs to the mapping and is the caller's, so a positive value is reported as measured here and the caller withholds it if its semantics are not.
A mapping that declares no zero rule for the field is treated as `unknown` and said so; it is not treated as measured.
type ModelUsage ¶
ModelUsage is one entry of the source-provided per-model split.
type Object ¶
type Object struct {
// contains filtered or unexported fields
}
Object is a JSON object with its keys in source order.
func NewObject ¶ added in v0.15.0
func NewObject() *Object
NewObject returns an empty object to build a body in. It is the only exported way to make one, and it exists so a caller can construct the Value that Canonicalize, ContentID and Seal have always taken.
func (*Object) Get ¶
Get returns a member and whether it was present. Present-with-null and absent are different answers, which is the whole point of the presence rules further down.
On a SEALED object -- one the strict parser built, so one whose digest is an identity somebody holds -- an array member is returned as a copy, for the same reason: the read measured on this API was `src.Get("event_key").([]Value)[0] = sentinel`. A nested *Object needs no copy because it is sealed too and defends itself. An object a caller built with NewObject is theirs, and Get hands back exactly what they put in.
func (*Object) Keys ¶
Keys returns the member names in source order, as a COPY.
It used to return o.keys itself, and that made the seal below a half-seal: a whole-PR read of #146 measured `env.Body.Keys()[0] = sentinel` moving ContentID(env.Body) away from env.ID with no error, which is the live-handle shape one method over from the exported member list. A reader of a validated body gets the names, not the array they live in.
func (*Object) Set ¶ added in v0.15.0
Set adds one member and returns a refusal rather than overwriting.
A name already present is RuleDuplicateKey, the same rule the strict parser applies to the bytes: a writer that kept the last of two would produce a body its own reader refuses, and the two sides of one grammar would disagree about a record that already had an ID. The first value stands; a refused Set changes nothing.
A value with no wire form -- a Go map, a struct, an int -- is refused HERE, at the member, rather than three levels down in the canonicaliser as "the value has no canonical form" about a path the caller cannot place.
A refusal names the member's ORDINAL and never the name itself, the way exactKeys, rawUsage and receipt all report indexPath(path, i). The name is the caller's string: it can hold a prompt or a private path, a carriage return that renders one refusal as two lines, or a forged "refused: ..." token, and a diagnostic is a shared file too (#146's adversarial read, finding 3: this was the one place in the package where a caller's string reached a rendered refusal).
An object that came from a validated envelope is sealed and answers ErrSealed: its digest is already an identity. An object from NewObject is the caller's own and stays writable.
type Observation ¶
type Observation struct {
Schema string
Source Source
Kind string
Revision Revision
Time Times
Origin Origin
Model Model
Repository Repository
RawUsage map[string]RawField
ModelUsage []ModelUsage
MappingID string
Receipt map[string]string
}
Observation is a validated nova.tokens.observation/2 body.
func (Observation) Body ¶ added in v0.15.0
func (o Observation) Body() (Value, error)
Body returns the observation's body, the value Seal and ContentID take. It is the inverse of the validator's read: every member the schema names is written, present even when null, and nothing else is.
type RawField ¶
type RawField struct {
Presence string // present, absent, unavailable
Value *string // the original lexeme; nil exactly when presence is not present
NumberKind string // integer or decimal
Unit string
Reason *string // a bounded code; nil exactly when presence is present
}
A RawField is one allowlisted source usage field as it was observed. The value is the ORIGINAL lexeme, kept as a string: an integer above 2^53 and a decimal with trailing digits both survive byte-for-byte because nothing here parses them into a float64.
type Refusal ¶
type Refusal struct {
Rule string // one of the Rule* constants
Field string // dotted path into the envelope, e.g. body.raw_usage.input.value
Note string // a fixed phrase, never source data
}
A Refusal is the only failure this package produces. It names the rule and the field, and it deliberately does NOT carry the offending value: the format says to report the field name and location without echoing its value, because a prompt or a private path can sit in an unsupported source field and a diagnostic is a shared file too.
type Repository ¶
Repository is the repository identity and the evidence that assigned it.
type ShardRef ¶ added in v0.15.0
ShardRef defines a referenced shard and its disjoint tagged inventory.
type Source ¶
type Source struct {
Kind string
ProducerVersion *string
Namespace string
SessionID string
EventKey []string
}
Source is source: the declared shape and the identity within it.
type Times ¶
Times keeps the ORIGINAL offset lexemes. Reporting converts known instants to UTC; this package does not, because a converted timestamp has lost the offset that said which day the source meant.
type Validator ¶
type Validator struct {
// contains filtered or unexported fields
}
A Validator performs the publisher boundary's structural validation. It holds the mapping-supplied allowlists and nothing else: it opens no file, resolves no path and loads no code, which is the property the format asks of the publisher.
func NewValidator ¶
func NewValidator(a Allowlists) *Validator
NewValidator returns a validator over one mapping's allowlists.
The two slices are COPIED. Finding 1 of #146's adversarial read was the schema's closed set held in mutable state a caller could reach; a validator that kept the caller's own backing arrays is the same shape one level down -- what the boundary enforces would change under it whenever the caller appended to the slice it had handed over, including between a SealObservation's seal and its strict read-back. The allowlist is the caller's DECLARATION, made once when the validator is built; it is not a live handle.
func (*Validator) SealObservation ¶ added in v0.15.0
func (v *Validator) SealObservation(o Observation) ([]byte, string, error)
SealObservation is the one call an adapter needs: the envelope bytes and the ID for an observation it has assembled.
It seals and then VALIDATES ITS OWN BYTES through ValidateEnvelope, under this validator's mapping allowlists. So a record that would be refused at the publisher boundary is refused here instead, with the same rule and the same field named, before it reaches a shard -- and the writer cannot drift from the reader, because the writer runs the reader. The bytes returned have already passed the boundary; nothing else needs to trust them.
The returned bytes are the envelope exactly as it goes on the wire: canonical, with no trailing newline. A shard writer adds the newline that separates JSONL records; it is not part of the digest.
func (*Validator) ValidateEnvelope ¶
ValidateEnvelope is the boundary. It takes the bytes as they were written, refuses with the field named, and returns the accepted record otherwise. It NEVER returns a repaired record: there is no path through this function that edits a value.
func (*Validator) Without ¶
Without returns a copy of the validator with the named rules not enforced. It exists for exactly one caller: the test that proves each refusal rule is load-bearing by showing the fixture that fails only because of it. It is never used by the publisher; a record that fails is refused with the field named, never repaired and never waved through.
type Value ¶
type Value any
Value is nil, bool, string, json.Number, *Object or []Value.
func Strings ¶ added in v0.15.0
Strings is the array member a caller wants most: event_key, supersedes and touched are all arrays of strings. An empty one seals as [] and never as null -- the format says "Empty means no split supplied, not no usage", and a null where an array belongs is a different fact. (A nil []Value would also canonicalise as [], because the dynamic type is what the canonicaliser switches on; the slice is returned non-nil for the caller who inspects it, not to fix the bytes, and a test measured that before this sentence was written.)
type ZeroSemantics ¶
type ZeroSemantics string
ZeroSemantics is field_rules.zero_semantics from the mapping, as written: measured, default_may_mask_absence, or unknown.
const ( ZeroMeasured ZeroSemantics = "measured" ZeroMayMaskAbsent ZeroSemantics = "default_may_mask_absence" ZeroUnknown ZeroSemantics = "unknown" )