README
¶
internal/protocol/usage
Centralized token extraction and normalization. All handlers call into this package instead of re-implementing provider rules inline.
Normalization rules
Different providers report tokens with incompatible semantics. We normalize to a consistent internal representation so the front-end cache-hit formula works correctly:
cache_hit_ratio = CacheInputTokens / (InputTokens + CacheInputTokens)
| Provider | Wire semantics | InputTokens stored as | CacheInputTokens stored as |
|---|---|---|---|
| OpenAI Chat / Responses | prompt_tokens = total (cached + uncached) |
prompt_tokens - cached_tokens |
cached_tokens |
| Anthropic | input_tokens = uncached only; cache_creation_input_tokens = write cost |
input_tokens + cache_creation_input_tokens |
cache_read_input_tokens |
Why add
cache_creationto input? Creation tokens are billable at a write-cost rate, so they belong in the denominator to correctly represent total prompt spend.
Anthropic streaming event split
Anthropic splits usage across two events:
message_start→input_tokens,cache_creation_input_tokens,cache_read_input_tokensmessage_delta→output_tokens(some non-standard providers also sendinput_tokenshere)
AnthropicAccumulator handles this split, priority fallback, and normalization transparently.
API
Non-streaming (pure functions)
usage.FromOpenAIChatCompletion(resp.Usage) // openai.CompletionUsage
usage.FromOpenAIResponses(resp.Usage) // responses.ResponseUsage
usage.FromAnthropicMessage(resp.Usage) // anthropic.Usage
usage.FromAnthropicBetaMessage(resp.Usage) // anthropic.BetaUsage
Streaming — Anthropic accumulator
acc := usage.NewAnthropicAccumulator()
// In event loop:
acc.Consume(&evt) // MessageStreamEventUnion (non-beta)
acc.ConsumeBeta(&evt) // BetaRawMessageStreamEventUnion (beta)
// At return:
if acc.HasUsage() {
return acc.Result(), nil
}
return protocol.ZeroTokenUsage(), nil
Coverage
internal/protocol/nonstream/
| Function | Extractor |
|---|---|
HandleOpenAIChatNonStream |
FromOpenAIChatCompletion |
HandleOpenAIResponsesNonStream |
FromOpenAIResponses |
HandleAnthropicV1NonStream |
FromAnthropicMessage |
HandleAnthropicV1BetaNonStream |
FromAnthropicBetaMessage |
internal/protocol/stream/
| Function | Mechanism |
|---|---|
HandleAnthropic |
AnthropicAccumulator.Consume |
HandleAnthropicBeta |
AnthropicAccumulator.ConsumeBeta |
AnthropicToOpenAIStreamWithMCPHooks |
AnthropicAccumulator.ConsumeBeta |
HandleAnthropicBetaToOpenAIResponsesStream |
AnthropicAccumulator.ConsumeBeta |
internal/server/ (dispatch layer)
| Code site | Extractor |
|---|---|
protocol_dispatch — Anthropic Beta non-stream (×2) |
FromAnthropicBetaMessage |
protocol_dispatch — Responses → Anthropic Beta |
FromAnthropicBetaMessage |
protocol_dispatch — OpenAI Chat non-stream (×2) |
FromOpenAIChatCompletion |
protocol_dispatch — OpenAI Responses non-stream (×2) |
FromOpenAIResponses |
anthropic_message_v1 — Responses → Anthropic v1 |
FromOpenAIResponses |
anthropic_message_beta — Responses → Anthropic Beta |
FromOpenAIResponses |
Intentional inline extraction (not migrated)
| File | Reason |
|---|---|
stream/openai_passthrough.go |
Per-chunk accumulation + estimated usage injection fallback |
stream/openai_to_anthropic*.go |
Uses StreamTokenCounter (incremental counting, not extraction) |
stream/openai_{chat,responses}_to_*.go |
state fields are dual-use: also build the wire response body |
stream/google_to_any.go |
Google SDK has no structured cache sub-fields |
nonstream/anthropic_to_openai.go |
Returns wire format (map[string]interface{}), not *TokenUsage |
nonstream/openai_to_anthropic.go |
Same |
server/protocol_dispatch — Google non-stream |
Google schema, no cached tokens in SDK struct |
Documentation
¶
Overview ¶
Package usage centralizes token extraction and normalization logic for all supported provider protocols. Every handler calls into this package instead of re-implementing provider-specific rules inline.
Normalization rules:
- OpenAI (Chat / Responses): prompt_tokens = total (cached + uncached). Store inputTokens = total - cached so the frontend ratio formula gives cache_read / (cache_read + uncached) = correct hit rate.
- Anthropic: input_tokens = uncached only; cache_creation_input_tokens is an additional write cost that belongs in the denominator. Store inputTokens = input + creation so the formula covers total prompt cost.
Index ¶
- func ChatUsage(u *protocol.TokenUsage) openai.CompletionUsage
- func FromAnthropicBetaMessage(u anthropic.BetaUsage) *protocol.TokenUsage
- func FromAnthropicMessage(u anthropic.Usage) *protocol.TokenUsage
- func FromOpenAIChatCompletion(u openai.CompletionUsage) *protocol.TokenUsage
- func FromOpenAIResponses(u responses.ResponseUsage) *protocol.TokenUsage
- type AnthropicAccumulator
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func ChatUsage ¶
func ChatUsage(u *protocol.TokenUsage) openai.CompletionUsage
ChatUsage converts normalized TokenUsage into an OpenAI Chat Completions CompletionUsage wire struct. OpenAI wire semantics: PromptTokens = TOTAL (uncached + cached), CachedTokens is a reported subset.
func FromAnthropicBetaMessage ¶
func FromAnthropicBetaMessage(u anthropic.BetaUsage) *protocol.TokenUsage
FromAnthropicBetaMessage extracts normalized TokenUsage from an Anthropic beta BetaMessage usage block. Same normalization as the non-beta path.
func FromAnthropicMessage ¶
func FromAnthropicMessage(u anthropic.Usage) *protocol.TokenUsage
FromAnthropicMessage extracts normalized TokenUsage from an Anthropic v1 (non-beta) Message usage block. CacheCreationInputTokens is added to InputTokens so the denominator covers all non-cache-read prompt cost.
func FromOpenAIChatCompletion ¶
func FromOpenAIChatCompletion(u openai.CompletionUsage) *protocol.TokenUsage
FromOpenAIChatCompletion extracts normalized TokenUsage from an OpenAI Chat Completions usage block. CachedTokens is a SUBSET of PromptTokens, so we subtract it to get the uncached portion.
func FromOpenAIResponses ¶
func FromOpenAIResponses(u responses.ResponseUsage) *protocol.TokenUsage
FromOpenAIResponses extracts normalized TokenUsage from an OpenAI Responses API usage block. Same semantics as Chat: InputTokens = total, CachedTokens is a subset.
Types ¶
type AnthropicAccumulator ¶
type AnthropicAccumulator struct {
// contains filtered or unexported fields
}
AnthropicAccumulator accumulates token usage across a streaming Anthropic response. The Anthropic protocol splits usage across two event types:
- message_start → input_tokens, cache_creation, cache_read
- message_delta → output_tokens (occasionally input for non-standard providers)
Both non-beta (MessageStreamEventUnion) and beta (BetaRawMessageStreamEventUnion) streams are supported via Consume and ConsumeBeta respectively.
func NewAnthropicAccumulator ¶
func NewAnthropicAccumulator() *AnthropicAccumulator
NewAnthropicAccumulator returns a zeroed accumulator ready to consume events.
func (*AnthropicAccumulator) Consume ¶
func (a *AnthropicAccumulator) Consume(evt *anthropic.MessageStreamEventUnion)
Consume updates the accumulator from a non-beta streaming event. It is safe to call on every event in the stream; only usage-carrying events (message_start, message_delta) have any effect.
func (*AnthropicAccumulator) ConsumeBeta ¶
func (a *AnthropicAccumulator) ConsumeBeta(evt *anthropic.BetaRawMessageStreamEventUnion)
ConsumeBeta updates the accumulator from a beta streaming event.
func (*AnthropicAccumulator) HasUsage ¶
func (a *AnthropicAccumulator) HasUsage() bool
HasUsage reports whether any non-zero usage was observed.
func (*AnthropicAccumulator) Result ¶
func (a *AnthropicAccumulator) Result() *protocol.TokenUsage
Result returns the normalized TokenUsage built from accumulated events.
Source Files