Documentation
¶
Overview ¶
Package core provides core interfaces and utilities for TokMan.
The core package defines fundamental abstractions used throughout the codebase, including the CommandRunner interface for shell command execution and token estimation utilities.
CommandRunner Interface ¶
The CommandRunner interface abstracts shell command execution, enabling dependency injection and testability:
type CommandRunner interface {
Run(ctx context.Context, args []string) (output string, exitCode int, err error)
LookPath(name string) (string, error)
}
Token Estimation ¶
EstimateTokens provides a fast heuristic for counting tokens in text:
tokens := core.EstimateTokens("some text to analyze")
Index ¶
- Variables
- func CalculateSavings(tokensSaved int, model string) float64
- func CalculateTokensSaved(original, filtered string) int
- func EstimateTokens(text string) int
- func EstimateTokensFast(text string) int
- func RegisterModelPricing(model string, input, output float64)
- func WarmupBPETokenizer()
- type BPETokenizer
- type CommandRunner
- type ModelPricing
- type OSCommandRunner
Constants ¶
This section is empty.
Variables ¶
var CommonModelPricing = map[string]ModelPricing{
"gpt-4o": {
Model: "gpt-4o",
InputPerMillion: 2.50,
OutputPerMillion: 10.00,
},
"gpt-4o-mini": {
Model: "gpt-4o-mini",
InputPerMillion: 0.15,
OutputPerMillion: 0.60,
},
"gpt-4.1": {
Model: "gpt-4.1",
InputPerMillion: 2.00,
OutputPerMillion: 8.00,
},
"gpt-4.1-mini": {
Model: "gpt-4.1-mini",
InputPerMillion: 0.40,
OutputPerMillion: 1.60,
},
"gpt-4.1-nano": {
Model: "gpt-4.1-nano",
InputPerMillion: 0.10,
OutputPerMillion: 0.40,
},
"claude-3.5-sonnet": {
Model: "claude-3.5-sonnet",
InputPerMillion: 3.00,
OutputPerMillion: 15.00,
},
"claude-3-haiku": {
Model: "claude-3-haiku",
InputPerMillion: 0.25,
OutputPerMillion: 1.25,
},
"claude-4-sonnet": {
Model: "claude-4-sonnet",
InputPerMillion: 3.00,
OutputPerMillion: 15.00,
},
"claude-4-opus": {
Model: "claude-4-opus",
InputPerMillion: 15.00,
OutputPerMillion: 75.00,
},
}
CommonModelPricing provides pricing for popular models (updated April 2026). Use RegisterModelPricing to add or override entries at runtime.
Functions ¶
func CalculateSavings ¶
CalculateSavings computes dollar savings from token reduction.
func CalculateTokensSaved ¶
CalculateTokensSaved computes token savings between original and filtered.
func EstimateTokens ¶
EstimateTokens is the single source of truth for token estimation. Uses BPE tokenization when available and loaded, falls back to heuristic. Optimized with fast path for short strings.
func EstimateTokensFast ¶ added in v0.28.0
EstimateTokensFast provides a fast estimate without BPE. Use this when exact count isn't critical.
func RegisterModelPricing ¶
RegisterModelPricing adds or overwrites pricing for a model at runtime. This allows users and plugins to keep pricing current without code changes.
func WarmupBPETokenizer ¶
func WarmupBPETokenizer()
WarmupBPETokenizer preloads the codec in a background goroutine. Call this during application startup to avoid latency on the first token estimation. Safe to call multiple times.
Types ¶
type BPETokenizer ¶
type BPETokenizer struct {
// contains filtered or unexported fields
}
BPETokenizer wraps tiktoken for accurate BPE token counting. P1.1: Replaces heuristic len/4 with real BPE tokenization. ~20-30% more accurate than heuristic estimation.
func (*BPETokenizer) Count ¶
func (b *BPETokenizer) Count(text string) int
Count returns the accurate BPE token count with caching.
type CommandRunner ¶
type CommandRunner interface {
Run(ctx context.Context, args []string) (output string, exitCode int, err error)
LookPath(name string) (string, error)
}
CommandRunner abstracts shell command execution.
type ModelPricing ¶
type ModelPricing struct {
Model string
InputPerMillion float64 // Cost per 1M input tokens
OutputPerMillion float64 // Cost per 1M output tokens
}
ModelPricing holds per-token pricing for a model.
func GetModelPricing ¶
func GetModelPricing(model string) ModelPricing
GetModelPricing returns the pricing for a model, or the default if unknown.
type OSCommandRunner ¶
type OSCommandRunner struct {
Env []string
}
OSCommandRunner executes real shell commands using os/exec.
func NewOSCommandRunner ¶
func NewOSCommandRunner() *OSCommandRunner
NewOSCommandRunner creates a command runner with the current environment.
Source Files