Why Go
- Case Studies
  
  Common problems companies solve with Go
- Use Cases
  
  Stories about how and why companies use Go
- Security
  
  How Go can help keep you secure by default
Learn
Docs
- Effective Go
  
  Tips for writing clear, performant, and idiomatic Go code
- Go User Manual
  
  A complete introduction to building software with Go
- Standard library
  
  Reference documentation for Go's standard library
- Release Notes
  
  Learn what's new in each Go release
- API
  
  Reference documentation for the pkg.go.dev API
Packages
Community
- Recorded Talks
  
  Videos from prior events
- Meetups
  
  Meet other local Go developers
- Conferences
  
  Learn and network with Go developers from around the world
- Go blog
  
  The Go project's official blog.
- Go project
  
  Get help and stay informed from Go
- Get connected

core

package

Go to main page

Versions in this module

v0

v0.5.1

May 12, 2026

v0.3.0

May 26, 2026

v0.2.0

May 16, 2026

v0.1.2

Jul 4, 2026

v0.1.1

Jun 25, 2026

Changes in this version

+ func EstimateTokensForModel(text string, model string) int

+ func EstimateTokensWithEncoding(text string, encoding string) int

type BPETokenizer

+ func GetTokenizerForEncoding(encoding string) (*BPETokenizer, error)

v0.1.0

May 12, 2026

Changes in this version

+ var CommonModelPricing = map[string]ModelPricing

+ func CalculateSavings(tokensSaved int, model string) float64

+ func CalculateTokensSaved(original, filtered string) int

+ func EstimateTokens(text string) int

+ func EstimateTokensFast(text string) int

+ func EstimateTokensPrecise(text string) int

+ func HasModelPricing(model string) bool

+ func RegisterModelPricing(model string, input, output float64)

+ func WarmupBPETokenizer()

+ type BPETokenizer struct

+ func (b *BPETokenizer) Count(text string) int

+ type BatchProcessor struct

+ func NewBatchProcessor(coordinator Processor, workers int) *BatchProcessor

+ func (bp *BatchProcessor) ProcessBatch(inputs []string) []BatchResult

+ type BatchResult struct

+ Error error

+ Index int

+ Output string

+ Stats interface{}

+ type CommandRunner interface

+ LookPath func(name string) (string, error)

+ Run func(ctx context.Context, args []string) (output string, exitCode int, err error)

+ type LimitedWriter struct

+ Dropped int64

+ N int64

+ W io.Writer

+ func (lw *LimitedWriter) Write(p []byte) (n int, err error)

+ type ModelPricing struct

+ InputPerMillion float64

+ Model string

+ OutputPerMillion float64

+ func GetModelPricing(model string) ModelPricing

+ type OSCommandRunner struct

+ Env []string

+ func NewOSCommandRunner() *OSCommandRunner

+ func (r *OSCommandRunner) LookPath(name string) (string, error)

+ func (r *OSCommandRunner) Run(ctx context.Context, args []string) (string, int, error)

+ type Processor interface

+ Process func(string) (string, interface{}, error)