tokens

package

v0.0.0 Latest Latest Go to latest Published: Apr 9, 2026 License: MIT Imports: 4 Imported by: 0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/idelchi/aura

Links

Open Source Insights

Documentation ¶

Overview ¶

Package tokens provides token count estimation and limit checks.

Index ¶

type EstimateBreakdown
type Estimator
- func NewEstimator(method, encoding string, divisor int) (*Estimator, error)

Constants ¶

This section is empty.

Variables ¶

This section is empty.

Functions ¶

This section is empty.

Types ¶

type EstimateBreakdown ¶

type EstimateBreakdown struct {
	Rough         int
	Tiktoken      int
	RoughTiktoken int
}

EstimateBreakdown holds per-method token estimates for debug logging.

type Estimator ¶

type Estimator struct {
	// contains filtered or unexported fields
}

Estimator provides configurable token estimation with eagerly-initialized tiktoken encoding. Thread-safe: tiktoken-go is not safe for concurrent use, so encoding is guarded by a mutex.

Two estimation paths:

Estimate(ctx, text) — uses native provider estimation when available, falls back to local.
EstimateLocal(text) — always uses local methods (rough/tiktoken). Never calls the provider.

func NewEstimator ¶

func NewEstimator(method, encoding string, divisor int) (*Estimator, error)

NewEstimator creates an Estimator with the given method, encoding, and divisor. Valid methods: "rough", "tiktoken", "rough+tiktoken", "native". The "native" method falls back to "rough+tiktoken" for EstimateLocal calls — actual native estimation requires UseNative() to wire the provider.

If the method uses tiktoken (anything except "rough"), the encoding is initialized eagerly and an error is returned on failure.

func (*Estimator) Estimate ¶

func (e *Estimator) Estimate(ctx context.Context, text string) int

Estimate returns a token count using native provider estimation when available, falling back to local estimation on error or when native is not configured. Use for low-frequency, accuracy-critical call sites (builder, result guard).

func (*Estimator) EstimateAll ¶

func (e *Estimator) EstimateAll(text string) EstimateBreakdown

EstimateAll returns token estimates for all methods (rough, tiktoken, rough+tiktoken). Used for debug logging — the "winner" depends on the configured method.

func (*Estimator) EstimateLocal ¶

func (e *Estimator) EstimateLocal(text string) int

EstimateLocal returns a token count for text using the configured local method. Never calls the provider — use for high-frequency estimation (chunking, pruning).

func (*Estimator) HasNative ¶

func (e *Estimator) HasNative() bool

HasNative reports whether a native estimation function has been registered.

func (*Estimator) Method ¶

func (e *Estimator) Method() string

Method returns the configured estimation method name.

func (*Estimator) Rough ¶

func (e *Estimator) Rough(text string) int

Rough returns a character-based token estimate using the configured divisor.

func (*Estimator) SetDebug ¶

func (e *Estimator) SetDebug(fn func(string, ...any))

SetDebug registers a debug logging function for per-call estimation breakdowns.

func (*Estimator) Tiktoken ¶

func (e *Estimator) Tiktoken(text string) int

Tiktoken returns a token count using the configured tiktoken encoding. Returns 0 when enc is nil (method "rough" skips tiktoken init).

func (*Estimator) UseNative ¶

func (e *Estimator) UseNative(fn func(context.Context, string) (int, error))

UseNative registers a function that calls the provider's native estimation endpoint. The Estimator falls back to local estimation if the native function returns an error.

Source Files ¶

View all Source files

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL