Documentation
¶
Overview ¶
Package ollamatokenizer provides Tokenizer implementations used by llmrepo to count and split tokens for a given model.
NewHTTPClient talks to an Ollama-compatible tokenizer endpoint; EstimateTokenizer is a dependency-free heuristic fallback; MockTokenizer is intended for tests.
Index ¶
- type ConfigHTTP
- type EstimateTokenizer
- func (e *EstimateTokenizer) CountTokens(ctx context.Context, modelName string, prompt string) (int, error)
- func (e *EstimateTokenizer) OptimalModel(ctx context.Context, baseModel string) (string, error)
- func (e *EstimateTokenizer) Tokenize(ctx context.Context, modelName string, prompt string) ([]int, error)
- type HTTPClient
- type MockTokenizer
- type Tokenizer
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type ConfigHTTP ¶
type ConfigHTTP struct {
BaseURL string
}
ConfigHTTP contains configuration for the HTTP client.
type EstimateTokenizer ¶
type EstimateTokenizer struct{}
EstimateTokenizer implements Tokenizer using simple character-based estimates. Use for local single-process mode where no tokenizer service is available. CountTokens uses ~4 chars per token (rough heuristic for typical LLM tokenizers).
func NewEstimateTokenizer ¶
func NewEstimateTokenizer() *EstimateTokenizer
NewEstimateTokenizer returns a tokenizer that estimates token counts without a remote service.
func (*EstimateTokenizer) CountTokens ¶
func (e *EstimateTokenizer) CountTokens(ctx context.Context, modelName string, prompt string) (int, error)
CountTokens returns an estimated token count (runes / 4, min 1).
func (*EstimateTokenizer) OptimalModel ¶
OptimalModel returns the base model unchanged (no proxy tokenizer model).
func (*EstimateTokenizer) Tokenize ¶
func (e *EstimateTokenizer) Tokenize(ctx context.Context, modelName string, prompt string) ([]int, error)
Tokenize returns a dummy slice of length equal to the estimated token count. No caller in the task engine uses the actual token IDs; the length is sufficient.
type HTTPClient ¶
type HTTPClient struct {
// contains filtered or unexported fields
}
HTTPClient implements the Tokenizer interface using HTTP calls to the tokenizer service.
func (*HTTPClient) CountTokens ¶
CountTokens uses the dedicated /count endpoint, with a backward-compatible fallback to /tokenize.
func (*HTTPClient) OptimalModel ¶
OptimalModel returns the optimal model for tokenization based on the given model. This is a client-side implementation mirroring the server's logic.
type MockTokenizer ¶
MockTokenizer is a mock implementation of the Tokenizer interface.
func (MockTokenizer) CountTokens ¶
func (MockTokenizer) OptimalModel ¶
type Tokenizer ¶
type Tokenizer interface {
Tokenize(ctx context.Context, modelName string, prompt string) ([]int, error)
CountTokens(ctx context.Context, modelName string, prompt string) (int, error)
OptimalModel(ctx context.Context, baseModel string) (string, error)
}
func NewHTTPClient ¶
NewHTTPClient creates a new HTTP-based tokenizer client.
func WithActivityTracker ¶
func WithActivityTracker(client Tokenizer, tracker libtracker.ActivityTracker) Tokenizer
WithActivityTracker decorates the given Tokenizer with activity tracking
Source Files