Documentation
¶
Overview ¶
Package tokenizer defines small, provider-neutral capabilities for counting and encoding text tokens. Concrete vocabularies live in implementation packages such as github.com/Tangerg/scope/tokenizers/tiktoken.
Example ¶
package main
import (
"context"
"fmt"
"strings"
"github.com/Tangerg/scope/core/tokenizer"
)
type wordEstimator struct{}
func (wordEstimator) EstimateText(_ context.Context, text string) (int, error) {
return len(strings.Fields(text)), nil
}
func main() {
var estimator tokenizer.TextEstimator = wordEstimator{}
count, err := estimator.EstimateText(context.Background(), "small stable contract")
if err != nil {
panic(err)
}
fmt.Println(count)
}
Output: 3
Index ¶
Examples ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Decoder ¶
type Decoder interface {
// Decode rejects token IDs outside the selected vocabulary and does not
// retain the caller's slice. Context cancellation remains identifiable.
Decode(ctx context.Context, tokens []int) (string, error)
}
Decoder converts vocabulary token IDs back into text.
type Encoder ¶
type Encoder interface {
// Encode returns token IDs in vocabulary order. The caller owns the returned
// slice; implementations must not expose a mutable internal buffer.
Encode(ctx context.Context, text string) ([]int, error)
}
Encoder converts text into vocabulary token IDs.
type TextEstimator ¶
type TextEstimator interface {
// EstimateText returns a non-negative count under the implementation's
// explicitly selected vocabulary or provider model. Remote implementations
// must honor cancellation and preserve context errors.
EstimateText(ctx context.Context, text string) (int, error)
}
TextEstimator reports the token count a model would assign to text. Implementations may use a local vocabulary or a provider API.
Source Files
Click to show internal directories.
Click to hide internal directories.