Documentation
¶
Overview ¶
Package tokenizer defines small, provider-neutral capabilities for counting and encoding text tokens. BatchConfig packs the exact rendered texts into stable, ordered groups under item and token budgets. Concrete vocabularies live in implementation packages such as github.com/Tangerg/scope/tokenizers/tiktoken.
Example ¶
package main
import (
"context"
"fmt"
"strings"
"github.com/Tangerg/scope/core/tokenizer"
)
type wordCounter struct{}
func (wordCounter) CountText(_ context.Context, text string) (int, error) {
return len(strings.Fields(text)), nil
}
func main() {
var counter tokenizer.TextCounter = wordCounter{}
count, err := counter.CountText(context.Background(), "small stable contract")
if err != nil {
panic(err)
}
fmt.Println(count)
}
Output: 3
Index ¶
Examples ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type BatchConfig ¶ added in v0.59.0
type BatchConfig struct {
MaxItems int
MaxTokens int
Counter TextCounter
}
BatchConfig bounds an ordered group of rendered texts by item count, token count or both. The zero value keeps all texts in one group. Counter is required exactly when MaxTokens is positive.
func (BatchConfig) Batches ¶ added in v0.59.0
Batches measures the exact supplied texts once and returns ordered slices backed by that input. Every item fits its limit. Errors return no slices.
Example ¶
A zero token limit permits batching solely by item count.
package main
import (
"context"
"fmt"
"github.com/Tangerg/scope/core/tokenizer"
)
func main() {
config := tokenizer.BatchConfig{MaxItems: 2}
groups, err := config.Batches(context.Background(), []string{"alpha", "beta", "gamma"})
if err != nil {
panic(err)
}
fmt.Println(groups)
}
Output: [[alpha beta] [gamma]]
func (BatchConfig) Validate ¶ added in v0.59.0
func (b BatchConfig) Validate() error
type Decoder ¶
type Decoder interface {
// Decode rejects token IDs outside the selected vocabulary and does not
// retain the caller's slice. Context cancellation remains identifiable.
Decode(ctx context.Context, tokens []int) (string, error)
}
Decoder converts vocabulary token IDs back into text.
type Encoder ¶
type Encoder interface {
// Encode returns token IDs in vocabulary order. The caller owns the returned
// slice; implementations must not expose a mutable internal buffer.
Encode(ctx context.Context, text string) ([]int, error)
}
Encoder converts text into vocabulary token IDs.
type TextCounter ¶ added in v0.21.0
type TextCounter interface {
// CountText returns a non-negative count under the implementation's
// explicitly selected vocabulary or provider model. Remote implementations
// must honor cancellation and preserve context errors.
CountText(ctx context.Context, text string) (int, error)
}
TextCounter reports the token count a model would assign to text. Implementations may use a local vocabulary or a provider API.
Source Files