chunking

package
v0.12.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 2, 2026 License: MIT, MIT Imports: 21 Imported by: 0

README

chunking

Text chunking utilities for Go.

This module provides deterministic chunking strategies for retrieval, embedding pipelines, summarization, and LLM context assembly.

For the complete API reference — every strategy, the span contract, semantic options, and token budgets — see docs/api-guide.md.

Install

go get github.com/dotcommander/reliquary/chunking

Usage

package main

import (
	"fmt"
	"log"

	"github.com/dotcommander/reliquary/chunking"
)

func main() {
	chunker, err := chunking.NewChunker(chunking.SmartBoundary)
	if err != nil {
		log.Fatal(err)
	}

	chunks := chunker.Chunk("First paragraph. Second paragraph.", 1200, 100)
	fmt.Println(len(chunks))
}

Strategies

  • SmartBoundary
  • SentenceBoundary
  • WordBoundary
  • MarkdownAware
  • HeadingAware
  • ParagraphAware
  • HardCut
  • TokenBased
  • Semantic

Semantic chunking is created with NewSemanticChunker because it needs an embedder that satisfies:

type BatchEmbedder interface {
	EmbedBatch(ctx context.Context, texts []string) ([][]float32, error)
}

Source Spans

Each Chunk has StartChar and EndChar fields that are byte offsets into the original input text. When set, the chunk text appears verbatim in the source:

text[ch.StartChar:ch.EndChar] == ch.Text

Spans are cleared (set to 0, 0) when a post-processing step such as overlap or hard-limit splitting cannot map the chunk text back to an exact contiguous source slice. Always check for non-zero spans before using them.

Separator Profiles

DefaultTextSeparatorProfile and CJKThaiSeparatorProfile expose ordered, pure separator lists for callers that run recursive splitting outside this package. The CJK/Thai profile includes ideographic punctuation, fullwidth punctuation, and zero-width space so scripts without word-boundary spaces can avoid poor fallback splits.

Token Chunking

// Default encoding (cl100k_base)
chunker, _ := chunking.NewChunker(chunking.TokenBased)
chunks := chunker.Chunk(text, 500, 50)

To use a different tiktoken encoding:

// Specific encoding
chunker, err := chunking.NewTokenChunker("o200k_base")
if err != nil {
	log.Fatal(err)
}
chunks := chunker.Chunk(text, 500, 50)

NewTokenChunker validates the encoding at construction time and returns an error for unrecognized names. Empty encoding defaults to cl100k_base.

Token Counting

tokens, err := chunking.CountTokens("hello world", "cl100k_base")

Token encoders are cached by encoding name.

Token Budget Composition

Use TokenBased when token boundaries should drive the primary split. Use ChunkWithTokenLimit when a normal boundary strategy (MarkdownAware, HeadingAware, SmartBoundary, etc.) should drive the split, but final chunks must fit a model-specific token budget.

base, err := chunking.NewChunker(chunking.MarkdownAware)
if err != nil {
	log.Fatal(err)
}

counter, err := chunking.NewTiktokenCounter("cl100k_base", 500)
if err != nil {
	log.Fatal(err)
}

chunks := chunking.ChunkWithTokenLimit(base, markdownText, 1600, 100, counter)

Source spans keep the existing contract: pass-through chunks preserve spans; token-split chunks may have unknown spans (zero). Construct with NewTiktokenCounter("", 500) to use the default encoding (cl100k_base).

Markdown Table Context

MarkdownAware splits oversized markdown tables by rows while preserving the header row in every chunk. This retains column context for embedding and retrieval. Table chunks use zero spans since the header is duplicated in later chunks.

chunker, _ := chunking.NewChunker(chunking.MarkdownAware)
chunks := chunker.Chunk(tableMarkdown, 200, 0)
// Each chunk contains the header row followed by body rows.

Documentation

Overview

Package chunking splits text into reusable chunks for search, retrieval, summarization, and LLM context assembly.

Each chunk carries StartChar and EndChar byte offsets into the original input text, such that original[StartChar:EndChar] == chunk.Text when the text appears verbatim. Sub-chunks from section/block fallbacks are rebased via adjustChunkSpans rather than cleared. Spans are cleared to 0 only by post-processing steps (EnforceHardLimits, EnforceTokenLimits) that cannot map the result back to the source.

Chunk fields

Each Chunk carries structured metadata beyond raw text:

  • Path: section breadcrumb from headings (nil for non-heading strategies)
  • Metadata: block-type metadata from goldmark parsing (nil for non-goldmark strategies)
  • ContentHash: first 16 hex characters of SHA-256(text); always set

The Metadata map contains keys like "heading_level", "language", "line_count", and "word_count" for blocks parsed via goldmark.

Prose filtering

FilterProse removes code and table chunks, dropping paragraphs below 5 words and headings below 3 words. Chunks without Metadata pass through unchanged.

Goldmark dependency

The MarkdownAware and HeadingAware strategies use github.com/yuin/goldmark for structural parsing. Headings inside fenced code blocks are never detected as heading boundaries — fence-gating is structural, not stateful.

Offset helpers

  • LineForOffset: converts a byte offset to a 1-based line number
  • Locate: finds byte offsets of a fragment in content (exact then normalized)
  • FillTokenCounts: populates TokenCount on chunks that lack it

Error sentinels

  • ErrUnknownStrategy: returned by NewChunker for unrecognized strategy names
  • ErrNilEmbedder: returned by NewSemanticChunker when embedder is nil

Index

Examples

Constants

View Source 
const (
	SeparatorProfileDefaultTextID = "default_text"
	SeparatorProfileCJKThaiID     = "cjk_thai"
)
View Source 
const HeadingWordFloor = 3

HeadingWordFloor is the minimum word count for a heading block to be admitted by FilterProse (lower than ProseWordFloor because headings are typically short but still meaningful).

View Source 
const ProseWordFloor = 5

ProseWordFloor is the minimum word count for any block type to be admitted by FilterProse.

Variables

View Source 
var ErrNilEmbedder = errors.New("chunking: embedder must not be nil")

ErrNilEmbedder is returned by NewSemanticChunker when a nil embedder is provided.

View Source 
var ErrUnknownStrategy = errors.New("chunking: unknown strategy")

ErrUnknownStrategy is returned by NewChunker when an unrecognized strategy name is provided.

Functions

func CountTokens 

func CountTokens(text string, encoding string) (int, error)

CountTokens returns the token count for text using the named tiktoken encoding. The encoder is cached after first use for the given encoding name.

func FillTokenCounts 

func FillTokenCounts(chunks []Chunk, encoding string) error

FillTokenCounts sets TokenCount on each chunk whose TokenCount is 0. Uses the named tiktoken encoding (e.g. "cl100k_base"). Chunks that already have a non-zero TokenCount are skipped.

func FillTokenCountsWithTokenizer added in v0.8.0 

func FillTokenCountsWithTokenizer(chunks []Chunk, tokenizer Tokenizer) error

FillTokenCountsWithTokenizer fills missing counts using the caller-selected provider or model tokenizer.

func LineForOffset 

func LineForOffset(content string, offset int) int

LineForOffset returns the 1-based line number for the given byte offset in content. offset is clamped to [0, len(content)]. An empty content returns 1.

func LineRangeForSpan 

func LineRangeForSpan(content string, span ChunkSpan) (startLine int, endLine int)

LineRangeForSpan returns the inclusive 1-based line range for a byte span within content. The end offset is adjusted by -1 before line counting so that a span ending exactly at a newline reports the previous content line as the end line, not the next.

func Locate 

func Locate(content, fragment string, cursor int) (int, int, bool)

Locate returns the byte offsets [start, end) of fragment in content. It tries exact substring match from cursor, then exact match from 0, then whitespace-normalized match. ok is false if no match is found.

The locate machinery lives in textutil; this delegates with ExactFirst ordering (exact matches preferred over normalized) so the chunking pipeline does not duplicate the implementation.

func NextChunkCursor 

func NextChunkCursor(span ChunkSpan) int

NextChunkCursor returns the next search cursor after a resolved span. It advances to Start+1 (not End) so that overlapping chunks and repeated phrases can still be found while avoiding the same match.

func SplitSentences 

func SplitSentences(text string) []string

SplitSentences is the exported entry point for token-aware chunk validation in the actions layer.

func StrategicSample 

func StrategicSample(text string, optimalLen int) string

StrategicSample returns a bounded representation of text by keeping the first 67% and splicing midpoint + tail samples (each remainingQuota/3), separated by seam markers. Returns text unchanged when the rune count of text is less than or equal to optimalLen.

Rune-boundary safety: the function operates on a slice of runes to ensure multi-byte UTF-8 codepoints are never split.

Types

type BatchEmbedder 

type BatchEmbedder interface {
	EmbedBatch(ctx context.Context, texts []string) ([][]float32, error)
}

BatchEmbedder is satisfied by any embedder that can embed multiple texts. Defined here so semantic chunking can accept any embedding implementation.

type Chunk 

type Chunk struct {
	ID         int
	Text       string
	CharCount  int
	WordCount  int
	TokenCount int
	// StartChar and EndChar are byte offsets into the original input text,
	// such that originalText[StartChar:EndChar] == Text when the chunk text
	// appears verbatim in the source. For overlap chunks, the span points to
	// the chunk's non-overlap contribution. Spans are rebased from
	// section-relative to original-relative coordinates by adjustChunkSpans
	// (base.go). Spans are cleared (set to 0) when a post-processing step
	// (e.g. EnforceHardLimits) cannot map the result back to the original text.
	StartChar int
	EndChar   int
	// Path is the section breadcrumb from headings; nil for non-heading-aware strategies.
	Path []string
	// Metadata holds block-type metadata from goldmark parsing; nil for non-goldmark strategies.
	Metadata map[string]string
	// ContentHash is the first 16 hex characters of SHA-256(text); always set.
	ContentHash string
}

Chunk represents a segment of text produced by a chunking strategy.

func ChunkWithTokenLimit 

func ChunkWithTokenLimit(c Chunker, text string, size int, overlap int, tc TokenCounter) []Chunk

ChunkWithTokenLimit composes any Chunker with model-specific token-budget enforcement. It runs the base chunker, then applies EnforceTokenLimits.

If c is nil, returns nil. If tc is nil, returns the base chunks unchanged. This is the recommended way to combine a boundary strategy (MarkdownAware, HeadingAware, SmartBoundary, etc.) with a token ceiling.

func EnforceHardLimits 

func EnforceHardLimits(chunks []Chunk, opts LimitOptions) []Chunk

EnforceHardLimits ensures no chunk exceeds the configured character limit. Oversized chunks are split using cascading boundary logic: 

paragraph → sentence → word → hard cut

Undersized chunks are left untouched. Chunk IDs are rebuilt to be sequential. Empty chunks are dropped. Text order is preserved.

When an oversized chunk is split, the resulting sub-chunks have StartChar and EndChar cleared to 0 because the split text cannot be reliably mapped back to the original source byte offsets. Pass-through chunks retain their spans.

func EnforceTokenLimits 

func EnforceTokenLimits(chunks []Chunk, tc TokenCounter) []Chunk

EnforceTokenLimits ensures no chunk exceeds the token counter's MaxTokens budget. It layers on top of EnforceHardLimits (character budgets) and handles the token dimension.

Oversized chunks are split using cascading boundary logic: 

sentence → word → hard cut

Chunk IDs are rebuilt to be sequential. Empty chunks are dropped. Source spans (StartChar/EndChar) are cleared on split sub-chunks since the split text cannot be reliably mapped back to the original source.

func EnforceTokenLimitsWithTokenizer added in v0.8.0 

func EnforceTokenLimitsWithTokenizer(chunks []Chunk, tokenizer Tokenizer, maxTokens int) ([]Chunk, error)

EnforceTokenLimitsWithTokenizer applies an exact provider/model tokenizer to a chunk budget and propagates tokenization failures.

func FilterProse 

func FilterProse(chunks []Chunk) []Chunk

FilterProse returns only prose-bearing chunks, skipping code and tables, and dropping chunks below word-count floors. It reads Chunk.Metadata keys "type" and "word_count". Chunks without Metadata are passed through unchanged — they cannot be classified, so FilterProse does not drop them.

type ChunkSpan 

type ChunkSpan struct {
	Start int
	End   int
}

ChunkSpan represents a resolved byte range [Start, End) within source text.

func ResolveChunkSpan 

func ResolveChunkSpan(content string, chunk Chunk, cursor int) (ChunkSpan, bool)

ResolveChunkSpan determines the byte span of chunk.Text within content.

If chunk.StartChar and chunk.EndChar describe a valid byte range and content[start:end] == chunk.Text, that range is returned directly. Otherwise, Locate is used to find the fragment starting from cursor. Returns false if the chunk text is empty or cannot be located.

type Chunker 

type Chunker interface {
	Chunk(text string, size int, overlap int) []Chunk
	Strategy() Strategy
}

Chunker splits text into segments according to a strategy.

func NewChunker 

func NewChunker(strategy Strategy) (Chunker, error)

NewChunker creates a Chunker for the given strategy.

Example 
package main

import (
	"fmt"

	"github.com/dotcommander/reliquary/chunking"
)

func main() {
	chunker, err := chunking.NewChunker(chunking.SentenceBoundary)
	if err != nil {
		panic(err)
	}

	chunks := chunker.Chunk("Alpha sentence. Beta sentence. Gamma sentence.", 24, 0)
	fmt.Println(len(chunks) > 0)
}

Output:
true

func NewTokenChunker 

func NewTokenChunker(encoding string) (Chunker, error)

NewTokenChunker creates a Chunker that splits text at token boundaries using the specified tiktoken encoding. Empty encoding defaults to "cl100k_base". Returns an error if the encoding name is not recognized by tiktoken.

type LimitOptions 

type LimitOptions struct {
	MaxChars     int
	Overlap      int
	OriginalText string // when set, sub-chunks from splits carry sub-spans mapped back into this text
}

LimitOptions configures the hard-limit finalizer.

type OptimalChunker 

type OptimalChunker struct {
	OptimalLength int // target chunk size for SplitIntoChunks
	MinLength     int // below this, content passes through unchanged
	MaxLength     int // above this, StrategicSample is applied
}

OptimalChunker gates content into size bands and splits large inputs using boundary-aware greedy chunking.

Three-tier gate:

  • len(text) <= MinLength: pass-through (insufficient context for splitting)
  • len(text) <= MaxLength: pass-through (optimal range for downstream AI)
  • len(text) > MaxLength: StrategicSample (head/midpoint/tail with seam markers)

OptimalLength, MinLength, and MaxLength are caller-supplied; NewOptimalChunker provides research-backed defaults.

func NewOptimalChunker 

func NewOptimalChunker() *OptimalChunker

NewOptimalChunker creates an OptimalChunker with defaults based on AI behavior research: OptimalLength 10000, MinLength 5000, MaxLength 15000.

func (*OptimalChunker) Chunk 

func (c *OptimalChunker) Chunk(text string, size int, _ int) []Chunk

Chunk satisfies the Chunker interface. size maps to OptimalLength; overlap is ignored (reserved for future spec). Returns nil for empty text or non-positive size.

func (*OptimalChunker) ChunkContent 

func (c *OptimalChunker) ChunkContent(text string) string

ChunkContent applies three-tier gating and returns a single string:

  • below MinLength: input unchanged
  • below MaxLength: input unchanged
  • above MaxLength: StrategicSample output with seam markers

Size comparisons use rune counts to match the Chunker interface semantics.

func (*OptimalChunker) SplitIntoChunks 

func (c *OptimalChunker) SplitIntoChunks(text string) []string

SplitIntoChunks divides content greedily into chunks of at most OptimalLength runes. It prefers \n\n (paragraph) or ". " (sentence) break points only when the break position is past OptimalLength/2, avoiding tiny leading chunks. Otherwise it hard-cuts at OptimalLength runes.

func (*OptimalChunker) StrategicSample 

func (c *OptimalChunker) StrategicSample(text string) string

StrategicSample on OptimalChunker delegates to the package function.

func (*OptimalChunker) Strategy 

func (c *OptimalChunker) Strategy() Strategy

Strategy returns the Optimal strategy constant.

type SemanticChunker 

type SemanticChunker struct {
	// contains filtered or unexported fields
}

SemanticChunker splits text at topic boundaries detected by embedding similarity between consecutive sentences. Produces variable-length chunks where each chunk covers one coherent topic.

func NewSemantic 

func NewSemantic(embedder BatchEmbedder, opts ...SemanticOption) (*SemanticChunker, error)

NewSemantic is a fluent constructor over NewSemanticChunker: it starts from zero-value SemanticOpts (package defaults then apply) and applies the given options. Returns ErrNilEmbedder if embedder is nil.

func NewSemanticChunker 

func NewSemanticChunker(embedder BatchEmbedder, opts SemanticOpts) (*SemanticChunker, error)

NewSemanticChunker creates a semantic chunker that falls back to smart boundary chunking on embedding failure. Returns ErrNilEmbedder if embedder is nil.

func (*SemanticChunker) ChunkSemantic 

func (sc *SemanticChunker) ChunkSemantic(ctx context.Context, text string, fallbackSize, fallbackOverlap int) []Chunk

ChunkSemantic splits text at topic boundaries. Falls back to smart boundary chunking if embedding fails or the text is too short for semantic analysis. When the input contains structural markers (conversation turns, headings, horizontal rules, or paragraph blocks), those are used as semantic atoms instead of individual sentences, reducing embedding calls and preserving source byte spans where possible.

type SemanticOption 

type SemanticOption func(*SemanticOpts)

SemanticOption configures SemanticOpts fluently for NewSemantic.

func WithBreakSensitivity 

func WithBreakSensitivity(f float64) SemanticOption

WithBreakSensitivity sets the stddev multiplier (higher = fewer breaks).

func WithCoherenceWindow 

func WithCoherenceWindow(n int) SemanticOption

WithCoherenceWindow sets the two-sided coherence gate (0 disables).

func WithMaxChunkChars 

func WithMaxChunkChars(n int) SemanticOption

WithMaxChunkChars sets the hard ceiling per chunk.

func WithMinChunkChars 

func WithMinChunkChars(n int) SemanticOption

WithMinChunkChars merges groups smaller than n.

func WithSmoothingWindow 

func WithSmoothingWindow(n int) SemanticOption

WithSmoothingWindow sets the similarity smoothing window (0 disables).

type SemanticOpts 

type SemanticOpts struct {
	MaxChunkChars    int     // hard ceiling per chunk (default 1600)
	MinChunkChars    int     // merge groups smaller than this (default 200)
	BreakSensitivity float64 // stddev multiplier: higher = fewer breaks (default 1.0)
	SmoothingWindow  int     // centered moving-average window for similarity smoothing; 0 disables (default). Odd values recommended (e.g. 3).
	CoherenceWindow  int     // two-sided coherence gate: require N coherent neighbors on each side of a candidate break; 0 disables (default). Recommended: 2.
}

SemanticOpts configures semantic chunking behavior.

type SemanticPlan 

type SemanticPlan struct {
	Units        []SemanticUnit
	Similarities []float64
	Breaks       []int
	Chunks       []Chunk
}

SemanticPlan describes an accepted semantic chunking plan.

func PlanSemanticChunks 

func PlanSemanticChunks(text string, units []SemanticUnit, embeddings [][]float32, opts SemanticPlanOptions) (SemanticPlan, bool)

PlanSemanticChunks plans semantic chunks from caller-supplied units and embeddings. It returns false when the supplied data is not suitable for semantic planning; callers should then use their preferred fallback chunker.

type SemanticPlanOptions 

type SemanticPlanOptions struct {
	MaxChunkChars    int
	MinChunkChars    int
	BreakSensitivity float64
	SmoothingWindow  int
	CoherenceWindow  int
	FallbackSize     int
	FallbackOverlap  int
}

SemanticPlanOptions configures semantic chunk planning from precomputed embeddings. FallbackSize and FallbackOverlap are applied only to the final hard-limit pass; callers should use their own fallback chunker when PlanSemanticChunks returns false.

type SemanticUnit 

type SemanticUnit struct {
	Text      string
	StartChar int
	EndChar   int
}

SemanticUnit is a structural unit used by semantic chunk planning. StartChar and EndChar are byte offsets into the original input text when known. A zero span means the source location is unknown.

func SemanticUnits 

func SemanticUnits(text string) []SemanticUnit

SemanticUnits returns the structural semantic units that semantic chunking uses before embedding. Units carry source byte offsets when they can be mapped exactly.

type SeparatorProfile 

type SeparatorProfile struct {
	ID         string
	Separators []string
}

SeparatorProfile names an ordered list of separators for recursive text splitting. The package provides pure profile data only; callers own the splitting algorithm that consumes it.

func CJKThaiSeparatorProfile 

func CJKThaiSeparatorProfile() SeparatorProfile

CJKThaiSeparatorProfile returns separators for scripts that commonly lack whitespace word boundaries, including fullwidth and ideographic punctuation plus a zero-width space.

Example 
package main

import (
	"fmt"

	"github.com/dotcommander/reliquary/chunking"
)

func main() {
	profile := chunking.CJKThaiSeparatorProfile()
	separators := profile.SeparatorStrings()

	fmt.Println(profile.ID, len(separators) > len(chunking.DefaultTextSeparatorProfile().Separators))
}

Output:
cjk_thai true

func DefaultTextSeparatorProfile 

func DefaultTextSeparatorProfile() SeparatorProfile

DefaultTextSeparatorProfile returns separators for whitespace-delimited prose, ordered from broad to narrow.

func (SeparatorProfile) SeparatorStrings 

func (profile SeparatorProfile) SeparatorStrings() []string

SeparatorStrings returns a detached separator slice.

type Strategy 

type Strategy string

Strategy identifies a chunking algorithm. 

const (
	SmartBoundary    Strategy = "smart_boundary"
	SentenceBoundary Strategy = "sentence_boundary"
	WordBoundary     Strategy = "word_boundary"
	MarkdownAware    Strategy = "markdown_aware"
	HeadingAware     Strategy = "heading_aware"
	ParagraphAware   Strategy = "paragraph_aware"
	HardCut          Strategy = "hard_cut"
	TokenBased       Strategy = "token_based"
	Semantic         Strategy = "semantic"
)
const Optimal Strategy = "optimal"

Optimal strategy registers a size-gating pass-through chunker that prefers boundary-aware splits only when the accumulated text exceeds OptimalLength/2.

type TiktokenCounter 

type TiktokenCounter struct {
	// contains filtered or unexported fields
}

TiktokenCounter implements TokenCounter using a cached tiktoken encoder. Create one with NewTiktokenCounter.

func NewTiktokenCounter 

func NewTiktokenCounter(encoding string, maxTokens int) (*TiktokenCounter, error)

NewTiktokenCounter creates a TiktokenCounter for the given encoding and maximum token budget. Empty encoding defaults to "cl100k_base". maxTokens <= 0 disables token limiting (pass-through in EnforceTokenLimits). Returns an error if the encoding name is not recognized by tiktoken.

func (*TiktokenCounter) CountTokens 

func (t *TiktokenCounter) CountTokens(text string) int

CountTokens returns the number of tokens in text using the configured encoding. Returns 0 if the counter is nil or if the encoder fails (should be unreachable after constructor validation).

func (*TiktokenCounter) MaxTokens 

func (t *TiktokenCounter) MaxTokens() int

MaxTokens returns the maximum allowed tokens per chunk. Returns 0 if the counter is nil, which disables token limiting.

type TiktokenTokenizer added in v0.8.0 

type TiktokenTokenizer struct {
	// contains filtered or unexported fields
}

TiktokenTokenizer provides OpenAI-compatible preflight estimates. API responses remain authoritative for actual request usage.

func NewTiktokenTokenizer added in v0.8.0 

func NewTiktokenTokenizer(encoding string) (*TiktokenTokenizer, error)

NewTiktokenTokenizer creates an OpenAI preflight tokenizer for encoding. Empty encoding defaults to cl100k_base.

func (*TiktokenTokenizer) Count added in v0.8.0 

func (t *TiktokenTokenizer) Count(text string) (int, error)

Count returns the number of tokens in text.

func (*TiktokenTokenizer) Encode added in v0.8.0 

func (t *TiktokenTokenizer) Encode(text string) ([]int, error)

Encode returns token IDs for text.

type TokenCounter 

type TokenCounter interface {
	// CountTokens returns the number of tokens in text.
	CountTokens(text string) int
	// MaxTokens returns the maximum allowed tokens per chunk.
	// Returns 0 to disable token limiting (passthrough).
	MaxTokens() int
}

TokenCounter abstracts token counting for budget enforcement. Implementations wrap a token encoder (e.g., tiktoken) to provide model-specific token budgets.

type Tokenizer added in v0.8.0 

type Tokenizer interface {
	Encode(text string) ([]int, error)
	Count(text string) (int, error)
}

Tokenizer is the consumer-owned tokenization boundary used for chunk sizing and preflight estimates. Implementations must use the tokenizer associated with the target provider or local model.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.