Documentation
¶
Overview ¶
Package utils provides utility functions for v2.
Package utils provides utility functions for v2.
Index ¶
Constants ¶
const ( // DefaultHectorDir is the default directory for Hector data DefaultHectorDir = ".hector" // DefaultConfigFileName is the default config file name DefaultConfigFileName = "config.yaml" // DefaultDatabaseFileName is the default database file name DefaultDatabaseFileName = "hector.db" )
Default paths for Hector files
Variables ¶
This section is empty.
Functions ¶
func DefaultConfigPath ¶
func DefaultConfigPath() string
DefaultConfigPath returns the default config file path: .hector/config.yaml
func DefaultDatabasePath ¶
func DefaultDatabasePath() string
DefaultDatabasePath returns the default database file path: .hector/hector.db
func EnsureHectorDir ¶
EnsureHectorDir ensures the .hector directory exists at the given base path. If basePath is empty or ".", it creates ./.hector in the current directory. Otherwise, it creates {basePath}/.hector.
This is used by various facilities that need to store data in .hector: - Config file: ./.hector/config.yaml - Tasks database: ./.hector/hector.db - Document store index state: {sourcePath}/.hector/index_state_*.json - Checkpoints: {sourcePath}/.hector/checkpoints/ - Vector stores: {sourcePath}/.hector/vectors/
Returns the full path to the .hector directory and any error.
func EstimateTokens ¶
EstimateTokens provides a quick token estimate without a counter. Uses ~4 characters per token as a rough approximation.
func GetEncodingForModel ¶
GetEncodingForModel returns the appropriate encoding name for a model.
Types ¶
type TokenCounter ¶
type TokenCounter struct {
// contains filtered or unexported fields
}
TokenCounter provides token counting functionality using tiktoken. Ported from pkg/utils/tokens.go for use in v2.
func NewTokenCounter ¶
func NewTokenCounter(model string) (*TokenCounter, error)
NewTokenCounter creates a new token counter for the given model.
func (*TokenCounter) Count ¶
func (tc *TokenCounter) Count(text string) int
Count returns the number of tokens in the given text.
func (*TokenCounter) CountMessages ¶
func (tc *TokenCounter) CountMessages(messages []Message) int
CountMessages returns the approximate token count for a list of messages. Uses the ChatML format estimation (3 tokens per message + role + content).
func (*TokenCounter) EstimateTokensForText ¶
func (tc *TokenCounter) EstimateTokensForText(text string) int
EstimateTokensForText returns an estimate of tokens for text. If the counter is not initialized, falls back to character-based estimation.
func (*TokenCounter) FitWithinLimit ¶
func (tc *TokenCounter) FitWithinLimit(messages []Message, maxTokens int) []Message
FitWithinLimit returns messages that fit within the token limit. It works backwards from the most recent message, keeping as many as fit. This preserves the most recent context which is typically most relevant.
func (*TokenCounter) GetModel ¶
func (tc *TokenCounter) GetModel() string
GetModel returns the model name used by this counter.
Source Files