Why Go
- Case Studies
  
  Common problems companies solve with Go
- Use Cases
  
  Stories about how and why companies use Go
- Security
  
  How Go can help keep you secure by default
Learn
Docs
- Effective Go
  
  Tips for writing clear, performant, and idiomatic Go code
- Go User Manual
  
  A complete introduction to building software with Go
- Standard library
  
  Reference documentation for Go's standard library
- Release Notes
  
  Learn what's new in each Go release
Packages
Community
- Recorded Talks
  
  Videos from prior events
- Meetups
  
  Meet other local Go developers
- Conferences
  
  Learn and network with Go developers from around the world
- Go blog
  
  The Go project's official blog.
- Go project
  
  Get help and stay informed from Go
- Get connected

api

package

Go to main page

Versions in this module

v0

v0.2.2

Jun 27, 2025

v0.2.1

Jun 3, 2025

v0.2.0

Apr 30, 2025

v0.1.2

Apr 8, 2025

v0.1.1

Dec 9, 2024

Changes in this version

+ func SpecialTokenStrings() []string

type SpecialToken

+ func SpecialTokenString(s string) (SpecialToken, error)

+ func SpecialTokenValues() []SpecialToken

+ func (SpecialToken) Values() []string

+ func (i *SpecialToken) UnmarshalJSON(data []byte) error

+ func (i *SpecialToken) UnmarshalText(text []byte) error

+ func (i *SpecialToken) UnmarshalYAML(unmarshal func(interface{}) error) error

+ func (i SpecialToken) IsASpecialToken() bool

+ func (i SpecialToken) MarshalJSON() ([]byte, error)

+ func (i SpecialToken) MarshalText() ([]byte, error)

+ func (i SpecialToken) MarshalYAML() (interface{}, error)

+ func (i SpecialToken) String() string

v0.1.0

Nov 9, 2024

Changes in this version

+ type Config struct

+ AddBosToken bool

+ AddEosToken bool

+ AddedTokensDecoder map[int]TokensDecoder

+ AdditionalSpecialTokens []string

+ BosToken string

+ ChatTemplate string

+ CleanUpTokenizationSpaces bool

+ ClsToken string

+ ConfigFile string

+ DoBasicTokenize bool

+ DoLowerCase bool

+ EosToken string

+ MaskToken string

+ MaxLength float64

+ ModelMaxLength float64

+ NameOrPath string

+ NeverSplit any

+ PadToken string

+ SepToken string

+ SpModelKwargs map[string]any

+ SpacesBetweenSpecialTokens bool

+ Stride int

+ StripAccents any

+ TokenizeChineseChars bool

+ TokenizerClass string

+ TruncationSide string

+ TruncationStrategy string

+ UnkToken string

+ UseDefaultSystemPrompt bool

+ func ParseConfigContent(jsonContent []byte) (*Config, error)

+ func ParseConfigFile(filePath string) (*Config, error)

+ type SpecialToken int

+ const TokBeginningOfSentence

+ const TokClassification

+ const TokEndOfSentence

+ const TokMask

+ const TokPad

+ const TokSpecialTokensCount

+ const TokUnknown

+ type Tokenizer interface

+ Decode func([]int) string

+ Encode func(text string) []int

+ SpecialTokenID func(token SpecialToken) (int, error)

+ type TokensDecoder struct

+ Content string

+ Lstrip bool

+ Normalized bool

+ Rstrip bool

+ SingleWord bool

+ Special bool