Versions in this module Expand all Collapse all v1 v1.1.0 Oct 29, 2026 v1.0.0 Oct 26, 2026 Changes in this version + var DefaultSeparators = []rune + type Tokenizer struct + func New(r io.Reader) *Tokenizer + func NewFromBytes(b []byte) *Tokenizer + func NewFromString(str string) *Tokenizer + func (t *Tokenizer) IgnoreParenthesis(state bool) + func (t *Tokenizer) KeepEmptyTokens(state bool) + func (t *Tokenizer) KeepSeparator(state bool) + func (t *Tokenizer) Scan() (string, error) + func (t *Tokenizer) ScanAll() ([]string, error) + func (t *Tokenizer) Separators(separators ...rune)
Go to main page