Versions in this module Expand all Collapse all v0 v0.16.0 Apr 28, 2023 Changes in this version + var DefaultSeparators = []rune + type Tokenizer struct + func New(r io.Reader) *Tokenizer + func NewFromBytes(b []byte) *Tokenizer + func NewFromString(str string) *Tokenizer + func (s *Tokenizer) KeepSeparator(state bool) + func (s *Tokenizer) Scan() (string, error) + func (s *Tokenizer) ScanAll() ([]string, error) + func (s *Tokenizer) Separators(separators ...rune)
Go to main page