Documentation
¶
Overview ¶
Package tokenizer provides a struct that handles tokenisation of a string.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type NumberFlags ¶
type NumberFlags byte
NumberFlags represents the type of number being parsed.
const ( // NumberFlagFloat represents a floating point number. NumberFlagFloat NumberFlags = 1 << iota // NumberFlagExponent represents an exponent number. NumberFlagExponent )
type Tokenizer ¶
type Tokenizer struct {
// contains filtered or unexported fields
}
Tokenizer defines the tokenizer itself.
func NewTokenizer ¶
NewTokenizer creates a new instance of the Tokenizer struct.
Source Files
¶
Click to show internal directories.
Click to hide internal directories.