Documentation
¶
Overview ¶
Package tokenizer provides a struct that handles tokenisation of a string.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type NumberFlags ¶
type NumberFlags byte
NumberFlags represents the type of number being parsed.
const ( // NumberFlagFloat represents a floating point number. NumberFlagFloat NumberFlags = 1 << iota // NumberFlagExponent represents an exponent number. NumberFlagExponent )
type Tokenizer ¶
type Tokenizer struct {
// contains filtered or unexported fields
}
Tokenizer defines the tokenizer itself.
func NewTokenizer ¶
NewTokenizer creates a new instance of the Tokenizer struct.
Source Files
¶
- get_next.go
- handle_ampersand_sign.go
- handle_asterisk_sign.go
- handle_equal_sign.go
- handle_exclamation_sign.go
- handle_greater_than_sign.go
- handle_identifier.go
- handle_less_than_sign.go
- handle_minus_sign.go
- handle_number_or_spread.go
- handle_percent_sign.go
- handle_pipe_sign.go
- handle_plus_sign.go
- handle_slash_sign.go
- handle_string.go
- handle_unknown_char.go
- parse_number.go
- parse_spread.go
- peek.go
- tokenize.go
- tokenizer.go
Click to show internal directories.
Click to hide internal directories.