tokenizer

package
v0.0.0-...-1339caa Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 17, 2019 License: MIT Imports: 3 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Token 

type Token struct {
	Type TokenType
	Raw  string
}

Token represents a single token.

func (*Token) String 

func (t *Token) String() string

type TokenType 

type TokenType int8

TokenType represents a Token's type. 

const (
	Unknown TokenType = iota

	Identifier
	Subquery

	Select
	From
	Where

	As
	Or
	And
	Not

	In
	Is
	Like
	RLike

	Equals
	NotEquals
	GreaterThanEquals
	GreaterThan
	LessThanEquals
	LessThan

	Comma
	Hyphen
	ExclamationMark
	OpenParen
	CloseParen
	OpenBracket
	CloseBracket
)

All TokenType constants.

func (TokenType) String 

func (t TokenType) String() string

type Tokenizer 

type Tokenizer struct {
	// contains filtered or unexported fields
}

Tokenizer represents a token worker.

func NewTokenizer 

func NewTokenizer(input string) *Tokenizer

NewTokenizer initializes a new Tokenizer.

func (*Tokenizer) All 

func (t *Tokenizer) All() []Token

All parses all tokens for this Tokenizer.

func (*Tokenizer) Next 

func (t *Tokenizer) Next() *Token

Next finds and returns the next Token in the input string.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.