Documentation
¶
Index ¶
- Constants
- func HasUChars(r rune, l *Lexer) bool
- func HasXChars(r rune, l *Lexer) bool
- func IRIRef(l *Lexer, styp ItemType) error
- func IsEndOfLine(r rune) bool
- type CheckRune
- type CheckRuneRec
- type Item
- type ItemIterator
- func (p *ItemIterator) Errorf(format string, args ...interface{}) error
- func (p *ItemIterator) Item() Item
- func (p *ItemIterator) Next() bool
- func (p *ItemIterator) Peek(num int) ([]Item, error)
- func (p *ItemIterator) PeekOne() (Item, bool)
- func (p *ItemIterator) Prev() bool
- func (p *ItemIterator) Restore(pos int)
- func (p *ItemIterator) Save() int
- type ItemType
- type Lexer
- func (l *Lexer) AcceptRun(c CheckRune) (lastr rune, validr bool)
- func (l *Lexer) AcceptRunRec(c CheckRuneRec)
- func (l *Lexer) AcceptRunTimes(c CheckRune, times int) int
- func (l *Lexer) AcceptUntil(c CheckRune)
- func (l *Lexer) Backup()
- func (l *Lexer) Emit(t ItemType)
- func (l *Lexer) Errorf(format string, args ...interface{}) StateFn
- func (l *Lexer) Ignore()
- func (l *Lexer) IgnoreRun(c CheckRune)
- func (l *Lexer) IsEscChar(r rune) bool
- func (l *Lexer) LexQuotedString() error
- func (l *Lexer) NewIterator() *ItemIterator
- func (l *Lexer) Next() (result rune)
- func (l *Lexer) Peek() rune
- func (l *Lexer) PeekTwo() []rune
- func (l *Lexer) Reset(input string)
- func (l *Lexer) Run(f StateFn) *Lexer
- func (l *Lexer) ValidateResult() error
- type RuneWidth
- type StateFn
Constants ¶
const EOF = -1
EOF indicates the end of the an input.
Variables ¶
This section is empty.
Functions ¶
func HasUChars ¶
HasUChars returns whether the lexer is at the beginning of a escaped Unicode character. UCHAR ::= '\u' HEX HEX HEX HEX | '\U' HEX HEX HEX HEX HEX HEX HEX HEX
func HasXChars ¶
HasXChars returns whether the lexer is at the start of a escaped hexadecimal byte (e.g \xFE) XCHAR ::= '\x' HEX HEX
func IsEndOfLine ¶
IsEndOfLine returns true if the rune is a Linefeed or a Carriage return.
Types ¶
type CheckRuneRec ¶
CheckRuneRec is like CheckRune with Lexer as extra argument. This can be used to recursively call other CheckRune(s).
type Item ¶
Item represents a unit emitted by the lexer.
type ItemIterator ¶
type ItemIterator struct {
// contains filtered or unexported fields
}
ItemIterator iterates over the items emitted by a lexer.
func (*ItemIterator) Errorf ¶
func (p *ItemIterator) Errorf(format string, args ...interface{}) error
Errorf returns an error message using the location of the next item in the iterator.
func (*ItemIterator) Peek ¶
func (p *ItemIterator) Peek(num int) ([]Item, error)
Peek returns the next n items without consuming them.
func (*ItemIterator) PeekOne ¶
func (p *ItemIterator) PeekOne() (Item, bool)
PeekOne returns the next 1 item without consuming it.
func (*ItemIterator) Restore ¶
func (p *ItemIterator) Restore(pos int)
Restore restores the iterator to position specified.
func (*ItemIterator) Save ¶
func (p *ItemIterator) Save() int
Save returns the current position of the iterator which we can use for restoring later.
type ItemType ¶
type ItemType int
ItemType is used to set the type of a token. These constants can be defined in the file containing state functions. Note that their value should be >= 5.
type Lexer ¶
type Lexer struct {
// NOTE: Using a text scanner wouldn't work because it's designed for parsing
// Golang. It won't keep track of Start Position, or allow us to retrieve
// slice from [Start:Pos]. Better to just use normal string.
Input string // string being scanned.
Start int // Start Position of this item.
Pos int // current Position of this item.
Width int // Width of last rune read from input.
Depth int // nesting of {}
BlockDepth int // nesting of blocks (e.g. mutation block inside upsert block)
ArgDepth int // nesting of ()
Mode StateFn // Default state to go back to after reading a token.
Line int // the current line number corresponding to Start
Column int // the current column number corresponding to Start
// contains filtered or unexported fields
}
Lexer converts a raw input into tokens.
func (*Lexer) AcceptRun ¶
AcceptRun accepts tokens based on CheckRune until it returns false or EOF is reached. Returns last rune accepted and valid flag for rune.
func (*Lexer) AcceptRunRec ¶
func (l *Lexer) AcceptRunRec(c CheckRuneRec)
AcceptRunRec accepts tokens based on CheckRuneRec until it returns false or EOF is reached.
func (*Lexer) AcceptRunTimes ¶
AcceptRunTimes accepts tokens with CheckRune given number of times. returns number of times it was successful.
func (*Lexer) AcceptUntil ¶
AcceptUntil accepts tokens based on CheckRune till it returns false or EOF is reached.
func (*Lexer) Backup ¶
func (l *Lexer) Backup()
Backup moves the lexer back to its previous position.
func (*Lexer) Ignore ¶
func (l *Lexer) Ignore()
Ignore skips the current token. Meant to be used for tokens that do not have any syntactical meaning (e.g comments).
func (*Lexer) IsEscChar ¶
IsEscChar returns true if the run is an escape character (ECHAR ::= '\' [uvtbnrf"'\])
func (*Lexer) LexQuotedString ¶
LexQuotedString properly processes a quoted string (by taking care of escaped characters).
func (*Lexer) NewIterator ¶
func (l *Lexer) NewIterator() *ItemIterator
NewIterator returns a new ItemIterator instance that uses the lexer.
func (*Lexer) ValidateResult ¶
ValidateResult verifies whether the entire input can be lexed without errors.
type RuneWidth ¶
type RuneWidth struct {
// contains filtered or unexported fields
}
A RuneWidth represents a consecutive string of runes with the same width and the number of runes is stored in count. The reason we maintain this information is to properly backup when multiple look-aheads happen. For example, if the following sequence of events happen 1. Lexer.Next() consumes 1 byte 2. Lexer.Next() consumes 1 byte 3. Lexer.Next() consumes 3 bytes we would create two RunWidthTrackers, the 1st having width 1 and count 2, while the 2nd having width 3 and count 1, then the following backups can be done properly: 4. Lexer.Backup() should decrement the pos by 3 5. Lexer.Backup() should decrement the pos by 1 6. Lexer.Backup() should decrement the pos by 1
Source Files