Documentation
¶
Index ¶
- Constants
- func TokenEnd(tok *Token) (endLine, endCol, endPos int)
- type Location
- type LocationError
- type Rune
- type Scanner
- func (s *Scanner) Accept(fn func(rune) bool) bool
- func (s *Scanner) AcceptAny(charset string) bool
- func (s *Scanner) AcceptDigit() bool
- func (s *Scanner) AcceptRune(c rune) bool
- func (s *Scanner) AcceptSeq(fn func(rune) bool) int
- func (s *Scanner) AcceptSeqAny(charset string) int
- func (s *Scanner) AcceptSeqDigit() int
- func (s *Scanner) AcceptSeqRune(c rune) int
- func (s *Scanner) AcceptSeqSpace() int
- func (s *Scanner) AcceptSpace() bool
- func (s *Scanner) AcceptString(literal string) (int, bool)
- func (s *Scanner) EOF() bool
- func (s *Scanner) EmitToken(typ Type) *Token
- func (s *Scanner) Err() error
- func (s *Scanner) Ignore()
- func (s *Scanner) Loc() *Location
- func (s *Scanner) LocStart() *Location
- func (s *Scanner) Peek() (rune, bool)
- func (s *Scanner) Rune() rune
- func (s *Scanner) ScanRune() error
- func (s *Scanner) SetPath(path string)
- func (s *Scanner) Text() string
- type Source
- type Token
- type Type
Constants ¶
const DefaultBufSize = 128 << 10
DefaultBufSize is the size of the sliding window NewScanner allocates. It is also the largest single token NewScanner can scan: the window never grows, so a token that fills it fails with "token exceeds maximum allowable size".
Variables ¶
This section is empty.
Functions ¶
Types ¶
type Location ¶
type Location struct {
File string // a name representing the source stream
Path string // a physical location which may differ from File
Pos int
Line int // line number (starting at 1 when tracked)
Col int // line column number (starting at 1 when tracked)
EndPos int // byte position past end of token/expr (0 = not tracked)
EndLine int // end line (1-based, 0 = not tracked)
EndCol int // end column (1-based, exclusive, 0 = not tracked)
}
type LocationError ¶
type LocationError struct {
Err error
Source *Location
Code string // error classification (empty = unclassified)
}
func (*LocationError) Error ¶
func (err *LocationError) Error() string
func (*LocationError) Unwrap ¶ added in v1.17.0
func (err *LocationError) Unwrap() error
Unwrap returns the underlying error for use with errors.Is/errors.As.
type Rune ¶
Rune contains a rune that read by Scanner during peeking operations.
func (Rune) IsRuneError ¶
IsRuneError returns true if Rune represents an invalid utf-8 sequence read by utf8.DecodeRune.
type Scanner ¶
type Scanner struct {
// contains filtered or unexported fields
}
Scanner facilitates construction of tokens from a byte stream (io.Reader).
func NewScanner ¶
NewScanner initializes and returns a new Scanner reading through a DefaultBufSize sliding window.
func NewScannerString ¶ added in v1.49.0
NewScannerString initializes and returns a new Scanner reading src, sizing the sliding window to src rather than allocating the DefaultBufSize window NewScanner uses. src is already in memory, so the window costs nothing extra and no token can overrun it.
This exists because the fixed window is charged per SCANNER, not per byte scanned, and the parser re-reads short strings: readsBackAsSymbol scans each #' operand and each half of every package-qualified symbol to check that it reads back as a symbol (issue #319). Through NewScanner that is 128KiB per check -- 2.6GB and 1.7s to parse 10k qualified symbols, which is both a pointless cost on ordinary source and an allocation amplification an attacker controls, in a parser whose job is to survive untrusted phylum source.
func (*Scanner) AcceptDigit ¶
func (*Scanner) AcceptRune ¶
func (*Scanner) AcceptSeqAny ¶
func (*Scanner) AcceptSeqDigit ¶
func (*Scanner) AcceptSeqRune ¶
func (*Scanner) AcceptSeqSpace ¶
func (*Scanner) AcceptSpace ¶
func (*Scanner) EmitToken ¶
EmitToken returns a token containing the text scanned since the last call to either EmitToken or Ignore.
func (*Scanner) Err ¶
Err returns an error encountered during the last read on the input stream. Err will always return false while there are still buffered runes that need to be accepted.
func (*Scanner) Ignore ¶
func (s *Scanner) Ignore()
Ignore causes the scanner to skip all text scanned since the last call to either EmitToken or Ignore.
func (*Scanner) Loc ¶
Loc returns a Location referencing the current scanner position, the last position of the current token.
func (*Scanner) LocStart ¶
LocStart returns a Location referencing the beginning of the current token, just beyond the end of the previous token.
func (*Scanner) Peek ¶
Peek returns the next rune to be scanned, if there are any. If an invalid utf-8 sequence or EOF prevents futher runes from being scanned Peek returns a false second value. If Peek returns a false value the next call to s.ScanRune will return an error that reflects of the cause.
func (*Scanner) Rune ¶
Rune returns the current unicode rune that is being scanned. The rune returned by Rune is the last rune in a token returned by EmitToken.
func (*Scanner) ScanRune ¶
ScanRune attempts to scan a utf-8 rune from the input for inclusion in the current token. If an error prevents a valid unicode rune from being scanned then an error will be returned.
type Source ¶
type Source interface {
// Token returns the current token. Token returns nil if Scan has not been
// called.
Token() *Token
// Peek returns the next token in the stream. At the end of the stream
// Peek should return a value to indicate the lack of a token (EOF).
Peek() *Token
// Scan advances the token stream if possible. If there are no tokens
// remaining Scan returns false.
Scan() bool
}
Source is an abstract stream of tokens which allows one token lookahead.
type Type ¶
type Type uint
const ( INVALID Type = iota ERROR EOF HASH_BANG // Atomic expressions & literals SYMBOL INT INT_OCTAL_MACRO INT_OCTAL INT_HEX_MACRO INT_HEX FLOAT STRING STRING_RAW COMMENT // Operators NEGATIVE // arithmetic negation is parsed specially QUOTE UNBOUND FUN_REF // Delimiters PAREN_L PAREN_R BRACE_L BRACE_R )
Type constants used for the elps lexer/parser. These constants aren't necessary to use the package.
Source Files