Documentation
¶
Overview ¶
Package lex implements the TDL lexer, turning source text into a stream of tokens for the parser.
Index ¶
Constants ¶
const ( IdentPattern = `[_A-Za-z][_A-Za-z0-9]*` IntPattern = `-?[0-9]+` FloatPattern = `-?[0-9]+\.[0-9]+` StringPattern = `"([^"\\\n]|\\["\\nt])*"` DocPattern = `///[^\n]*` RegexPattern = `/([^/\\\n]|\\[^\n])*/` // LineCommentPattern is the comment the lexer skips; it has no Kind. // It also matches a doc comment, so a consumer tries DocPattern first. LineCommentPattern = `//[^\n]*` )
Patterns for the token classes scanned by shape, as accepted by Lexer.Next and Lexer.RescanRegexAt. TestPatternsMatchTheLexer holds them to the lexer. They are unanchored.
Variables ¶
This section is empty.
Functions ¶
func Keywords ¶ added in v0.1.1
func Keywords() []string
Keywords returns every reserved keyword, sorted.
func Punctuation ¶ added in v0.1.1
func Punctuation() []string
Punctuation returns the spelling of every operator and delimiter, sorted.
Types ¶
type Comment ¶ added in v0.1.7
type Comment struct {
Text string // the text after the slashes, with one leading space removed
Pos Position
End int // offset just past the comment's last character
}
Comment is an ordinary `//` comment, which Lexer.Next skips and collects for the formatter to place by position. A `///` doc comment is a DOC token instead.
type Kind ¶
type Kind int
Kind identifies the lexical class of a Token.
const ( ILLEGAL Kind = iota EOF IDENT DOC // /// doc comment, text only STRING // "..." INT // 123 FLOAT // 1.23 REGEX // /.../, scanned only on demand; see [Lexer.RescanRegexAt] // Reserved keywords. Modifiers and constraint names are contextual and // lex as IDENT. PACKAGE IMPORT AS PRIMITIVE UNIT ALIAS TYPE ENUM CLASS MIXIN INSTANCE TARGET FOR REQUIRES WHERE INCLUDE NULL TRUE FALSE LBRACE // { RBRACE // } LPAREN // ( RPAREN // ) LBRACK // [ RBRACK // ] LT // < GT // > COLON // : COMMA // , EQUAL // = QUESTION // ? DOT // . PIPE // | ARROW // -> RANGE // .. CARET // ^ STAR // * SLASH // / FATARROW // => )
func Lookup ¶ added in v0.1.1
Lookup returns the kind the lexer produces for a fixed spelling, whether keyword, operator, or delimiter. It reports false for anything scanned by shape, which has no single spelling to look up.
func LookupIdent ¶
LookupIdent returns the keyword Kind for ident, or IDENT if ident is not a reserved keyword.
type Lexer ¶
type Lexer struct {
// contains filtered or unexported fields
}
Lexer scans TDL source text into a stream of [Token]s. It emits no newline tokens; whitespace is insignificant.
func (*Lexer) Comments ¶ added in v0.1.7
Comments returns a copy of every ordinary comment scanned so far, in source order.
func (*Lexer) Next ¶
Next scans and returns the next token. It returns an EOF token forever once the end of input is reached.
func (*Lexer) RescanRegexAt ¶
RescanRegexAt rescans the input from pos as a regex literal and leaves the lexer positioned after it.
`/` is also division in a unit expression, and only the parser knows which one it wants. Every other token is scanned without context.
Source Files