Documentation
¶
Overview ¶
Package lex implements the TDL lexer, turning source text into a stream of tokens for the parser.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type Kind ¶
type Kind int
Kind identifies the lexical class of a Token.
const ( ILLEGAL Kind = iota EOF IDENT // identifiers and keywords share a scanning path; Kind distinguishes them DOC // /// doc comment, text only STRING // "..." INT // 123 FLOAT // 1.23 REGEX // /.../, scanned only on demand; see [Lexer.RescanRegexAt] // Reserved keywords. Declaration keywords are reserved; modifiers and // constraint names (key, owned, deprecated, min, length, ...) are // contextual and lex as IDENT. PACKAGE IMPORT AS PRIMITIVE UNIT ALIAS TYPE VALUE ENTITY ENUM CLASS MIXIN INSTANCE TARGET FOR REQUIRES WHERE INCLUDE NULL TRUE FALSE UNION // reserved, not yet implemented by the parser // Punctuation LBRACE // { RBRACE // } LPAREN // ( RPAREN // ) LBRACK // [ RBRACK // ] LT // < GT // > COLON // : COMMA // , EQUAL // = QUESTION // ? DOT // . PIPE // | ARROW // -> RANGE // .. CARET // ^ STAR // * SLASH // / FATARROW // => )
func LookupIdent ¶
LookupIdent returns the keyword Kind for ident, or IDENT if ident is not a reserved keyword.
type Lexer ¶
type Lexer struct {
// contains filtered or unexported fields
}
Lexer scans TDL source text into a stream of [Token]s.
Whitespace is insignificant in TDL: the lexer emits no newline tokens and the parser has no separator rules. An item ends where the next begins.
func (*Lexer) Next ¶
Next scans and returns the next token. It returns an EOF token forever once the end of input is reached.
func (*Lexer) RescanRegexAt ¶
RescanRegexAt rescans the input from pos as a regex literal and leaves the lexer positioned after it.
`/` is division in a unit expression and the delimiter of a regex literal, and nothing local to the token tells them apart: `matches` is a contextual keyword, so the preceding token is an ordinary identifier either way. The parser knows which it wants, so it asks. Every other token is scanned by Lexer.Next without context.
Source Files