Documentation
¶
Overview ¶
Package api defines the Lua lexer (scanner) types and interface.
The lexer converts Lua source text into a stream of tokens. It handles all escape sequences, long strings, comments, number parsing, and reserved word detection.
Reference: .analysis/06-compiler-pipeline.md §2
Full Lua 5.5.1 lexer implementation.
Converts Lua source text into a stream of tokens. Handles all escape sequences, long strings, comments, number parsing, and reserved word detection.
Reference: lua-master/llex.c
Index ¶
Constants ¶
const EOZ = -1
EOZ signals end of input (-1, matching C's EOZ).
const FirstReserved = 256
FirstReserved is the boundary between single-char tokens and reserved words.
const NumReservedCount = NumReserved - FirstReserved
NumReservedCount is the actual count of reserved words (22). NumReserved above is a token value, not a count. Note: "global" is a context-sensitive (soft) keyword handled by the parser, not a reserved word in the lexer. It scans as TK_NAME.
Variables ¶
var ReservedWords = map[string]TokenType{ "and": TK_AND, "break": TK_BREAK, "do": TK_DO, "else": TK_ELSE, "elseif": TK_ELSEIF, "end": TK_END, "false": TK_FALSE, "for": TK_FOR, "function": TK_FUNCTION, "goto": TK_GOTO, "if": TK_IF, "in": TK_IN, "local": TK_LOCAL, "nil": TK_NIL, "not": TK_NOT, "or": TK_OR, "repeat": TK_REPEAT, "return": TK_RETURN, "then": TK_THEN, "true": TK_TRUE, "until": TK_UNTIL, "while": TK_WHILE, }
--------------------------------------------------------------------------- ReservedWords maps reserved word strings to their token types. Used during identifier scanning to detect keywords. ---------------------------------------------------------------------------
Functions ¶
func SetInput ¶
func SetInput(ls *LexState)
SetInput reads the first character and prepares the lexer for scanning.
Types ¶
type LexReader ¶
type LexReader interface {
// NextByte returns the next byte, or -1 on EOF.
NextByte() int
}
LexReader is the interface for reading source characters.
type LexState ¶
type LexState struct {
Current int // current character (rune or -1 for EOF)
Line int // current input line number
LastLine int // line of last consumed token
Token Token // current token
Lookahead Token // lookahead token (Type == TK_EOS if empty)
HasAhead bool // whether lookahead is populated
Source string // source name (for error messages)
EnvName string // "_ENV" string
BreakName string // "break" string (used as label name)
// Input reading (set during initialization)
Reader LexReader // character source
Buf []byte // token buffer
// Back-reference to parser state (set by parser)
// Uses any to avoid circular import with parse module.
FuncState any // current FuncState (set by parser)
DynData any // shared dynamic data (set by parser)
// Parser nesting depth counter (mirrors C Lua's L->nCcalls)
NestLevel int // current nesting depth
MaxNestLevel int // maximum allowed nesting (default 200)
}
--------------------------------------------------------------------------- LexState is the lexer state. It holds the input stream, current token, lookahead token, and line tracking.
This is a concrete struct (not an interface) because the parser directly accesses its fields for performance and simplicity. ---------------------------------------------------------------------------
func NewLexState ¶
NewLexState creates a new lexer state from a reader and source name.
type SyntaxError ¶
type SyntaxError struct {
Source string
Line int
Token string // token that caused the error (or "")
Message string
}
SyntaxError is raised (via panic) for lexical/syntax errors.
func (*SyntaxError) Error ¶
func (e *SyntaxError) Error() string
type Token ¶
type Token struct {
Type TokenType
IntVal int64 // for TK_INT
FltVal float64 // for TK_FLT
StrVal string // for TK_NAME, TK_STRING
}
--------------------------------------------------------------------------- Token carries the token type and its semantic value. ---------------------------------------------------------------------------
type TokenType ¶
type TokenType int
TokenType is the type of a lexical token.
const ( TK_AND TokenType = iota + FirstReserved TK_BREAK TK_DO TK_ELSE TK_ELSEIF TK_END TK_FALSE TK_FOR TK_FUNCTION TK_GOTO TK_IF TK_IN TK_LOCAL TK_NIL TK_NOT TK_OR TK_REPEAT TK_RETURN TK_THEN TK_TRUE TK_UNTIL TK_WHILE NumReserved // count of reserved words = 22 )
Reserved words (FirstReserved + index)
const ( TK_IDIV TokenType = iota + NumReserved // // TK_CONCAT // .. TK_DOTS // ... TK_EQ // == TK_GE // >= TK_LE // <= TK_NE // ~= TK_SHL // << TK_SHR // >> TK_DBCOLON // :: TK_EOS // <eof> TK_FLT // <number> (float) TK_INT // <integer> TK_NAME // <name> TK_STRING // <string> )
Multi-char operators and value tokens
Source Files