Documentation
¶
Overview ¶
Package lexer implements SQLite's tokenizer.
It is a direct port of sqlite3GetToken() in src/tokenize.c together with the token post-processing sqlite3RunParser() performs around it: runs of whitespace and comments are dropped, and the WINDOW/OVER/FILTER keywords are resolved against their neighbours (analyzeWindowKeyword and friends).
Illegal input is not reported here. sqlite3RunParser aborts at the first TK_ILLEGAL token it reaches, which is exactly as far as the parser had got, so meyer keeps ILLEGAL tokens in the stream and lets the parser report "unrecognized token" when it reaches one. That preserves SQLite's ordering between tokenizer errors and syntax errors.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Lex ¶
Lex splits src into tokens. Whitespace and comments are dropped; the returned slice always ends with a single EOF token whose Pos is the end of the consumed input.
A NUL byte terminates the input, matching SQLite, whose tokenizer walks a NUL-terminated buffer.
func LexFile ¶ added in v0.1.2
LexFile splits src into tokens like Lex, and also returns the trivia Lex drops: the runs of whitespace and comments, in source order, as SPACE and COMMENT tokens. "Trivia" is Roslyn's name — the C# compiler calls the channel of source text the grammar never sees "syntax trivia", and swift-syntax and rust-analyzer use the same term — adopted here because meyer's consumers (formatters) speak it too.
Tokens and trivia together tile the consumed input exactly: every byte before the EOF token's position belongs to exactly one token or trivia span, in order, with no overlap. Both slices come from the single pass Lex already makes; nothing is scanned twice.
Types ¶
This section is empty.
Source Files