lexer

Overview ¶

Package lexer implements SQLite's tokenizer.

It is a direct port of sqlite3GetToken() in src/tokenize.c together with the token post-processing sqlite3RunParser() performs around it: runs of whitespace and comments are dropped, and the WINDOW/OVER/FILTER keywords are resolved against their neighbours (analyzeWindowKeyword and friends).

Illegal input is not reported here. sqlite3RunParser aborts at the first TK_ILLEGAL token it reaches, which is exactly as far as the parser had got, so meyer keeps ILLEGAL tokens in the stream and lets the parser report "unrecognized token" when it reaches one. That preserves SQLite's ordering between tokenizer errors and syntax errors.