Documentation
¶
Overview ¶
Package sqltok splits SQL into tokens and does nothing else with them.
IT HAS TWO CALLERS AND THAT IS WHY IT EXISTS. `refs` asks which relations a query reads; `serve` asks whether a statement only reads. Both questions turn on the same thing -- telling CODE from a string literal, a quoted identifier and a comment -- and both are dialect-specific in the same places: `#` starts a comment in BigQuery and is an operator in Postgres, `"..."` is a string in BigQuery and an identifier in Postgres.
Those rules in two copies would be two copies that drift, and the one that drifts is a classifier deciding that `/* SELECT */ DROP TABLE t` is a SELECT. One lexer, one place to be wrong, one place to fix.
IT IS NOT A PARSER, deliberately. The measurement behind that choice is in refs's own doc comment: the official parsers weigh 14 to 37 MB and one of them read 6 of 447 real statements. This is 1.9 MB of standard library, and what it cannot do is the reason its callers refuse rather than guess.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Kind ¶
type Kind int
Kind is what a token IS, which is the only classification this package makes. A Literal keeps no text: nothing downstream may read inside a string a customer wrote, and the surest way to guarantee that is to not carry it.
type Token ¶
type Token struct {
Kind Kind
Text string // as written; a quoted identifier without its quotes
Lower string
}
Token is one piece of a statement.
func Tokenize ¶
Tokenize drops comments and the contents of string literals and splits the rest. `#` starts a comment in BigQuery only; in Postgres it is an operator.
THE SECOND RETURN SAYS WHETHER IT REACHED THE END, and no caller may ignore it by accident. An unterminated quote, backtick or block comment leaves this with no way to know where code resumes, so it stops -- and a caller that took the tokens alone would be judging a PREFIX. CHECKPOINT B found exactly that: `SELECT 1 /* ; DROP TABLE t` read as `SELECT 1`.
It was not exploitable, because a statement that truncates this also fails to parse at the warehouse. But that is two lexers agreeing rather than a decision anybody made, and the classifier's own rule is that what it cannot follow, it refuses.
Source Files