Documentation
¶
Overview ¶
Package lexer is a hand-written port of PostgreSQL's scanner as patched and pinned by libpg_query 18.0.0 (internal/reference/scan.l), plus the base_yylex token-merge filter from internal/reference/parser.c.
The scanner is byte-oriented and eager-capable but pull-based: each call to Next returns exactly what one core_yylex call would return, including the SQL_COMMENT / C_COMMENT tokens libpg_query's patches add. Token kinds are the protobuf Token enum values, which are defined to equal the bison token numbers of the pinned grammar.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var Keywords = map[string]Keyword{}/* 494 elements not displayed */
Keywords holds all 494 keywords of the pinned grammar, keyed by their lower-case SQL spelling.
Functions ¶
This section is empty.
Types ¶
type Error ¶
Error is a scanner error, pre-formatted exactly as the reference reports it (scanner_yyerror / in-action ereport calls). Filename and Funcname mirror the C error data; Cursorpos is the 1-based character (not byte) position, per scanner_errposition.
type Filter ¶
type Filter struct {
// contains filtered or unexported fields
}
Filter is parser.c's base_yylex: the one-token-lookahead layer between the core scanner and the grammar that keeps the grammar LALR(1) by merging token pairs (NOT_LA, NULLS_LA, WITH_LA, WITHOUT_LA, FORMAT_LA), converts UIDENT/USCONST (with optional UESCAPE) into plain IDENT/SCONST, and drops the comment tokens the patched scanner emits.
func (*Filter) Cursorpos ¶
Cursorpos is scanner_errposition for a byte offset: the 1-based character position.
func (*Filter) ParserError ¶
ParserError is an ereport(ERROR, ...) from a gram.y action or support function: no "at or near" decoration; funcname names the C function containing the ereport (base_yyparse for inline rule actions).
func (*Filter) SyntaxError ¶
SyntaxError builds bison's yyerror report for the given lookahead token: `syntax error at or near "<token text>"` (or "at end of input"), positioned at the token's start. The quoted text spans the token's match — parser.c's base_yylex pokes a NUL at the current token's end before any lookahead, so merged tokens (NOT_LA etc.) report only their first word.
type Keyword ¶
type Keyword struct {
Name string
Token string
Category KeywordCategory
BareLabel bool
}
Keyword is one kwlist.h entry. Token is the bison token name from gram.y (e.g. "ABORT_P"); BareLabel is the PG 14+ bare_label_keyword attribute (SELECT expr alias without AS).
type KeywordCategory ¶
type KeywordCategory int
KeywordCategory is PostgreSQL's keyword reserved-ness classification (kwlist.h). It maps 1:1 onto the protobuf KeywordKind enum minus NO_KEYWORD.
const ( Unreserved KeywordCategory = iota // unreserved_keyword ColName // col_name_keyword TypeFuncName // type_func_name_keyword Reserved // reserved_keyword )
type Scanner ¶
type Scanner struct {
// contains filtered or unexported fields
}
Scanner lexes one input string. The zero value is not usable; call New.
func New ¶
New returns a Scanner over input. Like the C scanner (which receives a NUL-terminated string), input is truncated at the first NUL byte.
func NewKeepingComments ¶ added in v0.2.0
NewKeepingComments returns a Scanner that also records the comment tokens it produces, for Comments to hand back after the scan.
func (*Scanner) Comments ¶ added in v0.2.0
Comments returns the SQL_COMMENT and C_COMMENT tokens scanned so far, in source order, for a Scanner from NewKeepingComments (nil otherwise). They do not overlap, and every one lies within Input.
type Token ¶
type Token struct {
Kind ast.Token
Start, End int32
Str string // IDENT/UIDENT/SCONST/USCONST/BCONST/XCONST/FCONST/Op: yylval.str; keywords: canonical spelling
Ival int32 // ICONST/PARAM: yylval.ival
KeywordKind ast.KeywordKind
}
Token is one core_yylex result. Start/End are byte offsets into the input ([Start,End) spans the token text after any flex yyless put-back, matching yylloc and the patched scanner's yyllocend). Str/Ival carry the semantic value (core_YYSTYPE) for the token kinds that have one.
Source Files