Documentation
¶
Overview ¶
Package token defines the lexical tokens for the Phase 0 subset.
The token set is intentionally richer than strictly necessary (it carries SpaceBefore, the seed of MRI's spaceSeen) so the parser can disambiguate command calls (`foo -1` vs `foo - 1`) without the lexer being rewritten as the grammar grows (plan-rbgo.md §10).
Index ¶
Constants ¶
This section is empty.
Variables ¶
View Source
var Keywords = map[string]Type{ "def": DEF, "class": CLASS, "module": MODULE, "end": END, "if": IF, "elsif": ELSIF, "else": ELSE, "unless": UNLESS, "while": WHILE, "until": UNTIL, "return": RETURN, "break": BREAK, "next": NEXT, "begin": BEGIN, "rescue": RESCUE, "ensure": ENSURE, "case": CASE, "when": WHEN, "in": IN, "for": FOR, "retry": RETRY, "then": THEN, "do": DO, "true": TRUE, "false": FALSE, "nil": NIL, "self": SELF, "super": SUPER, "yield": YIELD, "and": AND, "or": OR, "not": NOT, "alias": ALIAS, "undef": UNDEF, }
Keywords maps reserved words to their token type.
Functions ¶
This section is empty.
Types ¶
type Token ¶
type Token struct {
Type Type
Lit string
Flags string // regexp flag letters in source order (i, m, x, o and the encoding flags n, u, e, s), only set for REGEXP tokens
Words []string // the words of a WORDS/SYMBOLS literal, already split and unescaped; only set for those two types
Line int
Col int
SpaceBefore bool // whitespace immediately preceded this token (MRI spaceSeen)
}
Token is a single lexed token.
type Type ¶
type Type int
const ( EOF Type = iota ILLEGAL NEWLINE // \n or ; INT FLOAT STRING STRBEG // "…#{ (start of an interpolated string; Lit = literal prefix) STRMID // }…#{ (literal between two interpolations) STREND // }…" (literal suffix ending an interpolated string) IDENT // local variable or method name (lowercase / _ leading) CONST // Capitalized identifier IVAR // @instance_variable CVAR // @@class_variable GVAR // $global / $~ / $1 SYMBOL // :name LABEL // name: in a hash literal REGEXP // /pattern/flags (Lit = pattern source, Flags = matched flag letters) XSTRING // `cmd` / %x{cmd} backtick command literal (Lit = raw command source) WORDS // %w[…] word-array literal (Lit = raw body, Words = the split words) SYMBOLS // %i[…] symbol-array literal (Lit = raw body, Words = the split words) // Keywords. DEF CLASS MODULE END IF ELSIF ELSE UNLESS WHILE UNTIL RETURN BREAK NEXT BEGIN RESCUE ENSURE CASE WHEN IN FOR RETRY THEN DO TRUE FALSE NIL SELF SUPER YIELD AND // `and` low-precedence logical-and keyword OR // `or` low-precedence logical-or keyword NOT // `not` low-precedence logical-not keyword ALIAS // `alias` method-aliasing keyword UNDEF // `undef` method-removal keyword // Operators and delimiters. PLUS MINUS STAR POW // ** SLASH PERCENT ASSIGN EQ EQQ // === MATCH // =~ NEQ LT GT LE GE SPACESHIP // <=> OPASSIGN // compound assignment (+=, -=, ||=, …); Lit holds the operator SHOVEL // << ANDAND // && OROR // || BANG QUESTION // ? COLON // : (ternary separator) SCOPE // :: (constant scope resolution) LPAREN RPAREN LBRACE RBRACE LBRACKET RBRACKET PIPE HASHROCKET // => COMMA DOT DOTDOT // .. DOTDOTDOT // ... AMPER // & (block-pass / block param) SAFEDOT // &. (safe navigation) ARROW // -> (stabby lambda) CARET // ^ (pattern-matching pin operator) RSHIFT // >> TILDE // ~ (bitwise complement) NMATCH // !~ (does-not-match operator) )
func LookupIdent ¶
LookupIdent returns the keyword type for s, or IDENT/CONST otherwise.
Source Files
Click to show internal directories.
Click to hide internal directories.