token

package
v1.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 5, 2026 License: MIT Imports: 1 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Token

type Token struct {
	Type    Type
	Literal string
	Line    int
	Column  int
	// Raw is the token's source text with escape sequences still intact.
	// Set only for InterpString, whose `{expr}` spans the parser has to
	// re-scan: Literal has already been unescaped, so a `\"` inside an
	// interpolated expression (or a `\u{7B}` anywhere) would be
	// indistinguishable from real interpolation structure.
	Raw string
}

Token is a lexed lexeme tagged with its type and source location.

func CreateOperator

func CreateOperator(literal string, line, column int) Token

CreateOperator builds an operator token. Panics if literal is not a known operator — that is a lexer bug, not user input, and silently producing an Illegal token would hide the cause until it surfaced in the parser.

func CreateSeparator

func CreateSeparator(literal string, line, column int) Token

CreateSeparator builds a separator token, with the same panic-on-unknown contract as CreateOperator.

type Type

type Type string

Type is the discriminator for a Token.

const (
	Illegal Type = "ILLEGAL"
	EOF     Type = "EOF"

	// Literals
	Ident  Type = "IDENT"
	Int    Type = "INT"
	Float  Type = "FLOAT"
	String Type = "STRING"
	// InterpString is a backtick-quoted string subject to `{expr}`
	// interpolation. Kept distinct from String so plain `"..."`/`'...'`
	// strings containing `{` are not mis-parsed as interpolated.
	InterpString Type = "INTERP_STRING"

	// Arithmetic
	Plus     Type = "+"
	Minus    Type = "-"
	Asterisk Type = "*"
	Slash    Type = "/"
	Percent  Type = "%"
	Caret    Type = "^"
	FloorDiv Type = "//"

	// Unary-only operators.
	// `Hash` is length-of (`#t`). `Not` is logical not. `Tilde` is also a
	// unary operator (bitwise not) but lives in the bitwise section because
	// it doubles as binary XOR.
	Hash Type = "#"

	// Compound assignment.
	// `BorAssign` and `BandAssign` are *bitwise* OR/AND assign (`|=`, `&=`),
	// not logical — Lua's logical `or`/`and` have no compound form.
	PlusAssign   Type = "+="
	MinusAssign  Type = "-="
	MulAssign    Type = "*="
	DivAssign    Type = "/="
	BorAssign    Type = "|="
	BandAssign   Type = "&="
	LShiftAssign Type = "<<="
	RShiftAssign Type = ">>="

	// Bitwise
	Ampersand Type = "&"
	Pipe      Type = "|"
	Tilde     Type = "~"
	LShift    Type = "<<"
	RShift    Type = ">>"

	// Comparison
	Eq    Type = "=="
	NotEq Type = "~="
	LT    Type = "<"
	LTE   Type = "<="
	GT    Type = ">"
	GTE   Type = ">="

	// Assignment
	Assign Type = "="

	// Type-syntax operators (Luau-style annotations).
	// `Question` is the postfix-optional sugar (`T?` ≡ `T | nil`).
	// `Arrow` separates a function type's params from its return (`(A) -> B`)
	// and also acts as the arm separator in `match` expressions; the parser
	// disambiguates by context.
	// `::` is represented by Label and is reused for type assertions
	// (`expr :: T`); again the parser disambiguates by context.
	Question Type = "?"
	Arrow    Type = "->"

	// Logical (keywords in Lua, but typed for AST use)
	And Type = "AND"
	Or  Type = "OR"
	Not Type = "NOT"

	// Delimiters
	Comma     Type = ","
	Semicolon Type = ";"
	Colon     Type = ":"
	Dot       Type = "."
	Concat    Type = ".."
	Vararg    Type = "..."
	Label     Type = "::"

	LParen   Type = "("
	RParen   Type = ")"
	LBrace   Type = "{"
	RBrace   Type = "}"
	LBracket Type = "["
	RBracket Type = "]"

	// Keywords
	True     Type = "TRUE"
	False    Type = "FALSE"
	Nil      Type = "NIL"
	If       Type = "IF"
	ElseIf   Type = "ELSEIF"
	Else     Type = "ELSE"
	Then     Type = "THEN"
	End      Type = "END"
	Do       Type = "DO"
	While    Type = "WHILE"
	Repeat   Type = "REPEAT"
	Until    Type = "UNTIL"
	For      Type = "FOR"
	In       Type = "IN"
	Function Type = "FUNCTION"
	Local    Type = "LOCAL"
	Return   Type = "RETURN"
	Break    Type = "BREAK"
	Goto     Type = "GOTO"
	Match    Type = "MATCH"
	Enum     Type = "ENUM"
	Defer    Type = "DEFER"
	Try      Type = "TRY"
	Catch    Type = "CATCH"
	Throw    Type = "THROW"
)

func LookupIdent

func LookupIdent(ident string) Type

LookupIdent returns the keyword type for ident, or Ident if it is not a reserved word.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL