lexer

package
v0.0.13 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 18, 2026 License: MIT Imports: 2 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source
var KeywordTokens = map[string]token.TokenType{
	"let":       token.LET,
	"if":        token.IF,
	"else":      token.ELSE,
	"while":     token.WHILE,
	"for":       token.FOR,
	"break":     token.BREAK,
	"continue":  token.CONTINUE,
	"function":  token.FUNCTION,
	"return":    token.RETURN,
	"const":     token.CONST,
	"import":    token.IMPORT,
	"export":    token.EXPORT,
	"try":       token.TRY,
	"catch":     token.CATCH,
	"throw":     token.THROW,
	"switch":    token.SWITCH,
	"case":      token.CASE,
	"default":   token.DEFAULT,
	"in":        token.IN,
	"do":        token.DO,
	"print":     token.PRINT,
	"true":      token.TRUE,
	"false":     token.FALSE,
	"null":      token.NULL,
	"typeof":    token.TYPEOF,
	"match":     token.MATCH,
	"async":     token.ASYNC,
	"await":     token.AWAIT,
	"init":      token.INIT,
	"new":       token.NEW,
	"type":      token.IDENT,
	"lazy":      token.IDENT,
	"yield":     token.YIELD,
	"macro":     token.MACRO,
	"for_await": token.FOR_AWAIT,
	"select":    token.SELECT,
	"spawn":     token.SPAWN,
	"abstract":  token.ABSTRACT,
	"extends":   token.EXTENDS,
	"super":     token.SUPER,
}

KeywordTokens maps keyword strings to their token types.

Functions

func LookupIdent

func LookupIdent(ident string) token.TokenType

LookupIdent checks if the given identifier is a keyword and returns the corresponding token type. Returns token.IDENT for non-keywords.

Types

type Lexer

type Lexer struct {
	// contains filtered or unexported fields
}

Lexer performs lexical analysis on the input string, producing a stream of tokens.

func NewLexer

func NewLexer(input string) *Lexer

NewLexer creates a new Lexer for the given input string.

func (*Lexer) Input

func (l *Lexer) Input() string

Input returns the original input string.

func (*Lexer) NextToken

func (l *Lexer) NextToken() token.Token

NextToken reads and returns the next token from the input.

func (*Lexer) RestoreState

func (l *Lexer) RestoreState(s LexerState)

RestoreState rewinds the lexer to a previously saved state.

func (*Lexer) SaveState

func (l *Lexer) SaveState() LexerState

SaveState returns a snapshot of the lexer's current position.

type LexerState

type LexerState struct {
	Position     int
	ReadPosition int
	Ch           byte
	Line         int
	Column       int
}

LexerState captures the internal state of a Lexer so it can be saved and restored (e.g. for parser back-tracking).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL