lexer

package

v0.2.1 Latest Latest Go to latest Published: Oct 13, 2026 License: MIT Imports: 8 Imported by: 0

Details

Package lexer turns Kukel source text into tokens (syntax.ebnf, section 4).

This section is empty.

This section is empty.

func Tokenize(file string, src []byte) ([]token.Token, error)

Tokenize scans the whole of src. The result ends with an EOF token. On a lexical error it returns the tokens scanned so far and the error.

type Error = diag.Error

Error is a lexical error at a source position.

type Lexer struct {
	// contains filtered or unexported fields
}

Lexer scans one source file.

func New(file string, src []byte) *Lexer

New returns a lexer for src. A raw NUL byte ends its source line: the bytes from the NUL up to the next newline are dropped before scanning.

func (l *Lexer) Next() (token.Token, error)

Next returns the next token, or an error.