lexer

package

v0.35.0 Latest Latest Go to latest Published: Oct 28, 2026 License: MIT Imports: 8 Imported by: 0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/maarten-boot/kukel

Links

Open Source Insights

Documentation ¶

Overview ¶

Package lexer turns Kukel source text into tokens (syntax.ebnf, section 4).

Index ¶

func Tokenize(file string, src []byte) ([]token.Token, error)
type Error
type Lexer
- func New(file string, src []byte) *Lexer
- func (l *Lexer) Next() (token.Token, error)

Constants ¶

This section is empty.

Variables ¶

This section is empty.

Functions ¶

func Tokenize ¶

func Tokenize(file string, src []byte) ([]token.Token, error)

Tokenize scans the whole of src. The result ends with an EOF token. On a lexical error it returns the tokens scanned so far and the error.

Types ¶

type Error ¶

type Error = diag.Error

Error is a lexical error at a source position.

type Lexer ¶

type Lexer struct {

	// KeepComments makes Next return comments as COMMENT tokens (with the
	// "//", "#" or "/* */" in their text) instead of skipping them.
	KeepComments bool
	// contains filtered or unexported fields
}

Lexer scans one source file.

func New ¶

func New(file string, src []byte) *Lexer

New returns a lexer for src. A raw NUL byte ends its source line: the bytes from the NUL up to the next newline are dropped before scanning.

func (*Lexer) Next ¶

func (l *Lexer) Next() (token.Token, error)

Next returns the next token, or an error.

Source Files ¶

View all Source files

lexer.go

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL