token

package
v1.6.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jan 28, 2026 License: Apache-2.0 Imports: 2 Imported by: 1

Documentation

Index

Constants

View Source
const (
	// Special tokens
	ILLEGAL Token = iota
	EOF
	COMMENT

	// Identifiers and basic type literals
	// (these tokens stand for classes of literals)
	IDENT  // main
	INT    // 12345
	FLOAT  // 123.45
	IMAG   // 123.45i
	CHAR   // 'a'
	STRING // "abc"
	RAT    // 3r, 3.4r
	UNIT   // 1m, 2.3s, 3ms, 4us, 5ns, 6.5m, 7h, 8d, 9w, 10y

	ADD = '+'
	SUB = '-'
	MUL = '*'
	QUO = '/'
	REM = '%'

	AND = '&'
	OR  = '|'
	XOR = '^'

	LT     = '<'
	GT     = '>'
	ASSIGN = '='
	NOT    = '!'

	LPAREN = '('
	LBRACK = '['
	LBRACE = '{'
	COMMA  = ','
	PERIOD = '.'

	RPAREN    = ')'
	RBRACK    = ']'
	RBRACE    = '}'
	SEMICOLON = ';'
	COLON     = ':'
	QUESTION  = '?'
	TILDE     = '~'
	AT        = '@'
	ENV       = '$'
)
View Source
const (
	Break = -1
)
View Source
const (
	// NoPos - The zero value for Pos is NoPos; there is no file and line
	// information associated with it, and NoPos.IsValid() is false. NoPos
	// is always smaller than any other Pos value. The corresponding
	// Position value for NoPos is the zero value for Position.
	NoPos = token.NoPos
)

Variables

This section is empty.

Functions

func ForEach

func ForEach(from Token, f func(tok Token, lit string) int)

ForEach iterates tokens.

Types

type File

type File = token.File

A File is a handle for a file belonging to a FileSet. A File has a name, size, and line offset table.

type FileSet

type FileSet = token.FileSet

A FileSet represents a set of source files. Methods of file sets are synchronized; multiple goroutines may invoke them concurrently.

func NewFileSet

func NewFileSet() *FileSet

NewFileSet creates a new file set.

type Pos

type Pos = token.Pos

Pos is a compact encoding of a source position within a file set. It can be converted into a Position for a more convenient, but much larger, representation.

The Pos value for a given file is a number in the range [base, base+size], where base and size are specified when adding the file to the file set via AddFile.

To create the Pos value for a specific source offset (measured in bytes), first add the respective file to the current file set using FileSet.AddFile and then call File.Pos(offset) for that file. Given a Pos value p for a specific file set fset, the corresponding Position value is obtained by calling fset.Position(p).

Pos values can be compared directly with the usual comparison operators: If two Pos values p and q are in the same file, comparing p and q is equivalent to comparing the respective source file offsets. If p and q are in different files, p < q is true if the file implied by p was added to the respective file set before the file implied by q.

type Position

type Position = token.Position

Position describes an arbitrary source position including the file, line, and column location. A Position is valid if the line number is > 0.

type Token

type Token uint
const (
	SHL     Token // <<
	SHR           // >>
	AND_NOT       // &^

	ADD_ASSIGN // +=
	SUB_ASSIGN // -=
	MUL_ASSIGN // *=
	QUO_ASSIGN // /=
	REM_ASSIGN // %=

	AND_ASSIGN     // &=
	OR_ASSIGN      // |=
	XOR_ASSIGN     // ^=
	SHL_ASSIGN     // <<=
	SHR_ASSIGN     // >>=
	AND_NOT_ASSIGN // &^=

	LAND  // &&
	LOR   // ||
	ARROW // <-
	INC   // ++
	DEC   // --

	EQ       // ==
	NE       // !=
	LE       // <=
	GE       // >=
	DEFINE   // :=
	ELLIPSIS // ...

	DRARROW   // =>
	SRARROW   // ->
	BIDIARROW // <>
	POW       // **

)

func (Token) Len

func (tok Token) Len() int

Len returns 1) len of is token literal, if token is an operator. 2) 0 for else.

func (Token) String

func (tok Token) String() (s string)

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL