Documentation
¶
Overview ¶
Package tokenizer implements R7RS Scheme lexical analysis.
The tokenizer converts a Unicode rune stream into tokens with source positions:
Token Categories ¶
- Delimiters: (, ), [, ], (), .
- Quotation: ', `, ,, ,@ and syntax variants
- Numbers: integers, decimals, rationals, scientific, complex, polar
- Special: +inf.0, -inf.0, +nan.0, imaginary variants
- Big numbers: #z prefix for BigInteger, #m for BigFloat
- Literals: symbols, strings, characters
- Booleans: #t, #f, #true, #false
- Comments: line (;), block (#|...|#), datum (#;)
- Vectors: #(, #u8(
- Labels: #n=, #n#
Usage ¶
tok := tokenizer.NewTokenizer(reader, caseInsensitive)
for {
token, err := tok.Next()
if errors.Is(err, io.EOF) {
break
}
// process token
}
Each Token provides source position via Start() and End(), raw text via String(), and processed value (escapes resolved) via Value().
Index ¶
- Constants
- Variables
- type ErrorCode
- type SimpleToken
- func (p *SimpleToken) End() syntax.SourceIndexes
- func (p *SimpleToken) EqualTo(v values.Value) bool
- func (p *SimpleToken) HasHashDigit() bool
- func (p *SimpleToken) IsVoid() bool
- func (p *SimpleToken) Radix() int
- func (p *SimpleToken) SchemeString() string
- func (p *SimpleToken) Start() syntax.SourceIndexes
- func (p *SimpleToken) String() string
- func (p *SimpleToken) Type() TokenizerState
- func (p *SimpleToken) Value() string
- type Token
- type Tokenizer
- type TokenizerError
- type TokenizerState
Constants ¶
const ( MessageRuneError = "rune error" MessageExpectingNumber = "expecting number" MessageExpectingExponentDigits = "expecting exponent digits" MessageExpectingImaginary = "expecting imaginary" MessageExpectingDecimalFraction = "expecting decimal fraction" MessageExpectingNan = "expecting NaN" MessageExpectingInf = "expecting Inf" MessageExpectingTrue = "expecting true" MessageExpectingFalse = "expecting false" MessageExpectingToken = "expecting token" MessageExpectingEscape = "expecting escape" MessageExpectingHexSequenceTerminator = "expecting hex sequence terminator" MessageExpectingLineEnding = "expecting line ending" MessageExpectingHexDigit = "expecting hex digit" MessageExpectingCharacterMnemonicOrHexEscape = "expecting character mnemonic or hex escape" MessageExpectingDirective = "expecting directive" MessageCannotParseNumber = "cannot parse number" MessageCodePointExceedsUnicodeMaximum = "character code point exceeds Unicode maximum (0x10FFFF)" MessageCodePointIsSurrogate = "character code point is a surrogate (0xD800-0xDFFF)" MessageInvalidHexEscape = "invalid hex escape" MessageInvalidCharacterHexEscape = "invalid character hex escape" MessageInvalidCharacterMnemonic = "invalid character mnemonic" MessageUnterminatedExtendedSymbol = "unterminated extended symbol" MessageUnterminatedString = "unterminated string" MessageUnterminatedBlockComment = "unterminated block comment" MessageExpectingByteVectorPrefix = "expecting #u8( byte vector prefix" )
Error messages returned by the tokenizer.
Variables ¶
var CharMnemonics = map[string]rune{
"alarm": '\a',
"backspace": '\b',
"delete": '\x7F',
"escape": '\x1B',
"newline": '\n',
"null": '\x00',
"return": '\r',
"space": ' ',
"tab": '\t',
}
CharMnemonics maps R7RS §6.6 character mnemonic names to their rune values. Keys must be lowercase — lookup normalizes via strings.ToLower.
Functions ¶
This section is empty.
Types ¶
type ErrorCode ¶
type ErrorCode int
ErrorCode is currently unused: no value is ever constructed and no error carries one. TokenizerError classifies by message string instead (Is compares mess).
type SimpleToken ¶
type SimpleToken struct {
// contains filtered or unexported fields
}
SimpleToken is the concrete implementation of Token used by the tokenizer.
func NewSimpleToken ¶
func NewSimpleToken(typ TokenizerState, src, val string, sti, eni *syntax.SourceIndexes, hash bool, radix int) *SimpleToken
NewSimpleToken creates a new SimpleToken with the given type, source, value, position, hash-digit flag, and radix. radix is the effective parse base for integer tokens (2/8/10/16) and 0 for tokens where base is not meaningful.
func (*SimpleToken) End ¶
func (p *SimpleToken) End() syntax.SourceIndexes
End returns the source position where the token ends.
func (*SimpleToken) EqualTo ¶
func (p *SimpleToken) EqualTo(v values.Value) bool
EqualTo returns true if this token equals the given value.
func (*SimpleToken) HasHashDigit ¶
func (p *SimpleToken) HasHashDigit() bool
HasHashDigit returns true if the token contained # as an inexact digit placeholder. R7RS §7.1.1: # can appear in place of digits after at least one real digit, representing an unknown digit (treated as 0). Its presence forces the number to be inexact.
func (*SimpleToken) IsVoid ¶
func (p *SimpleToken) IsVoid() bool
IsVoid returns true if the token is nil.
func (*SimpleToken) Radix ¶
func (p *SimpleToken) Radix() int
Radix returns the effective parse base for an integer token (2, 8, 10, or 16). It is 0 for tokens where base is not meaningful (non-numeric tokens and numeric shapes other than integers). The tokenizer records the base here so the parser reads it directly rather than re-deriving a base literal from the token state.
func (*SimpleToken) SchemeString ¶
func (p *SimpleToken) SchemeString() string
SchemeString returns the Scheme representation of the token.
func (*SimpleToken) Start ¶
func (p *SimpleToken) Start() syntax.SourceIndexes
Start returns the source position where the token begins.
func (*SimpleToken) String ¶
func (p *SimpleToken) String() string
func (*SimpleToken) Value ¶
func (p *SimpleToken) Value() string
Value returns the processed value of the token (e.g., with escape sequences converted).
type Token ¶
type Token interface {
Type() TokenizerState
Start() syntax.SourceIndexes
End() syntax.SourceIndexes
String() string
Value() string // Returns processed value (e.g., with escape sequences converted)
HasHashDigit() bool // R7RS §7.1.1: whether # appeared as inexact digit placeholder
Radix() int // Effective parse base for integer tokens (2/8/10/16); 0 for non-integer tokens
}
Token is the interface for tokenizer output tokens.
type Tokenizer ¶
type Tokenizer struct {
// contains filtered or unexported fields
}
Tokenizer reads Scheme source code and produces a stream of tokens.
func NewTokenizer ¶
func NewTokenizer(rdr io.RuneReader, ci bool) *Tokenizer
NewTokenizer creates a new tokenizer that reads from the given RuneReader. The tokenizer is initialized with the first rune already read.
func (*Tokenizer) Next ¶
Next returns the next token from the input stream. Returns io.EOF when the input is exhausted. End of input is carried by p.err (readNextRune sets io.EOF), never by p.cur: utf8.RuneError is both the decode sentinel and a writable character, so testing the rune reported a literal U+FFFD as EOF and silently discarded the rest of the input. Comment tokens are always emitted; callers that want them elided (the parser, when constructed with skipComments) drop them themselves.
func (*Tokenizer) Reader ¶
func (p *Tokenizer) Reader() io.RuneReader
Reader returns the underlying RuneReader.
type TokenizerError ¶
type TokenizerError struct {
// contains filtered or unexported fields
}
TokenizerError represents an error that occurred during tokenization.
func NewTokenizerError ¶
func NewTokenizerError(mess string) *TokenizerError
NewTokenizerError creates a new tokenizer error with the given message.
func NewTokenizerErrorWithWrap ¶
func NewTokenizerErrorWithWrap(err error, mess string) *TokenizerError
NewTokenizerErrorWithWrap creates a new tokenizer error that wraps another error.
func (*TokenizerError) Error ¶
func (p *TokenizerError) Error() string
func (*TokenizerError) Is ¶
func (p *TokenizerError) Is(err error) bool
Is implements errors.Is for TokenizerError. Two tokenizer errors match when they carry the same message, so errors.Is can distinguish the message constants (e.g. errors.Is(err, NewTokenizerError(MessageUnterminatedString))); a wrapped cause is still reached through Unwrap.
func (*TokenizerError) Unwrap ¶
func (p *TokenizerError) Unwrap() error
type TokenizerState ¶
type TokenizerState int
TokenizerState represents the type of token recognized by the tokenizer. Each state corresponds to a distinct lexical element in Scheme syntax.
const ( // TokenizerStateFailed indicates tokenization failed. TokenizerStateFailed TokenizerState = iota // TokenizerStateSyntax represents #'expr (syntax quote). TokenizerStateSyntax // TokenizerStateUnsyntax represents #,expr (unsyntax). TokenizerStateUnsyntax // TokenizerStateUnsyntaxSplicing represents #,@expr (unsyntax-splicing). TokenizerStateUnsyntaxSplicing // TokenizerStateQuasisyntax represents #`expr (quasisyntax). TokenizerStateQuasisyntax // TokenizerStateQuote represents 'expr (quote). TokenizerStateQuote // TokenizerStateUnquote represents ,expr (unquote). TokenizerStateUnquote // TokenizerStateUnquoteSplicing represents ,@expr (unquote-splicing). TokenizerStateUnquoteSplicing // TokenizerStateQuasiquote represents `expr (quasiquote). TokenizerStateQuasiquote // TokenizerStateSignedInf represents +inf.0 or -inf.0 (infinity). TokenizerStateSignedInf // TokenizerStateSignedNan represents +nan.0 or -nan.0 (not a number). TokenizerStateSignedNan // TokenizerStateSignedImaginaryInf represents +inf.0i or -inf.0i (imaginary infinity). TokenizerStateSignedImaginaryInf // TokenizerStateSignedImaginaryNan represents +nan.0i or -nan.0i (imaginary NaN). TokenizerStateSignedImaginaryNan // TokenizerStateSignedImaginary represents +i, -i, +3i, -3.5i (pure imaginary). TokenizerStateSignedImaginary // TokenizerStateSignedComplex represents +1+2i, 3.5-2.5i (rectangular complex). TokenizerStateSignedComplex // TokenizerStateSignedComplexPolar represents +1@1.5708 (polar complex: magnitude@angle). TokenizerStateSignedComplexPolar // TokenizerStateUnsignedImaginaryInf represents inf.0i (unsigned imaginary infinity). TokenizerStateUnsignedImaginaryInf // TokenizerStateUnsignedImaginaryNan represents nan.0i (unsigned imaginary NaN). TokenizerStateUnsignedImaginaryNan // TokenizerStateUnsignedImaginary represents 3i, 3.5i (unsigned pure imaginary). TokenizerStateUnsignedImaginary // TokenizerStateUnsignedComplex represents 1+2i (unsigned rectangular complex). TokenizerStateUnsignedComplex // TokenizerStateUnsignedComplexPolar represents 1@1.5708 (unsigned polar complex). TokenizerStateUnsignedComplexPolar // TokenizerStateMarker represents a generic # marker. TokenizerStateMarker // TokenizerStateMarkerBooleanFalse represents #f or #false. TokenizerStateMarkerBooleanFalse // TokenizerStateMarkerBooleanTrue represents #t or #true. TokenizerStateMarkerBooleanTrue // TokenizerStateMarkerNumberInexact represents #i prefix (inexact). TokenizerStateMarkerNumberInexact // TokenizerStateMarkerNumberExact represents #e prefix (exact). TokenizerStateMarkerNumberExact // TokenizerStateSignedInteger represents a signed integer in any base // (-123, +456, #x-FF, #b+101). The base is carried on the token via Radix(). TokenizerStateSignedInteger // TokenizerStateUnsignedInteger represents an unsigned integer in any base // (123, #xFF, #b101). The base is carried on the token via Radix(). TokenizerStateUnsignedInteger // TokenizerStateBigFloat represents #m arbitrary-precision decimal. TokenizerStateBigFloat // TokenizerStateBigInteger represents #z arbitrary-precision integer (decimal). TokenizerStateBigInteger // TokenizerStateMarkerBase2 represents #b prefix (binary). TokenizerStateMarkerBase2 // TokenizerStateMarkerBase8 represents #o prefix (octal). TokenizerStateMarkerBase8 // TokenizerStateMarkerBase10 represents #d prefix (decimal). TokenizerStateMarkerBase10 // TokenizerStateMarkerBase16 represents #x prefix (hexadecimal). TokenizerStateMarkerBase16 // TokenizerStateSignedDecimalFraction represents -1.23 or +4.56. TokenizerStateSignedDecimalFraction // TokenizerStateSignedRationalFraction represents -1/2 or +3/4. TokenizerStateSignedRationalFraction // TokenizerStateUnsignedRationalFraction represents 1/2 or 3/4. TokenizerStateUnsignedRationalFraction // TokenizerStateUnsignedDecimalFraction represents 1.23 or 4.56. TokenizerStateUnsignedDecimalFraction // TokenizerStateSignedScientificNotation represents integers with exponents like +1e10, -2e-5. // Parser determines if result is integer or float based on exponent sign and mantissa. TokenizerStateSignedScientificNotation // TokenizerStateUnsignedScientificNotation represents integers with exponents like 1e10, 2e-5. // Parser determines if result is integer or float based on exponent sign and mantissa. TokenizerStateUnsignedScientificNotation // TokenizerStateEmptyList represents () (empty list). TokenizerStateEmptyList // TokenizerStateOpenParen represents ( (open parenthesis). TokenizerStateOpenParen // TokenizerStateCloseParen represents ) (close parenthesis). TokenizerStateCloseParen // TokenizerStateOpenBracket represents [ (open square bracket). // R7RS §2.1: Square brackets are equivalent to parentheses but must match. TokenizerStateOpenBracket // TokenizerStateCloseBracket represents ] (close square bracket). // R7RS §2.1: Square brackets are equivalent to parentheses but must match. TokenizerStateCloseBracket // TokenizerStateCons represents . (dot for improper lists). TokenizerStateCons // TokenizerStateString represents complete "string". TokenizerStateString // TokenizerStateCharMnemonicOrHexEscape represents intermediate character state. TokenizerStateCharMnemonicOrHexEscape // TokenizerStateCharMnemonic represents #\newline, #\space, etc. TokenizerStateCharMnemonic // TokenizerStateCharHexEscape represents #\x0A (hex escape). TokenizerStateCharHexEscape // TokenizerStateCharGraphic represents #\a (single graphic char). TokenizerStateCharGraphic // TokenizerStateLineCommentBody represents comment text. TokenizerStateLineCommentBody // TokenizerStateBlockCommentBody represents block content. TokenizerStateBlockCommentBody // TokenizerStateDatumCommentBegin represents the #; marker; the following // datum is read by the parser. TokenizerStateDatumCommentBegin // TokenizerStateSymbol represents an identifier or symbol. TokenizerStateSymbol // TokenizerStateOpenVector represents #( (vector). TokenizerStateOpenVector // TokenizerStateOpenVectorUnsignedByteMarker represents #u8( (bytevector). TokenizerStateOpenVectorUnsignedByteMarker // TokenizerStateDirective represents #!fold-case, etc. TokenizerStateDirective // TokenizerStateLabelReference represents #123# (datum label reference). TokenizerStateLabelReference // TokenizerStateLabelAssignment represents #123= (datum label assignment). TokenizerStateLabelAssignment )
TokenizerState values for different token types.
Source Files