Documentation
¶
Overview ¶
Package tokenizer implements R7RS Scheme lexical analysis.
The tokenizer converts a Unicode rune stream into tokens with source positions:
Token Categories ¶
- Delimiters: (, ), [, ], (), .
- Quotation: ', `, ,, ,@ and syntax variants
- Numbers: integers, decimals, rationals, scientific, complex, polar
- Special: +inf.0, -inf.0, +nan.0, imaginary variants
- Big numbers: #z prefix for BigInteger, #m for BigFloat
- Literals: symbols, strings, characters
- Booleans: #t, #f, #true, #false
- Comments: line (;), block (#|...|#), datum (#;)
- Vectors: #(, #u8(
- Labels: #n=, #n#
Usage ¶
tok := tokenizer.NewTokenizer(reader, caseInsensitive)
for {
token, err := tok.Next()
if errors.Is(err, io.EOF) {
break
}
// process token
}
Each Token provides source position via Start() and End(), raw text via String(), and processed value (escapes resolved) via Value().
Index ¶
- Constants
- Variables
- type SimpleToken
- func (p *SimpleToken) End() syntax.SourceIndexes
- func (p *SimpleToken) EqualTo(v values.Value) bool
- func (p *SimpleToken) HasHashDigit() bool
- func (p *SimpleToken) IsVoid() bool
- func (p *SimpleToken) Radix() int
- func (p *SimpleToken) SchemeString() string
- func (p *SimpleToken) Start() syntax.SourceIndexes
- func (p *SimpleToken) String() string
- func (p *SimpleToken) Type() TokenizerState
- func (p *SimpleToken) Value() string
- type Token
- type Tokenizer
- type TokenizerError
- type TokenizerState
Constants ¶
const ( MessageRuneError = "rune error" MessageExpectingNumber = "expecting number" MessageExpectingDelimiterAfterNumber = "expecting delimiter after radix-prefixed number" MessageExpectingExponentDigits = "expecting exponent digits" MessageExpectingImaginary = "expecting imaginary" MessageExpectingDecimalFraction = "expecting decimal fraction" MessageExpectingNan = "expecting NaN" MessageExpectingInf = "expecting Inf" MessageExpectingToken = "expecting token" MessageExpectingEscape = "expecting escape" MessageExpectingHexSequenceTerminator = "expecting hex sequence terminator" MessageExpectingLineEnding = "expecting line ending" MessageExpectingHexDigit = "expecting hex digit" MessageExpectingCharacterMnemonicOrHexEscape = "expecting character mnemonic or hex escape" MessageExpectingDirective = "expecting directive" MessageCannotParseNumber = "cannot parse number" MessageCodePointExceedsUnicodeMaximum = "character code point exceeds Unicode maximum (0x10FFFF)" MessageCodePointIsSurrogate = "character code point is a surrogate (0xD800-0xDFFF)" MessageInvalidCharacterHexEscape = "invalid character hex escape" MessageInvalidCharacterMnemonic = "invalid character mnemonic" MessageUnterminatedExtendedSymbol = "unterminated extended symbol" MessageUnterminatedString = "unterminated string" MessageUnterminatedBlockComment = "unterminated block comment" MessageExpectingByteVectorPrefix = "expecting #u8( byte vector prefix" )
Error messages returned by the tokenizer.
Variables ¶
var CharMnemonics = map[string]rune{
"alarm": '\a',
"backspace": '\b',
"delete": '\x7F',
"escape": '\x1B',
"newline": '\n',
"null": '\x00',
"return": '\r',
"space": ' ',
"tab": '\t',
}
CharMnemonics maps R7RS §6.6 character mnemonic names to their rune values. Keys must be lowercase — lookup normalizes via strings.ToLower.
Functions ¶
This section is empty.
Types ¶
type SimpleToken ¶
type SimpleToken struct {
// contains filtered or unexported fields
}
SimpleToken is the concrete implementation of Token used by the tokenizer.
func NewSimpleToken ¶
func NewSimpleToken(typ TokenizerState, src, val string, sti, eni *syntax.SourceIndexes, hash bool, radix int) *SimpleToken
NewSimpleToken creates a new SimpleToken with the given type, source, value, position, hash-digit flag, and radix. radix is the effective parse base for integer tokens (2/8/10/16) and 0 for tokens where base is not meaningful.
func (*SimpleToken) End ¶
func (p *SimpleToken) End() syntax.SourceIndexes
End returns the source position where the token ends.
func (*SimpleToken) EqualTo ¶
func (p *SimpleToken) EqualTo(v values.Value) bool
EqualTo returns true if this token equals the given value.
func (*SimpleToken) HasHashDigit ¶
func (p *SimpleToken) HasHashDigit() bool
HasHashDigit returns true if the token contained # as an inexact digit placeholder. R7RS §7.1.1: # can appear in place of digits after at least one real digit, representing an unknown digit (treated as 0). Its presence forces the number to be inexact.
func (*SimpleToken) IsVoid ¶
func (p *SimpleToken) IsVoid() bool
IsVoid returns true if the token is nil.
func (*SimpleToken) Radix ¶
func (p *SimpleToken) Radix() int
Radix returns the effective parse base for an integer or decimal-fraction token (2, 8, 10, or 16). It is 0 for tokens where base is not meaningful (non-numeric tokens, and numeric shapes other than those two). The tokenizer records the base here so the parser reads it directly rather than re-deriving a base literal from the token state.
Fractions carry it for the same reason integers do, plus one of their own: the reader's #e converts a decimal literal from its digits, and must not do that to a hex one (makeExactLiteral, pkg/parser).
func (*SimpleToken) SchemeString ¶
func (p *SimpleToken) SchemeString() string
SchemeString returns the Scheme representation of the token.
func (*SimpleToken) Start ¶
func (p *SimpleToken) Start() syntax.SourceIndexes
Start returns the source position where the token begins.
func (*SimpleToken) String ¶
func (p *SimpleToken) String() string
func (*SimpleToken) Value ¶
func (p *SimpleToken) Value() string
Value returns the processed value of the token (e.g., with escape sequences converted).
type Token ¶
type Token interface {
Type() TokenizerState
Start() syntax.SourceIndexes
End() syntax.SourceIndexes
String() string
Value() string // Returns processed value (e.g., with escape sequences converted)
HasHashDigit() bool // R7RS §7.1.1: whether # appeared as inexact digit placeholder
Radix() int // Effective parse base for integer and decimal-fraction tokens (2/8/10/16); 0 otherwise
}
Token is the interface for tokenizer output tokens.
type Tokenizer ¶
type Tokenizer struct {
// contains filtered or unexported fields
}
Tokenizer reads Scheme source code and produces a stream of tokens.
func NewTokenizer ¶
func NewTokenizer(rdr io.RuneReader, ci bool) *Tokenizer
NewTokenizer creates a new tokenizer that reads from the given RuneReader. The tokenizer is initialized with the first rune already read.
func (*Tokenizer) Err ¶ added in v1.20.0
Err returns the scanner's pending error, or nil.
It exists because Next reports a token and the error that ended it on separate calls: a token of type TokenizerStateFailed is handed to the parser with its explanation — message, index, character, and lexical state — still stashed here. Without this accessor the parser can only report the token's shape ("unknown token type"), discarding the reason the scanner already knew.
io.EOF is a normal terminator on this field, not a fault; callers that treat every non-nil result as a defect will misread a clean end of input.
func (*Tokenizer) Next ¶
Next returns the next token from the input stream. Returns io.EOF when the input is exhausted. End of input is carried by p.err (readNextRune sets io.EOF), never by p.cur: utf8.RuneError is both the decode sentinel and a writable character, so testing the rune reported a literal U+FFFD as EOF and silently discarded the rest of the input. Comment tokens are always emitted; callers that want them elided (the parser, when constructed with skipComments) drop them themselves.
func (*Tokenizer) Reader ¶
func (p *Tokenizer) Reader() io.RuneReader
Reader returns the underlying RuneReader.
type TokenizerError ¶
type TokenizerError struct {
// contains filtered or unexported fields
}
TokenizerError represents an error that occurred during tokenization.
Beyond the message it carries the scanner's state at the moment the fault was detected: where in the input it stopped, the rune it stopped on, and which lexical state it was scanning in. That triple is the whole reason the type exists rather than a bare sentinel — the parser sits above a streaming scanner and cannot recover any of it after the fact, because it drops the tokenizer on error (Parser.locateReaderErr). Stamping at the throw site is what makes the facts survive.
Identity is still the message alone (see Is): position is provenance, not classification, so errors.Is against a bare NewTokenizerError sentinel keeps working.
func NewTokenizerError ¶
func NewTokenizerError(mess string) *TokenizerError
NewTokenizerError creates an unlocated tokenizer error with the given message.
Unlocated is the point: this constructor builds the comparison sentinels for errors.Is. Errors raised while scanning go through Tokenizer.fail, which stamps position, rune, and state.
func NewTokenizerErrorWithWrap ¶
func NewTokenizerErrorWithWrap(err error, mess string) *TokenizerError
NewTokenizerErrorWithWrap creates an unlocated tokenizer error that wraps another error. See NewTokenizerError on why unlocated.
func (*TokenizerError) At ¶ added in v1.20.0
func (p *TokenizerError) At() (syntax.SourceIndexes, bool)
At returns the position of the offending rune and whether the error is located at all. Sentinels built by the New* constructors are not.
Located-ness is decided by the line being 1-based, matching SourceContext.Location: the zero SourceIndexes has line 0, which no real position ever has.
func (*TokenizerError) Error ¶
func (p *TokenizerError) Error() string
Error renders the message followed by the scanner facts that make it actionable: where it stopped, what it stopped on, and the state it was in.
The context is appended rather than prefixed so the message stays the leading text under further wrapping, and so an unlocated sentinel renders as the bare message it always did.
func (*TokenizerError) Is ¶
func (p *TokenizerError) Is(err error) bool
Is implements errors.Is for TokenizerError. Two tokenizer errors match when they carry the same message, so errors.Is can distinguish the message constants (e.g. errors.Is(err, NewTokenizerError(MessageUnterminatedString))); a wrapped cause is still reached through Unwrap.
Position, rune, and state are deliberately excluded: they describe where a failure happened, not which failure it is, and including them would make every sentinel comparison fail.
func (*TokenizerError) Rune ¶ added in v1.20.0
func (p *TokenizerError) Rune() (rune, bool)
Rune returns the rune the scanner stopped on, and whether it is a real character. It is false at end of input, where the scanner reports utf8.RuneError as a sentinel rather than as a character — a distinction the tokenizer has to preserve because a literal U+FFFD is writable source text (see Tokenizer.Next).
func (*TokenizerError) State ¶ added in v1.20.0
func (p *TokenizerError) State() TokenizerState
State returns the lexical state the scanner was in when it stopped.
func (*TokenizerError) Unwrap ¶
func (p *TokenizerError) Unwrap() error
type TokenizerState ¶
type TokenizerState int
TokenizerState represents the type of token recognized by the tokenizer. Each state corresponds to a distinct lexical element in Scheme syntax.
const ( // TokenizerStateFailed indicates tokenization failed. TokenizerStateFailed TokenizerState = iota // TokenizerStateSyntax represents #'expr (syntax quote). TokenizerStateSyntax // TokenizerStateUnsyntax represents #,expr (unsyntax). TokenizerStateUnsyntax // TokenizerStateUnsyntaxSplicing represents #,@expr (unsyntax-splicing). TokenizerStateUnsyntaxSplicing // TokenizerStateQuasisyntax represents #`expr (quasisyntax). TokenizerStateQuasisyntax // TokenizerStateQuote represents 'expr (quote). TokenizerStateQuote // TokenizerStateUnquote represents ,expr (unquote). TokenizerStateUnquote // TokenizerStateUnquoteSplicing represents ,@expr (unquote-splicing). TokenizerStateUnquoteSplicing // TokenizerStateQuasiquote represents `expr (quasiquote). TokenizerStateQuasiquote // TokenizerStateSignedInf represents +inf.0 or -inf.0 (infinity). TokenizerStateSignedInf // TokenizerStateSignedNan represents +nan.0 or -nan.0 (not a number). TokenizerStateSignedNan // TokenizerStateSignedImaginaryInf represents +inf.0i or -inf.0i (imaginary infinity). TokenizerStateSignedImaginaryInf // TokenizerStateSignedImaginaryNan represents +nan.0i or -nan.0i (imaginary NaN). TokenizerStateSignedImaginaryNan // TokenizerStateSignedImaginary represents +i, -i, +3i, -3.5i (pure imaginary). TokenizerStateSignedImaginary // TokenizerStateSignedComplex represents +1+2i, 3.5-2.5i (rectangular complex). TokenizerStateSignedComplex // TokenizerStateSignedComplexPolar represents +1@1.5708 (polar complex: magnitude@angle). TokenizerStateSignedComplexPolar // TokenizerStateUnsignedImaginaryInf represents inf.0i (unsigned imaginary infinity). TokenizerStateUnsignedImaginaryInf // TokenizerStateUnsignedImaginaryNan represents nan.0i (unsigned imaginary NaN). TokenizerStateUnsignedImaginaryNan // TokenizerStateUnsignedImaginary represents 3i, 3.5i (unsigned pure imaginary). TokenizerStateUnsignedImaginary // TokenizerStateUnsignedComplex represents 1+2i (unsigned rectangular complex). TokenizerStateUnsignedComplex // TokenizerStateUnsignedComplexPolar represents 1@1.5708 (unsigned polar complex). TokenizerStateUnsignedComplexPolar // TokenizerStateMarker represents a generic # marker. TokenizerStateMarker // TokenizerStateMarkerBooleanFalse represents #f or #false. TokenizerStateMarkerBooleanFalse // TokenizerStateMarkerBooleanTrue represents #t or #true. TokenizerStateMarkerBooleanTrue // TokenizerStateMarkerNumberInexact represents #i prefix (inexact). TokenizerStateMarkerNumberInexact // TokenizerStateMarkerNumberExact represents #e prefix (exact). TokenizerStateMarkerNumberExact // TokenizerStateSignedInteger represents a signed integer in any base // (-123, +456, #x-FF, #b+101). The base is carried on the token via Radix(). TokenizerStateSignedInteger // TokenizerStateUnsignedInteger represents an unsigned integer in any base // (123, #xFF, #b101). The base is carried on the token via Radix(). TokenizerStateUnsignedInteger // TokenizerStateBigFloat represents #m arbitrary-precision decimal. TokenizerStateBigFloat // TokenizerStateBigInteger represents #z arbitrary-precision integer (decimal). TokenizerStateBigInteger // TokenizerStateMarkerBase2 represents #b prefix (binary). TokenizerStateMarkerBase2 // TokenizerStateMarkerBase8 represents #o prefix (octal). TokenizerStateMarkerBase8 // TokenizerStateMarkerBase10 represents #d prefix (decimal). TokenizerStateMarkerBase10 // TokenizerStateMarkerBase16 represents #x prefix (hexadecimal). TokenizerStateMarkerBase16 // TokenizerStateSignedDecimalFraction represents -1.23 or +4.56. TokenizerStateSignedDecimalFraction // TokenizerStateSignedRationalFraction represents -1/2 or +3/4. TokenizerStateSignedRationalFraction // TokenizerStateUnsignedRationalFraction represents 1/2 or 3/4. TokenizerStateUnsignedRationalFraction // TokenizerStateUnsignedDecimalFraction represents 1.23 or 4.56. TokenizerStateUnsignedDecimalFraction // TokenizerStateSignedScientificNotation represents integers with exponents like +1e10, -2e-5. // Parser determines if result is integer or float based on exponent sign and mantissa. TokenizerStateSignedScientificNotation // TokenizerStateUnsignedScientificNotation represents integers with exponents like 1e10, 2e-5. // Parser determines if result is integer or float based on exponent sign and mantissa. TokenizerStateUnsignedScientificNotation // TokenizerStateEmptyList represents () (empty list). TokenizerStateEmptyList // TokenizerStateOpenParen represents ( (open parenthesis). TokenizerStateOpenParen // TokenizerStateCloseParen represents ) (close parenthesis). TokenizerStateCloseParen // TokenizerStateOpenBracket represents [ (open square bracket). // R7RS §2.1: Square brackets are equivalent to parentheses but must match. TokenizerStateOpenBracket // TokenizerStateCloseBracket represents ] (close square bracket). // R7RS §2.1: Square brackets are equivalent to parentheses but must match. TokenizerStateCloseBracket // TokenizerStateCons represents . (dot for improper lists). TokenizerStateCons // TokenizerStateString represents complete "string". TokenizerStateString // TokenizerStateCharMnemonicOrHexEscape represents intermediate character state. TokenizerStateCharMnemonicOrHexEscape // TokenizerStateCharMnemonic represents #\newline, #\space, etc. TokenizerStateCharMnemonic // TokenizerStateCharHexEscape represents #\x0A (hex escape). TokenizerStateCharHexEscape // TokenizerStateCharGraphic represents #\a (single graphic char). TokenizerStateCharGraphic // TokenizerStateLineCommentBody represents comment text. TokenizerStateLineCommentBody // TokenizerStateBlockCommentBody represents block content. TokenizerStateBlockCommentBody // TokenizerStateDatumCommentBegin represents the #; marker; the following // datum is read by the parser. TokenizerStateDatumCommentBegin // TokenizerStateSymbol represents an identifier or symbol. TokenizerStateSymbol // TokenizerStateOpenVector represents #( (vector). TokenizerStateOpenVector // TokenizerStateOpenVectorUnsignedByteMarker represents #u8( (bytevector). TokenizerStateOpenVectorUnsignedByteMarker // TokenizerStateDirective represents #!fold-case, etc. TokenizerStateDirective // TokenizerStateLabelReference represents #123# (datum label reference). TokenizerStateLabelReference // TokenizerStateLabelAssignment represents #123= (datum label assignment). TokenizerStateLabelAssignment // TokenizerStateBoxBegin represents the #& marker; like #;, it introduces a // datum that the parser reads as the next token. TokenizerStateBoxBegin )
TokenizerState values for different token types.
func (TokenizerState) String ¶ added in v1.20.0
func (p TokenizerState) String() string
String returns the state's kebab-case name, for reader diagnostics.
An out-of-range value renders as its number rather than a fixed "unknown", so a state added to the const block without a tokenizerStateNames entry is still identifiable from a user's bug report.
Source Files