Documentation
¶
Overview ¶
Package token provides a token stream implementation for parsing with backtracking support.
Index ¶
- type Stream
- func (s *Stream) Advance() bool
- func (s *Stream) AtEnd() bool
- func (s *Stream) ConsumeChars(n int) (string, bool)
- func (s *Stream) Current() (Token, bool)
- func (s *Stream) Peek(offset int) (Token, bool)
- func (s *Stream) Remaining() []Token
- func (s *Stream) RemainingText() string
- func (s *Stream) RestorePosition(pos int)
- func (s *Stream) SavePosition() int
- func (s *Stream) SetPartialToken(remaining string)
- func (s *Stream) Skip(predicate func(string) bool)
- func (s *Stream) Text() string
- type Token
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Stream ¶
type Stream struct {
// contains filtered or unexported fields
}
Stream wraps a slice of tokens with position tracking and backtracking support.
func (*Stream) Advance ¶
Advance moves to the next token. Returns false if at the end. Clears any partial token before advancing.
func (*Stream) ConsumeChars ¶
ConsumeChars consumes n characters from the current token and sets the rest as partial. Returns the consumed text and whether the operation succeeded.
func (*Stream) Current ¶
Current returns the current token and whether it exists. If there's a partial token buffered, returns that instead.
func (*Stream) Peek ¶
Peek returns the token at the given offset from the current position. Offset 0 returns the current token, 1 returns the next, etc.
func (*Stream) RemainingText ¶
RemainingText returns the text of all remaining tokens joined by spaces.
func (*Stream) RestorePosition ¶
RestorePosition restores a previously saved position.
func (*Stream) SavePosition ¶
SavePosition returns the current position for later restoration.
func (*Stream) SetPartialToken ¶
SetPartialToken sets the remaining text from a partially consumed token. This allows consuming only part of a token (e.g., "13015" -> "130" consumed, "15" remains).
Source Files