Why Go
- Case Studies
  
  Common problems companies solve with Go
- Use Cases
  
  Stories about how and why companies use Go
- Security
  
  How Go can help keep you secure by default
Learn
Docs
- Effective Go
  
  Tips for writing clear, performant, and idiomatic Go code
- Go User Manual
  
  A complete introduction to building software with Go
- Standard library
  
  Reference documentation for Go's standard library
- Release Notes
  
  Learn what's new in each Go release
Packages
Community
- Recorded Talks
  
  Videos from prior events
- Meetups
  
  Meet other local Go developers
- Conferences
  
  Learn and network with Go developers from around the world
- Go blog
  
  The Go project's official blog.
- Go project
  
  Get help and stay informed from Go
- Get connected

token

package

Go to main page

Versions in this module

v1

v1.8.1

Mar 20, 2026

v1.8.0

Feb 18, 2026

v1.7.0

Dec 7, 2025

Changes in this version

+ type Stream struct

+ func New(normalizedText string) *Stream

+ func (s *Stream) Advance() bool

+ func (s *Stream) AtEnd() bool

+ func (s *Stream) ConsumeChars(n int) (string, bool)

+ func (s *Stream) Current() (Token, bool)

+ func (s *Stream) Peek(offset int) (Token, bool)

+ func (s *Stream) Remaining() []Token

+ func (s *Stream) RemainingText() string

+ func (s *Stream) RestorePosition(pos int)

+ func (s *Stream) SavePosition() int

+ func (s *Stream) SetPartialToken(remaining string)

+ func (s *Stream) Skip(predicate func(string) bool)

+ func (s *Stream) Text() string

+ type Token struct

+ Position int

+ Text string