lexer

Overview ¶

Package lexer turns source bytes into tokens.

Phase 0 carries the seeds of MRI's stateful lexer — SpaceBefore on every token and a lexState field — without yet exercising the hard cases (regex vs division, heredocs, interpolation). Those land in later phases (plan §10); the state plumbing is here from the start so they slot in without a rewrite.