The highest tagged major version is
Documentation
¶
Overview ¶
Package lexer implements the CalcMark lexer/tokenizer.
The lexer converts CalcMark source code into a stream of tokens that can be consumed by the parser. It handles Unicode-aware tokenization including international characters, emojis, and various number formats.
Architecture ¶
The lexer is implemented as a hand-written scanner (not generated) to provide precise control over token recognition and error messages. It uses a single-pass, character-by-character approach with lookahead.
Usage ¶
Basic tokenization:
lexer := lexer.NewLexer("x = 5 + 3")
tokens, err := lexer.Tokenize()
if err != nil {
log.Fatal(err)
}
for _, token := range tokens {
fmt.Printf("%s: %s\n", token.Type, token.Value)
}
Token Types ¶
The lexer recognizes:
- Numbers: integers, decimals, with thousands separators (1,000 or 1_000)
- Multipliers: k, M, B, T (e.g., "1.5M" → 1,500,000)
- Percentages: 20% → 0.20
- Currency: $, €, £, ¥ and ISO 4217 codes (USD, EUR, etc.)
- Identifiers: Unicode-aware including emoji
- Operators: +, -, *, /, %, ^, =, ==, !=, <, >, <=, >=
- Keywords: Reserved words and boolean values
- Functions: avg, sqrt and natural language variants
Number Formats ¶
The lexer supports flexible number formatting:
42 // Integer 3.14 // Decimal 1,000 // Thousands separator (comma) 1_000_000 // Thousands separator (underscore) 1.5M // Multiplier (1,500,000) 20% // Percentage (0.20) 1.2e6 // Scientific notation (1,200,000)
Currency Recognition ¶
Currency can be specified with symbols or ISO 4217 codes:
$100 // Symbol prefix 100 USD // Code suffix (space required) USD100 // Code prefix (no space)
Unicode Support ¶
Identifiers can contain:
- Letters from any Unicode script
- Digits (after first character)
- Underscores
- Emoji characters
Examples:
salary // ASCII 給料 // Japanese café // French with accents 💰_total // Emoji + underscore
Reserved Keywords ¶
The following keywords are reserved and cannot be used as identifiers:
- Booleans: true, false, yes, no, t, f, y, n (case-insensitive)
- Functions: avg, sqrt
- Control flow: if, then, else, for, while, return (reserved for future)
- Logical: and, or, not (reserved for future)
Error Handling ¶
Lexer errors include position information:
Lexer error at line 1, column 5: unexpected character '!'
Performance ¶
The lexer is optimized for speed and typically completes in hundreds of nanoseconds:
- Currency token: ~600ns
- Number with multiplier: ~780ns
- Identifier: ~900ns
See benchmark_test.go in the parser package for detailed metrics.
Package lexer implements the CalcMark lexer/tokenizer
Index ¶
Constants ¶
const ( // MaxIdentifierLength limits identifier length // Example: prevents extremely long variable names MaxIdentifierLength = 256 // MaxNumberLength limits number literal length // Example: prevents pathological number parsing MaxNumberLength = 100 )
Security limits to prevent DOS attacks
Variables ¶
var BooleanKeywords = map[string]bool{ "true": true, "false": true, }
BooleanKeywords defines the valid boolean keyword values (case-insensitive). This map is exported for grammar introspection.
ContextualKeywords are identifiers that should NOT be consumed as unit names when they follow a number. They are used as separators in NL function syntax (e.g., "compound 1000 by 5%") but are not full reserved keywords, so they remain valid as standalone identifiers in other contexts.
var DateKeywords = map[string]TokenType{ "today": DATE_TODAY, "tomorrow": DATE_TOMORROW, "yesterday": DATE_YESTERDAY, }
DateKeywords maps date keyword strings to token types Performance: O(1) lookup via map
var EmojiRanges = []EmojiRange{
{Start: 0x2600, End: 0x26FF, Name: "Miscellaneous Symbols"},
{Start: 0x2700, End: 0x27BF, Name: "Dingbats"},
{Start: 0x2B50, End: 0x2B55, Name: "Stars and Circles"},
{Start: 0x1F600, End: 0x1F64F, Name: "Emoticons"},
{Start: 0x1F300, End: 0x1F5FF, Name: "Miscellaneous Symbols and Pictographs"},
{Start: 0x1F680, End: 0x1F6FF, Name: "Transport and Map Symbols"},
{Start: 0x1F900, End: 0x1F9FF, Name: "Supplemental Symbols and Pictographs"},
{Start: 0x1FA00, End: 0x1FA6F, Name: "Symbols and Pictographs Extended-A"},
}
EmojiRanges defines the Unicode emoji ranges supported for identifiers. This list is exported for grammar introspection.
var MonthNames = map[string]string{
"jan": "January",
"january": "January",
"feb": "February",
"february": "February",
"mar": "March",
"march": "March",
"apr": "April",
"april": "April",
"may": "May",
"jun": "June",
"june": "June",
"jul": "July",
"july": "July",
"aug": "August",
"august": "August",
"sep": "September",
"sept": "September",
"september": "September",
"oct": "October",
"october": "October",
"nov": "November",
"november": "November",
"dec": "December",
"december": "December",
}
MonthNames maps month abbreviations and full names to canonical month names Performance: O(1) lookup via map
var RelativeDateKeywords = map[string]TokenType{ "this week": DATE_THIS_WEEK, "this month": DATE_THIS_MONTH, "this year": DATE_THIS_YEAR, "next week": DATE_NEXT_WEEK, "next month": DATE_NEXT_MONTH, "next year": DATE_NEXT_YEAR, "last week": DATE_LAST_WEEK, "last month": DATE_LAST_MONTH, "last year": DATE_LAST_YEAR, }
RelativeDateKeywords maps multi-word relative date keywords to token types These are checked as phrases (e.g., "this week") Performance: O(1) lookup via map
var ReservedKeywords = map[string]TokenType{ "and": AND, "or": OR, "not": NOT, "if": IF, "then": THEN, "else": ELSE, "elif": ELIF, "end": END, "for": FOR, "from": FROM, "in": IN, "as": AS, "at": AT, "napkin": NAPKIN, "precise": PRECISE, "of": OF, "per": PER, "over": OVER, "with": WITH, "while": WHILE, "return": RETURN, "break": BREAK, "continue": CONTINUE, "let": LET, "const": CONST, "avg": FUNC_AVG, "sqrt": FUNC_SQRT, "sum": FUNC_SUM, "number": FUNC_NUMBER, }
ReservedKeywords defines reserved keywords (Go spec compliant + future control flow). This map is exported for grammar introspection. See: https://go.dev/ref/spec#Keywords
var TimeUnits = map[string]string{
"millisecond": "millisecond",
"milliseconds": "millisecond",
"ms": "millisecond",
"second": "second",
"seconds": "second",
"sec": "second",
"secs": "second",
"minute": "minute",
"minutes": "minute",
"min": "minute",
"mins": "minute",
"hour": "hour",
"hours": "hour",
"hr": "hour",
"hrs": "hour",
"day": "day",
"days": "day",
"week": "week",
"weeks": "week",
"month": "month",
"months": "month",
"year": "year",
"years": "year",
"yr": "year",
"yrs": "year",
}
TimeUnits maps time unit keywords to canonical forms Performance: O(1) lookup via map
Functions ¶
func IsQuantityUnit ¶
IsQuantityUnit checks if a string is a known unit for quantities (mass, length, volume, currency). Exported for use by document detector.
Types ¶
type EmojiRange ¶
EmojiRange represents a Unicode emoji range
type Lexer ¶
type Lexer struct {
// contains filtered or unexported fields
}
Lexer tokenizes CalcMark expressions
type LexerError ¶
LexerError represents a lexer error
func (*LexerError) Error ¶
func (e *LexerError) Error() string
type LexerSecurityError ¶
LexerSecurityError represents a security limit violation in the lexer
func (*LexerSecurityError) Error ¶
func (e *LexerSecurityError) Error() string
type Token ¶
type Token struct {
Type TokenType
Value string // Normalized value (e.g., "1000" with separators stripped)
OriginalText string // Original text from source (e.g., "1,000")
Line int
Column int
StartPos int // Byte offset in source where token starts
EndPos int // Byte offset in source where token ends
}
Token represents a lexical token
func TokenizeOld ¶
TokenizeOld scans the input string and returns a slice of tokens. Deprecated: Use Tokenize() from adapter.go instead.
type TokenType ¶
type TokenType int
TokenType represents the type of a token
const ( // Literals NUMBER TokenType = iota CURRENCY // DEPRECATED: Use QUANTITY (kept for backward compatibility) QUANTITY // Unified type for numbers with units (currency, measurements, etc.) BOOLEAN IDENTIFIER // Currency (split for parser) CURRENCY_SYM // $ CURRENCY_CODE // USD // Multiplier Numbers (for gocc integration) NUMBER_PERCENT // 5% NUMBER_K // 12k NUMBER_M // 1.2M NUMBER_B // 5B NUMBER_T // 2.5T NUMBER_SCI // 1.2e10, 4.5e-7 // Operators PLUS MINUS MULTIPLY DIVIDE MODULUS EXPONENT ASSIGN // Comparison operators GREATER_THAN LESS_THAN GREATER_EQUAL LESS_EQUAL EQUAL NOT_EQUAL // Logical operators (Go spec compliant) // See: https://go.dev/ref/spec#Logical_operators AND // "and" OR // "or" NOT // "not" // Grouping LPAREN RPAREN // Punctuation COMMA // "," DOT // "." - for qualified names AT_SIGN // "@" - for directive references (@scale, @globals.name) // Reserved keywords for future control flow IF THEN ELSE ELIF END FOR // Keywords for natural language constructs AS // "as" - for conversions like "as napkin" AT // "at" - capacity planning: "10 TB at 2 TB per disk" FROM // "from" - for date expressions: "2 days from today" IN // "in" - for unit conversions: "10 meters in feet" NAPKIN // "napkin" - for human-readable number formatting (e.g., "1234567 as napkin") PRECISE // "precise" - for full-precision display (e.g., "1 second as hour as precise") OF // "of" - for percentage expressions: "10% of 200" PER // "per" - rate expressions: "100 MB per second" OVER // "over" - rate accumulation: "100 MB/s over 1 day" WITH // "with" - capacity planning: "10000 req/s with 450 req/s capacity" (legacy) DOWNTIME // "downtime" - for availability (e.g., "99.9% downtime per month") WHILE RETURN BREAK CONTINUE LET CONST // Reserved function names (cannot be used as variables) FUNC_AVG // "avg" - canonical name FUNC_SQRT // "sqrt" - canonical name FUNC_SUM // "sum" - canonical name FUNC_NUMBER // "number" - canonical name // Multi-token function keywords (aliases) FUNC_AVERAGE_OF // "average of" → maps to "avg" FUNC_SQUARE_ROOT_OF // "square root of" → maps to "sqrt" FUNC_SUM_OF // "sum of" → maps to "sum" // Date keywords DATE_TODAY // "today" DATE_TOMORROW // "tomorrow" DATE_YESTERDAY // "yesterday" // Relative date keywords DATE_THIS_WEEK // "this week" DATE_THIS_MONTH // "this month" DATE_THIS_YEAR // "this year" DATE_NEXT_WEEK // "next week" DATE_NEXT_MONTH // "next month" DATE_NEXT_YEAR // "next year" DATE_LAST_WEEK // "last week" DATE_LAST_MONTH // "last month" DATE_LAST_YEAR // "last year" // Date/Duration literals (combined by lexer) DATE_LITERAL // "Dec 12", "December 25 2025" DURATION_LITERAL // "2 days", "3 weeks and 4 days" // Special NEWLINE EOF ERROR // Security/syntax error token )
Source Files