Documentation
¶
Overview ¶
Package protolex implements a pull-based lexer for .proto files that converts raw source text into a stream of typed tokens with position tracking. It supports proto2, proto3, and editions syntax as the first stage of the .proto file processing pipeline.
The core types are TokenType, a named int8 enum discriminating 8 token kinds (EOF, identifier, integer literal, float literal, string literal, keyword, symbol, comment), and Token, a struct carrying the token type, raw source text, and 1-based line and column position.
The entry point is NewLexer, which accepts a filename (for error messages) and the source bytes. The caller drives scanning by calling Next in a loop; each call returns the next token until EOF:
lex := protolex.NewLexer("example.proto", src)
for {
tok, err := lex.Next()
if err != nil {
// handle error; lexer recovers and continues
}
if tok.Type == protolex.TokenEOF {
break
}
// process tok
}
The lexer recognizes identifiers, 43 protobuf keywords (general, built-in type, boolean literal, and special constant keywords), integer literals (decimal, octal, hex), float literals, string literals with escape sequences, 14 single-character symbols, and both single-line (//) and block (/* */) comments. Whitespace is consumed between tokens and never emitted.
Errors are returned as the second value from Next with the format "protolex: filename:line:col: message". After an error the lexer advances past the problematic input, allowing callers to collect multiple diagnostics in a single pass.
The lexer slices into the caller-provided input buffer rather than copying bytes, avoiding heap allocations on the hot path. The input must not be modified while the lexer is in use.
Parsing, import resolution, semantic validation, string literal concatenation, and Unicode escapes (\u, \U) are out of scope.
This package imports only fmt from the standard library and has zero external or internal module dependencies.
Package protolex -- lexer_number.go contains integer and float literal scanning functions, handling decimal, hexadecimal, octal, and floating-point number formats.
Package protolex -- lexer_string.go contains string literal scanning functions, including escape sequence handling for octal, hex, and Unicode escapes.
Package protolex -- lexer_token.go contains the Lexer type, constructor, main token dispatch (Next), identifier/comment scanning, and low-level position tracking helpers.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type LexError ¶
type LexError struct {
// Filename is the source file being lexed.
Filename string
// Line is the 1-based line number where the error occurred.
Line int
// Column is the 1-based column number where the error occurred.
Column int
// Detail is a human-readable description of the failure.
Detail string
// Cause is the underlying error, if any.
Cause error
}
LexError is a typed error returned when lexical analysis encounters an invalid token or character at a known source location. It includes file, line, and column context and wraps an optional underlying cause.
type Lexer ¶
type Lexer struct {
// contains filtered or unexported fields
}
Lexer is a pull-based scanner for .proto source files. Callers create a Lexer with NewLexer and then call Next repeatedly to obtain tokens until TokenEOF is returned. The lexer slices into the caller-provided input buffer rather than copying bytes, avoiding heap allocations on the hot path.
func NewLexer ¶
NewLexer creates a Lexer that will tokenize the given input bytes. The filename is stored for inclusion in error messages but is not otherwise interpreted. The lexer does not copy the input buffer; it slices into it, so the caller must not modify the buffer while the lexer is in use.
NewLexer panics if input is nil. An empty slice ([]byte{}) is valid and produces only TokenEOF.
func (*Lexer) Next ¶
Next advances the lexer and returns the next token. Whitespace is consumed before each token. At end of input, Next returns TokenEOF with a nil error; subsequent calls continue returning TokenEOF. On invalid input, Next returns TokenInvalid with a non-nil error and advances past the problematic byte so that scanning can continue.
type Token ¶
Token represents a single token scanned from a .proto source file. Value holds the raw text exactly as it appears in the source, including quote characters for string literals and // or /* */ delimiters for comments. Line is the 1-based line number of the first character of the token. Column is the 1-based byte offset from the start of the line.
fieldalignment: fields ordered for semantic clarity, not padding
type TokenType ¶
type TokenType int8
TokenType represents the kind of token produced by the lexer. It is a named type over int8 to provide type safety. The zero value (TokenInvalid) is intentionally invalid.
const ( TokenEOF TokenType = iota + 1 // 1 TokenIdentifier // 2 TokenIntLiteral // 3 TokenFloatLiteral // 4 TokenStringLiteral // 5 TokenKeyword // 6 TokenSymbol // 7 TokenComment // 8 )
Token type constants for the 8 valid token kinds. Values start at 1 so that the zero value of TokenType is invalid by design.
const TokenInvalid TokenType = 0
TokenInvalid is the zero value of TokenType, used for error tokens.
Source Files