Documentation
¶
Overview ¶
Package protoparse implements a recursive descent parser for Protocol Buffer definition files (.proto). It consumes a token stream from the protolex package and produces a typed abstract syntax tree (AST) representing the complete structure of a .proto file, including all declarations, nested constructs, and attached comments.
The two entry points are Parse and ParseTokens. Parse accepts a filename and raw byte input, creates a lexer internally, and returns the parsed AST. ParseTokens accepts a pre-constructed protolex.Lexer, allowing callers to control lexer construction or reuse. Both return a *File (the AST root) and an error. On success the error is nil; on failure the error is an ErrorList containing one or more parse errors, and the returned *File is a partial AST representing everything that was successfully parsed before and between errors.
file, err := protoparse.Parse("example.proto", src)
if err != nil {
var errs protoparse.ErrorList
if errors.As(err, &errs) {
for _, e := range errs {
fmt.Println(e)
}
}
}
The File type is the root of the AST. It contains optional SyntaxDecl or EditionDecl (mutually exclusive), an optional PackageDecl, a slice of ImportDecl entries, top-level OptionDecl entries, and a Decls slice holding all top-level declarations in source order. The Decls slice contains values implementing the sealed Decl interface: *MessageDecl, *EnumDecl, *ServiceDecl, *ExtendDecl, and *OptionDecl.
Message bodies contain elements implementing the sealed MessageElement interface: *FieldDecl, *MapFieldDecl, *OneofDecl, *GroupDecl, *EnumDecl, *MessageDecl, *ExtendDecl, *ExtensionsDecl, *ReservedDecl, and *OptionDecl. All elements within a message body are stored in source order in the Body slice.
The parser supports all three protobuf dialects: proto2 (with required fields, groups, extensions, and extend blocks), proto3 (with label-less fields, maps, oneofs, and services), and editions (edition declaration in place of syntax). It handles string literal concatenation, aggregate (message) literal values in options, fully qualified type names with leading dots, and inline field options in bracket notation.
Comments are preserved in the AST. Leading comments (those immediately preceding a declaration with no intervening blank line) are attached to the declaration's Comments slice. Trailing comments (those on the same line after a semicolon or closing brace) are marked with Comment.Trailing set to true. Detached comments (those separated from any declaration by blank lines) are collected in File.Comments.
The package follows a split error convention. Programmer errors cause panics with a "protoparse: " prefix: Parse panics if input is nil, and ParseTokens panics if the lexer is nil. Data-driven failures (malformed .proto input) are returned as an ErrorList. Each error in the list is formatted as "protoparse: filename:line:col: message" for consistent grep-friendly output.
Error recovery uses synchronization on semicolons, closing braces, and top-level keywords. When the parser encounters an unexpected token, it skips forward to a synchronization point and resumes parsing, accumulating all errors into the ErrorList. This allows a single parse pass to report multiple errors rather than stopping at the first one.
The parser is a pure syntax pass. It does not perform semantic validation such as checking for duplicate field numbers, verifying that enum value zero exists, resolving type references, or enforcing proto2 versus proto3 rules beyond syntactic structure. Those responsibilities belong to a downstream linker or validation pass (see roadmap item 17).
This package depends on protolex from this module and the Go standard library (fmt, math, strconv, strings). It has zero external dependencies.
parse_enum.go handles parsing of enum declarations and enum value declarations in .proto files, and extraction of Go-specific edition features from parsed enum options.
parse_field.go handles parsing of field declarations, map fields, oneof declarations, group declarations, and field-related utilities (type names, field options, ranges) in .proto files.
parse_file.go contains the top-level file parsing logic for .proto files. It handles the file-level declarations: syntax, edition, import, and package. It also provides resolution functions for Go-specific edition features that apply at the file level, such as symbol visibility, naming style, and API mode.
parse_message.go handles parsing of message declarations, message body elements, labeled elements, extend blocks, extensions declarations, and reserved declarations in .proto files.
parse_option.go handles parsing of option declarations, option names, literal values, and aggregate (message) literals in .proto files.
parse_service.go handles parsing of service declarations and RPC method declarations in .proto files.
parser.go contains the core parser infrastructure: the parser struct, public Parse/ParseTokens entry points, token advancement, error recovery, and low-level token consumption helpers.
Index ¶
- func ResolveGoAPIMode(options []*OptionDecl) descriptor.GoAPIMode
- func ResolveNamingStyle(options []*OptionDecl) descriptor.NamingStyle
- func ResolveStripEnumPrefix(options []*OptionDecl) descriptor.StripEnumPrefix
- func ResolveSymbolVisibility(options []*OptionDecl) descriptor.SymbolVisibility
- type AggregateField
- type Comment
- type Decl
- type EditionDecl
- type EnumDecl
- type EnumValueDecl
- type ErrorList
- type ExtendDecl
- type ExtensionsDecl
- type FieldDecl
- type FieldOption
- type File
- type GroupDecl
- type ImportDecl
- type ImportModifier
- type Literal
- type LiteralKind
- type MapFieldDecl
- type MessageDecl
- type MessageElement
- type MethodDecl
- type OneofDecl
- type OptionDecl
- type OptionName
- type OptionNamePart
- type PackageDecl
- type ParseError
- type Position
- type Range
- type ReservedDecl
- type ServiceDecl
- type SyntaxDecl
- type TypeName
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func ResolveGoAPIMode ¶
func ResolveGoAPIMode(options []*OptionDecl) descriptor.GoAPIMode
ResolveGoAPIMode examines the parsed file-level options for a Go-specific edition feature extension setting the api_mode behavior. It looks for options with the extension name pattern "(go.features)" and returns the resolved GoAPIMode value. If no matching option is found, it returns the zero value (no override).
func ResolveNamingStyle ¶
func ResolveNamingStyle(options []*OptionDecl) descriptor.NamingStyle
ResolveNamingStyle examines the parsed file-level options for a Go-specific edition feature extension setting the enforce_naming_style behavior. It looks for options with the extension name pattern "(go.features)" and returns the resolved NamingStyle value. If no matching option is found, it returns the zero value (no override).
func ResolveStripEnumPrefix ¶
func ResolveStripEnumPrefix(options []*OptionDecl) descriptor.StripEnumPrefix
ResolveStripEnumPrefix examines the parsed enum options for a Go-specific edition feature extension setting the strip_enum_prefix behavior. It looks for options with the extension name pattern "(go.features)" or "(go.enum).strip_enum_prefix" and returns the resolved StripEnumPrefix value. If no matching option is found, it returns the zero value (no override).
func ResolveSymbolVisibility ¶
func ResolveSymbolVisibility(options []*OptionDecl) descriptor.SymbolVisibility
ResolveSymbolVisibility examines the parsed file-level options for a Go-specific edition feature extension setting the default_symbol_visibility behavior. It looks for options with the extension name pattern "(go.features)" and returns the resolved SymbolVisibility value. If no matching option is found, it returns the zero value (no override).
Types ¶
type AggregateField ¶
type AggregateField struct {
Pos Position
Name OptionName
Value Literal
}
AggregateField represents a field: value or field { ... } entry inside an aggregate (message) literal value.
type Comment ¶
Comment holds a single comment from the source text. Text includes the delimiters (// or /* */). Trailing is true when the comment appears on the same line after a declaration.
type Decl ¶
type Decl interface {
// contains filtered or unexported methods
}
Decl is a sealed interface representing a top-level declaration in a .proto file. It is implemented by *MessageDecl, *EnumDecl, *ServiceDecl, and *ExtendDecl.
type EditionDecl ¶
EditionDecl represents an edition declaration (e.g., edition = "2023";). Value holds the string literal value without quotes.
type EnumDecl ¶
type EnumDecl struct {
Pos Position
Name string
Values []*EnumValueDecl
Options []*OptionDecl
Reserved []*ReservedDecl
Comments []Comment
}
EnumDecl represents an enum declaration. It implements both Decl and MessageElement because enums can appear at the top level and nested inside messages.
type EnumValueDecl ¶
type EnumValueDecl struct {
Pos Position
Name string
Number int32
Options []*FieldOption
Comments []Comment
}
EnumValueDecl represents a single value within an enum declaration.
fieldalignment: fields ordered to match .proto source declaration order
type ErrorList ¶
type ErrorList []error
ErrorList collects multiple parse errors encountered during a single parse operation. It implements the error interface; its Error method joins all contained messages with newlines. The Unwrap method returns the underlying slice for use with errors.As and errors.Is.
type ExtendDecl ¶
type ExtendDecl struct {
Pos Position
TypeName TypeName
Fields []*FieldDecl
Groups []*GroupDecl
Comments []Comment
}
ExtendDecl represents an extend block. It implements both Decl and MessageElement because extend blocks can appear at the top level and inside messages.
type ExtensionsDecl ¶
type ExtensionsDecl struct {
Pos Position
Ranges []Range
Options []*FieldOption
Comments []Comment
}
ExtensionsDecl represents an extensions range declaration within a message. It implements MessageElement.
type FieldDecl ¶
type FieldDecl struct {
Pos Position
Label string
TypeName TypeName
Name string
Number int32
Options []*FieldOption
Comments []Comment
}
FieldDecl represents a field declaration within a message or oneof. Label is one of "optional", "required", "repeated", or "" for label-less proto3 fields.
fieldalignment: fields ordered to match .proto source declaration order
type FieldOption ¶
type FieldOption struct {
Pos Position
Name OptionName
Value Literal
}
FieldOption represents a single option within the inline option list of a field declaration (the [...] after the field number).
type File ¶
type File struct {
Pos Position
Syntax *SyntaxDecl
Edition *EditionDecl
Package *PackageDecl
Imports []*ImportDecl
Options []*OptionDecl
Decls []Decl
Comments []Comment
}
File is the root AST node representing a parsed .proto file. Exactly one of Syntax or Edition may be non-nil; both nil means the file had no syntax/edition declaration.
type GroupDecl ¶
type GroupDecl struct {
Pos Position
Label string
Name string
Number int32
Body []MessageElement
Comments []Comment
}
GroupDecl represents a proto2 legacy group declaration.
fieldalignment: fields ordered to match .proto source declaration order
type ImportDecl ¶
type ImportDecl struct {
Pos Position
Path string
Modifier ImportModifier
Comments []Comment
}
ImportDecl represents an import declaration. Path holds the import path without quotes. Modifier indicates whether the import is weak or public.
fieldalignment: fields ordered for semantic clarity, not padding
type ImportModifier ¶
type ImportModifier int8
ImportModifier represents the modifier on an import declaration. The zero value ImportNone means no modifier. ImportWeak and ImportPublic correspond to the weak and public keywords respectively.
const ( // ImportNone indicates a plain import with no modifier. ImportNone ImportModifier = 0 // ImportWeak indicates a weak import. ImportWeak ImportModifier = iota // 1 // ImportPublic indicates a public import. ImportPublic // 2 )
func (ImportModifier) String ¶
func (m ImportModifier) String() string
String returns the name of the import modifier. For valid values (0-2) it returns "none", "weak", or "public". For out-of-range values it returns a formatted string like "ImportModifier(5)".
func (ImportModifier) Valid ¶
func (m ImportModifier) Valid() bool
Valid reports whether m is one of the three defined import modifiers (0 through 2 inclusive).
type Literal ¶
type Literal struct {
Pos Position
Kind LiteralKind
Value string
Aggregate []*AggregateField
Sign string
}
Literal holds a parsed literal value. Kind indicates the type of the literal. Value holds the raw text for scalar literals. Aggregate holds the fields for aggregate literals. Sign holds "-" or "+" if a sign preceded the literal.
type LiteralKind ¶
type LiteralKind int8
LiteralKind represents the type of a literal value in the AST. The zero value is intentionally invalid.
const ( // LitString represents a string literal value. LitString LiteralKind = iota + 1 // 1 // LitInt represents an integer literal value. LitInt // 2 // LitFloat represents a floating-point literal value. LitFloat // 3 // LitBool represents a boolean literal value (true or false). LitBool // 4 // LitIdentifier represents an identifier literal (e.g., inf, nan, or enum // value names). LitIdentifier // 5 // LitAggregate represents an aggregate (message) literal value enclosed // in braces. LitAggregate // 6 )
func (LiteralKind) String ¶
func (k LiteralKind) String() string
String returns the name of the literal kind. For valid values (1-6) it returns names such as "string" and "aggregate". For out-of-range values it returns a formatted string like "LiteralKind(7)".
func (LiteralKind) Valid ¶
func (k LiteralKind) Valid() bool
Valid reports whether k is one of the six defined literal kinds (1 through 6 inclusive).
type MapFieldDecl ¶
type MapFieldDecl struct {
Pos Position
KeyType string
ValueType TypeName
Name string
Number int32
Options []*FieldOption
Comments []Comment
}
MapFieldDecl represents a map field declaration (e.g., map<string, int32> name = 1;). KeyType holds the map key type keyword.
fieldalignment: fields ordered to match .proto source declaration order
type MessageDecl ¶
type MessageDecl struct {
Pos Position
Name string
Body []MessageElement
Comments []Comment
}
MessageDecl represents a message declaration. Body holds all nested elements in source order. It implements both Decl and MessageElement because messages can appear at the top level and nested inside other messages.
type MessageElement ¶
type MessageElement interface {
// contains filtered or unexported methods
}
MessageElement is a sealed interface representing an element that can appear inside a message body. It is implemented by *FieldDecl, *MapFieldDecl, *OneofDecl, *EnumDecl, *MessageDecl, *ExtendDecl, *ExtensionsDecl, *ReservedDecl, *OptionDecl, and *GroupDecl.
type MethodDecl ¶
type MethodDecl struct {
Pos Position
Name string
InputType TypeName
OutputType TypeName
ClientStreaming bool
ServerStreaming bool
Options []*OptionDecl
Comments []Comment
}
MethodDecl represents an RPC method declaration within a service. ClientStreaming and ServerStreaming indicate whether the stream keyword appeared before the input or output type respectively.
fieldalignment: fields ordered to match .proto source declaration order
type OneofDecl ¶
type OneofDecl struct {
Pos Position
Name string
Fields []*FieldDecl
Options []*OptionDecl
Comments []Comment
}
OneofDecl represents a oneof declaration. Fields inside a oneof do not carry a label.
type OptionDecl ¶
type OptionDecl struct {
Pos Position
Name OptionName
Value Literal
Comments []Comment
}
OptionDecl represents an option declaration. It implements both Decl and MessageElement because options can appear at the top level and inside message bodies.
type OptionName ¶
type OptionName struct {
Parts []OptionNamePart
}
OptionName represents the full path of an option name, such as (com.example.my_option).field.subfield, as a sequence of parts.
type OptionNamePart ¶
OptionNamePart represents a single component of an option name path. When IsExtension is true, the part was parenthesized in the source (e.g., (com.example.my_option)).
type PackageDecl ¶
PackageDecl represents a package declaration. Name holds the full dotted package name (e.g., "foo.bar.baz").
type ParseError ¶
type ParseError struct {
// Pos is the source location where the error occurred.
Pos Position
// Detail is a human-readable description of the failure.
Detail string
// Cause is the underlying error, if any.
Cause error
}
ParseError is a typed error returned when proto file parsing encounters a syntax or semantic error at a known source location. It includes file, line, and column context and wraps an optional underlying cause.
func (*ParseError) Error ¶
func (e *ParseError) Error() string
Error returns a human-readable message with the source location prefix.
func (*ParseError) Unwrap ¶
func (e *ParseError) Unwrap() error
Unwrap returns the underlying cause so that errors.Is and errors.As work.
type Position ¶
Position records the source location of a token or AST node within a .proto file. Line and Column are 1-based.
type Range ¶
Range represents a numeric range used in extensions and reserved declarations. For single-value ranges Start equals End and Max is false. Max is true when the range endpoint was the keyword max.
type ReservedDecl ¶
ReservedDecl represents a reserved declaration within a message or enum. A single reserved statement contains either ranges or names, never both. It implements MessageElement.
type ServiceDecl ¶
type ServiceDecl struct {
Pos Position
Name string
Methods []*MethodDecl
Options []*OptionDecl
Comments []Comment
}
ServiceDecl represents a service declaration. It implements Decl because services are top-level declarations.
type SyntaxDecl ¶
SyntaxDecl represents a syntax declaration (e.g., syntax = "proto3";). Value holds the string literal value without quotes.