Documentation
¶
Overview ¶
Package syntax provides a lossless lexer, AST, and recursive-descent parser for the Apache Thrift IDL.
Grammar reference: the Apache Thrift compiler (compiler/cpp/src/thrift/thriftl.ll and thrifty.yy).
Losslessness: comments are first-class tokens in the token stream, in source order. Whitespace itself is not preserved; instead each token (comments included) records how many blank lines preceded it, which is the only layout information a formatter is allowed to act on.
Index ¶
- Variables
- func Dump(d *Document) string
- func IsComment(k TokenKind) bool
- func IsKeyword(k TokenKind) bool
- func IsTypeKeyword(k TokenKind) bool
- func Lex(src []byte) ([]Token, []Error)
- func NextReal(toks []Token, i int) int
- func Parse(src []byte) (*Document, []Error)
- func ParseTokens(toks []Token) (*Document, []Error)
- func PrevReal(toks []Token, i int) int
- func Walk(n Node, visit func(Node) bool)
- type Annotation
- type Annotations
- type CPPInclude
- type Const
- type ConstMapEntry
- type ConstValue
- type ConstValueKind
- type Document
- func (d *Document) CPPIncludes() []*CPPInclude
- func (d *Document) Consts() []*Const
- func (d *Document) Contains(n Node, pos Position) bool
- func (d *Document) EachAnnotation(fn func(*Annotations))
- func (d *Document) EachStructuredAnnotation(fn func(*StructuredAnnotation))
- func (d *Document) Enums() []*Enum
- func (d *Document) Exceptions() []*Struct
- func (d *Document) Includes() []*Include
- func (d *Document) Namespaces() []*Namespace
- func (d *Document) Range(n Node) (start, end Position)
- func (d *Document) SearchNodePathByPosition(pos Position) []Node
- func (d *Document) Services() []*Service
- func (d *Document) Structs() []*Struct
- func (d *Document) TokEnd() int
- func (d *Document) TokStart() int
- func (d *Document) TokenEndPosition(i int) Position
- func (d *Document) TokenIndex(t *Token) int
- func (d *Document) TokenPosition(i int) Position
- func (d *Document) TokenRange(t *Token) (start, end Position)
- func (d *Document) Typedefs() []*Typedef
- func (d *Document) Unions() []*Struct
- func (d *Document) WalkFieldLists(fn func(fields []*Field, kind FieldListKind))
- type Enum
- type EnumValue
- type Error
- type Field
- type FieldListKind
- type FieldType
- type FieldTypeKind
- type Function
- type Identifier
- type Include
- type Namespace
- type Node
- type Position
- type Service
- type Severity
- type Struct
- type StructKind
- type StructuredAnnotation
- type Throws
- type Token
- type TokenKind
- type Typedef
Constants ¶
This section is empty.
Variables ¶
var InvalidPosition = Position{}
InvalidPosition is the zero Position.
Functions ¶
func Dump ¶
Dump renders a parsed document as a debug tree: every token (comments included) with its kind, position, and blank-line count, followed by the node spans. Deterministic and stable for a given input, so dumps can be diffed across versions.
func IsComment ¶
IsComment reports whether the token kind is a comment trivia token. The parser skips comment tokens when matching the grammar, so they only appear between real tokens in the stream.
func IsKeyword ¶ added in v0.1.6
IsKeyword reports whether the token kind is one of the reserved words.
func IsTypeKeyword ¶ added in v0.1.6
IsTypeKeyword reports whether the token kind is a base or container type keyword, which always appears in a type position.
func Lex ¶
Lex tokenizes src into a flat token stream. The final token is always TokenEOF. Lexical errors are collected and returned alongside the tokens; lexing continues past errors so the parser can still recover.
func NextReal ¶ added in v0.1.6
NextReal returns the index of the next real (non-comment) token at or after i.
func Parse ¶
Parse lexes and parses src into a Document. It always returns a document (possibly partial) and every lexical and parse error that was found. Documents with errors must not be formatted.
func ParseTokens ¶
ParseTokens parses an already-lexed token stream. The LSP can reuse this to reparse a file from its cached tokens without re-lexing.
func PrevReal ¶ added in v0.1.6
PrevReal returns the index of the previous real (non-comment) token strictly before i, or -1. Comments are stream tokens but never participate in the grammar, so every adjacency lookup skips them.
func Walk ¶ added in v0.1.7
Walk visits n and its descendants depth-first in source order, preorder: n first, then each child's subtree. visit receives every node and returns whether to descend into that node's children. Use it to collect facts across a document instead of writing another hand-rolled recursion; it inherits nodeChildren's exhaustiveness, so new node kinds cannot be silently skipped.
Types ¶
type Annotation ¶
type Annotation struct {
Name *Identifier
Value *Token // the optional string literal; nil means bare
Sep TokenKind
// contains filtered or unexported fields
}
Annotation is one entry of an annotation group: <name> [= "value"]. A bare <name> is legal and means an implicit value of "1".
type Annotations ¶
type Annotations struct {
Items []*Annotation
// contains filtered or unexported fields
}
Annotations is a parenthesized annotation group: ( <annotations> ).
type CPPInclude ¶
type CPPInclude struct {
Path *Token // the path string literal
// contains filtered or unexported fields
}
CPPInclude is a C++ include: cpp_include "path".
type Const ¶
type Const struct {
Type *FieldType
Name *Identifier
Value *ConstValue
Sep TokenKind // trailing , or ; (0 if none)
Structured []*StructuredAnnotation
// contains filtered or unexported fields
}
Const declares a constant: const <type> <name> = <value>.
type ConstMapEntry ¶
type ConstMapEntry struct {
Key *ConstValue
Value *ConstValue
}
ConstMapEntry is one key/value pair of a map constant. Order is preserved.
type ConstValue ¶
type ConstValue struct {
Kind ConstValueKind
Text string // raw source text of scalar values
List []*ConstValue
Map []ConstMapEntry
// contains filtered or unexported fields
}
ConstValue is a constant value: a scalar (int, double, string, identifier, true/false), a list, or a map. Scalar values keep their raw source text in Text (e.g. "0x1F", "-1.5e-3") so formatting is lossless.
type ConstValueKind ¶
type ConstValueKind uint8
ConstValueKind discriminates the forms a constant value can take.
const ( ValueInt ConstValueKind = iota ValueDouble ValueString ValueIdent ValueList ValueMap )
type Document ¶
Document is a parsed thrift file: its token stream and the top-level nodes in source order. It implements Node with a range spanning the whole file.
func (*Document) CPPIncludes ¶
func (d *Document) CPPIncludes() []*CPPInclude
CPPIncludes returns the cpp_include headers in source order.
func (*Document) EachAnnotation ¶ added in v0.1.7
func (d *Document) EachAnnotation(fn func(*Annotations))
EachAnnotation visits every annotation group attached to any node of the document: namespaces, typedefs, structs, fields, enum values, services, functions, arguments, throws members, and container types.
Annotation groups are not tree children — they decorate nodes the way trivia decorates tokens — so a plain tree walk does not reach them; consumers must come here instead.
func (*Document) EachStructuredAnnotation ¶ added in v0.1.8
func (d *Document) EachStructuredAnnotation(fn func(*StructuredAnnotation))
EachStructuredAnnotation visits every structured annotation in the document in source order: the ones leading definitions, functions and their arguments and throws entries, and struct fields. Unlike legacy annotation groups, structured annotations are tree nodes and are reached by Walk; this accessor exists for consumers that want the flat list.
func (*Document) Exceptions ¶
Exceptions returns the exception declarations in source order.
func (*Document) Namespaces ¶
Namespaces returns the namespace headers in source order.
func (*Document) Range ¶
Range returns the span of a node: from the start of its first token to the end of its last token.
func (*Document) SearchNodePathByPosition ¶
SearchNodePathByPosition returns the path of nodes containing pos, from the document root to the innermost node. The LSP uses the deepest node to decide what the cursor is on and its ancestors for context.
func (*Document) TokenEndPosition ¶
TokenEndPosition returns the position immediately after token i. Tokens never span lines, so the end position is on the same line.
func (*Document) TokenIndex ¶
TokenIndex returns the index of a token pointer in the document's token stream. Token pointers are stable: the parser stores pointers into the document's token slice. It returns 0 for nil or foreign tokens.
func (*Document) TokenPosition ¶
TokenPosition returns the start position of token i.
func (*Document) TokenRange ¶
TokenRange returns the span of a token.
func (*Document) WalkFieldLists ¶ added in v0.1.2
func (d *Document) WalkFieldLists(fn func(fields []*Field, kind FieldListKind))
WalkFieldLists visits the field lists of every struct, union, exception, service function argument, and throws clause in document order.
type Enum ¶
type Enum struct {
Name *Identifier
Values []*EnumValue
Annotations *Annotations
Structured []*StructuredAnnotation
// contains filtered or unexported fields
}
Enum declares an enum: enum <name> { <values> }.
type EnumValue ¶
type EnumValue struct {
Name *Identifier
Value *Token // the optional int token; nil means auto-incremented
Sep TokenKind
Annotations *Annotations
// contains filtered or unexported fields
}
EnumValue is one enum member: <name> [= <int>].
type Field ¶
type Field struct {
FieldID *Token // the optional id int token; nil means implicit
Req *Token // the required/optional keyword token; nil means unqualified
Type *FieldType
Reference bool // & prefix (field reference)
Name *Identifier
Value *ConstValue
Sep TokenKind
Annotations *Annotations
Structured []*StructuredAnnotation
// contains filtered or unexported fields
}
Field is a struct field, function argument, or throws entry: [<id>:] [required|optional] <type> [&] <name> [= <value>].
type FieldListKind ¶ added in v0.1.2
type FieldListKind uint8
FieldListKind identifies the declaration a field list belongs to.
const ( StructFields FieldListKind = iota UnionFields ExceptionFields FunctionArgs // service function arguments ThrowsFields // a service function's throws clause )
type FieldType ¶
type FieldType struct {
Kind FieldTypeKind
Base TokenKind // TypeBase: the base type keyword
Ident *Identifier
KeyType *FieldType // TypeMap: key type
ValueType *FieldType // TypeMap/TypeList/TypeSet: value or element type
CPPType *Token // optional cpp_type "..." literal on containers
Annotations *Annotations
// contains filtered or unexported fields
}
FieldType is a type reference: a base type keyword, a named identifier, or a container (map/list/set). Base and container types may carry annotations; identifier types may not, matching the compiler.
type FieldTypeKind ¶
type FieldTypeKind uint8
FieldTypeKind discriminates the forms a type reference can take.
const ( TypeBase FieldTypeKind = iota TypeIdent TypeMap TypeList TypeSet )
type Function ¶
type Function struct {
Oneway *Token // oneway/async keyword, nil for synchronous functions
Type *FieldType
Void *Token // the void keyword; Type is nil when Void is set
Name *Identifier
Args []*Field
Throws *Throws
Sep TokenKind
Annotations *Annotations
Structured []*StructuredAnnotation
// contains filtered or unexported fields
}
Function is a service method: [oneway] <type|void> <name> ( <args> ) [throws ( <exceptions> )].
type Identifier ¶
type Identifier struct {
Text string
// contains filtered or unexported fields
}
Identifier is a name: a plain identifier token.
type Include ¶
type Include struct {
Path *Token // the path string literal
// contains filtered or unexported fields
}
Include is a thrift include: include "path".
type Namespace ¶
type Namespace struct {
Scope *Token // scope identifier or the '*' token
Name *Identifier
Annotations *Annotations
Structured []*StructuredAnnotation
// contains filtered or unexported fields
}
Namespace declares a namespace: namespace <scope> <name>.
type Node ¶
Node is any AST node. All nodes embed nodeBase, which provides the token range and seals the interface to this package.
type Position ¶
Position is a source position: 1-based line, 1-based rune column, and 0-based byte offset.
type Service ¶
type Service struct {
Name *Identifier
Extends *Identifier // optional base service
Functions []*Function
Annotations *Annotations
Structured []*StructuredAnnotation
// contains filtered or unexported fields
}
Service declares a service: service <name> [extends <base>] { <functions> }.
type Struct ¶
type Struct struct {
Kind StructKind
Name *Identifier
Fields []*Field
Annotations *Annotations
Structured []*StructuredAnnotation
// contains filtered or unexported fields
}
Struct is a struct, union, or exception: <kind> <name> { <fields> }.
type StructKind ¶
type StructKind = TokenKind
StructKind distinguishes struct, union, and exception declarations. It is the token kind of the leading keyword.
const ( StructDecl StructKind = TokenStruct UnionDecl StructKind = TokenUnion ExceptionDecl StructKind = TokenException )
type StructuredAnnotation ¶ added in v0.1.8
type StructuredAnnotation struct {
Name *Identifier
Value *ConstValue
// contains filtered or unexported fields
}
StructuredAnnotation is a Java-style structured annotation:
@Name <value>
where the value is a const map, a const list, or a parenthesized scalar constant, and is mandatory — matching the upfluence compiler, where a bare @Name is a syntax error. The name refers to a declared type; the value is a constant of that type.
type Throws ¶
type Throws struct {
Fields []*Field
// contains filtered or unexported fields
}
Throws is a function's throws clause: throws ( <fields> ).
type Token ¶
type Token struct {
Kind TokenKind
Text string // exact source text
Offset int // byte offset of the first character
Line int // 1-based line of the first character
Col int // 1-based rune column of the first character
// BlankLinesBefore is the number of empty lines between the previous
// stream entry (token or comment) and this one.
BlankLinesBefore int
}
Token is a single lexical token, comments included.
type TokenKind ¶
type TokenKind uint8
TokenKind identifies the lexical class of a Token.
const ( TokenInvalid TokenKind = iota TokenEOF TokenIdentifier TokenIntConstant TokenDoubleConstant TokenStringLiteral // Keywords. TokenInclude TokenCPPInclude TokenCPPType TokenNamespace TokenStruct TokenUnion TokenException TokenService TokenEnum TokenConst TokenTypedef TokenOneway TokenAsync // deprecated alias for oneway TokenThrows TokenExtends TokenRequired TokenOptional TokenVoid TokenBool TokenByte // deprecated, accepted with a warning TokenI8 TokenI16 TokenI32 TokenI64 TokenDouble TokenString TokenBinary TokenSlist // no longer supported by the compiler, accepted for old files TokenUUID TokenMap TokenList TokenSet TokenTrue TokenFalse // Punctuation. TokenLBrace TokenRBrace TokenLParen TokenRParen TokenLBracket TokenRBracket TokenLt TokenGt TokenComma TokenSemicolon TokenColon TokenEqual TokenStar TokenAmp // Comment trivia. A line comment consumes the rest of its source // line, so whatever follows always starts a fresh line. TokenLineComment TokenBlockComment TokenDocComment TokenAt // @, structured annotation marker )
type Typedef ¶
type Typedef struct {
Type *FieldType
Name *Identifier
Sep TokenKind
Annotations *Annotations
Structured []*StructuredAnnotation
// contains filtered or unexported fields
}
Typedef declares a type alias: typedef <type> <name>.
Source Files