token

package

v0.2.0 Latest Latest Go to latest Published: Aug 16, 2022 License: Apache-2.0 Imports: 5 Imported by: 0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/cloudwego/thriftgo

Links

Open Source Insights

Documentation ¶

Index ¶

Constants
type Pos
type Span
type Tok
- func (t Tok) String() string
type Token
type Tokenizer
- func NewTokenizer(src io.Reader) *Tokenizer

Constants ¶

View Source

const LexicalError = Tok(-1)

LexicalError .

Variables ¶

This section is empty.

Functions ¶

This section is empty.

Types ¶

type Pos ¶

type Pos struct {
	Line   int // start from 1
	Offset int // start from 1
}

Pos represents a position in the source file.

type Span ¶

type Span struct {
	Beg int
	End int
}

Span represents a segment in a stream.

type Tok ¶

type Tok int

Tok .

const (
	EOF           Tok = iota
	BlockComment      // /\*(?:\*[^/]|[^*])*\*/
	LineComment       // //[^\n]*
	UnixComment       // #[^\n]*
	Whitespaces       //  \t\v\r
	NewLine           // \r?\n
	Bool              // bool
	Byte              // byte
	I8                // i8
	I16               // i16
	I32               // i32
	I64               // i64
	Double            // double
	String            // string
	Binary            // binary
	Const             // const
	Oneway            // oneway
	Typedef           // typedef
	Map               // map
	Set               // set
	List              // list
	Void              // void
	Throws            // throws
	Exception         // exception
	Extends           // extends
	Required          // required
	Optional          // optional
	Service           // service
	Struct            // struct
	Union             // union
	Enum              // enum
	Include           // include
	CppInclude        // cpp_include
	Namespace         // namespace
	Asterisk          // *
	LBracket          // [
	RBracket          // ]
	LBrace            // {
	RBrace            // }
	LParenthesis      // (
	RParenthesis      // )
	LChevron          // <
	RChevron          // >
	Equal             // =
	Comma             // ,
	Colon             // :
	Semicolon         // ;
	StringLiteral     // '(?:\\'|[^'])*'|"(?:\\"|[^"])*"
	Identifier        // [_a-zA-Z][_a-zA-Z0-9]*(?:\.[_a-zA-Z][_a-zA-Z0-9]*)*
	IntLiteral        // [-+]?(?:0x[0-9a-fA-F]+|0o[0-7]+|0|[1-9][0-9]*)
	FloatLiteral      //

)

Toks .

func (Tok) String ¶

func (t Tok) String() string

type Token ¶

type Token struct {
	Tok
	Span
	Data []byte
}

Token is the set of lexical tokens of the thrift IDL.

func (*Token) AsFloat ¶

func (t *Token) AsFloat() float64

AsFloat returns the data of the token as a float64.

func (*Token) AsInt ¶

func (t *Token) AsInt() int64

AsInt returns the data of the token as a int64.

func (*Token) AsString ¶

func (t *Token) AsString() string

AsString returns the data of the token as a string.

func (*Token) String ¶

func (t *Token) String() string

func (*Token) Unquote ¶

func (t *Token) Unquote() string

Unquote interprets the token's data as a quote string and returns the string it quotes.

type Tokenizer ¶

type Tokenizer struct {
	Span
	// contains filtered or unexported fields
}

Tokenizer is a lexical analyzer for thrift IDL.

func NewTokenizer ¶

func NewTokenizer(src io.Reader) *Tokenizer

NewTokenizer returns a Tokenizer over the given source.

func (*Tokenizer) LineSpan ¶

func (p *Tokenizer) LineSpan(s Span) Span

LineSpan returns the first and last line that the given span exists.

func (*Tokenizer) Next ¶

func (p *Tokenizer) Next() Token

Next returns the next token from the source stream.

func (*Tokenizer) Pos2Pos ¶

func (p *Tokenizer) Pos2Pos(pos int) Pos

Pos2Pos converts an index in the byte stream to a Pos.

Source Files ¶

View all Source files

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL