Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Keyword ¶
type Keyword struct {
Word string // The actual keyword text
Reserved bool // Whether this is a reserved keyword
}
Keyword represents a lexical keyword with its properties
type Location ¶
Location represents a position in the source code using 1-based indexing. Both Line and Column are 1-based to match SQL standards.
type Token ¶
type Token struct {
Type TokenType
Value string
Word *Word // For TokenTypeWord
Long bool // For TokenTypeNumber to indicate if it's a long number
Quote rune // For quoted strings and identifiers
}
Token represents a SQL token with its value and metadata
type TokenType ¶
type TokenType int
TokenType represents the type of a SQL token
const ( // Basic token types TokenTypeEOF TokenType = iota TokenTypeWord TokenTypeNumber TokenTypeChar TokenTypeSingleQuotedString TokenTypeDoubleQuotedString TokenTypeTripleSingleQuotedString TokenTypeTripleDoubleQuotedString TokenTypeDollarQuotedString TokenTypeByteStringLiteral TokenTypeNationalStringLiteral TokenTypeEscapedStringLiteral TokenTypeUnicodeStringLiteral TokenTypeHexStringLiteral TokenTypeWhitespace // Operators and punctuation TokenTypeComma TokenTypeDoubleEq TokenTypeEq TokenTypeNeq TokenTypeLt TokenTypeGt TokenTypeLtEq TokenTypeGtEq TokenTypeSpaceship TokenTypePlus TokenTypeMinus TokenTypeMul TokenTypeDiv TokenTypeDuckIntDiv TokenTypeMod TokenTypeStringConcat TokenTypeLParen TokenTypeRParen TokenTypePeriod TokenTypeColon TokenTypeDoubleColon TokenTypeAssignment TokenTypeSemicolon TokenTypeBackslash TokenTypeLBracket TokenTypeRBracket TokenTypeAmpersand // Keywords TokenTypeKeyword TokenTypeSelect TokenTypeJoin TokenTypeInner TokenTypeLeft TokenTypeRight TokenTypeOuter TokenTypeGroup TokenTypeHaving TokenTypeWhere TokenTypeOrder TokenTypeLimit TokenTypeOffset TokenTypeOn TokenTypeAnd TokenTypeLike TokenTypeAsc TokenTypeFrom TokenTypeBy TokenTypeOr TokenTypeNot TokenTypeIn TokenTypeCount TokenTypeSum TokenTypeAvg TokenTypeMin TokenTypeMax TokenTypeBetween TokenTypeIs TokenTypeNull TokenTypeTrue TokenTypeFalse TokenTypeDesc TokenTypeCase TokenTypeWhen TokenTypeThen TokenTypeElse TokenTypeEnd TokenTypeAs TokenTypeGroupBy TokenTypeOrderBy TokenTypeLeftJoin TokenTypeRightJoin TokenTypeInnerJoin TokenTypeOuterJoin TokenTypePipe TokenTypeCaret TokenTypeLBrace TokenTypeRBrace TokenTypeRArrow TokenTypeSharp TokenTypeTilde TokenTypeTildeAsterisk TokenTypeExclamationMarkTilde TokenTypeExclamationMarkTildeAsterisk TokenTypeDoubleTilde TokenTypeDoubleTildeAsterisk TokenTypeExclamationMarkDoubleTilde TokenTypeExclamationMarkDoubleTildeAsterisk TokenTypeShiftLeft TokenTypeShiftRight TokenTypeOverlap TokenTypeExclamationMark TokenTypeDoubleExclamationMark TokenTypeAtSign TokenTypeCaretAt TokenTypePGSquareRoot TokenTypePGCubeRoot TokenTypePlaceholder TokenTypeArrow TokenTypeLongArrow TokenTypeHashArrow TokenTypeHashLongArrow TokenTypeAtArrow TokenTypeArrowAt TokenTypeHashMinus TokenTypeAtQuestion TokenTypeAtAt TokenTypeQuestion TokenTypeQuestionAnd TokenTypeQuestionPipe TokenTypeCustomBinaryOperator // Additional token types referenced in tests TokenTypeString TokenTypeIdentifier TokenTypeOperator TokenTypeLeftParen TokenTypeRightParen TokenTypeDot )
These constants define the token types used in the SQL tokenizer
type TokenWithSpan ¶
TokenWithSpan represents a token with its location in the source code
func NewEOFToken ¶
func NewEOFToken(pos Location) TokenWithSpan
NewEOFToken creates a new EOF token with span
func NewTokenWithSpan ¶
func NewTokenWithSpan(tokenType TokenType, value string, start, end Location) TokenWithSpan
NewTokenWithSpan creates a new TokenWithSpan with the given type, value, and location
func TokenAtLocation ¶
func TokenAtLocation(token Token, start, end Location) TokenWithSpan
TokenAtLocation creates a new TokenWithSpan from a Token and location
func WrapToken ¶
func WrapToken(token Token) TokenWithSpan
WrapToken wraps a token with an empty location
type TokenizerError ¶
TokenizerError represents an error during tokenization
func (TokenizerError) Error ¶
func (e TokenizerError) Error() string
type Whitespace ¶
type Whitespace struct {
Type WhitespaceType
Content string // For comments
Prefix string // For single line comments
}
Whitespace represents different types of whitespace tokens
type WhitespaceType ¶
type WhitespaceType int
WhitespaceType represents the type of whitespace
const ( WhitespaceTypeSpace WhitespaceType = iota WhitespaceTypeNewline WhitespaceTypeTab WhitespaceTypeSingleLineComment WhitespaceTypeMultiLineComment )
Source Files
Click to show internal directories.
Click to hide internal directories.