parser

package
v0.17.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 6, 2026 License: Apache-2.0, Apache-2.0 Imports: 15 Imported by: 0

README

Parser

A MySQL-compatible SQL parser, used by Spirit to parse DDL and understand schemas.

This package is a hard fork of the TiDB parser (github.com/pingcap/tidb/pkg/parser), taken from upstream master as of May 2026 (via block/tidb@e528fd979fc8, which added spatial type/index support). We are grateful to PingCAP and the TiDB contributors for building and maintaining an excellent MySQL-compatible parser for over a decade — and for accepting many of our compatibility fixes upstream over the years. The fork retains their Apache-2.0 license and copyright headers.

Why fork?

Spirit only needs to parse the MySQL dialect, and only cares about the syntax a schema-change tool encounters. TiDB's parser necessarily carries TiDB-specific extensions (TiDB system functions, hints, ADMIN/BRIE statements, syntax bound to TiKV/TiFlash concepts) and some MariaDB syntax. Forking lets us:

  • strip everything that is not MySQL, so what remains is auditable against the MySQL manual;
  • add the functionality we need quickly, without waiting on an upstream release cadence;
  • keep Spirit's dependency graph small (the parser no longer drags in pingcap/* modules, zap, or their transitive dependencies).

What changed relative to upstream

  • MySQL-only surface. TiDB-specific statements, keywords, system functions, optimizer hints, and MariaDB syntax (e.g. SYSTEM_TIME partitioning, ILIKE, FLUSH CLIENT_ERRORS_SUMMARY) are removed. The reserved-word set now matches MySQL 8.0 (see Testing).
  • Smaller charset catalog. Only the encodings the lexer actually transforms remain (utf8/utf8mb4, ascii, latin1, binary); other charset names are still recognized in DDL but are not transcoded. Custom charset registration was removed.
  • Modern Go, stdlib errors. The terror/pingcap/errors machinery was replaced with plain wrapped errors (errors.Is/As work as expected); zap logging was removed; the code passes this repo's golangci-lint configuration.
  • Dead weight removed. The legacy Format(io.Writer) pretty-printer, the driver indirection, keyword-listing generators, and APIs unused by Spirit are deleted. deadcode -test ./... reports only interface-conformance markers.
  • Upstream fixes ported. MySQL-compatibility fixes that landed in TiDB after the fork base are ported when relevant (e.g. the parser depth DoS guard, INSERT ... AS row_alias, dual-password syntax, SET_VAR decimal hints, and GROUP_CONCAT separator charset handling), as is the opt-in format.RestoreSkipRedundantParentheses restore flag, which lets expression Restore drop parentheses that MySQL's precedence and associativity rules make unnecessary (a + (b * c) restores as a + b * c; a - (b - c) keeps its parentheses). Spirit uses it to canonicalize CHECK-constraint and generated-column expressions.
  • Fixes beyond upstream. Parenthesized default values keep their parentheses through a parse/restore round trip (DEFAULT ('{}') is an ast.ParenthesesExpr): MySQL 8.0.13+ treats DEFAULT ('{}') and DEFAULT '{}' as different DDL, and BLOB/TEXT/JSON/GEOMETRY columns only accept the parenthesized form. The upstream parser still restores both to the bare form (pingcap/tidb#57768). Keyword-named functions also parse in DEFAULT expressions (DEFAULT (point(0,0)) — the MySQL manual's own expression-default example; upstream special-cases only REPLACE), and the spatial constructors (linestring(), polygon(), multipoint(), ...) parse as function calls in every expression context — as keyword tokens they previously only worked as column types (block/spirit#1128).

The AST (ast package), format restore machinery, charset, mysql constants, opcode, and types packages keep their upstream shapes, so code written against the TiDB parser API generally ports with an import change.

Usage

import (
    "github.com/block/spirit/pkg/parser"
    "github.com/block/spirit/pkg/parser/ast"
)

p := parser.New()
stmts, warns, err := p.ParseSQL("ALTER TABLE t1 ADD COLUMN b INT")
_ = warns
if err != nil {
    // handle parse error
}
alter := stmts[0].(*ast.AlterTableStmt)

Regenerating the parser

parser.go and hintparser.go are generated from parser.y and hintparser.y by goyacc. After editing a .y file:

cd pkg/parser
make parser

The build fails if the grammar introduces shift/reduce or reduce/reduce conflicts. Generated files are checked in; the parser-regen CI job deletes them, regenerates from the grammar, and fails on any diff, so a .y edit cannot ship without its regenerated output.

Testing against real MySQL

Most tests run offline. One additional suite compares the grammar's reserved-word set against a live MySQL server:

MYSQL_DSN="user:pass@tcp(127.0.0.1:3306)/" \
  go test -tags reserved_words_test -run TestCompareReservedWordsWithMySQL ./pkg/parser

License

Apache License 2.0, same as the upstream TiDB parser. See the LICENSE file at the repository root and the per-file copyright headers.

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	ErrWarnOptimizerHintUnsupportedHint = mysql.NewStdErr("parser", mysql.ErrWarnOptimizerHintUnsupportedHint)
	ErrWarnOptimizerHintInvalidToken    = mysql.NewStdErr("parser", mysql.ErrWarnOptimizerHintInvalidToken)
	ErrWarnOptimizerHintParseError      = mysql.NewStdErr("parser", mysql.ErrWarnOptimizerHintParseError)
	ErrWarnOptimizerHintInvalidInteger  = mysql.NewStdErr("parser", mysql.ErrWarnOptimizerHintInvalidInteger)
	ErrWarnOptimizerHintWrongPos        = mysql.NewStdErr("parser", mysql.ErrWarnOptimizerHintWrongPos)
)
View Source 
var (
	// ErrSyntax returns for sql syntax error.
	ErrSyntax = mysql.NewStdErr("parser", mysql.ErrSyntax)
	// ErrParse returns for sql parse error.
	ErrParse = mysql.NewStdErr("parser", mysql.ErrParse)
	// ErrUnknownCharacterSet returns for no character set found error.
	ErrUnknownCharacterSet = mysql.NewStdErr("parser", mysql.ErrUnknownCharacterSet)
	// ErrInvalidYearColumnLength returns for illegal column length for year type.
	ErrInvalidYearColumnLength = mysql.NewStdErr("parser", mysql.ErrInvalidYearColumnLength)
	// ErrWrongArguments returns for illegal argument.
	ErrWrongArguments = mysql.NewStdErr("parser", mysql.ErrWrongArguments)
	// ErrWrongFieldTerminators returns for illegal field terminators.
	ErrWrongFieldTerminators = mysql.NewStdErr("parser", mysql.ErrWrongFieldTerminators)
	// ErrTooBigDisplayWidth returns for data display width exceed limit .
	ErrTooBigDisplayWidth = mysql.NewStdErr("parser", mysql.ErrTooBigDisplaywidth)
	// ErrTooBigPrecision returns for data precision exceed limit.
	ErrTooBigPrecision = mysql.NewStdErr("parser", mysql.ErrTooBigPrecision)
	// ErrUnknownAlterLock returns for no alter lock type found error.
	ErrUnknownAlterLock = mysql.NewStdErr("parser", mysql.ErrUnknownAlterLock)
	// ErrUnknownAlterAlgorithm returns for no alter algorithm found error.
	ErrUnknownAlterAlgorithm = mysql.NewStdErr("parser", mysql.ErrUnknownAlterAlgorithm)
	// ErrWrongValue returns for wrong value
	ErrWrongValue = mysql.NewStdErr("parser", mysql.ErrWrongValue)
	// ErrWarnDeprecatedSyntax return when the syntax was deprecated
	ErrWarnDeprecatedSyntax = mysql.NewStdErr("parser", mysql.ErrWarnDeprecatedSyntax)
	// ErrWarnDeprecatedSyntaxNoReplacement return when the syntax was deprecated and there is no replacement.
	ErrWarnDeprecatedSyntaxNoReplacement = mysql.NewStdErr("parser", mysql.ErrWarnDeprecatedSyntaxNoReplacement)
	// ErrWrongUsage returns for incorrect usages.
	ErrWrongUsage = mysql.NewStdErr("parser", mysql.ErrWrongUsage)
	// ErrWrongDBName returns for incorrect DB name.
	ErrWrongDBName = mysql.NewStdErr("parser", mysql.ErrWrongDBName)
	// ErrDataOutOfRange returns for incorrect range.
	ErrDataOutOfRange = mysql.NewStdErr("parser", mysql.ErrDataOutOfRange)
)

Functions

func ParseErrorWith 

func ParseErrorWith(errstr string, lineno int) error

ParseErrorWith returns "You have a syntax error near..." error message compatible with mysql.

func ParseHint 

func ParseHint(input string, sqlMode mysql.SQLMode, initPos Pos) ([]*ast.TableOptimizerHint, []error)

ParseHint parses an optimizer hint (the interior of `/*+ ... */`).

Types

type CharsetConnection 

type CharsetConnection string

CharsetConnection is used for literals specified without a character set.

func (CharsetConnection) ApplyOn 

func (c CharsetConnection) ApplyOn(p *Parser) error

ApplyOn implements ParseParam interface.

type CollationConnection 

type CollationConnection string

CollationConnection is used for literals specified without a collation.

func (CollationConnection) ApplyOn 

func (c CollationConnection) ApplyOn(p *Parser) error

ApplyOn implements ParseParam interface.

type ParseParam 

type ParseParam interface {
	ApplyOn(*Parser) error
}

ParseParam represents the parameter of parsing.

type Parser 

type Parser struct {
	// contains filtered or unexported fields
}

Parser represents a parser instance. Some temporary objects are stored in it to reduce object allocation during Parse function.

func New 

func New() *Parser

New returns a Parser object with default SQL mode.

func (*Parser) EnableWindowFunc 

func (parser *Parser) EnableWindowFunc(val bool)

EnableWindowFunc controls whether the parser to parse syntax related with window function.

func (*Parser) Parse 

func (parser *Parser) Parse(sql, charset, collation string) (stmt []ast.StmtNode, warns []error, err error)

Parse parses a query string to raw ast.StmtNode. If charset or collation is "", default charset and collation will be used.

func (*Parser) ParseOneStmt 

func (parser *Parser) ParseOneStmt(sql, charset, collation string) (ast.StmtNode, error)

ParseOneStmt parses a query and returns an ast.StmtNode. The query must have one statement, otherwise ErrSyntax is returned.

func (*Parser) ParseSQL 

func (parser *Parser) ParseSQL(sql string, params ...ParseParam) (stmt []ast.StmtNode, warns []error, err error)

ParseSQL parses a query string to raw ast.StmtNode.

func (*Parser) Reset 

func (parser *Parser) Reset()

Reset resets the parser.

func (*Parser) SetParserConfig 

func (parser *Parser) SetParserConfig(config ParserConfig)

SetParserConfig sets the parser config.

func (*Parser) SetSQLMode 

func (parser *Parser) SetSQLMode(mode mysql.SQLMode)

SetSQLMode sets the SQL mode for parser.

func (*Parser) SetStrictDoubleTypeCheck 

func (parser *Parser) SetStrictDoubleTypeCheck(val bool)

SetStrictDoubleTypeCheck enables/disables strict double type check.

type ParserConfig 

type ParserConfig struct {
	EnableWindowFunction        bool
	EnableStrictDoubleTypeCheck bool
	SkipPositionRecording       bool
}

ParserConfig is the parser config.

type Pos 

type Pos struct {
	Line   int
	Col    int
	Offset int
}

Pos represents the position of a token.

type Scanner 

type Scanner struct {
	// contains filtered or unexported fields
}

Scanner implements the yyLexer interface.

func NewScanner 

func NewScanner(s string) *Scanner

NewScanner returns a new scanner object.

func (*Scanner) AppendError 

func (s *Scanner) AppendError(err error)

AppendError sets error into scanner. Scanner satisfies yyLexer interface which need this function.

func (*Scanner) AppendWarn 

func (s *Scanner) AppendWarn(err error)

AppendWarn sets warning into scanner.

func (*Scanner) EnableWindowFunc 

func (s *Scanner) EnableWindowFunc(val bool)

EnableWindowFunc controls whether the scanner recognize the keywords of window function.

func (*Scanner) Errorf 

func (s *Scanner) Errorf(format string, a ...interface{}) (err error)

Errorf tells scanner something is wrong. Scanner satisfies yyLexer interface which need this function.

func (*Scanner) Errors 

func (s *Scanner) Errors() (warns []error, errs []error)

Errors returns the errors and warns during a scan.

func (*Scanner) GetSQLMode 

func (s *Scanner) GetSQLMode() mysql.SQLMode

GetSQLMode return the SQL mode of scanner.

func (*Scanner) InheritScanner 

func (s *Scanner) InheritScanner(sql string) *Scanner

InheritScanner returns a new scanner object which inherits configurations from the parent scanner.

func (*Scanner) Lex 

func (s *Scanner) Lex(v *yySymType) int

Lex returns a token and store the token value in v. Scanner satisfies yyLexer interface. 0 and invalid are special token id this function would return: return 0 tells parser that scanner meets EOF, return invalid tells parser that scanner meets illegal character.

func (*Scanner) LexLiteral 

func (s *Scanner) LexLiteral() interface{}

LexLiteral returns the value of the converted literal

func (*Scanner) SetSQLMode 

func (s *Scanner) SetSQLMode(mode mysql.SQLMode)

SetSQLMode sets the SQL mode for scanner.

Source Files

View all Source files

Directories

Path Synopsis
Package ast is the abstract syntax tree parsed from a SQL statement by parser.
 Package ast is the abstract syntax tree parsed from a SQL statement by parser.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.