tokenizer

package

v1.0.10 Latest Latest Go to latest Published: Apr 3, 2026 License: GPL-3.0 Imports: 9 Imported by: 0

Details

This section is empty.

This section is empty.

func StandardRemoveQuotedTextAndComment(statement string) (string, error)

StandardRemoveQuotedTextAndComment removes the quoted text and comment.

type Tokenizer struct {
	// contains filtered or unexported fields
}

func NewTokenizer(statement string) *Tokenizer

NewTokenizer creates a new tokenizer. Notice: we append an additional eofRune in the statement. This is a sentinel rune.

func (t *Tokenizer) SetLineForMySQLCreateTableStmt(node *tidbast.CreateTableStmt, firstLine int) error

func (t *Tokenizer) SplitPostgreSQLMultiSQL() ([]base.SingleSQL, error)

SplitPostgreSQLMultiSQL splits the statement to a string slice. We mainly considered:

comments
- style /* comments */
- style -- comments
string
- style 'string'
- style $$ string $$
identifier
- style "indentifier"

Notice:

We support PostgreSQL CREATE PROCEDURE statement with $$ $$ style, but do not support BEGIN ATOMIC ... END; style. See https://www.postgresql.org/docs/14/sql-createprocedure.html.

func (t *Tokenizer) SplitStandardMultiSQL() ([]base.SingleSQL, error)

SplitStandardMultiSQL splits the statement to a string slice. We mainly considered:

comments
- style /* comments */
- style -- comments
string
- style 'string'
identifier
- style "identifier"

The difference between PostgreSQL and Oracle is that PostgreSQL supports dollar-quoted string, but Oracle does not.

func (t *Tokenizer) SplitTiDBMultiSQL() ([]base.SingleSQL, error)

SplitTiDBMultiSQL splits the statement to a string slice.