sqltoken

package module
v0.5.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 4, 2026 License: MIT Imports: 6 Imported by: 4

README

sqltoken - break SQL strings into a token array

GoDoc Coverage

Install:

go get github.com/muir/sqltoken

Sqltoken

Sqltoken is a high-performance hand-coded tokenizer for SQL strings. It's high-performance so that it can be used in situations where it is being run on every query.

The tokenization is somewhat rough: it correctly detects comments and numbers, but it depending on the situation and the SQL variant, it may think an indentifier is a literal.

It has support of MySQL/MariaDB/Singlestore, Postgres/CockroachDB, Oracle, and SQL server.

The return value is an array of simple tokens:

type Token struct {
	Type TokenType
	Text string
}

Concatting the Text portions togehter will reconstruct the original input:

TokenizeMySQL(s).String() == s
Use cases

SQL token can be used to strip comments from SQL code. It can be used to parse out variables to substitute.

Development status

This is new code. It has lots of tests and complete coverage, but feedback from users would be valuable.

Documentation

Index

Constants

View Source 
const Semicolon = Delimiter // Deprecated: for backwards compatibility only

Variables

This section is empty.

Functions

func TokenTypeStrings added in v0.4.0 

func TokenTypeStrings() []string

TokenTypeStrings returns a slice of all String values of the enum

Types

type BeginEndWordMode added in v0.5.0 

type BeginEndWordMode int
const (
	// BeginEndContextual treats BEGIN/END as context-dependent words that may be identifiers.
	//
	// WARNING: contextual detection is heuristic, not a full SQL parser. It is good for common
	// procedural SQL, but bareword identifiers named begin/end can still be ambiguous and may be
	// misclassified in edge cases. Prefer quoted identifiers (`begin`, `end`) when possible.
	BeginEndContextual BeginEndWordMode = iota
	// BeginEndReserved treats BEGIN/END as reserved keywords unless explicitly quoted/escaped.
	BeginEndReserved
)

type Config 

type Config struct {
	// Tokenize ? as type Question (used by MySQL, SQLite)
	NoticeQuestionMark bool

	// Tokenize ?<digits> (eg: "?7") as type Question (used by SQLite)
	NoticeQuestionNumber bool

	// Tokenize $<digits> (eg "$7") as type DollarNumber (PostgreSQL, SQLite)
	NoticeDollarNumber bool

	// Tokenize :word as type ColonWord (sqlx, Oracle, SQLite)
	NoticeColonWord bool

	// Tokenize :word with unicode as ColonWord (sqlx, SQLite)
	ColonWordIncludesUnicode bool

	// Tokenize # as type comment (MySQL)
	NoticeHashComment bool

	// $q$ stuff $q$ and $$stuff$$ quoting (PostgreSQL)
	NoticeDollarQuotes bool

	// NoticeHexValues 0xa0 x'af' X'AF' (MySQL)
	NoticeHexNumbers bool

	// NoticeLiteralBackslashEscape 'escape \' quote' (MySQL by default,
	// PostgreSQL optional).
	NoticeLiteralBackslashEscape bool

	// NoticeBinaryValues 0x01 b'01' B'01' (MySQL)
	NoticeBinaryNumbers bool

	// NoticeUAmpPrefix U& utf prefix U&"\0441\043B\043E\043D" (PostgreSQL)
	NoticeUAmpPrefix bool

	// NoticeCharsetLiteral _latin1'string' (MySQL, SingleStore)
	NoticeCharsetLiteral bool

	// NoticeNationalPrefix n'string' N'string' (MySQL)
	NoticeNationalPrefix bool

	// NoticeNotionalStrings [nN]'...”...' (Oracle, SQL Server)
	NoticeNotionalStrings bool

	// NoticeEscapedStrings [eE]'...' (PostgreSQL)
	NoticeEscapedStrings bool

	// NoticeDelimitedStrings [nN]?[qQ]'DELIM .... DELIM' (Oracle)
	NoticeDelimitedStrings bool

	// NoticeTypedNumbers nn.nnEnn[fFdD] (Oracle)
	NoticeTypedNumbers bool

	// NoticeMoneyConstants $10 $10.32 (SQL Server)
	NoticeMoneyConstants bool

	// NoticeAtWord @foo (SQL Server, SQLite)
	NoticeAtWord bool

	// NoticeAtIdentifiers _baz @fo$o @@b#ar #foo ##b@ar (SQL Server)
	NoticeIdentifiers bool

	// SeparatePunctuation prevents merging successive punctuation into a single token
	SeparatePunctuation bool

	// NoticeDelimiterStatement turns on recognition of custom statement delimiters
	// This is not 100%: the delimiter statement is always recognized after a newline even
	// if that's in the middle of a block that the client knows isn't the start of a
	// statement
	NoticeDelimiterStatement bool

	// NoticeBeginEndBlock tracks BEGIN/END block nesting for stored procedures,
	// functions, and triggers. When inside a block, semicolons are treated as
	// punctuation rather than statement delimiters. This is useful when DELIMITER
	// is not available (e.g., when sending SQL directly to database/sql drivers).
	// This option is mutually exclusive with NoticeDelimiterStatement in practice:
	// use NoticeDelimiterStatement for parsing SQL files that use DELIMITER syntax,
	// use NoticeBeginEndBlock for SQL going directly to drivers.
	NoticeBeginEndBlock bool

	// BeginEndWordMode controls whether BEGIN/END are interpreted contextually
	// (e.g. MySQL/MariaDB) or as reserved keywords (e.g. SingleStore).
	BeginEndWordMode BeginEndWordMode
}

Config specifies the behavior of Tokenize as relates to behavior that differs between SQL implementations

func MySQLAPIConfig added in v0.5.0 

func MySQLAPIConfig() Config

MySQLAPIConfig returns a parsing configuration that is appropriate for parsing MySQL and MariaDB SQL sent through driver APIs. This enables BEGIN/END block tracking with contextual begin/end detection.

WARNING: contextual begin/end handling is heuristic (not a full parser). If your SQL uses bareword identifiers named begin/end, quote them for predictable tokenization in all cases.

func MySQLConfig 

func MySQLConfig() Config

MySQLConfig returns a parsing configuration that is appropriate for parsing MySQL and MariaDB SQL. This includes support for DELIMITER and is compatible with the mysql client.

func OracleConfig 

func OracleConfig() Config

OracleConfig returns a parsing configuration that is appropriate for parsing Oracle's SQL

func PostgreSQLConfig 

func PostgreSQLConfig() Config

PostgreSQLConfig returns a parsing configuration that is appropriate for parsing PostgreSQL and CockroachDB SQL.

func SQLServerConfig 

func SQLServerConfig() Config

SQLServerConfig returns a parsing configuration that is appropriate for parsing SQLServer's SQL

func SQLiteConfig added in v0.5.0 

func SQLiteConfig() Config

SQLiteConfig returns a parsing configuration that is appropriate for SQLite SQL.

func SingleStoreAPIConfig added in v0.5.0 

func SingleStoreAPIConfig() Config

SingleStoreAPIConfig returns a parsing configuration that is appropriate for parsing SingleStore SQL sent through driver APIs. This enables BEGIN/END block tracking (instead of DELIMITER statement handling).

func SingleStoreConfig added in v0.4.0 

func SingleStoreConfig() Config

SingleStoreConfig returns a parsing configuration that is appropriate for parsing SingleStore SQL. This includes support for DELIMITER and is compatible with the mysql/singlestore client.

func (Config) WithBeginEndWordMode added in v0.5.0 

func (c Config) WithBeginEndWordMode(mode BeginEndWordMode) Config

WithBeginEndWordMode sets how BEGIN/END words are interpreted while block tracking is enabled.

For BeginEndContextual, use quoted identifiers if you have tables/columns/variables named begin/end and need deterministic behavior in all edge cases.

func (Config) WithColonWordIncludesUnicode added in v0.1.0 

func (c Config) WithColonWordIncludesUnicode() Config

WithColonWordIncludesUnicode enables unicode name parsing at a small performance cost

func (Config) WithNoticeAtWord added in v0.1.0 

func (c Config) WithNoticeAtWord() Config

WithNoticeAtWord enables parsing for '@foo' (SQL Server) using the AtWord token

func (Config) WithNoticeBeginEndBlock added in v0.5.0 

func (c Config) WithNoticeBeginEndBlock() Config

WithNoticeBeginEndBlock enables tracking of BEGIN/END blocks for stored procedures, functions, and triggers. Semicolons inside blocks are treated as punctuation rather than statement delimiters. This is mutually exclusive with NoticeDelimiterStatement - enabling this disables DELIMITER recognition.

func (Config) WithNoticeBinaryNumbers added in v0.1.0 

func (c Config) WithNoticeBinaryNumbers() Config

WithNoticeBinaryNumbers enables quoted binary number parsing (b'01') using the BinaryNumber token (MySQL)

func (Config) WithNoticeCharsetLiteral added in v0.1.0 

func (c Config) WithNoticeCharsetLiteral() Config

WithNoticeCharsetLiteral enables charset literal parsing (_latin1'string') using the Literal token (MySQL, SingleStore)

func (Config) WithNoticeColonWord added in v0.1.0 

func (c Config) WithNoticeColonWord() Config

WithNoticeColonWord enables parsing for named parameters using the ColonWord token

func (Config) WithNoticeDelimitedStrings added in v0.4.0 

func (c Config) WithNoticeDelimitedStrings() Config

WithNoticeDelimitedStrings enables deliminated string parsing (q'DELIM .... DELIM') using the Literal token (Oracle)

func (Config) WithNoticeDelimiterStatement added in v0.4.0 

func (c Config) WithNoticeDelimiterStatement() Config

WithNoticeDelimiterStatement enables recognition of custom statement delimiters (e.g., DELIMITER // ... DELIMITER ;). This is mutually exclusive with NoticeBeginEndBlock - enabling this disables BEGIN/END block tracking.

func (Config) WithNoticeDollarNumber added in v0.1.0 

func (c Config) WithNoticeDollarNumber() Config

WithNoticeDollarNumber enables parsing dollar parameters ($1) for PostgreSQL and SQLite using the DollarNumber token

func (Config) WithNoticeDollarQuotes added in v0.1.0 

func (c Config) WithNoticeDollarQuotes() Config

WithNoticeDollarQuotes enables dollar quote $$parsing$$ for PostgreSQL using the DollarNumber token

func (Config) WithNoticeEscapedStrings added in v0.4.0 

func (c Config) WithNoticeEscapedStrings() Config

WithNoticeEscapedStrings enables escaped string parsing (E'string') using the Literal token (PostgreSQL)

func (Config) WithNoticeHashComment added in v0.1.0 

func (c Config) WithNoticeHashComment() Config

WithNoticeHashComment enables parsing for '#' comments (MySQL) using the Punctuation token

func (Config) WithNoticeHexNumbers added in v0.1.0 

func (c Config) WithNoticeHexNumbers() Config

WithNoticeHexNumbers enables quoted hex number parsing (x'af') using the HexNumber token (MySQL)

func (Config) WithNoticeIdentifiers added in v0.1.0 

func (c Config) WithNoticeIdentifiers() Config

WithNoticeIdentifiers enables parsing for identifiers (SQL Server) using the Identifier token

func (Config) WithNoticeLiteralBackslashEscape added in v0.4.0 

func (c Config) WithNoticeLiteralBackslashEscape() Config

WithNoticeLiteralBackslashEscape enables 'escape \' quote' (MySQL by default, PostgreSQL optional).

func (Config) WithNoticeMoneyConstants added in v0.1.0 

func (c Config) WithNoticeMoneyConstants() Config

WithNoticeMoneyConstants enables money constant parsing ($10 $10.32) using the DollarNumber token (SQL Server)

func (Config) WithNoticeNationalPrefix added in v0.4.0 

func (c Config) WithNoticeNationalPrefix() Config

WithNoticeNationalPrefix enables national string prefix parsing (n'string' N'string') using the Literal token (MySQL)

func (Config) WithNoticeNotionalStrings added in v0.1.0 

func (c Config) WithNoticeNotionalStrings() Config

WithNoticeNotionalStrings enables notional string parsing (n'string') using the Literal token (Oracle, SQL Server)

func (Config) WithNoticeQuestionMark added in v0.1.0 

func (c Config) WithNoticeQuestionMark() Config

WithNoticeQuestionMark enables parsing question marks using the QuestionMark token

func (Config) WithNoticeQuestionNumber added in v0.5.0 

func (c Config) WithNoticeQuestionNumber() Config

WithNoticeQuestionNumber enables parsing numbered question marks (?<digits>) using the QuestionMark token.

func (Config) WithNoticeTypedNumbers added in v0.1.0 

func (c Config) WithNoticeTypedNumbers() Config

WithNoticeTypedNumbers enables typed number parsing (nn.nnEnn[fFdD]) using the Number token (Oracle)

func (Config) WithNoticeUAmpPrefix added in v0.1.0 

func (c Config) WithNoticeUAmpPrefix() Config

WithNoticeUAmpPrefix enables U& prefix parsing (U&"\0441\043B\043E\043D") using the Literal token (PostgreSQL)

func (Config) WithSeparatePunctuation added in v0.1.0 

func (c Config) WithSeparatePunctuation() Config

WithSeparatePunctuation enables separating successive punctuation into separate tokens

type DebugExtra added in v0.5.1 

type DebugExtra struct{}

type Token 

type Token struct {
	DebugExtra // defined in debug_true.go and debug_false.go; go test -tags debugSQLToken -failfast
	Type       TokenType
	Text       string
	// Split is set on the last token in a Tokens after CmdSplit/CmdSplitUnstripped
	// to capture the ; discarded by splitting.
	// Do not set manually.
	Split *Token
	// Strip captures what was before when you Strip() a tokens. It can be set on
	// any Token and it includes the Token itself
	// Do not set manually.
	Strip Tokens
}

func (Token) Copy added in v0.4.0 

func (t Token) Copy() Token

func (*Token) Debug added in v0.5.1 

func (t *Token) Debug() string

func (*Token) SetDebug added in v0.5.1 

func (t *Token) SetDebug(_ string)

type TokenType 

type TokenType int
const (
	Comment            TokenType = iota // 0
	Whitespace                          // 1
	QuestionMark                        // 2, used in MySQL and SQLite substitution
	AtSign                              // 3, used in sqlserver substitution
	DollarNumber                        // 4, used in PostgreSQL and SQLite substitution
	ColonWord                           // 5, used in sqlx substitution
	Literal                             // 6, strings
	Identifier                          // 7, used in SQL Server for many things
	AtWord                              // 8, used in SQL Server and SQLite, subset of Identifier
	Number                              // 9
	Delimiter                           // 10, semicolon except for MySQL when DELIMITER is used
	Punctuation                         // 11
	Word                                // 12
	Other                               // 13, control characters and other non-printables
	DelimiterStatement                  // 14, DELIMITER command - MySQL only
	Empty                               // 15, marker used in Split for a token that should be eliminated in join
)

func TokenTypeString added in v0.0.2 

func TokenTypeString(s string) (TokenType, error)

TokenTypeString retrieves an enum value from the enum constants string name. Throws an error if the param is not part of the enum.

func TokenTypeValues added in v0.0.2 

func TokenTypeValues() []TokenType

TokenTypeValues returns all values of the enum

func (TokenType) IsATokenType added in v0.0.2 

func (i TokenType) IsATokenType() bool

IsATokenType returns "true" if the value is listed in the enum definition. "false" otherwise

func (TokenType) MarshalJSON added in v0.0.2 

func (i TokenType) MarshalJSON() ([]byte, error)

MarshalJSON implements the json.Marshaler interface for TokenType

func (TokenType) String added in v0.0.2 

func (i TokenType) String() string

func (*TokenType) UnmarshalJSON added in v0.0.2 

func (i *TokenType) UnmarshalJSON(data []byte) error

UnmarshalJSON implements the json.Unmarshaler interface for TokenType

type Tokens 

type Tokens []Token

func Tokenize 

func Tokenize(s string, config Config) Tokens

Tokenize breaks up SQL strings into Token objects. No attempt is made to break successive punctuation.

func TokenizeMySQL 

func TokenizeMySQL(s string) Tokens

TokenizeMySQL breaks up MySQL / MariaDB strings into Token objects.

func TokenizeMySQLAPI added in v0.5.0 

func TokenizeMySQLAPI(s string) Tokens

TokenizeMySQLAPI breaks up MySQL / MariaDB strings into Token objects. Uses BEGIN/END block tracking for stored procedures.

func TokenizePostgreSQL 

func TokenizePostgreSQL(s string) Tokens

TokenizePostgreSQL breaks up PostgreSQL / CockroachDB SQL strings into Token objects.

func TokenizeSQLite added in v0.5.0 

func TokenizeSQLite(s string) Tokens

TokenizeSQLite breaks up SQLite strings into Token objects.

func TokenizeSingleStore added in v0.4.0 

func TokenizeSingleStore(s string) Tokens

TokenizeSingleStore breaks up SingleStore SQL strings into Token objects.

func TokenizeSingleStoreAPI added in v0.5.0 

func TokenizeSingleStoreAPI(s string) Tokens

TokenizeSingleStoreAPI breaks up SingleStore SQL strings into Token objects. Uses BEGIN/END block tracking for stored procedures.

func (Tokens) CmdSplit 

func (ts Tokens) CmdSplit() TokensList

CmdSplit breaks up the token array into multiple token arrays, one per command (splitting on ";" or on the delimiter if there is a delimiter set) and Strip()ing each of the returned Tokens. Empty (just comments/whitespace) commands are eliminated and are not recoverable with Join().

DELIMITER commands become an annotation on each command (in the first token) that has a special delimiter (rather than being a stand-alone command). That annotation will turn into bracketing each command with DELIMITER commands thus producing a result that is longer than the original but logically equivalent with each command self-contained.

func (Tokens) CmdSplitUnstripped added in v0.2.0 

func (ts Tokens) CmdSplitUnstripped() TokensList

CmdSplitUnstripped breaks up the token array into multiple token arrays, one per command (splitting on ";" or the current delimiter). It does not Strip() the commands.

DELIMITER statements will be repeated on each command so that each statement becomes self-contained.

func (Tokens) Copy added in v0.4.0 

func (ts Tokens) Copy() Tokens

func (Tokens) String 

func (ts Tokens) String() string

String returns exactly the original text unless the Tokens is a result of Strip() or CmdSplit(), or CmdSplitUnstripped().

func (Tokens) Strip 

func (ts Tokens) Strip() Tokens

Strip removes leading/trailing whitespace and semicolons and strips all internal comments. Internal whitespace is changed to a single space. Strip is not reversible if there is no command present (all whitespace and/or comment).

func (Tokens) Unstrip added in v0.4.0 

func (ts Tokens) Unstrip() Tokens

Unstrip reverses a Strip

type TokensList 

type TokensList []Tokens

func (TokensList) Copy added in v0.4.0 

func (tl TokensList) Copy() TokensList

func (TokensList) Join added in v0.3.0 

func (tl TokensList) Join() Tokens

Join reverses Split: adding back delimiters between the token lists

Join does not always recreate the original input. It tries to come close though. Use of DELIMITER will often create small differences.

func (TokensList) Strings 

func (tl TokensList) Strings() []string

Strings returns almost exactly the original text of each Tokens (except the semicolons) unless Strip() or CmdSplit() was called.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.