Splits SQL into tokens without parsing it, so it works on any statement the engine accepts,
including SQL the AST parser rejects. Every character of the input belongs to exactly one
token, so joining the tokens' text reproduces the input.
Recognizes string literals with doubled-quote escapes and the N prefix, double-quoted
identifiers, -- line comments (ended by a CR or an LF) and nested block comments everywhere,
and per dialect:
bracket or backtick identifiers when the dialect quotes identifiers that way, E'…' strings
with backslash escapes when SQLParserDialect.SupportsEscapeStringLiterals is set, and
dollar-quoted strings ($$…$$, $tag$…$tag$) when
SQLParserDialect.SupportsDollarQuotedStrings is set. An unterminated literal or comment
runs to the end of the input.
Splits SQL into tokens without parsing it, so it works on any statement the engine accepts, including SQL the AST parser rejects. Every character of the input belongs to exactly one token, so joining the tokens' text reproduces the input.
Recognizes string literals with doubled-quote escapes and the
Nprefix, double-quoted identifiers,--line comments (ended by a CR or an LF) and nested block comments everywhere, and per dialect: bracket or backtick identifiers when the dialect quotes identifiers that way,E'…'strings with backslash escapes when SQLParserDialect.SupportsEscapeStringLiterals is set, and dollar-quoted strings ($$…$$,$tag$…$tag$) when SQLParserDialect.SupportsDollarQuotedStrings is set. An unterminated literal or comment runs to the end of the input.