Splits SQL into tokens without parsing it, so it works on any statement the engine accepts,
including SQL the AST parser rejects. Every character of the input belongs to exactly one
token, so joining the tokens' text reproduces the input.
The dialect's lexical rules decide which string literal forms (prefixes, doubled-quote and
backslash escapes, dollar quoting, Oracle q'…' quoting), quoted identifier forms and comment
forms (--, #, //, nested or flat block comments) exist. An unterminated literal or
comment runs to the end of the input.
Splits SQL into tokens without parsing it, so it works on any statement the engine accepts, including SQL the AST parser rejects. Every character of the input belongs to exactly one token, so joining the tokens' text reproduces the input.
The dialect's lexical rules decide which string literal forms (prefixes, doubled-quote and backslash escapes, dollar quoting, Oracle
q'…'quoting), quoted identifier forms and comment forms (--,#,//, nested or flat block comments) exist. An unterminated literal or comment runs to the end of the input.