Kinds of token LexSQL produces.
word
SELECT
Name
__mj
number
string
'…'
N'…'
E'…'
$tag$…$tag$
identifier
"…"
[…]
`…`
comment
open
close
whitespace
comma
semicolon
dot
operator
Kinds of token LexSQL produces.
word: a bare keyword or identifier (SELECT,Name,__mj).number: a numeric literal.string: a string literal in any form ('…',N'…',E'…',$tag$…$tag$).identifier: a quoted identifier ("…",[…],`…`).comment: a line or block comment.open/close: a parenthesis.whitespace,comma,semicolon,dot,operator: the rest.