Tokens that carry meaning: everything except whitespace and comments.
Use SqlTokens.Lex(sql, dialect).Significant().
SqlTokens.Lex(sql, dialect).Significant()
Tokens that carry meaning: everything except whitespace and comments.