- [v2](#v2) ## v2 v2 was released in November 2020. It contains the following changes, some of which are backwards-incompatible: - Added optional `LexString()` and `LexBytes()` methods that lexer definitions can implement to fast-path lexing of bytes and strings. - A new stateful lexer has been added. - A `filename` must now be passed to all `Parse*()` and `Lex*()` methods. - The `text/scanner` lexer no longer automatically unquotes strings or supports arbitary length single quoted strings. The tokens it produces are identical to that of the `text/scanner` package. Use `Unquote()` to remove quotes. - `Tok` and `EndTok` will no longer be populated. - If a field named `Token []lexer.Token` exists it will be populated with the raw tokens that the node parsed from the lexer. - Support capturing directly into lexer.Token fields. eg. type ast struct { Head lexer.Token `@Ident` Tail []lexer.Token `@(Ident*)` } - Add an `experimental/codegen` for stateful lexers. This provides ~10x performance improvement with zero garbage when lexing strings. - The `regex` lexer has been removed. - The `ebnf` lexer has been removed. - All future work on lexing will be put into the stateful lexer. - The need for `DropToken` has been removed.