preview-tokens
Preview how ANTLR4 lexer rules tokenize input text. Test grammar changes, debug tokenization issues, and verify rule matches instantly.
Instructions
Preview how input text would be tokenized by lexer rules. Test grammar changes instantly!
š Native ANTLR4 Support: Automatically uses native ANTLR4 runtime if available for 100% accurate tokenization including:
ā Lexer modes (pushMode, popMode)
ā Semantic predicates ({...?})
ā Actions ({...})
ā All ANTLR4 features
Falls back to simulation if ANTLR4 is not installed (works for simple grammars without modes/predicates).
When to use:
Test if lexer rules match input as expected
Debug tokenization issues (especially with complex grammars!)
Verify grammar changes work correctly
Learn how ANTLR4 tokenizes input
Setup for 100% accuracy (optional):
Install Java: brew install openjdk
Install ANTLR4: wget https://www.antlr.org/download/antlr-4.13.1-complete.jar
Set env: export ANTLR4_JAR=/path/to/antlr-4.13.1-complete.jar
How it works:
Native mode: Compiles and executes actual ANTLR4 lexer (100% accurate)
Simulation mode: Best-effort tokenization (works for ~70% of grammars)
Example - Test basic tokenization: input: "x = 42;"
Example - Test with complex lexer (Palo Alto): from_file: "PaloAlto_lexer.g4" load_imports: true input: "set user-id-collector enable-mapping-timeout 1"
Example - Test specific rules only: input: "x + y * 2" rules_to_test: ["ID", "PLUS", "TIMES", "INT"]
Returns:
List of tokens with types and values
Character positions (start, end, line, column)
Channel information for channeled tokens
Errors for unmatched characters
Mode indicator (š Native or ā ļø Simulation)
Feature warnings if simulation used
Limitations:
ā Lexer modes not supported
ā Semantic predicates not evaluated
ā Actions not executed
ā ļø Fragment rules (basic support)
ā ļø Complex patterns (best-effort)
Alternative: For complex grammars, use test-parser-rule which tests parser rules without full lexer simulation.
Note: This is a simplified simulation. For 100% accuracy, use ANTLR4 tooling.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| input | Yes | Input text to tokenize. Can include newlines and special characters. | |
| from_file | No | Optional: path to a grammar file to read | |
| rules_to_test | No | Optional: Test only specific lexer rules (by name). Useful for focused testing. | |
| show_positions | No | If true, show detailed position information (line, column, start, end) for each token | |
| grammar_content | No | The ANTLR4 grammar file content with lexer rules |