Add a new lexer mode declaration to an ANTLR4 grammar.

When to use: Creating new modes for context-sensitive tokenization.

Features:

• Adds "mode MODE_NAME;" declaration

• Optional positioning with insert_after

• Validates mode name format

• Prevents duplicate mode names

Example - Add mode after specific rule: mode_name: "STRING_MODE" insert_after: "STRING" write_to_file: true

Example - Add mode at end of grammar: mode_name: "TEMPLATE_MODE" write_to_file: true

Add multiple lexer rules to a grammar in a single operation (bulk add).

When to use: Set up initial grammar structure, add multiple related tokens at once, or import rules from another grammar.

Example - Add common tokens: rules: [ { name: "ID", pattern: "[a-zA-Z_][a-zA-Z0-9_]" }, { name: "INT", pattern: "[0-9]+" }, { name: "WS", pattern: "[ \t\n\r]+", skip: true }, { name: "COMMENT", pattern: "//.?\n", channel: "COMMENTS" } ] write_to_file: true

Features:

• All rules inserted in alphabetical order

• Duplicate prevention per rule

• Partial success: some rules can succeed even if others fail

• Per-rule success/failure reporting

• Atomic per-rule operations

Returns:

• Summary: "Added X rules, Y failed"

• Per-rule results with success/failure status

• Modified grammar

• File write confirmation if applicable

Add multiple parser rules to a grammar in a single operation (bulk add).

When to use: Set up grammar structure, add multiple related parsing rules, or quickly prototype a grammar.

Example - Add expression rules: rules: [ { name: "program", definition: "statement+" }, { name: "statement", definition: "assignment | ifStatement | whileStatement" }, { name: "assignment", definition: "ID ASSIGN expression SEMI" }, { name: "expression", definition: "term ((PLUS | MINUS) term)*" } ] write_to_file: true

Features:

• All rules inserted in alphabetical order

• Duplicate prevention per rule

• Partial success: some rules can succeed even if others fail

• Per-rule success/failure reporting

• Atomic per-rule operations

Returns:

• Summary: "Added X rules, Y failed"

• Per-rule results with success/failure status

• Modified grammar

• File write confirmation if applicable

Add a new lexer or parser rule with automatic type detection and positioning.

Auto-detection: Rule type is determined by naming convention:

• UPPERCASE = lexer rule (e.g., ID, STRING, NUMBER)

• lowercase = parser rule (e.g., expression, statement, term)

When to use: Add any rule to your grammar - lexer tokens or parser rules.

Example - Add lexer rule (UPPERCASE): rule_name: "ID" pattern: "[a-zA-Z_][a-zA-Z0-9_]*"

Example - Add parser rule (lowercase): rule_name: "expression" definition: "term ((PLUS | MINUS) term)*"

Example - Add lexer with skip: rule_name: "WS" pattern: "[ \t\n\r]+" skip: true

Example - Add lexer with channel: rule_name: "COMMENT" pattern: "//.*?\n" channel: "COMMENTS"

Example - Add fragment: rule_name: "DIGIT" pattern: "[0-9]" fragment: true

Example - Add with positioning: rule_name: "STRING" pattern: "".*?"" insert_after: "ID"

Example - Add parser with return type: rule_name: "intLiteral" definition: "INT" return_type: "int value"

Features:

• Auto-detects lexer vs parser from rule name case

• Default: Alphabetical sorting within rule type

• Optional: Custom positioning with insert_after/insert_before

• Lexer-specific: skip, channel, fragment options

• Parser-specific: return_type option

• Prevents duplicate rule names

• Optional file persistence with write_to_file: true

• Diff output mode for large grammars

Returns: Modified grammar with new rule inserted, success message, position description, file write confirmation if applicable.

Add multiple mixed parser and lexer rules in a single operation (bulk add).

When to use: Set up complete grammar structure, add both tokens and parsing rules together, or quickly prototype.

Example - Add mixed rules: rules: [ { type: "lexer", name: "PLUS", pattern: "'+'" }, { type: "lexer", name: "MINUS", pattern: "'-'" }, { type: "parser", name: "expression", definition: "term ((PLUS | MINUS) term)*" }, { type: "parser", name: "term", definition: "INT | ID" } ] write_to_file: true

Features:

• Handles both lexer and parser rules in one operation

• Rules sorted alphabetically within their type category

• Duplicate prevention per rule

• Partial success: some rules can succeed even if others fail

• Per-rule success/failure reporting

• Atomic per-rule operations

Returns:

• Summary: "Added X rules, Y failed"

• Per-rule results with success/failure status

• Modified grammar

• File write confirmation if applicable

Add a lexer rule to a specific mode in an ANTLR4 grammar.

When to use: Adding tokens that only apply in specific lexical contexts.

Features:

• Places rule in the correct mode section

• Validates mode exists

• Supports all lexer rule options (skip, channel, fragment)

• Auto-sorts within the mode

Example - Add rule to STRING_MODE: rule_name: "INTERPOLATION_START" pattern: "\{" mode_name: "STRING_MODE" write_to_file: true

Example - Add with pushMode action: rule_name: "INTERPOLATION_START" pattern: "\{" mode_name: "STRING_MODE" action: "pushMode(EXPRESSION_MODE)" write_to_file: true

Add multiple similar lexer tokens at once using template-based generation.

When to use:

• Add multiple tokens that follow a similar pattern

• Generate tokens for command sequences (e.g., "config system X", "set X Y")

• Batch-add tokens with consistent naming conventions

Example - Add tokens for "config system X" patterns: base_names: ["ftm-push", "dns", "firewall", "admin"] preceding_tokens: ["SYSTEM", "CONFIG"]

Example - Add multiple keyword tokens: base_names: ["enable", "disable", "show", "hide"] pattern: "'{NAME}'"

Example - Add tokens with custom pattern: base_names: ["tcp", "udp", "icmp"] pattern: "'protocol-{NAME}'"

Features:

• Automatically generates proper token names (uppercase with underscores)

• Supports custom patterns with {NAME} placeholder

• Uses existing bulk add infrastructure for reliability

• All standard options supported (skip, channel, fragment)

Returns:

• Generated rules list

• Per-rule success/failure status

• Modified grammar

• Summary of results

Analyze grammar for common ambiguity patterns that may cause conflicts at runtime.

When to use: Before compiling grammar, after making changes, or when diagnosing parser warnings.

Detection Capabilities:

1. Identical Alternatives (ERROR)

   • Detects exact duplicate alternatives in rules

   • Example: expr: ID | NUMBER | ID → duplicate ID alternative

2. Overlapping Prefixes (WARNING)

   • Finds alternatives that start with same tokens

   • Example: stmt: IF expr THEN stmt | IF expr THEN stmt ELSE stmt

   • Suggestion: Factor out common prefix

3. Ambiguous Optionals (WARNING)

   • Detects A? A patterns (should be A+)

   • Detects A? A* patterns (A* is sufficient)

4. Hidden Left Recursion (ERROR)

   • Detects indirect left recursion via other rules

   • Example: expr: term, term: expr PLUS → hidden recursion

5. Lexer Conflicts (WARNING)

   • Identifies lexer rules that may overlap

   • Example: ID: [a-z]+ and KEYWORD: 'if' → keyword is also valid ID

Options:

• Selective checks: Enable/disable specific ambiguity patterns

• Minimum prefix length: Set threshold for prefix overlap warnings

• Severity levels: ERROR (must fix), WARNING (should review), INFO (optional)

Returns:

• List of issues with severity, type, rule name, line number

• Detailed descriptions and actionable suggestions

• Summary counts (errors, warnings, infos)

Example usage: from_file: "MyGrammar.g4" checkIdenticalAlternatives: true checkOverlappingPrefixes: true minPrefixLength: 2

Analyze grammar for performance bottlenecks and optimization opportunities.

When to use: Performance optimization, grammar refactoring, large grammar analysis.

Detects:

• High-branching rules: Rules with many alternatives (10+, 20+, 50+)

• Tilde negation patterns: ~NEWLINE, ~[ ] that could use lexer modes

• Missing lexer mode opportunities: String handling, line-based content, multi-line blocks

• Greedy loop issues: Nested quantifiers, reluctant patterns

• Deep recursion: Rules with potential stack overflow risk

• Token prefix collisions: Keywords that are prefixes of other keywords

Returns:

• Bottlenecks with severity (high/medium/low)

• Specific suggestions for each issue

• Estimated performance improvement potential

• Prioritized recommendations

Example: from_file: "MyGrammar.g4"

Analyze an ANTLR4 grammar file and extract its complete structure.

When to use: First step when exploring an unfamiliar grammar, understanding architecture, or extracting metadata.

Multi-file support: Set load_imports=true to automatically resolve and analyze imported grammars.

Example usage: from_file: "examples/MyGrammar.g4"

Multi-file example: from_file: "PaloAlto_rulebase.g4" base_path: "/path/to/grammars" load_imports: true

Returns:

• Grammar name and type (parser/lexer/combined)

• All parser rules with definitions and line numbers

• All lexer rules with patterns

• Rule references and dependencies (which rules reference which)

• Import declarations

• Grammar options

• Validation issues (undefined rules, unused rules, recursion)

• With load_imports=true: Includes rules from all imported grammars

Analyze lexer mode structure in ANTLR4 grammars.

When to use: Understanding mode-based tokenization, debugging mode transitions, documenting mode structure.

Lexer modes allow context-sensitive tokenization by switching between different sets of lexer rules. Common use cases:

• String interpolation (switching modes inside strings)

• Nested comments

• Template parsing

• Context-specific keywords

Features:

• Lists all defined modes with their rules

• Identifies mode entry points (pushMode actions)

• Identifies mode exit points (popMode actions)

• Detects common issues (undefined modes, unreachable modes, popMode in DEFAULT_MODE)

Returns:

• modes: List of modes with their rules and line numbers

• entryPoints: Rules that push to each mode

• exitPoints: Rules that pop from each mode

• issues: Problems detected (undefined modes, empty modes, etc.)

Example usage: from_file: "MyLexer.g4"

Analyze mode transition graph and detect issues in ANTLR4 lexer modes.

When to use: Debugging mode switching logic, ensuring balanced push/pop, detecting circular transitions.

Features:

• Builds complete mode transition graph

• Detects circular mode transitions

• Checks for balanced pushMode/popMode usage

• Identifies modes with no exit points

• Suggests improvements for mode structure

Returns:

• transitions: List of all mode transitions (from, to, via action, rule)

• issues: Problems detected (circular transitions, unbalanced push/pop)

• suggestions: Recommendations for improving mode structure

Example usage: from_file: "MyLexer.g4"

Benchmark grammar parsing performance with sample input.

When to use: Performance testing, comparing grammar versions, optimization validation.

Measures:

• Total tokens produced

• Average/min/max parse time (ms)

• Tokens per second throughput

• Performance rating (excellent/good/fair/slow)

Features:

• Warmup iterations to account for JIT

• Multiple iterations for statistical accuracy

• Performance rating based on parse time

• Optimization suggestions for slow grammars

Parameters:

• grammar_content or from_file: The grammar to test

• input: Sample input text to parse

• iterations: Number of timed iterations (default: 10)

• warmup_iterations: Warmup runs before timing (default: 3)

Example: from_file: "MyGrammar.g4" input: "x = 42 + y * 10" iterations: 20

Check grammar style and best practices with quality scoring.

When to use: Code review, maintaining grammar quality, enforcing conventions.

Checks:

Naming Conventions:

• Lexer rules should use UPPER_CASE

• Parser rules should use lowerCamelCase

Best Practices:

• Missing grammar declaration

• Unused/orphan rules

• Overly complex rules

Maintainability:

• Missing documentation on complex rules

• Rule complexity warnings

Returns:

• Issues with severity (error/warning/info)

• Style score (0-100)

• Specific suggestions for each issue

Example: from_file: "MyGrammar.g4"

Compare two ANTLR4 grammars side-by-side to identify differences.

When to use: Understand changes between versions, merge grammars, or analyze variations.

Example usage: from_file1: "v1/MyGrammar.g4" from_file2: "v2/MyGrammar.g4"

Returns:

• Common rules (unchanged)

• Rules unique to grammar 1

• Rules unique to grammar 2

• Modified rules (exist in both but differ)

• Statistical summary (counts, percentages)

Compare two parsing profiles to measure optimization impact.

When to use: Validating grammar optimizations, A/B testing changes, regression testing.

Parameters:

• profile1: First profile result (from profile-parsing)

• profile2: Second profile result (from profile-parsing)

Returns:

• Comparison metrics with % change

• Performance verdict (improved, degraded, unchanged)

• Key differences highlighted

• Recommendations based on changes

Example: profile1: { parseTimeMs: 150, ambiguityCount: 5, ... } profile2: { parseTimeMs: 80, ambiguityCount: 0, ... }

Output:

Verdict: ✅ Improved - Parse time reduced by 46.7%

Run native ANTLR4 compilation checks for grammar syntax/tooling errors.

When to use: Before committing grammar edits, after refactors, or when validate-grammar looks clean but ANTLR toolchain still fails.

Supports:

• ANTLR4 native diagnostics (error/warning with line/column)

• Multi-file imports/tokenVocab with load_imports=true and from_file

• Compile timing and generated file summary

Example usage: from_file: "MyGrammar.g4" load_imports: true

Returns: compile success/failure, native diagnostics, timing, generated artifacts.

Create a new ANTLR4 grammar from scratch with optional mode structure.

When to use: Starting a new grammar project, scaffolding grammar structure, creating grammar templates.

Features:

• Creates lexer, parser, or combined grammar

• Optionally includes boilerplate rules (WS, ID, NUMBER, STRING, comments)

• Adds specified modes with placeholder comments

• Ready-to-use structure for common patterns

Example - Simple lexer grammar: grammar_name: "MyLexer" type: "lexer"

Example - Lexer with modes: grammar_name: "TemplateLexer" type: "lexer" modes: ["STRING_MODE", "COMMENT_MODE", "INTERPOLATION_MODE"] include_boilerplate: true

Example - Combined grammar: grammar_name: "Calculator" type: "combined" include_boilerplate: true

Detect incomplete parsing patterns (anti-patterns that discard content).

When to use: When grammar parses configs but doesn't capture structure, or uses placeholder patterns.

Anti-patterns detected:

1. null_rest_of_line usage → Discards content instead of parsing it → Example: ss_ssl_tls_service_profile: ... null_rest_of_line → Problem: Loses protocol-settings, certificates, etc.

2. Overly broad negation patterns → Example: rule: ~[\r\n]+ (matches "anything until newline") → Better: Define specific tokens for expected content

Real-world impact:

• ss_ssl_tls_service_profile used null_rest_of_line

• Lost: protocol-settings min-version/max-version, certificate options

• Result: Thousands of warnings about unparsed structure

Recommendations:

• Replace null_rest_of_line with actual structure

• Define specific lexer tokens instead of broad negations

• Implement proper parser rules for complex structures

Returns: List of incomplete parsing patterns with suggestions.

Detect suspicious quantifier patterns that may prevent parsing real configs.

When to use: After seeing "unrecognized syntax" warnings or when rules don't match multi-line configs.

Common patterns detected:

1. Rule names with _rule, _setting, _property using ? instead of * → bgpp_export: EXPORT bgp_policy_rule? should be *

2. Multiple optional elements that should be alternatives → source? destination? action? should be (source | destination | action)*

3. Same optional reference appearing multiple times → rule: setting? ... setting? should use setting*

Real-world example: Rule: srs_definition: ... source_setting? destination_setting? action_setting? Issue: Config has multiple 'set source', 'set destination' lines Fix: Change to (source_setting | destination_setting | action_setting)*

Returns: List of suspicious patterns with suggestions and reasoning.

Detect ReDoS (Regular Expression Denial of Service) vulnerabilities in lexer patterns.

When to use: Security audit, performance optimization, validating lexer patterns.

Detects:

• Nested quantifiers: (a+)+, (a*)*

• Overlapping alternatives: (a|a)+

• Alternatives with common prefix: (ab|ac)

• Unbounded repetition of broad character classes

• Multiple optional elements in sequence

Returns:

• List of vulnerabilities with severity (high/medium/low)

• Line numbers and affected rules

• Specific suggestions for each issue

Example: from_file: "MyLexer.g4"

Duplicate a lexer mode with all its rules.

When to use: Creating similar modes, refactoring mode structure, creating mode templates.

Features:

• Copies all rules from source mode to new mode

• Optional prefix for cloned rule names

• Creates new mode declaration automatically

Example - Duplicate mode without prefix: source_mode: "STRING_MODE" new_mode: "TEMPLATE_MODE"

Example - Duplicate with rule prefix: source_mode: "STRING_MODE" new_mode: "INTERPOLATION_MODE" prefix_rules: "INTERP_"

Rules will be named: INTERP_STRING_CONTENT, etc.

Generate comprehensive Markdown documentation for your grammar.

When to use: Create README files, generate reference documentation, document grammar structure, share grammar specs.

Example usage: from_file: "MyGrammar.g4"

Generated documentation includes:

• Grammar name, type, and metadata

• Overview section with rule counts

• Parser rules section with definitions and references

• Lexer rules section with patterns

• Import declarations

• Grammar options

• Validation issues and warnings

• Summary statistics

Output format: Complete Markdown document ready for use in README.md or documentation sites.

Extract a reusable fragment rule from a pattern to reduce duplication.

When to use: Share common patterns, improve maintainability, reduce duplication in lexer rules.

Example - Extract digit pattern: fragment_name: "DIGIT" pattern: "[0-9]"

Example - Extract letter pattern: fragment_name: "LETTER" pattern: "[a-zA-Z]"

After extraction, use the fragment in other rules: ID: LETTER (LETTER | DIGIT)*

Benefits:

• Single source of truth for common patterns

• Easier maintenance

• Clearer lexer organization

• Fragments are not tokens themselves (helper patterns only)

Returns: Modified grammar with fragment added, original pattern preserved in existing rules.

Find rules with multiple matching modes: exact, regex, wildcard, or partial matching.

When to use: Locate specific rules, understand rule relationships, or discover rules matching a pattern.

Matching Modes:

1. Exact (default) - Exact rule name match: rule_name: "expression"

2. Regex - Regular expression pattern: rule_name: "^[A-Z]+$" match_mode: "regex" (Finds all lexer rules)

3. Wildcard - Shell-style wildcards (* and ?): rule_name: "expr*" match_mode: "wildcard" (Finds expression, expr, exprStatement, etc.)

   rule_name: "stat?" match_mode: "wildcard" (Finds stat1, stat2, stats, etc.)

4. Partial - Substring/contains search (case-insensitive): rule_name: "token" match_mode: "partial" (Finds tokenList, getToken, TOKEN_TYPE, etc.)

Examples:

• All lexer rules: rule_name="^[A-Z]+$", match_mode="regex"

• All parser rules: rule_name="^[a-z]+$", match_mode="regex"

• Rules starting with "expr": rule_name="expr*", match_mode="wildcard"

• Rules containing "statement": rule_name="statement", match_mode="partial"

Returns (exact match):

• Rule definition, type, and line number

• Referenced rules (fan-out: what this rule uses)

• Referencing rules (fan-in: what uses this rule)

• Usage count

Returns (pattern match):

• All matching rules with their details

• Match count

Find all locations where a specific rule is referenced with line numbers and context.

When to use: Before renaming/removing rules to understand impact, analyze dependencies, or trace rule usage.

Example usage: rule_name: "expression"

Returns:

• Total usage count

• Per-usage details: line number, rule context (which rule contains the reference)

• Complete usage report

Multi-file support:

• Set load_imports: true to search across imported grammars

• Useful for finding cross-file dependencies

Use before:

• Renaming rules (to see what will be affected)

• Removing rules (to identify breaking changes)

• Refactoring (to understand dependencies)

Selectively fix suspicious quantifier patterns - change )? to )* for specific rules.

When to use: After detect-quantifier-issues identifies problems.

Workflow:

1. Run detect-quantifier-issues to see what's suspicious

2. Review the suggestions

3. Run fix-quantifier-issues with specific rule_names to fix

What it fixes:

• Rules with alternatives: (a | b | c)? → (a | b | c)*

• Multiple optional elements that suggest repetition

• Collection-named rules (_rules, _settings) using )?

Examples:

// Step 1: Detect issues detect-quantifier-issues(from_file: "PaloAlto_interface.g4") → Shows: snie_ethernet, snie_lacp, sniel_high_availability, snil_units

// Step 2: Fix specific rules fix-quantifier-issues( from_file: "PaloAlto_interface.g4", rule_names: ["snie_ethernet", "snie_lacp", "snil_units"] ) → Fixes only those 3 rules

// Fix all detected issues fix-quantifier-issues(from_file: "PaloAlto_interface.g4") → Fixes all suspicious patterns

// Preview without changing fix-quantifier-issues( from_file: "PaloAlto_interface.g4", dry_run: true ) → Shows what would change

Real-world: Palo Alto grammar had 15 rules flagged. User fixed 12, left 3 as-is (they were correct).

Returns: List of changes with line numbers and reasoning

Display a structured summary of the grammar with proper formatting.

When to use: Get a high-level overview of grammar organization and structure.

Returns:

• Grammar name and type

• Complete rule list (organized by type)

• Import declarations

• Grammar options

• Issue summary with counts

Generate stress test inputs for grammar performance testing.

When to use: Testing grammar robustness, identifying performance issues, benchmarking.

Generation strategies:

• nested: Deep nesting of recursive rules (tests stack depth)

• wide: Many alternatives in choice rules (tests branching)

• repetition: Repeated sequences (tests loops)

• mixed: Combination of all strategies (default)

Parameters:

• grammar_content: The grammar to generate tests for

• strategy: Generation strategy (nested, wide, repetition, mixed)

• depth: Nesting depth for nested strategy (default: 50)

• count: Number of alternatives for wide strategy (default: 100)

• repetitions: Repetition count for repetition strategy (default: 100)

Returns:

• Generated test input

• Expected characteristics (depth, width, size)

• Warnings if grammar structure can't support requested strategy

Example: grammar_content: "grammar Expr; ..." strategy: "nested" depth: 30

Output: "(((...(1 + 2)...)))" (30 levels deep)

Generate a concise summary of grammar structure and health metrics.

When to use: Quick health checks, progress tracking, overview reports, or grammar comparisons.

Example usage: from_file: "MyGrammar.g4"

Returns:

• Grammar name and type

• Total rule count (parser/lexer breakdown)

• Import declarations

• Top 5 most referenced rules (indicating key grammar components)

• Issue summary (error/warning/info counts)

• Grammar health assessment

• Complexity indicators

Perfect for: Status reports, quick assessments, tracking changes over time.

Generate lexer tokens automatically from natural language input patterns.

When to use:

• Quick token generation from sample input text

• Convert configuration snippets into grammar rules

• Generate tokens from command examples

• Prototype grammars from example input

Example - Generate tokens from command: input_pattern: "ignore config system ftm-push" → Generates: IGNORE, CONFIG, SYSTEM, FTM_PUSH tokens

Example - Generate single compound token: input_pattern: "config-system-admin" tokenize: false → Generates: CONFIG_SYSTEM_ADMIN token

Example - Add prefix to generated tokens: input_pattern: "show running-config" prefix: "CMD" → Generates: CMD_SHOW, CMD_RUNNING_CONFIG tokens

Features:

• Automatic tokenization (splits on whitespace by default)

• Intelligent name generation (uppercase with underscores)

• Optional prefix for token namespacing

• Supports all standard token options (skip, channel, fragment)

• Generates proper ANTLR4 string literal patterns

Returns:

• List of generated tokens with names and patterns

• Per-rule success/failure status

• Modified grammar

• Summary of results

Get actionable improvement suggestions for an ANTLR4 grammar.

When to use: Optimize grammar quality, identify issues, get best practice recommendations.

Analyzes:

• Naming convention compliance (uppercase lexer, lowercase parser)

• Rule complexity and performance concerns

• Unused rules that could be removed

• Undefined references

• Fragment opportunities for code reuse

• Left recursion patterns

Returns: Categorized suggestions with specific recommendations for improvement.

Calculate comprehensive grammar metrics including branching estimation, complexity, and dependencies.

When to use: Understanding grammar complexity, identifying optimization opportunities, estimating parsing performance.

Metrics included:

Size Metrics:

• Total/parser/lexer rule counts

• Fragment counts

• Lines of code, average rule length

Branching Metrics:

• Average/max alternatives per rule

• Branching depth (subrule nesting)

• Branching distribution (1-2, 3-5, 6-10, 10+)

• Rules with most branching

Complexity Metrics:

• Cyclomatic complexity (per rule and total)

• Recursive rules detection

• Estimated parse complexity (low/medium/high/very-high)

Dependency Metrics:

• Fan-in/fan-out averages

• Orphan rules (unused)

• Hub rules (highly referenced)

• Most referenced rules

Example: from_file: "MyGrammar.g4"

Get comprehensive help about ANTLR4 MCP tools, common workflows, and usage examples.

USE THIS TOOL FIRST when starting work with ANTLR4 grammars.

Available topics:

• "overview": Summary of all 27 available tools by category

• "workflows": Common multi-step workflows for typical tasks

• "analysis": Detailed info about analysis and inspection tools

• "authoring": Detailed info about editing and modification tools

• "refactoring": Detailed info about refactoring and optimization tools

• "examples": Practical examples of tool usage

Returns: Comprehensive documentation for the requested topic.

Analyze change impact for a specific rule (dependencies, dependents, and risk).

When to use: Before renaming, deleting, or heavily modifying a rule.

Returns:

• Direct/transitive dependencies (what this rule needs)

• Direct/transitive dependents (what would be affected)

• Usage count and recursion status

• Entry-rule detection and risk level

• Actionable impact summary

Analyze an ANTLR4 grammar and infer its formatting style.

When to use: Understand the formatting conventions used in a grammar, or verify that formatting will be preserved when making changes.

Example usage: from_file: "MyGrammar.g4"

Analyzes:

• Colon placement (same-line vs new-line after rule name)

• Semicolon placement (same-line vs new-line after definition)

• Space before colon (e.g., "rule :" vs "rule:")

• Indentation style (spaces or tabs, and how many)

• Blank lines between rules

Returns: Formatting style object with detected patterns.

Note: The update-rule, add-lexer-rule, and add-parser-rule tools automatically use this inference to preserve your grammar's formatting style.

Inline a rule by replacing all references with its definition, then delete the original rule.

When to use:

• Remove "pass-through" or helper rules

• Simplify grammar structure

• Clean up unnecessary indirection

• Inverse of extract-fragment refactoring

How it works:

1. Validates rule can be inlined (no circular references, not recursive, actually used)

2. Extracts rule body (removes labels, actions preserved)

3. Finds all references to the rule

4. Replaces references with rule body (adds parentheses if needed)

5. Removes original rule definition

Example - Simple pass-through: rule_name: "additiveExpression" // Before: expression: additiveExpression; // additiveExpression: term ((PLUS | MINUS) term); // After: expression: term ((PLUS | MINUS) term);

Example - Helper rule: rule_name: "value" // Before: assignment: ID ASSIGN value SEMI; // value: NUMBER | STRING | ID; // After: assignment: ID ASSIGN (NUMBER | STRING | ID) SEMI;

Features:

• Circular reference detection

• Smart parenthesization

• Multi-reference support

• Dry-run mode available

• Detailed statistics

Returns:

• Modified grammar with rule inlined

• Number of references replaced

• List of affected rules

• Original rule definition

Validates:

• Rule exists

• Not self-recursive

• No circular dependencies

• Actually used somewhere

List all lexer rules in a specific mode.

When to use: Quick inspection of mode contents, debugging mode issues, understanding mode structure.

Returns:

• List of rules with names, patterns, and line numbers

• Total count of rules in the mode

Example - List rules in STRING_MODE: mode_name: "STRING_MODE"

Example - List rules in DEFAULT_MODE: mode_name: "DEFAULT_MODE"

List all rules in an ANTLR4 grammar with optional filtering.

When to use: Get a quick overview of all rules, or filter to see only lexer or parser rules.

Example usage: filter_type: "lexer" // Shows only lexer rules (uppercase) filter_type: "parser" // Shows only parser rules (lowercase) filter_type: "all" // Shows all rules (default)

Returns: Alphabetically sorted list of rules with:

• Rule names

• Rule types (lexer/parser)

• Complete definitions

• Line numbers

Merge two related rules into one rule with alternatives.

When to use: Consolidate similar rules, reduce rule count, group related alternatives.

Example - Merge literal rules: rule1_name: "intLiteral" rule2_name: "floatLiteral" new_rule_name: "numericLiteral"

Result: numericLiteral: intLiteral | floatLiteral

Benefits:

• Reduces grammar complexity

• Groups related alternatives logically

• Simplifies parser structure

Note: Original rules are removed; references to them should be updated manually or use rename-rule first.

Returns: Modified grammar with merged rule created and original rules removed.

Move an existing rule to a new position relative to another rule.

When to use: Reorganizing grammar, grouping related rules, fixing rule order.

Use cases:

• Move related rules together for better organization

• Position rules before/after their dependencies

• Group similar functionality

• Manual rule ordering (alternative to sort-rules)

Features:

• Preserves rule formatting (multi-line, comments)

• Maintains blank lines between rules

• Validates both rule and anchor exist

• Detects if rule is already in target position

Example usage:

Move expr rule before term rule: rule_name: "expr" position: "before" anchor_rule: "term"

Move NUMBER token after PLUS token: rule_name: "NUMBER" position: "after" anchor_rule: "PLUS" write_to_file: true

Note: This moves EXISTING rules. To insert NEW rules at specific positions, use add-parser-rule or add-lexer-rule with insert_before/insert_after.

Move an existing lexer rule to a different mode.

When to use: Reorganizing lexer rules, fixing mode placement, refactoring grammar structure.

Features:

• Moves lexer rules between modes

• Preserves rule definition exactly

• Validates source rule exists and is a lexer rule

• Validates target mode exists

Example - Move rule to STRING_MODE: rule_name: "STRING_CONTENT" target_mode: "STRING_MODE" write_to_file: true

Note: Parser rules cannot be moved to modes (they don't have mode context).

Benchmark grammar using actual ANTLR4 Java runtime (most accurate).

When to use: Final performance testing, comparing grammars, production validation.

Requirements:

• Java must be installed

• ANTLR4 JAR must be available (auto-downloads to ~/.local/lib/)

Features:

• Uses real ANTLR4 parser (100% accurate)

• Low-overhead Java driver (avoids JVM startup per iteration)

• Warmup iterations for JIT optimization

• Supports multi-file grammars

Parameters:

• grammar_files: Object mapping filename to content {"Expr.g4": "grammar Expr..."}

• start_rule: Parser rule to start from

• input: Sample input text

• iterations: Timed iterations (default: 10)

• warmup_iterations: Warmup runs (default: 3)

Example: grammar_files: {"Expr.g4": "grammar Expr; start: expr EOF; ..."} start_rule: "start" input: "1 + 2 * 3" iterations: 20

Returns:

• Avg/min/max parse time

• Throughput (chars/sec, tokens/sec)

• Performance rating (excellent/good/fair/slow)

Preview how input text would be tokenized by lexer rules. Test grammar changes instantly!

🚀 Native ANTLR4 Support: Automatically uses native ANTLR4 runtime if available for 100% accurate tokenization including:

• ✅ Lexer modes (pushMode, popMode)

• ✅ Semantic predicates ({...?})

• ✅ Actions ({...})

• ✅ All ANTLR4 features

Falls back to simulation if ANTLR4 is not installed (works for simple grammars without modes/predicates).

When to use:

• Test if lexer rules match input as expected

• Debug tokenization issues (especially with complex grammars!)

• Verify grammar changes work correctly

• Learn how ANTLR4 tokenizes input

Setup for 100% accuracy (optional):

1. Install Java: brew install openjdk

2. Install ANTLR4: wget https://www.antlr.org/download/antlr-4.13.1-complete.jar

3. Set env: export ANTLR4_JAR=/path/to/antlr-4.13.1-complete.jar

How it works:

• Native mode: Compiles and executes actual ANTLR4 lexer (100% accurate)

• Simulation mode: Best-effort tokenization (works for ~70% of grammars)

Example - Test basic tokenization: input: "x = 42;"

Example - Test with complex lexer (Palo Alto): from_file: "PaloAlto_lexer.g4" load_imports: true input: "set user-id-collector enable-mapping-timeout 1"

Example - Test specific rules only: input: "x + y * 2" rules_to_test: ["ID", "PLUS", "TIMES", "INT"]

Returns:

• List of tokens with types and values

• Character positions (start, end, line, column)

• Channel information for channeled tokens

• Errors for unmatched characters

• Mode indicator (🚀 Native or ⚠️ Simulation)

• Feature warnings if simulation used

Limitations: ❌ Lexer modes not supported ❌ Semantic predicates not evaluated
❌ Actions not executed ⚠️ Fragment rules (basic support) ⚠️ Complex patterns (best-effort)

Alternative: For complex grammars, use test-parser-rule which tests parser rules without full lexer simulation.

Note: This is a simplified simulation. For 100% accuracy, use ANTLR4 tooling.

Profile grammar parsing with detailed performance metrics.

When to use: Deep performance analysis, debugging slow parsing, optimizing grammars.

Measures:

• Parse time (ms)

• Token count

• Parse tree depth

• Decision evaluations (ATN transitions)

• Ambiguity count (conflicting alternatives)

• Context sensitivity (SLL→LL fallbacks)

• Rule invocation frequency

Parameters:

• grammar_files: Object mapping filename to content

• start_rule: Parser rule to start from

• input: Sample input text

Returns:

• Detailed profile metrics

• Most frequently invoked rules

• Optimization suggestions

Example: grammar_files: {"Expr.g4": "grammar Expr; ..."} start_rule: "program" input: "x = 1 + 2 * 3"

Interpretation:

• ambiguityCount > 0: Grammar has ambiguous alternatives

• contextSensitivityCount > 10: Many SLL→LL fallbacks (slow)

• treeDepth > 100: Excessive nesting

Remove a rule cleanly from an ANTLR4 grammar.

When to use: Delete obsolete rules, clean up unused definitions, or refactor grammar structure.

Example usage: rule_name: "oldExpression" write_to_file: true

Warning: Does not update references to this rule in other rules. Use find-rule-usages first to check impact.

Returns: Modified grammar with rule removed, success message, file write confirmation if applicable.

Safely rename a rule and automatically update ALL references throughout the grammar.

When to use: Refactor rule names for clarity, fix naming conventions, or improve code readability.

Example usage: old_name: "expr" new_name: "expression" write_to_file: true

Multi-file example: from_file: "MyGrammar.g4" base_path: "/path/to/grammars" load_imports: true write_to_file: true

Features:

• Uses whole-word matching (prevents "expr" from matching "subexpr")

• Updates rule definition and ALL references in other rules

• Preserves rule position in grammar

• Reports number of references updated

• Multi-file support: Set load_imports=true to rename across imported grammars

Recommended workflow:

1. find-rule-usages with load_imports=true to see full impact

2. rename-rule with load_imports=true to perform refactoring

3. validate-grammar to verify correctness

Returns: Modified grammar with rule and all references renamed, update count, file write confirmation if applicable.

Analyze rule complexity, dependencies, and performance characteristics.

When to use: Understand rule complexity, identify bottlenecks, find heavily-used rules, plan refactoring.

Example usage: rule_name: "expression"

Returns:

• Rule definition and type

• Complexity metrics: number of alternatives

• Fan-out: rules that this rule references (dependencies)

• Fan-in: rules that reference this rule (dependents)

• Recursion analysis: direct/indirect recursion detection

• Usage statistics

Use cases:

• Identify complex rules for optimization

• Find highly-coupled rules

• Detect recursion issues

• Plan refactoring priorities

Smart grammar validation with aggregated, actionable insights.

When to use: When validate-grammar returns too many issues and you need to see patterns, not individual warnings.

Improvements over basic validation:

• Groups similar issues (e.g., 15,000 undefined refs → "9 missing tokens")

• Prioritizes by impact (most-referenced undefined tokens first)

• Suggests specific fixes with reasoning

• Detects anti-patterns (null_rest_of_line usage)

• Flags suspicious quantifiers (? that should be *)

Example output: Summary: 17,234 issues

1. Undefined tokens (15,890 refs, 9 unique) → Add ADDRESS_REGEX (89 refs), EVENT_TYPE (67 refs), ...

2. Suspicious quantifiers (8 rules) → bgpp_export: bgp_policy_rule? should be *

3. Incomplete parsing (3 rules) → ss_ssl_tls_service_profile uses null_rest_of_line

Parameters:

• include_suggestions: Generate smart token suggestions

• detect_quantifiers: Flag suspicious ? patterns

• detect_incomplete: Flag null_rest_of_line usage

Returns: Aggregated summary, grouped issues, and actionable recommendations.

Reorder rules in a grammar according to various sorting strategies.

When to use:

• Clean up messy grammar files

• Organize rules logically

• Improve readability and maintenance

• Group related rules together

Sorting Strategies:

1. alphabetical (default)

   • Sorts parser rules alphabetically

   • Then sorts lexer rules alphabetically

   • Most common for general organization

2. type

   • Groups by rule type

   • parser_first: true (default) - parser rules first

   • parser_first: false - lexer rules first

3. dependency

   • Orders rules based on relationship to anchor rule

   • Requires anchor_rule option

   • Order: dependencies → anchor → dependents → rest

   • Useful for understanding rule relationships

4. usage

   • Most-referenced rules first

   • Helps identify "core" rules

   • Useful for understanding grammar structure

Example - Alphabetical: strategy: "alphabetical"

Example - Dependency-based: strategy: "dependency" anchor_rule: "expression"

Example - Type-based: strategy: "type" parser_first: false // Lexer rules first

Features:

• Preserves multi-line rule formatting

• Maintains blank lines after rules

• Preserves header (grammar declaration, imports, options)

• Handles all rule types (parser, lexer, fragment)

Returns:

• Reordered grammar

• Statistics (total rules, strategy used)

Parse error logs and automatically suggest missing tokens to add to grammar.

When to use:

• Debugging parser failures with error logs

• Identifying missing tokens from Batfish errors

• Analyzing ANTLR parse error output

• Incremental grammar development based on test failures

Supported error log formats:

1. Batfish-style: "unexpected token: 'word'"

2. ANTLR-style: "mismatched input 'word'"

3. ANTLR-style: "no viable alternative at input 'word'"

4. Generic: any quoted strings in error context

Example - Analyze Batfish error log: error_log: "Error parsing config: unexpected token: 'ftm-push' at line 10" → Suggests: FTM_PUSH token with pattern 'ftm-push'

Example - Parse ANTLR errors: error_log: "line 5:10 mismatched input 'admin' expecting {CONFIG, SYSTEM}" → Suggests: ADMIN token with pattern 'admin'

Features:

• Multi-format error log parsing

• Confidence scoring (high/medium/low)

• Automatic deduplication

• Skips tokens that already exist in grammar

• Provides reasoning for each suggestion

• Handles multiple errors in batch

Returns:

• List of suggested tokens with:

  • Token name (uppercase with underscores)

  • Pattern (string literal)

  • Reason for suggestion

  • Confidence level

• Summary of suggestions found

Note: This tool only suggests tokens. Use add-lexer-rules to actually add them to your grammar.

Test if input text matches a specific parser rule. Now with native ANTLR4 support!

🚀 Native ANTLR4 Support: Automatically uses native ANTLR4 runtime if available for 100% accurate parsing including:

• ✅ Lexer modes (pushMode, popMode)

• ✅ Semantic predicates ({...?})

• ✅ Actions ({...})

• ✅ All complex parser patterns

• ✅ Multi-file grammars with imports

Falls back to simulation if ANTLR4 is not installed.

When to use:

• Rapid iteration on rule syntax during development

• Verify if text matches a parser rule structure

• Test rules with complex features (modes, predicates)

• Test rules that reference imported tokens/rules

• Debug parsing issues in complex grammars

Setup for 100% accuracy (optional):

1. Install Java: brew install openjdk

2. Install ANTLR4: wget https://www.antlr.org/download/antlr-4.13.1-complete.jar

3. Set env: export ANTLR4_JAR=/path/to/antlr-4.13.1-complete.jar

How it works:

• Native mode: Compiles and executes actual ANTLR4 parser (100% accurate)

• Simulation mode: Best-effort matching (~70-90% accuracy)

Example - Test expression rule: rule_name: "expression" input: "x + y * 2"

Example - Test with multi-file grammar: from_file: "/path/to/MyParser.g4" base_path: "/path/to/grammar/dir" load_imports: true rule_name: "expression" input: "x + y"

Example - Show parse tree: rule_name: "statement" input: "if (x) y = 1;" show_tree: true

Returns:

• Match result (✅ matches or ❌ doesn't match)

• Parse tree (if show_tree enabled)

• Parse errors with line/column information

• Mode indicator (🚀 Native or ⚠️ Simulation)

Update an existing rule definition in place. Now supports multi-line definitions!

When to use: Modify rule logic, add alternatives, refine patterns, or fix issues.

Example - Simple update: rule_name: "expression" new_definition: "term ((PLUS | MINUS) term)*"

Example - Multi-line update (string with \n): rule_name: "srp_uuid_null" new_definition: "UUID\n | ~(\n ACTION\n | FROM\n )"

Example - Multi-line update (JSON array): rule_name: "srp_uuid_null" new_definition: ["UUID", " | ~(", " ACTION", " | FROM", " )"]

Multi-line support:

• Pass a string with embedded \n characters, OR

• Pass an array of strings (one per line) - more readable!

• Formatting is automatically preserved based on grammar style

Preserves rule position in grammar. Use rename-rule if changing the rule name.

Returns: Modified grammar with rule updated, success message, file write confirmation if applicable.

Validate ANTLR4 grammar syntax and detect common issues.

When to use: After making changes to verify correctness, or to diagnose problems in an existing grammar.

Example usage: from_file: "MyGrammar.g4"

Detects:

• Undefined rule references (rules used but not defined)

• Unused rules (defined but never referenced)

• Direct left recursion issues

• Fragment rule misuse

• Naming convention violations

Returns: List of issues with severity (error/warning/info), descriptions, line numbers, and affected rule names.

Visualize the parse tree structure for a given input.

When to use: Understanding parse results, debugging grammar structure, documentation.

Output formats:

• ascii: Text-based tree with indentation (default)

• json: Structured JSON tree representation

• lisp: S-expression style (rule child1 child2 ...)

Parameters:

• grammar_files: Object mapping filename to content

• start_rule: Parser rule to start from

• input: Sample input text

• format: Output format (ascii, json, lisp)

Example: grammar_files: {"Expr.g4": "grammar Expr; ..."} start_rule: "expr" input: "1 + 2 * 3" format: "ascii"

Returns: ASCII example:

expr
├── term
│   └── factor
│       └── NUMBER '1'
├── PLUS '+'
└── term
    ├── factor
    │   └── NUMBER '2'
    ├── TIMES '*'
    └── factor
        └── NUMBER '3'