Scenic MCP

Scenic MCP - Input Control for Scenic Applications

A Model Context Protocol (MCP) server that enables external keyboard and mouse input injection into Scenic GUI applications.

Features

• Keyboard Input: Send text and special keys to Scenic applications
• Mouse Control: Move cursor and click at specific coordinates
• Visual Feedback: Get descriptions of what's displayed on screen (v0.2.0)
• MCP Integration: Works with any MCP-compatible client (Claude Desktop, etc.)
• Real-time Communication: TCP-based connection for low-latency input
• Scenic Compatible: Uses proper Scenic ViewPort input routing

Installation

1. Add to your Scenic application's mix.exs:

2. Configure your Scenic viewport with proper naming:

IMPORTANT: For scenic_mcp to work correctly, your Scenic application MUST name both the viewport and driver. Update your scenic_config() function:

3. Install Node.js dependencies and build:

4. Configure for Claude Code:

Note: Replace /path/to/scenic_mcp with the actual path to your scenic_mcp directory.

Usage

Using with Claude Code

Once configured, you can use the MCP tools directly within Claude Code conversations:

1. Start your Scenic application (with the ScenicMcp.Server running on port 9999)
2. Use MCP tools in Claude Code:
   • connect_scenic - Test connection to your Scenic app
   • get_scenic_status - Check connection status
   • send_keys - Send keyboard input
   • send_mouse_move - Move mouse cursor
   • send_mouse_click - Click at coordinates
   • inspect_viewport - Get visual description of current screen

Example conversation:

MCP Tools

The server provides these MCP tools:

connect_scenic

Test connection to the Scenic application.

get_scenic_status

Check server status and available commands.

send_keys

Send keyboard input to the Scenic application.

Parameters:

• text (string): Text to type (each character sent as individual key press)
• key (string): Special key name (enter, escape, tab, backspace, delete, up, down, left, right, home, end, page_up, page_down, f1-f12)
• modifiers (array): Modifier keys (ctrl, shift, alt, cmd, meta)

send_mouse_move

Move mouse cursor to specific coordinates.

Parameters:

• x (number): X coordinate
• y (number): Y coordinate

send_mouse_click

Click mouse at specific coordinates.

Parameters:

• x (number): X coordinate
• y (number): Y coordinate
• button (string): Mouse button (left, right, middle) - default: left

get_scenic_graph (NEW in v0.2.0)

Return the script table for a ViewPort, providing a visual description of the scene.

take_screenshot

Capture a screenshot of the current Scenic display.

Parameters:

• filename (string, optional): Custom filename for the screenshot
• format (string, optional): Output format - "path" (default) or "base64"

Examples

Send text:

Send special key:

Send key with modifiers:

Move mouse:

Click mouse:

Get visual feedback:

Architecture

Key Components

• ScenicMcp.Server: TCP server that receives commands from the TypeScript bridge
• ScenicMcp.Probes: Direct interface to Scenic internals, sends input via Scenic.Driver.send_input/2
• Process Names: Uses :main_viewport and :scenic_driver registered process names

Development

Start the Elixir server:

Test the MCP server:

Testing

This project includes comprehensive testing to ensure reliability and guide improvements.

Test Suites

1. Elixir Unit Tests - Core server functionality
   mix test mix test --cover # With coverage reporting
2. TypeScript Tests - MCP integration testing
   npm test npm run test:coverage # With coverage npm run test:watch # Watch mode
3. LLM Tool Testing - Validates tool descriptions and usability
   npm run test:llm-tools

Testing Innovation: LLM-Driven Tool Enhancement

One of the unique aspects of this project is how we use LLM testing to improve the tool descriptions:

1. Tool Usage Analysis: We run scenarios through LLMs to see which tools they discover and use
2. Description Enhancement: Based on usage patterns, we automatically enhance tool descriptions
3. Real-World Validation: The enhanced descriptions are tested against real development scenarios

This approach has led to significant improvements in tool discoverability and correct usage by AI assistants.

Running All Tests

For detailed testing documentation, see TESTING.md.

Requirements

• Elixir/OTP 24+
• Node.js 18+
• Scenic 0.11+

License

MIT License