How do I use Qt Pilot?

1. Click on "Install Server". 2. Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state. 3. In the chat, type @ followed by the MCP server name and your instructions, e.g., "@Qt Pilot Launch myapp.py, click the 'submit_button', and capture a screenshot of the result." That's it! The server will respond to your query, and you can continue using it as needed. Here is a step-by-step guide with screenshots.

Qt Pilot

License: MIT

An MCP server for headless Qt/PySide6 GUI testing. Enables AI assistants like Claude to visually test and interact with Qt desktop applications.

Repository: github.com/neatobandit0/qt-pilot

Features

Launch Qt apps headlessly via Xvfb virtual display
Capture screenshots for visual verification
Simulate interactions: clicks, hovers, keyboard input
Widget discovery by object name
App health monitoring with stderr capture
Full Qt introspection via QTest and Qt APIs

Installation

From GitHub

git clone https://github.com/neatobandit0/qt-pilot.git ~/.claude/plugins/qt-pilot
pip install -r ~/.claude/plugins/qt-pilot/requirements.txt

Manual Installation

Copy the plugin to your Claude plugins directory:

cp -r qt-pilot ~/.claude/plugins/

Then add to your ~/.claude.json:

{
  "mcpServers": {
    "qt-pilot": {
      "type": "stdio",
      "command": "python3",
      "args": ["/path/to/qt-pilot/server/main.py"]
    }
  }
}

Dependencies

pip install mcp PySide6

Also requires Xvfb for headless display:

# Debian/Ubuntu
sudo apt install xvfb

# RHEL/CentOS/Fedora
sudo yum install xorg-x11-server-Xvfb

# macOS (via Homebrew)
brew install xquartz

MCP Tools

`launch_app`

Launch a Qt application headlessly.

# Script mode
launch_app(script_path="/path/to/test_gui.py")

# Module mode
launch_app(module="myapp.main", working_dir="/path/to/project")

`capture_screenshot`

Capture the current window.

capture_screenshot(output_path="/tmp/screenshot.png")

`click_widget`

Click a widget by its object name.

click_widget(widget_name="submit_button", button="left")

`hover_widget`

Hover over a widget.

hover_widget(widget_name="menu_item")

`type_text`

Type text into a widget or focused widget.

type_text(text="hello world", widget_name="search_input")
type_text(text="hello")  # Types into currently focused widget

`press_key`

Simulate a key press with optional modifiers.

press_key(key="Enter")
press_key(key="S", modifiers=["Ctrl"])  # Ctrl+S
press_key(key="Tab")

`find_widgets`

List widgets matching a name pattern.

find_widgets(name_pattern="*")  # All named widgets
find_widgets(name_pattern="btn_*")  # Widgets starting with "btn_"

`get_widget_info`

Get detailed widget information.

get_widget_info(widget_name="submit_button")
# Returns: type, visible, enabled, size, position, text, checked state, etc.

`get_app_status`

Check if the application is still running and get diagnostics.

get_app_status()
# Returns: {"running": true, "exit_code": null, "stderr": "", "display": ":99"}

`wait_for_idle`

Wait for the Qt event queue to settle after actions.

click_widget(widget_name="load_button")
wait_for_idle(timeout=5.0)  # Wait for async operations to complete
capture_screenshot()

`close_app`

Close the running application.

close_app()

Requirements for Target Applications

For widget interactions to work, your Qt application must:

Set object names on interactive widgets:

button = QPushButton("Click Me")
button.setObjectName("my_button")  # Required for widget discovery

Use QApplication (not QCoreApplication)
Show at least one window

Architecture

┌─────────────────────────────┐
│  AI Assistant (Claude)      │
└─────────────┬───────────────┘
              │ MCP Protocol (stdio)
              ▼
┌─────────────────────────────┐
│  MCP Server (main.py)       │
│  - Tool definitions         │
│  - Process management       │
└─────────────┬───────────────┘
              │ Unix Socket (IPC)
              ▼
┌─────────────────────────────┐
│  Test Harness (harness.py)  │
│  - Runs inside Xvfb         │
│  - QTest interactions       │
│  - Widget introspection     │
├─────────────────────────────┤
│  Your Qt Application        │
└─────────────────────────────┘

Example Workflow

# 1. Launch a test app
launch_app(module="myapp.main", working_dir="/path/to/project")

# 2. List available widgets
find_widgets()

# 3. Interact with the UI
click_widget(widget_name="login_button")
wait_for_idle()

# 4. Type into a field
type_text(text="user@example.com", widget_name="email_input")
press_key(key="Tab")
type_text(text="password123", widget_name="password_input")

# 5. Submit and capture result
click_widget(widget_name="submit_button")
wait_for_idle(timeout=3.0)
capture_screenshot(output_path="/tmp/result.png")

# 6. Clean up
close_app()

Troubleshooting

Ensure the widget has setObjectName() called
Use find_widgets() to list available widget names

"No app is running"

Call launch_app() first
Check that the script/module path is correct

App crashes silently

Use get_app_status() to check for errors
The stderr field contains crash information

Screenshots are blank

Ensure the application creates and shows a window
Use wait_for_idle() after launch for window to render

License

MIT License - see LICENSE file.

Qt Pilot

Qt Pilot

Features

Installation

From GitHub

Manual Installation

Dependencies

MCP Tools

`launch_app`

`capture_screenshot`

`click_widget`

`hover_widget`

`type_text`

`press_key`

`find_widgets`

`get_widget_info`

`get_app_status`

`wait_for_idle`

`close_app`

Requirements for Target Applications

Architecture

Example Workflow

Troubleshooting

"Widget not found"

"No app is running"

App crashes silently

Screenshots are blank

License

Resources

Looking for Admin?

Latest Blog Posts

MCP directory API