Enables AI agents to inspect and drive a running Blender application's Qt-based UI: find widgets by meaning, take screenshots, record GIFs, click/type/press keys, inspect widget trees and properties, and route requests to multiple running Blender instances.
Enables AI agents to inspect and drive a running Houdini application's Qt-based UI: find widgets by meaning, take screenshots, record GIFs, click/type/press keys, inspect widget trees and properties, and route requests to multiple running Houdini instances.
Enables AI agents to inspect and drive a running Nuke application's Qt-based UI: find widgets by meaning, take screenshots, record GIFs, click/type/press keys, inspect widget trees and properties, and route requests to multiple running Nuke instances.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@qtpy-mcp screenshot the 'Name goes here:' form in the DEV-1515 window"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
MCP server that lets an AI agent find, screenshot, GIF-record and drive Qt widgets in running applications and DCCs (Houdini, Blender, Maya, Nuke, ...), built on Qt.py so it works with PySide6, PyQt6, PySide2 and PyQt5.
It is a fork of 0xCarbon/qt-mcp (MIT OR Apache-2.0): same tool set, plus Qt.py, many-apps-at-once routing, semantic widget search and GIFs.
"screenshot the PIPE-1234 window's docked Render Queue" ┌────────────────────┐
"now the widget labelled 'Name goes here:' in DEV-1515" ──► │ Houdini pid 4121 │ probe
"record a GIF of Houdini process 1252151's 'Foo bar'" ├────────────────────┤
│ │ Houdini pid 1252 │ probe
MCP client ──stdio──► qtpy-mcp ── registry + JSON-RPC ───► ├────────────────────┤
(routes) (127.0.0.1, token) │ Blender pid 7770 │ probe
└────────────────────┘Find by meaning, not by ref: text, window title, class, objectName, placeholder,
tooltip, whatsThis, statusTip, accessible name, tab text, dynamic properties, even
style sheet. Fuzzy-ranked; each hit says which field matched.
Screenshot a whole window, a dialog, or any sub-widget such as a docked panel. A match on a
label like Name goes here: captures the form around it (expand="auto"; also self,
parent, buddy, window).
Record a GIF (qt_record_gif, or qt_record_start / qt_record_stop while you interact).
The probe grabs frames with QWidget.grab() and Pillow encodes them, so it behaves the same
on Windows, Linux and macOS: no ffmpeg or screen-capture APIs.
Many applications at once. Several Houdinis, Blender and Maya can be open together; one agent connection addresses any of them (see below).
Everything upstream provides: widget tree snapshots, properties, click / type / keys, menus, signals, layout checks, QGraphicsScene and VTK inspection.
Related MCP server: kwin-mcp
Two pieces: the probe lives inside the Qt application; the server runs next to your MCP client.
# MCP client side (any Python 3.10+):
uvx qtpy-mcp # or: pip install qtpy-mcp
# Inside a DCC, whose Python already has a Qt binding: install without dependencies
hython -m pip install --no-deps qtpy-mcp Qt.py # Houdini
<blender-python> -m pip install --no-deps qtpy-mcp Qt.py PySide6-Essentials # BlenderStandalone apps can use an extra: pip install "qtpy-mcp[pyside6]" (or pyqt6, pyqt5, pyside2).
Choose a binding with QT_PREFERRED_BINDING=PyQt6 (Qt.py's own variable).
MCP client config:
{ "mcpServers": { "qtpy": { "command": "uvx", "args": ["qtpy-mcp"] } } }from qtpy_mcp.probe import install
install(name="houdini") # after the QApplication existsor set QTPY_MCP_PROBE=1 in the environment before launching (no code change).
Working examples: examples/houdini,
examples/blender.
Every tool takes an optional instance. The server finds running apps through a per-user
registry, so apps may start and stop at any time. A reference can be any of:
You say |
|
a window title or part of one |
|
a process id |
|
an app name (if unique) |
|
the instance id |
|
If a reference matches more than one app, the tool errors with the candidates (pid, name,
window titles) rather than guessing. With one app running, instance can be omitted.
qt_list_instances shows what is running; qt_find_anywhere searches every app for a
description and names the owner of each hit; qt_select_instance sets a default.
Routing is covered by tests against fake probes, and against real separate Qt processes
(tests/test_multiprocess.py).
Instances |
|
Finding |
|
Pictures |
|
Driving |
|
Inspecting |
|
Variable | Meaning |
| auto-install the probe when a |
| label for this instance (default: application name) |
| fixed probe port (default: ephemeral); on the server side, connect to this one port and skip the registry |
| host the server connects to in fixed-port mode |
| shared secret instead of a random one |
| where instances register (default: per-user runtime dir) |
| cap screenshot size |
| Qt.py: |
The probe can click, type and call slots in the host application. It listens on 127.0.0.1
only, on an ephemeral port, and every request needs a per-process token kept in a registry
file readable only by your user. See SECURITY.md.
See CONTRIBUTING.md. Quality gates in CI: ruff, mypy --strict, tests on
PySide6 / PyQt6 / PyQt5 (Linux, Windows, macOS), a Blender smoke test, build checks and
pip-audit. Pushing a vX.Y.Z tag publishes to PyPI.
MIT OR Apache-2.0, at your option (LICENSE-MIT, LICENSE-APACHE).
Portions derive from qt-mcp; see NOTICE.
Check for active popup or modal dialog widgets.
Inputs: instance. Returns: result
Execute multiple Qt operations in a single round trip.
Chains clicks, types, key presses, waits, snapshots, and reads into one server call — dramatically reduces latency compared to sequential tool calls.
Each step dict: method (str, required): RPC method — same names as probe methods: click, type_text, key_press, set_property, invoke_slot, get_text, trigger_action, wait_for, snapshot, find_widget, screenshot. Special: "wait" with params {"ms": N} sleeps N ms processing events. params (dict, optional): Parameters forwarded to the method. wait_ms (int, optional): Extra pause after the step (default 0).
Execution stops on the first error; partial results are returned. Screenshot steps return size only (not base64 data) — call qt_screenshot separately when you actually need the image.
Note: refs must be known before calling qt_batch — steps cannot reference results from earlier steps within the same batch. Use qt_find_widget first to obtain refs, then pass them into the batch call.
Example — click a button twice then read a label in one round trip: qt_batch([ {"method": "click", "params": {"ref": "w5"}}, {"method": "click", "params": {"ref": "w5"}}, {"method": "get_text", "params": {"ref": "w12"}}, ])
Example — type multi-line code into IPython console and wait for output: qt_batch([ {"method": "type_text", "params": {"ref": "w42", "text": "%run /tmp/x.py", "use_clipboard": true}}, {"method": "key_press", "params": {"key": "Return"}}, {"method": "wait", "params": {"ms": 3000}}, {"method": "snapshot", "params": {"max_depth": 4}}, ])
Inputs: steps (required), instance. Returns: result
Click a widget.
Args: ref: Widget ref from qt_snapshot. button: Mouse button - 'left', 'right', or 'middle'. modifiers: Keyboard modifiers - 'shift', 'ctrl', 'alt', 'meta'. position: [x, y] relative to widget top-left. Default: center.
Inputs: ref (required), button, instance, position, +1 more. Returns: result
Semantic widget search across ALL running Qt apps at once.
Use when you do not know which app has the widget. Each hit names its instance; pass
that id as instance to qt_screenshot, qt_click, ... together with the ref.
Inputs: describe, min_score, class_name, max_results, +4 more. Returns: result
Search the widget tree for widgets matching criteria. Prefer over qt_snapshot.
Two modes. With describe and/or window_title it is a semantic search: describe
is matched fuzzily (typos, punctuation and case ignored) against label/button text,
group and dock titles, tooltip, what's-this, status tip, accessible name/description,
objectName, placeholder, tab titles, dynamic properties, class name and style sheet;
results are ranked with the reason each matched. Otherwise the plain substring filters
below apply.
Registers found widgets in the ref registry — returned refs are immediately usable with qt_click, qt_type, qt_get_text, etc.
Args:
pattern: Case-insensitive substring matched against class name, objectName,
or text content (OR logic across fields).
class_name: Exact class name, case-insensitive (e.g. 'QLineEdit', 'QPushButton',
'ControlWidget'). Must match the full class name, not a substring.
object_name: Substring match on objectName.
text: Substring match on widget text/label content.
root_ref: Restrict search to subtree of this widget.
visible_only: Only return visible widgets (default True).
max_results: Cap on results returned (default 20).
describe: Free-text description, e.g. 'Name goes here:' or 'save the profile'.
window_title: Fuzzy match on the top-level window title (e.g. 'PIPE-1234').
search_fields: Limit describe to some of: text, title, placeholder, tooltip,
whats_this, status_tip, accessible_name, accessible_description, object_name,
window_title, tab_text, properties, class_name, style_sheet.
min_score: Minimum semantic match score 0-1 (default 0.5).
At least one of pattern, class_name, object_name, text, describe or window_title must be provided.
Inputs: text, pattern, describe, instance, +8 more. Returns: result
Extract text content from a text editor or input widget.
Args: ref: Widget ref from qt_snapshot (e.g., 'w5').
Works with QPlainTextEdit, QTextEdit, QLineEdit, QLabel, QComboBox, and any widget with a text() or toPlainText() method.
Inputs: ref (required), instance. Returns: result
Invoke a slot or method on a QObject.
Args: ref: Widget ref from qt_snapshot. method_name: Slot/method name to invoke.
Inputs: ref (required), method_name (required), instance. Returns: result
Send a key event to a widget or the focused widget.
Args: key: Key name (e.g., 'Return', 'Escape', 'Ctrl+S', 'a'). ref: Widget ref. If omitted, sends to the currently focused widget.
Inputs: key (required), ref, instance. Returns: result
Detect layout issues in the visible widget tree.
Scans all visible widgets and reports problems like:
zero_size: Visible widget with 0x0 geometry (collapsed)
no_layout: Container with visible children but no layout manager
smaller_than_hint: Widget smaller than its sizeHint (content likely clipped)
text_truncated: Text content wider than widget width
overlapping_siblings: Sibling widgets with intersecting geometries
Inputs: instance. Returns: result
List running Qt apps that have the qtpy-mcp probe (Houdini, Blender, Maya, ...).
Shows each instance id, process, binding and visible window titles. Use the id with
the instance argument of any tool, or with qt_select_instance.
Inputs: no named parameters. Returns: result
List all top-level windows and their types.
Inputs: instance, skip_hidden. Returns: result
Returns all console messages
Args: level: Level of the console messages to return. Each level includes the messages of more severe levels. Defaults to "info".
Inputs: level, instance. Returns: result
Get the full QObject parent-child tree (not just visible widgets).
Args: root_ref: Starting ref. Defaults to QApplication root. max_depth: Maximum traversal depth.
Inputs: instance, root_ref, max_depth. Returns: result
Record an animated GIF of a window or widget for duration seconds.
Identify the widget exactly like qt_screenshot (ref, or describe/window_title/ class_name/object_name; the active window if nothing is given). The app keeps running normally while recording; interact with it meanwhile via other tools or by hand. Works on Windows and Linux (frames come from Qt's own rendering, no screen capture needed). For longer or interleaved recordings use qt_record_start / qt_record_stop.
Inputs: fps, ref, index, expand, +9 more. Returns: result
Start a background GIF recording; stop it later with qt_record_stop.
Returns a recording id that already names the app, so qt_record_stop needs no
instance. Recording stops by itself after max_seconds.
Inputs: fps, ref, index, expand, +9 more. Returns: result
Stop a recording started with qt_record_start and return the GIF.
Args: recording_id: The id returned by qt_record_start (includes the app it belongs to).
Inputs: recording_id (required). Returns: result
Get all items in a QGraphicsScene.
Args: ref: Ref to a QGraphicsView widget.
Inputs: ref (required), instance. Returns: result
Take a screenshot of a window or any widget, identified by ref OR by description.
Identify the widget semantically instead of by ref, e.g. describe="Name goes here:"
(a label: captures the widget that holds it), describe="save the profile" (tooltip),
window_title="PIPE-1234" with class_name="QDockWidget" (a docked widget), or
object_name="nameEdit". Add instance (pid, id or window title of the app) when
several Qt apps are running.
Args:
ref: Widget ref from qt_snapshot/qt_find_widget. If nothing identifies a widget,
the active window is captured.
full_window: If True, captures the first visible top-level window.
max_width: Max width before downscaling (default 1920).
max_height: Max height before downscaling (default 1080).
format: Image format - 'png' or 'jpeg' (default 'png').
quality: JPEG quality 1-100 (default 80, ignored for PNG).
describe: Free-text description matched against text, titles, tooltip, what's-this,
status tip, accessible name, objectName, placeholder, QSS ... (fuzzy).
window_title: Fuzzy match on the top-level window title.
class_name: Qt class, subclasses match (e.g. 'QDockWidget').
object_name: objectName, supports * and ? wildcards.
search_fields: Limit describe to specific fields (see qt_find_widget).
expand: What to capture relative to the match: 'auto' (a matched label captures the
widget holding it; anything else captures itself), 'self', 'parent', 'buddy'
(the label's field) or 'window'.
index: Which ranked match to use (0 = best).
min_score: Minimum semantic match score 0-1.
Returns an image plus a note saying what was captured.
Inputs: ref, index, expand, format, +11 more. Returns: result
Make one running Qt app the default target for later tool calls.
Args: instance: Instance id, pid, name, or words from a window title, e.g. "PIPE-1234".
Inputs: instance (required). Returns: result
Set a Qt property on a widget.
Args: ref: Widget ref from qt_snapshot. property_name: Qt property name. value: New value for the property.
Inputs: ref (required), value (required), property_name (required), instance. Returns: result
Inspect signal connections on a QObject.
For a given widget/object, enumerates all signals and reports which are connected and how many receivers each has. Useful for debugging "nothing happens when I click" problems.
Args: ref: Widget ref from qt_snapshot (e.g., 'w5').
Inputs: ref (required), instance. Returns: result
Capture the full Qt widget tree as a structured accessibility-like snapshot.
Returns a YAML-like text tree with widget types, object names, text content, geometry, visibility, enabled state, and interaction refs (w1, w2, ...). Use the refs to interact with specific widgets via other tools.
Inputs: instance, root_ref, max_depth, skip_hidden. Returns: result
Check thread affinity of all QObjects to detect threading issues.
Reports which threads own QObjects, flags QWidget subclasses on non-GUI threads (the #1 cause of GUI freezes), and lists QThread instances.
Inputs: instance. Returns: result
Trigger a menu action or toolbar action by text or index.
Args: ref: Ref to a QMenu, QMenuBar, QToolBar, or any widget with actions. action_text: The action's display text (e.g., 'Save', '&Open'). Ampersands are stripped for matching. action_index: The 0-based index of the action (matching qt_menu_items order).
Provide either action_text or action_index (not both). Use qt_menu_items first to see available actions.
Inputs: ref (required), instance, action_text, action_index. Returns: result
Type text into a widget.
Args: ref: Widget ref from qt_snapshot. text: Text to type. clear_first: If True, select all and delete before typing. use_clipboard: If True, insert via Ctrl+V paste instead of key-by-key events. Use this for multi-line text in console/terminal widgets — character-by-character typing treats each newline as Enter (submit), corrupting multi-line input.
Inputs: ref (required), text (required), instance, clear_first, +1 more. Returns: result
Get VTK scene state from a PyVista/VTK widget.
Args: ref: Ref to a widget containing a VTK render window.
Inputs: ref (required), instance. Returns: result
Capture a VTK render window to an image.
Args: ref: Ref to a widget containing a VTK render window.
Inputs: ref (required), instance. Returns: result
Wait for a UI state change.
Args: condition: One of 'widget_visible', 'window_count_changed', 'property_equals'. timeout_ms: Max time to wait in milliseconds (default 5000). object_name: Widget objectName (for widget_visible). ref: Widget ref (for property_equals). property_name: Property name (for property_equals). value: Expected value (for property_equals).
Inputs: condition (required), ref, value, instance, +3 more. Returns: result
Get detailed properties of a specific widget.
Args: ref: Widget ref from qt_snapshot (e.g., 'w5').
Returns all Qt properties, geometry, parent chain, and more.
Inputs: ref (required), instance. Returns: result
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
Scored across 30 tools
Most tools target clearly distinct operations, and the long descriptions actively steer selection (e.g. qt_find_widget vs qt_find_anywhere, qt_record_gif vs qt_record_start/stop, qt_screenshot vs qt_vtk_screenshot). A few pairs still require reading the docs to separate: qt_snapshot vs qt_object_tree, qt_scene_snapshot vs qt_snapshot, and qt_widget_details vs qt_snapshot.
All tools share a qt_ prefix and snake_case with verb-first names (qt_find_widget, qt_click, qt_set_property, qt_trigger_action, qt_wait_for), which is highly predictable. A handful drop the verb and are noun phrases (qt_object_tree, qt_menu_items, qt_widget_details, qt_active_popup, qt_messages), a minor inconsistency in an otherwise uniform scheme.
30 tools is above the comfortable range and includes consolidation candidates: the GIF trio (qt_record_gif / qt_record_start / qt_record_stop), the VTK pair, and the two find tools. The scope (introspection, interaction, recording, multi-instance, threads/signals, VTK) is genuinely broad, so most tools do earn their place, but the surface is heavier than ideal.
Coverage is unusually thorough: discovery (snapshot, find, object tree, windows, popups), interaction (click, type, key press, set property, invoke slot, trigger action), async (wait, batch), diagnostics (layout, threads, signals, messages), plus VTK and recording. Minor gaps remain around drag-and-drop, explicit scrolling, and app/window lifecycle (launch/close), though these are largely workaroundable via key_press or invoke_slot.
A visual brain for your agents: live, reusable UI fragments and skills to pull into code.
Connect ChatGPT, Claude, Cursor or VS Code to your Windows PC: files, terminal, browser, apps.
Connect AI assistants to your approved computer for files, commands, browser and desktop tasks.
Connect your AI assistant to Blender for 3D modeling, Geometry Nodes, materials, lighting, animation, rendering and export. Inspect real scenes and viewport images, refine the result and restore checkpoints. OAuth connects your chat to Blender Bridge Desktop on an online Windows 10/11 x64 computer running Blender 4.2+. The chat can run on another device. Requires an active Blender Bridge account; 7-day trial available. AI model access is separate.
