Skip to main content
Glama
qt-commander
by tiefeiyu

qt-commander

Live Qt UI introspection and automation for AI agents.

qt-commander links a small in-process runtime into a Qt application, exposes its live object tree over a local JSON-RPC connection, and drives it through an MCP server. Queries are evaluated in the target process every time; there are no snapshot files and no stale index of the UI.

Install

The first PyPI release is being prepared. Until it is out, install from source (option 2); the command line below already exists in this repository.

Requirements

  • Python 3.10+ and uv

  • A Qt 5 or Qt 6 target application. Verified on Qt 5.15.2 and Qt 6.8.3; other Qt 5.x/6.x releases are expected to work but are not yet validated.

  • Windows or Linux

  • Python is only needed for the MCP server; the target process does not need Python or gdb

Option 1 — package plus prebuilt runtime

pip install qt-commander-mcp
qt-commander doctor            # what this machine offers
qt-commander install-artifacts # injector + runtime for this OS/architecture

install-artifacts downloads the release archive, verifies SHA256SUMS, and unpacks it into the artifact root (QT_COMMANDER_ARTIFACTS, or a per-user directory for a wheel install). No C++ compiler is needed.

Option 2 — from source

git clone https://github.com/tiefeiyu/qt-commander.git
cd qt-commander
uv sync
uv run qt-commander doctor

Source-checkout artifacts built with qt_build or scripts/package_artifacts.py land in artifacts/ next to the checkout.

Register it with your MCP client

Claude Code:

claude mcp add qt-commander -- qt-commander-mcp          # pip install
claude mcp add qt-commander -- uv run --project /absolute/path/to/qt-commander qt-commander-mcp   # source install

Claude Desktop / Cursor / other JSON clients:

{
  "mcpServers": {
    "qt-commander": {
      "command": "qt-commander-mcp"
    }
  }
}

For a source checkout use "command": "uv" with "args": ["run", "--project", "/absolute/path/to/qt-commander", "qt-commander-mcp"]. Use a native absolute path (D:/Develop/qt-commander is fine on Windows). Restart the client after changing the config; qt_status should then list descriptors or live sessions.

3. Connect a target

  • Linked runtime: build/instrument the application or test app once (see Build the linked runtime); it writes a descriptor.

  • Already-running target: call qt_attach(pid=<pid>, mode="injection"); use qt_processes to discover a PID, then qt_target_info when the target's Qt version/toolchain is unknown. Build the matching injector and runtime if needed — normally qt_build(for_pid=<pid>), which derives every argument from the target and reports restart_required/next_step.

The usual first agent interaction is:

qt_status -> qt_attach -> qt_windows -> qt_query -> qt_act

Related MCP server: native-devtools-mcp

Status

  • Qt 5.15.2 verified with MinGW/MSVC; Qt 6.8.3 verified with both MinGW and MSVC for linked and injection. Ubuntu 24.04 + Qt 5.15.2 gcc_64 (XCB) verified for linked, the 99-test Python suite, and MCP stdio E2E.

  • Linked mode: the application calls qt_commander::start() after QApplication and writes a rendezvous descriptor.

  • Native injection (Windows and Linux): qt_attach(pid=..., mode="injection") invokes the C++ qtc-inject executable, which loads qt_commander_inject.dll/.so; the target-side library starts the same in-process server. auto injects when no live descriptor exists. Python is only the MCP orchestration layer; the target does not need Python or gdb. Linux ptrace still needs Yama permission (run the injector as the target's parent, or set kernel.yama.ptrace_scope=0). The native binary can be overridden with QT_COMMANDER_INJECTOR.

  • Launch-time injection (Linux and macOS): qt_launch(program, library=...) starts the application with the runtime preloaded (LD_PRELOAD / DYLD_INSERT_LIBRARIES), waits for its descriptor and attaches. No ptrace, no Yama configuration, and on macOS it is the only way in — the system refuses injection into an already-running hardened or notarized process. Windows has no preload hook, so qt_launch is not registered there.

  • 21 MCP tools are registered. Working today: status, target_info, processes, environments, build, build_status, attach, detach, windows, query, items, act, batch, wait, screenshot, get, set, call, methods, source, task. source searches the target's source tree (and QT_COMMANDER_SOURCE_DIR / an explicit source_dir) for matching .ui or .qml; unresolved targets return identity with E_NO_DEBUGINFO.

  • Every data request is executed from a per-server FIFO queue driven by the target's event loop, so a slot blocked in exec() no longer holds the socket: reads, acts and qt_task keep working on the same connection. qt_act/qt_call/qt_set take wait_ms (defaults 1000/3000/3000 ms); when it expires they return {"state": "running", "task_id": ...} and the result is stored for qt_task(task_id, wait_ms=...). Such a call has already been dispatched — never retry it. qt_ping reports queued/inflight/active for diagnostics.

  • items covers QTabWidget/QTabBar, QMenuBar/QMenu, QHeaderView, QTableView, QTreeView, QListView, QComboBox, QScrollBar, QSlider, QSpinBox, checkable QGroupBox, QDockWidget title buttons, QSplitter handles and QToolBar actions; custom/proxy models work through the item-view adapter.

  • QML deep objects under QQuickWidget are returned by query with qml_type/qml_id/objectName/local rect; use their vqml IDs with qt_get, qt_act, qt_screenshot and qt_source.

  • QML-only apps are first-class too: QQuickWindow roots work in qt_windows, qt_query, qt_items, qt_get, qt_set, qt_act (QML click/key/type) and qt_screenshot (window or element crop).

  • qt_query limit is a soft truncation: partial tree plus count/total/limit/truncated instead of a hard error.

  • qt_call invokes only Qt metaobject methods (slots / Q_INVOKABLE) on widgets and QML items; property setters such as QPushButton::setText are not methods — use qt_set instead. Non-void return values are returned in result; object returns use {class, id} so they can be reused as targets. qt_methods lists invokable methods, parameter types and return types (Qt _q_ internals are hidden unless include_internal=true). Pass signature to disambiguate overloads; enum arguments accept their integer values.

  • qt_target_info inspects a process's PE/ELF architecture, full Qt runtime version (e.g. 5.15.2), build type, and toolchain before qt_build; it does not inject or launch. detected_from says which evidence won (module — the Qt runtime actually mapped into the process, authoritative — adjacent, loader_path or descriptor). It also reports loaded_injector (path, artifact, build_id, stale), expected_artifact (what qt_attach would inject), injection_state (none/loaded/stale/mismatch) and restart_required. qt_processes lists discoverable running Qt targets with their attach hint, without injecting.

  • qt_source auto-discovers nearby source trees (including src ancestors) and accepts source_dir for qrc/embedded QML whose source tree is not next to the executable; it returns file/line/url when found.

  • qt_screenshot(annotate=...) supports annotate_offset for paging large annotation sets; legend n numbers stay continuous across pages.

  • qt_batch steps are exactly one of {"act": ...}, {"wait": ...}, {"set": ...}, {"get": ...}; this allows an ordered write/read/restore sequence (no implicit rollback).

  • qt_act(target=<window id>, action="minimize"|"restore") controls a QWindow/QQuickWindow; restore preserves the pre-minimize geometry. action="set_geometry" with geometry={x,y,w,h} reports the actual resolved.rect and an applied flag.

  • qt_act supports click/dblclick/type/focus/key/scroll/hover/drag and declarative expect (where/props/gone). type finds the first editable child for container targets and returns E_NOT_EDITABLE when none exists. Virtual items support click/dblclick.

  • Targets may use a plain string as a QML qml_id, or combine qml_id/text/class/objectName (child text is included). Text matches fold into the nearest actionable ancestor; e.g. a label inside a button clicks the button. Hidden targets are excluded by default and return E_HIDDEN unless include_hidden=true is passed.

  • Screenshots are capped at about 6 MiB at the MCP layer; oversized images are automatically re-captured at a smaller width (auto_downscaled).

Build the linked runtime

cmake -S tests/test-apps/linked -B build/linked -G Ninja \
  -DQT_MAJOR_VERSION=5 \
  -DQt5_DIR=/path/to/Qt/5.15.2/mingw81_64/lib/cmake/Qt5 \
  -DCMAKE_CXX_COMPILER=/path/to/mingw/g++.exe
cmake --build build/linked

MSVC users can instead call the MCP qt_build tool; it discovers Visual Studio via vswhere and loads vcvars64.bat automatically (or accepts vcvars_path).

On Linux with an official Qt kit, if Qt5GuiConfigExtras complains that GL/gl.h is missing under /usr/include/libdrm, add -D_qt5gui_OPENGL_INCLUDE_DIR=/usr/include to the configure command.

On Windows, put the Qt bin directory and the MinGW runtime on PATH, and point QT_PLUGIN_PATH at the Qt plugins directory. Then run:

$env:QT_COMMANDER_RENDEZVOUS = "$PWD\.test-workspace\rendezvous"
.\build\linked\qt-linked-test.exe

The app writes descriptor-<pid>.json into that directory (one file per process; owner-only permissions on POSIX).

Injection artifacts

qt_build writes injectors and injection libraries into one root only:

artifacts/<os>-<arch>/<compiler>-<version>/qt-<qt-version>-<debug|release>/
  manifest.json
  bin/qtc-inject.exe
  lib/qt_commander_inject.dll   # libqt_commander_inject.so on Linux

Examples:

artifacts/windows-x64/msvc-19.29/qt-5.15.2-debug/
artifacts/windows-x64/mingw-gcc-13.1/qt-5.15.2-release/
artifacts/linux-x64/gcc-13.3/qt-6.8.3-release/

manifest.json records build_id in the canonical qt<version>/<compiler>-<arch>/<build_type> form (for example qt5.15.2/msvc-x64/Release), the same string qt_build returns. The runtime's descriptor carries the compiled-in id instead (qt5/MSVC-AMD64/Release), so do not compare the two as plain strings; use qt_target_info.loaded_injector.build_id against expected_artifact.build_id.

When the artifact was built from a git checkout, the manifest also carries a source block with the release version and the commit it came from; qt_target_info and qt-commander doctor --pid print it.

qt_attach(mode="injection") matches the target's OS/architecture, compiler family and version, Qt version, and Qt debug/release against a manifest under artifacts/; it does not scan build/ or pick the newest file. Set QT_COMMANDER_INJECTOR / QT_COMMANDER_INJECT_DLL only for an explicit override.

Artifacts come from one of three places, in order of convenience:

  1. qt-commander install-artifacts — downloads qt-commander-artifacts-<os>-<arch>-*.zip from the latest GitHub release, verifies SHA256SUMS, and unpacks it. --from-dir/--from-url cover offline machines, --list shows what a release carries.

  2. qt_build(for_pid=<pid>) from an agent or MCP client — compiles against the Qt kit of a live target.

  3. scripts/package_artifacts.py for a build you made yourself with CMake.

The artifact root is artifacts/ in a source checkout. For an installed wheel it is read-only, so install-artifacts falls back to a per-user directory; set QT_COMMANDER_ARTIFACTS to pin one location explicitly (the MCP server and the CLI both honour it).

The Qt debug/release flavour comes from evidence, never from a guess: the Qt runtime actually mapped into the process wins, then the executable's own directory, then the loader PATH, then the descriptor. A directory holding both Qt5Core.dll and Qt5Cored.dll (a full Qt kit bin does) reports debug: null with flavor_ambiguous: true, and artifact selection then asks for an explicit artifact_dir instead of silently picking release. Picking the wrong flavour injects a DLL that links the other Qt runtime, loads a second Qt into the target, and times out waiting for QCoreApplication — which is why qt_build refuses to default an unknown build_type.

Run the MCP server

uv sync
uv run --project . qt-commander-mcp

If Windows reports qt-commander-mcp.exe is in use, stop the running Claude/MCP process first, then rerun uv run.

Point the MCP client at the descriptor with qt_attach(descriptor=...), or set QT_COMMANDER_RENDEZVOUS before starting the server. For an already-running Windows Qt app, use qt_attach(pid=<pid>, mode="injection"); the DLL path can be overridden with QT_COMMANDER_INJECT_DLL.

The MCP server carries an instructions block for agent startup, target selection, ambiguity/hidden handling, and destructive-action safety; each tool also has a schema-aware description.

Known limitations

  • Qt Quick Controls Popup roots (for example indicatorPopup) may not be discoverable by qml_id; query their text or child items instead. A Popup rendered in the main window's Overlay is not a separate window, so qt_windows still reports only the main window. Independent Qt.Popup windows do appear and have popup: true.

  • After changing C++ runtime code, rebuilding the artifact and restarting the target is required. Restarting only the MCP server leaves the already-loaded runtime in the old target process. A library already mapped into a process cannot be replaced by injecting again: LoadLibraryW on the same path only increments its reference count, so DllMain never runs. qt_target_info reports this as loaded_injector.loaded with restart_required: true, and qt_attach refuses with reason="injector_already_loaded" instead of silently doing nothing.

  • The injector's Qt debug/release flavour must match the flavour of the Qt runtime the target loaded (Qt5Core.dll vs Qt5Cored.dll). See Injection artifacts.

Troubleshooting

  • Start with qt-commander doctor --json (add --pid <pid> to inspect one target). It reports Python and dependencies, Qt kits, toolchains, installed artifacts and live descriptors, plus the artifact qt_attach would pick for that target and whether it needs a restart.

  • qt_status has no descriptors: start a linked target, or use qt_processes and then qt_attach(pid=..., mode="injection").

  • A desktop client cannot find uv: use the absolute path to uv as command in the client config.

  • qt-commander-mcp.exe is locked: stop the old Claude/MCP process, then restart the client.

  • Injection fails or attaches to an incompatible runtime: read error.reason first. injector_already_loaded and qt_runtime_flavor_mismatch mean the target must be restarted; artifact_missing means run qt-commander install-artifacts or qt_build(for_pid=<pid>). error.details carries the target snapshot, the artifact that was picked, the injector already loaded, and the artifact qt_attach expected. Do not bypass a flavour mismatch by forcing artifact_dir — the library will not start.

For multiple windows, call qt_windows first and pass that window's id to qt_screenshot. Without a target, a screenshot is refused when more than one visible top-level window exists; popups/tooltips do not count toward that ambiguity. qt_query(scope=...) can limit a query to one window/widget subtree; without props, nodes default to text/visible/enabled, which is usually enough to identify a target. where accepts both object_name and the objectName alias, and strings support wildcards or contains/startsWith/regex/equals matchers. For QAction/ menu items use qt_items(owner=menu, where=...); they are virtual items, not part of the QWidget query tree.

Test

uv run pytest -q          # 152 tests, no Qt installation required

The pure-QML regression app is built with the same options as the linked app:

cmake -S tests/test-apps/qmlwindow -B build/qmlwindow-mingw5 -G Ninja \
  -DQT_MAJOR_VERSION=5 \
  -DQt5_DIR=/path/to/Qt/5.15.2/mingw81_64/lib/cmake/Qt5
cmake --build build/qmlwindow-mingw5

End-to-end / benchmark scripts:

# linked smoke (start tests/test-apps/linked first)
uv run python scripts/linked_smoke.py
uv run python scripts/mcp_linked_smoke.py
uv run python scripts/window_regression.py
uv run python scripts/qml_window_test.py

# native injection smoke (pass the target pid)
uv run python scripts/injection_smoke.py <pid> --mode injection

# async task queue: pending -> task_id -> qt_task, nested exec, batch, soak
cmake -S tests/test-apps/modal -B build/modal-mingw5 -G Ninja \
  -DCMAKE_BUILD_TYPE=Release -DQT_MAJOR_VERSION=5 \
  -DQt5_DIR=/path/to/Qt/5.15.2/mingw81_64/lib/cmake/Qt5
cmake --build build/modal-mingw5
uv run python scripts/task_regression.py

# destructive input: Enter / a typed character / hover must never dispatch a
# second event to a widget the first one destroyed
cmake -S tests/test-apps/modal -B build/modal-msvc5 -G Ninja \
  -DCMAKE_BUILD_TYPE=Debug -DQT_MAJOR_VERSION=5 \
  -DQt5_DIR=/path/to/Qt/5.15.2/msvc2019_64/lib/cmake/Qt5
cmake --build build/modal-msvc5
uv run python scripts/type_submit_regression.py \
  --exe build/modal-msvc5/qt-modal-test.exe

# P0 benchmarks
uv run python scripts/p0_large.py --label large-7000
uv run python scripts/p0_matrix.py --exe build/matrix-mingw5/qt-matrix-test.exe

Layout

qt_commander/        Python MCP server, protocol, client, session, ops, injector adapter
qt_commander/cli.py  qt-commander serve / doctor / install-artifacts
src/                 C++ runtime (linked library + injection DLL)
src/injector/        native C++ qtc-inject (Windows remote thread / Linux ptrace)
tests/unit_server/   Python unit tests
tests/test-apps/     linked / large / injected / matrix / qmlwindow test applications
scripts/             smoke, injection, benchmark and release packaging scripts
docs/design.md       current design notes
docs/agent-eval.md   agent acceptance tasks and metrics
docs/release/        release checklist and licence decision notes
docs/benchmarks/     large-7000.json and matrix.json P0 results
ROADMAP.md           what comes next, and what is explicitly out of scope
CHANGELOG.md         release history
SECURITY.md          injection threat model, hardening, disclosure
CONTRIBUTING.md      setup, repository map, how to add a tool
CODE_STANDARDS.md    coding and review rules

Project

License

MIT. See LICENSE.

Earlier releases (v0.8.0 and before) were published under MIT AND Anti-996; they keep the terms they were published with. The reasoning behind the change is recorded in docs/release/LICENSE-NOTES.md, and the old anti-996 text is kept as a non-binding note in docs/community/anti-996.md.

Available Tools

25 tools
qt_attachA

Inject the helper library into a running Qt process and open a session.

Find the target pid with qt_list_processes first (only Qt processes are listed, each with its qt_version/arch/bitness). The build artifacts in .qt-commander/bin must match the target process: same Qt major (qt_major used in qt_build), same toolchain (msvc/mingw), same Debug/Release build type, same 64/32-bit — a mismatch makes injection fail with code 2002. A process that is already attached fails with 2006; detach first.

If the build has not been completed, this returns error code 2001 with the steps to follow (qt_detect_msvc_and_qt → AskUserQuestion → qt_build → qt_attach again); it does not build automatically.

Injection modifies the target process (loads a DLL, starts a local RPC thread/listener). If the target process exits, the session goes dead: operations then fail with connection/timeout errors — check qt_list_sessions and re-attach.

Inputs: pid (required). Returns: result

qt_buildA

Build the Qt injection library and injector for the specified environment.

toolchain selects the compiler: "msvc" (default; vcvars_path is a vcvars bat, qt_env a qtenv2.bat) or "mingw" (vcvars_path is the MinGW bin dir; qt_env the kit's qtenv2.bat — MinGW Qt kits ship one too; the Qt-bundled Ninja is used automatically).

Before calling this tool, the AI MUST present available build environments (from qt_detect_msvc_and_qt) to the user and let them choose. Do NOT guess or auto-pick paths — the user decides which VS or MinGW toolchain and which Qt installation to use.

The AI MUST also ask the user whether to enable QML/QQuick support (with_qml). QML support is required for inspecting QML/Qt Quick applications but adds build dependencies.

The AI MUST also ask the user to choose the build_type (Debug or Release). The build type must match the target process — a Debug process requires a Debug build, a Release process requires Release.

qt_major (default 5) must match the target application's Qt major version — read it from qt_list_processes' qt_version field or from the qt_detect_msvc_and_qt result; a Qt6 target built with qt_major=5 fails to inject. vcvars_args — extra args for the VS vcvars batch (e.g. "-arch x64"). generator — CMake generator override; omit to auto-select. Before rebuilding, detach any session that still holds the old DLL (qt_detach with purge=True) or the build fails on a locked file.

Inputs: qt_env, qt_major, with_qml, generator, +4 more. Returns: result

qt_call_methodA

Invoke a QMetaObject-invokable method on a UI element.

DESTRUCTIVE: directly changes the target application's state with no undo. Methods like close(), deleteLater() (destroys the element and invalidates its id) or any app slot execute for real. Prefer simulating user input (clicks/keys); call methods only when a direct API invocation is intended. Argument types are coerced silently (up to 10 args).

Inputs: method (required), element_id (required), session_id (required), args. Returns: result

qt_detachA

Disconnect from a Qt process session.

purge=True (default False) also ejects the injected DLL from the target process and deletes the session's saved files (snapshots, screenshots, session metadata) — irreversible. Use purge=True before rebuilding the library: otherwise the DLL file stays locked by the target process and qt_build fails. With purge=False the DLL stays loaded in the target until that process exits.

Inputs: session_id (required), purge. Returns: result

qt_detect_msvc_and_qtA

Detect MSVC (Visual Studio), MinGW toolchains, and Qt installations available for building.

IMPORTANT — this tool only discovers and displays what is found on this machine. It does NOT select an environment automatically.

After calling this tool, the AI MUST use AskUserQuestion to present the found installations to the user and let them choose which compiler and Qt to use with qt_build. For an MSVC Qt kit (kit == "msvc") pass the VS vcvars_path + the kit's qtenv_path with toolchain="msvc"; for a MinGW kit (kit == "mingw") pass a MinGW toolchain's bin dir (from mingw_toolchains) + the kit's qtenv_path (MinGW kits ship qtenv2.bat too) with toolchain="mingw". If detection finds nothing (or misses an installation the user knows about), the user can type the paths manually.

Returns three lists: vs_installations, mingw_toolchains, and qt_installations.

Inputs: no named parameters. Returns: result

qt_find_elementA

Find UI elements matching a query in a session.

Query fields (all AND-ed): type (exact C++ class name), type_inherits (superclass chain — C++ classes only; custom QML components are generated classes like X_QMLTYPE_8 and do NOT match a QML type name), text / text_contains (display text, CJK supported), object_name, window_title / window_title_contains, properties (all key-value pairs must match), ancestor_id / window_id (scope limits), depth ("exact" = direct children, "shallow" = 2 levels, integer = that many, "deep"/omitted = whole tree), limit (max matches). include_hidden is a field INSIDE the query dict (default false) — hidden elements are only matched when it is true.

Example: {"text_contains": "OK", "type_inherits": "QPushButton", "depth": "shallow"} or {"object_name": "searchBox", "limit": 5}.

Result: {"ok": true, "count": N, "elements": [{id, className, objectName, ...}]}; when nothing matches: {"ok": false, "message": "No matching element found"} — adjust the query or set include_hidden: true. No prior snapshot is needed; this call rebuilds the element map itself.

ID LIFECYCLE: the rebuild invalidates EVERY element_id/window_id from previous snapshots/finds. Use the returned ids immediately — if an operation reports "Element not found: id=N", the id is stale or the element was destroyed: re-run qt_find_element (or a snapshot) and retry with the fresh id; never reuse old ids.

Inputs: query (required), session_id (required). Returns: result

qt_focusA

Set focus on a UI element.

Best effort: QWidget focus works directly; some QML items may need an explicit focus request from within the app and report ok without changing focus.

Inputs: element_id (required), session_id (required). Returns: result

qt_get_propertyA

Read a property from a UI element.

Read-only, no side effects on the target. Note: hidden, disabled and zero-size elements are rejected even for reads ("Element is not visible") — pass an id from a recent find/snapshot of a visible element.

Inputs: name (required), element_id (required), session_id (required). Returns: result

qt_keyboard_inputA

Send keyboard input (typed text) to a UI element.

Real input: the text lands in whatever the target would receive — it can overwrite existing text and trigger real app behaviour. If element_id is 0 or does not resolve, the input goes to the widget that currently has focus (be careful: a stale id silently redirects the text). Widgets that rely on focus (QLineEdit etc.) may ignore input to an unfocused element — call qt_focus first when text does not land. Use qt_key_combo for shortcuts like Enter or Ctrl+C.

Inputs: text (required), element_id (required), session_id (required), modifiers. Returns: result

qt_key_comboA

Send a keyboard shortcut to an element, e.g. "Ctrl+C", "Ctrl+Shift+A".

Format: modifier names ("Ctrl", "Alt", "Shift", "Meta") joined with '+' followed by the key name ("C", "F5", "Enter", "Tab", "Escape", "Home", arrows, ...). element_id 0 targets the widget that currently has focus. Delivered as a real key press/release pair with the modifier state set — shortcuts can trigger real app actions (close window, save, submit).

Inputs: keys (required), element_id (required), session_id (required). Returns: result

qt_list_processesA

List running Qt processes that may be attachable.

Inputs: no named parameters. Returns: result

qt_list_sessionsA

List all active sessions.

Inputs: no named parameters. Returns: result

qt_mouse_clickA

Send a mouse click to a UI element (direct delivery).

For plain QtWidgets (QPushButton, QLineEdit, ...) direct delivery works and is the cheapest option. Use qt_mouse_click_region (element center) or qt_mouse_click_at (exact coordinates) for QML custom components (buttons, nav rows, list items) — those route through the real Qt input pipeline with real hit testing and are far more reliable. button: "left"/"right"/"middle" (anything else falls back to left); modifiers: list of "Ctrl"/"Alt"/"Shift"/"Meta". A result with ok:false means the element was rejected (hidden/ disabled/zero-size/stale id) — re-find and retry.

Inputs: element_id (required), session_id (required), button, modifiers. Returns: result

qt_mouse_click_atA

Click at an exact window coordinate, exactly like a real mouse click at that position.

x/y are relative to the target window's client area (same space as screenshots, logical pixels — DPI is handled internally). The click goes through the real Qt input pipeline, so the hit test (scene graph for QML, widget tree for QtWidgets) determines what receives it — identical behavior to a human clicking there, including real consequences (menus, close buttons, destructive actions). window_id: id of a top-level window from the MOST RECENT snapshot/find (ids expire on refresh); 0 uses the session's first visible top-level window — pass it explicitly when the session has multiple windows.

Inputs: x (required), y (required), session_id (required), button, +2 more. Returns: result

qt_mouse_click_regionA

Click at the center of an element's on-screen region.

Unlike qt_mouse_click (which delivers straight to the element), this routes through the real Qt input pipeline with real hit testing: for a QML container (e.g. a Rectangle with a MouseArea inside) the scene graph hit test delivers the click to the MouseArea, exactly as a human click would.

Inputs: element_id (required), session_id (required), button, modifiers. Returns: result

qt_mouse_context_menuA

Open the context menu at an element (right-click menu).

x/y are optional logical-pixel coordinates relative to the element's top-left corner; either both or neither (omitted = element center).

Inputs: element_id (required), session_id (required), x, y. Returns: result

qt_mouse_dbl_clickB

Send a double-click to a UI element (direct delivery).

x/y are optional logical-pixel coordinates relative to the element's top-left corner; either both or neither (omitted = element center).

Inputs: element_id (required), session_id (required), x, y, +2 more. Returns: result

qt_mouse_moveA

Move the mouse pointer to an element-local position (no buttons).

Use between qt_mouse_press and qt_mouse_release to drag, or alone to hover. x/y are coordinates relative to the element's top-left corner.

Inputs: x (required), y (required), element_id (required), session_id (required). Returns: result

qt_mouse_pressA

Press a mouse button on an element WITHOUT releasing it.

Pair with qt_mouse_release to split a click, or qt_mouse_move for a drag (press -> move -> release). x/y are optional logical-pixel coordinates relative to the element's top-left corner; either both or neither (omitted = element center). IMPORTANT: after a press you must release in the same session, or the target app stays in a pressed/ dragging state.

Inputs: element_id (required), session_id (required), x, y, +2 more. Returns: result

qt_mouse_releaseA

Release a previously pressed mouse button on an element.

Completes a press/release pair (a click) or finishes a drag started with qt_mouse_press + qt_mouse_move. x/y are optional logical-pixel coordinates relative to the element's top-left corner; either both or neither (omitted = element center).

Inputs: element_id (required), session_id (required), x, y, +2 more. Returns: result

qt_mouse_wheelA

Scroll the mouse wheel over an element.

dx/dy are the wheel deltas in either pixel (pixel=True) or line (default) units. x/y optionally position the wheel inside the element (element-local logical px, both or neither — omitted = center). Use this to scroll lists, tables, canvases and other scrollable content.

Inputs: element_id (required), session_id (required), x, y, +3 more. Returns: result

qt_prune_snapshotA

Occlusion-prune a saved snapshot.

snapshot_id is the value returned by the last qt_snapshot call of this session. When to use: after a snapshot, when elements overlap inside the same window and you need to know which one is really on top / how much of a target is visible — e.g. before deciding which element to click or whether a covered element matters.

Reads the saved snapshot JSON and computes what is actually visible: elements fully covered by higher-z opaque elements are removed (their still-visible descendants are reparented up), partially covered ones get a visible_ratio field, fully hidden (opacity 0) elements are dropped. Writes snapshot_<id>_pruned.json next to the original — read the tree at the returned uri (same objIDs as the source snapshot plus visible_ratio annotations). Result field pruned = {"removed", "kept", "removed_ratio"}; window roots are never removed.

The solver is a geometric heuristic (axis-aligned rects, per-window z-order with same-z tree order; widgets and QML rectangles/images occlude unless semi-transparent, transparent containers, text and custom QML components do not; a parent never occludes its own children). Occlusion is per top-level window on purpose: the agent must stay able to operate an app that the user has covered or minimised, so windows never occlude each other. The objIDs in the pruned file expire with the next snapshot/find refresh like any other id.

Inputs: session_id (required), snapshot_id (required). Returns: result

qt_screenshotA

Capture a screenshot of a UI element or the entire window.

element_id 0 captures the active (or first visible) top-level window. The PNG is written to the file at the returned uri — read that resource (qt-commander://sessions/.../screenshots/N.png) to obtain the image bytes. On failure the result reports the error and no image is written.

Inputs: session_id (required), element_id. Returns: result

qt_set_propertyA

Write a property value on a UI element.

DESTRUCTIVE: directly modifies the target application's state (may trigger property-notify signals, change geometry/visibility/business state) with no undo. Prefer real user input (clicks/typing) to simulate user actions; use this only when a direct API write is intended.

value is first parsed as JSON: "true" -> bool, "123" -> int, ""hello"" -> string. Unparseable text is sent as a plain string — pass quoted strings explicitly when a string is intended.

Inputs: name (required), value (required), element_id (required), session_id (required). Returns: result

qt_snapshotA

Capture the UI element tree of the session's windows.

The tree is NOT in this result — it is written to the file exposed at the returned uri (resource qt-commander://sessions/.../snapshots/ snapshot_N.json); read that resource to inspect the tree. The result carries only session_id / snapshot_id / uri. Node fields: className, objID, objectName, rect (window-local logical px), global_rect (screen coords), z_order, visible/enabled/opacity/color_alpha/clip, text, topLevelId, windowTitle, properties (per detail).

detail — property tier per node: "core" (first-class fields only, no properties), "extended" (DEFAULT; common interaction-state properties like text/checked/value), "full" (every Q_PROPERTY; slow on large UIs). include_hidden (default False) — hidden elements are pruned together with their whole subtree (hidden tabs, unopened dialogs). Set True to include them; note hidden elements are still rejected by click/input/property tools. max_depth — levels of children: 0 = roots only, 1 = roots + direct children, -1 = entire tree (default 1). prop_depth — levels of QObject property expansion (0 = none, -1 = unlimited). root_id — 0 = all top-level windows; >0 = subtree of an element from the MOST RECENT snapshot/find. Ids expire on every refresh: if root_id no longer resolves, the tool silently falls back to all top-level windows — pass it only when you know it is fresh.

ID LIFECYCLE: this call rebuilds the element map, so every element_id and window_id from previous snapshots/finds becomes invalid. The snapshot runs on the target process's GUI thread (brief UI freeze); if the target is busy, calls can time out after ~30s.

Inputs: session_id (required), detail, root_id, max_depth, +2 more. Returns: result

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A4.1/5.0

    Scored across 25 tools

    Disambiguation4/5

    Most tools target distinct resources and actions, but the three mouse click variants (qt_mouse_click, qt_mouse_click_region, qt_mouse_click_at) are closely related and could be confused. Their descriptions clearly explain the differences in delivery mechanism and use cases, so ambiguity is mostly resolved.

    Naming Consistency5/5

    All tool names follow a consistent qt_verb_noun pattern with lowercase underscores (e.g., list_processes, get_property, mouse_click). Minor abbreviations like 'dbl' in qt_mouse_dbl_click are negligible and do not disrupt the overall predictability.

    Tool Count4/5

    With 25 tools, the server is on the heavier side, but each tool maps to a distinct step in Qt UI automation—from environment detection and build to attach, inspection, and input simulation. The granularity is justified by the complexity of the domain.

    Completeness5/5

    The tool set covers the entire lifecycle: detect environment, build, attach, snapshot/find elements, read/write properties, call methods, simulate mouse and keyboard input, and detach. No obvious gaps prevent agents from accomplishing UI automation tasks.

    Maintenance

    ActivityActive
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      A
      quality
      D
      maintenance
      An MCP server for inspecting, debugging, and interacting with PySide6 desktop applications. It enables AI agents to capture UI snapshots, read widget properties, and perform actions like clicking buttons or typing text.
      22
      3
      Apache 2.0
    • A
      license
      Not graded
      quality
      B
      maintenance
      Gives AI agents and MCP clients direct control over native desktop apps, Chrome/Electron browsers, and Android devices with screenshots, OCR, accessibility-based element lookup, input simulation, window management, CDP, and ADB in one local server.
      142
      MIT
    • A
      license
      A
      quality
      D
      maintenance
      An MCP server for capturing screenshots of Qt/desktop windows and performing filesystem operations, enabling AI clients to inspect and modify project files.
      15
      MIT
    • A
      license
      A
      quality
      C
      maintenance
      Cross-platform desktop automation MCP server that lets AI agents capture screenshots, run OCR with UI-element classification, control mouse/keyboard, and launch programs on Linux, macOS, and Windows.
      20
      2
      MIT