qt-commander
It lets an AI agent discover, attach to, inspect, and automate live Qt desktop applications through an MCP server.
List running Qt processes and active sessions; attach via DLL injection or detach, optionally purging the injected DLL and session files.
Detect available MSVC/MinGW toolchains and Qt installations, then build a matching injection library/injector with chosen Qt major version, toolchain, QML support, and Debug/Release type.
Capture UI element trees as snapshots, prune snapshots for occlusion/visibility, and find elements by type, text, object name, window title, properties, ancestry, depth, etc.
Read/write element properties and invoke QMetaObject-invokable methods on live UI elements.
Take screenshots of windows or specific elements.
Simulate mouse input: click, double-click, coordinate/region clicks, press/release/drag, move/hover, wheel scroll, and context menus.
Send keyboard input, key combinations/shortcuts, and set focus on UI elements.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@qt-commander Attach to the running Qt app and show its UI snapshot"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
qt-commander
Live Qt UI introspection and automation for AI agents.
qt-commander links a small in-process runtime into a Qt application, exposes its live object tree over a local JSON-RPC connection, and drives it through an MCP server. Queries are evaluated in the target process every time; there are no snapshot files and no stale index of the UI.
Install
The first PyPI release is being prepared. Until it is out, install from source (option 2); the command line below already exists in this repository.
Requirements
Python 3.10+ and uv
A Qt 5 or Qt 6 target application. Verified on Qt 5.15.2 and Qt 6.8.3; other Qt 5.x/6.x releases are expected to work but are not yet validated.
Windows or Linux
Python is only needed for the MCP server; the target process does not need Python or gdb
Option 1 — package plus prebuilt runtime
pip install qt-commander-mcp
qt-commander doctor # what this machine offers
qt-commander install-artifacts # injector + runtime for this OS/architectureinstall-artifacts downloads the release archive, verifies SHA256SUMS, and
unpacks it into the artifact root (QT_COMMANDER_ARTIFACTS, or a per-user
directory for a wheel install). No C++ compiler is needed.
Option 2 — from source
git clone https://github.com/tiefeiyu/qt-commander.git
cd qt-commander
uv sync
uv run qt-commander doctorSource-checkout artifacts built with qt_build or
scripts/package_artifacts.py land in artifacts/ next to the checkout.
Register it with your MCP client
Claude Code:
claude mcp add qt-commander -- qt-commander-mcp # pip install
claude mcp add qt-commander -- uv run --project /absolute/path/to/qt-commander qt-commander-mcp # source installClaude Desktop / Cursor / other JSON clients:
{
"mcpServers": {
"qt-commander": {
"command": "qt-commander-mcp"
}
}
}For a source checkout use "command": "uv" with
"args": ["run", "--project", "/absolute/path/to/qt-commander", "qt-commander-mcp"].
Use a native absolute path (D:/Develop/qt-commander is fine on Windows).
Restart the client after changing the config; qt_status should then list
descriptors or live sessions.
3. Connect a target
Linked runtime: build/instrument the application or test app once (see Build the linked runtime); it writes a descriptor.
Already-running target: call
qt_attach(pid=<pid>, mode="injection"); useqt_processesto discover a PID, thenqt_target_infowhen the target's Qt version/toolchain is unknown. Build the matching injector and runtime if needed — normallyqt_build(for_pid=<pid>), which derives every argument from the target and reportsrestart_required/next_step.
The usual first agent interaction is:
qt_status -> qt_attach -> qt_windows -> qt_query -> qt_actRelated MCP server: native-devtools-mcp
Status
Qt 5.15.2 verified with MinGW/MSVC; Qt 6.8.3 verified with both MinGW and MSVC for linked and injection. Ubuntu 24.04 + Qt 5.15.2 gcc_64 (XCB) verified for linked, the 99-test Python suite, and MCP stdio E2E.
Linked mode: the application calls
qt_commander::start()afterQApplicationand writes a rendezvous descriptor.Native injection (Windows and Linux):
qt_attach(pid=..., mode="injection")invokes the C++qtc-injectexecutable, which loadsqt_commander_inject.dll/.so; the target-side library starts the same in-process server.autoinjects when no live descriptor exists. Python is only the MCP orchestration layer; the target does not need Python or gdb. Linux ptrace still needs Yama permission (run the injector as the target's parent, or setkernel.yama.ptrace_scope=0). The native binary can be overridden withQT_COMMANDER_INJECTOR.Launch-time injection (Linux and macOS):
qt_launch(program, library=...)starts the application with the runtime preloaded (LD_PRELOAD/DYLD_INSERT_LIBRARIES), waits for its descriptor and attaches. No ptrace, no Yama configuration, and on macOS it is the only way in — the system refuses injection into an already-running hardened or notarized process. Windows has no preload hook, soqt_launchis not registered there.21 MCP tools are registered. Working today:
status,target_info,processes,environments,build,build_status,attach,detach,windows,query,items,act,batch,wait,screenshot,get,set,call,methods,source,task.sourcesearches the target's source tree (andQT_COMMANDER_SOURCE_DIR/ an explicitsource_dir) for matching.uior.qml; unresolved targets return identity withE_NO_DEBUGINFO.Every data request is executed from a per-server FIFO queue driven by the target's event loop, so a slot blocked in
exec()no longer holds the socket: reads, acts andqt_taskkeep working on the same connection.qt_act/qt_call/qt_settakewait_ms(defaults 1000/3000/3000 ms); when it expires they return{"state": "running", "task_id": ...}and the result is stored forqt_task(task_id, wait_ms=...). Such a call has already been dispatched — never retry it.qt_pingreportsqueued/inflight/activefor diagnostics.itemscoversQTabWidget/QTabBar,QMenuBar/QMenu,QHeaderView,QTableView,QTreeView,QListView,QComboBox,QScrollBar,QSlider,QSpinBox, checkableQGroupBox,QDockWidgettitle buttons,QSplitterhandles andQToolBaractions; custom/proxy models work through the item-view adapter.QML deep objects under
QQuickWidgetare returned byquerywithqml_type/qml_id/objectName/local rect; use theirvqmlIDs withqt_get,qt_act,qt_screenshotandqt_source.QML-only apps are first-class too:
QQuickWindowroots work inqt_windows,qt_query,qt_items,qt_get,qt_set,qt_act(QML click/key/type) andqt_screenshot(window or element crop).qt_querylimitis a soft truncation: partial tree pluscount/total/limit/truncatedinstead of a hard error.qt_callinvokes only Qt metaobject methods (slots/Q_INVOKABLE) on widgets and QML items; property setters such asQPushButton::setTextare not methods — useqt_setinstead. Non-void return values are returned inresult; object returns use{class, id}so they can be reused as targets.qt_methodslists invokable methods, parameter types and return types (Qt_q_internals are hidden unlessinclude_internal=true). Passsignatureto disambiguate overloads; enum arguments accept their integer values.qt_target_infoinspects a process's PE/ELF architecture, full Qt runtime version (e.g. 5.15.2), build type, and toolchain beforeqt_build; it does not inject or launch.detected_fromsays which evidence won (module— the Qt runtime actually mapped into the process, authoritative —adjacent,loader_pathordescriptor). It also reportsloaded_injector(path, artifact, build_id, stale),expected_artifact(whatqt_attachwould inject),injection_state(none/loaded/stale/mismatch) andrestart_required.qt_processeslists discoverable running Qt targets with their attach hint, without injecting.qt_sourceauto-discovers nearby source trees (includingsrcancestors) and acceptssource_dirfor qrc/embedded QML whose source tree is not next to the executable; it returnsfile/line/urlwhen found.qt_screenshot(annotate=...)supportsannotate_offsetfor paging large annotation sets; legendnnumbers stay continuous across pages.qt_batchsteps are exactly one of{"act": ...},{"wait": ...},{"set": ...},{"get": ...}; this allows an ordered write/read/restore sequence (no implicit rollback).qt_act(target=<window id>, action="minimize"|"restore")controls a QWindow/QQuickWindow; restore preserves the pre-minimize geometry.action="set_geometry"withgeometry={x,y,w,h}reports the actualresolved.rectand anappliedflag.qt_actsupports click/dblclick/type/focus/key/scroll/hover/drag and declarativeexpect(where/props/gone).typefinds the first editable child for container targets and returnsE_NOT_EDITABLEwhen none exists. Virtual items support click/dblclick.Targets may use a plain string as a QML
qml_id, or combineqml_id/text/class/objectName(child text is included). Text matches fold into the nearest actionable ancestor; e.g. a label inside a button clicks the button. Hidden targets are excluded by default and returnE_HIDDENunlessinclude_hidden=trueis passed.Screenshots are capped at about 6 MiB at the MCP layer; oversized images are automatically re-captured at a smaller width (
auto_downscaled).
Build the linked runtime
cmake -S tests/test-apps/linked -B build/linked -G Ninja \
-DQT_MAJOR_VERSION=5 \
-DQt5_DIR=/path/to/Qt/5.15.2/mingw81_64/lib/cmake/Qt5 \
-DCMAKE_CXX_COMPILER=/path/to/mingw/g++.exe
cmake --build build/linkedMSVC users can instead call the MCP qt_build tool; it discovers Visual
Studio via vswhere and loads vcvars64.bat automatically (or accepts
vcvars_path).
On Linux with an official Qt kit, if Qt5GuiConfigExtras complains that
GL/gl.h is missing under /usr/include/libdrm, add
-D_qt5gui_OPENGL_INCLUDE_DIR=/usr/include to the configure command.
On Windows, put the Qt bin directory and the MinGW runtime on PATH, and
point QT_PLUGIN_PATH at the Qt plugins directory. Then run:
$env:QT_COMMANDER_RENDEZVOUS = "$PWD\.test-workspace\rendezvous"
.\build\linked\qt-linked-test.exeThe app writes descriptor-<pid>.json into that directory (one file per
process; owner-only permissions on POSIX).
Injection artifacts
qt_build writes injectors and injection libraries into one root only:
artifacts/<os>-<arch>/<compiler>-<version>/qt-<qt-version>-<debug|release>/
manifest.json
bin/qtc-inject.exe
lib/qt_commander_inject.dll # libqt_commander_inject.so on LinuxExamples:
artifacts/windows-x64/msvc-19.29/qt-5.15.2-debug/
artifacts/windows-x64/mingw-gcc-13.1/qt-5.15.2-release/
artifacts/linux-x64/gcc-13.3/qt-6.8.3-release/manifest.json records build_id in the canonical
qt<version>/<compiler>-<arch>/<build_type> form (for example
qt5.15.2/msvc-x64/Release), the same string qt_build returns. The runtime's
descriptor carries the compiled-in id instead (qt5/MSVC-AMD64/Release), so do
not compare the two as plain strings; use
qt_target_info.loaded_injector.build_id against expected_artifact.build_id.
When the artifact was built from a git checkout, the manifest also carries a
source block with the release version and the commit it came from;
qt_target_info and qt-commander doctor --pid print it.
qt_attach(mode="injection") matches the target's OS/architecture,
compiler family and version, Qt version, and Qt debug/release against a
manifest under artifacts/; it does not scan build/ or pick the newest file.
Set QT_COMMANDER_INJECTOR / QT_COMMANDER_INJECT_DLL only for an explicit
override.
Artifacts come from one of three places, in order of convenience:
qt-commander install-artifacts— downloadsqt-commander-artifacts-<os>-<arch>-*.zipfrom the latest GitHub release, verifiesSHA256SUMS, and unpacks it.--from-dir/--from-urlcover offline machines,--listshows what a release carries.qt_build(for_pid=<pid>)from an agent or MCP client — compiles against the Qt kit of a live target.scripts/package_artifacts.pyfor a build you made yourself with CMake.
The artifact root is artifacts/ in a source checkout. For an installed wheel
it is read-only, so install-artifacts falls back to a per-user directory; set
QT_COMMANDER_ARTIFACTS to pin one location explicitly (the MCP server and the
CLI both honour it).
The Qt debug/release flavour comes from evidence, never from a guess: the Qt
runtime actually mapped into the process wins, then the executable's own
directory, then the loader PATH, then the descriptor. A directory holding
both Qt5Core.dll and Qt5Cored.dll (a full Qt kit bin does) reports
debug: null with flavor_ambiguous: true, and artifact selection then asks
for an explicit artifact_dir instead of silently picking release. Picking the
wrong flavour injects a DLL that links the other Qt runtime, loads a second Qt
into the target, and times out waiting for QCoreApplication — which is why
qt_build refuses to default an unknown build_type.
Run the MCP server
uv sync
uv run --project . qt-commander-mcpIf Windows reports qt-commander-mcp.exe is in use, stop the running
Claude/MCP process first, then rerun uv run.
Point the MCP client at the descriptor with qt_attach(descriptor=...), or
set QT_COMMANDER_RENDEZVOUS before starting the server. For an already-running
Windows Qt app, use qt_attach(pid=<pid>, mode="injection"); the DLL path can
be overridden with QT_COMMANDER_INJECT_DLL.
The MCP server carries an instructions block for agent startup, target
selection, ambiguity/hidden handling, and destructive-action safety; each
tool also has a schema-aware description.
Known limitations
Qt Quick Controls
Popuproots (for exampleindicatorPopup) may not be discoverable byqml_id; query their text or child items instead. APopuprendered in the main window's Overlay is not a separate window, soqt_windowsstill reports only the main window. IndependentQt.Popupwindows do appear and havepopup: true.After changing C++ runtime code, rebuilding the artifact and restarting the target is required. Restarting only the MCP server leaves the already-loaded runtime in the old target process. A library already mapped into a process cannot be replaced by injecting again:
LoadLibraryWon the same path only increments its reference count, soDllMainnever runs.qt_target_inforeports this asloaded_injector.loadedwithrestart_required: true, andqt_attachrefuses withreason="injector_already_loaded"instead of silently doing nothing.The injector's Qt debug/release flavour must match the flavour of the Qt runtime the target loaded (
Qt5Core.dllvsQt5Cored.dll). See Injection artifacts.
Troubleshooting
Start with
qt-commander doctor --json(add--pid <pid>to inspect one target). It reports Python and dependencies, Qt kits, toolchains, installed artifacts and live descriptors, plus the artifactqt_attachwould pick for that target and whether it needs a restart.qt_statushas no descriptors: start a linked target, or useqt_processesand thenqt_attach(pid=..., mode="injection").A desktop client cannot find
uv: use the absolute path touvascommandin the client config.qt-commander-mcp.exeis locked: stop the old Claude/MCP process, then restart the client.Injection fails or attaches to an incompatible runtime: read
error.reasonfirst.injector_already_loadedandqt_runtime_flavor_mismatchmean the target must be restarted;artifact_missingmeans runqt-commander install-artifactsorqt_build(for_pid=<pid>).error.detailscarries the target snapshot, the artifact that was picked, the injector already loaded, and the artifactqt_attachexpected. Do not bypass a flavour mismatch by forcingartifact_dir— the library will not start.
For multiple windows, call qt_windows first and pass that window's id to
qt_screenshot. Without a target, a screenshot is refused when more than one
visible top-level window exists; popups/tooltips do not count toward that
ambiguity. qt_query(scope=...) can limit a query to one window/widget subtree;
without props, nodes default to text/visible/enabled, which is usually
enough to identify a target. where accepts both object_name and the objectName alias, and strings
support wildcards or contains/startsWith/regex/equals matchers. For QAction/
menu items use qt_items(owner=menu, where=...); they are virtual items, not
part of the QWidget query tree.
Test
uv run pytest -q # 152 tests, no Qt installation requiredThe pure-QML regression app is built with the same options as the linked app:
cmake -S tests/test-apps/qmlwindow -B build/qmlwindow-mingw5 -G Ninja \
-DQT_MAJOR_VERSION=5 \
-DQt5_DIR=/path/to/Qt/5.15.2/mingw81_64/lib/cmake/Qt5
cmake --build build/qmlwindow-mingw5End-to-end / benchmark scripts:
# linked smoke (start tests/test-apps/linked first)
uv run python scripts/linked_smoke.py
uv run python scripts/mcp_linked_smoke.py
uv run python scripts/window_regression.py
uv run python scripts/qml_window_test.py
# native injection smoke (pass the target pid)
uv run python scripts/injection_smoke.py <pid> --mode injection
# async task queue: pending -> task_id -> qt_task, nested exec, batch, soak
cmake -S tests/test-apps/modal -B build/modal-mingw5 -G Ninja \
-DCMAKE_BUILD_TYPE=Release -DQT_MAJOR_VERSION=5 \
-DQt5_DIR=/path/to/Qt/5.15.2/mingw81_64/lib/cmake/Qt5
cmake --build build/modal-mingw5
uv run python scripts/task_regression.py
# destructive input: Enter / a typed character / hover must never dispatch a
# second event to a widget the first one destroyed
cmake -S tests/test-apps/modal -B build/modal-msvc5 -G Ninja \
-DCMAKE_BUILD_TYPE=Debug -DQT_MAJOR_VERSION=5 \
-DQt5_DIR=/path/to/Qt/5.15.2/msvc2019_64/lib/cmake/Qt5
cmake --build build/modal-msvc5
uv run python scripts/type_submit_regression.py \
--exe build/modal-msvc5/qt-modal-test.exe
# P0 benchmarks
uv run python scripts/p0_large.py --label large-7000
uv run python scripts/p0_matrix.py --exe build/matrix-mingw5/qt-matrix-test.exeLayout
qt_commander/ Python MCP server, protocol, client, session, ops, injector adapter
qt_commander/cli.py qt-commander serve / doctor / install-artifacts
src/ C++ runtime (linked library + injection DLL)
src/injector/ native C++ qtc-inject (Windows remote thread / Linux ptrace)
tests/unit_server/ Python unit tests
tests/test-apps/ linked / large / injected / matrix / qmlwindow test applications
scripts/ smoke, injection, benchmark and release packaging scripts
docs/design.md current design notes
docs/agent-eval.md agent acceptance tasks and metrics
docs/release/ release checklist and licence decision notes
docs/benchmarks/ large-7000.json and matrix.json P0 results
ROADMAP.md what comes next, and what is explicitly out of scope
CHANGELOG.md release history
SECURITY.md injection threat model, hardening, disclosure
CONTRIBUTING.md setup, repository map, how to add a tool
CODE_STANDARDS.md coding and review rulesProject
ROADMAP.md — priorities and non-goals
CHANGELOG.md — what changed, per release
CONTRIBUTING.md — build, test, and how a tool is wired
SECURITY.md — read this before injecting into anything you care about
License
MIT. See LICENSE.
Earlier releases (v0.8.0 and before) were published under
MIT AND Anti-996; they keep the terms they were published with. The reasoning
behind the change is recorded in
docs/release/LICENSE-NOTES.md, and the old
anti-996 text is kept as a non-binding note in
docs/community/anti-996.md.
Available Tools
25 toolsqt_attachA
Inject the helper library into a running Qt process and open a session.
Find the target pid with qt_list_processes first (only Qt processes are listed, each with its qt_version/arch/bitness). The build artifacts in .qt-commander/bin must match the target process: same Qt major (qt_major used in qt_build), same toolchain (msvc/mingw), same Debug/Release build type, same 64/32-bit — a mismatch makes injection fail with code 2002. A process that is already attached fails with 2006; detach first.
If the build has not been completed, this returns error code 2001 with the steps to follow (qt_detect_msvc_and_qt → AskUserQuestion → qt_build → qt_attach again); it does not build automatically.
Injection modifies the target process (loads a DLL, starts a local RPC thread/listener). If the target process exits, the session goes dead: operations then fail with connection/timeout errors — check qt_list_sessions and re-attach.
Inputs: pid (required). Returns: result
qt_buildA
Build the Qt injection library and injector for the specified environment.
toolchain selects the compiler: "msvc" (default; vcvars_path
is a vcvars bat, qt_env a qtenv2.bat) or "mingw"
(vcvars_path is the MinGW bin dir; qt_env the kit's qtenv2.bat
— MinGW Qt kits ship one too; the Qt-bundled Ninja is used
automatically).
Before calling this tool, the AI MUST present available build
environments (from qt_detect_msvc_and_qt) to the user and let them
choose. Do NOT guess or auto-pick paths — the user decides which VS
or MinGW toolchain and which Qt installation to use.
The AI MUST also ask the user whether to enable QML/QQuick support
(with_qml). QML support is required for inspecting QML/Qt Quick
applications but adds build dependencies.
The AI MUST also ask the user to choose the build_type (Debug or
Release). The build type must match the target process — a Debug
process requires a Debug build, a Release process requires Release.
qt_major (default 5) must match the target application's Qt major
version — read it from qt_list_processes' qt_version field or from
the qt_detect_msvc_and_qt result; a Qt6 target built with qt_major=5
fails to inject. vcvars_args — extra args for the VS vcvars batch
(e.g. "-arch x64"). generator — CMake generator override; omit to
auto-select. Before rebuilding, detach any session that still holds
the old DLL (qt_detach with purge=True) or the build fails on a locked
file.
Inputs: qt_env, qt_major, with_qml, generator, +4 more. Returns: result
qt_call_methodA
Invoke a QMetaObject-invokable method on a UI element.
DESTRUCTIVE: directly changes the target application's state with no
undo. Methods like close(), deleteLater() (destroys the
element and invalidates its id) or any app slot execute for real.
Prefer simulating user input (clicks/keys); call methods only when a
direct API invocation is intended. Argument types are coerced
silently (up to 10 args).
Inputs: method (required), element_id (required), session_id (required), args. Returns: result
qt_detachA
Disconnect from a Qt process session.
purge=True (default False) also ejects the injected DLL from the
target process and deletes the session's saved files (snapshots,
screenshots, session metadata) — irreversible. Use purge=True before
rebuilding the library: otherwise the DLL file stays locked by the
target process and qt_build fails. With purge=False the DLL stays
loaded in the target until that process exits.
Inputs: session_id (required), purge. Returns: result
qt_detect_msvc_and_qtA
Detect MSVC (Visual Studio), MinGW toolchains, and Qt installations available for building.
IMPORTANT — this tool only discovers and displays what is found on this machine. It does NOT select an environment automatically.
After calling this tool, the AI MUST use AskUserQuestion to present
the found installations to the user and let them choose which compiler
and Qt to use with qt_build. For an MSVC Qt kit
(kit == "msvc") pass the VS vcvars_path + the kit's
qtenv_path with toolchain="msvc"; for a MinGW kit
(kit == "mingw") pass a MinGW toolchain's bin dir (from
mingw_toolchains) + the kit's qtenv_path (MinGW kits ship
qtenv2.bat too) with toolchain="mingw".
If detection finds nothing (or misses an installation the user knows
about), the user can type the paths manually.
Returns three lists: vs_installations, mingw_toolchains, and
qt_installations.
Inputs: no named parameters. Returns: result
qt_find_elementA
Find UI elements matching a query in a session.
Query fields (all AND-ed): type (exact C++ class name),
type_inherits (superclass chain — C++ classes only; custom QML
components are generated classes like X_QMLTYPE_8 and do NOT match
a QML type name), text / text_contains (display text, CJK
supported), object_name, window_title /
window_title_contains, properties (all key-value pairs must
match), ancestor_id / window_id (scope limits), depth
("exact" = direct children, "shallow" = 2 levels, integer = that many,
"deep"/omitted = whole tree), limit (max matches). include_hidden
is a field INSIDE the query dict (default false) — hidden elements
are only matched when it is true.
Example: {"text_contains": "OK", "type_inherits": "QPushButton", "depth": "shallow"} or {"object_name": "searchBox", "limit": 5}.
Result: {"ok": true, "count": N, "elements": [{id, className, objectName, ...}]}; when nothing matches: {"ok": false, "message": "No matching element found"} — adjust the query or set include_hidden: true. No prior snapshot is needed; this call rebuilds the element map itself.
ID LIFECYCLE: the rebuild invalidates EVERY element_id/window_id from previous snapshots/finds. Use the returned ids immediately — if an operation reports "Element not found: id=N", the id is stale or the element was destroyed: re-run qt_find_element (or a snapshot) and retry with the fresh id; never reuse old ids.
Inputs: query (required), session_id (required). Returns: result
qt_focusA
Set focus on a UI element.
Best effort: QWidget focus works directly; some QML items may need an explicit focus request from within the app and report ok without changing focus.
Inputs: element_id (required), session_id (required). Returns: result
qt_get_propertyA
Read a property from a UI element.
Read-only, no side effects on the target. Note: hidden, disabled and zero-size elements are rejected even for reads ("Element is not visible") — pass an id from a recent find/snapshot of a visible element.
Inputs: name (required), element_id (required), session_id (required). Returns: result
qt_keyboard_inputA
Send keyboard input (typed text) to a UI element.
Real input: the text lands in whatever the target would receive — it can overwrite existing text and trigger real app behaviour. If element_id is 0 or does not resolve, the input goes to the widget that currently has focus (be careful: a stale id silently redirects the text). Widgets that rely on focus (QLineEdit etc.) may ignore input to an unfocused element — call qt_focus first when text does not land. Use qt_key_combo for shortcuts like Enter or Ctrl+C.
Inputs: text (required), element_id (required), session_id (required), modifiers. Returns: result
qt_key_comboA
Send a keyboard shortcut to an element, e.g. "Ctrl+C", "Ctrl+Shift+A".
Format: modifier names ("Ctrl", "Alt", "Shift", "Meta") joined with '+' followed by the key name ("C", "F5", "Enter", "Tab", "Escape", "Home", arrows, ...). element_id 0 targets the widget that currently has focus. Delivered as a real key press/release pair with the modifier state set — shortcuts can trigger real app actions (close window, save, submit).
Inputs: keys (required), element_id (required), session_id (required). Returns: result
qt_list_processesA
List running Qt processes that may be attachable.
Inputs: no named parameters. Returns: result
qt_list_sessionsA
List all active sessions.
Inputs: no named parameters. Returns: result
qt_mouse_clickA
Send a mouse click to a UI element (direct delivery).
For plain QtWidgets (QPushButton, QLineEdit, ...) direct delivery
works and is the cheapest option. Use qt_mouse_click_region (element
center) or qt_mouse_click_at (exact coordinates) for QML custom
components (buttons, nav rows, list items) — those route through the
real Qt input pipeline with real hit testing and are far more
reliable. button: "left"/"right"/"middle" (anything else falls
back to left); modifiers: list of "Ctrl"/"Alt"/"Shift"/"Meta".
A result with ok:false means the element was rejected (hidden/
disabled/zero-size/stale id) — re-find and retry.
Inputs: element_id (required), session_id (required), button, modifiers. Returns: result
qt_mouse_click_atA
Click at an exact window coordinate, exactly like a real mouse click at that position.
x/y are relative to the target window's client area (same space as screenshots, logical pixels — DPI is handled internally). The click goes through the real Qt input pipeline, so the hit test (scene graph for QML, widget tree for QtWidgets) determines what receives it — identical behavior to a human clicking there, including real consequences (menus, close buttons, destructive actions). window_id: id of a top-level window from the MOST RECENT snapshot/find (ids expire on refresh); 0 uses the session's first visible top-level window — pass it explicitly when the session has multiple windows.
Inputs: x (required), y (required), session_id (required), button, +2 more. Returns: result
qt_mouse_click_regionA
Click at the center of an element's on-screen region.
Unlike qt_mouse_click (which delivers straight to the element), this routes through the real Qt input pipeline with real hit testing: for a QML container (e.g. a Rectangle with a MouseArea inside) the scene graph hit test delivers the click to the MouseArea, exactly as a human click would.
Inputs: element_id (required), session_id (required), button, modifiers. Returns: result
qt_mouse_dbl_clickB
Send a double-click to a UI element (direct delivery).
x/y are optional logical-pixel coordinates relative to the element's top-left corner; either both or neither (omitted = element center).
Inputs: element_id (required), session_id (required), x, y, +2 more. Returns: result
qt_mouse_moveA
Move the mouse pointer to an element-local position (no buttons).
Use between qt_mouse_press and qt_mouse_release to drag, or alone to hover. x/y are coordinates relative to the element's top-left corner.
Inputs: x (required), y (required), element_id (required), session_id (required). Returns: result
qt_mouse_pressA
Press a mouse button on an element WITHOUT releasing it.
Pair with qt_mouse_release to split a click, or qt_mouse_move for a drag (press -> move -> release). x/y are optional logical-pixel coordinates relative to the element's top-left corner; either both or neither (omitted = element center). IMPORTANT: after a press you must release in the same session, or the target app stays in a pressed/ dragging state.
Inputs: element_id (required), session_id (required), x, y, +2 more. Returns: result
qt_mouse_releaseA
Release a previously pressed mouse button on an element.
Completes a press/release pair (a click) or finishes a drag started with qt_mouse_press + qt_mouse_move. x/y are optional logical-pixel coordinates relative to the element's top-left corner; either both or neither (omitted = element center).
Inputs: element_id (required), session_id (required), x, y, +2 more. Returns: result
qt_mouse_wheelA
Scroll the mouse wheel over an element.
dx/dy are the wheel deltas in either pixel (pixel=True) or line
(default) units. x/y optionally position the wheel inside the
element (element-local logical px, both or neither — omitted = center).
Use this to scroll lists, tables, canvases and other scrollable
content.
Inputs: element_id (required), session_id (required), x, y, +3 more. Returns: result
qt_prune_snapshotA
Occlusion-prune a saved snapshot.
snapshot_id is the value returned by the last qt_snapshot call of
this session. When to use: after a snapshot, when elements overlap
inside the same window and you need to know which one is really on
top / how much of a target is visible — e.g. before deciding which
element to click or whether a covered element matters.
Reads the saved snapshot JSON and computes what is actually visible:
elements fully covered by higher-z opaque elements are removed (their
still-visible descendants are reparented up), partially covered ones
get a visible_ratio field, fully hidden (opacity 0) elements are
dropped. Writes snapshot_<id>_pruned.json next to the original —
read the tree at the returned uri (same objIDs as the source
snapshot plus visible_ratio annotations). Result field
pruned = {"removed", "kept", "removed_ratio"}; window roots are
never removed.
The solver is a geometric heuristic (axis-aligned rects, per-window z-order with same-z tree order; widgets and QML rectangles/images occlude unless semi-transparent, transparent containers, text and custom QML components do not; a parent never occludes its own children). Occlusion is per top-level window on purpose: the agent must stay able to operate an app that the user has covered or minimised, so windows never occlude each other. The objIDs in the pruned file expire with the next snapshot/find refresh like any other id.
Inputs: session_id (required), snapshot_id (required). Returns: result
qt_screenshotA
Capture a screenshot of a UI element or the entire window.
element_id 0 captures the active (or first visible) top-level
window. The PNG is written to the file at the returned uri —
read that resource (qt-commander://sessions/.../screenshots/N.png) to
obtain the image bytes. On failure the result reports the error and
no image is written.
Inputs: session_id (required), element_id. Returns: result
qt_set_propertyA
Write a property value on a UI element.
DESTRUCTIVE: directly modifies the target application's state (may trigger property-notify signals, change geometry/visibility/business state) with no undo. Prefer real user input (clicks/typing) to simulate user actions; use this only when a direct API write is intended.
value is first parsed as JSON: "true" -> bool, "123" -> int,
""hello"" -> string. Unparseable text is sent as a plain string —
pass quoted strings explicitly when a string is intended.
Inputs: name (required), value (required), element_id (required), session_id (required). Returns: result
qt_snapshotA
Capture the UI element tree of the session's windows.
The tree is NOT in this result — it is written to the file exposed at
the returned uri (resource qt-commander://sessions/.../snapshots/ snapshot_N.json); read that resource to inspect the tree. The result
carries only session_id / snapshot_id / uri. Node fields: className,
objID, objectName, rect (window-local logical px), global_rect (screen
coords), z_order, visible/enabled/opacity/color_alpha/clip, text,
topLevelId, windowTitle, properties (per detail).
detail — property tier per node: "core" (first-class fields only,
no properties), "extended" (DEFAULT; common interaction-state
properties like text/checked/value), "full" (every Q_PROPERTY; slow on
large UIs).
include_hidden (default False) — hidden elements are pruned
together with their whole subtree (hidden tabs, unopened dialogs).
Set True to include them; note hidden elements are still rejected by
click/input/property tools.
max_depth — levels of children: 0 = roots only, 1 = roots +
direct children, -1 = entire tree (default 1).
prop_depth — levels of QObject property expansion (0 = none,
-1 = unlimited).
root_id — 0 = all top-level windows; >0 = subtree of an element
from the MOST RECENT snapshot/find. Ids expire on every refresh: if
root_id no longer resolves, the tool silently falls back to all
top-level windows — pass it only when you know it is fresh.
ID LIFECYCLE: this call rebuilds the element map, so every element_id and window_id from previous snapshots/finds becomes invalid. The snapshot runs on the target process's GUI thread (brief UI freeze); if the target is busy, calls can time out after ~30s.
Inputs: session_id (required), detail, root_id, max_depth, +2 more. Returns: result
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 25 tools
Most tools target distinct resources and actions, but the three mouse click variants (qt_mouse_click, qt_mouse_click_region, qt_mouse_click_at) are closely related and could be confused. Their descriptions clearly explain the differences in delivery mechanism and use cases, so ambiguity is mostly resolved.
All tool names follow a consistent qt_verb_noun pattern with lowercase underscores (e.g., list_processes, get_property, mouse_click). Minor abbreviations like 'dbl' in qt_mouse_dbl_click are negligible and do not disrupt the overall predictability.
With 25 tools, the server is on the heavier side, but each tool maps to a distinct step in Qt UI automation—from environment detection and build to attach, inspection, and input simulation. The granularity is justified by the complexity of the domain.
The tool set covers the entire lifecycle: detect environment, build, attach, snapshot/find elements, read/write properties, call methods, simulate mouse and keyboard input, and detach. No obvious gaps prevent agents from accomplishing UI automation tasks.
Maintenance
Related MCP Connectors
Trading MCP server for AI agents, with live market data, account reads and controlled execution.
MCP server for building and testing AI agents with multi-model experimentation and insights.
MCP server for your apps' tools and custom tools, plus hosted AI agents and approval-gated workflows
MCP server for AI agents to plan, verify, and deploy Cloudflare-native apps.
Related MCP Servers
- AlicenseAqualityDmaintenanceAn MCP server for inspecting, debugging, and interacting with PySide6 desktop applications. It enables AI agents to capture UI snapshots, read widget properties, and perform actions like clicking buttons or typing text.223Apache 2.0
- AlicenseNot gradedqualityBmaintenanceGives AI agents and MCP clients direct control over native desktop apps, Chrome/Electron browsers, and Android devices with screenshots, OCR, accessibility-based element lookup, input simulation, window management, CDP, and ADB in one local server.142MIT
- AlicenseAqualityDmaintenanceAn MCP server for capturing screenshots of Qt/desktop windows and performing filesystem operations, enabling AI clients to inspect and modify project files.15MIT
- AlicenseAqualityCmaintenanceCross-platform desktop automation MCP server that lets AI agents capture screenshots, run OCR with UI-element classification, control mouse/keyboard, and launch programs on Linux, macOS, and Windows.202MIT