ApplicationController

An MCP server that exposes deterministic desktop UI automation tools (PyAutoGUI + UIAutomation) for MCP-compatible clients (for example GitHub Copilot desktop app).

What improved

This version focuses on speed and reliability for real WPF automation:

1. Coordinate/DPI correctness

   • Coordinate space is explicit: physical (default) or logical.

   • Added convert_coordinates.

   • Coordinate-aware tools return coordinate metadata (space, dpi, scale).

2. Deterministic scoped UIA actions

   • scope_hwnd support added for deterministic exact-handle scoping.

   • Scope resolution order: scope_hwnd exact → exact title variants → robust contains fallback.

   • ua_find, ua_invoke, ua_get_value, ua_set_value return structured diagnostics and error codes.

3. Fast-path action

   • Added click_in_window(title|hwnd, button|automation_id|name, wait_until_idle, retries).

   • Handles focus, control lookup, invoke/click fallback, retries, and optional post-action idle wait.

4. Dialog/modal handling

   • Added close_foreground_dialog().

   • Added dismiss_dialog(title|hwnd, button, wait_timeout).

   • Added post-action new-child-window detection in click_in_window.

5. State waits (instead of sleeps)

   • wait_for_window now supports regex and fast polling.

   • Added wait_for_control_change (exists, visible, enabled, value/text change/equality).

   • Added wait_ui_idle (input-idle + stable window rect settle).

6. Batch efficiency

   • batch now shares resolved scope context across steps to avoid repeated lookups.

7. Observability

   • Structured error codes (WINDOW_NOT_FOUND, CONTROL_AMBIGUOUS, TIMEOUT, etc.).

   • Timing breakdowns in responses (lookup_ms, action_ms, total_ms where applicable).

8. Backward compatibility

   • Existing tool names remain: ua_*, wait_for_element, auto_dismiss_dialog, batch, etc.

   • wait_for_element remains available (implemented as compatible wait path).

   • auto_dismiss_dialog remains available (alias of improved dismiss_dialog).

Related MCP server: mcp-autogui-multinode

Tool summary

Screenshot & observation

Mouse & keyboard

UIAutomation

Waiting & dialogs

Other

Coordinate-space behavior

• Default input space is physical for coordinate tools (click, move_mouse, drag, scroll, capture_region, etc.).

• Pass coordinate_space: "logical" for DPI-scaled inputs.

• Output responses include coordinate metadata:

{
	"coordinate": {
		"space": "logical",
		"dpi": 144,
		"scale": 1.5
	}
}

Use convert_coordinates when moving between desktop APIs that report different spaces.

Diagnostics and error codes

Errors are structured:

{
	"success": false,
	"error": {
		"code": "CONTROL_NOT_FOUND",
		"message": "No matching control found"
	},
	"diagnostics": {
		"matched_windows": [],
		"controls_considered": []
	}
}

Common codes:

• WINDOW_NOT_FOUND

• WINDOW_AMBIGUOUS

• CONTROL_NOT_FOUND

• CONTROL_AMBIGUOUS

• STALE_HANDLE

• TIMEOUT

• ACTION_FAILED

• DEPENDENCY_MISSING

• INVALID_ARGUMENT

Recommended fast flow (WPF)

1. list_windows and pick hwnd.

2. click_in_window with hwnd + automation_id/button.

3. wait_for_control_change or wait_ui_idle for deterministic completion.

4. ua_get_value for assertion/readback.

Migration notes

1. Legacy scripts using coordinate clicks continue to work (physical remains default).

2. For DPI-sensitive scenarios, switch to coordinate_space: "logical" or explicit convert_coordinates.

3. Replace sleep-heavy sequences with wait_for_window, wait_for_control_change, and wait_ui_idle.

4. For scope flakiness, pass scope_hwnd instead of fuzzy title-only scoping.

Setup

1. Install dependencies

pip install -r requirements.txt

To enable image search:

pip install opencv-python

or:

pip install ".[image-search]"

2. Add to GitHub Copilot desktop app

{
	"mcpServers": {
		"application-controller": {
			"command": "python",
			"args": ["C:\\path\\to\\ApplicationController\\server.py"]
		}
	}
}

3. Safety note

PyAutoGUI failsafe is enabled: move mouse to top-left to abort.

Benchmark script and results

scripts/benchmark_before_after.py provides a short before-vs-after benchmark harness for:

1. app launch + dialog dismissal

2. opening a target window via button click

3. waiting for ready state and reading control value

Example run results (HPMAGS Travel Sheet scenario):

Detailed machine-readable output is included in scripts/benchmark_results.json.