Windows Computer Use MCP
by cooolie
README.md
# Windows Computer Use MCP
[](https://github.com/cooolie/windows-computer-use-mcp/actions/workflows/tests.yml)
[](LICENSE)
A native Windows MCP server for desktop automation. It lets an MCP client
inspect and control ordinary Windows applications through UI Automation,
`SendInput`, and GDI screen capture — no macOS app bundle, Node runtime, or
vendor-specific agent required.
## Features
- **Accessibility-first**: every `get_app_state` returns a compact UI Automation
tree with element indices, roles, names, values, bounds, and supported actions.
- **Ten tools** for window discovery, snapshots, clicking, typing, key presses,
scrolling, dragging, text selection, value setting, and accessibility actions.
- **State-safe element actions**: element indices are tied to the `state_id` of
the snapshot that produced them. Bounds, enabled state, and offscreen state
are re-read when an action runs, so a moved or vanished control fails instead
of clicking a stale location.
- **DPI and multi-monitor aware**: windows, screenshots, and input use absolute
physical screen pixels across the virtual desktop.
- **Safety checks**: UIPI integrity preflight, top-level window hit testing,
cursor-movement verification, out-of-desktop coordinate rejection, and a
read-only mode that hides all input tools.
- **Small dependency footprint**: the only runtime dependency is `comtypes`;
PNG encoding is implemented with the standard library.
## Requirements
- Windows 10 or Windows 11, 64-bit.
- Python 3.10 or newer.
- [`comtypes`](https://pypi.org/project/comtypes/) for UI Automation. Without
it, `list_apps`, screenshots, and coordinate input still work; the
accessibility tree is unavailable.
The server does not require administrator rights. It checks process token
integrity before sending input and refuses targets at a higher integrity level
(Windows UIPI). It cannot operate on the secure desktop (UAC prompt or lock
screen).
## Install
```powershell
git clone https://github.com/cooolie/windows-computer-use-mcp.git
cd windows-computer-use-mcp
python -m pip install -e .
```
Verify the test suite:
```powershell
python -m unittest discover -s tests -v
```
### Configure an MCP client
Add a stdio server entry to your MCP client's configuration. For Claude Code,
the user-scope configuration lives in the top-level `mcpServers` object of
`~/.claude.json`:
```json
{
"mcpServers": {
"computer-use-windows": {
"type": "stdio",
"command": "C:\\path\\to\\python.exe",
"args": ["-m", "computer_use_windows"],
"env": {
"PYTHONUNBUFFERED": "1",
"PYTHONIOENCODING": "utf-8",
"PYTHONUTF8": "1"
},
"timeout": 120000
}
}
}
```
Use the absolute path to the Python interpreter where the package is installed
(for example, a virtual environment's `python.exe`). Claude Code inherits only a
small allowlist of environment variables on Windows, so a bare `python` may not
resolve to the intended interpreter.
To run directly from a checkout without installing the package, install the
dependency and point the client at the launcher instead:
```powershell
python -m pip install -r requirements.txt
```
```json
{
"command": "C:\\path\\to\\python.exe",
"args": ["C:\\path\\to\\windows-computer-use-mcp\\bin\\computer-use-mcp-server.py"]
}
```
The server also includes `.claude-plugin/plugin.json` if you package it in a
Claude Code plugin marketplace.
### Optional Claude Code skill
The repository ships a skill that teaches Claude Code when and how to use the
tools. Copy it into your personal skills directory:
```powershell
Copy-Item skills\computer-use\SKILL.md `
"$HOME\.claude\skills\computer-use-windows\SKILL.md"
```
Restart Claude Code. The tools appear as `mcp__computer-use-windows__*`.
## Tools
| Tool | Purpose |
|---|---|
| `list_apps` | List visible top-level windows with process, title, class, hwnd, and bounds. |
| `get_app_state` | Activate a window, return its accessibility tree, and capture a screenshot. |
| `click` | Click an element center or absolute screen coordinates with optional modifiers. |
| `set_value` | Set a value through ValuePattern, falling back to focus + select-all + typing. |
| `type_text` | Type Unicode text into the focused control. |
| `press_key` | Press a key or combination such as `Return`, `ctrl+s`, or `alt+f4`. |
| `scroll` | Rotate the mouse wheel at an element or coordinate. |
| `drag` | Drag between two absolute screen coordinates. |
| `select_text` | Select matching text inside a TextPattern element. |
| `perform_secondary_action` | Invoke an action the element explicitly advertises. |
### Typical loop
1. `list_apps` to find the target window.
2. `get_app_state` with the app name, title, process, or hwnd. Record the
returned `state_id`.
3. Act on an `element_index` from that state, passing the same `state_id`.
4. Call `get_app_state` again before deciding the next action.
All element indices and `state_id` values expire on the next snapshot.
## Coordinates and screenshots
- Element rectangles and input coordinates are absolute physical screen pixels
on the virtual desktop. Multi-monitor coordinates may be negative.
- Screenshots may be scaled down. The result reports `origin` and `scale`, where
`scale` is image pixels divided by screen pixels:
`screen = origin + image_pixel / scale`.
- A screenshot captures the visible screen rectangle, not an offscreen window
surface. If the target cannot be brought to the foreground, the screenshot is
omitted rather than capturing an unrelated window.
- UI Automation trees from Chromium/Electron applications are often sparse; use
the screenshot and coordinates for those apps.
## Safety and limits
- The server targets existing windows; it does not launch applications. Start
the app with the normal shell or CLI first.
- Set `COMPUTER_USE_WINDOWS_READ_ONLY=1` to expose only `list_apps` and
`get_app_state`. In read-only mode `get_app_state` does not activate the
window and captures no screenshot unless the target is already foreground.
- Screenshots can contain sensitive data. Do not send them to external services
or include them in reports without the user's approval.
- Input is refused when the server or target integrity level cannot be read, or
when the target is elevated above the server.
- Cloaked windows are excluded from `list_apps`; minimized windows are listed
with `minimized=yes` and `get_app_state` restores them.
- The server is single-threaded and handles one tool call at a time.
See [SECURITY.md](SECURITY.md) for the security policy and private reporting
instructions.
## Project layout
```text
bin/computer-use-mcp-server.py dependency-free checkout launcher
computer_use_windows/
server.py MCP stdio loop and protocol handling
schemas.py tool definitions and JSON schemas
tools.py dispatch and tool implementations
uia.py UI Automation snapshots and actions
win32.py ctypes bindings
apps.py window enumeration and activation
input.py SendInput keyboard and mouse injection
capture.py GDI screen capture
keys.py key-combination parser
png.py dependency-free PNG encoder
geometry.py rectangle helpers
skills/computer-use/SKILL.md optional Claude Code skill
tests/ unit and stdio integration tests
```
## Development
```powershell
python -m pip install -e .
python -m unittest discover -s tests -v
```
The tests run on Windows and do not inject mouse or keyboard input. The stdio
integration test starts the server, performs the MCP handshake, lists tools, and
calls `list_apps`. GitHub Actions runs the suite on Python 3.10 and 3.12.
## License
MIT. See [LICENSE](LICENSE).
This server cannot be deployed
Maintenance
ActivityMaintained
ResponsivenessNo recent community issues