Skip to main content
Glama
Windows Computer Use MCP

Windows Computer Use MCP

by cooolie
README.md
# Windows Computer Use MCP

[![tests](https://github.com/cooolie/windows-computer-use-mcp/actions/workflows/tests.yml/badge.svg)](https://github.com/cooolie/windows-computer-use-mcp/actions/workflows/tests.yml)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)

A native Windows MCP server for desktop automation. It lets an MCP client
inspect and control ordinary Windows applications through UI Automation,
`SendInput`, and GDI screen capture — no macOS app bundle, Node runtime, or
vendor-specific agent required.

## Features

- **Accessibility-first**: every `get_app_state` returns a compact UI Automation
  tree with element indices, roles, names, values, bounds, and supported actions.
- **Ten tools** for window discovery, snapshots, clicking, typing, key presses,
  scrolling, dragging, text selection, value setting, and accessibility actions.
- **State-safe element actions**: element indices are tied to the `state_id` of
  the snapshot that produced them. Bounds, enabled state, and offscreen state
  are re-read when an action runs, so a moved or vanished control fails instead
  of clicking a stale location.
- **DPI and multi-monitor aware**: windows, screenshots, and input use absolute
  physical screen pixels across the virtual desktop.
- **Safety checks**: UIPI integrity preflight, top-level window hit testing,
  cursor-movement verification, out-of-desktop coordinate rejection, and a
  read-only mode that hides all input tools.
- **Small dependency footprint**: the only runtime dependency is `comtypes`;
  PNG encoding is implemented with the standard library.

## Requirements

- Windows 10 or Windows 11, 64-bit.
- Python 3.10 or newer.
- [`comtypes`](https://pypi.org/project/comtypes/) for UI Automation. Without
  it, `list_apps`, screenshots, and coordinate input still work; the
  accessibility tree is unavailable.

The server does not require administrator rights. It checks process token
integrity before sending input and refuses targets at a higher integrity level
(Windows UIPI). It cannot operate on the secure desktop (UAC prompt or lock
screen).

## Install

```powershell
git clone https://github.com/cooolie/windows-computer-use-mcp.git
cd windows-computer-use-mcp
python -m pip install -e .
```

Verify the test suite:

```powershell
python -m unittest discover -s tests -v
```

### Configure an MCP client

Add a stdio server entry to your MCP client's configuration. For Claude Code,
the user-scope configuration lives in the top-level `mcpServers` object of
`~/.claude.json`:

```json
{
  "mcpServers": {
    "computer-use-windows": {
      "type": "stdio",
      "command": "C:\\path\\to\\python.exe",
      "args": ["-m", "computer_use_windows"],
      "env": {
        "PYTHONUNBUFFERED": "1",
        "PYTHONIOENCODING": "utf-8",
        "PYTHONUTF8": "1"
      },
      "timeout": 120000
    }
  }
}
```

Use the absolute path to the Python interpreter where the package is installed
(for example, a virtual environment's `python.exe`). Claude Code inherits only a
small allowlist of environment variables on Windows, so a bare `python` may not
resolve to the intended interpreter.

To run directly from a checkout without installing the package, install the
dependency and point the client at the launcher instead:

```powershell
python -m pip install -r requirements.txt
```

```json
{
  "command": "C:\\path\\to\\python.exe",
  "args": ["C:\\path\\to\\windows-computer-use-mcp\\bin\\computer-use-mcp-server.py"]
}
```

The server also includes `.claude-plugin/plugin.json` if you package it in a
Claude Code plugin marketplace.

### Optional Claude Code skill

The repository ships a skill that teaches Claude Code when and how to use the
tools. Copy it into your personal skills directory:

```powershell
Copy-Item skills\computer-use\SKILL.md `
  "$HOME\.claude\skills\computer-use-windows\SKILL.md"
```

Restart Claude Code. The tools appear as `mcp__computer-use-windows__*`.

## Tools

| Tool | Purpose |
|---|---|
| `list_apps` | List visible top-level windows with process, title, class, hwnd, and bounds. |
| `get_app_state` | Activate a window, return its accessibility tree, and capture a screenshot. |
| `click` | Click an element center or absolute screen coordinates with optional modifiers. |
| `set_value` | Set a value through ValuePattern, falling back to focus + select-all + typing. |
| `type_text` | Type Unicode text into the focused control. |
| `press_key` | Press a key or combination such as `Return`, `ctrl+s`, or `alt+f4`. |
| `scroll` | Rotate the mouse wheel at an element or coordinate. |
| `drag` | Drag between two absolute screen coordinates. |
| `select_text` | Select matching text inside a TextPattern element. |
| `perform_secondary_action` | Invoke an action the element explicitly advertises. |

### Typical loop

1. `list_apps` to find the target window.
2. `get_app_state` with the app name, title, process, or hwnd. Record the
   returned `state_id`.
3. Act on an `element_index` from that state, passing the same `state_id`.
4. Call `get_app_state` again before deciding the next action.

All element indices and `state_id` values expire on the next snapshot.

## Coordinates and screenshots

- Element rectangles and input coordinates are absolute physical screen pixels
  on the virtual desktop. Multi-monitor coordinates may be negative.
- Screenshots may be scaled down. The result reports `origin` and `scale`, where
  `scale` is image pixels divided by screen pixels:
  `screen = origin + image_pixel / scale`.
- A screenshot captures the visible screen rectangle, not an offscreen window
  surface. If the target cannot be brought to the foreground, the screenshot is
  omitted rather than capturing an unrelated window.
- UI Automation trees from Chromium/Electron applications are often sparse; use
  the screenshot and coordinates for those apps.

## Safety and limits

- The server targets existing windows; it does not launch applications. Start
  the app with the normal shell or CLI first.
- Set `COMPUTER_USE_WINDOWS_READ_ONLY=1` to expose only `list_apps` and
  `get_app_state`. In read-only mode `get_app_state` does not activate the
  window and captures no screenshot unless the target is already foreground.
- Screenshots can contain sensitive data. Do not send them to external services
  or include them in reports without the user's approval.
- Input is refused when the server or target integrity level cannot be read, or
  when the target is elevated above the server.
- Cloaked windows are excluded from `list_apps`; minimized windows are listed
  with `minimized=yes` and `get_app_state` restores them.
- The server is single-threaded and handles one tool call at a time.

See [SECURITY.md](SECURITY.md) for the security policy and private reporting
instructions.

## Project layout

```text
bin/computer-use-mcp-server.py   dependency-free checkout launcher
computer_use_windows/
  server.py                      MCP stdio loop and protocol handling
  schemas.py                     tool definitions and JSON schemas
  tools.py                       dispatch and tool implementations
  uia.py                         UI Automation snapshots and actions
  win32.py                       ctypes bindings
  apps.py                        window enumeration and activation
  input.py                       SendInput keyboard and mouse injection
  capture.py                     GDI screen capture
  keys.py                        key-combination parser
  png.py                         dependency-free PNG encoder
  geometry.py                    rectangle helpers
skills/computer-use/SKILL.md     optional Claude Code skill
tests/                           unit and stdio integration tests
```

## Development

```powershell
python -m pip install -e .
python -m unittest discover -s tests -v
```

The tests run on Windows and do not inject mouse or keyboard input. The stdio
integration test starts the server, performs the MCP handshake, lists tools, and
calls `list_apps`. GitHub Actions runs the suite on Python 3.10 and 3.12.

## License

MIT. See [LICENSE](LICENSE).