# computer-mcp 🖥️

**A Windows desktop-control MCP server for AI agents** — Python / FastMCP, stdio transport, **45 tools** in one file.

Give any MCP-capable agent real eyes and hands on Windows — UIA element-based control, native-image screenshots, instant app launching, batched action sequences, smart waits, and safety gates — in a **0–1000 normalized coordinate system** so it never guesses pixels.

![badge-python](https://img.shields.io/badge/python-3.10%2B-blue) ![badge-platform](https://img.shields.io/badge/OS-Windows%2010%2F11-0078d6) ![badge-mcp](https://img.shields.io/badge/MCP-stdio-black) ![badge-tools](https://img.shields.io/badge/tools-45-green) ![badge-license](https://img.shields.io/badge/licence-MIT-yellow)

*This is NOT the npm `@zavora-ai/computer-use-mcp` and NOT Codex's bundled `computer-use` — it is this repo's single-file Python server.*

---

## Why this exists

Most computer-use setups share two fatal habits:

1. **Screenshot-and-guess** — the model receives images, estimates pixel coordinates, and mis-clicks at any non-standard resolution/DPI.
2. **Token burn** — every state check costs a full screenshot.

`computer-mcp` inverts that:

| Habit | Instead |
|---|---|
| Guess pixels from images | **UIA element trees** — exact names, roles, and coords of every button/label/field (`get_ui_tree`, `find_element`, `click_element`) — *zero image tokens* |
| Screenshot to check state | **Cheap state checks** — `get_screen_info`, `list_windows`, `diff_screenshot` (returns *changed regions*, not images), `wait_for_pixel`, `get_pixel_color` |
| Many slow round-trips | **`batch()`** — click → paste → wait → verify as **one tool call** (40 actions, 120 s cap) |
| Type via simulated keys | **`paste_text`** (clipboard + Ctrl+V) for unicode/long text — preserves the user clipboard by default |
| Launch apps by poking the Start menu | **`open_app()`** resolves Start-Menu shortcuts, Microsoft Store AUMIDs, App-Paths registry and PATH — launches *instantly*, no UI theatre |

Everything drives the **real desktop** at your user privileges; designed to run quietly in the background while you keep working.

---

## Requirements

| | |
|---|---|
| OS | Windows 10 or 11 (single-file server; runs on your session, no admin needed) |
| Python | 3.10+ on PATH ([python.org](https://www.python.org/downloads/)) |
| Screen | Any resolution / DPI — coordinates are normalized 0–1000 |
| Optional | `opencv-python` for confidence-based template matching (included in requirements) |

## Install

```powershell
git clone https://github.com/Tb3bhargav/computer-mcp.git
cd computer-mcp

# (recommended) isolated venv
python -m venv .venv
.\.venv\Scripts\pip install -r requirements.txt

# smoke test — should exit 0 and print nothing
python -m py_compile server.py
```

The server is **not** started by hand: your MCP client launches it over stdio and talks the Model Context Protocol. The config snippets below show exactly how.

> **💡 Pick the right interpreter**
> In client configs, point `command` at your venv's interpreter for zero global installs:
> `C:/path/to/computer-mcp/.venv/Scripts/python.exe`
> On Windows the launcher is typically `python` (or `python.exe` under `...\Programs\Python\Python311\`). Use **forward slashes** in JSON paths.

---

## Add it to your MCP client

All snippets assume the repo lives at
`C:/Users/nj322ws/OneDrive/Desktop/My Projects/computer-mcp` — adjust to where you cloned it.

### Generic `mcp.json` — Claude Desktop · Cursor · Cline · Continue · Windsurf

| Client | Config file |
|---|---|
| Claude Desktop | `%APPDATA%\Claude\claude_desktop_config.json` |
| Cursor | `~/.cursor/mcp.json` |
| Cline | `cline_mcp_settings.json` (Cline → MCP Servers → Configure) |
| Continue | `~/.continue/config.json` |
| Windsurf | `~/.codeium/windsurf/mcp_config.json` |

```json
{
  "mcpServers": {
    "computer": {
      "command": "python",
      "args": ["C:/Users/nj322ws/OneDrive/Desktop/My Projects/computer-mcp/server.py"]
    }
  }
}
```

### OpenCode — global `~/.config/opencode/opencode.json` (production-tested setup)

```json
{
  "mcp": {
    "computer": {
      "type": "local",
      "command": [
        "C:/Users/nj322ws/AppData/Local/Programs/Python/Python311/python.exe",
        "C:/Users/nj322ws/OneDrive/Desktop/My Projects/computer-mcp/server.py"
      ],
      "cwd": "C:/Users/nj322ws/OneDrive/Desktop/My Projects/computer-mcp",
      "enabled": true,
      "timeout": 180000
    }
  },
  "permission": {
    "computer_*": "allow"
  }
}
```

`"permission": { "computer_*": "allow" }` whitelists every tool name of this server in OpenCode. Drop it if you prefer to approve each desktop-control call manually.

OpenCode also supports per-project config with the same shape (e.g. project `opencode.jsonc` or `.opencode/opencode.jsonc`):

```jsonc
{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "servers": {
      "computer": {
        "type": "local",
        "command": ["python", "C:/Users/nj322ws/OneDrive/Desktop/My Projects/computer-mcp/server.py"]
      }
    }
  },
  "permission": { "computer_*": "allow" }
}
```

### VS Code (GitHub Copilot) — `.vscode/mcp.json`

```json
{
  "servers": {
    "computer": {
      "type": "stdio",
      "command": "python",
      "args": ["C:/Users/nj322ws/OneDrive/Desktop/My Projects/computer-mcp/server.py"]
    }
  }
}
```

### Codex CLI — `~/.codex/config.toml`

```toml
[mcp_servers.computer]
command = "python"
args = ["C:/Users/nj322ws/OneDrive/Desktop/My Projects/computer-mcp/server.py"]
```

### Claude Code — CLI one-liner

```powershell
claude mcp add computer -- python "C:/Users/nj322ws/OneDrive/Desktop/My Projects/computer-mcp/server.py"
```

### Gemini CLI — `~/.gemini/settings.json`

```json
{
  "mcpServers": {
    "computer": {
      "command": "python",
      "args": ["C:/Users/nj322ws/OneDrive/Desktop/My Projects/computer-mcp/server.py"]
    }
  }
}
```

> **Verify it works:** restart the client, start a chat, and ask *"what's on my screen?"* — the agent should call `get_screen_info()` (reported in plain text, free tokens) instead of grabbing an image.

---

## The 0–1000 coordinate system

Every position the agent passes in — clicks, regions, polls — is normalized:

- `(0,0)` = top-left · `(1000,1000)` = bottom-right of the primary display.
- Internally converted to real pixels: resolution- and DPI-independent.
- Anything returned (element boxes, image matches, diff cells) is also 0–1000.

---

## All 45 tools

### Perception — see without burning image tokens

| Tool | What it does |
|---|---|
| `get_screen_info()` | OS, screen size, mouse position, active window + owning process, monitor list — **no image** |
| `get_system_info()` | Capability report: OS/arch/python, pygetwindow/opencv availability |
| `list_windows()` | Open windows (title, rect, active) — ghost/DWM-cloaked windows filtered out |
| `read_screen_text(root, window, depth)` | All visible UI text of the active/named window as one string — **no image** |
| `diff_screenshot(region, threshold)` | What changed since last capture: changed %, region cells, union bbox — **no image** |

### UI elements (UI Automation) — the fast path

| Tool | What it does |
|---|---|
| `get_ui_tree(root, window, depth, interactive_only)` | The AI DOM dump: every element with index/role/name/box/center in 0–1000 |
| `find_element(text, ...)` | Case-insensitive match on visible text → roles + coords |
| `click_element(text, occurrence, double_click, right_click)` | Click an element **by its visible label** — re-searched fresh at click time, never stale |
| *(all above also accept `root:"screen"`)* | Read the whole desktop, not one window |

### Screenshots & vision

| Tool | What it does |
|---|---|
| `take_screenshot(region, max_width, quality, grayscale)` | Native image to the agent; `region=[x1,y1,x2,y2]` crops to save tokens |
| `save_screenshot(path, region)` | Full-res PNG to disk — for evidence or template building |
| `find_image(image_path, confidence, region)` | Locate a template PNG → 0–1000 center + box |
| `get_pixel_color(x, y)` | RGB/hex of one pixel — poll toggles/spinners without images |

### Windows & apps

| Tool | What it does |
|---|---|
| `activate_window(title_contains)` | Focus by title substring (exact > startswith > process-name > contains), restores minimized |
| `window_control(title_contains, action)` | minimize / maximize / restore / close |
| `open_app(name)` | **Instant** launch of *any* installed app: Start-Menu shortcuts, Store apps (AUMID), App-Paths registry, PATH, URIs (`ms-settings:`) |
| `launch_app(query, wait_after)` | OS-aware launch with wait (Spotlight on macOS, direct exec on Linux; Start-flow fallback on Windows) |
| `list_installed_apps(filter, limit)` | Everything openable, cheap and cached |
| `list_agent_desktops()` / `create_agent_desktop()` / `switch_agent_desktop(to_agent)` / `close_agent_desktop()` | Dedicated Windows virtual desktop that only the agent works on — user's desktop untouched |

### Mouse

| Tool | What it does |
|---|---|
| `mouse_move(x, y)` | Hover / tooltips |
| `mouse_click(x, y, button, clicks)` | Left/right/middle, single or multi-click |
| `mouse_down(button)` / `mouse_up(button)` | Press-and-hold primitives |
| `scroll(x, y, clicks, horizontal)` | Scroll wheel at a position |
| `drag(from_x, from_y, to_x, to_y, button, duration)` | Sliders, selections, window moves |

### Keyboard & text

| Tool | What it does |
|---|---|
| `type_text(text)` | ASCII typewriting (~50 chars/s) |
| `paste_text(text, restore_clipboard)` | ⭐ Reliable entry for long/unicode/symbol text — clipboard + Ctrl+V, **restores your clipboard afterwards** |
| `press_key(keys)` | `enter`, `ctrl+c`, `alt+tab`, `win+r` … |
| `key_down(key)` / `key_up(key)` | Hold-and-release (e.g. shift for multi-select) |
| `set_clipboard(text)` / `get_clipboard()` | Direct clipboard control |

### Batch — 12 action types, one call

| Tool | What it does |
|---|---|
| `batch(actions, stop_on_error, verify_screenshot)` | Runs a sequence in ONE round-trip: `click`, `double_click`, `right_click`, `move`, `type`, `paste`, `press`, `scroll`, `wait`, `focus_window`, `launch`, `click_text` — with a 120 s wall-clock cap and optional trailing verification screenshot |

### Waits — sync without blind sleeping

| Tool | What it does |
|---|---|
| `wait(seconds)` | Sleep (max 300 s); prefer the targeted waits below |
| `wait_for_image(image_path, timeout, confidence, region, poll)` | Poll until a template appears (dialog/loader done) |
| `wait_for_stable(timeout, poll, threshold, region)` | Block until the screen stops changing (page load, progress bar) |
| `wait_for_pixel(x, y, hex_color, timeout, tolerance)` | Poll one pixel until it matches a color — spinners, toggle states |
| `wait_until_image_gone(...)` | Block until a template disappears (modal closed) |

### Safety & control state

| Tool | What it does |
|---|---|
| `set_safety_mode(enable)` | ON: destructive actions require explicit confirmation |
| `confirm_destructive(action_id)` | Approve a pending destructive op returned by the safety gate |
| `control_status()` | Live state of the control machinery (mode, cancel flag, overlay state) |
| `set_overlay_mode(auto/always/off)` | Optional "AI is using your computer" overlay preference (silent `off` is default) |
| `resume_control()` | Clears the Esc-cancel latch after a user pause |

---

## The agent workflow this server is built around

The golden loop — absorb it once with the agent of your choice:

```
STATE (free)         get_screen_info / list_windows
TARGETS (free)       get_ui_tree → find_element → read_screen_text
ACT                  click_element / batch / paste_text / press_key
VERIFY (free)        diff_screenshot / wait_for_pixel / wait_for_image
ONLY IF NEEDED       take_screenshot(region=...) for actual visual truth
```

**Design principles baked into the tools:**

- Perception first, pixels last — a healthy session uses screenshots only for final visual proof.
- `click_element` re-finds its target **at click time** — no stale coordinates between two UI states.
- Windows are matched by **exact title → prefix → owning process name → substring**, so `"whatsapp"` finds WhatsApp even when the title says "John • WhatsApp".
- `list_windows`/window matching filters DWM-cloaked ghosts invisible to the user.

## Recipes

<details>
<summary><b>1 · Launch an app and act inside it</b></summary>

```python
open_app("chrome", wait_after=4)
find_element("New tab")          # zero-image locate
click_element("New tab")
```
</details>

<details>
<summary><b>2 · Send a WhatsApp message in ONE call</b></summary>

```python
batch([
  {"focus_window": "WhatsApp"},
  {"press": "ctrl+f"}, {"paste": "Bhavna"},
  {"wait": 1.5}, {"press": "enter"}, {"wait": 2},
  {"click_text": "Bhavna Didi"},
  {"paste": "Compliments on the launch 🎉"},
  {"press": "enter"}
], verify_screenshot=True)
```
</details>

<details>
<summary><b>3 · Fill a form, keep the clipboard intact</b></summary>

```python
click_element("Email")        # focus first field
paste_text("user@example.com", restore_clipboard=True)
press_key("tab")
paste_text("S3cret Password")
press_key("enter")
diff_screenshot()             # what changed — without an image
```
</details>

<details>
<summary><b>4 · Verify an effect without a screenshot</b></summary>

```python
click_element("Submit")
wait_for_pixel(500, 820, "#22c55e")   # success badge turns green
diff_screenshot(region=[380, 760, 620, 860])
```
</details>

<details>
<summary><b>5 · Template-based UI matching</b></summary>

```python
save_screenshot("templates/save-btn.png", region=None)   # build templates once
find_image("templates/save-btn.png", confidence=0.9)
wait_until_image_gone("templates/save-btn.png")          # modal closed?
```
</details>

<details>
<summary><b>6 · Work on an isolated desktop</b></summary>

```python
create_agent_desktop()       # sandbox ahead
switch_agent_desktop()       # agent now operates there
# ... any work ...
switch_agent_desktop(False)  # back to the user's screen
close_agent_desktop()
```
</details>

<details>
<summary><b>7 · Safety mode end-to-end</b></summary>

```python
set_safety_mode(True)
press_key("delete")
# → "SAFETY MODE: destructive action pending — press_key('delete') ...
#     Call confirm_destructive('perm_1')"
confirm_destructive("perm_1")    # now it runs
```
</details>

---

## Safety

> [!WARNING]
> This server gives the agent **your hands**: real mouse, keyboard, clipboard and app launches are executed with your user privileges on your real desktop. Do **not** attach it to untrusted agents or models.

- **Explicit permissions** — In OpenCode, only set `"permission": { "computer_*": "allow" }` for agents you trust; leave it out to approve every desktop call.
- **Safety mode** — `set_safety_mode(True)` pends destructive actions (delete/backspace key events) behind `confirm_destructive`; default OFF.
- **Isolated desktops** — route risky visual work to `create_agent_desktop()` so the user's screen stays untouched.
- **Silent control by default** — control runs in the background; the takeover-overlay engine remains available (`control_status`, `set_overlay_mode`).
- **Esc-to-cancel** — the overlay engine's global Esc hook latches an emergency pause state whenever it is active; `resume_control()` clears it.

## How it works (in one screen)

- **One file, `server.py`** — a FastMCP stdio server; every tool is a plain typed function.
- **OS + screen detected at startup** and logged to stderr (stdout is reserved for the MCP protocol).
- **UIA all on one thread** — Windows UI Automation COM objects are apartment-bound, so every UIA call is serialized onto a single worker with per-call timeouts.
- **App launcher index** — a TTL-cached (10 min) merge of Start-Menu shortcuts + `Get-StartApps` (incl. Microsoft Store AUMIDs) + App-Paths registry + PATH.
- **Screenshot baselines** — `diff_screenshot` keeps a downscaled grayscale baseline and returns changed grid cells + union bbox instead of pixels, so the agent only grabs a cropped image if it truly needs to see the change.
- **Graceful degradation** — missing optional deps reduce features (no numpy → slower diffing) but never crash the server; window matching has layered fallbacks (exact → prefix → process name → substring, then "best available window").

## Testing — real results on a real machine

15-file report suite: **12 scenarios, all PASS**, run October 2026 on the author's
daily Windows 11 machine (OneDrive-synced, in active use). Full evidence in
[`reports/`](reports/README.md):

| Test | Proves | Result |
|---|---|---|
| [Foundation 1–3](reports/README.md) | OneDrive path resolution, write persistence (7 formats), surgical edits | PASS |
| [Self-Healing Coder](reports/Level-4-1-Self-Healing-Coder.md) | build → serve → verify → find own 404 → fix → re-verify | PASS w/ honors |
| [Librarian from Hell](reports/Level-4-2-Librarian-From-Hell.md) | 460-file scan + OCR + honest "no text found" | PASS |
| [GitHub Janitor](reports/Level-4-3-GitHub-Janitor.md) | 36-repo scan → branch → PR, `main` untouched | PASS |
| [Persistence](reports/Level-4-4-Persistence.md) | reboot-proof scheduled task | PASS |
| [Mission 1 – Executive Assistant](reports/Mission-1-Executive-Assistant.md) | Gmail → Excel → PPTX, zero-hallucination data flow | PASS |
| [Mission 2 – Data Janitor](reports/Mission-2-Data-Janitor-Analyst.md) | OCR + **winget self-install of a missing dependency** + citations | PASS |
| [Mission 3 – Chaos Organizer](reports/Mission-3-Chaos-Organizer.md) | organize + **byte-perfect undo**, non-disturb operation | PASS |
| [Mission 4 – Full-Stack Ghost](reports/Mission-4-Full-Stack-Ghost.md) | local edit → push → live PR verification | PASS |
| [Mission 5 – Deep State](reports/Mission-5-Deep-State.md) | live research + honest N/A + no-focus-steal screenshots | PASS |

## Troubleshooting

<details>
<summary><b><code>python</code> not recognized / opens Microsoft Store</b></summary>

Install Python from python.org and tick *Add to PATH*. If the Windows Store alias persists, disable *App execution aliases* (Settings → Apps → Advanced app settings), or point configs at `python.exe`'s full path.
</details>

<details>
<summary><b>Server starts but the client shows no tools</b></summary>

Check the client's MCP logs — the server writes an OS/screen banner to stderr at startup. Confirm the absolute path in your config is exact (it lives under user folders by design) and restart the client fully.
</details>

<details>
<summary><b>UIA tools return nothing in Chrome/apps</b></summary>

Chromium browsers and some apps only expose accessibility info on demand. Chrome/Edge: enable accessibility in the app or set the automation flag — UIA then sees the DOM. Win32 apps (Notepad, Explorer, Office, most desktop apps) expose it natively.
</details>

<details>
<summary><b>find_image weaker than expected</b></summary>

Template matching needs `opencv-python` for the confidence parameter (listed in requirements). Rebuild templates with `save_screenshot` right before matching, same scale/theme.
</details>

<details>
<summary><b>Coordinates feel off</b></summary>

They're normalized — scenarios: mixed resolutions or Windows scaling changes make real pixel deltas unreliable; any x/y you pass is /1000-mapped to the primary display and never needs recalibration.
</details>

---

## Contributing

Issues and PRs welcome — especially for the per-OS abstractions. Keep the runtime a single importable `server.py` (the whole point), and please add a short report under `reports/` for behavior changes.

## License

[MIT](LICENSE) — free for any use, including commercial; keep the copyright notice in copies. © 2026 Tejus Bhargav.

---

<br>
<p align="center">
  <sub>Engineered and torture-tested on a real Windows machine — <a href="https://github.com/Tb3bhargav">Tb3bhargav</a> · If this saved you time, give it a star ⭐</sub>
</p>