computer-mcp 🖥️

A Windows desktop-control MCP server for AI agents — Python / FastMCP, stdio transport, 45 tools in one file.

Give any MCP-capable agent real eyes and hands on Windows — UIA element-based control, native-image screenshots, instant app launching, batched action sequences, smart waits, and safety gates — in a 0–1000 normalized coordinate system so it never guesses pixels.

This is NOT the npm @zavora-ai/computer-use-mcp and NOT Codex's bundled computer-use — it is this repo's single-file Python server.

Why this exists

Most computer-use setups share two fatal habits:

1. Screenshot-and-guess — the model receives images, estimates pixel coordinates, and mis-clicks at any non-standard resolution/DPI.

2. Token burn — every state check costs a full screenshot.

computer-mcp inverts that:

Everything drives the real desktop at your user privileges; designed to run quietly in the background while you keep working.

Related MCP server: PointHand

Requirements

Install

git clone https://github.com/Tb3bhargav/computer-mcp.git
cd computer-mcp

# (recommended) isolated venv
python -m venv .venv
.\.venv\Scripts\pip install -r requirements.txt

# smoke test — should exit 0 and print nothing
python -m py_compile server.py

The server is not started by hand: your MCP client launches it over stdio and talks the Model Context Protocol. The config snippets below show exactly how.

💡 Pick the right interpreter In client configs, point command at your venv's interpreter for zero global installs: C:/path/to/computer-mcp/.venv/Scripts/python.exe On Windows the launcher is typically python (or python.exe under ...\Programs\Python\Python311\). Use forward slashes in JSON paths.

Add it to your MCP client

All snippets assume the repo lives at C:/Users/nj322ws/OneDrive/Desktop/My Projects/computer-mcp — adjust to where you cloned it.

Generic mcp.json — Claude Desktop · Cursor · Cline · Continue · Windsurf

{
  "mcpServers": {
    "computer": {
      "command": "python",
      "args": ["C:/Users/nj322ws/OneDrive/Desktop/My Projects/computer-mcp/server.py"]
    }
  }
}

OpenCode — global ~/.config/opencode/opencode.json (production-tested setup)

{
  "mcp": {
    "computer": {
      "type": "local",
      "command": [
        "C:/Users/nj322ws/AppData/Local/Programs/Python/Python311/python.exe",
        "C:/Users/nj322ws/OneDrive/Desktop/My Projects/computer-mcp/server.py"
      ],
      "cwd": "C:/Users/nj322ws/OneDrive/Desktop/My Projects/computer-mcp",
      "enabled": true,
      "timeout": 180000
    }
  },
  "permission": {
    "computer_*": "allow"
  }
}

"permission": { "computer_*": "allow" } whitelists every tool name of this server in OpenCode. Drop it if you prefer to approve each desktop-control call manually.

OpenCode also supports per-project config with the same shape (e.g. project opencode.jsonc or .opencode/opencode.jsonc):

{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "servers": {
      "computer": {
        "type": "local",
        "command": ["python", "C:/Users/nj322ws/OneDrive/Desktop/My Projects/computer-mcp/server.py"]
      }
    }
  },
  "permission": { "computer_*": "allow" }
}

VS Code (GitHub Copilot) — .vscode/mcp.json

{
  "servers": {
    "computer": {
      "type": "stdio",
      "command": "python",
      "args": ["C:/Users/nj322ws/OneDrive/Desktop/My Projects/computer-mcp/server.py"]
    }
  }
}

Codex CLI — ~/.codex/config.toml

[mcp_servers.computer]
command = "python"
args = ["C:/Users/nj322ws/OneDrive/Desktop/My Projects/computer-mcp/server.py"]

Claude Code — CLI one-liner

claude mcp add computer -- python "C:/Users/nj322ws/OneDrive/Desktop/My Projects/computer-mcp/server.py"

Gemini CLI — ~/.gemini/settings.json

{
  "mcpServers": {
    "computer": {
      "command": "python",
      "args": ["C:/Users/nj322ws/OneDrive/Desktop/My Projects/computer-mcp/server.py"]
    }
  }
}

Verify it works: restart the client, start a chat, and ask "what's on my screen?" — the agent should call get_screen_info() (reported in plain text, free tokens) instead of grabbing an image.

The 0–1000 coordinate system

Every position the agent passes in — clicks, regions, polls — is normalized:

• (0,0) = top-left · (1000,1000) = bottom-right of the primary display.

• Internally converted to real pixels: resolution- and DPI-independent.

• Anything returned (element boxes, image matches, diff cells) is also 0–1000.

All 45 tools

Perception — see without burning image tokens

UI elements (UI Automation) — the fast path

Screenshots & vision

Windows & apps

Mouse

Keyboard & text

Batch — 12 action types, one call

Waits — sync without blind sleeping

Safety & control state

The agent workflow this server is built around

The golden loop — absorb it once with the agent of your choice:

STATE (free)         get_screen_info / list_windows
TARGETS (free)       get_ui_tree → find_element → read_screen_text
ACT                  click_element / batch / paste_text / press_key
VERIFY (free)        diff_screenshot / wait_for_pixel / wait_for_image
ONLY IF NEEDED       take_screenshot(region=...) for actual visual truth

Design principles baked into the tools:

• Perception first, pixels last — a healthy session uses screenshots only for final visual proof.

• click_element re-finds its target at click time — no stale coordinates between two UI states.

• Windows are matched by exact title → prefix → owning process name → substring, so "whatsapp" finds WhatsApp even when the title says "John • WhatsApp".

• list_windows/window matching filters DWM-cloaked ghosts invisible to the user.

Recipes

open_app("chrome", wait_after=4)
find_element("New tab")          # zero-image locate
click_element("New tab")

batch([
  {"focus_window": "WhatsApp"},
  {"press": "ctrl+f"}, {"paste": "Bhavna"},
  {"wait": 1.5}, {"press": "enter"}, {"wait": 2},
  {"click_text": "Bhavna Didi"},
  {"paste": "Compliments on the launch 🎉"},
  {"press": "enter"}
], verify_screenshot=True)

click_element("Email")        # focus first field
paste_text("user@example.com", restore_clipboard=True)
press_key("tab")
paste_text("S3cret Password")
press_key("enter")
diff_screenshot()             # what changed — without an image

click_element("Submit")
wait_for_pixel(500, 820, "#22c55e")   # success badge turns green
diff_screenshot(region=[380, 760, 620, 860])

save_screenshot("templates/save-btn.png", region=None)   # build templates once
find_image("templates/save-btn.png", confidence=0.9)
wait_until_image_gone("templates/save-btn.png")          # modal closed?

create_agent_desktop()       # sandbox ahead
switch_agent_desktop()       # agent now operates there
# ... any work ...
switch_agent_desktop(False)  # back to the user's screen
close_agent_desktop()

set_safety_mode(True)
press_key("delete")
# → "SAFETY MODE: destructive action pending — press_key('delete') ...
#     Call confirm_destructive('perm_1')"
confirm_destructive("perm_1")    # now it runs

Safety

• Explicit permissions — In OpenCode, only set "permission": { "computer_*": "allow" } for agents you trust; leave it out to approve every desktop call.

• Safety mode — set_safety_mode(True) pends destructive actions (delete/backspace key events) behind confirm_destructive; default OFF.

• Isolated desktops — route risky visual work to create_agent_desktop() so the user's screen stays untouched.

• Silent control by default — control runs in the background; the takeover-overlay engine remains available (control_status, set_overlay_mode).

• Esc-to-cancel — the overlay engine's global Esc hook latches an emergency pause state whenever it is active; resume_control() clears it.

How it works (in one screen)

• One file, server.py — a FastMCP stdio server; every tool is a plain typed function.

• OS + screen detected at startup and logged to stderr (stdout is reserved for the MCP protocol).

• UIA all on one thread — Windows UI Automation COM objects are apartment-bound, so every UIA call is serialized onto a single worker with per-call timeouts.

• App launcher index — a TTL-cached (10 min) merge of Start-Menu shortcuts + Get-StartApps (incl. Microsoft Store AUMIDs) + App-Paths registry + PATH.

• Screenshot baselines — diff_screenshot keeps a downscaled grayscale baseline and returns changed grid cells + union bbox instead of pixels, so the agent only grabs a cropped image if it truly needs to see the change.

• Graceful degradation — missing optional deps reduce features (no numpy → slower diffing) but never crash the server; window matching has layered fallbacks (exact → prefix → process name → substring, then "best available window").

Testing — real results on a real machine

15-file report suite: 12 scenarios, all PASS, run October 2026 on the author's daily Windows 11 machine (OneDrive-synced, in active use). Full evidence in reports/:

Troubleshooting

Install Python from python.org and tick Add to PATH. If the Windows Store alias persists, disable App execution aliases (Settings → Apps → Advanced app settings), or point configs at python.exe's full path.

Check the client's MCP logs — the server writes an OS/screen banner to stderr at startup. Confirm the absolute path in your config is exact (it lives under user folders by design) and restart the client fully.

Chromium browsers and some apps only expose accessibility info on demand. Chrome/Edge: enable accessibility in the app or set the automation flag — UIA then sees the DOM. Win32 apps (Notepad, Explorer, Office, most desktop apps) expose it natively.

Template matching needs opencv-python for the confidence parameter (listed in requirements). Rebuild templates with save_screenshot right before matching, same scale/theme.

They're normalized — scenarios: mixed resolutions or Windows scaling changes make real pixel deltas unreliable; any x/y you pass is /1000-mapped to the primary display and never needs recalibration.

Contributing

Issues and PRs welcome — especially for the per-OS abstractions. Keep the runtime a single importable server.py (the whole point), and please add a short report under reports/ for behavior changes.

License

MIT — free for any use, including commercial; keep the copyright notice in copies. © 2026 Tejus Bhargav.