# com-use-mcp

Computer-use MCP server for [OpenCode](https://opencode.ai): mouse, keyboard, screenshots, Windows OCR, window control and shell — so an AI agent can see and operate a Windows PC.

Built by **Nanon** + **OpenCode** (Muse Spark).

> Windows only. Tested on Python 3.12.

## How it works

- `com_server.py` exposes tools over MCP (stdio) using `FastMCP`.
- `com_use/actions.py` implements everything: human-like mouse moves, bulk-paste typing, native-resolution screenshots, Windows built-in OCR (`winocr`) with x/y coordinates, contour-based UI element detection, color search, clipboard and window management via win32 APIs.
- The agent loop is: `look()` (screenshot + OCR with coordinates) → decide → `click` / `click_text` / `type_text` / `hotkey` → repeat.

Key accuracy decisions (v3):

- **Primary monitor only** — captures can never drift to a secondary display.
- **Native 1:1 pixels** — the screenshot the model sees is exactly the coordinate space it clicks in (`scale: 1.0`), so no mental rescaling error.
- **Text-grounded clicks** — `click_text()` resolves OCR word centers in code instead of guessing pixels.
- **DPI aware** — coordinates match real pixels on 125%/150% scaling.

## Install

```bat
pip install -r requirements.txt
```

## OpenCode config

Copy `opencode.json.example` to your project as `opencode.json` and fix the path to `com_server.py`:

```json
{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "com-use": {
      "type": "local",
      "command": ["python", "C:\\path\\to\\com-use-mcp\\com_server.py"],
      "enabled": true,
      "timeout": 30000
    }
  }
}
```

The 30s timeout matters: the first import (`cv2`, `numpy`, …) is slow and a short timeout kills the server before it is ready.

## Tools

Mouse: `click`, `double_click`, `right_click`, `mouse_move`, `drag`, `scroll`, `mouse_position`
Keyboard: `type_text` (bulk paste), `press_key`, `hotkey`, `key_hold`
Screen: `look` (screenshot + OCR, primary tool), `screenshot`, `screen_size`, `ocr`
Grounding: `ground_text` (exact center of text), `click_text` (ground + click, preferred for buttons)
Detection: `find_elements`, `find_color`
Clipboard: `get_clipboard`, `set_clipboard`
Windows: `list_windows`, `active_window`, `focus_window`, `close_window`
Shell: `run_command`, `open_app`, `open_url`, `wait`