com-use-mcp

Computer-use MCP server for OpenCode: mouse, keyboard, screenshots, Windows OCR, window control and shell — so an AI agent can see and operate a Windows PC.

Built by Nanon + OpenCode (Muse Spark).

Windows only. Tested on Python 3.12.

How it works

• com_server.py exposes tools over MCP (stdio) using FastMCP.

• com_use/actions.py implements everything: human-like mouse moves, bulk-paste typing, native-resolution screenshots, Windows built-in OCR (winocr) with x/y coordinates, contour-based UI element detection, color search, clipboard and window management via win32 APIs.

• The agent loop is: look() (screenshot + OCR with coordinates) → decide → click / click_text / type_text / hotkey → repeat.

Key accuracy decisions (v3):

• Primary monitor only — captures can never drift to a secondary display.

• Native 1:1 pixels — the screenshot the model sees is exactly the coordinate space it clicks in (scale: 1.0), so no mental rescaling error.

• Text-grounded clicks — click_text() resolves OCR word centers in code instead of guessing pixels.

• DPI aware — coordinates match real pixels on 125%/150% scaling.

Related MCP server: jenny-mcp-server

Install

pip install -r requirements.txt

OpenCode config

Copy opencode.json.example to your project as opencode.json and fix the path to com_server.py:

{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "com-use": {
      "type": "local",
      "command": ["python", "C:\\path\\to\\com-use-mcp\\com_server.py"],
      "enabled": true,
      "timeout": 30000
    }
  }
}

The 30s timeout matters: the first import (cv2, numpy, …) is slow and a short timeout kills the server before it is ready.

Tools

Mouse: click, double_click, right_click, mouse_move, drag, scroll, mouse_position Keyboard: type_text (bulk paste), press_key, hotkey, key_hold Screen: look (screenshot + OCR, primary tool), screenshot, screen_size, ocr Grounding: ground_text (exact center of text), click_text (ground + click, preferred for buttons) Detection: find_elements, find_color Clipboard: get_clipboard, set_clipboard Windows: list_windows, active_window, focus_window, close_window Shell: run_command, open_app, open_url, wait