opencode-gui-bridge

Give opencode (or any MCP client) computer-use capabilities: it can see (understand screen state), act (click/type/scroll), and verify (confirm the action took effect).

Built on PySide6 + Win32 API + Windows UI Automation + local OCR, with zero system-level dependencies. All basic operations run locally with no network requirement (only the vision describe optionally uses a network API).

Quick Start

1. Extract the project to any directory (e.g. D:\gui-bridge\), double-click setup.bat, and wait for it to show Done.

2. Put an opencode.json in your opencode working directory (see "Integrating with opencode"), and change both paths to the actual paths from step 1

3. Restart opencode

4. Just say it in the AI chat:

   • "List the windows on the computer" → get the list_targets result

   • "Open Notepad and type hello in it" → it will automatically run open → bind → snapshot → click → input → verify

Installation

.\setup.bat

The script does everything in one pass: creates a venv virtual environment (skipped if it already exists) → pip installs dependencies → runs a smoke test. Seeing Done. means installation succeeded; on failure it exits and prints the reason.

Manual installation has the same effect:

python -m venv venv
venv\Scripts\python -m pip install -e .
venv\Scripts\python tests\smoke_test.py

Requirements: Windows 10/11 + Python 3.10+ (check Add python.exe to PATH during installation).

Integrating with opencode

Put opencode.json in the working directory where you run opencode (not inside the project):

{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "gui-bridge": {
      "type": "local",
      "command": [
        "D:\\gui-bridge\\venv\\Scripts\\python.exe",
        "D:\\gui-bridge\\server.py"
      ],
      "enabled": true,
      "environment": {
        "SILICONFLOW_API_KEY": "{env:SILICONFLOW_API_KEY}"
      }
    }
  }
}

Two changes to make:

1. Replace both D:\\gui-bridge\\... paths with your actual paths (\ must be written as \\ in JSON)

2. The SILICONFLOW_API_KEY line: local OCR and click/input don't need any key; you only need to configure it if you plan to use vision describe (see next section). Delete the line if you have no key.

Verify the integration: after restarting opencode, tell the AI "list the windows on the computer"; if the AI returns a window list, the python.exe and server.py paths are configured correctly.

Vision Channel Configuration (for describe, optional)

list_targets returns channels.vision with a status of ready (has key) or no-key (doesn't). It uses an OpenAI-compatible API, any vendor:

Three ways to set it, pick any one:

a) Embedded in opencode.json (travels with the config, recommended)

"environment": {
  "VISION_BASE_URL": "https://api.siliconflow.cn/v1",
  "VISION_API_KEY": "{env:OPENAI_API_KEY}",
  "VISION_MODEL": "Qwen/Qwen3-VL-32B-Instruct"
}

{env:XXX} means read an existing environment variable of the same name from your machine.

b) System-level persistence (applies to all terminals):

setx VISION_API_KEY "sk-xxxx"
setx VISION_BASE_URL "https://api.siliconflow.cn/v1"

After setting, you must reopen the terminal and restart opencode for it to take effect.

c) Only for the current terminal session:

$env:VISION_API_KEY = "sk-xxxx"

CDP Channel Configuration (WebView2 / Tauri / Electron)

For Web-kernel apps like Tauri, WebView2, and Electron, UIA can only see the outer shell and can't read the DOM. Once the CDP debug port is enabled, snapshots automatically go through the CDP channel (element ids prefixed with d:), and full-text reads are millisecond-fast.

Enable the debug port by app type:

$env:WEBVIEW2_ADDITIONAL_BROWSER_ARGUMENTS = "--remote-debugging-port=9222 --remote-allow-origins=*"
Start-Process 目标应用

After launching, confirm with list_targets: the returned channels.cdp will show the port number (e.g. 9222). From then on, snapshot automatically uses CDP and act automatically routes DOM operations:

• Read full page text: DOM innerText, <10ms (OCR takes 1~6s)

• Click: native DOM click (bypasses physical hit-test overlays)

• Input: Input.insertText real input pipeline (compatible with editors like Quill)

• Element coordinates: CSS×DPR+window position approximation (operations don't depend on coordinates)

Not enabling it doesn't break anything: such apps automatically fall back to the local OCR channel and can still read the screen and operate.

Toolbox: 7 MCP Tools

Rules: snapshot/act must be called after focus_target.

act Action Details

Return structure {ok, verify, detail}:

• ok: whether the action executed

• verify: result of automatic verification after execution

  • changed / matched: the UI actually changed / the input content is confirmed to have appeared

  • no_change / no_match: expected change not detected (the action may not have taken effect; re-snapshot to see the latest state)

  • cdp_insert / skipped: went through CDP input or verification was explicitly disabled

  • failed: execution failed, detail carries the reason; click-type failures automatically retry physically and attach a diagnostic screenshot path

• detail: human-readable result description, may include Diagnostic screenshot: <path>

Architecture

┌─ Agent (AI)
│   7 个 MCP 工具: list_targets / focus_target / snapshot /
│   act / wait_change / screenshot / describe
├─ server.py      会话编排: 目标窗口绑定, 通道选择, 验证闭环
├─ snapshot.py    统一元素抽象: {id, type, text, bbox, enabled, focused}
│                 通道融合 + 稳定 id (u:路径链 / o:OCR索引)
├─ executor.py    动作路由: click/input/press/scroll + 内置验证
├─ uia.py         UIA 控件树通道 (L1, 毫秒级, 原生应用)
├─ ocr.py         本地 OCR 通道 (L2, 1~6s, WebView 兜底)
├─ win32io.py     Win32 底层: 窗口/鼠标/键盘/截图/PostMessage/PrintWindow
└─ vision.py      视觉模型通道 (L3, 兜底理解, 需 API key)

运行日志写入 `logs/gui-bridge.log`（JSON lines：每次工具调用的耗时/通道/结果）。

Core Design

1. The AI only operates by element id, never coordinates. The snapshot provides ids, and act automatically routes each id to the best channel.

2. Automatic channel fallback: CDP → UIA → OCR → vision; click: InvokePattern → PostMessage → physical.

3. Built-in verification loop: act returns verify=changed/no_match/failed + reason.

4. Occlusion-safe capture: OCR and verification use PrintWindow to grab the target window's real content directly, so even if the target is covered by other windows, content doesn't get mixed up.

Element ID Rules

For o: ids and u: ids that go stale after UI changes, re-run snapshot to get fresh ids before clicking.

Testing

venv\Scripts\python tests\smoke_test.py   # 7 工具 + UIA 全链路（自建测试窗口）
venv\Scripts\python tests\ocr_test.py     # OCR 通道兜底链路
venv\Scripts\python tests\stdio_e2e.py    # 端到端：真实 MCP stdio 会话

Known Limitations

• WebView2/Tauri double-shell DOM isn't exposed to UIA → automatically falls back to the OCR channel (verified to fully read the screen and operate)

• Windows may prevent background processes from stealing focus → focus_target will warn; manually click the target window once if needed

• OCR channel takes 1~6s per snapshot (sub-second with snapshot cache hits when the screen is static), the main latency source for WebView apps

• Currently Windows-only