# Parallel CUA

**English** | [简体中文](README.zh-CN.md)

[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
[![Python](https://img.shields.io/badge/python-3.10%2B-blue.svg)](https://www.python.org/)
[![Platform](https://img.shields.io/badge/platform-Windows%2010%2F11-blue.svg)](https://www.microsoft.com/windows)
[![MCP](https://img.shields.io/badge/MCP-server-green.svg)](https://modelcontextprotocol.io)
[![Version](https://img.shields.io/badge/version-0.8.0-orange.svg)](https://github.com/Shinyeiceblue/parallel-cua-mcp)

## The computer is no longer a screenshot for the model — it’s a pair of hands

You already have models that can *see* a screen. What you usually don’t have is a model that can **use** a computer the way a person does: hold `W` and turn the camera at the same time, open a CAD menu it has never seen, find “保存” without a hard-coded pixel, and come back tomorrow already knowing the app.

**Parallel CUA** is a Windows [MCP](https://modelcontextprotocol.io) server built for that.

Most “computer use” stacks still look like this: screenshot → one click → screenshot → one keystroke. Fine for filling a form. Useless when the job is *moving*, *aiming*, and *firing* in the same second — or when the button you need only exists after you learn the software.

This project starts from a different question:

> If an agent is going to operate my PC, it should feel less like a remote control toy — and more like someone sitting at the desk.

### What that actually means here

- **Two hands, not one finger.** Keys stay *held* while the mouse keeps moving. Keyboard, aim, and clicks can run as parallel streams with real timing — not serialized RPC steps.
- **It can look things up.** Unfamiliar app? `research_app` → save a playbook → next time it already knows the shortcuts.
- **It already knows the workshop.** 47 engineering apps (AutoCAD, SOLIDWORKS, KiCad, MATLAB, Revit, CAXA, …) ship with shortcuts and intents like `draw_line` / `extrude` / `run`.
- **It aims like a person.** UIA element names first, OCR text anchors second, coordinates last — plus `human_*` motion with curves, jitter, and pauses.
- **It doesn’t care which agent you use.** MCP for Claude/Cursor/MiMo; OpenAI tools; Anthropic computer-use actions; plain HTTP.

> Games · CAD/EDA/CAE · desktop agents · serious RPA — anywhere “one action at a time” is not enough.

### Three minutes in someone’s day

**11:04 — FPS warm-up**  
The agent holds forward, strafes, and keeps the crosshair tracking while clicking fire — three streams at once. Not “press W, wait for tool result, then move mouse.”

**14:20 — A CAD job nobody documented**  
Foreground is AutoCAD. `observe_environment` attaches the knowledge base. The request is “draw a line.” It doesn’t guess ribbon pixels — it does `Esc` → `L` → `Enter` like a drafter who has done it a thousand times.

**16:50 — A weird in-house tool**  
No UIA names, no memory of this app. The agent screenshots, OCRs the Chinese label `导出 PDF`, clicks the text, types a filename with human pauses, and saves a playbook so tomorrow is boring in the best way.

That’s the point: **not a demo of clicking — a layer that makes agents *operate*.**

---

## Table of contents

- [Why it feels different](#why-it-feels-different)
- [Feature comparison](#feature-comparison)
- [Architecture](#architecture)
- [Quick start](#quick-start)
- [Multi-harness](#multi-harness)
- [Core capabilities](#core-capabilities)
- [Engineering software knowledge base](#engineering-software-knowledge-base)
- [Examples](#examples)
- [Performance](#performance)
- [Tools](#tools)
- [Environment variables](#environment-variables)
- [Development](#development)
- [Security](#security)
- [Limitations](#limitations)
- [Roadmap](#roadmap)
- [License](#license)

---

## Why it feels different

| What usually hurts | What you get instead |
|---|---|
| CUA that does **one action**, then waits | Held-state keys/buttons + multi-stream `parallel` + timed `game_macro` |
| Tap-and-forget injection (pyautogui style) | Windows `SendInput` with an explicit “what is still held down” brain |
| Straight-line robot moves, metronome clicks | `human_*`: bezier paths, overshoot, random pauses, optional typos |
| Unknown software → blind clicking | Research, playbooks, and reuse next session |
| Engineering apps = memorized shortcuts | 47-app KB + `run_engineer_task(intent='draw_line')` |
| Locked to one client | One core: **MCP** + **OpenAI/Anthropic** + **HTTP** |
| Buttons without stable coordinates | **UIA names** → **OCR text** → coordinates |

### The one-liner we actually stand behind

> Not “a tool that can click the screen.”  
> A **Windows operation layer** that runs **in parallel**, **learns software**, and **plugs into the agent you already use**.

---

## Feature comparison

| Capability | Typical CUA / Windows-MCP | Parallel CUA |
|---|---|---|
| Screenshot / click / type / scroll | ✅ | ✅ |
| Action model | One-by-one (serial) | **Held-state + parallel streams + timeline macros** |
| Hold W while moving mouse | ❌ | ✅ `hold_keys` + `move_rel` |
| Keyboard / aim / fire streams in parallel | ❌ | ✅ `parallel` |
| Timed concurrent game macros | ❌ | ✅ `game_macro` |
| Scancode injection (Raw Input games) | Rare | ✅ `set_keyboard_mode(vk\|scan\|both)` |
| Human-like input | Rare | ✅ `human_*` |
| Learn unknown software first | ❌ | ✅ `research_app` + playbooks |
| Engineering shortcuts / intents | ❌ | ✅ 47-app KB + `run_engineer_task` |
| UIA element targeting | Partial | ✅ `list_ui_elements` / `click_element` |
| OCR text anchors | ❌ | ✅ `find_text` / `click_text` |
| Multi-agent / multi-protocol | Usually MCP only | ✅ MCP + OpenAI + Anthropic + HTTP |

---

## Architecture

```text
┌─────────────────────────────────────────────────────────────┐
│                     AI Harness / Agent                       │
│  MiMo · Claude · Cursor · OpenAI Agents · Anthropic · LC    │
└───────────────┬─────────────────────────────┬───────────────┘
                │ MCP (stdio / HTTP)          │ HTTP JSON
                ▼                             ▼
┌──────────────────────────┐   ┌──────────────────────────────┐
│   MCP Tool Surface       │   │  HTTP Gateway                 │
│   (61 tools)             │   │  /tools /computer /v1/...     │
└────────────┬─────────────┘   └───────────────┬──────────────┘
             │          Unified Executor       │
             └───────────────┬─────────────────┘
                             ▼
        ┌────────────────────────────────────────────┐
        │  ConcurrentInputController (SendInput)     │
        │  held-keys · parallel streams · scancode   │
        ├────────────────────────────────────────────┤
        │  HumanDriver  │  UIA  │  OCR  │  Knowledge │
        └────────────────────────────────────────────┘
                             │
                             ▼
                      Windows Desktop
```

**Layers**

- **Input core** — injection + concurrency only
- **Dialects** — map OpenAI CUA / Anthropic actions to one executor
- **Knowledge** — process match → shortcuts/intents → executable plan
- **Human / UIA / OCR** — pacing, accessibility tree, text anchors

---

## Quick start

### Requirements

- Windows 10 / 11
- Python 3.10+

### Install

```powershell
cd parallel-cua-mcp
python -m venv .venv
.\.venv\Scripts\python.exe -m pip install -e .
```

### Run as MCP (stdio)

```powershell
.\.venv\Scripts\python.exe -m parallel_cua serve --transport stdio
```

### MiMo Desktop

Merge into `~/.config/mimocode/mimocode.jsonc` (adjust paths):

```json
{
  "mcp": {
    "parallel-cua": {
      "type": "local",
      "command": [
        "C:\\path\\to\\parallel-cua-mcp\\.venv\\Scripts\\python.exe",
        "-m",
        "parallel_cua",
        "serve",
        "--transport",
        "stdio"
      ],
      "environment": {
        "PYTHONPATH": "C:\\path\\to\\parallel-cua-mcp\\src"
      },
      "enabled": true
    }
  }
}
```

Restart the engine or open a new chat.

### Claude Desktop / Cursor

```json
{
  "mcpServers": {
    "parallel-cua": {
      "command": "C:\\path\\to\\parallel-cua-mcp\\.venv\\Scripts\\python.exe",
      "args": ["-m", "parallel_cua", "serve", "--transport", "stdio"],
      "env": { "PYTHONPATH": "C:\\path\\to\\parallel-cua-mcp\\src" }
    }
  }
}
```

Generate configs:

```powershell
.\.venv\Scripts\python.exe -m parallel_cua harness --list
.\.venv\Scripts\python.exe -m parallel_cua harness --name claude_desktop
.\.venv\Scripts\python.exe -m parallel_cua harness --name openai_sdk
```

---

## Multi-harness

| Harness | Protocol | Command |
|---|---|---|
| MiMo / Claude / Cursor / Gemini / Codex | MCP stdio | `serve --transport stdio` |
| Any MCP HTTP client | Streamable HTTP / SSE | `serve --transport streamable-http --port 8000` |
| OpenAI Agents / custom tool loop | HTTP JSON | `serve --transport http --port 8765` |
| Anthropic SDK computer-use | HTTP `/computer` | same, `dialect=anthropic` |
| LangChain / LangGraph | HTTP generic | `POST /tools/call` |

### Schema export

```powershell
python -m parallel_cua tools --format openai
python -m parallel_cua tools --format anthropic
python -m parallel_cua tools --format generic
```

### HTTP endpoints

| Method | Path | Description |
|---|---|---|
| GET | `/health` | Health check |
| GET | `/tools?dialect=openai\|anthropic\|generic\|mcp` | Tool list |
| POST | `/tools/call` | `{"name":"human_click","arguments":{"x":10,"y":20}}` |
| POST | `/computer` | OpenAI / Anthropic action dialects |
| POST | `/v1/chat/completions` | Execute already-chosen `tool_calls` (model stays external) |

Optional auth: set `PARALLEL_CUA_HTTP_TOKEN` and send `Authorization: Bearer <token>`.

### Dialects

- **OpenAI CUA**: `click / type / key / scroll / drag / screenshot / hold_key / wait`
- **Anthropic computer-use**: `left_click / type / key / scroll / left_click_drag / screenshot …`
- **Generic / MCP**: full surface (`hold_keys`, `parallel`, `game_macro`, UIA, OCR, playbooks…)

---

## Core capabilities

### 1. True parallel (games / high-frequency ops)

- `hold_keys` / `hold_mouse` — keep down until `release_*` / `release_all`
- `move_mouse(dx, dy)` — relative aim while keys held
- `parallel(streams)` — thread-level concurrent action streams
- `game_macro(steps)` — timed holds + continuous look + clicks
- `set_keyboard_mode("scan"|"both")` — Raw Input-friendly games

### 2. Human-like input (desktop apps)

- `human_move_to` — bezier + jitter + overshoot
- `human_click` / `human_type` — random hold times, word pauses, optional typos
- `human_think` / `human_glance` — reading pauses and micro-moves
- `human_set_speed` — 0.5 careful → 1.5 hurried
- `run_script` — high-level action scripts

### 3. Learning loop (unknown software)

```text
observe_environment + screenshot
  → get_playbook
  → research_app(name, goal)     # Bing first, fetch_page_text
  → save_playbook(steps, shortcuts)
  → human_* operate
  → screenshot verify; release_all if needed
```

Playbooks: `%USERPROFILE%\.parallel-cua\playbooks\`

### 4. UIA element targeting (0.7)

- `list_ui_elements` / `find_element` / `click_element`
- `set_edit_text` / `get_element_text` / `focus_element`
- Prefer Invoke/Toggle patterns, else center click
- Verified on Windows 11 Notepad (`DocumentControl`)

### 5. OCR text anchors (0.8)

- `ocr_region` / `find_text` / `click_text` / `ocr_with_screenshot`
- Windows OCR (`winsdk`), CJK space-insensitive matching
- Typical window-region OCR ~200–300ms

**Suggested order:** UIA → OCR → coordinate click.

### 6. Observation & windows

- `observe_environment` — foreground process/title/rect + engineering shortcuts when known
- `list_windows` / `focus_window` / `window_action` / `open_app`
- `screenshot` — `region` + `scale` (prefer `scale=0.5`)

---

## Engineering software knowledge base

**47 apps** (including Chinese CAD/BIM tools) with shortcuts, workflows, and notes:

| Category | Apps |
|---|---|
| CAD | AutoCAD, CAXA, ZWCAD, GstarCAD, TArch, Rhino |
| MCAD | SOLIDWORKS, Fusion, Inventor, SketchUp, ZW3D, Gstar 3D, FreeCAD, CATIA, NX, Creo, Solid Edge |
| BIM | Revit, Glodon |
| EDA | KiCad, Altium, EAGLE, Multisim, Proteus, LTspice |
| CAE | ANSYS, Abaqus, COMSOL |
| Analysis / 3D / CAM | MATLAB, Origin, LabVIEW, Blender, OpenSCAD, Mastercam |
| Dev / Office | VS Code, PyCharm, Eclipse, Keil, IAR, Jupyter, Excel, Word, PPT, Visio, Project, Acrobat |

Data: [`knowledge/engineering_apps.json`](knowledge/engineering_apps.json)  
Fetcher: [`scripts/fetch_engineering_docs.py`](scripts/fetch_engineering_docs.py)

### Intent-level execution

```text
run_engineer_task(intent='save')
run_engineer_task(intent='draw_line', app_id='autocad')
run_engineer_task(intent='run', app_id='matlab')
run_engineer_task(intent='zoom_extents', dry_plan_only=True)
```

Intents: `save` · `undo` · `cancel` · `new` · `open` · `run` · `draw_line` · `extrude` · `zoom_extents` · `rebuild`

Also: `list_engineering_apps` · `match_engineering_app` · `suggest_shortcuts` · `apply_shortcut` · `seed_engineering_playbooks`

---

## Examples

### FPS: move forward while turning left

```text
hold_keys(["w", "shift"])
move_mouse(dx=-80, dy=0)
move_mouse(dx=-80, dy=0)
release_all()
```

### Move + shoot (parallel streams)

```text
parallel(streams=[
  [{"type":"hold_keys","keys":["w"],"ms":1000}],
  [{"type":"move_rel","dx":50,"dy":0},{"type":"wait","ms":150},
   {"type":"move_rel","dx":50,"dy":0},{"type":"wait","ms":150},
   {"type":"move_rel","dx":50,"dy":0}],
  [{"type":"wait","ms":300},{"type":"click","button":"left"},
   {"type":"wait","ms":200},{"type":"click","button":"left"}]
])
```

### game_macro: advance + turn + fire

```text
game_macro(steps=[
  {
    "duration_ms": 600,
    "keys_down": ["w"],
    "mouse_rel": {"dx": -200, "dy": 0, "steps": 24},
    "clicks": [
      {"at_ms": 80, "button": "left"},
      {"at_ms": 280, "button": "left"}
    ]
  },
  {"duration_ms": 80, "keys_up": ["w"]}
], release_at_end=True)
```

### CAD line (knowledge intent)

```text
run_engineer_task(intent='draw_line', app_id='autocad')
# Esc → wait → l → enter (human pacing)
```

### HTTP (OpenAI-style)

```bash
curl -s http://127.0.0.1:8765/tools/call \
  -H "Content-Type: application/json" \
  -d '{"name":"screenshot","arguments":{"scale":0.4}}'

curl -s http://127.0.0.1:8765/computer \
  -H "Content-Type: application/json" \
  -d '{"dialect":"anthropic","action":{"action":"left_click","coordinate":[120,80]}}'
```

---

## Performance

1. **Lock never spans sleep / long I/O** — only held-set updates  
   Regression: stream A `press` 120ms while stream B finishes in ~11ms (serialized would be ~120ms)
2. **Batched SendInput** — chord downs/ups in one call; `release_all` once
3. **Scancode mode** — default `both` (VK + scancode)
4. **Short waits** — &lt;15ms spin for tighter chords
5. **Counters** — `get_input_stats`

---

## Tools

Currently **61** MCP tools (HTTP generic catalog mirrors them).

<details>
<summary>Expand full list</summary>

**Observe / screenshot**  
`screenshot` · `observe_environment` · `get_cursor_pos` · `get_screen_info` · `get_held_state` · `get_input_stats`

**UIA**  
`list_ui_elements` · `find_element` · `click_element` · `set_edit_text` · `get_element_text` · `focus_element`

**OCR**  
`ocr_region` · `find_text` · `click_text` · `ocr_with_screenshot`

**Basic input**  
`click` · `double_click` · `move_mouse` · `drag` · `scroll` · `type_text` · `key` · `hotkey` · `wait` · `ui_shortcut`

**Concurrent / held**  
`hold_keys` · `release_keys` · `hold_mouse` · `release_mouse` · `release_all` · `parallel` · `game_macro` · `set_keyboard_mode`

**Human-like**  
`human_set_speed` · `human_move_to` · `human_click` · `human_type` · `human_hotkey` · `human_think` · `human_glance` · `human_scroll` · `run_script`

**Windows / apps**  
`list_windows` · `focus_window` · `window_action` · `open_app`

**Learning**  
`search_web` · `fetch_page_text` · `research_app` · `save_playbook` · `get_playbook` · `list_playbooks` · `find_playbook_for_window`

**Engineering KB**  
`list_engineering_apps` · `match_engineering_app` · `suggest_shortcuts` · `seed_engineering_playbooks` · `apply_shortcut` · `list_engineering_intents` · `run_engineer_task`

</details>

---

## Environment variables

| Variable | Default | Description |
|---|---|---|
| `PARALLEL_CUA_DRY_RUN` | off | `1/true/yes/on` — no real input injection |
| `PARALLEL_CUA_KEYBOARD_MODE` | `both` | `vk` / `scan` / `both` |
| `PARALLEL_CUA_HTTP_TOKEN` | empty | HTTP gateway Bearer auth |

---

## Development

```powershell
$env:PARALLEL_CUA_DRY_RUN = "1"
$env:PYTHONPATH = ".\src"

.\.venv\Scripts\python.exe -m tests.test_basic
.\.venv\Scripts\python.exe -m tests.test_concurrency
.\.venv\Scripts\python.exe -m tests.test_agent_layer
.\.venv\Scripts\python.exe -m tests.test_harness
.\.venv\Scripts\python.exe -m tests.test_engineering_kb
.\.venv\Scripts\python.exe -m tests.test_engineer_tasks
.\.venv\Scripts\python.exe -m tests.test_uia
.\.venv\Scripts\python.exe tests\smoke_mcp_client.py

# Real E2E (opens Notepad, injects real input)
.\.venv\Scripts\python.exe tests\e2e_notepad.py
.\.venv\Scripts\python.exe tests\e2e_ocr_simple.py
```

### Layout

```text
parallel-cua-mcp/
├── knowledge/engineering_apps.json
├── scripts/fetch_engineering_docs.py
├── src/parallel_cua/
│   ├── server.py      # MCP tools
│   ├── wininput.py    # SendInput concurrent core
│   ├── human.py       # human-like driver
│   ├── uia.py         # UIA elements
│   ├── ocr.py         # Windows OCR anchors
│   ├── knowledge.py   # engineering KB + intents
│   ├── dialects.py    # OpenAI / Anthropic
│   ├── http_api.py    # HTTP gateway
│   └── harness.py     # client config profiles
└── tests/
```

---

## Security

This service can inject arbitrary OS input:

- Do **not** bind `0.0.0.0` without auth; set `PARALLEL_CUA_HTTP_TOKEN` for remote HTTP
- Do not run unattended with unrestricted model access
- Call `release_all` after holds to avoid stuck keys
- Anti-cheat games may reject injection (expected)

---

## Limitations

- Some anti-cheat titles block injected input
- `type_text` is Unicode injection (chat/forms); use `key`/`hold_keys` for game skills
- Multi-monitor uses virtual-desktop coordinates; DPI follows the system coordinate space
- Engineering KB covers common versions; local menus may differ
- `research_app` needs network (Bing first); offline → local KB/playbooks

---

## Roadmap

- [ ] Vision-assisted control detection (OCR + optional detection → click)
- [ ] More Chinese/vertical engineering packs
- [ ] Record & replay (human ops → game_macro / run_script)
- [ ] Multi-monitor partition ops
- [ ] Optional UIA element selector (Playwright-like)

---

## License

This project is licensed under the [MIT License](LICENSE).

See also: [Changelog](CHANGELOG.md) · [Contributing](CONTRIBUTING.md) · [Security](SECURITY.md)

---

<p align="center">
  <sub>For agents that need to actually sit at a Windows desk — two hands, not one click at a time.</sub>
</p>