Parallel CUA

English | 简体中文

The computer is no longer a screenshot for the model — it’s a pair of hands

You already have models that can see a screen. What you usually don’t have is a model that can use a computer the way a person does: hold W and turn the camera at the same time, open a CAD menu it has never seen, find “保存” without a hard-coded pixel, and come back tomorrow already knowing the app.

Parallel CUA is a Windows MCP server built for that.

Most “computer use” stacks still look like this: screenshot → one click → screenshot → one keystroke. Fine for filling a form. Useless when the job is moving, aiming, and firing in the same second — or when the button you need only exists after you learn the software.

This project starts from a different question:

If an agent is going to operate my PC, it should feel less like a remote control toy — and more like someone sitting at the desk.

What that actually means here

• Two hands, not one finger. Keys stay held while the mouse keeps moving. Keyboard, aim, and clicks can run as parallel streams with real timing — not serialized RPC steps.

• It can look things up. Unfamiliar app? research_app → save a playbook → next time it already knows the shortcuts.

• It already knows the workshop. 47 engineering apps (AutoCAD, SOLIDWORKS, KiCad, MATLAB, Revit, CAXA, …) ship with shortcuts and intents like draw_line / extrude / run.

• It aims like a person. UIA element names first, OCR text anchors second, coordinates last — plus human_* motion with curves, jitter, and pauses.

• It doesn’t care which agent you use. MCP for Claude/Cursor/MiMo; OpenAI tools; Anthropic computer-use actions; plain HTTP.

Games · CAD/EDA/CAE · desktop agents · serious RPA — anywhere “one action at a time” is not enough.

Three minutes in someone’s day

11:04 — FPS warm-up
The agent holds forward, strafes, and keeps the crosshair tracking while clicking fire — three streams at once. Not “press W, wait for tool result, then move mouse.”

14:20 — A CAD job nobody documented
Foreground is AutoCAD. observe_environment attaches the knowledge base. The request is “draw a line.” It doesn’t guess ribbon pixels — it does Esc → L → Enter like a drafter who has done it a thousand times.

16:50 — A weird in-house tool
No UIA names, no memory of this app. The agent screenshots, OCRs the Chinese label 导出 PDF, clicks the text, types a filename with human pauses, and saves a playbook so tomorrow is boring in the best way.

That’s the point: not a demo of clicking — a layer that makes agents operate.

Related MCP server: Windows MCP Server

Table of contents

• Why it feels different

• Feature comparison

• Architecture

• Quick start

• Multi-harness

• Core capabilities

• Engineering software knowledge base

• Examples

• Performance

• Tools

• Environment variables

• Development

• Security

• Limitations

• Roadmap

• License

Why it feels different

The one-liner we actually stand behind

Not “a tool that can click the screen.”
A Windows operation layer that runs in parallel, learns software, and plugs into the agent you already use.

Feature comparison

Architecture

┌─────────────────────────────────────────────────────────────┐
│                     AI Harness / Agent                       │
│  MiMo · Claude · Cursor · OpenAI Agents · Anthropic · LC    │
└───────────────┬─────────────────────────────┬───────────────┘
                │ MCP (stdio / HTTP)          │ HTTP JSON
                ▼                             ▼
┌──────────────────────────┐   ┌──────────────────────────────┐
│   MCP Tool Surface       │   │  HTTP Gateway                 │
│   (61 tools)             │   │  /tools /computer /v1/...     │
└────────────┬─────────────┘   └───────────────┬──────────────┘
             │          Unified Executor       │
             └───────────────┬─────────────────┘
                             ▼
        ┌────────────────────────────────────────────┐
        │  ConcurrentInputController (SendInput)     │
        │  held-keys · parallel streams · scancode   │
        ├────────────────────────────────────────────┤
        │  HumanDriver  │  UIA  │  OCR  │  Knowledge │
        └────────────────────────────────────────────┘
                             │
                             ▼
                      Windows Desktop

Layers

• Input core — injection + concurrency only

• Dialects — map OpenAI CUA / Anthropic actions to one executor

• Knowledge — process match → shortcuts/intents → executable plan

• Human / UIA / OCR — pacing, accessibility tree, text anchors

Quick start

Requirements

• Windows 10 / 11

• Python 3.10+

Install

cd parallel-cua-mcp
python -m venv .venv
.\.venv\Scripts\python.exe -m pip install -e .

Run as MCP (stdio)

.\.venv\Scripts\python.exe -m parallel_cua serve --transport stdio

MiMo Desktop

Merge into ~/.config/mimocode/mimocode.jsonc (adjust paths):

{
  "mcp": {
    "parallel-cua": {
      "type": "local",
      "command": [
        "C:\\path\\to\\parallel-cua-mcp\\.venv\\Scripts\\python.exe",
        "-m",
        "parallel_cua",
        "serve",
        "--transport",
        "stdio"
      ],
      "environment": {
        "PYTHONPATH": "C:\\path\\to\\parallel-cua-mcp\\src"
      },
      "enabled": true
    }
  }
}

Restart the engine or open a new chat.

Claude Desktop / Cursor

{
  "mcpServers": {
    "parallel-cua": {
      "command": "C:\\path\\to\\parallel-cua-mcp\\.venv\\Scripts\\python.exe",
      "args": ["-m", "parallel_cua", "serve", "--transport", "stdio"],
      "env": { "PYTHONPATH": "C:\\path\\to\\parallel-cua-mcp\\src" }
    }
  }
}

Generate configs:

.\.venv\Scripts\python.exe -m parallel_cua harness --list
.\.venv\Scripts\python.exe -m parallel_cua harness --name claude_desktop
.\.venv\Scripts\python.exe -m parallel_cua harness --name openai_sdk

Multi-harness

Schema export

python -m parallel_cua tools --format openai
python -m parallel_cua tools --format anthropic
python -m parallel_cua tools --format generic

HTTP endpoints

Optional auth: set PARALLEL_CUA_HTTP_TOKEN and send Authorization: Bearer <token>.

Dialects

• OpenAI CUA: click / type / key / scroll / drag / screenshot / hold_key / wait

• Anthropic computer-use: left_click / type / key / scroll / left_click_drag / screenshot …

• Generic / MCP: full surface (hold_keys, parallel, game_macro, UIA, OCR, playbooks…)

Core capabilities

1. True parallel (games / high-frequency ops)

• hold_keys / hold_mouse — keep down until release_* / release_all

• move_mouse(dx, dy) — relative aim while keys held

• parallel(streams) — thread-level concurrent action streams

• game_macro(steps) — timed holds + continuous look + clicks

• set_keyboard_mode("scan"|"both") — Raw Input-friendly games

2. Human-like input (desktop apps)

• human_move_to — bezier + jitter + overshoot

• human_click / human_type — random hold times, word pauses, optional typos

• human_think / human_glance — reading pauses and micro-moves

• human_set_speed — 0.5 careful → 1.5 hurried

• run_script — high-level action scripts

3. Learning loop (unknown software)

observe_environment + screenshot
  → get_playbook
  → research_app(name, goal)     # Bing first, fetch_page_text
  → save_playbook(steps, shortcuts)
  → human_* operate
  → screenshot verify; release_all if needed

Playbooks: %USERPROFILE%\.parallel-cua\playbooks\

4. UIA element targeting (0.7)

• list_ui_elements / find_element / click_element

• set_edit_text / get_element_text / focus_element

• Prefer Invoke/Toggle patterns, else center click

• Verified on Windows 11 Notepad (DocumentControl)

5. OCR text anchors (0.8)

• ocr_region / find_text / click_text / ocr_with_screenshot

• Windows OCR (winsdk), CJK space-insensitive matching

• Typical window-region OCR ~200–300ms

Suggested order: UIA → OCR → coordinate click.

6. Observation & windows

• observe_environment — foreground process/title/rect + engineering shortcuts when known

• list_windows / focus_window / window_action / open_app

• screenshot — region + scale (prefer scale=0.5)

Engineering software knowledge base

47 apps (including Chinese CAD/BIM tools) with shortcuts, workflows, and notes:

Data: knowledge/engineering_apps.json
Fetcher: scripts/fetch_engineering_docs.py

Intent-level execution

run_engineer_task(intent='save')
run_engineer_task(intent='draw_line', app_id='autocad')
run_engineer_task(intent='run', app_id='matlab')
run_engineer_task(intent='zoom_extents', dry_plan_only=True)

Intents: save · undo · cancel · new · open · run · draw_line · extrude · zoom_extents · rebuild

Also: list_engineering_apps · match_engineering_app · suggest_shortcuts · apply_shortcut · seed_engineering_playbooks

Examples

FPS: move forward while turning left

hold_keys(["w", "shift"])
move_mouse(dx=-80, dy=0)
move_mouse(dx=-80, dy=0)
release_all()

Move + shoot (parallel streams)

parallel(streams=[
  [{"type":"hold_keys","keys":["w"],"ms":1000}],
  [{"type":"move_rel","dx":50,"dy":0},{"type":"wait","ms":150},
   {"type":"move_rel","dx":50,"dy":0},{"type":"wait","ms":150},
   {"type":"move_rel","dx":50,"dy":0}],
  [{"type":"wait","ms":300},{"type":"click","button":"left"},
   {"type":"wait","ms":200},{"type":"click","button":"left"}]
])

game_macro: advance + turn + fire

game_macro(steps=[
  {
    "duration_ms": 600,
    "keys_down": ["w"],
    "mouse_rel": {"dx": -200, "dy": 0, "steps": 24},
    "clicks": [
      {"at_ms": 80, "button": "left"},
      {"at_ms": 280, "button": "left"}
    ]
  },
  {"duration_ms": 80, "keys_up": ["w"]}
], release_at_end=True)

CAD line (knowledge intent)

run_engineer_task(intent='draw_line', app_id='autocad')
# Esc → wait → l → enter (human pacing)

HTTP (OpenAI-style)

curl -s http://127.0.0.1:8765/tools/call \
  -H "Content-Type: application/json" \
  -d '{"name":"screenshot","arguments":{"scale":0.4}}'

curl -s http://127.0.0.1:8765/computer \
  -H "Content-Type: application/json" \
  -d '{"dialect":"anthropic","action":{"action":"left_click","coordinate":[120,80]}}'

Performance

1. Lock never spans sleep / long I/O — only held-set updates
   Regression: stream A press 120ms while stream B finishes in ~11ms (serialized would be ~120ms)

2. Batched SendInput — chord downs/ups in one call; release_all once

3. Scancode mode — default both (VK + scancode)

4. Short waits — <15ms spin for tighter chords

5. Counters — get_input_stats

Tools

Currently 61 MCP tools (HTTP generic catalog mirrors them).

Observe / screenshot
screenshot · observe_environment · get_cursor_pos · get_screen_info · get_held_state · get_input_stats

UIA
list_ui_elements · find_element · click_element · set_edit_text · get_element_text · focus_element

OCR
ocr_region · find_text · click_text · ocr_with_screenshot

Basic input
click · double_click · move_mouse · drag · scroll · type_text · key · hotkey · wait · ui_shortcut

Concurrent / held
hold_keys · release_keys · hold_mouse · release_mouse · release_all · parallel · game_macro · set_keyboard_mode

Human-like
human_set_speed · human_move_to · human_click · human_type · human_hotkey · human_think · human_glance · human_scroll · run_script

Windows / apps
list_windows · focus_window · window_action · open_app

Learning
search_web · fetch_page_text · research_app · save_playbook · get_playbook · list_playbooks · find_playbook_for_window

Engineering KB
list_engineering_apps · match_engineering_app · suggest_shortcuts · seed_engineering_playbooks · apply_shortcut · list_engineering_intents · run_engineer_task

Environment variables

Development

$env:PARALLEL_CUA_DRY_RUN = "1"
$env:PYTHONPATH = ".\src"

.\.venv\Scripts\python.exe -m tests.test_basic
.\.venv\Scripts\python.exe -m tests.test_concurrency
.\.venv\Scripts\python.exe -m tests.test_agent_layer
.\.venv\Scripts\python.exe -m tests.test_harness
.\.venv\Scripts\python.exe -m tests.test_engineering_kb
.\.venv\Scripts\python.exe -m tests.test_engineer_tasks
.\.venv\Scripts\python.exe -m tests.test_uia
.\.venv\Scripts\python.exe tests\smoke_mcp_client.py

# Real E2E (opens Notepad, injects real input)
.\.venv\Scripts\python.exe tests\e2e_notepad.py
.\.venv\Scripts\python.exe tests\e2e_ocr_simple.py

Layout

parallel-cua-mcp/
├── knowledge/engineering_apps.json
├── scripts/fetch_engineering_docs.py
├── src/parallel_cua/
│   ├── server.py      # MCP tools
│   ├── wininput.py    # SendInput concurrent core
│   ├── human.py       # human-like driver
│   ├── uia.py         # UIA elements
│   ├── ocr.py         # Windows OCR anchors
│   ├── knowledge.py   # engineering KB + intents
│   ├── dialects.py    # OpenAI / Anthropic
│   ├── http_api.py    # HTTP gateway
│   └── harness.py     # client config profiles
└── tests/

Security

This service can inject arbitrary OS input:

• Do not bind 0.0.0.0 without auth; set PARALLEL_CUA_HTTP_TOKEN for remote HTTP

• Do not run unattended with unrestricted model access

• Call release_all after holds to avoid stuck keys

• Anti-cheat games may reject injection (expected)

Limitations

• Some anti-cheat titles block injected input

• type_text is Unicode injection (chat/forms); use key/hold_keys for game skills

• Multi-monitor uses virtual-desktop coordinates; DPI follows the system coordinate space

• Engineering KB covers common versions; local menus may differ

• research_app needs network (Bing first); offline → local KB/playbooks

Roadmap

• Vision-assisted control detection (OCR + optional detection → click)

• More Chinese/vertical engineering packs

• Record & replay (human ops → game_macro / run_script)

• Multi-monitor partition ops

• Optional UIA element selector (Playwright-like)

License

This project is licensed under the MIT License.

See also: Changelog · Contributing · Security