windeep-computer-control
Allows registering the MCP server with Hermes agent for Windows desktop automation, enabling AI agents to control Windows applications through the Hermes platform.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@windeep-computer-control打开记事本,输入'Hello MCP',然后截图"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
windeep-computer-control
Production-Grade Windows Desktop Automation Engine
生产级 Windows 桌面自动化引擎Intelligent routing, fault self-healing, UIA COM, SendInput, virtual display — fully autonomous Windows control. 智能路由、故障自愈、UIA COM 直调、SendInput 真实击键、虚拟显示器——完全自主的 Windows 控制。
🌟 Features | 特性
Core Architecture | 核心架构
Feature | Description |
Route Integrity Guard | 3-function bytecode SHA256 fingerprint — SSDT-style protection against runtime hooking |
Fault Lockdown | Auto-lock after 3 consecutive route failures, energy-linked auto-recovery |
Energy Model | 3-zone ATP metabolism (HIGH/WARNING/COMA) with automatic |
Async Verification |
|
NT Kernel Mapping | IRQL → input routing priority, SSDT → route guard, Object Manager → layout knowledge |
Input Layer | 输入层
Feature | Description |
SendInput Real Keystrokes |
|
UIA COM Direct |
|
PostMessage Fallback | Background-safe WM_CHAR/WM_CLICK for Win32 controls |
XAML Click Adapter | 4-tier strategy: comtypes UIA COM → dispatch=auto → foreground SendInput → PostMessage |
Capture Layer | 截图层
Feature | Description |
DXGI Desktop Duplication | DirectX-based full-screen capture, works on Electron/DirectX windows |
PrintWindow API | Occluded/minimized Win32 window capture |
BitBlt + MSS | Multi-monitor visible content capture |
PIL ImageGrab | Universal fallback |
System Infrastructure | 基础设施
Feature | Description |
Virtual Display | VirtualDrivers signed driver (v25.7.23) — 2 screens, no test mode required |
Clipboard Guard | Multi-format backup/restore (CF_UNICODETEXT + CF_HDROP + CF_DIB) |
Resilient Element Matching | Weighted multi-attribute: AutoId(0.50) > Name(0.35) > Class(0.08) > Type(0.05) > Parent(0.02) |
Cross-Session Layout Knowledge | LRU disk persistence with fuzzy title matching |
UiTreeCache | 2s TTL with resilient fallback via element fingerprint |
One-Click Deployment |
|
Cognitive Unity |
|
Browser Control | 浏览器控制
Feature | Description |
Playwright + Chromium | Direct browser automation — navigate, fill, click, screenshot |
CDP Bridge | Chrome DevTools Protocol for multi-tab orchestration |
Related MCP server: OmniFlow
📊 Comparison vs Computer Use Plugin | 对比
Dimension | WindEep v3.2 | Computer Use Plugin |
Route Auto-Degradation | ✅ 3-tier SSDT guard | ❌ Manual primitive selection |
Clipboard Protection | ✅ Multi-format guard | ❌ None |
Async Verify + Self-Heal | ✅ | ❌ Manual re-snapshot |
Virtual Display | ✅ 2 screens (signed driver) | ❌ Physical desktop only |
UIPI Elevation | ✅ | ❌ Not addressed |
Cross-Session Memory | ✅ LayoutKnowledge LRU | ❌ Stateless |
SendInput Real Keystrokes | ✅ | ✅ Native |
WGC Occlusion Capture | ✅ DXGI + PrintWindow + BitBlt | ✅ Windows.Graphics.Capture |
Browser Control | ✅ Playwright + Chromium | ✅ CDP multi-tab |
macOS / iOS | ❌ Windows only | ✅ |
Overall: Hermes wins 9 : 4 for Windows desktop control (Codex++ evaluation, 2026-06-29).
🚀 Quick Start | 快速开始
One-Click Headless Deployment | 一键部署
# Run as Administrator | 以管理员身份运行
PowerShell -ExecutionPolicy Bypass -File scripts\install_headless.ps1This script:
✅ Checks admin privileges
✅ Verifies Python availability
✅ Downloads & installs VirtualDrivers signed display driver
✅ Validates the windeep project
✅ Runs Python syntax checks on all modules
✅ Checks route integrity
✅ Verifies DXGI screenshot capability
✅ Creates optional startup scheduled task
Manual Setup | 手动安装
# 1. Install dependencies | 安装依赖
pip install comtypes Pillow mss playwright
python -m playwright install chromium
# 2. Verify virtual display | 验证虚拟显示器
python -c "import ctypes; u=ctypes.windll.user32; print(f'Screens: {u.GetSystemMetrics(80)}')"
# Expected: Screens: 2 (with virtual display) | 预期: 2屏(含虚拟显示器)
# 3. Test route integrity | 测试路由完整性
python computer_control_enhanced.py route-status
# Expected: integrity ✅ OK, locked routes ✅ none
# 4. Start MCP server | 启动MCP服务器
python winctl_mcp_server.py --port 59322
# 5. Register with Hermes | 注册到Hermes
hermes mcp add winctl --url http://127.0.0.1:59322CLI Usage | 命令行使用
# Route health | 路由健康检查
python computer_control_enhanced.py route-status --json
# Click with UIA COM (XAML-safe) | UIA COM点击(XAML安全)
python computer_control_enhanced.py click --pid 1234 --element 6
# Type text | 输入文本
python computer_control_enhanced.py type-text --pid 1234 "hello world"
# Screenshot | 截图
python computer_control_enhanced.py screenshot --pid 1234
# Cache info | 缓存信息
python computer_control_enhanced.py cache-infoPython API
from computer_control_enhanced import (
capture, click, type_text, paste_text,
type_keys, scroll, drag, list_apps,
_check_routing_integrity, _get_route_status,
_uia_com_invoke, _xaml_click,
)
# Check system health | 检查系统健康
assert _check_routing_integrity()["ok"]
status = _get_route_status()
# XAML-safe click via UIA COM | UIA COM点击(XAML安全)
result = _uia_com_invoke(
pid=1234,
element_name="文件(F)", # File menu in Chinese Notepad
hwnd=0x12345678
)
# SendInput real click | SendInput真实点击
from scripts.sendinput import send_click, send_type_text, send_hotkey
send_click(x=500, y=300)
send_type_text("Hello from Hermes!")
send_hotkey(['ctrl'], 's')
# Capture with occlusion support | 遮挡窗口截图
result = _capture_window_wgc(pid=1234, window_id=0x12345678)
# Async verification | 异步验证
_enqueue_verify(pid=1234, window_id=12345678, settle_ms=200)
# Returns immediately, verifier runs in background thread🏗 System Architecture | 系统架构
windeep/ ← Project root | 项目根目录
│
├── computer_control_enhanced.py ← Main orchestration | 主编排层 (1425+ lines)
│ Route guard / Energy model / Async verify
│ UIA COM / XAML click / WGC capture
│
├── winctl_mcp_server.py ← MCP Server (22 tools, HTTP :59322)
│
├── compress_image.py ← Progressive compression pipeline | 渐进式压缩管线
│
├── scripts/
│ ├── sendinput.py ← SendInput real keystrokes | 真实击键 (NEW v3.1)
│ ├── dxgi_capture.py ← DXGI + fallback capture | DXGI截图管线 (NEW v3.2)
│ ├── clipboard_guard.py ← Multi-format clipboard guard | 剪贴板守卫
│ ├── resilient_matcher.py ← Weighted element matching | 弹性元素匹配
│ ├── layout_knowledge.py ← Cross-session layout persistence | 布局知识
│ ├── ui_tree_cache.py ← UI tree cache | UI树缓存
│ ├── element_fingerprint.py ← SHA256 element fingerprints | 元素指纹
│ ├── ocr_finder.py ← WinRT OCR engine | 原生OCR
│ ├── smart_matcher.py ← 5-strategy smart matching | 智能匹配
│ ├── assertion_verifier.py ← 4-type assertion system | 断言系统
│ ├── shared_ui_state.py ← Cross-agent shared state | 共享状态
│ ├── action_queue.py ← Deferred action queue | 延迟队列
│ └── install_headless.ps1 ← One-click deploy | 一键部署Input Routing Pipeline | 输入路由管线
UIA COM (comtypes, 10ms)
├─ ✅ XAML menus, buttons, text fields
▼ (fails)
SendInput (real keystrokes, 50ms)
├─ ✅ Electron/SPA/UWP apps
▼ (fails)
PostMessage (background-safe, 1ms)
├─ ✅ Win32 native controls
▼ (fails)
ctypes BM_CLICK (last resort)Capture Pipeline | 截图管线
DXGI Desktop Duplication
├─ ✅ Full desktop, occluded windows, Electron/DirectX
▼ (fails)
PrintWindow API
├─ ✅ Occluded Win32 windows
▼ (fails)
BitBlt + MSS
├─ ✅ Multi-monitor visible content
▼ (fails)
PIL ImageGrab
├─ ✅ Universal full-desktop fallback⚡ Performance | 性能
Operation | Latency | Notes |
Route integrity check | <0.1ms | Bytecode SHA256 |
Route status query | <0.05ms | Lock/energy/stats |
UIA COM invoke | 10-350ms | First call includes comtypes init |
SendInput click | ~1ms | Kernel-level event |
Clipboard guard | ~2ms | Backup → set → restore |
Layout knowledge | ~3-6ms | Learn + lookup |
get_window_state (MCP) | 10-100ms | DXGI + UIA tree |
type_text (10 chars) | ~300ms | 30ms/char |
capture (DXGI) | 10-50ms | Full screen |
📋 Requirements | 依赖
Mandatory | 必须
OS: Windows 10+ (x64)
Python: 3.10+
comtypes:
pip install comtypes(for UIA COM)
Recommended | 推荐
Virtual Display Driver: VirtualDrivers/Virtual-Display-Driver v25.7.23 (signed, no test mode)
Pillow:
pip install Pillow(screenshot compression)mss:
pip install mss(multi-monitor capture)Playwright:
pip install playwright && python -m playwright install chromium(browser control)Hermes Agent: Hermes Agent (for MCP integration)
Zero Dependencies | 零依赖
Core functions (window management, input control, UIA COM) work via ctypes + comtypes only. No npm, no Node.js, no .NET runtime required.
🧪 Verification | 验证
# Run full system audit | 全系统审计
python -c "
import computer_control_enhanced as cce
# Route integrity | 路由完整性
assert cce._check_routing_integrity()['ok']
# Energy | 能量
assert cce._get_route_status()['energy_ok']
# No locked routes | 无锁定
assert not cce._get_route_status()['locked_routes']
# UIA COM | UIA COM可用
assert hasattr(cce, '_uia_com_invoke')
# XAML click | XAML点击可用
assert hasattr(cce, '_xaml_click')
# WGC capture | 截图可用
assert hasattr(cce, '_capture_window_wgc')
print('ALL CHECKS PASSED')
"📜 License | 许可证
🤝 Architecture Credits | 架构贡献
Hermes Agent (Nous Research) — Agent runtime & MCP framework
cua-driver — Windows desktop control backend (38 MCP tools)
Codex++ (bobliang1979) — Cognitive bridge, UIA COM research, architecture audit
VirtualDrivers — Signed virtual display driver
Computer Use Plugin (OpenAI) — Reference architecture for comparison
© 2026 BOBLIANG. All rights reserved. Built by Hermes Agent × Codex++
This server cannot be deployed
Maintenance
Related MCP Connectors
Open-source Zapier/n8n alternative as an MCP server: agents build, run and debug your workflows.
Personal assistant MCP server with search, execute, packages, jobs, secrets, and integrations.
MCP server for progressive tool usage at any scale (see https://klavis.ai)
Automate 1,000+ services from any MCP-compatible AI agent: build Applets, run actions and queries.
Related MCP Servers
- AlicenseNot gradedqualityDmaintenanceWindows automation MCP offering * AI Vision (e.g. Click by Description) * Windows UI Automation Tree tools * Chrome Automation via Playwright * Mouse control * Keyboard control * a lot more (>40 tools) Also comes with Python/TypeScript/C# client libs and a Windows Desktop tool to try all the tools.2MIT
- AlicenseNot gradedqualityDmaintenanceEnables AI clients to automate Windows desktop applications through window manipulation, image recognition, OCR, keyboard/mouse simulation, and memory operations via the MCP protocol.MIT
- AlicenseNot gradedqualityCmaintenanceWindows desktop automation MCP server supporting mouse, keyboard, screenshot, and image recognition, with native CJK text input via clipboard.MIT
- AlicenseNot gradedqualityCmaintenanceMCP server for Windows desktop automation, enabling agents to control mouse, keyboard, and UI elements via 22 tools, with OCR, screenshots, macro recording, and autonomous mission execution.MIT