Skip to main content
Glama
bobliang1979

windeep-computer-control

by bobliang1979

windeep-computer-control

Production-Grade Windows Desktop Automation Engine
生产级 Windows 桌面自动化引擎

Intelligent routing, fault self-healing, UIA COM, SendInput, virtual display — fully autonomous Windows control. 智能路由、故障自愈、UIA COM 直调、SendInput 真实击键、虚拟显示器——完全自主的 Windows 控制。

License Python Windows Version


🌟 Features | 特性

Core Architecture | 核心架构

Feature

Description

Route Integrity Guard

3-function bytecode SHA256 fingerprint — SSDT-style protection against runtime hooking

Fault Lockdown

Auto-lock after 3 consecutive route failures, energy-linked auto-recovery

Energy Model

3-zone ATP metabolism (HIGH/WARNING/COMA) with automatic enter_comath() recovery

Async Verification

_verifier_loop background thread — settle → capture → element assertion → auto lockdown

NT Kernel Mapping

IRQL → input routing priority, SSDT → route guard, Object Manager → layout knowledge

Input Layer | 输入层

Feature

Description

SendInput Real Keystrokes

send_click(x, y), send_type_text(text), send_hotkey([ctrl], 's') — kernel-level events

UIA COM Direct

_uia_com_invoke() — InvokePattern / ExpandCollapsePattern / ValuePattern via Python comtypes

PostMessage Fallback

Background-safe WM_CHAR/WM_CLICK for Win32 controls

XAML Click Adapter

4-tier strategy: comtypes UIA COM → dispatch=auto → foreground SendInput → PostMessage

Capture Layer | 截图层

Feature

Description

DXGI Desktop Duplication

DirectX-based full-screen capture, works on Electron/DirectX windows

PrintWindow API

Occluded/minimized Win32 window capture

BitBlt + MSS

Multi-monitor visible content capture

PIL ImageGrab

Universal fallback

System Infrastructure | 基础设施

Feature

Description

Virtual Display

VirtualDrivers signed driver (v25.7.23) — 2 screens, no test mode required

Clipboard Guard

Multi-format backup/restore (CF_UNICODETEXT + CF_HDROP + CF_DIB)

Resilient Element Matching

Weighted multi-attribute: AutoId(0.50) > Name(0.35) > Class(0.08) > Type(0.05) > Parent(0.02)

Cross-Session Layout Knowledge

LRU disk persistence with fuzzy title matching

UiTreeCache

2s TTL with resilient fallback via element fingerprint

One-Click Deployment

scripts/install_headless.ps1 — 8-step automated setup

Cognitive Unity

cognitive_unity.py — Hermes↔Codex++ unified cognitive loop with ARD Federation

Browser Control | 浏览器控制

Feature

Description

Playwright + Chromium

Direct browser automation — navigate, fill, click, screenshot

CDP Bridge

Chrome DevTools Protocol for multi-tab orchestration


Related MCP server: OmniFlow

📊 Comparison vs Computer Use Plugin | 对比

Dimension

WindEep v3.2

Computer Use Plugin

Route Auto-Degradation

✅ 3-tier SSDT guard

❌ Manual primitive selection

Clipboard Protection

✅ Multi-format guard

❌ None

Async Verify + Self-Heal

_verifier_loop + lockdown

❌ Manual re-snapshot

Virtual Display

✅ 2 screens (signed driver)

❌ Physical desktop only

UIPI Elevation

_get_window_integrity + _uia_route

❌ Not addressed

Cross-Session Memory

✅ LayoutKnowledge LRU

❌ Stateless

SendInput Real Keystrokes

scripts/sendinput.py

✅ Native

WGC Occlusion Capture

✅ DXGI + PrintWindow + BitBlt

✅ Windows.Graphics.Capture

Browser Control

✅ Playwright + Chromium

✅ CDP multi-tab

macOS / iOS

❌ Windows only

Overall: Hermes wins 9 : 4 for Windows desktop control (Codex++ evaluation, 2026-06-29).


🚀 Quick Start | 快速开始

One-Click Headless Deployment | 一键部署

# Run as Administrator | 以管理员身份运行
PowerShell -ExecutionPolicy Bypass -File scripts\install_headless.ps1

This script:

  1. ✅ Checks admin privileges

  2. ✅ Verifies Python availability

  3. ✅ Downloads & installs VirtualDrivers signed display driver

  4. ✅ Validates the windeep project

  5. ✅ Runs Python syntax checks on all modules

  6. ✅ Checks route integrity

  7. ✅ Verifies DXGI screenshot capability

  8. ✅ Creates optional startup scheduled task

Manual Setup | 手动安装

# 1. Install dependencies | 安装依赖
pip install comtypes Pillow mss playwright
python -m playwright install chromium

# 2. Verify virtual display | 验证虚拟显示器
python -c "import ctypes; u=ctypes.windll.user32; print(f'Screens: {u.GetSystemMetrics(80)}')"
# Expected: Screens: 2 (with virtual display) | 预期: 2屏(含虚拟显示器)

# 3. Test route integrity | 测试路由完整性
python computer_control_enhanced.py route-status
# Expected: integrity ✅ OK, locked routes ✅ none

# 4. Start MCP server | 启动MCP服务器
python winctl_mcp_server.py --port 59322

# 5. Register with Hermes | 注册到Hermes
hermes mcp add winctl --url http://127.0.0.1:59322

CLI Usage | 命令行使用

# Route health | 路由健康检查
python computer_control_enhanced.py route-status --json

# Click with UIA COM (XAML-safe) | UIA COM点击(XAML安全)
python computer_control_enhanced.py click --pid 1234 --element 6

# Type text | 输入文本
python computer_control_enhanced.py type-text --pid 1234 "hello world"

# Screenshot | 截图
python computer_control_enhanced.py screenshot --pid 1234

# Cache info | 缓存信息
python computer_control_enhanced.py cache-info

Python API

from computer_control_enhanced import (
    capture, click, type_text, paste_text,
    type_keys, scroll, drag, list_apps,
    _check_routing_integrity, _get_route_status,
    _uia_com_invoke, _xaml_click,
)

# Check system health | 检查系统健康
assert _check_routing_integrity()["ok"]
status = _get_route_status()

# XAML-safe click via UIA COM | UIA COM点击(XAML安全)
result = _uia_com_invoke(
    pid=1234,
    element_name="文件(F)",   # File menu in Chinese Notepad
    hwnd=0x12345678
)

# SendInput real click | SendInput真实点击
from scripts.sendinput import send_click, send_type_text, send_hotkey
send_click(x=500, y=300)
send_type_text("Hello from Hermes!")
send_hotkey(['ctrl'], 's')

# Capture with occlusion support | 遮挡窗口截图
result = _capture_window_wgc(pid=1234, window_id=0x12345678)

# Async verification | 异步验证
_enqueue_verify(pid=1234, window_id=12345678, settle_ms=200)
# Returns immediately, verifier runs in background thread

🏗 System Architecture | 系统架构

windeep/                          ← Project root | 项目根目录
│
├── computer_control_enhanced.py  ← Main orchestration | 主编排层 (1425+ lines)
│                                      Route guard / Energy model / Async verify
│                                      UIA COM / XAML click / WGC capture
│
├── winctl_mcp_server.py          ← MCP Server (22 tools, HTTP :59322)
│
├── compress_image.py             ← Progressive compression pipeline | 渐进式压缩管线
│
├── scripts/
│   ├── sendinput.py              ← SendInput real keystrokes | 真实击键 (NEW v3.1)
│   ├── dxgi_capture.py           ← DXGI + fallback capture | DXGI截图管线 (NEW v3.2)
│   ├── clipboard_guard.py        ← Multi-format clipboard guard | 剪贴板守卫
│   ├── resilient_matcher.py      ← Weighted element matching | 弹性元素匹配
│   ├── layout_knowledge.py       ← Cross-session layout persistence | 布局知识
│   ├── ui_tree_cache.py          ← UI tree cache | UI树缓存
│   ├── element_fingerprint.py    ← SHA256 element fingerprints | 元素指纹
│   ├── ocr_finder.py             ← WinRT OCR engine | 原生OCR
│   ├── smart_matcher.py          ← 5-strategy smart matching | 智能匹配
│   ├── assertion_verifier.py     ← 4-type assertion system | 断言系统
│   ├── shared_ui_state.py        ← Cross-agent shared state | 共享状态
│   ├── action_queue.py           ← Deferred action queue | 延迟队列
│   └── install_headless.ps1      ← One-click deploy | 一键部署

Input Routing Pipeline | 输入路由管线

UIA COM (comtypes, 10ms)
  ├─ ✅ XAML menus, buttons, text fields
  ▼ (fails)
SendInput (real keystrokes, 50ms)
  ├─ ✅ Electron/SPA/UWP apps
  ▼ (fails)
PostMessage (background-safe, 1ms)
  ├─ ✅ Win32 native controls
  ▼ (fails)
ctypes BM_CLICK (last resort)

Capture Pipeline | 截图管线

DXGI Desktop Duplication
  ├─ ✅ Full desktop, occluded windows, Electron/DirectX
  ▼ (fails)
PrintWindow API
  ├─ ✅ Occluded Win32 windows
  ▼ (fails)
BitBlt + MSS
  ├─ ✅ Multi-monitor visible content
  ▼ (fails)
PIL ImageGrab
  ├─ ✅ Universal full-desktop fallback

⚡ Performance | 性能

Operation

Latency

Notes

Route integrity check

<0.1ms

Bytecode SHA256

Route status query

<0.05ms

Lock/energy/stats

UIA COM invoke

10-350ms

First call includes comtypes init

SendInput click

~1ms

Kernel-level event

Clipboard guard

~2ms

Backup → set → restore

Layout knowledge

~3-6ms

Learn + lookup

get_window_state (MCP)

10-100ms

DXGI + UIA tree

type_text (10 chars)

~300ms

30ms/char

capture (DXGI)

10-50ms

Full screen


📋 Requirements | 依赖

Mandatory | 必须

  • OS: Windows 10+ (x64)

  • Python: 3.10+

  • comtypes: pip install comtypes (for UIA COM)

  • Virtual Display Driver: VirtualDrivers/Virtual-Display-Driver v25.7.23 (signed, no test mode)

  • Pillow: pip install Pillow (screenshot compression)

  • mss: pip install mss (multi-monitor capture)

  • Playwright: pip install playwright && python -m playwright install chromium (browser control)

  • Hermes Agent: Hermes Agent (for MCP integration)

Zero Dependencies | 零依赖

Core functions (window management, input control, UIA COM) work via ctypes + comtypes only. No npm, no Node.js, no .NET runtime required.


🧪 Verification | 验证

# Run full system audit | 全系统审计
python -c "
import computer_control_enhanced as cce
# Route integrity | 路由完整性
assert cce._check_routing_integrity()['ok']
# Energy | 能量
assert cce._get_route_status()['energy_ok']
# No locked routes | 无锁定
assert not cce._get_route_status()['locked_routes']
# UIA COM | UIA COM可用
assert hasattr(cce, '_uia_com_invoke')
# XAML click | XAML点击可用
assert hasattr(cce, '_xaml_click')
# WGC capture | 截图可用
assert hasattr(cce, '_capture_window_wgc')
print('ALL CHECKS PASSED')
"

📜 License | 许可证

Apache 2.0


🤝 Architecture Credits | 架构贡献

  • Hermes Agent (Nous Research) — Agent runtime & MCP framework

  • cua-driver — Windows desktop control backend (38 MCP tools)

  • Codex++ (bobliang1979) — Cognitive bridge, UIA COM research, architecture audit

  • VirtualDrivers — Signed virtual display driver

  • Computer Use Plugin (OpenAI) — Reference architecture for comparison


© 2026 BOBLIANG. All rights reserved. Built by Hermes Agent × Codex++

Related MCP Connectors

Related MCP Servers

  • A
    license
    Not graded
    quality
    D
    maintenance
    Windows automation MCP offering * AI Vision (e.g. Click by Description) * Windows UI Automation Tree tools * Chrome Automation via Playwright * Mouse control * Keyboard control * a lot more (>40 tools) Also comes with Python/TypeScript/C# client libs and a Windows Desktop tool to try all the tools.
    2
    MIT
  • A
    license
    Not graded
    quality
    D
    maintenance
    Enables AI clients to automate Windows desktop applications through window manipulation, image recognition, OCR, keyboard/mouse simulation, and memory operations via the MCP protocol.
    MIT