Skip to main content
Glama
bobliang1979

windeep-computer-control

by bobliang1979

windeep-computer-control

Production-Grade Windows Desktop Automation Engine
生产级 Windows 桌面自动化引擎

Intelligent routing, fault self-healing, UIA COM, SendInput, virtual display — fully autonomous Windows control. 智能路由、故障自愈、UIA COM 直调、SendInput 真实击键、虚拟显示器——完全自主的 Windows 控制。

License Python Windows Version


🌟 Features | 特性

Core Architecture | 核心架构

Feature

Description

Route Integrity Guard

3-function bytecode SHA256 fingerprint — SSDT-style protection against runtime hooking

Fault Lockdown

Auto-lock after 3 consecutive route failures, energy-linked auto-recovery

Energy Model

3-zone ATP metabolism (HIGH/WARNING/COMA) with automatic enter_comath() recovery

Async Verification

_verifier_loop background thread — settle → capture → element assertion → auto lockdown

NT Kernel Mapping

IRQL → input routing priority, SSDT → route guard, Object Manager → layout knowledge

Input Layer | 输入层

Feature

Description

SendInput Real Keystrokes

send_click(x, y), send_type_text(text), send_hotkey([ctrl], 's') — kernel-level events

UIA COM Direct

_uia_com_invoke() — InvokePattern / ExpandCollapsePattern / ValuePattern via Python comtypes

PostMessage Fallback

Background-safe WM_CHAR/WM_CLICK for Win32 controls

XAML Click Adapter

4-tier strategy: comtypes UIA COM → dispatch=auto → foreground SendInput → PostMessage

Capture Layer | 截图层

Feature

Description

DXGI Desktop Duplication

DirectX-based full-screen capture, works on Electron/DirectX windows

PrintWindow API

Occluded/minimized Win32 window capture

BitBlt + MSS

Multi-monitor visible content capture

PIL ImageGrab

Universal fallback

System Infrastructure | 基础设施

Feature

Description

Virtual Display

VirtualDrivers signed driver (v25.7.23) — 2 screens, no test mode required

Clipboard Guard

Multi-format backup/restore (CF_UNICODETEXT + CF_HDROP + CF_DIB)

Resilient Element Matching

Weighted multi-attribute: AutoId(0.50) > Name(0.35) > Class(0.08) > Type(0.05) > Parent(0.02)

Cross-Session Layout Knowledge

LRU disk persistence with fuzzy title matching

UiTreeCache

2s TTL with resilient fallback via element fingerprint

One-Click Deployment

scripts/install_headless.ps1 — 8-step automated setup

Cognitive Unity

cognitive_unity.py — Hermes↔Codex++ unified cognitive loop with ARD Federation

Browser Control | 浏览器控制

Feature

Description

Playwright + Chromium

Direct browser automation — navigate, fill, click, screenshot

CDP Bridge

Chrome DevTools Protocol for multi-tab orchestration


Related MCP server: Orbination AI Desktop Vision & Control

📊 Comparison vs Computer Use Plugin | 对比

Dimension

WindEep v3.2

Computer Use Plugin

Route Auto-Degradation

✅ 3-tier SSDT guard

❌ Manual primitive selection

Clipboard Protection

✅ Multi-format guard

❌ None

Async Verify + Self-Heal

_verifier_loop + lockdown

❌ Manual re-snapshot

Virtual Display

✅ 2 screens (signed driver)

❌ Physical desktop only

UIPI Elevation

_get_window_integrity + _uia_route

❌ Not addressed

Cross-Session Memory

✅ LayoutKnowledge LRU

❌ Stateless

SendInput Real Keystrokes

scripts/sendinput.py

✅ Native

WGC Occlusion Capture

✅ DXGI + PrintWindow + BitBlt

✅ Windows.Graphics.Capture

Browser Control

✅ Playwright + Chromium

✅ CDP multi-tab

macOS / iOS

❌ Windows only

Overall: Hermes wins 9 : 4 for Windows desktop control (Codex++ evaluation, 2026-06-29).


🚀 Quick Start | 快速开始

One-Click Headless Deployment | 一键部署

# Run as Administrator | 以管理员身份运行
PowerShell -ExecutionPolicy Bypass -File scripts\install_headless.ps1

This script:

  1. ✅ Checks admin privileges

  2. ✅ Verifies Python availability

  3. ✅ Downloads & installs VirtualDrivers signed display driver

  4. ✅ Validates the windeep project

  5. ✅ Runs Python syntax checks on all modules

  6. ✅ Checks route integrity

  7. ✅ Verifies DXGI screenshot capability

  8. ✅ Creates optional startup scheduled task

Manual Setup | 手动安装

# 1. Install dependencies | 安装依赖
pip install comtypes Pillow mss playwright
python -m playwright install chromium

# 2. Verify virtual display | 验证虚拟显示器
python -c "import ctypes; u=ctypes.windll.user32; print(f'Screens: {u.GetSystemMetrics(80)}')"
# Expected: Screens: 2 (with virtual display) | 预期: 2屏(含虚拟显示器)

# 3. Test route integrity | 测试路由完整性
python computer_control_enhanced.py route-status
# Expected: integrity ✅ OK, locked routes ✅ none

# 4. Start MCP server | 启动MCP服务器
python winctl_mcp_server.py --port 59322

# 5. Register with Hermes | 注册到Hermes
hermes mcp add winctl --url http://127.0.0.1:59322

CLI Usage | 命令行使用

# Route health | 路由健康检查
python computer_control_enhanced.py route-status --json

# Click with UIA COM (XAML-safe) | UIA COM点击(XAML安全)
python computer_control_enhanced.py click --pid 1234 --element 6

# Type text | 输入文本
python computer_control_enhanced.py type-text --pid 1234 "hello world"

# Screenshot | 截图
python computer_control_enhanced.py screenshot --pid 1234

# Cache info | 缓存信息
python computer_control_enhanced.py cache-info

Python API

from computer_control_enhanced import (
    capture, click, type_text, paste_text,
    type_keys, scroll, drag, list_apps,
    _check_routing_integrity, _get_route_status,
    _uia_com_invoke, _xaml_click,
)

# Check system health | 检查系统健康
assert _check_routing_integrity()["ok"]
status = _get_route_status()

# XAML-safe click via UIA COM | UIA COM点击(XAML安全)
result = _uia_com_invoke(
    pid=1234,
    element_name="文件(F)",   # File menu in Chinese Notepad
    hwnd=0x12345678
)

# SendInput real click | SendInput真实点击
from scripts.sendinput import send_click, send_type_text, send_hotkey
send_click(x=500, y=300)
send_type_text("Hello from Hermes!")
send_hotkey(['ctrl'], 's')

# Capture with occlusion support | 遮挡窗口截图
result = _capture_window_wgc(pid=1234, window_id=0x12345678)

# Async verification | 异步验证
_enqueue_verify(pid=1234, window_id=12345678, settle_ms=200)
# Returns immediately, verifier runs in background thread

🏗 System Architecture | 系统架构

windeep/                          ← Project root | 项目根目录
│
├── computer_control_enhanced.py  ← Main orchestration | 主编排层 (1425+ lines)
│                                      Route guard / Energy model / Async verify
│                                      UIA COM / XAML click / WGC capture
│
├── winctl_mcp_server.py          ← MCP Server (22 tools, HTTP :59322)
│
├── compress_image.py             ← Progressive compression pipeline | 渐进式压缩管线
│
├── scripts/
│   ├── sendinput.py              ← SendInput real keystrokes | 真实击键 (NEW v3.1)
│   ├── dxgi_capture.py           ← DXGI + fallback capture | DXGI截图管线 (NEW v3.2)
│   ├── clipboard_guard.py        ← Multi-format clipboard guard | 剪贴板守卫
│   ├── resilient_matcher.py      ← Weighted element matching | 弹性元素匹配
│   ├── layout_knowledge.py       ← Cross-session layout persistence | 布局知识
│   ├── ui_tree_cache.py          ← UI tree cache | UI树缓存
│   ├── element_fingerprint.py    ← SHA256 element fingerprints | 元素指纹
│   ├── ocr_finder.py             ← WinRT OCR engine | 原生OCR
│   ├── smart_matcher.py          ← 5-strategy smart matching | 智能匹配
│   ├── assertion_verifier.py     ← 4-type assertion system | 断言系统
│   ├── shared_ui_state.py        ← Cross-agent shared state | 共享状态
│   ├── action_queue.py           ← Deferred action queue | 延迟队列
│   └── install_headless.ps1      ← One-click deploy | 一键部署

Input Routing Pipeline | 输入路由管线

UIA COM (comtypes, 10ms)
  ├─ ✅ XAML menus, buttons, text fields
  ▼ (fails)
SendInput (real keystrokes, 50ms)
  ├─ ✅ Electron/SPA/UWP apps
  ▼ (fails)
PostMessage (background-safe, 1ms)
  ├─ ✅ Win32 native controls
  ▼ (fails)
ctypes BM_CLICK (last resort)

Capture Pipeline | 截图管线

DXGI Desktop Duplication
  ├─ ✅ Full desktop, occluded windows, Electron/DirectX
  ▼ (fails)
PrintWindow API
  ├─ ✅ Occluded Win32 windows
  ▼ (fails)
BitBlt + MSS
  ├─ ✅ Multi-monitor visible content
  ▼ (fails)
PIL ImageGrab
  ├─ ✅ Universal full-desktop fallback

⚡ Performance | 性能

Operation

Latency

Notes

Route integrity check

<0.1ms

Bytecode SHA256

Route status query

<0.05ms

Lock/energy/stats

UIA COM invoke

10-350ms

First call includes comtypes init

SendInput click

~1ms

Kernel-level event

Clipboard guard

~2ms

Backup → set → restore

Layout knowledge

~3-6ms

Learn + lookup

get_window_state (MCP)

10-100ms

DXGI + UIA tree

type_text (10 chars)

~300ms

30ms/char

capture (DXGI)

10-50ms

Full screen


📋 Requirements | 依赖

Mandatory | 必须

  • OS: Windows 10+ (x64)

  • Python: 3.10+

  • comtypes: pip install comtypes (for UIA COM)

  • Virtual Display Driver: VirtualDrivers/Virtual-Display-Driver v25.7.23 (signed, no test mode)

  • Pillow: pip install Pillow (screenshot compression)

  • mss: pip install mss (multi-monitor capture)

  • Playwright: pip install playwright && python -m playwright install chromium (browser control)

  • Hermes Agent: Hermes Agent (for MCP integration)

Zero Dependencies | 零依赖

Core functions (window management, input control, UIA COM) work via ctypes + comtypes only. No npm, no Node.js, no .NET runtime required.


🧪 Verification | 验证

# Run full system audit | 全系统审计
python -c "
import computer_control_enhanced as cce
# Route integrity | 路由完整性
assert cce._check_routing_integrity()['ok']
# Energy | 能量
assert cce._get_route_status()['energy_ok']
# No locked routes | 无锁定
assert not cce._get_route_status()['locked_routes']
# UIA COM | UIA COM可用
assert hasattr(cce, '_uia_com_invoke')
# XAML click | XAML点击可用
assert hasattr(cce, '_xaml_click')
# WGC capture | 截图可用
assert hasattr(cce, '_capture_window_wgc')
print('ALL CHECKS PASSED')
"

📜 License | 许可证

Apache 2.0


🤝 Architecture Credits | 架构贡献

  • Hermes Agent (Nous Research) — Agent runtime & MCP framework

  • cua-driver — Windows desktop control backend (38 MCP tools)

  • Codex++ (bobliang1979) — Cognitive bridge, UIA COM research, architecture audit

  • VirtualDrivers — Signed virtual display driver

  • Computer Use Plugin (OpenAI) — Reference architecture for comparison


© 2026 BOBLIANG. All rights reserved. Built by Hermes Agent × Codex++

A
license - permissive license
-
quality - not tested
B
maintenance

Maintenance

Maintainers
Response time
Release cycle
Releases (12mo)
Commit activity

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/bobliang1979/windeep-computer-control'

If you have feedback or need assistance with the MCP directory API, please join our Discord server