Computer Use v2
This server does not currently have a deployable release on Glama. Follow the repository's setup instructions to install and configure it for your MCP client, including any required credentials and local dependencies.
Computer Use v2 (Presentation & Perception Engine)
基于《桌面 Agent Computer Use 插件开发方案》全新重构的新一代桌面控制 MCP 插件。
针对上一代 local-computer-use 在使用体验上的关键痛点(抢占物理鼠标、缺乏实时操作视觉反馈、多屏/高 DPI 缩放偏差、缺少主动人类接管机制),本插件引入了透明置顶呈现图层(Presentation Overlay)、AI 专属虚拟光标、主动人机共存安全守护与高性能无障碍语义树。
核心架构与创新特性
1. 仿 ChatGPT 桌面版的呈现图层 (Presentation Overlay)
零依赖原生加速:基于 Python 3.13 与 PyQt6 6.11.0 构建全屏穿透透明图层(
WS_EX_TRANSPARENT | WS_EX_LAYERED | WS_EX_NOACTIVATE)。AI 专属虚拟光标:在屏幕上以 60 FPS 三次贝塞尔曲线平滑呈现 AI 思考与操作轨迹(带渐变发光质感与运动残影)。
点击水波涟漪:左键(天蓝)、右键(琥珀)、中键(紫罗兰)同心圆扩散渐隐动画。
目标高亮框:圆角微光矩形框选当前正在操作的目标控件。
全屏流光微光呼吸边框:四边流动渐变边框,直观提示「AI 正在操作桌面」。
顶部状态胶囊与 Keycast 按键气泡:实时展示当前动作描述与键盘键入按键。
截图彻底隐身 (WDA_EXCLUDEFROMCAPTURE):调用 Win32
SetWindowDisplayAffinity(hwnd, 0x11),所有截屏与录屏完全由 DWM 硬件级剔除 Overlay,绝不污染模型视觉上下文。
2. 双执行模式 (Dual Execution Modes)
virtual模式(默认,AI 第二光标):AI 虚拟光标负责在屏幕上平滑演出移动、涟漪与路径引导,后台执行点击/拖拽/滚轮后立即恢复物理光标原位,且不因实体鼠标移动/点击而暂停(实体鼠标与 AI 互不干扰并行工作)。real模式:通过平滑贝塞尔缓动真正驱动系统物理光标,并启用实体鼠标接管暂停保护,适合全自动录制或接管演示。
3. 紧急熔断与安全护栏 (Emergency Fuse & Safety Guard)
单按 Esc 熔断 (
esc_double_interval_sec: 0.0):无论处于virtual还是real模式,在活跃回合中单次按下 Esc 键立即触发紧急停止(EMERGENCY_STOP_TRIGGERED),瞬间阻断本轮 Agent 后续操作;空闲未执行回合时按 Esc 不会误触发锁定。物理鼠标接管暂停 (
real模式):在real模式下基于pynput监听物理鼠标(位移 > 15px 或人为点击),检测到用户介入时立即PAUSE让出控制权;virtual模式下保持关闭以便人类自由使用实体鼠标。高危操作防线:支持配置支付、敏感删除、对外发送信息等策略护栏(默认可通过
config.json调整)。
4. 高精度坐标感知与 Windows UIA
Per-Monitor DPI V2 感知:自动适应 100%、125%、150% 等任意笔记本缩放与多显示器。
跨屏虚拟桌面负坐标:正确处理多屏幕扩展布局中出现的负坐标。
BFS 高效无障碍树遍历:
< 100ms生成紧凑带有element_id的控件树,支持根据元素直接定位点击。
Related MCP server: Extra
目录结构
computer-use-v2/
├── overlay/ # 呈现图层模块
│ ├── window.py # PyQt6 全屏透明置顶穿透窗口,WDA_EXCLUDEFROMCAPTURE
│ ├── cursor.py # 渐变发光虚拟光标与三次贝塞尔 60fps 平滑移动
│ ├── ripple.py # 点击同心水波涟漪动效
│ ├── highlight.py # 目标控件高亮边框
│ ├── border.py # 全屏流光呼吸边框、状态胶囊条与按键气泡
│ └── service.py # 独立子进程 IPC 驱动与队列管理
├── core/ # 感知、执行与安全核心
│ ├── geometry.py # Per-Monitor DPI 感知、负坐标处理与坐标换算
│ ├── capture.py # 高保真截屏(MSS + PrintWindow 降级,排除 Overlay)
│ ├── input.py # SendInput 封装(virtual / real 模式、Unicode/Emoji、拖拽)
│ ├── safety.py # pynput 鼠标接管监听、Esc 急停与敏感动作拦截
│ ├── audit.py # JSONL 审计日志、敏感文本脱敏与回合预算控制
│ ├── uia.py # Windows UI Automation 树遍历与语义动作
│ └── window.py # 窗口枚举、状态查询与激活
├── tests/ # 自动化测试套件
│ ├── selftest.py # 13 项端到端集成自检
│ ├── test_core.py # 核心模块单元测试
│ ├── test_overlay.py # 呈现图层单元测试
│ └── test_server_mcp.py # JSON-RPC 2.0 stdio 协议测试
├── server.py # MCP 标准 stdio 服务入口
├── config.json # 默认配置(模式、动效速度、安全阈值)
├── plugin.json # Antigravity 插件清单
└── skill/
└── SKILL.md # 技能规范与 Agent 提示词指导环境要求与安装
操作系统:Windows 10 / Windows 11(依赖 Win32 API、DWM
WDA_EXCLUDEFROMCAPTURE及 Windows UI Automation)Python:Python 3.10+(推荐 Python 3.12 / 3.13)
1. 克隆仓库并安装依赖
git clone https://github.com/enenhengrusty/computer-use-v2.git
cd computer-use-v2
pip install -r requirements.txt
# 或使用 uv:
# uv pip install -r requirements.txt2. 配置使用
方式 A:作为 Antigravity 插件加载
将仓库克隆或复制到 %USERPROFILE%\.gemini\config\plugins\computer-use-v2:
git clone https://github.com/enenhengrusty/computer-use-v2.git "$env:USERPROFILE\.gemini\config\plugins\computer-use-v2"如需独立注册 Skill,可将 skill\SKILL.md 复制到:
%USERPROFILE%\.gemini\config\skills\computer-use-v2\SKILL.md
方式 B:在通用 MCP 客户端中注册(Antigravity / Claude Desktop / Cursor 等)
在你的 MCP 配置文件(如 %USERPROFILE%\.gemini\config\mcp_config.json 或客户端的 mcpServers 配置)中添加(将 <path-to-repo> 替换为本地实际路径,若使用虚拟环境请将 command 指向对应 python.exe):
{
"mcpServers": {
"computer-use-v2": {
"command": "python",
"args": [
"<path-to-repo>\\computer-use-v2\\server.py"
],
"env": {
"PYTHONIOENCODING": "utf-8",
"PYTHONUTF8": "1"
}
}
}
}运行自动化自检
端到端集成自检(在仓库根目录下运行):
python tests/selftest.py单元测试集:
python -m unittest discover -s tests -p "test_*.py"Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
This server cannot be deployed
Maintenance
Related MCP Connectors
Eyes and hands on real Windows PCs — observe, click, type via Glasswarp API.
Connect AI assistants to your approved computer for files, commands, browser and desktop tasks.
Real desktops for AI agents, on machines you choose: screenshot, type, click, hire more agents.
Securely control computers you explicitly pair through files, terminals, processes, screenshots, desktop UI/input, clipboard, browser automation, diagnostics, and document tools.
Related MCP Servers
- AlicenseNot gradedqualityCmaintenanceEnables safe Windows desktop automation and computer use through natural language, including window observation, UI Automation, and execution of verified actions like clicking, typing, and scrolling.2MIT
- AlicenseNot gradedqualityBmaintenanceEnables AI assistants to autonomously control Windows 11 and 10 desktops via sub-10ms screen capture, native UI Automation element inspection, and zero-lag keyboard and mouse input. Combines a visual plane with a semantic plane and stall detection so agents can operate real applications reliably without vision-only guessing.262 npm171 PyPI7MIT
- AlicenseNot gradedqualityCmaintenanceEnables AI assistants to drive Windows desktop automation — opening and interacting with applications, native UI controls, keyboard/mouse input, files, processes, clipboard, and Chrome/Edge browsing via UI Automation and CDP. All actions pass through a safety policy with human approval, emergency stop, and deterministic verification.1MIT
- AlicenseNot gradedqualityBmaintenanceEnables AI clients to inspect and control allowlisted Windows applications using screenshots, mouse actions, Unicode keyboard input, UI Automation, and an Esc/F12 emergency stop.1Apache 2.0