WinAgent-Lite 🤖🖱️

A GUI Agent driven by local vision models — also a plug-and-play MCP plugin that gives any AI platform "eyes" and "hands".

An ordinary computer without a dedicated GPU: a local small model acts as the eyes (recognizing screen elements) plus real keyboard and mouse events act as the hands (operating any software like a human), and an evaluation suite quantifies the success rate. From "able to demo" to "able to measure", and then to "able to be called by any platform".

What It Can Do

Call it on demand like a plugin: the platform mounts the 9 tools into its own tool list (look / click / type_text / key / act / run_scenario / doctor / screenshot / discover); when the business needs to "click a button" it calls act, and when it needs to "see the screen" it calls screenshot. Multi-software collaboration is orchestrated by the host, while WinAgent only provides the real "eyes" and "hands". See docs/PLUGINS.md for how to connect.

Related MCP server: atomic-computer-mcp

Architecture

vision.py    眼睛: 截屏 -> 降采样 -> Ollama VLM -> 坐标/问答      （跨平台）
hand/        手:   win32(user32) | x11(XTest) | macos(Quartz)   （平台后端分发）
agent.py     脑干: 闭环 + 步骤解释器 + 焦点等待 + 容差自愈
bench.py     度量: 任务 YAML(含 setup/teardown) -> 成功率报告
planner.py   规划: 目标 -> 步骤 JSON（本地模型，云 API 适配器预留）
mcp_server.py 插件: MCP stdio server，9 工具，协议一致性 CI 验证

Quick Start

git clone https://github.com/ZYYDI1959/winagent-lite && cd winagent-lite
python -m venv .venv && .venv/Scripts/pip install -e .
# 需要本机 Ollama（默认模型 qwen2.5vl:7b）
cp config.example.yaml config.yaml

winagent doctor                          # 环境自检（一条命令诊断一切）
winagent look "任务栏右下角的时钟"         # 眼睛：FOUND x,y
winagent click 1824 1056                 # 手：真实点击
winagent run scenarios/notepad_save.yaml # 闭环：YAML 步骤
winagent bench                           # 评测：全任务成功率报告
winagent plan "打开记事本输入你好并保存"     # 规划：目标 -> 步骤
winagent-mcp                             # 以 MCP 插件模式运行

Using as a MCP Plugin (30-second integration)

{ "mcpServers": { "winagent": {
    "command": "D:\\winagent-lite\\.venv\\Scripts\\python.exe",
    "args": ["-m", "winagent.mcp_server"] } } }

Your AI platform can then: screenshot to look at your screen → act to click any software → type_text to type. Detailed configuration, security boundaries, and troubleshooting for each platform: ** docs/PLUGINS.md**.

Platform Support

Hmm, wait — let me re-do that macOS row properly:

| macOS | 🚧 Code ready | Quartz CGEvent backend; awaiting real-hardware validation to become official |

Benchmark Results and Actual Findings

• Baseline v0.2: 7/10 tasks at 100% in a clean environment; the other 3 are honest model-capability boundary data points

• Full report: ** docs/baseline_v0.2.md**; every run trace is reproducible (winagent bench)

• Core conclusions (seven, all backed by controlled acac):

  1. Save-as style dialogs are immune to synthetic keyboard input → the interaction design takes a "no-dialog" route

  2. Visual coordinates have a 1~3 keV systematic bias → the closed loop "verify + tolerant retry" empirility proof of self-healing

  3. Typing is dispatched by character kind: alphanumeric Chinese goes via UNICODE (bypassing IME), while punctuation and operators go through virtual keys atomically

  4. Betting on a fixed sleep that the window is ready always loses → poll the active-window title (focus waiting wait)

  5. Win11 Notepad session restore resurrects old tabs → the eval setup clears the TabState

  6. Elevated-privilege windows are immune to non-elevated agents → the eval environment must be cleared

  7. Vision-model gradient: 4b is blind / 7b + tolerance is the sweet spot for price-performance / 27b is slow and biased (on GPU-less machines)

Performance and Configuration

• image_format: jpeg reduces the request body ~10x; capture_monitor avoids over-capturing screens; typing_interval_ms adjusts speed

• Keep the model resident using the server-side OLLAMA_KEEP_ALIVE (applies globally)

• Complete configuration keys are in config.example.yaml; invalid values are falling back as soon as loaded

Security Boundaries

• Only allow access：requests are LAN-only /local-machine (SSRF protection); subprocess allowlists only; file operations confined to the Temp directory

• Action tools operate the real keyboard/mouse — the host should authorize them before calling; doctor/discover/screenshot are entirely passive all the way

Project Status

• Version history and changes: ** CHANGELOG.md** (v0.1 → v0.4)

• Community files: contributing guide / Issue templates / PR templates / Code of Conduct / Security policy — health score 100

• Student work: from a computer without an NVIDIA card, an " unsure about the AI direction" question, to a 4-version, measurable Agent project

License

MIT