win-dialog-agent

EN: Semantic Windows dialog automation for AI agents — OpenAI Agents SDK function tools + MCP stdio server.
中文： 面向 AI Agent 的语义级 Windows 对话框操作模块（Agents SDK 工具 + MCP stdio 服务）。

Agent says “save to D:\out.txt” — not click (1024, 768).
Agent 说的是意图，不是坐标。

Features / 功能

• P0 semantic tools: wait / list / describe / read / click / set text / Save As / Open / MessageBox
  语义工具：等待、列举、描述、读文本、点按钮、填文本、另存为、打开、消息框

• Bilingual buttons: 保存 ≈ Save(&S) via community-editable aliases.py
  中英文按钮别名，欢迎 PR 扩充

• P1 UIA fallback for non-standard dialogs (with warning logs)
  非标准对话框走通用 UIA，并打告警日志

• Dual expose: @function_tool (Agents SDK) + MCP stdio (win-dialog-mcp)
  一份实现，两种暴露

Related MCP server: Windows-MCP

Install / 安装

# from this directory / 在本目录
pip install -e ".[dev]"

# or with uv
uv pip install -e ".[dev]"

Copy env template:

copy .env.example .env

Quick start — Agents SDK / 快速开始

from agents import Agent, Runner
from win_dialog_agent.tools import ALL_TOOLS

agent = Agent(
    name="DialogOperator",
    instructions=(
        "You operate Windows dialogs via tools. "
        "Prefer dialog_handle_save_as / dialog_handle_open_file / "
        "dialog_handle_messagebox for standard #32770 dialogs."
    ),
    tools=ALL_TOOLS,
)

# async example
# result = await Runner.run(agent, "Save the current Save As dialog to D:\\out.txt")

Core library API (no LLM required):

from win_dialog_agent import dialogs

info = dialogs.dialog_wait_for(timeout=10)
dialogs.dialog_handle_save_as(info.handle_id, r"D:\out.txt")

MCP server / MCP 服务

Run locally

win-dialog-mcp
# or
python -m win_dialog_agent.server

MCP client config (Cursor / Claude Desktop)

{
  "mcpServers": {
    "win-dialog-agent": {
      "command": "uvx",
      "args": [
        "--from",
        "git+https://github.com/tigersaint88-app/win-dialog-agent",
        "win-dialog-mcp"
      ]
    }
  }
}

After PyPI publish you can simplify to uvx win-dialog-agent.

Cursor Marketplace / Agent Plugin

This repo includes:

• .cursor-plugin/plugin.json — Cursor Plugin manifest

• plugin.json — Agent Plugins (open standard) manifest

• mcp.json — MCP server entry for marketplace install

• skills/windows-dialog-ops/ — agent skill for dialog workflows

Submit / update listing: cursor.com/marketplace/publish
Community directory: cursor.directory

Local plugin test (optional): copy this repo into ~/.cursor/plugins/local/win-dialog-agent, then Reload Window.

Inspector self-test

npx @modelcontextprotocol/inspector win-dialog-mcp

Call dialog_list from the Inspector Tools tab.

Catalog targets

Suitable for listing on mcp.so, Smithery, Glama, and Windows 11 native MCP registry (follow Microsoft registration docs).

Tool reference / 工具一览

Discovery (P0)

Semantic actions (P0)

UIA fallback (P1)

Architecture / 架构

dialog_wait_for / dialog_list
        │
        ▼
 class == #32770 ──yes──▶ semantic fast path (dialogs.py)
        │
        no
        ▼
 UIA fallback (uia_fallback.py) + warning log

• Find-window and do-action are separated for mockable unit tests.
  找窗口与做动作分离，便于单测 mock。

• Timeouts / retries come from env (WIN_DIALOG_*), never hard-coded at call sites.
  超时与重试全部走配置。

• Logs store text hashes / digests, not full dialog bodies or file contents.
  日志只记摘要，不落用户正文。

Safety notes / 安全与限制

1. Elevation: If the target dialog runs as Administrator, this process must too. Tools return a clear error instead of failing silently.
   目标以管理员运行时，本进程也需管理员；错误信息明确，不静默失败。

2. UAC secure desktop: Not reachable via UIA — returns unsupported.
   安全桌面上的对话框不可自动化。

3. 32/64-bit: Match Python bitness to the target process when possible.
   尽量与目标进程位数一致。

4. Password fields: dialog_set_text refuses password controls.
   密码框拒绝写入。

5. P0 has no coordinate clicks: dialogs.py must not contain click_input / move_mouse.
   P0 层禁止坐标点击。

Development / 开发

pip install -e ".[dev]"
pytest
pytest -m integration   # real desktop scenarios
ruff check src tests

Acceptance scenarios (manual) / 验收场景

1. Notepad → Ctrl+S → dialog_wait_for → dialog_handle_save_as → file on disk

2. MessageBox → dialog_read_text → dialog_handle_messagebox("yes")

3. Open dialog → dialog_handle_open_file

4. Repeat on Chinese Windows (aliases 保存 / 确定)

5. Custom WPF dialog → uia_* + warning log

6. MCP Inspector lists tools and calls dialog_list

7. rg "click_input|move_mouse" src/win_dialog_agent/dialogs.py → zero hits

Contributing aliases / 贡献别名

The highest-leverage contribution is expanding button labels in
src/win_dialog_agent/aliases.py.
See CONTRIBUTING.md.

License

MIT — see LICENSE.