desktop-automation-mcp
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@desktop-automation-mcplocate and click the 'Send' button"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
🖱️ desktop-automation-mcp
Windows 桌面自动化 MCP 服务:截图 → 视觉定位 → 精确点击 一步闭环
让任何 MCP 客户端(Pi / Claude / Cursor...)通过 4 个工具控制 Windows 桌面: 精确点击任何坐标、全屏截图、视觉定位 UI 元素(三阶段校准,±1~5px)、定位+点击一步完成。
零依赖(仅 Node 18+ 内置 API + 可选 sharp),原生解决"注入点击被应用过滤、视觉模型定位不准"两大痛点。 视觉模型通过 config.json / 环境变量 配置,代码内不含任何密钥,可放心上传 GitHub。
✨ 4 个工具
工具 | 功能 |
| 鼠标单击指定物理像素坐标(高完整性 + 真人式按压) |
| 全屏截图,返回 PNG 路径 |
| 视觉定位目标元素,返回物理坐标(三阶段校准) |
| 定位 + 点击 一步完成 |
Related MCP server: computer-use-windows
🚀 安装
1. 克隆
git clone https://github.com/loser1727/desktop-automation-mcp
cd desktop-automation-mcp2. 编译点击工具(可选,仓库已带编译好的 exe)
cd tools
# Windows 自带 .NET Framework csc(或任意 C# 编译器)
C:\Windows\Microsoft.NET\Framework64\v4.0.30319\csc.exe /nologo /target:exe ^
/out:CursorClick.exe /win32manifest:CursorClick.manifest CursorClick.cs
CursorClick.exe是自提权程序(manifestrequireAdministrator)——点击时自动 UAC 提升 (需系统ConsentPromptBehaviorAdmin=0自动批准,或无干扰时手动点"是")。
3. 确保 sharp 可用(定位裁剪/放大需要)
sharp 一般已随 Pi/其他依赖就绪;若缺,在 server.js 所在目录执行:
npm install sharp4. 配置视觉模型(关键,见下节)
cp config.example.json config.json # 然后编辑填入你的 baseUrl / apiKey / model5. 注册到 MCP(Pi 的 ~/.pi/agent/mcp.json)
{
"mcpServers": {
"desktop-automation": {
"command": "node",
"args": ["C:\\path\\to\\desktop-automation-mcp\\server.js"]
}
}
}其他 MCP 客户端(Claude Desktop 等)同理放入其 mcp 配置。重启 Pi/客户端生效。
⚙️ 视觉模型配置(换模型超简单)
配置优先级:环境变量 > config.json > 内置默认值。config.json 位于 server.js 同目录,已被 .gitignore 忽略(不会泄露密钥)。
方式一:config.json(推荐)—— 支持多模型阶梯式自动切换
{
"models": [
{
"name": "智谱 GLM-4V-Flash (免费, 优先)",
"baseUrl": "https://open.bigmodel.cn/api/paas/v4",
"apiKey": "你的智谱密钥",
"model": "glm-4v-flash",
"disableThinking": false
},
{
"name": "火山方舟 Doubao (备用)",
"baseUrl": "https://ark.cn-beijing.volces.com/api/v3",
"apiKey": "ark-你的密钥",
"model": "doubao-seed-2-1-turbo-260628",
"disableThinking": true
}
],
"maxWidth": 1024,
"timeoutMs": 90000,
"retries": 2,
"maxTokens": 500,
"disableThinking": true,
"extraBody": {},
"fineCropWidth": 320,
"fineCropHeight": 240,
"fineZoom": 3,
"ultraCropWidth": 150,
"ultraCropHeight": 110,
"ultraZoom": 6,
"clickExe": ""
}阶梯式自动切换:models 数组按顺序尝试——第一个失败(网络/HTTP 错误/超时/空响应)自动切第二个,以此类推;全部失败才报错。启动日志会打印当前阶梯:视觉模型阶梯: A → B → C。
💡 建议把免费模型放最前面(如智谱 GLM-4V-Flash),付费/更强模型作后备——零成本优先,挂了再自动升级。
字段 | 说明 |
| 视觉模型阶梯数组(按顺序尝试)。每项可覆盖全局设置: |
| 任意 OpenAI 兼容 |
| 你的密钥(只放这里,绝不进代码) |
| 视觉模型 ID —— 换模型只改这一行 |
| 粗定位输入图最大宽度(默认 1024;越小越快,精度略降) |
|
|
| 附加请求参数, |
| 细/超细定位的裁剪尺寸与放大倍数 |
| 单次视觉请求超时、重试次数、最大输出 token |
| 自定义点击工具路径(默认 |
兼容:不写
models数组时,使用传统单模型字段baseUrl/apiKey/model(等价于一个元素的阶梯)。
方式二:环境变量
变量 | 说明 |
| 视觉 API 网关地址(兼容别名 |
| API 密钥(兼容别名 |
| 模型 ID(兼容别名 |
| 对应 config 字段 |
| 点击工具路径 |
兼容性说明:旧的
LITTLEAPI_*/ARK_*环境变量继续有效,升级无缝。
💡 使用示例
A. 定位一个按钮
desktop_locate(target="下载按钮")
→ {"x":1180,"y":341,"method":"ultra(粗→细→超细 6x)"}B. 定位并点击(一步)
desktop_click_target(target="开始游戏")
→ {"located":{"x":700,"y":400,"method":"ultra..."},"click":{"ok":true}}C. 先截图再人工检查
desktop_screenshot() → 返回路径 → 任何工具打开查看🔬 工作原理
定位:三阶段渐进放大 + 十字校准
视觉模型报"绝对像素坐标"不可靠(后端把图缩放进内部网格,无尺子,误差 100px+),但在局部放大图上定位非常准。所以:
① 粗定位:全图 → maxWidth(1024)px → 四角红十字校准 → 仿射变换 → ±50px
② 细定位:320×240 裁剪 3× 放大 → 局部十字校准 → ±10px
③ 超细定位:150×110 裁剪 6× 放大 → ±1~5px ✅实测误差(合成 UI):Cancel (341,306) vs (340,305) / Settings (661,181) vs (660,180) = 1px
点击:高完整性真人式点击
CursorClick.exe(自提权 manifest)
→ per-monitor DPI aware(物理像素坐标系,适配 125%/150% 缩放)
→ SetCursorPos 精确定位
→ SendInput DOWN → 保持 120ms(真实按压时长,防止极短按压被 UI 忽略)→ UP解决了两类常见问题:
UIPI 权限:普通权限注入点击被管理员级应用(如 Steam)静默丢弃 → 自提权运行
注入过滤:Steam 等应用会检测
LLMHF_INJECTED过滤合成输入 → 保持时长/多阶段重试提高命中
📁 文件结构
desktop-automation-mcp/
├── server.js ← MCP 服务本体(零依赖)
├── config.example.json ← 配置模板(含多模型阶梯示例,复制为 config.json 后填写)
├── .gitignore ← 忽略 config.json / 日志(防密钥泄露)
├── tools/
│ ├── CursorClick.cs ← 点击工具源码(C#,可编译)
│ ├── CursorClick.manifest ← 自提权清单(requireAdministrator)
│ └── CursorClick.exe ← 编译好的可执行文件
└── README.md⚠️ 注意事项
坐标 = 物理像素:截图尺寸与点击坐标同源,天然一致;DPI 缩放系统下也可直接使用
点击前确保目标窗口在前台(SendInput 投递到前台窗口)
desktop_click_target依赖视觉 API 可用;网络不佳时自动重试(可配置)未配置 API Key 时服务仍可启动(截图/点击可用),但定位会报错——请按上文配置
与 pi-vision-locate 扩展的配合
pi-vision-locate(Pi 扩展):聊天内直接识图/定位
本项目(MCP):程序化截图/定位/点击闭环
两者可独立使用,也可组合(扩展定位 → MCP 点击)。
📄 License
MIT
Available Tools
4 toolsdesktop_clickA
在屏幕指定物理像素坐标 (x,y) 处执行鼠标单击。高完整性进程注入,带真人式按压时长(默认120ms)。返回是否成功。
| Name | Required | Description | Default |
|---|---|---|---|
| x | Yes | 物理像素 X(截图坐标系) | |
| y | Yes | 物理像素 Y | |
| times | No | 点击次数(默认1,最多5) | |
| hold_ms | No | 按压保持毫秒数(默认120) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of behavioral disclosure. It discloses high-integrity process injection, human-like press duration (default 120ms), and states that it returns success/failure. This goes beyond a simple 'click' description and gives the agent useful behavioral traits.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences, front-loaded with the primary action and resource. Every clause adds value: the coordinate system, the injection method, the press behavior, and the return value. No wasted words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple click tool with 4 parameters and no output schema, the description covers the purpose, key behaviors, and return value. It lacks explicit edge-case handling (e.g., off-screen coordinates) but is otherwise complete enough for correct invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the baseline is 3. The description adds almost nothing beyond the schema: it repeats the physical pixel coordinate framing and the default hold duration, both already in the schema. No new parameter meaning is introduced.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb and resource: 'performs a mouse click at specified physical pixel coordinates'. It differentiates from the sibling desktop_click_target by explicitly saying the click is at raw physical pixel coordinates, not a located target.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies when to use the tool (when you have physical pixel coordinates to click) but does not explicitly mention when not to use it or point to alternatives like desktop_click_target or desktop_locate. The 'high integrity process injection' hint adds context but no clear routing guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
desktop_click_targetA
定位目标并点击一步完成:先视觉定位(多阶段精定位),再鼠标单击。返回定位坐标与点击结果。
| Name | Required | Description | Default |
|---|---|---|---|
| target | Yes | 要点击的目标描述 | |
| hold_ms | No | 按压时长(默认120) | |
| image_path | No | 截图路径(可选) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It discloses the order of operations (visual localization then click), mentions multi-stage precision, and states it returns coordinates and click result. However, it does not disclose potential failure modes (e.g., target not found), side effects (e.g., moving the cursor), or any access/permission requirements. This is a gap given no annotation support.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, front-loaded sentence that conveys the core purpose, the steps, and the return value without redundancy. It is appropriately sized and contains no wasted words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool that combines two actions (locate and click) with three parameters, no output schema, and no annotations, the description is minimal. It explains the process and return but lacks usage guidance relative to siblings, error handling, and behavior when the target is not found. This leaves the agent without enough context to fully anticipate the tool's behavior in edge cases.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, with each parameter (target, hold_ms, image_path) having a description. The tool description adds no additional parameter semantics beyond what the schema provides; it only mentions the overall return value. Therefore, the description does not compensate for any missing schema detail.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: '定位目标并点击一步完成' (locate target and click in one step), specifying that it first performs visual localization (multi-stage precise localization) then a mouse click, and returns coordinates and click result. This distinguishes it from siblings like desktop_click (click only) and desktop_locate (locate only).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies the tool performs both locate and click, but it does not explicitly state when to use this over the sibling tools. There is no mention that desktop_locate should be used if only coordinates are needed, or that desktop_click should be used if the target is already known. Guidance is only implied by the combined action.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
desktop_locateB
在截图中定位一个 UI 元素/文字/按钮,返回物理像素坐标 (x,y)。采用多阶段精定位:全图校准粗定位 → 局部放大精定位,误差约 ±5px。
| Name | Required | Description | Default |
|---|---|---|---|
| refine | No | 是否做精定位(默认 true) | |
| target | Yes | 要定位的目标描述,如 '开始游戏按钮'、'下载按钮'、'游戏版本(测试版)' | |
| image_path | No | 截图路径(可选,缺省自动截图) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description must carry the full behavioral burden. It discloses the multi-stage refinement process and ±5px accuracy, which is useful, but it does not explicitly state whether the tool is read-only (no state changes) or what happens on failure (e.g., element not found). The mention of auto-screenshot via optional image_path is implicit but not fully explicit.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, efficient sentence in Chinese that front-loads the core purpose (locate UI element, return coordinates) before diving into methodology. No wasted words, though it could be slightly more structured with a period after the main clause.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple locate tool with 100% parameter coverage and no output schema, the description covers the key aspects: what it does, method, accuracy. However, it omits failure behavior (e.g., if the target is not found) and does not clarify whether the tool performs any side effects (it likely doesn't, but this is not stated). The missing output schema shifts some burden to the description, but the core usage is adequately covered.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema covers all parameters with descriptions, so the baseline is 3. The tool description adds context about the refinement process that relates to the 'refine' parameter, but it does not add meaning beyond what the schema already states for target and image_path. The schema descriptions are sufficient.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action (locate), the resource (a UI element/text/button in a screenshot), and the output (physical pixel coordinates x,y). It is distinct from sibling tools like desktop_click (clicking) and desktop_screenshot (capturing), making its purpose unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus alternatives, nor does it mention any exclusions or prerequisites. An agent cannot infer when to prefer desktop_locate over desktop_click_target or other siblings based on the description alone.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
desktop_screenshotA
全屏截图并保存为 PNG 文件,返回文件路径。
| Name | Required | Description | Default |
|---|---|---|---|
| path | No | 保存路径(可选,默认临时目录) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must carry the full behavioral disclosure. It does state the primary behavior, file format, and return value. However, it does not disclose side effects such as overwriting existing files, whether the path is expected to be absolute or relative, or any permission/system requirements.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, compact sentence that front-loads the action, format, and return value. There is no filler or repetition of schema details.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple tool with one optional parameter and no output schema, the description adequately covers the key facts: what is captured, the saved format, and the returned value. It could add edge-case behavior such as error handling or path creation, but the core contract is complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, with the path parameter already documented as optional and defaulting to a temporary directory. The tool description adds the overall PNG-saving context but does not add parameter-specific meaning beyond what the schema already provides, so baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb, resource, and outcome: take a full-screen screenshot, save it as a PNG, and return the file path. This also distinguishes it clearly from the sibling tools (desktop_click, desktop_locate, desktop_click_target), which perform different actions entirely.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description makes the use case clear: capturing the full screen to a PNG file. It does not explicitly mention when not to use it or name an alternative, but the sibling tools are action-type tools, so there is little ambiguity about when this screenshot tool is appropriate.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
4 tool updates
v2.0.0- First observed
desktop_click - First observed
desktop_click_target - First observed
desktop_locate - First observed
desktop_screenshot
TDQS
Scored across 4 tools
每个工具都有明确且独立的用途:点击坐标、截图、定位元素、组合定位并点击。即使click和click_target有相似性,但一个是精确点击给定坐标,另一个自动定位后点击,描述清晰,不存在混淆。
所有工具统一使用desktop_前缀,后接明确的动词(click, screenshot, locate)或动词组合(click_target),命名模式完全一致,易于预测和记忆。
仅有4个工具,但每个工具都覆盖了桌面自动化核心操作:截图、定位、点击、点击目标,没有冗余或遗漏,规模小而精悍,符合最小可用原则。
工具集缺少键盘输入、双击、右键、拖动等常见桌面自动化操作,仅覆盖了基本的鼠标左键单击和截图定位,对于完整桌面自动化场景存在明显功能缺口。
Maintenance
Related MCP Connectors
MCP server for visual regression testing: triage a PR's UI diffs from your coding agent.
MCP server for progressive tool usage at any scale (see https://klavis.ai)
Personal assistant MCP server with search, execute, packages, jobs, secrets, and integrations.
MCP server for Qwen Image 3 AI image generation
Related MCP Servers
- AlicenseAqualityAmaintenanceWindows desktop automation MCP server — screenshot, mouse, keyboard & UI Automation. Lets LLM agents see and control your Windows desktop directly.30185 npm22MIT
- AlicenseNot gradedqualityDmaintenanceA standalone MCP server for Windows desktop control, enabling screenshots, mouse and keyboard input, app launch, window/display management, and clipboard access via natural language.2MIT
- AlicenseNot gradedqualityBmaintenanceGUI automation MCP server that enables AI agents to see and control the Windows desktop using a local Vision LLM (Ollama), supporting screenshot analysis, mouse/keyboard actions, and autonomous task execution.4MIT
- AlicenseNot gradedqualityDmaintenanceVision-based desktop automation MCP server that controls any application via screenshot and AI vision, enabling UI automation through natural language commands.12 npm5Business Source 1.1