Skip to main content
Glama

desktop-touch-mcp

MCP server

Windows 10/11 专用 Computer-use MCP 服务器 — 截图、UI Automation、Chrome CDP、键鼠输入、终端 + 可靠后台 shell_session,原生 Rust 引擎 + PowerShell 回退。


快速开始

方式 A:下载 zip(推荐)

  1. 从 Releases 下载 zip

  2. 解压到任意文件夹

  3. 双击 start.bat — 完事

start.bat 首次运行自动安装依赖,之后直接启动。无需 npm/npx/构建工具。

方式 B:npx

npx -y @harusame64/desktop-touch-mcp

HTTP 模式

双击 start.bat 时加参数即可开启 HTTP 模式:

start.bat --http --port 23847 --key YOUR_KEY

注册到 Claude CLI

在 ~/.claude.json 的 mcpServers 中添加:

{
  "desktop-touch": {
    "type": "stdio",
    "command": "node",
    "args": ["C:/Tools/desktop-touch-mcp/dist/index.js"]
  }
}

默认绑定 0.0.0.0。绑定到非 localhost 地址时 API 密钥为必填,客户端须带 Authorization: Bearer <KEY> 请求头。健康检查:http://<地址>:<端口>/health。


Related MCP server: Glasswarp

核心理念

  1. 发现-操作 — desktop_discover 返回带租约的可交互实体(非原始坐标),desktop_act 操作你意图的目标而非它曾经的位置

  2. 感知防护 — 每次操作前自动验证目标窗口身份和边界,防止误窗口输入

  3. Rust 原生加速 — UIA 焦点查询 2ms(160× 加速),图像差分 SSE2 SIMD 13~15× 加速;引擎不可用时透明回退 PowerShell


环境要求

要求

版本

操作系统

Windows 10/11(64 位)

Node.js

v20+(推荐 v22+)

PowerShell

5.1+(仅作 Rust 引擎回退)

VC++ 运行时

下载(nut-js 需要)


工具一览(32+ 个)

分类

工具

说明

🌐 主体路径

desktop_discover

观察桌面,返回带租约的可交互实体

desktop_act

通过租约验证对实体操作(点击/输入/拖拽/选择)

👁️ 观察

desktop_state

轻量焦点/窗口/光标/注意信号检查

screenshot

多模式捕获(text/diff/dotByDot/background)

screenshot_query / screenshot_gc

截图缓存查询与清理

workspace_snapshot

所有窗口缩略图 + UI 摘要

server_status

原生引擎健康诊断

⌨️ 输入

keyboard

键盘输入,支持 IME 旁路

mouse_click / mouse_drag

坐标交互 + 归位 + 强制焦点

scroll

滚轮 / 定位元素 / 智能列表 / 拼接

click_element

UIA 按名称/ID 点击(回退)

🌐 浏览器 CDP

browser_open / browser_navigate

幂等调试启动 + 导航

browser_click / browser_fill

跨重绘稳定的 DOM 交互

browser_eval

JS 执行 / DOM 提取 / SPA 状态

🛠️ 工作流

shell_session

可靠后台命令通道(进程 stdin/stdout + exit code)

terminal

驱动可见控制台窗口(run/send/read)

wait_until

窗口/焦点/文本/URL 状态轮询

window_dock / focus_window

窗口吸附/置顶/聚焦

workspace_launch

启动应用 + 自动检测新窗口

run_macro

最多 50 个操作批量执行

clipboard / notification_show

剪贴板 + 通知

📊 Office

excel

VBA 宏写入与运行


标准工作流

desktop_state          → 定向:焦点窗口/元素、模态、注意信号
desktop_discover       → 查找可操作实体(返回 lease + windows[])
desktop_act(lease, …)  → 操作实体(返回 attention + post.perception)
desktop_state          → 确认世界按预期变化

命令执行:选对通道(重要)

场景

用哪个

原因

装软件、跑脚本、文件操作、包管理

shell_session

真实 OS 进程 + stdin/stdout + exit code;不依赖焦点/IME/剪贴板/SoM 日志

用户必须看见某个控制台窗口

terminal

驱动可见窗口(UIA/WM_CHAR),适合交互演示

# 一次性命令(推荐)
shell_session({ action: "exec", input: "whoami" })
# → ok:true 仅当 exitCode===0;非零退出返回 ok:false + stdout/stderr

# 交互式多步
shell_session({ action: "start", id: "ops", shell: "powershell" })
shell_session({ action: "write", id: "ops", input: "Get-ChildItem", waitMs: 500 })
shell_session({ action: "read",  id: "ops", clear: true })
shell_session({ action: "stop",  id: "ops" })

禁止假成功: write 只有 stdin 被 OS 接受才 ok:true;exec 只有 exitCode===0 才 ok:true。

点击优先级

  1. browser_click(selector) — Chrome/Edge CDP(跨重绘稳定)

  2. desktop_act(lease) — 原生/对话框/视觉(基于实体)

  3. click_element(name | automationId) — UIA 回退

  4. mouse_click(x, y, origin?, scale?) — 像素级最后手段

恢复提示

信号

处理

lease_expired / *_mismatch / entity_not_found

重新 desktop_discover

modal_blocking

click_element(name=blockingElement.name) 关闭后重试

entity_outside_viewport

scroll(to_element) 后重调 desktop_discover

executor_failed

回退到 click_element / mouse_click / browser_click

租约 TTL 自适应(上限 60s),softExpiresAtMs 约 60% 处 LLM 应考虑刷新。


截图参数速查

参数

效果

Token

detail="image"

PNG/WebP 像素(默认)

~443

detail="text"

UIA 元素 JSON + clickAt 坐标

~100-300

detail="meta"

仅标题+区域

~20/窗口

dotByDot=true

1:1 WebP,图像像素=屏幕坐标

~800

dotByDotMaxDimension=N

限制最长边,响应含 scale

—

grayscale=true

灰度,文本类减小约 50%

—

region={x,y,w,h}

窗口局部裁剪

—

diffMode=true

仅变化窗口(P帧)

~160

ocrFallback="auto"

UIA 稀疏时自动触发 Windows OCR

—

推荐流程:

workspace_snapshot()                     → 全面定位
screenshot(detail="text", windowTitle=X) → 获取 clickAt 坐标
mouse_click(x, y)                        → 直接点击
screenshot(diffMode=true)                → 仅检查变化

浏览器 CDP 自动化

无需 Selenium/Playwright,只需启用 Chrome 远程调试端口:

chrome.exe --remote-debugging-port=9222 --user-data-dir=C:\tmp\cdp
browser_open({launch:{}})                          → 启动 CDP Chrome + 列出标签
browser_click({selector:"#submit"})                → 查找+点击一步完成
browser_eval({action:"js", expression:"..."})      → 执行 JS
browser_fill({selector:"#email", value:"..."})     → 填充受控输入(React/Vue/Svelte 安全)
browser_navigate({url:"https://example.com"})      → CDP 导航

browser_locate 返回的坐标已含浏览器 UI 偏移和 DPI 缩放,可直接传给 mouse_click。


终端命令完成判定

terminal(action='run') 通过 until 参数控制"命令是否完成":

模式

等待内容

适用场景

quiet(默认)

输出安静持续 quietMs

短命令

pattern

输出匹配预期的字符串/正则

有已知结束标记的长命令

exit

命令真正结束 + 返回退出码

需要完成码时

exit 模式注入与回显不同的完成标记,彻底解决哨兵误匹配问题:

terminal({
  action: 'run', windowTitle: 'pwsh',
  input: 'npm run build',
  until: { mode: 'exit', shell: 'powershell' },
})
// → { reason: 'exited', exitCode: 0 }

支持的 shell:bash、powershell。cmd.exe 尚不支持。不安全输入(未关闭引号等)直接拒绝。


鼠标归位校正

截图获取坐标后窗口可能已移动,归位系统自动校正:

层级

启用方式

功能

1

始终可用

(dx, dy) 偏移修正

2

传 windowTitle

窗口被遮挡时自动前置

3

传 elementName + windowTitle

UIA 重查缩放后的新坐标

mouse_click(x=500, y=300)                                      # 层级 1
mouse_click(x=500, y=300, windowTitle="记事本")                  # 层级 1+2
mouse_click(x=500, y=300, windowTitle="记事本", elementName="保存")  # 层级 1+2+3
mouse_click(x=500, y=300, homing=false)                         # 关闭归位

origin + scale 坐标换算

dotByDot 截图带 dotByDotMaxDimension 时响应含 origin 和 scale,直接传入即可自动换算:

mouse_click(x=640, y=300, origin={x:0, y:120}, scale=0.6667, windowTitle="Chrome")
# 服务器换算: screen = (0 + 640/0.6667, 120 + 300/0.6667) = (960, 570)

自动防护

操作工具传入 windowTitle 时自动防护:

  • ✅ 验证目标窗口身份(检测进程重启/HWND 替换)

  • ✅ 确认点击坐标在窗口矩形内

  • ✅ 失败时返回 post.perception.status,LLM 可无截图恢复

状态

含义

ok

防护通过

unguarded

未提供 windowTitle

target_not_found

无匹配窗口

identity_changed

窗口已被替换

unsafe_coordinates

坐标在窗口矩形外

needs_escalation

需用 browser_click 或指定 windowTitle

unsafe_coordinates 或 identity_changed 时可传 fixId 批准一次性恢复(15 秒过期)。

设置 DESKTOP_TOUCH_AUTO_GUARD=0 可禁用自动防护。


强制焦点

Windows 前台保护可能阻止按键到达目标窗口。mouse_click、keyboard、terminal(send) 均支持 forceFocus: true,通过 AttachThreadInput 绕过:

{ "name": "mouse_click", "arguments": { "x": 500, "y": 300, "windowTitle": "Chrome", "forceFocus": true } }

全局默认:设 DESKTOP_TOUCH_FORCE_FOCUS=1。被拒绝时返回 ok:false + code: "ForegroundRestricted",操作被抑制不误投。


自动停靠 CLI

MCP 启动时自动停靠承载 Claude 的终端:

{
  "mcpServers": {
    "desktop-touch": {
      "env": {
        "DESKTOP_TOUCH_DOCK_TITLE": "@parent",
        "DESKTOP_TOUCH_DOCK_CORNER": "bottom-right",
        "DESKTOP_TOUCH_DOCK_WIDTH": "480",
        "DESKTOP_TOUCH_DOCK_HEIGHT": "360"
      }
    }
  }
}

环境变量

默认值

说明

DOCK_TITLE

—

@parent 沿进程树查找终端,或用字面子串

DOCK_CORNER

bottom-right

四角可选

DOCK_WIDTH/HEIGHT

480/360

px 或比例(如 "25%")

DOCK_PIN

true

置顶

DOCK_MONITOR

主显示器

显示器 ID

DOCK_MARGIN

8

屏幕边缘填充(px)

⚠️ 置顶窗口活动时按键会发到它而非目标。键盘操作前先 focus_window(title=...)。


截图缓存

screenshot 返回廉价的 screenshot://by-ref/{id} 链接而非内联像素,减少 token 消耗。

环境变量

默认值

说明

SCREENSHOTS_DIR

用户缓存目录

固定缓存路径

SCREENSHOT_MAX_COUNT

200

缓存上限

SCREENSHOT_MAX_BYTES

256 MiB

磁盘上限

SCREENSHOT_MAX_AGE_MS

—

超龄丢弃(opt-in)

SCREENSHOT_AUTOPRUNE

on

新增时自动清理,0 禁用


自动感知(Always-on)

每个 desktop_state 和 desktop_act 响应自动附加 attention 信号,操作工具传 windowTitle 时自动防护。lensId 参数在操作工具上保留供高级固定目标使用。


安全机制

紧急停止(唯一安全机制)

将鼠标移至屏幕左上角(0,0 附近 10px 以内)即立即终止 MCP 服务器。

  • 每次工具调用前检查 checkFailsafe()

  • 500ms 后台轮询作为长操作后备

  • 触发半径:10px

所有按键组合和应用启动均不受限制。键盘黑名单和应用黑名单已移除。


鼠标移动速度

mouse_click、mouse_drag、scroll 均支持 speed 参数:

值

行为

省略

默认 1500 px/秒

0

瞬移,无动画

1~N

N px/秒动画

全局设置:DESKTOP_TOUCH_MOUSE_SPEED=3000。常用:0=瞬移,1500=柔和,3000=快速,5000=极速。


性能(Rust 原生引擎)

UIA 基准

函数

Rust 原生

PowerShell

加速比

getFocusedElement

2.2 ms

366 ms

163.9×

getUiElements(~60 元素)

106.5 ms

346 ms

3.3×

加权平均

~82×

图像差分(SSE2 SIMD)

函数

Rust

TypeScript

加速比

computeChangeFraction

0.26 ms

3.8 ms

~15×

dHash

0.09 ms

1.2 ms

~13×

架构

MCP 客户端
    │  stdio / HTTP
    ▼
TypeScript 服务层
    ├── Rust 原生引擎(.node 插件)
    │   ├── UIA: napi-rs + windows-rs,MTA COM 线程
    │   └── 图像: SSE2 SIMD 差分 + 感知哈希
    └── PowerShell 回退(引擎不可用时自动激活)

UI 操作层 V2

desktop_discover / desktop_act 默认开启。设 DESKTOP_TOUCH_DISABLE_FUKUWARAI_V2=1 可禁用回退到 V1。


已知限制

限制

变通方法

游戏/DirectX 全屏截图可能黑屏

用 screenshot(mode:'background') 或 BitBlt 回退

Chrome/WinUI3 UIA 元素为空

自动 OCR 回退,或用 browser_open + CDP

browser_* 需 Chrome 以 --remote-debugging-port 启动

先关 Chrome,用 browser_open({launch:{}})

diff 缓冲区 90s 不活动后清除

长等待后调 workspace_snapshot 重置

置顶窗口活动时键盘发到错误窗口

先 focus_window,验证 isActive=true

Chrome 中长破折号/智能引号被拦截

用 use_clipboard=true

React/Vue/Svelte 受控输入

用 browser_fill(原生 setter + InputEvent)


Token 成本参考

模式

Token

用途

screenshot(768px)

~443

一般视觉

screenshot(dotByDot)

~800

精确点击

screenshot(diffMode)

~160

操作后差异

screenshot(detail="text")

~100-300

UI 交互(无图像)

workspace_snapshot

~2000

全会话概览


环境变量汇总

变量

默认值

说明

DESKTOP_TOUCH_API_KEY

—

HTTP 模式 API 密钥,绑定非 localhost 时必填

DESKTOP_TOUCH_HOST

0.0.0.0

HTTP 监听地址

DESKTOP_TOUCH_MOUSE_SPEED

1500

鼠标移动速度(px/秒)

DESKTOP_TOUCH_FORCE_FOCUS

—

设 1 全局强制焦点

DESKTOP_TOUCH_AUTO_GUARD

on

设 0 禁用自动防护

DESKTOP_TOUCH_DOCK_TITLE

—

自动停靠标题(@parent 查终端)

DESKTOP_TOUCH_DOCK_CORNER

bottom-right

停靠角落

DESKTOP_TOUCH_DOCK_WIDTH/HEIGHT

480/360

停靠尺寸

DESKTOP_TOUCH_DOCK_PIN

true

停靠置顶

DESKTOP_TOUCH_DOCK_MARGIN

8

停靠边距

DESKTOP_TOUCH_DISABLE_FUKUWARAI_V2

—

设 1 禁用 V2 工具

DESKTOP_TOUCH_MCP_HOME

—

npx 缓存根目录

DESKTOP_TOUCH_SCREENSHOTS_DIR

用户缓存

截图缓存路径

DESKTOP_TOUCH_SCREENSHOT_MAX_COUNT

200

缓存数量上限

DESKTOP_TOUCH_SCREENSHOT_MAX_BYTES

256 MiB

缓存大小上限


许可证

MIT

Available Tools

30 tools
browser_clickA

Click a DOM element in Chrome/Edge. Two ways to target: (1) selector — a CSS selector (combines browser_locate + mouse_click; stable across repaints); or (2) by-axis (semantic) — by:'text'|'regex'|'role'|'ariaLabel' + pattern, so you do not have to build a CSS selector for dynamic-class SPAs. by-axis resolves to a SINGLE actionable element (climbing to a clickable ancestor up to 3 levels, hit-testing for occlusion) and STOPS with code:'BrowserAmbiguousTarget' (candidates[] + next[] hints) when 2+ actionable elements match, or code:'BrowserNoActionableTarget' when matches exist but none is clickable — it never guesses. If the target is behind a modal dialog blocking the page, BOTH targeting modes STOP with code:'BrowserModalBlocking' (context.blockingElement {name, role}) instead of clicking through to the backdrop — dismiss the dialog (its close button or Escape) and retry; a plain navigation drawer does not count as blocking. Optionally add role to filter (by:'text',pattern:'Save',role:'button') and scope to narrow the search. Provide EITHER selector OR by+pattern (not both). Pass tabId+port so the server auto-guards (verifies tab readyState and identity) and returns post.perception.status. lensId is optional for advanced pinned-tab workflows. Caveats: selector mode fails if the element is outside the visible viewport — scroll it into view with browser_eval("document.querySelector('sel').scrollIntoView()") first (by-axis only resolves in-viewport actionable targets). hints.verifyDelivery:{status:'delivered'|'unverifiable', reason, observedSignals:{mutationCount,urlChanged,activeElementChanged}} reports the post-click observation in 2 values: 'delivered' fires only when mutationCount>0 OR urlChanged (activeElementChanged is recorded in observedSignals but intentionally NOT a delivery signal — plain clicks on focusable controls always update focus, treating that as 'delivered' would mask silent-fail regressions); 'unverifiable' reason ∈ {'iframe_context_mismatch','no_dom_mutation','probe_install_failed','probe_read_failed'}. CDP emits 2 values only (focus_only is a UIA-path concept, N/A here). BrowserClickNotDelivered is reserved-only (false-positive risk too high to emit) — degradation reads from 'unverifiable' status.

Inputs: by, port, role, fixId, +9 more.

browser_evalA

Purpose: Inspect or operate on a browser tab via 3 actions: 'js' (evaluate JS), 'dom' (get HTML), 'appState' (extract SSR-injected SPA state). Details: action='js' — Run a JS expression. withPerception:true wraps in {ok, result, post}. action='dom' — Return outerHTML of selector (or document.body), truncated to maxLength. action='appState' — Scan Next/Nuxt/Remix/Apollo/GitHub/Redux SSR injected JSON; pass selectors to override defaults. Prefer: Use action='appState' BEFORE 'dom' or 'js' on SPAs where rendered HTML is sparse — single CDP call. Use 'dom' when 'appState' is empty and you need page structure. Use 'js' as the escape hatch for arbitrary scripting. Caveats: DOM nodes cannot be returned from action='js' directly (circular refs are serialized safely). React/Vue/Svelte controlled inputs cannot be set via element.value — use keyboard(action='type') / browser_fill instead. readyState is strictly checked; guard blocks if page is still loading. Typed errors: code:'BrowserNotConnected' on CDP disconnect (re-attach via browser_open); code:'AutoGuardBlocked' when the auto-guard refuses (e.g. page still loading) — the error message preserves the guard's 1-sentence recommended next step (most often wait_until({condition:'ready_state'}) or browser_eval readyState polling, then retry). Examples: browser_eval({action:'js', expression:'document.title'}) → page title browser_eval({action:'dom', selector:'#main', maxLength:5000}) → outerHTML browser_eval({action:'appState'}) → default SPA state probes

Inputs: action (required), port, tabId, lensId, +8 more.

browser_fillA

Fill a form input with a value via CDP — works on React/Vue/Svelte controlled inputs that reject browser_eval value assignment. Two ways to target: (1) selector — a CSS selector (use browser_overview / browser_locate to find one); or (2) by-axis (semantic) — by:'text'|'regex'|'role'|'ariaLabel' + pattern (e.g. by:'ariaLabel', pattern:'Email address', or by:'role', pattern:'textbox'), so you do not have to build a CSS selector. by-axis resolves to a SINGLE fillable element and STOPS with code:'BrowserAmbiguousTarget' (candidates[] + next[] hints) when 2+ match, or code:'BrowserNoActionableTarget' when the match is not a fillable input/textarea/contenteditable — it never guesses. Optionally add role to filter and scope to narrow. Provide EITHER selector OR by+pattern (not both). Use this over browser_eval when setting a controlled input's value via JS does not update framework state. Caveats: Requires browser_open (CDP active). actual in the response shows the element's value after fill; verify it matches the intended value. Typed errors: code:'BrowserFillNotDelivered' on post-fill value mismatch — note the false-positive case where a React controlled input's onChange transforms the value (delivery actually succeeded; hints.verifyDelivery.subReason:'controlled_input_transform' for that case; the actual value is authoritative).

Inputs: value (required), by, port, role, +7 more.

browser_formA

Inspect all form fields (input, select, textarea, button) within a CSS-selector-specified container and return their name, type, id, current value, hint text, disabled/readOnly state, and associated label text (resolved via for[id], ancestor LABEL, aria-labelledby, aria-label in that order). Use this before browser_fill to discover exact field selectors and avoid accidentally targeting the wrong input (e.g. a global search bar). Caveats: Requires browser_open (CDP active). Hidden inputs (type=hidden) are excluded by default — set includeHidden:true if needed. Value text is truncated at 200 chars.

Inputs: selector (required), port, tabId, include, +3 more.

browser_locateA

Find a DOM element by CSS selector and return its physical screen coordinates — compatible directly with mouse_click. Prefer browser_click to find+click in one step. Prefer browser_overview to discover selectors. Caveats: Coordinates are captured at call time; if the page reflows before mouse_click, coords may be stale. Typed errors: code:'BrowserNotConnected' (call browser_open first), code:'ElementNotFound' (selector did not match — re-discover via browser_overview / browser_search).

Inputs: selector (required), port, tabId, include, +1 more.

browser_navigateA

Navigate a browser tab to a URL via CDP Page.navigate — more reliable than clicking the address bar. Pass tabId+port so the server auto-guards (verifies tab readyState) and returns post.perception.status. lensId is optional for advanced pinned-tab workflows. Caveats: Does not block until page load completes — the Page.navigate ack confirms only that the navigation request was accepted (frameStoppedLoading / loaderId observation is internal). Follow with wait_until({condition:'ready_state' or 'element_matches'}) or repeated browser_eval polling for slow pages. Typed errors: code:'NavigateFailed' (Page.navigate rejected — DNS failure, malformed URL, network unreachable; check URL + connectivity), code:'BrowserNotConnected' (CDP disconnect — re-attach via browser_open), code:'AutoGuardBlocked' when the auto-guard refuses (e.g. tab still loading) — the error message preserves the guard's 1-sentence recommended next step (most often wait_until({condition:'ready_state'}) then retry).

Inputs: url (required), port, tabId, lensId, +4 more.

browser_openA

Connect to Chrome/Edge running with --remote-debugging-port and return open tab IDs — required before all other browser_* tools. Pass launch:{} (or with overrides) to auto-spawn a debug-mode browser when no CDP endpoint is live (idempotent: an already-running endpoint is preferred). Returns tabs[] with id, url, title, active — pass tabId to browser_* tools to target a specific tab. Caveats: CDP connection is per-process; if Chrome restarts, call browser_open again to get fresh tab IDs. A Chrome session started without --remote-debugging-port cannot be taken over — close it first or use a separate userDataDir. If the CDP endpoint is unreachable and launch is omitted, returns ok:false (typically code:'BrowserNotConnected' when the fetch surfaces ECONNREFUSED, otherwise code:'ToolError' with error 'Cannot reach Chrome/Edge CDP...'); re-call with launch:{} (idempotent) to auto-spawn or start Chrome manually with --remote-debugging-port=9222.

Inputs: port, launch, include.

browser_overviewA

List all interactive elements (links, buttons, inputs, ARIA controls) on the current page with CSS selectors, visible text or value for inputs, and viewport status — use before browser_click to discover stable selectors, and prefer this over screenshot when verifying button/toggle state after submission (no image tokens, structured output). scope limits to a CSS subsection (e.g. '.sidebar'). Returns state (checked/pressed/selected/expanded) for ARIA custom controls. Also returns a modal: section — whether a true modal dialog is blocking the page (isModal + blocker {name, role} + the signals it was judged on); it is ALWAYS present (isModal:false when no modal), and a navigation drawer is NOT reported as a modal (only an aria-modal / alertdialog / native showModal dialog, or a backdrop-backed dialog that locks the page, is treated as modal). Caveats: Selectors are CDP-generated snapshots — re-call after page navigates or re-renders. Input text reflects the empty-field hint text when defined (takes priority over typed value) — use browser_eval('document.querySelector(sel).value') to read actual typed content. Typed errors: code:'BrowserNotConnected' (CDP not attached — call browser_open or browser_open({launch:{}})). Note: a non-matching scope CSS selector silently falls back to the full document (does not raise an error) — verify the selector via browser_eval if scoped enumeration is required.

Inputs: port, scope, tabId, types, +4 more.

click_elementA

Invoke a UI element by name or automationId via UIA InvokePattern — no screen coordinates needed. The server auto-guards using windowTitle (verifies identity, foreground, modal) and returns post.perception.status. Prefer over mouse_click for buttons, menu items, and links in native Windows apps. Use desktop_discover first to discover automationIds. Pass fixId from a suggestedFix to re-target after window identity drift. lensId is optional for advanced pinned-lens use. Caveats: Typed errors: code:'InvokePatternNotSupported' — the control does not expose InvokePattern, fall back to mouse_click; code:'ElementDisabled' — the element is in a disabled state, re-check preconditions before retry; code:'GuardFailed' — read the perception envelope (attention / guard fields) and choose recovery (re-focus, wait, or pass the suggestedFix.fixId on the next call). Some custom controls do not expose InvokePattern at all; fall back to mouse_click for those.

Inputs: windowTitle (required), hwnd, name, fixId, +5 more.

clipboardA

Read or write the Windows clipboard. action='read' returns current text content (empty string if non-text). action='write' replaces clipboard with given text and verifies delivery via Get-Clipboard -Raw read-back, comparing the bytes (UTF-16LE) for exact equality. Caveats: Non-text clipboard payloads (images, files) return empty string on read. Overwrites existing clipboard content on write. action='write' delivery-verification failure returns code:'ClipboardWriteNotDelivered' — typical causes: a third-party clipboard manager intercepts SetClipboardData, DLP / endpoint protection blocks the payload, RDP / Citrix clipboard transcoding strips the text, or another process clears the clipboard between Set and the read-back. Recovery: retry the write, or fall back to keyboard(action='type', use_clipboard=false) for short text. Examples: clipboard({action:'write', text:'hello'}) → write+verify; clipboard({action:'read'}) → returns current text.

Inputs: action (required), text, include.

desktop_stateA

Purpose: Read-only observation of the current desktop state. Returns focused window/element, modal flag, attention signal from Auto Perception. Phase 4 absorbs former get_active_window / get_cursor_position / get_screen_info / get_document_state via include* flags. Details: Always returns: focusedWindow (title, hwnd, processName), focusedElement (name, type, value, automationId), cursorPos {x,y}, cursorOverElement (name, type), cursorOverWindow, hasModal (boolean), pageState ('ready'|'loading'|'dialog'), attention, visibleWindows count. Optional fields (default off): includeCursor:true → cursor {x,y,monitorId} (richer than cursorPos). includeScreen:true → screen {virtualScreen, displays[], displayCount, primaryIndex}. includeDocument:true → document {url, title, readyState, selection, scroll, viewport} via CDP (silently omitted on non-Chromium foreground). includeSessionContext:true (or include:['sessionContext']) → sessionContext {origin, consoleSessionId, sessionLabel, sessionState, ownWinStation} for Terminal Services session classification (ADR-017, observability-only). Chromium: cursorOverElement is null (UIA sparse); focusedElement may fall back to CDP document.activeElement; hints.focusedElementSource reports which path produced the row ('view' = engine-perception latest_focus, 'uia' = direct UIA query, 'cdp' = document.activeElement). Does NOT enumerate descendants — use desktop_discover for actionable entity list and window list. Prefer: Use after each action to confirm state. Cheapest observation tool — cheaper than any screenshot. attention='ok' means safe to proceed; other values require recovery (see suggest[]). Set include* flags only when you need the extra data (each adds one syscall or CDP round-trip). Caveats: Cannot detect non-UIA elements (custom-drawn UIs, game overlays). hasModal only detects modal dialogs exposed via UIA — browser alert/confirm dialogs may not appear here. includeDocument requires browser_open (CDP active); silently omitted otherwise with hints.documentUnavailable.

Inputs: port, tabId, include, includeCursor, +3 more.

excelA

Purpose: Author and run VBA macros against Excel via COM late binding (ADR-015). Headline differentiator against Claude for Excel which writes formulas but cannot run VBA. Details: action='run_vba' authors a Sub in a fresh workbook, saves into the managed Trusted Location (%LOCALAPPDATA%\desktop-touch-mcp\trusted-vba), and Application.Run the macro. Requires HKCU AccessVBOM=1 + VBAWarnings=1 + a registered Trusted Location (all configured by node scripts/enable-access-vbom.mjs). Trust setup: Excel must restart after the CLI runs (values cached at process start). action='check_access_vbom' is a read-only preflight returning {trusted, lockedByPolicy, scope}. Prefer: Run check_access_vbom first when a workflow depends on macro execution; the remediation hint pre-empts an opaque HRESULT 0x800a03ec failure inside run_vba. Caveats: macroName MUST appear as Sub <name>(...) in code (else VbaMacroNotFound). VBA Editor UI is structurally bypassed — no UIA tree walk needed. Excel COM is STA: each call serialises through the bridge's worker thread, so long-running macros block subsequent excel() calls on the same MCP server. Examples: excel({action:'check_access_vbom'}) → {trusted:true, scope:'hkcu'} excel({action:'run_vba', code:'Sub DesktopTouchAdHoc()\n Range("A1").Value = "Hello"\nEnd Sub'}) → {ok:true, workbookPath:'...\trusted-vba\dt_vba_.xlsm'} excel({action:'run_vba', code:'Sub Demo()\n MsgBox "hi"\nEnd Sub', macroName:'Demo', visible:true}) → demo recording path

Inputs: no named parameters.

focus_windowA

Bring a window to the foreground by partial title match (case-insensitive). Use when a tool does not accept a windowTitle param, or when you need to switch focus before a sequence of actions. Use chromeTabUrlContains to activate a specific Chrome/Edge tab by URL substring before focusing — only the active tab's title appears in the windows list. If CDP is unavailable, chromeTabUrlContains is silently skipped — check response.hints.warnings. Returns WindowNotFound if no match exists; call desktop_discover to see available titles. Caveats: On some apps focus may be immediately stolen back (modal dialogs, UAC prompts) — verify with desktop_state after focusing. Win11 foreground refusal (UIPI cross-elevation / admin-only target / call from a background process or service) returns code:'ForegroundRestricted' ok:false instead of silently failing — recover by switching to a tool that does not require foreground transfer: desktop_act / click_element use UIA InvokePattern (no foreground needed); keyboard BG path bypasses foreground for terminal-class targets only (Windows Terminal / cmd / PowerShell — keyboard with windowTitle on non-terminal apps still hits the same ForegroundRestricted refusal). browser_* tools target by tabId/selector, not windowTitle.

Inputs: title (required), cdpPort, include, forceFocus, +1 more.

keyboardA

Purpose: Send keyboard input to a window: 'type' for text, 'press' for key combos, 'sequence' for atomic multi-step chords. Details: action='type' inserts text (auto-clipboard for non-ASCII / IME-safe). action='press' sends key combos like 'ctrl+c'/'alt+tab'. action='sequence' runs ordered steps in one keyboard lock — use for Alt+letter, letter mnemonic chains where intermediate tool calls would close the menu. Pass windowTitle to auto-focus and auto-guard (identity, foreground, modal) before input. Omitting windowTitle acts on the active window (unguarded). Prefer: Use windowTitle to auto-focus before injection. Set lensId for perception guards. Use desktop_act({action:'setValue'}) for UIA ValuePattern text fields. Caveats: win+r/win+x/win+s/win+l blocked. action='type' does not handle CJK IME composition — use use_clipboard=true or desktop_act({action:'setValue'}). Non-ASCII text (CJK / emoji / diacritics / smart-quote-class punctuation) auto-clipboards to prevent silent-drop and Chrome accelerator hijack; pass forceKeystrokes:true to disable. Background (PostMessage/WM_CHAR) auto-engages for terminal-class windows (Windows Terminal / cmd / PowerShell); DTM_BG_AUTO=1 enables globally. Foreground non-terminal type runs a per-chunk leash; user focus-steal mid-stream aborts with FocusLostDuringType + context.typed/remaining; pass abortOnFocusLoss:false to disable. BG type verifies WM_CHAR via UIA TextPattern read-back; mismatch returns BackgroundInputNotDelivered (see SUGGESTS for false-positive notes). BG press read-back is scoped to terminal-class + enter/tab/arrow; other combos return verifyDelivery:'unverifiable', failure returns BackgroundKeyNotDelivered. action='sequence' is FG-only (BG/foreground_flash schema-rejected); emits verifyDelivery:'focus_only'; mid-loop focus theft returns MenuFocusLostMidSequence + context.remaining: Step[]. Win11 FG refusal returns ForegroundRestricted — terminal-class targets auto-engage BG; non-terminal switch to desktop_act / click_element. Examples: keyboard({action:'type', text:'hello', windowTitle:'Notepad'}) → text injected (guarded) keyboard({action:'type', text:'hello'}) → text injected (unguarded) keyboard({action:'press', keys:'ctrl+c'}) → copy keyboard({action:'press', keys:'escape', windowTitle:'Dialog'}) → dismiss dialog keyboard({action:'sequence', steps:[{keys:'alt+i', gapMs:100},{keys:'m'}], windowTitle:'Microsoft Visual Basic'}) → Insert > Module (atomic)

Inputs: action (required), hwnd, keys, text, +15 more.

mouse_clickA

Click at screen coordinates. Normally pass windowTitle so the server auto-guards the click (verifies target identity, foreground, coordinate is inside the target rect) and returns post.perception without a confirmation screenshot. origin+scale from dotByDot=true screenshots are converted to screen coords before guarding. doubleClick:true for double-click; tripleClick:true for triple-click (selects a full line of text). Prefer click_element (UIA) for native apps, prefer browser_click for Chrome. Examples: mouse_click({windowTitle:'Notepad', x:200, y:150}) // guarded — post.perception.status='ok'. mouse_click({x:100, y:100}) // unguarded — post.perception.status='unguarded'. If a guard failure returns a suggestedFix, pass its fixId to approve the fix: mouse_click({fixId:'fix-...'}) // one-shot, expires in 15s. lensId is optional and only for advanced pinned-target workflows; omit it for normal use. Caveats: origin+scale are meaningful ONLY with dotByDot=true screenshot responses. hints.verifyDelivery:{status:'delivered'|'focus_only'|'unverifiable', reason} reports the post-click observation in 3 values (focused-element shift, window-foreground change, or no signal). Win11 foreground refusal during the homing path (UIPI cross-elevation / admin-only target / call from a background process or service) returns code:'ForegroundRestricted' ok:false rather than landing the click on the wrong window — recover by switching to a tool that accepts windowTitle directly (click_element / desktop_act) — browser_* tools target by tabId/selector, not windowTitle. MouseClickNotDelivered is reserved-only (false-positive risk is too high to emit a typed code), so degradation is expressed via the 'unverifiable' status, not a separate error.

Inputs: x (required), y (required), verifyDelivery (required), hwnd, +17 more.

mouse_dragA

Click and drag from (startX, startY) to (endX, endY) holding the left mouse button — for sliders, drag-and-drop, canvas drawing, and window resizing. Pass windowTitle so the server auto-guards the start coordinate and returns post.perception. Examples: mouse_drag({windowTitle:'Notepad', startX:50, startY:50, endX:200, endY:200}). lensId is optional and only for advanced pinned-target workflows. Caveats: Left button only. Both start and endpoint are guarded. Cross-window and desktop drags are blocked by default — pass allowCrossWindowDrag:true to confirm intent. hints.verifyDelivery:{status:'delivered'|'focus_only'|'unverifiable', reason} reports the post-drop observation in the same 3-value shape as mouse_click. MouseDragNotDelivered is SUGGESTS-registered but reserved-only (not emitted) — degradation is expressed via the 'unverifiable' status rather than a typed code. Win11 foreground refusal (UIPI cross-elevation / admin-only target / call from a background process or service) returns code:'ForegroundRestricted' ok:false from the homing path.

Inputs: endX (required), endY (required), startX (required), startY (required), +10 more.

notification_showA

Show a Windows system tray balloon notification to alert the user. Use at the end of a long-running task so the user knows it finished without watching the screen. Caveats: toast の user reach は原理的に観測不能 (matrix §3.1 line 158 規範整合)。Focus Assist (Do Not Disturb) / Notifications-off setting / consent UI sink いずれも tool 側からは判別不能のため、successful response は常に hints.verifyDelivery を含む (status="unverifiable", reason="user_visible_side_effect_uninspectable", channel="win32_balloon_tip" — 全 double-quoted JSON literal)。caller は user 側の post-notification behavior (例: wait_until(focus_changes)) で間接観測することが望ましい。Uses System.Windows.Forms — no external modules needed.

Inputs: body (required), title (required), include.

run_macroA

Purpose: Execute multiple tools sequentially in one MCP call — eliminates round-trip latency for predictable multi-step workflows. Details: steps[] is an array of {tool, params} objects. Accepts all desktop-touch tools plus a special sleep pseudo-step: {tool:"sleep", params:{ms:N}} (max 10000ms per step). stop_on_error=true (default) halts on first failure. Max 50 steps. The LLM cannot inspect intermediate results during execution — all steps run to completion (or first error) before any output is returned. Prefer: Use for predictable fixed sequences (focus → sleep → type → screenshot). Do not use for conditional logic — return to the LLM between branches so it can inspect intermediate state. Caveats: If any step may fail conditionally (e.g. a dialog that may or may not appear), split the macro at that point. Each screenshot step within a macro incurs the same token cost as a standalone call. Examples: [{tool:'focus_window',params:{windowTitle:'Notepad'}},{tool:'sleep',params:{ms:300}},{tool:'keyboard',params:{action:'type',text:'Hello'}},{tool:'screenshot',params:{detail:'text',windowTitle:'Notepad'}}] [{tool:'browser_navigate',params:{url:'https://example.com'}},{tool:'wait_until',params:{condition:'element_matches',target:{by:'text',pattern:'Example Domain'}}}]

Inputs: steps, include, stop_on_error.

screenshotA

Purpose: Capture desktop, window, or region across detail levels (meta / text / image / som / ocr) and capture modes (normal / background). Details: detail='meta' (default) returns window titles+positions only (~20 tok/window, no image). detail='text' returns UIA actionable elements with clickAt coords, no image (~100-300 tok). detail='som' returns OCR-detected elements with IDs plus a Set-of-Marks annotated image delivered by-ref by default (bypasses UIA entirely). detail='ocr' returns Windows OCR words with screen-pixel clickAt coords (Phase 4: absorbs former screenshot_ocr — use when UIA is sparse and you want to force OCR unconditionally). detail='image' and detail='som' both return a cheap by-ref resource_link by default (no inline base64); pass confirmImage=true to also embed the inline image (the annotated bitmap for som). mode='background' captures hidden/minimised/occluded windows via PrintWindow (Phase 4: absorbs former screenshot_background) — pair with windowTitle/hwnd. dotByDot=true returns 1:1 pixel WebP; compute screen coords: screen_x = origin_x + image_x (or screen_x = origin_x + image_x / scale when dotByDotMaxDimension is set — scale printed in response). diffMode=true returns only changed windows after the first call (~160 tok). region={x,y,width,height} captures a sub-rectangle (Phase 4: absorbs former scope_element when paired with windowTitle/hwnd — discover element bounds via desktop_discover, then pass region here). Data reduction: grayscale=true (−50%), dotByDotMaxDimension=1280 (caps longest edge), windowTitle+region (sub-crop to exclude browser chrome — e.g. region={x:0, y:120, width:1920, height:900}). Prefer: Use meta to orient, text before clicking, dotByDot only when precise pixel coords are needed. Use detail='som' for native apps or games that do not expose UIA elements (UIA-Blind). Use detail='ocr' for OCR-only (skip UIA entirely). Use mode='background' when the target window must stay hidden or cannot be brought to foreground. Prefer browser_* tools for Chrome. Use diffMode after actions to confirm state changed. Only use image+confirmImage when text returned 0 actionable elements and visual inspection is genuinely required. Caveats: Default mode scales to maxDimension=768 — image pixels ≠ screen pixels; apply the scale formula before passing to mouse_click. Foreground detail='image' returns a by-ref resource_link by default; pass confirmImage=true to also receive inline pixels. diffMode requires a prior full-capture baseline (non-diff call or workspace_snapshot) — calling diffMode cold returns a full frame, not a diff. mode='background' requires windowTitle or hwnd, and only composes with detail in {'image','meta'} — detail='text'/'som'/'ocr' run only against foreground capture (the dispatcher rejects the conflicting combination). Passing mode='background' is itself the acknowledgement that image pixels are wanted, so confirmImage is NOT required for it (matches the former screenshot_background contract). fullContent=false enables legacy mode (faster but GPU windows may be black). detail='ocr' requires windowTitle or hwnd; first call may take ~1s (WinRT cold-start) and the matching OCR language pack must be installed. Examples: screenshot() → meta orientation of all windows screenshot({detail:'text', windowTitle:'Notepad'}) → clickable elements with coords screenshot({detail:'ocr', windowTitle:'PDF', ocrLanguage:'ja'}) → OCR words with screen-pixel coords screenshot({mode:'background', windowTitle:'Chrome', dotByDot:true, dotByDotMaxDimension:1280, grayscale:true}) → background-capture pixel-accurate Chrome screenshot({windowTitle:'Notepad', region:{x:0,y:120,width:600,height:400}}) → cropped sub-region (zoom into element after desktop_discover)

Inputs: hwnd, mode, detail, region, +15 more.

screenshot_gcA

Reclaim disk space from cached screenshots by retention policy. By DEFAULT this is a dry run: it returns the captures that WOULD be deleted (candidates) plus a count/size of leftover orphan files, and deletes nothing. To actually delete, pass BOTH dryRun:false AND confirm:true. Retention caps (all optional): maxCount (keep newest N), maxTotalBytes (keep newest under a byte budget), maxAgeMs (delete older than). When you pass none, the cache's env defaults apply (newest 200 / 256 MiB). Scope to a single tag with tag (other tags are never touched); includeOrphans (default true) also reclaims leftover on-disk files with no index entry. The newest capture is always kept by the count/byte caps. Only ever touches files inside the screenshot cache — never any other path.

Inputs: tag, dryRun, confirm, include, +4 more.

screenshot_queryA

List screenshots already saved in the disk-cache WITHOUT re-reading any pixels. The screenshot tools return each capture as a cheap by-ref link (screenshot://by-ref/{captureId}); this lists what is in the cache — captureId + by-ref uri, dimensions, size in bytes, timestamp, and tag/window — so you can find and re-open a specific earlier capture, or check how much the cache holds before reclaiming space with screenshot_gc. The response also carries whole-cache totals (totalCaptures / totalBytes). Reading a capture's bytes still costs tokens, so open a by-ref link only when you actually need to inspect the pixels. Filter by tag (case-insensitive) / windowUuid / since / until; page with limit (default 50) and offset. Results are newest-first and never include a filesystem path.

Inputs: tag, limit, since, until, +3 more.

scrollA

Purpose: Scroll a window or page. 5 strategies via action: 'raw' (wheel notches), 'to_element' (UIA name/automationId or CSS selector), 'smart' (auto-detect target with multi-strategy fallback), 'capture' (full-page stitched image), 'read' (scroll+OCR+dedupe → stitched text). Details: action='raw': send raw mouse-wheel notches at (x,y) or current cursor, optional window focus. Scroll scale — UIA Tier 1 (ScrollPattern apps): empirically ≈1 text line per notch; amount:3 (default) ≈ 3 lines (small nudge), amount:10 ≈ 10 lines (~½ visible area). Legacy SendInput: each amount unit = 3 wheel ticks; ≈9 text lines per unit at Windows default (app/OS-setting dependent). action='to_element': scroll a named element into viewport (UIA or CDP). action='smart': handles nested scroll layers, virtualised lists, sticky-header occlusion. action='capture': stitches full-page images (caps at ~700KB raw); sizeReduced=true means downscaled. action='read': scrolls page-by-page, OCRs each viewport, deduplicates overlapping lines, returns stitched text; language auto-detected from OS locale if omitted. Prefer: Use action='to_element' or action='smart' for click target out-of-viewport recovery (entity_outside_viewport). Use action='capture' for reading long pages as images. Use action='read' for extracting text from long native-app documents (PDF readers, text editors, terminals) where copy-paste is unavailable. For simple scroll without target, use action='raw'. Caveats: action='capture' returns stitched image — pixels do NOT match screen coords when sizeReduced=true, use for reading only, not mouse_click. action='smart' CDP path requires browser_open. action='to_element' native path requires element to implement UIA ScrollItemPattern. action='read' uses OCR (imperfect accuracy) and requires the window to be visible; for browser pages prefer browser_eval or browser_overview for accurate DOM text. action='raw' typed errors: code:'ScrollNotDelivered' on silent drop (overlay / non-scrollable / UIPI low-IL); already-at-boundary is success via pre/post-percent disambiguation. hints.verifyDelivery.{channel,reason} per ADR-018 §2.6 (Phase 1b: Tier 1 UIA dispatch for HWNDs exposing ScrollPattern; other apps use legacy SendInput). action='smart' typed errors: code:'OverflowHiddenAncestor' (retry with expandHidden:true), code:'VirtualScrollExhausted' (provide virtualIndex). Examples: scroll({action:'raw', direction:'down', amount:5, windowTitle:'Chrome'}) scroll({action:'to_element', name:'OK', windowTitle:'Dialog'}) scroll({action:'smart', target:'#create-release-btn'}) scroll({action:'capture', windowTitle:'Chrome', maxScrolls:10}) scroll({action:'read', windowTitle:'Acrobat', maxPages:15}) // OCR + dedupe long PDF

Inputs: action (required), x, y, hint, +28 more.

server_statusA

Return MCP server status. engine: native engine availability — uia: 'native' = Rust UIA addon (~2 ms focus / ~100 ms tree); 'powershell' = PS fallback (~366 ms focus). imageDiff: 'native' = Rust SSE2 SIMD (0.26 ms @ 1080p); 'typescript' = TS fallback (~3.8 ms). health: process diagnostic snapshot (issue #365) — uptimeSec, memory.{rssBytes,heapUsedBytes,heapTotalBytes}, cpu.{userUs,systemUs} (cumulative since startup), shutdown.{pending,graceMs,inflightCount} (pending=true means stdin EOF received and grace timer is running), lastRpc.{receivedAt(ISO),method} (last JSON-RPC request observed on stdio transport; HTTP transport is not tracked). Diagnostic metadata — do not surface unless the user asks about performance/troubleshooting. engine values are stable for the process lifetime; health values change per call.

Inputs: include.

shell_sessionA

Purpose: Reliable background shell channel backed by a real OS process (stdin/stdout/stderr + exit code). Use this for installs, scripts, file ops, and any command that must actually finish — not the visible-terminal terminal tool. Details: Actions: start (spawn hidden shell), write (stdin + optional wait/read), read (drain ring buffer), exec (one-shot run-to-completion with exit code), list, stop. Never reports ok:true on write unless stdin accepted AND process still tracked; exec only returns ok:true when exitCode===0. Non-zero exits return ok:false code:ShellSessionNonZeroExit with stdout/stderr. Default shell: powershell on Windows, bash elsewhere. Independent of GUI focus, IME, clipboard, and SoM terminal pollution. Prefer: Prefer shell_session({action:'exec', input:'...'}) for one-shot commands. Prefer start+write+read for interactive multi-step shells. Keep terminal for when you must drive a visible console window the user is watching. Caveats: This does NOT open a visible window — do not use it when the user needs to see the console UI. Ring buffer is capped (default 1MB). Max 16 concurrent sessions. On Windows, powershell/cmd are windowsHide; do not send Ctrl+C via keyboard tools into these sessions (use action:'stop'). Examples: shell_session({action:'exec', input:'whoami'}) → {ok:true, exitCode:0, stdout:'...'} shell_session({action:'start', id:'ops', shell:'powershell'}) → {ok:true, session:{pid,...}} shell_session({action:'write', id:'ops', input:'Get-ChildItem', waitMs:500}) → stdout/stderr shell_session({action:'read', id:'ops', clear:true}) shell_session({action:'stop', id:'ops'})

Inputs: no named parameters.

terminalA

Purpose: Interact with a terminal window: read output, send input, or run+wait+read in one call. action='read' / action='send' absorb the formerly-standalone read/send tools (Phase 4). Details: action='run' is the recommended high-level workflow: send command → wait until quiet/pattern/timeout → read output. The command text is passed as input (the legacy parameter name command is also accepted as a deprecated alias — see issue #245). Returns completion={reason, elapsedMs} first-class plus outputIntegrity:'ok'|'baseline_lost' so callers can detect when scrollback could not be anchored to the pre-send buffer. action='read' reads current text via UIA TextPattern (falls back to OCR); use sinceMarker for incremental diff. action='send' sends a command with focus management. Prefer: action='run' for command execution + result. For long-running commands (test runners, builds, deploys) use until:{mode:'pattern', pattern:''} — the default quiet mode is tuned for short interactive commands and may complete prematurely on multi-second silent gaps mid-run. Use action='read'/'send' for fine-grained control or when you need to interleave other actions. Caveats: Do not screenshot the terminal — action='read' is cheaper and structured. action='run' supports completion reasons: quiet | pattern_matched | exited | timeout | window_closed | window_not_found | send_failed (send rejected on a live window — see warnings for the underlying error code). until:{mode:'exit', shell:'bash'|'powershell'} (issue #386) returns completion.exitCode + reason:'exited' via an echo-immune sentinel that works for multiline input that pattern mode cannot anchor; pass shell explicitly (auto fails as ExitModeShellAmbiguous on WT/conhost/SSH), cmd is unsupported (ExitModeShellUnsupported), open-construct input is rejected (ExitModeUnsafeInput). When outputIntegrity:'baseline_lost' is returned, output is forced to '' and readError.code='BaselineMarkerLost' is set: rerun with until:{mode:'pattern',...} or longer timeoutMs. action='run' may also emit warnings prefixed FileLockCollision: when output reveals an EBUSY/Windows-lock/EAGAIN-EDEADLK file collision (e.g. shell '>' redirect colliding with the script's own writer — issue #236). Default quietMs=1500 (issue #196); long silences require pattern mode. preferClipboard=true (send default) overwrites clipboard. Hidden-input prompts emit verifyDelivery.unverifiable (reason:'hidden_input_prompt') — use method:'foreground'. action='read' typed errors: TerminalWindowNotFound, TerminalTextPatternUnavailable (force source:'ocr'); stale sinceMarker → hints.terminalMarker.previousMatched:false on ok:true (omit sinceMarker). FG-path Win11 foreground refusal returns code:'ForegroundRestricted' — switch to method:'background' or DTM_BG_AUTO=1. BG path auto-engages only when (a) the target window class is ConsoleWindowClass (conhost: cmd / PowerShell / pwsh classic hosts) OR (b) env DTM_BG_AUTO=1 is set globally. Windows Terminal (CASCADIA_HOSTING_WINDOW_CLASS) is intentionally EXCLUDED from auto-engage (issue #173): WT runs on WinUI/XAML and silently drops WM_CHAR posted to its HWND, so the FG path is used by default — pass sendOptions:{method:'background'} only if you have verified your WT build accepts BG input. Examples: terminal({action:'run', windowTitle:'PowerShell', input:'npm test', until:{mode:'pattern', pattern:'Test Files'}}) → recommended for test runners; matches when vitest summary appears terminal({action:'run', windowTitle:'pwsh', input:'ls'}) → quiet 1500ms wait, returns output (short interactive) terminal({action:'run', windowTitle:'pwsh', command:'ls'}) → identical to the above; command is a deprecated alias of input (issue #245) terminal({action:'read', windowTitle:'PowerShell', sinceMarker:'...'}) → incremental diff using the read action terminal({action:'send', windowTitle:'PowerShell', input:'echo hello'}) → sends text + Enter using the send action

Inputs: action (required), input, until, command, +5 more.

wait_untilA

Purpose: Server-side poll for an observable condition — eliminates screenshot-polling loops when waiting for state changes. Details: condition selects what to watch: window_appears/window_disappears (target.windowTitle required), focus_changes (optional target.fromHwnd), element_appears/value_changes (target.windowTitle + target.elementName required, UIA; min 500ms interval), ready_state (target.windowTitle; visible + not minimized), terminal_output_contains (target.windowTitle + target.pattern required [+target.regex:true], needs terminal tools loaded), element_matches (target.by + target.pattern required, needs browser tools loaded), url_matches (target.pattern required [+target.regex:true]; matches the active tab's location.href via CDP — use for SPA route changes, redirects, OAuth flows). Returns {ok:true, elapsedMs, observed} on success, or WaitTimeout error with suggest hints. timeoutMs default 5000 (max 60000). Prefer: Use instead of run_macro({sleep:N}) + screenshot loops. Use terminal_output_contains to detect CLI command completion. Use element_matches for browser DOM readiness after navigation. Use url_matches when the URL is the most reliable signal (SPA routing / redirect cascades). Caveats: terminal_output_contains, element_matches, and url_matches require a browser CDP connection (open --remote-debugging-port=9222 first). element_appears/value_changes spawn a UIA process per poll — interval clamped to 500ms minimum. On elapsed-timeout the response is {ok:false, code:'WaitTimeout', error, suggest:[...]}; the suggest[] array lists three fixed actions: 'Increase timeoutMs', 'Verify the target is correct', 'Inspect intermediate state with screenshot(detail='meta')'. Non-timeout failures also occur — pre-poll validation and missing-hook errors classify as code:'ToolError' (read the descriptive error message), and CDP probe errors (url_matches / element_matches conditions) surface as code:'BrowserNotConnected' (re-attach via browser_open). Branch on code rather than assume WaitTimeout. Examples: wait_until({condition:'window_appears', target:{windowTitle:'Save As'}, timeoutMs:10000}) wait_until({condition:'terminal_output_contains', target:{windowTitle:'Terminal', pattern:'$ '}, timeoutMs:30000}) wait_until({condition:'element_matches', target:{by:'text', pattern:'Submit', scope:'#checkout-form'}}) wait_until({condition:'url_matches', target:{pattern:'/dashboard'}, timeoutMs:15000}) wait_until({condition:'url_matches', target:{pattern:'^https://app\\.example\\.com/orders/[0-9]+$', regex:true}})

Inputs: condition (required), target, include, timeoutMs, +1 more.

window_dockA

Purpose: Decorate a window: pin (always-on-top), unpin, or dock (move + resize + optional pin). Details: action='pin' makes window always-on-top until unpin/duration_ms. action='unpin' removes always-on-top. action='dock' positions to corner with width/height (default 480×360 bottom-right) and optionally pins. Minimized windows are automatically restored before docking. Prefer: Use action='dock' for terminal/CLI window auto-positioning at session start. Use action='pin' alone when you only need always-on-top without moving or resizing. Caveats: Pin survives minimize/restore; explicit action='unpin' needed to release. Dock fails on elevated processes. Dock overrides any existing Win+Arrow snap arrangement. Examples: window_dock({action:'dock', title:'PowerShell', corner:'bottom-right', width:480, height:360}) window_dock({action:'pin', title:'Settings', duration_ms:5000}) window_dock({action:'unpin', title:'Settings'})

Inputs: action (required), pin, title, width, +6 more.

workspace_launchA

Purpose: Launch an application and wait for its new window to appear, returning title, HWND, and PID. Details: Runs the command via ShellExecute, snapshots the window list before launch, then polls until a new HWND appears (compared by HWND, not title). Returns {windowTitle, hwnd, pid, elapsedMs}. Works for localized window titles (e.g. '電卓' for calc.exe) because detection is HWND-based, not title-based. timeoutMs default 10000. detach=true fires without waiting and returns no window info. Prefer: Use instead of run_macro({exec, sleep, desktop_discover}) combos. Follow with focus_window(windowTitle) to interact with the launched app. Caveats: Single-instance apps that reuse an existing window will not register as a new HWND — call desktop_discover first to check if the window is already open. detach=true returns immediately with no window title or hwnd. Examples: workspace_launch({command:'notepad.exe'}) → {windowTitle:'', hwnd:'...', pid:...} workspace_launch({command:'calc.exe', timeoutMs:15000})

Inputs: command (required), args, waitMs, include.

workspace_snapshotA

Purpose: Orient fully in one call — returns display layouts, all window thumbnails (WebP), and per-window actionable element lists with clickAt coords. Details: uiSummary.actionable[] per window includes: action ('click'|'type'|'expand'|'select'), clickAt {x,y} (pass directly to mouse_click), value (current text for editable fields). Runs parallel internally; latency ≈ max(single screenshot), not N×screenshots. Also resets the diffMode buffer so subsequent screenshot(diffMode=true) returns only changes (P-frame). Prefer: Use at session start or after major workspace changes. Use screenshot(detail='meta') for cheap re-orientation within a session. Use screenshot(detail='text', windowTitle=X) for a single-window update. Caveats: Thumbnails are scaled, not 1:1 — use screenshot(dotByDot=true, windowTitle=X) for pixel-accurate coords on a specific window after snapshot. Also: this call resets the screenshot diff baseline (I-frame) and identity tracker as a side effect, so subsequent screenshot(diffMode=true) starts fresh from this snapshot. The reset is not currently exposed in causal/working memory — record an explicit 'workspace_snapshot' step if you need to track the reset point in your causal trail (ADR-010 §11 OQ carry-over for full visibility).

Inputs: include, includeUiSummary, thumbnailMaxDimension.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A3.8/5.0

    Scored across 30 tools

    Disambiguation3/5

    Multiple clusters overlap in function: three click tools (browser_click, click_element, mouse_click), four browser-discovery tools (overview/search/locate/form), and two shell tools (shell_session, terminal). The descriptions' consistent 'Prefer X over Y' guidance resolves most ambiguity, but an agent must track CDP-vs-UIA-vs-coordinates distinctions across several near-synonym tools.

    Naming Consistency3/5

    The browser_* prefix (9 tools) plus mouse_*, workspace_*, and screenshot_* clusters are highly consistent, but the remaining tools mix conventions: verb_noun (click_element, focus_window, run_macro), noun_verb (window_dock, notification_show), and bare nouns (keyboard, scroll, excel). All lowercase snake_case keeps it readable, but action placement is unpredictable across the set.

    Tool Count2/5

    At 30 tools this exceeds the 25+ threshold and creates a heavy context and selection burden for agents. Several tools are peripheral or mergeable (screenshot_query/screenshot_gc are both cache management, server_status is diagnostics, notification_show is a one-shot utility), though the genuinely broad desktop-automation scope softens the excess somewhat.

    Completeness2/5

    The browser lifecycle is well covered (open, navigate, click, fill, observe, wait), but the native-desktop workflow has dead ends: desktop_discover and desktop_act are referenced as required precursors in multiple descriptions yet are absent from the tool set, so agents following documented guidance will hit tool-not-found errors. Missing browser_close/tab management and first-class element enumeration are notable gaps.

    Maintenance

    ActivityNo data
    ResponsivenessOutdated data

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      D
      maintenance
      Enables AI clients to automate Windows desktop applications through window manipulation, image recognition, OCR, keyboard/mouse simulation, and memory operations via the MCP protocol.
      MIT
    • A
      license
      A
      quality
      C
      maintenance
      Allow AI agents to see and control a real Windows PC you own: observe (UIA + screenshots), click/type/drag/scroll, launch apps, owner Live View. BYOH — your machine, your key.
      16
      Apache 2.0
    • A
      license
      Not graded
      quality
      C
      maintenance
      Enables AI clients on Windows to control the local mouse, keyboard, and screen understanding via MCP stdio, allowing automated workflows like viewing the screen, locating elements, clicking, typing, and verifying results.
      1
      Apache 2.0