OpenEyes
OpenEyes exposes an MCP server for safely observing, interacting with, and verifying Windows desktop GUI and browser automation.
List visible windows, capture screenshots, and enumerate interactive UI elements via the accessibility tree.
Click by coordinates, name/selector, stable automation ID, or grid cell; send hotkeys and type Unicode text.
Resolve elements through a structured-first
uia -> cdp -> visionchain and parse OmniParser-compatible detections.Describe a window as text using UIA, OCR, and OpenCV, including coordinates and digest output.
Launch and control browsers over Edge CDP or Firefox BiDi: list tabs, scan DOM elements, click, type, and screenshot.
Ensure apps are running or usable, and discover native automation surfaces such as CLI, URL schemes, executables, or Store AUMIDs.
Probe and wait for OpenCV visual anchors against a single window frame.
Perform actions and verify expected UI state changes; replay recorded trajectories in dry-run or live mode.
Stay safety-oriented with dry-run defaults, pointer-click budgets, structured JSON, and explicit verification.
Provides computer-use primitives on Linux desktops, enabling GUI capture, UI element detection through AT-SPI accessibility APIs, and mouse/keyboard actuation via XTest, exposed as MCP tools.
Provides computer-use primitives on macOS, enabling desktop GUI capture, UI element detection through macOS accessibility APIs, and mouse/keyboard actuation via CG, exposed as MCP tools.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@OpenEyes capture the active window and list its interactive elements"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
OpenEyes(开源点睛)
AI-friendly computer-use primitives for Windows desktops. Capture. Detect. Click. Let any LLM agent drive any desktop GUI.
GitHub: https://github.com/yangpeng366/openeyes · 中文 · English · Quick start · Architecture · Roadmap · License
一键安装(面向 Agent)
npx skills add yangpeng366/openeyes # 装 skill
npx skills add yangpeng366/openeyes --list # 不装,只查会收录什么装好之后 agent 拿到的是 skills/openeyes/SKILL.md 这份指引:它负责判断什么时候该用 OpenEyes、以及怎么用(命令表、退出码、降级链)。本仓本体是 pip 包 openeyes(提供 eyes CLI 与 eyes-mcp),skill 只是说明书,不替代安装本体:
pip install openeyesRelated MCP server: helix-pilot
中文
OpenEyes(点睛) 是一个开源的 AI 友好电脑使用平台。基于「see → resolve → act」三段式原语,让任何 LLM agent 都能可靠地操控桌面 GUI 应用;提供 Vimium 风格字母 hint overlay 作为人类可调试的回退通道,并通过 MCP server 暴露给所有 agent。
为什么做这个
现状 | 痛点 |
Neverclick | 闭源付费,仅 Windows,无 API |
OmniParser / OS-Atlas | 只做 detection,没有 actuation,模型重 |
Anthropic Computer Use / OpenAI Operator | 仅 Linux VM,看不见本机,不可定制 |
UiBot / APA | 录制型,对动态 UI 适应性差 |
pywinauto / xdotool | 纯库,无 AI 友好的 element schema |
OpenEyes 把这些能力合成一个 MIT 开源、跨平台、AI 友好的统一平台。
特性
AI 友好 element schema:每个 UI 元素输出结构化 JSON(bbox / center / control_type / name / automation_id / class / state)
易点击:UIA 命中 → 自动点中心;不命中 → vision bbox → 点中心;都不命中 → 字母 hint overlay
安全:默认 dry-run,所有状态改变操作可审计
MCP 优先:所有原语通过 MCP server 暴露给任何 agent
结构化优先:
uia -> cdp -> vision降级链,能走无障碍树 / DOM 就不看像素,截图只作最后手段解析式 vision backend:OmniParser v2 兼容,屏幕先被解析成结构化元素再决策,不让模型对着像素点坐标
文字场景层:
eyes scene把窗口读成文字清单(无障碍树 → OCR → OpenCV 轮廓),不支持看图的 agent 也能知道页面上有什么、在哪里;窗口被别的窗口盖住时用PrintWindow读窗口自身Windows 优先:当前只有 Windows(UIA)后端真正可用;macOS / Linux 在路线图上,尚未实现
远程浏览器:经 SSH 转发驱动另一台机器上的 Firefox(WebDriver BiDi),隧道与 Host 头一并处理
会话恢复:
recover_session把「掉线 → 拉起进程 → 重连 → 状态稳定」收进一个预算(默认 20 秒),报告带实际耗时,可断言、可 dry-run
设计文档
完整设计书:https://my.feishu.cn/docx/Ul6gdMULGo5VfzxDVAYcVoX6n9e (中文,含架构图 / 模块清单 / 路线图)
面向 Agent 的接入
下游多半是大模型 agent,所以兼容做到接口层面即可,不必先读源码:
CLI:
eyes <command> --json,字段固定;子命令一律先 dry-run,再--go真执行MCP:直接挂
openeyes-mcp,工具名与 CLI 一一对应,参数即 JSON schema退出码:0 成功、1 失败、2 参数错、3 没匹配到目标、4 后端不可达、5 动作没生效 (JSON 走 stdout,人读的错误行走 stderr,所以 stdout 永远可直接喂解析器)
元素 schema:
bbox / center / control_type / name / automation_id / class / state,可直接喂给模型降级链:
uia -> cdp -> vision,拿到的source字段说明这答案从哪来
要改行为或做深度集成时再读 Architecture 与 Record & Replay;交接要点见 AGENTS.md。
English
OpenEyes is an open-source, AI-friendly computer-use platform. Built on a "see → resolve → act" pipeline, it lets any LLM agent reliably drive any desktop GUI. Includes a Vimium-style letter-hint overlay as a human-debuggable fallback, and exposes every primitive via MCP server for any agent.
Why
Today | Pain |
Neverclick | Closed-source, paid, Windows-only, no API |
OmniParser / OS-Atlas | Detection only, no actuation, heavy models |
Anthropic Computer Use / Operator | Linux VMs only, opaque, no audit |
UiBot / APA | Record/playback, weak on dynamic UIs |
pywinauto / xdotool | Raw libs, no AI-friendly schema |
OpenEyes fuses these into one MIT-licensed, AI-friendly platform for Windows desktops.
Features
AI-friendly element schema — every interactive element exposes structured JSON
Easy click — UIA hit → bbox center; fallback vision bbox → center; final fallback letter hint
Auditable degradation —
resolvereports which backend answered and why the chain fell throughSafe by default — dry-run; full audit trail for state-changing actions
MCP-first — every primitive exposed as an MCP tool to any agent
Structured first — a
uia -> cdp -> visiondegradation chain: prefer the accessibility tree or the DOM, treat pixels as the last resortParsing-style vision backend — OmniParser v2 compatible; the screen is parsed into structured elements before anything decides, never raw coordinate clicking
Scene text tier —
eyes scenereads a window as a list of named elements (accessibility tree → OCR → OpenCV contours), so an agent that cannot look at a picture still knows what is on screen and where;PrintWindowreads the window itself even when something covers itWindows first — the Windows (pywinauto/UIA) backend is the one that works today; macOS and Linux are on the roadmap, not in the box
Remote browser — drive a Firefox on another machine through an SSH forward (WebDriver BiDi); the tunnel and the Host header are handled for you
Session recovery —
recover_sessionchases a dropped session (detect -> start the process -> reconnect -> settle) inside one budget (20 s by default) and reports the seconds it took, so the promise can be asserted instead of hoped for
Quick start
Windows
git clone https://github.com/yangpeng366/openeyes.git
cd openeyes
pip install -e ".[windows,mcp]"
# 1. list visible top-level windows; take an hwnd from the table
eyes windows list
# 2. capture that window (0 = the whole screen)
eyes capture --window <hwnd> --out shot.png
# 3. enumerate interactive elements
eyes detect --window <hwnd> --pretty
# 4. click by text: dry-run first, then --go
eyes click --window <hwnd> --name-contains "Submit"
eyes click --window <hwnd> --name-contains "Submit" --go --budget 1
# 5. start MCP server (for Codex / Claude / Cursor)
eyes-mcp--budget N allows at most N physical clicks in one run (default 0:
semantic clicks only). A control with no invoke path then fails with
BACKGROUND_UNAVAILABLE instead of moving your mouse; add --budget 1,
or ask for it by name with --method pixel.
Output contract
Every command answers the same way, so a script (or an agent) never has to guess:
--jsonis a global flag - it works before or after the subcommand:eyes --json detect --window 123andeyes detect --window 123 --jsonare the same call.The JSON goes to stdout; the human-readable error line goes to stderr. Piping stdout into a JSON parser therefore always works.
A failure is a non-zero exit code:
3= no target matched,4= the backend could not be reached,5= the action did not take effect. The message names the cause and the parameters that were in play.
# machine-readable
eyes --json detect --window 123456 --automation-id "SendButton"
eyes --json app state --title-contains "Any VPN" --process-name "Any VPN"First showcase — 飞书 client
See examples/feishu_first_test.py.
python examples\feishu_first_test.py --dry-runArchitecture
┌────────────────────────────────────────────────────────────────┐
│ L5 Orchestration Codex / Claude / Cursor / 自定义 Agent │
├────────────────────────────────────────────────────────────────┤
│ L4 AI Layer Intent → Element (LLM 驱动 + 缓存) │
├────────────────────────────────────────────────────────────────┤
│ L3 Resolver selector → coord (UIA / Vision / Hint) │
├────────────────────────────────────────────────────────────────┤
│ L2 Perceive capture + detect (UIA / AX / AT-SPI) │
├────────────────────────────────────────────────────────────────┤
│ L1 Actuate mouse / key / drag (Win32 / CG / XTest) │
├────────────────────────────────────────────────────────────────┤
│ L0 Platform Windows (UIA) - macOS/Linux planned │
└────────────────────────────────────────────────────────────────┘See docs/architecture.md for the full design.
Record & Replay (v0.2.0)
No screenshots, no vision model required. Every recorded step is a structured (element, action) pair — pure text that any LLM can read, reason about, and replay.
Two capture paths (not mutually exclusive)
Path | How | When |
A — Show | Human demonstrates; | monkeybot-style, real workflow |
B — Prompt | Human describes intent; LLM or optional JEV backend generates a | WorkBuddy-style, zero-demo |
Quick example
from openeyes.record import Recorder, Replayer, to_natural_language
from openeyes.record.export import to_codex_skill
# Path A: show
r = Recorder()
r.start(name="rotate-vpn", app_name="AnyVPN")
r.record_click(hwnd, element, intent="open VPN client")
r.record_click(hwnd, element, intent="click Connect")
r.record_wait(2000, intent="wait for tunnel")
traj = r.stop()
# Inspect
print(to_natural_language(traj)) # plain English steps
print(to_codex_skill(traj)) # Codex SKILL.md
# Replay (dry-run)
report = Replayer(dry_run=True).replay(traj)CLI replay
eyes record replay trajectory.jsonl --json # dry-run: plan only, sends nothing
eyes record replay trajectory.jsonl --live # send real input, judge each step
eyes record replay trajectory.jsonl --live --json # same, report.to_dict() on stdoutWithout --live nothing is sent and every step is planned. With --live the
steps are delivered and each one is verified / unverified / failed; the exit
code is 1 when any step failed. sent: true means delivered, not effective:
the text tail prints delivery: N delivered / N not delivered (plus
refused_activation when a window refused focus) so a step that never reached
the desktop is not read as one that ran and did nothing.
See docs/RECORD_DESIGN.md for the three-state semantics.
Export a loadable skill
eyes record export trajectory.jsonl --skill-dir .skills\rotate-vpnWrites SKILL.md (front-matter + step table + a Run it section) and copies the
source trajectory.jsonl byte-for-byte, so the folder replays itself:
eyes record replay .skills\rotate-vpn\trajectory.jsonl --json prints a dry-run plan.
--json reports skill_dir / skill_md / trajectory / steps / name; a non-empty
target directory is refused unless --force is passed.
Export formats
Format | Use case |
| Generate a Codex / OpenAI skill from recording |
| ai-action-trace-receipt-standard v0.5 YAML audit trail |
| OpenCUA AgentNet-compatible JSONL |
| Build a DAG from N recordings of the same workflow |
| Plain-English step list for LLM consumption |
Demos
python examples\record_demo_mock.py # no UI needed — fake elements
python examples\record_demo_prompt.py # LLM generates trajectory from prompt
python examples\record_demo_real.py # real Edge window (needs openeyes installed)generate_trajectory_from_prompt(..., backend="jev") uses the deterministic
fake JEV scorer by default and switches to the real System One wire when
TYPESAFE_API_KEY is configured, or when TYPESAFE_BASE_URL points at a
loopback node (a local XERJ node may answer keyless).
The thresholds follow the scale of the answer, not the scorer: the hosted
judge answers with a probability and keeps 0.85 / 0.60; a local history
vote answers with a support share (the winner's share of the vote weight)
and gets its own measured profile, per language family (Chinese default) - see
docs/jev-local-judge.md.
$env:TYPESAFE_BASE_URL = "http://127.0.0.1:9201" # local XERJ node
$env:XERJ_API_KEY = (Get-Content D:\AI-Portable\xerj-data\admin.key -Raw).Trim()
python examples\systemone-local-acceptance.py # one JSON line, exit 0
python examples\seed-systemone-history.py --index judgements --recreate # Chinese default
python examples\systemone-calibration.py # measure the judge
python examples\seed-systemone-history.py --index judgements --recreate --lang en # English opt-inSee docs/RECORD_DESIGN.md for the full design rationale.
Backend priority & degradation chain (v0.2.0)
The rule the 2026 computer-use field converged on: reach for structure first, pixels last. openEyes makes that executable instead of a doc line.
uia -> cdp -> vision
accessibility tree DOM screen parsing# which tier answers for this window, and why did the earlier ones fall through?
eyes resolve --window 1234 --pretty
# turn OmniParser v2 detections into structured elements (no model needed)
eyes vision --detections det.json --prettyfrom openeyes import resolve_elements
result = resolve_elements(hwnd=1234, port=9222)
print(result.used_backend) # "uia" | "cdp" | "vision" | None
print([a.to_dict() for a in result.attempts])The vision tier is parsing-style: the screen is parsed into the same
Element[] shape the other backends emit, so the selector, hint and decision
layers work unchanged. See docs/architecture.md.
Roadmap
v0.1.0 — MVP: UIA capture/detect/click + CLI + MCP + 飞书 first showcase
v0.2.0 — Record & Replay: show-and-prompt trajectory capture, no vision required
v0.2.0 — Parsing-style vision backend (OmniParser v2) + structured-first degradation chain
v0.3.0 — macOS (AXUIElement) + Linux (AT-SPI)
v0.3.0 — Florence-2 vision backend
v0.4.0 — Vimium-style letter hint overlay
v0.5.0 — LLM intent resolver
v0.6.0 — Audit log + replay
v1.0.0 — Rust hot path + cross-platform binaries
Quality gate
Lint, tests, encoding and the MCP contract are one command, and CI runs the same steps on every push:
pwsh -NoProfile -File examples\quality-gate.ps1Step | What it proves |
| the tree is lint-clean under the repo's own rule set |
| every source file is UTF-8 without BOM |
| the MCP server starts and exposes the 19-tool contract |
| the behaviour holds |
Repository layout
openeyes/
├── openeyes/ # main package
│ ├── core/ # windows + capture + detect / click primitives
│ │ ├── windows.py # EnumWindows wrapper
│ │ ├── capture.py # PIL.ImageGrab wrapper
│ │ ├── schema.py # Element / WindowInfo dataclasses
│ │ ├── selector.py # element find / filter
│ │ ├── scene.py # window -> text scene (uia / ocr / opencv)
│ │ ├── scene_text.py # scene -> compact text digest
│ │ ├── actuator.py # mouse / hotkey / type_text façade
│ │ ├── tunnel.py # SSH local forward to a remote browser
│ │ └── hints.py # Vimium-style letter hint overlay
│ ├── backends/
│ │ ├── uia.py # Windows UIA backend (pywinauto)
│ │ └── cdp.py # Edge / Chrome DevTools Protocol backend
│ ├── actuators/
│ │ └── win32.py # Windows mouse/keyboard input
│ ├── cli/ # `eyes` command
│ │ └── main.py
│ ├── record/ # Record & Replay (v0.2.0, no vision required)
│ │ ├── event.py # UIAEvent / InputEvent / Trajectory dataclasses
│ │ ├── recorder.py # Path A: show-and-capture
│ │ ├── replayer.py # Trajectory → OpenEyes actuate() calls
│ │ ├── abstract.py # merge_trajectories DAG + natural-language rendering
│ │ ├── generator.py # Path B: prompt → trajectory (LLM-driven)
│ │ ├── prompt_template.py
│ │ └── export/ # codex_skill / action_receipt / opencua exporters
│ └── mcp/ # MCP server — 20 tools (7 native + 6 browser + 2 structured-first + 1 lifecycle + 2 act-then-prove + 1 discovery + 1 record-replay)
│ └── server.py
├── docs/
│ ├── architecture.md
│ ├── RECORD_DESIGN.md # Record & Replay design rationale
│ ├── capability-contract.md # 19-tool MCP contract (read / write / high-risk)
│ ├── remote-firefox-bidi.md # remote Firefox over SSH runbook
│ └── dsh-web-acceptance.md # manual acceptance guide for dsh web
├── examples/
│ ├── feishu_first_test.py
│ ├── remote_firefox_bidi.py # tunnel + scan/click/type on a remote Firefox
│ ├── record_demo_mock.py # Record: fake elements, all exports
│ ├── record_demo_prompt.py # Record: LLM generates trajectory
│ ├── record_demo_real.py # Record: real Edge window
│ ├── record-out/ # Sample export outputs
│ ├── anyvpn_keepalive.py # + launch-anyvpn-keepalive.ps1
│ ├── mcp-stdio-probe.py # MCP stdio mount smoke test
│ ├── dsh-preflight.ps1 # dsh web preflight (read-only)
│ ├── maintenance-round-probes.ps1 # five-probe patrol orchestrator (read-only)
│ ├── dsh-fetch-stall-probe.py # CDP probe for the dsh fetch-stall symptom
│ ├── dsh-session-diagnostic.py
│ ├── dsh-openeyes-mcp.yml # web profile for dsh
│ └── README.md
├── tests/
│ ├── test_record.py # Record & Replay unit tests (no UI needed)
│ ├── test_smoke.py
│ ├── test_cdp.py
│ ├── test_hints.py
│ ├── test_mcp_contract.py # locks the 19-tool contract
│ ├── test_mcp_stdio_probe.py
│ ├── test_maintenance_round_probes.py
│ ├── test_dsh_mount_contract.py
│ └── test_dsh_fetch_stall_probe.py
├── .codex-plugin/
│ └── plugin.json
├── skills/openeyes/
│ └── SKILL.md
├── pyproject.toml
├── LICENSE # MIT
└── README.mdLicense
MIT — see LICENSE.
Available Tools
23 toolsact_and_verifyA
Act, then prove the act worked. Runs an action and reports ok=true ONLY when the expected state was observed - never merely because the action was delivered. The expectation is read from the live accessibility tree (an element that must appear or vanish, or a control whose label must match), so a click that changed nothing is a miss, not a success.
Inputs: hwnd (required), expect (required), budget, method, +7 more.
anchor_probeA
Judge several OpenCV template anchors against ONE window frame. Read-only, no clicking. Returns matched / ambiguous / missing / unreadable, a per-anchor state map and the frame sha256, so a running-vs-idle decision comes from one capture instead of N racing ones. A minimized window returns WINDOW_MINIMIZED instead of an all-missing verdict.
Inputs: hwnd (required), names (required), app, store, +4 more.
anchor_waitA
Poll anchor_probe until every named anchor is present (until=present, the default) or all are gone (until=absent). Read-only; it never resends a task. Timeout is a verdict (ANCHOR_TIMEOUT), not an exception. Ambiguous counts as visible.
Inputs: hwnd (required), names (required), app, store, +7 more.
app_ensureA
Ensure an app is alive and usable, then prove it. Reports whether the app is running, window-gone or not running; restores a hidden window without stealing focus, or launches it (exe / Store AUMID) and waits until it is actually usable. ok is true only when usability was observed, not merely when a launch was issued.
Inputs: exe, aumid, dry_run, timeout, +3 more.
browser_clickB
Click a single element resolved by --hint, --idx, or --name-contains. Default dry_run=true; set go=true to actually click.
Inputs: go, idx, hint, port, +5 more.
browser_launchB
Launch a dedicated Edge with --remote-debugging-port=PORT. Returns {port, pid, profile_dir}. Reuse --profile-dir to keep cookies across scripts.
Inputs: url, port, seed, backend, +3 more.
browser_scanB
DOM probe the active page; returns interactive elements with Vimium-style letter hints. Each entry carries backend/control_type/name/bbox/center/hint. Use protocol=bidi for Firefox Remote Agent.
Inputs: url, port, protocol, host_header, +4 more.
browser_shotC
Capture the page viewport as PNG. Pass --url-contains to scope capture to one page target.
Inputs: out (required), port, dry_run, protocol, +2 more.
browser_tabsC
List browser page targets on the configured port. Use protocol=bidi for Firefox Remote Agent.
Inputs: port, protocol, host_header.
browser_typeC
Type text into a focused/input element. Optionally pass --hint / --idx / --name-contains to focus a specific element first. press_enter=true submits.
Inputs: text (required), idx, hint, port, +8 more.
capture_windowC
Capture a window (or the full screen if hwnd=0) to PNG.
Inputs: out (required), hwnd, dry_run.
clickB
Click by coordinates OR by element selector. Default dry_run=true (returns target without clicking).
Inputs: x, y, hwnd, regex, +7 more.
click_by_idA
Click an element addressed by its stable automation_id. Use this instead of click-by-name when the visible label changes (a live countdown, a spinner): the automation id is the anchor that does not move. Prefers the UIA Invoke pattern, so it needs no pixel maths, does not move the user's cursor and does not steal focus. Default dry_run=true (resolves the target, clicks nothing).
Inputs: hwnd (required), automation_id (required), budget, method, +2 more.
detect_elementsB
Enumerate interactive elements in a window via the platform accessibility tree. Returns AI-friendly JSON with bbox/center.
Inputs: hwnd (required), depth, regex, restore, +3 more.
gridC
Click the center of a Vimium-style grid cell on the window.
Inputs: col (required), row (required), hwnd (required), cols, +2 more.
hotkeyC
Press a hotkey chord, e.g. 'ctrl+a' or 'alt+F4'.
Inputs: combo (required), dry_run.
list_windowsB
List all visible top-level windows. Optional filters: title_contains / class_name / regex.
Inputs: regex, class_name, title_contains.
native_surfacesA
Report the native ways an application can be driven: a command line it ships, a URL scheme it registers, a registered executable, or a Store AUMID. These are cheaper and more deterministic than driving the window, so look here before clicking. Read-only.
Inputs: app (required), schemes, cli_names, exe_names.
record_replayA
Replay a recorded trajectory (JSONL) and report, per step, whether it was verified / unverified / failed. live=false (the default) sends no input: the actions come back as 'planned'. live=true delivers real keyboard and mouse input and judges each step against the expectation recorded in the trajectory. 'sent' only means delivered - it is not proof of effect, and a step with no recorded expectation stays 'unverified'. Failed steps keep their reason and error in actions[] and in failed_steps[], together with the step's 'activation' receipt (null when no activation was attempted).
Inputs: trajectory_path (required), live, retries, timeout, +2 more.
resolve_elementsA
Walk the structured-first backend chain (uia -> cdp -> vision) and return the first non-empty element set, which tier answered, and why the earlier tiers fell through. Read-only.
Inputs: hwnd, port, origin_x, origin_y, +3 more.
scene_describeA
Describe a window as text: elements with names and coordinates plus a text digest. Use it when the target has no usable accessibility tree (Electron / canvas apps) or when you cannot look at a picture. Every bbox/center is window-relative physical pixels in the grid window.size describes, so a center is a valid click point inside the window; the text digest prints the window's screen origin (@x,y), so absolute = origin + center, added once. source=uia reads the accessibility tree only and never runs OCR; source=ocr reads pixels only; auto blends both. capture.verified=false means the frame could not be proven to be this window, so its OCR and OpenCV results were suppressed; a failed read carries error_code and reason (INVALID_SOURCE, INVALID_REGION, CAPTURE_FAILED, OCR_UNAVAILABLE, UIA_EMPTY, SCENE_FILTERED_EMPTY, UNVERIFIED_FRAME). Read-only: it never clicks or types, and the window may sit behind other windows.
Inputs: hwnd (required), source, max_items, include_shapes.
type_textC
Type a literal string verbatim. Characters are sent as Unicode packets, so an active Chinese IME cannot swallow a URL into pinyin composition.
Inputs: text (required), dry_run, interval.
vision_detectB
Parse screen-parsing detections (OmniParser v2 compatible) into AI-friendly elements with bbox/center. Read-only; no model runs on this host.
Inputs: score, origin_x, origin_y, detections, +2 more.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 23 tools
The descriptions do real work differentiating close pairs (click_by_id vs click vs grid vs browser_click; anchor_probe vs anchor_wait; browser_scan vs detect_elements vs resolve_elements), but there are still four element-enumeration tools and four click paths whose boundaries an agent must infer from prose rather than names. Overlap between scene_describe, capture_window, and browser_shot for 'look at the screen' adds further selection risk.
Everything is snake_case, and prefix families (browser_*, anchor_*) are used consistently, which helps. But the grammar is mixed: verb_noun (click_by_id, capture_window, list_windows, detect_elements), noun_verb (app_ensure, scene_describe, anchor_probe, record_replay), and bare nouns (grid, hotkey, native_surfaces), so the pattern isn't predictable across the set.
23 tools is on the heavy side, and the surface spans three tiers (browser CDP, Windows UIA, vision/anchors) plus verification and replay. Each tool does map to a distinct capability tier, but the count is high enough that an agent must triage before acting.
Core lifecycle is well covered: launch/ensure app, enumerate, click, type, hotkey, capture, verify, and replay. Notable gaps remain — there is no scroll, drag, or navigation/URL-open tool, and no way to close or kill an app or window, which are common dead ends in an automation loop.
Maintenance
Related MCP Connectors
MCP server for your apps' tools and custom tools, plus hosted AI agents and approval-gated workflows
MCP server for progressive tool usage at any scale (see https://klavis.ai)
Remote MCP server for supportsheep: run AI interviews and manage support content for your blog.
Remote MCP server for supportsheep: run AI interviews and manage support content for your blog.
Related MCP Servers
- AlicenseAqualityAmaintenanceWindows desktop automation MCP server — screenshot, mouse, keyboard & UI Automation. Lets LLM agents see and control your Windows desktop directly.30983 npm26MIT
- AlicenseNot gradedqualityCmaintenanceGUI automation MCP server that enables AI agents to see and control the Windows desktop using a local Vision LLM (Ollama), supporting screenshot analysis, mouse/keyboard actions, and autonomous task execution.5MIT
- AlicenseNot gradedqualityBmaintenanceAn MCP server that gives any AI assistant eyes and hands on your desktop — screenshots, clicking, typing, OCR, window management, accessibility-tree queries, workflow recording.5Apache 2.0
- AlicenseAqualityCmaintenanceCross-platform desktop automation MCP server that lets AI agents capture screenshots, run OCR with UI-element classification, control mouse/keyboard, and launch programs on Linux, macOS, and Windows.202MIT