Skip to main content
Glama

reflex-computer-use

reflex-computer-use is an open-source computer use MCP server for Claude Code, Codex and other AI agents. It controls macOS and Windows desktops, including a remote Windows machine over SSH, by reading the accessibility / UI Automation tree instead of taking a screenshot at every step, and uses Jev (TypeSafe's System One decision model; a TypeSafe, OpenRouter or Command Code key works) as a fast judge. Since 0.4, Jev can also look at the screen itself (look: Jev 1.14 image input and boxes) to find icons and things in game scenes. → English README

给编码代理(Claude Code、Codex 等)用的桌面操作(computer use)MCP 服务(原名 reflex-cu):让 AI 自己操作 Mac 和 Windows 的桌面,也能通过 SSH 操作另一台 Windows。

名字里的 reflex 是"反射":除了常规的截图、点击、打字,它把"点哪个""成功了没""加载完没"这类不需要思考的高频小判断,交给一个反射式的快速判断模型去做,代理不用每一步都看截图。目前接的判断模型是 Jev(TypeSafe 的 System One 模型,一次约 0.3 秒)。

0.4 新功能:Jev 能看画面了。 给 find、check、wait 加上 look: true,Jev 直接看截图:能找到纯图标、游戏画面里的人物和物件,也能回答"画面里有没有某样东西"。截图只发给 Jev,不进代理的上下文。目前是实验性功能,只支持 TypeSafe 官方密钥,详见下文看画面。

English

三种用法

用法

被操作的机器

怎么连

本机 Mac

运行代理的这台 Mac

MCP 服务进程内直接调用系统接口

本机 Windows

运行代理的这台 Windows

同上

远程

另一台 Windows

那台机器的桌面会话里跑一个只监听本机的服务,MCP 服务通过 SSH 隧道访问

三种用法的工具、参数、坐标约定完全相同。

Related MCP server: ScreenHand

工具

工具

作用

status

分辨率、前台窗口、用户多久没动鼠标键盘

screenshot

截图,可截窗口或局部

windows / focus

列出窗口、切到前台

observe

不截图,直接读出窗口里的界面元素和它们的坐标;打开着的菜单也会列出

click / move / drag / scroll

鼠标,可以用坐标,也可以用 observe 返回的元素编号

key / type

按键和组合键;输入文字(含中文),可走剪贴板

find

Jev:一句话描述要操作的元素,返回最匹配的那个和置信度,可以直接点;加 look 时看截图找(图标、游戏画面)

check

Jev:对当前屏幕问一个是非题;加 look 时连截图一起看

wait

Jev:轮询直到是非题成立或超时

steps

一次执行一串操作,Jev 没把握或某步失败就停下

带 Jev 标记的三个工具需要你自备 Jev 密钥(见下文);没有密钥时它们不可用,其余工具照常工作。

界面是怎么读出来的

  • Windows 用 UI Automation,Mac 用辅助功能树:能拿到控件的名称、类型、位置和是否禁用,通常 0.1–0.5 秒。

  • 控件的状态会一起读出来:开关是开还是关,下拉框当前选的是什么、有没有展开,哪一项被选中,输入框里填了什么(密码框除外)。几个控件重名时标出各自属于哪个分组。

  • Windows 上,滚动区域里还没滚到的项也会列出;用元素编号点击或用 find 点击时会先自动滚动到它。

  • 读不到东西的窗口(游戏、自绘界面)自动改用文字识别:Windows 用 RapidOCR,Mac 用系统自带的 Vision。

  • 默认只把文字交给 Jev。纯图标、游戏画面这类没有文字的内容,给 find、check、wait 加上 look: true,Jev 会直接看截图(见下一节)。

看画面(look,实验性)

Jev 1.13 起能看图,1.14 又加了"框出某个东西"的问题类型。加上 look: true 后:

  • find 先在整个窗口上粗框一次,再把那一块放大重新框一次,返回中心坐标;confidence 是放大之后 Jev 认为它确实在那里的概率,够高才会点。

  • check、wait 把窗口截图和读出的文字一起交给 Jev 判断。

  • 截图只发给 Jev,不进代理的上下文,代理那边不花看图的钱。

实测(本机,不点击,只比对位置):一张游戏截图上的 8 个目标(人物、灯笼、箭靶、旗帜等)两轮框选 8/8 命中;浏览器工具栏上 8 个小按钮命中 6 个,其中一个低置信度没点,一个框在了同一个标签页的标题上而不是它的关闭按钮。一次 find 约 1~3 秒,check 约 0.6 秒。

在一台 4K 显示器的全屏游戏(黑神话:悟空)上问"画面里有人物吗?":只读文字时 Jev 只拿到 1 个识别出的元素,答 0.06(错);加 look 后答 0.98。

限制:

  • 这是 TypeSafe 标为实验性("Dreamer-ready")的功能,格式和准确度都可能变。紧挨着的小图标仍可能框错,重要的点击先用 check 确认,或者自己看截图。

  • 目前只有 TypeSafe 官方密钥能用。OpenRouter 已经上了 Jev 1.14,但会把图片丢掉、让模型闭着眼回答,所以通过 OpenRouter 或 Command Code 时 look 会直接报错,而不是瞎答。status 的 jev_images 表示当前来源能不能看图。

  • 框选需要 Jev 1.14,默认用 jev-preview;TypeSafe 把 jev-latest 切到 1.14 之后可以用 CU_JEV_VISION_MODEL 改回。

坐标

所有坐标都以不带 zoom 的截图为准:主显示器等比缩放到最宽 1600(环境变量 CU_VIEW_W 可改)。截图上看到的位置就是要点的位置。

安装

想让 AI 替你装:把 INSTALL_FOR_AI.md 的链接发给你的编码代理,让它照着做。下面是手动安装的步骤。

需要 Python 3.10 以上。

快速安装(操作本机 Mac 或 Windows):装了 uv 后,可以直接从 GitHub 源码包注册:

claude mcp add reflexcu -s user -- uvx --from https://github.com/8itlab/reflex-computer-use/archive/refs/heads/main.tar.gz reflex-computer-use
codex mcp add reflexcu -- uvx --from https://github.com/8itlab/reflex-computer-use/archive/refs/heads/main.tar.gz reflex-computer-use

或者 pip install https://github.com/8itlab/reflex-computer-use/archive/refs/heads/main.tar.gz 后用命令 reflex-computer-use 启动。远程操作另一台 Windows 仍按下文"远程 Windows"安装。

Jev 密钥(需自备)

本项目不附带、也不代理任何 Jev 额度。find、check、wait 三个工具要用你自己的密钥,费用由你自己的账户承担。不配置密钥也能用,只是这三个工具会报错,其余工具不受影响。

密钥有三种来源,任选其一:

来源

环境变量

或者写进文件

看画面(look)

TypeSafe 官方

TYPESAFE_API_KEY

~/.reflexcu/typesafe_key

支持

OpenRouter(不需要 TypeSafe 账号)

OPENROUTER_API_KEY

~/.reflexcu/openrouter_key

暂不支持(图片会被丢掉)

Command Code 的 Provider API

COMMANDCODE_API_KEY

~/.reflexcu/commandcode_key

未验证,默认不发图

密钥放在被操作的那台机器上,文件里只写密钥本身(Windows 上 ~ 是 %USERPROFILE%)。配了不止一种时按表里的顺序取第一个;要指定就把 CU_JEV_PROVIDER(或文件 ~/.reflexcu/jev_provider)设成 typesafe、openrouter 或 commandcode。status 的 jev_provider 会显示当前用的是哪一个。

别的兼容同一协议(System One)的网关也能接:用 CU_JEV_URL 和 CU_JEV_MODEL(或文件 ~/.reflexcu/jev_url、~/.reflexcu/jev_model)覆盖地址和模型名,密钥仍按上面的方式放。

如果所在网络访问不了 TypeSafe 的接口(返回 HTTP 451),可以换一种来源(比如 OpenRouter),或者配一个代理:环境变量 CU_PROXY,或文件 ~/.reflexcu/proxy,内容形如 http://127.0.0.1:7890。Windows 上由计划任务启动的远程服务读不到登录之后才设置的环境变量,这种情况用文件。

本机 Mac

git clone https://github.com/8itlab/reflex-computer-use && cd reflex-computer-use
python3 -m venv ~/.reflexcu/venv
~/.reflexcu/venv/bin/pip install -r requirements-macos.txt
claude mcp add reflexcu -s user -- ~/.reflexcu/venv/bin/python "$PWD/reflexcu/server.py"

启动代理的那个应用(终端、IDE)需要"辅助功能"和"屏幕录制"两项权限,status 会报告它们是否到位。

本机 Windows

git clone https://github.com/8itlab/reflex-computer-use
cd reflex-computer-use
python -m venv venv
venv\Scripts\pip install -r requirements-windows.txt
claude mcp add reflexcu -s user -- "%CD%\venv\Scripts\python.exe" "%CD%\reflexcu\server.py"

要操作以管理员身份运行的程序,代理本身也得以管理员身份运行,否则输入会被系统拦掉。

远程 Windows

在被操作的机器上(需要已经能从你的机器 SSH 过去):

git clone https://github.com/8itlab/reflex-computer-use C:\reflexcu
powershell -ExecutionPolicy Bypass -File C:\reflexcu\scripts\install-windows-daemon.ps1

脚本会建虚拟环境、装依赖,并注册一个登录时启动的计划任务。服务必须跑在桌面会话里:从 SSH 直接启动的进程看不到屏幕。

在运行代理的机器上(这一侧只用标准库):

claude mcp add winbox -s user -e CU_SSH=<ssh 别名> -e 'CU_DIR=C:\reflexcu' -- python3 /path/to/reflexcu/reflexcu/server.py

要连多台机器,就各注册一个,并用 CU_LPORT 给每台分一个不同的本地端口。

在 Codex(ChatGPT 桌面应用)里用

它是标准的 stdio MCP 服务,Codex 同样能用。把上面 claude mcp add 换成 codex mcp add,环境变量用 --env:

codex mcp add reflexcu -- ~/.reflexcu/venv/bin/python /path/to/reflex-computer-use/reflexcu/server.py
codex mcp add winbox --env CU_SSH=<ssh 别名> --env 'CU_DIR=C:\reflexcu' -- python3 /path/to/reflex-computer-use/reflexcu/server.py

只读工具(status、screenshot、windows、observe、check、wait)带有只读标注,codex exec 这类非交互运行可以直接调用;会动鼠标键盘的工具需要你在交互界面里批准。在 Mac 上,系统权限是按启动服务的应用给的:从 ChatGPT 应用里用,就要给 ChatGPT 开"辅助功能"和"屏幕录制"。

配置

环境变量

用在哪

含义

CU_SSH

MCP 服务

不设或 local 是本机;否则是远程机器的 SSH 别名

CU_DIR

MCP 服务

远程机器上的安装目录,用来读 token.txt(默认 C:\reflexcu)

CU_TOKEN

MCP 服务

直接给出令牌,不通过 SSH 读取

CU_LPORT / CU_RPORT

MCP 服务

隧道的本地端口(默认 18765)和远程服务端口(默认 8765)

CU_PORT

远程服务

监听端口(默认 8765)

CU_VIEW_W

后端

坐标系宽度(默认 1600)

TYPESAFE_API_KEY

后端

你自己的 TypeSafe 密钥;也可以写在 ~/.reflexcu/typesafe_key

OPENROUTER_API_KEY

后端

通过 OpenRouter 用 Jev 时的密钥;也可以写在 ~/.reflexcu/openrouter_key

COMMANDCODE_API_KEY

后端

通过 Command Code 用 Jev 时的密钥;也可以写在 ~/.reflexcu/commandcode_key

CU_JEV_PROVIDER

后端

typesafe、openrouter 或 commandcode;不设时按这个顺序用第一个配了密钥的。也可以写在 ~/.reflexcu/jev_provider

CU_JEV_URL / CU_JEV_MODEL

后端

覆盖接口地址和模型名,用来接别的兼容网关;也可以写在 ~/.reflexcu/jev_url、~/.reflexcu/jev_model

CU_JEV_VISION_MODEL

后端

look 用的模型(默认 jev-preview);也可以写在 ~/.reflexcu/jev_vision_model

CU_JEV_IMAGES

后端

设成 1 时,即使当前来源没被确认支持图片也照样发图(给已经支持图片的兼容网关用)

CU_PROXY

后端

访问 Jev 用的代理,例如 http://127.0.0.1:7890;也可以写在 ~/.reflexcu/proxy。系统代理不会被自动使用

命令行调试

server.py 带参数时直接执行一个工具并打印结果:

python reflexcu/server.py status
python reflexcu/server.py find '{"goal":"保存按钮"}'
python reflexcu/server.py find '{"goal":"右上角的齿轮图标","look":true}'
python reflexcu/server.py check '{"question":"画面里有没有弹出的对话框?","look":true}'
CU_SHOT=shot.jpg python reflexcu/server.py screenshot

tests/local_smoke.py 通过真实的 MCP 协议做一遍只读检查;tests/selftest_mac.py 在一个临时的"文本编辑"文档里测鼠标键盘。

安全

这个工具能看到屏幕、能控制鼠标键盘,请按这个分量对待它。

  • 远程服务只监听 127.0.0.1,每个请求都要带 token.txt 里的令牌,对外只走 SSH 隧道。不要把端口暴露到网络上。

  • 远程服务以最高权限运行,这样才能操作管理员窗口。能访问那个端口和令牌的人就能控制那台机器。

  • find、check、wait 会把屏幕上的文字发给 TypeSafe 的 Jev 接口(用 OpenRouter 或 Command Code 的密钥时经由它们转发)。加了 look 时还会发送那个窗口的截图。

  • 用 OpenRouter 的密钥时,请求会带上本项目的名称和仓库地址(OpenRouter 的应用归属标头,用于它公开的应用榜单),不含你的任何信息;设 CU_NO_ATTRIBUTION=1 可以关掉。屏幕上有敏感内容时不要用这三个工具,或者不配置密钥。

  • 输入是发给前台窗口的。先切对窗口再打字,否则按键会进别的程序;有人正在用那台机器时,status 的 idle_seconds 会很小。

  • 带内核反作弊的在线游戏可能把模拟输入当作外挂,请自行判断,不要在这类游戏里使用。

已知限制

  • 只操作主显示器。

  • look 是实验性功能:紧挨着的小图标可能框错,Jev 偶尔对框错的位置也给出高置信度;重要的点击先确认。分辨率很高的窗口更慢:4K 全屏游戏上一次看图判断约 8 秒,其中文字识别约 4 秒;游戏里只想看画面时加 source: "tree" 跳过文字识别,约 4 秒。

  • key 发的字符键会经过输入法;输入文字用 type。key 的长按只是保持按下,不会连发字符。

  • Mac 上打开的菜单靠文字识别读取,没有禁用状态;Windows 上经典菜单和归属于该窗口的弹出层走 UI Automation。

  • 远程模式目前只有 Windows 的安装脚本。Mac 后端也能跑 daemon.py,但没有测过。

  • 游戏里的相对移动(转镜头)没有系统测试过。

实测数据

都是同一个代理(Claude Opus 5.5)、同一句任务描述,只换工具:一组用 reflex-computer-use 的全部工具,另一组把同一个服务限制成只有截图、点击、按键(CU_TOOLS=screenshot)。对照组是本项目自己的截图模式,不是别家产品。每格只跑了 2 次,数字只能看量级。

任务

reflex-computer-use 全部工具

只用截图和点击

Mac,只说一句话:"打开 ChatGPT 应用"

8 / 9 秒,4 次调用

18 / 30 秒,7 / 12 次调用

Mac,只说一句话:"打开计算器,算出 1234 乘以 56"

23 / 22 秒,5–6 次调用

35 / 38 秒,21 次调用

Mac,只说一句话:"打开系统设置,查 macOS 版本号"

25 / 24 秒,4 次调用

31 / 26 秒,11–13 次调用

远程 Windows,在"设置"里连改五项

61 / 88 秒,11–15 次调用,约 $0.27

54 / 72 秒,28–38 次调用,$0.40–0.56

短任务上更快,调用次数少一半以上;步骤很多的长任务上速度没有优势,但花费更低。

常见问题

reflex-computer-use 是什么? 一个标准的 stdio MCP 服务,给 AI 代理提供"看屏幕、动鼠标键盘"的工具,也就是通常说的 computer use。它装在你自己的机器上,由你已经在用的代理(Claude Code、Codex、ChatGPT 桌面应用里的 Codex)来调用。

和模型自带的 computer use 有什么不同? 常见做法是每一步截一张图交给大模型看。reflex-computer-use 优先直接读出界面上的控件(名称、状态、坐标),只有图标、游戏画面这类没有文字的内容才看图,而且看图找东西也可以交给 Jev(look),截图不必进大模型的上下文;"点哪个""成功了没"这类小判断交给 Jev。它还能通过 SSH 操作另一台 Windows 的真实桌面。

没有 Jev 密钥能用吗? 能。截图、读界面、鼠标、键盘都不需要密钥,只有 find、check、wait 三个工具需要。

Jev 能看图吗? 能,Jev 1.13 起支持图片,目前还是实验性功能。reflex-computer-use 0.4 起用它来找图标和游戏画面里的东西(look),只支持 TypeSafe 官方密钥,见上文"看画面"。

Jev 的密钥从哪来? TypeSafe 官方、OpenRouter、Command Code 三选一,见上文"Jev 密钥"。在访问不了 TypeSafe 的网络里,可以用 OpenRouter 的密钥。

支持哪些系统和客户端? 被操作的机器:macOS、Windows;远程模式目前只支持操作 Windows。客户端:在 Claude Code 和 Codex(含 ChatGPT 桌面应用)里实测过,其他支持 stdio MCP 的客户端按同样方式注册。

代码结构

reflexcu/core.py     与平台无关:合并界面树和文字识别、Jev 判断、各个工具的实现
reflexcu/win.py      Windows 后端
reflexcu/mac.py      Mac 后端
reflexcu/server.py   MCP 服务(stdio):本机直连,或经 SSH 隧道转发
reflexcu/daemon.py   远程模式下跑在被操作机器上的 HTTP 服务

许可证

MIT

Available Tools

15 tools
checkA
Read-only

【Jev】对当前屏幕上的文字问一个是非题,返回概率。用来确认上一步有没有成功。问的是画面本身(游戏里、图片里、颜色)时加 look=true。

NameRequiredDescriptionDefault
lookNo让 Jev 直接看截图,而不只是读文字:图标、游戏画面、图片、颜色这类没有文字的内容用它。截图只发给 Jev,不进你的上下文。目前只有 TypeSafe 官方密钥支持(status 里 jev_images=true),每次多 1~3 秒
sourceNoauto:先读界面树,读不到(游戏、自绘界面)再用文字识别
windowNo窗口标题或应用名的一部分,或 windows 返回的 hwnd;不填就是前台窗口
questionYes例如 "保存对话框是否已经打开?"
thresholdNo

TDQS

A4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true and openWorldHint=false, so the safety profile is covered. The description adds genuinely new behavior: it returns a probability rather than a definitive answer, and there is no output schema to otherwise convey that. Cost/latency caveats for look live only in the schema, so the description is not fully self-sufficient.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three short sentences: what it does and returns, the verification use case, then the mode switch. Front-loaded with the core capability and zero filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With five parameters, one enum, and no output schema, the description covers the essential contract: a probability answer about current on-screen text plus the visual-mode switch. It leaves the source/window/threshold parameters entirely to the schema, which is acceptable at this coverage level but slightly thin for a tool whose accuracy depends on which window is read.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 80%, so the enum and look semantics are already well documented in the schema. The description's look=true clause largely restates that schema text rather than adding new nuance, and it says nothing about source, window, or threshold. Baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

Names a specific action (ask a yes/no question about on-screen text) and the return type (probability), which is more precise than a generic 'observe'/'check screen'. It never names a sibling tool to contrast with, so the agent must infer that this differs from observe or screenshot rather than being told.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives a concrete usage context: verifying whether the previous step succeeded. It also gates a mode (look=true when the subject is the picture itself: in-game, images, colors). It stops short of stating when NOT to use this tool or naming the alternative action tool for those cases.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

clickC

点击坐标或元素 id。

NameRequiredDescriptionDefault
xNo
yNo
idNo上一次 observe/find 返回的元素 id,代替 x/y
holdNo按住秒数
countNo2 是双击
buttonNo

TDQS

C2.8/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=false, destructiveHint=false and openWorldHint=false, so the safety profile is covered. The description adds no behavior beyond that: no mention of whether it waits for the target to be actionable, what happens on a miss, whether held clicks fire continuously, or whether the call blocks.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness3/5

Is the description appropriately sized, front-loaded, and free of redundancy?

It is a single front-loaded clause with no filler, which is structurally clean. However, one sentence is too thin for a six-parameter action tool, so brevity here reads as under-specification rather than economy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With six optional parameters, no required args, no output schema and no annotations about side effects, the description should explain targeting precedence, what a click requires from prior calls, and the outcome. None of that is present, so an agent must reverse-engineer behavior from parameter names.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 50%, with `id`, `hold` and `count` documented in-schema while `x`, `y` and `button` are not. The description's "coordinates or element id" usefully signals that x/y and id are alternative targeting modes, but it says nothing about hold, count or mouse button, leaving that half to the schema alone.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description names a specific action and its two targeting modes: click by coordinates or by element reference. An agent can tell it apart from siblings like move, drag and scroll, though it does not explicitly contrast with check, which is the nearest overlapping verb.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no when-to-use guidance at all: nothing says to prefer `id` over x/y, nothing says to call observe/find first, and nothing distinguishes a plain click from `check` or a double click. Usage is only inferable from the parameter names in the schema.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

dragD

按住拖动。

NameRequiredDescriptionDefault
x0Yes
x1Yes
y0Yes
y1Yes
buttonNo
secondsNo

TDQS

D1.8/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=false, destructiveHint=false, and openWorldHint=false, so the safety profile is covered. The description adds nothing beyond that — not whether it needs focus, whether the drag is visible, or how timing/button affect behavior. No contradiction with annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness2/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Four characters are technically concise, but this is under-specification rather than economy. There is no waste, yet nothing is front-loaded because there is nothing to load.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness1/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

A GUI automation tool with six parameters, four of them required and none documented by the schema, demands far more than "press and drag." The agent cannot determine coordinate semantics, button defaults, or drag duration from this definition.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters1/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0% across six parameters. The description does not explain that x0/y0 is the start point and x1/y1 the end point, whether coordinates are screen or window relative, what values 'button' accepts, or what 'seconds' controls.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose3/5

Does the description clearly state what the tool does and how it differs from similar tools?

"按住拖动" (press and drag) names a specific action and implicitly distinguishes itself from the sibling "move" by indicating the button is held down. However, it never states that it drags from one point to another, nor what it operates on (desktop, window, element).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines1/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no guidance on when to use this instead of the sibling tools move, click, or scroll, and no prerequisites or exclusions are mentioned. The agent must infer everything from the name alone.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

findA

【Jev】用一句话描述要操作的东西,Jev 在界面元素里选出对应的那个并给出置信度。click=true 时置信度够就直接点。约 0.3 秒,不需要截图。默认只认得有文字或有名称的元素;纯图标、游戏画面里的东西加 look=true,Jev 会看截图框出它(先整窗粗框、再放大局部细框,约 2~5 秒),confidence 是放大后它确实在那里的概率。还不行再用 screenshot 自己看。

NameRequiredDescriptionDefault
goalYes例如 "保存按钮"、"把语言改成中文的那个下拉框"
lookNo让 Jev 直接看截图,而不只是读文字:图标、游戏画面、图片、颜色这类没有文字的内容用它。截图只发给 Jev,不进你的上下文。目前只有 TypeSafe 官方密钥支持(status 里 jev_images=true),每次多 1~3 秒
clickNo
countNo
buttonNo
regionNolook=true 时只在这块区域里找
sourceNoauto:先读界面树,读不到(游戏、自绘界面)再用文字识别
windowNo窗口标题或应用名的一部分,或 windows 返回的 hwnd;不填就是前台窗口
min_confidenceNo默认 0.75

TDQS

A3.9/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Beyond the annotations (readOnly=false, destructive=false, openWorld=false), the description discloses latency (about 0.3s vs 2–5s for look), that look mode two-passes the screenshot (coarse window box, then zoomed detail), that screenshots are sent only to Jev and not into the caller's context, and that look requires an official key (status jev_images=true). It leaves open what is returned on failure or when confidence falls below min_confidence.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The core capability is front-loaded in the first clause, followed by the fast path, the visual path, and the fallback in a logical escalation order. It is a dense run-on paragraph with several chained qualifiers, but almost every clause carries operational information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 9-parameter tool with no output schema, the description explains the main modes and confidence semantics but does not say what the response contains (element handle, coordinates, ranked candidates, failure shape) or what the remaining parameters do. An agent can invoke it but may be unsure how to consume or retry the result.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

At 67% schema description coverage the description still carries extra meaning: it explains the click=true behavior and its confidence gate, the semantics of confidence as a post-zoom probability for look mode, and region's role as a look-mode search restriction. The `count` and `button` parameters remain undocumented in both description and schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a concrete verb and resource: describe the target in one sentence and Jev selects the matching UI element and returns a confidence. It also distinguishes the fast text/tree path from the visual path and names `screenshot` as an escape hatch. It does not distinguish itself from the sibling `click` or `observe` tools, which overlap in intent.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Clear routing rules are given for the modes: default text/name matching, add look=true for icons/game visuals/images/colors, escalate to `screenshot` if that fails. What is missing is guidance on when to call `find` versus the sibling `click` tool, given that click=true makes this tool perform clicks directly.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

focusC

把窗口切到前台。

NameRequiredDescriptionDefault
windowYes窗口标题或应用名的一部分,或 windows 返回的 hwnd;不填就是前台窗口

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=false, destructiveHint=false, openWorldHint=false, so the safety profile is covered. The description adds nothing beyond them: it does not say what happens if the window is minimized, absent, or ambiguous, and it does not state whether focus steals input or fails silently. For a side-effecting window action, that gap is meaningful.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

It is a single short sentence with zero waste and the action is front-loaded. It borders on under-specification given what it omits, but nothing in it is redundant.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool is simple, has one parameter with full schema coverage, and has no output schema, so there is little that must be explained. Still, for an action with side effects on window state, the description should say more about failure modes and ordering relative to windows or click.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the window parameter's accepted forms (title fragment, app name, or hwnd from windows) are already documented in the schema. The description adds no meaning beyond that, so the baseline of 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a concrete action on a specific resource using a verb and resource (切到前台 / window), which an agent can easily distinguish from siblings such as click, move, or windows. It does not explicitly differentiate itself from the sibling status, but its purpose is clear.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives no guidance on when to use focus rather than windows or status, no prerequisites, and no exclusions. An agent must infer that this is the call used to activate a window before sending input to it.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

keyA

按键或组合键,如 "ctrl+s"(Mac 上是 "cmd+s")、"esc"、"w"。hold 是按住秒数(游戏里走路用),times 是重复次数。字符键会经过输入法,输入文字请用 type。

NameRequiredDescriptionDefault
holdNo
keysYes
timesNo

TDQS

A4.6/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations (readOnlyHint=false, destructiveHint=false) correctly signal a non-read, non-destructive action, and the description adds real behavioral context beyond them: the IME interception of character keys and the game-walking use of hold. It stops short of explaining which window receives the keystroke (focus dependency) or what the call returns, but the essential behavior is disclosed.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the action and examples, then parameter meanings, then the sibling redirect — three tight sentences with no wasted wording. Every clause carries information an agent needs.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple input-simulation tool with no output schema, the description covers purpose, all parameters, and the alternative tool, which is nearly everything needed. The one gap relative to the sibling set is not mentioning the `focus` dependency, i.e., that the target window must be focused first.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must carry all parameter meaning, and it does: `keys` is a key or combination with examples, `hold` is seconds held down, `times` is repeat count. Units and intent are clear for all three; only edge cases (e.g., valid key names) are left implied.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource — press a key or key combination — with concrete examples ("ctrl+s", "esc", "w") and Mac variant. It explicitly distinguishes itself from the sibling `type` for text entry, so an agent can tell the two apart without opening either schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives explicit routing: character keys go through the input method, so use `type` for entering text. It also clarifies the purpose of `hold` (walking in games) and `times` (repeat count), telling the agent when those options apply rather than leaving it to guesswork.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

moveB

移动鼠标。relative=true 时 x/y 是相对位移(用来转游戏镜头)。

NameRequiredDescriptionDefault
xNo
yNo
idNo上一次 observe/find 返回的元素 id,代替 x/y
relativeNo

TDQS

B3.4/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly=false, destructive=false, openWorld=false, so the safety profile is covered. The description adds that relative mode rotates the in-game camera, which is meaningful state-changing context, but it omits whether motion is instant or animated, what coordinate space absolute x/y uses, and any dependency on prior focus.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two short sentences with the core action front-loaded and the parameter caveat immediately after. No filler, though the brevity leaves gaps that are penalized in other dimensions rather than being padding.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 4-parameter input tool with no output schema and sparse schema descriptions, the description covers the trickiest case (relative movement for camera control) but should also specify the absolute coordinate frame and default behavior for `relative`. It is minimally sufficient but not complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is only 25% (only id documented), so the description must compensate. It does clarify the crucial x/y semantics in relative mode, but leaves absolute-mode coordinates undefined and does not state the default for `relative` or how id interacts with x/y when both are supplied.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb+resource ("移动鼠标" / move the mouse), which is unambiguous and clearly distinct from siblings like click, drag, and scroll. It does not explicitly name a sibling to contrast with, but the purpose is immediately clear.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It supplies one useful conditional use case: relative=true is for turning the game camera. However, it gives no guidance on when to move the cursor versus using click/drag directly, nor any prerequisites such as needing a target window in focus first.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

observeA
Read-only

读出窗口里的界面元素(名称、类型、中心坐标),不经过截图。开关带开/关,下拉框带当前值和是否展开,重名的控件标出所属分组;打开着的菜单和下拉选项也会列出(popup=true)。滚动区域里还没滚到的项也会列出(offscreen=true,没有坐标):用 id 点击或 find 点击时会先自动滚动到它。filter 按文字过滤,多个词用 | 隔开。

NameRequiredDescriptionDefault
limitNo
filterNo
regionNo
sourceNoauto:先读界面树,读不到(游戏、自绘界面)再用文字识别
windowNo窗口标题或应用名的一部分,或 windows 返回的 hwnd;不填就是前台窗口

TDQS

A4.3/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations only declare readOnlyHint and openWorldHint; the description adds substantial behavior beyond that — toggles report on/off state, dropdowns report current value and expanded state, duplicate-named controls are tagged with their group, offscreen items appear without coordinates, and clicking by id auto-scrolls first. This is exactly the kind of semantics an agent needs before deciding to observe-then-click.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loads what the tool returns, then layers behavior details in a compact single paragraph with semicolon-separated clauses. Dense but every clause carries information; it would read better as a short bullet list given the number of distinct behaviors.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

No output schema exists, so the description carries the burden of describing return content, and it does so well (fields, state attributes, coordinate presence/absence). Remaining gaps are `limit`/`region` semantics and ordering/pagination, which matter less for a zero-required-parameter read tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is only 40%, and the description compensates for `filter` by explaining its semantics (text filtering, multiple terms separated by `|`). However `limit` and `region` are documented in neither the schema nor the description, leaving two of five parameters opaque.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource — reading out UI elements (name, type, center coordinates) in a window — and immediately distinguishes itself from the sibling `screenshot` with "不经过截图". An agent knows exactly what it gets back without opening the schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives concrete situational guidance: `source=auto` reads the UI tree first and falls back to OCR for games/self-drawn UIs, `popup=true` surfaces open menus, `offscreen=true` surfaces unscrolled items. It stops short of explicitly saying when to prefer `screenshot` or `find` instead, so the alternative-selection rule is implied rather than stated.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

screenshotA
Read-only

截图。所有坐标都以不带 zoom 的截图为准。region 截局部;zoom=true 按原始分辨率截(只用来看清小字)。

NameRequiredDescriptionDefault
zoomNo
regionNo[x0,y0,x1,y1]
windowNo窗口标题或应用名的一部分,或 windows 返回的 hwnd;不填就是前台窗口

TDQS

A3.9/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already establish readOnlyHint=true and openWorldHint=false, so safety is covered. The description adds genuinely non-obvious behavior: all coordinates are defined against the non-zoom screenshot, so a zoomed capture must not be used for coordinate reference, and zoom is explicitly view-only for small text. That is exactly the kind of trap an agent would otherwise fall into.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three short clauses, zero redundancy, with the most error-prone constraint (coordinate reference frame) front-loaded. Every sentence earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a three-parameter, all-optional capture tool with no output schema, the description covers the risky semantics: coordinate frame, region behavior, and zoom's limited purpose. Only the default capture behavior when window is omitted is left to the schema, which handles it.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 67%, with region and window already documented in the schema. The description compensates for the remaining gap by defining the coordinate reference frame (non-zoom screenshot), clarifying that region means partial capture, and pinning down zoom as original-resolution viewing only. This adds meaning well beyond the bare types.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a concrete verb+resource ('截图' / screenshot) and immediately qualifies scope with 'region 截局部'. It is clear what the tool does, but it never contrasts itself with nearby state-reading siblings like observe, find, or check, so the agent must infer when a screenshot is the right call.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives conditional guidance for the zoom and region parameters ('zoom=true 按原始分辨率截(只用来看清小字)'), which is real when-to-use advice at the parameter level. However, there is no statement about when this tool should be preferred over observe/windows/status, and no exclusions or prerequisites.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

scrollC

滚轮。dy>0 向下,单位是格。

NameRequiredDescriptionDefault
xNo
yNo
dxNo
dyNo
idNo上一次 observe/find 返回的元素 id,代替 x/y

TDQS

C2.9/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=false, openWorldHint=false and destructiveHint=false, so the safety profile is covered. The description adds genuinely new behavioral detail — that dy>0 means downward and the unit is grid cells — but says nothing about whether scrolling targets the page or a specific element, whether it is animated or how far a unit actually travels.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two short sentences with the core meaning front-loaded and no filler. The terseness borders on under-specification for a five-parameter tool, but nothing written is wasted.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with five parameters, zero required fields, low schema coverage and no output schema, the description is far too thin. An agent still cannot tell whether x/y target an element or the viewport, what dx does, what the defaults are, or what the call returns.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is only 20% (only id is documented, as an alternative to x/y), so the description must carry the load. It clarifies dy's sign and unit but leaves x, y, dx, their defaults, and whether the unit applies to dx as well completely unexplained.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

"滚轮" (scroll wheel) names a specific action, and the dy direction convention makes the intent unambiguous versus siblings like click, move, drag or key. It stops short of stating what is being scrolled (page vs. element under the cursor), so it is clear but not fully differentiated.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no guidance on when to use scroll versus observe/screenshot/find, nor any prerequisite such as needing a prior observe for the id parameter. The only usable hint is the sign convention for dy, which is direction semantics rather than when-to-use guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

statusA
Read-only

机器状态:分辨率、坐标系、前台窗口、用户多久没动鼠标键盘(idle_seconds)。操作前先看 idle_seconds,用户正在用电脑时不要抢鼠标。

NameRequiredDescriptionDefault

No parameters

TDQS

A3.9/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With readOnlyHint=true and openWorldHint=false already declaring the safety profile, the description's added value is the idle_seconds signal and the associated 'don't steal the mouse' policy, which is genuine behavioral context. It says nothing about cost, latency, or freshness of the reported values, so it is useful but not rich.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two tight sentences: the first lists the returned signals, the second front-loads the actionable constraint. No filler and no repetition of the tool name.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

There is no output schema and no parameters, so the description carries the burden of explaining what comes back, which it largely does by enumerating the four fields and naming idle_seconds explicitly. It stops short of giving types or units (e.g. resolution format, idle_seconds granularity).

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Zero parameters, so the baseline is 4; there is no parameter meaning for the description to carry. The prose does describe the conceptual outputs (idle_seconds, foreground window), which partially substitutes for the absent output schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description names the resource (machine status) and enumerates the specific fields returned: resolution, coordinate system, foreground window, and idle_seconds. This lets an agent separate it from sibling state-readers such as screenshot, windows, focus, and observe, though the opening is a bare noun phrase rather than an explicit verb.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives an explicit when-to-use rule — check idle_seconds before acting — plus a concrete exclusion: don't take over the mouse while the user is actively using the computer. It doesn't name which sibling to call instead (e.g. observe or screenshot) when idle, so it falls short of full alternative routing.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

stepsA

一次执行一串操作,中途不回来问。每步是 {"op": 上面任一工具名(screenshot 除外), ...该工具的参数},例如 [{"op":"find","goal":"个性化","click":true},{"op":"wait","question":"是否出现了颜色这一项?"},{"op":"find","goal":"颜色","click":true},{"op":"observe","filter":"模式"}]。find 没把握、wait/check 不成立或出错时立刻停下并返回已做到哪一步。最后一步是 observe 时会带回读到的元素。适合路径明确的多步操作。

NameRequiredDescriptionDefault
pauseNo每次动鼠标键盘之后的停顿秒数,默认 0.35
stepsYes

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The annotations state that the tool is non-readOnly and non-destructive, but the description adds the important behavior: it stops immediately and reports the step reached when find is uncertain or wait/check fails. It also states that a trailing observe returns the elements it read, and it documents how the pause parameter is applied after each mouse or keyboard action.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is dense but efficient: the core behavior comes first, followed by the step format and an example, then the failure and return semantics. The op list refers back to sibling tools instead of repeating the tool names, which is a saving.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

There are only two parameters and no output schema, so the remaining burden is the step structure and the failure behavior. Both are covered. The main omission is the overall shape of the return value when the batch succeeds or fails early.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema coverage is only 50%; the steps array is typed only as a generic object and has no schema description. The description compensates by defining each step's shape ({"op": <tool name>, ...that tool's params}) and by providing a concrete four-step example. It does not enumerate every valid op or its required fields.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb and resource: it executes a batch of operations in one call and deliberately avoids returning midway to ask. It is clearly distinguishable from the atomic siblings (click, type, find) because it batches them.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives explicit usage guidance: "适合路径明确的多步操作", and it excludes screenshot from valid ops. However, it gives no guidance on when not to use it, for example when later steps depend on the result of an earlier one.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

typeB

输入文字(支持中文)。paste=true 走剪贴板粘贴;enter=true 输完按回车。

NameRequiredDescriptionDefault
textYes
enterNo
pasteNo

TDQS

B3.4/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare this is a non-read-only, non-destructive, non-open-world action, so the safety profile is covered. The description adds genuinely useful mechanism detail: two distinct input paths exist and the paste path goes through the system clipboard. It omits the consequence of that mechanism (clipboard contents are overwritten) and does not say where the text is delivered (currently focused control).

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two short clauses, front-loaded with the primary action, and each clause maps to a real parameter. Nothing is wasted, though the terseness borders on telegraphic and a single added sentence about the focus requirement would cost little.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple three-parameter tool with annotations covering safety and no output schema, the essentials are mostly there. What an agent still cannot infer is the precondition (a focused input must exist) and the side effect of the clipboard mode, both of which matter for correct invocation in a GUI automation flow.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description carries the burden, and it explains the semantics of `paste` and `enter` but leaves `text` undocumented (its format, length limits, or escaping behavior) and never states that both flags default to false. Two of three parameters gain meaning; the core required one does not.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific action and resource: input text, with the added note that Chinese characters are supported. It does not distinguish this from the sibling tool `key`, which is the most likely point of confusion (typing a string vs. sending key combos), so differentiation is left to inference.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The two clauses give conditional guidance on the optional flags (paste=true for clipboard path, enter=true to submit afterward), which implies when to pick each mode. However, there is no statement of prerequisites, e.g. that a target window must already be focused via `focus`, and no explicit when-not guidance versus `key`.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

waitA
Read-only

【Jev】反复看屏幕,直到是非题成立或超时(默认 10 秒,最长 120 秒)。等加载、等弹窗时用,不用一遍遍截图。

NameRequiredDescriptionDefault
lookNo让 Jev 直接看截图,而不只是读文字:图标、游戏画面、图片、颜色这类没有文字的内容用它。截图只发给 Jev,不进你的上下文。目前只有 TypeSafe 官方密钥支持(status 里 jev_images=true),每次多 1~3 秒
sourceNoauto:先读界面树,读不到(游戏、自绘界面)再用文字识别
windowNo窗口标题或应用名的一部分,或 windows 返回的 hwnd;不填就是前台窗口
timeoutNo
intervalNo
questionYes
thresholdNo

TDQS

A3.9/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true and openWorldHint=false, so safety is covered. The description adds real behavioral context — polling loop, 10s default, 120s ceiling — but never says what happens on timeout (false result vs. error) or what the call returns, which matters for a blocking poll.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two tight sentences: the mechanism and its limits come first, the usage cue second. No filler, and the anti-pattern (repeated screenshots) is packed into the same clause rather than a separate sentence.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 7-parameter polling tool with no output schema and 43% coverage, the description leaves notable holes: return value on success vs. timeout, and the role of interval and threshold. The core loop and timeout bounds are covered, so it is adequate but not complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is only 43%, and the description compensates partially: it explains that 'question' is a yes/no predicate and supplies the default and maximum for 'timeout', which the schema leaves undocumented. However, 'interval' and 'threshold' get no meaning from either source, so the gap is only half closed.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and mechanism: repeatedly inspect the screen until a yes/no question holds or a timeout fires. Defaults (10s, max 120s) are included, and the closing clause distinguishes it from the screenshot sibling by saying it replaces repeated screenshots.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives concrete usage conditions — waiting for loading, waiting for a popup — and implicitly routes the agent away from manual repeated screenshots. It does not name an explicit when-not case or name a sibling tool like observe or check, which keeps it short of a 5.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

windowsA
Read-only

列出可见窗口(标题、进程、位置、是否前台)。

NameRequiredDescriptionDefault

No parameters

TDQS

A3.7/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The annotations already declare readOnlyHint=true and openWorldHint=false, so the safety profile is covered. The description adds value beyond that by disclosing the visible-window scope and the exact fields returned in each entry, which matters since no output schema exists. It omits ordering and whether minimized/hidden windows are excluded, keeping it below a 5.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single sentence with the action front-loaded and the return fields compactly parenthesized; every token earns its place and nothing is repeated from the schema.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a trivial zero-parameter read tool with annotations and no output schema, listing the returned fields is exactly the context an agent needs. Minor gaps remain around result ordering and whether the list is filtered to normal windows, but nothing essential is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool takes zero parameters and the schema is an empty object, so there is nothing for the description to disambiguate. The baseline for a no-parameter tool applies, with no contradictory parameter language present.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description gives a specific verb+resource ('list visible windows') and even enumerates the returned fields (title, process, position, foreground status), so an agent knows exactly what it gets. It does not explicitly distinguish itself from siblings such as 'focus', which operates on the same domain, so it stops short of a 5.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no statement of when to use this tool versus alternatives, no prerequisite (e.g. that 'focus' requires a window target that this tool supplies), and no exclusions. The intended use case is only inferable from the enumeration semantics.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 15 tool updatesv0.1.0
    • First observedcheck
    • First observedclick
    • First observeddrag
    • First observedfind
    • First observedfocus
    • First observedkey
    • First observedmove
    • First observedobserve
    • First observedscreenshot
    • First observedscroll
    • First observedstatus
    • First observedsteps
    • First observedtype
    • First observedwait
    • First observedwindows

TDQS

B3.3/5.0

Scored across 15 tools

Disambiguation4/5

Each tool targets a distinct action, and the descriptions give clear guidance on when to use each, for example screenshot compared with observe, or find compared with check and wait. The only minor overlaps are that find can also click, and that screenshot, observe, and check all read the screen, but their descriptions make the boundaries clear.

Naming Consistency5/5

Every name is a single lowercase English word with no mixed casing or naming styles. Although the set mixes nouns and verbs, that mix is conventional for a desktop-control API and is highly predictable.

Tool Count4/5

The 15 tools are tightly scoped to input, screen reading, and verification, so each one earns its place. It is at the upper end of the ideal range, but nothing feels redundant or bloated.

Completeness4/5

The surface covers window management, mouse and keyboard input, element inspection, AI-assisted locating, verification, waiting, and multi-step macros. Minor gaps remain, since there is no explicit app-launch or close-window tool and no clipboard read; however, these can be worked around with key, type, and focus.

Maintenance

ActivityMaintained
ResponsivenessNo recent community issues

Related MCP Connectors

Related MCP Servers

  • A
    license
    Not graded
    quality
    F
    maintenance
    An experimental MCP server providing full control over the macOS user interface through mouse, keyboard, and window management tools. It enables AI assistants to automate desktop tasks by utilizing native accessibility APIs and OCR for real-time screen comprehension.
    7
    Creative Commons Zero v1.0 Universal
  • A
    license
    Not graded
    quality
    D
    maintenance
    An open-source MCP server for macOS and Windows that provides native desktop control via Accessibility APIs, OCR, and Chrome CDP. It enables AI agents to interact with applications, manage browser sessions, and automate workflows with high-speed native UI actions.
    35 npm
    15
    AGPL 3.0
  • A
    license
    Not graded
    quality
    B
    maintenance
    An MCP server that gives any AI assistant eyes and hands on your desktop — screenshots, clicking, typing, OCR, window management, accessibility-tree queries, workflow recording.
    5
    Apache 2.0