Skip to main content
Glama
liufeicc

cc-computer-use

by liufeicc

get_screen_text

Read on-screen text and coordinates to locate and click UI elements, even in apps without accessibility support.

Instructions

读取屏幕上的文字及其坐标,返回紧凑文本列表(每行 [ref] 文字 @ (x,y) 宽x高)。灰区应用(无元素树:SWT/Java、自绘控件、游戏、远程桌面)首选感知方式——它比 screenshot 省得多:你读文字就能定位,不需要看图、也不需要从图像里估算像素位置再换算回屏幕坐标(那正是灰区操作慢和点偏的主因)。拿到结果后可直接 click(x=..,y=..),或 click(ref=N) 由服务端代点(避免抄错坐标)。scope='window'(默认)只识别当前活动窗口那块区域,23 秒;scope='screen' 整屏,文字密集时要 810 秒,非必要别用。也可传 region=[x,y,w,h] 自己指定区域。注意:坐标是快照,界面变化(切窗/弹新对话框)后请重新调用,不要复用旧 ref。⚠️ 已知限制:文字紧邻深色图标时那一行会识别失败(实测「确认删除该文件吗?」挨着问号图标时被认成乱码)。若发现某行文字明显不对,把 region 收窄到只含该文字的小块再调一次即可(可先用本次返回的该块 bbox 定位)。

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
scopeNowindow
regionNo
min_confNo
max_itemsNo

Output Schema

TableJSON Schema
NameRequiredDescriptionDefault
resultYes

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observedv0.1.0

TDQS

A4.7/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full behavioral burden and delivers richly: it discloses timing expectations (2-3s vs 8-10s), the snapshot semantics of coordinates with the warning not to reuse stale refs after UI changes, and a concrete known failure mode (text next to dark icons misrecognized) plus a mitigation (narrowing the region). This is exemplary disclosure for a sensing tool.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is long but front-loaded and tightly organized: function+format, when to use, how to chain with click, scope timing, region, then snapshot warning and known limitations. Each sentence earns its place given the tool's timing and failure-mode complexity, though the size is on the higher end and could tolerate minor trimming.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with 4 parameters, no annotations, and an output schema, the description covers nearly everything an agent needs: return format, scoping behavior with latencies, follow-up click usage, snapshot invalidation, and a failure-mode workaround. Small gaps remain — the semantics of min_conf/max_items, behavior when no text is found, and the coordinate reference frame (screen vs window-relative) are only implied.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0%, so the description must compensate. It adds deep semantics for scope ('window' default targeting the active window, 'screen' full-screen with heavy cost) and region (arbitrary [x,y,w,h] specification), which the bare schema cannot convey. However, min_conf and max_items receive no explanation at all, leaving two of the four parameters semantically undocumented.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb+resource: '读取屏幕上的文字及其坐标' (read on-screen text and its coordinates), and specifies the exact return format `[ref] 文字 @ (x,y) 宽x高`. It differentiates from siblings by positioning itself as the preferred sensing method for gray-area apps with no element tree (SWT/Java, custom-drawn controls, games, remote desktop), which clearly separates it from get_ui_tree/find_element and screenshot.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly states when to use it (gray-area applications) and contrasts it against the alternative (screenshot): reading text avoids guessing pixel positions, which is 'the main cause of slow and offset clicks'. It also gives scope-specific guidance — window scope for the active window (2-3s) vs screen scope (8-10s) with an explicit '非必要别用' (don't use unless necessary) — and prescribes the follow-up action of click(x,y) or click(ref=N).

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.