Skip to main content
Glama
Sign In
Computer Use

Computer Use

by munimtechnologies
OverviewSchemaSourceRelated ServersDiscussionsAdmin
Rust
Local

Read accessibility tree

get_app_state
Read-onlyIdempotent

Read an app's accessibility tree as an indented outline in which interactive elements carry ids like [e12] that click, type_text, set_value, scroll, hover and select_text accept.

Instructions

Read an app's accessibility tree as an indented outline in which interactive elements carry ids like [e12] that click, type_text, set_value, scroll, hover and select_text accept. Use it instead of screenshot whenever you intend to act: it is far cheaper in tokens and gives exact targets. Call it before interacting and again after the UI changes, because ids are per-snapshot. Each id is rechecked before an action: one whose element is gone or has changed fails instead of acting on something else. The outline lists what is on screen: rows of long lists and content scrolled out of view are skipped (scroll, or pass offscreen), and while a dialog, sheet or popover blocks a window only its controls are listed. When an app exposes little to accessibility, its on-screen text is read with OCR and listed with ids too. Read-only; it does not change focus.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "app": "<app>"
    },
    "name": "get_app_state"
  }
}

Input Schema

NameRequiredDescriptionDefault
appYesApp name, bundle id, or pid exactly as reported by list_apps, or "frontmost" for the app the user is looking at
ocrNoRead on-screen text with OCR and list it with ids that click, hover, right_click and drag accept: auto (default) only when the app exposes no labelled controls to accessibility, always, or never. macOS and Windows; macOS needs Screen Recording permission.
queryNoOnly list elements whose role, label or value contains this text (case-insensitive). Ids stay valid. It searches the whole outline, not just the first max_elements elements, and says when the read stopped early. Use it instead of raising max_elements when you know what you are looking for.
max_depthNoMaximum nesting depth to descend (default 18). Lower it for a quick overview of a large window.
offscreenNoAlso list elements scrolled out of view, such as every row of a long list (default false). Prefer scrolling or `query`: this can make the outline far larger.
max_elementsNoMaximum elements to emit before the outline is truncated (default 800). With `query`, the maximum matches to list instead: a query searches past this cap. Prefer `query` over raising this.

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed2 schema fields changedv0.7.3
    • changedInput schema / properties / max_elements / description
      Previous value: -"Maximum elements to emit before the outline is truncated (default 800). Prefer `query` over raising this."New value: +"Maximum elements to emit before the outline is truncated (default 800). With `query`, the maximum matches to list instead: a query searches past this cap. Prefer `query` over raising this."
    • changedInput schema / properties / query / description
      Previous value: -"Only list elements whose role, label or value contains this text (case-insensitive). Ids stay valid. Use it instead of raising max_elements when you know what you are looking for."New value: +"Only list elements whose role, label or value contains this text (case-insensitive). Ids stay valid. It searches the whole outline, not just the first max_elements elements, and says when the read stopped early. Use it instead of raising max_elements when you know what you are looking for."
  2. Changed2 schema fields changedv0.7.1
    • addedInput schema / properties / ocr
      Added value: +{
+  "description": "Read on-screen text with OCR and list it with ids that click, hover, right_click and drag accept: auto (default) only when the app exposes no labelled controls to accessibility, always, or never. macOS and Windows; macOS needs Screen Recording permission.",
+  "enum": [
+    "auto",
+    "always",
+    "never"
+  ],
+  "type": "string"
+}
    • addedInput schema / properties / offscreen
      Added value: +{
+  "description": "Also list elements scrolled out of view, such as every row of a long list (default false). Prefer scrolling or `query`: this can make the outline far larger.",
+  "type": "boolean"
+}
  3. Changed1 schema field changedv0.5.2
    • changedInput schema / properties / app / description
      Previous value: -"App name, bundle id, or pid exactly as reported by list_apps"New value: +"App name, bundle id, or pid exactly as reported by list_apps, or \"frontmost\" for the app the user is looking at"
  4. Changed3 schema fields changedv0.3.1
    • changedInput schema / properties / app / description
      Previous value: -"App name, bundle id, or pid"New value: +"App name, bundle id, or pid exactly as reported by list_apps"
    • changedInput schema / properties / max_depth / description
      Previous value: -"Max tree depth (default 18)"New value: +"Maximum nesting depth to descend (default 18). Lower it for a quick overview of a large window."
    • changedInput schema / properties / max_elements / description
      Previous value: -"Max elements to emit (default 800)"New value: +"Maximum elements to emit before the outline is truncated (default 800). Prefer `query` over raising this."
  5. First observedv0.3.0

TDQS

A4.9/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Goes well past the readOnly/idempotent annotations: ids are per-snapshot, stale ids fail rather than mis-clicking, offscreen rows are skipped, only blocking-dialog controls appear when a modal is up, and OCR substitutes for apps with weak accessibility exposure. It also discloses the non-effect on focus, which annotations cannot express.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Six tight sentences, each carrying distinct load: what it returns, why over screenshot, when to re-call, what ids guarantee, what is excluded, and the OCR fallback. The most decision-critical point (use this before acting) lands in sentence two.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

No output schema exists, and the description compensates by describing the return format, the id scheme, truncation behavior, and the modal/offscreen omissions an agent must anticipate. Nothing needed to call it correctly on a six-parameter, read-only tool is left unstated.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is already 100%, so the baseline is 3; the description adds real guidance by recommending `query` over `max_elements` and over `offscreen`, and explaining that query searches the whole outline and preserves id validity. It does not restate enum mechanics, which is fine since the schema does.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource — reading an app's accessibility tree — and immediately characterizes the output shape (indented outline with [e12] ids) and who consumes those ids. It distinguishes itself from `screenshot` and implicitly from `browser_snapshot` by framing itself as the app-level, act-before-you-click reader.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicit routing: 'Use it instead of screenshot whenever you intend to act', 'Call it before interacting and again after the UI changes'. It also names in-parameter alternatives (scroll or pass `offscreen`, use `query` instead of raising `max_elements`) and states preconditions for the OCR path (Screen Recording permission).

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server

Other Tools