mobilerun-mcp

Perceive Screen

perceive_screen

Captures annotated screenshots with element tap points and flags, enabling AI agents to perceive and interact with mobile device screens.

Instructions

LOOK at the screen: an annotated screenshot plus every element and its tap point.

e is an array whose INDEX is the som_id (first entry = som_id 1): [center_x, center_y, name, flags] (name/flags omitted when empty). elements is the same list as text. Box colours: BLUE tappable, GREEN text input (type_text), MAGENTA scrollable, AMBER toggle, GREY nothing declared, RED on-host vision (detector/OCR; a good guess, not a fact). Flags (only when true): e editable, c checked, o unchecked, d disabled, f focused, l long-pressable, ? low-confidence vision box, w scroll host (aim inside it). offscreen: text that exists but is not on screen (cannot be tapped). detail="full" adds the visual pass (OmniParser YOLOv8 icon detector + OCR) for icons the tree does not describe; perception_tier reports tree_only or full. description is logged only. ids go stale after any action.

Tool call template

This tool declares no required parameters. Add optional parameters from the input schema as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {},
    "name": "perceive_screen"
  }
}

Input Schema

Name	Required	Default
`ocr`	No	auto
`lang`	No	eng
`detail`	No
`device`	No
`max_marks`	No
`description`	No
`include_image`	No

Schema Changelog

Changes observed during successful MCP inspections.

First observedv0.1.0Sep 27, 2026

TDQS

B3.4/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations at all, the description carries the full behavioral burden and does so richly: som_id indexing, box colour semantics, flag meanings, offscreen elements being untappable, the cost/benefit of detail="full", perception_tier reporting, and that ids go stale after any action. It is silent on the screen-capture permission requirement (a sibling, request_screen_capture_permission, implies one) and on whether the visual pass is slow or expensive.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Dense but well front-loaded — the single-line summary comes first, then output anatomy, then the detail-mode caveat. Every sentence carries load-bearing information, though the colour/flag enumeration is terse to the point of requiring careful re-reading.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Return-value interpretation is thoroughly covered, which matters given there is no output schema, but the description is thin on the input side for a 7-parameter tool and omits any permission or prerequisite context. An agent can call it and read the result, but cannot reason about half its knobs.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0% across 7 parameters, so the description must compensate, and it only covers two: detail="full" and the fact that description is logged only. ocr, lang, device, max_marks, and include_image are left completely undocumented, including ocr's "auto" value and lang's "eng" default.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The first sentence states a specific verb and resource — 'LOOK at the screen: an annotated screenshot plus every element and its tap point' — which is far more concrete than a tautology. It does not explicitly distinguish itself from closely related siblings such as get_ui_tree, read_screen, ui_json, or screenshot, so an agent must infer the boundary from the detailed output anatomy.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is implied rather than stated: 'ids go stale after any action' hints that this tool should be re-run before acting, and detail="full" is framed as an escalation for undescribed icons. There is no explicit when-to-use guidance and no named alternative (e.g. get_ui_tree for a cheap text-only pass, get_screenshot for pixels only).

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server