Skip to main content
Glama

Robot Actions — Remote Device Control

by com.robotactions.test
OverviewDiscussionsAdmin

device_tap

Tap at (x,y) coordinates on a device screen. Coordinates are in DEVICE TAP-COORD SPACE (the "Tap-coord space" dims printed in the device_screenshot footer; same space as device_page_source bounds). First call starts a device control session (~3s).

COORDINATE SOURCES — in priority order:

  1. PRIMARY: device_page_source bounds [L,T][R,B] (or the "Labeled elements" block bundled with device_screenshot) → tap center = ((L+R)/2, (T+B)/2). NO scaling. Pixel-exact.

  2. FALLBACK ONLY (element not in page_source — image-only widget / custom Canvas): visual estimate from the screenshot pixels, scaled with the formula below.

VISUAL → TAP COORDINATE FORMULA (Android): scale = device_width / rendered_chat_width tap_x = visual_x × scale tap_y = visual_y × scale where device_width is the "Tap-coord space" width from the device_screenshot footer and rendered_chat_width is the "Image" width from the same footer. Both axes share one scale (aspect preserved). The footer prints concrete values per device — never assume any constant.

Skipping the scale on a visual estimate is the #1 cause of taps landing in the wrong place — the agent sees a downscaled image but device_tap expects full-resolution tap-space coords.

Pass input="hid" to require the virtual touchscreen: the gesture then reaches the app as real kernel input and fails loudly instead of silently falling back to injected input. That strict path used to be device_uhid_tap, which stays callable under its old name.

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
xYesX coordinate in device tap-coord space (NOT visual screenshot pixels — apply scale = device_width / rendered_chat_width if you started from a visual estimate)
yYesY coordinate in device tap-coord space (NOT visual screenshot pixels — apply scale = device_width / rendered_chat_width if you started from a visual estimate)
udidYesDevice serial number (UDID)
inputNoInput path. 'auto' (default) prefers the virtual touchscreen and falls back to injected input. 'hid' requires the virtual touchscreen and fails if the device does not provide one — use it when the gesture must be real kernel input or must fail loudly.
holdMsNoHow long the contact stays down, in ms. Only applies with input="hid" (default 60ms).

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed3 schema fields changed
    • changedInput schema / properties / input / description
      Previous value: -"Input path. 'auto' (default) prefers the virtual touchscreen and falls back to injected input. 'hid' requires the virtual touchscreen and fails if the device does not provide one — use it when the gesture must be real kernel input or must fail loudly."New value: +"Input path. 'auto' (default) prefers the virtual touchscreen and falls back to injected input. 'hid' requires the virtual touchscreen and fails if the device does not provide one — use it when the gesture must be real kernel input or must fail loudly."
    • changedInput schema / properties / x / description
      Previous value: -"X coordinate in device tap-coord space (NOT visual screenshot pixels — apply scale = device_width / rendered_chat_width if you started from a visual estimate)"New value: +"X coordinate in device tap-coord space (NOT visual screenshot pixels — apply scale = device_width / rendered_chat_width if you started from a visual estimate)"
    • changedInput schema / properties / y / description
      Previous value: -"Y coordinate in device tap-coord space (NOT visual screenshot pixels — apply scale = device_width / rendered_chat_width if you started from a visual estimate)"New value: +"Y coordinate in device tap-coord space (NOT visual screenshot pixels — apply scale = device_width / rendered_chat_width if you started from a visual estimate)"
  2. Changed3 schema fields changed
    • changedInput schema / properties / input / description
      Previous value: -"Input path. 'auto' (default) prefers the virtual touchscreen and falls back to injected input. 'hid' requires the virtual touchscreen and fails if the device does not provide one — use it when the gesture must be real kernel input or must fail loudly."New value: +"Input path. 'auto' (default) prefers the virtual touchscreen and falls back to injected input. 'hid' requires the virtual touchscreen and fails if the device does not provide one — use it when the gesture must be real kernel input or must fail loudly."
    • changedInput schema / properties / x / description
      Previous value: -"X coordinate in device tap-coord space (NOT visual screenshot pixels — apply scale = device_width / rendered_chat_width if you started from a visual estimate)"New value: +"X coordinate in device tap-coord space (NOT visual screenshot pixels — apply scale = device_width / rendered_chat_width if you started from a visual estimate)"
    • changedInput schema / properties / y / description
      Previous value: -"Y coordinate in device tap-coord space (NOT visual screenshot pixels — apply scale = device_width / rendered_chat_width if you started from a visual estimate)"New value: +"Y coordinate in device tap-coord space (NOT visual screenshot pixels — apply scale = device_width / rendered_chat_width if you started from a visual estimate)"
  3. Changed2 schema fields changed
    • addedInput schema / properties / holdMs
      Added value: +{
+  "description": "How long the contact stays down, in ms. Only applies with input=\"hid\" (default 60ms).",
+  "maximum": 5000,
+  "minimum": 1,
+  "type": "integer"
+}
    • addedInput schema / properties / input
      Added value: +{
+  "description": "Input path. 'auto' (default) prefers the virtual touchscreen and falls back to injected input. 'hid' requires the virtual touchscreen and fails if the device does not provide one — use it when the gesture must be real kernel input or must fail loudly.",
+  "enum": [
+    "auto",
+    "hid"
+  ],
+  "type": "string"
+}
  4. First observed

TDQS

A4.6/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the behavioral disclosure burden. It covers first-call session startup (~3s), the coordinate-space scaling requirement, failure-mode differences between 'auto' and 'hid', and the legacy behavior. Minor deduction because it doesn't describe what happens on invalid coordinates or when session setup fails, but the critical operational behaviors are disclosed.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is long but well-organized with clear headers, a priority list, and a concrete formula. Every section contributes actionable information, and the most failure-prone warning (skipping the scale) is emphasized. Slightly verbose in places, but the structure supports quick scanning.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Despite having no output schema and no annotations, the description fully equips an agent to call this tool correctly. It explains coordinate spaces, how to derive coordinates from both primary and fallback sources, the scaling formula, and input path semantics. All parameters are effectively covered, and an agent can confidently perform a tap without additional lookups.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the baseline is 3. The description adds meaning by defining the DEVICE TAP-COORD SPACE, providing the scaling formula, and clarifying that 'auto' falls back to injected input while 'hid' fails loudly. This meaningfully supplements the schema for the two most error-prone parameters, x and y.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Description opens with a specific verb and resource: 'Tap at (x,y) coordinates on a device screen.' It clearly defines the coordinate space, distinguishes the primary source from the fallback, and names the legacy device_uhid_tap path, so an agent can differentiate it from related input tools without opening schemas.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides explicit when-to-use guidance: use device_page_source bounds as the primary source, use visual estimation only as a fallback, and use input='hid' when real kernel input is required. It explains the fallback chain, the exact scaling formula, and the relationship to device_uhid_tap, giving the agent clear alternatives and conditions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

All Robot Actions — Remote Device Control toolsInspect
Try in Browser
One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources