Skip to main content
Glama

Robot Actions — Remote Device Control

by com.robotactions.test
OverviewDiscussionsAdmin

flow_recording_action

Perform and record one action step on the device. Performs the action and records it with automatic page source + element capture. Coordinates are in video / iOS-automation space — use flow_recording_start's windowSize as the reference.

Actions:

  • tap / doubleTap / longPress / swipe — gesture actions (require x,y; swipe also needs toX,toY)

  • sendKeys — type text into the focused field

  • keyPress — press an Android keycode (e.g. 3=HOME, 4=BACK)

  • wait — pause for waitMs milliseconds

  • pressButton — press a hardware button by name ('home', 'volumeUp', 'volumeDown', 'lock'). Performed live on the device during recording. Android maps to hardware keycodes; iOS invokes the native hardware-button command.

  • assert — record a UI assertion check (no device gesture). Identifies the target element at (x,y), then at replay time verifies the assertion condition. Assertion failures mark the step as FAILED and surface the mismatch in errorMessage. Makes the recording a real verifiable test. assertType 'visualMatch' is the exception: it takes no x/y and no expected, and at replay time compares the whole screen against the screenshot captured now.

  • scrollToElement — record a scroll-until-visible checkpoint (no device gesture). Binds the element at (x,y) now; at replay time it scrolls until that element is back on screen, searching downwards first and then upwards, and reports the element's live coordinates. Use it before acting on something whose position varies between runs (a row in a long list, a button below the fold). This FAILS the run if the element never comes into view, because the step exists to bring a target into view for the step that follows it. Note you do NOT need a separate step to resolve an element before acting on it — every element-bound action re-resolves its own target at replay; use scrollToElement only when the target is genuinely off screen, and assert/exists when its absence should fail the run.

  • appLaunch — launch an app mid-flow by appId (Android package name, iOS bundle id; optional appActivity on Android). Launches live during recording and relaunches the same app at replay time. Use it to switch apps or to return to the app under test after leaving it.

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
xNoX coordinate (tap/doubleTap/longPress/swipe start; assert & scrollToElement target element)
yNoY coordinate (tap/doubleTap/longPress/swipe start; assert & scrollToElement target element)
toXNoSwipe end X
toYNoSwipe end Y
textNoText to type (sendKeys)
appIdNoApp to launch for an appLaunch step: Android package name, iOS bundle id. Required when action=appLaunch.
actionYesAction type
waitMsNoWait duration ms (for wait action)
keyCodeNoAndroid keycode (keyPress, e.g. 3=HOME, 4=BACK)
keyNameNoHardware button name for pressButton (e.g. 'home', 'volumeUp', 'volumeDown', 'lock'). Required when action=pressButton.
expectedNoExpected value (required for textEquals and textContains assertions).
optionalNoMark this step as one whose target is EXPECTED to be absent sometimes — a cookie banner, a first-run tutorial, an occasional interstitial. When the element cannot be found at replay the step is SKIPPED instead of failing, and no gesture is dispatched at the recorded position. Leave it off for anything whose absence is a bug: a step that is NOT optional now FAILS the replay when its element is missing, which is what tells a real regression apart from a banner that simply did not appear this run.
attributeNoAttribute to read for text assertions. iOS: label, value, name. Android: text, content-desc, resource-id. Defaults to the platform primary text attribute when omitted.
assertTypeNoAssertion type (required when action=assert). exists/notExists check element presence; textEquals/textContains compare an attribute value; visualMatch compares the whole screen against the screenshot captured now (takes no x/y and no expected).
durationMsNoSwipe/longPress duration ms (default 300)
appActivityNoOptional Android activity to launch with appId. Ignored on iOS.
recordingIdYesRecording ID from flow_recording_start
visionFallbackNoLet a vision model settle this assertion when the accessibility tree cannot find the element. Off by default, and only honoured for assertType 'exists' and 'notExists'. Turn it on for UI the tree cannot describe — canvas/WebGL screens, game engines, image-only controls with their text baked into the bitmap — where 'a person can see it' is the only check available. Leave it off for ordinary native UI: an assertion's value is that it reports what is really there, and a model asked to find something tends to find it. Ignored for text assertions, which compare exact strings that OCR cannot supply reliably.

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed10 schema fields changed
    • changedInput schema / properties / action / enum
      Previous value: -[
-  "tap",
-  "doubleTap",
-  "longPress",
-  "swipe",
-  "sendKeys",
-  "keyPress",
-  "wait",
-  "assert",
-  "pressButton"
-]New value: +[
+  "tap",
+  "doubleTap",
+  "longPress",
+  "swipe",
+  "sendKeys",
+  "keyPress",
+  "wait",
+  "assert",
+  "pressButton",
+  "scrollToElement",
+  "appLaunch"
+]
    • addedInput schema / properties / appActivity
      Added value: +{
+  "description": "Optional Android activity to launch with appId. Ignored on iOS.",
+  "type": "string"
+}
    • addedInput schema / properties / appId
      Added value: +{
+  "description": "App to launch for an appLaunch step: Android package name, iOS bundle id. Required when action=appLaunch.",
+  "type": "string"
+}
    • changedInput schema / properties / assertType / description
      Previous value: -"Assertion type (required when action=assert). exists/notExists check element presence; textEquals/textContains compare an attribute value."New value: +"Assertion type (required when action=assert). exists/notExists check element presence; textEquals/textContains compare an attribute value; visualMatch compares the whole screen against the screenshot captured now (takes no x/y and no expected)."
    • changedInput schema / properties / assertType / enum
      Previous value: -[
-  "exists",
-  "notExists",
-  "textEquals",
-  "textContains"
-]New value: +[
+  "exists",
+  "notExists",
+  "textEquals",
+  "textContains",
+  "visualMatch"
+]
    • changedInput schema / properties / attribute / description
      Previous value: -"Attribute to read for text assertions. iOS: label, value, name. Android: text, content-desc, resource-id. Defaults to platform primary text attribute when omitted."New value: +"Attribute to read for text assertions. iOS: label, value, name. Android: text, content-desc, resource-id. Defaults to the platform primary text attribute when omitted."
    • addedInput schema / properties / optional
      Added value: +{
+  "description": "Mark this step as one whose target is EXPECTED to be absent sometimes — a cookie banner, a first-run tutorial, an occasional interstitial. When the element cannot be found at replay the step is SKIPPED instead of failing, and no gesture is dispatched at the recorded position. Leave it off for anything whose absence is a bug: a step that is NOT optional now FAILS the replay when its element is missing, which is what tells a real regression apart from a banner that simply did not appear this run.",
+  "type": "boolean"
+}
    • addedInput schema / properties / visionFallback
      Added value: +{
+  "description": "Let a vision model settle this assertion when the accessibility tree cannot find the element. Off by default, and only honoured for assertType 'exists' and 'notExists'. Turn it on for UI the tree cannot describe — canvas/WebGL screens, game engines, image-only controls with their text baked into the bitmap — where 'a person can see it' is the only check available. Leave it off for ordinary native UI: an assertion's value is that it reports what is really there, and a model asked to find something tends to find it. Ignored for text assertions, which compare exact strings that OCR cannot supply reliably.",
+  "type": "boolean"
+}
    • changedInput schema / properties / x / description
      Previous value: -"X coordinate (tap/doubleTap/longPress/swipe start; assert target element)"New value: +"X coordinate (tap/doubleTap/longPress/swipe start; assert & scrollToElement target element)"
    • changedInput schema / properties / y / description
      Previous value: -"Y coordinate (tap/doubleTap/longPress/swipe start; assert target element)"New value: +"Y coordinate (tap/doubleTap/longPress/swipe start; assert & scrollToElement target element)"
  2. First observed

TDQS

A5/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full behavioral burden and meets it: actions execute live while recording, coordinates reference flow_recording_start's windowSize, assert failures mark the step FAILED and surface mismatches in errorMessage, scrollToElement searches down then up and fails if absent, and appLaunch relaunches at replay. It also clarifies replay-time re-resolution so agents do not insert redundant steps.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is long, but the tool is a polymorphic 18-parameter dispatcher; each action is a short, self-contained entry with requirements and replay behavior. The overview and coordinate reference are front-loaded, and there is no filler or needless repetition of schema text.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For an 18-parameter tool with 11 action variants and no output schema, the description covers invocation requirements, live vs replay behavior, failure semantics, and per-action parameter constraints. No critical call-time information appears to be missing; the only minor omission is an explicit statement of the return value, which is not essential for correct invocation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3, but the description adds substantial value beyond the schema: it maps each action to the exact parameters it requires (swipe needs toX/toY, assert needs assertType, appLaunch needs appId), defines the coordinate space, explains the visualMatch exception, and clarifies the meaning of optional and visionFallback beyond their schema text.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb+resource: 'Perform and record one action step on the device', and enumerates the 11 supported action types. It distinguishes this recorder from generic device_* gesture tools by emphasizing automatic page-source and element capture during recording.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives explicit when-to-use and when-not-to-use guidance for the ambiguous actions: scrollToElement is 'use it before acting on something whose position varies', with the exclusion 'use scrollToElement only when the target is genuinely off screen'; appLaunch is for switching apps; optional is for expected absences; visionFallback is only honored for exists/notExists. These are concrete usage rules, not vague hints.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

All Robot Actions — Remote Device Control toolsInspect
Try in Browser
One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources