Robot Actions — Remote Device Control

Flow recording action

flow_recording_action

Perform and record one action step on the device. Performs the action and records it with automatic page source + element capture. Coordinates are in video / iOS-automation space — use flow_recording_start's windowSize as the reference.

Actions:

tap / doubleTap / longPress / swipe — gesture actions (require x,y; swipe also needs toX,toY)
sendKeys — type text into the focused field
keyPress — press an Android keycode (e.g. 3=HOME, 4=BACK)
wait — pause for waitMs milliseconds
pressButton — press a hardware button by name ('home', 'volumeUp', 'volumeDown', 'lock'). Performed live on the device during recording. Android maps to hardware keycodes; iOS invokes the native hardware-button command.
assert — record a UI assertion check (no device gesture). Identifies the target element at (x,y), then at replay time verifies the assertion condition. Assertion failures mark the step as FAILED and surface the mismatch in errorMessage. Makes the recording a real verifiable test. assertType 'visualMatch' is the exception: it takes no x/y and no expected, and at replay time compares the whole screen against the screenshot captured now.
scrollToElement — record a scroll-until-visible checkpoint (no device gesture). Binds the element at (x,y) now; at replay time it scrolls until that element is back on screen, searching downwards first and then upwards, and reports the element's live coordinates. Use it before acting on something whose position varies between runs (a row in a long list, a button below the fold). This FAILS the run if the element never comes into view, because the step exists to bring a target into view for the step that follows it. Note you do NOT need a separate step to resolve an element before acting on it — every element-bound action re-resolves its own target at replay; use scrollToElement only when the target is genuinely off screen, and assert/exists when its absence should fail the run.
appLaunch — launch an app mid-flow by appId (Android package name, iOS bundle id; optional appActivity on Android). Launches live during recording and relaunches the same app at replay time. Use it to switch apps or to return to the app under test after leaving it.

Input Schema

TableJSON Schema

Name	Required	Description
`x`	No	X coordinate (tap/doubleTap/longPress/swipe start; assert & scrollToElement target element)
`y`	No	Y coordinate (tap/doubleTap/longPress/swipe start; assert & scrollToElement target element)
`toX`	No	Swipe end X
`toY`	No	Swipe end Y
`text`	No	Text to type (sendKeys)
`appId`	No	App to launch for an appLaunch step: Android package name, iOS bundle id. Required when action=appLaunch.
`action`	Yes	Action type
`waitMs`	No	Wait duration ms (for wait action)
`keyCode`	No	Android keycode (keyPress, e.g. 3=HOME, 4=BACK)
`keyName`	No	Hardware button name for pressButton (e.g. 'home', 'volumeUp', 'volumeDown', 'lock'). Required when action=pressButton.
`expected`	No	Expected value (required for textEquals and textContains assertions).
`optional`	No	Mark this step as one whose target is EXPECTED to be absent sometimes — a cookie banner, a first-run tutorial, an occasional interstitial. When the element cannot be found at replay the step is SKIPPED instead of failing, and no gesture is dispatched at the recorded position. Leave it off for anything whose absence is a bug: a step that is NOT optional now FAILS the replay when its element is missing, which is what tells a real regression apart from a banner that simply did not appear this run.
`attribute`	No	Attribute to read for text assertions. iOS: label, value, name. Android: text, content-desc, resource-id. Defaults to the platform primary text attribute when omitted.
`assertType`	No	Assertion type (required when action=assert). exists/notExists check element presence; textEquals/textContains compare an attribute value; visualMatch compares the whole screen against the screenshot captured now (takes no x/y and no expected).
`durationMs`	No	Swipe/longPress duration ms (default 300)
`appActivity`	No	Optional Android activity to launch with appId. Ignored on iOS.
`recordingId`	Yes	Recording ID from flow_recording_start
`visionFallback`	No	Let a vision model settle this assertion when the accessibility tree cannot find the element. Off by default, and only honoured for assertType 'exists' and 'notExists'. Turn it on for UI the tree cannot describe — canvas/WebGL screens, game engines, image-only controls with their text baked into the bitmap — where 'a person can see it' is the only check available. Leave it off for ordinary native UI: an assertion's value is that it reports what is really there, and a model asked to find something tends to find it. Ignored for text assertions, which compare exact strings that OCR cannot supply reliably.

Schema Changelog

Changes observed during successful MCP inspections.

Changed10 schema fields changedSep 16, 2026

changedInput schema / properties / action / enum

Previous value: -[
-  "tap",
-  "doubleTap",
-  "longPress",
-  "swipe",
-  "sendKeys",
-  "keyPress",
-  "wait",
-  "assert",
-  "pressButton"
-]New value: +[
+  "tap",
+  "doubleTap",
+  "longPress",
+  "swipe",
+  "sendKeys",
+  "keyPress",
+  "wait",
+  "assert",
+  "pressButton",
+  "scrollToElement",
+  "appLaunch"
+]

addedInput schema / properties / appActivity

Added value: +{
+  "description": "Optional Android activity to launch with appId. Ignored on iOS.",
+  "type": "string"
+}

addedInput schema / properties / appId

Added value: +{
+  "description": "App to launch for an appLaunch step: Android package name, iOS bundle id. Required when action=appLaunch.",
+  "type": "string"
+}

changedInput schema / properties / assertType / description

Previous value: -"Assertion type (required when action=assert). exists/notExists check element presence; textEquals/textContains compare an attribute value."New value: +"Assertion type (required when action=assert). exists/notExists check element presence; textEquals/textContains compare an attribute value; visualMatch compares the whole screen against the screenshot captured now (takes no x/y and no expected)."

changedInput schema / properties / assertType / enum

Previous value: -[
-  "exists",
-  "notExists",
-  "textEquals",
-  "textContains"
-]New value: +[
+  "exists",
+  "notExists",
+  "textEquals",
+  "textContains",
+  "visualMatch"
+]

changedInput schema / properties / attribute / description

Previous value: -"Attribute to read for text assertions. iOS: label, value, name. Android: text, content-desc, resource-id. Defaults to platform primary text attribute when omitted."New value: +"Attribute to read for text assertions. iOS: label, value, name. Android: text, content-desc, resource-id. Defaults to the platform primary text attribute when omitted."

addedInput schema / properties / optional

Added value: +{
+  "description": "Mark this step as one whose target is EXPECTED to be absent sometimes — a cookie banner, a first-run tutorial, an occasional interstitial. When the element cannot be found at replay the step is SKIPPED instead of failing, and no gesture is dispatched at the recorded position. Leave it off for anything whose absence is a bug: a step that is NOT optional now FAILS the replay when its element is missing, which is what tells a real regression apart from a banner that simply did not appear this run.",
+  "type": "boolean"
+}

addedInput schema / properties / visionFallback

Added value: +{
+  "description": "Let a vision model settle this assertion when the accessibility tree cannot find the element. Off by default, and only honoured for assertType 'exists' and 'notExists'. Turn it on for UI the tree cannot describe — canvas/WebGL screens, game engines, image-only controls with their text baked into the bitmap — where 'a person can see it' is the only check available. Leave it off for ordinary native UI: an assertion's value is that it reports what is really there, and a model asked to find something tends to find it. Ignored for text assertions, which compare exact strings that OCR cannot supply reliably.",
+  "type": "boolean"
+}

changedInput schema / properties / x / description

Previous value: -"X coordinate (tap/doubleTap/longPress/swipe start; assert target element)"New value: +"X coordinate (tap/doubleTap/longPress/swipe start; assert & scrollToElement target element)"

changedInput schema / properties / y / description

Previous value: -"Y coordinate (tap/doubleTap/longPress/swipe start; assert target element)"New value: +"Y coordinate (tap/doubleTap/longPress/swipe start; assert & scrollToElement target element)"

First observedAug 4, 2026

TDQS

A4.7/5.0

Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations declare non-readonly, non-idempotent, non-destructive, but the description adds far more: actions execute live on the device during recording and re-execute at replay, assertions can FAIL a step and surface errorMessage, scrollToElement fails the run if the element never appears, and optional steps are SKIPPED rather than failed. This is behavior an agent cannot infer from the annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The core purpose and coordinate-space caveat are front-loaded, then a clean bulleted action list follows. The assert, scrollToElement, optional, and visionFallback paragraphs are somewhat verbose and repeat the replay-time semantics, but for an 11-action tool the length is largely earned.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Covers the recording-vs-replay contract, failure semantics, and per-action requirements thoroughly for a complex 18-param tool. The one notable gap is that, with no output schema, it never says what the call returns (e.g. a step id/index needed for flow_recording_delete_step or flow_recording_replace_step).

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3, but the description adds real meaning beyond the schema: it maps each action to the params it requires (swipe needs toX/toY, pressButton needs keyName), documents the coordinate space relative to flow_recording_start's windowSize, notes durationMs defaults to 300, and explains the visualMatch exception that takes no x/y or expected.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a precise verb+resource: 'Perform and record one action step on the device,' which separates it from siblings like flow_recording_start, flow_recording_replay, and the low-level device_tap/device_swipe tools. The subsequent per-action enumeration makes the scope unambiguous.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives explicit when-to-use and when-not-to-use guidance for the tricky cases: use scrollToElement only when the target is genuinely off screen (and assert/exists when absence should fail the run), and no separate resolve step is needed because element-bound actions re-resolve at replay. visionFallback is scoped with 'turn it on for X, leave it off for ordinary native UI.'

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

All Robot Actions — Remote Device Control tools Inspect

Try in Browser

Let your AI agent see the webWeb-to-screenshot MCP for Claude, Cursor and Codex. No cookie banners. Free to start.snap-render.comAd

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources

Need Help?