LOOK at the screen: an annotated screenshot plus every element and its tap point.

e is an array whose INDEX is the som_id (first entry = som_id 1): [center_x, center_y, name, flags] (name/flags omitted when empty). elements is the same list as text. Box colours: BLUE tappable, GREEN text input (type_text), MAGENTA scrollable, AMBER toggle, GREY nothing declared, RED on-host vision (detector/OCR; a good guess, not a fact). Flags (only when true): e editable, c checked, o unchecked, d disabled, f focused, l long-pressable, ? low-confidence vision box, w scroll host (aim inside it). offscreen: text that exists but is not on screen (cannot be tapped). detail="full" adds the visual pass (OmniParser YOLOv8 icon detector + OCR) for icons the tree does not describe; perception_tier reports tree_only or full. description is logged only. ids go stale after any action.

Read the screen now (waits for it to stop moving first): the screen drawn as a character grid, each element a box with its som_id and label, then a table of what can be acted on: som (tap by this), in (som_id of the smallest box containing it), flg, label (only when it did not fit on the grid). Flags: * tappable, e text input (type_text, not tap), S scrollable, c toggle ON, o toggle OFF, l long-pressable, d disabled, - nothing declared (usually still tappable). N+k = som_id N plus k more elements with exactly those bounds; tap N. Header IDLE/BUSY: BUSY means it was still moving when the wait expired. Use perceive_screen instead for how something looks, when this ends with ESCALATE, or when what you need is missing.

Compact accessibility tree (class, id, label, flags C/L/E/S/K/P, bounds).

Plain screenshot as an image.

Plain screenshot. hide_overlay hides the Portal's element overlay first.

Take a screenshot, save it as a PNG file and return the path.

Tap at (x, y) or at the center of a numbered mark (som_id from perceive_screen / read_screen). stealth=true uses mobilerun-core's humanized tap.

Double-tap at (x, y) or a mark.

Press and hold at (x, y), a mark (som_id) or a get_state element (index). Hold time: duration_ms or ms (default 1000).

Long press at (x, y) (mobilerun agent action).

Swipe from (x1, y1) to (x2, y2) over duration_ms / ms (default 300). mobilerun agent form: coordinate=[x, y], coordinate2=[x, y], duration in seconds.

Scroll the content down (reveal what is below): a centered swipe over half the screen (amount), or inside a scrollable mark (som_id).

Scroll the content up (reveal what is above).

Scroll the content left (reveal what is to the left).

Scroll the content right (reveal what is to the right).

Scroll the content in direction (up | down | left | right) by distance (fraction of the screen). verify=true reports whether the screen actually moved (mobilerun-core).

Two modes. With x1, y1, x2, y2: drag the content from one point to the other (precise scroll). With text: scroll in direction until an element containing text is visible and return its mark.

Type into the focused field (tap it first, or pass som_id). submit presses Enter after.

Type text (mobilerun). index taps that get_state element first; stealth=true types key by key like a person (mobilerun-core), at wpm words per minute.

Press the Home button.

Press Back (also closes the keyboard without leaving the screen).

Press Enter (submits search bars and forms).

Open the recent-apps overview.

Press a key by mobilerun-core name (back, home, menu, enter, delete, escape, tab, space, search, page_up, page_down, volume_up, volume_down, wakeup, media_play_pause, ...) or by Android keycode number.

Open an app by name (fuzzy) or exact package_name. An ambiguous name returns ranked candidates instead of guessing. If the app is already in the foreground it is left as is (already_foreground=true) unless force=true.

Start an app by id (Android package / iOS bundle id), optionally a specific activity.

Search installed apps by name or package; returns ranked candidates with scores.

List installed apps (user apps only unless include_system_apps=true). include_protected_apps is honoured on Mobilerun Cloud devices only (as in mobilerun-core).

Deep links into an app, best first. source: shortcut (launcher shortcut, app-shortcut://pkg/id, open with open_deeplink) > resolved (curated and verified on this device) > catalog (curated) > discovered (declared by the app's intent filters).

Which app would open this URI (or intent action such as android.settings.WIFI_SETTINGS)? handler_kind: app | browser_only | none.

Jump straight to a screen via a URI, an app-shortcut://pkg/id from list_app_deeplinks, or an intent action. package_name / app_name pin the target app. intent:, file:, content: and javascript: URIs are refused.

Battery, screen power, foreground app, size, storage, network addresses, volume.

Devices you can control. scope: local (adb devices) | cloud (Mobilerun Cloud, needs MOBILERUN_CLOUD_API_KEY) | all. Cloud filters: state (creating, assigned, ready, terminated, ...), type, name, country, page, pageSize (or a filters dict). Any listed id works as the device argument of every tool.

Is the device reachable? For adb devices, reports the Portal transport (http or content_provider).

(Re)connect adb and the Portal for a device; use after the network path came back.

Disconnect a TCP/IP adb device (adb disconnect host:port) and drop its session.

Install and enable the Mobilerun Portal on the device (mobilerun setup); path installs a specific Portal APK.

Health check of adb, the Portal and the device (mobilerun doctor).

Compatibility no-op: screenshots use the Portal / adb screencap, no prompt is needed.

Returns text verbatim: a check that the MCP transport is alive (no device access).

Press home, back or enter (kept for old clients; prefer press_home/back/enter).

Current status-bar notifications, newest first, without touching the screen: key, app, title, text, action labels. Ongoing ones (music, navigation, downloads) only with include_ongoing=true; package_name filters to one app; limit default 20, cap 30. With the safety policy on, banking and authenticator notifications are withheld.

Dismiss one notification (by key, or package/title) or every clearable one.

Tap one of a notification's own buttons (reply, archive, stop...); reply_text fills an inline reply field and sends it. Best-effort: it drives the notification shade.

Active media sessions (app, playback state, title/artist) and the music volume.

Control playback in any app without touching the screen: play, pause, play_pause, next, previous, stop, rewind, fast_forward. Goes to the active media session; package_name (from get_media_sessions) is checked against it and reported.

Raise the music volume by steps.

Lower the music volume by steps.

Toggle mute on the media stream (muted=true/false forces a state). Muting remembers the previous level for unmute.

Search the device's media index by name, newest first: images, videos, audio and documents (downloads included). kind: image | video | audio | document | any. Returns content:// URIs for open_file (limit default 10, cap 25). With path, searches that shared-storage folder by file name instead (any file type).

Open a file in its default viewer. uri: a content://media/... URI from find_files (never build one by hand); path: a file under shared storage.

One-call Android actions (action = the verb; verb= is accepted too). Verbs: set_alarm(hour, minute, label), set_timer(seconds, label), dial(phone_number), compose_sms(phone_number, body), add_calendar_event(title, start, end, location, notes; ISO datetimes), share_text(text, subject), navigate( destination, mode drive|walk|bike|transit). dial/compose_sms only prefill; the user sends.

Find contacts by (partial) name and return their phone numbers.

LONG waits only (downloads, uploads, processing, status changes); gestures already settle. With text / package / activity: wait until that is on screen (gone=true: until it disappears). Otherwise wait until loading finishes (no progress bar or "Loading" text) and the screen is still; condition is echoed back, you judge the returned state. Timeouts: timeout_ms (default 5000, max 30000) / poll_interval_ms (default 500, min 100), or timeout / interval in seconds.

Collect device events for up to timeout_seconds (default 10, max 30), returning early once max_events (default 50) arrive: foreground app, keyboard, screen content, notifications posted/removed. kinds filters: foreground, keyboard, screen, notifications.

Pre-check a planned action against the safety policy (and, for our action set, that its target exists) without doing it. gesture_type is the action (tap, type_text, launch_app, open_deeplink, ...); target is the app name/package, deep-link URI or text you plan to use. Returns allowed=false with a category for blocked apps and text.

Check an outcome against the live screen. kind: text (visible), gone (not visible), app (foreground package or name), activity, changed (the last action changed the screen).

Search the web for how to do something in an app ('how to in android'). Returns ranked sources (title, url, snippet) and, with TAVILY_API_KEY set, a synthesized answer. max_results 1-10 (default 5); topic general | news. The screen overrules results.

Start a plan checklist. target_count > 0 means 'N items must be recorded' before end_session(success) is allowed. With 3+ steps and a search_query, the first web search rides along in the reply.

Update a plan step: pending | in_progress | done | skipped | failed. Put facts you read off the screen in note; the pixels are gone next turn.

Record one item you found. quote must be copied exactly from the CURRENT screen.

Mark the end of the task (the server keeps listening; the next call starts fresh). reason: short summary of what was done. outcome: success (goal state verified) | partial | failure. goal_type: play_media | send_message | send_email | purchase | post | open_app | search | navigate | other. For send_message, send_email, purchase and post, success is refused unless you looked at the screen (read_screen / perceive_screen) after your last action. Success is also refused while fewer findings than the plan's target_count are recorded. failure is never refused.

How to use this server well. Topics: overview, shortcuts, text_entry, failures, ledger, safety, stop, efficiency, full.

Raw UI snapshot (a11y_tree, phone_state, device_context, ...), as Device.ui().

The UI snapshot serialized as JSON text.

UI snapshot that retries past a dead or empty accessibility tree.

Backend, platform and the actions this device supports.

Whether this device supports a Device action (e.g. execute_script, get_clipboard).

[width, height] in pixels.

Package / bundle id of the foreground app.

The device clock.

Nodes matching every given filter (exact text/desc/resource_id/class_name, or *_contains substrings), including off-screen ones.

Like find_nodes, limited to nodes inside the visible screen.

Tap the first on-screen node whose text/description contains text.

Tap the center of a node returned by find_nodes / find_nodes_on_screen.

Tap a text (or node) and wait until the UI has been idle for idle seconds.

Scroll until a matching node is on screen; result is the node (or null).

Clear the focused text field.

Fail unless app_id is in the foreground.

Fail unless text becomes visible on screen within timeout seconds.

Wait until app_id is in the foreground.

Wait until the UI stops changing.

Wait until the UI differs from now.

Wait until a node containing text exists (off-screen nodes count).

Poll find_nodes until something matches (or timeout, returning []).

Start an app and wait until it is in front and idle.

Force-stop an app; clear_data also wipes its data.

Install an APK (host path) on the device.

Uninstall an app.

Grant a runtime permission (android.permission.*) to an app.

Dispatch a deep link / intent (default action VIEW), optionally pinned to a package.

Run JavaScript in the foreground browser page and return its JSON result.

The clipboard's text (Android needs the Mobilerun Keyboard as the active IME).

Put text on the clipboard.