Read what is on the phone screen as a list of UI elements with ids, text, and tap coordinates. Call this first and after every action to see the result. Element ids are only valid until the next read_screen.

Tap an element (by id from the last read_screen) or a screen coordinate in pixels.

Press and hold an element (by id from the last read_screen) or a screen coordinate in pixels, for about 0.7 s. Use it for what apps put behind a long press: a message's or item's menu (reply, copy, delete, react), selecting items or text, moving icons. Use tap for normal taps. Returns "Long-pressed" and where; call read_screen to see the menu or selection it opened.

Type into the focused text field (or the given element). Replaces its content unless append is true. Set submit to press the keyboard's enter/send action afterwards.

Scroll a list or page by about a third of its height or width, to reach content that isn't on screen yet: 'down' shows what's further down (like reading on), 'up' goes back, 'left'/'right' for carousels and tabs. Without element it scrolls the middle of the screen; give the id of a scrollable element (marked scroll in read_screen) to scroll just that list. Returns "Scrolled"; call read_screen to see what came into view. For a precise drag, use swipe.

Drag a finger in a straight line from (x1, y1) to (x2, y2), in screen pixels. Use it for gestures scroll doesn't cover: moving a slider, swiping a card or a message aside, pulling down to refresh, dragging on a map. A very short swipe counts as a tap. Returns "Swiped"; call read_screen to see the result.

Press one of Android's system buttons: back (leave the current screen or close a dialog or the keyboard), home (go to the home screen), recents (the recent apps list), notifications (pull down the notification shade), quick_settings (open Quick Settings). To switch apps, open_app is faster than home and recents. Returns "Pressed" or "Could not press".

Open an installed app by its name (e.g. 'Messenger') or package name.

List launchable apps with their package names.

Open a link on the phone the way tapping it would: https:// pages open in the user's browser or in the app that handles them (a YouTube link in YouTube), and deep links open their app (mailto: an email draft, tel: the dialer with the number, geo: or a maps link a map, market:// the Play Store). Nothing is sent or called by opening a link. For public web pages you only need to read, the browser_* tools work without taking over the screen. Returns "Opened"; call read_screen to see it.

Take a screenshot (JPEG). Use only when read_screen is not enough (images, games, canvas UIs). The reply says how to convert image coordinates to screen pixels.

Wait for the UI to settle, up to 10000 ms.

Ask the user to approve an action in the Buddy bubble and wait for the answer. REQUIRED before sending messages, posting, buying, deleting, or changing settings. Show the exact text or action. Returns APPROVED, DENIED, or CHANGES REQUESTED with what the user wants changed: then make those changes and call confirm again.

Ask the user a question in the Buddy bubble and wait for the answer: 2 to 4 answer buttons, and the user can always type their own answer instead. Use it when you need the user to choose or decide (which reply, which item, which time) instead of guessing. With highlight, it also hands a step over to the user (a password or code, their details in a form, a file picker, a captcha): e.g. "Type your password, then tap Done" with options ["Done", "I can't"] and the field in highlight. Returns ANSWER with the option picked, ANSWER (typed) with the user's own words, or NO ANSWER. It is not an approval: still call confirm before anything outward-facing.

Show the user where to look or what to do on the screen: a box, circle or arrow with a short label around up to 6 elements, drawn over the app without blocking their touches. When the user does what a mark points at (taps that button, flips that switch, fills in that field and moves on), it turns green and goes. With wait=true the call returns only then (DONE: what they did) or when they don't (GONE, LEFT, NOT DONE): use it to guide the user step by step through something they want to learn to do themselves, one step per call, instead of doing it for them. Marks follow their element as the screen scrolls, and go when it's gone, when the app changes, when you use another phone action, or after seconds. Call with no marks to remove them.

Show a short status line in the Buddy bubble without stopping.

Answer the notification that started this chat through its own reply field, without opening the app. Only in chats started by one of the user's notification rules, when the notification has a reply field. Buddy shows the user an approval card with the text first, unless you called confirm with it just before.

(Needs Linux tools) Open Android's installer for an APK built in Buddy's Linux, e.g. /root/.buddy/work/myapp/build/app.apk, or served by another app on this phone at a local URL (http://127.0.0.1:/app.apk, e.g. from Termux). The user taps Install/Update themselves; call wait_for_install next.

Check whether a newer Buddy is out (the official GitHub release) and what's new; with install=true, download it and open Android's installer. Use it when the user asks to update Buddy or whether there's a new version. Only the user can tap Update.

After opening the Android installer for an APK, wait until the user has tapped Install/Update and confirmed. Returns as soon as the package is installed (at least min_version_code if given), or early if the installer is no longer on screen, or after the timeout. Use this instead of shell polling loops.

Open a web page in Buddy's background browser (a separate browser the user can watch in this chat; it doesn't use the phone's screen). Returns the page as text, like browser_read.

Read the whole page in the background browser as text. Items you can use are numbered, e.g. [12] button "Add to cart"; the numbers stay valid until the page changes. Long pages come in parts: pass from to read on.

Tap an item on the page by its number from browser_read (a link, button, checkbox, …).

Type into a text field by its number from browser_read (replaces its text unless append is true), or pick an option of a select by its text. Set submit to press Enter afterwards.

Scroll the page in the background browser (loads more on endless pages).

A picture of what the background browser shows, for images, maps or layouts browser_read can't describe.

Go back to the previous page in the current tab of the background browser, like a browser's back button. If the page warns that leaving loses unsaved changes, Buddy stays the first time and says so. Returns where the tab is now, or that there's no previous page; then call browser_read.

Manage the background browser's tabs (this chat's own): with no arguments, list them as numbered lines with title and address, the current one marked; with switch_to, make that tab current (the browser_* tools act on the current tab); with close, close it. Tabs open with browser_open new_tab or when a page opens a link in a new tab.