Drives a physical Android phone unrooted as uid 2000: reads the screen as a ~1-3 KB list of typed elements instead of pixels, then taps, types (any Unicode, verified by reading the field back) and swipes by element reference, launches apps and deep links. Exposes tool packs for phone capabilities (SMS, calls, camera, GPS, sensors, torch, TTS, notifications, Wi-Fi), system state (battery, network, storage, clipboard, waiting for a screen), file read/write/search across both filesystems, and shell commands at both privilege levels, plus a pack for teaching the agent an unseen app by exploring and saving selectors. Goals can be delegated from a laptop over MCP, with read-only vs destructive tools annotated and risky taps (sends, payments, logins/2FA/CAPTCHA checkpoints) gated by policy.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@DroidPilot start a phone task to check my battery, then show its status"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
An LLM agent that runs on your Android phone and drives it. It reads the screen as structured data, taps, types and swipes, and can be handed whole tasks from a laptop over MCP. Unrooted: no root, no Shizuku.
droidpilot run "What Android version is this phone running?"
# the agent calls device_info, reads "Android 12", answers, and stops:
# one step, recorded to artifacts/runs/ and verifiable with `droidpilot verify`This is a full-control agent. Run it on a phone you treat as expendable. It has a shell, your SMS, your camera and your files. See Safety.
13 ms screen reads | A custom AccessibilityService ( |
Shell privilege, unrooted | Termux connects adb to the phone's own adbd over loopback, so the phone drives itself as uid 2000. One Android dialog, once. ARCHITECTURE.md |
Screens as data, not pixels | A ~70 KB view hierarchy becomes a 1–3 KB list of typed elements. No vision model in the loop. |
Packs keep cheap models cheap | 102 tools in 6 packs; only |
Taps are judged by target | "Send", "Pay now", "Log out" and typing into a comment box are refused unless the operator allows that rule for the run, whatever tool the model used. |
It notices when it is stuck | The same call on an unchanged screen gets a warning, then ends the run, instead of burning the step budget. |
Delegate, don't proxy | A laptop sends one goal; a cheap model on the phone runs the 30 steps against localhost. One expensive turn instead of thirty. |
Measured on a realme narzo 50 Pro (Android 14) and a Samsung M21 (Android 12). Every number has its method in BENCHMARKS.md.
Related MCP server: mcp-android
On the phone, in Termux (the GitHub build, not the Play Store one: the add-ons need matching signatures):
curl -sL https://raw.githubusercontent.com/Arnavvs/droidpilot/main/scripts/bootstrap_phone.sh | bashIt installs the packages, performs the adb self-connect, waits for you to tap
Allow, installs DroidPilot, and runs doctor to prove the chain works.
Full walkthrough, including the reboot caveat: SETUP-PHONE.md.
Optional, for the 13 ms backend (needs a JDK and the Android SDK on a laptop):
cd android && ./build.sh installexport OPENROUTER_API_KEY=sk-or-v1-...
export DROIDPILOT_MODEL=deepseek/deepseek-v4-flash-0731or run offline against llama.cpp on the phone (DROIDPILOT_PROVIDER=local,
DROIDPILOT_TOOL_MODE=json). See MODELS.md.
droidpilot doctor # check every link in the chain
droidpilot run "<goal>" # give it a task, watch it work
droidpilot run --mode readonly "<goal>" # it cannot change anything
droidpilot tools --pack phone # browse the tool surface
droidpilot tool ui_dump --args '{"limit":10}' # call one tool, no model
droidpilot runs # every run, with verdicts
droidpilot verify latest --expect '<regex>' # label a run pass/fail
droidpilot macro make latest --name X # a verified run becomes a guarded macro
droidpilot macro run X # replay it with no model; stops on a mismatch
droidpilot serve --host 0.0.0.0 # accept tasks from a laptop# phone
droidpilot serve --host 0.0.0.0
# laptop
export DROIDPILOT_URL=http://<phone-ip>:8765
export DROIDPILOT_TOKEN=<token printed by serve>
claude mcp add droidpilot -- python bridge/mcp_bridge.pyThe laptop gets phone_task (delegate a goal), phone_tool (one call),
phone_task_status / phone_task_stop / phone_reply for background tasks,
and phone_run_inspect. Every tool carries MCP annotations: read-only tools are
marked so a client can auto-approve them, and sends, shell and file writes are
marked destructive.
Pack | Tools | What |
| 37 | Screen reading, tapping, typing, swipes, app launch, deep links, notes and recall, diagnose, hand-back to a person, tool discovery |
| 29 | SMS, calls, camera, GPS, sensors, torch, TTS, notifications, Wi-Fi (via Termux:API) |
| 14 | Battery, network, storage, clipboard, waiting for a screen |
| 10 | Teach the agent an app it has never seen: explore, diff, propose and save selectors |
| 8 | Read, write, search and move files across both filesystems |
| 4 | Arbitrary commands at both privilege levels |
Full catalogue: TOOLS.md. Adding a tool is one decorated function; adding an app is a selector file plus a module. See CONTRIBUTING.md.
See, act, see again. Every tap goes by a ref from the last read
(ref="<ver>_<i>"), and the element is found again on a fresh read before the
tap. If it moved, the tap follows; if it is gone, covered or ambiguous, the tap
is refused with the reason.
Text entry is verified. Text goes in through the bridge (any Unicode) and
is read back from the field; plain adb input text crashes on Devanagari and is
only an ASCII fallback.
Verification is a program, not a vibe. A task can declare checks (answer regex, text reached, final app) and each run is labelled pass, fail or inconclusive automatically.
Old results become one-line capsules that say what was observed, never "success", so the model's own history cannot mislead it later.
Everything is recorded, before compaction, as JSONL, with a decision record per step (candidates, choice, outcome).
Design and the measurements behind it: ARCHITECTURE.md.
Use an expendable phone.
phone_sms_send and phone_call are denied by default even in auto
mode, because they reach real people and cost money. Enable them by name.
Taps are judged by what they hit (policy/writes.json): sends, payments
and account-level actions are refused unless allowed for the run, and typing
into a comment or message box counts as a send.
A checkpoint stops the run. Login, 2FA, CAPTCHA, "unusual activity" and
"action blocked" screens end the run with human_required, whatever the model
intended.
The HTTP server binds loopback unless you pass --host, and then requires a
bearer token. The accessibility bridge requires its own token, readable only
by adb shell, and refuses browser-originated requests.
policy/reads.py holds hook points for rate-limiting reads. The default build
counts nothing; plug in your own limiter if you automate an app with limits.
Out of scope by design: CAPTCHA solving, device-identity spoofing, root exploits, consent-dialog bypass, and anything meant to make automation look human to defeat a platform's enforcement. Automating an app does not change what its terms allow.
pip install -e . pytest
python -m pytest -q # 238 tests, no phone neededThe device layer, structured-UI extraction and selector registry come from MobileAgentMCP. DroidPilot is the open-source core of a larger private build; app-specific packs are not included.
MIT licensed.
This server cannot be deployed
Melaya is a remote MCP server. It gives an assistant hands on your own Android phone and browser: it reads the screen through the accessibility tree, then taps, types and navigates inside the apps and sites you allow-list, with no per-app API. It also builds, schedules and runs agent pipelines across 6k+ connected tools. OAuth 2.1, nothing to install.
Reach your own phone from an AI agent: notifications, approval questions, reminders, ring, files.
Operate Linux, macOS and Windows from your LLM. Every action runs through an auditable allowlist.
Control real Android and iOS devices with LLM agents — tap, swipe, type, automate flows.
