# Agentic Android

Control an Android phone or emulator with an LLM over ADB. The agent reads the
screen, picks an action (tap, swipe, type, launch an app), runs it through `adb`,
checks the result, and repeats until the task is done. No root, and nothing to
install on the device.

It works with Claude (via the `claude` CLI or the Anthropic API), Codex CLI,
OpenAI or any OpenAI-compatible endpoint, and local models through Ollama or
LM Studio.

<p align="center">
  <img src="assets/demo-wallpaper.gif" alt="Agentic Android installing a wallpaper app from the Play Store and setting a new wallpaper, with the agent's console beside it" width="700">
  <br>
  <em>The phone is on the left, the agent's console on the right. The task: install a wallpaper app from the Play Store and set a new wallpaper. It installs Zedge, corrects a wrong package name with <code>list_apps</code> (<code>com.zedge.android</code> → <code>net.zedge.android</code>), and when Zedge's preview returns a blank screenshot it switches to reading the UI tree with <code>dump_ui</code> to finish. Brain is the free <code>claude-cli</code>; no API key.</em>
  <br><br>
  <a href="assets/demo-wallpaper.mp4">MP4</a> (2.2 MB) · <a href="assets/demo-wallpaper.gif">GIF</a>
</p>

## What you can ask it

You give it a task the way you'd text a friend who's holding your phone, and it
works through the apps to do it:

- "Send a sweet good night text to my girlfriend"
- "Reply to Mom's last WhatsApp and tell her I'll call her tomorrow"
- "Install Spotify and play something chill"
- "Set an alarm for 6:30 and turn on Do Not Disturb until then"
- "Order my usual from the Starbucks app"
- "Book an Uber home"
- "Go through my notifications and tell me what actually needs a reply"
- "Find a butter chicken recipe and read me the ingredients"
- "Download a wallpaper app and set a new wallpaper" (the demo above)

It only sends, posts, buys, or signs in when you tell it to. For anything it can't
undo (a purchase, a 2FA code, a password) it stops and asks first, even at high
persistence. See [Persistence level](#persistence-level-05-and-cost) to tune how
much it does on its own before checking in.

## Providers

Set `provider` in `agentic-android.toml`.

| `provider`   | Backend | Needs |
|--------------|---------|-------|
| `claude-cli` | Your logged-in **Claude Code** CLI, run as a live chat | the `claude` CLI, no API key |
| `codex-cli`  | Your logged-in **Codex CLI**, run as a live chat with the Android MCP tools | the `codex` CLI, no app API key |
| `anthropic`  | Anthropic API | `ANTHROPIC_API_KEY` |
| `openai`     | OpenAI, or any OpenAI-compatible endpoint (OpenRouter, Azure, a local server) | `OPENAI_API_KEY` |
| `ollama`     | A local model via Ollama | nothing, runs on-device |
| `lmstudio`   | A local model via LM Studio | nothing, runs on-device |

The agent reads the screen one of two ways: as a screenshot (vision mode), or as
a text list of on-screen elements parsed from the UI tree (text mode). Text mode
is cheaper and faster and works with text-only models that can't see images. Both
are configured in `agentic-android.toml`.

## Requirements

1. A device: a phone with USB debugging on, or an emulator. A networked emulator
   works too (for example MuMu or an emulator VM at `192.168.1.79:5555`).
2. adb: the `adbutils` dependency ships a bundled binary, so you don't need to
   install adb yourself. A system `adb` on PATH is used if there is one. Check
   with `python -m agentic_android --list-devices`.
3. A brain. Pick one and set it in `agentic-android.toml`:
   - `claude-cli`: the `claude` CLI installed and logged in (run `claude` once).
     No API key. This is the default.
   - `codex-cli`: the `codex` CLI installed and logged in (run `codex login`
     once). No app API key.
   - `anthropic`: an Anthropic API key.
   - `openai`: an OpenAI key, or any OpenAI-compatible endpoint via a custom base
     URL.
   - `ollama`: a local model, no key. Auto-detects the loaded model from
     `http://localhost:11434/v1`. The model needs tool calling (for example
     `qwen3`, `llama3.1`/`3.2`, `mistral-nemo`); vision models such as `qwen3-vl`
     or `llava` can set `vision = true`. Uses Ollama's native API so it can set a
     usable context window, since the 4096 default is too small.
   - `lmstudio`: a local model via LM Studio, no key. Start the server (Developer
     tab), load a tool-calling model (Gemma 4, Qwen, Llama 3.x), and it
     auto-detects it from `http://localhost:1234/v1`. Set a generous context
     length when you load the model.

## Install

Use the launcher. It creates its own virtualenv on first run, so it never touches
the system Python:

```bash
./run.sh --list-devices          # Linux/macOS
run.bat --list-devices           # Windows (or double-click run.bat)
```

`run.sh` and `run.bat` are wrappers around `python -m agentic_android`, so any
argument in this README works after them (for example
`./run.sh --provider openai "…"`).

To install manually instead:

```bash
cd /mnt/windows/Work/Ai/projects/agentic-android
python3 -m venv .venv
.venv/bin/pip install -e .        # or: pip install -r requirements.txt
.venv/bin/python -m agentic_android --list-devices
```

Use the venv's Python, not the system one. Running `python -m agentic_android`
with `/usr/bin/python` fails with "No module named agentic_android", because that
Python doesn't have the package. Use `./run.sh`, activate the venv
(`source .venv/bin/activate`), or call `.venv/bin/python` directly.

Pillow is optional. It only downscales large screenshots to cut token cost.
Without it (for example on Python 3.14 where wheels may be missing) the agent
sends full-resolution screenshots and still works.

## Configuration

Copy the template to create your own config (it holds your API keys and device
serial, so it is git-ignored and never committed):

```bash
cp agentic-android.toml.example agentic-android.toml
```

Everything you'd tweak lives in `agentic-android.toml`, which is commented. The
three settings that matter most:

```toml
provider = "claude-cli"          # claude-cli | codex-cli | anthropic | openai | ollama | lmstudio

[device]
serial = "192.168.1.79:5555"     # your phone/emulator (see --list-devices)

[agent]
effort = 3                       # 0 = ask early and cheap … 5 = try everything
```

Then fill in the section for your provider. For an OpenAI-compatible server, point
`[openai].base_url` at it:

```toml
[openai]
api_key  = ""                            # or set env OPENAI_API_KEY
base_url = "https://api.openai.com/v1"   # OpenRouter, http://localhost:11434/v1, etc.
model    = "gpt-4o"                       # must be vision-capable for vision mode
```

Run with no arguments and it uses the config:

```bash
python -m agentic_android
```

### Unlocking the phone

Give the agent your lock-screen passcode and it stops being blocked by a sleeping
phone: it wakes the device and unlocks it at the start of a run, again if the screen
goes dark mid-run, and whenever it calls the `unlock_device` tool.

The passcode is a secret, so it lives in `.env` only — never in
`agentic-android.toml` (which you might commit) and never in a CLI flag (which
lands in your shell history and the process list):

```bash
# .env
AGENTIC_ANDROID_PASSCODE=1234
AGENTIC_ANDROID_UNLOCK_METHOD=pin    # pin (digits) | password (alphanumeric)
```

```toml
# agentic-android.toml
[device]
auto_unlock = true                   # false = only unlock when the agent explicitly asks
unlock_method = "pin"
```

Unlock the device by hand and exit — handy for checking your setup:

```bash
python -m agentic_android --unlock
```

How it works: wake with `KEYCODE_WAKEUP`, swipe the lock-screen curtain away
(enough on its own for a swipe-only lock), then type the passcode — a PIN as digit
key events, since the PIN pad ignores `input text` on many ROMs. Lock state is read
back from `dumpsys` (`deviceLocked`, `isKeyguardShowing`, `mWakefulness`) before and
after, so the unlock is verified rather than assumed.

Some deliberate limits:

- The model never sees the passcode. It calls `unlock_device`, which takes no
  arguments; the code is read from the environment inside the device layer and is
  never logged, echoed, or put in a tool result. The prompts tell the agent not to
  ask you for it either.
- The passcode is only ever typed when the keyguard is confirmed to be up. If the
  lock state can't be read, nothing is typed — better a stalled run than a PIN
  typed into a chat box.
- Three wrong attempts per process and it stops trying, since Android escalates on
  repeated failures (long lockouts, and a wipe on devices set to auto-reset). The
  error tells you to check the passcode.
- Pattern locks aren't supported — only PIN and password.

## Reliability and diagnostics

A few things keep runs working outside a demo. They're on by default and tunable
in `[agent]`:

- Blank-screen fallback. Some app surfaces can't be screen-captured and come back
  black. The agent detects this (a dead capture is a tiny PNG) and reads the screen
  from the UI element list instead, so it keeps going rather than stalling.
- Wait-for-idle. After each action it waits for the screen to stop changing (up to
  `settle_timeout`) instead of a fixed delay, so it doesn't act on a half-loaded screen.
- Retries and timeouts. Transient `adb` failures (a dropped network device) are
  retried with backoff and a reconnect; LLM calls have a timeout so a stalled
  connection can't hang the run.
- Confirmation gate (opt-in). Set `confirm_destructive = true` (or pass
  `--confirm-destructive`) and the agent stops for a `y/N` before high-risk actions:
  uninstall, buy/pay, delete, factory reset, sign out.

Run a preflight check anytime:

```bash
python -m agentic_android --doctor
```
```
Agentic Android — preflight doctor
  provider=ollama · device=192.168.1.79:5555

  [PASS] adb            bundled adbutils binary
  [PASS] device         192.168.1.79:5555 connected
  [PASS] screenshot     1080x2400 PNG, non-blank
  [FAIL] ollama server  no response from http://localhost:11434/v1
           → Start `ollama serve`.
```

The same checks run automatically before each task (skip with `--no-preflight`), so
setup problems show up as a clear message instead of a cryptic error. If exactly one
device is attached it's selected automatically, so `--serial` is optional.

## Usage

### Live chat through the `claude` CLI (no API key)

This is the default. It spawns a headless `claude` agent using your logged-in
Claude Code subscription (no `ANTHROPIC_API_KEY`), connects the device tools over
MCP, and relays your terminal to and from the agent. You type, it acts, and you
can keep typing to steer it while it works.

```bash
python -m agentic_android --chat -s 192.168.1.79:5555
```
```
you> Install WhatsApp on my phone
   · screenshot {}
agent> Opening the Play Store and searching for WhatsApp.
   · launch_app {"package":"com.android.vending"}
   · tap {"x":360,"y":140}
   · type_text {"text":"WhatsApp"}
   ...
agent> WhatsApp is installing. I'll tell you when it's done.
you> also install Telegram while you're at it          # steer mid-task
```

Chat needs the `claude` CLI installed and logged in (run `claude` once). Options:
`--model sonnet|haiku|opus` (default `sonnet`), `--budget 2` for a USD cap (passed
to `claude --max-budget-usd`). The agent runs with its working directory set to
this project, so its Claude Code session history stays separate from other
projects.

### Live chat through the `codex` CLI (no app API key)

Set `provider = "codex-cli"` or pass `--provider codex-cli`. It launches your
installed Codex CLI in JSON mode, attaches the same Android MCP server, and
renders Codex's stream through Agentic Android's terminal UI. Codex uses your
normal Codex CLI authentication and defaults unless you set `[codex_cli].model`
or pass `--model`.

```bash
python -m agentic_android --provider codex-cli -s 192.168.1.79:5555
python -m agentic_android --provider codex-cli "Open Settings and tell me the Android version"
```

Chat needs the `codex` CLI installed and logged in (`codex login`). The Codex path
uses `codex exec --json`, so output stays in the same `agent>` / tool-call style
as the rest of Agentic Android.

### How it works: the MCP device server

The interesting part of the CLI paths is that the agent is just Claude Code or
Codex CLI, with your phone handed to it as tools over
[MCP](https://modelcontextprotocol.io) (the Model Context Protocol). You reuse
the CLI's whole agent loop — planning, tool use, recovery — pointed at a phone,
running on your existing CLI login, so there is no app-side API key.

When you run `--chat` / `--provider claude-cli`, `chat.py` does three things:

1. Writes a temporary MCP config that points at `python -m agentic_android.mcp_server`,
   a small [FastMCP](https://modelcontextprotocol.io) server that talks over stdio.
   The device serial, adb path, and the reliability settings are passed to it as
   environment variables.
2. Spawns the headless agent with that server attached:
   ```
   claude -p --input-format stream-json --output-format stream-json \
     --permission-mode bypassPermissions \
     --allowedTools ToolSearch,mcp__agentic_android \
     --mcp-config <tmp.json> --strict-mcp-config \
     --append-system-prompt "<device rules>" --model sonnet
   ```
3. Relays both directions. It prints the agent's stream-json events (its text and
   each tool call) as they arrive, and feeds whatever you type back in as a user
   message — so you can steer mid-task without stopping it.

For `--provider codex-cli`, `codex_chat.py` starts `codex exec --json --cd
<project>` and passes the same MCP server through temporary `-c
mcp_servers.agentic_android.*` configuration overrides, leaving your normal Codex
config file untouched.

The server (`agentic_android/mcp_server.py`) exposes the device as MCP tools:
`screenshot`, `tap`, `tap_element`, `swipe`, `type_text`, `press_key`,
`launch_app`, `list_apps`, `dump_ui`, and `unlock_device`. Every screen-changing tool returns a
fresh screenshot in its result, so the agent sees the effect of each action (the
computer-use loop). If a capture comes back blank/black, it returns the UI element
list instead, so a protected surface can't blind it.

Why this is nice:

- No app-side API key. The CLI providers use the account/auth you already set up
  in their own CLIs.
- You get the whole agent loop from the CLI. Planning, retrying, and tool chaining
  stay in the provider; this project only has to expose adb cleanly.
- MCP is portable. `agentic_android` is a standard MCP server, so any MCP client
  (Claude Desktop, Codex CLI, another agent) can attach it and drive the phone too — nothing
  is tied to this CLI.
- It's genuinely multimodal: the tools hand back real screenshots, not just text.
- It's isolated and autonomous. It runs in this project's directory with the
  Android MCP server attached for that session.

### Persistence level (0–5) and cost

When the agent gets stuck or hits an ambiguous choice, it asks you a question with
options instead of guessing. How hard it tries to recover on its own before asking
is the persistence level, 0 to 5, set in `agentic-android.toml` or with
`--effort N`:

| Level | Behaviour |
|------:|-----------|
| 0 | Ask at the first ambiguity or failure; almost no self-recovery |
| 1 | One quick retry, then ask |
| 2 | A couple of recovery attempts, then ask |
| 3 | Default. Several strategies (re-screenshot, `dump_ui`, scroll, alternate path) before asking |
| 4 | Exhausts visual and UI-tree approaches; asks only when truly blocked |
| 5 | Tries everything; asks only for what it can't resolve (passwords, 2FA, purchase confirmation) |

Higher levels make more screenshots, tool calls, and model turns before the agent
pauses, so they cost more. Lower levels check in sooner and spend less.

```bash
python -m agentic_android --chat -s 192.168.1.79:5555 --effort 4   # try hard before asking
python -m agentic_android --chat -s 192.168.1.79:5555 --effort 0   # ask early, spend little
```

When it asks in chat, type your answer (a number or free text) and it continues.
The API-agent path uses an `ask_user` tool that prompts in the terminal.

### API agent (`anthropic` or `openai`)

Set `provider` to `anthropic` or `openai` in the config or with `--provider`. The
loop is provider-agnostic: it keeps one ongoing conversation, sends a fresh
screenshot with each message, and acts. `ask_user` lets it ask a question with
options when stuck.

```bash
# OpenAI, or any OpenAI-compatible endpoint (note the custom --base-url)
python -m agentic_android --provider openai "Open Settings and turn on Airplane mode"
python -m agentic_android --provider openai                  # interactive chat

# Local models, no key (auto-detects the loaded model)
./run.sh --provider ollama "Open the Chrome browser app"
./run.sh --provider lmstudio "Open the Chrome browser app"

# Anthropic
python -m agentic_android --provider anthropic --model claude-opus-4-8
```

Keys come from the config file or the environment (`OPENAI_API_KEY`,
`ANTHROPIC_API_KEY`). Example tasks: "Open the Clock app and set an alarm for 7:30
AM", "In Settings, find the Android version".

### Text mode (for models that can't see screenshots)

Many models, including most local and text-only ones, can't read images. Set
`vision = false` (or pass `--no-vision`) and the agent stops sending screenshots.
Instead it describes the screen as a numbered list of on-screen elements parsed
from the Android UI tree, each with a tap point:

```
Screen 720x1280px — 45 elements. Tap with the @(x,y) point.
#1 [ImageView] "Back" @(48,57) [tap]
#4 [ImageView] "Open notification settings" @(560,57) [tap]
#9 [View] "1,380 posts" id=profile_header_post_count @(298,236) [tap]
#20 [EditText] "Search" id=search_input @(360,140) [tap,input]
```

The model taps the `@(x,y)` of the element it wants (coordinates are real device
pixels). Enable it per provider:

```toml
[openai]
model  = "llama3.2-vision"   # or any text model
vision = false
```
```bash
python -m agentic_android --provider openai --base-url http://localhost:11434/v1 --no-vision
```

Vision is still better when the model supports it. Text mode is the fallback that
makes non-vision models usable.

### Skills: learn once, run free

Record a task the first time (the LLM does it) and the steps are saved as a
deterministic, self-healing skill. Replaying it drives the device with no LLM calls;
if the UI changed, it repairs just the affected step and updates the skill.

```bash
# record (needs an API provider — the controllable loop)
python -m agentic_android --provider openai --record "open the Clock app and start a timer" --as timer
python -m agentic_android --run-skill timer      # replay, ~zero LLM cost
python -m agentic_android --skills               # list saved skills
```

Skills live in `data/skills/<name>.json` (git-ignored). On replay the agent first
tries a mined `am start` shortcut to jump past navigation, then runs each step,
verifying it against the screen it expected; a step that no longer matches is healed
with one LLM call and the fix is written back, so the cost is paid once per change,
not every run. You can also turn a `--chat` session's debug log into a skill with
`--import-trace debug/session-*.jsonl`. Secrets typed into password/OTP fields are
never stored — those steps are marked for a live rerun instead.

### Guardrails and self-verifying tasks

Give a task rules it must respect and a success condition to verify. Deterministic
rules are checked from the UI tree after every step (free); a natural-language
success/forbidden check is judged by the model at the end (and optionally every N
steps with `--judge-frequency`).

```bash
python -m agentic_android --provider openai "open my cart and check the total" \
  --stay-in-app com.amazon.mShop.android.shopping \
  --forbid "Buy now" --forbid "Place order" \
  --assert "the cart total is shown and no order was placed" \
  --on-violation rewind
```

On a violation it stops (default), rewinds toward the last good screen and tries
another path (`--on-violation rewind`), or asks you (`ask`). At the end it prints a
verdict (which rules held, whether success was verified) and exits non-zero (code 3)
if a guardrail failed — useful in scripts. Rules can also live in a file:
`--guardrails task.toml`. On the `claude-cli` and `codex-cli` paths the rules are
enforced as a device-level block on forbidden taps plus instructions in the
agent's prompt; the API providers enforce them in the loop.

### A nicer CLI (styled output + inline screenshots)

With `rich` installed (`pip install rich`, or the `ui` / `full` extra) the chat gets a
status bar, panels, and colored tool calls. In a terminal that supports inline images
(kitty, iTerm2) it also draws the phone screen after each action; elsewhere it falls
back to text. Control it with `--ui auto|rich|plain` and `--inline-screen auto|on|off`
(or the `[ui]` section). Piped or redirected output is always plain with no image
escapes, so recordings and logs stay clean.

## Debugging

Add `--debug` (or `[agent] debug = true`) to save every API request and response
for the run to `debug/session-<timestamp>.jsonl`, one JSON object per line:

```bash
./run.sh --provider openai --no-vision --debug "Open the Clock app"
```

- For `anthropic` and `openai`: each request (model, messages, tools) and its
  response, including token `usage`. Base64 screenshots are redacted to a size
  marker so the file stays readable; the text element list is kept as-is.
- For `claude-cli`: the agent's full raw stream-json (every tool call, result, and
  message).
- For `codex-cli`: Codex's JSON event stream is saved, including agent messages,
  MCP tool calls, and usage events.

`debug/` is git-ignored. Use `--debug-dir <path>` to change where it's written.

## Layout

| File | Role |
|------|------|
| `agentic_android/adb.py` | Thin adb wrapper: screencap, input tap/swipe/text/key, uiautomator dump, app launch, keyguard state + passcode entry. |
| `agentic_android/device.py` | Screenshot capture with optional downscaling; maps image coordinates back to real device pixels; wake/unlock. |
| `agentic_android/mcp_server.py` | MCP stdio server exposing the device tools (used by the CLI providers). |
| `agentic_android/chat.py` | Spawns a `claude` CLI agent with the MCP server attached and relays the terminal chat. |
| `agentic_android/codex_chat.py` | Spawns `codex exec --json`, attaches the MCP server through temporary config overrides, and relays events through the app renderer. |
| `agentic_android/brains.py` | Provider backends (`AnthropicBrain`, `OpenAIBrain`, `OllamaBrain`) behind one interface. |
| `agentic_android/agent.py` | Provider-agnostic agent loop; one-shot `run()` and interactive `chat()`. |
| `agentic_android/tools.py` | Tool schemas (converted to OpenAI function format by `brains.py`). |
| `agentic_android/config.py` | Loads `agentic-android.toml` and the persistence-level text. |
| `agentic_android/doctor.py` | Preflight checks (`--doctor`), device auto-detection, provider reachability. |
| `agentic_android/skills.py` | Record / replay / self-heal skills (learn once, run free). |
| `agentic_android/guardrails.py` | Forbidden-state + success-assertion checks, verdict, rewind. |
| `agentic_android/ui.py` | Styled CLI output + inline screenshots (optional `rich`). |
| `agentic_android/__main__.py` | CLI: provider, device, and dispatch. |

The API agent defaults to `claude-opus-4-8`. Coordinates the model returns are in
the shown image's pixel space; `Device.scale` converts them to physical pixels so
taps land correctly even when screenshots are downscaled.

## Notes and limits

- `type_text` goes through `adb shell input text`. It escapes common characters
  but works best with plain ASCII. For unusual unicode, use an on-screen keyboard
  or a per-app text method.
- This automates whatever device you point it at. Review a task before running a
  destructive one, and turn on `confirm_destructive` to be prompted before
  high-risk actions (uninstall, buy, delete, factory reset, sign out).
- A passcode in `.env` is stored in plain text, like any other credential there.
  Anything that can read that file can unlock your phone while it's connected —
  leave `AGENTIC_ANDROID_PASSCODE` unset on a machine you don't trust, and the
  agent will simply report that the phone is locked. Pattern locks are not
  supported (PIN and password only).
- One action set per step keeps the loop readable. Raise `--max-steps` for longer
  flows.

## Possible additions

- `scroll_to_text` / `tap_text` helpers built on `dump_ui` for steadier targeting.
- A persistent navigation graph for fast cross-app routing; multi-device fan-out.
- Trigger/watch mode (act when a notification or screen condition fires).

## License

MIT. See [LICENSE](LICENSE). Copyright (c) 2026 Bhavik Patel.