# android-agent

An [MCP](https://modelcontextprotocol.io) server that lets a coding agent (Claude Code, Cursor, or
anything else that speaks MCP) **build an Android app, drive it on a real device, and debug it** —
instead of editing Kotlin blind and asking you what the screen shows.

- **Screenshots come back as real image blocks**, so the model looks at the pixels directly.
- **Video goes through a local vision model** ([LM Studio](https://lmstudio.ai)), so a 60-second
  recording returns as a paragraph instead of 200 frames of context.
- **UI dumps are ref-addressable** (`n1`, `n2`, …), so taps hit real controls rather than guessed
  coordinates.
- **Builds don't fight Android Studio.** They run in Studio's own Gradle daemon pool, wait while
  Studio is building, and name the process holding a locked file. No more
  *"R.jar: The process cannot access the file because it is being used by another process"*.
- **Failures come back complete**: compile errors with source context, crashes and ANRs with stack
  frames mapped to your files, install errors explained — plus a self-contained brief you can hand
  to a fixer subagent.

Scope is live UI driving plus build/install. It does not run Espresso/JUnit suites.

---

## Install

Needs [uv](https://docs.astral.sh/uv/) and the Android SDK platform-tools (`adb`).

**Claude Code:**

```bash
claude mcp add android-agent --scope user -- uvx --from git+https://github.com/Fdondi/android-agent-mcp android-agent
```

**Cursor** — in `~/.cursor/mcp.json` (or `.cursor/mcp.json` in a project):

```json
{
  "mcpServers": {
    "android-agent": {
      "command": "uvx",
      "args": ["--from", "git+https://github.com/Fdondi/android-agent-mcp", "android-agent"]
    }
  }
}
```

Restart the host (or reload its MCP servers) afterwards. To pin a version, append `@v0.1.0` to the
git URL.

Check the install, which verifies binaries, devices, the crash log, Gradle daemons and the local
model in one go:

```bash
uvx --from git+https://github.com/Fdondi/android-agent-mcp android-agent --selftest
```

### Requirements

| Piece | Needed for |
| --- | --- |
| Android SDK platform-tools (`adb`) | Everything. Found via `ANDROID_HOME`, else the usual per-OS SDK location. |
| A device or emulator with USB debugging | Everything. |
| Android SDK emulator | Only `android_emulator_start`. Resolved by path, not `PATH`. |
| Gradle wrapper in your project | Only `android_build` / `reinstall=True`. |
| ffmpeg | Only video analysis. |
| LM Studio with a vision (VLM) model | Only `android_assert`, `android_record_stop`, `android_analyze` and `analyze=`. |

### Platform support

Developed and tested on **Windows 11**. The device, build, crash and vision features are plain
`adb`/Gradle and are portable. The Gradle **daemon inventory and file-lock detection** are
OS-specific: Windows uses the Restart Manager API and `Get-CimInstance`; macOS and Linux use
`/proc` or `lsof` plus `ps`. The POSIX paths are implemented but **have not been exercised on a
real macOS or Linux machine** — bug reports welcome.

---

## Tools

### Build, install, run
| Tool | What it does |
| --- | --- |
| `android_build(project_dir, tasks, install, release_locks)` | Lock-safe Gradle build. On failure returns a full diagnostic report plus a brief path. `tasks=[":app:compileDebugKotlin"]` gives a fast compile-only check. |
| `android_install(project_dir \| apk, allow_uninstall)` | `adb install -r -d -t`, with `INSTALL_FAILED_*` codes explained. |
| `android_session_start(package, project_dir, reinstall, fresh)` | The entry point: optionally build+install, launch, and return **screenshot + UI dump**. A crash on launch returns the mapped stack trace instead. |
| `android_prepare(package)` | Grants what `pm grant` cannot: usage-stats and overlay appops, notifications, optionally an accessibility service. |

### Look and interact
| Tool | What it does |
| --- | --- |
| `android_screenshot(scale, analyze)` | Screenshot as an image block; `analyze="..."` adds a local-VLM answer. |
| `android_ui_dump(compact)` | Ref-addressable elements: text, content-desc, class, centre, bounds, flags. |
| `android_tap(ref \| text \| x,y)` | Tap by ref, visible text, or coordinates. |
| `android_swipe(direction \| x1,y1,x2,y2)` | Scroll or gesture. |
| `android_type(text, ref, submit)` | Type into a field (ASCII only — an adb limitation). |
| `android_key(key)` | `BACK`, `HOME`, `ENTER`, `APP_SWITCH`, … |
| `android_wait(ms \| until_text \| until_gone)` | Wait for a condition instead of screenshotting mid-animation. |
| `android_devices()` / `android_emulator_start(avd)` | List devices and AVDs; boot an AVD and wait for it. |

Every interaction result checks the device's crash log, so a crash is never silent: results carry
`!! APP CRASHED` or `!! APP NOT RESPONDING`.

### Diagnose
| Tool | What it does |
| --- | --- |
| `android_crashes(package, project_dir, only_new)` | Java exceptions, native signals and ANRs, with app frames mapped to source and a fixer brief. |
| `android_logcat(package, grep)` | Recent logcat scoped to the app's pid. |
| `android_gradle_daemons(project_dir, stop)` | Every Gradle/Kotlin daemon with its user home, JDK and IDLE/BUSY state, and which hold your build outputs. `stop="lockers" \| "foreign" \| "all-idle"` stops idle ones only. |
| `android_vlm_log(last, call, full)` | Which local-model calls were made, by which tool, and what the model saw and said. |

### Local vision model
| Tool | What it does |
| --- | --- |
| `android_assert(question)` | Screenshot → local VLM → `PASS`/`FAIL` + evidence. Costs the driving model almost no context. |
| `android_record_start` / `android_record_stop(analyze, fps)` | Record the screen, then have the local model narrate what happened. |
| `android_analyze(question, images, video)` | Ask the local model about media already on disk. |

---

## A typical loop

```
android_session_start(package="com.example.app", project_dir="/path/to/project", reinstall=True)
  -> builds, installs, launches, returns a screenshot + addressable elements

android_tap(text="Sign in")
android_type(text="demo@example.com", ref="n7", submit=True)
android_wait(until_text="Welcome")
android_assert("is the welcome screen visible with a Continue button?")
android_crashes(only_new=True)
```

For a whole flow rather than one screen:

```
android_record_start(max_seconds=45)
  ... taps and swipes ...
android_record_stop(handle="rec1", analyze="did any screen flash an error or render blank?")
```

## Building alongside Android Studio

On Windows a Gradle daemon keeps build outputs such as `R.jar` open for a while after a build. A
build in a **different** daemon then cannot rewrite them, which is the classic
*"being used by another process"* failure. Agent terminals make this routine: some run Gradle under
their own sandboxed `GRADLE_USER_HOME`, and a daemon there can never be reused by Android Studio.

`android_build` avoids it by:

1. **Building in Studio's pool** — its Gradle user home (your persistent `GRADLE_USER_HOME`, else
   `~/.gradle`) and its Gradle JDK (the project's daemon JVM criteria, else `.idea/gradle.xml`'s
   `gradleJvm`, else Studio's bundled JBR). A sandboxed `GRADLE_USER_HOME` in the environment is
   ignored. Sequential builds from Studio and the agent then share one daemon, and a process cannot
   lock files against itself.
2. **Never starting a second daemon** while one in that pool is BUSY; it waits (`wait_if_busy_s`).
3. **Checking locks before starting**, and refusing with the holder named when something outside
   Studio's pool holds the outputs.
4. **`release_locks=True`**, which stops *idle* Gradle/Kotlin daemons holding the outputs. Busy
   builds and the IDE are never touched; an idle daemon costs only warm-up time.

**So: agents should never run `gradlew` in a terminal — use `android_build`.**

## What a failure report contains

- **Category and next step**: `COMPILE_ERROR`, `RESOURCE_ERROR`, `FILE_LOCKED`, `DAEMON_BUSY`,
  `INSTALL_FAILED`, `DEPENDENCY_ERROR`, `BUILD_SCRIPT_ERROR`, `JDK_ERROR`, `SDK_ERROR`,
  `OUT_OF_MEMORY`, `CONFIGURATION_CACHE`, `TIMEOUT`, `UNKNOWN`.
- **Every error** from Kotlin (K1 and K2), javac, KSP/kapt, AAPT, manifest merger and build scripts,
  with file:line:col and ±3 lines of source, each marked `[changed in working tree]` or
  `[not changed]`. An error at an unchanged call site usually means a callee changed elsewhere.
- **Gradle's own explanation**, the **working-tree changes**, the **lock holders** (PID, daemon kind,
  Gradle user home, JDK, IDLE/BUSY), the **environment**, and the **full log path**.

Crash reports lead with the root cause (`Caused by`), collapse framework frames, and map app frames
to your source with surrounding lines.

### Briefs and the fixer subagent

Every failure also writes a markdown brief (`build-failure-*.md` / `crash-*.md` in the run
directory) holding the task, the exact tool call that reproduces and verifies the fix, the project's
own rule files, and the full report — ready to hand to a subagent.

`agents/android-fixer.md` is such a subagent. Copy it to `~/.cursor/agents/` (Cursor) or
`~/.claude/agents/` (Claude Code), then: *"use android-fixer on `<brief path>`"*.

## Was a local model used?

Only `android_screenshot(analyze=...)`, `android_assert`, `android_record_stop` and
`android_analyze` ever call LM Studio. Everything else is plain adb/Gradle.

Each such result ends with a tag like
`[local model gemma-4-26b-a4b-it, call v1 in run-20260922-174231, 3.0s, 1 image(s)]`. No tag means no
local model was involved. Beyond that:

- `~/.android-agent/vlm-calls.jsonl` — one JSON line per call: calling tool, model requested versus
  served, prompt, response, usage, duration, and any error.
- `~/.android-agent/run-*/vlm/<id>/image-NN.png` — the exact images the model received.
- `android_vlm_log()`, or `android-agent --cli vlm-log [n | v3]`.

## Configuration

| Variable | Default | Meaning |
| --- | --- | --- |
| `ANDROID_AGENT_DEVICE` | first physical device | Pin the default target serial. |
| `ANDROID_AGENT_GRADLE_USER_HOME` | persistent `GRADLE_USER_HOME`, else `~/.gradle` | The Gradle user home Android Studio uses. |
| `ANDROID_STUDIO_JDK` | Studio's bundled JBR | JDK for projects without a pinned daemon JVM. |
| `ANDROID_AGENT_GRADLE_TIMEOUT` | `1800` | Build time limit, seconds. |
| `ANDROID_AGENT_SCALE` | `1280` | Long edge of screenshots returned to the model. |
| `ANDROID_AGENT_VLM_SCALE` | `768` | Long edge of images sent to the local model. |
| `ANDROID_AGENT_LMSTUDIO_URL` | `http://127.0.0.1:1234` | LM Studio endpoint. |
| `ANDROID_AGENT_LMSTUDIO_MODEL` | `gemma-4-26b-a4b-it` | Model id for analysis; set this to whatever vision model you run. |
| `ANDROID_AGENT_LMSTUDIO_TIMEOUT` | `300` | Seconds; a cold model load is slow. |
| `ANDROID_AGENT_OUT` | `~/.android-agent` | Where screenshots, videos, logs, briefs and the VLM log go. |
| `ANDROID_HOME` | auto-detected | Android SDK root. |

Device selection: explicit `device=` → `ANDROID_AGENT_DEVICE` → first **physical** device → first
available. A plugged-in phone wins over a running emulator.

## Development

```bash
git clone https://github.com/Fdondi/android-agent-mcp
cd android-agent-mcp
uv run --script tests/test_parsers.py          # offline: no device, Gradle or LM Studio
uv run --script src/android_agent/server.py --selftest
uv run --script src/android_agent/server.py --cli devices
```

The server is one module, `src/android_agent/server.py`. It carries a PEP 723 header so it also runs
standalone with `uv run --script`; its dependency list must stay in sync with `pyproject.toml`.

Notes for anyone editing it:

- **stdout is the MCP wire** — log to stderr only.
- **MCP 2.x hides exception text** unless it is a `ToolError`; the `@tool` decorator converts
  everything, which is what makes failures readable.
- **Screenshots must be binary-clean**: `adb exec-out screencap -p` with a list argv, never a shell
  pipe.
- **Stopping a recording takes exactly one SIGINT**; a second signal truncates the mp4 before the
  `moov` atom is written.
- **Compose exposes no resource-ids** — match on text/content-desc, or fall back to coordinates.

## License

MIT