# apkscope

See and drive a running Android app **as text**, for an agent whose model
rejects image input.

Most LLM-based coding agents cannot look at a screenshot: the model refuses
image content, and the usual "describe this image" helpers call a hosted vision
API that bills per request or rate-limits you to nothing. This MCP server exists
so such an agent can still work on an Android UI — reading what is on screen,
measuring it, and tapping it.

Everything is converted into characters the model can read.

## Why it exists

Developed while building a Flutter app (SahayakAI) with an agent that had no
image input at all. Two capabilities made UI work possible:

- **Reading the semantics tree.** Flutter publishes its accessibility tree only
  while an accessibility service is bound. With none bound, `uiautomator dump`
  against a Flutter screen returns a bare `FrameLayout` with no children.
  `apkscope` enables a stock system service for the duration of the read and
  restores the previous setting afterwards.
- **Rendering pixels as glyphs.** Enough to judge layout, spacing, colour and
  gross breakage — not a substitute for vision, but far better than nothing.

## Install

```bash
pip install -e .
```

Requires `adb` on `PATH`, or `$ANDROID_HOME` / `$ANDROID_SDK_ROOT` set, or
`$APKSCOPE_ADB` pointing at the binary.

## MCP setup

```jsonc
{
  "mcp": {
    "apkscope": {
      "type": "local",
      "command": ["apkscope-mcp"],
      "environment": { "ANDROID_HOME": "C:\\Users\\you\\AppData\\Local\\Android\\Sdk" }
    }
  }
}
```

Or run it directly to check it starts:

```bash
apkscope-mcp
```

## Tools

| tool | mutates device | what it gives you |
|---|---|---|
| `screen_tree` | no | every labelled element: class, flags, pixel bounds, centre, label |
| `audit_layout` | no | tap targets under 48dp, overlapping controls, off-screen nodes |
| `screenshot` | no | the screen rendered as text, in one of three modes |
| `tap` | **yes** | taps the element whose label matches |

### `screen_tree` — start here

The primary way to read a Flutter UI. Labels, roles and exact bounds are
precise and cheap; a pixel render is approximate and token-heavy.

```
View       [-FE--]   530x98    @(  540,  842)  Good morning
View       [CFE--]   477x283   @(  283, 1750)  Lesson Plan
Button     [CFE--]   126x126   @(  765,  202)  Network
```

Flags: `C` clickable, `F` focusable, `E` enabled, `k` checked, `s` selected.
**Only `C` means tappable.** Flutter also marks plain text focusable, so `F`
alone is not a control — that distinction is what keeps headings out of the
tap list.

Pass `tappable=True` for just the tap-target table.

### `audit_layout` — turn observation into findings

Reports `TAP-SMALL` (below Material's 48dp guidance, converted using the
device's real density), `OVERLAP` (two controls sharing more than a third of the
smaller one) and, with `check_clip=True`, `CLIPPED` off-screen nodes.

Full-bleed wrappers are excluded. Flutter routinely wraps a whole screen in one
`GestureDetector` (a "tap anywhere to speak" surface); without that exclusion
every other control is reported as overlapping it.

### `screenshot` — pixels as text

Three modes:

| mode | output | cost | good for |
|---|---|---|---|
| `palette` (default) | one glyph per pixel, plus a legend mapping glyph → true RGB | ~2k tokens | layout, colour, shape |
| `ramp` | luminance ASCII (`@%#*+=-:. `) | cheap | structure only |
| `ansi` | truecolour half-blocks `▀`, two pixels per character | very high (~40 chars/pixel) | detail crops only |

**Keep `colors` at 8 or below.** On a light-themed app a large palette gets
spent on imperceptible differences between near-identical backgrounds
(`#f3f1ec` vs `#f7f5f0`) and the one meaningful hue is lost. Fewer bins force
the quantiser to group the near-whites.

The legend always reports RGB from the unaltered image, so colour is never
misrepresented.

`crop` accepts `all`, `top`, `mid`, `bot`, or `'a,b'` fractions of height. A
phone screen is tall and mostly empty near the bottom; cropping a band is the
difference between a legible render and a hundred rows of background.

### `tap` — drive the UI

Resolves a label to coordinates through the semantics tree, which survives
layout changes and screen-size differences in a way hardcoded pixels do not.

Refuses an ambiguous label and lists the candidates, because a silent wrong tap
navigates somewhere unexpected and costs more to debug than asking. Pass
`first=True` when you know what you want.

## CLI

The same operations without an MCP host:

```bash
apkscope tree --taps
apkscope audit --clip
apkscope shot --cols 72 --colors 8 --crop 0.3,0.6
apkscope shot --mode ramp --crop top
apkscope tap "Continue with Google"
```

`audit` exits non-zero when it finds something, so it works in a CI check.

## Honest limits

A glyph render is **not** vision. It conveys structure, colour, alignment,
spacing and obvious breakage. It does not reliably convey small body text,
subtle typography, or whether a screen matches a design intent. For fine visual
judgement a human still has to look.

`screen_tree` and `audit_layout` are the higher-value tools for UI work —
exact, cheap, directly actionable. Reach for `screenshot` to catch layout
problems, not to proofread.

The tree only shows what Flutter marks semantic. Text painted directly to a
canvas is invisible to it, and to `uiautomator` generally.

## Tests

```bash
pip install -e ".[dev]"
pytest
```

97 tests, no device required. They pin down the bugs this tool actually hit
while being built: an inverted luminance ramp, a palette legend that crashed on
low-colour crops, a whole-screen overlap false positive from Flutter's
full-bleed wrapper, and unlabelled icon buttons being invisible to the
tap-target filter.

## License

MIT