Skip to main content
Glama

apkscope

See and drive a running Android app as text, for an agent whose model rejects image input.

Most LLM-based coding agents cannot look at a screenshot: the model refuses image content, and the usual "describe this image" helpers call a hosted vision API that bills per request or rate-limits you to nothing. This MCP server exists so such an agent can still work on an Android UI — reading what is on screen, measuring it, and tapping it.

Everything is converted into characters the model can read.

Why it exists

Developed while building a Flutter app (SahayakAI) with an agent that had no image input at all. Two capabilities made UI work possible:

  • Reading the semantics tree. Flutter publishes its accessibility tree only while an accessibility service is bound. With none bound, uiautomator dump against a Flutter screen returns a bare FrameLayout with no children. apkscope enables a stock system service for the duration of the read and restores the previous setting afterwards.

  • Rendering pixels as glyphs. Enough to judge layout, spacing, colour and gross breakage — not a substitute for vision, but far better than nothing.

Related MCP server: scrcpy-mcp

Install

pip install -e .

Requires adb on PATH, or $ANDROID_HOME / $ANDROID_SDK_ROOT set, or $APKSCOPE_ADB pointing at the binary.

MCP setup

{
  "mcp": {
    "apkscope": {
      "type": "local",
      "command": ["apkscope-mcp"],
      "environment": { "ANDROID_HOME": "C:\\Users\\you\\AppData\\Local\\Android\\Sdk" }
    }
  }
}

Or run it directly to check it starts:

apkscope-mcp

Tools

tool

mutates device

what it gives you

screen_tree

no

every labelled element: class, flags, pixel bounds, centre, label

audit_layout

no

tap targets under 48dp, overlapping controls, off-screen nodes

screenshot

no

the screen rendered as text, in one of three modes

tap

yes

taps the element whose label matches

screen_tree — start here

The primary way to read a Flutter UI. Labels, roles and exact bounds are precise and cheap; a pixel render is approximate and token-heavy.

View       [-FE--]   530x98    @(  540,  842)  Good morning
View       [CFE--]   477x283   @(  283, 1750)  Lesson Plan
Button     [CFE--]   126x126   @(  765,  202)  Network

Flags: C clickable, F focusable, E enabled, k checked, s selected. Only C means tappable. Flutter also marks plain text focusable, so F alone is not a control — that distinction is what keeps headings out of the tap list.

Pass tappable=True for just the tap-target table.

audit_layout — turn observation into findings

Reports TAP-SMALL (below Material's 48dp guidance, converted using the device's real density), OVERLAP (two controls sharing more than a third of the smaller one) and, with check_clip=True, CLIPPED off-screen nodes.

Full-bleed wrappers are excluded. Flutter routinely wraps a whole screen in one GestureDetector (a "tap anywhere to speak" surface); without that exclusion every other control is reported as overlapping it.

screenshot — pixels as text

Three modes:

mode

output

cost

good for

palette (default)

one glyph per pixel, plus a legend mapping glyph → true RGB

~2k tokens

layout, colour, shape

ramp

luminance ASCII (@%#*+=-:. )

cheap

structure only

ansi

truecolour half-blocks ▀, two pixels per character

very high (~40 chars/pixel)

detail crops only

Keep colors at 8 or below. On a light-themed app a large palette gets spent on imperceptible differences between near-identical backgrounds (#f3f1ec vs #f7f5f0) and the one meaningful hue is lost. Fewer bins force the quantiser to group the near-whites.

The legend always reports RGB from the unaltered image, so colour is never misrepresented.

crop accepts all, top, mid, bot, or 'a,b' fractions of height. A phone screen is tall and mostly empty near the bottom; cropping a band is the difference between a legible render and a hundred rows of background.

tap — drive the UI

Resolves a label to coordinates through the semantics tree, which survives layout changes and screen-size differences in a way hardcoded pixels do not.

Refuses an ambiguous label and lists the candidates, because a silent wrong tap navigates somewhere unexpected and costs more to debug than asking. Pass first=True when you know what you want.

CLI

The same operations without an MCP host:

apkscope tree --taps
apkscope audit --clip
apkscope shot --cols 72 --colors 8 --crop 0.3,0.6
apkscope shot --mode ramp --crop top
apkscope tap "Continue with Google"

audit exits non-zero when it finds something, so it works in a CI check.

Honest limits

A glyph render is not vision. It conveys structure, colour, alignment, spacing and obvious breakage. It does not reliably convey small body text, subtle typography, or whether a screen matches a design intent. For fine visual judgement a human still has to look.

screen_tree and audit_layout are the higher-value tools for UI work — exact, cheap, directly actionable. Reach for screenshot to catch layout problems, not to proofread.

The tree only shows what Flutter marks semantic. Text painted directly to a canvas is invisible to it, and to uiautomator generally.

Tests

pip install -e ".[dev]"
pytest

97 tests, no device required. They pin down the bugs this tool actually hit while being built: an inverted luminance ramp, a palette legend that crashed on low-colour crops, a whole-screen overlap false positive from Flutter's full-bleed wrapper, and unlabelled icon buttons being invisible to the tap-target filter.

License

MIT

Available Tools

4 tools
audit_layoutA
Read-onlyIdempotent

Check tap-target sizes, overlaps and (optionally) off-screen elements.

Reports TAP-SMALL (a control below min_tap_dp, Material's guidance is 48), OVERLAP (two controls sharing more than a third of the smaller one), and with check_clip CLIPPED (a node extending past the viewport).

Full-bleed wrappers are excluded: Flutter often wraps a whole screen in one GestureDetector, which would otherwise appear to overlap everything.

Inputs: device, check_clip, min_tap_dp.

screenshotA
Read-onlyIdempotent

Render the current screen as text.

mode: 'palette' (default) one glyph per pixel with a legend mapping glyphs to true RGB -- layout, colour and shape. 'ramp' luminance ASCII, cheapest and structure only. 'ansi' truecolour half-blocks, roughly 40 characters per pixel, so keep it to small crops and low cols.

crop: 'all', 'top', 'mid', 'bot', or 'a,b' as fractions of screen height. cols: output width in characters. squash: vertical scale factor, default 1.0 (true aspect); raise it to compress a tall screen.

Does not convey small body text, subtle typography, or whether a screen matches a design intent -- use screen_tree for exact content.

Inputs: cols, crop, mode, colors, +2 more.

screen_treeA
Read-onlyIdempotent

List every labelled element on screen with its bounds.

The primary way to read a Flutter UI: Flutter publishes its semantics tree only while an accessibility service is bound, which this tool arranges temporarily and then reverts.

Each line gives class, flags, pixel size, centre coordinate and label. Flags: C=clickable, F=focusable, E=enabled, k=checked, s=selected. Only elements marked C are real controls -- Flutter also marks plain text focusable, so do not treat F alone as tappable.

Pass tappable=True to get just the tap-target table.

Inputs: device, tappable, max_depth.

tapA

Tap the on-screen element whose label contains label.

Refuses an ambiguous label and lists the candidates, because a silent wrong tap navigates somewhere unexpected. Pass first=True to take the first match.

Returns the resolved label and coordinates actually tapped.

Inputs: label (required), first, device.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A4.1/5.0

    Scored across 4 tools

    Disambiguation5/5

    Each tool has a clearly distinct role: screenshot renders pixels visually, screen_tree extracts semantic structure, audit_layout checks geometry heuristics, and tap performs an action. The descriptions explicitly cross-reference each other (e.g. screenshot points to screen_tree for exact content), eliminating overlap.

    Naming Consistency3/5

    All names are lowercase snake_case, but the grammatical pattern is mixed: 'screenshot' and 'screen_tree' are output nouns, 'audit_layout' is verb_noun, and 'tap' is a bare verb. Readable and non-chaotic, but not a predictable single convention.

    Tool Count4/5

    Four tools is a lean, well-scoped set covering the read/audit/interact loop for a Flutter screen. It is slightly thin (no gesture, input, or navigation helpers), but nothing feels redundant or out of place.

    Completeness3/5

    Read and audit coverage is solid, and tap handles basic interaction, but the surface lacks text input, scrolling/swiping, back navigation, and app launch/wait operations. These are notable gaps for a UI inspection/automation domain that will force agents to work around them.

    Maintenance

    ActivitySlowing
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      B
      quality
      C
      maintenance
      Enables AI agents to measure the real geometry of a live Android UI over adb and uiautomator, reporting overlapping views, clipping, touch targets under 48dp, occluded or unlabeled controls, and portrait/landscape diffs. Analysis also runs against saved uiautomator dumps, so layouts can be checked in CI without a device attached.
      6
      MIT
    • A
      license
      Not graded
      quality
      A
      maintenance
      Enables AI agents to visually perceive and directly control Android devices via ADB and screen-control tooling, including screenshots, taps, swipes, typing, app management, UI inspection, file transfer, shell commands, and audio/video streaming.
      468 npm
      MIT
    • A
      license
      B
      quality
      C
      maintenance
      Enables any MCP assistant to see and operate an Android phone or emulator over plain adb, reading the accessibility tree to tap elements by name and also supporting screenshots, typing, swiping, app launching and installation, and logcat — with nothing installed on the device and no root required.
      13
      MIT
    • A
      license
      A
      quality
      B
      maintenance
      Enables AI agents to see and drive a real Android phone over plain ADB: screenshots and numbered UI trees, gestures, typing with accents and emoji, SMS/RCS reading and prepared replies, notifications, files, and settings, with no root and no app installed on the phone. Safeguards mask security codes and require explicit confirmation before sending, paying, deleting or confirming anything.
      18
      2
      MIT