Reads and drives a running Android app as text via adb, without any image input. Provides tools to inspect the on-screen semantics/accessibility tree (element classes, flags, pixel bounds, centres and labels), audit layouts for undersized tap targets, overlapping controls and clipped off-screen nodes, render the screen as a coloured glyph/ASCII image with crop and palette controls, and tap elements by their label. This lets an agent inspect and interact with Android UIs, including Flutter screens whose semantics tree is normally hidden without a bound accessibility service.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@apkscope dump the tappable elements on the current Android screen"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
See and drive a running Android app as text, for an agent whose model rejects image input.
Most LLM-based coding agents cannot look at a screenshot: the model refuses image content, and the usual "describe this image" helpers call a hosted vision API that bills per request or rate-limits you to nothing. This MCP server exists so such an agent can still work on an Android UI — reading what is on screen, measuring it, and tapping it.
Everything is converted into characters the model can read.
Developed while building a Flutter app (SahayakAI) with an agent that had no image input at all. Two capabilities made UI work possible:
Reading the semantics tree. Flutter publishes its accessibility tree only
while an accessibility service is bound. With none bound, uiautomator dump
against a Flutter screen returns a bare FrameLayout with no children.
apkscope enables a stock system service for the duration of the read and
restores the previous setting afterwards.
Rendering pixels as glyphs. Enough to judge layout, spacing, colour and gross breakage — not a substitute for vision, but far better than nothing.
Related MCP server: android-layout-inspector-mcp
pip install -e .Requires adb on PATH, or $ANDROID_HOME / $ANDROID_SDK_ROOT set, or
$APKSCOPE_ADB pointing at the binary.
{
"mcp": {
"apkscope": {
"type": "local",
"command": ["apkscope-mcp"],
"environment": { "ANDROID_HOME": "C:\\Users\\you\\AppData\\Local\\Android\\Sdk" }
}
}
}Or run it directly to check it starts:
apkscope-mcptool | mutates device | what it gives you |
| no | every labelled element: class, flags, pixel bounds, centre, label |
| no | tap targets under 48dp, overlapping controls, off-screen nodes |
| no | the screen rendered as text, in one of three modes |
| yes | taps the element whose label matches |
screen_tree — start hereThe primary way to read a Flutter UI. Labels, roles and exact bounds are precise and cheap; a pixel render is approximate and token-heavy.
View [-FE--] 530x98 @( 540, 842) Good morning
View [CFE--] 477x283 @( 283, 1750) Lesson Plan
Button [CFE--] 126x126 @( 765, 202) NetworkFlags: C clickable, F focusable, E enabled, k checked, s selected.
Only C means tappable. Flutter also marks plain text focusable, so F
alone is not a control — that distinction is what keeps headings out of the
tap list.
Pass tappable=True for just the tap-target table.
audit_layout — turn observation into findingsReports TAP-SMALL (below Material's 48dp guidance, converted using the
device's real density), OVERLAP (two controls sharing more than a third of the
smaller one) and, with check_clip=True, CLIPPED off-screen nodes.
Full-bleed wrappers are excluded. Flutter routinely wraps a whole screen in one
GestureDetector (a "tap anywhere to speak" surface); without that exclusion
every other control is reported as overlapping it.
screenshot — pixels as textThree modes:
mode | output | cost | good for |
| one glyph per pixel, plus a legend mapping glyph → true RGB | ~2k tokens | layout, colour, shape |
| luminance ASCII ( | cheap | structure only |
| truecolour half-blocks | very high (~40 chars/pixel) | detail crops only |
Keep colors at 8 or below. On a light-themed app a large palette gets
spent on imperceptible differences between near-identical backgrounds
(#f3f1ec vs #f7f5f0) and the one meaningful hue is lost. Fewer bins force
the quantiser to group the near-whites.
The legend always reports RGB from the unaltered image, so colour is never misrepresented.
crop accepts all, top, mid, bot, or 'a,b' fractions of height. A
phone screen is tall and mostly empty near the bottom; cropping a band is the
difference between a legible render and a hundred rows of background.
tap — drive the UIResolves a label to coordinates through the semantics tree, which survives layout changes and screen-size differences in a way hardcoded pixels do not.
Refuses an ambiguous label and lists the candidates, because a silent wrong tap
navigates somewhere unexpected and costs more to debug than asking. Pass
first=True when you know what you want.
The same operations without an MCP host:
apkscope tree --taps
apkscope audit --clip
apkscope shot --cols 72 --colors 8 --crop 0.3,0.6
apkscope shot --mode ramp --crop top
apkscope tap "Continue with Google"audit exits non-zero when it finds something, so it works in a CI check.
A glyph render is not vision. It conveys structure, colour, alignment, spacing and obvious breakage. It does not reliably convey small body text, subtle typography, or whether a screen matches a design intent. For fine visual judgement a human still has to look.
screen_tree and audit_layout are the higher-value tools for UI work —
exact, cheap, directly actionable. Reach for screenshot to catch layout
problems, not to proofread.
The tree only shows what Flutter marks semantic. Text painted directly to a
canvas is invisible to it, and to uiautomator generally.
pip install -e ".[dev]"
pytest97 tests, no device required. They pin down the bugs this tool actually hit while being built: an inverted luminance ramp, a palette legend that crashed on low-colour crops, a whole-screen overlap false positive from Flutter's full-bleed wrapper, and unlabelled icon buttons being invisible to the tap-target filter.
MIT
Check tap-target sizes, overlaps and (optionally) off-screen elements.
Reports TAP-SMALL (a control below min_tap_dp, Material's guidance is 48),
OVERLAP (two controls sharing more than a third of the smaller one), and
with check_clip CLIPPED (a node extending past the viewport).
Full-bleed wrappers are excluded: Flutter often wraps a whole screen in one GestureDetector, which would otherwise appear to overlap everything.
| Name | Required | Description | Default |
|---|---|---|---|
| device | No | ||
| check_clip | No | ||
| min_tap_dp | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations cover the safety profile (readOnly, idempotent, non-destructive), and the description adds substantial behavior beyond that: the emitted codes TAP-SMALL, OVERLAP and CLIPPED, the overlap threshold (more than a third of the smaller control), the default Material tap guidance of 48, and the important exclusion of full-bleed wrappers. This is unusually rich disclosure of what the tool detects and why.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with the core action, then codes, then the wrapper exclusion rationale. Every line carries information, though the multi-line formatting is slightly more expansive than strictly necessary for three checks.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
No output schema exists, so the description usefully enumerates the report codes an agent will receive, which is the key missing piece. The only gap is the unexplained 'device' parameter, but otherwise an agent has enough to invoke and interpret this diagnostic tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so the description must compensate. It adds real meaning for min_tap_dp (defines the below-threshold condition and cites Material's 48) and check_clip (node extending past the viewport), but the 'device' parameter is never explained anywhere, leaving one of three params undocumented.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a precise verb+resource: it audits layout, enumerating exactly what it checks (tap-target sizes, overlaps, off-screen elements) and what diagnostic codes it reports. An agent can distinguish this from screenshot/screen_tree/tap siblings purely from the description.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is implied through the checks performed and the conditional 'with check_clip CLIPPED' phrasing, which hints when clipping detection activates. However, there is no explicit statement of when to prefer this over screen_tree or screenshot, nor any when-not guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Render the current screen as text.
mode: 'palette' (default) one glyph per pixel with a legend mapping glyphs to true RGB -- layout, colour and shape. 'ramp' luminance ASCII, cheapest and structure only. 'ansi' truecolour half-blocks, roughly 40 characters per pixel, so keep it to small crops and low cols.
crop: 'all', 'top', 'mid', 'bot', or 'a,b' as fractions of screen height. cols: output width in characters. squash: vertical scale factor, default 1.0 (true aspect); raise it to compress a tall screen.
Does not convey small body text, subtle typography, or whether a screen matches a design intent -- use screen_tree for exact content.
| Name | Required | Description | Default |
|---|---|---|---|
| cols | No | ||
| crop | No | ||
| mode | No | palette | |
| colors | No | ||
| device | No | ||
| squash | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnly/idempotent/non-destructive, so the safety profile is covered. The description adds genuinely useful behavior beyond that: the relative cost and fidelity of each mode ('cheapest, structure only', 'roughly 40 characters per pixel'), and the explicit limitation that it cannot convey subtle typography or design intent. No return-format or pagination detail is given, keeping it from a 5.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Purpose is front-loaded in one line, followed by a compact per-parameter block and a closing limitation. The mode tests ('layout, colour and shape' / 'structure only') are terse and each sentence carries information, though the run-on mode paragraph is slightly dense.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
There is no output schema, so the description is right to characterize output fidelity and limits. For a 6-parameter renderer it covers the parameters that matter most and explains the output's nature, with only 'colors' and 'device' left unexplained.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so the description must carry the load, and it does for four of six parameters: mode (all three values explained), crop (all named values plus the 'a,b' fraction format), cols (output width), and squash (scale factor with default 1.0 semantics). 'colors' and 'device' are never mentioned, leaving a gap that prevents a 5.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Opens with a specific verb+resource: 'Render the current screen as text.' It immediately distinguishes itself from the sibling screen_tree by stating its output is a rendered approximation rather than exact content. An agent can identify the tool's role without opening the schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Names screen_tree as the alternative and states the condition that selects it ('exact content'), and warns that this tool does not convey small body text or design-intent matching. It also gives the mode-specific guidance to keep 'ansi' to small crops. No explicit when-not for the other siblings (tap, audit_layout), so not a full 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
List every labelled element on screen with its bounds.
The primary way to read a Flutter UI: Flutter publishes its semantics tree only while an accessibility service is bound, which this tool arranges temporarily and then reverts.
Each line gives class, flags, pixel size, centre coordinate and label. Flags: C=clickable, F=focusable, E=enabled, k=checked, s=selected. Only elements marked C are real controls -- Flutter also marks plain text focusable, so do not treat F alone as tappable.
Pass tappable=True to get just the tap-target table.
| Name | Required | Description | Default |
|---|---|---|---|
| device | No | ||
| tappable | No | ||
| max_depth | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations cover the safety profile (readOnly, idempotent, non-destructive), and the description adds material side effects they cannot express: it temporarily binds an accessibility service to make Flutter publish its semantics tree and then reverts it. It also warns that F alone does not mean tappable, which prevents a real mis-invocation.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loads the core purpose, then the mechanism, then the output format and flags. Every sentence earns its place, and the flag legend is necessary because there is no output schema to carry it.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema, the description correctly documents the return format (class, flags, pixel size, centre coordinate, label) and the flag legend. The remaining gaps are the device and max_depth parameters, which an agent still has to guess at.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description carries the full burden, yet it only explains one of three parameters (tappable, with its effect on output). device and max_depth are never mentioned, leaving their format and purpose (e.g. depth truncation) entirely undocumented.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Opening sentence gives a specific verb and resource ('List every labelled element on screen with its bounds') and the second sentence positions it precisely as the Flutter semantics-tree reader. This distinguishes it from sibling screenshot (pixels) and audit_layout (layout diagnostics) without needing to name them.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
States it is 'the primary way to read a Flutter UI,' which tells the agent when to reach for this over screenshot, and explains the tappable=True shortcut. It stops short of explicit when-not-to-use guidance or naming sibling alternatives for non-Flutter surfaces.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tap the on-screen element whose label contains label.
Refuses an ambiguous label and lists the candidates, because a silent wrong tap navigates somewhere unexpected. Pass first=True to take the first match.
Returns the resolved label and coordinates actually tapped.
| Name | Required | Description | Default |
|---|---|---|---|
| first | No | ||
| label | Yes | ||
| device | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations declare the safety profile (readOnlyHint=false, idempotentHint=false, openWorldHint=true), and the description adds genuinely non-redundant behavior: it refuses ambiguous labels, lists the candidates instead, and explains why ('a silent wrong tap navigates somewhere unexpected'). It also states the return content. Device targeting behavior is not disclosed.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with the core action, then the ambiguity/override rule, then the return value. Every sentence is useful, though the parenthetical rationale about unexpected navigation is slightly verbose for its value.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
No output schema exists, so the description helpfully states what is returned ('the resolved label and coordinates actually tapped'). Combined with the ambiguity behavior and the `first` override, an agent has nearly everything needed, with only device selection left unexplained.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must carry parameter meaning. It clarifies `label` as a substring match ('contains') and explains `first`, but the third parameter `device` is never mentioned, leaving a real gap at zero baseline coverage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb+resource: 'Tap the on-screen element whose label contains `label`.' This is unambiguous and clearly distinct from the read-only siblings (screenshot, screen_tree, audit_layout), so an agent can route to it without inspecting the schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives a concrete use-condition for the optional parameter ('Pass first=True to take the first match') and explains the default behavior of refusing ambiguous labels. It doesn't name alternatives, but the siblings are complementary rather than competing tools, so the guidance is adequate.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v0.1.0audit_layoutscreen_treescreenshottapScored across 4 tools
Each tool has a clearly distinct role: screenshot renders pixels visually, screen_tree extracts semantic structure, audit_layout checks geometry heuristics, and tap performs an action. The descriptions explicitly cross-reference each other (e.g. screenshot points to screen_tree for exact content), eliminating overlap.
All names are lowercase snake_case, but the grammatical pattern is mixed: 'screenshot' and 'screen_tree' are output nouns, 'audit_layout' is verb_noun, and 'tap' is a bare verb. Readable and non-chaotic, but not a predictable single convention.
Four tools is a lean, well-scoped set covering the read/audit/interact loop for a Flutter screen. It is slightly thin (no gesture, input, or navigation helpers), but nothing feels redundant or out of place.
Read and audit coverage is solid, and tap handles basic interaction, but the surface lacks text input, scrolling/swiping, back navigation, and app launch/wait operations. These are notable gaps for a UI inspection/automation domain that will force agents to work around them.
Control real Android and iOS devices with LLM agents — tap, swipe, type, automate flows.
Cloud Android phones for AI agents: create a phone, read the screen, tap, type, install APKs, park.
Disposable cloud Android emulators for coding agents: run an APK or PR build, tap, type, screenshot.
Drive real devices from your AI Coding tool. Embed a client SDK (Unity, Godot, Flutter, iOS/macOS, Android, React Native, Web) in your app, then capture screenshots, traverse the UI tree, inject taps and key events, and run automated test tasks on the physical device over a secure relay.
