ThumbAgent

Local-first platform that gives AI agents a real thumb on mobile devices.

中文 | English

A local-first, cross-platform mobile device Skills platform for AI Agents.

Current Progress

The project has completed ITER-0052 Desktop Device Screen & Live Observation: the desktop workbench displays a device screen panel while executing Agent tasks, updating with real screenshots after each action in real time; task reports can expand screenshot evidence per round. The Runtime has added a read-only content endpoint GET /v1/artifacts/{artifact_id}/content (Bearer token authentication, screenshot PNG only, single file up to 8 MiB, no-store), and the task.step_completed event carries the round's screenshot_artifact_id.

The desktop workbench (Tauri 2) automatically launches and authenticates the local Runtime, with the home page showing unified readiness diagnostics and a list of discovered devices, supporting natural language task submission, execution timeline, and full reports. See apps/desktop/README.md for desktop development. Using Python 3.11+:

make check
make run

The Runtime listens on 127.0.0.1:8765 by default, providing /v1/health, /v1/devices, and POST /v1/devices/{device_id}/observe.

Related MCP server: Android MCP Server

MCP Skills Developer Preview

When performing local real-device acceptance on macOS + Codex desktop, you can use the one-click script:

./scripts/run-mcp-preview.zsh

On first run, the script prompts for a model Key and saves the model Key and a stable local Runtime token separately to the macOS login Keychain; subsequent launches will not ask again. The script safely stops old mobile_agent.api.server processes occupying the target port, reuses unchanged MCP registrations, and starts a new Runtime. Codex/ChatGPT does not need to be closed or reopened if already running. Only on first registration, explicit --refresh-mcp, or when the MCP configuration or Tool Catalog changes does a running Codex need to be restarted once and a new task created to refresh the cached MCP environment; ordinary Runtime restarts do not require this.

The model Key only enters the Keychain and the Runtime process environment, never the repository or script output. The script stays in the foreground; press Ctrl+C to stop the Runtime. To only check Python, ADB, Codex, and model configuration paths without reading keys or modifying MCP configuration, use:

./scripts/run-mcp-preview.zsh --check

To force-refresh the MCP registration or delete the preview Secret:

./scripts/run-mcp-preview.zsh --refresh-mcp
./scripts/run-mcp-preview.zsh --forget-secrets

Refreshing the registration does not rotate the token in the Keychain. If the target port is occupied by another program, the script refuses to kill it; it only automatically stops processes explicitly identified on the command line as mobile_agent.api.server.

To let Web, CLI, and MCP share the same Runtime, start the service with an explicit local token:

MOBILE_AGENT_API_TOKEN=<local-random-token> \
MOBILE_AGENT_ADB_PATH=/usr/local/platform-tools/adb \
make run

Then configure a stdio Server in the MCP Host using the same token. See mcp-server.example.json for an example. The actual command the MCP Host launches is:

PYTHONPATH=runtime \
MOBILE_AGENT_API_TOKEN=<same-local-random-token> \
python3.11 -m mobile_agent.mcp

MCP provides goal-level Tools covering readiness diagnostics, device and installed-app inspection, app lifecycle, Agent async tasks, task query/cancel, redacted logs, aggregated performance snapshots, diagnostic evidence bundles, performance comparison, and local Artifact retention cleanup. It does not expose ADB, arbitrary Shell, arbitrary file paths, or atomic Tools such as input.tap. Actions requiring confirmation are only allowed to pass confirmed=true after the MCP Host has shown the user the parameters and impact and obtained confirmation.

After startup, you can view the unified readiness diagnostics:

PYTHONPATH=runtime python3.11 -m mobile_agent.cli.runtime_diagnose

GET /v1/readiness and the Web UI show Android Gateway, device connection/authorization, Session, Lease occupancy, and remediation suggestions. When ADB is not installed or the path is wrong, the Runtime still starts the diagnostics UI instead of exiting directly with ADB_NOT_FOUND.

View a single device's current capabilities, risks, confirmation requirements, and limitations:

PYTHONPATH=runtime python3.11 -m mobile_agent.cli.device_inspect <device_id>

MCP also provides read-only mobile_list_apps and mobile_inspect_app for bounded listing of app identifiers and querying a single app's version, install source, and enabled state. They do not return APK paths, signatures, permissions, or raw dumpsys, and do not launch or modify apps.

Local APK installation only accepts a single .apk in the <data-dir>/apks directory. An external Agent must first call mobile_prepare_apk_install to obtain a short-lived Approval containing the file name, size, SHA-256, Manifest package id, and replacement impact; the MCP Host shows this summary to the user and, after obtaining explicit confirmation, may call mobile_install_apk. The Approval expires in ten minutes and can only be used once by default. The Runtime does not download URLs and does not accept split APKs or arbitrary ADB arguments.

App uninstall uses a separate two-phase mobile_prepare_app_uninstall → mobile_uninstall_app flow. Prepare only returns the app version, system-app determination, and data deletion impact in read-only mode; system apps or apps with unknown attributes are rejected outright. The user must explicitly confirm the summary again before an async uninstall task can be submitted. Failed or unknown-outcome tasks must not be automatically retried.

App lifecycle provides mobile_inspect_app_state, mobile_launch_app, and mobile_stop_app. State checks only return whether the process exists, whether it is in the foreground, and the stopped flag; launch and stop return an async task_id, and explicit confirmation is required before stopping a non-system app. To permanently clear app data, you must first call mobile_prepare_app_data_clear, show the package name, version, and data deletion impact, obtain a fresh explicit confirmation, and then call mobile_clear_app_data. Clearing app data does not uninstall the app, and failed or unknown outcomes must not be automatically retried.

After explicit confirmation, collect a recent log snapshot (requires the local API token generated at Runtime startup):

PYTHONPATH=runtime python3.11 -m mobile_agent.cli.device_logs_collect \
  <device_id> --max-lines 500 --minimum-level info --confirm --token <runtime-token>

Logs are redacted first, then saved as a local Artifact up to 1 MiB; CLI and REST do not return log bodies. Add --async-task to immediately get a task_id and use unified execution status, events, cancellation, and task reports:

PYTHONPATH=runtime python3.11 -m mobile_agent.cli.device_logs_collect \
  <device_id> --confirm --async-task --deadline-seconds 60 --token <runtime-token>

Collect aggregated CPU, memory, battery temperature, and system load snapshots:

PYTHONPATH=runtime python3.11 -m mobile_agent.cli.device_performance_snapshot \
  <device_id> --async-task --deadline-seconds 90 --token <runtime-token>

Performance Artifacts contain only aggregated JSON metrics, not raw dumpsys, process names, or app details.

Collect screenshots, UI Tree, redacted logs, aggregated performance, and optional app state in one pass, and generate a local ZIP with a SHA-256 manifest:

PYTHONPATH=runtime python3.11 -m mobile_agent.cli.diagnostic_bundle_collect \
  <device_id> --app-id <package-id> --max-log-lines 500 \
  --minimum-log-level info --confirm --token <runtime-token>

Diagnostic bundles are Medium risk and require explicit confirmation. CLI, Web, REST, and MCP only return Artifact metadata and a safety summary, never inlining screenshots, UI Tree, logs, or ZIP contents; file names inside the bundle are fixed, total size does not exceed 24 MiB, and nothing is uploaded or sent externally.

View local Artifact usage and read-only pre-check evidence exceeding the default 7-day retention period:

PYTHONPATH=runtime python3.11 -m mobile_agent.cli.local_storage
PYTHONPATH=runtime python3.11 -m mobile_agent.cli.local_data_cleanup_prepare \
  --retention-days 7 --max-artifacts 500 --token <runtime-token>

Prepare does not delete files; it only returns candidate count, size, cutoff time, and a short-lived Approval. Only after the user reviews the impact summary and explicitly confirms again can an async cleanup task be submitted:

PYTHONPATH=runtime python3.11 -m mobile_agent.cli.local_data_cleanup \
  <approval-id> --confirm --token <runtime-token>

Cleanup only accepts system-generated Artifact IDs, relative paths, sizes, and SHA-256 hashes bound to the Approval; it does not accept arbitrary paths, does not delete the task database, configuration, keys, or APKs, and does not run automatically in the background or auto-retry failures.

Compare two completed performance snapshot tasks on the same device:

PYTHONPATH=runtime python3.11 -m mobile_agent.cli.device_performance_compare \
  <baseline_task_id> <candidate_task_id> --token <runtime-token>

The Web task report can also set one successful snapshot as a baseline and select another snapshot for comparison. Comparison results only represent the numeric direction and stability threshold of two point samples, and do not automatically determine causality or performance regression.

If adb is not in PATH, you can configure it explicitly:

MOBILE_AGENT_ADB_PATH=/usr/local/platform-tools/adb

ITER-0003 adds GET /v1/tools, POST /v1/tools/{tool_id}/invoke, and POST /v1/skills/app.open/invoke. input.tap is Medium risk and requires explicit confirmation by default.

ITER-0004 adds safe UI hierarchy parsing, semantic Selectors, input.tap_element, and POST /v1/skills/settings.navigate/invoke. Semantic taps are Medium risk and are rejected when the match is not unique.

ITER-0005 adds policy-constrained input.swipe, input.text, bounded semantic scroll search, and POST /v1/skills/settings.scroll_navigate/invoke. Scrolling and input are both Medium risk and require explicit confirmation by default; password, verification code, payment, account security, and auto-submit scenarios are out of scope for this iteration.

ITER-0006 adds a minimal Task Runner, TaskRun evidence reports, and the preview synchronous endpoint POST /v1/tasks/settings.scroll_navigate/run. This endpoint only wraps the existing settings.scroll_navigate Skill and does not replace the future async task queue design.

ITER-0007 adds an in-process Task Store, TaskEvent, and query endpoints GET /v1/tasks/{task_id} and GET /v1/tasks/{task_id}/events. This Store is only valid for the current Runtime process lifetime and does not yet provide restart recovery.

ITER-0008 adds the first version of the CLI task report view, which can render TaskRun and TaskEvent as user-readable reports:

PYTHONPATH=runtime python3.11 -m mobile_agent.cli.task_report <task_id>

This command queries tasks and events from the local Runtime API.

ITER-0009 adds a SQLite Task Store, saving tasks and events to <data-dir>/mobile-agent.db by default. When MOBILE_AGENT_DATA_DIR is set, the database is located in that directory; otherwise, the platform default local data directory is used.

ITER-0010 adds a historical task list:

PYTHONPATH=runtime python3.11 -m mobile_agent.cli.task_list --limit 20

The list shows recent task summaries; you can copy the task_id and use task_report to view details.

ITER-0011 adds a local Web UI. After starting the Runtime, open:

http://127.0.0.1:8765/ui

to view task history and task report details.

ITER-0012 adds a "Run Safe Demo" button in the Web UI. This button selects an online Android device and runs a fixed task: open system settings and navigate to the Display/Brightness page. POST requests still use the local Runtime token and can only be triggered from same-origin loopback pages.

ITER-0013 adds the Agent Loop Preview before model integration: POST /v1/tasks/agent.run. This endpoint uses a deterministic Planner to generate constrained decisions, currently supporting only the safe demonstration goal "navigate to the Display/Brightness page in system settings", and writes observation summaries, Planner decisions, Skill execution results, and evidence into the task report.

ITER-0014 adds a natural language task input box and a "Run Agent Preview" button in the Web UI. The page calls POST /v1/tasks/agent.run and refreshes the history list and opens the task report after the task returns.

ITER-0015 adds the internal preview contract for the LLM Planner and MockLLMPlanner. Model-style output must first pass structured parsing and field validation, then be re-validated by the Agent Runner's Skill allowlist; this iteration does not call real model services or read model keys.

ITER-0016 adds an OpenAI-compatible Planner Provider preview that is disabled by default. The Provider can construct chat-completions-style requests, parse structured responses through an injectable transport, and reuse ITER-0015's Planner output validation; the default Runtime does not enable a real Provider, and tests do not depend on the network or model keys.

ITER-0017 adds the model Provider configuration gate: the default configuration still returns RuleBasedPlanner; only when openai_compatible is explicitly enabled, base_url, model, and api_key_ref are provided, and the key is resolved through the injected SecretResolver, is an OpenAI-compatible Planner constructed. This iteration does not integrate with the default Runtime or read real keys.

ITER-0018 adds a read-only model Provider status entry: GET /v1/model-provider/status and a "Model Provider" status panel in the Web UI. The status only shows whether it is enabled, the provider, the model, and whether a key reference is configured; it does not return real keys or the raw api_key_ref; the default Runtime still does not enable a real model.

ITER-0019 adds local model configuration reading: at startup, the Runtime reads <data-dir>/model-provider.json or the file specified by MOBILE_AGENT_MODEL_CONFIG, and allows MOBILE_AGENT_MODEL_* environment variables to override configuration fields. The configuration file only stores api_key_ref; the development preview SecretResolver only resolves env:MOBILE_AGENT_MODEL_SECRET_* references; the default Agent Runner still does not call a real model.

ITER-0020 brings the model Planner into the default Runtime in a controlled manner: when configuration is off, the rule Planner continues to be used; when configuration is on and the key reference is resolvable, the OpenAI-compatible Planner is used; when configuration is on but unavailable, Agent tasks fail explicitly with MODEL_UNAVAILABLE rather than silently falling back to the rule Planner. Model output must still pass structured parsing, the Skill allowlist, the Policy Engine, and the Device Gateway.

ITER-0021 enhances the Web UI model Provider status card: distinguishing not enabled, connected, configuration unavailable, and configuration read, and prompting to check the configuration file, MOBILE_AGENT_MODEL_CONFIG, and MOBILE_AGENT_MODEL_SECRET_* when unavailable. The repository provides a configuration example: model-provider.example.json.

ITER-0022 upgrades the Agent Preview from "one round of model decisions calling a large Skill" to "multi-round model decisions + atomic Tool execution + re-observation each round". The Planner can output run_tool and finish; the Runtime only allows whitelisted Tools and performs deterministic verification via UI Selector on finish; the old run_skill path is retained for compatibility.

ITER-0023 promotes AgentObservationSummary, AgentDecision, and AgentStepResult in each Agent round report to public JSON Schemas and updates the TaskRun Schema to officially support agent.run. Desktop, CLI, and future external Agents can stably consume multi-round Observe–Plan–Act reports.

ITER-0024 adds Agent action progress feedback: the Runtime compares the foreground app and UI tree before and after a Tool, feeds changed / unchanged feedback to the next model round, and blocks re-dispatching the same no-progress action. The Web report also shows the actual Tool, parameters, and page progress.

ITER-0025 optimizes model-side Observation: filters out nodes without semantic layout, prioritizes visible text and actionable nodes, adds summary truncation metadata, and redacts common phone numbers, emails, and long numeric identifiers from UI text before it enters the model Prompt and task summary.

ITER-0026 adds a strict Agent ToolCall Contract, one bounded repair for invalid model parameters, and evidence retention for failed rounds. A multi-round model closed loop for "navigate to Display and Brightness" has been completed on a real device.

ITER-0027 establishes a goal-driven online Agent evaluation foundation: the real model re-plans against the current device UI each time; evaluation only constrains the goal, final state, disabled Tools, and round budget, and does not compare fixed action paths. Completed agent.run tasks can be evaluated via POST /v1/tasks/{task_id}/evaluate; a scenario example is at agent-evaluation-scenario.example.json.

ITER-0042 organizes multiple path-independent scenarios into a versioned Suite. First execute the Suite's goals via Web or MCP separately, then hand the completed task_ids to the read-only aggregation CLI; this command only calls existing evaluation APIs and does not submit or replay device actions:

./scripts/report-mcp-evaluation.zsh \
  --suite evaluations/android-settings-smoke-v1.json \
  --task settings.bluetooth.v1=task_<id> \
  --task settings.display-brightness.v1=task_<id> \
  --task settings.battery.v1=task_<id>

The report shows overall success rate, per-scenario success rate, p50/p95 latency, average rounds and Tool count, plus Provider retry, NO_PROGRESS, MODEL_UNAVAILABLE, and policy violation statistics. The Suite defines goals and independent success conditions, not fixed action paths. The script only reads the mobile-agent local connection info registered in Codex, does not print tokens, and does not submit device tasks.

ITER-0028 strengthens reliability: side-effect-free goal targeting and finish verification failures can be fed back to the model as failed rounds for continued planning; finish can combine foreground app/activity and UI Selector; taps in the top system area and bottom gesture area are intercepted before dispatch. Provider timeout, HTTP, connection, and response format errors are classified and recorded, and retryable model requests are retried at most once; invalid Selectors only show field-level redacted diagnostics. When the model omits the non-security-critical reason, the Runtime generates a fixed audit note without issuing an additional model repair request; Tool, Selector, Policy, and completion conditions remain strictly validated.

ITER-0029 adds optional caller-provided Runtime-owned success conditions. POST /v1/tasks/agent.run can accept acceptance, using all-of semantics over the foreground app id, Activity, and unique UI Selector to verify the model's finish; the path is still dynamically planned by the model based on live Observation. The task report persists and displays goal_acceptance and completion_source. A request example is at agent-run-runtime-acceptance.example.json.

ITER-0030 adds two-phase goal compilation: POST /v1/goals/compile converts a short natural language goal into a reviewable AgentGoalSpec draft containing an enhanced execution goal, assumptions, confidence, and optional success conditions. The model draft must be explicitly confirmed by the user before being passed to agent.run; the task is still dynamically planned by the model based on live Observation, without generating fixed action paths. See agent-goal-spec.example.json for an example.

ITER-0031 adds async Agent execution: POST /v1/tasks/agent.run/async immediately returns 202 Accepted and a task_id; GET /v1/task-executions/{task_id} and /events provide persisted state and per-round events; POST /v1/task-executions/{task_id}/cancel requests cancellation at a safe boundary. Async creation supports Idempotency-Key; the original synchronous POST /v1/tasks/agent.run remains compatible. The local Web UI uses the async entry by default.

ITER-0032 adds a same-device exclusive lease for Runtime public write entries and an optional deadline_seconds for synchronous/asynchronous agent.run (default 600 seconds, range 1–1800). When the device is occupied by another task, DEVICE_LOCKED is returned; tasks exceeding the budget at a safe boundary end with timed_out/TASK_DEADLINE_EXCEEDED, and evidence from completed actions is retained.

ITER-0033 adds a Runtime single-instance lock for the same data directory and generates a session_id for each continuous online device connection. Tasks and leases are bound to the current Session; after device disconnect or reconnect, old tasks stop with DEVICE_SESSION_CHANGED and do not send subsequent actions to the new connection. Device, TaskExecution, TaskRun, and Web/CLI reports all show the session identifier.

ITER-0034 adds unified Runtime/Device Readiness: Web and CLI use the same read-only Contract to interpret ADB, device connection and authorization, Session, and Lease states; only ready devices can start tasks from the Web. When ADB is missing, the Runtime enters diagnostics mode and provides remediation suggestions, without auto-installing tools or modifying device configuration.

ITER-0035 adds Device Inspection and a Capability Catalog. In the Web UI, clicking a device shows eight basic V1 capabilities; GET /v1/devices/{device_id}/inspection and the CLI show current availability, risk, idempotency, verification requirements, associated Tools, and limitations. Inspection only reads device discovery and Lease; it does not take screenshots, read the UI, or execute actions.

ITER-0036 adds the first engineering diagnostic Skill: POST /v1/skills/device.logs.collect/invoke. The Android Adapter only accepts bounded line counts and fixed log levels, using fixed logcat arguments to collect a snapshot; the Skill requires explicit Medium risk confirmation and generates a device_log Artifact after local redaction. Web and CLI only show Artifact metadata. Continuous streaming collection, arbitrary logcat filters, and log upload are out of scope for this iteration.

ITER-0037 integrates log collection into the unified async task pipeline: POST /v1/tasks/device.logs.collect/async returns 202 Accepted and reuses TaskExecution state, incremental events, Idempotency-Key, cancellation, Deadline, Device Session, Lease, and persisted TaskRun reports. The Web log button submits asynchronously by default; the synchronous Skill endpoint remains. The executor only allows Agent and log task types registered in code; clients cannot submit arbitrary handlers.

ITER-0038 adds device.performance.snapshot: the Android Adapter collects total CPU, Total/Free RAM, battery level/temperature, uptime, and load average via fixed read-only commands, writing only normalized values to a local JSON Artifact. Synchronous Skill, async Task, Web, and CLI share the same Contract; no app/PID details or continuous sampling are provided.

ITER-0039 adds POST /v1/performance-comparisons, taking two successful performance snapshot TaskRuns on the same device as input and computing two-point differences and threshold trends for CPU, memory, battery, temperature, and load. Comparison is entirely local, reading structured task evidence without accessing the device, model, or raw dumpsys; Web and CLI clearly note that two point samples cannot alone prove causality or performance regression.

ITER-0040 adds the MCP 2025-11-25 stdio developer preview. The MCP subprocess only calls the fixed localhost REST API of an already-started Runtime, so it shares tasks, Session, Lease, and Policy with the Web; all time-consuming capabilities return a ThumbAgent task_id asynchronously. Tool inputs come from public Contracts, are strictly validated and rate-limited before invocation, and domain errors are returned as structuredContent. MCP Tasks, remote transport, Resources, and Prompts are not yet implemented.

ITER-0047 adds device.diagnostics.bundle. One confirmed async task combines Observation, redacted logs, aggregated performance, and optional app state within the same Device Session and Lease, generating a local ZIP with fixed contents. The Manifest records the names, sizes, and SHA-256 hashes of the four source Artifacts; the Runtime re-validates source integrity and the ZIP file set before publishing the diagnostic bundle, retaining already-completed safe Artifact references on failure.

ITER-0048 adds local Artifact storage summary and two-phase expired cleanup. Preview only scans system-generated files and returns an aggregated impact summary; Submit only accepts a ten-minute-valid, single-use, scope-bound High risk Approval, verifying path, size, SHA-256, and cutoff time item by item before deletion. Async tasks do not acquire a Device Session or Lease; cancellation and Deadline at the safe boundary between Artifacts block subsequent deletions and retain the completed deletion summary.

If a real Provider consistently completes responses near the default 30-second budget, you can raise timeout_seconds to 60 (allowed range 1–120) in the local configuration, or override with MOBILE_AGENT_MODEL_TIMEOUT_SECONDS=60. Timeout retries may incur additional model calls; the task report shows the retry count.

Product Documentation

• Product Positioning

• V1 Product Proposal

• V1 Technical Design

Engineering Standards

• Agent Development Entry

• Contribution Guide

• Documentation and Iteration Standards

• Engineering Development Standards

• Contract and API Evolution Standards

• Capability Model

• Skill Development Standards

• Reliability and Execution Semantics

• Data and Migration Standards

• Error and Diagnostic Standards

• Architecture Boundaries

• Testing Standards

• Security Standards

• Multi-Agent Collaboration Standards

• Architecture Decision Records

• Iteration Index

License

Apache-2.0