AutoPlayQA

Android game QA automation testing framework — a deterministic "eyes (perception) + hands (actions)" recognition-gated task engine. The brain is delegated to an external AI agent (Claude Code / Codex driven via MCP or CLI); the project itself calls no LLM, zero token cost.

The framework is not bound to any specific game: the scene label taxonomy, task JSON, template images, and YOLO models are all provided by the integrator (the project hosting the game under test); this repository only handles the generic parts — perception channels, action backends, the recognition-gated task engine, QA forensics, and the MCP / CLI interfaces.

Table of Contents

• Features

• Quick Start

  • Environment

  • Option 1: MCP (Recommended)

  • Option 2: Local CLI

• Task Engine (Core)

• Creating Tasks

• Directory Structure

• Testing

• About

• Acknowledgments

Related MCP server: scrcpy-mcp

Features

Perception and Actions (deterministic eyes and hands)

• 📱 Auto-detects connected Android emulators and physical devices (ADB); supports wireless adb: connect / pair (Android 11+) / tcpip one-click switching, config list auto-connects on startup

• 🤖 One device, one Agent; Agent pool manages multiple devices

• 👁️ Free screen-locating dual channel: uiautomator dump widget matching → local OCR (rapidocr, also works for games rendered on a single Surface)

• 🧩 OpenCV template matching: recognizes graphics invisible to the text channel (game buildings / icons and other pure textures) — multi-scale scanning + transparent channel mask + multi-instance NMS; capture_template to collect → find_template to locate / tasks use template recognition-gated clicks

• 🧷 ORB feature matching: the deformation-resistant sibling of template matching — describes local keypoints rather than per-pixel correlation, holding up against minor revisions / scaling and rotation / partial occlusion; tasks use feature recognition gating. Only suitable for texture-rich anchors (flat solid-color icons yield no keypoints; use template for those)

• 🔎 YOLO object detection (optional, onnxruntime inference, no PyTorch): trained models locate + classify objects in the frame, resistant to displacement · scaling · occlusion (the Achilles' heel of template matching); models are trained and provided by the integrator — drop a .onnx into task/models/ to enable it, detect_objects for detection / tasks use yolo recognition gating, auto-yields lazily when no model is present; model versions are recorded in task/models/models.json (filename → version / date / notes / classes / training_ref), list_yolo_classes also reports the version to the agent

• 🧭 Scene classification (scene): answers "which screen am I on" for the whole frame, used to confirm position after a run-away and for abnormal-branch assertions (returns no coordinates, does no anchor locating). The framework only ships one built-in label blank (near-black / screen-off / empty frames), plus the non-scene signals other_app and the never-guessed unknown; all other labels are registered by the integrator via register_scene_probe(label, fn, *, description=..., order=...) (also unregister_scene_probe / clear_scene_probes / registered_scene_probes). expected matches by dot-prefix ("popup" hits popup.error, "menu" hits menu.settings); MCP classify_scene returns the currently active taxonomy

• 🏷️ Set-of-Marks annotated image: screenshot overlaid with numbered badges (red = clickable widget, blue = plain text), the agent clicks by number (click_index) without guessing coordinates

• 🎯 Click / drag / text input / key press / wait

• ✋ Rootless multi-touch gestures: app_process launches a lightweight dex helper that goes through the system-hidden injectInputEvent (same privileged path as input), no root, no writing to /dev/input (modern MIUI / HyperOS have SELinux-blocked shell domain) — can inject multi-touch MotionEvents; the gesture action accepts frames frame sequences or the pinch convenience parameter, solving dynamic gesture bottlenecks like two-finger pinch / rotate / two-finger drag (the dex is not committed to the repo; build it from auditable source via injector/build.ps1)

Set-of-Marks annotation illustration (synthetic screen, not a real game screenshot): screenshot_marked gives clickable widgets red numbers and plain text blue numbers, returning an indexed element table; the agent then calls click_index(N) to click by number without guessing coordinates.

Task Engine (deterministic replay)

• 🔁 Recognition-gated task engine: task JSON state machine (recognition confirms arrival at the expected screen before executing actions), supporting branches, timeout recovery, and resume from breakpoint — one operation, zero-token replay

• 🧩 Composable tasks: includes shares node files (write a generic popup handler once and reference it everywhere) + custom in-process deterministic actions (built-in swipe_until swipe-to-find-target, launch_app cold start, gm_command to issue GM commands, etc.)

• 🗂️ Replay anchor cache: OCR checks the cached ROI first, then falls back to full-screen for speed; anchor displacement is reported as an anchor_drift finding rather than silently self-healing

• ⏭️ Report-then-skip (bug-skip): when a bug is detected (watchdog hit / logcat crash·ANR), it can be reported with evidence and then jump to a recovery node to continue testing, without aborting; pure lag / timeout never triggers a jump (that's on_timeout's job) — only reported bugs, not slowness, reroute the flow

• 🧹 Benign popup whitelist: the task's popups field explicitly lists known benign popups (user agreements / in-game alerts and other expected noise); when recognition gets stuck, they are auto-dismissed without recording a finding; popups not on the list still stall into timeout / get caught by the watchdog — distinguishing "noise" from "anomaly" without silently swallowing bugs

• 🔙 BACK fallback: when the whitelist is exhausted and the task is still stuck (unknown popup covering the screen), the finding is first pinned to that frame as evidence, then BACK is pressed once and pixel-diff confirms the screen actually changed before granting one more recognition round; when the node has its own on_timeout, it yields to the author's recovery branch, and never jumps (jumping is bug-skip's job)

• 🩺 Anchor health: each round tallies node hit sources (direct hit / timeout recovery / popup assist / BACK fallback / drift) into node_stats; nodes that repeatedly only pass via fallbacks are flagged anchor_rot_suspect (task anchor rot, not a game bug); CLI task health aggregates across runs to see trends, task lint pre-checks fragile patterns before saving (W001-W007)

• 🔗 Suite chaining (suite): multiple cases compiled into one suite JSON (cases + mandatory resume_after/case_entry/landing, no framework defaults), sharing one cold start + login and running consecutively; each case is still an independent run with its own findings directory; case failure follows on_case_failure to restart-and-retry / skip / abort

QA Forensics (anomalies are test findings)

• 🔬 Three trigger types (findings are always recorded, independent of debug switches): task-level watchdogs negative assertions (forbidden text / white screens that must not appear), node finding fields (popup·error branches self-report), logcat crash·ANR monitoring — any hit records a finding

• 🛫 Trigger-then-preserve-evidence: on-the-spot screenshot (the exact error frame) + ui_dump on failure, plus a "flight recorder" black box — ~60s of context before the issue: logcat excerpts · flow timeline · on-device rolling screen recording (real MP4)

• 📦 Result delivery: run results carry findings (listed even on task success), the whole evidence folder can be exported (screenshots + logs + screen recording + report.json, self-contained relative paths)

• 📄 Human-readable report: the same data is rendered into report.html — a zero-external-link single file, double-click to open offline, safe to email around; screenshots embedded as <img>, screen recordings as <video>, logcat and flow timeline collapsible — for QA colleagues who don't read JSON

• 🧾 Evidence retention: outputs/findings/<date>/<device>/<run_id>/ is self-contained and browsable; startup cleans expired date directories per findings.retention_days (default 14 days); with findings.export_dir configured, runs with findings are auto-packed into a single zip (timestamp_task_device_status.zip)

• 🛰️ Gap-period sentinel: after a task finishes / during agent handoff, that engine round's run is sealed and the screen and logcat would otherwise be unwatched — a background frame monitor mounts a sentinel that reuses existing frames (no extra screenshots, no extra adb round-trips) to keep checking for white-screen freezes (N consecutive frames with grayscale stddev below threshold counts as one episode, reported once, re-armed after recovery) and crash / ANR; on a hit it writes an ordinary findings run (task name monitor_sentinel) with an extra lossless original image as evidence. Gated per device: while the engine runs on device A, the sentinel still watches device B

• 📣 Result push: unattended runs don't require someone to go dig through directories — with findings.notifiers configured (Feishu custom bot / generic webhook), each run wraps up by pushing one Chinese summary (task / device / status / counts per severity / top 3 findings / report and evidence package paths); min_findings, on_status filtering, clean runs stay quiet by default, push failures only log and never affect run results

Offline report report.html structure illustration (synthetic screen, not a real game screenshot): one finding = evidence screenshot of the error frame + fields + embedded screen recording + collapsible logcat excerpt and flow timeline; zero-external-link single file, double-click to open offline, safe to forward.

Performance and Integration

• ⚡ scrcpy frame-stream screenshot backend (default): resident H.264 stream locally decoded at ~13ms/frame, screencap as fallback (any failure auto-falls back, repeated failures latch off); set capture.backend: screencap when exact pixels are needed or scrcpy is unavailable

• 🔌 MCP server: plug-and-play with Claude Code / Codex

• 📝 Multiple task creation methods: hand-written JSON / agent pathfinding on a real device / observational recording (user demonstrates manually, agent monitors and generates) / CLI session recording drafts

• 🎨 Visual orchestration: pipeline_editor/'s Web canvas editor (truth-value validation + lint, screenshot ROI/template capture, real-device run highlighting, real-time collaboration with the agent via embedded MCP), see Visual Orchestration: PipelineEditor

Quick Start

Environment

# Python 3.11 环境（conda / venv 均可）
conda create -n autoplayqa python=3.11 -y
conda activate autoplayqa
pip install -r requirements.txt

# adb 需在 PATH（Android SDK platform-tools 默认安装位置）
$env:PATH = "$env:LOCALAPPDATA\Android\Sdk\platform-tools;$env:PATH"

Option 1: MCP (Recommended, Claude Code / Codex as the brain)

Copy .mcp.json.example to .mcp.json, change command to the absolute path of your environment's Python interpreter; then start Claude Code in this project directory and it will auto-discover the autoplayqa server.

The template also includes a second entry pipeline-editor (http, http://127.0.0.1:8930/mcp): the editing-side MCP embedded in the visual editor PipelineEditor's backend, only available after the editor starts; when it's not running, the agent automatically uses only the stdio autoplayqa (the editing tools share the same names and semantics, the user just doesn't see the live canvas). Division of labor is detailed in docs/MCP_INTEGRATION.md.

For Codex CLI, add to ~/.codex/config.toml (adjust paths for your machine):

[mcp_servers.autoplayqa]
command = "C:\\path\\to\\python.exe"
args = ["C:\\path\\to\\autoplayqa\\mcp_server.py"]

Then simply tell the agent: "Connect the device, open Settings and set brightness to 50%, save it as a task when done."

MCP Tool List

Option 2: Local CLI

python main.py       # 交互式 CLI（确保 adb 已在 PATH）

A glimpse of an interactive CLI session (illustrative output, not a real transcript).

Task Engine (Core)

A task is a recognition-gated state machine (task/task_definitions/*.json): each node first recognizes (ui_text control matching / ocr / template icon matching / feature ORB features / yolo object detection / scene full-screen scene / blank_screen / always) to confirm the UI, and only on a hit executes the action, then polls the next candidate list to transition—whichever candidate recognizes first wins (naturally supporting popup branches); on timeout, the on_timeout recovery node is taken.

flowchart TD
    R["识别当前节点锚点<br/>ui_text / ocr / template / feature / yolo"] -->|命中| ACT["执行动作<br/>click · swipe · custom · agent"]
    ACT --> NEXT{"轮询 next 候选<br/>谁先识别命中走谁"}
    NEXT -->|命中某候选| R
    R -->|迟迟不命中| POP["popups 良性弹窗白名单清扫<br/>（不记 finding）"]
    POP -->|仍卡住| BK["BACK 兜底<br/>先记 finding 留证再按 BACK"]
    BK -->|还不行| TO["on_timeout 恢复节点"]
    ACT -->|agent 动作| HALT["挂起 agent_required<br/>外部智能体接手 → start_after 续跑"]
    WD["watchdog 断言 / logcat 崩溃·ANR"] -->|"命中：记 finding + 截图留证"| SKIP["bug-skip：skip_to / on_finding<br/>跳恢复节点继续测"]

{
  "entry": "点击设置",
  "nodes": {
    "点击设置": {
      "step": "1",
      "recognition": {"type": "ui_text", "expected": "设置"},
      "action": {"type": "click", "target": "recognized"},
      "next": ["进入显示设置"]
    },
    "进入显示设置": {
      "step": "2",
      "recognition": {"type": "ui_text", "expected": "显示"},
      "action": {"type": "click", "target": "recognized"},
      "next": [],
      "on_timeout": "点击设置"
    }
  }
}

Steps requiring intelligent judgment use the agent action: the engine suspends and returns status=agent_required + the instruction text; the agent completes that step using device tools, then calls run_task(start_after=<node>) to resume. See get_task_schema or action/action_schema.py for the full format; the handoff round-trip sequence diagram is in docs/MCP_INTEGRATION.md.

Step number (step): A node may carry a read-only step number identifying execution order—the main path (from entry along next[0]) is integer 1, 2, 3…, fallback branches (on_timeout / next[1:]) get dotted numbers like 2.1, 2.1.1, and unreachable nodes are ?. A task is a graph, not a list—reading the JSON top to bottom won't reveal execution order; the step number lets a human/agent instantly locate a node's position in the flow. The engine never reads it—purely for navigation. Step numbers are computed live from the graph: CLI task renumber <name> recomputes from the current graph and writes step back to the file (placed at the head of the node); task show <name> first prints the outline sorted by step number, then the original text; MCP get_task additionally returns _steps (name→step mapping) + _step_outline (flow listing). After editing a task, rerun renumber to refresh—no stale numbers are left behind.

Background execution (long tasks): run_task blocks synchronously and only returns when finished, with no progress in between—poor experience for long tasks / full-flow smoke tests. Switch to start_task to get a run_id immediately, then poll get_run_status(run_id): it returns status (running / agent_required / done / error) + current_node + steps + elapsed_s, and in the terminal state a complete result isomorphic to run_task (steps / findings / report / handoff). The engine is a singleton—only one background run at a time; when agent_required, complete that step per result.handoff, then call start_task(start_after=<node>) to resume.

Suite chained runs (suite): Multiple cases share one cold start + login and run consecutively (task/task_definitions/suites/*.json; MCP list_suites / run_suite, CLI task suites / task suite <name>). The suite JSON declares a cases list plus resume_after (skip the opening and resume directly at a node) / case_entry (case body entry) / landing (recognition spec verifying the landing screen between cases)—all three fields required, no framework defaults; validate_suite checks before running and errors out on missing fields. The first case does the full cold start; subsequent cases skip repeated login via resume_after, but each case is still an independent run (independent findings directory / report.json); case failure (crash / hang / landing screen mismatch) is handled per on_case_failure: restart_retry (default, cold-start retry) / restart_continue (no retry, next case cold-starts normally) / abort (terminate the suite, remaining cases marked skipped).

{
  "name": "smoke_mini",
  "cases": ["chat_smoke", "main_smoke"],
  "resume_after": "主场景确认",
  "case_entry": "用例开始",
  "landing": {"type": "ocr", "expected": "主界面", "roi": [0, 2280, 1080, 2448]}
}

Shared nodes (includes): Shared nodes like common popup handling live in task/task_definitions/common/*.json (containing only "nodes", single-level references); tasks pull them in with "includes": ["common/popups.json"]. Cross-file next/on_timeout references are validated as a whole on the merged node table, and only run if all pass (atomic loading). Duplicate node names error by default (strict); setting "on_conflict": "overwrite" lets the loader override—the main file merges last, so tasks can specialize shared nodes. References are preserved on save; include file updates take effect on the next run of referencing tasks.

Deterministic complex steps (custom): Multi-step deterministic logic between a single adb atomic action and an agent suspension (no intelligent judgment needed), written as Python handlers registered with @register("name") into task/custom_actions/, referenced in tasks with {"type": "custom", "name": "swipe_until", "params": {...}}, validated as registered at load time. Built-ins: swipe_until (repeatedly swipe until recognition hits—find targets in lists / scrollable pages), launch_app (wake screen + launch app, cold-start opening), gm_command (send commands via GM panel, auto-handles the input method), ensure_checkbox (flip a switch to the target state), set_text_field (clear then input). New task/custom_actions/<module>.py files are auto-discovered and registered within the package (pkgutil scans by name for import)—no manual __init__.py wiring needed; a module import failure throws immediately (fail-fast), never silently becoming a runtime "unregistered" error.

Report-then-skip (bug-skip): After detecting and reporting a bug, instead of aborting, jump to a specified recovery node and continue testing. Two layers of annotation—watchdog skip_to (jump on hit, highest priority, overrides fail_task) + task-level on_finding (global fallback target, also covers logcat crashes / ANR and other bugs without a watchdog). Key constraints: only a "newly recorded finding" triggers a jump (the same watchdog jumps at most once per round, deduplicated via a seen set), and the trigger source is only watchdog hits + logcat crashes / ANR—pure recognition timeouts (stalls, no bug) never jump and still go through on_timeout. At load time, skip_to / on_finding node references are validated to exist.

In a real task it looks like this (the core example above was pure flow and never showed these fields—task-level watchdogs / on_finding, node-level finding):

{
  "entry": "开始战斗",
  "on_finding": "回到主界面",
  "watchdogs": [
    {"type": "ocr", "expected": "网络错误", "skip_to": "回到主界面", "message": "战斗中弹出网络错误"},
    {"type": "blank_screen", "fail_task": true, "message": "黑屏卡死"}
  ],
  "nodes": {
    "开始战斗": {
      "recognition": {"type": "ui_text", "expected": "开始"},
      "action": {"type": "click", "target": "recognized"},
      "next": ["结算页", "战斗失败弹窗"]
    },
    "结算页": {
      "recognition": {"type": "ocr", "expected": "胜利"},
      "action": {"type": "none"},
      "next": []
    },
    "战斗失败弹窗": {
      "recognition": {"type": "ocr", "expected": "战斗失败"},
      "finding": {"severity": "warning", "message": "战斗失败弹窗（异常分支，自我上报）"},
      "action": {"type": "click", "target": "recognized"},
      "next": ["回到主界面"]
    },
    "回到主界面": {
      "recognition": {"type": "ui_text", "expected": "主界面"},
      "action": {"type": "none"},
      "next": []
    }
  }
}

• watchdogs[0].skip_to: OCR detects "network error" during combat → record finding, then jump to return to main screen to continue testing;

• watchdogs[1].fail_task: black screen → fail the task outright (no skip_to fallback for it);

• Task-level on_finding: bugs without a corresponding watchdog, like logcat crashes / ANR, all fall back to return to main screen;

• Node battle-failed popup.finding: the abnormal branch self-reports the moment it's entered, without relying on a watchdog.

Benign popup whitelist (popups): The task popups field explicitly lists known benign popups (user agreements, in-game alerts, and other expected noise) with their recognition + dismissal actions (only click / key / gesture). Only when recognition is stuck does it scan and dismiss them, without recording a finding (screenshots are the performance bottleneck, so no per-step overhead is added); popups not on the whitelist still stall into timeouts / get caught by watchdogs—unlike forks that silently swallow popups, this project insists on "anomalies are discoveries, don't self-heal silently". Dismissed popup names are returned in result["popups_dismissed"].

Unknown popup BACK fallback (back_fallback): If the whitelist is exhausted and still stuck, whatever is covering the screen is unexpected. The engine first pins an unknown_popup_backoff finding to the current frame (evidence first—BACK may wipe the scene), then presses BACK once, and uses pixel differencing to confirm the screen actually changed before granting one more recognition round. It only unblocks, never navigates (navigation is bug-skip's job); when a node has its own on_timeout, it fully yields to the author's recovery branch, so it only covers dead ends that would otherwise inevitably fail. Config engine.back_fallback is on by default; "back_fallback": false in the task JSON disables it per-task.

Anchor health (node_stats / task health / task lint): The engine tallies hit sources per node—direct hit, timeout recovery, popup assistance, BACK fallback, anchor drift—producing result["node_stats"] and writing it into report.json. Nodes that repeatedly only pass via fallbacks within a round, or whose anchors keep shifting, get an anchor_rot_suspect warning finding: this is task anchor rot, not a game bug (thresholds engine.rot_suspect_timeouts / engine.drift_tolerance_px). CLI task health [name] [--days N] aggregates historical runs' node_stats offline to see trends; task lint <name> (also auto-run on save_task) checks for "legal but fragile" patterns—dead nodes without recovery branches, branches that look like errors but don't report, cold-start tasks without a popup whitelist, hardcoded coordinates where anchors exist, tasks with zero QA assertions—warnings only by default; config lint.strict: true changes it to refuse saving.

Creating Tasks

Four ways, ordered by recommendation:

Whichever way, writing tasks follows the same convention: recognition anchors prefer ui_text (system UI) / ocr (game single-Surface text) / template (icons, sprites, and other textless elements) / feature (texture-rich anchors that change slightly) / yolo (trained object detection, robust to deformation and occlusion) / scene (only answers "where am I", produces no coordinates); actions use "target": "recognized" rather than hardcoded coordinates; abnormal branch nodes like popups get a finding field, and the task level gets watchdogs negative assertions—this project is positioned as a QA testing tool, so anomalies must be reported with evidence, not silently bypassed.

Visual Orchestration: PipelineEditor

Tasks can not only be written, but also drawn. pipeline_editor/ (FastAPI + React, shipped with this repo) is a web visual editor for task JSON, editing task/task_definitions/<task-name>.json directly with no separate copy:

• 🎨 Canvas orchestration: drag-and-drop connections to orchestrate the state machine—solid lines = next (with recognition priority labeled on the edge), orange dashed lines = on_timeout; nodes brought in via includes are grayed out, locked, read-only, and automatically stripped on save—never solidified into the main file

• ✅ Truth-value validation + lint: after 0.8 seconds of inactivity, the current graph is automatically sent to the backend for a dry run of task_loader.resolve_task and lint_task—the editor replicates no validation rules; what you see as errors on the canvas is exactly what the engine will report

• 🎯 Screenshot ROI / template capture: click the crosshair next to the roi field, drag a box on a real-device full-resolution screenshot to write back coordinates, and even do an on-the-spot "OCR trial read / template trial match" to see hit boxes and scores; the template field can crop a new template straight from the screenshot and save it to disk

• ▶️ Real-device run highlighting: a background thread runs the engine, WebSocket pushes each step, the canvas highlights the current node + visited trail in real time; stopping goes through the engine's cooperative clean shutdown, with report and evidence chain complete

• 🤝 MCP collaboration: the backend embeds an editing-surface MCP server at /mcp; the moment the agent saves, the user's canvas auto-reloads within ~2 seconds (a conflict banner appears if there are unsaved changes)—human draws, AI edits JSON, writing the same file and passing the same validation

One command from the repo root brings it up (backend :8930 + frontend :5173; open the printed address in a browser):

powershell -File editor.ps1 -Python <python>   # 转发到 pipeline_editor\scripts\dev.ps1，参数语义一致

Frontend dependencies need a one-time install (cd pipeline_editor\frontend; npm install); backend dependencies are already merged into the root requirements.txt. Full usage guide in pipeline_editor/README.md (docs live in pipeline_editor/docs/).

Directory Structure

flowchart TD
    BRAIN["外部智能体（大脑）<br/>Claude Code / Codex"] -->|"MCP（stdio）"| MCP
    USER["用户"] -->|交互式命令| CLI

    subgraph L1["接口层"]
        MCP["mcp_server.py"]
        CLI["main.py + user_interface/"]
    end
    subgraph L2["任务层"]
        TASK["task/<br/>识别门控引擎 · findings · suite · lint · 健康度"]
        AGT["agent/<br/>设备 Agent 池"]
    end
    subgraph L3["感知 / 执行层"]
        PER["perception/<br/>截图 scrcpy · OCR · dump · 模板/特征/YOLO · 场景 · logcat"]
        ACTL["action/<br/>click · swipe · 手势注入"]
    end
    subgraph L4["基础层"]
        CORE["core/<br/>配置 · ADB 设备 · 日志"]
        UTIL["utils/"]
    end

    L1 --> L2
    L2 --> L3
    L3 --> L4

Dependencies only flow top-down: the perception / execution layers must not import the task layer, and the base layer must not import upper layers. The project itself makes zero LLM calls—intelligence always lives on the external agent's side.

autoplayqa/
├── mcp_server.py                 # MCP 入口（FastMCP / stdio）：感知/动作/任务工具全集，装配复用 bootstrap.py
├── main.py                       # CLI 入口：config → 设备 → 感知 → 解析 → 任务引擎 → Agent 池 → CLI，装配复用 bootstrap.py
├── bootstrap.py                  # 双入口共用装配层：load_app（读配置建日志）+ build_runtime（拼感知/任务对象图）
├── .mcp.json.example             # Claude Code 自动发现配置模板（复制为 .mcp.json 改 Python 路径；含 autoplayqa + pipeline-editor 两个 server）
├── config.yaml.example           # 配置模板（缺省走默认值，无需任何凭证即可启动）
├── requirements.txt              # 依赖（框架 + PipelineEditor 后端，一次装齐）
├── pytest.ini                    # 测试范围：一条 pytest 同时跑 tests/ 与 pipeline_editor/tests/
├── editor.ps1                    # PipelineEditor 启动薄包装（转发 pipeline_editor\scripts\dev.ps1）
│
├── core/                         # 基础设施
│   ├── config.py                 #   配置加载（缺 config.yaml 返回空走默认）
│   ├── device_manager.py         #   ADB 设备发现 + 无线连接（connect/disconnect/pair/tcpip，启动自动连）
│   ├── text_resolver.py          #   LLM-free 指令解析：显式坐标正则 → 屏幕定位 → 失败引导走 MCP
│   ├── adb_timeout.py            #   全局 adb 超时：config `adb.timeout_s` 统一设定，卡死的 adb 调用不再无限等待
│   ├── notifier.py               #   run 汇总推送（飞书机器人 / 通用 webhook，一 run 一条，失败只记日志）
│   └── logger.py                 #   日志
│
├── agent/                        # 一设备一 Agent
│   ├── agent_pool.py             #   多设备 Agent 选择与分发
│   └── device_agent.py           #   单设备执行 + verify_steps 逐步像素差分校验
│
├── action/                       # 动作执行
│   ├── action_executor.py        #   动作路由（click/drag/input_text/wait/key/gesture）
│   ├── action_schema.py          #   动作 + 任务 JSON 格式 schema（TASK_SCHEMA_DOC 文档源）
│   └── backends/
│       ├── adb_backend.py        #     adb shell input 后端
│       └── motionevent_backend.py#     无 root 多指 MotionEvent 注入（app_process + dex helper）
│
├── perception/                   # 确定性感知（眼睛）
│   ├── screenshot_capturer.py    #   截图统筹（raw screencap 本地组装，热路径免 PNG 编解码）
│   ├── scrcpy_stream.py          #   默认 scrcpy 帧流后端（H.264 本地解码 ~13ms/帧，失败回退 screencap）
│   ├── ui_dump_matcher.py        #   uiautomator dump 控件匹配（tty 失败回退文件 dump）
│   ├── ocr_engine.py             #   rapidocr 本地 OCR（懒加载）
│   ├── template_matcher.py       #   OpenCV 模板匹配（多尺度 + 掩膜 + 多实例 NMS）
│   ├── feature_matcher.py        #   ORB 特征匹配（抗小改版/缩放/遮挡，需纹理丰富锚点）
│   ├── yolo_detector.py          #   YOLO 目标检测（onnxruntime，可选，无模型自动让位）
│   ├── scene_classifier.py       #   整屏场景分类（内置只有 blank；其余标签由接入方 register_scene_probe 注册）
│   ├── ui_detector.py            #   两级免费定位编排：dump → OCR
│   ├── screen_marker.py          #   Set-of-Marks 标注图（序号徽标，配合 click_index）
│   ├── screen_recorder.py        #   设备端 screenrecord 滚动分段录屏（findings 黑匣子视频）
│   └── logcat_monitor.py         #   轮询式 crash / ANR 检测（FATAL EXCEPTION / Fatal signal / ANR in）
│
├── task/                         # 识别门控任务引擎（核心）
│   ├── task_engine.py            #   状态机：识别→动作→next 轮询；agent 挂起交接 / 续跑 / bug-skip / 弹窗清扫
│   ├── suite_runner.py           #   套件连跑：登录一次连跑多个用例，冷启动只付一次，跑挂按策略重启/重试
│   ├── task_loader.py            #   加载校验（includes 合并、节点引用整体校验、custom 注册校验、suite 校验）
│   ├── recognizers.py            #   识别通道：ui_text / ocr / template / feature / yolo / scene / always / blank_screen
│   ├── findings.py               #   QA 发现一等公民：触发即留证 + 飞行记录仪黑匣子 + 保留策略 + 导出 zip + run 汇总推送
│   ├── sentinel.py               #   空窗期哨兵：搭后台帧监控查白屏 / crash，写成独立 findings run（monitor_sentinel）
│   ├── report_html.py            #   report.json → 自包含离线 report.html（截图/录屏/日志内嵌）
│   ├── replay_cache.py           #   回放锚点缓存（ROI 提速，锚点移位上报 anchor_drift）
│   ├── task_lint.py              #   任务加固体检 W001-W007（save_task / CLI task lint）
│   ├── anchor_health.py          #   跨 run 聚合 node_stats 巡检锚点腐烂（CLI task health）
│   ├── step_numbering.py         #   任务步骤编号（step 字段 / 流程大纲，引擎不读，纯导航）
│   ├── task_editor.py            #   录制会话 → 确定性回放草稿
│   ├── custom_actions/           #   进程内确定性动作：目录内新建 <模块>.py 即自动发现注册；内置 swipe_until / launch_app / gm_command / ensure_checkbox / set_text_field / click_topmost_text
│   ├── task_definitions/         #   任务 JSON（含 common/ 共享节点文件、suites/ 套件 JSON）——接入方资产，默认不入库
│   ├── templates/                #   模板匹配图库（feature 通道共用）——接入方采集，默认不入库
│   └── models/                   #   YOLO 模型库（接入方放入 .onnx 即启用；版本记 models.json，换模型必须同步更新）
│
├── record/                       # getevent 手势录制
│   ├── gesture_recorder.py       #   getevent -lt 流 → tap/长按/滑动/多指分段 + 面板→显示像素校准
│   ├── record_session.py         #   录制会话状态（MCP / CLI 共用的启停与产物落盘）
│   └── frame_stream.py           #   可选无 glow 帧流（复用 scrcpy v3.1 server，缺则回退 screencap）
│
├── injector/                     # 无 root 多指注入 dex helper
│   ├── GameInjector.java         #   可审源码（调隐藏 InputManager.injectInputEvent）
│   └── build.ps1                 #   构建脚本（dex 不入库，按此自建）
│
├── user_interface/               # 本地 CLI
│   ├── cli_handler.py            #   命令分发与交互循环
│   └── command_parser.py         #   命令 / 自然语言解析
│
├── utils/                        # 工具
│   ├── debug_tracer.py           #   调试落盘 outputs/debug/
│   ├── image_annotator.py        #   图片标注
│   └── helpers.py                #   像素差分等通用助手
│
├── pipeline_editor/              # 可视化任务编排器（FastAPI + React）：画布编排 · 真值校验 · 截图取 ROI/模板 · 真机运行高亮 · 内嵌编辑面 MCP（backend/ frontend/ docs/ tests/）
│
├── vendor/                       # 第三方二进制（scrcpy-server-v3.1，版本须与代码常量一致）
├── tests/                        # 单元测试（subprocess 全 mock，免真机）
├── outputs/                      # 运行时产物：截图 / 日志 / debug / findings（自包含证据夹）/ recordings
│
├── training/                     # YOLO 训练流水线（离线工具线，运行时代码不得 import）
│   ├── preannotate.py            #   已有模型预标注新帧，人工只做订正
│   ├── build_increment.py        #   增量数据集拼装（旧集 + 新标注）
│   └── train_and_export.py       #   训练 → 校验 → 导出 onnx → 部署到 task/models/
│
└── docs/                         # 使用手册与图示（`images/` 下是本 README 的插图）

Testing

python -m pytest              # 全量：框架 tests/ + PipelineEditor 后端 pipeline_editor/tests/
python -m pytest tests -q     # 只跑框架
python -m pytest pipeline_editor/tests -q   # 只跑编辑器后端

Neither suite depends on a real device (all subprocess calls are mocked); the combined run scope is defined by the root pytest.ini testpaths.

About

• Positioning: An Android game QA automation testing framework. It provides deterministic device perception (eyes) and operation (hands/feet), and a recognition-gated task engine; judgment and orchestration are delegated to external AI agents—the project itself makes zero LLM calls and spends zero tokens.

• Generic vs. specific: The framework side is game-agnostic generic capability (perception channels / action backends / task engine / findings forensics / MCP and CLI interfaces); game-specific parts are provided by the integrator—task/task_definitions/ tasks and suites, task/templates/ template images, task/models/ YOLO models, and scene labels registered via register_scene_probe are all local / integrator-project assets, not committed by default.

• Platform / tech stack: Windows + Android (ADB, emulator / real device); Python 3.11; rapidocr local OCR, OpenCV template matching / ORB feature matching, YOLO object detection (onnxruntime, optional), rule-based scene classification, uiautomator dump, default scrcpy frame-stream screenshots.

• Integration: MCP server (plug-and-play with Claude Code / Codex) or a local interactive CLI.

• Design orientation: A QA testing tool—anomalies are test discoveries that must be reported with evidence (watchdog assertions / node findings / crash monitoring / flight-recorder evidence), not silently bypassed.

Acknowledgments and Influences

This project is an independent implementation (Python + React, no code reuse). The following are documented influences from the engineering iteration record, acknowledged here:

• MaaFramework: The conceptual source of the task engine—task JSON organizes flow as "recognition confirm → execute action → next candidate polling (first hit wins)", with timeouts going to recovery branches; the engine's first version was built on its Pipeline idea; later node capabilities like combined recognition (all_of/any_of), action-level repeat bursts, and task-level defaults blocks also draw on its pipeline features.

• better-genshin-impact: Three things were implemented against its engineering practices: idle-window sentinel (background frame monitoring continues checking for white screens and crashes while a task finishes / during agent handoffs), findings result push (a run pushes an IM / webhook summary at the end), and model manifest (YOLO model versioning registered in models.json).

• MaaPipelineEditor: The interaction paradigm of this repo's pipeline_editor/ visual orchestrator follows it (canvas connection orchestration, property panel, real-time JSON sync, built-in recognition helper tools); see pipeline_editor/README.md.