The mk-qa-master server is a universal MCP server for AI-driven test execution, analysis, generation, and quality optimization across web, mobile, API, and edge AI frameworks.
Test Execution
Run tests across pytest/Playwright, Jest, Cypress, Go, Maestro (mobile), Schemathesis (API fuzzing), Newman (Postman), and Edge AI (RTSP + YOLO)
Re-run only failed tests, list available tests, auto-retry flaky tests, and filter by name/pattern or browser engine
Output JUnit XML for CI pipelines (GitHub Actions, Jenkins, GitLab)
Analysis & Test Generation
Analyze live web URLs (analyze_url): extract forms, nav, dialogs, CTAs, API endpoints, and candidate test cases
Analyze mobile screens (analyze_screen): dump view hierarchy via Maestro to identify UI modules
Analyze RTSP streams (analyze_stream): probe Edge AI streams for geometry, FPS, and YOLO-detected labels
Generate runnable test skeletons (generate_test) for Playwright, Maestro, Jest, Cypress, or Go — with concrete selectors, not stubs
Auto-generate tests for an entire URL in one shot (auto_generate_tests)
Launch Playwright codegen for interactive click-to-record test creation
Reporting & Failure Analysis
Retrieve per-run pass/fail/skip/duration summaries and per-failure root cause materials (screenshots, video, Playwright trace.zip, step sequences)
Generate self-contained HTML reports with embedded screenshots, sparkline trends, and failure cards
View run history for flake scoring, outcome patterns, and slow regression detection
QA Knowledge & Self-Improvement
Initialize (init_qa_knowledge) and retrieve (get_qa_context) a project-specific qa-knowledge.md with business rules, bug patterns, and testing methodology
Get prioritized optimization plans (get_optimization_plan) covering suite quality, MCP tool usage patterns, and AI generation effectiveness
Security & CAPTCHA
Run OWASP API Security Top 10 scans against OpenAPI specs (BOLA, Broken Auth, Mass Assignment, Function-Level Authz, Security Misconfiguration)
Inspect and solve reCAPTCHA v2 / hCaptcha image-grid challenges via AI vision and Playwright click execution
Planning & Verification
Declare success criteria upfront (qa_plan) and automatically verify them against results/findings (verify_plan)
Pass a plan_id to core tools (run_tests, analyze_url, auto_generate_tests, etc.) to auto-attach plan verification to any response
Cross-Host & Integrations
Functions as a single-file agent skill for Claude Code, Codex CLI, Gemini CLI, and OpenClaw
Composable with other MCP servers (GitHub, Slack, Sentry, Atlassian, Filesystem) for orchestrated QA workflows
Bilingual support: English or Traditional Chinese (QA_LANG=zh-tw)
Allows running Cypress tests, collecting artifacts such as screenshots and videos, and generating reports with failure details.
Produces JUnit XML output compatible with GitHub Actions for test result reporting in CI pipelines.
Produces JUnit XML output compatible with GitLab CI/CD for test result reporting in CI pipelines.
Produces JUnit XML output compatible with Jenkins for test result reporting in CI pipelines.
Allows running Jest tests and retrieving test reports with pass/fail/skip statistics and failure details.
Allows running pytest tests (including pytest-playwright) with failure screenshots, traces, videos, and auto-retry of flaky tests.
Universal MCP server for running tests across pytest / Jest / Cypress / Go, with built-in DOM analyzer, run history, and a self-improvement coach. Stable since v1.0.0 (2026-06-02) — see Stability promise below.
A Model Context Protocol server that lets Claude Desktop / Cursor / any MCP client drive your test suite end-to-end: run tests, inspect failures (screenshot + video + trace), analyze a live URL to draft test cases, and — after each run — produce a prioritized action plan telling you exactly what to fix or write next.
| Framework | Language | Target |
| pytest + Playwright | Python | Web |
| Jest | JavaScript | Web |
| Cypress | JavaScript | Web |
|
| Go | Backend |
| Maestro | YAML | iOS + Android |
| Schemathesis | OpenAPI 3.x / Swagger 2.0 | API (since v0.6.0) |
| Newman | Postman collection v2.x | API (since v0.6.1) |
Full design notes: docs/framework.md.
Run tests across multiple frameworks (web + mobile + API) via a single MCP surface
Mobile via Maestro (since v0.3.0): same MCP tools, iOS Simulator / Android Emulator / real device; YAML flows; cross-platform without rewrites
Native API testing — two runners (since v0.6.0 / v0.6.1): two peers now share the API testing slot, each fed by the artifact your team already maintains.
Schemathesis (QA_RUNNER=schemathesis, since v0.6.0): point at an
OpenAPI 3.x / Swagger 2.0 URL or file:// schema and get property-based
fuzzed tests covering status codes, response schemas, content types,
and 5xx-on-fuzz violations.
Newman (QA_RUNNER=newman, since v0.6.1): point at an exported
Postman 2.x collection (plus optional environment / globals files) and
Newman replays every request, runs the embedded pm.test(...)
assertions, and returns one mk-qa-master nodeid per assertion. Newman
is a system prerequisite (npm install -g newman) — it's an npm
package, not pip, so it doesn't ship as a Python extra.
Both drop into the same MCP tool surface as the web / mobile runners, and
both feed the same report.json / history / flake / optimizer pipeline.
Existing API tests written in pytest+httpx, Jest+supertest, Cypress
cy.request(), or Go net/http/httptest still ride their existing
runners — no migration needed. Pact provider verification stays on the
v0.7.0 conditional roadmap.
Failure artifacts: screenshot (base64-inlined), video, Playwright trace.zip / Maestro recordings
Run history: every run snapshotted; HTML report shows a sparkline trend
DOM / Screen analyzer — analyze_url for web (forms / nav / dialogs /
CTAs + the API endpoints the page hits) and analyze_screen for mobile
(maestro hierarchy → form / cta / tab_bar modules)
Smart test generation (generate_test): hand it an analyzer module
and it writes a runnable Playwright .py or Maestro .yaml with concrete
selectors, not # TODO stubs
Auto-retry flakes — pytest side via pytest-rerunfailures; Maestro
side via custom retry wrapper (no native --reruns); flaky tests
surfaced separately from real failures
Self-improvement coach (get_optimization_plan): post-run analysis
across three lenses — suite quality, MCP usability, AI generation effectiveness
JUnit XML output for CI integrations (GitHub Actions / Jenkins / GitLab)
Related MCP server: wopee-mcp
Two paths — pick the one that matches how you'll use it.
uvx (zero install, recommended for end users)Add mk-qa-master to your client config without installing anything globally; uv fetches and runs it in an ephemeral environment per session:
{
"mcpServers": {
"mk-qa-master": {
"command": "uvx",
"args": ["mk-qa-master"],
"env": { "QA_RUNNER": "pytest", "QA_PROJECT_ROOT": "/path/to/your-test-project" }
}
}
}That's the whole setup. First call downloads the package; subsequent calls are cached. Switching versions: uvx mk-qa-master@0.4.1 ....
pip install mk-qa-master # or: pip install -e . from a clone
playwright install # only if you use pytest-playwright
pip install pytest-rerunfailures # optional, enables auto-retryThen point your client config at the same Python interpreter:
"command": "/path/to/.venv/bin/python",
"args": ["-m", "mk_qa_master.server"]mk-qa-master doctor # human-readable check report
mk-qa-master doctor --json # for CI gates / host-LLM consumptionWalks Python version, ffmpeg + mediamtx on PATH, core deps, [edge] extras, runner registry, and MCP tool surface. Exits 0 when nothing critical is missing (warnings about unused features don't fail), 1 when mk-qa-master can't run cleanly. Run it after a fresh install or when an MCP tool returns missing_extras.
| You also need |
|
|
| A Node project with |
| A Node project with |
| Go toolchain on PATH |
| Maestro CLI + a booted simulator / emulator / device (or BlueStacks reachable via |
|
|
|
|
QA_RUNNER=schemathesis)Point the runner at any OpenAPI 3.x / Swagger 2.0 schema and Schemathesis
generates property-based test cases per operation — covering response
schema conformance, status code conformance, content-type checks, and
5xx-on-fuzz. Results flow through the same report.json / history /
flake / optimizer pipeline as your UI tests.
End-to-end walkthrough lives in docs/walkthrough-api.md;
a self-contained 3-endpoint sample lives at
examples/sample_api_project/.
"env": {
"QA_RUNNER": "schemathesis",
"QA_OPENAPI_URL": "https://api.example.com/openapi.json"
}Variable | Required | Default | What it does |
| yes | — | OpenAPI URL. |
| no |
| Comma-separated subset: |
| no | — | Authorization header value. Sent as |
| no |
| Hypothesis examples per operation. Higher = deeper fuzz, slower run. |
| no |
| Set to |
| no |
| Disables secret redaction in archived reports. Default redacts |
Standard QA_TIMEOUT_SECONDS still applies (default 600s).
QA_RUNNER=newman)Point the runner at any exported Postman 2.x collection and Newman 6.x
replays every request, runs the embedded pm.test(...) assertions, and
returns one mk-qa-master "test" per assertion. Results flow through the
same report.json / history / flake / optimizer pipeline as the
Schemathesis and UI runners.
System prerequisite: Newman ships via npm, not pip. Install once:
npm install -g newmanThere's no pip install 'mk-qa-master[postman]' extra — the runner
just shells out to the newman binary on PATH. If it's missing, the
runner raises a clear ImportError pointing at the npm install line.
The same 3-endpoint Library API that the OpenAPI sample targets
ships as a Postman collection at
examples/sample_api_project/postman-collection.json —
pair it with prism mock examples/sample_api_project/openapi.yaml for
a fully self-contained dev loop, or point at your own staging server.
"env": {
"QA_RUNNER": "newman",
"QA_POSTMAN_COLLECTION": "/absolute/path/to/your-collection.json"
}Variable | Required | Default | What it does |
| yes | — | Plain filesystem path to a Postman 2.x collection JSON. No |
| no | — | Plain path to a Postman environment file ( |
| no | — | Plain path to a Postman globals file ( |
| no |
| Replay the whole collection N times ( |
| no | — | CSV of Postman folder names to restrict the run to (repeated |
| no |
| Per-request HTTP timeout in milliseconds ( |
| no |
| Same redaction policy as the Schemathesis runner — disable only for short debug sessions. |
Standard QA_TIMEOUT_SECONDS still applies (default 600s).
When backend bypass isn't an option: Claude looks at the CAPTCHA, mk-qa-master does the clicks.
Supports reCAPTCHA v2 (since v0.7.0) and hCaptcha (since v0.7.1).
The first capability in the family where the AI client's vision is
load-bearing, not optional. Two new MCP tools
(inspect_visual_challenge + solve_visual_challenge) detect a
reCAPTCHA v2 or hCaptcha image-grid challenge on the active Playwright
page, screenshot it for the multimodal AI client, accept the
tile-selection the AI returns, and execute the click chain. The
runner is the eyes and hands; the AI client (Claude / Cursor / Gemini
/ GPT-4o) is the actual solver.
The built-in QA knowledge layer (get_qa_context section="CAPTCHA")
codifies three tiers. Reach for them in order:
Tier | Approach | When |
1 — bypass | reCAPTCHA test keys, feature flags, IP allowlist, test-mode headers | Default. Covers ~90% of cases. |
2 — degrade | Mark as | When you can't change the backend but the test isn't about the CAPTCHA itself. |
3 — AI visual judgment | This feature. | Only when 1 + 2 don't fit (client sites with authorization but no backend access, staging that mirrors prod CAPTCHA, mobile webviews where IP allowlist isn't reachable). |
The solver does nothing until you explicitly opt in. Two env vars drive it:
Variable | Required | Default | What it does |
| yes |
| Must be set to |
| no (recommended) | — | Comma-separated allowlist of domains where the tool may operate. When SET, refuses any other domain. When UNSET, warn-only — proceeds but stamps the response with a warning telling you to set one. Recommended for shared CI / multi-tenant environments. |
| no |
| Wall-clock budget in seconds for the inspect→solve cycle. Honors |
"env": {
"QA_RUNNER": "pytest",
"QA_PROJECT_ROOT": "/path/to/project",
"QA_VISUAL_CHALLENGE_CONSENT": "true",
"QA_VISUAL_CHALLENGE_AUTHORIZED_DOMAINS": "client-staging.example.com"
}Then, when a run_tests call surfaces an external_dependency
failure that points at a CAPTCHA, the AI client can escalate:
mk-qa-master.inspect_visual_challenge() # screenshot + tile grid
→ AI vision picks tiles [0, 4, 7]
mk-qa-master.solve_visual_challenge(
challenge_id="...", selected_tile_indices=[0, 4, 7], confirm=true,
)
→ status: "passed", token: "...", hint: "CAPTCHA verified. Resume your test."Full walkthrough lives in docs/walkthrough-visual-challenge.md.
PRD: docs/prd-v0.7-visual-challenge.md.
Regardless of consent or allowlist, the solver refuses to operate on
known third-party identity providers (accounts.google.com,
login.microsoftonline.com, id.apple.com, facebook.com,
login.live.com, etc.). No legitimate QA scenario justifies a
CAPTCHA solver against someone else's login portal.
No screenshot retention beyond the active inspect→solve cycle. Telemetry logs the boolean outcome only — never the screenshot, never the challenge text, never the tile selection. The 5-minute LRU cache holds at most 10 outstanding challenges per process and never touches disk.
The AI client's vision model does the actual judging — Claude Sonnet
4, GPT-4o, and Gemini 2.5 all ship with native vision but their
accuracy on a 3x3 reCAPTCHA varies. Plan for at least one retry per
challenge (reCAPTCHA gives you three before locking out). get_telemetry
will eventually surface aggregate pass-rate so you can size that
expectation per-client.
Scope: reCAPTCHA v2 image-grid only in v0.7.0. hCaptcha lands in v0.7.1. reCAPTCHA v3 / Cloudflare Turnstile are permanently out of scope — they don't surface a visible challenge to inspect.
Schemathesis catches correctness drift. v0.8.0 adds the layer that catches the security drift hiding behind a passing schema.
v0.8.0 ships an OWASP API Security Top 10 (2023) rule-based scanner
as a new MCP tool: run_api_security_scan. It loads an OpenAPI 3.x
spec, walks each (path × method), and dispatches five purely-HTTP-
observable rules:
OWASP # | Rule | Severity when triggered |
API1 | BOLA / IDOR — alice's token reads bob's object via path-id tampering | CRITICAL |
API2 | Broken Authentication — server accepts | MEDIUM / HIGH / CRITICAL by probe |
API3 | Mass Assignment — server persists dangerous extra fields like | HIGH |
API5 | Function-Level Authz — non-admin user accesses admin-shaped endpoints | HIGH |
API8 | Security Misconfiguration — missing HSTS/CSP/X-Frame headers, wildcard CORS with credentials | LOW / MEDIUM / HIGH |
API4 (rate limit DoS risk), API6 (business flow modeling), API7
(SSRF callback infra), API9 (prod recon), API10 (upstream APIs) are
deferred — see docs/prd-v0.8-api-security.md §3.
Mirrors the v0.7 visual-challenge consent model:
Variable | Required | What it does |
| yes | Must be |
| yes for external hosts | Comma-separated allowlist. Localhost / 127.0.0.1 are implicitly authorized. |
The mass_assignment rule mutates server state — it's excluded from
default categories. Callers must opt in:
categories=["headers", "broken_auth", "bola", "function_authz", "mass_assignment"].
"env": {
"QA_RUNNER": "pytest",
"QA_PROJECT_ROOT": "/path/to/project",
"QA_API_SECURITY_CONSENT": "true",
"QA_API_SECURITY_AUTHORIZED_DOMAINS": "api.staging.example.com"
}Then ask the AI client to scan:
mk-qa-master.run_api_security_scan(
spec_url="https://api.staging.example.com/openapi.yaml",
auth={
"token": "alice's bearer token",
"alt_user_token": "bob's bearer token",
"bola_test_ids": {"user_a": [101, 103], "user_b": [202]}
},
severity_threshold="medium"
)Returns the v0.8 security report block:
{
"scan_id": "a3f8d1c9b7e2",
"spec_url": "...",
"base_url": "https://api.staging.example.com",
"categories_run": ["headers", "broken_auth", "bola", "function_authz"],
"rules_ran": ["OWASP-API8-Headers", "OWASP-API2-BrokenAuth", ...],
"ops_scanned": 23,
"severity_threshold": "medium",
"findings": [
{
"rule_id": "OWASP-API1-BOLA-CrossUserDataExposure",
"severity": "critical",
"endpoint": "GET /orders/{id}",
"title": "user_a can read user_b's object id=202 — missing object-level authorization check",
"evidence": {"actor": "user_a", "target_owner": "user_b", "target_id": 202, "probed_path": "/orders/202", "status_code": 200, ...},
"remediation_hint": "Compare the caller's identity to the object's owner before returning..."
},
...
],
"summary": {"total": 7, "by_severity": {"critical": 2, "high": 4, "medium": 1, "low": 0, "info": 0}}
}examples/sample_vulnerable_api/ ships a deliberately-vulnerable
Flask app where every in-scope OWASP category has a vuln/safe
endpoint pair. Run it locally to see what each rule looks like in
action:
cd examples/sample_vulnerable_api
pip install -r requirements.txt
python app.py # binds 127.0.0.1:5099
# Then from another shell, point run_api_security_scan at
# http://127.0.0.1:5099 + the bundled openapi.yamlThe scanner finds all 5 categories on /vuln/* and produces zero
false positives on /safe/*. That property is enforced by the
Tier 1 dogfood tests on
every PR.
The scanner runs adversarial test cases. Do not point it at production systems you don't own, and do not point it at any system where you don't have authorization. The two env vars above are the contract.
PRD: docs/prd-v0.8-api-security.md.
The earlier v0.8 mobile attempt was parked — see
docs/v0.8-mobile-postmortem.md for
what we learned and how it shaped the API-security PRD's testing
gates.
Same skill folder loads in four different agent hosts via the agentskills.io convention.
v0.9.0 packages mk-qa-master as a cross-host agent skill in addition
to its MCP-server form. The skills/mk-qa-master/ folder is the single
source of truth — the same SKILL.md, slash commands, and reference
docs load into:
Claude Code — via .claude-plugin/plugin.json (this repo is a
plugin marketplace).
OpenAI Codex — via .codex-plugin/plugin.json (Codex reads Claude-
style marketplaces).
OpenClaw — install from local checkout: openclaw plugins install /path/to/mk-qa-master.
Hermes Agent — symlink the skill folder into ~/.hermes/skills/.
# Inside Claude Code:
/plugin marketplace add kao273183/mk-qa-master
/plugin install mk-qa-master@mk-qa-masterRestart Claude Code so the skill registers. Then any QA testing prompt auto-activates the skill — or explicitly invoke a slash command:
/mk-qa-master:run-tests login
/mk-qa-master:generate https://staging.example.com
/mk-qa-master:api-security https://api.staging.example.com/openapi.yamlThe skill is a single-file operating contract that teaches the host how to drive mk-qa-master's 22 MCP tools coherently. It encodes:
When to auto-activate — phrases like "run my tests", "why did this test fail", "scan this API for OWASP issues" trigger it.
Five flows — run tests / generate tests / debug failures / solve CAPTCHAs / scan APIs.
Hard rules — surface consent errors verbatim, don't silently re-run with relaxed filters, confirm before destructive runs.
Full reference at skills/mk-qa-master/SKILL.md.
The MCP server makes the 22 tools callable by any client. The skill makes them discoverable + governed: it gives the host's skill router enough context to decide when to use the tools and which flow to follow. Inspired by microsoft/Webwright, which uses the same pattern.
22 tools. Frozen schema. Versioned drift. Pin and go.
mk-qa-master shipped v1.0.0 on 2026-06-02. The MCP tool surface is locked: 22 tools, the consent gate env vars, the plan / bookend shapes, and the hard-stop blacklists don't change without a deprecation cycle.
If you pin… | What you get |
| Patch releases only (bugfixes; no surface change) |
| Minor releases (additive only: new tools, new optional args, new fields) |
| Same as above |
Breaking changes require a v2.0 bump. Deprecations get ≥ 1 minor of warning with DeprecationWarning raised at runtime, "Deprecated:" in the MCP tool description, and an entry in docs/MIGRATION-1.x-to-2.0.md (created when v2.0 work opens).
A CI snapshot test (tests/test_v1_schema_snapshot.py) freezes the 22-tool surface in tests/snapshots/v1/tool_surface.json. Any drift fails CI unless the PR sets BREAKING_CHANGE_ACK=true AND both docs/MIGRATION-0.x-to-1.0.md and docs/DEPRECATION-POLICY.md exist. The ack alone isn't a free pass — the docs must be in place.
A second test (tests/test_v1_doc_sync.py) scans every public doc for tool-count claims and fails if any disagree with the live server.
docs/MIGRATION-0.x-to-1.0.md — every additive shape change v0.7 → v1.0 enumerated. TL;DR: v0.10 → v1.0 is a no-op.
docs/DEPRECATION-POLICY.md — formal cycle. patch = bugfix, minor = additive, major = removal (only after deprecation).
docs/prd-v1.0-stability-lock.md — locked PRD.
MIT today. Apache 2.0 in v2.0.
mk-qa-master is announcing that it will relicense from MIT to Apache 2.0 in v2.0.0. This patch (v1.2.1) is the formal announcement and starts the deprecation clock.
If you pin... | What you get |
| MIT forever — every v1.x release stays MIT-licensed |
| MIT for as long as you stay on v1.x |
| Apache 2.0 |
Apache 2.0 grants strictly more rights than MIT (explicit patent grant + trademark protection) while keeping the same commercial-use permission. No scenario reduces your usage rights.
v1.2.1 (this release): announcement only. No code changes.
v1.3.x onwards: still MIT. Hold cycle for at least one minor before v2.0 lands.
v2.0.0 (TBD): actual relicense. Apache 2.0 LICENSE file, NOTICE file, source-header sweep, manifest sync.
Plus a commitment to maintain v1.x bugfix releases for ≥ 6 months after v2.0.0 ships. If your company can't move to Apache 2.0 immediately, you have a runway.
Long-term sustainability — patent peace, trademark protection, contributor IP unambiguity, broader corporate procurement compatibility. See docs/RELICENSING.md for the full rationale + mechanical v2.0 checklist.
RTSP stream + YOLO inference + pytest assertions in a single
QA_RUNNER=edgeflag.
v1.1.0 adds an Edge AI Inference Runner that drops into the same analyze → generate → run loop the web and mobile runners already use. The new analyze_stream MCP tool (tool #22) probes RTSP geometry and emits candidate test cases per detected label.
pip install "mk-qa-master[edge]" # opencv-python + ultralytics + requests
# Plus the binary deps the runner shells out to:
brew install ffmpeg mediamtx # macOS
# or: sudo apt install ffmpeg + download mediamtx from https://github.com/bluenviron/mediamtxThe bundled sample fixture at examples/sample_edge_fixture/ exercises the full loop. Tested against mk-qa-master==1.1.0 (Edge AI), mk-qa-master==1.1.1 (housekeeping), and 1.1.2 (this doc patch).
1. Configure the runner. Three env vars are enough for the desktop path:
export QA_RUNNER=edge
export QA_RTSP_SOURCE="$(pwd)/examples/sample_edge_fixture/factory.mp4"
export QA_MODEL_PATH=yolov8n.pt # ultralytics auto-downloads on first useOptional tuning (defaults in parentheses): QA_MIN_FPS (25), QA_LATENCY_SLA_MS (40), QA_IOU_THRESHOLD (0.5).
2. Ask Claude / Cursor / any MCP host. With mk-qa-master wired as an MCP server (see Wire into Claude Desktop), prompt:
"analyze the stream at
examples/sample_edge_fixture/factory.mp4with the bundled annotations sidecar, then generate detection tests for each label."
Claude calls analyze_stream → gets {width: 320, height: 240, fps: 5, labels: ["forklift", "person"], candidate_tcs: [...]} → calls generate_test per label → writes test_edge_factory_person.py and test_edge_factory_forklift.py to PROJECT_ROOT/tests/.
3. Run. The runner brings up local mediamtx + ffmpeg (the file source loops over RTSP), exports EDGE_* env vars from the QA_* you set, and invokes pytest. Each generated test:
Reads frames via cv2.VideoCapture(EDGE_RTSP_URL)
Pushes each frame through the YOLO backend
Tracks per-frame latency in a LatencyTracker
Asserts the per-label detection appears within the IoU threshold for at least one frame in the ground-truth window
Asserts p95 latency ≤ EDGE_LATENCY_SLA_MS
Asserts sustained throughput ≥ EDGE_MIN_FPS over a 150-frame window
The report lands in PROJECT_ROOT/report.json + junit.xml, gets archived under test-results/history/, and triggers get_optimization_plan like any other runner.
analyze_stream refuses RTSP URLs at known surveillance / IoT camera vendor domains (Dahua, Hikvision, Ezviz, Axis, Amcrest, Lorex, Swann, Reolink) by default. Keeps accidental probing of public camera feeds off the default path. Set QA_EDGE_ALLOW_VENDOR_HOSTS=true to opt in for own-camera testing.
v1.3.0 adds an opt-in network degradation harness for Edge runs. Pass resilience_mode="netem" to generate_test and the emitted pytest uses Linux tc qdisc (via mk_qa_master.edge.resilience.apply_netem) to inject 200 ms latency + 5 % packet loss on the loopback interface, asserts the runner stays within SLA under degradation, then clears the qdisc on teardown.
Double-guarded for safety:
apply_netem raises RuntimeError on non-Linux (macOS / Windows hosts → tests pytest.skip automatically).
Even on Linux it refuses to run until QA_EDGE_NETEM_ENABLED=true — explicit consent for the loopback impact.
The same module also ships three companion helpers: clear_netem (idempotent teardown), kill_ffmpeg_subprocess (process-loss scenario), and build_corrupted_gop_fixture (ffmpeg-driven bitstream noise injection). See src/mk_qa_master/edge/resilience.py and the v1.3.0 PRD for the full menu.
When tests run under resilience mode, the emitted report carries an additive per-test edge_metrics block (frame drops, recovery time, etc.). get_optimization_plan reads it to surface 4 Edge-specific flake signals (corrupted-frame rate, recovery-time skew, drop bursts, sustained latency violations) alongside its usual signal mix.
Phase | What | Status |
1 | Desktop YOLO runner + RTSP source mgmt + metrics | ✅ v1.1.0 |
2 |
| ✅ v1.1.0 |
housekeeping | Sample fixture + | ✅ v1.1.1 |
docs | README walkthrough + troubleshooting (this section) | ✅ v1.1.2 |
3 | Remote inference ( | ✅ v1.2.0 |
4 | Resilience injection + Edge flake signals + degradation scenarios | ✅ v1.3.0 |
Symptom | Likely cause | Fix |
| ffmpeg or mediamtx not on PATH; readiness probe timed out at 10 s | Verify |
| Port 8554 already in use by another mediamtx / RTSP server | Set |
| Base install without |
|
| Default-on blacklist (Dahua / Hikvision / etc.) | If it's your own camera: |
| You set | v1.1 ships LocalYolo only. Unset the remote env vars to fall back to desktop YOLO. Phase 3 follows in v1.2 |
| First-time torch download (~700 MB) | One-time cost. Cache |
Generated test asserts | Synthetic test pattern doesn't contain real persons / forklifts | Expected for the bundled fixture (it's plumbing verification only). Swap in real footage + real annotations for actual detection assertions; see |
p95 latency assertion fires on CPU but passes on GPU | Default | Raise |
ffmpeg complains about | Sample fixture is intentionally low fps (5) to keep the binary at 75KB | Expected. For real testing supply your own higher-fps source |
v1.0.0 → v1.1.0 is additive only — no existing tool changed shape. v1.1.0 → v1.1.1 → v1.1.2 are patch releases (housekeeping + docs). v1.2.0 added Phase 3 (remote inference). v1.3.0 added Phase 4 (resilience injection + Edge flake signals). See docs/MIGRATION-1.x.md for the full change log + the list of new QA_* env vars (QA_EDGE_NETEM_ENABLED, …).
Full PRD: docs/prd-v1.1-edge-ai-runner.md.
Declare success up front, run the work, get a checklist back — on every meaningful tool, not just one.
v0.10.0 generalizes the v0.9.4 bookend pattern (which lived only on
run_api_security_scan) to 5 core tools. Each one accepts an
optional plan_id kwarg returned by qa_plan. When you thread that
plan_id through, the tool's response gains a plan_verification
envelope that auto-verifies the work against the critical points you
declared — no separate verify_plan call needed.
Tool | Evidence shape | Typical CP |
| pytest-json-report's |
|
| Single record: |
|
| One row per discovered module (with |
|
| One row per generated test (success or failure) |
|
| One row per OWASP finding |
|
plan = qa_plan(
task="Smoke the signup flow",
critical_points=[
{"id": "CP1", "verification_hint": "test_happy_path passes"},
{"id": "CP2", "verification_hint": "BOLA-on-orders"},
],
)
result = run_tests(plan_id=plan["plan_id"])
# result["plan_verification"]["status"] == "passed" | "incomplete" | "failed"
# result["plan_verification"]["checklist"] tells you per-CP outcomesBackward compat: omitting plan_id keeps the v0.9.x response shape
intact. See docs/prd-v0.10-universal-bookend.md
for per-tool evidence contracts and the locked decisions.
If you prefer the bare MCP-server wiring (no plugin/skill layer), copy
examples/configs/claude_desktop_config.example.json to:
macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
Windows: %APPDATA%\Claude\claude_desktop_config.json
Two environment variables drive the runtime:
Variable | Example | What it does |
|
| Selects which test framework |
|
| Points at the project under test |
|
| Remote-ADB endpoint for BlueStacks / Genymotion / Nox / cloud Android. When set, the Maestro runner auto-runs |
|
| Hard ceiling on any single subprocess invocation (pytest / jest / cypress / go test / maestro). Returns |
pytest-playwright:
"env": { "QA_RUNNER": "pytest", "QA_PROJECT_ROOT": "/path/to/python-project" }Jest:
"env": { "QA_RUNNER": "jest", "QA_PROJECT_ROOT": "/path/to/node-project" }Cypress:
"env": { "QA_RUNNER": "cypress", "QA_PROJECT_ROOT": "/path/to/cypress-project" }Go test:
"env": { "QA_RUNNER": "go", "QA_PROJECT_ROOT": "/path/to/go-project" }Maestro (mobile, since v0.3.0):
"env": {
"QA_RUNNER": "maestro",
"QA_PROJECT_ROOT": "/path/to/maestro-flows",
"QA_ANDROID_HOST": "127.0.0.1:5555"
}QA_ANDROID_HOST is optional — only set it when targeting BlueStacks / Genymotion / cloud-Android-farm via remote ADB. iOS Simulator / Android Emulator / local USB device auto-discovered.
Schemathesis (API):
"env": {
"QA_RUNNER": "schemathesis",
"QA_OPENAPI_URL": "https://api.example.com/openapi.json"
}Newman (Postman):
"env": {
"QA_RUNNER": "newman",
"QA_POSTMAN_COLLECTION": "/absolute/path/to/collection.json"
}Edge AI (RTSP + YOLO, since v1.1.0):
"env": {
"QA_RUNNER": "edge",
"QA_RTSP_SOURCE": "/absolute/path/to/factory.mp4",
"QA_MODEL_PATH": "yolov8n.pt"
}Requires pip install "mk-qa-master[edge]" + ffmpeg + mediamtx on PATH. See the Edge AI Runner walkthrough for the full env-var table and troubleshooting.
MCP is an open protocol — this server isn't Claude-only. The same Python process talks to any MCP client over JSON-RPC stdio. What differs across clients is (1) the config file format and (2) how reliably the underlying model auto-chains tool calls.
Client | Config | Format | Model | Tool-chain quality |
Claude Desktop / Cursor |
| JSON | Claude Opus / Sonnet | Best tested |
Codex CLI |
| TOML | GPT-5 family | Strong (well-trained on tool chaining) |
Gemini CLI |
| JSON | Gemini 3.1 Pro / Flash | Works; prefers explicit prompts ("first analyze, then write") |
Cline / Continue / Zed | each has its own MCP config slot | varies | varies | depends on configured model |
Example configs ship in the repo:
codex-config.example.toml ·
gemini-config.example.json ·
claude_desktop_config.example.json.
Codex (TOML):
[mcp_servers.mk-qa-master]
command = "/path/to/.venv/bin/python"
args = ["-m", "mk_qa_master.server"]
cwd = "/path/to/mk-qa-master"
[mcp_servers.mk-qa-master.env]
QA_RUNNER = "pytest"
QA_PROJECT_ROOT = "/path/to/your-test-project"Gemini (JSON, same shape as Claude Desktop):
{
"mcpServers": {
"mk-qa-master": {
"command": "/path/to/.venv/bin/python",
"args": ["-m", "mk_qa_master.server"],
"cwd": "/path/to/mk-qa-master",
"env": {
"QA_RUNNER": "pytest",
"QA_PROJECT_ROOT": "/path/to/your-test-project"
}
}
}
}Tool descriptions already nudge the recommended chains
(analyze_url → generate_test, get_qa_context before generating
domain tests). Clients with weaker tool-selection benefit most from
explicit prompts that name the steps.
Shared across all runners (some tools degrade gracefully on non-pytest runners):
Tool | Purpose |
| Which runner is active + all available ones |
| Enumerate tests in the project |
| Run tests (filter / headed / browser; last two pytest-playwright only) |
| Re-run last failures ( |
| Summary (pass / fail / skipped / duration / flaky-in-run) |
| Per-failure message + screenshot / trace / video paths |
| Test skeleton; with |
| One-shot: analyze URL → generate one test per discovered module |
| Launch Playwright codegen (web) / hint to |
| Render the latest run as self-contained HTML |
| Last N archived run summaries (for trend / flake debugging) |
| Web: DOM probe → modules + selectors + candidate TCs + API endpoints + layout overflow warnings |
| Mobile: |
| Scaffold + read the project's QA knowledge layer (methodology + domain). Bilingual since v0.6.2 — methodology ships in English by default ( |
| Three-layer self-improvement coach (suite / MCP / AI strategy) |
| v0.7.0 AI Visual Challenge Solver — detect a reCAPTCHA v2 image-grid challenge, screenshot it, accept the AI client's tile selection, execute the click chain. Gated by |
| v0.8.0 OWASP API Security Top 10 (2023) rule-based scanner — load an OpenAPI 3.x spec, walk path × method, dispatch 5 in-scope rules (API1 BOLA, API2 Broken Auth, API3 Mass Assignment, API5 Function-Level Authz, API8 Misconfig). Gated by |
URI | What |
| Live-rendered HTML report (dark mode, self-contained) |
| Raw pytest-json-report JSON |
| Latest |
After every run, _archive_report() snapshots report.json into
test-results/history/ and writes a fresh optimization-plan.md covering:
Suite quality — outcomes string per test (PFPFP); transitions → flake
score; 3+ identical-signature fails → broken; rerun-passed → flaky-in-run
MCP usability — top tools, error rates, repeat-arg patterns, common A→B chains (from telemetry JSONL logs)
AI strategy — adoption rate of generate_test outputs, coverage gaps
from analyze_url modules with no matching test files
The plan emits prioritized actions (high / medium / low) each with
target + evidence + suggestion + optional auto_action_hint the MCP client
can chain into the next tool call.
mk-qa-master/
├── pyproject.toml
├── src/mk_qa_master/
│ ├── server.py # MCP entry (tool routing + telemetry wrap)
│ ├── config.py # Paths + env vars
│ ├── runners/ # Per-framework plugins
│ │ ├── base.py # TestRunner abstract interface
│ │ ├── pytest_playwright.py
│ │ ├── jest.py
│ │ ├── cypress.py
│ │ └── go_test.py
│ ├── reporters/
│ │ └── html.py # Self-contained HTML render
│ └── tools/ # Thin shims + analyzer + optimizer + telemetry
└── tests_project/ # Example project under testCreate src/mk_qa_master/runners/your_runner.py, subclass TestRunner,
implement the abstract methods
Register the name in runners/__init__.py's REGISTRY
Done
The intended pipeline — from a URL to "what should I improve next time":
flowchart LR
URL[URL] -->|analyze_url| MOD[modules<br/>+ candidate TCs<br/>+ API endpoints]
MOD -->|generate_test<br/>module=...| TEST[tests/test_*.py<br/>runnable skeleton]
TEST -->|run_tests| RES[report.json<br/>+ screenshots<br/>+ trace.zip<br/>+ junit.xml]
RES -->|auto archive| HIST[history/ snapshot]
RES -->|generate_html_report| HTML[HTML report<br/>self-contained]
HIST -->|auto write| PLAN[optimization-plan.md]
PLAN -.->|next session reads| URLThe loop is the point: every run feeds the optimizer, the optimizer points at the weakest link, the next run hits that link first.
In a Claude / Cursor session:
You: 分析
https://shop.example/login，幫我寫對應測試Claude: [
analyze_url] Found 1 form (email_password_form_0) + 3 API endpoints. 5 candidate TCs. [generate_testwith the form module] Wrotetests/test_login.py— runnable with concrete selectors, no# TODOstubs.
You: 跑
Claude: [
run_tests] 23 passed, 0 failed in 31s. Screenshots + step traces captured for every test.
You: 下一步該做什麼？
Claude: [opens
report://optimization] Top:tests/test_login.py::test_invalid_credentialsis flaky (flake_score=0.4, outcomes=PFPFP). Suggestion: addwait_for_response('/api/login')before asserting the error message.
The three optimizer lenses (suite quality / MCP usability / AI generation effectiveness) make every "下一步" answer data-driven, not gut feel.
Same shape, different runner. Requires Maestro CLI installed +
simulator/emulator booted + your app launched (or pass launch_app=true
with app_id):
You: 分析 your mobile app 首頁的條碼按鈕、寫對應測試
Claude: [
analyze_screen(app_id="com.example.app", launch_app=true)] Found 15 interactive elements; matchedbarcode_button(text="條碼", resource_id="barcodeButton") + candidate TCs. [generate_testwith the cta module] Wrotemaestro-flows/test_barcode.yaml—tapOn: { id: barcodeButton }+ waitForAnimationToEnd + takeScreenshot, ready tomaestro test.
You: 跑
Claude: [
run_tests] 5 flows pass, retry didn't fire. Screenshots embedded in HTML report.
You: 上面這個按鈕有時候會 fail、為什麼？
Claude: [
get_optimization_plan]barcode_button::barcode_buttonflagged flaky (flake_score=0.4, outcomes=PFPFP, rerun_count=1). Suggestion: 加waitForAnimationToEnd或extendedWaitUntil等動畫穩定後再 tap。
Mobile-specific notes:
The same qa-knowledge.md (built-in methodology + your domain) feeds
both web and mobile runs — write your business rules once.
analyze_screen filters out iOS status bar (signal / wifi / battery)
and asset-name labels (bg_*, *_filled); the result is signal-heavy.
Maestro's takeScreenshot: <name> directive controls which screens
show up as inline images in the HTML report.
Each row shows a phrase you can paste into a Claude / Cursor session and the underlying MCP tool call it should trigger. Use as a reference for "how do I get the AI to do X without naming the tool myself."
You say | Claude calls |
"Initialize the QA knowledge file." |
|
"Show me the current QA knowledge." |
|
"Open the ISTQB principles section." |
|
You say | Claude calls |
"Run all tests." |
|
"Run only login-related tests." |
|
"Re-run just the failures." |
|
"Show me the summary." |
|
"Which ones failed? Give me screenshots and trace." |
|
"Generate the HTML report." |
|
You say | Claude calls |
"Auto-generate tests for |
|
"Analyze |
|
"Analyze coupon page and write a regression test for our past idempotency bug." |
|
"Just record a checkout flow as a baseline." |
|
Requires QA_RUNNER=maestro, Maestro CLI, and a booted simulator/emulator/device.
You say | Claude calls |
"Analyze the current your mobile app screen and write a test for the barcode button." |
|
"Test the login form on this app." |
|
"Cover the tab bar — write one flow per tab." |
|
"Use Maestro Studio to record a flow." |
|
BlueStacks / remote Android instances: set QA_ANDROID_HOST=127.0.0.1:5555
(or whatever host:port BlueStacks exposes — see Settings → Advanced → Android
Debug Bridge). The Maestro runner will adb connect before each test and
analyze_screen, and bumps the hierarchy timeout to 60s to absorb the
slower TCP-ADB path. Genymotion / Nox / LDPlayer / WSA work the same way;
any host:port that responds to adb connect is fine.
You say | Claude calls |
"What should I fix next?" |
|
"Has |
|
"Why did it fail? Show me the trace." |
|
Mention QA knowledge explicitly — "reference qa knowledge when testing coupon" pushes Claude to call get_qa_context first; saying just "test coupon" may skip it.
State the order — "analyze first, then write" forces analyze_url before generate_test; "just write a test for X" skips analysis.
Batch vs precise — "auto-generate the whole page" → auto_generate_tests; "write one test per candidate_tc" → manual chain.
Failure debugging — Asking "why did it fail / show me the screenshot" reliably triggers get_failure_details (which now returns screenshot + trace + video paths).
❌ "Run it 5 times to see if it's flaky" — the runner has auto-retry + history; just ask "is it flaky" and let get_optimization_plan answer.
❌ "Generate 100 tests" — noise > signal. Use get_optimization_plan first to find what's missing.
❌ "Test all edge cases" — too vague. Phrase as "test every candidate_tc for this form" — concrete, bounded, traceable.
analyze_url (excerpt){
"url": "https://shop.example/login",
"page_title": "Login",
"module_count": 3,
"modules": [
{
"kind": "form",
"name": "email_password_form_0",
"selectors": {
"container": "#login",
"fields": [
{"label": "Email", "selector": "#email", "type": "email", "required": true},
{"label": "Password", "selector": "#password", "type": "password", "required": true}
],
"submit": "button[type='submit']"
},
"candidate_tcs": [
"所有必填欄位為空時送出，應顯示必填錯誤",
"Email 欄位填入格式錯誤的字串（無 @），應顯示格式錯誤",
"Password 欄位輸入後應預設遮蔽（type=password）",
"全部填入合法值後送出，應觸發成功流程"
]
}
],
"api_endpoints": [
{
"method": "POST",
"path": "/api/login",
"status": 401,
"candidate_tcs": [
"POST /api/login payload 缺必填欄位應回 400 + 欄位錯誤訊息",
"POST /api/login 合法 payload 應回 2xx",
"POST /api/login 缺少 auth header 應回 401/403"
]
}
]
}generate_test output (smart, with module)"""
Login happy path
Auto-generated from analyze_url module: email_password_form_0 (kind=form)
"""
from playwright.sync_api import Page, expect
def test_login(page: Page):
page.goto('https://shop.example/login')
page.locator('#email').fill('test@example.com')
page.locator('#password').fill('TestPass123!')
page.locator("button[type='submit']").click()
# TC: Email 欄位填入格式錯誤的字串（無 @），應顯示格式錯誤
# TC: Password 欄位輸入後應預設遮蔽
# TC: 正確 Email + 正確密碼 → 導向 dashboard
# TODO: 補上實際斷言，例如：
# expect(page).to_have_url(...)
# expect(page.get_by_text("成功")).to_be_visible()optimization-plan.md (excerpt)# Optimization Plan — 2026-05-12T14:03:40
_Based on 6 archived runs._
## Prioritized Actions
### 1. 🔴 HIGH — flaky
- **Target**: `tests/test_login.py::test_invalid_credentials`
- **Evidence**: flake_score=0.4, outcomes=PFPFP, rerun_count=1
- **Suggestion**: 加 explicit wait（wait_for_response / locator wait）
### 2. 🟡 MEDIUM — coverage_gap
- **Target**: `register_form`
- **Evidence**: 由 analyze_url 偵測但 repo 內找不到對應 test_*.py
- **Suggestion**: `call generate_test(description="...", filename="test_register_form.py")`Open the live rendered demo →
(served via GitHub Pages — clicking the link in GitHub's UI to
sample_report.html would only show source).
The demo shows the stats grid, trend sparkline, failure cards with embedded screenshots + step lists, and the collapsed Passed section.
mk-qa-master doesn't bundle third-party SDKs — it stays a pure
test-execution + analysis layer. Real QA workflows are composed by
running multiple MCP servers side-by-side in the same client config;
Claude orchestrates the chain across servers. There's no MCP-to-MCP
RPC — each server is independent, the AI client is the conductor.
The pairings below are the ones that complete the loop most often:
Pair with | Why | Example chain |
Atlassian MCP (JIRA + Confluence) | Auto-open bug tickets from failures; sync |
|
Notify channels on failure, share the rendered HTML report, mention oncall for flaky tests |
| |
Read PR description / linked issues for business context before generating tests; post results back as PR comments |
| |
Production errors drive regression priority: top crashes → matching regression tests |
| |
Read a shared |
|
Honorable mention — Google Drive MCP: pairs with Google-Sheet-based TC management (read TCs from a sheet → generate_test → write status back).
All five run as separate processes alongside mk-qa-master:
{
"mcpServers": {
"mk-qa-master": { "command": "python", "args": ["-m", "mk_qa_master.server"], "env": { "QA_RUNNER": "maestro" } },
"atlassian": { "command": "npx", "args": ["-y", "@atlassian/mcp"] },
"slack": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-slack"] },
"github": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-github"] }
}
}Then a single prompt walks the chain:
"Run the checkout suite. For each failure, open a JIRA in project QA with the RIDER format and the screenshot attached. Post the HTML report to #qa-bots when done."
Why this matters: mk-qa-master stays focused on the test loop
(analyze → generate → run → coach). JIRA / Slack / Sentry are entire
domains with their own dedicated servers — bolting them into this one
would dilute the scope, duplicate auth handling, and force every user
to inherit dependencies they may not want.
本 repo 不打包任何第三方 SDK——維持「測試執行 + 分析」單一職責。實務上 QA 工作流是多個 MCP server 並存、由 Claude 編排跨 server 的 tool chain達成的。範例配套：JIRA / Slack / GitHub / Sentry / Filesystem 各自獨立 MCP server，配上 mk-qa-master 拼出完整測試管線。
Releases ship to PyPI via Trusted Publishing — no API tokens stored in the repo. The flow:
Bump version = "x.y.z" in pyproject.toml (via a normal PR — main is branch-protected).
After merge, tag main and push:
git tag -a vX.Y.Z -m "vX.Y.Z — short summary"
git push origin vX.Y.ZCreate a GitHub Release for that tag (gh release create vX.Y.Z ...).
The release event fires .github/workflows/publish.yml → builds sdist + wheel → uploads to PyPI.
One-time PyPI setup (must be done once before the first publish works):
Sign in at https://pypi.org → enable 2FA.
Project page → Settings → Publishing → add a pending publisher with:
Owner: kao273183
Repository: mk-qa-master
Workflow filename: publish.yml
Environment name: pypi
After the first successful run, PyPI auto-promotes the pending publisher to a trusted one and subsequent releases authenticate via OIDC.
The workflow refuses to publish if the release tag doesn't match pyproject.version, which catches "tagged but forgot to bump" mistakes before they hit PyPI.
mk-qa-master is built and maintained solo on nights and weekends. If it saved you time or shaped how your team thinks about AI-driven QA, a coffee keeps the late-night Maestro debugging sessions going:
Your support funds: keeping this repo free + actively maintained, more device variants for Maestro testing (real iPhones / Android tablets / BlueStacks), recorded tutorials for the QA community, and the next 2am bug hunt.
No ads, no sponsorships, no enterprise upsell — just the work.
This repo is maintained solo. Ideas and bug reports are very welcome — please open an Issue or start a Discussion. I read every one and will implement what fits the project's direction.
External pull requests are auto-closed. Not because contributions aren't appreciated, but because keeping the codebase coherent under a single voice matters more here than the throughput a multi-contributor model would bring. If you really want a specific change, an Issue describing the problem gets you further than a PR.
本 repo 由我一人維護。歡迎透過 Issue / Discussion 提想法或回報問題，我會親自評估並實作。外部 PR 會自動關閉——不是不歡迎貢獻，而是想保持程式碼風格與走向一致。
MIT © 2026 Jack Kao — see LICENSE
(中文翻譯參考: LICENSE.zh-TW.md; the English version is
authoritative).
In plain English: you can use this for anything (personal projects, commercial work, modifications, redistribution). The only ask is that you keep the copyright + license notice in any copy you ship. There's no warranty — use at your own risk.
Mobile 版的 analyze_url：透過 maestro hierarchy dump 當前 iOS Simulator / Android Emulator / 實體機 / BlueStacks（透過 QA_ANDROID_HOST）前景 app 的 view tree，再分類成 form（具 hint_text 的輸入欄位）、cta（enabled + 有文字的可點元件）、tab_bar（selected 狀態 + 同 y 對齊的 2+ 個 tab）三種 modules 並附 candidate_tcs。內建 noise filter 自動排除 iOS 狀態列 + asset 命名標籤（bg_* / *_filled / 純數字 / 單一 ASCII 字元等）讓結果信號集中。需 Maestro CLI 已裝、裝置 booted、app 已在前景。若給 app_id + launch_app=true，會先用 launchApp 啟動再 dump。
| Name | Required | Description | Default |
|---|---|---|---|
| app_id | No | 選填，bundle id (iOS) / package name (Android)，格式如 `com.example.app`。搭配 launch_app=true 使用，或為了在輸出標註是分析哪個 app。 | |
| launch_app | No | 搭配 app_id：True 時在 hierarchy dump 前用 maestro launchApp 啟動 app。用 clearState: false（保留 app 狀態），確保看到「真實」起始畫面。省略則假設裝置上 app 已是當前前景。 | |
| timeout_ms | No | 選填，hierarchy 命令超時毫秒。預設 30000；BlueStacks / 遠端 ADB 較慢，QA_ANDROID_HOST 有設時會自動拉到 60000 起跳。 |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
In the absence of annotations, the description thoroughly discloses the tool's behavior: it dumps the hierarchy, classifies modules, applies noise filtering, and requires specific operational conditions. It also explains the launching behavior with clearState: false. This provides good transparency beyond the schema.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single paragraph but packs substantial information efficiently. It is front-loaded with the main purpose and then details. While slightly dense, it avoids unnecessary verbosity and is structured logically.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity and the lack of an output schema, the description is highly complete. It explains what the tool returns (hierarchy, classified modules, candidate_tcs), lists prerequisites, and covers optional behaviors comprehensively. No significant gaps remain.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has 100% coverage with descriptions for all three parameters. The description adds significant context: app_id is used for launching or labeling, launch_app details the exact behavior (clearState: false), and timeout_ms adjusts for BlueStacks. This goes beyond what the schema provides.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly identifies the tool as a mobile version of analyze_url, specifying that it dumps the view tree of the current foreground app on a mobile device, classifies UI elements into forms, CTAs, and tab bars, and filters noise. It differentiates from the sibling analyze_url by explicitly stating 'Mobile 版的 analyze_url'.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear context for when to use the tool, including prerequisites (Maestro CLI installed, device booted, app in foreground) and options to launch the app. It does not explicitly mention when not to use it or list alternatives, but the context is sufficient for selecting the tool.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
v1.1.0 — Edge AI version of analyze_url / analyze_screen. Probes an RTSP stream (or a file path destined for local mediamtx) and returns basic geometry (width / height / fps) plus a candidate_tcs list. When an annotations sidecar (JSON: per-frame expected detections) is supplied, candidate_tcs gets one entry per discovered label PLUS four runner-standard entries (throughput, latency SLA, reconnect, empty-frame). Strings only — same schema parity as analyze_url's candidate_tcs.
Vendor-host blacklist (default-on): refuses RTSP URLs at known surveillance / IoT camera vendor domains (Dahua / Hikvision / etc.) to keep accidental probing of public camera feeds off the default path. Set QA_EDGE_ALLOW_VENDOR_HOSTS=true to opt out for own-camera testing.
Requires the [edge] extras (pip install "mk-qa-master[edge]") — opencv-python is the probe driver. Tool returns {error: missing_extras, hint} when the extras aren't installed.
Returns on success: {url, width, height, fps, labels, candidate_tcs}. Returns on rejection: {error: bad_request | forbidden_vendor_host | missing_extras | stream_unreachable, hint, ...}.
| Name | Required | Description | Default |
|---|---|---|---|
| rtsp_url | Yes | Required. The stream to probe. Either an `rtsp://...` URL or a file path that the EdgeInferenceRunner will serve via mediamtx + ffmpeg at setup time. | |
| annotations_path | No | Optional. JSON sidecar with per-frame expected detections (format: {fps, frames: {frame_idx: [{label, bbox}, ...]}}). When supplied, candidate_tcs lists one entry per discovered label. Missing / malformed files are non-fatal — the tool falls back to label-free candidates. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description fully discloses behavior: vendor host blacklist, opt-out via env var, extra requirement, error returns, non-fatal missing annotations, and return structures for success and rejection.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Description is front-loaded with version and summary, then logically organizes details about return structure, restrictions, requirements, and error returns. Every sentence adds value without redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given two parameters and no output schema, the description comprehensively covers input semantics, return structures for all outcomes (success, error types), and edge cases (missing extras, vendor host blacklist, malformed annotations).
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema already has descriptions, but the description adds significant context: for rtsp_url it explains file path likely served via mediamtx, for annotations_path it details format and fallback behavior. This goes beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: probing an RTSP stream or file path for geometry and candidate_tcs, with explicit comparison to siblings analyze_url and analyze_screen. The verb 'probes' and output listing make it unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Provides use context (Edge AI), requirements (extras), and restrictions (vendor host blacklist with opt-out). It does not explicitly state when not to use compared to siblings, but the comparison to analyze_url and analyze_screen implies alternatives.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Probe a live web page in headless Chromium and return a structured map of testable modules plus the API endpoints the page actually called. The web counterpart of analyze_screen.
Behavior:
page.goto(url) with DOMContentLoaded + 5s networkidle wait
DOM probe extracts five module kinds: form (with fields[] + required flags), nav (link lists), dialog (modal containers), section (labeled regions), cta (action buttons matching action keywords like 登入/送出/ Login/Submit)
Each module gets a candidate_tcs[] — domain-aware test case strings ready to paste into generate_test
Records every fetch/XHR the page issues, dedupes by (method, path), adds endpoint-specific candidate TCs (401, 404, 4xx, payload-too-large…)
Layout overflow scan flags visible elements whose content escapes its container by >2 px horizontal / >10 px vertical (跑版 / text-overflow) Returns: {url, page_title, scanned_at, modules[], api_endpoints[], layout_warnings[]}
When to use:
User wants tests for a specific URL or page
Designing regression coverage from real user-facing behavior
Need backend API coverage hints (api_endpoints[] gives methods + paths)
Investigating layout bugs at the current viewport
Pair with generate_test(module=…) for one runnable test per module
When NOT to use:
Mobile apps (no DOM) → use analyze_screen
Want analysis + immediate test generation → use auto_generate_tests (one-shot version)
Looking for existing tests → use list_tests
Single-page testing prototype → use codegen instead
Edge cases:
URL unreachable / timeout → returns {error: 「打開頁面失敗…」, url}
Page has 0 forms / 0 ctas → modules[] is empty but the call succeeds
Login-walled URL with no auth_cookie → analyzes the login page (less useful) — pass auth_cookie to reach post-login pages
SPA with delayed hydration → bump timeout_ms to 30000+
Plan bookend (v0.10.0): pass plan_id from a prior qa_plan call and the response auto-attaches plan_verification. Each discovered module is passed as an evidence row with its kind field intact (form / nav / cta / dialog / section / tab_bar). CPs author verification_hint against the module kind / name / selector. Source URL is tacked onto each row for scoping context.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | 要分析的網頁 URL，需含 protocol（http:// 或 https://）。 | |
| timeout_ms | No | 選填，page.goto 等待 DOMContentLoaded 的逾時毫秒數。之後額外 wait 5 秒讓 networkidle（XHR 載入）穩定。預設 15000。慢站 / 需要 SSR / 重 JS hydration 的網站可拉到 30000+。 | |
| auth_cookie | No | 選填，預先注入登入 cookie，格式：`name1=value1; name2=value2`（一行 cookie header）。用法：先在瀏覽器 DevTools / Application / Cookies 複製值再貼進來。用於分析需要登入後才看得到的頁面。 | |
| plan_id | No | 選填，v0.10.0+. Plan id returned by qa_plan. When supplied, the response gains a `plan_verification` envelope that checks every critical point against the discovered modules. Each module is passed as evidence with its `kind` field (form / cta / nav / etc.) preserved; CPs target the kind/name/selector to assert module discovery. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description fully discloses behavior: page load strategy (DOMContentLoaded + 5s networkidle), DOM probe extraction, XHR recording, layout overflow scan, and return structure. Edge cases (unreachable URL, no forms, login-walled, SPA) are detailed.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well-structured with clear sections (behavior, usage, edge cases, plan bookend) and front-loaded with purpose. While comprehensive, it is slightly verbose but still efficient for the complexity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Despite no output schema, the description fully explains the return structure (url, page_title, modules, api_endpoints, layout_warnings) and error case. It covers all relevant context for a tool with 4 parameters and no output schema.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so baseline is 3. The description adds context beyond schema: timeout behavior (extra 5s wait), auth_cookie format, and plan_id purpose. This adds meaningful value.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: probing a live web page with headless Chromium to return testable modules and API endpoints. It uses specific verbs and distinguishes itself from sibling tools like analyze_screen.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description includes explicit 'When to use' and 'When NOT to use' sections, listing specific scenarios and alternative tools (e.g., mobile apps → analyze_screen, immediate generation → auto_generate_tests). Edge cases are also covered.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
一鍵交付：在內部依序做 analyze_url → 為每個偵測到的 module 用 candidate_tcs 內容各跑一次 generate_test，把整套 pytest 測試骨架寫進 PROJECT_ROOT/tests/。等同於『analyze_url 後對每個 module 手動跑 N 次 generate_test』的自動化版本，適合「給我一個 URL、其他你看著辦」這種快速覆蓋場景。每條 candidate_tc 變成對應 test 函式的 docstring，run_tests 跑完 HTML 報告會用 docstring 當 case 名稱顯示。回傳產生的檔案路徑列表 + 每個 module 對應幾個 test。預設每個 module 1 條，想要更密的覆蓋拉 tests_per_module。
Plan bookend (v0.10.0): pass plan_id from a prior qa_plan call and the response auto-attaches plan_verification. Each generated test record (or generation failure) becomes an evidence row with kind=generated_test, path, covers_module (form/cta/nav/etc.), module_name, error (None on success), and source url. CPs can assert coverage ("form module produced ≥1 test") or failure-mode invariants ("no module had generation errors").
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | 要分析並批次產測的 URL，需含 protocol（http:// 或 https://）。 | |
| timeout_ms | No | 選填，analyze_url 內部 page.goto 等 DOMContentLoaded 的逾時毫秒。預設 15000，慢站可拉到 30000+。 | |
| auth_cookie | No | 選填，登入後分析所需 cookie，格式：`name1=value1; name2=value2`。從 DevTools / Application / Cookies 抓現成值貼進來。 | |
| tests_per_module | No | 選填，每個 module 從 candidate_tcs 取前 N 條各產一條 test。1-10，預設 1（最少噪音）。想要更密的覆蓋拉 3-5；拉到 10 通常會產 garbage tests，因為 candidate_tcs 後段是泛例。 | |
| plan_id | No | 選填，v0.10.0+. Plan id returned by qa_plan. When supplied, the response gains a `plan_verification` envelope. Each generated test record (success or failure) becomes one evidence row with kind=generated_test, path, covers_module, module_name, error (None on success), and source url. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries full burden. It details the internal sequence (analyze_url, then generate_test per module), side effects (writing files to PROJECT_ROOT/tests/), behavior of tests_per_module (up to 10 may produce garbage), and the plan_id feature attaching evidence rows. It does not cover error handling for analyze_url failures, but the plan bookend section acknowledges generation failures.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is lengthy and contains non-English text (Chinese), which may reduce clarity. However, it is front-loaded with the core action and each subsequent sentence adds useful detail. It could be more concise by separating the plan bookend section more elegantly.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema, the description explains the return value (list of file paths and test counts), and the enhanced response with plan_id. It covers the workflow, parameters, and limitations. It is largely complete for a composite tool with 5 parameters, though the exact return structure without plan_id could be more explicit.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, baseline 3. The description adds value beyond schema for multiple parameters: tests_per_module warns about garbage at high values, auth_cookie provides practical source instructions, and plan_id explains the plan_verification envelope and evidence row structure.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool automates the process of analyzing a URL and generating tests for each detected module, writing pytest skeletons to a directory. It differentiates itself from siblings by being a one-click automation of manual sequential steps, as explicitly noted.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description specifies the tool is suitable for rapid coverage scenarios where the user provides a URL and expects automated test generation. It contrasts with manual execution of analyze_url and generate_test, and mentions the optional plan_id integration for QA plan workflows. However, it does not explicitly state when not to use it or provide alternatives for fine-grained control.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Launch interactive test recording for the active runner. Useful as a baseline-builder before refining with generate_test.
Behavior:
pytest-playwright: spawns playwright codegen -o <output> <url> — a real Chromium window opens, you click / type / navigate, Playwright transcribes every action into runnable pytest code, output is saved to PROJECT_ROOT/ on browser close
Maestro: returns a human-readable hint string pointing at maestro studio (no shell-able codegen exists for it)
jest / cypress / go runners: same Maestro-style fallback hint Returns: a string with the saved path or the manual-record hint.
When to use:
Building a baseline happy-path test interactively (you click, it transcribes)
Site has complex auth / JS state you'd rather not script by hand
Quick prototype before refining with generate_test
User says 「record / 錄製 / use codegen / 紀錄操作」
When NOT to use:
Headless CI / container environments → can't open Chromium
Need structured, AI-driven test generation from analysis → use generate_test or auto_generate_tests instead
One-shot per-module test coverage → use auto_generate_tests
Mobile UI flows → returns a hint anyway, consider analyze_screen + generate_test instead
Edge cases:
output contains .. or is absolute → blocked by security guardrail
Chromium not installed → playwright codegen fails; user sees the playwright install hint in stderr
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | 受測 URL。Playwright codegen 會開瀏覽器 navigate 到此網址、從這頁開始錄製你的互動。 | |
| output | No | 選填，輸出檔名（相對於 PROJECT_ROOT，不可絕對路徑、不可含 `..`）。預設 `recorded_test.py`。 | recorded_test.py |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations exist, so the description carries full burden. It details opening a real Chromium window, transcribing actions, saving output, security guards on output path, Chromium installation failures, and fallback hints for non-Playwright runners.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well-structured with clear headings and bullet points. Every section adds value, though it is somewhat lengthy. The purpose is front-loaded, and the structure aids scanning.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (multiple runners, security concerns, no output schema), the description covers return value, all relevant behaviors, and edge cases comprehensively.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, and the description adds meaningful context beyond parameter names: url is the starting page, output is a relative path with security constraints (no `..`, no absolute).
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a clear verb+resource: 'Launch interactive test recording for the active runner.' It distinguishes from sibling tools like generate_test by framing this as a baseline-builder before refinement.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicit 'When to use' and 'When NOT to use' sections list concrete scenarios and alternative tool names (e.g., generate_test, auto_generate_tests). The 'When NOT to use' section covers headless environments and mobile flows.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
把最近一次 run_tests 的結果渲染成單檔自包含 HTML——base64 內嵌截圖、嵌入式 step list、history sparkline 走勢、折疊的 Passed 區塊、展開的 Failed cards。沒外部 CSS/JS 依賴，可以直接寄信、丟靜態 host、貼到 Slack。預設輸出 PROJECT_ROOT/report.html。實作位於 reporters/html.py，走 sample_report.html 同款設計。
| Name | Required | Description | Default |
|---|---|---|---|
| output | No | 選填，輸出檔名（相對於 QA_PROJECT_ROOT）。預設 `report.html`。 | report.html |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must carry the full burden. It discloses key behaviors: base64 embedding, no external dependencies, default output path, and implementation location. However, it does not mention side effects like file overwriting, potential size issues, or whether it triggers a test run.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single paragraph that front-loads the main purpose, then details features and outputs. It is relatively concise for the amount of information provided, though could be slightly more structured.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (generating a rich HTML report) and no output schema, the description lacks completeness. It does not specify what the tool returns (e.g., success status, file path), when it fails, or how it handles missing test results. The usage context is clear but insufficient.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has 1 parameter with 100% coverage (its description is provided). The tool description adds minimal extra meaning beyond the schema, just reiterating the default output path. No additional constraints or usage details are given.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states that the tool renders results from the last run_tests into a self-contained HTML report, listing specific features. It distinguishes itself from siblings by its self-contained nature, but does not explicitly compare to get_test_report.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage after running tests ('最近一次 run_tests 的結果'), but does not explicitly state when to use this tool versus alternatives like get_test_report, nor does it mention prerequisites or when not to use.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
產生 pytest-playwright 測試骨架。推薦流程：先呼叫 analyze_url 拿 candidate_tcs，再對每條想覆蓋的 TC 呼叫一次 generate_test、把該 candidate_tc 整段字串當 description 傳入 — 這段會自動寫成 test 函式的 docstring，HTML 報告會把它當作 case 名稱顯示。若提供 url+module（來自 analyze_url 的 modules[]），會用 selectors 預填可執行版本。若想一次處理整個 URL、不想自己編排，請改用 auto_generate_tests。
| Name | Required | Description | Default |
|---|---|---|---|
| description | Yes | test 的描述文字。會直接寫成產出 test 函式的 docstring（pytest）或 YAML 開頭註解（Maestro），HTML 報告會用這段當 case 名稱顯示。建議直接傳 analyze_url / analyze_screen 回來的某個 candidate_tc 整段字串。 | |
| filename | Yes | 輸出檔名，相對於 PROJECT_ROOT。pytest 用 .py、Maestro 用 .yaml、Jest 用 .test.js、Cypress 用 .cy.js、Go 用 _test.go。不可絕對路徑、不可含 `..`（會被 security guardrail 擋）。 | |
| url | No | 選填，受測 URL；提供後 page.goto 會預填 | |
| module | No | 選填，analyze_url 結果 modules[] 中的一個項目；提供後會用 selectors 預填 | |
| business_context | No | 選填，業務規則 / 歷史 Bug / 標準斷言文字 等領域知識。提供後會以 `# Business context:` 註解區塊印進 test 函式內，讓人類 reviewer 與後續 AI 都能看到設計依據。建議先 call get_qa_context() 拿到相關 section 再傳入。 |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Discloses that description becomes docstring and HTML case name, url+module prefill selectors, filename restrictions (no absolute path or '..'), and business_context usage. No annotations provided, so description fully shoulders the burden.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with purpose and workflow, then details. Slightly lengthy but all sentences add value; no redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Covers purpose, usage, parameter details, and behavioral notes for a 5-parameter tool without output schema. Could explicitly state that it creates a file, but implied.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, but description adds valuable usage context for description (from candidate_tc) and business_context (call get_qa_context), going beyond schema descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states it generates pytest-playwright test skeletons and recommends a workflow with analyze_url, clearly distinguishing it from auto_generate_tests for whole URLs.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly describes when to use (after analyze_url, per candidate TC) and when not (for whole URL, use auto_generate_tests), with a clear workflow recommendation.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Extract full root-cause-analysis materials for every failed test in the most recent run.
Behavior:
Reads report.json, filters tests where outcome == 「failed」
pytest: parses Playwright trace.zip → extracts real API call sequence (Frame., Page., Locator., ElementHandle. events) as steps[]
Maestro: parses flow YAML for takeScreenshot: directives → resolves .png at PROJECT_ROOT root
Best-effort resolves screenshot / trace.zip / video / recording paths from --output / --debug-output artifact directories Returns: list[{nodeid, title, message, duration, steps[], screenshot, trace, video}]
When to use:
run_tests just reported failed > 0 → drill into each case
User asks 「why did it fail / show me the trace / what broke」
Filing a JIRA bug → use the artifact paths to attach screenshot+trace
Comparing failure signatures across runs (pair with get_test_history)
When NOT to use:
Want the summary count only → use get_test_report (lighter)
No tests have been run yet → returns [{error: 「找不到報告」}]
Want details for PASSING tests too → not supported here; the HTML reporter renders those via a different path
Edge cases:
test_id substring matches nothing → empty list, no error
screenshot/trace/video missing on disk → those fields are null but the entry stays
Retry-recovered flake (was failed, now passed) → not listed here; surfaces in summary.flaky_in_run instead
| Name | Required | Description | Default |
|---|---|---|---|
| test_id | No | 選填，僅回傳 nodeid 含此關鍵字的 case（substring match，不分大小寫）。省略則回傳全部失敗 case。常用模式：先全部抓→看到特定模式後再用 test_id 收斂。 |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, so description carries full burden. It discloses reading report.json, filtering by outcome, parsing behavior for pytest/Maestro, artifact resolution, error handling for missing files, and edge cases. Very transparent about behavior and limitations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well-structured with clear sections (Behavior, Use cases, Non-use cases, Edge cases) and front-loaded. However, it is somewhat lengthy and could be slightly more concise, though every sentence adds value.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the complexity (multi-framework, multiple artifact types), no annotations, and no output schema, the description covers behavior, parameter details, return format, and edge cases comprehensively. It is complete for an agent to understand and use the tool effectively.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Only one parameter (test_id) with 100% schema description coverage. The description adds meaning: 'optional, substring match, case-insensitive, omit for all failures, common usage pattern', which goes beyond the schema and helps the agent use it correctly.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool extracts full root-cause-analysis materials for every failed test in the most recent run. It specifies the verb 'extract' and the resource 'full root-cause-analysis materials for failed tests', and differentiates from siblings like get_test_report and get_test_history.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly provides when to use (after run_tests with failures, user asking about failures, filing bugs, comparing across runs) and when NOT to use (want summary only, no tests run, want passing tests). Also covers edge cases, giving comprehensive guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
綜合 history/ 快照、telemetry tool-usage、analyze_url 偵測過的 modules，產出三層自我強化分析：(1) 測試套件品質：每條 test 算 outcomes 字串（PFPFP 那種）→ flake_score、再對失敗 error signature 做指紋比對，連 3 次相同 signature 升級為 broken，duration 退化超 1.5x 標記 slow_regression，否則 stable_passing；(2) MCP 使用模式：top tool、重複 args、錯誤率、常見呼叫鏈（A→B 共現）；(3) AI 產測效益：generate_test 寫的 test 有沒有出現在下一次 run、analyze_url 偵測到的 module 對不對得到 test 檔（採用率 vs 覆蓋缺口）。回傳結構化 JSON 並同步寫進 PROJECT_ROOT/optimization-plan.md。每次 run_tests 結束會自動 trigger 一次、所以這個 tool 用來「即時讀」結果。
| Name | Required | Description | Default |
|---|---|---|---|
| history_limit | No | 選填，套件品質分析會看最近 N 次 history 快照。1-100，預設 10。flake score 至少要 5 次以上才穩，深度分析建議 30+。 | |
| telemetry_limit | No | 選填，MCP 使用模式分析會看 telemetry 最近 N 筆 tool-call。10-5000，預設 500。長期使用模式分析拉到 2000+，近期問題排查 100-200 就夠。 |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description fully carries the burden. It details the analysis logic (flake scoring, broken test detection, pattern mining) and discloses that results are written to a file (optimization-plan.md), which is a write operation with potential persistence effects.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single dense paragraph covering complex logic. While each sentence is necessary, it could be better structured (e.g., bullet points) to improve readability for an AI agent.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description thoroughly explains the three analysis layers and their logic. However, it only states that the tool returns 'structured JSON' without detailing the return format, and it lacks information about error handling or prerequisites, leaving minor gaps.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% with parameter descriptions already explaining history_limit and telemetry_limit. The description does not add extra meaning beyond what the schema provides, so the baseline score of 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool produces a three-layer analysis (test suite quality, MCP usage patterns, AI test generation effectiveness) and writes results to a file. This distinguishes it from sibling tools like get_test_history or analyze_url, which have narrower scopes.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explains that the tool is automatically triggered after run_tests, so it is used to 'read results in real-time.' This provides clear context, though it does not explicitly list when to avoid using it or name alternatives.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
讀取受測專案的 qa-knowledge.md（業務規則 / 歷史 Bug / 標準斷言文字 / User Journeys 等領域知識），用 ## H2 區段拆分。用法：先 call 拿到整份或指定 section，再把相關段落以 business_context 傳給 generate_test，產出的 test 就會自帶業務知識註解 — 跳脫 monkey testing。若檔案不存在會 fallback 到內建的 ISTQB 七大原則 + 等價分割 + 邊界值 + 決策表 + 狀態轉換 + Mobile checklist 通用知識，先用著也可以；之後跑 init_qa_knowledge 建立專案專屬版本。
| Name | Required | Description | Default |
|---|---|---|---|
| section | No | 選填，只取單一 H2 section（不區分大小寫、支援部分匹配）。省略則回整份檔 + 所有 section 名稱清單。 |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description fully covers behavioral traits: fallback to built-in knowledge, case-insensitive partial section matching, and the output structure (whole file with section list when no section specified). No contradictions.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is reasonably concise in Chinese (4-5 sentences), front-loading the core purpose and usage. While every sentence adds value, it could be slightly more compact without losing meaning.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given one optional parameter, no output schema, and no annotations, the description is fully adequate. It explains fallback, usage, and output, making the tool's behavior predictable for an AI agent.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The single optional parameter 'section' has 100% schema description coverage. The description adds significant value by explaining matching behavior (case-insensitive, partial match) and the effect of omission vs. specification, which goes beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool reads a project-specific qa-knowledge.md file, which distinguishes it from sibling tools like init_qa_knowledge (which creates the file). The verb '讀取' (read) and specific resource are well-defined.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly outlines the usage pattern: call this tool first to get knowledge, then pass sections to generate_test. It also notes fallback behavior when the file doesn’t exist. While it doesn’t explicitly state when not to use it or list alternatives, the guidance is clear for the intended workflow.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
回傳目前由 QA_RUNNER 環境變數選定的測試 runner（pytest / jest / cypress / go / maestro 五選一）加上 server 編譯時內建的全部 runner 清單。建議每個 session 第一個呼叫——AI 用它判斷後續該產 Playwright .py 還是 Maestro .yaml、要不要 headed browser，避免後面拿錯模板。也用來確認專案環境設定正確：QA_PROJECT_ROOT 指對地方、QA_RUNNER 沒拼錯。回傳 shape：{active: 'pytest', available: ['pytest', 'jest', ...]}。
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description discloses that the tool reads an environment variable and returns configuration data, implying a read-only operation with no side effects. Without annotations, it provides sufficient behavioral context.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single paragraph that covers all necessary points: purpose, usage guidance, and return format. It is clear and informative, though slightly verbose.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no parameters and no output schema, the description provides a complete overview including return shape and context. It could mention error cases (e.g., missing env variable) but is otherwise thorough.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has no parameters, so schema description coverage is 100%. The description does not add parameter meaning beyond the schema, which is expected. Baseline score of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool returns the active runner from the QA_RUNNER environment variable along with the list of all built-in runners. It distinguishes itself from sibling tools by focusing on runner configuration info.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly recommends calling this tool first in each session and explains how the returned information is used to decide which test framework to generate, whether to use headed browser, and to verify environment setup.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
遍歷 test-results/history/*.json 快照（每次 run_tests 完會自動歸檔），回傳逐次摘要：timestamp / total / passed / failed / skipped / duration / pass_rate(0-100)。用於 flake 分析（『這條測試上週一直 fail 嗎』）、速度退化分析（『duration 是不是越來越長』）、覆蓋趨勢圖。預設回最近 10 次，limit 可調 1-100。想要可執行行動建議的話接 get_optimization_plan，它已綜合 history + telemetry。
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | 選填，回最近 N 次 run 的摘要。1-100，預設 10。長期 flake 分析建議 30+。 |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description discloses that the tool reads snapshot files (non-destructive) and returns summary data. While no annotations are provided, the behavioral traits are adequately covered, though it does not mention any potential performance implications or error handling.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise enough, covering the core functionality in a few sentences with key information front-loaded. It could be slightly more compact, but it is not verbose.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description fully explains the tool's purpose, return fields, use cases, and parameter. Given the simple input schema and no output schema, the description provides sufficient context for correct usage.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% and the parameter 'limit' is well-documented in the schema with default, range, and use-case recommendation. The description adds minimal additional meaning, so baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool traverses test history snapshots and returns per-run summaries with specific fields (timestamp, total, passed, etc.). It also lists concrete use cases (flake analysis, speed degradation, coverage trends), making the purpose unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly states when to use the tool (e.g., flake analysis, speed degradation) and directs users to a sibling (get_optimization_plan) for actionable suggestions, providing clear guidance on tool selection.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
讀上一次 run_tests 留下的 report.json，回傳一個輕量摘要：total / passed / failed / skipped / flaky_in_run（auto-retry 救回的數量）/ duration（秒）。比再跑一次 suite 便宜得多——適合在連續操作中間反覆查狀態。未跑過時回 {error: 找不到報告，請先執行 run_tests}。拿到摘要後若 failed > 0，接 get_failure_details 拿錯誤細節。
v1.3.0+: Edge AI runner attaches an optional edge_metrics block to each test entry ({p95_latency_ms, fps, iou_per_frame, labels_covered}). get_optimization_plan reads these to surface 4 Edge-specific flake signals (latency_p95_exceeded_sla, fps_variance_across_runs, iou_jitter_per_tc, coverage_gap_per_label) alongside the standard flake/broken/slow_regression categories. Non-edge runs have no edge_metrics field and see no signal changes.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It discloses that it reads a file from a previous run, is non-destructive, and returns a summary. It also explains the conditional edge_metrics block. However, it does not explicitly state idempotency or potential side effects (though none likely).
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is detailed and contains important usage guidance. However, it includes version-specific details that could be separated (v1.3.0+ edge AI runner info). It is front-loaded with the core functionality.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema, the description thoroughly explains the return shape (fields total, passed, failed, skipped, flaky_in_run, duration) and the error case. It also covers the edge metrics block for Edge AI runner. The tool has no parameters, so this is sufficient for complete usage.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
There are zero parameters, so baseline 4 applies. The description correctly adds no parameter information since none exist.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states it reads the previous test report and returns a lightweight summary, specifying exact fields (total, passed, failed, skipped, flaky_in_run, duration). It distinguishes itself from run_tests (cheaper) and get_failure_details (for detailed failure info).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly states when to use (between continuous operations, checking status cheaply) and when not (if no previous run, returns error). It also provides alternative actions: if failed > 0, use get_failure_details. Mentions edge AI runner specifics and leads to get_optimization_plan.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
在受測專案根 (PROJECT_ROOT) 建立 qa-knowledge.md 起手範本，含業務規則 / 歷史 Bug / 標準斷言文字 / User Journeys / 技術約束 5 個 H2 區段，每段都有 TODO 提示。Idempotent：檔已存在不會覆蓋（除非 overwrite=true）。新用戶建議第一次跑 MCP 就先 call 一次。這份檔案後續會被 get_qa_context 讀、做為 business_context 傳進 generate_test，讓 AI 寫出有業務邏輯的測試（而不是泛例 monkey testing）。
| Name | Required | Description | Default |
|---|---|---|---|
| overwrite | No | 強制覆蓋既存檔案（會丟失你已填的內容、請先備份） |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, description discloses idempotency, overwrite behavior, and that overwrite=true causes content loss. Adequately informs about file creation and side effects.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Description is concise, front-loaded with purpose, and covers all essential aspects. Slightly dense but efficient.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given simple parameter and no output schema, description is complete: explains what file is created, sections included, idempotency, recommended usage, and integration with other tools.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% (overwrite param has description). The description adds value by explaining the backup recommendation and reinforcing the overwrite behavior beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool creates a starter template 'qa-knowledge.md' with specific sections (Business Rules, Historical Bugs, etc.). It distinguishes from siblings by focusing on initialization of a knowledge base, not analysis or test generation.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Provides guidance on idempotency, recommends calling on first run, and explains downstream use (read by get_qa_context, used in generate_test). Lacks explicit when-not-to-use, but context is clear.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Detect a reCAPTCHA v2 image-grid challenge on the active page, screenshot it, and return tile metadata. The AI client (Claude / Cursor / Gemini — multimodal) uses its own vision to identify which tiles to click, then calls solve_visual_challenge with the selected indices. Requires QA_VISUAL_CHALLENGE_CONSENT=true at the server level; without it, returns a structured consent_required error carrying the full legal disclaimer.
Returns: {challenge_id, screenshot_base64, challenge_text, grid_layout ('3x3'|'4x4'), tile_count, tiles[{index, viewport_x, viewport_y, w, h}], expires_at, fingerprint}.
Error shapes: consent_required / unauthorized_domain / forbidden_domain / no_challenge_present / no_active_page / detection_failed — same {error, retryable, hint} envelope as every other runner. Scope: reCAPTCHA v2 image-grid only in v0.7.0 (hCaptcha → v0.7.1; v3 / Turnstile permanently out of scope). Pair with solve_visual_challenge — this tool alone never clicks anything.
| Name | Required | Description | Default |
|---|---|---|---|
| page_id | No | Reserved for future multi-page sessions; ignored in v0.7.0 (the tool operates on the active Playwright page handed in by the runner). | |
| selector | No | Optional override for the iframe selector. Default auto-detection tries `iframe[title*="recaptcha challenge"]` (English UI) then `iframe[src*="recaptcha/api2/bframe"]` (URL pattern, locale-agnostic). |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries full burden. It discloses that the tool requires consent, only operates on the active page, and never clicks. It details return structure, error shapes, and scope (v0.7.0 reCAPTCHA v2 only). This is thorough and transparent.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is moderately long but well-structured. It front-loads the main purpose, then provides details on consent, return, errors, and scope. Every sentence adds value, though it could be slightly more compact.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the lack of output schema, the description compensates by detailing the return structure and error shapes. It covers consent, scope, and pairing. Minor omission: not fully describing challenge_text or fingerprint format, but sufficient for the tool's task.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% and parameters are well-documented. The description adds context: page_id is ignored in current version, and selector has default auto-detection logic. This provides meaningful guidance beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool detects a reCAPTCHA v2 image-grid challenge, screenshots it, and returns tile metadata. It distinguishes itself from the sibling solve_visual_challenge by noting that this tool never clicks anything, providing a clear purpose.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explains when to use the tool (detecting challenges), mentions the need for consent, and pairs it with solve_visual_challenge. However, it does not explicitly state when not to use it (e.g., if no challenge present), though error shapes hint at that. It lacks explicit alternatives but context is clear.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
用 runner 的原生 collection 機制列出受測專案內所有可執行測試：pytest 走 pytest --collect-only、Jest 走 npx jest --listTests、Cypress 走 cypress/e2e/*.cy.* glob、Go 走 go test -list .*、Maestro 走 *.yaml 遞迴掃。回傳一份逐行 nodeid / 檔名清單。用法：run_tests 前確認 collection 沒漏、generate_test 前避免跟既有 case 重複。
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description fully carries the burden. It details the underlying commands per framework (e.g., pytest --collect-only) and the return format (list of nodeid/filename). It could mention that the tool is read-only and safe to run, but overall it is transparent.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise and well-structured: first sentence lists mechanisms for each framework, second sentence describes the return format, and third provides usage guidance. No extraneous information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema, the description explains the return format (list of nodeid/filename). It is complete enough for a listing tool, though it could mention potential performance impacts for large projects. Overall, it provides sufficient context.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
There are zero parameters, and schema coverage is 100% (trivially). The description adds no parameter-level details because none are needed, but it does explain the tool's behavior without relying on params. Baseline 4 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states it lists all executable tests using native collection mechanisms for multiple frameworks (pytest, Jest, Cypress, Go, Maestro). It distinguishes itself from sibling tools like run_tests (execution) and generate_test (creation) by specifying its purpose.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides explicit usage scenarios: before run_tests to confirm coverage and before generate_test to avoid duplicates. It gives clear context but does not explicitly mention when not to use the tool.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
v0.9.1 — Store a critical-points checklist before acting on a QA task. The host LLM declares what success looks like (test passes, scan finds X, screenshot shows Y), this tool stores it, returns a plan_id. Later, call verify_plan with evidence (test result rows, scan findings, log lines, screenshot paths) and get a per-CP pass/fail verdict. Inspired by microsoft/Webwright's plan.md pattern: declaring success criteria up-front makes the verifier honest about whether the work was done.
Plans live 30 minutes (cache TTL) in memory and are LRU-bounded at 50 outstanding.
v0.9.3 — disk persistence: when QA_PROJECT_ROOT is set (or QA_PLAN_PERSIST=true), the plan is also dumped atomically to /test-results/plans/.json. verify_plan transparently falls back to disk on in-memory misses, so plans survive process restarts and cache eviction. Expiry is still honored on disk reads — a TTL'd plan won't silently reload. Persistence is best-effort: filesystem errors never raise into the caller.
Returns: {plan_id (12 hex chars), task, kind, critical_points [{id, description, verification_hint}], created_at, expires_at, persisted_to (filesystem path or null when persistence is off)}.
Error shapes: no_task / no_critical_points / bad_critical_points (duplicate id, missing description, wrong type) / bad_kind.
| Name | Required | Description | Default |
|---|---|---|---|
| task | Yes | Required. The natural-language goal — what the user wants done. Will be echoed back in verify_plan's output. | |
| critical_points | Yes | Required, non-empty. Each entry is either a string (used as description+verification_hint) or a dict {id?, description, verification_hint?}. IDs auto-assigned as CP1..CPn if omitted. verification_hint defaults to description — pick a substring that will literally appear in the evidence you'll later pass. | |
| kind | No | Optional. Hint for downstream verifiers about which evidence stream to expect. Omit if unsure. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description fully discloses behavioral traits: cache TTL of 30 minutes, LRU bound of 50 plans, disk persistence behavior, error shapes, and version history. This provides comprehensive transparency beyond what structured fields could offer.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is detailed but well-structured, starting with the primary purpose, then usage pattern, followed by technical details. It could be slightly more concise, but every sentence adds value.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description fully covers all aspects: purpose, usage, parameters, return value, error shapes, persistence, and expiry. It also links to the sibling tool verify_plan, making it complete for a tool of this complexity.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, but the description adds extra meaning: auto-assignment of IDs for critical_points, defaulting verification_hint to description, and the purpose of the 'kind' enum. This enhances understanding beyond the schema alone.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: 'Store a critical-points checklist before acting on a QA task.' It specifies the return of a plan_id and outlines the complementary workflow with verify_plan, distinguishing it from sibling tools like verify_plan.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explains when to use the tool (before acting on a QA task) and how it pairs with verify_plan. It does not explicitly list exclusions or alternatives, but the context is clear enough for an agent to understand proper usage.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
v0.8.0: OWASP API Security Top 10 (2023) rule-based scanner. Loads an OpenAPI 3.x spec, walks each path × method, and dispatches v0.8's 5 in-scope rules — BOLA (API1), Broken Authentication (API2), Mass Assignment (API3, opt-in), Function-Level Authz (API5), Security Misconfiguration (API8). Returns a v0.8 security report block with per-finding rule_id, severity (critical/high/medium/low/info), endpoint, evidence dict, and remediation_hint.
Requires QA_API_SECURITY_CONSENT=true at the server level. Non-localhost hosts must be in QA_API_SECURITY_AUTHORIZED_DOMAINS (comma-separated). mass_assignment mutates server state — opt in by passing it in categories. Tier 1 fixture (examples/sample_vulnerable_api/) ships with the package for self-tests.
v0.9.4 — Pass plan_id (from qa_plan) to auto-verify the scan's findings against the plan's critical points in the same call. The response gains a plan_verification block (per-CP checklist + overall status). One-shot equivalent of qa_plan → run_api_security_scan → verify_plan.
Returns: {scan_id, spec_url, base_url, categories_run, rules_ran, ops_scanned, severity_threshold, findings[...], summary{total, by_severity}, findings_below_threshold_count, plan_verification (only when plan_id given)}.
Error shapes: consent_required / unauthorized_domain / spec_load_failed / no_base_url / unknown_categories / bad_severity_threshold.
| Name | Required | Description | Default |
|---|---|---|---|
| spec_url | Yes | OpenAPI 3.x URL (http:// or https://) or local path (file:// or bare). YAML and JSON both accepted. | |
| auth | No | Auth config. `token` enables single-user rules (headers + broken_auth). Add `alt_user_token` to enable two-user rules (bola + function_authz). For BOLA: also provide `bola_test_ids: {user_a: [...], user_b: [...]}` listing the ids of objects each user owns. | |
| categories | No | Rules to run. Default: headers + broken_auth + bola + function_authz (mass_assignment excluded — it mutates server state, opt in explicitly). | |
| severity_threshold | No | Minimum severity to include in `findings`. Lower-severity findings counted in `findings_below_threshold_count`. | medium |
| base_url | No | Override spec's `servers[0].url`. Use when the spec is hosted separately from the API. | |
| timeout_s | No | Per-request timeout. Default 30s. | |
| plan_id | No | v0.9.4 — Optional. plan_id returned by qa_plan. When supplied, the scan auto-verifies its findings against the plan's critical points and adds a `plan_verification` block to the response (per-CP checklist + overall passed/incomplete/failed status). Only findings ABOVE severity_threshold are seen by the verifier — if a CP targets a low-severity finding, lower the threshold to 'low' or 'info' accordingly. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description fully covers behavioral traits: it discloses that mass_assignment mutates server state, describes error shapes, explains severity_threshold behavior, and details return structure including plan_verification.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is front-loaded with version and core purpose. It is fairly long but each sentence adds value. Minor improvements could be made to structure with bullet points, but overall it is appropriately sized.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (7 parameters, no output schema), the description provides comprehensive context: return shape, error shapes, authentication details, plan verification, version history. It leaves no critical gaps.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% so baseline is 3, but the description adds significant meaning beyond the schema, such as explaining how auth config enables different rules, default categories, and the effect of plan_id. This extra context justifies a score of 4.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool is an OWASP API Security Top 10 rule-based scanner that loads an OpenAPI spec and runs specific rules. It distinguishes itself from sibling tools like run_tests and qa_plan by focusing on security scanning.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides context for when to use the tool and prerequisites (consent, authorized domains). It explains opt-in for mass_assignment. However, it does not explicitly state when not to use it versus alternatives like qa_plan or verify_plan.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
只重跑上次失敗的測試——比跑整套套件快很多，適合修完一個 bug 後驗證迭代。pytest 走 --lf（last-failed）、Jest 走 --onlyFailures、Cypress 解析上次 report.json 的 failures[] 反查 spec 重跑、Go 撈失敗的 Test 名組成 regex 餵 -run、Maestro 反查 nodeid 對應 .yaml 重跑。需要先有過一次 run_tests（不然 report.json 不存在）。回傳 shape 跟 run_tests 一樣，接 get_test_report / get_failure_details 同樣方式檢視。
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Despite no annotations, the description fully discloses the tool's behavior: it runs only last-failed tests, explains framework-specific mechanisms (pytest --lf, Jest --onlyFailures, etc.), and states the return shape matches run_tests. No contradictions.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is moderately long but well-structured: it starts with the core purpose, then lists framework-specific details, and finishes with return shape and prerequisites. Every sentence adds value, though slightly verbose; could be trimmed slightly without loss.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (supporting multiple test frameworks) and lack of output schema, the description is comprehensive. It covers usage context, prerequisites, behavior per framework, and return shape, making the tool usable without additional context.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
There are zero parameters, and schema coverage is 100% vacuously. Following the guideline '0 params = baseline 4', the score is set to 4. The description does not add parameter information because none exists, but it compensates by explaining the tool's behavior and prerequisites.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states it reruns only the previously failed tests, using specific verbs like '只重跑' and contrasting with the full suite. It differentiates from sibling tool 'run_tests' by specifying the target as only failed tests.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly says when to use ('適合修完一個 bug 後驗證迭代') and notes the prerequisite that a prior run_tests must have been executed. It also mentions the tool is faster than running the full suite, guiding the agent to use it for efficient iteration.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Execute the test suite under the active QA_RUNNER and produce a structured report. The single most-called tool — invoke whenever a user says 「跑/run/test/check/驗證/執行」, after generate_test (verify new test), or after a fix (confirm bug gone).
Behavior:
Invokes the runner's native CLI under QA_PROJECT_ROOT — pytest with --screenshot=on / --tracing=on / --video=retain-on-failure, or npx jest --json, npx cypress run --reporter json, go test -json, maestro test --format junit
Optional filter narrows the scope: pytest -k expr, jest -t pattern, cypress --spec glob, go -run regex, maestro flow-name substring
Writes report.json (pytest-json-report shape, runner-agnostic) + JUnit XML
Snapshots the run into history/ and auto-triggers optimizer.write_plan() → optimization-plan.md is refreshed
Maestro: auto-retries flows that failed on first attempt (MAESTRO_RETRY=true), surfaces flaky_in_run count Returns: {exit_code, raw_exit_code, stdout_tail, stderr_tail, retry_enabled, flaky_in_run, ...}
When to use:
After writing a new test → verify it actually passes
Smoke before a release
Whenever the user prompt contains a run/test verb
When NOT to use:
Inspecting last results without re-running → use get_test_report (cheaper)
Re-running only failed cases → use run_failed (way faster)
Enumerating which tests exist → use list_tests
Edge cases:
No tests match filter → exit_code != 0 with 「no tests ran」 in stderr_tail
QA_TIMEOUT_SECONDS exceeded → exit_code 124 + [TIMEOUT…] tag in stderr_tail
filter starting with - or containing .. → blocked by security guardrail, returns {error: …}
Plan bookend (v0.10.0): pass plan_id from a prior qa_plan call and the response auto-attaches plan_verification — the critical points are checked against the just-written report.json via the same flow run_api_security_scan uses. Omit plan_id to keep the legacy shape (no plan_verification key). When verify_plan fails (unknown / expired plan_id), the run still succeeds; the error envelope is surfaced under plan_verification.
| Name | Required | Description | Default |
|---|---|---|---|
| filter | No | 選填，測試名稱關鍵字。pytest 走 -k 表達式（支援 and/or/not）、Jest 走 -t、Cypress 走 --spec '**/*<filter>*'、Go 走 -run regex、Maestro 在 flow 檔名作子字串比對。 | |
| headed | No | 選填，僅對 pytest-playwright 有效。True 時瀏覽器有 UI 模式跑（適合 debug、看 flake 視覺現象）；預設 headless 跑、CI / 大量套件用這個。 | |
| browser | No | 選填，僅對 pytest-playwright 有效，指定 Playwright 啟用的 browser engine。需事先 `playwright install <browser>` 過。 | chromium |
| plan_id | No | 選填，v0.10.0+。Plan id returned by qa_plan. When supplied, the response gains a `plan_verification` envelope that checks every critical point against the just-written report.json. Same shape as run_api_security_scan's plan bookend. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It details the invocation (pytest with screenshot/tracing/video flags, jest/cypress/maestro commands), output files (report.json, JUnit XML), side effects (snapshot to history, auto-trigger optimizer.write_plan), Maestro auto-retries, and edge cases (no match, timeout, security guardrail). This is comprehensive.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is structured with labeled sections (Behavior, When to use, Edge cases, Plan bookend) and front-loaded with the core purpose. While it is somewhat lengthy, every section earns its place by providing necessary detail. Minor redundancy in the 'When to use' list but overall efficient.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no annotations or output schema, the description covers all critical aspects: behavior, return shape (exit_code, stdout_tail, etc.), side effects, edge cases, and the plan bookend feature. It references sibling tools and explains when to use alternatives. The description is complete for a complex tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so baseline is 3. The description adds value by explaining how 'filter' works per runner (pytest -k, jest -t, etc.), that 'headed' is only for pytest-playwright, 'browser' requires pre-installation, and 'plan_id' ties to qa_plan with response shape change. It enriches the schema without redundancy.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool executes the test suite and produces a structured report. It identifies itself as the 'single most-called tool' for running tests and lists triggers like 'run/test/check/驗證/執行'. It distinguishes the primary action (execute) from related tools like get_test_report and run_failed.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly states when to use (after writing a new test, smoke before release, when user prompt contains run/test verbs) and when NOT to use (inspect results without re-running → get_test_report, re-run only failures → run_failed, list tests → list_tests). This provides clear alternatives and context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Apply the AI client's tile selection, execute the click chain, click Verify, wait for the reCAPTCHA token, and return the outcome. Pairs with inspect_visual_challenge — must be called with the challenge_id returned by the previous inspect call.
Requires confirm: true as a safety latch — an accidental call without confirm returns confirm_required without clicking anything. Also requires QA_VISUAL_CHALLENGE_CONSENT=true at the server level.
DYNAMIC-REPLACE MODE (v0.7.4): when the challenge prompt says 'Click verify once there are none left' (en) / '確定沒有遺漏' (zh), clicked tiles get replaced with new images. solve detects this and returns status: 'continue' with a FRESH screenshot + tile grid instead of clicking Verify. The AI should look at the new screenshot and call solve again with the next matches. To finalize (click Verify and check for a token), pass an empty selected_tile_indices: [].
Returns: {status: 'passed' | 'continue' | 'failed' | 'expired' | 'consent_required' | 'confirm_required' | 'challenge_not_found' | 'error', challenge_id, attempts_remaining, token (only on passed), hint, plus on 'continue': screenshot_base64, tiles, tile_count, grid_layout, rounds_used}. Telemetry logs the boolean outcome only — no screenshots, no challenge text, no tile selection are ever persisted.
Plan bookend (v0.10.0): pass plan_id from a prior qa_plan call and the response auto-attaches plan_verification. Evidence is a single-record summary {kind: 'captcha_solve', status, token_populated, rounds_used, fingerprint, challenge_id} — the raw token is NEVER included (telemetry hygiene). Verification only fires once solve actually executes; consent_required / confirm_required / challenge_not_found / expired all bypass it because they're usage errors, not solve outcomes.
| Name | Required | Description | Default |
|---|---|---|---|
| challenge_id | Yes | Required. The challenge_id returned by inspect_visual_challenge. Expires after 5 minutes; re-inspect to get a fresh id. | |
| selected_tile_indices | Yes | Required. The tiles the AI client wants to click, by index (0..tile_count-1). For a 3x3 grid: tile 0 = top-left, 4 = center, 8 = bottom-right. For a 4x4 grid: 0..15 row-major. | |
| confirm | No | Safety latch. MUST be set to true for the click chain to execute. Without it, returns `confirm_required` and clicks nothing — this prevents an accidental tool call from auto-submitting a CAPTCHA. | |
| plan_id | No | 選填，v0.10.0+. Plan id returned by qa_plan. When supplied AND solve actually executes, the response gains a `plan_verification` envelope with single-record evidence {kind: 'captcha_solve', status, token_populated, rounds_used, fingerprint, challenge_id}. Raw token never appears in evidence — CPs check token_populated. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Even without annotations, the description is highly transparent. It details the safety latch (confirm), dynamic-replace mode, telemetry logging constraints (only boolean outcome, no screenshots/challenge text/tile selection persisted), and plan bookend behavior. No contradictions with annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well-structured with clear sections for core behavior, dynamic-replace mode, and plan bookend. It is front-loaded with the main purpose. Some minor redundancy (e.g., explaining telemetry twice) but overall efficient for its complexity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema, the description fully enumerates all possible return statuses and their fields. It covers edge cases (e.g., consent_required, confirm_required) and explains dynamic mode continuation. Highly complete for a complex CAPTCHA-solving tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
While input schema already describes each parameter, the description adds valuable context: challenge_id expiry (5 min), tile index mapping for grids, confirm as safety latch, and plan_id optionality with evidence summary. Schema coverage is 100%, so baseline is 3; description adds enough to raise to 4.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb and resource: 'Apply the AI client's tile selection, execute the click chain, click Verify, wait for the reCAPTCHA token, and return the outcome.' It explicitly pairs with inspect_visual_challenge and differentiates by being the solve step, distinguishing it from siblings.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides explicit pairing with inspect_visual_challenge and requires challenge_id from that call. It explains dynamic-replace mode and when to pass empty selected_tile_indices to finalize. However, it does not explicitly state alternatives or when not to use.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
v0.9.1 (extended v0.9.2 with auto-discovery) — Walk a plan's critical points and check each against evidence. Pairs with qa_plan — must be called with the plan_id returned by a prior qa_plan call. Returns a structured checklist with per-CP satisfaction + an overall status (passed / incomplete / failed).
Matching rule: a CP is satisfied when its verification_hint appears (case-insensitively, as a substring) in any evidence item's stringified form. Evidence items may be strings, dicts, or nested structures — the matcher flattens them.
v0.9.2 — auto_discover mode: set auto_discover: true and the verifier reads the project's pytest-json-report at <QA_PROJECT_ROOT>/report.json (or MK_QA_REPORT_PATH, or the report_path arg) and adds its tests list to the evidence stream. Best-effort — missing or malformed report is silently skipped, NOT a hard error. The response's evidence_sources field reports what was used.
status semantics:
'passed': every CP satisfied
'incomplete': some satisfied, some not
'failed': zero CPs satisfied (or empty evidence)
Even if the host claims 'all good', verify_plan returns 'incomplete' when any CP is unsatisfied. That's the design — ground truth wins over capability claims.
v0.9.3 — When persistence is enabled (see qa_plan), an in-memory cache miss transparently falls back to disk. The response's plan_source field reports where the plan came from: 'memory' (cache hit) or 'disk' (loaded from /.json after a restart / eviction).
Returns: {plan_id, task, kind, status, checklist[{id, description, verification_hint, satisfied, matched_evidence}], unmet[], summary{total, satisfied, unsatisfied}, evidence_sources{explicit_count, autodiscovered, autodiscovered_count, report_path}, plan_source ('memory' or 'disk'), verified_at}.
Error shapes: no_plan_id / plan_not_found / no_evidence (only when both explicit evidence AND auto_discover are omitted) / bad_evidence.
| Name | Required | Description | Default |
|---|---|---|---|
| plan_id | Yes | Required. The plan_id returned by qa_plan. | |
| evidence | No | Optional when `auto_discover: true`. Each item is searched for each CP's verification_hint. Pass structured payloads — test result rows from `get_test_report`, scan findings from `run_api_security_scan`, log lines, screenshot paths, etc. | |
| auto_discover | No | v0.9.2 — When true, read the project's pytest-json-report and add its `tests` array to the evidence stream. Useful for verifying a CP set against the most recent test run without manually copying report rows into the call. | |
| report_path | No | v0.9.2 — Override the report.json location when auto_discover is true. Defaults to `MK_QA_REPORT_PATH` env, then `<QA_PROJECT_ROOT>/report.json`, then `./report.json`. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description fully discloses behavioral traits: matching rule (case-insensitive substring), flattening of evidence, auto_discover behavior (best-effort, silent skip), status semantics, persistence cache, and error shapes. Comprehensive.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Well-structured with layered detail: main purpose first, then matching rule, then version updates. Every sentence provides necessary information without verbosity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Completely describes return shape with field details, error shapes, and behavioral nuances. Without an output schema, the description fully compensates.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, and the description adds significant context: matching rule for evidence, auto_discover details, report_path fallback logic. Adds value beyond schema descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states it verifies a plan's critical points against evidence, names the sibling `qa_plan` for pairing, and specifies the return type. It distinguishes itself from other tools by focusing on verification of plans.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly mandates calling with `plan_id` from a prior `qa_plan` call and describes auto_discover usage. Does not explicitly state when not to use, but the pairing guidance is clear.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Most tools have distinct purposes (e.g., analyze_url vs analyze_screen vs analyze_stream for different platforms), and descriptions are thorough. However, some overlap exists between auto_generate_tests and the combination of analyze_url + generate_test, and the sheer number of tools (22) increases potential for misselection.
Tool names follow a consistent snake_case and verb_noun pattern (analyze_url, generate_test, run_tests). Minor deviations include auto_generate_tests (two verbs) and a few longer names like get_failure_details, but overall the pattern is predictable.
22 tools is on the heavy side, but the server covers a broad domain (QA for web, mobile, stream, API security). Some tools could be merged (e.g., auto_generate_tests vs analyze_url+generate_test), making the surface feel slightly crowded for the scope.
The tool set covers the QA lifecycle well: analysis (analyze_url, analyze_screen, analyze_stream), test generation (generate_test, auto_generate_tests), execution (run_tests, run_failed), reporting (get_test_report, generate_html_report), and optimization (get_optimization_plan). Minor gaps include lack of direct test editing or test data management tools.
Unclaimed servers have limited discoverability.
If you are the server author, to access and configure the admin panel.
MCP server for Mint — AI-powered QA that runs your app in a real browser on every PR.
MCP server for building and testing AI agents with multi-model experimentation and insights.
Official MCP server for Qase — manage test cases, runs, suites, defects via AI tools.
MCP server for AI agents to plan, verify, and deploy Cloudflare-native apps.
We provide all the information about MCP servers via our MCP API.
curl -X GET 'https://glama.ai/api/mcp/v1/servers/kao273183/mk-qa-master'
If you have feedback or need assistance with the MCP directory API, please join our Discord server
