run_tests

Instructions

Implementation Reference

• src/mk_qa_master/tools/runner.py:9-13 (handler)

  Tool-level entry point (runner.py): dispatches run_tests by getting the active runner and calling its run_tests method, with security filter validation first.

  def run_tests(filter=None, headed=False, browser="chromium") -> dict:
      ok, err = validate_filter(filter)
      if not ok:
          return {"error": err}
      return get_runner().run_tests(filter=filter, headed=headed, browser=browser)

• src/mk_qa_master/server.py:44-108 (registration)

  MCP tool registration in server.py: defines the 'run_tests' Tool with description, schema (filter, headed, browser), and detailed usage docs.

      name="run_tests",
      description=(
          "Execute the test suite under the active QA_RUNNER and produce a structured "
          "report. The single most-called tool — invoke whenever a user says "
          "「跑/run/test/check/驗證/執行」, after generate_test (verify new test), "
          "or after a fix (confirm bug gone).\n\n"
          "Behavior:\n"
          "- Invokes the runner's native CLI under QA_PROJECT_ROOT — pytest with "
          "  --screenshot=on / --tracing=on / --video=retain-on-failure, or "
          "  `npx jest --json`, `npx cypress run --reporter json`, `go test -json`, "
          "  `maestro test --format junit`\n"
          "- Optional `filter` narrows the scope: pytest -k expr, jest -t pattern, "
          "  cypress --spec glob, go -run regex, maestro flow-name substring\n"
          "- Writes report.json (pytest-json-report shape, runner-agnostic) + JUnit XML\n"
          "- Snapshots the run into history/ and auto-triggers optimizer.write_plan() "
          "  → optimization-plan.md is refreshed\n"
          "- Maestro: auto-retries flows that failed on first attempt (MAESTRO_RETRY=true), "
          "  surfaces flaky_in_run count\n"
          "Returns: {exit_code, raw_exit_code, stdout_tail, stderr_tail, retry_enabled, "
          "flaky_in_run, ...}\n\n"
          "When to use:\n"
          "- After writing a new test → verify it actually passes\n"
          "- Smoke before a release\n"
          "- Whenever the user prompt contains a run/test verb\n\n"
          "When NOT to use:\n"
          "- Inspecting last results without re-running → use get_test_report (cheaper)\n"
          "- Re-running only failed cases → use run_failed (way faster)\n"
          "- Enumerating which tests exist → use list_tests\n\n"
          "Edge cases:\n"
          "- No tests match `filter` → exit_code != 0 with 「no tests ran」 in stderr_tail\n"
          "- QA_TIMEOUT_SECONDS exceeded → exit_code 124 + `[TIMEOUT…]` tag in stderr_tail\n"
          "- `filter` starting with `-` or containing `..` → blocked by security "
          "  guardrail, returns {error: …}"
      ),
      inputSchema={
          "type": "object",
          "properties": {
              "filter": {
                  "type": "string",
                  "description": (
                      "選填，測試名稱關鍵字。pytest 走 -k 表達式（支援 and/or/not）、"
                      "Jest 走 -t、Cypress 走 --spec '**/*<filter>*'、Go 走 -run "
                      "regex、Maestro 在 flow 檔名作子字串比對。"
                  ),
              },
              "headed": {
                  "type": "boolean",
                  "default": False,
                  "description": (
                      "選填，僅對 pytest-playwright 有效。True 時瀏覽器有 UI 模式跑（適合 debug、"
                      "看 flake 視覺現象）；預設 headless 跑、CI / 大量套件用這個。"
                  ),
              },
              "browser": {
                  "type": "string",
                  "enum": ["chromium", "firefox", "webkit"],
                  "default": "chromium",
                  "description": (
                      "選填，僅對 pytest-playwright 有效，指定 Playwright 啟用的 browser engine。"
                      "需事先 `playwright install <browser>` 過。"
                  ),
              },
          },
      },
  ),

• src/mk_qa_master/server.py:644-650 (handler)

  Dispatch handler in server.py's _dispatch function: routes 'run_tests' calls to runner.run_tests() with filter, headed, and browser args.

  if name == "run_tests":
      result = runner.run_tests(
          filter=args.get("filter"),
          headed=args.get("headed", False),
          browser=args.get("browser", "chromium"),
      )
      return [TextContent(type="text", text=json.dumps(result, ensure_ascii=False, indent=2))]

• src/mk_qa_master/runners/base.py:4-13 (helper)

  Abstract base class defining the run_tests interface that all runners implement.

  class TestRunner(ABC):
      """通用測試 runner 介面。每個測試框架實作一個子類。"""
  
      name: str = "base"
  
      @abstractmethod
      def list_tests(self) -> str: ...
  
      @abstractmethod
      def run_tests(self, filter: str | None = None, **kwargs) -> dict: ...

• src/mk_qa_master/security.py:36-68 (schema)

  Security guardrail for run_tests filter argument: rejects args starting with '-' (CLI injection) or containing '..' (path traversal).

  def validate_filter(value: object) -> tuple[bool, str | None]:
      """Check a `filter` argument before it reaches subprocess argv.
  
      Pytest / Go / Jest / Cypress all accept fairly free-form filter
      syntax (regex, boolean expressions, globs), so the rule is *minimal*:
      reject the cases that smell like attacks, allow everything else.
  
      Returns (ok, error_message). When ok is True, caller passes the
      original value through unchanged; when False, caller surfaces the
      error string as a tool-result error.
      """
      if value is None or value == "":
          return True, None
      if not isinstance(value, str):
          return False, f"filter must be a string, got {type(value).__name__}"
      if len(value) > _MAX_FILTER_LEN:
          return False, f"filter too long ({len(value)} > {_MAX_FILTER_LEN} chars)"
      if value.startswith("-"):
          # A leading `-` would slot in as a new CLI option to the underlying
          # tool — e.g. `pytest -k --config=evil` parses `--config=evil` as a
          # pytest option, not as part of -k. Block at the boundary.
          return False, (
              "filter cannot start with '-' (looks like a CLI option, "
              "not a test name)"
          )
      if ".." in value:
          # Filters are not paths; '..' in them is almost always a sign that
          # someone is trying to escape an expected scope (cypress turns the
          # filter into a glob, where `..` would walk outside the project).
          return False, "filter cannot contain '..'"
      if _CONTROL_CHARS_RE.search(value):
          return False, "filter contains control characters"
      return True, None