verify_plan
Check each critical point in a QA plan against evidence to produce a structured checklist with per-point satisfaction and an overall status of passed, incomplete, or failed.
Instructions
v0.9.1 (extended v0.9.2 with auto-discovery) — Walk a plan's critical points and check each against evidence. Pairs with qa_plan — must be called with the plan_id returned by a prior qa_plan call. Returns a structured checklist with per-CP satisfaction + an overall status (passed / incomplete / failed).
Matching rule: a CP is satisfied when its verification_hint appears (case-insensitively, as a substring) in any evidence item's stringified form. Evidence items may be strings, dicts, or nested structures — the matcher flattens them.
v0.9.2 — auto_discover mode: set auto_discover: true and the verifier reads the project's pytest-json-report at <QA_PROJECT_ROOT>/report.json (or MK_QA_REPORT_PATH, or the report_path arg) and adds its tests list to the evidence stream. Best-effort — missing or malformed report is silently skipped, NOT a hard error. The response's evidence_sources field reports what was used.
status semantics:
'passed': every CP satisfied
'incomplete': some satisfied, some not
'failed': zero CPs satisfied (or empty evidence)
Even if the host claims 'all good', verify_plan returns 'incomplete' when any CP is unsatisfied. That's the design — ground truth wins over capability claims.
v0.9.3 — When persistence is enabled (see qa_plan), an in-memory cache miss transparently falls back to disk. The response's plan_source field reports where the plan came from: 'memory' (cache hit) or 'disk' (loaded from <plans_dir>/<plan_id>.json after a restart / eviction).
Returns: {plan_id, task, kind, status, checklist[{id, description, verification_hint, satisfied, matched_evidence}], unmet[], summary{total, satisfied, unsatisfied}, evidence_sources{explicit_count, autodiscovered, autodiscovered_count, report_path}, plan_source ('memory' or 'disk'), verified_at}.
Error shapes: no_plan_id / plan_not_found / no_evidence (only when both explicit evidence AND auto_discover are omitted) / bad_evidence.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| plan_id | Yes | Required. The plan_id returned by qa_plan. | |
| evidence | No | Optional when `auto_discover: true`. Each item is searched for each CP's verification_hint. Pass structured payloads — test result rows from `get_test_report`, scan findings from `run_api_security_scan`, log lines, screenshot paths, etc. | |
| auto_discover | No | v0.9.2 — When true, read the project's pytest-json-report and add its `tests` array to the evidence stream. Useful for verifying a CP set against the most recent test run without manually copying report rows into the call. | |
| report_path | No | v0.9.2 — Override the report.json location when auto_discover is true. Defaults to `MK_QA_REPORT_PATH` env, then `<QA_PROJECT_ROOT>/report.json`, then `./report.json`. |