# qa-mcp

Automated QA for a **.NET microservice backend** (several independent services, each with its own
Swagger/OpenAPI document) and a **React frontend**, running against **stage**, exposed to an MCP
client over stdio.

## The one architectural rule

**MCP is a thin layer on top of the test engine. The engine does not depend on the model.**

Tests are never generated by an LLM at call time. They are either derived deterministically from
OpenAPI, or written by hand as YAML flows / Playwright specs. That is what makes two consecutive
runs comparable — and comparability is the entire value of the tool. If every run produced a
different set of assertions, `qa_diff` could not tell a regression from noise.

Consequence: everything under `src/api`, `src/flows`, `src/ui` and `src/report` runs perfectly well
from CI without an MCP client. `src/index.ts` only registers tools and shrinks the output.

## Install

```bash
npm install
npm run build
```

Requires Node ≥ 20. `npm install` also pulls Playwright; download the browser once with:

```bash
npx playwright install chromium
```

## Configure

`qa.config.json` is the single source of truth. Its path comes from `QA_MCP_CONFIG`, defaulting to
`./qa.config.json`. Start from the example:

```bash
cp qa.config.example.json qa.config.json
```

| Section | What it does |
| --- | --- |
| `environment` | Free-form label stored in every run report. |
| `services[]` | `{ name, baseUrl, openapi, tags[], headers{} }`. `openapi` is either an absolute URL or a path relative to `baseUrl`. |
| `frontend` | `baseUrl`, `loginPath`, `loginSelectors{username,password,submit,successUrlPattern}`, `crawl{maxPages,ignore[],viewport}`. |
| `auth` | `tokenUrl`, `grantType`, `clientId`, `clientSecret`, `tokenField`, and one entry per role under `roles`. |
| `policy` | `readOnly`, `allowedMutationPaths[]`, `excludePaths[]`, `maxResponseMs`, `concurrency`, `requestTimeoutMs`. |
| `samples` | Real values used to fill required parameters, keyed by **parameter name**. |
| `artifactsDir` | Where runs, HTML reports and screenshots are written. Default `.qa-runs`. |

### Secrets never live in the config

Any string starting with `env:` is read from the environment; if the variable is unset, startup
fails with an explicit message naming the field and the variable.

```json
"password": "env:QA_ADMIN_PASSWORD"
```

```bash
export QA_ADMIN_PASSWORD=...
export QA_CUSTOMER_PASSWORD=...
export QA_CLIENT_SECRET=...
```

## Register the server in an MCP client

`command` + `args`, transport stdio:

```json
{
  "mcpServers": {
    "qa-mcp": {
      "command": "node",
      "args": ["G:/mcp/qatester/dist/index.js"],
      "env": {
        "QA_MCP_CONFIG": "G:/mcp/qatester/qa.config.json",
        "QA_ADMIN_PASSWORD": "...",
        "QA_CUSTOMER_PASSWORD": "...",
        "QA_CLIENT_SECRET": "..."
      }
    }
  }
}
```

Claude Code CLI equivalent:

```bash
claude mcp add qa-mcp -- node G:/mcp/qatester/dist/index.js
```

Verify the handshake at any time without touching stage:

```bash
npm run smoke
```

## Tools

| Tool | Input | What it does |
| --- | --- | --- |
| `qa_discover` | `services?` | Fetches every Swagger document, reports endpoints per service, planned assertions, operations skipped by policy **with the reason**, and services that could not be reached. Sends no request to the endpoints. |
| `qa_run_api` | `services?`, `role?`, `pathContains?` | Generates and runs the API matrix. Mutating endpoints are skipped while `policy.readOnly` is on. |
| `qa_run_flows` | `names?`, `flowsDir?` | Runs `tests/flows/*.yaml` sequentially. |
| `qa_run_ui` | `mode: specs\|crawl\|both`, `role?`, `maxPages?`, `startPaths?`, `grep?` | Playwright specs and/or the automatic crawl. |
| `qa_run_all` | `role?`, `services?`, `skipUi?`, `maxPages?` | Full pass, then an automatic diff against the previous stored run. Release gate. |
| `qa_report` | `runId?`, `status?`, `severity?`, `suite?`, `service?`, `limit` | Reads a stored run with filters. Use it when a digest says results were truncated. |
| `qa_diff` | `baselineRunId?`, `currentRunId?` | Compares two runs by stable test id. |

Every run tool returns a **digest**, not the raw results: the summary, failures grouped by service,
at most 40 failures sorted critical-first, how many were truncated, and the `runId` to drill into
with `qa_report`. A full pass over several services produces thousands of assertions; returning all
of them would bury the signal.

## What gets tested

### Generated API tests

Per operation, from the OpenAPI document:

| Check | Assertion | Severity |
| --- | --- | --- |
| `contract` | Valid request → status must be declared in the spec and the body must validate against that status' schema | major |
| `authz` | Secured operation called with no token → anything other than 401/403 is a leak | critical |
| `robustness` | Malformed value in a required typed parameter → expect 400/404/422 | major |
| `perf` | Response time over `policy.maxResponseMs` | minor |

**Any 5xx is an unconditional critical failure**, including when the input was deliberately broken.
A well-behaved service answers 400, not 500. On a .NET backend this single rule is the highest-yield
automated check there is — it drags unhandled exceptions straight out of the framework.

Parameter values come from `config.samples` (by parameter name) first, then `enum[0]`, `default`,
`example`, then a format/type heuristic (`uuid` → zero-uuid, `date` → today, `integer` → 1, …).
Malformed values match the type: `uuid` → `not-a-uuid`, numeric → `not-a-number`, date →
`31-31-9999`.

Each test carries a stable id — `service:METHOD:/path#check` — with no timestamp and no random
value in it. `qa_diff` depends entirely on that.

One valid request produces both the `contract` and `perf` results, so an operation costs at most
three HTTP calls (valid, unauthenticated, malformed).

### Safety model — read this before pointing it at anything

Running generated tests against stage can destroy data or fire notifications at real users.

* With `policy.readOnly: true` (the default) only `GET`/`HEAD`/`OPTIONS` are executed. Every
  mutating verb is refused unless its path starts with one of `policy.allowedMutationPaths`.
* `policy.excludePaths` is honoured **always**, including when `readOnly` is off.
* Every refused operation is reported with its reason by `qa_discover`, so you always know what is
  *not* covered.
* Destructive operations are only allowed inside YAML flows, where a human wrote the steps.

### Cross-service flows

`tests/flows/*.yaml`. Real microservice bugs live here, not in single-service tests: each service
passes its own tests while the order id never reaches the notification consumer.

```yaml
name: order-lifecycle
description: why this scenario matters
role: customer
severity: critical
steps:
  - name: create order
    service: orders          # must match a service name in qa.config.json
    method: POST
    path: /api/orders
    role: customer           # optional, overrides the flow role
    body: { productId: "{{productId}}" }
    expectStatus: [200, 201]
    expectBody: { status: Pending }   # dot-path -> expected value
    capture: { orderId: id }          # variable name -> dot-path in the response
    waitMs: 0                         # pause before the step, for eventual consistency
```

`{{var}}` interpolation works in `path`, `body`, `headers` and `expectBody`. Dot-paths understand
array indexes (`items[0].status`). Flows run sequentially — they mutate shared state. One flow
produces exactly one result: the business scenario either completes or it does not. On failure the
evidence holds the failing step name, the trace of the steps that did succeed, and every captured
variable. `waitMs` exists for the case where the target service only learns about the change through
a message broker.

### Frontend — two layers

**Fixed specs** (`tests/ui/`, standalone `playwright.config.ts`) are run as a child process with
`--reporter=json` and normalised into results. `QA_BASE_URL`, `QA_LOGIN_PATH` and `QA_LOGIN`
(credentials for the requested role) are passed in as environment variables, so the project also
runs directly in CI. If `playwright.config.ts` is missing you get a `skip` with a clear message, not
a crash. Put `@critical` or `@minor` in a test title to set its severity.

The three shipped specs assert business outcomes rather than "did the page load": a successful
login, a list that must contain rows (an empty table where data is expected is the classic silent
data-layer failure), and one that fakes a 500 with `page.route` to prove the UI shows the user an
error instead of hanging on a spinner.

**Automatic crawl** covers the pages nobody wrote a spec for. It optionally logs in with a role; if
that login fails it returns one critical result and stops immediately, because every later result
would be meaningless without a session. It then BFS-crawls same-origin links up to `maxPages`,
skipping `crawl.ignore` prefixes. A route fails on navigation error, HTTP ≥ 400, console errors, an
XHR ≥ 400, or an effectively empty render. Console noise (React DevTools, HMR, React Router future
flags, unused-preload warnings, favicon 404s) is filtered out — without that the output is unusable.
Every failure gets a full-page screenshot whose path lands in the evidence.

## Runs, reports and diff

Each run is written to `artifactsDir` as `run-<timestamp>.json` plus a readable `.html` next to it,
with a severity-coloured failure table. Colons and dots in the timestamp are replaced with `-`
(illegal in Windows filenames); the ids stay lexicographically sortable.

`qa_diff` compares two runs by stable id and splits the delta:

| Bucket | Meaning |
| --- | --- |
| `regressions` | Passed before, fails now. **The only bucket that should block a release.** |
| `fixed` | Failed before, passes now. |
| `newFailures` | Test did not exist in the baseline and fails now (new endpoint, new page). |
| `stillFailing` | Already broken before. |
| `removed` | Was in the baseline, gone now (endpoint or page deleted). |

When no `baselineRunId` is given, the baseline is the most recent earlier run covering the **same
suites** — comparing an API-only run against a UI-only run would mark every test as removed and mean
nothing.

## Expect the first run to be noisy

This is normal and it is not a bug. The first pass against a real stage environment will report
plenty of failures that are configuration, not defects: endpoints needing an id you did not provide,
admin-only paths, export jobs, health probes.

Work through it in this order:

1. Run `qa_discover` and read what was skipped by policy — that is your coverage gap.
2. Run `qa_run_api`, then push obvious non-defects into `policy.excludePaths`.
3. Fill `samples` with **real** stage ids (`orderId`, `customerId`, `productId`, …) so path and
   required-query parameters resolve to rows that actually exist. Most 404 noise disappears here.
4. Raise `maxResponseMs` if stage is simply slower than production.
5. Re-run until what remains is genuinely interesting.
6. **Keep that run as the baseline.** From then on `qa_diff` / `qa_run_all` answer the only question
   that matters before a release: *did anything that used to work stop working?*

## Layout

```
src/
  config.ts               zod config schema + env: resolution
  types.ts                TestResult / RunReport / summarize()
  discovery/openapi.ts    fetch + normalise specs, resolve $ref
  auth/session.ts         per-role token with cache
  api/generator.ts        test matrix per operation + safety gate
  api/runner.ts           bounded-concurrency execution + schema validation
  flows/runner.ts         cross-service YAML scenarios
  ui/crawler.ts           automatic Playwright crawl
  ui/specs.ts             runs Playwright specs, normalises the JSON report
  report/store.ts         store, diff, render HTML
  index.ts                MCP tool registration
tests/
  flows/*.yaml
  ui/                     playwright.config.ts + *.spec.ts
```

## Notes for whoever extends this

* stdio transport: **never write to stdout.** `console.log` corrupts JSON-RPC and the client
  disconnects with an opaque error. Every log goes to `process.stderr` (see `log()` in
  `src/index.ts`). `npm run smoke` fails if anything non-protocol reaches stdout.
* `ajv`/`ajv-formats` are CommonJS. Under `NodeNext` the constructable sits on `.default`, and
  `import type Ajv from "ajv"` is unusable as a type — `src/api/runner.ts` shows the working pattern.
* Relative ESM imports need the `.js` extension even from `.ts` sources.
* Every tool handler is wrapped in `try/catch` and returns `isError: true` rather than throwing.
* Every `fetch` carries `AbortSignal.timeout(policy.requestTimeoutMs)`.
* Uncompilable response schemas are treated as "no schema", not as an endpoint failure — the defect
  is in the spec, not in the service.
* `$ref` resolution has both a depth cap and a per-branch seen-set; self-referencing DTOs are common
  in .NET and would otherwise hang the process.