# playwright-kiwi-sync

`license: MIT` · `node: >=18` · `Model Context Protocol: server` · `tests: 20 e2e checks`

An MCP server that syncs automated test results — Playwright, Allure, or any JUnit
XML (Maestro, Cypress, pytest…) — straight into **existing** Kiwi TCMS test
executions. No more "mark case 4001 as passed" one at a time in chat.

It's a **companion** to the team's `kiwi-tcms-mcp`, not a replacement: that one is
for everything manual (browsing plans, creating cases, ad-hoc lookups); this one
runs after a test job and records what happened.

- **Reads** Playwright's `json` report, Allure's raw `allure-results/`, or a JUnit
  XML file — and matches each test to a Kiwi case via a `[C1234]` tag or annotation
- **Writes** the execution status (PASSED/FAILED/BLOCKED) into the run, adds the
  failure message as a comment, uploads screenshot/trace attachments
- **Fast**: one read for the whole run, then bounded-parallel writes — a 300-test
  suite is ~1 read + 300 writes (5 in flight), not 600 sequential calls
- **Safe by default**: `dryRun` shows exactly what would change; cases missing from
  the run are reported, not silently added (opt in with `addMissingCases`)
- **Correct by construction**: execution statuses are resolved **by name** from your
  instance (`TestExecutionStatus.filter`), never hardcoded ids; several results on one
  case (data-driven tests) collapse to the **worst** outcome, deterministically
- **Guarded escape hatch**: `kiwi_rpc` refuses `*.remove({})` outright and needs an
  explicit `allowDestructive` for any delete; oversized results are capped so one
  broad filter can't flood the model's context
- **Runs locally** with your own Kiwi login — it can only do what your account can

```
Playwright json / allure-results/ / junit.xml
                    |
MCP client  <--stdio-->  playwright-kiwi-sync (Node)  <--HTTPS JSON-RPC-->  Kiwi TCMS
```

## Tools

| Tool | What it does |
|---|---|
| `sync_playwright_run` | Playwright `--reporter=json` file → Kiwi executions |
| `sync_allure_run` | `allure-results/` directory (raw `*-result.json`) → Kiwi executions |
| `sync_junit_run` | JUnit XML → Kiwi executions. Playwright `--reporter=junit`, **Maestro `--format junit`**, Cypress, pytest… |
| `get_run_summary` | Pass/fail/blocked counts, % complete and failed case IDs for a run — regression status without opening Kiwi |
| `create_kiwi_run` | Create a new TestRun from a plan + list of case IDs |
| `kiwi_rpc` | Raw escape hatch — call any Kiwi JSON-RPC method directly. Destructive methods need `allowDestructive: true` and are **always** refused with an empty filter (Kiwi's `remove({})` deletes every row). Large results are truncated with a narrowing hint. |

All three sync tools take `runId`, `dryRun` and `addMissingCases`.

### A note on report formats

This reads **raw, structured** result data. It does **not** parse rendered HTML
reports (Playwright's or Allure's) — those are generated static sites with no
stable format meant to be read back, and scraping them would break on any reporter
version bump. Keep generating HTML for humans; just also emit `json`/`junit`
(Playwright) or keep `allure-results/` around (Allure). They run side by side.

## How it compares

| | Kiwi official `junit.xml-plugin` | `kiwi-tcms-mcp` (team) | **playwright-kiwi-sync** |
|---|---|---|---|
| Reads automation results | JUnit XML | no — chat-driven only | Playwright json, Allure, JUnit |
| Records against **existing** case IDs | no — creates/reuses cases **by name** (duplicates your migrated manual cases) | manually, per prompt | yes, via `[C1234]` tag / annotation |
| Failure comment + attachments | status only | via separate tools, manually | automatic on failure |
| Dry run | no | no | yes |
| Runtime | Python + `tcms-api` | Node | Node — same stack as `kiwi-tcms-mcp` |
| Chat / MCP native | no | yes | yes |

The official plugin is the right tool if you want Kiwi to *generate* cases from a
CI run. This is the right tool if you already have curated cases in Kiwi (e.g.
migrated from TestRail) and want automation results recorded against **those**.

## Tagging tests for Kiwi

A `[C1234]` tag in the test name works for every format. Annotation/label forms
keep titles clean but are reporter-specific:

```ts
// Works everywhere (Playwright json, Allure, JUnit, Maestro flow names)
test("[C4001] rejects an expired token", async ({ page }) => { ... });

// Playwright json reporter
test("rejects an expired token", async ({}, testInfo) => {
  testInfo.annotations.push({ type: "kiwi-case", description: "4001" });
});

// Allure
import { allure } from "allure-js-commons";
test("rejects an expired token", async () => { allure.label("kiwi-case", "4001"); });
```

## Setup

New here? Follow **[SETUP.md](./SETUP.md)** — step-by-step, cross-platform.

```bash
git clone https://github.com/ZainAbdeen/playwright-kiwi-sync
cd playwright-kiwi-sync
npm install
cp .env.example .env   # your own Kiwi login
npm test               # smoke + 20 end-to-end checks against a built-in mock Kiwi
```

Then add a `kiwi-sync` block under `mcpServers` in `~/.claude.json`, next to your
existing `kiwi` block — full JSON in [SETUP.md](./SETUP.md#5-wire-it-into-claude-code-cli).

## Example prompts

> "Dry-run sync results.json into Kiwi run 87."
>
> "Sync results.json into run 87, add any cases that aren't in it yet."
>
> "Sync the Maestro junit.xml into run 92."
>
> "What's the status of regression run 87?"

## Testing

`npm run e2e` boots an in-memory fake of Kiwi's JSON-RPC endpoint
(`scripts/mock-kiwi.js`), starts the real MCP server against it, and drives every
tool through a real MCP client. It covers: dry-run writes nothing, correct
executions updated, failure comments, missing-case reporting vs `addMissingCases`,
JUnit attribute decoding, run summary, `kiwi_rpc` passthrough, and the `-32098`
permission-denied path (surfaced per case with a hint, and not mistaken for an
expired session), the escape-hatch guards (empty-filter delete refused, explicit
delete needs `allowDestructive`, oversized output truncated), and a dead-tunnel
HTML/502 response diagnosed plainly instead of crashing on JSON parse.

Method names and the `values` shape were checked against the Kiwi 16.4 API docs and
source (`TestExecution.update` binds a ModelForm whose field is `status`, not
`status_id`; `add_comment`, `add_attachment` (since 15.3), `add_link` and
`TestRun.add_case` all exist). What the mock can't prove is your instance's
permissions and any local customisation — hence dry-run first, see
[SETUP.md](./SETUP.md#before-you-point-this-at-a-real-run).

## Known gaps

- Needs a Kiwi account that can **update** executions. A create-only account
  (the `-32098` case) will get a clear per-case error, but nothing will sync.
- No CI/webhook trigger yet — invoked from a Claude prompt. A CI step calling
  `sync_junit_run` after the test job is the natural next step.
- Several results on one case collapse to the worst status (FAILED/ERROR > BLOCKED >
  PASSED) with all failure messages kept — there's no per-variant record in Kiwi.
- JUnit has no attachment element, so `sync_junit_run` records status + failure
  message only.

## License

[MIT](./LICENSE)