e2e-verifier

MCP server that reproduces and verifies website bug tickets in a real browser and hands back evidence: an annotated video, a Playwright trace, screenshots, logs and a verdict per acceptance criterion.

• Input: a structured BugTicket (steps, expected behaviour, acceptance criteria) or a plain Scenario. The MCP client (for example Claude Code) turns free-text bug reports into tickets; the server itself never calls an LLM.

• Execution: Playwright, Chromium headless by default, one isolated browser context per run.

• Evidence: video.webm with a HUD overlay (step banner, red frame around the failing element), trace.zip for the Playwright trace viewer, per-step screenshots, console.jsonl, network.jsonl, chapters.vtt, report.md and result.json.

• Modes: reproduce before a fix (a failing assertion means reproduced) and verify after it (all criteria met means fixed).

The full design is in spec.md.

Requirements

• Python 3.12+ and uv

• Chromium for Playwright (uv run playwright install chromium)

• Optional: ffmpeg on the path for an additional mp4 export (E2E_FFMPEG_PATH)

Related MCP server: QualityMax QA MCP

Setup

uv sync
uv run playwright install chromium

Run the server

uv run e2e-verifier                 # stdio transport (default)
E2E_TRANSPORT=http uv run e2e-verifier   # Streamable HTTP on 127.0.0.1:8765

Claude Code configuration (.mcp.json in the project that contains the website under test):

{
  "mcpServers": {
    "e2e-verifier": {
      "command": "uv",
      "args": ["run", "e2e-verifier"],
      "env": {
        "E2E_ARTIFACT_DIR": "artifacts",
        "E2E_ALLOWED_HOSTS": "localhost,127.0.0.1"
      }
    }
  }
}

The paths are relative to the project root, which is the working directory Claude Code uses for a project-scoped .mcp.json. When the server is started from another directory, add "--directory", "<absolute path to end_to_end_testmcp>" after "run" and use an absolute E2E_ARTIFACT_DIR. The HTTP alternative is E2E_TRANSPORT=http uv run e2e-verifier and a client entry {"type": "http", "url": "http://127.0.0.1:8765/mcp"}.

Typical session

1. probe_target("http://127.0.0.1:5173/") – is the site up?

2. discover_elements(url, query="Save") – which targets exist? Returns suggested Targets.

3. Build a BugTicket (prompt ticket_from_report), then validate_ticket.

4. run_ticket(ticket, options={"mode": "reproduce"}) – attach video.webm to the ticket.

5. After the fix: run_ticket(ticket, options={"mode": "verify"}).

Minimal ticket:

{
  "id": "WEB-142",
  "title": "Save button does nothing",
  "url": "http://127.0.0.1:5173/settings",
  "steps": [
    {"id": "s1", "action": "fill", "target": {"label": "Name"}, "value": "Anna"},
    {"id": "s2", "action": "click", "target": {"role": {"role": "button", "name": "Save"}}},
    {"id": "s3", "action": "expect_visible", "target": {"role": {"role": "status", "name": "Saved"}}},
    {"id": "s4", "action": "expect_request", "method": "POST", "url_pattern": "**/api/settings", "status": 200}
  ],
  "expected": "A toast 'Saved' appears and the settings are posted.",
  "actual": "Nothing happens.",
  "acceptance_criteria": [
    {"id": "AC1", "description": "Toast appears", "step_ids": ["s3"]},
    {"id": "AC2", "description": "Settings are posted", "step_ids": ["s4"]}
  ]
}

Interactive sessions: letting an agent operate the site

An agent that does not know the site yet can drive it step by step. Every session is recorded (video with HUD banners, trace, screenshots, logs), and every tool result comes with a screenshot the agent can look at.

session_open(url)                         -> session id, first navigation result, screenshot
session_inspect(session_id, query="Save") -> ARIA snapshot + candidate targets with suggested Target JSON
session_act(session_id, step)             -> step result + screenshot; step = any action/assertion,
                                             e.g. {"id":"s1","action":"click","target":{"test_id":"save"}}
session_annotate(session_id, message, target?, tone?) -> callout/frame drawn into the video + screenshot
session_screenshot(session_id, full_page?) -> screenshot
session_close(session_id, verdict?, summary?) -> RunResult with video.webm, trace.zip, report.md ...

Failed assertions do not end a session; they are marked with a red frame in the video and the agent can keep going. get_schema(kind="step") returns the schema of all step types. Sessions are closed automatically after E2E_SESSION_IDLE_TIMEOUT_S (default 600 s); at most E2E_MAX_SESSIONS (default 3) are open at once. The session id doubles as run id for get_artifact and the runs:// resources.

Tools

Resources: schema://ticket, schema://scenario, schema://run-options, schema://run-result, runs://index, runs://{run_id}/{result,ticket,report,video,trace,console,network,chapters,har}, runs://{run_id}/screenshots/{index}.

Prompts: ticket_from_report, analyze_run, refine_target.

Configuration

All settings are environment variables with the prefix E2E_ (a .env file is read too).

Artifacts

artifacts/runs/<run_id>/
  result.json  report.md  ticket.json  options.json
  video.webm   chapters.vtt   trace.zip
  console.jsonl  network.jsonl  [network.har]  [video.mp4]
  screenshots/001_passed.png ... 003_failed.png 003_failure_fullpage.png

Open a trace with:

uv run playwright show-trace artifacts/runs/<run_id>/trace.zip

Development

uv run ruff format .
uv run ruff check .
uv run mypy src tests
uv run pytest tests/unit
uv run pytest tests/integration/test_run_ticket_broken_button.py -x --timeout=120

Integration tests start a real Chromium against a fixture site under tests/fixtures/site; run them per file. The code base follows a no-comments, no-docstrings rule that is enforced by a unit test; tool descriptions live in decorator arguments and Field(description=...).