qa-screens

An MCP server for visual QA during AI refactoring. It captures pages with Playwright, compares them with reference screenshots using SSIM, and returns the score, the changed regions and a labelled REFERENCE | LIVE preview image, so the agent can see what broke and fix it.

• Design conformance: compare the running site with golden or Figma screenshots.

• Refactor safety: capture before and after a change and prove it is pixel-neutral.

• A/B: compare two running servers, such as main and a worktree.

• Apps behind a login: OAuth (client credentials, password, refresh, authorization code + PKCE), bearer tokens, cookies, sessions, localStorage/sessionStorage and HTTP basic auth.

• Self-reporting: significant errors and crashes are filed as GitHub issues automatically, with secrets scrubbed. See Error reporting.

Quick start

No configuration is needed for a first run:

claude mcp add qa-screens -- uvx qa-screens

Then ask the agent: "Use qa-screens to make the current homepage the reference, then check it."

1. The first call downloads Chromium (it starts in the background as soon as the server starts).

2. update_reference(page="home", confirm=true) screenshots http://localhost:8080/home/ and saves it as screenshots/home.png. Append -mobile to the name for the mobile version.

3. run_qa compares the live site with the references from then on.

If your site runs somewhere else, the error message says so. Set base_url in .qa-screens.json (see Configure) or pass base_url to the tool.

Related MCP server: websight

Install

pip install qa-screens          # or: uv tool install qa-screens

Chromium is downloaded automatically on first use. To do it up front, run python -m playwright install chromium.

Claude Code

claude mcp add qa-screens -- uvx qa-screens

or in a project's .mcp.json:

{
  "mcpServers": {
    "qa-screens": { "command": "uvx", "args": ["qa-screens"] }
  }
}

To run the latest main without PyPI, use uvx --from git+https://github.com/astuanax/qa-screens qa-screens.

Copy skills/qa-screenshots into .claude/skills/ so the agent runs QA on its own after every UI change.

Other MCP clients

The server speaks MCP over stdio. The command is qa-screens (or python -m qa_screens), and it runs in the project directory, or wherever QA_SCREENS_ROOT points.

Configure

Put a .qa-screens.json in the project root. Every key is optional. A mistake in this file never stops the tools: bad JSON, unknown keys (with a "did you mean" hint) and invalid values fall back to the defaults, and every tool result lists them under config_warnings (the CLI prints them as warning: lines).

{
  "base_url": "http://localhost:8080",
  "references_dir": "qa/screenshots",
  "route_template": "/nl-be/{name}/",
  "routes": { "home": "/", "nl-be": "/nl-be/" },
  "threshold": 0.9,
  "mobile_device_scale_factor": 1,
  "mask_selectors": [".carousel", "[data-testid=clock]"],
  "default_profile": null
}

The environment variables QA_SCREENS_BASE_URL, QA_SCREENS_REFERENCES_DIR, QA_SCREENS_ROUTE_TEMPLATE, QA_SCREENS_THRESHOLD, QA_SCREENS_MAX_CHANGED_RATIO and QA_SCREENS_PROFILE override the file. QA_SCREENS_FIGMA_API changes the Figma API URL.

Tools

Page results are PASS, FAIL or ERROR. A page fails when its SSIM is below threshold or more than max_changed_ratio of it clearly changed colour. SSIM is a page-wide average, so a recoloured header scores about 0.99 and would pass on SSIM alone. ERROR means the comparison is meaningless: an HTTP 4xx/5xx, a navigation failure or a missing reference. The agent should fix the environment, not the CSS.

Authentication for apps

A profile is a named session that is applied to every browser context that uses it:

auth_update_profile(
  name="staging",
  origins=["https://app.staging.example.com"],
  oauth={"grant_type": "client_credentials",
         "token_url": "https://idp.example.com/oauth/token",
         "client_id": "qa-bot", "client_secret": "env:QA_CLIENT_SECRET",
         "audience": "https://api.example.com"},
  apply_token_as=["header", "local_storage:access_token"])

run_qa(profile="staging", base_url="https://app.staging.example.com")

• Secrets: any value can be "env:VAR". It is resolved from the server's environment at use time, so it never passes through the AI conversation.

• Scoping: headers and tokens go only to the profile's origins (default: the base URL's origin), never to CDNs or third parties. Cookies follow normal browser rules.

• Token placement (apply_token_as): header (Authorization: Bearer …), header:X-Api-Key, local_storage:<key>, session_storage:<key>, local_storage_json:<key> (the whole token object), cookie:<name>.

• Refresh: expired tokens are refreshed through refresh_token or by re-running the client-credentials/password grant, including mid-session in the long-lived server.

• Storage: profiles live in ~/.local/state/qa-screens/profiles/ with mode 0600, outside the project, so they are never committed. QA_SCREENS_STATE_DIR moves the location.

Error reporting

qa-screens reports its own bugs, so they reach the maintainers without a manual bug report.

• Significant errors (unexpected exceptions in a tool, not user errors like a missing reference or bad credentials) are written to ~/.local/state/qa-screens/reports/pending/ and posted in the background as a GitHub issue on astuanax/qa-screens.

• Crashes: uncaught exceptions are logged to a crash file, and hard crashes (segfault, abort) are captured by faulthandler. On the next start-up the server scans for these files and posts them.

• A report stays pending until it has been posted, so reports made offline or without a token are sent later.

• De-duplication: each error has a stable fingerprint. A repeat within 24h is not re-posted, and an open issue with the same fingerprint gets a comment instead of a new issue. At most 10 posts per day.

• Privacy: tokens, JWTs, cookies, passwords, URL query strings and credentials, and the home directory path are redacted. Tool arguments that hold secrets are dropped entirely.

Without a token nothing is posted; reports wait in pending/. qa-screens reports --flush posts them by hand.

CLI (CI-friendly)

qa-screens                       # MCP server over stdio (same as `qa-screens serve`)
qa-screens run                   # QA every reference; exit 1 on FAIL/ERROR
qa-screens run nl-be faq --viewport desktop --base-url http://localhost:8080 --json
qa-screens reports --flush       # post pending error reports

A report is written to .qa-screens/runtime/reports/latest.json on every run.

How it works

1. The reference name maps to a URL (route_template / routes). Names ending in -mobile use a mobile context (390px, touch, mobile UA).

2. The page is loaded with the HTTP cache disabled (so the CSS you just edited is what gets measured), with animations, transitions and scrollbars disabled, and after document.fonts.ready. It is captured full-page.

3. SSIM is computed per colour channel. Grayscale SSIM, as the original tool used, scores a pure hue change at 0.9996. Captures above 12MP are decoded at reduced resolution, so long pages never run out of memory. Different sizes are handled by align: resize (legacy default), crop or pad (height changes count as differences).

4. Separately, pixels whose colour clearly changed are measured on lightly blurred images, so JPEG artefacts and antialiasing don't count. Their share of the page is changed_ratio.

5. Changed pixels are grouped into region boxes. A red heatmap and a cropped side-by-side preview are written, and the preview is returned to the agent as an image.

Development

uv venv && uv pip install -e ".[dev]"
python -m playwright install chromium
pytest

The tests exercise the server the way an agent does: through an in-process MCP client, against a fake web app (login form, session cookies, OAuth provider with PKCE, basic auth) and fake GitHub and Figma APIs. Assertions are about what those services actually received.

Releases are published to PyPI by GitHub Actions when a v* tag is pushed (trusted publishing).

Links

• sus.bot

License

MIT