QualityForge

Evidence-first browser quality automation for modern web applications.

QualityForge is an open-source foundation on top of Playwright that turns a failing run into evidence a machine can read: normalized defect artifacts with trace, screenshot and video, a quality gate with explicit thresholds, capture of console, network and page-error signals — and a read-only MCP server that hands all of it to an AI assistant without granting it the ability to act.

Status — stable, published. qualityforge@0.2.0 is on npm since 2026-10-21 (0.1.0 on 2026-10-15 was the first stable release), and the defect contract is defect.v1 at 1.6.0. The 0.x line still allows breaking changes in a minor, so pin the version you install.

Contents

• Why this exists

• What you get

• Quick start

• The demo app

• The evidence pipeline

• Defect artifacts

• The MCP server

• Configuration

• Commands

• Rules enforced in CI

• Documentation

• Contributing

Related MCP server: visual-ui-debug-agent-mcp

Why this exists

Most teams end up with a pile of one-off UI tests per project: duplicated selectors, no shared failure evidence, no cross-project view of quality. QualityForge is a reusable core instead — one configuration, one evidence format, one set of quality gates, pointed at any web application.

The differentiator is not “AI writes tests”. Playwright already ships agents for that. It is the evidence layer between a failing run and any agent: defect artifacts normalized into one schema, readable without granting the reader the ability to act.

What you get

• An evidence pipeline. Trace on first retry, screenshot and video on failure, in Chromium, Firefox and WebKit. A run that fails still publishes its artifacts.

• One normalized defect format. defect.v1, contract 1.4.0: a versioned, validated artifact per failure that extends Playwright's error-context.md rather than duplicating it.

• A quality gate. The collector splits by spec × project, refuses duplicate ids, treats an aborted suite as one outage instead of a defect per test, and exits non-zero when thresholds are violated.

• Signal capture. Console errors, uncaught page errors, failed requests and HTTP errors above 400 — redacted at the source, bounded per category, with a signals.dropped count so a truncated capture never looks complete.

• A read-only MCP server. Protocol 2026-07-28, thirteen tools, resources and a triage_failure prompt, verified against OpenCode v2.0.16 over real stdio.

• Run history. One compact file per run, so “is this a regression or has it always been this way” has an answer.

• CI that checks itself. Six legs — lint and typecheck, unit tests on Node 22 and 24, the suite on three browsers — plus a job that installs the real OpenCode client and asserts the frames that cross the wire.

Quick start

Development requires Node.js 22 or newer — Node 20 reached end-of-life in March 2026, and "engines": ">=22.0.0" says so instead of promising install politeness. CI runs Node 22 and 24, and the floor is the lower of the two. .nvmrc pins 22 for contributors: tsc dies with a segmentation fault under one Node 24.21.0 build on this machine, reproducibly. If you use a version manager, run nvm use before anything else.

Platforms. Linux is what CI runs and therefore what is verified. The core is written to be platform-neutral — path.sep and path.relative throughout, no POSIX-only calls — so Windows and macOS should work, and have not been tested. Treat them as unverified rather than supported.

git clone https://github.com/ninelegsdog/qualityforge.git
cd qualityforge
npm ci
npx playwright install --with-deps chromium
npm test

npm test starts the bundled demo app automatically, so a fresh clone runs green with no configuration and no third-party network dependency. Open the HTML report with npm run report. For a step-by-step path — seeing a real failure with evidence, collecting defect artifacts, connecting an agent, pointing the suite at your own application — follow docs/quick-start.md, or docs/quick-start.ru.md.

The demo app

fixtures/ is a small multi-page app that exists so the suite has something real to test. It is not a mock: it renders, validates and fetches.

Every control is labelled, every message has a role, and every element the tests need has a stable handle — the suite asserts on what a user perceives, not on markup structure.

The evidence pipeline

The project's core promise is that a failure leaves enough evidence to diagnose it, and that claim is tested, not asserted:

QUALITYFORGE_EVIDENCE_CHECK=1 npx playwright test tests/smoke/evidence-pipeline.spec.ts

This runs two deliberately failing tests and exits non-zero. Afterwards test-results/ contains, per failure:

error-context.md is the most interesting one: it is written as an instruction to an assistant — “explain why, be concise, respect best practices” — followed by the test name, file location, error and the diff of expectations. It is the raw material an agent needs, and it costs nothing extra.

On CI, prefer the trace viewer over screenshots and videos when debugging: it gives the DOM snapshot at each action plus the full network log.

Defect artifacts

A failing test is only useful if the failure can be read later without re-running anything. QualityForge turns each failure into one normalized, versioned artifact:

npm test                    # writes artifacts/json/playwright-results.json
npm run defects:collect     # writes artifacts/defects/<runId>/

{
  "id": "form-validation-smoke-shows-an-error-when-email-is-empty",
  "status": "failed",
  "test": { "file": "tests/smoke/form-validation.smoke.spec.ts", "line": 15 },
  "failure": {
    "message": "Error: expect(locator).toHaveText(expected) failed\n\nExpected: \"Email is required\"",
    "errorContextRef": "test-results/.../error-context.md",
  },
  "evidence": { "screenshot": "test-results/.../test-failed-1.png" },
  "context": { "commit": "43c761f", "baseUrl": "http://127.0.0.1:4311" },
  "flakiness": { "verdict": "unknown", "attempts": 1 },
}

The artifact deliberately extends Playwright's error-context.md rather than duplicating it: it records a path to that file and adds stable identity, run correlation, VCS context, evidence pointers and a flakiness verdict. The signals block — console errors, uncaught page errors, failed requests, HTTP errors above 400 — is collected live by the bundled fixture, because “element not found” is a symptom while “the page threw a TypeError” is the cause. See docs/defect-schema.md for the full contract.

Accessibility is a second, separate measurement rather than a field on that artifact: npm run a11y:audit runs axe-core against a page and writes a11y-audit.v1 — capped at ten nodes per rule with the discarded count stated, html reduced to structure because it carries page content, and target kept because without it the nodes are indistinguishable. See docs/a11y-schema.md.

npm run defects:collect doubles as a CI quality gate: it exits 0 when the gate passes, 1 when thresholds are violated, and 2 when collection could not run at all.

Installing the package

npm i qualityforge@0.2.0

latest points at this release. Pin the version you want rather than trusting a tag name anyway: npm forbids removing latest, so a tag can only ever move forward, and a bare npm i qualityforge takes whatever it points at rather than what you tested against.

A consuming project imports the fixture from the build — Playwright refuses to transpile TypeScript under node_modules, so the fixture is shipped as a subpath in exports:

import { expect, test } from "qualityforge/fixtures/quality-context.js";

That one import change is the whole integration: console, page-error and network capture become automatic. Everything captured is redacted at the source — URLs keep scheme, host and path with query strings and credentials removed, console text has tokens and JWTs replaced, sensitive header values are masked while their names are kept — and capture is bounded at 40 entries per category.

@playwright/test is a peer dependency: a consuming project brings its own, and npm 7+ installs it automatically when it is missing, because the fixture cannot run without it. The declared range is ^1.63.0; the number CI verifies is 1.63.0, pinned exactly in devDependencies, and npm run package:check fails if the package stops declaring the peer or npm stops installing it for the consumer.

The bin entries publish the two commands a consuming project runs — always through npx --no-install, which provably cannot reach the registry for somebody else's code:

npx --no-install qualityforge [flags]                          # the collector
npx --no-install qualityforge-mcp [--root <dir>] [--history <dir>]

The MCP server

npm run mcp starts a read-only MCP server over stdio, speaking protocol 2026-07-28. It hands an agent the evidence layer: what failed, where, and what the page was doing while it failed. Verified against OpenCode v2.0.16 on Linux by connecting with the block below; Kilo and MiMo are out of scope by decision — see the end of this section.

{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "qualityforge": {
      "enabled": true,
      "type": "local",
      "command": ["node", "/absolute/path/to/qualityforge/dist/mcp/index.js"],
    },
  },
}

A project that installed the package needs no path at all: ["npx", "--no-install", "qualityforge-mcp"] runs the same server out of node_modules/.bin. The command points into dist/, which npm ci builds (prepare runs tsc) — or npm run build, if the install was done with --ignore-scripts. opencode mcp list then reports the connection status; run it more than once before you believe it — the status is a sample of a connection still forming, not a verdict.

Plus resources for the latest run summary and any defect, and a triage_failure prompt that asks for facts before hypotheses.

Before you connect

• Produce the artifacts first. The server exits 1 when it cannot find an artifacts root, and the message saying so goes to stderr, which the client discards — so all you see is failed: Connection closed. Run the suite and npm run defects:collect first, or pass --root <dir>.

• Use absolute paths. A client spawns the server from your project directory, not from this checkout, so a relative root resolves somewhere that does not exist.

• Spell the runner out. The block points at dist/, so nothing transpiles at connection time: initialize was answered in ~0.24 s here, against ~2.4 s for ["npx", "tsx", …, "src/mcp/index.ts"], and OpenCode's documented default MCP timeout is 5000 ms.

• Both config forms connect. The flat mcp.<name> shown above and the V2-documented mcp.servers.<name> were both connected on v2.0.16. The difference is which file wrote it, not whether it works.

Read-only by construction

There is no write path in the server: no tool mutates anything, and the store exposes no mutating method. Read-only is structural, not a promise.

Every client-supplied path is treated as hostile, because client-side path allowlists are a convenience rather than a boundary. Absolute paths, .. traversal before and after percent-decoding, NUL bytes and symlinks resolving outside the root are all rejected — and the rejection message never echoes the filesystem layout, since it can end up in a transcript.

Why the protocol is hand-written. The official @modelcontextprotocol/sdk was checked rather than assumed: version 1.32.0 declares LATEST_PROTOCOL_VERSION = "2025-11-25" and contains no server/discover, no resultType, no ttlMs/cacheScope and no subscriptions/listen — it does not implement 2026-07-28. The 2025-11-25 handshake is still accepted, because a client that never sends initialize must still get tools/list.

Kilo and MiMo are not supported, by decision — not merely untested. The owner decided on 2026-10-03 not to pursue them: the server-side work a third client would require is better spent on the one client that is real. If you use them, nothing in this document says whether the block above is accepted.

Configuration

config/project.json declares the project name, the origin under test, the evidence policy, gate thresholds, and where artifacts are written. It is validated on load, and every problem is reported at once with the exact path:

Configuration error: Invalid configuration in /…/config/project.json
  - thresholds.maxFailureRate must be a number between 0 and 1, got "high"
  - baseUrl must not contain credentials, a query string or a fragment

Run history

An optional history block keeps one compact file per run, so the question “is this a regression, or has it always been this way” has an answer:

"history": { "directory": "quality-history", "keep": 200 }

It is committed on purpose, unlike artifacts/: a history entry is a few hundred bytes of counts and outcomes, and it is the only thing that makes a run worth having had. Each entry records which specs did not pass, and two MCP tools read it: quality_flaky_tests separates a spec that recovered from one that never did, and quality_get_trend reports a direction over a window and answers unknown below four runs. Left unset, history is off and nothing extra is written.

Commands

npm test                    # full suite
npm run test:smoke          # Chromium only
npm run test:unit           # unit tests, no browser launched
npm run test:debug          # Playwright Inspector, step through a test
npm run report              # open the HTML report
npm run report:clean        # remove generated output
npm run build               # compile src/ to dist/ (an install runs this for you)
npm run defects:collect     # build defect artifacts from the last run
npm run defects:check       # prove two report-shape rules on a real Playwright run
npm run a11y:audit          # audit a page for accessibility, into artifacts/a11y/<runId>/
npm run mcp                 # start the read-only MCP server on stdio
npm run mcp:check:all       # drive the server over real stdio and check it
npm run mcp:schema          # cross-check capabilities against the client's binary
npm run mcp:live            # connect the real opencode client and assert the wire
npm run verify              # lint + typecheck + build + format check (CI runs this first)

The Python-based checks — mcp:check*, mcp:schema, mcp:live, defects:check, docs:numbers, docs:support, docs:tools and ci:validate — are separate scripts so that verify stays runnable with only Node installed. mcp:live additionally needs the opencode binary; without it the check fails rather than skips — a live-client check that quietly passes without a client would be decoration.

Rules enforced in CI

• page.waitForTimeout() is banned by a lint rule. It is the main cause of flaky tests; wait for a real condition instead.

• test.only() and describe.only() are banned. In CI they would silently skip the rest of the suite.

• @typescript-eslint/no-floating-promises is an error. A forgotten await on a Playwright call makes a test quietly assert nothing.

• Selector policy is a lint guard, not advice: CSS, XPath and the text= shorthand in a literal locator() argument fail npm run lint. The hierarchy — role, then label, then text, data-testid last — is in docs/selectors-and-testid.md.

• Formatting and type checks must pass.

Everything in this repository follows the same rule: nothing is done until it has run in the real environment and been observed to fail for the right reason. Claims about reports, protocol frames and runner verdicts are checked against the real producers, never against a hand-written fixture.

Documentation

• docs/quick-start.md — a step-by-step path from a fresh clone to an agent reading a real failure

• docs/architecture.md — how the pieces fit together

• docs/defect-schema.md — the defect.v1 contract

• docs/a11y-schema.md — the a11y-audit.v1 contract

• docs/selectors-and-testid.md — locator policy

• docs/client-support.md — who was connected to the MCP server, and what was observed

• docs/roadmap.md — where this is going, and where it is not

• docs/parallel-work.md — how several agents work in this repository without colliding

• AGENTS.md — rules for coding agents working in this repo

• CHANGELOG.md — release history

To start a project from rather than to read: the template, https://github.com/ninelegsdog/qualityforge-template, installs this package from git, runs green from a fresh clone, and ships the MCP block already written.

Contributing

See CONTRIBUTING.md — set up, pull-request rules, test expectations, and the expectations for AI-assisted contributions. Use the issue templates for a bug report or a feature request, and the PULL_REQUEST_TEMPLATE.md when you open a pull request.

Security issues: read SECURITY.md first — do not open a public issue for them.

License

Apache-2.0 — see LICENSE. Be straightforward and assume good faith: the community standards are in CODE_OF_CONDUCT.md.

Why evidence-first?

Обычный тест-раннер говорит «failed». QualityForge говорит «почему упало» — собирает артефакты для быстрого разбора агентом или человеком.

Артефакты при падении

• Screenshot — скриншот состояния страницы в момент падения

• Trace/Video — трассировка Playwright для воспроизведения

• Logs — консольные логи и сетевые события

• HAR — сетевой трафик (опционально)

• Context — URL, viewport, шаги, ожидания

Интеграция с агентами

В комплекте read-only MCP-сервер. Агент может прочитать артефакты падения, понять контекст и предложить точечный фикс без догадок.

Used in production

QualityForge is used to verify the BotKit ecosystem of 9 production Telegram bots via botkit-farm. The MCP server enables AI agents to analyze test failures with full evidence (traces, screenshots, network logs) without direct access to modify code.

Agent Infrastructure Focus

Built as part of a broader agentic development infrastructure - harnesses, quality gates, and verification-first workflows for AI coding agents.