QAIQ

QAIQ scores the hygiene of a Playwright test suite — waits, locators, isolation, assertions, CI config — and hands back findings and questions, not a verdict.

Quickstart

npx @rafaellabarrocas/qaiq .

Running against demo/sample-suite/ — a small Playwright suite written with deliberate mistakes, included in this repo so the output above is reproducible: npx @rafaellabarrocas/qaiq demo/sample-suite.

Real output, from running QAIQ on its own repository:

  Hygiene Score  99/100

  Wait discipline     ██████████  100
  Locator quality     ··········  n/a
  Test isolation      ██████████   98
  Assertion strength  ██████████  100
  CI hygiene          ██████████  100

  FINDINGS (9)
   ~scripts/__tests__/check-ip-boundary.test.ts:78  ISO-004  Hardcoded identity data that collides in parallel
    src/core/__tests__/suppression.test.ts:25  SUP-001  Suppression without a reason
    src/core/__tests__/suppression.test.ts:34  SUP-001  Suppression without a reason
    src/core/__tests__/suppression.test.ts:39  SUP-001  Suppression without a reason
    src/core/__tests__/suppression.test.ts:44  SUP-001  Suppression without a reason
    src/core/__tests__/suppression.test.ts:61  SUP-001  Suppression without a reason
    src/io/__tests__/config.test.ts:7  ISO-001  Mutable module-level state shared across tests
    src/io/__tests__/discover.test.ts:8  ISO-001  Mutable module-level state shared across tests
    src/surfaces/mcp/__tests__/server.test.ts:15  ISO-001  Mutable module-level state shared across tests

  ~ = heuristic rule, weighted at half. Confirm before acting.

  NOT MEASURED — these need a human
   · Whether you tested the right things (risk coverage)
   · Whether your assertions match real intent (oracle correctness)
   · Whether the gaps that matter are covered (test design)
   · Whether this is safe to release (release judgment)

  Questions for a human
   ? QAIQ measures how tests are written, not what they cover. Who reviewed the test design against the product's real risks?

  QAIQ scores test-code hygiene only. A high score means your tests are well-written — not that your product is well-tested.

The five SUP-001 findings are QAIQ correctly flagging its own test fixtures: src/core/__tests__/suppression.test.ts deliberately embeds qaiq-disable-next-line comments with missing or non-substantive reasons (-- x, -- ., a 21-character run with no spaces) as literal fixture text, to prove that FIX 1's substantive-reason bar rejects them. QAIQ's suppression parser reads raw file text and has no notion of "this text is inside a JS string literal argument," so it correctly reads these as real (and correctly rejected) suppression attempts when scanning its own repository.

Locator quality reads n/a, not a perfect score, because QAIQ found zero locator calls to judge in its own source tree — see The five dimensions for why a missing denominator is never rounded up to 100.

Other ways to run it:

npx @rafaellabarrocas/qaiq . --json                # machine-readable Scorecard
npx @rafaellabarrocas/qaiq . --min-score=80         # exit 1 if the headline score is below 80

Exit codes:

Related MCP server: Autonomous QA Engineer MCP

What a QA gets out of it

• A first read of an inherited suite from one command. Point QAIQ at a suite you did not write and get a list of path:line findings, not a week of reading every spec to learn where the waits and shared state live.

• Flake causes found before they flake. Fixed sleeps, networkidle waits, hardcoded identity data that collides under parallel workers, and retry counts high enough to hide failures are all things that pass today and go red at random later. QAIQ flags them while they are still green.

• Review comments that cite a rule, not a preference. "Don't use waitForTimeout" becomes WAIT-001, with a rationale and a bad/good example from qaiq_explain_rule. The conversation moves from taste to a shared standard.

• A CI gate that tells a low score apart from a broken run. --min-score fails the build with exit 1; an unreadable config or no test files exits 2. You can alert on one and page on the other.

• An audit trail for every exception. A suppression has to carry a real reason, and every one that applied is listed in the output. When someone asks why a rule was switched off, the answer is in the scorecard.

• A check on AI-written tests. Through MCP, an agent can run qaiq_review_snippet on the test it just wrote. Because no output ever carries an approval, the agent cannot report "QAIQ says it's fine" as a green light.

• Numbers you can track. --json gives a scorecard with a score per dimension, so hygiene can be trended across releases instead of argued from memory.

• Your judgment stays yours. Every run ends by naming what it did not measure and handing back at least one question for a person. That gives you the wording for the conversation where a high score is mistaken for "well tested".

What QAIQ does not do

QAIQ scores how your tests are written. It cannot tell you:

• Whether you tested the right things (risk coverage)

• Whether your assertions match real intent (oracle correctness)

• Whether the gaps that matter are covered (test design)

• Whether this is safe to release (release judgment)

Those are judgment calls. QAIQ is built to hand them to a person, not answer them itself — which is why no output of this tool, CLI or MCP, ever returns an approval, a pass, or a green light. There is no field named ok, approved, ready, or safe anywhere in its schema, and there will not be one added. Every scorecard also carries openQuestions, and that list is never empty — even a suite that triggers zero findings still gets handed at least one question only a person can answer. A high score means your tests are well-written, not that your product is well-tested.

The five dimensions

QAIQ scores a suite across five dimensions. Each is scored independently — null ("n/a") when the suite offers zero opportunities to judge it, never coerced to a number — and the headline score is the floor of the mean of whichever dimensions actually had something to measure.

Findings are weighted by severity (error > warning > info) and confidence (high vs. heuristic, at half weight) against the number of opportunities QAIQ found for that dimension in your suite — so a suite with ten waits and one bad one is not judged the same as a suite with one wait that is bad.

Absent is not the same as inapplicable

A dimension scores n/a when the suite offers nothing to judge — an API-only suite has no locators, and pretending otherwise would punish it for a choice it was entitled to make. But n/a must never be reachable by deleting the thing being judged: an n/a dimension drops out of the headline mean, so losing a below-average dimension would lift the average. Omission would pay.

So absence of an artefact QAIQ expects is a finding, not an exemption. A Playwright suite with no config raises CI-005, keeping ci-hygiene measured — and badly — rather than letting it vanish. These whole-suite checks live in the pack's suiteRules, because no per-file rule can fire on a file that is not there.

The headline also states its own coverage whenever it is partial:

Hygiene Score  84/100  ·  4 of 5 dimensions measured

A score over four dimensions is not comparable with a score over five, and the number travels further than any caveat around it — so the caveat ships inside the scorecard (coverage: { measured, total } in --json).

The 21 rules

Generated directly from playwrightPack (src/rules/playwright/index.ts) — listed here so this table cannot drift from what the tool actually runs.

wait-discipline

• WAIT-001 — Hardcoded wait

• WAIT-002 — Manual visibility polling

• WAIT-003 — waitForLoadState('networkidle')

• WAIT-004 — Redundant waitForLoadState('domcontentloaded')

• WAIT-005 — Hand-rolled retry loop

locator-quality

• LOC-001 — CSS or XPath selector instead of a user-facing locator

• LOC-002 — Positional selector

• LOC-003 — Long literal text selector

• LOC-004 — page.$ / page.$$ element handles

test-isolation

• ISO-001 — Mutable module-level state shared across tests

• ISO-002 — Serial execution mode

• ISO-003 — Setup hook with no matching teardown

• ISO-004 — Hardcoded identity data that collides in parallel

assertion-strength

• ASR-001 — Test with no assertion

• ASR-002 — Tautological assertion

• ASR-004 — Missing await on a retrying assertion

ci-hygiene

• CI-001 — No trace or screenshot retained on failure

• CI-002 — Retry count high enough to mask flake

• CI-003 — No parallelism configured

• CI-004 — Unpinned browser image in CI

• CI-005 — No Playwright config found (whole-suite check — see below)

Each rule also ships a qaiq_explain_rule entry (see MCP) with a rationale and a bad/good example — static, generic, and identical for every caller.

Suppressions

A finding can be suppressed line by line:

// qaiq-disable-next-line WAIT-001 -- flaky third-party widget has no better hook yet
await page.waitForTimeout(2000);

The text after -- is mandatory, and it must be substantive: at least 15 characters and at least 3 whitespace-separated words. A reason that is missing, or present but not substantive (-- x, -- ., a single long word with no spaces), is treated as no reason at all: the suppression does not apply — the underlying rule still reports — and it is itself flagged as SUP-001, "Suppression without a reason." A suppression records a deliberate human decision, and a non-substantive reason gives a false sense that one was made; without a real one, nobody can tell later whether it is still valid. This is the one place in QAIQ where an agent cannot silence a finding by pattern-matching a comment: it has to write down why, and write down enough that a person plausibly thought about it.

Every suppression that actually applied is recorded, not hidden: it appears in the scorecard's suppressed array, and the CLI prints a SUPPRESSED (n) block naming each path:line, rule, and reason whenever n > 0. A score reached with suppressions in play is never indistinguishable from a genuinely clean run.

Configuration

An optional qaiq.config.json at the scanned root:

{
  "exclude": ["node_modules", "dist", "build", ".git", "coverage"],
  "minScore": 80
}

Both fields are optional; the values above are the defaults. --min-score on the command line overrides minScore from the file.

MCP setup

QAIQ also ships an MCP server exposing four tools, all critique-only:

Every one of them returns openQuestions, notMeasured, and disclaimer — none of them returns anything an agent could read as permission to proceed. qaiq_scan and qaiq_review_snippet also return findings and suppressed — every suppression that actually silenced a finding, so an agent cannot read an empty or high-scoring result as "genuinely clean" without also seeing what was suppressed to get there. None of them rewrites your code: qaiq_review_snippet critiques the snippet you send, it does not send one back.

Point an MCP-capable client at the server:

{
  "mcpServers": {
    "qaiq": {
      "command": "npx",
      "args": ["-p", "@rafaellabarrocas/qaiq", "qaiq-mcp"]
    }
  }
}

The -p @rafaellabarrocas/qaiq is required: qaiq-mcp is a secondary bin of the qaiq package, not a package name in its own right, so npx needs to be told which package to fetch it from.

Writing a rule

See docs/writing-rules.md for the Rule interface field by field, both detector kinds, and the fixture pair every rule must ship with. In short: a rule without a passing "clean" fixture is a rule with a known false positive, and will not be merged.

Licence

Apache-2.0. See LICENSE. Forks and redistributions must carry NOTICE — that is a term of the licence, not a request.