Vedha

Is Playwright alone enough for agentic testing?

Run this demo in about 3 seconds: npx tsx examples/demo/run.ts

Not when your agent trusts the screen. Your agent clicks "Save", the page says "Saved", and it reports success. But the server returned an error, or stored nothing. Vedha catches that. It runs on Playwright and checks that each action really happened before it reports a pass. It is built for AI agents driving a browser and for people writing tests.

npx -y @vedhajs/mcp --allow-domain localhost     # MCP server for your agent
npm install -D @vedhajs/cli                      # or the CLI / SDK for your tests
npm install -D @vedhajs/playwright-test          # guard an existing Playwright suite: change one import

Head to head: AI agent testing a web app

Same model, same tasks, written before any run on them. Playwright MCP reports success whenever the page looks right; Vedha also checks what the server did.

The lead held on every held-out set: 10 vs 4 of 14 (set 2), 15 vs 10 of 18 (set 3) and 16 vs 10 of 18 (set 4), or 41 vs 24 caught overall.

Fine print: LLM results are from one free model (minimax/minimax-m3.1-flash) at 3 trials per cell. When an expert writes every check by hand, Vedha and Playwright tie; Vedha's lead is with typical checks and with AI agents. All numbers come from raw data in benchmarks/results/, and the known misses are listed under Measured results.

Why use Vedha instead of Playwright alone

Vedha does not replace Playwright, it builds on it. It reuses Playwright's auto-waiting and actionability, and can wrap an existing page in a Playwright test (see Migrating a Playwright test).

Related MCP server: periscope-mcp

Core ideas

• Verified semantic actions. An action is verified only when framework-run, deterministic postconditions pass (URL, element state/value/text, network response and JSON fields, read-only API probes). No postcondition ⇒ unverified, never silently "passed".

• Bounded recovery that keeps the evidence. Alternate locators, known-interstitial dismissal and a single safe retry are tried in a fixed ladder; every attempt is a RecoveryRecord containing the original failure. A run that needed recovery reports passed_with_recovery.

• Uncertainty is a first-class result. For mutating actions with an unobserved outcome the framework does not re-issue the action: it reconciles via a read-only probe, or reports outcome_unknown.

• Bounded observations. controls (interactive elements only), outline (ARIA tree, refs from Playwright), region/detail on demand, deltas between observations. Page-derived text is flagged untrusted.

• Explore → promote → replay. An actor (scripted, rule-based, or your LLM adapter) explores through the same verified runner; the result becomes a reviewable TestProposal and a plain @playwright/test spec, which can be replay-checked.

• Failure bundles. Goal, steps, verdicts, recovery records, network summary, redacted trace; replay re-drives the steps and reports divergences (and states what cannot be replayed).

• Agents can make mistakes without breaking the run. Failures come back with a hint (fill on a dropdown: "use select"); the explore loop continues, but a repeat guard refuses to run the same mutating action again while its earlier outcome is unresolved, and an independent check (confirms) can clear an unknown outcome.

• Server errors cannot hide behind an optimistic page. A page-health check fails a mutating step if a request returned 5xx, a write was rejected, or the page threw, even when the declared checks passed.

• Real-app reach. Beyond click/fill/select/check/press: hover, double click, uncheck, upload (fail-closed uploadRoots), scroll, drag, wait, back, reload, tab switching, iframe targets, dialog policy.

• Usable from any MCP client. @vedhajs/mcp exposes the same engine as a stdio MCP server.

• No LLM required. The core has no model dependency; LLM actors implement a small Actor interface outside the core.

It is not a Playwright wrapper that forwards commands: it uses Playwright as a replaceable browser substrate (BrowserAdapter) and owns the action/verification/recovery/promotion/bundle layers. It does not replace Playwright Test, and it reuses Playwright's actionability and auto-waiting (the strongest part of Playwright, per research/playwright-execution-model.md).

Status: v0.1 research MVP. Every number here comes from raw trials.jsonl files in benchmarks/results/, and each held-out set was written before any run on it. Vedha ties expert Playwright when the author declares every check. It leads when checks are typical or missing, and when an LLM agent does the testing. The LLM results come from one free model (minimax/minimax-m3.1-flash) at 3 trials per cell. Known misses are listed under Measured results.

Quick start

Install from npm (Node 20 or later):

npm install -D @vedhajs/cli          # the `vedha` command
npx playwright install chromium      # browser for Playwright, if you don't have it yet
npx vedha doctor                     # checks Node and the Playwright browser

Agent skill (Claude Code, Codex, Cursor and other agents that read SKILL.md): npx skills add HMAKT99/VEDHA installs vedha-verified-browser-testing, which teaches the agent to declare checks, mark writes and never repeat an unknown outcome.

From source instead: nvm use 22 && pnpm install && npx playwright install chromium && pnpm build, then use node packages/cli/dist/bin.js wherever this README says npx vedha.

Deterministic spec (no LLM)

order.json:

{
  "goal": "place an order",
  "steps": [
    {
      "id": "open",
      "intent": "open the form",
      "op": "goto",
      "value": "/",
      "postconditions": [{ "type": "state", "target": { "label": "Name" }, "state": "visible" }]
    },
    {
      "id": "name",
      "intent": "enter name",
      "op": "fill",
      "target": { "label": "Name" },
      "value": "Ada",
      "postconditions": [{ "type": "value", "target": { "label": "Name" }, "equals": "Ada" }]
    },
    {
      "id": "submit",
      "intent": "place the order",
      "op": "click",
      "target": { "role": "button", "name": "Place order" },
      "altTargets": [{ "testId": "place-order" }],
      "dismiss": [{ "role": "button", "name": "Accept" }],
      "mutating": true,
      "postconditions": [
        { "type": "text", "target": { "role": "status" }, "contains": "Order placed" },
        {
          "type": "network",
          "method": "POST",
          "urlPattern": "/api/orders",
          "status": "2xx",
          "json": { "path": "qty", "equals": 3 }
        }
      ],
      "reconcile": { "type": "api", "get": "/api/orders", "matchesAny": { "name": "Ada" } }
    }
  ]
}

npx vedha run order.json --base-url http://localhost:3000 --out ./failure-bundle
# exit 0 passed | 1 failed | 2 blocked | 4 outcome_unknown | 5 unverified ; JSON report on stdout
npx vedha replay ./failure-bundle --base-url http://localhost:3000

SDK

npm install -D @vedhajs/adapter-playwright playwright

import { openSession } from "@vedhajs/adapter-playwright";

const session = await openSession({ allowedDomains: ["localhost"] });
const report = await session.run(steps); // ActionSpec[]
console.log(
  report.verdict,
  report.steps.map((s) => [s.actionId, s.status, s.recovered]),
);
await session.close();

Wrap an existing Playwright page: sessionFromPage(page, policy).

Agent-driven exploration → regression test

npx vedha explore goal.json --start-url http://localhost:3000/ \
  --promote order.spec.ts --check http://localhost:3000   # generates the spec and replay-checks it

The shipped RuleActor is a heuristic form-filler used for tests and benchmarks. To use an LLM, implement Actor.decide({goal, observation, history, step}) and return an ActionSpec or done; the framework enforces policy, verification and budgets regardless of what the actor proposes. No provider is bundled and nothing is sent anywhere by default.

Use it from an MCP client

Claude Code:

claude mcp add vedha -- npx -y @vedhajs/mcp --allow-domain localhost --out ./vedha-out

Any client that takes an mcpServers config (Claude Desktop, Cursor, VS Code and others):

{
  "mcpServers": {
    "vedha": {
      "command": "npx",
      "args": ["-y", "@vedhajs/mcp", "--allow-domain", "localhost", "--out", "./vedha-out"]
    }
  }
}

Run npx playwright install chromium once first if Playwright's browser is not installed. npx -y @vedhajs/mcp --help lists the options (browser, headed, uploads, secrets to redact, step cap).

Tools: vedha_act (one action with the checks you declare in expect), vedha_observe, vedha_probe (GET a same-origin JSON path without moving the page), vedha_report, vedha_promote (verified steps to a Playwright spec), vedha_bundle, vedha_close. The server refuses to start without --allow-domain HOST or an explicit --allow-any-domain. Mark data-changing steps mutating: true; after an unknown outcome Vedha refuses to repeat them until a check confirms the effect (confirms: "<step id>").

Migrating a Playwright test

One line, no rewrite. Install @vedhajs/playwright-test and change the import in your existing tests:

// before
import { test, expect } from "@playwright/test";
// after
import { test, expect } from "@vedhajs/playwright-test";

Run this demo: npx tsx examples/guard-demo/run.ts

Your tests run exactly as before, with guard mode watching each one. A test whose own checks pass still fails if the app answered a request with a 5xx, or a write request (POST/PUT/PATCH/DELETE) failed outright: the page said "Saved", the server didn't. A 4xx answer to a write and an uncaught page error are added to the report as warnings, because validation tests and third-party scripts produce them on purpose. Requests to other sites (analytics, CDNs) are not watched. The findings are attached to the test as vedha-guard.json.

test.use({ vedhaGuard: "strict" }); // 4xx writes and page errors fail too
test.use({ vedhaGuard: { ignore: ["/api/flaky", /\/v2\//], hosts: ["api.other-domain.com"] } });
test.use({ vedhaGuard: false }); // off for these tests (for example a test that expects a 500)

Checked steps. For steps that must hold, Vedha can also run the action itself and verify it:

// before
await page.getByRole("button", { name: "Save" }).click();
await expect(page.getByRole("status")).toHaveText("Saved");

// after (wraps your existing page; the rest of the test is unchanged)
const af = await sessionFromPage(page);
const r = await af.step({
  id: "save",
  intent: "save the form",
  op: "click",
  target: { role: "button", name: "Save" },
  altTargets: [{ testId: "save" }],
  postconditions: [{ type: "text", target: { role: "status" }, equals: "Saved" }],
});
expect(r.status).toBe("verified"); // r.recoveries explains any recovery that was needed

More in design/migration-and-compatibility.md.

Measured results

Everything below is from raw trials.jsonl files under benchmarks/results/; protocols, limits and amendments are in benchmarks/preregistration.md and benchmarks/methodology.md. Scripted tracks use no model. "Expert" baselines (pw-strong, sel-strong) were given every technique the scenarios' declared checks allow, including state probes.

What this does not show: that Vedha beats expert Playwright or expert Selenium when the author declares everything (it ties); anything about models beyond one free model family; anything about real production apps; Chromium-only for Selenium. Several results are mechanism-specific by design (read-back needs a conventional REST convention; the page-health check needs a request that returned an error).

Reproduce a scripted track: pnpm bench -- --set held-out --trials 20 --concurrency 3 --run-id my-run (sets: seen, held-out, held-out-minimal, independent, multi; the last two need VEDHA_TODOMVC_DIR pointing at a clone of tastejs/todomvc checked out at ff43b02; Selenium drivers are opt-in via --drivers sel-trust,sel-strong). Merge valid parts of several runs with benchmarks/analysis/merge.ts.

Repository map

Security

Page and repository content is treated as untrusted data; see SECURITY.md for the threat model and known limits (notably: Playwright route handlers do not see redirect hops - Vedha validates them itself; WebSockets are gated and service workers blocked in launched sessions; Chromium only was tested).

Contributing / License

See CONTRIBUTING.md. Licensed under Apache-2.0 (see LICENSE).