dsh-verify
This MCP server runs real Chromium-based acceptance checks and returns deterministic PASS/FAIL verdicts with reports.
verify_spec: Runs a dsh-verify JSON spec or glob, serving a local static dir or targeting a URL, with optional URL override and headed debug mode.verify_url: Runs inline human-style checks against a live URL, including navigation, clicks, fills, waits, text/class/style/URL/navigation/console/network assertions, and screenshots.generate_and_verify: Uses an LLM to draft a QA checklist from plain-language requirements, then executes it in real Chromium; requires an API key.health: Checks that the MCP server and Chromium browser are ready.All verification outputs include per-step results and an HTML report path; reports are written to a configurable output directory.
dsh-verify
中文 | English
Witness — Check the behavior your browser spec describes. Browser assertions can expose some implementation gaps; they do not prove an app is complete or independently reviewed. (Witness is the product name;
dsh-verifyis the package name — same thing.)
If Witness catches something for you, ⭐ star the repo — it's how this project stays alive.
You asked an AI to build a web app. It said "done." Does it actually work?
dsh-verify opens a real browser and checks — so you never have to take the agent's word for it.
First success — no API key
Requires Node.js 20+ and a one-time Chromium download. Install the current npm release with npm install --save-dev dsh-verify@0.9.6, then run npx dsh-verify init. Alternatively, run from a clone checkout:
git clone https://github.com/263311487-ux/dsh-verify && cd dsh-verify
npm ci
npx playwright install chromium
node bin/verify.mjs init
cd dsh-verify-starter
node ../bin/verify.mjs --spec dsh-verify.spec.json
# PASS (5/5), exit 0 -> dsh-verify-out/report.html
node ../bin/verify.mjs --spec dsh-verify.fail.json --out failed-report
# Intentional FAIL (4/5), exit 1 -> failed-report/report.htmlOpen either HTML report with its adjacent PNG file. The failing spec deliberately expects wrong text; inspect the assertion and screenshot. Existing missing-CSS failure report is another inspectable example.
init [directory] only writes a local fixture and two JSON specs. It never calls an LLM, launches a browser, or downloads anything. The destination must be new (its parent must exist); existing directories, files, and symlinks are refused without changes. No force/overwrite option. GitHub package and browser installation require internet; these starter checks use localhost only. gen and MCP generate_and_verify are optional, separate features that require an LLM key.
To check your own app, edit the JSON steps; keep serve for static files or use base for an existing URL. serve paths resolve from the command's current directory, not the spec file. A PASS applies only to the supplied checks in that run; it does not establish complete coverage, security, accessibility, or independent authorship. The app and spec may come from the same agent; review the spec separately when independence matters. Reports retain assertion details and explicitly requested screenshots, not a complete audit trail.

The quality gate for agent-built web apps. Works with any agent — DeepSeek Harness (dsh), Claude Code, Cursor, Copilot, Codex — and with any CI. You write what a human would check in a browser; a real browser executes it and returns a PASS/FAIL verdict with receipts (screenshots + diff images).
No LLM judges the outcome. The browser is the judge.

Same task. Same AI. Two builds. One missing CSS rule — the agent's self-review passed, a real browser caught it.
Related MCP server: visual-ui-debug-agent-mcp
Why this exists
We ran a 4-agent web team (spec writer → frontend dev → QA → reviewer). Their own review said:
✅ "All requirements met. No issues found."
In a real browser, the dark-mode toggle did nothing — the .dark class was toggled, but the CSS rule was never written. Every agent self-test passed because there was nothing in the page for the agents to run. No one opened a real browser.
That's the gap: agents verify against what they believe they built, not against what a user actually experiences. These checks did not exercise the rendered CSS behavior; a browser assertion can cover it.
Build | What the agents said | What a real browser says |
| "No issues found" | ❌ FAIL — background never changes |
| one CSS rule added | ✅ PASS — theme flips |
Same page. Same JS. One missing CSS rule. Two different verdicts.
How it fits with other tools
Playwright scripts provide a general browser API. This package exposes common checks as a reviewable JSON spec and report; custom scripts remain useful for flows outside this schema.
LLM evaluation tools can assess open-ended outputs and may also integrate browsers. This package evaluates explicit browser assertions without an LLM deciding their result.
Agent browser tools let agents operate a page. Stored specs make repeat runs inspectable; independent review still depends on who writes and reviews the spec.
Visual regression tools detect pixel changes. This package combines screenshot comparisons with behavior assertions; coverage depends on the chosen checks.
Use it three ways
Entry point | What it's for | One-liner |
MCP server | Your AI agent verifies its own deliverable, mid-session |
|
CLI | You or your CI verify a build/URL |
|
GitHub Action | Every push runs real-browser checks |
|
From any AI agent (MCP)
claude mcp add dsh-verify -- npx -y -p dsh-verify dsh-verify-mcpThen tell your agent, in plain words:
Verify http://localhost:3000 — click
#dark-toggle, then checkbodybackground-color changed. Screenshot it.
Tools exposed: verify_spec (run a spec JSON), verify_url (inline checks, no files), generate_and_verify (the AI drafts the checklist, real Chromium executes it), health.
In CI (GitHub Action)
- uses: 263311487-ux/dsh-verify/.github/actions/dsh-verify@main
with:
spec: demo/fixed.json # spec file or glob
# url: https://staging.example.com # optional override
# out: dsh-verify-out # report output dir (default)The repo dogfoods it: the dogfood workflow asserts the fixed build passes and the buggy build fails on every push.
On the command line
npm install -g dsh-verify # or: npx dsh-verify
npx playwright install chromium # one-time browser download
npx dsh-verify --spec 'specs/*.json'
# [PASS] specs/home.json (5/5)
# [FAIL] specs/cart.json (4/5)
# ❌ expect_text #total: got "0" want "99"What's in the box
Deterministic assertions — Chromium, Firefox, or WebKit executes supplied checks: click, fill, text, classes, computed styles, URLs, console/network errors, and pixels. Coverage depends on the spec and browser.
Inspectable reports — HTML results accompany screenshot and difference image files when those actions run;
--jsonis available for CI. Exit 1 alone does not distinguish assertion failures from runtime errors; inspect the structured verdict and failed actions as the release checker does.Visual regression — screenshot baselines, pixel-diff with thresholds (
expect_screenshot), refresh with--update-baselines.AI-drafted checklists —
dsh-verify gen --url ... --prompt "..."learns the page in a real browser, has an LLM draft the checklist, then executes it deterministically. The AI drafts; it never judges.Multi-browser —
chromium|firefox|webkitper spec or--browser.Zero framework lock-in — a JSON spec is all there is. No config language, no SDK, no vendor.
Example spec
{
"title": "my app",
"serve": "dist",
"browser": "chromium",
"steps": [
{ "action": "goto", "path": "/index.html" },
{ "action": "click", "selector": "#count-btn", "count": 3 },
{ "action": "expect_text", "selector": "#count-btn", "text": "Clicked: 3" },
{ "action": "capture_style", "selector": "#page", "prop": "backgroundColor", "var": "bg_before" },
{ "action": "click", "selector": "#color-btn" },
{ "action": "expect_class", "selector": "#page", "class": "dark", "present": true },
{ "action": "expect_style_changed", "selector": "#page", "prop": "backgroundColor", "var": "bg_before" },
{ "action": "screenshot", "name": "final-state" }
]
}Top-level fields: title, serve (static dir) or base (target URL), browser, steps. Run many at once with a glob; exit is 0 only if all pass.
The report
An HTML report with adjacent image files — every step with a pass/fail badge, selector, and detail, plus screenshots:

Agent Arena — bring your agent
Real-browser benchmark for agent-built web apps: same 3 tasks, same human checks, open entry. Run your model on the board in ~10 minutes:
git clone https://github.com/263311487-ux/dsh-verify && cd dsh-verify
npm install && npx playwright install chromium
export LLM_API_KEY=sk-... # any OpenAI-compatible model
node arena/run.mjs --agent "gpt-5/single" --task all --repeat 1 --submitter yournameSubmit the result files for maintainer review before they appear on the leaderboard: agent-arena. Full rules in docs/ARENA.md.
Prove it (run it yourself)
git clone https://github.com/263311487-ux/dsh-verify && cd dsh-verify
npm install && npx playwright install chromium
npm run demo:fixed # → PASS (11/11)
npm run demo:buggy # → FAIL (exit 1) — the missing .dark rule, caught
npm test # engine self-testsThe repo's own CI runs exactly that — engine self-tests, then asserts fixed passes and buggy fails — so the tool verifies itself on every push.
Agent Arena — can agents ship working web apps?
Historical snapshot, not a current model ranking. The committed records span 2026-08-17–18 UTC: 48 records, 44 PASS, 2 FAIL, 2 ERROR. Among the 46 records with a browser verdict, 44 passed and 2 failed; the two ERROR records are runner failures. The records cover two model labels × two strategies × three static tasks (four records per cell). Self-check allows up to two repair rounds, so compute budgets differ. Model labels are recorded identifiers, not independently verified identities.
Records do not pin historical verifier/dependency versions. Generated apps, model replies, and local report files are not committed, limiting exact reproduction. No new LLM runs were made for the 0.9.5 or 0.9.6 release. See methodology and limitations.
See docs/ARENA.md — methodology, the tasks, and how to run your own agent.
Badge your agent-built app
The following static image is illustrative only. It is not bound to a particular app, commit, spec, or successful run:
[](https://github.com/263311487-ux/dsh-verify)For a result tied to a specific build, publish the commit, spec, action run, verdict, and report together. See docs/verified-badge.md.
Roadmap
MCP server · AI-drafted checklists · visual regression · multi-browser · GitHub Action · dsh plugin
Agent arena — a public benchmark: give the same task to different agent setups, grade them in real browsers, publish the leaderboard
Spec recorder (browser extension: click through once → spec generated)
Cloud runs + shareable report links + PR comments
Related
dsh-doublecheck — delivery quality gate for DeepSeek Harness (/gate): requirements grill + evidence discipline. Complementary pair: /gate keeps the evidence honest, dsh-verify keeps the browser honest.
falsify — the scientific thinking protocol for AI agents (hypothesis → falsify → evidence → calibrated conclusion). The pair: falsify catches the wrong conclusion, dsh-verify catches the broken output.
Featured in the DeepSeek Harness community — Show Your Plugins: dsh-verify (48-run Agent Arena results in-thread)
License
MIT
Available Tools
4 toolsgenerate_and_verifyA
AI-drafts an acceptance checklist for a URL (LLM writes it, a real browser executes it), then immediately runs it in real Chromium. Pass requirements as a plain-language description of what a human QA should verify (e.g. "dark-mode toggle changes the background color"). Requires DEEPSEEK_API_KEY or OPENAI_API_KEY in the server environment. Returns the drafted spec path plus the PASS/FAIL verdict, per-step results and report path.
Inputs: url (required), out, model, headed, +1 more.
healthA
Check that the dsh-verify MCP server and its Chromium browser are ready to run verifications.
Inputs: no named parameters.
verify_specA
Run a dsh-verify acceptance spec (JSON file, or glob) against a real headless Chromium. The spec may serve a local static dir (spec.serve) or target any URL (spec.base / step.url). Returns PASS/FAIL with per-step results and a self-contained HTML report path. Exit-code semantics: ok=true means every step passed. This is deterministic — no LLM judges the outcome.
Inputs: specPath (required), out, url, headed.
verify_urlA
Verify a live URL against a list of human-style checks, driven by a real headless Chromium. Checks are dsh-verify steps: goto/click/fill/wait/expect_text/expect_class/capture_style/expect_style_changed/expect_url_contains/expect_navigation/expect_console_errors/expect_network_errors/screenshot. If no goto step is given, a goto to the url is prepended. Returns PASS/FAIL with per-step results and a self-contained HTML report path. Deterministic — no LLM judges the outcome.
Inputs: url (required), checks (required), out, headed.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 4 tools
The tools are mostly distinct: health checks readiness, verify_spec runs a spec file/glob, verify_url runs ad-hoc checks on a URL, and generate_and_verify combines AI drafting with execution. There is slight overlap between verify_spec and verify_url, but input types clearly differentiate them.
Naming is mostly snake_case but inconsistent in style: health is a noun, verify_spec and verify_url follow verb_noun, while generate_and_verify is a compound verb phrase. The pattern is still readable but not uniform.
With 4 tools, the server is well-scoped. Each tool serves a distinct purpose in the verification workflow: readiness check, spec execution, live URL verification, and AI-assisted generation, with no redundancy.
The domain of running headless Chromium verifications is well covered: health, spec-based execution, ad-hoc checks, and AI-drafted specs. Minor gaps include no standalone spec generation without execution and no spec management, but these are not critical for the core workflow.
Maintenance
Related MCP Connectors
Browser-based QA for AI-built software. Test pages with real browsers via agents.
MCP server for Mint — AI-powered QA that runs your app in a real browser on every PR.
Write, run and check natural-language browser tests of your web app from your AI assistant.
Screenshot, diff, audit and sitemap-capture any web page — 5 MCP tools for AI agents.
Related MCP Servers
- AlicenseBqualityCmaintenanceBrowser automation MCP server with persistent memory for AI assistants, enabling automated web testing and workflow replay with self-healing selectors.543MIT
- AlicenseCqualityCmaintenanceAn MCP server that enables AI agents to autonomously test, debug, and analyze web interfaces visually using Playwright, with 30 tools for screenshots, workflows, performance, and visual comparison.3029 npm84ISC
- FlicenseNot gradedqualityDmaintenanceMCP server that enables AI agents to automate browser testing via Chromium, providing tools for navigation, interaction, and inspection.-
- AlicenseNot gradedqualityDmaintenanceMCP server for browser automation with AI agents, enabling network payload capture and QA workflows.1Apache 2.0