⚡ Omni-Tester: Zero-Hallucination MCP Testing Agent

Omni-Tester is a Model Context Protocol (MCP) server that equips AI agents (Claude, Cursor, Windsurf, and others) with a proof-based, anti-hallucination validation engine. Stop letting your LLM "guess" whether an endpoint works — demand programmatic proof.

The agent communicates directly with backend servers, inspecting HTTP status codes, response headers, and JSON payloads to produce deterministic, machine-verified results.

✅ Currently Implemented (Deterministic Proof)

These layers run today and return machine-verified evidence — no LLM judgment involved:

• Layer 1 — Protocol & Status Verification: Fires the request and verifies the returned HTTP status code matches your explicit expectation (e.g. 200 OK, 201 Created), flagging mismatches and 5xx crashes.

• Layer 2 — Schema & Payload Integrity: Checks the Content-Type header for application/json, parses the body as JSON, and returns a payload snapshot. When you pass an optional expected_schema, it also enforces the presence and type of each field deterministically — top-level ("id"), nested via dotted paths ("address.geo.lat"), and list indices ("items.0.id"). Supported types: int, float/number, str/string, bool, list/array, dict/object, null. (JSON true/false is not accepted as int.)

Both layers always run — a status-code failure does not suppress the payload diagnostics, so you still see the error body of a failing 4xx/5xx response.

Example with schema enforcement:

run_api_4_layer_validation(
    endpoint_url="https://jsonplaceholder.typicode.com/users/1",
    expected_status=200,
    method="GET",
    expected_schema={"id": "int", "email": "str", "address.geo.lat": "str"},
)

There is also an experimental run_ui_dom_checklist tool (Playwright) that loads a page in a headless browser and reports form counts, images missing alt tags, and console errors.

Related MCP server: QA Testing MCP Server

🗺️ Roadmap (Not Yet Implemented)

The following are planned but not currently in the code. They are documented here for transparency, not as available features:

• Layer 3 — Functional Business Logic: Use the LLM to check that returned data logically fits the scenario. (Note: this layer is LLM-based judgment, not deterministic proof.)

• Layer 4 — Security & Boundary Stressing: Inject malicious payloads (SQL injection strings, malformed headers, expired auth tokens) to verify the API fails safely.

• Deeper Layer 2 (nested type checks): ✅ Done — top-level, dotted nested paths, and list indices are all enforced. Remaining ideas: per-element validation across a whole array (e.g. "every item has an int id") and enum/format constraints.

• Sovereign API Agent: An autonomous test engineer that plans scenarios, generates Python test code, and self-heals when schemas shift.

• API Explorer Agent: Fuzzes and chains actions dynamically (e.g. pulling a token from a GET and injecting it into a PUT).

• Audit trails: Structured output to specs/api_plan.md, tests/api/, and data/reports/api_report_*.html.

📦 Installation

Using uv (recommended):

uv venv
source .venv/bin/activate        # On Windows: .venv\Scripts\activate
uv sync                          # Install dependencies from pyproject.toml

For the UI checklist tool, install the Playwright browser once:

uv run playwright install chromium

Verify the server starts (it speaks MCP over stdio and will wait for a client):

uv run omni-tester

🔌 Connecting to MCP Clients

The server is launched with uv run omni-tester. Add it to your client of choice below. If uv is not on your client's PATH, use the absolute path to the uv binary in the command field.

Claude Code (CLI)

Add it with one command:

claude mcp add --transport stdio omni-tester -- uv run omni-tester

Or add it manually to a project-scoped .mcp.json:

{
  "mcpServers": {
    "omni-tester": {
      "type": "stdio",
      "command": "uv",
      "args": ["run", "omni-tester"],
      "env": {}
    }
  }
}

Claude Desktop

Edit claude_desktop_config.json (Settings → Developer → Edit Config):

• macOS: ~/Library/Application Support/Claude/claude_desktop_config.json

• Windows: %APPDATA%\Claude\claude_desktop_config.json

{
  "mcpServers": {
    "omni-tester": {
      "command": "uv",
      "args": ["run", "omni-tester"],
      "env": {}
    }
  }
}

Restart Claude Desktop after saving.

Cursor

Edit .cursor/mcp.json (project-scoped) or ~/.cursor/mcp.json (global):

{
  "mcpServers": {
    "omni-tester": {
      "type": "stdio",
      "command": "uv",
      "args": ["run", "omni-tester"],
      "env": {}
    }
  }
}

Windsurf (Cascade)

Edit mcp_config.json (Settings → Manage MCP Servers → View raw config):

• macOS/Linux: ~/.codeium/windsurf/mcp_config.json

• Windows: %APPDATA%\Codeium\Windsurf\mcp_config.json

{
  "mcpServers": {
    "omni-tester": {
      "command": "uv",
      "args": ["run", "omni-tester"],
      "env": {}
    }
  }
}

Restart Windsurf after saving.

🛠️ Tools

Naming note: the tool is called run_api_4_layer_validation for forward compatibility, but only Layers 1–2 are implemented today (see Roadmap).

🧪 Example

Validating a live endpoint, expecting a 200:

run_api_4_layer_validation(
    endpoint_url="https://jsonplaceholder.typicode.com/users/1",
    expected_status=200,
    method="GET",
)

Produces:

Backend 4-Layer Validation Complete:
✅ Layer 1 PASS: Expected 200, received 200.
✅ Layer 2 PASS: Valid JSON payload returned.
Payload Snapshot: {'id': 1, 'name': 'Leanne Graham', ...}