# ⚡ Omni-Tester: Zero-Hallucination MCP Testing Agent

Omni-Tester is a [Model Context Protocol (MCP)](https://modelcontextprotocol.io)
server that equips AI agents (Claude, Cursor, Windsurf, and others) with a
**proof-based, anti-hallucination validation engine**. Stop letting your LLM
"guess" whether an endpoint works — demand programmatic proof.

The agent communicates directly with backend servers, inspecting **HTTP status
codes, response headers, and JSON payloads** to produce deterministic,
machine-verified results.

---

## ✅ Currently Implemented (Deterministic Proof)

These layers run today and return machine-verified evidence — **no LLM
judgment involved**:

- **Layer 1 — Protocol & Status Verification:** Fires the request and verifies
  the returned HTTP status code matches your explicit expectation (e.g. `200 OK`,
  `201 Created`), flagging mismatches and `5xx` crashes.
- **Layer 2 — Schema & Payload Integrity:** Checks the `Content-Type` header for
  `application/json`, parses the body as JSON, and returns a payload snapshot.
  When you pass an optional `expected_schema`, it also **enforces the presence
  and type of each field deterministically** — top-level (`"id"`), nested via
  dotted paths (`"address.geo.lat"`), and list indices (`"items.0.id"`).
  Supported types: `int`, `float`/`number`, `str`/`string`, `bool`,
  `list`/`array`, `dict`/`object`, `null`. (JSON `true`/`false` is *not*
  accepted as `int`.)

Both layers **always run** — a status-code failure does not suppress the payload
diagnostics, so you still see the error body of a failing `4xx`/`5xx` response.

Example with schema enforcement:

```
run_api_4_layer_validation(
    endpoint_url="https://jsonplaceholder.typicode.com/users/1",
    expected_status=200,
    method="GET",
    expected_schema={"id": "int", "email": "str", "address.geo.lat": "str"},
)
```

There is also an experimental **`run_ui_dom_checklist`** tool (Playwright) that
loads a page in a headless browser and reports form counts, images missing
`alt` tags, and console errors.

---

## 🗺️ Roadmap (Not Yet Implemented)

The following are planned but **not currently in the code**. They are documented
here for transparency, not as available features:

- **Layer 3 — Functional Business Logic:** Use the LLM to check that returned
  data logically fits the scenario. *(Note: this layer is LLM-based judgment, not
  deterministic proof.)*
- **Layer 4 — Security & Boundary Stressing:** Inject malicious payloads (SQL
  injection strings, malformed headers, expired auth tokens) to verify the API
  fails safely.
- **Deeper Layer 2 (nested type checks):** ✅ *Done* — top-level, dotted
  nested paths, and list indices are all enforced. Remaining ideas: per-element
  validation across a whole array (e.g. "every item has an `int` id") and
  enum/format constraints.
- **Sovereign API Agent:** An autonomous test engineer that plans scenarios,
  generates Python test code, and self-heals when schemas shift.
- **API Explorer Agent:** Fuzzes and chains actions dynamically (e.g. pulling a
  token from a `GET` and injecting it into a `PUT`).
- **Audit trails:** Structured output to `specs/api_plan.md`, `tests/api/`, and
  `data/reports/api_report_*.html`.

---

## 📦 Installation

Using [`uv`](https://docs.astral.sh/uv/) (recommended):

```bash
uv venv
source .venv/bin/activate        # On Windows: .venv\Scripts\activate
uv sync                          # Install dependencies from pyproject.toml
```

For the UI checklist tool, install the Playwright browser once:

```bash
uv run playwright install chromium
```

Verify the server starts (it speaks MCP over stdio and will wait for a client):

```bash
uv run omni-tester
```

---

## 🔌 Connecting to MCP Clients

The server is launched with `uv run omni-tester`. Add it to your client of
choice below. If `uv` is not on your client's `PATH`, use the absolute path to
the `uv` binary in the `command` field.

### Claude Code (CLI)

Add it with one command:

```bash
claude mcp add --transport stdio omni-tester -- uv run omni-tester
```

Or add it manually to a project-scoped `.mcp.json`:

```json
{
  "mcpServers": {
    "omni-tester": {
      "type": "stdio",
      "command": "uv",
      "args": ["run", "omni-tester"],
      "env": {}
    }
  }
}
```

### Claude Desktop

Edit `claude_desktop_config.json` (Settings → Developer → Edit Config):

- **macOS:** `~/Library/Application Support/Claude/claude_desktop_config.json`
- **Windows:** `%APPDATA%\Claude\claude_desktop_config.json`

```json
{
  "mcpServers": {
    "omni-tester": {
      "command": "uv",
      "args": ["run", "omni-tester"],
      "env": {}
    }
  }
}
```

Restart Claude Desktop after saving.

### Cursor

Edit `.cursor/mcp.json` (project-scoped) or `~/.cursor/mcp.json` (global):

```json
{
  "mcpServers": {
    "omni-tester": {
      "type": "stdio",
      "command": "uv",
      "args": ["run", "omni-tester"],
      "env": {}
    }
  }
}
```

### Windsurf (Cascade)

Edit `mcp_config.json` (Settings → Manage MCP Servers → View raw config):

- **macOS/Linux:** `~/.codeium/windsurf/mcp_config.json`
- **Windows:** `%APPDATA%\Codeium\Windsurf\mcp_config.json`

```json
{
  "mcpServers": {
    "omni-tester": {
      "command": "uv",
      "args": ["run", "omni-tester"],
      "env": {}
    }
  }
}
```

Restart Windsurf after saving.

---

## 🛠️ Tools

| Tool                          | Status         | Purpose                                                        |
| ----------------------------- | -------------- | ------------------------------------------------------------- |
| `run_api_4_layer_validation`  | ✅ Layers 1–2  | Verifies HTTP status, `Content-Type`, JSON validity, and optional per-field type/schema (nested + list paths) |
| `run_ui_dom_checklist`        | 🧪 Experimental | Playwright DOM checklist (forms, alt tags, console errors)     |

> **Naming note:** the tool is called `run_api_4_layer_validation` for forward
> compatibility, but only Layers 1–2 are implemented today (see Roadmap).

---

## 🧪 Example

Validating a live endpoint, expecting a `200`:

```
run_api_4_layer_validation(
    endpoint_url="https://jsonplaceholder.typicode.com/users/1",
    expected_status=200,
    method="GET",
)
```

Produces:

```
Backend 4-Layer Validation Complete:
✅ Layer 1 PASS: Expected 200, received 200.
✅ Layer 2 PASS: Valid JSON payload returned.
Payload Snapshot: {'id': 1, 'name': 'Leanne Graham', ...}
```