# MCP API Testing Agent

An AI-powered API testing agent that uses the **Model Context Protocol (MCP)**
to automate API test-case generation, execution, and failure analysis.

## What it does

1. **Discovers** endpoints from an OpenAPI/Swagger spec (via an MCP tool).
2. **Generates** positive and negative test scenarios for each endpoint using
   an LLM (LangChain + OpenAI) — valid inputs, missing required fields, wrong
   types, boundary values, auth failures, etc.
3. **Executes** each test case against the live API through MCP tools that
   send requests, validate responses, and analyze HTTP status codes.
4. **Analyzes failures** by diffing expected vs. actual responses and asking
   the LLM to explain *why* a test failed and how severe it is.
5. **Reports** results as a structured Markdown/JSON test report.

A FastAPI service wraps the whole pipeline so it can be triggered over HTTP
(`POST /agent/run`) — e.g. from CI, a scheduler, or a UI — and the MCP server
can also be run standalone and plugged into any MCP-compatible client
(Claude Desktop, Claude Code, etc.).

## Architecture

```
┌─────────────────────┐      OpenAPI spec / target base URL
│   FastAPI Service    │◄──────────────────────────────────
│   (api/main.py)      │
└──────────┬───────────┘
           │ triggers
┌──────────▼───────────┐
│   Testing Agent        │
│   (agent/*.py)         │
│                         │
│  1. TestGenerator       │──uses──► OpenAI (LangChain)
│     (positive/negative  │
│      scenarios)         │
│                         │
│  2. TestExecutor        │──calls──► MCP Client ──stdio──► MCP Server
│     (runs each case)    │                                  │
│                         │                          ┌───────┴────────┐
│  3. FailureAnalyzer     │                          │  MCP Tools:     │
│     (LLM explains diff) │                          │  - discover_    │
│                         │                          │    endpoints    │
│  4. ReportGenerator     │                          │  - send_request │
│     (md/json report)    │                          │  - validate_    │
└─────────────────────────┘                          │    response     │
                                                       │  - analyze_    │
                                                       │    status_code │
                                                       └────────┬───────┘
                                                                │ HTTP
                                                       ┌────────▼───────┐
                                                       │  Target API     │
                                                       │  (any REST API, │
                                                       │  e.g. sample_   │
                                                       │  target_api/)   │
                                                       └─────────────────┘
```

## Project layout

```
mcp-api-testing-agent/
├── mcp_server/
│   ├── server.py               # MCP server (FastMCP) exposing the 4 tools
│   └── tools/
│       ├── discover.py         # discover_endpoints — parses OpenAPI spec
│       ├── request_tool.py     # send_request — issues HTTP calls
│       ├── validate.py         # validate_response — schema/status checks
│       └── status_analyzer.py  # analyze_status_code — status code semantics
├── agent/
│   ├── mcp_client.py           # stdio MCP client used by the agent
│   ├── test_generator.py       # LLM-based positive/negative test generation
│   ├── test_executor.py        # runs generated test cases via MCP tools
│   ├── failure_analyzer.py     # LLM explains expected-vs-actual mismatches
│   └── report_generator.py     # Markdown + JSON report writer
├── api/
│   └── main.py                  # FastAPI app: POST /agent/run, GET /agent/reports/{id}
├── schemas/
│   └── models.py                # Pydantic models shared across the app
├── sample_target_api/
│   └── demo_api.py              # tiny FastAPI service to test the agent against
├── scripts/
│   └── run_agent.py             # CLI entrypoint (no FastAPI needed)
├── reports/                     # generated test reports land here
├── Dockerfile
├── docker-compose.yml
├── requirements.txt
└── .env.example
```

## Setup

```bash
python -m venv venv && source venv/bin/activate
pip install -r requirements.txt
cp .env.example .env   # add your OPENAI_API_KEY
```

## Run the demo target API (a small sample API to test against)

```bash
uvicorn sample_target_api.demo_api:app --port 9000
```

This exposes a toy "Task Manager" API with `/tasks` CRUD endpoints and a
generated OpenAPI spec at `http://localhost:9000/openapi.json`.

## Run the agent via CLI

```bash
python scripts/run_agent.py --spec http://localhost:9000/openapi.json --base-url http://localhost:9000
```

This will generate test cases, execute them, analyze any failures, and write
a report to `reports/report_<timestamp>.md` and `.json`.

## Run the agent as an HTTP service

```bash
uvicorn api.main:app --port 8000
```

```bash
curl -X POST http://localhost:8000/agent/run \
  -H "Content-Type: application/json" \
  -d '{"spec_url": "http://localhost:9000/openapi.json", "base_url": "http://localhost:9000"}'
```

## Run the MCP server standalone

To plug the tools into an MCP-compatible client (Claude Desktop, Claude
Code, etc.) instead of the built-in agent:

```bash
python -m mcp_server.server
```

Then add it to your MCP client config, e.g. for Claude Desktop
(`claude_desktop_config.json`):

```json
{
  "mcpServers": {
    "api-testing-agent": {
      "command": "python",
      "args": ["-m", "mcp_server.server"],
      "cwd": "/absolute/path/to/mcp-api-testing-agent"
    }
  }
}
```

## Run everything with Docker

```bash
docker compose up --build
```

This starts the demo target API, the MCP-backed testing agent FastAPI
service, and mounts `./reports` so generated reports are available on the
host.

## Sample report output

```
# API Test Report — 2026-02-03T10:15:00

Target: http://localhost:9000
Total: 18   Passed: 15   Failed: 3   Pass rate: 83%

## Failures

### POST /tasks — missing required field "title" (negative test)
Expected: 422 Unprocessable Entity
Actual:   500 Internal Server Error
Analysis: The endpoint does not validate the request body before hitting the
database layer, so a missing "title" causes an unhandled exception instead
of a client-error response. Severity: High — indicates missing input
validation.
```

## Notes on adapting this to a real project

- Swap `sample_target_api/` for your real service, or point `--spec` /
  `spec_url` at any live OpenAPI/Swagger JSON endpoint.
- `test_generator.py`'s prompt can be extended with domain rules (e.g.
  required auth headers, rate limits, tenant IDs).
- For CI, run `scripts/run_agent.py` as a pipeline step and fail the build
  if `report["summary"]["failed"] > 0`.