# Agentic API Testing

Upload an API definition, tell the app where the API lives, and test it by chatting: *"run negative tests for payments"*, *"which endpoints fail most?"*, *"create defects for the failures"*.

A local LLM (Ollama) understands the request and chooses what to run. **Deterministic code does the testing**, so results are repeatable and the model can never invent a pass.

## Example

Against the bundled sample API (which has two deliberate bugs), using `llama3.2:3b` on a 4 GB GPU:

```text
You:   Run all tests
Agent: The run_id is 240ea01e-…  This run failed 2 tests out of 9.
       1. POST /payments - the server did not return a 4xx status code when omitting the 'currency' field.
       2. GET /payments/{paymentId} - schema mismatch at 'paymentId': expected a string, received an integer.

You:   Create defects for the failed tests
Agent: Two defects were created for run 240ea01e-…: one for POST /payments (required_fields check),
       one for GET /payments/{paymentId} (schema check).
```

Without a token, the same run explains *why* it fails: `4 request(s) were rejected with 401/403, so the auth token may be missing, expired or wrong.`

## How it works

```mermaid
flowchart LR
    T([Tester]) --> UI[Streamlit UI]
    UI -->|upload spec, set target| API[FastAPI backend]
    UI -->|chat| API
    API --> ING[ingest adapters]
    ING --> DB[(SQLite: catalog, targets, runs, defects)]
    API --> AG[LangGraph ReAct agent + Ollama]
    AG -->|5 tools| TOOLS[mcp_tools]
    TOOLS --> DB
    TOOLS -->|pytest subprocess| LIB[test_library]
    LIB -->|HTTP| TARGET([API under test])
```

1. **Ingest.** An uploaded file is loaded (JSON or YAML), its format is detected from the content, and an adapter converts it to a *canonical API model* (`endpoint`, `method`, `required_fields`, `request_schema`, `response_schema`). Base URL and auth scheme are read from the file to pre-fill the target form. Everything downstream uses only the canonical model.
2. **Target.** The tester enters the base URL and, if needed, an auth token in the UI, per project. Nothing is configured through environment variables.
3. **Agent.** A LangGraph ReAct agent picks among five tools based on the request.
4. **Test library.** `run_tests` runs the pytest files in `test_library/` against the live API in a subprocess. Each file is one *category* of check, parametrized over the project's endpoints. Results are saved to SQLite.

### Design decisions

| Decision | Why |
|---|---|
| The LLM routes and summarizes; code tests | A small local model is unreliable at generating and judging tests. Deterministic checks are repeatable. |
| `project_id` is injected by the backend, not requested from the model | Small models forget or invent arguments. Tools never ask the LLM for what the system already knows. |
| Tokens never reach the LLM | They are entered in the UI, stored by the backend, and passed to the test subprocess through a temp file, not argv or chat. |
| Adapter-based ingestion | A new input format is one small adapter that emits the canonical model. The agent and tests don't change. |
| Failure hints computed in code | If requests get 401/403, the tool result says so; the model only relays it. |
| Negative tests must be conclusive | A "missing field returns 4xx" test is skipped as *inconclusive* unless the complete request succeeds first, so a 401 can't count as a pass. |
| Categories are files | Drop `test_<name>.py` into `test_library/` and the agent tool discovers it. |

## Test categories

| Category | Checks |
|---|---|
| `status_codes` | A valid request returns 2xx |
| `schema` | A successful response body matches the response schema |
| `required_fields` | Omitting each required field returns 4xx (negative tests) |

## Quick start

Requirements: Python 3.12, [Ollama](https://ollama.com) with a tool-calling model.

```powershell
ollama pull llama3.2:3b

python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install -r requirements.txt
```

Run three processes (three terminals):

```powershell
# 1. Sample API under test. It requires the bearer token "secret" and has two seeded bugs.
$env:SAMPLE_TARGET_TOKEN = "secret"
python -m uvicorn examples.sample_target:app --port 8080

# 2. Backend
$env:APP_OLLAMA_BASE_URL = "http://localhost:11434"
python -m uvicorn backend.main:app --port 8000

# 3. Frontend
$env:BACKEND_URL = "http://localhost:8000"
python -m streamlit run frontend/streamlit_app.py
```

Open http://localhost:8501, then:

1. Upload `examples/example_openapi.json` and click **Analyze Upload**.
2. Open **Target API**. The base URL is pre-filled. Run *"Run all tests"* once without a token to see the 401 explanation.
3. Enter the token `secret`, save, and run *"Run all tests"* again. It finds the two seeded bugs.

Docker Compose files (`docker-compose.yml`) are included but have not been build-tested. From inside a container, use `host.docker.internal` instead of `localhost` for the target URL.

## Agent tools

| Tool | Purpose |
|---|---|
| `list_apis` | Show the endpoints of the uploaded spec |
| `run_tests(category, endpoint_filter)` | Run the test library and save results |
| `get_results(run_id)` | Show a saved run (latest if empty) |
| `create_defects(run_id)` | Create defect records from a run's failures |
| `get_failure_trends` | Which endpoints fail most across runs |

The same functions are also exposed as an MCP server (`python -m mcp_tools.server`) for other MCP clients. The built-in agent calls them in-process.

## HTTP API

| Endpoint | Purpose |
|---|---|
| `POST /upload` | Upload a spec; returns project id, detected format, target hints |
| `GET/PUT /projects/{id}/target` | Read or set the base URL and token (the token is never returned) |
| `POST /agent/chat` | Send a message to the agent |
| `GET /projects`, `GET /projects/{id}/apis` | List projects and their canonical APIs |
| `GET /history`, `GET /reports` | Run history and failure trends |

## Configuration

Environment variables (prefix `APP_`, or a `.env` file). See `.env.example`.

| Variable | Default |
|---|---|
| `APP_OLLAMA_BASE_URL` | `http://host.docker.internal:11434` |
| `APP_OLLAMA_MODEL` | `llama3.2:3b` |
| `APP_OLLAMA_NUM_CTX` | `4096` |
| `APP_DATABASE_URL` | `sqlite+aiosqlite:///./data/api_testing.db` |

## Project layout

```text
ingest/        API-definition adapters -> canonical model (OpenAPI 3.x / Swagger 2.0)
agent/         LangGraph agent and prompt
mcp_tools/     Tool functions shared by the agent and the MCP server
test_library/  Pytest categories run against the target
backend/       FastAPI app
frontend/      Streamlit UI
database/      SQLAlchemy models and session
examples/      Sample specs and a sample target API with seeded bugs
tests/         Unit tests (pytest)
```

## Tests

```powershell
pytest
```

`tests/` covers the ingestion layer (OpenAPI 3, Swagger 2.0, nested and circular `$ref`, error cases). The test library itself is run on demand against a live API.

## Status and limitations

This is a working prototype, not a finished product. Known gaps:

- **Formats:** only OpenAPI 3.x / Swagger 2.0. Postman, HAR and cURL adapters are not built.
- **Extraction:** `allOf`/`oneOf`/`anyOf`, path/query/header parameters, external `$ref`s and per-endpoint `security` are not handled. Path parameters use the placeholder id `1`, which causes false failures on real APIs.
- **Auth:** static token only. There is no login-flow option, so expiring tokens must be re-entered.
- **Test coverage:** three categories. No boundary, authorization, workflow or security tests yet.
- **Model:** a 3B model is not fully reliable at tool selection. There is no evaluation harness yet, and the agent keeps no conversation memory between messages.
- **Storage:** tokens are stored in plain text in the local SQLite database.
- **Deployment:** Docker Compose is untested; there is no CI.

## Roadmap

1. Test data support (real path-parameter ids) and `allOf` handling.
2. Postman-collection adapter.
3. Login-flow authentication and multi-user authorization tests.
4. Failure explanation and business-rule tests generated by the LLM.
5. Evaluation harness for tool-selection accuracy across models.