Agentic API Testing

Upload an API definition, tell the app where the API lives, and test it by chatting: "run negative tests for payments", "which endpoints fail most?", "create defects for the failures".

A local LLM (Ollama) understands the request and chooses what to run. Deterministic code does the testing, so results are repeatable and the model can never invent a pass.

Example

Against the bundled sample API (which has two deliberate bugs), using llama3.2:3b on a 4 GB GPU:

You:   Run all tests
Agent: The run_id is 240ea01e-…  This run failed 2 tests out of 9.
       1. POST /payments - the server did not return a 4xx status code when omitting the 'currency' field.
       2. GET /payments/{paymentId} - schema mismatch at 'paymentId': expected a string, received an integer.

You:   Create defects for the failed tests
Agent: Two defects were created for run 240ea01e-…: one for POST /payments (required_fields check),
       one for GET /payments/{paymentId} (schema check).

Without a token, the same run explains why it fails: 4 request(s) were rejected with 401/403, so the auth token may be missing, expired or wrong.

Related MCP server: Allure TestOps MCP Server

How it works

flowchart LR
    T([Tester]) --> UI[Streamlit UI]
    UI -->|upload spec, set target| API[FastAPI backend]
    UI -->|chat| API
    API --> ING[ingest adapters]
    ING --> DB[(SQLite: catalog, targets, runs, defects)]
    API --> AG[LangGraph ReAct agent + Ollama]
    AG -->|5 tools| TOOLS[mcp_tools]
    TOOLS --> DB
    TOOLS -->|pytest subprocess| LIB[test_library]
    LIB -->|HTTP| TARGET([API under test])

1. Ingest. An uploaded file is loaded (JSON or YAML), its format is detected from the content, and an adapter converts it to a canonical API model (endpoint, method, required_fields, request_schema, response_schema). Base URL and auth scheme are read from the file to pre-fill the target form. Everything downstream uses only the canonical model.

2. Target. The tester enters the base URL and, if needed, an auth token in the UI, per project. Nothing is configured through environment variables.

3. Agent. A LangGraph ReAct agent picks among five tools based on the request.

4. Test library. run_tests runs the pytest files in test_library/ against the live API in a subprocess. Each file is one category of check, parametrized over the project's endpoints. Results are saved to SQLite.

Design decisions

Test categories

Quick start

Requirements: Python 3.12, Ollama with a tool-calling model.

ollama pull llama3.2:3b

python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install -r requirements.txt

Run three processes (three terminals):

# 1. Sample API under test. It requires the bearer token "secret" and has two seeded bugs.
$env:SAMPLE_TARGET_TOKEN = "secret"
python -m uvicorn examples.sample_target:app --port 8080

# 2. Backend
$env:APP_OLLAMA_BASE_URL = "http://localhost:11434"
python -m uvicorn backend.main:app --port 8000

# 3. Frontend
$env:BACKEND_URL = "http://localhost:8000"
python -m streamlit run frontend/streamlit_app.py

Open http://localhost:8501, then:

1. Upload examples/example_openapi.json and click Analyze Upload.

2. Open Target API. The base URL is pre-filled. Run "Run all tests" once without a token to see the 401 explanation.

3. Enter the token secret, save, and run "Run all tests" again. It finds the two seeded bugs.

Docker Compose files (docker-compose.yml) are included but have not been build-tested. From inside a container, use host.docker.internal instead of localhost for the target URL.

Agent tools

The same functions are also exposed as an MCP server (python -m mcp_tools.server) for other MCP clients. The built-in agent calls them in-process.

HTTP API

Configuration

Environment variables (prefix APP_, or a .env file). See .env.example.

Project layout

ingest/        API-definition adapters -> canonical model (OpenAPI 3.x / Swagger 2.0)
agent/         LangGraph agent and prompt
mcp_tools/     Tool functions shared by the agent and the MCP server
test_library/  Pytest categories run against the target
backend/       FastAPI app
frontend/      Streamlit UI
database/      SQLAlchemy models and session
examples/      Sample specs and a sample target API with seeded bugs
tests/         Unit tests (pytest)

Tests

pytest

tests/ covers the ingestion layer (OpenAPI 3, Swagger 2.0, nested and circular $ref, error cases). The test library itself is run on demand against a live API.

Status and limitations

This is a working prototype, not a finished product. Known gaps:

• Formats: only OpenAPI 3.x / Swagger 2.0. Postman, HAR and cURL adapters are not built.

• Extraction: allOf/oneOf/anyOf, path/query/header parameters, external $refs and per-endpoint security are not handled. Path parameters use the placeholder id 1, which causes false failures on real APIs.

• Auth: static token only. There is no login-flow option, so expiring tokens must be re-entered.

• Test coverage: three categories. No boundary, authorization, workflow or security tests yet.

• Model: a 3B model is not fully reliable at tool selection. There is no evaluation harness yet, and the agent keeps no conversation memory between messages.

• Storage: tokens are stored in plain text in the local SQLite database.

• Deployment: Docker Compose is untested; there is no CI.

Roadmap

1. Test data support (real path-parameter ids) and allOf handling.

2. Postman-collection adapter.

3. Login-flow authentication and multi-user authorization tests.

4. Failure explanation and business-rule tests generated by the LLM.

5. Evaluation harness for tool-selection accuracy across models.