SecureAgentBench
README.md
# AgentBench-MCP / SecureAgentBench
[](https://opensource.org/licenses/Apache-2.0)
[](https://www.python.org/downloads/)
[](https://modelcontextprotocol.io/)
[](https://gymnasium.farama.org/)
[-orange.svg)](https://ollama.com/)
> **A deterministic, sandboxed evaluation control plane and Reinforcement Learning (RL) environment for autonomous software engineering agents interacting via the Model Context Protocol (MCP).**
---
## Key Highlights
- **Zero-Cost Local Evaluation**: Built from the ground up for local LLMs via Ollama (`gemma`, `codestral`, `llama3`, `deepseek`) with resilient JSON repair handling broken markdown fences and trailing commas. Token cost: **$0.00**.
- **Multi-Ring Sandboxed Security**: Ephemeral Docker sandbox containers (`agentbench-box`), automated network air-gapping (`disconnect_network()`), path traversal defense (`SEC-006`), cgroups resource caps (1 CPU, 1GB RAM, 100 PIDs), and read-only test suite mounting.
- **20-Task Golden Fixture Catalog**: Real-world software engineering challenges across 6 core domains with visible, hidden generalization, and baseline regression test suites.
- **DMAIC Failure Taxonomy (F01–F12)**: Standardized root-cause failure classification engine diagnosing exactly why an agent failed (e.g., `F05 Flawed Logic`, `F06 Blind Patch`, `F08 Regression Degradation`).
- **Gymnasium RL Harness (`SWEGymEnv`)**: Standard OpenAI/Farama Gymnasium interface with multi-tier step and terminal rewards designed for RL fine-tuning (PPO / GRPO).
- **Engineering Economic Value (EEV)**: Quantitative business metric tracking developer hours saved, capacity value created, and deducting production defect risk penalties.
- **Live Interactive Dashboard**: High-contrast, clean web report at `http://localhost:8500` featuring expandable inspection drawers with syntax-highlighted `git diff` patches, raw `pytest` terminal logs, and step-by-step MCP tool call transcripts.
---
## System Architecture
```
┌─────────────────────────────────────────────────────────────┐
│ Autonomous Agent │
│ (Local Ollama / Open Source Model) │
└──────────────────────────────┬──────────────────────────────┘
│ JSON-RPC 2.0 (MCP Protocol)
▼
┌─────────────────────────────────────────────────────────────┐
│ MCP Protocol Server / Harness │
│ (tools: repo.*, terminal.*) │
└──────────────────────────────┬──────────────────────────────┘
│ SEC-006 Path Validation
▼
┌─────────────────────────────────────────────────────────────┐
│ Air-Gapped Docker Sandbox Container │
│ (Isolated filesystem, No Network, cgroups caps) │
│ │
│ /workspace (Ephemeral) /tests (Read-Only Mount) │
│ ├── source code ├── test_visible.py │
│ └── dependencies ├── test_hidden.py │
│ └── test_regression.py │
└──────────────────────────────┬──────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ Deterministic Evaluation Engine │
│ • Multi-Tier S_total (0-100%) │
│ • Failure Taxonomy Classifier (F01-F12) │
│ • Economic Value Calculation (EEV) │
│ • Interactive HTML Dashboard │
└─────────────────────────────────────────────────────────────┘
```
---
## 20-Task Benchmark Catalog
| ID | Domain | Name | Difficulty | Description |
| :--- | :--- | :--- | :---: | :--- |
| `bug_001` | Bug Fix | FastAPI Stock Race Condition | L2 | Threading race condition where account balances turn negative under concurrency. |
| `bug_002` | Bug Fix | TS Token Bucket Overflow | L2 | Integer arithmetic overflow in rate limiter token bucket allowing DDoS requests. |
| `bug_003` | Bug Fix | Async Deadlock Resolution | L3 | Circular lock dependency in async banking transfer service causing thread hangs. |
| `bug_004` | Bug Fix | Boundary Off-by-One | L1 | Off-by-one fencepost error causing array slice data truncation. |
| `bug_005` | Bug Fix | Memory Leak Closure | L3 | Circular closure reference preventing garbage collection in async worker processes. |
| `feat_001` | Feature | Keyset API Pagination | L2 | High-throughput cursor/keyset pagination replacing slow `OFFSET` queries. |
| `feat_002` | Feature | Idempotency Middleware | L2 | HTTP `Idempotency-Key` header cache to prevent duplicate financial mutations. |
| `feat_003` | Feature | Event Bus Pattern | L3 | Pub-Sub asynchronous event bus with error boundary isolation. |
| `feat_004` | Feature | Circuit Breaker | L2 | Fault-tolerant circuit breaker tripping OPEN after 5 failed upstream calls. |
| `repair_001` | Repair | Pytest Brittle Mock Repair | L1 | Fix broken mock assertions after internal database private method rename. |
| `repair_002` | Repair | Flaky Async Race Repair | L2 | CI/CD test failure caused by nondeterministic `asyncio.sleep` timing jitter. |
| `repair_003` | Repair | Dependency Semver Conflict | L2 | Semantic versioning mismatch where Pydantic v1 imports break under v2. |
| `perf_001` | Performance | SQL N+1 Query Elimination | L2 | Eliminates N+1 database queries using eager loading (`selectinload`). |
| `perf_002` | Performance | ReDoS Catastrophic Backtrack | L3 | Fixes regular expression that locks 100% CPU on crafted malicious input. |
| `perf_003` | Performance | Cache Stampede Defense | L3 | Prevents cache stampede thundering herds using probabilistic early expiration. |
| `refactor_001` | Refactoring | God Class Decomposition | L3 | Decomposes monolithic 800-line class into clean single-responsibility services. |
| `refactor_002` | Refactoring | Extract Strategy Pattern | L2 | Replaces nested `if/elif/else` spaghetti code with an extensible Strategy Pattern. |
| `sec_001` | Security | Path Traversal Defense | L2 | Patches `../../etc/passwd` arbitrary file read vulnerability in file server. |
| `sec_002` | Security | JWT None Algorithm Bypass | L3 | Rejects forged JWT tokens with `alg: none` header. |
| `sec_003` | Security | SSRF Metadata Filter | L3 | Blocks Server-Side Request Forgery attempts targeting AWS metadata IP. |
---
## Quickstart
### 1. Installation
```bash
git clone https://github.com/lawrenceemenike/AgentBench-MCP-SecureAgentBench.git
cd AgentBench-MCP-SecureAgentBench
python -m venv .venv
source .venv/bin/activate # On Windows: .venv\Scripts\activate
pip install -e .
```
### 2. Validate Benchmark Tasks
```bash
agentbench validate-tasks --tasks-dir=tasks
```
### 3. Launch Live Dashboard
```bash
agentbench dashboard --port=8500
```
Open **`http://localhost:8500`** in your browser to inspect evaluation results, click any task row to see the **git diff patch**, **pytest logs**, and **tool call transcripts**.
### 4. Run an Agent Evaluation
```bash
agentbench run --agent="ollama:gemma4:12b" --tasks="tasks/bug_001_fastapi_race"
```
---
## Reinforcement Learning Environment (`SWEGymEnv`)
```python
import gymnasium as gym
from src.gym.env import SWEGymEnv
env = SWEGymEnv(task_id="bug_001_fastapi_race", max_steps=20)
obs, info = env.reset()
# Step using MCP Tool Action
action = {
"name": "terminal.run_tests",
"arguments": {"test_target": "tests/test_concurrency.py"}
}
obs, reward, terminated, truncated, info = env.step(action)
print(f"Step Reward: {reward}, Terminated: {terminated}")
```
---
## License
Apache License 2.0. See `LICENSE` for details.
This server cannot be deployed
Maintenance
ActivityMaintained
ResponsivenessNo issues