Skip to main content
Glama
lawrenceemenike

SecureAgentBench

by lawrenceemenike
OverviewSchemaSourceRelated ServersDiscussionsAdmin
Local

AgentBench-MCP / SecureAgentBench

License: Apache-2.0 Python 3.11+ Protocol: Model Context Protocol Environment: Gymnasium Evaluation: 100% Free / Ollama

A deterministic, sandboxed evaluation control plane and Reinforcement Learning (RL) environment for autonomous software engineering agents interacting via the Model Context Protocol (MCP).

Key Highlights

  • Zero-Cost Local Evaluation: Built from the ground up for local LLMs via Ollama (gemma, codestral, llama3, deepseek) with resilient JSON repair handling broken markdown fences and trailing commas. Token cost: $0.00.

  • Multi-Ring Sandboxed Security: Ephemeral Docker sandbox containers (agentbench-box), automated network air-gapping (disconnect_network()), path traversal defense (SEC-006), cgroups resource caps (1 CPU, 1GB RAM, 100 PIDs), and read-only test suite mounting.

  • 20-Task Golden Fixture Catalog: Real-world software engineering challenges across 6 core domains with visible, hidden generalization, and baseline regression test suites.

  • DMAIC Failure Taxonomy (F01–F12): Standardized root-cause failure classification engine diagnosing exactly why an agent failed (e.g., F05 Flawed Logic, F06 Blind Patch, F08 Regression Degradation).

  • Gymnasium RL Harness (SWEGymEnv): Standard OpenAI/Farama Gymnasium interface with multi-tier step and terminal rewards designed for RL fine-tuning (PPO / GRPO).

  • Engineering Economic Value (EEV): Quantitative business metric tracking developer hours saved, capacity value created, and deducting production defect risk penalties.

  • Live Interactive Dashboard: High-contrast, clean web report at http://localhost:8500 featuring expandable inspection drawers with syntax-highlighted git diff patches, raw pytest terminal logs, and step-by-step MCP tool call transcripts.

Related MCP server: Coding Tools MCP

System Architecture

┌─────────────────────────────────────────────────────────────┐
│                       Autonomous Agent                      │
│             (Local Ollama / Open Source Model)              │
└──────────────────────────────┬──────────────────────────────┘
                               │  JSON-RPC 2.0 (MCP Protocol)
                               ▼
┌─────────────────────────────────────────────────────────────┐
│                 MCP Protocol Server / Harness               │
│               (tools: repo.*, terminal.*)                   │
└──────────────────────────────┬──────────────────────────────┘
                               │  SEC-006 Path Validation
                               ▼
┌─────────────────────────────────────────────────────────────┐
│            Air-Gapped Docker Sandbox Container              │
│       (Isolated filesystem, No Network, cgroups caps)       │
│                                                             │
│   /workspace (Ephemeral)        /tests (Read-Only Mount)    │
│   ├── source code               ├── test_visible.py         │
│   └── dependencies              ├── test_hidden.py          │
│                                 └── test_regression.py      │
└──────────────────────────────┬──────────────────────────────┘
                               │
                               ▼
┌─────────────────────────────────────────────────────────────┐
│                 Deterministic Evaluation Engine             │
│   • Multi-Tier S_total (0-100%)                             │
│   • Failure Taxonomy Classifier (F01-F12)                   │
│   • Economic Value Calculation (EEV)                        │
│   • Interactive HTML Dashboard                              │
└─────────────────────────────────────────────────────────────┘

20-Task Benchmark Catalog

ID

Domain

Name

Difficulty

Description

bug_001

Bug Fix

FastAPI Stock Race Condition

L2

Threading race condition where account balances turn negative under concurrency.

bug_002

Bug Fix

TS Token Bucket Overflow

L2

Integer arithmetic overflow in rate limiter token bucket allowing DDoS requests.

bug_003

Bug Fix

Async Deadlock Resolution

L3

Circular lock dependency in async banking transfer service causing thread hangs.

bug_004

Bug Fix

Boundary Off-by-One

L1

Off-by-one fencepost error causing array slice data truncation.

bug_005

Bug Fix

Memory Leak Closure

L3

Circular closure reference preventing garbage collection in async worker processes.

feat_001

Feature

Keyset API Pagination

L2

High-throughput cursor/keyset pagination replacing slow OFFSET queries.

feat_002

Feature

Idempotency Middleware

L2

HTTP Idempotency-Key header cache to prevent duplicate financial mutations.

feat_003

Feature

Event Bus Pattern

L3

Pub-Sub asynchronous event bus with error boundary isolation.

feat_004

Feature

Circuit Breaker

L2

Fault-tolerant circuit breaker tripping OPEN after 5 failed upstream calls.

repair_001

Repair

Pytest Brittle Mock Repair

L1

Fix broken mock assertions after internal database private method rename.

repair_002

Repair

Flaky Async Race Repair

L2

CI/CD test failure caused by nondeterministic asyncio.sleep timing jitter.

repair_003

Repair

Dependency Semver Conflict

L2

Semantic versioning mismatch where Pydantic v1 imports break under v2.

perf_001

Performance

SQL N+1 Query Elimination

L2

Eliminates N+1 database queries using eager loading (selectinload).

perf_002

Performance

ReDoS Catastrophic Backtrack

L3

Fixes regular expression that locks 100% CPU on crafted malicious input.

perf_003

Performance

Cache Stampede Defense

L3

Prevents cache stampede thundering herds using probabilistic early expiration.

refactor_001

Refactoring

God Class Decomposition

L3

Decomposes monolithic 800-line class into clean single-responsibility services.

refactor_002

Refactoring

Extract Strategy Pattern

L2

Replaces nested if/elif/else spaghetti code with an extensible Strategy Pattern.

sec_001

Security

Path Traversal Defense

L2

Patches ../../etc/passwd arbitrary file read vulnerability in file server.

sec_002

Security

JWT None Algorithm Bypass

L3

Rejects forged JWT tokens with alg: none header.

sec_003

Security

SSRF Metadata Filter

L3

Blocks Server-Side Request Forgery attempts targeting AWS metadata IP.

Quickstart

1. Installation

git clone https://github.com/lawrenceemenike/AgentBench-MCP-SecureAgentBench.git
cd AgentBench-MCP-SecureAgentBench
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate
pip install -e .

2. Validate Benchmark Tasks

agentbench validate-tasks --tasks-dir=tasks

3. Launch Live Dashboard

agentbench dashboard --port=8500

Open http://localhost:8500 in your browser to inspect evaluation results, click any task row to see the git diff patch, pytest logs, and tool call transcripts.

4. Run an Agent Evaluation

agentbench run --agent="ollama:gemma4:12b" --tasks="tasks/bug_001_fastapi_race"

Reinforcement Learning Environment (SWEGymEnv)

import gymnasium as gym
from src.gym.env import SWEGymEnv

env = SWEGymEnv(task_id="bug_001_fastapi_race", max_steps=20)
obs, info = env.reset()

# Step using MCP Tool Action
action = {
    "name": "terminal.run_tests",
    "arguments": {"test_target": "tests/test_concurrency.py"}
}
obs, reward, terminated, truncated, info = env.step(action)
print(f"Step Reward: {reward}, Terminated: {terminated}")

License

Apache License 2.0. See LICENSE for details.

This server cannot be deployed

One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

Maintenance

ActivityMaintained
ResponsivenessNo issues

Resources

Related MCP Connectors

Related MCP Servers