MCP Agent System
by Pasukalu
README.md
# MCP Agent System
> Production-grade MCP Server with Agent Tool Orchestration — a Model Context Protocol implementation featuring 6 sandboxed tools, an agent orchestration engine, and an evaluation suite with CI/CD quality gates.
---
## Table of Contents
- [Overview](#overview)
- [Architecture](#architecture)
- [Features](#features)
- [Project Structure](#project-structure)
- [Quick Start](#quick-start)
- [API Endpoints](#api-endpoints)
- [Tools](#tools)
- [Evaluation Suite](#evaluation-suite)
- [Configuration](#configuration)
- [Docker Deployment](#docker-deployment)
- [CI/CD Pipeline](#cicd-pipeline)
- [Testing](#testing)
- [Tech Stack](#tech-stack)
- [License](#license)
---
## Overview
MCP Agent System is a production-grade implementation of the [Model Context Protocol](https://modelcontextprotocol.io/) (MCP) with a built-in agent orchestration engine. It bridges LLM agents and executable tools through a standardized protocol layer, providing a safe and observable environment for autonomous task completion.
The system operates in two modes:
- **Stub mode** (default): Deterministic, offline, CI-friendly. Tool selection uses keyword matching — no external API calls or LLM dependencies.
- **Real mode**: LLM-powered agent with real tool execution (configurable via environment variables).
---
## Architecture
The system is organized into four core layers:
```
┌─────────────────────────────────────────────────────────────────┐
│ FastAPI Dashboard Layer │
│ (REST API + real-time tool call monitoring) │
├─────────────────────────────────────────────────────────────────┤
│ Agent Orchestration Engine │
│ Plan ──► Tool Selection ──► Execution ──► Verification │
│ (StubPlanner keyword matching / LLM mode) │
├─────────────────────────────────────────────────────────────────┤
│ MCP Protocol Layer │
│ Tools (actions) │ Resources (context) │ Prompts │
│ (server.py — registration, invocation, lifecycle) │
├─────────────────────────────────────────────────────────────────┤
│ Tool Implementation Layer │
│ ┌──────────┬────────────┬───────────┬────────────┬───────────┐ │
│ │Calculator│Code Executor│Web Searcher│File Manager│Data Analyzer│ │
│ └──────────┴────────────┴───────────┴────────────┴───────────┘ │
│ ┌──────────┐ │
│ │Git Helper│ + AST-validated sandbox safety │
│ └──────────┘ │
├─────────────────────────────────────────────────────────────────┤
│ Evaluation Suite │
│ Tool Selection Accuracy │ Task Completion │ Quality Score │
│ Pass-Rate Gate │ Cost & Latency Tracking │
└─────────────────────────────────────────────────────────────────┘
```
### MCP Protocol Layer
Implements the three MCP primitives:
- **Tools** — Executable actions the agent can invoke (POST-like semantics). Each tool has input/output JSON schemas and a safety flag.
- **Resources** — Read-only context data the agent can reference (GET-like semantics), identified by URI.
- **Prompts** — Reusable prompt templates with parameterized arguments.
### 6 Tools
Six fully sandboxed tools covering common agent task domains: calculation, code execution, web search, file management, data analysis, and Git operations.
### Agent Engine
Follows a **Plan → Execute → Verify** loop:
1. **Plan** — Analyze the task and select the appropriate tool via keyword scoring (stub) or LLM reasoning (real mode).
2. **Execute** — Call the selected tool with inferred arguments through the MCP server.
3. **Verify** — Check whether the result satisfies the task; retry with a fallback strategy if needed.
Safety limits enforced: max iterations per task, max total tool calls, and per-tool execution timeout.
### Evaluation Suite
Automated evaluation measuring tool selection accuracy, task completion rate, quality scores, latency, and cost — with a configurable pass-rate threshold that gates CI/CD.
---
## Features
- **MCP Protocol Compliant** — Full implementation of Tools, Resources, and Prompts primitives with registration, listing, and invocation.
- **6 Sandboxed Tools** — Calculator, code executor, web searcher, file manager, data analyzer, and Git helper.
- **AST-Validated Safety** — Code execution and calculation tools validate the AST before execution, blocking imports, dunder access, and dangerous builtins.
- **Agent Orchestration** — Plan → Execute → Verify loop with deterministic stub planner and safety limits.
- **Dual-Mode Operation** — Stub mode for CI-friendly deterministic runs; real mode for LLM-powered autonomy.
- **Evaluation Suite** — Automated quality gates with tool selection accuracy, task completion rate, and quality scoring.
- **Cost & Latency Tracking** — Every tool call records latency and estimated cost for observability.
- **Virtual Environments** — File manager uses an in-memory virtual filesystem; Git helper operates on a virtual repository — no host system access.
- **FastAPI Dashboard** — REST API for server stats, tool listing, and real-time tool call monitoring.
- **Docker Ready** — Production Dockerfile with slim Python image.
- **CI/CD Integrated** — GitHub Actions workflow with ruff linting, pytest, and eval gate enforcement.
---
## Project Structure
```
mcp-agent-system/
├── app/
│ ├── __init__.py # Package metadata
│ ├── config.py # Pydantic settings (env-based configuration)
│ ├── models.py # Pydantic data models (MCP + Agent + Eval)
│ ├── server/
│ │ ├── __init__.py
│ │ └── server.py # MCPServer — protocol layer implementation
│ ├── agent/
│ │ ├── __init__.py
│ │ └── engine.py # Agent + StubPlanner — orchestration engine
│ └── tools/
│ ├── __init__.py
│ ├── registry.py # Tool registration with the MCP server
│ ├── calculator.py # AST-validated math expression evaluator
│ ├── code_executor.py # Sandboxed Python code executor
│ ├── web_searcher.py # Simulated web search with knowledge base
│ ├── file_manager.py # Virtual in-memory filesystem manager
│ ├── data_analyzer.py # CSV data analysis (stats/filter/sort/aggregate)
│ └── git_helper.py # Virtual Git repository operations
├── eval/ # Evaluation suite and task datasets
├── scripts/ # CLI entry points (run_eval, etc.)
├── tests/ # pytest test suite
├── .github/
│ └── workflows/
│ └── ci.yml # CI workflow (lint + test + eval gate)
├── pyproject.toml # Build config, ruff & pytest settings
├── requirements.txt # Production dependencies
├── requirements-dev.txt # Development dependencies
├── Dockerfile # Container build definition
├── .dockerignore
├── .gitignore
└── README.md
```
---
## Quick Start
### Prerequisites
- Python 3.10 or higher
- pip (or your preferred package manager)
### Installation
```bash
# Clone the repository
git clone https://github.com/your-username/mcp-agent-system.git
cd mcp-agent-system
# Create and activate a virtual environment
python -m venv .venv
source .venv/bin/activate # On Windows: .venv\Scripts\activate
# Install production dependencies
pip install -r requirements.txt
# Or install with development dependencies
pip install -r requirements-dev.txt
```
### Run the Server
```bash
# Start the FastAPI server (default: 0.0.0.0:8000)
uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload
# Or run with custom configuration via environment variables
TRANSPORT=sse AGENT_MODE=stub PORT=8000 uvicorn app.main:app --host 0.0.0.0 --port 8000
```
The server will be available at `http://localhost:8000`. Interactive API docs are available at `http://localhost:8000/docs` (Swagger UI).
### Run the CLI
```bash
# Run the evaluation suite with default settings
python -m scripts.run_eval
# Run with custom parameters
python -m scripts.run_eval --max-tasks 30 --threshold 0.7
```
---
## API Endpoints
| Method | Endpoint | Description |
|--------|-----------------------|----------------------------------------------------------|
| GET | `/` | Health check and server info. |
| GET | `/tools` | List all registered MCP tools with their schemas. |
| POST | `/tools/call` | Invoke a tool by name with arguments. |
| GET | `/resources` | List all registered MCP resources. |
| GET | `/resources/{uri}` | Read a resource by URI. |
| GET | `/prompts` | List all registered MCP prompt templates. |
| GET | `/prompts/{name}` | Render a prompt template with arguments. |
| POST | `/agent/run` | Submit a task to the agent orchestration engine. |
| GET | `/stats` | Get server statistics (call count, success rate, etc.). |
| GET | `/health` | Liveness probe for container orchestration. |
---
## Tools
| Tool | Description | Key Operations / Features | Safety |
|------------------|-----------------------------------------------------------------------------|------------------------------------------------------------------------|---------------------------------|
| `calculator` | Evaluate mathematical expressions safely. | `+`, `-`, `*`, `/`, `**`, `%`, `//`, `abs`, `round`, `min`, `max`, `pi`, `e` | AST-validated; blocks non-math nodes |
| `code_executor` | Execute Python code in a sandboxed environment. | Arithmetic, string ops, loops, functions, comprehensions, lambdas | AST-validated; blocks imports, dunder access, `exec`/`eval`/`open` |
| `web_searcher` | Search the web for information (simulated in stub mode). | Keyword-matched results from a built-in knowledge base | Read-only; no network in stub mode |
| `file_manager` | Manage files in a virtual in-memory filesystem. | `read`, `write`, `list`, `delete`, `exists` | Virtual FS; blocks directory traversal (`..`) |
| `data_analyzer` | Analyze CSV-formatted data. | `stats`, `filter`, `sort`, `aggregate` (count/sum/avg/min/max) | Read-only computation; no side effects |
| `git_helper` | Perform Git operations on a virtual repository. | `status`, `log`, `diff`, `branch`, `commit` | Virtual repo; no host Git access |
---
## Evaluation Suite
The evaluation suite runs a set of predefined tasks through the agent and measures performance across multiple dimensions. It serves as a quality gate in CI/CD.
### Metrics
| Metric | Description | Range |
|-----------------------------|-----------------------------------------------------------------------------|-------------|
| **Tool Selection Accuracy** | Fraction of tasks where the agent selected the correct (`expected_tool`) tool. | 0.0 – 1.0 |
| **Task Completion Rate** | Fraction of tasks where the agent produced a valid, non-error answer. | 0.0 – 1.0 |
| **Quality Score** | Per-task quality score based on output correctness and completeness. | 0.0 – 1.0 |
| **Pass Rate** | Fraction of tasks with `quality_score >= quality_threshold`. | 0.0 – 1.0 |
| **Avg Latency** | Average total latency per task (milliseconds). | ms |
| **Avg Cost** | Average estimated cost per task (USD), based on token and tool-call costs. | USD |
| **Avg Tool Calls** | Average number of tool calls per task. | count |
### Quality Gate
The eval gate enforces a minimum pass rate before allowing deployment:
```bash
# Pass if pass_rate >= threshold (default: 0.7)
python -m scripts.run_eval --max-tasks 30 --threshold 0.7
```
If the pass rate falls below the threshold, the command exits with a non-zero status code, failing the CI pipeline.
### Recommendation Levels
The evaluation summary produces a recommendation based on the pass rate:
| Pass Rate | Recommendation | Action |
|---------------|----------------------|------------------------------------------|
| `>= 0.8` | `production_ready` | Safe to deploy. |
| `0.7 – 0.8` | `needs_work` | Deployable but improvements recommended. |
| `< 0.7` | `investigate` | Do not deploy; investigate failures. |
---
## Configuration
All configuration is handled via environment variables (with `.env` file support) using `pydantic-settings`. Sensible offline defaults are provided for stub mode.
| Variable | Default | Description |
|-----------------------------|--------------------------------------|-------------------------------------------------------------------|
| `AGENT_MODE` | `stub` | Agent mode: `stub` (deterministic) or `real` (LLM-powered). |
| `TRANSPORT` | `stdio` | MCP transport: `stdio`, `sse`, or `streamable_http`. |
| `HOST` | `0.0.0.0` | Server bind host. |
| `PORT` | `8000` | Server bind port. |
| `MAX_ITERATIONS` | `10` | Maximum plan-execute-verify iterations per task. |
| `MAX_TOOL_CALLS` | `20` | Maximum total tool calls per task. |
| `TIMEOUT_SECONDS` | `30` | Per-tool execution timeout in seconds. |
| `ENABLE_SANDBOX` | `True` | Enable AST-validated code execution sandbox. |
| `BLOCKED_IMPORTS` | `["os","subprocess","shutil","ctypes"]` | Python modules blocked in the code executor. |
| `BLOCKED_ATTRIBUTES` | `["__class__","__bases__",...]` | Dunder attributes blocked in the code executor. |
| `COST_PER_1K_INPUT_TOKENS` | `0.00015` | Estimated cost per 1,000 input tokens (USD). |
| `COST_PER_1K_OUTPUT_TOKENS` | `0.0006` | Estimated cost per 1,000 output tokens (USD). |
| `COST_PER_TOOL_CALL` | `0.001` | Estimated overhead cost per tool call (USD). |
| `EVAL_MAX_TASKS` | `30` | Maximum tasks per evaluation run. |
| `QUALITY_THRESHOLD` | `0.7` | Minimum quality score for a task to count as "pass". |
Create a `.env` file in the project root to override defaults:
```env
AGENT_MODE=stub
TRANSPORT=sse
HOST=0.0.0.0
PORT=8000
MAX_ITERATIONS=10
ENABLE_SANDBOX=True
QUALITY_THRESHOLD=0.7
```
---
## Docker Deployment
### Build the Image
```bash
docker build -t mcp-agent-system:latest .
```
### Run the Container
```bash
docker run -d \
--name mcp-agent \
-p 8000:8000 \
-e AGENT_MODE=stub \
-e TRANSPORT=sse \
mcp-agent-system:latest
```
The server will be available at `http://localhost:8000`.
### Verify the Deployment
```bash
# Health check
curl http://localhost:8000/health
# List tools
curl http://localhost:8000/tools
# Call a tool
curl -X POST http://localhost:8000/tools/call \
-H "Content-Type: application/json" \
-d '{"tool_name": "calculator", "arguments": {"expression": "2 + 2"}}'
```
The Dockerfile uses a multi-step build with `python:3.12-slim` and caches the dependency layer separately for faster rebuilds. The `.dockerignore` file excludes tests, scripts, and cache directories to keep the image lean.
---
## CI/CD Pipeline
The project includes a GitHub Actions workflow (`.github/workflows/ci.yml`) that runs on every push and pull request to the `main` branch.
### Pipeline Stages
| Step | Description |
|------|-------------|
| **Checkout** | Clone the repository on `ubuntu-latest`. |
| **Setup Python** | Install Python 3.12 with pip caching enabled. |
| **Install Dependencies** | `pip install -r requirements-dev.txt` (includes test and lint tools). |
| **Lint** | `ruff check .` — enforces code style and imports. |
| **Test** | `pytest` — runs the full test suite with verbose output. |
| **Eval Gate** | `python -m scripts.run_eval --max-tasks 30 --threshold 0.7` — enforces minimum quality. |
All steps must pass for the workflow to succeed. The eval gate ensures that regressions in tool selection accuracy or task completion rate are caught before merge.
### Badge
Add the CI status badge to your README (replace `your-username` with your GitHub username/org):
```markdown
```
---
## Testing
### Run the Test Suite
```bash
# Run all tests
pytest
# Run with coverage
pytest --cov=app --cov-report=term-missing
# Run a specific test file
pytest tests/test_calculator.py -v
```
### Run the Evaluation Suite
```bash
# Default evaluation (30 tasks, 0.7 threshold)
python -m scripts.run_eval
# Custom evaluation
python -m scripts.run_eval --max-tasks 50 --threshold 0.8
```
### Linting
```bash
# Check code style
ruff check .
# Auto-fix issues
ruff check . --fix
```
### Test Categories
- **Unit tests** — Individual tool handlers, the MCP server, and the agent engine.
- **Integration tests** — End-to-end agent task runs through the full plan-execute-verify loop.
- **Safety tests** — Verify the sandbox blocks dangerous code (imports, dunder access, `exec`/`eval`).
- **Evaluation tests** — Verify the eval suite produces correct metrics and gate decisions.
---
## Tech Stack
| Category | Technology |
|----------------|--------------------------------------------------------|
| Language | Python 3.10+ |
| Web Framework | FastAPI 0.104+ |
| ASGI Server | Uvicorn (with standard extras) |
| Data Validation| Pydantic 2.0+ |
| Configuration | pydantic-settings 2.0+ |
| Linting | Ruff 0.1+ |
| Testing | pytest 7.0+, pytest-asyncio, httpx |
| Containerization| Docker (python:3.12-slim) |
| CI/CD | GitHub Actions |
| Protocol | Model Context Protocol (MCP) |
---
## License
This project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.
This server cannot be deployed
Maintenance
ActivityStale
ResponsivenessNo issues