MCP Agent System

Production-grade MCP Server with Agent Tool Orchestration — a Model Context Protocol implementation featuring 6 sandboxed tools, an agent orchestration engine, and an evaluation suite with CI/CD quality gates.

Table of Contents

• Overview

• Architecture

• Features

• Project Structure

• Quick Start

• API Endpoints

• Tools

• Evaluation Suite

• Configuration

• Docker Deployment

• CI/CD Pipeline

• Testing

• Tech Stack

• License

Related MCP server: mcp-llm-eval

Overview

MCP Agent System is a production-grade implementation of the Model Context Protocol (MCP) with a built-in agent orchestration engine. It bridges LLM agents and executable tools through a standardized protocol layer, providing a safe and observable environment for autonomous task completion.

The system operates in two modes:

• Stub mode (default): Deterministic, offline, CI-friendly. Tool selection uses keyword matching — no external API calls or LLM dependencies.

• Real mode: LLM-powered agent with real tool execution (configurable via environment variables).

Architecture

The system is organized into four core layers:

┌─────────────────────────────────────────────────────────────────┐
│                     FastAPI Dashboard Layer                      │
│        (REST API + real-time tool call monitoring)               │
├─────────────────────────────────────────────────────────────────┤
│                   Agent Orchestration Engine                     │
│   Plan ──► Tool Selection ──► Execution ──► Verification         │
│        (StubPlanner keyword matching / LLM mode)                 │
├─────────────────────────────────────────────────────────────────┤
│                    MCP Protocol Layer                            │
│     Tools (actions)  │  Resources (context)  │  Prompts          │
│        (server.py — registration, invocation, lifecycle)         │
├─────────────────────────────────────────────────────────────────┤
│                    Tool Implementation Layer                     │
│  ┌──────────┬────────────┬───────────┬────────────┬───────────┐ │
│  │Calculator│Code Executor│Web Searcher│File Manager│Data Analyzer│ │
│  └──────────┴────────────┴───────────┴────────────┴───────────┘ │
│  ┌──────────┐                                                   │
│  │Git Helper│   + AST-validated sandbox safety                  │
│  └──────────┘                                                   │
├─────────────────────────────────────────────────────────────────┤
│                    Evaluation Suite                              │
│  Tool Selection Accuracy │ Task Completion │ Quality Score      │
│  Pass-Rate Gate │ Cost & Latency Tracking                        │
└─────────────────────────────────────────────────────────────────┘

MCP Protocol Layer

Implements the three MCP primitives:

• Tools — Executable actions the agent can invoke (POST-like semantics). Each tool has input/output JSON schemas and a safety flag.

• Resources — Read-only context data the agent can reference (GET-like semantics), identified by URI.

• Prompts — Reusable prompt templates with parameterized arguments.

6 Tools

Six fully sandboxed tools covering common agent task domains: calculation, code execution, web search, file management, data analysis, and Git operations.

Agent Engine

Follows a Plan → Execute → Verify loop:

1. Plan — Analyze the task and select the appropriate tool via keyword scoring (stub) or LLM reasoning (real mode).

2. Execute — Call the selected tool with inferred arguments through the MCP server.

3. Verify — Check whether the result satisfies the task; retry with a fallback strategy if needed.

Safety limits enforced: max iterations per task, max total tool calls, and per-tool execution timeout.

Evaluation Suite

Automated evaluation measuring tool selection accuracy, task completion rate, quality scores, latency, and cost — with a configurable pass-rate threshold that gates CI/CD.

Features

• MCP Protocol Compliant — Full implementation of Tools, Resources, and Prompts primitives with registration, listing, and invocation.

• 6 Sandboxed Tools — Calculator, code executor, web searcher, file manager, data analyzer, and Git helper.

• AST-Validated Safety — Code execution and calculation tools validate the AST before execution, blocking imports, dunder access, and dangerous builtins.

• Agent Orchestration — Plan → Execute → Verify loop with deterministic stub planner and safety limits.

• Dual-Mode Operation — Stub mode for CI-friendly deterministic runs; real mode for LLM-powered autonomy.

• Evaluation Suite — Automated quality gates with tool selection accuracy, task completion rate, and quality scoring.

• Cost & Latency Tracking — Every tool call records latency and estimated cost for observability.

• Virtual Environments — File manager uses an in-memory virtual filesystem; Git helper operates on a virtual repository — no host system access.

• FastAPI Dashboard — REST API for server stats, tool listing, and real-time tool call monitoring.

• Docker Ready — Production Dockerfile with slim Python image.

• CI/CD Integrated — GitHub Actions workflow with ruff linting, pytest, and eval gate enforcement.

Project Structure

mcp-agent-system/
├── app/
│   ├── __init__.py              # Package metadata
│   ├── config.py                # Pydantic settings (env-based configuration)
│   ├── models.py                # Pydantic data models (MCP + Agent + Eval)
│   ├── server/
│   │   ├── __init__.py
│   │   └── server.py            # MCPServer — protocol layer implementation
│   ├── agent/
│   │   ├── __init__.py
│   │   └── engine.py            # Agent + StubPlanner — orchestration engine
│   └── tools/
│       ├── __init__.py
│       ├── registry.py          # Tool registration with the MCP server
│       ├── calculator.py        # AST-validated math expression evaluator
│       ├── code_executor.py     # Sandboxed Python code executor
│       ├── web_searcher.py      # Simulated web search with knowledge base
│       ├── file_manager.py      # Virtual in-memory filesystem manager
│       ├── data_analyzer.py     # CSV data analysis (stats/filter/sort/aggregate)
│       └── git_helper.py        # Virtual Git repository operations
├── eval/                        # Evaluation suite and task datasets
├── scripts/                     # CLI entry points (run_eval, etc.)
├── tests/                       # pytest test suite
├── .github/
│   └── workflows/
│       └── ci.yml               # CI workflow (lint + test + eval gate)
├── pyproject.toml               # Build config, ruff & pytest settings
├── requirements.txt             # Production dependencies
├── requirements-dev.txt         # Development dependencies
├── Dockerfile                   # Container build definition
├── .dockerignore
├── .gitignore
└── README.md

Quick Start

Prerequisites

• Python 3.10 or higher

• pip (or your preferred package manager)

Installation

# Clone the repository
git clone https://github.com/your-username/mcp-agent-system.git
cd mcp-agent-system

# Create and activate a virtual environment
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Install production dependencies
pip install -r requirements.txt

# Or install with development dependencies
pip install -r requirements-dev.txt

Run the Server

# Start the FastAPI server (default: 0.0.0.0:8000)
uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload

# Or run with custom configuration via environment variables
TRANSPORT=sse AGENT_MODE=stub PORT=8000 uvicorn app.main:app --host 0.0.0.0 --port 8000

The server will be available at http://localhost:8000. Interactive API docs are available at http://localhost:8000/docs (Swagger UI).

Run the CLI

# Run the evaluation suite with default settings
python -m scripts.run_eval

# Run with custom parameters
python -m scripts.run_eval --max-tasks 30 --threshold 0.7

API Endpoints

Tools

Evaluation Suite

The evaluation suite runs a set of predefined tasks through the agent and measures performance across multiple dimensions. It serves as a quality gate in CI/CD.

Metrics

Quality Gate

The eval gate enforces a minimum pass rate before allowing deployment:

# Pass if pass_rate >= threshold (default: 0.7)
python -m scripts.run_eval --max-tasks 30 --threshold 0.7

If the pass rate falls below the threshold, the command exits with a non-zero status code, failing the CI pipeline.

Recommendation Levels

The evaluation summary produces a recommendation based on the pass rate:

Configuration

All configuration is handled via environment variables (with .env file support) using pydantic-settings. Sensible offline defaults are provided for stub mode.

Create a .env file in the project root to override defaults:

AGENT_MODE=stub
TRANSPORT=sse
HOST=0.0.0.0
PORT=8000
MAX_ITERATIONS=10
ENABLE_SANDBOX=True
QUALITY_THRESHOLD=0.7

Docker Deployment

Build the Image

docker build -t mcp-agent-system:latest .

Run the Container

docker run -d \
  --name mcp-agent \
  -p 8000:8000 \
  -e AGENT_MODE=stub \
  -e TRANSPORT=sse \
  mcp-agent-system:latest

The server will be available at http://localhost:8000.

Verify the Deployment

# Health check
curl http://localhost:8000/health

# List tools
curl http://localhost:8000/tools

# Call a tool
curl -X POST http://localhost:8000/tools/call \
  -H "Content-Type: application/json" \
  -d '{"tool_name": "calculator", "arguments": {"expression": "2 + 2"}}'

The Dockerfile uses a multi-step build with python:3.12-slim and caches the dependency layer separately for faster rebuilds. The .dockerignore file excludes tests, scripts, and cache directories to keep the image lean.

CI/CD Pipeline

The project includes a GitHub Actions workflow (.github/workflows/ci.yml) that runs on every push and pull request to the main branch.

Pipeline Stages

All steps must pass for the workflow to succeed. The eval gate ensures that regressions in tool selection accuracy or task completion rate are caught before merge.

Badge

Add the CI status badge to your README (replace your-username with your GitHub username/org):

![CI](https://github.com/your-username/mcp-agent-system/actions/workflows/ci.yml/badge.svg)

Testing

Run the Test Suite

# Run all tests
pytest

# Run with coverage
pytest --cov=app --cov-report=term-missing

# Run a specific test file
pytest tests/test_calculator.py -v

Run the Evaluation Suite

# Default evaluation (30 tasks, 0.7 threshold)
python -m scripts.run_eval

# Custom evaluation
python -m scripts.run_eval --max-tasks 50 --threshold 0.8

Linting

# Check code style
ruff check .

# Auto-fix issues
ruff check . --fix

Test Categories

• Unit tests — Individual tool handlers, the MCP server, and the agent engine.

• Integration tests — End-to-end agent task runs through the full plan-execute-verify loop.

• Safety tests — Verify the sandbox blocks dangerous code (imports, dunder access, exec/eval).

• Evaluation tests — Verify the eval suite produces correct metrics and gate decisions.

Tech Stack

License

This project is licensed under the MIT License. See the LICENSE file for details.