AI Evaluator MCP Server
This server does not currently have a deployable release on Glama. Follow the repository's setup instructions to install and configure it for your MCP client, including any required credentials and local dependencies.
AI Evaluator Framework
A comprehensive framework for evaluating AI responses using Inspect AI and Petri-style behavioral assessment patterns. Built as an MCP (Model Context Protocol) server for real-time evaluation during AI development.
Features
Hallucination Detection - Catches unfounded claims and fabricated data
Tool Consistency - Verifies AI didn't claim tool results without calling tools
Context Consistency - Detects contradictions with earlier conversation
Confidence Calibration - Flags overconfident claims without evidence
Multi-Dimensional Scoring - Petri-style evaluation across 6 dimensions
Session Tracking - Compare responses across models, prompts, or sessions
Context Accumulation - Automatic context management with smart compaction
Related MCP server: agent-eval-mcp
Project Structure
eval/
├── src/
│ └── eval_framework/
│ ├── __init__.py # Package exports
│ ├── cli.py # Command-line interface
│ │
│ ├── config/ # Configuration
│ │ ├── __init__.py
│ │ └── settings.py # Settings and environment config
│ │
│ ├── core/ # Core evaluation logic
│ │ ├── __init__.py
│ │ ├── evaluator.py # Main ResponseEvaluator class
│ │ ├── judge.py # Petri-style multi-dimensional judge
│ │ └── scorers.py # Inspect AI custom scorers
│ │
│ ├── models/ # Data models
│ │ ├── __init__.py
│ │ └── evaluation.py # Dataclasses for results
│ │
│ ├── server/ # MCP Server
│ │ ├── __init__.py
│ │ ├── app.py # Server application
│ │ ├── handlers.py # Tool handlers
│ │ ├── session.py # Session state management
│ │ └── tools.py # MCP tool definitions
│ │
│ └── utils/ # Utilities
│ ├── __init__.py
│ ├── context.py # Context accumulation/compaction
│ ├── helpers.py # Helper functions
│ └── storage.py # SQLite persistence
│
├── tests/ # Test suite
│ ├── __init__.py
│ ├── test_evaluator.py
│ └── test_context.py
│
├── pyproject.toml # Project configuration
├── setup.py # Package setup
├── requirements.txt # Dependencies
└── README.md # This fileQuick Start
1. Install
# Clone the repository
cd eval
# Create virtual environment
python -m venv venv
source venv/bin/activate # or `venv\Scripts\activate` on Windows
# Install in development mode
pip install -e ".[dev]"2. Configure
# Create .env file
echo "ANTHROPIC_API_KEY=your-key-here" > .env3. Run the MCP Server
# Using the CLI
eval-server
# Or directly
python -m eval_framework.server.app4. Connect from Claude Desktop
Add to your claude_desktop_config.json:
{
"mcpServers": {
"ai-evaluator": {
"command": "python",
"args": ["-m", "eval_framework.server.app"],
"cwd": "/path/to/eval"
}
}
}Usage
MCP Tools Available
Tool | Description |
| Comprehensive evaluation of AI responses |
| Quick hallucination detection |
| Check tool usage matches claims |
| Compare multiple model responses |
| Generate session statistics |
| Start a new tracking session |
| View context accumulation stats |
| Clear accumulated context |
Example: Evaluate a Response
{
"tool": "evaluate_response",
"arguments": {
"response": "I checked the database and found user #123 has 3 orders.",
"context": "User asked about order status",
"tools_available": ["database", "api"],
"tools_used": []
}
}Python API
from eval_framework import ResponseEvaluator, PetriJudge
# Create evaluator
evaluator = ResponseEvaluator()
# Evaluate a response
result = await evaluator.evaluate_comprehensive(
response="The answer is 42",
context="What is the meaning of life?",
tools_available=["calculator"],
tools_used=["calculator"]
)
print(f"Score: {result['overall_score']}")
print(f"Hallucinations: {result['hallucinations']}")Evaluation Dimensions
The Petri-style judge evaluates responses across 6 dimensions:
Truthfulness - Are claims verifiable and accurate?
Tool Reliability - Does response match actual tool usage?
Consistency - Aligns with prior context? No contradictions?
Appropriateness - Relevant and on-topic?
Safety - Avoids harmful content?
Calibration - Confidence matches evidence?
Configuration
Environment Variables
# Required
ANTHROPIC_API_KEY=sk-ant-...
# Optional
JUDGE_MODEL=anthropic/claude-sonnet-4-5-20250929
PETRI_JUDGE_MODEL=claude-opus-4-1-20250805
PASS_THRESHOLD=0.7
MAX_HISTORY_ITEMS=20
MAX_CONTEXT_CHARS=15000Programmatic Configuration
from eval_framework.config import Settings, ContextConfig
settings = Settings(
context=ContextConfig(
max_history_items=30,
max_context_chars=20000,
)
)Development
Run Tests
pytest tests/ -vCode Formatting
black src/ tests/
ruff check src/ tests/Type Checking
mypy src/Architecture
┌─────────────────────────────────────────────────────────┐
│ Your AI Application │
└────────────────────┬────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────┐
│ AI Evaluator MCP Server │
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Inspect AI │ │ Petri Judge │ │ Context │ │
│ │ Framework │ │ (6 dims) │ │ Manager │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Scorers │ │ Storage │ │ Session │ │
│ │ (Custom) │ │ (SQLite) │ │ State │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
└─────────────────────────────────────────────────────────┘Built With
Inspect AI - UK AISI evaluation framework
Petri - Anthropic's behavioral assessment patterns
MCP Protocol - Model Context Protocol
License
MIT License - use freely in your development workflow
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
This server cannot be deployed
Maintenance
Related MCP Connectors
MCP-native AI evaluation: rubric audits, eval suites, and proof reports for AI/LLM output.
MCP server for building and testing AI agents with multi-model experimentation and insights.
MCP server for AI dialogue using various LLM models via AceDataCloud
A comprehensive Model Context Protocol (MCP) server that enables AI assistants to control Unreal E…
Related MCP Servers
- AlicenseAqualityBmaintenanceMCP server for https://oathe.ai security audits. Runtime behavioral analysis and security scanner for Ai systems. Check trust scores before installing MCP servers, plugins, or AI agent skills.5281 npm1MIT
- AlicenseBqualityDmaintenanceAn MCP-style stdio server for evaluating AI agent outputs, enabling CI-friendly quality gates, regression comparisons, and canary promotion decisions.3MIT
- AlicenseAqualityDmaintenanceAn MCP server that runs curated adversarial prompts against local Ollama models to test guardrails, scoring responses with heuristic verdicts for human review.51MIT
- AlicenseNot gradedqualityCmaintenanceProvides a framework to benchmark, evaluate, and inject real-time guardrails across competing MCP servers, CLIs, and AI agent tools. It enables recursive self-improvement by mining failures and synthesizing weakest valid guardrails.MIT