A
licenseB
qualityB
maintenanceAn MCP-style stdio server for evaluating AI agent outputs, enabling CI-friendly quality gates, regression comparisons, and canary promotion decisions.
3
MIT
mcp-llm-eval
by berkayildi
Related Servers
Alternatives to mcp-llm-eval
No user-submitted related servers found.
Related Servers
- AlicenseNot gradedqualityDmaintenanceA local MCP server for prompt evaluation, enabling users to define test cases, run prompts against multiple LLM providers, score outputs with deterministic and LLM-graded assertions, and track quality over time, all within an AI coding environment.MIT
- AlicenseAqualityBmaintenanceAn MCP server that lets any AI agent evaluate RAG outputs -- faithfulness scoring, hallucination detection, and retrieval quality metrics -- with zero API keys, using MCP sampling.6MIT
- FlicenseNot gradedqualityBmaintenancePolicy-as-code gate for AI-SDLC, providing MCP tools to review prompts, diff tool manifests, vet MCP servers, and run evaluation suites for LLM agent repos.1-
- AlicenseNot gradedqualityBmaintenanceMCP server that enables AI agents to run a deterministic orchestration loop with decomposition, subagent execution, and review feedback across multiple LLM backends.59MIT
- FlicenseNot gradedqualityCmaintenanceA production-grade MCP server with 6 sandboxed tools and an agent orchestration engine for autonomous task completion, featuring an evaluation suite with CI/CD quality gates.-
TDQS
A3.7/5.0
Scored across 10 tools
Disambiguation4/5
Most tools have distinct purposes, but check_retrieval_drift and compare_runs both detect regressions, causing slight overlap. Descriptions help clarify, but an agent might confuse them.
Naming Consistency4/5
Most tools use verb_noun pattern (e.g., check_thresholds, evaluate_retrieval), but evaluate_rag_end_to_end is more verbose and deviates slightly. Overall consistent.
Tool Count5/5
10 tools is well-scoped for an evaluation server, covering retrieval, generation, comparison, and reporting without being overwhelming.
Completeness4/5
The set covers core evaluation workflows (retrieval, RAG, comparison, thresholds, reporting). Missing delete or batch management, but that's acceptable. The stub tool indicates a planned feature not yet implemented.
Maintenance
ActivityInactive
ResponsivenessNo issues