llm-eval-mcp
The server evaluates LLM agent reliability by generating adversarial tasks, scoring answers with an LLM judge, and reporting aggregate statistics.
Generate adversarial test tasks: Create 1–50 tasks targeting any of six failure categories (hallucination, instruction following, prompt injection, unsafe output, reasoning error, tool misuse), each with an expected behavior as reference.
Run LLM-as-judge evaluations: Submit a prompt, expected behavior, and an agent's answer to receive a structured verdict (pass/fail, score 1–5, justification).
Retrieve evaluation statistics: Get total evaluations, pass count, pass rate, and average score from stored results.
Access via MCP or HTTP: The same capabilities are exposed as an MCP server (stdio) and a FastAPI web API (with auth for the judge endpoint).
Click on "Install Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@llm-eval-mcpGenerate adversarial tasks and score my agent's responses"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
llm-eval-mcp
A small platform for evaluating the reliability of LLM agents. It generates adversarial test tasks, scores answers with an LLM acting as a judge, and keeps track of the results so you can see how often a model actually behaves the way it should.
The same logic is exposed two ways: as an MCP server (so any MCP client can call the tools) and as a web API (so anyone can try it from a browser). Both sit on top of one shared core.
What it does
The idea is simple. Testing whether an LLM answer is "good" with fixed rules is fragile, so this project leans on three moving parts that work together:
It builds adversarial tasks across six failure categories (hallucination, instruction following, prompt injection, unsafe output, reasoning errors, tool misuse). Each task comes with the behavior a reliable agent should show, which acts as the reference.
It runs an LLM as judge: a strong model reads the task, the expected behavior, and a candidate answer, then returns a structured verdict (pass or fail, a score from 1 to 5, and a short justification).
It stores every judgment and reports aggregate statistics, so a single answer becomes data you can count, compare, and later analyze.
Related MCP server: groundcheck
The tools
The MCP server and the web API both expose the same three capabilities:
generate_adversarial_tasks produces test tasks for a chosen failure category. It runs without any LLM, so it is free and deterministic.
run_llm_as_judge (or POST /judge) scores an answer against its expected behavior and records the result. This is the one call that talks to the judge model.
get_eval_stats (or GET /stats) returns the totals: how many evaluations ran, how many passed, the pass rate, and the average score.
How it is built
src/llm_eval_mcp/
├── domain/ # Pure business logic (no network, no provider, no SQL)
│ ├── adversarial.py # adversarial task generation
│ ├── judging.py # verdict model, Judge contract, prompt, orchestration
│ └── eval_run.py # EvalRun entity, EvalRunRepository contract, stats
├── adapters/ # Technical details (network, providers, database)
│ ├── groq_judge.py # the real judge, backed by Groq
│ └── persistence.py # the real repository, backed by SQLAlchemy
├── wiring.py # Composition root: assembles the real implementations
├── server.py # MCP input adapter (stdio transport)
└── api.py # HTTP input adapter (FastAPI) with API key auth
tests/
├── test_adversarial.py # generator tests
├── test_judging.py # judging tests (fake judge)
├── test_groq_judge.py # Groq adapter tests (fake client)
├── test_persistence.py # stats, SQL repository, and server integration
├── test_api.py # HTTP API tests (TestClient, auth), no network
└── fakes.py # test doubles (FakeJudge, InMemoryEvalRunRepository)Why it is built this way
The whole codebase follows one rule: the business logic never depends on a specific tool. It depends on contracts instead.
The domain package knows nothing about Groq, HTTP, or the database. It defines what a judge is (a Judge protocol) and what a store is (an EvalRunRepository protocol), and it works against those ideas. The real implementations live in adapters and plug into those contracts. A small wiring module is the only place that decides which real implementations to use and how to assemble them.
This buys three concrete things. The tests run with in memory doubles, so they need no API key, no network, and no database, and they always give the same result. Swapping Groq for another provider means writing one new adapter, not touching the core. And the two front doors, MCP and HTTP, reuse the exact same logic through wiring, so nothing is duplicated.
Getting started
You need Python 3.10 or newer. From the project root:
python -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
pip install -e ".[dev]"The judge needs a Groq API key, which is free and does not require a card at https://console.groq.com:
cp .env.example .env # then paste your key into GROQ_API_KEYListing the tools works without a key. Only the judge call needs one, since the key is read lazily.
Running the tests
pytestEverything runs offline thanks to the test doubles, so this is fast and does not touch Groq or any database.
Running the MCP server
The MCP server speaks the stdio transport, so you do not use it by hand. The easiest way to inspect it is the official inspector:
mcp dev src/llm_eval_mcp/server.pyThat opens MCP Inspector in your browser, where you can list the three tools and call them live.
Running the web API
uvicorn llm_eval_mcp.api:app --reloadThen open http://localhost:8000/docs for the interactive Swagger page. POST /judge expects an X-API-Key header whose value matches API_KEY in your .env. The health check, task generation, and stats endpoints are open.
Docker
The image uses a two stage build, so the final image carries only what it needs to run and none of the build tooling. It runs as a non root user and starts the API with Uvicorn.
docker build -t llm-eval-mcp .
docker run --rm -p 8000:8000 --env-file .env llm-eval-mcpDeployment
render.yaml describes a free web service on Render. Push the repository to GitHub, create a Blueprint that points at it, and set the GROQ_API_KEY secret in the dashboard. Render generates the API_KEY for you and gives you a public URL, with the interactive docs available at /docs.
By default the app uses SQLite. In a container that storage is temporary, so records reset on each redeploy, which is fine for a demo. For durable storage, point DATABASE_URL at a managed PostgreSQL instance. Thanks to SQLAlchemy and a small URL normalizer, the switch needs no code change, only the environment variable.
License
MIT.
Available Tools
3 toolsgenerate_adversarial_tasksA
Génère des tâches de test adversariales pour éprouver un agent LLM.
Appelle cet outil quand tu veux construire un jeu de tests ciblant un mode de défaillance précis (hallucination, injection de prompt, non-respect des consignes, sortie dangereuse, erreur de raisonnement, mauvais usage d'outil). Chaque tâche inclut le comportement attendu, qui sert de référence au juge.
Args: category: La famille de défaillance à cibler. count: Nombre de tâches à générer (entre 1 et 50). Par défaut 5.
Returns: La liste des tâches adversariales annotées.
| Name | Required | Description | Default |
|---|---|---|---|
| count | No | ||
| category | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the burden. It explains that each task includes expected behavior for the judge and that the return is a list of annotated tasks. However, it does not discuss side effects, permissions, or limitations, leaving some behavioral aspects undisclosed.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well-structured: a clear one-sentence purpose, a usage condition, then a concise Args/Returns breakdown. Every section contributes essential information without redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the output schema exists, the description covers the necessary context: purpose, when to use, parameters, and return type. It aligns with sibling tools and provides enough detail for an agent to invoke it appropriately.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema itself has 0% description coverage on properties, but the description compensates thoroughly. It explains 'category' as the failure family to target and 'count' with its range (1-50) and default (5). This adds meaningful context beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific action: 'Génère des tâches de test adversariales pour éprouver un agent LLM' (generates adversarial test tasks to test an LLM agent). This clearly distinguishes it from sibling tools like run_llm_as_judge (which evaluates) and get_eval_stats (which retrieves metrics).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It provides explicit guidance on when to call the tool: 'Appelle cet outil quand tu veux construire un jeu de tests ciblant un mode de défaillance précis' (call when building a test suite targeting a specific failure mode). It lists example failure modes but does not mention when not to use it or name alternatives, stopping short of a 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_eval_statsA
Renvoie des statistiques agrégées sur toutes les évaluations enregistrées.
Fournit le nombre total d'évaluations, le nombre de réussites, le taux de réussite (0 à 1) et le score moyen. Ne nécessite pas de clé API.
Returns: Les statistiques agrégées.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Output Schema
| Name | Required | Description |
|---|---|---|
| total | Yes | Nombre total d'évaluations. |
| passed | Yes | Nombre d'évaluations réussies. |
| avg_score | Yes | Score moyen (1 à 5). |
| pass_rate | Yes | Taux de réussite, entre 0 et 1. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the transparency burden. It adds value by stating that no API key is required and that it returns read-only statistics, implying no side effects. However, it doesn't explicitly state it is a read-only operation or mention rate limits/pagination.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is succinct and front-loaded with the main action. It uses a brief paragraph and a 'Returns:' label, making it easy to parse without fluff.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a zero-parameter tool with an output schema, the description is complete: it states the purpose, lists all expected output fields, and notes authentication requirements. No further context is needed.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has zero parameters, so the schema already covers everything. The description adds no parameter info, but baseline 4 applies because there's nothing to explain.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's function: returning aggregated statistics on all registered evaluations. It lists the specific metrics (total, successes, success rate, average score), which distinguishes it from the sibling tools that generate tasks or run a judge.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
While the description implies the tool is for retrieving overall evaluation stats, it does not provide explicit guidance on when to use it versus the sibling tools, nor does it mention alternative tools for specific cases. The context is clear but no exclusions are given.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
run_llm_as_judgeA
Évalue la réponse d'un agent à une consigne, via un juge LLM.
Compare answer au expected_behavior attendu pour la prompt donnée, et
renvoie un verdict structuré : réussite (booléen), score de 1 à 5, et une
courte justification. Utile pour noter automatiquement les réponses d'un
agent testé sur des tâches adversariales.
Args: prompt: La consigne qui avait été soumise à l'agent. expected_behavior: Le comportement attendu d'un bon agent (référence). answer: La réponse produite par l'agent, à évaluer.
Returns: Le verdict structuré du juge.
| Name | Required | Description | Default |
|---|---|---|---|
| answer | Yes | ||
| prompt | Yes | ||
| expected_behavior | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
| score | Yes | Qualité globale, de 1 (mauvais) à 5 (parfait). |
| passed | Yes | Vrai si la réponse respecte l'essentiel du comportement attendu. |
| justification | Yes | Explication courte et factuelle du verdict (1 à 2 phrases). |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description bears the full burden of behavioral disclosure. It transparently describes the evaluation process (comparing 'answer' to 'expected_behavior') and the structured verdict format (success boolean, 1-5 score, short justification). While it omits details like API costs, rate limits, or potential non-determinism of LLM judges, it provides a solid overview of what the tool does and returns.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well-structured: a concise paragraph explaining the purpose and a clear list of parameter definitions. Each sentence adds value, and the format is easy to parse. It is slightly verbose due to the parameter list, but not unnecessarily wordy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's modest complexity (3 string parameters, no nested objects, output schema present), the description adequately covers what the tool does, when to use it, and what it returns. It also lists all parameters with explanations. The only minor gap is not discussing edge cases or limitations of the LLM judge, but this is not essential for basic invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema lists only bare string parameters with no descriptions (0% coverage). The description compensates fully by defining each parameter: 'prompt' as the instruction given to the agent, 'expected_behavior' as the reference good behavior, and 'answer' as the response to evaluate. This gives complete semantic meaning beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's function: it evaluates an agent's response via an LLM judge. The verb 'Évalue' and the specific resource 'réponse d'un agent' make the purpose clear, and it distinguishes itself from siblings (generate_adversarial_tasks and get_eval_stats) by focusing on judging/rating answers rather than generating tasks or fetching stats.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear usage context: 'Utile pour noter automatiquement les réponses d'un agent testé sur des tâches adversariales' indicates when to use the tool. It does not explicitly mention alternatives or exclusions, but the tool's distinct role among siblings makes the usage context sufficiently clear.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
TDQS
Each tool has a distinct role: generating adversarial tasks, judging a single answer, and aggregating evaluation stats. There is no overlap in purpose or expected inputs.
All tool names follow a clear verb_noun pattern with snake_case: generate_adversarial_tasks, run_llm_as_judge, get_eval_stats. The verbs are action-oriented and descriptive.
With 3 tools, the server is tightly scoped to the core evaluation workflow: generate tasks, judge responses, and view stats. Each tool is essential and the count is within the ideal 3-15 range.
The set covers the main lifecycle of an evaluation run: generating adversarial cases, scoring a response against expected behavior, and retrieving aggregate metrics. A minor gap is the lack of task management (list, delete, update) but tasks are ephemeral, so this is acceptable.
Maintenance
Resources
Unclaimed servers have limited discoverability.
Looking for Admin?
If you are the server author, to access and configure the admin panel.
Related MCP Connectors
MCP server for building and testing AI agents with multi-model experimentation and insights.
MCP server providing access to the Scorecard API to evaluate and optimize LLM systems.
MCP server for agentverse documentation, generated by doc2mcp.
Security scanner for MCP servers. Detect vulnerabilities, prompt injection, and tool poisoning.
Related MCP Servers
- AlicenseAqualityBmaintenanceAn MCP server that lets any AI agent evaluate RAG outputs -- faithfulness scoring, hallucination detection, and retrieval quality metrics -- with zero API keys, using MCP sampling.6MIT
- AlicenseAqualityCmaintenanceAn MCP server that audits LLM-as-judge evaluations, detecting judge drift across runs, measuring bias through controlled probes, and comparing judge agreement with human raters.6MIT
- AlicenseNot gradedqualityBmaintenanceMCP server that analyzes AI agent execution logs to calculate reliability scores, detect failure patterns, and suggest concrete improvements for making AI agents more reliable.MIT
Latest Blog Posts
- Who's Calling? MCP Hosts Are an Identity Blind Spot (And the Spec Knows It)By Om-Shree-0709 on .mcpAgent IdentityOAuth 2.1
- Your AI Chatbot Just Exposed Your CEO's Salary to an InternBy Om-Shree-0709 on .Agent IdentityMCP SecurityOAuth Delegation
- Why MCP Servers Need Execution Sandboxing (And Why Your Current Stack Isn't Enough)By Om-Shree-0709 on .Agentic AiPrompt InjectionWebAssembly
MCP directory API
We provide all the information about MCP servers via our MCP API.
curl -X GET 'https://glama.ai/api/mcp/v1/servers/cdywolf/llm-eval-mcp'
If you have feedback or need assistance with the MCP directory API, please join our Discord server