Skip to main content
Glama

evalgate-mcp

An MCP server that lets an agent read its own eval results. Wraps evalgate — run a suite, explain a failure, check for drift, and ask whether the judge was ever calibrated.

Phase 1 is read-only. Nothing here writes a file. run_suite executes the system under test and may bill a provider, but the result comes back in memory rather than landing in .evalgate/result.json. Write access is phase 2, deliberately.


Why an agent should be allowed to do this

Handing an agent its own grader is circular. It's also useful, and the circularity is manageable as long as one thing is never hidden: whether anyone measured the grader.

So every result from run_suite carries a judgeStatus block, in one of five states:

state

meaning

no-judge

No judged assertions ran. Nothing to qualify.

calibrated

The judge has a passing stamp bound to its own id. Agreement is published.

uncalibrated

No stamp exists. Scores are unqualified — a judge nobody measured is a random number generator with good manners.

judge-mismatch

A stamp exists but belongs to a different judge. Agreement is withheld, because one judge inheriting another's credibility is worse than no stamp at all.

failed-calibration

The judge last failed calibration. Agreement is reported with the failure.

That block is the point of this server. An agent that can score its own output and can't see the state of its grader will report success it didn't earn.

Related MCP server: mcp-llm-eval

Tools

tool

cost

what it does

list_suites

free

Load and validate suites — cases, assertions, thresholds.

explain_result

free

Read a result artifact and explain what failed, down to the claim-level grounding breakdown.

check_drift

free

Analyze history.jsonl for slow decline the per-PR gate can't see.

judge_status

free

Read the calibration stamp and report agreement, bias state, and judge binding.

run_suite

executes the SUT; may call providers

Run the suites and return the scored result. Writes nothing.

Bad paths and malformed suites come back as tool errors with the message intact, not as thrown stack traces the model can't act on.

Install

Not on npm. evalgate isn't either — both install from git.

git clone https://github.com/HolisticShort/evalgate-mcp.git
cd evalgate-mcp
npm install          # builds on install via `prepare`
npm run smoke        # end-to-end check against evalgate's example suite

npm run smoke expects an evalgate checkout at ../evalgate, or set EVALGATE_DIR.

Use it with Claude Code

Add to .mcp.json in the project you want to gate:

{
  "mcpServers": {
    "evalgate": {
      "command": "node",
      "args": ["/absolute/path/to/evalgate-mcp/dist/server.js"]
    }
  }
}

Restart Claude Code, then /mcp to confirm the server connected and the five tools are listed.

Phase 2

Write access — letting an agent add cases, adjust thresholds, and write baselines. Held back on purpose. An agent editing the test cases that grade it is a different trust question from an agent reading them, and it deserves its own design rather than arriving as a convenience.

License

MIT

Available Tools

5 tools
check_driftCheck quality driftA

Analyze .evalgate/history.jsonl for slow decline the per-PR regression gate cannot see. A case sliding a little each run passes every individual gate by construction. Reports per-series delta and slope. Read-only.

ParametersJSON Schema
NameRequiredDescriptionDefault
windowNoMost recent N runs to consider.
thresholdNoDecline over the window that counts as drift. Defaults to evalgate’s own.
historyPathYesPath to .evalgate/history.jsonl.

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description must carry behavioral disclosure. It clearly states 'Read-only' and describes the output (per-series delta and slope), which covers the key behavioral traits. It does not document file-missing or empty-history behavior, but for a read-only analysis tool this is acceptable.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is four short sentences, each earning its place: the resource and purpose, the rationale, the output, and the read-only nature. No filler or redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with no output schema, it covers purpose, output, and safety. It lacks explicit handling of edge cases (e.g., missing file, no drift), but the available schema and description provide sufficient context for an agent to invoke it correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Input schema has 100% coverage with descriptive comments for all three parameters, so baseline is 3. The description does not add any parameter-specific details beyond noting the history file path and the concept of slow decline, which is already captured by the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb ('Analyze') and resource ('.evalgate/history.jsonl') with a clear purpose: detecting slow decline that per-PR gates miss. This differentiates it from siblings like run_suite or judge_status, which focus on individual results.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Implies usage for detecting cumulative performance drift over time rather than per-PR checks, noting that individual gates can miss gradual declines. Does not explicitly name alternatives or exclusions, but the contrast with 'per-PR regression gate' provides clear context.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

explain_resultExplain an eval resultA

Read an evalgate result artifact (.evalgate/result.json) and explain why it passed or failed: which gates tripped, which cases failed, and the per-assertion explanation for each — including the claim-level grounding breakdown. Read-only.

ParametersJSON Schema
NameRequiredDescriptionDefault
resultPathYesPath to a result artifact written by `evalgate run --json`.
failuresOnlyNoOnly include cases that failed or were flagged critical. Defaults to true.

TDQS

A4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden and does well by explicitly stating 'Read-only'. It also discloses what the explanation covers (gates, failed cases, per-assertion and claim-level breakdown), giving a clear picture of the tool's behavior and output without revealing internals.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two tight sentences, front-loaded with the primary action ('Read... and explain why it passed or failed') and structured to list the explanation contents. The final 'Read-only' is a concise, valuable note with no redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given no output schema or annotations, the description sufficiently explains what the tool does and what it returns. It covers read-only behavior and the scope of the explanation, which is adequate for a simple read tool. It doesn't discuss error cases or relationship to siblings, but that's not essential.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema descriptions cover 100% of parameters, so baseline is 3. The description adds the artifact path example (.evalgate/result.json) but does not go beyond the schema's per-parameter descriptions for resultPath or failuresOnly. It provides no additional meaning for the failuresOnly parameter.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's verb and resource: 'Read an evalgate result artifact' and 'explain why it passed or failed'. It also details specific output elements (which gates tripped, cases failed, per-assertion explanations, claim-level grounding breakdown), fully distinguishing it from siblings like list_suites or run_suite.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage after a run (explaining an existing result artifact) but does not explicitly state when to use it versus alternatives or mention any exclusions. It lacks a direct 'use this instead of X' or 'after run_suite' guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

judge_statusCheck whether the judge was calibratedA

Read .evalgate/calibration.json and report the judge’s measured agreement with human scores. An uncalibrated judge is a random number generator with good manners — call this before trusting any rubric or grounded score. Read-only.

ParametersJSON Schema
NameRequiredDescriptionDefault
judgeIdNoIf given, verify the stamp belongs to this judge id.
calibrationPathYesPath to .evalgate/calibration.json.

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden of behavioral disclosure. It states 'Read-only,' which is a critical safety trait, and explains that it reads a specific file and reports agreement. It does not mention failure modes or return format, but for a simple status read, this is sufficient transparency.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two sentences long, front-loaded with the core action, and uses the metaphor 'random number generator with good manners' to add context without wasted words. Every sentence earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple status-check tool with full schema coverage and no output schema, the description provides sufficient context: it names the file, explains what the report contains, and gives usage timing. It could describe the output shape in more detail, but the core purpose is fully covered.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, with both calibrationPath and judgeId described in the input schema. The description adds no additional parameter-level semantics, so the baseline score of 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states a specific verb and resource: 'Read .evalgate/calibration.json and report the judge’s measured agreement with human scores.' This distinguishes it from sibling tools like list_suites, explain_result, check_drift, and run_suite, which all focus on different aspects of the tool ecosystem.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives explicit usage guidance: 'call this before trusting any rubric or grounded score.' This tells the agent when to use the tool, though it does not mention alternatives or when not to use it, which would earn a 5.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_suitesList eval suitesA

Load and validate the eval suites in a directory or file. Returns each suite name, its cases, the assertions declared on each case, and the threshold policy that gates it. Read-only — no model calls, no cost.

ParametersJSON Schema
NameRequiredDescriptionDefault
suitePathYesDirectory of suite YAML files, or a single suite file.

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the burden of safety disclosure. It explicitly states 'Read-only — no model calls, no cost', which is important behavioral info. It also mentions 'validate', implying it may error on invalid input, but doesn't detail error behavior. Overall, good transparency for a simple tool.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two concise sentences: first states the action and return values, second provides a safety note. Every word adds value, and the key information is front-loaded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool has one simple parameter, no output schema, and no annotations. The description compensates well by listing the return components and explicitly stating the read-only behavior. Nothing critical is missing for an agent to use it correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% for the single parameter 'suitePath', and the description only restates the same meaning ('directory or file') without adding extra details like formats or examples. Baseline 3 is appropriate since the schema already documents the parameter fully.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states it loads and validates eval suites, and enumerates exactly what it returns (suite name, cases, assertions, threshold policy). This distinguishes it from siblings like run_suite, which would execute suites.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The read-only note ('no model calls, no cost') provides clear context for when to use this tool—for inspection rather than execution. It does not explicitly name alternative tools, but the context is clear enough for an agent to select it appropriately.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

run_suiteRun an eval suiteA

Run the suites at a path against a system-under-test module and return the scored result. EXECUTES the SUT module and may call model providers, so it can incur real cost. Writes nothing — the result is returned in memory, not saved as an artifact.

ParametersJSON Schema
NameRequiredDescriptionDefault
suitePathYesDirectory of suite YAML files, or a single suite file.
sutModuleYesPath to the module exporting the system under test as `default` or `sut`.
concurrencyNoCases evaluated at once.
calibrationPathNoPath to the calibration stamp. Defaults to .evalgate/calibration.json beside the suite.

TDQS

A4.4/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden and explicitly warns 'EXECUTES the SUT module and may call model providers, so it can incur real cost' and states 'Writes nothing — the result is returned in memory, not saved as an artifact.' This discloses side effects and persistence behavior, which is crucial for an agent deciding to run it.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two sentences: the first states the core function, the second delivers essential warnings about cost and non-persistence. Every sentence carries meaningful information with no filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the four parameters are fully documented in the schema and the description covers side effects and return notion, the tool is largely complete. However, the absence of an output schema makes the vague 'scored result' less informative than it could be, and there is no guidance on how to interpret the result or error cases.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

All four parameters have detailed descriptions in the schema (100% coverage), so the baseline is 3. The description itself only reiterates 'suites at a path' and 'system-under-test module' without adding new semantics for concurrency or calibrationPath, so it adds no significant value beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states it 'Run the suites at a path against a system-under-test module and return the scored result,' identifying the specific action (run), the resource (suites at a path + SUT module), and the output (scored result). This distinguishes it from siblings like list_suites and explain_result.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides clear context for what the tool does, and the SUT execution and cost note implies when to use it (when you need to actually evaluate a module). However, it does not explicitly compare with alternatives or state when not to use it, leaving the agent to infer from sibling names.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

TDQS

A4.3/5.0
Disambiguation5/5

Each tool targets a distinct operation: listing suites, running them, explaining results, checking drift, and assessing judge calibration. No two tools overlap in purpose or behavior.

Naming Consistency5/5

All tool names follow a consistent verb_noun pattern: list_suites, run_suite, explain_result, check_drift, judge_status. The naming is uniform and predictable.

Tool Count5/5

With 5 tools, the server covers the core evaluation lifecycle without bloat. Each tool earns its place and the count is well-scoped for the domain.

Completeness4/5

The surface covers listing, running, explaining, monitoring, and validating calibration. Minor gaps exist (e.g., no tool for editing suites or historical per-case comparison) but core workflows are complete.

Maintenance

ActivityMaintained
ResponsivenessNo issues

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Related MCP Connectors

Related MCP Servers

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/HolisticShort/evalgate-mcp'

If you have feedback or need assistance with the MCP directory API, please join our Discord server