Skip to main content
Glama
Mr-Lucasz

Conversational API Test MCP

by Mr-Lucasz
OverviewSchemaSourceRelated ServersDiscussionsAdmin
TypeScript
Local

run_eval

Evaluate variable or multi-input API endpoints in one call: repeat each case, check expected conditions, and return pass rates, failures, latency percentiles, answer variation, optional judge grading.

Instructions

Evaluate an endpoint whose answer varies (LLM / RAG / search) — or any endpoint over many inputs — in one call. Runs each case repeat times and passes it when passRate of the runs satisfy expect; returns one row per case, the distinct failures, latency p50/p95 and how many different answers came back. With judge it also returns question / answer / retrieved passages for you to grade what code cannot (faithfulness, relevance). Runs do not capture variables or replace the session's last response. Every run is a real, possibly paid, request.

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
casesNoInline cases: {name, variables: {question: "…"}, expect, reference}. expect = {status, jsonPathExists, maxDurationMs, checks: [{path (JSONPath; omit = whole body), contains | containsAny | notContains | matches | equals | min | max | minLength | maxLength | includesAll | includesAny | subsetOf (another JSONPath), ignoreCase, minRatio}]}
judgeNoReturn samples for you to grade: {answerPath, contextPath, questionVariable, criteria: [faithfulness | relevance | context_relevance | correctness | your own rubric]}.
expectNoApplied to every case.
repeatNoTimes each case is sent (default 1, or the eval's).
evalNameNoEval declared under `evals:` in the definition. Otherwise pass `requestId` + `cases`.
passRateNoShare of a case's runs that must pass (default 1).
caseNamesNoRun only these cases.
requestIdNo
timeoutMsNo
concurrencyNo
maxRequestsNoRefuse to start when cases × repeat exceeds this.
workspaceRootNoOptional when MCP_WORKSPACE_ROOT is set.
definitionRelativePathYes

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observedv0.1.0

TDQS

A4.1/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden and does so well: it discloses the repeat/passRate pass logic, the return shape (one row per case, distinct failures, p50/p95 latency, distinct-answer count), that runs neither capture variables nor replace the session's last response, and warns every run is a real, possibly paid request.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Four dense sentences that are front-loaded with the core purpose, followed by pass logic, output shape, the judge extension, and caveats. Every sentence earns its place, though the middle sentence is heavily packed.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 13-parameter tool with no output schema and no annotations, the description covers purpose, return shape, and cost but omits several parameters (concurrency, maxRequests guard behavior, timeoutMs, caseNames, evalName-vs-requestId routing, workspaceRoot). It is adequate for a first call but leaves gaps for full parameterization.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 69% and the description adds real meaning beyond it: `repeat`/`passRate` are explained through the passing rule, `judge` is described by its returned question/answer/passages, and `cases`/`expect` semantics are tied to per-case evaluation. It doesn't cover routing params like evalName vs requestId.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (Evaluate) and resource (an endpoint / any endpoint over many inputs), and scopes it to non-deterministic answers (LLM / RAG / search) versus deterministic inputs. It conveys what the tool does distinctly, though it never names the closest siblings (assert_response, run_vander_checks) to make the boundary explicit.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Clear context for when this applies — endpoints whose answers vary, or any endpoint across many inputs — and it explains the nested `judge` mode for grading what code cannot. There are no explicit exclusions or named alternatives, so the agent must infer when to prefer assert_response or run_vander_checks.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server

Other Tools