analyze_responses
Semantically analyze N already-produced model outputs for the SAME task (the MCP counterpart to the LLM Sandbox). Without a reference: computes consensus — pairwise cosine agreement, the most-representative output, and the outlier. With a reference (ground truth): also ranks every output by closeness (token cosine + ROUGE-L composite) and names the closest. No LLM, no key, runs locally. The reference mode is gate-able in CI; the consensus/outlier mode is a signal (no reference to fail against). You bring the outputs (2+). A 2-way head-to-head with structural JSON diff is compare_responses.
Example tool call
This call uses example arguments supplied in the tool’s input schema.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"reference": "Paris is the capital of France.",
"responses": [
{
"text": "Lyon is the capital.",
"label": "A"
},
{
"text": "Paris is the capital of France.",
"label": "B"
}
]
},
"name": "analyze_responses"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| reference | No | Optional ground-truth answer. If set, each output is also ranked by closeness to it and the closest one is named. | |
| responses | Yes | The outputs to analyze (same task, N models/prompts/versions). Each item is a plain string or { "label": "GPT-4o", "text": "..." }. At least 2 required. |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| count | No | ||
| summary | No | ||
| consensus | No | ||
| reference_ranking | No |