Skip to main content
Glama

analyze_responses

Read-onlyIdempotent

Semantically analyze N already-produced model outputs for the SAME task (the MCP counterpart to the LLM Sandbox). Without a reference: computes consensus — pairwise cosine agreement, the most-representative output, and the outlier. With a reference (ground truth): also ranks every output by closeness (token cosine + ROUGE-L composite) and names the closest. No LLM, no key, runs locally. The reference mode is gate-able in CI; the consensus/outlier mode is a signal (no reference to fail against). You bring the outputs (2+). A 2-way head-to-head with structural JSON diff is compare_responses.

Example tool call

This call uses example arguments supplied in the tool’s input schema.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "reference": "Paris is the capital of France.",
      "responses": [
        {
          "text": "Lyon is the capital.",
          "label": "A"
        },
        {
          "text": "Paris is the capital of France.",
          "label": "B"
        }
      ]
    },
    "name": "analyze_responses"
  }
}

Input Schema

NameRequiredDescriptionDefault
referenceNoOptional ground-truth answer. If set, each output is also ranked by closeness to it and the closest one is named.
responsesYesThe outputs to analyze (same task, N models/prompts/versions). Each item is a plain string or { "label": "GPT-4o", "text": "..." }. At least 2 required.

Output Schema

NameRequiredDescriptionDefault
countNo
summaryNo
consensusNo
reference_rankingNo

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed1 schema field changed
    • addedInput schema / examples
      Added value: +[
      +  {
      +    "reference": "Paris is the capital of France.",
      +    "responses": [
      +      {
      +        "label": "A",
      +        "text": "Lyon is the capital."
      +      },
      +      {
      +        "label": "B",
      +        "text": "Paris is the capital of France."
      +      }
      +    ]
      +  }
      +]
  2. First observed

TDQS

A4.6/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Beyond the readOnly/idempotent/non-destructive annotations, it discloses the execution model ("No LLM, no key, runs locally") and the exact computations performed in each mode (pairwise cosine agreement, most-representative output, outlier, token-cosine + ROUGE-L composite ranking). It does not cover limits such as input size caps or runtime cost on large N, which is the main remaining gap.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Dense but front-loaded: the core action comes first, then the two modes, then the guarantee, then the sibling exclusion. The parentheticals ("the MCP counterpart to the LLM Sandbox", the composite metric breakdown) add some weight, but nearly every clause carries decision-relevant information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 2-parameter, read-only analysis tool with an output schema already present, the description covers what an agent needs: the two operating modes, the requirement of 2+ outputs, the local/no-key execution profile, and the sibling alternative. Return shapes are correctly left to the output schema.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3; the description adds real semantic weight by explaining that supplying `reference` switches the tool from consensus-only to also ranking every output and naming the closest. That clarifies the consequence of the optional parameter rather than restating its type.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ("semantically analyze N already-produced model outputs for the SAME task") and immediately splits into two well-named modes: consensus without a reference, ranking against ground truth with one. It also names the sibling compare_responses for the 2-way case, so an agent can distinguish it without opening any schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicit when-to-use: bring 2+ outputs of the same task; add `reference` when ground truth exists. Explicit when-not/alternative: "A 2-way head-to-head with structural JSON diff is compare_responses." It further clarifies the decision context by noting the reference mode is CI-gateable while consensus/outlier is only a signal.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.