WhitePact

Evaluate Responses Against Benchmark

rai_benchmark

Read-onlyIdempotent

Evaluate pre-collected model responses against truthfulqa, bbq, or hellaswag benchmark suites to measure factual accuracy, bias, and reasoning.

Instructions

Evaluate pre-collected model responses against a standard benchmark suite. Suites: truthfulqa (factual accuracy), bbq (bias in questions), hellaswag (reasoning). Call rai_benchmark_prompts first to get the question set, collect responses, then pass them here.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "model_name": "<model_name>",
      "provider": "<provider>",
      "responses": "<responses>"
    },
    "name": "rai_benchmark"
  }
}

Input Schema

Name	Required	Description	Default
`suite`	No		truthfulqa
`provider`	Yes
`responses`	Yes	Map of sample_id → model response text
`model_name`	Yes

Schema Changelog

Changes observed during successful MCP inspections.

First observedv1.2.6Sep 11, 2026

TDQS

A4.2/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly, idempotent, non-destructive and closed-world, so safety is covered. The description adds the non-obvious behavioral requirement that responses must be collected externally before invocation, which is real context beyond the annotations. Return-format behavior is left unstated.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three sentences, front-loaded with the action, then suite meanings, then the workflow. No filler and every clause carries usable information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description cannot lean on structure to explain results, and it does not describe what an evaluation returns (scores, per-suite metrics). However, the suite semantics and the end-to-end workflow make the call path unambiguous for a nested-input evaluation tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is only 25%, so the description must compensate. It partially does by glossing each enum suite value (truthfulqa=factual accuracy, bbq=bias, hellaswag=reasoning), but 'provider' and 'model_name' get no explanation. The responses map semantics live only in the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Evaluate pre-collected model responses against a standard benchmark suite') and enumerates the three supported suites with a one-word meaning for each. It also distinguishes itself from the sibling rai_benchmark_prompts by role in the workflow (prompts generate questions, this one scores answers).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives an explicit ordering: 'Call rai_benchmark_prompts first to get the question set, collect responses, then pass them here.' That names the prerequisite tool and the condition under which this tool is appropriate. It stops short of saying when NOT to use it or which suite to pick for a given goal.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.