Skip to main content
Glama

compare_paired

Read-onlyIdempotent

Compare two complete experiment arms task by task to see fixes, breaks, net score change, per-task deltas and cost or latency ratios. Judges a candidate against its parent before you record a decision.

Instructions

Compare two complete arms task by task: fixes, breaks, net score change, per-task deltas, per-group tables, and cost and latency ratios. Use it to judge a candidate against its parent; use gate_check to record a decision with noise and budget checks. Both arms must be complete, error-free and match on tasks, split, repeats and anchors. Read-only. Returns the comparison object.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "config": "<config>",
      "parent_run": "<parent_run>",
      "child_run": "<child_run>",
      "aggregation": "<aggregation>"
    },
    "name": "compare_paired"
  }
}

Input Schema

NameRequiredDescriptionDefault
configYesPath of the scenario YAML file on the server host, absolute or relative to the server's working directory, for example `scenario.yaml`. It selects the workspace; a server started with --config rejects any other path.
child_runYesRun id of the candidate arm.
parent_runYesRun id of the parent (incumbent) arm.
aggregationYesHow repeats combine per task: `majority` for pass/fail outcomes (ties count as failure) or `mean` for numeric scores.

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed4 schema fields changedv0.1.9
    • addedInput schema / properties / aggregation / description
      Added value: +"How repeats combine per task: `majority` for pass/fail outcomes (ties count as failure) or `mean` for numeric scores."
    • addedInput schema / properties / child_run / description
      Added value: +"Run id of the candidate arm."
    • addedInput schema / properties / config / description
      Added value: +"Path of the scenario YAML file on the server host, absolute or relative to the server's working directory, for example `scenario.yaml`. It selects the workspace; a server started with --config rejects any other path."
    • addedInput schema / properties / parent_run / description
      Added value: +"Run id of the parent (incumbent) arm."
  2. First observedv0.1.7

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint, idempotentHint and destructiveHint=false, so safety is covered and the bar is lower. The description still contributes real behavioral context the annotations cannot: the hard preconditions on arm completeness and task/split/repeat/anchor alignment, plus what the call produces. It does not say what happens when a precondition fails, which keeps it below a 5.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The output inventory is front-loaded in the first sentence and the routing and preconditions follow in logical order with no filler. Minor redundancy: "Read-only" restates the annotations, and "Returns the comparison object" is thin given the first sentence already enumerated the contents.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description does an adequate job by enumerating the result components, and it covers the preconditions an agent must satisfy before calling. It is slightly incomplete on failure behavior and on result size or pagination, which matters for a per-task, per-group comparison over two full runs.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already explains config path resolution, run ids on both arms, and the mean/majority aggregation semantics including tie handling. The description adds no parameter-level detail beyond the passing mention of anchors, so the baseline of 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description names a specific verb and resource ("Compare two complete arms task by task") and enumerates exactly what the comparison yields: fixes, breaks, net score change, per-task deltas, per-group tables, cost and latency ratios. It also distinguishes itself from gate_check, so an agent can separate the two without opening either schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It states the selection condition explicitly ("Use it to judge a candidate against its parent") and routes the alternative decision-recording case to gate_check with its noise and budget checks. It further lists the preconditions for a valid call: both arms complete, error-free, and matching on tasks, split, repeats and anchors.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.