Skip to main content
Glama
rupturetrace-mcp

rupturetrace-mcp

by Susan571

evaluate_run

Evaluates a verified run against gold labels and writes a JSON report with MAE/MSE/exact-match, initial-error AUROC, and optional localization metrics.

Instructions

Evaluate a verified run against gold labels (MAE/MSE/Exact, initial-error AUROC, optional localization).

results_path must be <run_dir>/ta2_claim_results.jsonl; gold rows need problem_id and gold_label or likert_score in {-2..2}. The JSON report is written to out_path (outside the run directory) and returned.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "results_path": "<results_path>",
      "gold_path": "<gold_path>",
      "out_path": "<out_path>"
    },
    "name": "evaluate_run"
  }
}

Input Schema

NameRequiredDescriptionDefault
out_pathYes
gold_pathYes
input_pathNo
results_pathYes
allow_partialNo
annotations_pathNo
annotation_sourceNo
str_baseline_pathNo
localization_traces_pathNo

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observedv0.1.0

TDQS

B3.2/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden. It does disclose a key side effect (a JSON report is written to out_path, deliberately outside the run directory) and that the report is returned, which is real behavioral value. It says nothing about error handling, partial-run behavior (allow_partial), or what happens when gold labels are malformed beyond the value range.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three tight sentences with the purpose front-loaded and constraints following. No filler, though the parameter constraints are packed into a single dense sentence that could be split for scanability.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

A 9-parameter evaluation tool with no output schema and no annotations needs more than this. The description covers the core data flow and metric suite, but the majority of optional parameters — notably localization_traces_path, which relates to the 'optional localization' metric it advertises — are never connected to behavior.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate and it only covers 3 of 9 parameters: results_path (with a required filename pattern), gold_path (row schema with problem_id and gold_label/likert_score in {-2..2}), and out_path. input_path, allow_partial, annotations_path, annotation_source, str_baseline_path, and localization_traces_path are entirely unexplained in both schema and description.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Evaluate a verified run against gold labels') and enumerates the metrics produced (MAE/MSE/Exact, initial-error AUROC, localization). The word 'verified' signals a dependency on the verify_run sibling, but the tool is not explicitly contrasted with any alternative, so it stops short of full sibling differentiation.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies the run must already be verified and constrains the results file location to <run_dir>/ta2_claim_results.jsonl, which is useful pre-call guidance. However, it never says when to choose this tool over siblings like assess_claim or verify_run, nor when the optional localization path should be supplied.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.