evaluate_run
Evaluates a verified run against gold labels and writes a JSON report with MAE/MSE/exact-match, initial-error AUROC, and optional localization metrics.
Instructions
Evaluate a verified run against gold labels (MAE/MSE/Exact, initial-error AUROC, optional localization).
results_path must be <run_dir>/ta2_claim_results.jsonl; gold rows need problem_id and gold_label or likert_score in {-2..2}. The JSON report is written to out_path (outside the run directory) and returned.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"results_path": "<results_path>",
"gold_path": "<gold_path>",
"out_path": "<out_path>"
},
"name": "evaluate_run"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| out_path | Yes | ||
| gold_path | Yes | ||
| input_path | No | ||
| results_path | Yes | ||
| allow_partial | No | ||
| annotations_path | No | ||
| annotation_source | No | ||
| str_baseline_path | No | ||
| localization_traces_path | No |