Check a rubric against human labels
check_against_humansRun a rubric over human-judged gold sets to measure pass/fail and per-criterion agreement, flag low-confidence escalations, and preview each disagreement.
Instructions
Run a rubric over a gold set (subjects people already judged) and report how often Verdict reached the same verdict: overall pass/fail agreement, per-criterion agreement, how many subjects the judge handed to a human (low-confidence escalation), and every disagreement with a preview. rubricPath is a local YAML/JSON rubric file, re-read on every call; gold is a local JSON array or JSON-lines file of {id?, name, inputs, expected?: {criterion: label}, human?: {passed?, criteria?}, labelSource?, split?}; choice criteria are labelled with an option id, and unreadable labels are rejected before any judge runs. Local mode only.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"gold": "<gold>"
},
"name": "check_against_humans"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| gold | Yes | Local gold-set file: JSON array or JSON lines | |
| rubric | No | Stored rubric id or inline rubric, instead of rubricPath | |
| rubricPath | No | Local rubric file (YAML or JSON) |