falsepass_reach
Counts how many tasks a failing probe reaches and how often completions fail or earn full reward, after an audit, using selected probes, templates, or a task sample.
Instructions
After an audit: how many tasks a failing probe reaches. Scores only the rows asked for, e.g. probe=["P15/denied"] (a code, or code/variant as in the results), plus the reference (P1) as a control, on every task (tasks=0) or a sample of tasks. templates counts completions of your own instead of writing a script (rows U1/t1, t2, ...): "{answer}", "{wrong}", "{answer_in_format}" and "{wrong_in_format}" are filled per task, other braces stay, e.g. "\boxed{{answer} \text{ is wrong}}"; expect "zero" counts where they are paid, "full" where they are not paid in full. Each template row also returns first_completion, the text it became on the first task. Returns per row the tasks counted, how many failed (expected 0: paid; expected full: not paid in full), full-reward and error counts, the failing task keys and up to three failing completions. Per-example probes only (P1-P9, P13-P17, A1, X1). Runs in the background like falsepass_audit: status "running" means call falsepass_audit_status. Whole datasets whose reward starts a process per call (uv scripts) take minutes to an hour.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"env": "<env>"
},
"name": "falsepass_reach"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| env | Yes | the audit's name as falsepass_init returned it: the folder under workdir/audits/ | |
| wait | No | seconds to wait for the background job before returning its progress instead (max 600) | |
| probe | No | probe rows to count, e.g. ["P15/denied"] (code, or code/variant as in the results) | |
| tasks | No | how many tasks to score (0 = every task in the dataset) | |
| expect | No | for templates: "zero" counts where they are paid, "full" where they are not paid in full | zero |
| source | No | take the environment from this patched copy (relative to workdir) | |
| offline | No | read Hugging Face datasets from the local cache without checking the Hub first: seconds saved per run once the data is cached (a dataset not cached yet fails to load) | |
| workdir | No | the folder whose audits/ holds the audits (default: the current directory) | . |
| templates | No | completions of your own to count, with {answer}, {wrong}, {answer_in_format} or {wrong_in_format} filled per task |