falsepass_repro
Writes a standalone script to re-score exact failing completions using the environment’s packages; optionally runs it twice in fresh virtualenvs to verify identical, reproducible output.
Instructions
Write audits//repro_auto.py: a script that depends only on the environment's own packages and re-scores the exact failing completions. probe limits it to some probes, e.g. ["P5"] or ["X1/denies_both_figures"] (code/variant). Identical output from two clean runs is what makes a lead reproducible: with run=true the script is also run twice, each in a fresh virtualenv, in the background like an audit (status "running": call falsepass_audit_status). The result says whether the runs were identical, where the first is stored (headed by its command), and how many cases still fail. script runs an existing script in the audit folder instead (e.g. a hand-written repro.py).
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"env": "<env>"
},
"name": "falsepass_repro"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| env | Yes | the audit's name as falsepass_init returned it: the folder under workdir/audits/ | |
| run | No | true: also run the script twice in fresh virtualenvs and compare (in the background) | |
| wait | No | seconds to wait for the background job before returning its progress instead (max 600) | |
| probe | No | limit the script to these probes, e.g. ["P5"] or ["X1/denies_both_figures"] (code, or code/variant) | |
| script | No | run an existing script in the audit folder instead, e.g. repro.py | |
| source | No | with run=true: take the environment from this patched copy (relative to workdir); its output is stored apart from the unpatched one | |
| offline | No | read Hugging Face datasets from the local cache without checking the Hub first: seconds saved per run once the data is cached (a dataset not cached yet fails to load) | |
| workdir | No | the folder whose audits/ holds the audits (default: the current directory) | . |