falsepass_audit
Installs an environment in a throwaway virtualenv and scores known-good and known-bad completions to detect false passes and false fails before training. Runs in the background; check progress with its status tool.
Instructions
Probe one environment set up with falsepass_init: installs it in a throw-away virtualenv and scores known-good and known-bad completions with its own reward code.
Takes from under a minute to several minutes, so it runs in the background: this call waits up to wait seconds (max 600). If the audit is done, it returns the headline numbers, the failing probes with one example completion each, and static hints (status "finished"); otherwise {"status": "running", ...}: then call falsepass_audit_status. Calling it again while an audit of env runs does not start a second one. n is the number of tasks sampled; search > 0 adds mutation search (scorer calls per direction and example; 120 is a good value). source installs the environment from a local folder (a patched copy, relative to workdir) instead of its release: the whole probe suite against a fix.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"env": "<env>"
},
"name": "falsepass_audit"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| n | No | tasks sampled from the dataset (default 25) | |
| env | Yes | the audit's name as falsepass_init returned it: the folder under workdir/audits/ | |
| wait | No | seconds to wait for the background job before returning its progress instead (max 600) | |
| search | No | mutation search: scorer calls per direction and task (0 = off; 120 is a good value) | |
| source | No | install the environment from this local folder (a patched copy, relative to workdir) instead of its release | |
| offline | No | read Hugging Face datasets from the local cache without checking the Hub first: seconds saved per run once the data is cached (a dataset not cached yet fails to load) | |
| workdir | No | the folder whose audits/ holds the audits (default: the current directory) | . |