falsepass

Overview Schema Source Related Servers Discussions Admin

falsepass_verify_fix

Reruns a stored repro against a patched source to list changed output lines and, optionally, detect correct-answer rows the patch breaks.

Instructions

Check a patch: rerun the stored repro (repro.py, else repro_auto.py) with the environment taken from source and list the lines that change (no stored output: the unpatched script runs first and is stored as the baseline). source is a patched package folder, or a patched copy of the audit's src/ without pyproject.toml, which is laid over the release. known_good >= 0 also scores every correct-answer row (P1, P2, P14) with and without the patch, on every task (0) or a sample, and lists the tasks whose correct answer the patch stops paying. Runs in the background like an audit (status "running": call falsepass_audit_status).

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "env": "<env>",
      "source": "<source>"
    },
    "name": "falsepass_verify_fix"
  }
}

Input Schema

Name	Required	Description	Default
`env`	Yes	the audit's name as falsepass_init returned it: the folder under workdir/audits/
`wait`	No	seconds to wait for the background job before returning its progress instead (max 600)
`source`	Yes	a patched package folder, or a patched copy of the audit's src/ (laid over the release); relative to workdir
`offline`	No	read Hugging Face datasets from the local cache without checking the Hub first: seconds saved per run once the data is cached (a dataset not cached yet fails to load)
`workdir`	No	the folder whose audits/ holds the audits (default: the current directory)	.
`known_good`	No	-1: off; 0: also score every correct-answer row with and without the patch; N: on a sample of N tasks

Schema Changelog

Changes observed during successful MCP inspections.

First observedv0.15.0Nov 9, 2026

TDQS

A4/5.0

Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations, so the description carries the full burden, and it does disclose real behavior: the job runs in the background with status 'running', an unpatched baseline is computed and stored when no stored output exists, and known_good triggers extra correct-answer scoring on all or sampled tasks. It omits cost/runtime expectations beyond the wait default, what happens to that stored baseline on repeat calls, and error modes when source doesn't match the release.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness3/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the verb, and each sentence carries information, but the parenthetical-heavy nesting ('(no stored output: the unpatched script runs first and is stored as the baseline)') makes it hard to scan and forces the agent to re-parse. It could be split into shorter statements without losing content.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 6-parameter, background-executing tool with no output schema, the description covers the key mechanics: what is compared, where the baseline comes from, what known_good adds, and how to poll status. The one gap is that the shape of the reported 'lines that change' result is not described, which matters given there is no output schema.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so baseline is 3; the description still adds genuine meaning, notably the two accepted shapes of `source` (a patched package folder, or a patched copy of the audit's src/ without pyproject.toml, which is laid over the release) and the effect of known_good >= 0 extending scoring to P1/P2/P14 rows. That is more than the schema restates.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and object ('Check a patch'), then names the exact mechanism: rerun the stored repro (repro.py, else repro_auto.py) with the environment from `source` and list the lines that change. That distinguishes it from the plain audit/repro siblings, since this is the post-patch verification step, and an agent can tell what it produces without opening the schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage context is clear (verifying that a patch fixes the repro, optionally that it doesn't break known-good answers), and it routes the agent correctly for follow-up: when status is 'running', call falsepass_audit_status. It doesn't explicitly say when to prefer this over falsepass_repro or falsepass_audit, so it stops short of full when/when-not guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server