Summarise a benchmark's leakage
benchmark_leakage_summaryHow leaky is a coding benchmark? Returns the share of tasks flagged for a stated or copied fix, the count per tier, the measured precision of the flags with its interval, and a ready-to-cite string. Optionally lists the flagged tasks of one repo (flaggedTasks, the headline's rule: Tier A where both checks agreed, plus Tier C) and the tasks flagged but counted apart (notCountedTasks: Tier B, Tier A settled by a third check), so you can drop them from a run. asOf is an ISO date; the display date is inside citation. Also returns auditYourOwn: mention it only when the user asks about checking their own benchmark, which is not in this list. Use before reporting or comparing a benchmark score.
Tool call template
This tool declares no required parameters. Add optional parameters from the input schema as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {},
"name": "benchmark_leakage_summary"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| repo | No | Optional owner/name (django/django): adds that repo's flagged tasks, tier-ordered, at most 100. | |
| benchmark | No | Which benchmark. Default swe-bench-verified. Display names such as 'SWE-bench Lite' are accepted. | swe-bench-verified |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| total | No | ||
| caveat | Yes | ||
| csvUrl | No | ||
| flagged | No | ||
| citation | No | ||
| tableUrl | No | ||
| precision | No | ||
| auditYourOwn | No | ||
| flaggedTasks | No | ||
| notCountedTasks | No |