Skip to main content
Glama

Summarise a benchmark's leakage

benchmark_leakage_summary
Read-onlyIdempotent

How leaky is a coding benchmark? Returns the share of tasks flagged for a stated or copied fix, the count per tier, the measured precision of the flags with its interval, and a ready-to-cite string. Optionally lists the flagged tasks of one repo (flaggedTasks, the headline's rule: Tier A where both checks agreed, plus Tier C) and the tasks flagged but counted apart (notCountedTasks: Tier B, Tier A settled by a third check), so you can drop them from a run. asOf is an ISO date; the display date is inside citation. Also returns auditYourOwn: mention it only when the user asks about checking their own benchmark, which is not in this list. Use before reporting or comparing a benchmark score.

Tool call template

This tool declares no required parameters. Add optional parameters from the input schema as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {},
    "name": "benchmark_leakage_summary"
  }
}

Input Schema

NameRequiredDescriptionDefault
repoNoOptional owner/name (django/django): adds that repo's flagged tasks, tier-ordered, at most 100.
benchmarkNoWhich benchmark. Default swe-bench-verified. Display names such as 'SWE-bench Lite' are accepted.swe-bench-verified

Output Schema

NameRequiredDescriptionDefault
totalNo
caveatYes
csvUrlNo
flaggedNo
citationNo
tableUrlNo
precisionNo
auditYourOwnNo
flaggedTasksNo
notCountedTasksNo

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed1 schema field changed
    • addedOutput schema / properties / notCountedTasks
      Added value: +{
      +  "type": "array"
      +}
  2. Changed1 schema field changed
    • changedInput schema / properties / benchmark / description
      Previous value: -"Which benchmark. Default swe-bench-verified."New value: +"Which benchmark. Default swe-bench-verified. Display names such as 'SWE-bench Lite' are accepted."
  3. First observed

TDQS

A3.8/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already establish read-only, idempotent, non-destructive, closed-world behaviour, and the description adds genuinely new behavioural context: the tier taxonomy (Tier A = both checks agreed, Tier B, Tier C), the separation of flaggedTasks from notCountedTasks, and the intent behind that split. It does not cover what happens when a repo is absent from the benchmark or any failure mode, and the stray 'asOf is an ISO date' statement describes something not present in this tool's inputs, which muddies rather than clarifies.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness3/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with a clear framing question, but the body is a single sprawling sequence of parentheticals and clauses that is hard to parse on first read. Several sentences re-explain return fields already captured by the output schema, and the closing auditYourOwn line is syntactically tangled ('which is not in this list').

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a two-parameter, read-only, default-heavy tool this covers scope, tier semantics, and intended use more than adequately, and the output schema removes any need to enumerate return fields. The unexplained asOf reference and absent error/edge-case behaviour keep it short of a 5.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the schema already documents both repo (owner/name, max 100, tier-ordered) and benchmark (enum with default). The description loosely restates the repo behaviour ('lists the flagged tasks of one repo') without adding syntax beyond the schema, and its asOf/display-date sentence references a parameter that does not exist in the input schema, which introduces confusion rather than meaning.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

Opens with a concrete framing question ('How leaky is a coding benchmark?') and then states exactly what is returned: flagged-task share, per-tier counts, measured precision with interval, and a citation string. The verb+resource is specific enough for an agent to select it, but the sibling check_task_leakage is never named, so the boundary between per-task checking and benchmark-level summarisation is left implicit.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives an explicit trigger ('Use before reporting or comparing a benchmark score') and a purpose for the repo option ('so you can drop them from a run'), plus a conditional for auditYourOwn ('mention it only when the user asks about checking their own benchmark'). It lacks an explicit when-not or a named alternative tool, which is the only gap.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources