leaklens

Check tasks for leakage

check_task_leakage

Read-onlyIdempotent

Check whether coding tasks from SWE-bench-style benchmarks leaked their fix. Pass one instance ID for the full evidence, or up to 50 IDs in the queries field, as a JSON array of strings or one string separated by commas, spaces or new lines (a results file's task list), to get back which are flagged. Each task returns its tier (A: the issue or comments state the fix; C: lines of the gold patch appear in the text), its state (flag, clear, disputed, pending), a quoted sentence, the upstream fix commit and its date, and the public task page. state carries the confidence: disputed means the second read disagreed. Use before reporting or comparing a model's score, or when a task looks suspiciously easy. Covers SWE-bench Verified and Lite.

Tool call template

This tool declares no required parameters. Add optional parameters from the input schema as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {},
    "name": "check_task_leakage"
  }
}

Input Schema

Name	Required	Description
`query`	No	One-task form of queries. Give queries or query, not both.
`queries`	No	Several tasks at once: 1 to 50 identifiers separated by commas, spaces or new lines, or a JSON array of strings (also accepted). Each is a SWE-bench instance ID (django__django-11099), a GitHub issue or PR URL (https://github.com/django/django/pull/11099), or repo#number (django/django#11099).
`benchmark`	No	Optional: restrict the match to one benchmark. Display names such as 'SWE-bench Verified' are accepted.

Output Schema

Name	Required	Description	Default
`note`	No
`found`	No
`tasks`	No
`caveat`	Yes
`matches`	No
`notFound`	No
`flaggedCount`	No
`totalMatches`	No

Schema Changelog

Changes observed during successful MCP inspections.

Changed3 schema fields changedNov 13, 2026

changedInput schema / properties / benchmark / description

Previous value: -"Optional: restrict the match to one benchmark."New value: +"Optional: restrict the match to one benchmark. Display names such as 'SWE-bench Verified' are accepted."

changedInput schema / properties / queries / description

Previous value: -"Several tasks at once: 1 to 50 identifiers separated by commas, spaces or new lines. Each is a SWE-bench instance ID (django__django-11099), a GitHub issue or PR URL (https://github.com/django/django/pull/11099), or repo#number (django/django#11099)."New value: +"Several tasks at once: 1 to 50 identifiers separated by commas, spaces or new lines, or a JSON array of strings (also accepted). Each is a SWE-bench instance ID (django__django-11099), a GitHub issue or PR URL (https://github.com/django/django/pull/11099), or repo#number (django/django#11099)."

changedInput schema / properties / queries / maxLength
```
Previous value: -10000New value: +10100
```

First observedNov 12, 2026

TDQS

A4.3/5.0

Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly, idempotent, non-destructive and closed-world, so safety is covered; the description adds substantial behavior beyond that. It explains the tier semantics (A: the issue or comments state the fix; C: gold-patch lines appear in the text), what 'state' means and that 'disputed' means the second read disagreed, and that a single ID returns fuller evidence than the batch form. Those are non-obvious traits an agent could not get from the annotations alone.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

It is front-loaded with purpose, then input forms, then return content, then usage — a sensible order. The tier/state enumeration and 'disputed means the second read disagreed' clause are dense and slightly run-on, but each sentence carries information the agent needs, so nothing reads as filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a read-only checker with three parameters and an output schema, the definition supplies everything material: what it checks, both input shapes with documented ID formats, the 50-ID cap, benchmark scoping, return-field semantics, and when to run it. The return-value detail is arguably redundant given the output schema, but it clarifies the meaning of the tier and state values rather than merely restating field names.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3. The description adds real meaning beyond the schema: it explains the query-versus-queries tradeoff and, critically, that a single instance ID yields 'the full evidence' while the batch form only returns which tasks are flagged. That asymmetry is not stated in the schema and affects how an agent should call the tool.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb and resource: 'Check whether coding tasks from SWE-bench-style benchmarks leaked their fix.' That is far more concrete than a name restatement, and the 'Covers SWE-bench Verified and Lite' line scopes it. It never names or distinguishes itself from the sibling benchmark_leakage_summary, so the agent must infer that this is the per-task check while the sibling aggregates.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives explicit trigger conditions: 'Use before reporting or comparing a model's score, or when a task looks suspiciously easy.' That covers the common cases well. It offers no when-not guidance and does not point to benchmark_leakage_summary for aggregate use, so the alternative selection is left to inference.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

All leaklens tools Inspect

Try in Browser

Claude Code, inside the Godot 4 editorFlockbay puts an AI agent in the real Godot 4 editor: it writes GDScript, runs your game and fixes what breaks. Free, or bring your own Claude Code or Codex.flockbay.comAd

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources

Need Help?