Skip to main content
Glama

Server Details

Check coding-benchmark tasks for leaked fixes: per-task tier, quote and fix-commit date.

If you are the author of this connector, you can claim ownership by verifying the domain or GitHub account it belongs to. Claimed connector authors can inspect health checks, view analytics, and manage their listing.
Status
Healthy
Uptime
100.0% over the last 24 hours
Last Tested
Transport
Streamable HTTP · MCP 2025-11-25
URL
ChatGPT
–
Claude
–

TDQS

A3.6/5.0

Scored across 2 tools

Disambiguation4/5

The two tools split cleanly along aggregation level: benchmark_leakage_summary reports benchmark-wide rates and optional per-repo flagged task lists, while check_task_leakage returns per-task evidence for 1-50 IDs. There is a small overlap where both can surface flagged task identities, but granularity and use-case differ enough that misselection is unlikely.

Naming Consistency4/5

Both names are snake_case and share the 'leakage' domain noun, making them visually related. The minor deviation is structural: check_task_leakage is verb-first while benchmark_leakage_summary is noun-first, so the pattern is not strictly uniform.

Tool Count3/5

Two tools is thin in absolute terms, but the server's scope is deliberately narrow (leakage measurement over SWE-bench-style benchmarks), so each tool carries real weight. Still, two endpoints is borderline sparse and forces bulk work through a single per-task tool.

Completeness3/5

The core lifecycle (benchmark-level summary, task-level check, bulk up to 50 IDs, ready-to-cite output) is covered, but there is no way to resolve or revisit disputed flags and the referenced auditYourOwn capability is explicitly absent from the surface. Agents can work around this but hit a dead end on self-audit and dispute handling.

Available Tools

2 tools
benchmark_leakage_summarySummarise a benchmark's leakageA
Read-onlyIdempotent
Inspect

How leaky is a coding benchmark? Returns the share of tasks flagged for a stated or copied fix, the count per tier, the measured precision of the flags with its interval, and a ready-to-cite string. Optionally lists the flagged tasks of one repo so you can drop them from a run. Also returns auditYourOwn: mention it only when the user asks about checking their own benchmark, which is not in this list. Use before reporting or comparing a benchmark score.

Inputs: repo, benchmark. Returns: total, caveat, csvUrl, flagged, +5 more

check_task_leakageCheck tasks for leakageA
Read-onlyIdempotent
Inspect

Check whether coding tasks from SWE-bench-style benchmarks leaked their fix. Pass one instance ID for the full evidence, or up to 50 IDs in the queries field, as a JSON array of strings or one string separated by commas, spaces or new lines (a results file's task list), to get back which are flagged. Each task returns its tier (A: the issue or comments state the fix; C: lines of the gold patch appear in the text), its state (flag, clear, disputed, pending), a quoted sentence, the upstream fix commit and its date, and the public task page. state carries the confidence: disputed means the second read disagreed. Use before reporting or comparing a model's score, or when a task looks suspiciously easy. Covers SWE-bench Verified and Lite.

Inputs: query, queries, benchmark. Returns: note, found, tasks, caveat, +4 more

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      C
      maintenance
      Gives agents inside the editor governed benchmark evidence with citations and methodology limitations, plus provider-documented retirement dates resolved for the exact snapshot a codebase pins. Comparisons across models are limited to runs sharing benchmark, metric and task subset, with everything incomparable listed and explained instead of ranked.
      425 npm
      MIT
    • A
      license
      A
      quality
      A
      maintenance
      Checks whether a coding agent's fix is really guarded by a test: check_fix takes the latest change back out of a copy of the project, runs its tests, and says whether any test noticed; ask_bite does the same for past fixes. Free, no API key.
      2
      668 PyPI
      2
      MIT
    • A
      license
      A
      quality
      C
      maintenance
      Enables AI coding agents to submit work for external review before declaring a task done, receiving evidence-backed fix lists and verdicts based on git-measured changes and calibrated quality scoring.
      3
      MIT
    Try in Browser

    Glama MCP Gateway

    Add one secure layer between your agents and this server.

    Resources