Skip to main content
Glama

Check the workspace for leakage without scoring

check_leakage

Runs integrity checks without spending a reveal, detecting train/sealed overlap, code leakage or tampering, and files containing hidden canary labels; optionally limit code checks to specific paths.

Instructions

Run the integrity checks without spending a reveal.

Reports overlap between the sealed split and train (HV-ROW), leakage and tampering patterns in the workspace's Python files and notebooks (HV-COD), and data files carrying the hidden canary labels (HV-CAN-001). paths limits the code checks to some files or folders.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "dataset": "<dataset>"
    },
    "name": "check_leakage"
  }
}

Input Schema

NameRequiredDescriptionDefault
pathsNo
datasetYes

Output Schema

NameRequiredDescriptionDefault
datasetYes
messageYesPlain-English summary
blockingYesFindings that would refuse a score
findingsNo
unparsedNoFiles that could not be parsed
warningsYes
files_scannedYesPython files and notebooks analysed
tables_scannedYesData files compared against the hidden canary labels

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observedv0.1.0

TDQS

A3.6/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description has to carry the full behavioral burden. It does well on scope — it discloses exactly which three check categories run and that `paths` only narrows the code checks — but says nothing about whether the call is read-only, whether it is expensive, or what a failing result implies. Reasonable, but not a rich behavioral disclosure for an unannotated tool.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The key differentiator ("without spending a reveal") is front-loaded and the follow-on sentence efficiently enumerates the checks with their rule codes. Dense but each clause carries information; no filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a two-parameter read-style check tool with an output schema covering return values, the definition covers what runs and how `paths` narrows it. The main gap is the unexplained required `dataset` argument and the absence of any safety/read-only indication given there are no annotations.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0%, so the description must compensate and it only half does. It meaningfully clarifies `paths` (limits only the code checks, accepts files or folders), but the required `dataset` parameter is never mentioned, leaving its meaning and expected format entirely to guesswork.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description names a specific action ("run the integrity checks") and enumerates the three concrete check families it performs (HV-ROW split overlap, HV-COD leakage/tampering in Python files and notebooks, HV-CAN-001 canary labels). That is enough to distinguish it from scoring-style siblings, though it relies on the undefined term "reveal" to convey the distinction rather than saying plainly what it does not do.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

"Run the integrity checks without spending a reveal" gives clear selection context: use this when you want the checks but don't want to consume a scoring/reveal budget, which implicitly points at the score/evaluate siblings. It stops short of an explicit rule such as "use score when you want a graded result," and no exclusions are stated.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.