Skip to main content
Glama

Check reported metrics against the ledger

audit_claims

Trace every metric claim in reports to the vault score that produced it, then label each as traced, mismatched, orphaned, cherry-picked, stale, or untrusted-source.

Instructions

Trace every metric claim in reports to the score that produced it.

Reads Markdown, text and notebook markdown in the workspace (or only paths, or only text), finds claims like "AUC of 0.91", and labels each one: traced (matches a vault score), mismatched, orphaned, cherry-picked (best of several scores, others undisclosed), stale (from a commit no longer in history) or untrusted-source (only in MLflow/JSON run records). Free.

Tool call template

This tool declares no required parameters. Add optional parameters from the input schema as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {},
    "name": "audit_claims"
  }
}

Input Schema

NameRequiredDescriptionDefault
textNo
pathsNo

Output Schema

NameRequiredDescriptionDefault
claimsNo
countsYesclaims per label
recordsYesrun records read, per source
summaryYese.g. '12 metric claim(s), 9 traced, 3 unsupported'
documentsYesdocuments read

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observedv0.1.0

TDQS

A4.1/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations supplied, the description carries the full disclosure burden and does well: it declares the sources scanned (Markdown, text, notebook markdown, MLflow/JSON run records), defines each classification label, and notes the tool is free. It stops short of stating explicitly that it is read-only and side-effect-free, or how it behaves on large workspaces, which keeps it below a 5.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The core action is front-loaded in the first sentence, followed by input scope and then the label taxonomy, which is dense but each item earns its place by defining a possible verdict. Minor waste only in the line-break formatting and the trailing one-word "Free."

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so return values need no explanation, and the description covers inputs, scope, and the meaning of every result label. What remains thin is operational context — how large workspaces are handled and whether scanning is bounded — but that is a modest gap for a read-only analysis tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0%, so the description must compensate, and it largely does: it explains that the tool scans the workspace by default, and that `paths` or `text` can be supplied instead — clarifying mutual exclusivity and defaults that the bare schema does not. It does not specify path format (relative, absolute, globs) or how the two parameters interact if both are given.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

"Trace every metric claim in reports to the score that produced it" gives a specific verb (trace) and resource (metric claims) plus the linkage to vault scores, which separates it from scoring/evaluation siblings. Its enumeration of outcomes (traced, mismatched, orphaned, cherry-picked, stale, untrusted-source) further pins down what the tool actually returns.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It explains input modes (whole workspace, only `paths`, or only `text`), which is useful guidance about how to invoke it, and "Free" hints there is no cost barrier to running it liberally. However, it never says when to prefer this over siblings like `score`, `evaluate`, or `vault_status`, and no exclusions are stated.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.