Check reported metrics against the ledger
audit_claimsTrace every metric claim in reports to the vault score that produced it, then label each as traced, mismatched, orphaned, cherry-picked, stale, or untrusted-source.
Instructions
Trace every metric claim in reports to the score that produced it.
Reads Markdown, text and notebook markdown in the workspace (or only paths, or only text),
finds claims like "AUC of 0.91", and labels each one: traced (matches a vault score),
mismatched, orphaned, cherry-picked (best of several scores, others undisclosed), stale (from a
commit no longer in history) or untrusted-source (only in MLflow/JSON run records). Free.
Tool call template
This tool declares no required parameters. Add optional parameters from the input schema as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {},
"name": "audit_claims"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| text | No | ||
| paths | No |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| claims | No | ||
| counts | Yes | claims per label | |
| records | Yes | run records read, per source | |
| summary | Yes | e.g. '12 metric claim(s), 9 traced, 3 unsupported' | |
| documents | Yes | documents read |