Skip to main content
Glama

SCRIPTMASTERLABS — Calibration Court

Model Calibration Report

court_model_report
Read-only

Calibration report for one model, computed from its resolved predictions recorded on the ledger: Brier score, expected calibration error (ECE), reliability bins with Wilson score intervals, overconfidence measures and drift over time. Free. Returns not_found when the model has no resolved predictions.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "model": "<model>"
    },
    "name": "court_model_report"
  }
}

Input Schema

NameRequiredDescriptionDefault
modelYesModel identifier to report on, exactly as recorded on the ledger (e.g. 'jev-1.14.0').
task_typeNoOptional task category filter; omit to aggregate the model's resolved predictions across all task types.

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed2 schema fields changed
    • addedInput schema / properties / model / description
      Added value: +"Model identifier to report on, exactly as recorded on the ledger (e.g. 'jev-1.14.0')."
    • addedInput schema / properties / task_type / description
      Added value: +"Optional task category filter; omit to aggregate the model's resolved predictions across all task types."
  2. First observed

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations only declare readOnlyHint=true, so the description carries most of the behavioral burden and does so well: it discloses the data source (resolved predictions on the ledger), that the call is free, and the exact error condition (not_found when no resolved predictions exist). It does not cover latency, caching, or how drift is windowed, which keeps it short of a 5.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three tight sentences: what it is, what it returns (metric list front-loaded after the resource), and the one failure mode. No filler, no repetition of the schema, and the cost note is a single word.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description must convey return content, and it does so by listing the metrics, plus it names the not_found error case. What is missing is minor: no indication of report size, time window for drift, or whether results are cached, none of which block a correct invocation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so both 'model' and 'task_type' are already documented at the schema level, and the description adds no per-parameter syntax or examples. The only added semantic is that the report is computed from resolved predictions, which indirectly clarifies what the aggregation means. Baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description names a specific verb and resource ('Calibration report for one model') and then enumerates exactly what that report contains: Brier score, ECE, reliability bins with Wilson intervals, overconfidence measures, drift over time. The 'one model' scoping also distinguishes it from a multi-model leaderboard sibling without needing to name it.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The single-model scope plus the 'Free' note and the 'Returns not_found when the model has no resolved predictions' precondition give clear context for when this call is appropriate and when it will fail. It stops short of explicitly routing the agent to a sibling for cross-model comparison or for recomputation, so it is not a full when/when-not/alternatives treatment.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources