Skip to main content
Glama

LLM Cutover

Run an LLM cutover audit

run_cutover_audit
Read-only

Measures how often a cheaper model (Sage 2) agrees with your current LLM on your samples, the confidence threshold for routing, and the monthly cost now vs a hybrid. Returns the full report as JSON.

Input Schema

NameRequiredDescriptionDefault
modeYesyesno: a yes/no decision per text. extract: fields pulled out of each text.
usageNo
fieldsNoextract only. Up to 12 fields: {"name": {"type": "string|number|integer|boolean|date|money", "definition": "..."}}
targetNoAgreement you need on the calls the cheap model keeps.
currentYesYour current model: {"model": "<id from GET /api/models>", "effort": "<level>"} or custom prices {"in": USD per 1M input tokens, "out": USD per 1M output tokens}.
samplesYesReal inputs with the answer your CURRENT model gave. Array of objects or a JSONL/CSV string. yesno: {"text","label":"yes|no"}; extract: {"text","expected":{...}}. Each text ≤ 4000 chars. Min 5. Max 10 anonymous, 25 free on the website, 500 yesno / 200 extract with a pass.
questionNoyesno only. The yes/no question your current model answers, e.g. "Is this ticket about billing?"

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observed

TDQS

B3.4/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The annotations already establish the safety profile: readOnlyHint=true, openWorldHint=true, and idempotentHint=false. The description adds that the operation produces a full JSON report and quantifies agreement, threshold, and cost, which is useful context beyond the annotations. It does not disclose turnaround time, sample-size throttling behavior, or why the call is not idempotent, which leaves a few behavioral gaps.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two sentences with no filler, and the main capability is front-loaded before the return-format note. The first sentence is dense, packing model agreement, threshold, and cost into one clause, but every phrase earns its place. Slightly harder to scan than it could be.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

This is a moderately complex tool with 7 parameters, nested objects, and no output schema. Because there is no output schema, the description should say more about the report structure than 'returns the full report as JSON'. It covers purpose and headline outputs but omits what the report contains and any limits on sample counts or runtime.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 86%, so the schema already documents mode, fields, target, current, and samples in detail, including the enum semantics. The description echoes the concepts (samples, confidence threshold, cost) without adding syntax or format detail beyond the schema. A baseline of 3 is appropriate when the schema already provides most of the meaning.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb and resource: it measures agreement between a cheaper model (Sage 2) and the caller's current LLM, derives a routing confidence threshold, and compares monthly cost. That is far more than a restatement of the name. It does not explicitly contrast with the sibling list_models, but that tool is clearly unrelated, so the only gap is the missing sibling routing.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The framing implies the scenario (evaluating a cutover from the current model to a cheaper one), and it mentions 'your samples' and 'your current LLM', which signals prerequisites. However, it never states when to choose this tool over alternatives, what to do first (for example, list_models to obtain a valid model id), or what conditions make an audit inappropriate. Usage is inferable rather than stated.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources