Skip to main content
Glama

IA-QA — 152 QA & Dev Tools for AI Agents

metamorphic_check

Read-onlyIdempotent

Reference-free stability check: does an assistant's answer stay the same when the QUESTION is transformed (case, typo, paraphrase, reorder, translation)? You bring the outputs; no model is called (tfidf mode is deterministic and free). Default thresholds: case .95, typo .90, paraphrase .80, reorder .80, translation .75 (embeddings only), specialization .60 (advisory, never gated). Returns PASS / FAIL / INVALID — INVALID when the BASE answer is a refusal or too short, so a model that refuses everything never scores 1.0. Pair with run_semantic_tests for correctness.

Input Schema

NameRequiredDescriptionDefault
baseYesThe original run.
modeNotfidf (default, lexical: gate on case/typo, read paraphrase as a trend) or embeddings (OpenAI, needs api_key; required for translation).
api_keyNoOpenAI key, embeddings mode only.
variantsYesAnswers to transformed versions of the question.
thresholdsNoPer-relation overrides, 0-1. Calibrate on your own data before gating.
require_allNoDefault true: every gated variant must pass. Keep it for a gate — false lets one easy typo row hold the run at PASS while a paraphrase fails.
baseline_guardNoFloor on the BASE answer; failing it returns INVALID.

Output Schema

NameRequiredDescriptionDefault
modeNo
summaryNo
verdictNo
weakestNo
baselineNo
variantsNo
warningsNo
thresholdsNo

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed16 schema fields changed
    • addedInput schema / examples
      Added value: +[
      +  {
      +    "base": {
      +      "output": "Paris is the capital and largest city of France."
      +    },
      +    "variants": [
      +      {
      +        "output": "Paris is the capital and largest city of France.",
      +        "relation": "case"
      +      }
      +    ]
      +  }
      +]
    • changedInput schema / properties / api_key / description
      Previous value: -"OpenAI API key — required only when mode is embeddings."New value: +"OpenAI key, embeddings mode only."
    • changedInput schema / properties / base / description
      Previous value: -"The reference run: the original question and the answer your system produced for it."New value: +"The original run."
    • changedInput schema / properties / base / properties / output / description
      Previous value: -"Required — the answer your system produced for the original question."New value: +"The answer to the original question."
    • changedInput schema / properties / base / properties / question / description
      Previous value: -"Optional — the original question, echoed back in the output for readability."New value: +"Optional, echoed back."
    • changedInput schema / properties / baseline_guard / description
      Previous value: -"Correctness floor applied to the BASE answer before anything is scored. Failing it returns INVALID, not FAIL."New value: +"Floor on the BASE answer; failing it returns INVALID."
    • changedInput schema / properties / baseline_guard / properties / min_length / description
      Previous value: -"Minimum base answer length in chars (default 40)."New value: +"Default 40 chars."
    • changedInput schema / properties / baseline_guard / properties / must_not_match / description
      Previous value: -"Refusal patterns, matched case-insensitively in the FIRST 200 CHARS of the base answer (refusals lead; matching anywhere would flag a long correct answer that merely mentions one). Replaces the FR+EN default list, never merges — pass [] to disable."New value: +"Refusal patterns, checked in the first 200 chars. Replaces the default FR+EN list; [] disables."
    • changedInput schema / properties / mode / description
      Previous value: -"tfidf (default): free, lexical, deterministic — but a genuine paraphrase rarely reaches 0.80, so gate on case/typo and treat paraphrase as a trend. embeddings: OpenAI text-embedding-3-small, true semantic similarity, requires api_key. translation requires this mode."New value: +"tfidf (default, lexical: gate on case/typo, read paraphrase as a trend) or embeddings (OpenAI, needs api_key; required for translation)."
    • changedInput schema / properties / require_all / description
      Previous value: -"If true (default), every gated variant must pass. KEEP THE DEFAULT for any run you gate on. Setting it false is not a tolerance dial but an off switch: relations have asymmetric pass rates (a typo variant usually scores ~1.0 because the answer really is identical), so one trivial row is enough to hold the whole run at PASS while a paraphrase fails. When that happens the result carries an explicit warning naming the failed rows."New value: +"Default true: every gated variant must pass. Keep it for a gate — false lets one easy typo row hold the run at PASS while a paraphrase fails."
    • changedInput schema / properties / thresholds / description
      Previous value: -"Per-relation threshold overrides. Calibrate on your own corpus before gating — the defaults are starting points, not measurements."New value: +"Per-relation overrides, 0-1. Calibrate on your own data before gating."
    • changedInput schema / properties / variants / description
      Previous value: -"Answers produced for transformed versions of the same question, each tagged with the relation that was applied."New value: +"Answers to transformed versions of the question."
    • removedInput schema / properties / variants / items / properties / id / description
      Removed value: -"Optional identifier (default: variant_<n>)."
    • changedInput schema / properties / variants / items / properties / output / description
      Previous value: -"Required — the answer your system produced for the transformed question."New value: +"The answer to the transformed question."
    • changedInput schema / properties / variants / items / properties / question / description
      Previous value: -"Optional — the transformed question, echoed back for readability."New value: +"Optional, echoed back."
    • changedInput schema / properties / variants / items / properties / relation / description
      Previous value: -"Transformation applied to the QUESTION. Decides the metric and the default threshold."New value: +"Transformation applied to the question; sets the metric and threshold."
  2. Changed1 schema field changed
    • changedInput schema / properties / require_all / description
      Previous value: -"If true (default), every gated variant must pass. If false, one passing gated variant is enough."New value: +"If true (default), every gated variant must pass. KEEP THE DEFAULT for any run you gate on. Setting it false is not a tolerance dial but an off switch: relations have asymmetric pass rates (a typo variant usually scores ~1.0 because the answer really is identical), so one trivial row is enough to hold the whole run at PASS while a paraphrase fails. When that happens the result carries an explicit warning naming the failed rows."
  3. Added

TDQS

A4.6/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations declare readOnly/idempotent/non-destructive and the description is consistent, adding context annotations cannot carry: no model invocation, deterministic vs paid mode, default thresholds, and the crucial INVALID rule when the BASE answer is a refusal or too short — which pre-empts a gaming failure mode. That is real behavior disclosure, not restatement.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the core question it answers, and every sentence carries information (modes, thresholds, verdict semantics, pairing guidance). It is dense with parentheticals and slightly run-on, which costs a little scannability.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 7-parameter tool with nested objects and an output schema, the description covers everything an agent needs to decide and call: input expectations, mode trade-offs, threshold defaults, verdict meanings, and baseline guard behavior. Return-value detail is correctly delegated to the output schema.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so baseline is 3, but the description goes further by stating the actual default threshold values per relation (case .95, typo .90, paraphrase .80, reorder .80, translation .75, specialization .60 advisory) and the rationale behind require_all=true, which the schema does not provide.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific operation (reference-free stability check) on a specific resource (an assistant's answer under question transformations) and enumerates the transformations. It explicitly separates itself from run_semantic_tests (correctness) and implicitly from similarity/consistency siblings, so an agent can route without opening the schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Says what you must supply ('You bring the outputs; no model is called'), how the modes differ (tfidf deterministic and free vs embeddings needing api_key), and names the complement tool for correctness. It lacks an explicit 'do not use this when…' exclusion, so it stops short of full alternatives guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.