Skip to main content
Glama
Sign In

IA-QA — 152 QA & Dev Tools for AI Agents

by io.github.JcJamet
OverviewDiscussionsAdmin

consistency_check

Read-onlyIdempotent

Compare multiple LLM responses to the same prompt and detect inconsistencies using Jaccard word-overlap similarity and fact drift (number comparison). Fast, deterministic, no API key needed. Limitations: relies on surface-level word matching — "Paris is the capital of France" vs "Paris is the French capital" may score low despite semantic equivalence. True semantic consistency: run_semantic_tests in embedding mode. Essential for determinism testing.

Input Schema

NameRequiredDescriptionDefault
responsesYesArray of 2+ LLM responses to compare (same prompt, different runs)
check_factsNoCheck for contradictory numbers/facts across responses (default: true)

Output Schema

NameRequiredDescriptionDefault
verdictNo
fact_driftNo
avg_similarityNo
response_countNo
pairwise_scoresNo
fact_contradictionNo
length_variance_percentNo

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed1 schema field changed
    • addedInput schema / examples
      Added value: +[
+  {
+    "responses": [
+      "The capital of France is Paris.",
+      "Paris is the capital of France."
+    ]
+  }
+]
  2. Changed1 schema field changed
    • addedOutput schema / properties
      Added value: +{
+  "avg_similarity": {},
+  "fact_contradiction": {},
+  "fact_drift": {},
+  "length_variance_percent": {},
+  "pairwise_scores": {},
+  "response_count": {
+    "type": "number"
+  },
+  "verdict": {}
+}
  3. Changed1 schema field changed
    • changedOutput schema / (root)
      Previous value: -nullNew value: +{
+  "additionalProperties": true,
+  "type": "object"
+}
  4. First observed

TDQS

Score is being calculated.

All IA-QA — 152 QA & Dev Tools for AI Agents toolsInspect
Try in Browser
One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources