Skip to main content
Glama
Sign In

IA-QA — 152 QA & Dev Tools for AI Agents

by io.github.JcJamet
OverviewDiscussionsAdmin

run_semantic_tests

Read-onlyIdempotent

Semantic assertion primitive: compare actual vs expected text pairs using cosine similarity + ROUGE-L. Two modes: tfidf (default, free, no API key) or embeddings (OpenAI text-embedding-3-small, BYOK, true semantic similarity). Returns per-case PASS/FAIL verdicts and an overall verdict. CI-ready: pipe the JSON verdict field to gate a build.

Input Schema

NameRequiredDescriptionDefault
modeNotfidf (default): fast, free, lexical. embeddings: OpenAI text-embedding-3-small, true semantic similarity, requires api_key.
casesYesArray of (actual, expected) pairs to evaluate.
api_keyNoOpenAI API key — required only when mode is embeddings.
thresholdsNoPass/fail thresholds (defaults: cosine 0.75, rouge_l 0.5).
require_allNoIf true (default), all cases must pass for overall PASS. If false, at least one case passing returns PASS.

Output Schema

NameRequiredDescriptionDefault
modeNo
totalNo
failedNo
passedNo
resultsNo
verdictNo

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed1 schema field changed
    • addedInput schema / examples
      Added value: +[
+  {
+    "cases": [
+      {
+        "actual": "Paris is the capital of France",
+        "expected": "The capital of France is Paris"
+      }
+    ]
+  }
+]
  2. Changed1 schema field changed
    • changedInput schema / properties / api_key / description
      Previous value: -"OpenAI API key — required only when mode is embeddings."New value: +"OpenAI API key — required only when mode is embeddings."
  3. Changed1 schema field changed
    • addedOutput schema / properties
      Added value: +{
+  "failed": {
+    "type": "number"
+  },
+  "mode": {
+    "type": "string"
+  },
+  "passed": {
+    "type": "number"
+  },
+  "results": {
+    "items": {
+      "type": "object"
+    },
+    "type": "array"
+  },
+  "total": {
+    "type": "number"
+  },
+  "verdict": {
+    "type": "string"
+  }
+}
  4. Changed1 schema field changed
    • changedOutput schema / (root)
      Previous value: -nullNew value: +{
+  "additionalProperties": true,
+  "type": "object"
+}
  5. First observed

TDQS

A4.1/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already cover the safety profile (readOnly, idempotent, non-destructive). The description adds meaningful context beyond them: the BYOK/API-key requirement for embeddings mode, that tfidf needs no key, and the per-case plus overall verdict behavior. Good added value though not exhaustive on edge cases.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Four tight sentences, front-loaded with the core purpose, then modes, then output and CI use. Every sentence carries information with no waste.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With an output schema, full annotation coverage, and 100% schema description coverage, the description provides everything else an agent needs — mode tradeoffs, key requirement, and verdict output — to select and invoke it correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents all five parameters including mode enum and thresholds. The description reinforces mode semantics and the API-key dependency, but adds little beyond the schema for thresholds or require_all, so the baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource — comparing actual vs expected text pairs via cosine similarity + ROUGE-L — plus the PASS/FAIL verdict output. Purpose is unambiguous, but it doesn't explicitly distinguish itself from sibling similarity tools like similarity_score, embedding_similarity, or compare_responses, so no sibling differentiation.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explains the two modes and when to pick each (tfidf default free/no key vs embeddings BYOK/true semantic), and notes the CI-ready gating use case. It stops short of naming alternative tools or stating when-not to use this one.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

All IA-QA — 152 QA & Dev Tools for AI Agents toolsInspect
Try in Browser
One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources