Skip to main content
Glama

run_semantic_tests

Read-onlyIdempotent

Semantic assertion primitive: compare actual vs expected text pairs using cosine similarity + ROUGE-L. Two modes: tfidf (default, free, no API key) or embeddings (OpenAI text-embedding-3-small, BYOK, true semantic similarity). Returns per-case PASS/FAIL verdicts and an overall verdict. CI-ready: pipe the JSON verdict field to gate a build.

Example tool call

This call uses example arguments supplied in the tool’s input schema.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "cases": [
        {
          "actual": "Paris is the capital of France",
          "expected": "The capital of France is Paris"
        }
      ]
    },
    "name": "run_semantic_tests"
  }
}

Input Schema

NameRequiredDescriptionDefault
modeNotfidf (default): fast, free, lexical. embeddings: OpenAI text-embedding-3-small, true semantic similarity, requires api_key.
casesYesArray of (actual, expected) pairs to evaluate.
api_keyNoOpenAI API key — required only when mode is embeddings.
thresholdsNoPass/fail thresholds (defaults: cosine 0.75, rouge_l 0.5).
require_allNoIf true (default), all cases must pass for overall PASS. If false, at least one case passing returns PASS.

Output Schema

NameRequiredDescriptionDefault
modeNo
casesNo
summaryNo
verdictNo
thresholdsNo

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed9 schema fields changed
    • addedOutput schema / properties / cases
      Added value: +{}
    • removedOutput schema / properties / failed
      Removed value: -{
      -  "type": "number"
      -}
    • removedOutput schema / properties / mode / type
      Removed value: -"string"
    • removedOutput schema / properties / passed
      Removed value: -{
      -  "type": "number"
      -}
    • removedOutput schema / properties / results
      Removed value: -{
      -  "items": {
      -    "type": "object"
      -  },
      -  "type": "array"
      -}
    • addedOutput schema / properties / summary
      Added value: +{
      +  "properties": {
      +    "avg_cosine": {},
      +    "avg_rouge_l": {},
      +    "failed": {},
      +    "passed": {},
      +    "total": {}
      +  }
      +}
    • addedOutput schema / properties / thresholds
      Added value: +{}
    • removedOutput schema / properties / total
      Removed value: -{
      -  "type": "number"
      -}
    • removedOutput schema / properties / verdict / type
      Removed value: -"string"
  2. Changed1 schema field changed
    • addedInput schema / examples
      Added value: +[
      +  {
      +    "cases": [
      +      {
      +        "actual": "Paris is the capital of France",
      +        "expected": "The capital of France is Paris"
      +      }
      +    ]
      +  }
      +]
  3. First observed

TDQS

A3.8/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already cover safety (readOnly, idempotent, non-destructive, closed-world), so the bar is lower. The description adds useful context: it returns per-case PASS/FAIL plus an overall verdict, and embeddings mode requires an external API key and incurs cost. It doesn't mention case limits (maxItems 50) or how thresholds interact with scoring.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three dense sentences, front-loaded with the core operation, then mode tradeoffs, then output and CI usage. No filler and nothing repeated unnecessarily.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 5-parameter tool with nested cases, an output schema and full annotations, this covers the essentials: what it scores, how modes differ, what it returns, and how to use in CI. Minor gaps are case-count limits and threshold interaction, largely covered by the schema.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so all parameters including mode, thresholds, api_key and require_all are already documented in the schema. The description adds only the cost/free framing for tfidf versus embeddings, which the schema description largely repeats; baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource: 'Semantic assertion primitive: compare actual vs expected text pairs using cosine similarity + ROUGE-L.' The metric and inputs are named concretely, but no sibling tool (e.g. embedding_similarity, compare_responses) is named to differentiate it.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives clear when-to-use guidance for the two modes: tfidf is 'default, free, no API key', embeddings gives 'true semantic similarity' via OpenAI BYOK. Also flags the CI gating use case. It stops short of explicit exclusions or pointing at a specific alternative tool.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.