Skip to main content
Glama

IA-QA — 152 QA & Dev Tools for AI Agents

guardrail_test

Read-onlyIdempotent

Test an LLM response against a set of guardrail rules: must-include, must-not-include, max length, required format, language, forbidden patterns, and custom regex. Returns pass/fail per rule.

Input Schema

NameRequiredDescriptionDefault
rulesYesArray of guardrail rules to check
responseYesThe LLM response to test

Output Schema

NameRequiredDescriptionDefault
passNo
ruleNo
labelNo
valueNo
detailNo
failedNo
passedNo
resultsNo
all_passedNo
total_rulesNo

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed1 schema field changed
    • addedInput schema / examples
      Added value: +[
      +  {
      +    "response": "Your refund is approved.",
      +    "rules": [
      +      {
      +        "type": "must_include",
      +        "value": "refund"
      +      },
      +      "no_pii"
      +    ]
      +  }
      +]
  2. Changed1 schema field changed
    • addedOutput schema / properties
      Added value: +{
      +  "all_passed": {
      +    "type": "boolean"
      +  },
      +  "detail": {
      +    "type": "string"
      +  },
      +  "failed": {},
      +  "label": {},
      +  "pass": {
      +    "type": "boolean"
      +  },
      +  "passed": {},
      +  "results": {},
      +  "rule": {},
      +  "total_rules": {
      +    "type": "number"
      +  },
      +  "value": {}
      +}
  3. Changed1 schema field changed
    • changedOutput schema / (root)
      Previous value: -nullNew value: +{
      +  "additionalProperties": true,
      +  "type": "object"
      +}
  4. First observed

TDQS

B3.4/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly, idempotent, non-destructive and closed-world, so the safety profile needs no restating. The description adds the response contract ("Returns pass/fail per rule"), which is real value, but says nothing about how invalid or unrecognized rule types are handled, or whether evaluation short-circuits.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two tight sentences, front-loaded with the core action and closed with the return shape; nothing is padded. The single flaw is that the inline rule enumeration is long enough to be misleading rather than merely terse, given the enum mismatch.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With an output schema present, the return values are covered, and the description correctly conveys the input contract at a high level. However, for a tool whose whole difficulty is authoring valid rules, the description neither enumerates the actual supported rule types nor clarifies the string-vs-object rule shape shown in the example, leaving the agent to reverse-engineer the schema.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3; the description adds little beyond restating rule categories. Worse, its enumeration is partly wrong – "language" is not in the type enum, and fully half the enum (min_length, starts_with, ends_with, word_count_max/min) goes unmentioned, while the example's bare-string shorthand "no_pii" is nowhere explained.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ("Test an LLM response against a set of guardrail rules") and enumerates the constraint categories, so the agent knows this is a rule-based response validator rather than a scanner. It does not differentiate against adjacent siblings like shield_analyze, prompt_injection_scan or toxicity_scan, and two enumerated categories ("language", "forbidden patterns") have no matching rule type in the schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is only implied by the phrasing "Test an LLM response" – the agent can infer this is for validating generated text against explicit rule sets. There is no statement of when to prefer this over the many sibling scanners, nor any prerequisite or exclusion (e.g. what to do with an unknown rule type).

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.