Skip to main content
Glama

IA-QA — 152 QA & Dev Tools for AI Agents

prompt_test_suite

Read-onlyIdempotent

Define a test suite for a prompt: provide the system prompt, user prompt, and expected output criteria. Returns a test plan with scored rubric — use this as input for manual or automated LLM evaluation.

Input Schema

NameRequiredDescriptionDefault
max_tokensNoMax token budget for the test
temperatureNoTemperature to use
user_promptYesThe user prompt to send
check_safetyNoInclude safety/PII checks in the rubric
must_includeNoRequired content (comma-separated)
system_promptYesThe system prompt under test
expected_formatNoExpected output format
must_not_includeNoForbidden content (comma-separated)
expected_behaviorNoDescription of what the LLM should do (free text)
adversarial_promptsNoAuto-generate adversarial test variants (jailbreak, injection, edge cases)

Output Schema

NameRequiredDescriptionDefault
rubricNo
categoriesNo
total_testsNo
instructionsNo
test_suite_nameNo

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed1 schema field changed
    • addedInput schema / examples
      Added value: +[
      +  {
      +    "system_prompt": "You are a helpful assistant.",
      +    "user_prompt": "Say hi"
      +  }
      +]
  2. Changed1 schema field changed
    • addedOutput schema / properties
      Added value: +{
      +  "categories": {
      +    "type": "array"
      +  },
      +  "instructions": {
      +    "type": "string"
      +  },
      +  "rubric": {},
      +  "test_suite_name": {
      +    "type": "string"
      +  },
      +  "total_tests": {
      +    "type": "number"
      +  }
      +}
  3. Changed1 schema field changed
    • changedOutput schema / (root)
      Previous value: -nullNew value: +{
      +  "additionalProperties": true,
      +  "type": "object"
      +}
  4. First observed

TDQS

A3.8/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true, idempotentHint=true, destructiveHint=false, so the safety profile is covered. The description adds meaningful context beyond that: the tool returns a scored rubric test plan and is a precursor to evaluation rather than an evaluator itself. It still doesn't say whether adversarial_prompts actually generates variants at call time or just annotates the plan, which is the one behavioral unknown left.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two tight sentences: the first states the action and required inputs, the second states the return value and downstream use. Nothing is wasted and the essential information is front-loaded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With 10 parameters, an output schema, and full annotation coverage, the description only needs to frame the tool's role and data flow, which it does. The remaining gap is routing: an agent can't tell from the description alone why it should pick this over run_semantic_tests or generate_test_cases, which matters given the crowded sibling set.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100% with per-property descriptions and an example object, so the schema carries the parameter burden. The description restates the three primary concepts (system prompt, user prompt, expected output criteria) but adds no syntax or interaction detail, such as how must_include/must_not_include feed the rubric or how adversarial_prompts changes the returned plan. Baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

Specific verb+resource: 'Define a test suite for a prompt,' plus the required inputs (system prompt, user prompt, expected criteria) and the artifact returned (a test plan with scored rubric). It is clearly a planning/authoring tool rather than an execution tool, which separates it from run_semantic_tests or guardrail_test, though it never names those siblings.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The phrase 'use this as input for manual or automated LLM evaluation' implies a workflow position (build the plan first, then evaluate), which is useful guidance. But there is no explicit when-not clause and no named alternative for agents that want to actually run the suite rather than define it.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.