Skip to main content
Glama

Design an eval suite

design_eval_suite
Read-onlyIdempotent

Build an evaluation suite for a planned AI feature with AI PM Lab's method: test cases (typical, edge, out of scope, attacks), graders with a judge rubric, pass bars, and a JSON test set. Returns instructions to follow with the user's text.

Tool call template

This tool declares no required parameters. Add optional parameters from the input schema as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {},
    "name": "design_eval_suite"
  }
}

Input Schema

NameRequiredDescriptionDefault
featureNoWhat the AI feature does, who uses it, and what it must never do

Output Schema

NameRequiredDescriptionDefault
textYesThe answer in markdown, as shown to the user; for the guides, instructions to follow with the user's text
linksYesThe AI PM Lab pages the answer links to, where each idea can be tried hands-on

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed1 schema field changed
    • changedOutput schema / (root)
      Previous value: -nullNew value: +{
      +  "$schema": "https://json-schema.org/draft/2020-12/schema",
      +  "additionalProperties": false,
      +  "properties": {
      +    "links": {
      +      "description": "The AI PM Lab pages the answer links to, where each idea can be tried hands-on",
      +      "items": {
      +        "type": "string"
      +      },
      +      "type": "array"
      +    },
      +    "text": {
      +      "description": "The answer in markdown, as shown to the user; for the guides, instructions to follow with the user's text",
      +      "type": "string"
      +    }
      +  },
      +  "required": [
      +    "text",
      +    "links"
      +  ],
      +  "type": "object"
      +}
  2. First observed

TDQS

A3.7/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The annotations already cover the safety profile (readOnly, idempotent, non-destructive, closed-world), so the bar is lower. The description still adds genuinely useful context that annotations cannot convey: "Returns instructions to follow with the user's text," which tells the agent this is an instruction/prompt generator rather than a direct executor — a behavioral fact that changes how the result must be handled. It doesn't say anything about length, latency, or formatting of that output, keeping it below 5.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, no filler, and the core action is front-loaded before the enumerated artifacts. The final sentence ("Returns instructions to follow with the user's text") is slightly awkward in phrasing but carries real information that would be costly to drop.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so the description needn't explain return values, and it correctly doesn't try. With one optional parameter and full schema coverage, the main things an agent needs — what it builds and that the result is instructions to apply — are both present. The only gap is that required-parameter count is zero and the description never says what happens if `feature` is omitted.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

There is a single parameter with 100% schema description coverage, and the schema defines what to include (what the feature does, who uses it, what it must never do). The description only gestures at this with "a planned AI feature" and adds no format, granularity, or length guidance beyond the schema's 50000-char cap. Schema does the work here, so baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

Names a specific verb and resource ("Build an evaluation suite") and enumerates the concrete artifacts produced: test cases across four categories, graders with a judge rubric, pass bars, a JSON test set. That is far more specific than a tautology. It stops short of a 5 because it doesn't distinguish itself from adjacent siblings like write_acceptance_criteria or review_ai_spec, which an agent could plausibly confuse with eval design.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

"For a planned AI feature" implies the usage context (pre-build, not post-incident or spec review), which is enough to infer when this fires. But there is no explicit statement of when NOT to use it and no named alternative among the 18 siblings. Usage is implied, not guided.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources