AI PM Lab

Design an eval suite

design_eval_suite

Read-onlyIdempotent

Build an evaluation suite for a planned AI feature with AI PM Lab's method: test cases (typical, edge, out of scope, attacks), graders with a judge rubric, pass bars, and a JSON test set. Returns instructions to follow with the user's text.

Input Schema

Name	Required	Description	Default
`feature`	No	What the AI feature does, who uses it, and what it must never do

Output Schema

Name	Required	Description	Default
`text`	Yes	The answer in markdown, as shown to the user; for the guides, instructions to follow with the user's text
`links`	Yes	The AI PM Lab pages the answer links to, where each idea can be tried hands-on

Schema Changelog

Changes observed during successful MCP inspections.

Changed1 schema field changedOct 12, 2026

changedOutput schema / (root)

Previous value: -nullNew value: +{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "additionalProperties": false,
+  "properties": {
+    "links": {
+      "description": "The AI PM Lab pages the answer links to, where each idea can be tried hands-on",
+      "items": {
+        "type": "string"
+      },
+      "type": "array"
+    },
+    "text": {
+      "description": "The answer in markdown, as shown to the user; for the guides, instructions to follow with the user's text",
+      "type": "string"
+    }
+  },
+  "required": [
+    "text",
+    "links"
+  ],
+  "type": "object"
+}

First observedOct 10, 2026

TDQS

A3.7/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare this read-only, idempotent, non-destructive, closed-world. The description adds the important behavioral detail that the tool 'Returns instructions to follow with the user's text' — i.e., it emits guidance rather than generating the suite itself — which is not in the annotations and materially affects how an agent should use the result. It is consistent with readOnlyHint=true.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, front-loaded with the action and deliverables, then immediately the return-behavior caveat. Dense and mostly waste-free, with only the light branding phrase 'with AI PM Lab's method' being marginal.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists and the description explicitly flags that the output is instructions to follow, so return values don't need elaboration. With one fully documented required-capable parameter and clear annotations, the definition is essentially complete, missing only sibling-relative guidance.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% and the single parameter carries its own description ('What the AI feature does, who uses it, and what it must never do'), so the schema does the heavy lifting. The description adds only the framing 'planned AI feature' and does not enrich the parameter beyond that, making the baseline 3 appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

Specific verb 'Build' plus resource 'evaluation suite' and an explicit enumeration of deliverables (test cases, graders/judge rubric, pass bars, JSON test set). An agent knows exactly what this produces. It does not, however, name any sibling (e.g., write_acceptance_criteria or review_ai_spec) to disambiguate against a dense set of similar AI‑PM tools.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

'for a planned AI feature' implies the usage context (pre‑launch/planning), but there is no explicit when-to-use, when-not, or alternative-tool routing. Usage must be inferred against siblings like write_acceptance_criteria.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

All AI PM Lab tools Inspect

Try in Browser

One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources

Need Help?