Skip to main content
Glama

agent_eval_plan

Read-onlyIdempotent

Build an AI agent evaluation plan from defined capabilities and risks: size a golden set, pick metrics, write a judge rubric, add red-team cases, and set regression cadence.

Instructions

Design an evaluation plan for an AI agent: golden set sizing, metrics, judge rubric, red-team cases and regression cadence from explicit capabilities and risks.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "agent": "<agent>",
      "capabilities": "<capabilities>",
      "risks": "<risks>"
    },
    "name": "agent_eval_plan"
  }
}

Input Schema

NameRequiredDescriptionDefault
agentYes
risksYes
toolsNo
capabilitiesYes
releaseCadenceNoweekly

Output Schema

NameRequiredDescriptionDefault
kindNo
agentNo
metricsNo
redTeamNo
sourcesNo
goldenSetNo
nextToolsNo
validatedNo
limitationNo
judgeRubricNo
requirementsNo
mutatingToolsNo

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed2 schema fields changedv0.5.0
    • removedInput schema / properties / tools / items / additionalProperties
      Removed value: -false
    • changedInput schema / required
      Previous value: -[
      -  "agent",
      -  "capabilities",
      -  "tools",
      -  "risks",
      -  "releaseCadence"
      -]New value: +[
      +  "agent",
      +  "capabilities",
      +  "risks"
      +]
  2. First observedv0.4.1

TDQS

B3/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint, idempotentHint, destructiveHint=false and openWorldHint=false, so the safety profile is fully covered. The description adds that the plan is derived from supplied capabilities and risks and lists what the plan contains, which is useful, but it says nothing about depth, assumptions, or how inputs shape output. A 3 fits given the annotation coverage.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single front-loaded sentence with a compact component list and zero filler. Density is high but every clause carries information; only the trailing prepositional phrase is slightly compressed at the cost of clarity.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so return values need not be explained, and annotations cover behavior. However, for a five-parameter generator with an all-required core and a nested tools array, the description leaves the role of half the inputs unclear, which is a real gap for correct invocation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0% across five parameters, so the description must carry the burden. It names only 'capabilities' and 'risks' and gestures at cadence via 'regression cadence'; it never explains the 'agent' identifier, the 'tools' array with its required 'mutating' flag (which plausibly drives red-team case selection), or the cadence enum and default.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description names a specific verb ('Design') and resource ('an evaluation plan for an AI agent') and enumerates the plan's components: golden set sizing, metrics, judge rubric, red-team cases, regression cadence. That is enough to distinguish it from near-neighbors like agent_tool_contract or prompt_injection_checklist, though no sibling is named explicitly.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no statement of when to reach for this tool versus alternatives, no prerequisites beyond the implicit 'from explicit capabilities and risks', and no exclusions. The agent must infer the situation (pre-release eval design) from the description alone.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.