Skip to main content
Glama

Run an agent evaluation

run_agent_evaluation

Ask a product's AI agent up to 20 test questions and grade the answers against what you expect; the run is saved in Training → Evals. It spends the workspace's AI budget. Admins only.

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
casesYesThe test questions.
modelNoAn OpenRouter model id to try instead of the workspace's model.
productIdYesProduct id from list_products.

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observed

TDQS

A4.4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already mark this as a non-read-only, non-destructive, non-idempotent operation, so the safety profile is covered. The description adds value beyond that: it discloses cost (spends the workspace's AI budget), an authorization requirement (Admins only), and a persistence side effect (the run is saved in Training → Evals). It does not describe runtime behavior such as whether grading is synchronous or how long the run takes.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single sentence packed with the highest-value facts, ordered action → outcome → cost → permission. Zero filler and nothing repeated from the title.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a mutation tool with no output schema and full schema coverage, the description supplies the non-obvious operational facts an agent needs (cost, admin gate, persisted result location). It is still silent on result shape and timing, which is a minor gap given no output schema exists.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the baseline is 3; the description still adds meaning by framing `cases` as test questions and the `expect` field as the grading target, and by surfacing the 20-question cap in prose rather than leaving it buried in maxItems. It adds no guidance on `model` beyond what the schema already says.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb-action chain (ask the product's AI agent test questions and grade the answers) plus the resource (product agent) and a concrete scope limit (up to 20 questions). This cleanly separates it from look-alike siblings such as test_journey, test_proactive_message, and get_training_overview.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives real operating context for deciding whether to invoke: it consumes the workspace's AI budget and is restricted to admins, and it says the run lands in Training → Evals. It stops short of naming alternatives or a when-not-to-use condition (e.g. lightweight single-question testing), so it is clear context without exclusions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources