Skip to main content
Glama

IA-QA — 152 QA & Dev Tools for AI Agents

generate_eval_yaml

Read-only

Generate a complete .ia-eval.yaml evaluation contract from a plain-language description of what your LLM should do. Uses Groq openai/gpt-oss-20b (server-side, no API key needed). Returns ready-to-run YAML for the LLM Test Runner (run_eval_contract). Picks appropriate evaluators (cosine_similarity, contains_check, hallucination_check, etc.) based on the task type.

Input Schema

NameRequiredDescriptionDefault
task_typeNoOptional task type hint to guide evaluator selection.
descriptionYesPlain-language description of what the LLM under test should do. Be specific: describe inputs, expected behaviour, and constraints.
system_promptNoOptional system prompt of the LLM under test. Helps generate more accurate test cases.
scenario_countNoNumber of scenarios to generate (default: 5). Covers happy path + edge cases + adversarial.

Output Schema

NameRequiredDescriptionDefault
yamlNo
task_typeNo
model_usedNo
scenario_countNo

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed1 schema field changed
    • addedInput schema / examples
      Added value: +[
      +  {
      +    "description": "Summarize a customer support ticket in one sentence."
      +  }
      +]
  2. Changed1 schema field changed
    • addedOutput schema / properties
      Added value: +{
      +  "model_used": {
      +    "type": "string"
      +  },
      +  "scenario_count": {
      +    "type": "number"
      +  },
      +  "task_type": {
      +    "type": "string"
      +  },
      +  "yaml": {
      +    "type": "string"
      +  }
      +}
  3. Changed1 schema field changed
    • changedOutput schema / (root)
      Previous value: -nullNew value: +{
      +  "additionalProperties": true,
      +  "type": "object"
      +}
  4. First observed

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already cover safety (readOnlyHint=true, destructiveHint=false), so the description earns credit for adding deployment context the annotations don't: it runs server-side via Groq openai/gpt-oss-20b with no API key, and it auto-selects evaluators (cosine_similarity, contains_check, hallucination_check). That explains the openWorldHint dependency. It does not mention latency, rate limits, or that output varies run-to-run despite idempotentHint=false.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Four short sentences, front-loaded with the verb and artifact, then progressively adding model, return value, and evaluator behavior. No filler and no repetition beyond the necessary naming of the consumer tool.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Output schema exists, so return-value details are unnecessary, and annotations handle the safety profile; the description covers what's left — external model, no auth, downstream use. Minor gap: nothing about failure modes (e.g. what happens with a vague description) or how deterministic the generated YAML is.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so parameters are fully documented in the schema itself, including defaults and enum values. The description adds one genuinely useful link — that task_type drives evaluator selection — but gives no guidance on system_prompt or scenario_count trade-offs. Baseline 3 is appropriate when the schema does the heavy lifting.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and artifact: generate a complete .ia-eval.yaml evaluation contract from a plain-language description. It also names the downstream consumer (run_eval_contract / LLM Test Runner), which separates it cleanly from siblings like generate_test_cases or prompt_test_suite that produce different artifacts.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description makes the usage context clear: you have a plain-language spec of an LLM's behavior and want a runnable eval contract. Naming run_eval_contract as the follow-up step is a useful routing signal. It stops short of explicit exclusions — it never says when to prefer generate_test_cases, prompt_test_suite, or json_schema_generate instead.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.