generate_eval_yaml
Generate a complete .ia-eval.yaml evaluation contract from a plain-language description of what your LLM should do. Uses Groq openai/gpt-oss-20b (server-side, no API key needed). Returns ready-to-run YAML for the LLM Test Runner (run_eval_contract). Picks appropriate evaluators (cosine_similarity, contains_check, hallucination_check, etc.) based on the task type.
Example tool call
This call uses example arguments supplied in the tool’s input schema.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"description": "Summarize a customer support ticket in one sentence."
},
"name": "generate_eval_yaml"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| task_type | No | Optional task type hint to guide evaluator selection. | |
| description | Yes | Plain-language description of what the LLM under test should do. Be specific: describe inputs, expected behaviour, and constraints. | |
| system_prompt | No | Optional system prompt of the LLM under test. Helps generate more accurate test cases. | |
| scenario_count | No | Number of scenarios to generate (default: 5). Covers happy path + edge cases + adversarial. |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| yaml | No | ||
| task_type | No | ||
| model_used | No | ||
| scenario_count | No |