Skip to main content
Glama
Coval MCP Server

Coval MCP Server

Official
by coval-ai

Create test case

create_test_case

Adds a synthetic evaluation test case to a specified test set, defining the agent input, expected behaviors, and optional scripted turns for reproducible testing.

Instructions

Create a synthetic evaluation test case in a test set. Scenario content is evaluation fixture data, never the current ChatGPT conversation. For IVR/phone-tree agents, keypad presses are DTMF: script them as typed {"type": "dtmf", "digits": ...} turns in simulation_metadata_input.script_turns (input_type SCRIPT), or have the scenario instruct the persona to use its dtmf_tool — never as spoken digit text.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "test_set_id": "<test_set_id>",
      "input_str": "<input_str>"
    },
    "name": "create_test_case"
  }
}

Input Schema

NameRequiredDescriptionDefault
input_strYesThe test input or scenario that will be presented to the agent.
input_typeNoInput type; defaults to SCENARIO. Use SCRIPT with simulation_metadata_input.script_turns for exact ordered turns.
user_notesNoInternal notes about this test case.
descriptionNoHuman-readable description of what this test case validates.
test_set_idYesThe 8-character ID of the test set to add this test case to.
metric_inputNoCustom inputs for metric evaluation.
expected_behaviorsNoList of expected agent behaviors or responses for evaluation.
simulation_metadata_inputNoAdditional context passed to the simulation environment. For scripted (input_type SCRIPT) cases, set {"script_turns": [...]} here: entries are plain strings the persona speaks verbatim, or typed keypad presses {"type": "dtmf", "digits": "1"} (digits allow 0-9, *, # plus phone punctuation for full numbers). Never write DTMF as spoken text like "dtmf:1" — it would be read aloud. For SCENARIO cases against IVRs, instead instruct the persona in input_str to use its dtmf_tool for each keypress.

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed4 schema fields changedv0.5.0
    • removedInput schema / additionalProperties
      Removed value: -false
    • addedInput schema / properties / metric_input / propertyNames
      Added value: +{
      +  "type": "string"
      +}
    • changedInput schema / properties / simulation_metadata_input / additionalProperties
      Previous value: -trueNew value: +{}
    • changedInput schema / properties / simulation_metadata_input / properties / script_turns / items / anyOf
      Previous value: -[
      -  {
      -    "minLength": 1,
      -    "type": "string"
      -  },
      -  {
      -    "anyOf": [
      -      {
      -        "additionalProperties": false,
      -        "properties": {
      -          "text": {
      -            "minLength": 1,
      -            "type": "string"
      -          },
      -          "type": {
      -            "const": "text",
      -            "type": "string"
      -          }
      -        },
      -        "required": [
      -          "type",
      -          "text"
      -        ],
      -        "type": "object"
      -      },
      -      {
      -        "additionalProperties": false,
      -        "properties": {
      -          "digits": {
      -            "allOf": [
      -              {
      -                "pattern": "^[0-9#*+(). -]+$"
      -              },
      -              {
      -                "pattern": "[0-9#*]"
      -              }
      -            ],
      -            "type": "string"
      -          },
      -          "type": {
      -            "const": "dtmf",
      -            "type": "string"
      -          }
      -        },
      -        "required": [
      -          "type",
      -          "digits"
      -        ],
      -        "type": "object"
      -      },
      -      {
      -        "additionalProperties": false,
      -        "properties": {
      -          "type": {
      -            "const": "skip",
      -            "type": "string"
      -          }
      -        },
      -        "required": [
      -          "type"
      -        ],
      -        "type": "object"
      -      }
      -    ]
      -  }
      -]New value: +[
      +  {
      +    "minLength": 1,
      +    "type": "string"
      +  },
      +  {
      +    "oneOf": [
      +      {
      +        "additionalProperties": false,
      +        "properties": {
      +          "text": {
      +            "minLength": 1,
      +            "type": "string"
      +          },
      +          "type": {
      +            "const": "text",
      +            "type": "string"
      +          }
      +        },
      +        "required": [
      +          "type",
      +          "text"
      +        ],
      +        "type": "object"
      +      },
      +      {
      +        "additionalProperties": false,
      +        "properties": {
      +          "digits": {
      +            "allOf": [
      +              {
      +                "pattern": "^[0-9#*+(). -]+$",
      +                "type": "string"
      +              },
      +              {
      +                "pattern": "[0-9#*]",
      +                "type": "string"
      +              }
      +            ],
      +            "type": "string"
      +          },
      +          "type": {
      +            "const": "dtmf",
      +            "type": "string"
      +          }
      +        },
      +        "required": [
      +          "type",
      +          "digits"
      +        ],
      +        "type": "object"
      +      },
      +      {
      +        "additionalProperties": false,
      +        "properties": {
      +          "type": {
      +            "const": "skip",
      +            "type": "string"
      +          }
      +        },
      +        "required": [
      +          "type"
      +        ],
      +        "type": "object"
      +      }
      +    ]
      +  }
      +]
  2. Changed4 schema fields changedv0.4.1
    • addedInput schema / properties / input_type
      Added value: +{
      +  "description": "Input type; defaults to SCENARIO. Use SCRIPT with simulation_metadata_input.script_turns for exact ordered turns.",
      +  "enum": [
      +    "SCENARIO",
      +    "TRANSCRIPT",
      +    "IVR",
      +    "AUDIO",
      +    "MANUAL",
      +    "SCRIPT"
      +  ],
      +  "type": "string"
      +}
    • changedInput schema / properties / simulation_metadata_input / additionalProperties
      Previous value: -{}New value: +true
    • changedInput schema / properties / simulation_metadata_input / description
      Previous value: -"Additional context passed to the simulation environment."New value: +"Additional context passed to the simulation environment. For scripted (input_type SCRIPT) cases, set {\"script_turns\": [...]} here: entries are plain strings the persona speaks verbatim, or typed keypad presses {\"type\": \"dtmf\", \"digits\": \"1\"} (digits allow 0-9, *, # plus phone punctuation for full numbers). Never write DTMF as spoken text like \"dtmf:1\" — it would be read aloud. For SCENARIO cases against IVRs, instead instruct the persona in input_str to use its dtmf_tool for each keypress."
    • addedInput schema / properties / simulation_metadata_input / properties
      Added value: +{
      +  "script_turns": {
      +    "items": {
      +      "anyOf": [
      +        {
      +          "minLength": 1,
      +          "type": "string"
      +        },
      +        {
      +          "anyOf": [
      +            {
      +              "additionalProperties": false,
      +              "properties": {
      +                "text": {
      +                  "minLength": 1,
      +                  "type": "string"
      +                },
      +                "type": {
      +                  "const": "text",
      +                  "type": "string"
      +                }
      +              },
      +              "required": [
      +                "type",
      +                "text"
      +              ],
      +              "type": "object"
      +            },
      +            {
      +              "additionalProperties": false,
      +              "properties": {
      +                "digits": {
      +                  "allOf": [
      +                    {
      +                      "pattern": "^[0-9#*+(). -]+$"
      +                    },
      +                    {
      +                      "pattern": "[0-9#*]"
      +                    }
      +                  ],
      +                  "type": "string"
      +                },
      +                "type": {
      +                  "const": "dtmf",
      +                  "type": "string"
      +                }
      +              },
      +              "required": [
      +                "type",
      +                "digits"
      +              ],
      +              "type": "object"
      +            },
      +            {
      +              "additionalProperties": false,
      +              "properties": {
      +                "type": {
      +                  "const": "skip",
      +                  "type": "string"
      +                }
      +              },
      +              "required": [
      +                "type"
      +              ],
      +              "type": "object"
      +            }
      +          ]
      +        }
      +      ]
      +    },
      +    "type": "array"
      +  }
      +}
  3. First observedv0.1.0

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare this is a non-read-only, non-destructive, non-idempotent creation tool, so the description adds valuable behavioral context: scenario content is fixture data, not conversation context, and DTMF must be represented as structured turns rather than spoken text. It does not describe permissions or return behavior, but the annotations cover the safety profile.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The purpose is front-loaded, and the remaining sentences address the two highest-risk ambiguities: scenario fixture data and DTMF formatting. There is no wasted text.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the rich schema and mutation annotations, the description covers the important behavioral traps an agent would otherwise miss. It does not explain the response shape or test-set prerequisites, but those are either absent or documented in the schema.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3, but the description adds meaningful semantics for input_type SCRIPT and simulation_metadata_input.script_turns, including the distinction between typed DTMF objects and spoken digit text.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb and resource: create a synthetic evaluation test case in a test set. It clearly separates creation from sibling operations such as update_test_case and create_test_set.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is implied by the create verb and the conditional IVR/SCRIPT guidance, but the description does not explicitly say when to use this tool versus update_test_case or how to decide between input modes beyond the DTMF-specific case.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.