Create test case
create_test_caseAdds a synthetic evaluation test case to a specified test set, defining the agent input, expected behaviors, and optional scripted turns for reproducible testing.
Instructions
Create a synthetic evaluation test case in a test set. Scenario content is evaluation fixture data, never the current ChatGPT conversation. For IVR/phone-tree agents, keypad presses are DTMF: script them as typed {"type": "dtmf", "digits": ...} turns in simulation_metadata_input.script_turns (input_type SCRIPT), or have the scenario instruct the persona to use its dtmf_tool — never as spoken digit text.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"test_set_id": "<test_set_id>",
"input_str": "<input_str>"
},
"name": "create_test_case"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| input_str | Yes | The test input or scenario that will be presented to the agent. | |
| input_type | No | Input type; defaults to SCENARIO. Use SCRIPT with simulation_metadata_input.script_turns for exact ordered turns. | |
| user_notes | No | Internal notes about this test case. | |
| description | No | Human-readable description of what this test case validates. | |
| test_set_id | Yes | The 8-character ID of the test set to add this test case to. | |
| metric_input | No | Custom inputs for metric evaluation. | |
| expected_behaviors | No | List of expected agent behaviors or responses for evaluation. | |
| simulation_metadata_input | No | Additional context passed to the simulation environment. For scripted (input_type SCRIPT) cases, set {"script_turns": [...]} here: entries are plain strings the persona speaks verbatim, or typed keypad presses {"type": "dtmf", "digits": "1"} (digits allow 0-9, *, # plus phone punctuation for full numbers). Never write DTMF as spoken text like "dtmf:1" — it would be read aloud. For SCENARIO cases against IVRs, instead instruct the persona in input_str to use its dtmf_tool for each keypress. |