prompt_test_suite
Define a test suite for a prompt: provide the system prompt, user prompt, and expected output criteria. Returns a test plan with scored rubric — use this as input for manual or automated LLM evaluation.
Example tool call
This call uses example arguments supplied in the tool’s input schema.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"user_prompt": "Say hi",
"system_prompt": "You are a helpful assistant."
},
"name": "prompt_test_suite"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| max_tokens | No | Max token budget for the test | |
| temperature | No | Temperature to use | |
| user_prompt | Yes | The user prompt to send | |
| check_safety | No | Include safety/PII checks in the rubric | |
| must_include | No | Required content (comma-separated) | |
| system_prompt | Yes | The system prompt under test | |
| expected_format | No | Expected output format | |
| must_not_include | No | Forbidden content (comma-separated) | |
| expected_behavior | No | Description of what the LLM should do (free text) | |
| adversarial_prompts | No | Auto-generate adversarial test variants (jailbreak, injection, edge cases) |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| rubric | No | ||
| categories | No | ||
| total_tests | No | ||
| instructions | No | ||
| test_suite_name | No |