regression_test
Run regression test for a prompt variant against its golden test suite. Tests the variant against pre-validated scenarios with known success criteria to ensure it doesn't break existing functionality. Returns pass/fail status per scenario with fluke detection for suspected regressions. If no golden suite exists, one will be automatically generated from prompt analysis. Three ways to specify the variant: (1) variantPrompt — full prompt content; (2) variantPromptId — reference an existing stored prompt; (3) variantOverrides — array of find/replace ops applied on top of promptId's content.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| promptId | Yes | The agent ID that has a golden regression suite. Required. | |
| customerId | No | (Converra superadmin only) Target customer ID to act on behalf of | |
| variantPrompt | No | The variant system prompt content to test. Use this OR variantPromptId OR variantOverrides. | |
| variantPromptId | No | Alternative to variantPrompt: ID of stored agent to use as variant. | |
| variantOverrides | No | Construct the variant by applying find/replace overrides on top of promptId's content. Avoids sending the full prompt when the change is small. Each override's `find` string must occur exactly once in the prompt (after prior overrides have been applied) — multiple or zero matches throw a validation error. Order matters: each find runs against the result of prior overrides. Anchors are case-sensitive and match any substring (including inside larger words) — include surrounding whitespace/punctuation to disambiguate short anchors. Example: [{find: "Be concise.\n", replace: "Be concise and warm.\n"}]. Use this OR variantPrompt OR variantPromptId. |