compare_responses
Compare two ALREADY-PRODUCED outputs (e.g. model A vs model B on the same task) side by side. Returns deterministic metrics (token cosine, ROUGE-L, Jaccard, length/structure deltas, JSON diff) and a verdict. If a reference (ground truth) is given, scores each output against it and picks the closer one. If model + api_key are given, an LLM judge also picks a qualitative winner for the task. No re-execution — you bring the outputs.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| task | No | The task/prompt both outputs were answering — used by the LLM judge for context | |
| model | No | Optional judge model id (BYOK). When set with api_key, an LLM judge picks a qualitative winner. | |
| api_key | No | Optional API key for the judge model (BYOK). Used only for the judge call; never stored. | |
| label_a | No | Label for output A (e.g. "GPT-4o", "v1.0") | |
| label_b | No | Label for output B (e.g. "GPT-5-nano", "v1.1") | |
| reference | No | Optional ground-truth / expected answer. If set, each output is scored against it and the closer one wins (deterministic). | |
| check_json | No | Try to parse as JSON and compare structurally (keys, types, values) | |
| response_a | Yes | First output (e.g. model A's answer) | |
| response_b | Yes | Second output (e.g. model B's answer) |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| judge | No | ||
| labelA | No | ||
| labelB | No | ||
| metrics | No | ||
| summary | No | ||
| verdict | No |