compare
Run blind A/B comparisons between two outputs using multiple AI judges to determine preference without position bias. Get objective scores for translations, code, or summaries.
Instructions
Blind A/B comparison of two items judged by N models.
When blind=True (default), randomizes which item is shown as A vs B to each judge independently, then de-randomizes scores. This prevents position bias where judges consistently favor the first item shown.
Use for comparing translations, code solutions, summaries, or any pair of outputs where you want an objective preference.
Args: item_a: First item to compare item_b: Second item to compare context: Optional context for the comparison (e.g. the original task) dimensions: Scoring dimensions (default ["quality"]) scale: Rating scale as "min-max" (default "1-5") judge_count: Number of judge models (default 3) blind: Randomize A/B order per judge to prevent position bias (default true) min_tier: Minimum quality tier for judge selection (default "A") free_only: If true, only use free models as judges max_tokens: Max response tokens per judge (default 512) temperature: Sampling temperature (default 0.0)
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| blind | No | ||
| scale | No | 1-5 | |
| item_a | Yes | ||
| item_b | Yes | ||
| context | No | ||
| min_tier | No | A | |
| free_only | No | ||
| dimensions | No | ||
| max_tokens | No | ||
| judge_count | No | ||
| temperature | No |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| result | Yes |