batch_judge
Run batch evaluations on multiple items using a diverse pool of judges. Returns per-item scores, summary statistics, and error counts with incremental results and resume support.
Instructions
Run judge evaluations at scale on a list of items.
Processes items with bounded concurrency using a shared pool of diverse judges. Returns per-item scores, summary statistics (mean, stdev, min, max per dimension), and error counts.
Each item in the list should have a "prompt" key with the evaluation prompt, and an optional "metadata" key for tracking (e.g. language, entry ID).
When results_file is set, each scored item is appended as a JSON line immediately after scoring. On interruption, the file contains all completed items. On resume (same results_file), already-scored indices are skipped automatically.
Args: items: List of {"prompt": "...", "metadata": {...}} dicts rubric: List of scoring dimensions (e.g. ["accuracy", "naturalness"]) scale: Rating scale as "min-max" (default "1-5") judge_count: Judges per item (default 3) min_tier: Minimum quality tier for judge selection (default "A") free_only: If true, only use free models as judges output_format: How judges format scores — "csv" (default) or "json" concurrency: Max items evaluated in parallel (default 5) max_tokens: Max response tokens per judge (default 256) temperature: Sampling temperature (default 0.0) results_file: Path to JSONL file for incremental writes and resume support
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| items | Yes | ||
| scale | No | 1-5 | |
| rubric | Yes | ||
| min_tier | No | A | |
| free_only | No | ||
| max_tokens | No | ||
| concurrency | No | ||
| judge_count | No | ||
| temperature | No | ||
| results_file | No | ||
| output_format | No | csv |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| result | Yes |