score_list
Rank a list of existing refs by scoring with an evaluator, returning the top-k winners. Avoids parameter size limits by resolving artifacts server-side.
Instructions
Score and rank a list of existing refs against an evaluator.
This is the missing piece for "generate N candidates, then pick the best"
pipelines where the N candidates already exist as refs from an upstream
combinator (map, par, etc.) — beam is the wrong shape because
it fans out width-N proposers of the same prompt, whereas score_list
takes N different outputs and ranks them.
The key infrastructure point: refs are resolved to their artifact text
server-side, so callers do not need to pipe full artifact text
through tool parameters. This clears the ~4KB param-size wall you would
otherwise hit scoring 6+ medium-length artifacts through map.
Evaluator forms are the same as iterate / beam:
validate:<type>— LLM validator against a registered typescore:<criterion>— ad-hoc haiku rubricexec:<shell-cmd>— ground-truth shell command (exit code scoring)
Each ref is tagged with a search stamp carrying its score and
beam_rank. Refs outside the top-k are marked pruned=True
with a reason pointing at the beam cut. top_k=0 means return all
without pruning.
Args:
refs: JSON array of refs — either ["run_id/agent_id", ...] or
[{"ref": "run_id/agent_id"}, ...]. Both forms are accepted.
evaluator: Scoring directive (validate:<type>, score:<criterion>,
or exec:<cmd>).
top_k: How many top-scoring refs to surface as winners. 0 means
rank-only, no pruning stamp applied.
max_concurrency: Upper bound on parallel evaluator calls (default: 5).
Returns:
JSON with run_id, evaluator, total, top_k,
winners (list of winning ref strings), and ranked (the full
per-ref trace: rank, ref, score, verdict, issues, reason).
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| refs | Yes | ||
| evaluator | Yes | ||
| top_k | No | ||
| max_concurrency | No |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| result | Yes |