create_metric_group
Creates a named, ordered group of Evaluators to score pipeline Sessions, with optional weights that define how metrics contribute to a Composite score.
Instructions
Creates a Metric Group: a named, ordered set of Evaluators to score Sessions of any pipeline with.
The group names Evaluators, not versions, so each run judges with their newest versions. Running
it costs money if an Evaluator is judged. Find Evaluator ids with list_evaluators.
weights say how much each Metric counts toward the group's Composite, as
{evaluator_id, metric_key, weight} entries. weight is a number above 0 and relative: weights
are divided by their sum, so 2, 1, 1 count as 50%, 25%, 25%. Each Evaluator must be in the group,
each metric_key must be one its newest version declares with a Normalized mean (a Score
with both range and better, or a Label with a scale), and no pair may repeat; otherwise
hub-api rejects the request. Without weights a group has no Composite. A stored weight with
usable false and an unusable_reason means a newer Evaluator version dropped the Metric or
its normalization: it is kept but no longer counts in new runs.
:param name: The Metric Group's name.
:param evaluator_ids: The Evaluators to run, in the order results show them (at least one).
:param weights: Optional Metric weights, as described above.
:returns: The created Metric Group, or an error message.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| name | Yes | ||
| weights | No | ||
| evaluator_ids | Yes |