compare_models_for_task
Rank LLMs by expected cost per successful task from token counts and success rates, so cheap but unreliable models do not rank above reliable ones.
Instructions
Rank models for a task by expected cost per successful task.
Cost per token is the wrong ranking for agent work: a cheap model that fails half its tasks costs more per success than a pricier, reliable one. Pass measured success rates (0-1 per model_id) if you have them; models without a supplied rate are ranked on raw cost.
Args: input_tokens: Input tokens for one attempt at the task. output_tokens: Output tokens for one attempt at the task. success_rates: Optional map of model_id to success rate in (0, 1].
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"input_tokens": "<input_tokens>",
"output_tokens": "<output_tokens>"
},
"name": "compare_models_for_task"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| input_tokens | Yes | ||
| output_tokens | Yes | ||
| success_rates | No |