lbb_model_choice_manage
Test candidate models on checked calls, compare scores and cost, then switch, revert, or set workflow model choices only after user confirmation.
Instructions
Model choice, writes. start tests a candidate model on the uses in jobs (without jobs, every use it can do that has checked calls): the server answers those checked calls again with the candidate and scores it and the model in use against the same ground truth. It changes no model and spends at most the trial budget ($3.00 by default) of model and judge cost; an open trial of the same use and candidate comes back as it is. Pick candidates from lbb_model_choice action=options (provider anthropic with an effort, or typesafe jev-latest). A trial runs by itself: poll lbb_model_choice action=trial until its status is no longer running. stop ends a trial; it keeps what it compared. switch makes the trial's use run its candidate on this graph from the next call on; only a trial that qualifies and compared with the model in use switches (ask to a Claude model, route to a Jev model). Show the user the trial's report (score, difference and interval, cost per call, time) and switch only what the user confirmed. revert puts the use back on LBB's model. put_choice chooses the model of a workflow use (use extract or decide, provider, model, effort) for the environment, or for one workflow with workflow: the next calls run it. Pick the model from the row's options in lbb_model_choice action=choices, show the user the price per million tokens of the model in use and of the new one, and set only what the user confirmed. delete_choice removes a choice: the use runs the environment's choice, else its default; it answers {id, object, deleted: true}. Both change only the model of future calls and remove no data, so neither asks for approval.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"action": "<action>"
},
"name": "lbb_model_choice_manage"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| job | No | revert: the use to put back on LBB's model. | |
| use | No | put_choice / delete_choice: the workflow use. | |
| days | No | start: the days the trial takes new checks (default 14). | |
| jobs | No | start: the uses to test it on. | |
| graph | No | Graph to target; defaults to the connection's graph | |
| model | No | start / put_choice: the model id, e.g. claude-haiku-5-5 or jev-latest. | |
| action | Yes | ||
| detail | No | Response detail level. Defaults to compact. | |
| effort | No | start / put_choice: the effort of a Claude model. | |
| target | No | start: the calls to compare (default 40). | |
| provider | No | start: the candidate's provider; put_choice: the model's provider (mock only on a local server). | |
| trial_id | No | stop / switch: the trial id. | |
| workflow | No | put_choice / delete_choice: a named workflow of the graph; absent for the environment. | |
| experiment | No | put_choice: an experiment (exp_…) whose B this choice applies, with workflow; use and the model are B's change. The workflow must still run the experiment's A, else experiment_outdated; when it runs B already, nothing changes. |