run_eval_contract
Run an IA-EVAL test suite (.ia-eval.yaml contract): call the model per scenario; return passed (true, false, null if undecided) and a verdict (PASS, FAIL, PARTIAL). Expectations are assertions that fail a scenario whatever its score; evaluators give the score. Hosted: pass inline_contract. validate_only calls no model.
Example tool call
This call uses example arguments supplied in the tool’s input schema.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"inline_contract": {
"metadata": {
"name": "smoke",
"version": "1.0"
},
"scenarios": [
{
"id": "capital",
"input": "Capital of France? One word.",
"ground_truth": "Paris"
}
],
"expectations": {
"contains": [
"Paris"
],
"min_score": 70
}
}
},
"name": "run_eval_contract"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| api_keys | No | Your keys per provider. Only Groq falls back to the server key; any other provider needs yours. | |
| overrides | No | Override contract defaults | |
| contract_path | No | A .ia-eval.yaml under the server's working directory (local use). | |
| validate_only | No | Check the contract against the IA-EVAL schema and stop: no model call, no key, deterministic. Returns valid + issues instead of throwing. Gate a PR that edits a contract on valid. | |
| inline_contract | No | The contract as an object: "metadata" ({name, version, model?, provider?}), "expectations", "evaluators" and "scenarios" ([{id, input, expectations?}]). generate_eval_yaml scaffolds one. |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| valid | No | ||
| issues | No | ||
| summary | No | ||
| metadata | No | ||
| warnings | No | ||
| contract_path | No | ||
| scenario_results | No |