IA-QA — 152 QA & Dev Tools for AI Agents

run_eval_contract

Read-only

Run an IA-EVAL test suite (.ia-eval.yaml contract): call the model per scenario; return passed (true, false, null if undecided) and a verdict (PASS, FAIL, PARTIAL). Expectations are assertions that fail a scenario whatever its score; evaluators give the score. Hosted: pass inline_contract. validate_only calls no model.

Input Schema

Name	Required	Description
`api_keys`	No	Your keys per provider. Only Groq falls back to the server key; any other provider needs yours.
`overrides`	No	Override contract defaults
`contract_path`	No	A .ia-eval.yaml under the server's working directory (local use).
`validate_only`	No	Check the contract against the IA-EVAL schema and stop: no model call, no key, deterministic. Returns valid + issues instead of throwing. Gate a PR that edits a contract on valid.
`inline_contract`	No	The contract as an object: "metadata" ({name, version, model?, provider?}), "expectations", "evaluators" and "scenarios" ([{id, input, expectations?}]). generate_eval_yaml scaffolds one.

Output Schema

Name	Required	Description	Default
`valid`	No
`issues`	No
`summary`	No
`metadata`	No
`warnings`	No
`contract_path`	No
`scenario_results`	No

Schema Changelog

Changes observed during successful MCP inspections.

Changed3 schema fields changedNov 3, 2026

changedInput schema / examples

Previous value: -[
-  {
-    "inline_contract": {
-      "expectations": {
-        "min_score": 70
-      },
-      "metadata": {
-        "name": "smoke",
-        "version": "1.0"
-      },
-      "scenarios": [
-        {
-          "ground_truth": "Paris",
-          "id": "capital",
-          "input": "Capital of France? One word."
-        }
-      ]
-    }
-  }
-]New value: +[
+  {
+    "inline_contract": {
+      "expectations": {
+        "contains": [
+          "Paris"
+        ],
+        "min_score": 70
+      },
+      "metadata": {
+        "name": "smoke",
+        "version": "1.0"
+      },
+      "scenarios": [
+        {
+          "ground_truth": "Paris",
+          "id": "capital",
+          "input": "Capital of France? One word."
+        }
+      ]
+    }
+  }
+]

addedInput schema / properties / api_keys / properties / mistral
```
Added value: +{
+  "type": "string"
+}
```

changedInput schema / properties / inline_contract / description

Previous value: -"The contract as an object: \"metadata\" ({name, version, model?, provider?}), \"expectations\" ({min_score?}) and \"scenarios\" ([{id, input, ground_truth?}]). generate_eval_yaml scaffolds one."New value: +"The contract as an object: \"metadata\" ({name, version, model?, provider?}), \"expectations\", \"evaluators\" and \"scenarios\" ([{id, input, expectations?}]). generate_eval_yaml scaffolds one."

Changed4 schema fields changedOct 31, 2026

addedInput schema / examples

Added value: +[
+  {
+    "inline_contract": {
+      "expectations": {
+        "min_score": 70
+      },
+      "metadata": {
+        "name": "smoke",
+        "version": "1.0"
+      },
+      "scenarios": [
+        {
+          "ground_truth": "Paris",
+          "id": "capital",
+          "input": "Capital of France? One word."
+        }
+      ]
+    }
+  }
+]

changedInput schema / properties / api_keys / description

Previous value: -"API keys to use for LLM generation (all optional — falls back to server env vars)"New value: +"Your keys per provider. Only Groq falls back to the server key; any other provider needs yours."

changedInput schema / properties / contract_path / description

Previous value: -"Absolute or relative path to a .ia-eval.yaml file (required unless inline_contract is provided)"New value: +"A .ia-eval.yaml under the server's working directory (local use)."

changedInput schema / properties / inline_contract / description

Previous value: -"Raw contract object (alternative to contract_path). Must contain top-level \"metadata\" ({name, version, model?, provider?}), \"expectations\" ({min_score?}), and \"scenarios\" ([{id, input, ground_truth?}]) — scenarios alone are rejected. Use generate_eval_yaml to scaffold one."New value: +"The contract as an object: \"metadata\" ({name, version, model?, provider?}), \"expectations\" ({min_score?}) and \"scenarios\" ([{id, input, ground_truth?}]). generate_eval_yaml scaffolds one."

Changed2 schema fields changedOct 28, 2026
- addedInput schema / properties / api_keys / properties / deepseek
```
Added value: +{
+  "type": "string"
+}
```
- addedInput schema / properties / api_keys / properties / xai
```
Added value: +{
+  "type": "string"
+}
```

Changed3 schema fields changedOct 21, 2026

addedInput schema / properties / validate_only

Added value: +{
+  "description": "Check the contract against the IA-EVAL schema and stop: no model call, no key, deterministic. Returns valid + issues instead of throwing. Gate a PR that edits a contract on valid.",
+  "type": "boolean"
+}

addedOutput schema / properties / issues
```
Added value: +{}
```
addedOutput schema / properties / valid
```
Added value: +{
+  "type": "boolean"
+}
```

Changed1 schema field changedJul 24, 2026

changedInput schema / properties / inline_contract / description

Previous value: -"Raw contract object (alternative to contract_path)"New value: +"Raw contract object (alternative to contract_path). Must contain top-level \"metadata\" ({name, version, model?, provider?}), \"expectations\" ({min_score?}), and \"scenarios\" ([{id, input, ground_truth?}]) — scenarios alone are rejected. Use generate_eval_yaml to scaffold one."

Changed1 schema field changedJul 7, 2026

changedInput schema / properties / api_keys / description

Previous value: -"API keys to use for LLM generation (all optional â€” falls back to server env vars)"New value: +"API keys to use for LLM generation (all optional — falls back to server env vars)"

Changed1 schema field changedJun 21, 2026

addedOutput schema / properties

Added value: +{
+  "contract_path": {},
+  "metadata": {
+    "type": "object"
+  },
+  "scenario_results": {},
+  "summary": {
+    "type": "object"
+  },
+  "warnings": {}
+}

Changed1 schema field changedJun 18, 2026
- addedInput schema / properties / inline_contract / additionalProperties
```
Added value: +true
```

Changed1 schema field changedJun 10, 2026

changedOutput schema / (root)

Previous value: -nullNew value: +{
+  "additionalProperties": true,
+  "type": "object"
+}

First observedMay 26, 2026

TDQS

A4/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Beyond the annotations it discloses that a model is invoked per scenario, that null means undecided, the PASS/FAIL/PARTIAL verdict space, and that validate_only is deterministic with no model call or key. The distinction that expectations are hard assertions while evaluators produce the score is genuinely useful behavioral context; it does not, however, discuss cost, latency, or partial-run failure behavior.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the core action and dense throughout; every clause carries information. Some compression hurts readability ('Hosted:' is undefined shorthand, semicolon chaining obscures the passed/verdict relationship).

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 5-parameter, nested-object tool with a full output schema and annotations, the description covers the execution model, the two input paths, and the validation mode adequately. Minor gaps remain around what PARTIAL means and what happens when a scenario errors mid-run.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3, but the description adds real meaning by positioning inline_contract as the hosted path versus contract_path and by restating validate_only's no-key/no-model semantics. The api_keys fallback nuance and overrides fields are left entirely to the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Run an IA-EVAL test suite (.ia-eval.yaml contract)') and describes what happens per scenario (call the model, produce passed + verdict). It clarifies the two entry modes but does not name or distinguish itself from close siblings such as run_semantic_tests, prompt_test_suite, or run_vlm_test_suite.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives a concrete routing rule between the two contract sources ('Hosted: pass inline_contract') and a clear when-to-use for validate_only ('gate a PR that edits a contract on valid'). It lacks explicit exclusions telling an agent when to prefer the sibling eval tools instead.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

All IA-QA — 152 QA & Dev Tools for AI Agents tools Inspect

Try in Browser

One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

Glama MCP Gateway

Add one secure layer between your agents and this server.

run_eval_contract

Input Schema

Output Schema

Schema Changelog

TDQS

Glama MCP Gateway

Resources