Skip to main content
Glama

IA-QA — 130+ QA & Dev Tools for AI Agents

run_eval_contract

Read-only

Parse a .ia-eval.yaml LLM test suite, call the specified LLM model for each scenario, run all configured scorers, and return a structured JSON report with per-scenario Pass/Fail verdicts and a Markdown summary. Use list_local_tests to discover available test files.

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
api_keysNoAPI keys to use for LLM generation (all optional — falls back to server env vars)
overridesNoOverride contract defaults
contract_pathNoAbsolute or relative path to a .ia-eval.yaml file (required unless inline_contract is provided)
inline_contractNoRaw contract object (alternative to contract_path). Must contain top-level "metadata" ({name, version, model?, provider?}), "expectations" ({min_score?}), and "scenarios" ([{id, input, ground_truth?}]) — scenarios alone are rejected. Use generate_eval_yaml to scaffold one.

Output Schema

TableJSON Schema
NameRequiredDescriptionDefault
summaryNo
metadataNo
warningsNo
contract_pathNo
scenario_resultsNo

TDQS

A4/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true and destructiveHint=false, so the safety profile is known. The description adds that the tool makes external LLM calls and generates a report, which aligns with openWorldHint. It doesn't disclose potential costs, rate limits, or auth requirements, but with annotation coverage, the added behavioral context is adequate but not extensive.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two sentences with no filler. The first sentence front-loads the core purpose and workflow; the second provides a practical pointer to a sibling tool. Every part contributes to usability.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (4 parameters, nested objects, output schema), the description covers the main workflow and points to list_local_tests for discovery. It doesn't dwell on edge cases, but the schema and output schema cover parameter constraints and return format. This is sufficient for a capable agent.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, and the schema already explains all parameters, including nested objects and fallback behavior for API keys. The description adds no parameter-level semantics, so a baseline of 3 is appropriate since the schema does the heavy lifting.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a clear, specific action: parse a .ia-eval.yaml LLM test suite, call the LLM model per scenario, run scorers, and return a structured JSON report with per-scenario verdicts and a Markdown summary. It distinguishes itself from generic eval or test tools by naming the file format and the workflow, making its purpose unique among siblings.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies the tool is for running an existing .ia-eval.yaml test suite and explicitly directs the user to list_local_tests for discovering test files, providing useful workflow context. However, it doesn't explicitly exclude alternatives like run_semantic_tests or run_vlm_test_suite, so it stops short of full when-not-to-use guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

TDQS

A3.6/5.0
Disambiguation2/5

Multiple tools overlap significantly: compare_models/llm_fit_finder/model_info/list_llm_models all compare models; similarity_score/embedding_similarity/run_semantic_tests all measure text similarity; detect_secrets/secret_scan/analyze_diff_bugs/pr_gatekeeper all scan for secrets. Descriptions attempt to differentiate, but the boundaries between many tools are unclear, making selection error-prone.

Naming Consistency4/5

The vast majority of tools follow a snake_case verb_noun pattern (validate_email, generate_uuid, parse_csv), making the set mostly predictable. A few notable deviations exist (pr_gatekeeper, llm_fit_finder, cot_analyzer, jira_to_test_suite, needle_haystack_generate) but they are the exception rather than the rule.

Tool Count1/5

With 149 tools, this set is far beyond the 50+ threshold for an extreme mismatch. Even as a general-purpose QA & Dev toolkit, the sheer number overwhelms and exceeds any reasonable scope, making discovery and selection impractical.

Completeness4/5

The toolkit covers an impressively broad range: text processing, LLM evaluation, security auditing, web checks, MCP validation, Jira/Confluence integration, and more. Minor gaps exist, such as missing delete/update for webhooks and Confluence pages, and no create/update for Jira issues, but these are workable around.

Resources