Skip to main content
Glama

IA-QA — 152 QA & Dev Tools for AI Agents

run_vlm_test_suite_batch

Read-only

Compare several vision models on the same image and test cases, in parallel (same assertions as run_vlm_test_suite). Returns per-model PASS/FAIL, pass rates, latency and a comparison table. BYOK: one key per model.

Input Schema

NameRequiredDescriptionDefault
modelsYes
api_keysYesModel id → API key.
image_urlNoPublic image URL (or image_base64).
thresholdNoPass rate for an overall PASS (default 80).
test_casesYes
image_base64No
system_promptNo
image_mime_typeNoFor image_base64 (default image/jpeg).

Output Schema

NameRequiredDescriptionDefault
suitesNo
verdictNo
total_failedNo
total_passedNo

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed14 schema fields changed
    • addedInput schema / examples
      Added value: +[
      +  {
      +    "api_keys": {
      +      "gemini-2.5-flash": "<your-google-key>",
      +      "gpt-4o-mini": "<your-openai-key>"
      +    },
      +    "image_url": "https://www.ia-qa.com/media/discover-report-poster.png",
      +    "models": [
      +      "gpt-4o-mini",
      +      "gemini-2.5-flash"
      +    ],
      +    "test_cases": [
      +      {
      +        "question": "Does the image contain text? Answer yes or no."
      +      }
      +    ]
      +  }
      +]
    • changedInput schema / properties / api_keys / description
      Previous value: -"Map of model ID → API key. Example: { \"gpt-4o\": \"sk-...\", \"claude-3-5-sonnet-20241022\": \"sk-ant-...\" }"New value: +"Model id → API key."
    • removedInput schema / properties / image_base64 / description
      Removed value: -"Base64-encoded image data (required unless image_url is provided)."
    • changedInput schema / properties / image_mime_type / description
      Previous value: -"MIME type of the image if using image_base64 (default: image/jpeg)."New value: +"For image_base64 (default image/jpeg)."
    • changedInput schema / properties / image_url / description
      Previous value: -"Public URL of the image to evaluate (required unless image_base64 is provided)."New value: +"Public image URL (or image_base64)."
    • removedInput schema / properties / models / description
      Removed value: -"Array of model IDs to compare (runs in parallel)."
    • changedInput schema / properties / models / items / enum
      Previous value: -[
      -  "gpt-4o",
      -  "gpt-4o-mini",
      -  "claude-3-5-sonnet-20241022",
      -  "claude-3-5-haiku-20241022",
      -  "gemini-3.5-flash",
      -  "gemini-2.5-flash"
      -]New value: +[
      +  "gpt-4o",
      +  "gpt-4o-mini",
      +  "claude-sonnet-4-6",
      +  "claude-haiku-4-5-20251001",
      +  "gemini-3.5-flash",
      +  "gemini-2.5-flash"
      +]
    • removedInput schema / properties / system_prompt / description
      Removed value: -"Optional system prompt sent to every VLM."
    • removedInput schema / properties / test_cases / description
      Removed value: -"Array of test cases to run against every model."
    • removedInput schema / properties / test_cases / items / properties / assertion_type / description
      Removed value: -"Assertion to run on the VLM response."
    • changedInput schema / properties / test_cases / items / properties / assertion_value / description
      Previous value: -"Expected value for the assertion (not needed for json_format)."New value: +"Not needed for json_format."
    • removedInput schema / properties / test_cases / items / properties / id / description
      Removed value: -"Optional identifier for this case."
    • removedInput schema / properties / test_cases / items / properties / question / description
      Removed value: -"Question to ask the VLM about the image."
    • changedInput schema / properties / threshold / description
      Previous value: -"Pass rate threshold for overall verdict (default: 80, 0–100)."New value: +"Pass rate for an overall PASS (default 80)."
  2. Changed1 schema field changed
    • changedInput schema / properties / models / items / enum
      Previous value: -[
      -  "gpt-4o",
      -  "gpt-4o-mini",
      -  "claude-3-5-sonnet-20241022",
      -  "claude-3-5-haiku-20241022",
      -  "gemini-1.5-flash",
      -  "gemini-2.0-flash"
      -]New value: +[
      +  "gpt-4o",
      +  "gpt-4o-mini",
      +  "claude-3-5-sonnet-20241022",
      +  "claude-3-5-haiku-20241022",
      +  "gemini-3.5-flash",
      +  "gemini-2.5-flash"
      +]
  3. Changed2 schema fields changed
    • changedInput schema / properties / api_keys / description
      Previous value: -"Map of model ID → API key. Example: { \"gpt-4o\": \"sk-...\", \"claude-3-5-sonnet-20241022\": \"sk-ant-...\" }"New value: +"Map of model ID → API key. Example: { \"gpt-4o\": \"sk-...\", \"claude-3-5-sonnet-20241022\": \"sk-ant-...\" }"
    • changedInput schema / properties / threshold / description
      Previous value: -"Pass rate threshold for overall verdict (default: 80, 0–100)."New value: +"Pass rate threshold for overall verdict (default: 80, 0–100)."
  4. Changed1 schema field changed
    • addedOutput schema / properties
      Added value: +{
      +  "suites": {
      +    "items": {
      +      "type": "object"
      +    },
      +    "type": "array"
      +  },
      +  "total_failed": {
      +    "type": "number"
      +  },
      +  "total_passed": {
      +    "type": "number"
      +  },
      +  "verdict": {
      +    "type": "string"
      +  }
      +}
  5. Changed1 schema field changed
    • changedOutput schema / (root)
      Previous value: -nullNew value: +{
      +  "additionalProperties": true,
      +  "type": "object"
      +}
  6. First observed

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already cover the safety profile (readOnly, non-destructive, openWorld, non-idempotent), so the bar is lower; the description still adds the parallel-execution trait and the BYOK auth requirement (one key per model), which is the main operational gotcha. It does not mention cost/latency implications of fanning out across up to six external providers, nor what happens when a model errors mid-batch.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three short sentences, front-loaded with the action and the parallelism trait, then the return shape, then the auth prerequisite. No filler or repetition of the tool name.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With an output schema present, the description needn't enumerate return fields, and it still names the headline results (PASS/FAIL, pass rates, latency, comparison table). Given a nested 8-parameter batch tool with BYOK auth, the main gap is failure/partial-success semantics and any batch-size or cost constraints.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is only 50% across 8 parameters, and the description clarifies only the key mapping ('one key per model') and the shared image/test-case inputs, both of which the schema already conveys. It adds no meaning for threshold, system_prompt, image_mime_type, or the assertion_type/assertion_value pairing.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (compare) and resource (vision models on the same image and test cases), and explicitly scopes it against the sibling run_vlm_test_suite by saying it uses the same assertions but runs models in parallel. An agent can distinguish this from compare_models/compare_responses without opening any schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The parenthetical '(same assertions as run_vlm_test_suite)' implicitly routes single-model runs to that sibling, and 'BYOK: one key per model' states the prerequisite for calling this. There is no explicit 'do not use when' clause or cost/rate-limit warning, so it stops short of a 5.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.