Skip to main content
Glama

IA-QA — 152 QA & Dev Tools for AI Agents

run_vlm_test_suite

Read-only

Test a vision model (OpenAI, Anthropic or Gemini) on one image: each test case asks a question and asserts on the answer (contains, not_contains, json_format, min_length, max_length, semantic_contains = TF-IDF cosine ≥ 0.4). Returns per-case PASS/FAIL, the pass rate, an overall PASS/WARNING/FAIL against threshold, and latency. BYOK: your key for the model's provider; the image is fetched only from a public address.

Input Schema

NameRequiredDescriptionDefault
modelYes
api_keyYesKey for the model's provider.
image_urlNoPublic image URL (or image_base64).
thresholdNoPass rate for an overall PASS (default 80).
test_casesYes
image_base64No
system_promptNo
image_mime_typeNoFor image_base64 (default image/jpeg).

Output Schema

NameRequiredDescriptionDefault
modelNo
totalNo
failedNo
passedNo
resultsNo
verdictNo

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed14 schema fields changed
    • addedInput schema / examples
      Added value: +[
      +  {
      +    "api_key": "<your-api-key>",
      +    "image_url": "https://www.ia-qa.com/media/discover-report-poster.png",
      +    "model": "gemini-2.5-flash",
      +    "test_cases": [
      +      {
      +        "assertion_type": "contains",
      +        "assertion_value": "yes",
      +        "question": "Does the image contain text? Answer yes or no."
      +      }
      +    ]
      +  }
      +]
    • changedInput schema / properties / api_key / description
      Previous value: -"API key for the model provider (OpenAI sk-, Anthropic sk-ant-, or Google AIzaSy...)."New value: +"Key for the model's provider."
    • removedInput schema / properties / image_base64 / description
      Removed value: -"Base64-encoded image data (required unless image_url is provided)."
    • changedInput schema / properties / image_mime_type / description
      Previous value: -"MIME type of the image if using image_base64 (default: image/jpeg)."New value: +"For image_base64 (default image/jpeg)."
    • changedInput schema / properties / image_url / description
      Previous value: -"Public URL of the image to evaluate (required unless image_base64 is provided)."New value: +"Public image URL (or image_base64)."
    • removedInput schema / properties / model / description
      Removed value: -"VLM model to use."
    • changedInput schema / properties / model / enum
      Previous value: -[
      -  "gpt-4o",
      -  "gpt-4o-mini",
      -  "claude-3-5-sonnet-20241022",
      -  "claude-3-5-haiku-20241022",
      -  "gemini-3.5-flash",
      -  "gemini-2.5-flash"
      -]New value: +[
      +  "gpt-4o",
      +  "gpt-4o-mini",
      +  "claude-sonnet-4-6",
      +  "claude-haiku-4-5-20251001",
      +  "gemini-3.5-flash",
      +  "gemini-2.5-flash"
      +]
    • removedInput schema / properties / system_prompt / description
      Removed value: -"Optional system prompt sent to the VLM."
    • removedInput schema / properties / test_cases / description
      Removed value: -"Array of test cases to run."
    • removedInput schema / properties / test_cases / items / properties / assertion_type / description
      Removed value: -"Assertion to run on the VLM response. semantic_contains uses TF-IDF cosine similarity ≥ 0.4."
    • changedInput schema / properties / test_cases / items / properties / assertion_value / description
      Previous value: -"Expected value for the assertion (not needed for json_format)."New value: +"Not needed for json_format."
    • removedInput schema / properties / test_cases / items / properties / id / description
      Removed value: -"Optional identifier for this case."
    • removedInput schema / properties / test_cases / items / properties / question / description
      Removed value: -"Question to ask the VLM about the image."
    • changedInput schema / properties / threshold / description
      Previous value: -"Pass rate threshold for overall verdict (default: 80, 0–100)."New value: +"Pass rate for an overall PASS (default 80)."
  2. Changed1 schema field changed
    • changedInput schema / properties / model / enum
      Previous value: -[
      -  "gpt-4o",
      -  "gpt-4o-mini",
      -  "claude-3-5-sonnet-20241022",
      -  "claude-3-5-haiku-20241022",
      -  "gemini-1.5-flash",
      -  "gemini-2.0-flash"
      -]New value: +[
      +  "gpt-4o",
      +  "gpt-4o-mini",
      +  "claude-3-5-sonnet-20241022",
      +  "claude-3-5-haiku-20241022",
      +  "gemini-3.5-flash",
      +  "gemini-2.5-flash"
      +]
  3. Changed2 schema fields changed
    • changedInput schema / properties / test_cases / items / properties / assertion_type / description
      Previous value: -"Assertion to run on the VLM response. semantic_contains uses TF-IDF cosine similarity ≥ 0.4."New value: +"Assertion to run on the VLM response. semantic_contains uses TF-IDF cosine similarity ≥ 0.4."
    • changedInput schema / properties / threshold / description
      Previous value: -"Pass rate threshold for overall verdict (default: 80, 0–100)."New value: +"Pass rate threshold for overall verdict (default: 80, 0–100)."
  4. Changed1 schema field changed
    • addedOutput schema / properties
      Added value: +{
      +  "failed": {
      +    "type": "number"
      +  },
      +  "model": {
      +    "type": "string"
      +  },
      +  "passed": {
      +    "type": "number"
      +  },
      +  "results": {
      +    "items": {
      +      "type": "object"
      +    },
      +    "type": "array"
      +  },
      +  "total": {
      +    "type": "number"
      +  },
      +  "verdict": {
      +    "type": "string"
      +  }
      +}
  5. Changed1 schema field changed
    • changedOutput schema / (root)
      Previous value: -nullNew value: +{
      +  "additionalProperties": true,
      +  "type": "object"
      +}
  6. First observed

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations cover the safety profile (readOnly, non-destructive, openWorld, non-idempotent), and the description adds real value on top: BYOK auth requirement, the public-address-only fetch restriction, and the shape of the result (per-case PASS/FAIL, pass rate, overall verdict, latency). It does not explain cost/rate implications of calling an external provider, which is the main residual gap.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two dense sentences, front-loaded with the core action and the key behavioral facts; nothing is filler. The second sentence is a long run-on chain of clauses, which slightly hurts scannability but costs no information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so spelling out return values was optional, and the description covers auth, input constraints, and assertion semantics for a 3-required-of-8 parameter tool. Minor gaps remain around system_prompt's effect and base64 input handling, but an agent has enough to call it correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

With only 50% schema coverage, the description compensates well by defining assertion semantics that the schema only enumerates — notably that semantic_contains means TF-IDF cosine ≥ 0.4 — and clarifying that the threshold governs the overall PASS. It adds little for system_prompt, image_base64, or image_mime_type, keeping it short of a 5.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource: test a vision model (with named providers) on one image with question/assertion test cases. The 'one image' scope implicitly separates it from run_vlm_test_suite_batch, and the enumerated assertion types make the tool's job concrete.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives clear operating context: BYOK, key must match the model's provider, and the image must be reachable at a public address. It never names an explicit alternative (e.g. run_vlm_test_suite_batch for multiple images, or prompt_test_suite for text-only), so routing is implied rather than stated.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.