Skip to main content
Glama

run_vlm_test_suite

Read-only

Test a vision model (OpenAI, Anthropic or Gemini) on one image: each test case asks a question and asserts on the answer (contains, not_contains, json_format, min_length, max_length, semantic_contains = TF-IDF cosine ≥ 0.4). Returns per-case PASS/FAIL, the pass rate, overall PASS/WARNING/FAIL against threshold (N/A when no assertion could run), and latency. BYOK: your key for the model's provider; the image is fetched only from a public address.

Example tool call

This call uses example arguments supplied in the tool’s input schema.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "model": "gemini-2.5-flash",
      "api_key": "<your-api-key>",
      "image_url": "https://www.ia-qa.com/media/discover-report-poster.png",
      "test_cases": [
        {
          "question": "Does the image contain text? Answer yes or no.",
          "assertion_type": "contains",
          "assertion_value": "yes"
        }
      ]
    },
    "name": "run_vlm_test_suite"
  }
}

Input Schema

NameRequiredDescriptionDefault
modelYes
api_keyYesKey for the model's provider.
image_urlNoPublic image URL (or image_base64).
thresholdNoPass rate for an overall PASS (default 80).
test_casesYes
image_base64No
system_promptNo
image_mime_typeNoFor image_base64 (default image/jpeg).

Output Schema

NameRequiredDescriptionDefault
modelNo
totalNo
passedNo
overallNo
resultsNo
image_urlNo
pass_rateNo
thresholdNo
latencyStatsNo

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed12 schema fields changed
    • removedOutput schema / properties / failed
      Removed value: -{
      -  "type": "number"
      -}
    • addedOutput schema / properties / image_url
      Added value: +{}
    • addedOutput schema / properties / latencyStats
      Added value: +{}
    • removedOutput schema / properties / model / type
      Removed value: -"string"
    • addedOutput schema / properties / overall
      Added value: +{
      +  "enum": [
      +    "PASS",
      +    "WARNING",
      +    "FAIL",
      +    "N/A"
      +  ]
      +}
    • addedOutput schema / properties / pass_rate
      Added value: +{}
    • removedOutput schema / properties / passed / type
      Removed value: -"number"
    • removedOutput schema / properties / results / items
      Removed value: -{
      -  "type": "object"
      -}
    • removedOutput schema / properties / results / type
      Removed value: -"array"
    • addedOutput schema / properties / threshold
      Added value: +{}
    • removedOutput schema / properties / total / type
      Removed value: -"number"
    • removedOutput schema / properties / verdict
      Removed value: -{
      -  "type": "string"
      -}
  2. Changed14 schema fields changed
    • addedInput schema / examples
      Added value: +[
      +  {
      +    "api_key": "<your-api-key>",
      +    "image_url": "https://www.ia-qa.com/media/discover-report-poster.png",
      +    "model": "gemini-2.5-flash",
      +    "test_cases": [
      +      {
      +        "assertion_type": "contains",
      +        "assertion_value": "yes",
      +        "question": "Does the image contain text? Answer yes or no."
      +      }
      +    ]
      +  }
      +]
    • changedInput schema / properties / api_key / description
      Previous value: -"API key for the model provider (OpenAI sk-, Anthropic sk-ant-, or Google AIzaSy...)."New value: +"Key for the model's provider."
    • removedInput schema / properties / image_base64 / description
      Removed value: -"Base64-encoded image data (required unless image_url is provided)."
    • changedInput schema / properties / image_mime_type / description
      Previous value: -"MIME type of the image if using image_base64 (default: image/jpeg)."New value: +"For image_base64 (default image/jpeg)."
    • changedInput schema / properties / image_url / description
      Previous value: -"Public URL of the image to evaluate (required unless image_base64 is provided)."New value: +"Public image URL (or image_base64)."
    • removedInput schema / properties / model / description
      Removed value: -"VLM model to use."
    • changedInput schema / properties / model / enum
      Previous value: -[
      -  "gpt-4o",
      -  "gpt-4o-mini",
      -  "claude-3-5-sonnet-20241022",
      -  "claude-3-5-haiku-20241022",
      -  "gemini-3.5-flash",
      -  "gemini-2.5-flash"
      -]New value: +[
      +  "gpt-4o",
      +  "gpt-4o-mini",
      +  "claude-sonnet-4-6",
      +  "claude-haiku-4-5-20251001",
      +  "gemini-3.5-flash",
      +  "gemini-2.5-flash"
      +]
    • removedInput schema / properties / system_prompt / description
      Removed value: -"Optional system prompt sent to the VLM."
    • removedInput schema / properties / test_cases / description
      Removed value: -"Array of test cases to run."
    • removedInput schema / properties / test_cases / items / properties / assertion_type / description
      Removed value: -"Assertion to run on the VLM response. semantic_contains uses TF-IDF cosine similarity ≥ 0.4."
    • changedInput schema / properties / test_cases / items / properties / assertion_value / description
      Previous value: -"Expected value for the assertion (not needed for json_format)."New value: +"Not needed for json_format."
    • removedInput schema / properties / test_cases / items / properties / id / description
      Removed value: -"Optional identifier for this case."
    • removedInput schema / properties / test_cases / items / properties / question / description
      Removed value: -"Question to ask the VLM about the image."
    • changedInput schema / properties / threshold / description
      Previous value: -"Pass rate threshold for overall verdict (default: 80, 0–100)."New value: +"Pass rate for an overall PASS (default 80)."
  3. First observed

TDQS

A3.8/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Beyond the annotations (which already establish a non-destructive, open-world read), the description discloses the credential model (BYOK — your key for the provider), a network constraint (images fetched only from public addresses), the scoring mechanics (semantic_contains = TF-IDF cosine ≥ 0.4), and the verdict ladder PASS/WARNING/FAIL with an N/A case when no assertion can run. That N/A edge case and the auth requirement are genuine behavioral context not derivable from the schema or annotations. The only shortfall is that it repeats return-shape detail that an output schema already covers.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

It is front-loaded with the core action, then layers assertion semantics, return values, and the BYOK/network caveat in a compact three-sentence block with no filler. The middle sentence is dense with parenthetical detail but stays readable, and nothing is stated twice.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with an output schema and meaningful annotations, the description supplies everything an agent needs that structured data does not: credential expectations, image-source restriction, assertion-type semantics, and the threshold-relative verdict. Nothing essential to a correct invocation is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 50% across 8 parameters, so the description has to carry part of the load — and it does explain the trickiest nested parameter by defining each assertion type's meaning, including the 0.4 TF-IDF threshold for semantic_contains. However, `system_prompt`, `image_base64`, `image_mime_type`, the `model` enum choices, and the 10-case cap on `test_cases` get no treatment in the description, and threshold/api_key largely restate what the schema already says.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description gives a specific verb and resource — 'Test a vision model (OpenAI, Anthropic or Gemini) on one image' — and enumerates the assertion types it supports, so the agent knows exactly what class of work this tool performs. It implicitly separates itself from text-prompt siblings by scoping to a single image and named VLM providers, but it never names an alternative tool, so the differentiation stays implicit rather than explicit.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is implied rather than stated: the 'one image' framing and the BYOK requirement tell the agent this is a single-image VLM evaluation, and 'the image is fetched only from a public address' rules out local/private images. There is no explicit when-to-use/when-not-to-use guidance and no named alternative for multi-image or text-only testing, leaving the agent to infer the boundary from sibling names alone.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.