run_vlm_test_suite
Test a vision model (OpenAI, Anthropic or Gemini) on one image: each test case asks a question and asserts on the answer (contains, not_contains, json_format, min_length, max_length, semantic_contains = TF-IDF cosine ≥ 0.4). Returns per-case PASS/FAIL, the pass rate, overall PASS/WARNING/FAIL against threshold (N/A when no assertion could run), and latency. BYOK: your key for the model's provider; the image is fetched only from a public address.
Example tool call
This call uses example arguments supplied in the tool’s input schema.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"model": "gemini-2.5-flash",
"api_key": "<your-api-key>",
"image_url": "https://www.ia-qa.com/media/discover-report-poster.png",
"test_cases": [
{
"question": "Does the image contain text? Answer yes or no.",
"assertion_type": "contains",
"assertion_value": "yes"
}
]
},
"name": "run_vlm_test_suite"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| model | Yes | ||
| api_key | Yes | Key for the model's provider. | |
| image_url | No | Public image URL (or image_base64). | |
| threshold | No | Pass rate for an overall PASS (default 80). | |
| test_cases | Yes | ||
| image_base64 | No | ||
| system_prompt | No | ||
| image_mime_type | No | For image_base64 (default image/jpeg). |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| model | No | ||
| total | No | ||
| passed | No | ||
| overall | No | ||
| results | No | ||
| image_url | No | ||
| pass_rate | No | ||
| threshold | No | ||
| latencyStats | No |