addedInput schema / examples
Added value: +[
+ {
+ "api_keys": {
+ "gemini-2.5-flash": "<your-google-key>",
+ "gpt-4o-mini": "<your-openai-key>"
+ },
+ "image_url": "https://www.ia-qa.com/media/discover-report-poster.png",
+ "models": [
+ "gpt-4o-mini",
+ "gemini-2.5-flash"
+ ],
+ "test_cases": [
+ {
+ "question": "Does the image contain text? Answer yes or no."
+ }
+ ]
+ }
+]
changedInput schema / properties / api_keys / description
Previous value: -"Map of model ID → API key. Example: { \"gpt-4o\": \"sk-...\", \"claude-3-5-sonnet-20241022\": \"sk-ant-...\" }"New value: +"Model id → API key."
removedInput schema / properties / image_base64 / description
Removed value: -"Base64-encoded image data (required unless image_url is provided)."
changedInput schema / properties / image_mime_type / description
Previous value: -"MIME type of the image if using image_base64 (default: image/jpeg)."New value: +"For image_base64 (default image/jpeg)."
changedInput schema / properties / image_url / description
Previous value: -"Public URL of the image to evaluate (required unless image_base64 is provided)."New value: +"Public image URL (or image_base64)."
removedInput schema / properties / models / description
Removed value: -"Array of model IDs to compare (runs in parallel)."
changedInput schema / properties / models / items / enum
Previous value: -[
- "gpt-4o",
- "gpt-4o-mini",
- "claude-3-5-sonnet-20241022",
- "claude-3-5-haiku-20241022",
- "gemini-3.5-flash",
- "gemini-2.5-flash"
-]New value: +[
+ "gpt-4o",
+ "gpt-4o-mini",
+ "claude-sonnet-4-6",
+ "claude-haiku-4-5-20251001",
+ "gemini-3.5-flash",
+ "gemini-2.5-flash"
+]
removedInput schema / properties / system_prompt / description
Removed value: -"Optional system prompt sent to every VLM."
removedInput schema / properties / test_cases / description
Removed value: -"Array of test cases to run against every model."
removedInput schema / properties / test_cases / items / properties / assertion_type / description
Removed value: -"Assertion to run on the VLM response."
changedInput schema / properties / test_cases / items / properties / assertion_value / description
Previous value: -"Expected value for the assertion (not needed for json_format)."New value: +"Not needed for json_format."
removedInput schema / properties / test_cases / items / properties / id / description
Removed value: -"Optional identifier for this case."
removedInput schema / properties / test_cases / items / properties / question / description
Removed value: -"Question to ask the VLM about the image."
changedInput schema / properties / threshold / description
Previous value: -"Pass rate threshold for overall verdict (default: 80, 0–100)."New value: +"Pass rate for an overall PASS (default 80)."