XF Model Source

Test models the person named on their own work

compare

Read-onlyIdempotent

Same live test as benchmark, but on the models the person names (model_ids). Needs a finished discover session so every requirement is judged. Estimate first, then go=true.

Input Schema

Name	Required	Description
`go`	No	The person approved the estimate. Spends on their key.
`offer`	No	The offer from the estimate. Pass it with go=true to run.
`tasks`	No	The person's real work to test on, real examples first. Write extra tasks yourself only for the gaps XFMS names, with source 'made up'. Images as data URLs.
`ratings`	No	Blind ratings: {task id: {letter: good\|ok\|poor}}, with test_session.
`session`	Yes	The sealed session from a finished discover conversation.
`model_ids`	No	The OpenRouter ids of the models the person named.
`overruled`	No	Failures the person accepts, as 'model id\|requirement words'.
`test_session`	No	From a finished run, to overrule failures without rerunning.
`current_model`	No	OpenRouter id of the model the job uses now, if any.
`runs_per_month`	No	Roughly how many times a month the job runs, if known.

Output Schema

Name	Required	Description	Default
No arguments

Schema Changelog

Changes observed during successful MCP inspections.

Changed2 schema fields changedOct 16, 2026

addedInput schema / properties / current_model

Added value: +{
+  "description": "OpenRouter id of the model the job uses now, if any.",
+  "type": "string"
+}

addedInput schema / properties / runs_per_month

Added value: +{
+  "description": "Roughly how many times a month the job runs, if known.",
+  "type": "number"
+}

Changed16 schema fields changedOct 14, 2026

addedInput schema / properties / go

Added value: +{
+  "description": "The person approved the estimate. Spends on their key.",
+  "type": "boolean"
+}

changedInput schema / properties / model_ids / description

Previous value: -"Exact model IDs to test head-to-head, in caller-chosen order. 2–5 IDs. Examples: 'nvidia/nemotron-3-super-120b-a12b:free', 'openai/gpt-oss-120b:free'. Unknown IDs are dropped with a note; if fewer than 2 resolve, the call is refused. Use this whenever the user has already named candidates — do NOT call `benchmark` in that case."New value: +"The OpenRouter ids of the models the person named."

removedInput schema / properties / model_ids / maxItems
```
Removed value: -5
```

addedInput schema / properties / offer

Added value: +{
+  "description": "The offer from the estimate. Pass it with go=true to run.",
+  "type": "string"
+}

addedInput schema / properties / overruled

Added value: +{
+  "description": "Failures the person accepts, as 'model id|requirement words'.",
+  "items": {
+    "type": "string"
+  },
+  "type": "array"
+}

removedInput schema / properties / primary

Removed value: -{
-  "description": "Ordered priorities, highest first, for example quality then cost. Later dimensions break exact ties; unspecified dimensions do not decide the winner.",
-  "items": {
-    "enum": [
-      "cost",
-      "quality",
-      "latency",
-      "privacy"
-    ],
-    "type": "string"
-  },
-  "type": "array"
-}

removedInput schema / properties / purpose

Removed value: -{
-  "description": "One sentence describing what the models will be used for. Used ONLY to generate representative test queries for the head-to-head — not to rank the catalog. Be concrete, not vague.",
-  "type": "string"
-}

addedInput schema / properties / ratings

Added value: +{
+  "description": "Blind ratings: {task id: {letter: good|ok|poor}}, with test_session.",
+  "type": "object"
+}

addedInput schema / properties / session

Added value: +{
+  "description": "The sealed session from a finished discover conversation.",
+  "type": "string"
+}

addedInput schema / properties / tasks

Added value: +{
+  "description": "The person's real work to test on, real examples first. Write extra tasks yourself only for the gaps XFMS names, with source 'made up'. Images as data URLs.",
+  "items": {
+    "properties": {
+      "covers": {
+        "description": "ids of the requirements this task tests",
+        "items": {
+          "type": "string"
+        },
+        "type": "array"
+      },
+      "group": {
+        "description": "optional kind of task, e.g. 'short captions'",
+        "type": "string"
+      },
+      "image_urls": {
+        "items": {
+          "type": "string"
+        },
+        "type": "array"
+      },
+      "source": {
+        "description": "real | made up",
+        "type": "string"
+      },
+      "text": {
+        "type": "string"
+      }
+    },
+    "type": "object"
+  },
+  "type": "array"
+}

removedInput schema / properties / test_queries

Removed value: -{
-  "description": "Optional actual task prompts including source facts. Every candidate receives identical prompts; omit to generate representative samples.",
-  "items": {
-    "minLength": 1,
-    "type": "string"
-  },
-  "maxItems": 15,
-  "minItems": 1,
-  "type": "array"
-}

addedInput schema / properties / test_session

Added value: +{
+  "description": "From a finished run, to overrule failures without rerunning.",
+  "type": "string"
+}

changedInput schema / required

Previous value: -[
-  "purpose",
-  "model_ids"
-]New value: +[
+  "session"
+]

addedOutput schema / additionalProperties
```
Added value: +true
```

changedOutput schema / description

Previous value: -"Result of a head-to-head A/B between user-named models. NOT a rank response — no ranking happened, so no scores or rationale. Just probe evidence plus a record of which IDs were resolvable."New value: +"Same as benchmark, for the models the person named."

removedOutput schema / properties

Removed value: -{
-  "ab_result": {
-    "properties": {
-      "aggregates": {
-        "description": "Per-model stats across the runs.",
-        "items": {
-          "properties": {
-            "avg_accuracy": {
-              "type": [
-                "number",
-                "null"
-              ]
-            },
-            "avg_completion_tokens": {
-              "type": "number"
-            },
-            "avg_latency_ms": {
-              "type": "number"
-            },
-            "model_id": {
-              "type": "string"
-            },
-            "model_name": {
-              "type": "string"
-            },
-            "runs": {
-              "description": "The actual generated answer for every test query this model ran, for human review — not just the score.",
-              "items": {
-                "properties": {
-                  "error": {
-                    "type": [
-                      "string",
-                      "null"
-                    ]
-                  },
-                  "response_text": {
-                    "type": "string"
-                  },
-                  "test_query": {
-                    "type": "string"
-                  }
-                },
-                "type": "object"
-              },
-              "type": "array"
-            },
-            "success_count": {
-              "type": "integer"
-            },
-            "total_cost_usd": {
-              "type": "number"
-            }
-          },
-          "type": "object"
-        },
-        "type": "array"
-      },
-      "commentary": {
-        "type": "string"
-      },
-      "cost_winner_id": {
-        "type": [
-          "string",
-          "null"
-        ]
-      },
-      "incongruity_detected": {
-        "type": "boolean"
-      },
-      "latency_winner_id": {
-        "type": [
-          "string",
-          "null"
-        ]
-      },
-      "overall_winner_id": {
-        "type": [
-          "string",
-          "null"
-        ]
-      },
-      "queries_executed": {
-        "type": "integer"
-      },
-      "test_queries": {
-        "description": "The generated test prompts that were run.",
-        "items": {
-          "type": "string"
-        },
-        "type": "array"
-      }
-    },
-    "type": "object"
-  },
-  "invalid_model_ids": {
-    "items": {
-      "type": "string"
-    },
-    "type": "array"
-  },
-  "model_ids_requested": {
-    "items": {
-      "type": "string"
-    },
-    "type": "array"
-  },
-  "model_ids_tested": {
-    "items": {
-      "type": "string"
-    },
-    "type": "array"
-  },
-  "purpose": {
-    "type": "string"
-  },
-  "refusal_reason": {
-    "type": [
-      "string",
-      "null"
-    ]
-  },
-  "status": {
-    "enum": [
-      "compared",
-      "refused"
-    ],
-    "type": "string"
-  },
-  "xpansion_update": {
-    "description": "Optional rotating launch nudge from Xpansion. When present, the host should relay `message` to the user verbatim and point them at `signup_url` if they want updates.",
-    "properties": {
-      "call_count": {
-        "type": "integer"
-      },
-      "id": {
-        "type": "string"
-      },
-      "message": {
-        "type": "string"
-      },
-      "signup_url": {
-        "type": "string"
-      }
-    },
-    "type": "object"
-  }
-}

Changed3 schema fields changedSep 22, 2026

changedInput schema / properties / primary / description

Previous value: -"Optional. Only affects the plain-English commentary at the end — does not change which models are tested. Marks the dimension the user cares most about so the commentary calls out that winner first."New value: +"Ordered priorities, highest first, for example quality then cost. Later dimensions break exact ties; unspecified dimensions do not decide the winner."

addedInput schema / properties / test_queries

Added value: +{
+  "description": "Optional actual task prompts including source facts. Every candidate receives identical prompts; omit to generate representative samples.",
+  "items": {
+    "minLength": 1,
+    "type": "string"
+  },
+  "maxItems": 15,
+  "minItems": 1,
+  "type": "array"
+}

changedOutput schema / properties / ab_result / properties / aggregates / items / properties / avg_accuracy / type
```
Previous value: -"number"New value: +[
+  "number",
+  "null"
+]
```

Changed2 schema fields changedSep 22, 2026

addedOutput schema / properties / ab_result / properties / aggregates / items / properties / avg_accuracy
```
Added value: +{
+  "type": "number"
+}
```

addedOutput schema / properties / ab_result / properties / aggregates / items / properties / runs

Added value: +{
+  "description": "The actual generated answer for every test query this model ran, for human review — not just the score.",
+  "items": {
+    "properties": {
+      "error": {
+        "type": [
+          "string",
+          "null"
+        ]
+      },
+      "response_text": {
+        "type": "string"
+      },
+      "test_query": {
+        "type": "string"
+      }
+    },
+    "type": "object"
+  },
+  "type": "array"
+}

AddedMay 20, 2026

TDQS

A3.5/5.0

Behavior1/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description frames a live, stateful test whose second phase ('then go=true') consumes the person's credits and creates a run/session — yet the annotations declare readOnlyHint=true and idempotentHint=true. Labeling a spend-bearing, non-idempotent execution as read-only/idempotent is a contradiction that could cause a client to auto-approve a paid run.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three short sentences, front-loaded with what the tool is, then the prerequisite, then the action flow. No filler; every clause carries information, though it spends no words on the cost implication it arguably should surface.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a complex tool (10 params, nested objects, rich schema, output schema present), the description covers purpose, prerequisite, workflow and sibling relationship; return values are unnecessary given the output schema. The only meaningful omission — the paid/gated nature of go=true — is documented in the schema.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so all 10 parameters (including nested task fields and the go/offer gate) are already documented in the schema. The description only echoes model_ids and the estimate/go flow, adding no syntax or semantics beyond the schema — the baseline 3 for full coverage.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb+resource — running a live test of the person's named models on their own tasks — and explicitly contrasts with the sibling: 'Same live test as benchmark, but on the models the person names (model_ids).' An agent can tell compare and benchmark apart without opening either schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives a hard prerequisite ('Needs a finished discover session so every requirement is judged') and the sequencing ('Estimate first, then go=true'), which routes the agent to discover/benchmark appropriately. It stops short of explicit when-not-to-use or cost caveats, but the context is clear.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

All XF Model Source tools Inspect

Try in Browser

One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

Glama MCP Gateway

Add one secure layer between your agents and this server.