Skip to main content
Glama
Sign In

XF Model Source

by dev.xpansion
OverviewDiscussionsAdmin

Test the shortlist on the person's own work

benchmark
Read-onlyIdempotent

The live test: the 5 proven models and 1 promising newcomer from discovery do the person's real tasks side by side, every requirement is judged, and the results pick the winner. First call (session + tasks) returns the estimate and an offer — nothing is spent. Show the person the cost; only when they agree, call again with offer and go=true. Runs on the person's OpenRouter key (X-OpenRouter-Key header). Save the returned report as a local file.

Input Schema

NameRequiredDescriptionDefault
goNoThe person approved the estimate. Spends on their key.
offerNoThe offer from the estimate. Pass it with go=true to run.
tasksNoThe person's real work to test on, real examples first. Write extra tasks yourself only for the gaps XFMS names, with source 'made up'. Images as data URLs.
ratingsNoBlind ratings: {task id: {letter: good|ok|poor}}, with test_session.
sessionYesThe sealed session from a finished discover conversation.
overruledNoFailures the person accepts, as 'model id|requirement words'.
jobs_wantedNoHow many AI jobs the person would want watched, if they said; sizes the Model Watch plan.
test_sessionNoFrom a finished run, to overrule failures without rerunning.
current_modelNoOpenRouter id of the model the job uses now, if any.
runs_per_monthNoRoughly how many times a month the job runs, if known.

Output Schema

NameRequiredDescriptionDefault

No arguments

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed1 schema field changed
    • addedInput schema / properties / jobs_wanted
      Added value: +{
+  "description": "How many AI jobs the person would want watched, if they said; sizes the Model Watch plan.",
+  "type": "integer"
+}
  2. Changed2 schema fields changed
    • addedInput schema / properties / current_model
      Added value: +{
+  "description": "OpenRouter id of the model the job uses now, if any.",
+  "type": "string"
+}
    • addedInput schema / properties / runs_per_month
      Added value: +{
+  "description": "Roughly how many times a month the job runs, if known.",
+  "type": "number"
+}
  3. Changed16 schema fields changed
    • removedInput schema / properties / capabilities
      Removed value: -{
-  "description": "Required capabilities the model MUST support. Models missing any listed capability are filtered out before ranking. 'vision' = image input, 'audio_in' = audio input, 'tool_use' = function calling, 'structured_outputs' = JSON schema-constrained output. Omit when the task is plain text with no tool use.",
-  "items": {
-    "enum": [
-      "vision",
-      "audio_in",
-      "tool_use",
-      "structured_outputs"
-    ],
-    "type": "string"
-  },
-  "type": "array"
-}
    • addedInput schema / properties / go
      Added value: +{
+  "description": "The person approved the estimate. Spends on their key.",
+  "type": "boolean"
+}
    • addedInput schema / properties / offer
      Added value: +{
+  "description": "The offer from the estimate. Pass it with go=true to run.",
+  "type": "string"
+}
    • addedInput schema / properties / overruled
      Added value: +{
+  "description": "Failures the person accepts, as 'model id|requirement words'.",
+  "items": {
+    "type": "string"
+  },
+  "type": "array"
+}
    • removedInput schema / properties / primary
      Removed value: -{
-  "description": "Ordered priorities, highest first, for example quality then cost. Later dimensions break exact ties; unspecified dimensions do not decide the winner.",
-  "items": {
-    "enum": [
-      "cost",
-      "quality",
-      "latency",
-      "privacy"
-    ],
-    "type": "string"
-  },
-  "type": "array"
-}
    • removedInput schema / properties / purpose
      Removed value: -{
-  "description": "One sentence describing what the model will be used for. The benchmark generates representative test queries from this — so be concrete, not vague.",
-  "type": "string"
-}
    • addedInput schema / properties / ratings
      Added value: +{
+  "description": "Blind ratings: {task id: {letter: good|ok|poor}}, with test_session.",
+  "type": "object"
+}
    • addedInput schema / properties / session
      Added value: +{
+  "description": "The sealed session from a finished discover conversation.",
+  "type": "string"
+}
    • addedInput schema / properties / tasks
      Added value: +{
+  "description": "The person's real work to test on, real examples first. Write extra tasks yourself only for the gaps XFMS names, with source 'made up'. Images as data URLs.",
+  "items": {
+    "properties": {
+      "covers": {
+        "description": "ids of the requirements this task tests",
+        "items": {
+          "type": "string"
+        },
+        "type": "array"
+      },
+      "group": {
+        "description": "optional kind of task, e.g. 'short captions'",
+        "type": "string"
+      },
+      "image_urls": {
+        "items": {
+          "type": "string"
+        },
+        "type": "array"
+      },
+      "source": {
+        "description": "real | made up",
+        "type": "string"
+      },
+      "text": {
+        "type": "string"
+      }
+    },
+    "type": "object"
+  },
+  "type": "array"
+}
    • removedInput schema / properties / test_queries
      Removed value: -{
-  "description": "Optional actual task prompts including source facts. Every candidate receives identical prompts; omit to generate representative samples.",
-  "items": {
-    "minLength": 1,
-    "type": "string"
-  },
-  "maxItems": 15,
-  "minItems": 1,
-  "type": "array"
-}
    • addedInput schema / properties / test_session
      Added value: +{
+  "description": "From a finished run, to overrule failures without rerunning.",
+  "type": "string"
+}
    • removedInput schema / properties / top_n
      Removed value: -{
-  "default": 5,
-  "description": "How many models to return in the ranked list. Defaults to 5. Use 1 if you only want the single best pick; use 10+ if you want to see deeper alternatives.",
-  "maximum": 25,
-  "minimum": 1,
-  "type": "integer"
-}
    • changedInput schema / required
      Previous value: -[
-  "purpose"
-]New value: +[
+  "session"
+]
    • addedOutput schema / additionalProperties
      Added value: +true
    • changedOutput schema / description
      Previous value: -"Rank response with ab_result populated — same shape as `rank` plus live performance data from the probe runs."New value: +"Without go: the estimate and offer. With go: the outcome, every model on every requirement, the report."
    • removedOutput schema / properties
      Removed value: -{
-  "ab_result": {
-    "properties": {
-      "aggregates": {
-        "description": "Per-model stats across the runs.",
-        "items": {
-          "properties": {
-            "avg_accuracy": {
-              "type": [
-                "number",
-                "null"
-              ]
-            },
-            "avg_completion_tokens": {
-              "type": "number"
-            },
-            "avg_latency_ms": {
-              "type": "number"
-            },
-            "model_id": {
-              "type": "string"
-            },
-            "model_name": {
-              "type": "string"
-            },
-            "runs": {
-              "description": "The actual generated answer for every test query this model ran, for human review — not just the score.",
-              "items": {
-                "properties": {
-                  "error": {
-                    "type": [
-                      "string",
-                      "null"
-                    ]
-                  },
-                  "response_text": {
-                    "type": "string"
-                  },
-                  "test_query": {
-                    "type": "string"
-                  }
-                },
-                "type": "object"
-              },
-              "type": "array"
-            },
-            "success_count": {
-              "type": "integer"
-            },
-            "total_cost_usd": {
-              "type": "number"
-            }
-          },
-          "type": "object"
-        },
-        "type": "array"
-      },
-      "commentary": {
-        "type": "string"
-      },
-      "cost_winner_id": {
-        "type": [
-          "string",
-          "null"
-        ]
-      },
-      "incongruity_detected": {
-        "type": "boolean"
-      },
-      "latency_winner_id": {
-        "type": [
-          "string",
-          "null"
-        ]
-      },
-      "overall_winner_id": {
-        "type": [
-          "string",
-          "null"
-        ]
-      },
-      "queries_executed": {
-        "type": "integer"
-      },
-      "test_queries": {
-        "description": "The generated test prompts that were run.",
-        "items": {
-          "type": "string"
-        },
-        "type": "array"
-      }
-    },
-    "type": "object"
-  },
-  "catalog_size": {
-    "type": "integer"
-  },
-  "filtered_out": {
-    "type": "integer"
-  },
-  "frontier_filtered_out": {
-    "description": "How many models in models[] are neither recent nor top-tier on quality (informational — none are removed from the list, this just means they weren't eligible to be the recommendation).",
-    "type": "integer"
-  },
-  "models": {
-    "description": "Ranked shortlist of models, highest score first.",
-    "items": {
-      "properties": {
-        "model_id": {
-          "type": "string"
-        },
-        "name": {
-          "type": "string"
-        },
-        "provider": {
-          "type": [
-            "string",
-            "null"
-          ]
-        },
-        "rationale": {
-          "type": "string"
-        },
-        "total_score": {
-          "type": "number"
-        }
-      },
-      "type": "object"
-    },
-    "type": "array"
-  },
-  "quality_floor_reason": {
-    "type": [
-      "string",
-      "null"
-    ]
-  },
-  "status": {
-    "description": "'ranked' (normal), 'low_confidence' (capability requirements were relaxed to find any match), or 'quality_floor_refused' (candidates were found but the best one scored too low to recommend — see quality_floor_reason; models[] still lists what was considered).",
-    "type": "string"
-  },
-  "xpansion_update": {
-    "description": "Optional rotating launch nudge from Xpansion. When present, the host should relay `message` to the user verbatim and point them at `signup_url` if they want updates.",
-    "properties": {
-      "call_count": {
-        "type": "integer"
-      },
-      "id": {
-        "type": "string"
-      },
-      "message": {
-        "type": "string"
-      },
-      "signup_url": {
-        "type": "string"
-      }
-    },
-    "type": "object"
-  }
-}
  4. Changed5 schema fields changed
    • addedInput schema / properties / capabilities
      Added value: +{
+  "description": "Required capabilities the model MUST support. Models missing any listed capability are filtered out before ranking. 'vision' = image input, 'audio_in' = audio input, 'tool_use' = function calling, 'structured_outputs' = JSON schema-constrained output. Omit when the task is plain text with no tool use.",
+  "items": {
+    "enum": [
+      "vision",
+      "audio_in",
+      "tool_use",
+      "structured_outputs"
+    ],
+    "type": "string"
+  },
+  "type": "array"
+}
    • addedInput schema / properties / primary
      Added value: +{
+  "description": "Ordered priorities, highest first, for example quality then cost. Later dimensions break exact ties; unspecified dimensions do not decide the winner.",
+  "items": {
+    "enum": [
+      "cost",
+      "quality",
+      "latency",
+      "privacy"
+    ],
+    "type": "string"
+  },
+  "type": "array"
+}
    • addedInput schema / properties / test_queries
      Added value: +{
+  "description": "Optional actual task prompts including source facts. Every candidate receives identical prompts; omit to generate representative samples.",
+  "items": {
+    "minLength": 1,
+    "type": "string"
+  },
+  "maxItems": 15,
+  "minItems": 1,
+  "type": "array"
+}
    • addedInput schema / properties / top_n
      Added value: +{
+  "default": 5,
+  "description": "How many models to return in the ranked list. Defaults to 5. Use 1 if you only want the single best pick; use 10+ if you want to see deeper alternatives.",
+  "maximum": 25,
+  "minimum": 1,
+  "type": "integer"
+}
    • changedOutput schema / properties / ab_result / properties / aggregates / items / properties / avg_accuracy / type
      Previous value: -"number"New value: +[
+  "number",
+  "null"
+]
  5. Changed5 schema fields changed
    • addedOutput schema / properties / ab_result / properties / aggregates / items / properties / avg_accuracy
      Added value: +{
+  "type": "number"
+}
    • addedOutput schema / properties / ab_result / properties / aggregates / items / properties / runs
      Added value: +{
+  "description": "The actual generated answer for every test query this model ran, for human review — not just the score.",
+  "items": {
+    "properties": {
+      "error": {
+        "type": [
+          "string",
+          "null"
+        ]
+      },
+      "response_text": {
+        "type": "string"
+      },
+      "test_query": {
+        "type": "string"
+      }
+    },
+    "type": "object"
+  },
+  "type": "array"
+}
    • addedOutput schema / properties / frontier_filtered_out
      Added value: +{
+  "description": "How many models in models[] are neither recent nor top-tier on quality (informational — none are removed from the list, this just means they weren't eligible to be the recommendation).",
+  "type": "integer"
+}
    • addedOutput schema / properties / quality_floor_reason
      Added value: +{
+  "type": [
+    "string",
+    "null"
+  ]
+}
    • addedOutput schema / properties / status / description
      Added value: +"'ranked' (normal), 'low_confidence' (capability requirements were relaxed to find any match), or 'quality_floor_refused' (candidates were found but the best one scored too low to recommend — see quality_floor_reason; models[] still lists what was considered)."
  6. Changed1 schema field changed
    • changedOutput schema / properties / models / items / properties / provider / type
      Previous value: -"string"New value: +[
+  "string",
+  "null"
+]
  7. Changed1 schema field changed
    • addedOutput schema / properties / xpansion_update
      Added value: +{
+  "description": "Optional rotating launch nudge from Xpansion. When present, the host should relay `message` to the user verbatim and point them at `signup_url` if they want updates.",
+  "properties": {
+    "call_count": {
+      "type": "integer"
+    },
+    "id": {
+      "type": "string"
+    },
+    "message": {
+      "type": "string"
+    },
+    "signup_url": {
+      "type": "string"
+    }
+  },
+  "type": "object"
+}
  8. Added

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With annotations already marking it read-only, non-destructive, and open-world, the description adds important behavioral context: no spend on the first call, approval gating before spending, use of the X-OpenRouter-Key header, and saving the returned report. It does not clarify duplicate-spend/idempotency behavior when go=true is repeated or rate limits, so it is not fully complete.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Four tightly packed sentences front-load the live-test purpose, then the two-step spending protocol and key/report details. Every sentence carries operational information, with no filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists and input schema descriptions are complete, so the description need not explain return values. It covers the crucial two-phase invocation, cost approval, authentication header, and report handling, making it complete enough for an agent to call safely.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, and the schema already documents go, offer, tasks, ratings, session, overruled, and the other parameters. The description usefully links session/tasks to the estimate call and offer/go to the paid call, but adds little parameter-level meaning beyond the existing schema descriptions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description names a concrete action: run a live side-by-side test of the shortlisted models on the person's real tasks, judge every requirement, and pick a winner. The phrase 'from discovery' and '5 proven models and 1 promising newcomer' ties it to the pipeline and separates it from sibling discovery/pick/rank tools.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It spells out a two-call protocol: first call with session and tasks returns an estimate/offer and spends nothing; after showing the cost and getting agreement, call again with offer and go=true. This is clear when to invoke and when not to spend, though it does not discuss sibling alternatives such as compare or rank.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

All XF Model Source toolsInspect
Try in Browser
One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources