XFMS — Xpansion Framework Model Source

Benchmark the engine's top picks with real test queries

benchmark

Read-onlyIdempotent

Run a live A/B test against the engine's TOP 3 PICKS for a stated purpose — the engine chooses the candidates from the full catalog. Generates 5 representative test queries (auto-expands to 10 or 15 if results are too close to call), runs them through the picked models in parallel, and returns real cost, latency, and plain-English commentary on who won what. Use AFTER pick or rank when the user wants the engine's own picks stress-tested with live data. DO NOT use this when the user has already named specific candidate models — the engine will ignore the names and test its own picks. Use compare instead in that case. Costs more than rank (15+ live LLM calls).

Input Schema

TableJSON Schema

Name	Required	Description
`top_n`	No	How many models to return in the ranked list. Defaults to 5. Use 1 if you only want the single best pick; use 10+ if you want to see deeper alternatives.
`primary`	No	Ordered priorities, highest first, for example quality then cost. Later dimensions break exact ties; unspecified dimensions do not decide the winner.
`purpose`	Yes	One sentence describing what the model will be used for. The benchmark generates representative test queries from this — so be concrete, not vague.
`capabilities`	No	Required capabilities the model MUST support. Models missing any listed capability are filtered out before ranking. 'vision' = image input, 'audio_in' = audio input, 'tool_use' = function calling, 'structured_outputs' = JSON schema-constrained output. Omit when the task is plain text with no tool use.
`test_queries`	No	Optional actual task prompts including source facts. Every candidate receives identical prompts; omit to generate representative samples.

Output Schema

TableJSON Schema

Name	Required	Description
`models`	No	Ranked shortlist of models, highest score first.
`status`	No	'ranked' (normal), 'low_confidence' (capability requirements were relaxed to find any match), or 'quality_floor_refused' (candidates were found but the best one scored too low to recommend — see quality_floor_reason; models[] still lists what was considered).
`ab_result`	No
`catalog_size`	No
`filtered_out`	No
`xpansion_update`	No	Optional rotating launch nudge from Xpansion. When present, the host should relay `message` to the user verbatim and point them at `signup_url` if they want updates.
`quality_floor_reason`	No
`frontier_filtered_out`	No	How many models in models[] are neither recent nor top-tier on quality (informational — none are removed from the list, this just means they weren't eligible to be the recommendation).

Schema Changelog

Changes observed during successful MCP inspections.

Changed5 schema fields changedSep 22, 2026

addedInput schema / properties / capabilities

Added value: +{
+  "description": "Required capabilities the model MUST support. Models missing any listed capability are filtered out before ranking. 'vision' = image input, 'audio_in' = audio input, 'tool_use' = function calling, 'structured_outputs' = JSON schema-constrained output. Omit when the task is plain text with no tool use.",
+  "items": {
+    "enum": [
+      "vision",
+      "audio_in",
+      "tool_use",
+      "structured_outputs"
+    ],
+    "type": "string"
+  },
+  "type": "array"
+}

addedInput schema / properties / primary

Added value: +{
+  "description": "Ordered priorities, highest first, for example quality then cost. Later dimensions break exact ties; unspecified dimensions do not decide the winner.",
+  "items": {
+    "enum": [
+      "cost",
+      "quality",
+      "latency",
+      "privacy"
+    ],
+    "type": "string"
+  },
+  "type": "array"
+}

addedInput schema / properties / test_queries

Added value: +{
+  "description": "Optional actual task prompts including source facts. Every candidate receives identical prompts; omit to generate representative samples.",
+  "items": {
+    "minLength": 1,
+    "type": "string"
+  },
+  "maxItems": 15,
+  "minItems": 1,
+  "type": "array"
+}

addedInput schema / properties / top_n

Added value: +{
+  "default": 5,
+  "description": "How many models to return in the ranked list. Defaults to 5. Use 1 if you only want the single best pick; use 10+ if you want to see deeper alternatives.",
+  "maximum": 25,
+  "minimum": 1,
+  "type": "integer"
+}

changedOutput schema / properties / ab_result / properties / aggregates / items / properties / avg_accuracy / type
```
Previous value: -"number"New value: +[
+  "number",
+  "null"
+]
```

Changed5 schema fields changedSep 22, 2026

addedOutput schema / properties / ab_result / properties / aggregates / items / properties / avg_accuracy
```
Added value: +{
+  "type": "number"
+}
```

addedOutput schema / properties / ab_result / properties / aggregates / items / properties / runs

Added value: +{
+  "description": "The actual generated answer for every test query this model ran, for human review — not just the score.",
+  "items": {
+    "properties": {
+      "error": {
+        "type": [
+          "string",
+          "null"
+        ]
+      },
+      "response_text": {
+        "type": "string"
+      },
+      "test_query": {
+        "type": "string"
+      }
+    },
+    "type": "object"
+  },
+  "type": "array"
+}

addedOutput schema / properties / frontier_filtered_out

Added value: +{
+  "description": "How many models in models[] are neither recent nor top-tier on quality (informational — none are removed from the list, this just means they weren't eligible to be the recommendation).",
+  "type": "integer"
+}

addedOutput schema / properties / quality_floor_reason

Added value: +{
+  "type": [
+    "string",
+    "null"
+  ]
+}

addedOutput schema / properties / status / description

Added value: +"'ranked' (normal), 'low_confidence' (capability requirements were relaxed to find any match), or 'quality_floor_refused' (candidates were found but the best one scored too low to recommend — see quality_floor_reason; models[] still lists what was considered)."

Changed1 schema field changedJun 9, 2026
- changedOutput schema / properties / models / items / properties / provider / type
```
Previous value: -"string"New value: +[
+  "string",
+  "null"
+]
```

Changed1 schema field changedMay 19, 2026

addedOutput schema / properties / xpansion_update

Added value: +{
+  "description": "Optional rotating launch nudge from Xpansion. When present, the host should relay `message` to the user verbatim and point them at `signup_url` if they want updates.",
+  "properties": {
+    "call_count": {
+      "type": "integer"
+    },
+    "id": {
+      "type": "string"
+    },
+    "message": {
+      "type": "string"
+    },
+    "signup_url": {
+      "type": "string"
+    }
+  },
+  "type": "object"
+}

AddedMay 19, 2026

TDQS

A4.7/5.0

Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The annotations already declare readOnlyHint/openWorldHint/idempotentHint, and the description adds substantial behavioral context BEYOND those hints. It discloses that the engine silently ignores user-named candidate models, that test queries auto-expand from 5 to 10 or 15 when results are close, that models run in parallel, and that the operation costs 15+ live LLM calls. This is exactly the kind of behavioral disclosure that helps an agent invoke it safely.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is about five sentences but packs only high-value information: main behavior, auto-expansion behavior, return values, when-to-use, when-not-to-use, and cost. No filler or repetition of schema details. The usage guidance is front-loaded after the core behavior.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a complex tool with 100% schema coverage and an output schema, the description is complete: it covers candidate selection, dynamic test query generation, return payload (cost, latency, commentary), and placement among siblings. An agent has everything needed to decide whether to call it and what to expect.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the structured schema already documents all five parameters. The description adds some behavioral context (e.g., that purpose drives generated queries and that test_queries can be supplied) but does not add syntax/type/enum meaning beyond the schema. Per the calibration baseline, 3 is appropriate when the schema does the heavy lifting.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource: 'Run a live A/B test against the engine's TOP 3 PICKS for a stated purpose.' It also names the key output (cost, latency, commentary) and distinguishes from siblings by naming compare/pick/rank placement. An agent can tell it apart from compare without opening the schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly says 'Use AFTER `pick` or `rank` when the user wants the engine's own picks stress-tested with live data.' It also gives a negative rule: 'DO NOT use this when the user has already named specific candidate models — the engine will ignore the names and test its own picks. Use `compare` instead in that case.' It even adds cost guidance ('Costs more than `rank`'). Nothing is left to inference.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

All XFMS — Xpansion Framework Model Source tools Inspect

Try in Browser

One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources

Need Help?