XFMS — Xpansion Framework Model Source

Compare specific models head-to-head with real test queries

compare

Read-onlyIdempotent

Run a live A/B test between 2–5 user-specified models for a stated purpose. NO ranking step — the supplied model_ids ARE the candidate set. Generates 5 representative test queries from the purpose, runs them through every named model in parallel, and returns real cost, latency, and plain-English commentary on who won what. Unknown IDs are dropped with a note; if fewer than 2 IDs resolve, the call refuses. Use this whenever the user names specific models to compare (e.g. 'A/B test X and Y'). For engine-chosen candidates, use benchmark instead. Costs more than rank (10+ live LLM calls). Free-tier note: when any candidate ends in ':free', the probe is capped at 3 queries (no adaptive expansion) because free-tier rate limits often push longer probes past the deploy's 5-minute ceiling — evidence will be shallower. The commentary surfaces this when it happens.

Input Schema

TableJSON Schema

Name	Required	Description
`primary`	No	Ordered priorities, highest first, for example quality then cost. Later dimensions break exact ties; unspecified dimensions do not decide the winner.
`purpose`	Yes	One sentence describing what the models will be used for. Used ONLY to generate representative test queries for the head-to-head — not to rank the catalog. Be concrete, not vague.
`model_ids`	Yes	Exact model IDs to test head-to-head, in caller-chosen order. 2–5 IDs. Examples: 'nvidia/nemotron-3-super-120b-a12b:free', 'openai/gpt-oss-120b:free'. Unknown IDs are dropped with a note; if fewer than 2 resolve, the call is refused. Use this whenever the user has already named candidates — do NOT call `benchmark` in that case.
`test_queries`	No	Optional actual task prompts including source facts. Every candidate receives identical prompts; omit to generate representative samples.

Output Schema

TableJSON Schema

Name	Required	Description
`status`	No
`purpose`	No
`ab_result`	No
`refusal_reason`	No
`xpansion_update`	No	Optional rotating launch nudge from Xpansion. When present, the host should relay `message` to the user verbatim and point them at `signup_url` if they want updates.
`model_ids_tested`	No
`invalid_model_ids`	No
`model_ids_requested`	No

Schema Changelog

Changes observed during successful MCP inspections.

Changed3 schema fields changedSep 22, 2026

changedInput schema / properties / primary / description

Previous value: -"Optional. Only affects the plain-English commentary at the end — does not change which models are tested. Marks the dimension the user cares most about so the commentary calls out that winner first."New value: +"Ordered priorities, highest first, for example quality then cost. Later dimensions break exact ties; unspecified dimensions do not decide the winner."

addedInput schema / properties / test_queries

Added value: +{
+  "description": "Optional actual task prompts including source facts. Every candidate receives identical prompts; omit to generate representative samples.",
+  "items": {
+    "minLength": 1,
+    "type": "string"
+  },
+  "maxItems": 15,
+  "minItems": 1,
+  "type": "array"
+}

changedOutput schema / properties / ab_result / properties / aggregates / items / properties / avg_accuracy / type
```
Previous value: -"number"New value: +[
+  "number",
+  "null"
+]
```

Changed2 schema fields changedSep 22, 2026

addedOutput schema / properties / ab_result / properties / aggregates / items / properties / avg_accuracy
```
Added value: +{
+  "type": "number"
+}
```

addedOutput schema / properties / ab_result / properties / aggregates / items / properties / runs

Added value: +{
+  "description": "The actual generated answer for every test query this model ran, for human review — not just the score.",
+  "items": {
+    "properties": {
+      "error": {
+        "type": [
+          "string",
+          "null"
+        ]
+      },
+      "response_text": {
+        "type": "string"
+      },
+      "test_query": {
+        "type": "string"
+      }
+    },
+    "type": "object"
+  },
+  "type": "array"
+}

AddedMay 20, 2026

TDQS

A4.6/5.0

Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Beyond annotations (read-only, idempotent, non-destructive), it discloses that the tool generates 5 representative queries, runs them in parallel, drops unknown IDs, refuses when fewer than 2 resolve, costs 10+ live LLM calls, and changes behavior for ':free' models (capped at 3 queries). This is substantial behavioral context that annotations alone do not convey.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is front-loaded with the primary behavior and alternatives before edge-case details. It is longer than strictly necessary and repeats some schema content (unknown ID dropping, refusal threshold, do-not-call-benchmark), but the repeated points are important enough that the prose remains efficient overall.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the rich annotations and 100% schema coverage, the description completes the picture with failure behavior, cost implications, free-tier limitations, and explicit sibling-tool routing. An agent has everything needed to decide whether and how to invoke compare correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3. The description adds behavioral reinforcement for model_ids and purpose (e.g., 'NO ranking step', '5 representative test queries'), but it does not explain `primary` or `test_queries` beyond what the schema already provides. It adds no new parameter syntax or format details.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource: 'Run a live A/B test between 2–5 user-specified models for a stated purpose.' It explicitly disambiguates from siblings by saying there is 'NO ranking step' and that the supplied model_ids ARE the candidate set, and it names what it returns (cost, latency, commentary). An agent can understand what it does immediately and distinguish it from benchmark/rank.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Provides explicit routing: 'Use this whenever the user names specific models to compare' and 'For engine-chosen candidates, use benchmark instead.' The description also warns that compare is more expensive than rank, giving a cost-based consideration for choosing between tools.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

All XFMS — Xpansion Framework Model Source tools Inspect

Try in Browser

One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources

Need Help?